Telemetry Governor

The Telemetry Governor is CollectorCtrl's centralized data plane profiler. It enables you to tap a configurable percentage of live telemetry from any connected agent, inspect traffic patterns, and run What-If cost simulations to predict pipeline savings — all without touching production telemetry paths.

Non-Blocking Telemetry Tapping

Tapping operates exclusively on cloned data streams. Production telemetry pipelines remain completely isolated and unaffected.

1. How Agent Tapping Works

When a tap is activated, the Supervisor agent clones a percentage of live signals and forwards them over OTLP/HTTP to the Governor receiver:

1. Open Telemetry Governor

Navigate to Telemetry Governor in the sidebar.

2. Toggle Tap Target

Select target agents and click Toggle Tap (POST /api/telemetry_governor/taps/toggle).

3. Set Sampling Rate

Configure the sampling rate (e.g. 5.0%). The agent clones that percentage of logs, metrics, and spans for profiling.

2. Ingestion Storage Architecture

Tapped telemetry is stored in either SQLite or ClickHouse:

Storage Backends

Property	SQLite (Default)	ClickHouse (Enterprise)
Target Scale	Trials & fleets < 100 nodes	Production fleets > 1,000 nodes
Setup	Zero-config built-in database	Decentralized ClickHouse cluster
Performance	Basic local query engine	Sub-second queries over billions of rows
Retention	Rolling 1-hour window	Rolling 1-hour window

ClickHouse Configuration

.env / Server Environment Variables

COLLECTORCTRL_CLICKHOUSE_ADDR=your-clickhouse-host:9000
COLLECTORCTRL_CLICKHOUSE_DB=telemetry_governor
COLLECTORCTRL_CLICKHOUSE_USER=default
COLLECTORCTRL_CLICKHOUSE_PASSWORD=your_secure_password

3. Deploying the Governor Engine

1. Deploy Docker Engine

Log into your target Linux VM and run the automated installer:

$mkdir telemetry-governor && cd telemetry-governor
$curl -fsSL https://raw.githubusercontent.com/CollectorCtrl/CollectorCtrl/main/deploy/telemetry-governor/linux/install.sh -o install.sh
$chmod +x install.sh
$sudo ./install.sh

2. Configure Network Ingress

Allow inbound ports on the Governor host:

Port 9000 (TCP): ClickHouse native TCP port for server queries.
Ports 4317 / 4318 (TCP): OTel Collector gRPC / HTTP ingestion endpoints.

3. Connect Management Server

Configure the server environment variables and restart the server service:

PS >Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\CollectorCtrl" -Name Environment -Value "COLLECTORCTRL_CLICKHOUSE_ADDR=YOUR_VM_IP:9000" -Type MultiString
PS >Restart-Service CollectorCtrl

4. What-If Cost Simulation Engine

The simulation engine evaluates filter expressions over tapped telemetry, predicting exactly how much data (and SaaS ingestion cost) you would eliminate by applying OTTL filters before data leaves your network boundary.

Executing a Simulation

Submit a POST request to /api/telemetry_governor/simulate:

{
  "signal": "logs",
  "condition": "severity == \"info\" and body contains \"healthcheck\""
}

Simulation Output Response

{
  "total_records": 48230,
  "matched_records": 35682,
  "saved_percentage": 74
}

Cost Optimization Result

Applying this OTTL filter drops 74% of log volume on the edge before it reaches your SaaS observability vendor — translating directly into reduced ingestion and egress bills.

5. Expression Syntax Reference

Operator	Example Expression	Description
`==`	`severity == "error"`	Exact equality match
`!=`	`agent_id != "web-prod-01"`	Non-equality match
`contains`	`body contains "healthcheck"`	Substring match
`>` / `<`	`duration_ms > 1000`	Numerical comparison
`and`	`severity == "info" and body contains "ping"`	Logical AND condition

6. Rolling Window Data Pruning

To prevent storage bloat, the background pruner runs every 5 minutes:

Automatically purges all records older than 1 hour across logs, metrics, and spans.
Operates transparently on both SQLite and ClickHouse backends.