Fleet Orchestration & Policy Management

Orchestrating telemetry pipelines across thousands of enterprise servers requires a policy-driven control engine. CollectorCtrl replaces manual SSH edits with Fleet Policies, dynamic target selectors, and progressive rollout workflows.

Policy-Driven Fleet Management

A Fleet Policy defines a unified OpenTelemetry configuration baseline that automatically applies to matching collectors. Rather than binding agents to static files, CollectorCtrl evaluates dynamic label queries on every heartbeat.

1. Fleet Policies & Selection Criteria

Policies target agents using dynamic, label-based queries. You can combine exact labels and logical expressions to build targeted rollout rules:

Target Selection Expressions

matchLabels: Key-value pairs that must match agent tags exactly.
matchExpressions: Advanced operators supporting set operations:
- In / NotIn: Value must exist inside or outside a list.
- Exists / DoesNotExist: Validates the presence of a tag.
- Equals / NotEquals: Direct string comparison operations.

{
  "matchLabels": {
    "app.class": "databases"
  },
  "matchExpressions": [
    {
      "key": "host.os",
      "operator": "In",
      "values": ["windows", "linux"]
    },
    {
      "key": "compliance.pci",
      "operator": "Exists"
    }
  ]
}

2. Policy Priority & Resolution Order

When an agent matches multiple policies simultaneously, CollectorCtrl evaluates policies by Priority (descending order):

1. High-Priority Evaluation

Enterprise security and compliance policies (e.g., PCI-DSS audit rules, PII redactors) are processed first.

2. Sequential Policy Deep-Merge

Policies are merged sequentially. Settings from higher-priority policies override corresponding fields in lower-priority policies.

3. Node Override Layer

The compiled policy baseline is merged with the node's local Override Config to generate the final Effective Configuration.

3. Progressive Rollouts & Canary Rings

To prevent fleet-wide telemetry outages, CollectorCtrl supports progressive release workflows:

Canary Ring Safety Protocols

Target Rollout Percentage: Specify a rollout percentage (e.g., 10% of matching nodes).
Automated Health Guards: The server tracks agent health metrics during the rollout. If an agent reports a config_error or crash loop, the rollout automatically pauses.
One-Click Promotion / Abort: Administrators can inspect canary health metrics and choose to Promote to 100% or Abort to instantly rollback all canary nodes.

4. Version History & Policy Snapshots

Every policy publish action creates an immutable Policy Snapshot in the system database:

Full Audit Trail: Logs user identity, publish timestamp, target selectors, and complete YAML state.
Visual Color-Coded Diffs: View side-by-side line diffs comparing active configurations against historical versions.
Instant Rollback: One-click rollback restores previous policy states and pushes updated configuration hashes to all matching Supervisors.

5. Deep-Merge Mechanics & Pipeline Alignment

When merging global policy templates with local overrides, CollectorCtrl applies intelligent deep-merge rules to avoid OpenTelemetry validation failures:

Pipeline Key Alignment (`AlignPolicyConfig`)

[Policy Template]            [Local Override]            [Compiled Result]
metrics                      metrics/local               metrics/local
  receivers: [otlp]     +      processors: [batch]  =>     receivers: [otlp]
                                                           processors: [batch]

List Overwrite Behavior

To allow policies to cleanly add or remove receivers, YAML lists (such as receivers: [otlp, hostmetrics]) are completely replaced by overrides during a merge, rather than appended.

policy-template.yaml

service:
pipelines:
  metrics:
    receivers: [otlp, hostmetrics]

override-config.yaml

service:
pipelines:
  metrics:
    receivers: [otlp] # Replaces policy list cleanly