BetaFree-to-use OpAMP fleet management for Windows & LinuxQuick Start →
Get Started

CollectorCtrl — Understanding Config Sources & Governance

Note

Audience: End users, DevOps engineers, SREs, and Platform / Observability teams using CollectorCtrl.

1. Overview: What is "Config Source"?

In the CollectorCtrl Fleet Overview, each OpenTelemetry Collector displays a badge under the CONFIG SOURCE column.

This column tells you where the collector's running configuration originates and how it is currently governed:

Config Source BadgeBackend ValueMeaning at a Glance
GitOps Baseline Badge GitOps BaselinepolicyGoverned 100% by centralized fleet policies synchronized with Git. Standard, compliant state.
GitOps Node Override Badge GitOps Node OverridemanualHas a node-specific custom configuration applied directly to this agent, overriding the baseline.
Default Badge DefaultdefaultNot attached to any fleet policy or override; runs standalone default settings.
                                 CONFIG SOURCE HIERARCHY
                                 
      Central Git Repository (Base + Templates)
                         │
                         ▼
        ┌──────────────────────────────────┐
        │       GitOps Baseline            │ ◄── Standard Fleet Configuration
        │   (Governed by Fleet Policy)     │     (Green Badge)
        └─────────────────┬────────────────┘
                          │
         Does node have a local override?
                          │
             ┌────────────┴────────────┐
          NO │                         │ YES
             ▼                         ▼
   ┌──────────────────┐      ┌─────────────────────────┐
   │ GitOps Baseline  │      │  GitOps Node Override   │ ◄── Custom Node Tweaks
   │  (Pure Policy)   │      │ (Merged: Policy + Node) │     (Amber Badge)
   └──────────────────┘      └─────────────────────────┘

2. Deep Dive: The Three Config Sources

1. GitOps Baseline (Green Badge)

  • What it means:
    The collector is strictly inheriting its configuration from an Authoritative Fleet Policy defined by the Central Observability / Platform Team and stored in Git version control.
  • How it works:
    1. The collector reports its metadata (environment, operating system, hostname, compute type, cluster tags) during its OpAMP handshake.
    2. CollectorCtrl matches these attributes against policy target selectors (matchLabels and matchExpressions).
    3. The matching policy’s YAML configuration (from Git repositories such as base/ and templates/) is compiled and deployed to the collector.
  • When you see this:
    This is the ideal operational state for 99%+ of collectors in production. It guarantees fleet consistency, compliance, security guardrails (PII masking, memory limiters), and cost controls.
  • Modifying configuration:
    Changes should be made upstream in the GitOps repository via Pull Request (PR) or in the Central Policy management view.

2. GitOps Node Override (Amber / Orange Badge)

  • What it means:
    An operator has applied a custom, instance-level configuration override (CustomInstanceConfig) directly to this specific collector.
  • Why it is flagged with an amber warning:
    While node overrides are fully supported as an operational escape hatch, they create a divergent state ("snowflake") where the collector is no longer purely governed by fleet-wide policies.
  • How it works (Precedence & Deep Merge):
    CollectorCtrl does not erase the policy baseline. Instead, its deep merge engine combines the baseline with the node override:
    • Key Overrides: If the baseline sets batch: { timeout: 5s } and the node override specifies batch: { timeout: 1s }, the override wins.
    • Additive Blocks: If the override introduces a new receiver (e.g. prometheus/local: ...), it is merged into the active pipeline.
  • Common Use Cases:
    • Emergency Troubleshooting: Enabling verbose debug logging (log: { level: debug }) on a single problematic node without flooding the rest of the fleet.
    • Temporary High-Frequency Sampling: Lowering scrape intervals to 1s to capture an ongoing incident.
    • Hardware/Host Unique Requirements: Adding a custom path or local file log scraper unique to a single server.
  • Reverting to Baseline:
    When troubleshooting is complete, operators can click Remove Override in the Collector Detail View. The collector immediately returns to GitOps Baseline status.

3. Default (Gray Badge)

  • What it means:
    The collector is connected to CollectorCtrl, but:
    1. It does not match any active Fleet Policy (agent.PolicyID == "").
    2. It has no custom instance override (agent.CustomInstanceConfig == "").
  • How it behaves:
    The agent operates using its built-in fallback configuration (default supervisor baseline).
  • Recommended Action:
    Check the collector's tags and environment attributes. Assign the collector to an appropriate environment or create a fleet policy matching its labels to bring it under centralized GitOps governance.

3. Configuration Merge Engine & Precedence

When CollectorCtrl generates the executable YAML sent to a collector via OpAMP, it evaluates layers according to the 4-Tier Policy Architecture:

Tier 0: Global Base Layer (e.g., base/00-global-base.yaml)
        └── Global OTLP sinks, proxy certs, memory limiter guardrails
Tier 1: Template / Policy Layer (e.g., templates/linux-vm-standard.yaml)
        └── Host metrics, standard file logs, common processors
Tier 2: Cluster Layer (e.g., clusters/aws-eks-prod.yaml)
        └── K8s attributes processor, cluster-specific endpoints
Tier 3: Node Override Layer (e.g., overrides/<hostname>.yaml)
        └── Node-specific custom YAML defined in CollectorCtrl UI
─────────────────────────────────────────────────────────────────────────────
                             ▼ DEEP MERGE
                  EFFECTIVE CONFIGURATION (Live on Agent)

Precedence Rules:

  1. Tier 3 (Node Override) wins over Tier 2, Tier 1, and Tier 0 for conflicting scalar fields.
  2. Pipelines and maps are merged cleanly, ensuring baseline security processors (like memory_limiter or resourcedetection) remain intact unless explicitly altered.
  3. Telemetric Taps (if active) are dynamically injected at runtime.

4. How to Inspect & Manage Config Sources in the UI

A. In the Fleet Table

  • Look at the CONFIG SOURCE column.
  • Use the Quick Filter bar:
    • Click Manual Config to filter for all nodes currently running with GitOps Node Override.
    • Easily audit snowflake nodes across thousands of agents.

B. In the Collector Detail View

Navigating to an individual collector reveals the Configuration Hierarchy Panel:

  1. Step 1: Policy Baseline
    Shows the assigned fleet policy name and environment (e.g., linux-prod-policy).
  2. Step 2: Node Override
    • Displays "No Override" (Inheriting policy baseline) if running pure GitOps.
    • Displays "Custom Active" (Custom rules taking precedence) if a node override exists.
  3. Step 3: Effective Config
    Displays the final, compiled YAML that is currently running live on the collector.

Working with the Config Tabs:

  • 📄 Effective Config (Live) Tab:
    Read-only view of the exact compiled YAML delivered to the collector over OpAMP WebSocket.
  • ✏️ Node Override Tab:
    Interactive YAML editor allowing authorized engineers to add custom overrides.
  • Remove Override Button:
    Located above the YAML editor. Clicking this deletes the instance override from the database and Git, automatically restoring the collector to the pure GitOps Baseline.

5. GitOps Integration & Auditability

When GitOps synchronization is enabled:

  • Policies are synchronized with Git repositories (/base and /templates).
  • Node Overrides created in the UI are automatically tracked and exported to the /overrides/<hostname>.yaml directory in Git.
  • Drift Detection: CollectorCtrl continuously checks the running configuration hash reported by the OpAMP supervisor against the Git target hash. Any unauthorized local tampering on the host machine triggers a ⚠️ Config Drift indicator.

6. Best Practices for End Users

  1. Aim for GitOps Baseline: Keep collectors on GitOps Baseline whenever possible for predictable telemetry pipelines and zero configuration drift.
  2. Treat Overrides as Ephemeral: Use GitOps Node Override for active troubleshooting or local testing, and remove the override once the issue is resolved.
  3. Promote Recurring Overrides to Policies: If multiple nodes require the same override settings, request the Central Observability Team to incorporate those settings into a reusable GitOps template.