Package Management & Asset Distribution

CollectorCtrl features an integrated binary registry and asset distribution system. This repository enables administrators to upload, version, verify, and push OpenTelemetry Collector binaries and auxiliary sidecar tools directly to edge nodes without relying on third-party package managers.

---

1. Asset Registry Overview

The Package Management console acts as a secure, local package repository for your telemetry binaries.

Package Classifications

We classify package assets into two categories to support flexible data plane architectures:

A. Standard Fleet Packages (Type: Primary)

These are core OpenTelemetry Collector binaries (e.g. OTel Contrib, ADOT, or custom builds) designed to run as the primary daemon on target hosts. Standard packages are typically matched to OS and CPU architectures (such as windows/amd64 or linux/arm64).

B. Custom Sidecar Packages (Type: Addon)

These are secondary helper binaries, custom scripts, or plugins. The Supervisor manages sidecar processes in tandem with the primary collector, automatically launching, restarting, and configuring them based on the policies assigned to the node.

Storage & Cryptographic Verification

• Configurable Storage Path: Packages are saved in the directory defined by the environment variable COLLECTORCTRL_PACKAGES_DIR on the Main Server.
• Integrity Validation (SHA-256): When a package is uploaded, the server calculates its SHA-256 checksum and saves it in the package database.
• Package Manifests: An automated JSON manifest is generated, mapping the architecture compatibility, version strings, dependencies, and checksums.

---

2. Step-by-Step Upgrade Strategy

Updating collector binaries across production servers can lead to telemetry blackouts if a broken executable is deployed. CollectorCtrl prevents this by using a step-by-step rollout and fallback verification process.

Follow this sequential strategy when rolling out a binary upgrade:

Step 1: Upload and Stage

Upload the new pre-compiled collector binary to the Package Management console. The server registers the version and calculates the SHA-256 hash.

Step 2: Target the Canary Ring

Create an upgrade policy targeting a small subset of non-critical hosts (e.g. env: dev or canary: true).

Step 3: Trigger the Upgrade

Associate the package version with the upgrade policy. The control plane updates the database models, generating a new configuration package offer.

Step 4: Download and Verification

Upon the next heartbeat, the target Supervisors identify the package offer. The Supervisor:

1. Downloads the package archive to a local temp folder.
2. Calculates the file's SHA-256 hash.
3. Compares the hash against the server-provided signature to ensure the file was not corrupted during transit.

Step 5: Dry-Run Configuration Check

The Supervisor executes a validation test using the newly downloaded binary and the current running YAML configuration: otelcol --config /etc/otelcol/config.yaml validate If the validation checks fail (e.g. because of an unsupported processor keyword in the new binary version), the Supervisor aborts the upgrade, reports UpgradeFailed to the server, and continues running the existing daemon.

Step 6: Swapping Processes and Verification

If the validation succeeds:

1. The Supervisor stops the active OTel Collector child process.
2. It backs up the existing executable binary to a .bak path.
3. Swaps in the new executable binary.
4. Spawns the new Collector process.
5. Monitors the process for 60 seconds, verifying the collector successfully binds to network ports and reports a healthy status.

[!WARNING] Automatic Fallback Recovery: If the new process crashes, fails to start, or reports unhealthy within the 60-second window, the Supervisor immediately stops the process, restores the backup binary from the .bak path, and restarts the original working collector to prevent loss of telemetry stream data.