Semantic Mapping Registry

Expand

1. The Core Challenge: Semantic Telemetry Fragmentation

Every major observability platform was developed with its own proprietary taxonomy and attribute naming conventions. Even when using OpenTelemetry at the edge, organizations face significant semantic friction when routing telemetry across multiple backends:

| Telemetry Concept | OpenTelemetry Standard | Datadog | Splunk HEC | Coralogix | Dynatrace | Elasticsearch | | :--- | :--- | :--- | :--- | :--- | :--- | | Service Name | service.name | service | service_name | service.name | service.name | service.name | | Environment | deployment.environment | env | environment | subsystemName | deployment.environment | service.environment | | Host Identifier | host.name | host | host | host.name | host.name | host.name | | Log Body | body | message | _raw | body / message | content | message | | Application Domain | service.namespace | env | service.namespace | applicationName | service.namespace | service.namespace | | Client IP Address | client.address | network.client.ip | client.address | client.address | client.address | client.ip |

This fragmentation forces platform teams to write fragile, bespoke regexes and manual processor renames every time a new vendor or data stream is onboarded.

2. What We Are Doing with the Registry Today

The Semantic Mapping Registry provides a centralized, vendor-neutral dictionary that maps proprietary vendor attribute keys to canonical OpenTelemetry Semantic Conventions (v1.26+).

Domain-Indexed Schema Hierarchy

The registry organizes thousands of telemetry attributes into structured domain categories:

• Resource & Service: service.name, service.version, service.namespace, service.instance.id
• Host & OS: host.name, host.id, host.arch, os.type
• Cloud Infrastructure: cloud.provider, cloud.region, cloud.account.id, cloud.zone
• Kubernetes & Containers: k8s.pod.name, k8s.namespace.name, k8s.node.name, container.name
• HTTP & Web: http.request.method, http.response.status_code, url.path, client.address
• Database & SQL: db.system, db.name, db.statement, db.operation
• RPC & Network: rpc.system, rpc.service, rpc.method, network.transport
• Messaging & Streams: messaging.system, messaging.destination.name, messaging.operation

How the Mappings Are Maintained

1. One-Click OTel Seeding: By clicking ✨ Seed OTel Semantic Conventions, the registry populates baseline canonical attributes directly from the official OpenTelemetry semantic convention specification.
2. Multi-Vendor Crosswalk: Each attribute maps across supported vendor backends (Coralogix, Datadog, Dynatrace, Elasticsearch, New Relic, Splunk HEC).
3. Translation Classifications:
   • direct: The vendor natively supports the OpenTelemetry attribute key without alteration.
   • rename: The attribute must be renamed to match the vendor's expected ingestion schema (e.g. service.name $\to$ service).
   • transform: Advanced translation involving compound expressions or conditional logic.

3. The Strategic Roadmap: Universal Observability Cross-Compiler

The long-term vision for the Semantic Mapping Registry is to serve as the industry's universal translation engine, enabling frictionless telemetry migration and multi-vendor portability.

┌────────────────────────────────────────────────────────────────────────┐
│                   SEMANTIC MAPPING REGISTRY (DATABASE)                 │
│         Exhaustive Attribute Dictionary Across All Major Vendors       │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
       ┌────────────────────────────┼────────────────────────────┐
       ▼                            ▼                            ▼
┌──────────────┐             ┌──────────────┐             ┌──────────────┐
│     AIM:     │             │  EXECUTION:  │             │OTHER PURPOSE:│
│  OTTL Code   │             │ Copilot &    │             │  Alerts &    │
│  Generator   │             │  MCP Server  │             │  Dashboard   │
│  (YAML A➔B)  │             │ Integration  │             │ Translations │
└──────────────┘             └──────────────┘             └──────────────┘

Aim: The OTTL Translation Engine for Cross-Vendor Bridging

The primary goal is to build an exhaustive, verified database covering every major observability vendor. This database will power automated OpenTelemetry Transformation Language (OTTL) code generation:

• The Problem: When migrating from Vendor A (e.g., Splunk or Datadog) to Vendor B (e.g., Coralogix or an open OTLP lake), engineers must rewrite hundreds of lines of collector YAML, adjusting tag names, resource attributes, and log bodies to satisfy the new backend.
• The Solution: The Semantic Registry acts as an automated compiler. Given a source vendor and target vendor, it generates the exact OTTL transformation statements needed to make the data immediately compatible:

# Automated OTTL Translation (Vendor A [Datadog] ➔ Vendor B [Coralogix])
processors:
transform/vendor_translation:
  error_mode: ignore
  log_statements:
    - context: log
      statements:
        # 1. Map Application & Subsystem metadata
        - set(attributes["applicationName"], attributes["service"]) where attributes["service"] != nil
        - set(attributes["subsystemName"], attributes["env"]) where attributes["env"] != nil
        
        # 2. Standardize log message payload
        - set(body, attributes["message"]) where attributes["message"] != nil and body == nil
        
        # 3. Preserve source trace context
        - set(trace_id.string, attributes["dd.trace_id"]) where attributes["dd.trace_id"] != nil
        - set(span_id.string, attributes["dd.span_id"]) where attributes["dd.span_id"] != nil

Execution: Native Integration with Copilot & MCP

The Semantic Registry will serve as a ground-truth knowledge base for AI agents operating within the CollectorCtrl ecosystem:

• CollectorCtrl Copilot Integration:
  Engineers can query the Copilot directly from the dashboard:
  • "How do I translate our legacy Splunk HEC tags into Coralogix application and subsystem attributes?"
  • "Generate an OTTL transform rule to convert Datadog distributed trace headers to W3C TraceContext standards."
• Model Context Protocol (MCP) Server Integration:
  External AI developer tools (Cursor, Claude Desktop, autonomous agents) will access the registry via dedicated MCP tools:
  • lookup_semantic_mapping(concept, source_vendor, target_vendor)
  • generate_ottl_migration_policy(source_vendor, target_vendor, signal)
    This eliminates AI hallucinations by grounding all configuration proposals in verified semantic mappings.

Additional Strategic Purpose: Alert & Dashboard Payload Translation

Beyond real-time data plane telemetry, the single biggest blocker in observability migrations is content lock-in:

• Organizations have accumulated hundreds of proprietary alert rules, monitors, and dashboards (e.g., Datadog Monitors, Splunk SPL saved searches, Grafana panels, Coralogix alerts).
• Manually rewriting alert thresholds and dashboard queries across distinct query languages is a multi-month engineering effort.
• The Semantic Registry as a Query Cross-Compiler:
  Because the registry understands both metric naming differences (e.g., system.cpu.idle vs. system.cpu.time{state="idle"}) and label dimensions (host:web-01 vs. host.name="web-01"), it provides the foundational dictionary required to translate alert queries and dashboard JSON definitions from Vendor A into Vendor B.

4. Current State vs. Roadmap Overview

Summary of Alpha Guidelines

1. Exploration Only: Use the Semantic Mapping Registry to review naming conventions and plan upcoming migrations.
2. Do Not Expect Runtime Effect: Remember that changes made in the registry are not yet injected into live collector pipelines.
3. Report Feedback: Suggest missing vendor mappings or domain extensions to the engineering team as we expand the semantic database.

Next Steps

• CollectorCtrl Copilot — Discover how AI assists with fleet operations and OTTL generation
• Model Context Protocol (MCP) Server — Connect external AI agents to CollectorCtrl APIs
• Custom Binary Builder (OCB) — Compile tailored OpenTelemetry binaries with custom exporters
• Fleet Orchestration & Policies — Manage fleet-wide collector configurations and rollouts