Loading

Send data from a contrib OpenTelemetry Collector

This guide shows how to forward telemetry data from an upstream contrib OpenTelemetry Collector to a self-managed Elastic Stack using an EDOT Collector in gateway mode.

Use this pattern if you:

  • Already run a contrib OpenTelemetry Collector and want to add Elastic as a backend without replacing your existing setup
  • Need to fan out telemetry to multiple observability backends from a single contrib Collector
  • Evaluate Elastic alongside another backend before committing to a full migration
  • Use a technology or language for which Elastic doesn't provide an EDOT distribution

Your services send telemetry to the contrib Collector (otelcol-contrib), which forwards it over OTLP/gRPC to the EDOT Collector gateway. The gateway applies Elastic-specific processing and writes directly to Elasticsearch.

flowchart LR
    S["Your services"] -->|OTLP| U["Upstream otelcol-contrib<br/>(one per host)"]
    U -->|"OTLP/gRPC :4317"| G

    subgraph G["EDOT Collector (gateway)"]
        P["elasticapm processor<br/>enriches spans"]
        C["elasticapm connector<br/>generates aggregated metrics"]
    end

    G -->|"elasticsearch exporter<br/>mapping.mode: otel"| E[("ES")]

The elasticsearch exporter with mapping.mode: otel is the recommended path for self-managed deployments. The Managed OTLP endpoint isn't available for self-managed installations. Sending directly to APM Server or the managed intake service through OTLP is also possible but not recommended. Use the EDOT gateway path when available.

  • A running self-managed Elasticsearch cluster
  • The EDOT Collector installed on the gateway host. It ships as part of the Elastic Agent package and runs as Elastic Agent in otel mode.
  • The contrib OpenTelemetry Collector installed on your agent hosts
  • Network connectivity from the contrib Collector hosts to the EDOT gateway host on port 4317

  1. Create an Elasticsearch API key

    The EDOT gateway authenticates to Elasticsearch using an API key.

    1. In Kibana, navigate to Stack Management → API keys.
    2. Select Create API key.
    3. Give the key a name (for example, edot-gateway) and assign it the necessary privileges for writing to Elasticsearch data streams.
    4. Copy the encoded key to use as the ELASTIC_API_KEY value in the gateway configuration.

  2. Configure the EDOT gateway

    Create a configuration file for the EDOT Collector running in gateway mode. Save this as gateway.yml on the gateway host.

    Set the following environment variables on the gateway host before starting the Collector:

    export ELASTIC_ENDPOINT=https://your-elasticsearch:9200
export ELASTIC_API_KEY=your-encoded-api-key

    Then create gateway.yml:

    receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

connectors:
  elasticapm: {}

processors:
  batch:
    send_batch_size: 1000
    timeout: 1s
    send_batch_max_size: 1500
  batch/metrics:
    send_batch_max_size: 0
    timeout: 1s
  elasticapm: {}

exporters:
  elasticsearch/otel:
    endpoints:
      - ${env:ELASTIC_ENDPOINT}
    api_key: ${env:ELASTIC_API_KEY}
    mapping:
      mode: otel

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, elasticapm]
      exporters: [elasticapm, elasticsearch/otel]
    metrics:
      receivers: [otlp]
      processors: [batch/metrics]
      exporters: [elasticsearch/otel]
    metrics/aggregated-otel-metrics:
      receivers: [elasticapm]
      processors: []
      exporters: [elasticsearch/otel]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [elasticsearch/otel]

    Key components in this configuration:

    • elasticapm processor (under processors): Enriches spans with attributes required by the APM UI.
    • elasticapm connector (under connectors): Generates pre-aggregated APM metrics from trace data. It appears as an exporter in the traces pipeline and as a receiver in the metrics/aggregated-otel-metrics pipeline.
    • elasticsearch/otel exporter: Writes data directly to Elasticsearch using native OpenTelemetry data streams (mapping.mode: otel).
    Note

    The elasticapm connector and processor are required for full APM functionality (service maps, transaction histograms, service-level indicators). You only need them when exporting directly to Elasticsearch. If you send to the Managed OTLP endpoint or APM Server or the managed intake service, they are not required.

    Refer to APM services missing due to misconfigured elasticapm connector for more information.

    3. Start the EDOT gateway. The EDOT Collector is the Elastic Agent binary run in otel mode, so start it with the otel subcommand:

    elastic-agent otel --config gateway.yml

  3. Configure the contrib Collector

    On each contrib Collector host, configure the OTLP exporter to point to the EDOT gateway. Add or update the exporters and service sections in your existing config.yml:

    exporters:
  otlp:
    endpoint: "gateway-host:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      exporters: [otlp]
    metrics:
      exporters: [otlp]
    logs:
      exporters: [otlp]
    1. Set to false and configure ca_file for production

    Replace gateway-host with the hostname or IP of your EDOT gateway host. In production, configure TLS to secure communication between the contrib Collector and the gateway.

    Tip

    Set the deployment.environment resource attribute in your contrib Collector so that services appear under the correct environment in the Kibana APM Service Map. Without it, all services show as "unset" in the environment selector.

    processors:
  resource:
    attributes:
      - key: deployment.environment
        action: insert
        value: production

    Refer to Attributes and labels for more details.

    4. Restart the contrib Collector to apply the changes.

  4. Verify data in Kibana

    After starting both Collectors, wait a few minutes for data to appear. Then verify in Kibana:

    • Navigate to Observability → APM → Services to confirm your services appear.
    • Navigate to Observability → APM → Service Map to confirm environment-based filtering works.
    • Navigate to Discover and check the traces-generic.otel-default, logs-generic.otel-default, and metrics-generic.otel-default data streams for incoming data.

    If no data appears, refer to No logs, metrics, or traces visible in Kibana.