Loading

Send data from an upstream OpenTelemetry Collector

This guide shows how to forward telemetry data from an existing (upstream) OpenTelemetry Collector to a self-managed Elastic Stack using an Elastic Distribution of OpenTelemetry (EDOT) Collector configured as a gateway. The examples use the contrib distribution (otelcol-contrib), but the same approach works with any OTel Collector distribution, including vendor distributions and custom builds assembled with the OpenTelemetry Collector Builder.

Use this setup if you:

  • Already run an existing OpenTelemetry Collector and want to add Elastic as a backend without replacing your current setup
  • Need to send telemetry to multiple observability backends from a single Collector
  • Evaluate Elastic alongside another backend before committing to a full migration
  • Use a technology or language for which Elastic doesn't provide an EDOT SDK

Your services send telemetry to an OpenTelemetry Collector (for example, otelcol-contrib), which forwards it over OTLP/gRPC to the EDOT Collector gateway. The gateway applies Elastic-specific processing and writes directly to Elasticsearch.

flowchart LR
    S["Your services"] -->|OTLP| U["Upstream otelcol-contrib<br/>(one per host)"]
    U -->|"OTLP/gRPC :4317"| G

    subgraph G["EDOT Collector (gateway)"]
        P["elasticapm processor<br/>enriches spans"]
        C["elasticapm connector<br/>generates aggregated metrics"]
    end

    G -->|"elasticsearch exporter<br/>mapping.mode: otel"| E[("Elasticsearch")]

The elasticsearch exporter with mapping.mode: otel is the recommended path for self-managed deployments.

Note

The Managed OTLP endpoint is an alternative ingest path available only on Elastic Cloud, so it doesn't apply to self-managed deployments. Sending directly to APM Server or the managed intake service through OTLP is also possible, but the EDOT gateway path in this guide is recommended for full APM functionality.

You’ll need:

  • A running self-managed Elasticsearch cluster
  • The EDOT Collector installed on the gateway host. It ships as part of the Elastic Agent package and runs as Elastic Agent in otel mode.
  • An existing OpenTelemetry Collector installed on your agent hosts. This guide uses otelcol-contrib.
  • Network connectivity from your Collector hosts to the EDOT gateway host on port 4317

  1. Create an Elasticsearch API key

    The EDOT gateway authenticates to Elasticsearch using an API key.

    1. Find API keys in the navigation menu or use the global search field.
    2. Select Create API key.
    3. Give the key a name (for example, edot-gateway) and assign it auto_configure and create_doc index privileges on the logs-*.otel-*, metrics-*.otel-*, and traces-*.otel-* indices.
    4. Copy the encoded key to use as the value of the ELASTIC_API_KEY environment variable in the gateway configuration.

  2. Configure the EDOT gateway

    1. Set the following environment variables on the gateway host before starting the Collector:

      export ELASTIC_ENDPOINT=https://your-elasticsearch:9200
export ELASTIC_API_KEY=your-encoded-api-key

    2. Create the following configuration file and save it as gateway.yml on the gateway host:

      receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

connectors:
  elasticapm: {}

processors:
  elasticapm: {}

exporters:
  elasticsearch/otel:
    endpoints:
      - ${env:ELASTIC_ENDPOINT}
    api_key: ${env:ELASTIC_API_KEY}
    mapping:
      mode: otel

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [elasticapm]
      exporters: [elasticapm, elasticsearch/otel]
    metrics:
      receivers: [otlp]
      processors: []
      exporters: [elasticsearch/otel]
    metrics/aggregated-otel-metrics:
      receivers: [elasticapm]
      processors: []
      exporters: [elasticsearch/otel]
    logs:
      receivers: [otlp]
      processors: []
      exporters: [elasticsearch/otel]

      Key components in this configuration:

      • elasticapm processor (under processors): Enriches spans with attributes required by the APM UI.
      • elasticapm connector (under connectors): Generates pre-aggregated APM metrics from trace data. It appears as an exporter in the traces pipeline and as a receiver in the metrics/aggregated-otel-metrics pipeline.
      • elasticsearch/otel exporter: Writes data directly to Elasticsearch using native OpenTelemetry data streams (mapping.mode: otel). The exporter handles batching automatically using sending_queue. Refer to Performance and batching to customize throughput for your environment.
      Note

      The elasticapm connector and processor are required for full APM functionality (service maps, transaction histograms, service-level indicators). You only need them when exporting directly to Elasticsearch. If you send data to the Managed OTLP endpoint or APM Server or the managed intake service, they are not required.

      Refer to APM services missing due to misconfigured elasticapm connector for more information.

    3. Start the EDOT gateway. The EDOT Collector is the Elastic Agent binary run in otel mode, so start it with the otel subcommand:

      elastic-agent otel --config gateway.yml

  3. Configure the contrib Collector

    1. Configure the OTLP exporter to point to the EDOT gateway. On each contrib Collector host, add or update the exporters and service sections in your existing config.yml:

      exporters:
  otlp:
    endpoint: "gateway-host:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      exporters: [otlp]
    metrics:
      exporters: [otlp]
    logs:
      exporters: [otlp]
      1. Set to false and configure ca_file for production

      Replace gateway-host with the hostname or IP of your EDOT gateway host. In production, set insecure: false and configure ca_file with the path to the CA certificate used to secure communication between the contrib Collector and the gateway. Refer to the TLS configuration settings for the full list of options.

    2. Set the deployment.environment resource attribute in your contrib Collector so that services appear under the correct environment in the Kibana APM Service Map. Without it, all services show as "unset" in the environment selector:

      processors:
  resource:
    attributes:
      - key: deployment.environment
        action: insert
        value: production

      Refer to Attributes and labels for more details.

    3. Restart the contrib Collector to apply the changes.

  4. Verify data in Kibana

    After starting both Collectors, wait a few minutes for data to appear. Then verify in Kibana:

    • Find Applications in the navigation menu or use the global search field, then go to Services to confirm your services appear.
    • From Applications, go to Service Map to confirm environment-based filtering works.
    • Find Discover in the navigation menu or use the global search field and check the traces-generic.otel-default, logs-generic.otel-default, and metrics-generic.otel-default data streams for incoming data.

    If no data appears, refer to No logs, metrics, or traces visible in Kibana for troubleshooting tips.