--- title: Send data from an upstream OpenTelemetry Collector description: Send data from an upstream OpenTelemetry Collector to a self-managed Elastic Stack by routing it through an EDOT Collector gateway. url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7168/solutions/observability/get-started/opentelemetry/use-cases/upstream-collector products: - Elastic Distribution of OpenTelemetry Collector - Elastic Observability applies_to: - Self-managed Elastic deployments: Generally available since 9.2 --- # Send data from an upstream OpenTelemetry Collector This guide shows how to forward telemetry data from an existing (upstream) OpenTelemetry Collector to a self-managed Elastic Stack using an [Elastic Distribution of OpenTelemetry (EDOT) Collector](https://docs-v3-preview.elastic.dev/elastic/elastic-agent/tree/main/reference/edot-collector) configured as a gateway. The examples use the [contrib distribution](https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib) (`otelcol-contrib`), but the same approach works with any OTel Collector distribution, including vendor distributions and custom builds assembled with the [OpenTelemetry Collector Builder](https://opentelemetry.io/docs/collector/custom-collector/). ## When to use this setup Use this setup if you: - Already run an existing OpenTelemetry Collector and want to add Elastic as a backend without replacing your current setup - Need to send telemetry to multiple observability backends from a single Collector - Evaluate Elastic alongside another backend before committing to a full migration - Use a technology or language for which Elastic doesn't provide an EDOT SDK ## Architecture Your services send telemetry to an OpenTelemetry Collector (for example, `otelcol-contrib`), which forwards it over OTLP/gRPC to the EDOT Collector gateway. The gateway applies Elastic-specific processing and writes directly to Elasticsearch. ```mermaid flowchart LR S["Your services"] -->|OTLP| U["Upstream otelcol-contrib
(one per host)"] U -->|"OTLP/gRPC :4317"| G subgraph G["EDOT Collector (gateway)"] P["elasticapm processor
enriches spans"] C["elasticapm connector
generates aggregated metrics"] end G -->|"elasticsearch exporter
mapping.mode: otel"| E[("Elasticsearch")] ``` The `elasticsearch` exporter with `mapping.mode: otel` is the recommended path for self-managed deployments. The [Managed OTLP endpoint](https://docs-v3-preview.elastic.dev/elastic/opentelemetry/tree/main/reference/managed-inputs/managed-otlp-endpoint) is an alternative ingest path available only on Elastic Cloud, so it doesn't apply to self-managed deployments. Sending directly to APM Server or the managed intake service through OTLP is also possible, but the EDOT gateway path in this guide is recommended for full APM functionality. ## Before you begin You’ll need: - A running [self-managed](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7168/deploy-manage/deploy/self-managed) Elasticsearch cluster - The EDOT Collector installed on the gateway host. It ships as part of the Elastic Agent package and runs as Elastic Agent in `otel` mode. - An existing OpenTelemetry Collector installed on your agent hosts. This guide uses [`otelcol-contrib`](https://opentelemetry.io/docs/collector/installation/). - Network connectivity from your Collector hosts to the EDOT gateway host on port 4317 ## Set up the EDOT gateway The EDOT gateway authenticates to Elasticsearch using an API key. 1. Find **API keys** in the navigation menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7168/explore-analyze/find-and-organize/find-apps-and-objects). 2. Select **Create API key**. 3. Give the key a name (for example, `edot-gateway`) and assign it `auto_configure` and `create_doc` index privileges on the `logs-*.otel-*`, `metrics-*.otel-*`, and `traces-*.otel-*` indices. 4. Copy the encoded key to use as the value of the `ELASTIC_API_KEY` environment variable in the gateway configuration. 1. Set the following environment variables on the gateway host before starting the Collector: ```bash export ELASTIC_ENDPOINT=https://your-elasticsearch:9200 export ELASTIC_API_KEY=your-encoded-api-key ``` 2. Create the following configuration file and save it as `gateway.yml` on the gateway host: ```yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 connectors: elasticapm: {} processors: elasticapm: {} exporters: elasticsearch/otel: endpoints: - ${env:ELASTIC_ENDPOINT} api_key: ${env:ELASTIC_API_KEY} mapping: mode: otel service: pipelines: traces: receivers: [otlp] processors: [elasticapm] exporters: [elasticapm, elasticsearch/otel] metrics: receivers: [otlp] processors: [] exporters: [elasticsearch/otel] metrics/aggregated-otel-metrics: receivers: [elasticapm] processors: [] exporters: [elasticsearch/otel] logs: receivers: [otlp] processors: [] exporters: [elasticsearch/otel] ``` Key components in this configuration: - **`elasticapm` processor** (under `processors`): Enriches spans with attributes required by the APM UI. - **`elasticapm` connector** (under `connectors`): Generates pre-aggregated APM metrics from trace data. It appears as an exporter in the `traces` pipeline and as a receiver in the `metrics/aggregated-otel-metrics` pipeline. - **`elasticsearch/otel` exporter**: Writes data directly to Elasticsearch using native OpenTelemetry data streams (`mapping.mode: otel`). The exporter handles batching automatically using `sending_queue`. Refer to [Performance and batching](https://docs-v3-preview.elastic.dev/elastic/elastic-agent/tree/main/reference/edot-collector/components/elasticsearchexporter#performance-and-batching) to customize throughput for your environment. The `elasticapm` connector and processor are required for full APM functionality (service maps, transaction histograms, service-level indicators). You only need them when exporting directly to Elasticsearch. If you send data to the Managed OTLP endpoint or APM Server or the managed intake service, they are not required.Refer to [APM services missing due to misconfigured elasticapm connector](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7168/troubleshoot/ingest/opentelemetry/edot-collector/misconfigured-elasticapm-connector) for more information. 3. Start the EDOT gateway. The EDOT Collector is the Elastic Agent binary run in `otel` mode, so start it with the `otel` subcommand: ```bash elastic-agent otel --config gateway.yml ``` 1. Configure the OTLP exporter to point to the EDOT gateway. On each contrib Collector host, add or update the `exporters` and `service` sections in your existing `config.yml`: ```yaml exporters: otlp: endpoint: "gateway-host:4317" tls: insecure: true service: pipelines: traces: exporters: [otlp] metrics: exporters: [otlp] logs: exporters: [otlp] ``` Replace `gateway-host` with the hostname or IP of your EDOT gateway host. In production, set `insecure: false` and configure `ca_file` with the path to the CA certificate used to secure communication between the contrib Collector and the gateway. Refer to the [TLS configuration settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md) for the full list of options. 2. Set the `deployment.environment` resource attribute in your contrib Collector so that services appear under the correct environment in the Kibana APM Service Map. Without it, all services show as "unset" in the environment selector: ```yaml processors: resource: attributes: - key: deployment.environment action: insert value: production ``` Refer to [Attributes and labels](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7168/solutions/observability/apm/opentelemetry/attributes) for more details. 3. Restart the contrib Collector to apply the changes. After starting both Collectors, wait a few minutes for data to appear. Then verify in Kibana: - Find **Applications** in the navigation menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7168/explore-analyze/find-and-organize/find-apps-and-objects), then go to **Services** to confirm your services appear. - From **Applications**, go to **Service Map** to confirm environment-based filtering works. - Find **Discover** in the navigation menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7168/explore-analyze/find-and-organize/find-apps-and-objects) and check the `traces-generic.otel-default`, `logs-generic.otel-default`, and `metrics-generic.otel-default` data streams for incoming data. If no data appears, refer to [No logs, metrics, or traces visible in Kibana](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7168/troubleshoot/ingest/opentelemetry/no-data-in-kibana) for troubleshooting tips. ## Related pages - [No logs, metrics, or traces visible in Kibana](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7168/troubleshoot/ingest/opentelemetry/no-data-in-kibana) - [APM services missing due to misconfigured elasticapm connector](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7168/troubleshoot/ingest/opentelemetry/edot-collector/misconfigured-elasticapm-connector) - [Attributes and labels](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7168/solutions/observability/apm/opentelemetry/attributes) - [EDOT Collector](https://docs-v3-preview.elastic.dev/elastic/elastic-agent/tree/main/reference/edot-collector) - [Managed OTLP endpoint](https://docs-v3-preview.elastic.dev/elastic/opentelemetry/tree/main/reference/managed-inputs/managed-otlp-endpoint)