---
title: Kafka-based ingest pipelines with EDOT
description: Use Kafka as a transport buffer in EDOT ingest pipelines for self-managed Elasticsearch or Elastic Cloud Managed OTLP (mOTLP).
url: https://www.elastic.co/elastic/docs-builder/docs/3028/reference/opentelemetry/architecture/kafka
products:
  - Elastic Cloud Serverless
  - Elastic Distribution of OpenTelemetry Collector
  - Elastic Observability
applies_to:
  - Serverless Observability projects: Generally available
  - Elastic Stack: Generally available
---

# Kafka-based ingest pipelines with EDOT
Kafka can act as a transport buffer between telemetry sources (applications and edge collectors) and your backend, decoupling production from ingestion. Use this pattern when you need buffering during outages or maintenance, independent scaling of collection and ingestion, or a shared transport layer across environments or networks.
In this pipeline, the collector that receives OTLP and produces to Kafka runs at the edge (the producer side). It is not the same as the [Gateway Collector](/elastic/docs-builder/docs/3028/reference/opentelemetry/architecture#understanding-the-elastic-observability-backend) that serves as the backend ingestion layer in direct-ingestion architectures. This page uses "edge gateway collector" for the producer-side collector and "backend collector" for the consumer-side collector that sends data to Elasticsearch or mOTLP.

## Reference architectures

The following patterns cover self-managed Elasticsearch and Elastic Cloud using the Elastic Cloud Managed OTLP Endpoint.

### Self-managed (on-prem) Elasticsearch

The pipeline flows as follows:
```mermaid
flowchart LR
  A["SDKs (OTLP)"] --> B["Edge gateway collector"]
  B --> C["Kafka"]
  C --> D["Backend collector"]
  D --> E["Elasticsearch"]
```

In this model:
- An edge gateway collector receives OTLP from applications (for example, EDOT SDKs or other OTLP-compatible SDKs) and exports OTLP payloads to Kafka.
- An EDOT backend collector consumes OTLP payloads from Kafka and exports to Elasticsearch.


### Elastic Cloud (Hosted/Serverless) using mOTLP

The pipeline flows as follows:
```mermaid
flowchart LR
  A["SDKs (OTLP)"] --> B["Edge gateway collector"]
  B --> C["Kafka"]
  C --> D["Backend collector"]
  D --> E["mOTLP"]
```

In this model:
- An EDOT backend collector consumes OTLP payloads from Kafka and exports to the Elastic Cloud Managed OTLP endpoint (mOTLP) using the OTLP/HTTP exporter.


## Components

These pipelines rely on the following EDOT Collector components:
- `kafkaexporter` to write OTLP payloads to Kafka.
- `kafkareceiver` to read OTLP payloads from Kafka.

For EDOT, only the `otlp_proto` and `otlp_json` encodings are supported for the Kafka receiver and exporter. Partitioning options (for example, `partition_traces_by_id`) are not supported. Refer to the [EDOT Collector components list](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/edot-collector/components) for the full list and support notes.

## Example configuration

The following examples show a minimal split deployment for:
- Edge gateway collector (produces to Kafka)
- Backend collector (consumes from Kafka and exports to Elasticsearch or mOTLP)

<note>
  Use an OTLP encoding on Kafka (for example, `otlp_proto`). Ensure the receiver and exporter use the same encoding and topics.
</note>


### Edge gateway collector

This example receives OTLP and exports to Kafka.
```yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  kafka:
    brokers: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]
    logs:
      topic: "otel-otlp"
      encoding: otlp_proto
    metrics:
      topic: "otel-otlp"
      encoding: otlp_proto
    traces:
      topic: "otel-otlp"
      encoding: otlp_proto

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [kafka]
    metrics:
      receivers: [otlp]
      exporters: [kafka]
    logs:
      receivers: [otlp]
      exporters: [kafka]
```


### Backend collector for self-managed

This example receives from Kafka and exports to Elasticsearch.
```yaml
receivers:
  kafka:
    brokers: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]
    logs:
      topics: ["otel-otlp"]
      encoding: otlp_proto
    metrics:
      topics: ["otel-otlp"]
      encoding: otlp_proto
    traces:
      topics: ["otel-otlp"]
      encoding: otlp_proto

exporters:
  elasticsearch:
    endpoints: ["https://elasticsearch.example:9200"]
    api_key: "${ELASTICSEARCH_API_KEY}"

service:
  pipelines:
    traces:
      receivers: [kafka]
      exporters: [elasticsearch]
    metrics:
      receivers: [kafka]
      exporters: [elasticsearch]
    logs:
      receivers: [kafka]
      exporters: [elasticsearch]
```


### Backend collector for Elastic Cloud

This example receives from Kafka and exports to Elastic Cloud Managed OTLP Endpoint. The Elastic Cloud Managed OTLP Endpoint endpoint uses the OTLP/HTTP protocol, so the configuration uses the `otlphttp` exporter to send to the HTTPS endpoint correctly.
```yaml
receivers:
  kafka:
    brokers: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]
    logs:
      topics: ["otel-otlp"]
      encoding: otlp_proto
    metrics:
      topics: ["otel-otlp"]
      encoding: otlp_proto
    traces:
      topics: ["otel-otlp"]
      encoding: otlp_proto

exporters:
  otlphttp:
    endpoint: "${MOTLP_ENDPOINT}"
    headers:
      Authorization: "ApiKey ${MOTLP_API_KEY}"

service:
  pipelines:
    traces:
      receivers: [kafka]
      exporters: [otlphttp]
    metrics:
      receivers: [kafka]
      exporters: [otlphttp]
    logs:
      receivers: [kafka]
      exporters: [otlphttp]
```


## Operational notes

Monitor backpressure and export failures on both the edge gateway collector and the backend collector. A Kafka buffer can mask downstream ingestion problems until retention is exhausted. To avoid it, size retention and partitions for peak ingest and expected outage windows.
For APM UI optimizations on self-managed backends, align the backend collector's mode and processors with the recommended EDOT gateway architecture for your deployment.