elastic-agent
Loading

Custom data stream routing with EDOT

Serverless Observability Stack EDOT Collector

Elastic Distribution of OpenTelemetry (EDOT) uses opinionated defaults for data stream naming to ensure compatibility with Elastic dashboards, APM visualizations, and curated UIs.

While most use cases rely on these defaults, EDOT also supports advanced dynamic routing.

Important

We strongly recommend not changing the default data stream names. Customizing data stream routing diverges from the standard ingestion model and there's no guarantee it will be valid for future versions.

The only recommended use case for customizing data stream routing is to separate data by environment (for example: dev, staging, and prod).

In this case, we recommend changing only data_stream.namespace, not data_stream.dataset.

A data stream name follows this structure:

<type>-<dataset>-<namespace>

The namespace is intended as the configurable part of the name. Elastic dashboards, detectors, and UIs support multiple namespaces automatically.

Changing the dataset value can cause:

  • Dashboards and APM views to fail to load
  • Loss of compatibility with built-in correlations and cross-linking
  • Inconsistent field mappings
  • Proliferation of data streams and increased shard counts

Only modify dataset if it's absolutely necessary and you're aware of the trade-offs.

To enable dynamic data stream routing:

  1. Set mapping.mode: otel in the Elasticsearch exporter. When using otel mapping mode, the exporter appends .otel to the data_stream.dataset value automatically.
  2. Use a resource processor to set the desired namespace or dataset from resource attributes.
  3. Add the processor to your pipeline.
Note

The example is purely illustrative, with no guarantee of it being production ready.

exporters:
  elasticsearch/otel:
    api_key: ${env:ELASTIC_API_KEY}
    endpoints: [${env:ELASTIC_ENDPOINT}]
    mapping:
      mode: otel

processors:
  resource/env-namespace:
    attributes:
      - key: data_stream.namespace
        from_attribute: k8s.namespace.name
        action: upsert

service:
  pipelines:
    metrics/otel:
      processors:
        - batch
        - resource/env-namespace
      exporters:
        - elasticsearch/otel
  1. <-- make sure you are using `resource` and not `attributes`
  2. <-- add the processor to the pipeline

Any dynamic value used in data_stream.namespace or data_stream.dataset must comply with Elasticsearch index naming rules:

  • Lowercase only
  • No spaces
  • Must not start with _
  • Must not contain: ", \, *, ,, <, >, |, ?, /
  • Avoid hyphens in environment names (use produs instead of prod-us)

Invalid names prevent data stream creation.

This configuration diverges from the standard ingestion model. Be aware of the following:

  • Future EDOT versions may not support this configuration or may introduce breaking changes.
  • Changes might lead to an increase in data streams and shard counts.
  • Dashboards and UIs may not recognize non-standard datasets.
  • Some data streams might fail to be created if there are non-allowed characters in the values set for data_stream.namespace or data_stream.dataset.

Use this feature only when necessary and validate in non-production environments first.