Send data to Elastic Observability using the contrib Collector

While Elastic Agent provides a streamlined experience with pre-selected components, you can also use the contrib OpenTelemetry Collector or a custom distribution to send data to Elastic Observability. This approach requires more configuration but gives you more control over your OpenTelemetry setup.

The contrib OpenTelemetry Collector is the community-maintained version that provides the foundation for all OpenTelemetry distributions. To configure it to work with Elastic Observability, you need to:

  • Manually select and configure components.
  • Set up proper data processing pipelines.
  • Handle authentication and connection details.
  • Ensure required components have been properly configured in accordance to your use case.

The configuration requirements vary depending on your use case and the Elastic deployment model you want to send data to. The following sections outline what you need for each scenario.

Elastic Cloud Serverless provides a Elastic Cloud Managed OTLP Endpoint that accepts OpenTelemetry data in its native format. This makes it the simplest scenario for using contrib components because scaling and signal processing (for example producing metrics from events) is handled by Elastic.

The following configuration example shows how to send data to the Elastic Cloud Managed OTLP Endpoint:

exporters:
  otlp:
    endpoint: "https://<motlp-endpoint>"
    headers:
      Authorization: "ApiKey <your-api-key>"

service:
  pipelines:
    traces:
      exporters: [otlp]
    metrics:
      exporters: [otlp]
    logs:
      exporters: [otlp]

To find your endpoint and create an API key, refer to Elastic Cloud Managed OTLP Endpoint.

Elastic Cloud Hosted provides a Elastic Cloud Managed OTLP Endpoint. Point your contrib Collector OTLP exporter at the endpoint using the same configuration as Elastic Cloud Serverless.

Note

On Elastic Cloud Hosted, the Elastic Cloud Managed OTLP Endpoint requires a deployment version 9.0 or later.

If you need local processing before ingest, such as deriving metrics from events in APM, you can set up Elastic Agent as a gateway. Refer to Gateway configuration.

The Elastic Cloud Managed OTLP Endpoint is not available for self-managed, Elastic Cloud Enterprise, or Elastic Cloud on Kubernetes deployments. With these deployment methods, set up Elastic Agent as a gateway. The gateway handles processing required for some use cases, like deriving metrics from events in APM, and writes data directly to Elasticsearch.

Point your contrib Collector OTLP exporter to the Elastic Agent gateway. Refer to Gateway configuration and Gateway mode.

You also need to:

  • Point the gateway to your Elasticsearch instance.
  • Configure appropriate security settings.
  • Ensure your Elasticsearch version is compatible.
  • Set up proper index templates and mappings.

When using the contrib OpenTelemetry Collector with Elastic Observability, follow these best practices:

Always include the resourcedetectionprocessor to automatically add host, cloud, and Kubernetes metadata:

processors:
  resourcedetection:
    detectors: [env, system, gcp, ecs, ec2, azure, aks, eks, gke]
    timeout: 5s
    override: true

Use the attributesprocessor to ensure consistent attribute naming and add required metadata:

processors:
  attributes:
    actions:
      - key: service.name
        value: "your-service-name"
        action: insert
      - key: service.version
        value: "1.0.0"
        action: insert

Configure the batchprocessor for optimal performance:

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
    send_batch_max_size: 2048

For production deployments, always use secure connections:

exporters:
  elasticsearch:
    tls:
      insecure: false
      ca_file: "/path/to/ca.crt"
    user: "elastic"
    password: "YOUR_PASSWORD"

Using the contrib OpenTelemetry Collector instead of Elastic Agent comes with some trade-offs. Refer to Elastic Agent compared to the contrib Collector for more information.