Contrib OpenTelemetry Collectors and language SDKs

The Elastic Stack natively supports the OpenTelemetry protocol (OTLP). This means logs, metrics, and trace data collected from your applications and infrastructure can be sent directly to the Elastic Stack.

Send data to Elastic from a contrib OpenTelemetry Collector
Send data to Elastic from a contrib OpenTelemetry language SDK

To compare approaches and choose the best one for your use case, refer to OpenTelemetry.

Important

The Elastic Distribution of OpenTelemetry Collector (EDOT Collector) includes additional features and configurations to seamlessly integrate with Elastic. Refer to EDOT compared to contrib OpenTelemetry for a comparison.

Note

The EDOT Collector runs embedded inside Elastic Agent, sharing a single elastic-agent.yml configuration file. If you're running Elastic Agent 9.2 or later, refer to Elastic Agent as an OpenTelemetry Collector instead of installing a separate Collector binary.

Send data from a contrib OpenTelemetry Collector

Connect your OpenTelemetry Collector instances to Elastic Observability or Elastic Observability Serverless using the OTLP exporter:

		receivers:
  # ...
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  # ...
  memory_limiter:
    check_interval: 1s
    limit_mib: 2000

exporters:
  debug:
    verbosity: detailed
  otlp:
    # Elastic endpoint without the "https://" prefix
    endpoint: "${env:ELASTIC_OTLP_ENDPOINT}" <5>
    headers:
      # Elastic secret token (for API key, use the ApiKey prefix)
      Authorization: "Bearer ${env:ELASTIC_SECRET_TOKEN}" <6>

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [..., memory_limiter]
      exporters: [debug, otlp]
    metrics:
      receivers: [otlp]
      processors: [..., memory_limiter]
      exporters: [debug, otlp]
    logs:
      receivers: [otlp]
      processors: [..., memory_limiter]
      exporters: [debug, otlp]
		
	

The receivers, like the OTLP receiver, that forward data emitted by OpenTelemetry SDKs, or the host metrics receiver.
Use the memory limiter processor to prevent out-of-memory failures. For more information, refer to recommended processors.
The debug exporter is helpful for troubleshooting, and supports configurable verbosity levels: basic (default), normal, and detailed.
Elastic endpoint configuration. Elastic supports a ProtoBuf payload via both the OTLP protocol over gRPC transport (OTLP/gRPC) and the OTLP protocol over HTTP transport (OTLP/HTTP). To learn more about these exporters, refer to the OpenTelemetry Collector documentation: OTLP/HTTP Exporter or OTLP/gRPC exporter. When adding an endpoint to an existing configuration an optional name component can be added, like otlp/elastic, to distinguish endpoints as described in the OpenTelemetry Collector Configuration Basics.
Hostname and port of the Elastic endpoint. For self-managed, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes deployments, use the address of your EDOT Collector configured as a gateway (for example, edot-collector:4317 for gRPC or edot-collector:4318 for HTTP). For Elastic Cloud Hosted, use the Managed OTLP endpoint.
Credential for Elastic secret token authorization (Authorization: "Bearer a_secret_token") or API key authorization (Authorization: "ApiKey an_api_key").
Environment-specific configuration parameters can be conveniently passed in as environment variables documented in the OpenTelemetry Collector environment variables reference (for example, ELASTIC_OTLP_ENDPOINT and ELASTIC_SECRET_TOKEN).
To send OpenTelemetry logs to Elastic Stack version 8.0+, declare a logs pipeline.

		receivers:
  # ...
  otlp:

processors:
  # ...
  memory_limiter:
    check_interval: 1s
    limit_mib: 2000

exporters:
  logging:
    loglevel: warn
  otlp/elastic:
    # Elastic endpoint without the "https://" prefix
    endpoint: "${ELASTIC_OTLP_ENDPOINT}" <5>
    headers:
      # Elastic API key
      Authorization: "ApiKey ${ELASTIC_API_KEY}" <6>

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [..., memory_limiter]
      exporters: [logging, otlp/elastic]
    metrics:
      receivers: [otlp]
      processors: [..., memory_limiter]
      exporters: [logging, otlp/elastic]
    logs:
      receivers: [otlp]
      processors: [..., memory_limiter]
      exporters: [logging, otlp/elastic]
		
	

The receivers, like the OTLP receiver, that forward data emitted by OpenTelemetry SDKs, or the host metrics receiver.
Use the memory limiter processor to prevent out-of-memory failures. For more information, refer to recommended processors.
The logging exporter is helpful for troubleshooting and supports various logging levels, like debug, info, warn, and error.
Elastic Observability Serverless endpoint configuration. Elastic supports a ProtoBuf payload via both the OTLP protocol over gRPC transport (OTLP/gRPC) and the OTLP protocol over HTTP transport (OTLP/HTTP). To learn more about these exporters, refer to the OpenTelemetry Collector documentation: OTLP/HTTP Exporter or OTLP/gRPC exporter.
URL of the Managed OTLP endpoint. Find your endpoint URL in the Elastic Cloud Serverless project settings.
Credential for Elastic API key authorization (Authorization: "ApiKey an_api_key").
Environment-specific configuration parameters can be conveniently passed in as environment variables documented in the OpenTelemetry Collector environment variables reference (for example, ELASTIC_OTLP_ENDPOINT and ELASTIC_API_KEY).
To send OpenTelemetry logs to your project, declare a logs pipeline.

You’re now ready to export traces and metrics from your services and applications.

Important

When using the OpenTelemetry Collector, send data through the OTLP exporter. Using the elasticsearch exporter instead bypasses Elastic's data processing pipeline and is not supported for use with Elastic Observability.

Send data from a contrib OpenTelemetry SDK

Note

The following instructions show how to send data directly from an OpenTelemetry SDK to Elastic, which is appropriate when getting started. However, sending data from an OpenTelemetry SDK to the OpenTelemetry Collector is preferred, as the Collector processes and exports data to Elastic. Read more about when and how to use a collector in the OpenTelemetry documentation.

To export traces and metrics to Elastic, instrument your services and applications with the OpenTelemetry API, SDK, or both. For example, if you are a Java developer, you need to instrument your Java app with the OpenTelemetry agent for Java. Refer to the OpenTelemetry Instrumentation guides to download the OpenTelemetry agent or SDK for your language.

Define environment variables to configure the OpenTelemetry agent or SDK and enable communication with Elastic. For example, if you are instrumenting a Java app, define the following environment variables:

		export OTEL_RESOURCE_ATTRIBUTES=service.name=checkoutService,service.version=1.1,deployment.environment=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://elastic-endpoint:4318
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer a_secret_token"
export OTEL_METRICS_EXPORTER="otlp" \
export OTEL_LOGS_EXPORTER="otlp" \
java -javaagent:/path/to/opentelemetry-javaagent-all.jar \
     -classpath lib/*:classes/ \
     com.mycompany.checkout.CheckoutServiceServer
		
	

The OpenTelemetry logs intake through Elastic is in technical preview.

OTEL_RESOURCE_ATTRIBUTES

Fields that describe the service and the environment that the service runs in. Refer to attributes for more information.

OTEL_EXPORTER_OTLP_ENDPOINT

Elastic endpoint URL. For self-managed, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes deployments, use the address of your EDOT Collector configured as a gateway or APM Server OTLP endpoint. For Elastic Cloud Hosted, use the Managed OTLP endpoint.

OTEL_EXPORTER_OTLP_HEADERS

Authorization header that includes the Elastic secret token or API key: "Authorization=Bearer a_secret_token" or "Authorization=ApiKey an_api_key".

For information on how to format an API key, refer to API keys.

Note the required space between Bearer and a_secret_token, and ApiKey and an_api_key.

Note

If you are using a version of the Python OpenTelemetry agent earlier than 1.27.0, the content of the header must be URL-encoded. You can use the Python standard library’s urllib.parse.quote function to encode the content of the header.

OTEL_METRICS_EXPORTER

Metrics exporter to use. Refer to exporter selection for more information.

OTEL_LOGS_EXPORTER

Logs exporter to use. Refer to exporter selection for more information.

		export OTEL_RESOURCE_ATTRIBUTES=service.name=checkoutService,service.version=1.1,deployment.environment=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://your-motlp-endpoint:443
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey an_api_key"
export OTEL_METRICS_EXPORTER="otlp" \
export OTEL_LOGS_EXPORTER="otlp" \
java -javaagent:/path/to/opentelemetry-javaagent-all.jar \
     -classpath lib/*:classes/ \
     com.mycompany.checkout.CheckoutServiceServer
		
	

The OpenTelemetry logs intake through Elastic is in technical preview.

OTEL_RESOURCE_ATTRIBUTES

Fields that describe the service and the environment that the service runs in. Refer to attributes for more information.

OTEL_EXPORTER_OTLP_ENDPOINT

Elastic endpoint URL. The URL of the Managed OTLP endpoint.

OTEL_EXPORTER_OTLP_HEADERS

Authorization header that includes the Elastic API key: "Authorization=ApiKey an_api_key". Note the required space between ApiKey and an_api_key.

For information on how to format an API key, refer to API keys.

Note

OTEL_METRICS_EXPORTER

Metrics exporter to use. Refer to exporter selection for more information.

OTEL_LOGS_EXPORTER

Logs exporter to use. Refer to exporter selection for more information.

You are now ready to collect traces and metrics, and then verify and visualize them.

Proxy requests to APM Server

Note

For new users, Elastic recommends sending OpenTelemetry data to the EDOT Collector or Managed OTLP endpoint instead of to the APM Server.

APM Server supports both the OTLP/gRPC and OTLP/HTTP protocol on the same port as Elastic APM agent requests. For ease of setup, use OTLP/HTTP when proxying or load balancing requests to Elastic.

If you use the OTLP/gRPC protocol, requests to Elastic must use either HTTP/2 over TLS or HTTP/2 Cleartext (H2C). No matter which protocol is used, OTLP/gRPC requests will have the "Content-Type: application/grpc" header.

When using a layer 7 (L7) proxy like AWS ALB, proxy the requests in a way that ensures they follow the rules outlined previously. For example, with ALB you can create rules to select an alternative backend protocol based on the headers of requests coming into ALB. In this example, you’d select the gRPC protocol when the "Content-Type: application/grpc" header exists on a request.

Many L7 load balancers handle HTTP and gRPC traffic separately and rely on explicitly defined routes and service configurations to correctly proxy requests. Since APM Server serves both protocols on the same port, it may not be compatible with some L7 load balancers. For example, to work around this issue in Ingress NGINX Controller for Kubernetes, either:

Use the otlp exporter in the EDOT collector. Set annotation nginx.ingress.kubernetes.io/backend-protocol: "GRPC" on the K8s Ingress object proxying to APM Server.
Use the otlphttp exporter in the EDOT collector. Set annotation nginx.ingress.kubernetes.io/backend-protocol: "HTTP" (or "HTTPS" if APM Server expects TLS) on the K8s Ingress object proxying to APM Server.

The preferred approach is to deploy a L4 (TCP) load balancer (for example, NLB on AWS) in front of APM Server, which forwards raw TCP traffic transparently without protocol inspection.

For more information on how to configure an AWS ALB to support gRPC, refer to this AWS blog post: Application Load Balancer Support for End-to-End HTTP/2 and gRPC.

For more information on how APM Server services gRPC requests, refer to Muxing gRPC and HTTP/1.1.

APM Server vs managed intake service

In Elastic Cloud Hosted, the APM Server receives data from Elastic APM agents and transforms it into Elasticsearch documents. In Elastic Cloud Serverless there is in fact no APM Server running, instead the managed intake service receives and transforms data.

Next steps

Collect metrics
Add resource attributes
Learn about the limitations of this integration