Quickstart for Docker on Elastic Cloud Hosted

Learn how to set up the Elastic Agent and EDOT SDKs in a Docker environment with Elastic Cloud Hosted (ECH) to collect host metrics, logs, and application traces. This quickstart uses the Elastic Cloud Managed OTLP Endpoint, which is the recommended ingestion path for ECH.

  • An Elastic Cloud Hosted deployment running version 9.0 or later.
  • Docker and Docker Compose installed on the host.

The fastest way to get started is the Add data screen in Elastic Observability. It generates install commands with your endpoint and API key already included.

  1. Open Elastic Observability.
  2. Go to Add data.
  3. Select what you want to monitor.
  4. Follow the instructions.

If you need to manage credentials manually, for example, to use them in automation or to configure multiple environments, follow the steps in the Manual installation section.

Follow these steps to deploy the Elastic Agent and EDOT SDKs in Docker with ECH:

  1. Create the config file

    Create an Elastic Agent configuration file for the Elastic Cloud Managed OTLP Endpoint. This example uses the filename otel-collector-config.yml.

    Start from the logs, metrics, and traces sample for the Elastic Cloud Managed OTLP Endpoint. The sample is written for a host process, so adapt it for the Compose mounts in this quickstart:

    1. In file_log/platformlogs, set include to [/hostfs/var/log/*.log].

    2. In hostmetrics/system, set root_path: /hostfs.

    3. Add a docker_stats receiver and a pipeline that exports those metrics:

      receivers:
  docker_stats: {}

service:
  pipelines:
    metrics/docker:
      receivers: [docker_stats]
      processors: [resourcedetection]
      exporters: [otlp_grpc/ingest_metrics_traces]

    Keep the other receivers, processors, exporters, and pipelines from the sample. The result is a Docker-specific configuration, as the adaptations replace host paths with /hostfs mount paths and add Docker metrics collection. For details about the pipelines, refer to Using the Managed OTLP Endpoint.

  2. Find your endpoint and create an API key

    Find your endpoint

    The easiest way to get your endpoint and API key is from the Add data screen in Kibana:

    1. Go to Add data.
    2. In the Connect directly to the endpoint section, select the OpenTelemetry tab.
    3. Copy the Endpoint value.
    4. Click Create key to generate an API key with the required privileges.

    Alternatively, retrieve the endpoint from the Elastic Cloud Console and create an API key manually:

    1. Log in to the Elastic Cloud Console.
    2. Find your deployment in Hosted deployments, and select Manage.
    3. In the Application endpoints, cluster and component IDs section, select Managed OTLP.
    4. Copy the public endpoint value.

    Create an API key

    Note

    The Elastic Cloud Managed OTLP Endpoint validates API keys using APM application privileges. Index-level privilege scoping is not yet supported, meaning that API keys with custom index-level role descriptors return a PermissionDenied error.

    Using Kibana

    1. Go to Stack Management → API keys.

    2. Click Create API key, enter a name, and enable Control security privileges.

    3. In the role descriptors box, enter the following privileges:

      {
  "otlp_writer": {
    "applications": [
      {
        "application": "apm",
        "resources": ["*"],
        "privileges": ["event:write"]
      }
    ]
  }
}

    4. Click Create API key and copy the encoded value.

    Using the Elasticsearch API

    Use the Create API key API:

    				POST /_security/api_key
					{
  "name": "otlp-writer",
  "role_descriptors": {
    "otlp_writer": {
      "applications": [
        {
          "application": "apm",
          "resources": ["*"],
          "privileges": ["event:write"]
        }
      ]
    }
  }
}

    The event:write privilege for the apm application is the minimum required to send data through the Elastic Cloud Managed OTLP Endpoint.

  3. Create the .env file

    Create a .env file with the following content. Replace the placeholders with your Elastic Cloud credentials and the path to the configuration file you created:

    HOST_FILESYSTEM=/
DOCKER_SOCK=/var/run/docker.sock
ELASTIC_AGENT_OTEL=true
COLLECTOR_CONTRIB_IMAGE=elastic/elastic-agent:9.5.3
ELASTIC_API_KEY=<your_api_key_here>
ELASTIC_OTLP_ENDPOINT=<your_motlp_endpoint_here>
OTEL_COLLECTOR_CONFIG=/path/to/otel-collector-config.yml

  4. Create the compose file

    Create a compose.yml file with the following content:

    services:
 otel-collector:
   image: ${COLLECTOR_CONTRIB_IMAGE}
   container_name: otel-collector
   deploy:
     resources:
       limits:
         memory: 1.5G
   restart: unless-stopped
   command: ["--config", "/etc/otelcol-config.yml" ]
   network_mode: host
   user: "0:0"
   volumes:
     - ${HOST_FILESYSTEM}:/hostfs:ro
     - ${DOCKER_SOCK}:/var/run/docker.sock:ro
     - ${OTEL_COLLECTOR_CONFIG}:/etc/otelcol-config.yml
   environment:
     - HOST_FILESYSTEM
     - ELASTIC_AGENT_OTEL
     - ELASTIC_API_KEY
     - ELASTIC_OTLP_ENDPOINT
     - STORAGE_DIR=/usr/share/elastic-agent

  5. Start the Collector

    To start the Collector, run:

    docker compose up -d

  6. (Optional) Instrument your applications

    To collect telemetry from applications and use the Elastic Agent as a gateway, instrument your target applications following the setup instructions:

    Configure your SDKs to send the data to the local Elastic Agent using OTLP/gRPC (http://localhost:4317) or OTLP/HTTP (http://localhost:4318).

    Tip

    Enable Central Configuration to configure your EDOT SDKs from within Kibana. Refer to EDOT SDKs Central Configuration.

  7. Install the content packs

    Install the System OpenTelemetry Assets integration and the Docker OpenTelemetry Assets integration in Kibana.

  8. Explore your data

    Go to Kibana and select Dashboards to explore your newly collected data.

If you need to write telemetry directly to Elasticsearch (for example, for pipeline customizations not yet supported through Elastic Cloud Managed OTLP Endpoint), use a different Collector configuration, .env file, and compose file. For a full list of features and limitations that apply to each path, refer to Elastic features available with Elastic OpenTelemetry.

Start from the logs, metrics, and traces sample for direct ingestion into Elasticsearch. The sample is written for a host process, so adapt it for the Compose mounts in this quickstart:

  1. In file_log/platformlogs, set include to [/hostfs/var/log/*.log].

  2. In hostmetrics/system, set root_path: /hostfs.

  3. Add a docker_stats receiver and a pipeline that exports those metrics:

    receivers:
  docker_stats: {}

service:
  pipelines:
    metrics/docker:
      receivers: [docker_stats]
      processors: [resourcedetection]
      exporters: [elasticsearch/otel]

Keep the other receivers, processors, exporters, and pipelines from the sample. For details about the pipelines, refer to Direct ingestion into Elasticsearch.

Retrieve your Elasticsearch URL and your API key:

  1. Retrieve the Elasticsearch URL for your Elastic Cloud deployment:

    1. Go to the Elastic Cloud console.
    2. Next to your deployment, select Manage.
    3. Under Applications next to Elasticsearch, select Copy endpoint.

  2. Create an API Key following these instructions.

Create a .env file with your Elasticsearch endpoint, credentials, and the path to the configuration file:

HOST_FILESYSTEM=/
DOCKER_SOCK=/var/run/docker.sock
ELASTIC_AGENT_OTEL=true
COLLECTOR_CONTRIB_IMAGE=elastic/elastic-agent:9.5.3
ELASTIC_API_KEY=<your_api_key_here>
ELASTIC_ENDPOINT=<your_elasticsearch_endpoint_here>
OTEL_COLLECTOR_CONFIG=/path/to/otel-collector-config.yml

Use the following compose file, which passes ELASTIC_ENDPOINT instead of ELASTIC_OTLP_ENDPOINT:

services:
 otel-collector:
   image: ${COLLECTOR_CONTRIB_IMAGE}
   container_name: otel-collector
   deploy:
     resources:
       limits:
         memory: 1.5G
   restart: unless-stopped
   command: ["--config", "/etc/otelcol-config.yml" ]
   network_mode: host
   user: "0:0"
   volumes:
     - ${HOST_FILESYSTEM}:/hostfs:ro
     - ${DOCKER_SOCK}:/var/run/docker.sock:ro
     - ${OTEL_COLLECTOR_CONFIG}:/etc/otelcol-config.yml
   environment:
     - HOST_FILESYSTEM
     - ELASTIC_AGENT_OTEL
     - ELASTIC_API_KEY
     - ELASTIC_ENDPOINT
     - STORAGE_DIR=/usr/share/elastic-agent

The following issues might occur.

The following error is due to an improperly formatted API key, and typically occurs when credentials are configured manually:

Exporting failed. Dropping data.
{"kind": "exporter", "data_type": }
"Unauthenticated desc = ApiKey prefix not found"

For a Collector, format the header as "Authorization": "ApiKey <api-key>". For an SDK, format it as "Authorization=ApiKey <api-key>".

For additional troubleshooting, refer to Troubleshooting common issues with the Elastic Agent and Troubleshooting the EDOT SDKs.