---
title: Send any log file using OTel Collector
description: This guide shows you how to manually configure the Elastic OpenTelemetry Collector to send your log data to Elasticsearch by configuring the otel.yml...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7434/solutions/observability/logs/stream-any-log-file-using-edot-collector
products:
  - Elastic Cloud Serverless
  - Elastic Observability
applies_to:
  - Elastic Cloud Serverless: Generally available
  - Elastic Stack: Generally available
---

# Send any log file using OTel Collector
This guide shows you how to manually configure the Elastic OpenTelemetry Collector to send your log data to Elasticsearch by configuring the `otel.yml` file. For an Elastic Agent equivalent, refer to [Send any log file using Elastic Agent](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7434/solutions/observability/logs/stream-any-log-file).
For more OpenTelemetry quickstarts, refer to [Elastic OpenTelemetry quickstarts](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7434/solutions/observability/get-started/opentelemetry/quickstart).

## Prerequisites

<applies-switch>
  <applies-item title="stack:" applies-to="Elastic Stack: Generally available">
    To follow the steps in this guide, you need an Elastic Stack deployment that includes:
    - Elasticsearch for storing and searching data
    - Kibana for visualizing and managing data
    - Kibana user with `All` privileges on Fleet and Integrations. Because many Integrations assets are shared across spaces, users need the Kibana privileges in all spaces.
    - Integrations Server (included by default in every Elastic Cloud Hosted deployment)
    To get started quickly, create an Elastic Cloud Hosted deployment and host it on AWS, GCP, or Azure. [Try it out for free](https://cloud.elastic.co/registration?page=docs&placement=docs-body).
  </applies-item>

  <applies-item title="serverless:" applies-to="Elastic Cloud Serverless: Generally available">
    The **Admin** role or higher is required to onboard log data. To learn more, refer to [Assign user roles and privileges](/elastic/docs-content/pull/7434/deploy-manage/users-roles/cloud-organization/user-roles#general-assign-user-roles).
  </applies-item>
</applies-switch>


## Install Elastic Agent

Complete these steps to install and configure Elastic Agent and send your log data to Elastic Observability.
<stepper>
  <step title="Download and install {{agent}}">
    On your host, download the Elastic Agent installation package that corresponds with your system:
    <tab-set>
      <tab-item title="Linux">
        ```shell
        curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.4.3-linux-x86_64.tar.gz
        tar xzvf elastic-agent-9.4.3-linux-x86_64.tar.gz
        ```
      </tab-item>

      <tab-item title="macOS">
        ```shell
        curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.4.3-darwin-x86_64.tar.gz
        tar xzvf elastic-agent-9.4.3-darwin-x86_64.tar.gz
        ```
      </tab-item>

      <tab-item title="Windows">
        ```powershell
        # PowerShell 5.0+
        wget https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.4.3-windows-x86_64.zip -OutFile elastic-agent-9.4.3-windows-x86_64.zip
        Expand-Archive .\elastic-agent-9.4.3-windows-x86_64.zip
        ```
      </tab-item>
    </tab-set>
  </step>

  <step title="Configure {{agent}}">
    Follow these steps to retrieve the managed OTLP endpoint URL for your Serverless project:
    1. In Elastic Cloud Serverless, open your Observability project.
    2. Go to **Add data** → **Application** → **OpenTelemetry**.
    3. Select **Managed OTLP Endpoint** in the second step.
    4. Copy the OTLP endpoint configuration value.
    5. Select **Create API Key** to generate an API key.
    Replace `<ELASTIC_OTLP_ENDPOINT>` and `<ELASTIC_API_KEY>` before applying the following commands:
    <tab-set>
      <tab-item title="Linux">
        ```bash
        ELASTIC_OTLP_ENDPOINT=<ELASTIC_OTLP_ENDPOINT> && \
        ELASTIC_API_KEY=<ELASTIC_API_KEY> && \
        cp ./otel_samples/managed_otlp/logs_metrics_traces.yml ./otel.yml && \
        mkdir -p ./data/otelcol && \
        sed -i "s#\${env:STORAGE_DIR}#${PWD}/data/otelcol#g" ./otel.yml && \
        sed -i "s#\${env:ELASTIC_OTLP_ENDPOINT}#${ELASTIC_OTLP_ENDPOINT}#g" ./otel.yml && \
        sed -i "s#\${env:ELASTIC_API_KEY}#${ELASTIC_API_KEY}#g" ./otel.yml
        ```
      </tab-item>

      <tab-item title="macOS">
        ```bash
        ELASTIC_OTLP_ENDPOINT=<ELASTIC_OTLP_ENDPOINT> && \
        ELASTIC_API_KEY=<ELASTIC_API_KEY> && \
        cp ./otel_samples/managed_otlp/logs_metrics_traces.yml ./otel.yml && \
        mkdir -p ./data/otelcol && \
        sed -i '' "s#\${env:STORAGE_DIR}#${PWD}/data/otelcol#g" ./otel.yml && \
        sed -i '' "s#\${env:ELASTIC_OTLP_ENDPOINT}#${ELASTIC_OTLP_ENDPOINT}#g" ./otel.yml && \
        sed -i '' "s#\${env:ELASTIC_API_KEY}#${ELASTIC_API_KEY}#g" ./otel.yml
        ```
      </tab-item>

      <tab-item title="Windows">
        ```powershell
        Remove-Item -Path .\otel.yml -ErrorAction SilentlyContinue
        Copy-Item .\otel_samples\managed_otlp\logs_metrics_traces.yml .\otel.yml
        New-Item -ItemType Directory -Force -Path .\data\otelcol | Out-Null

        $content = Get-Content .\otel.yml
        $content = $content -replace '\${env:STORAGE_DIR}', "$PWD\data\otelcol"
        $content = $content -replace '\${env:ELASTIC_OTLP_ENDPOINT}', "<ELASTIC_OTLP_ENDPOINT>"
        $content = $content -replace '\${env:ELASTIC_API_KEY}', "<ELASTIC_API_KEY>"
        $content | Set-Content .\otel.yml
        ```
      </tab-item>
    </tab-set>
  </step>

  <step title="Configure log file collection">
    To collect logs from specific log files, you need to modify the `otel.yml` configuration file. The configuration includes receivers, processors, and exporters that handle log data.
    <applies-switch>
      <applies-item title="stack:" applies-to="Elastic Stack: Generally available">
        Here's an example configuration for collecting log files with Elastic Stack:
        <dropdown title="otel.yml for logs collection (Elastic Stack)">
          ```yaml
          receivers:
            # Receiver for platform specific log files
            filelog/platformlogs:
              include: [ /var/log/*.log ]
              retry_on_failure:
                enabled: true
              start_at: end
              storage: file_storage
          #   start_at: beginning

          extensions:
            file_storage:
              directory: ${env:STORAGE_DIR}

          processors:
            resourcedetection:
              detectors: ["system"]
              system:
                hostname_sources: ["os"]
                resource_attributes:
                  host.name:
                    enabled: true
                  host.id:
                    enabled: false
                  host.arch:
                    enabled: true
                  host.ip:
                    enabled: true
                  host.mac:
                    enabled: true
                  host.cpu.vendor.id:
                    enabled: true
                  host.cpu.family:
                    enabled: true
                  host.cpu.model.id:
                    enabled: true
                  host.cpu.model.name:
                    enabled: true
                  host.cpu.stepping:
                    enabled: true
                  host.cpu.cache.l2.size:
                    enabled: true
                  os.description:
                    enabled: true
                  os.type:
                    enabled: true

          exporters:
            # Exporter to print the first 5 logs/metrics and then every 1000th
            debug:
              verbosity: detailed
              sampling_initial: 5
              sampling_thereafter: 1000

            # Exporter to send logs and metrics to Elasticsearch
            elasticsearch/otel:
              endpoints: ["${env:ELASTIC_ENDPOINT}"]
              api_key: ${env:ELASTIC_API_KEY}
              mapping:
                mode: otel

          service:
            extensions: [file_storage]
            pipelines:
              logs/platformlogs:
                receivers: [filelog/platformlogs]
                processors: [resourcedetection]
                exporters: [debug, elasticsearch/otel]
          ```
        </dropdown>
      </applies-item>

      <applies-item title="serverless:" applies-to="Elastic Cloud Serverless: Generally available">
        Here's an example configuration for collecting log files with Elastic Cloud Serverless:
        <dropdown title="otel.yml for logs collection (Serverless)">
          ```yaml
          receivers:
            # Receiver for platform specific log files
            filelog/platformlogs:
              include: [/var/log/*.log]
              retry_on_failure:
                enabled: true
              start_at: end
              storage: file_storage
          #   start_at: beginning

          extensions:
            file_storage:
              directory: ${env:STORAGE_DIR}

          processors:
            resourcedetection:
              detectors: ["system"]
              system:
                hostname_sources: ["os"]
                resource_attributes:
                  host.name:
                    enabled: true
                  host.id:
                    enabled: false
                  host.arch:
                    enabled: true
                  host.ip:
                    enabled: true
                  host.mac:
                    enabled: true
                  host.cpu.vendor.id:
                    enabled: true
                  host.cpu.family:
                    enabled: true
                  host.cpu.model.id:
                    enabled: true
                  host.cpu.model.name:
                    enabled: true
                  host.cpu.stepping:
                    enabled: true
                  host.cpu.cache.l2.size:
                    enabled: true
                  os.description:
                    enabled: true
                  os.type:
                    enabled: true

          exporters:
            # Exporter to print the first 5 logs/metrics and then every 1000th
            debug:
              verbosity: detailed
              sampling_initial: 5
              sampling_thereafter: 1000

            # Exporter to send logs and metrics to Elasticsearch Managed OTLP Input
            otlp/ingest:
              endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
              headers:
                Authorization: ApiKey ${env:ELASTIC_API_KEY}

          service:
            extensions: [file_storage]
            pipelines:
              logs/platformlogs:
                receivers: [filelog/platformlogs]
                processors: [resourcedetection]
                exporters: [debug, otlp/ingest]
          ```
        </dropdown>
      </applies-item>
    </applies-switch>
    Key configuration elements:
    - `receivers.filelog/platformlogs.include`: Specifies the path to your log files. You can use patterns like `/var/log/*.log`.
    - `processors.resourcedetection`: Automatically detects and adds host system information to your logs.
    - `extensions.file_storage`: Provides persistent storage for the Collector's state.
    - `exporters`: Configures how data is sent to Elasticsearch (Elastic Stack) or OTLP endpoint (Serverless).
  </step>

  <step title="Run {{agent}}">
    Run the following command to run Elastic Agent:
    <tab-set>
      <tab-item title="Linux and macOS">
        ```bash
        sudo ./otelcol --config otel.yml
        ```
      </tab-item>

      <tab-item title="Windows">
        ```powershell
        .\elastic-agent.exe otel --config otel.yml
        ```
      </tab-item>
    </tab-set>

    <note>
      The Collector opens ports `4317` and `4318` to receive application data from locally running OTel SDKs without authentication. This allows the SDKs to send data without any further configuration needed as they use this endpoint by default.
    </note>
  </step>
</stepper>


## Troubleshoot your Elastic Agent configuration

If you're not seeing your log files in the UI, verify the following:
- The path to your logs file under `include` is correct.
- Your API key is properly set in the environment variables.
- The OTLP endpoint URL is correct and accessible.
- The Collector is running without errors (check the console output).

If you're still running into issues, see [Elastic Agent troubleshooting](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7434/troubleshoot/ingest/opentelemetry/edot-collector) and [Configure Elastic Agent](https://docs-v3-preview.elastic.dev/elastic/elastic-agent/tree/main/reference/edot-collector/config).

## Next steps

After you have Elastic Agent configured and are streaming log data to Elasticsearch:
- Refer to the [Explore log data](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7434/solutions/observability/logs/discover-logs) documentation for information on exploring your log data in the UI, including searching and filtering your log data, getting information about the structure of log fields, and displaying your findings in a visualization.
- Refer to the [Parse and organize logs](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7434/solutions/observability/logs/parse-route-logs) documentation for information on extracting structured fields from your log data, rerouting your logs to different data streams, and filtering and aggregating your log data.
- Refer to the [Filter and aggregate logs](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7434/solutions/observability/logs/filter-aggregate-logs) documentation for information on filtering and aggregating your log data to find specific information, gain insight, and monitor your systems more efficiently.
- To collect telemetry from applications and use Elastic Agent as a gateway, instrument your target applications following the setup instructions:
  - [Android](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/android/)
- [.NET](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/dotnet/setup/)
- [iOS](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/ios/)
- [Java](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/java/setup/)
- [Node.js](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/nodejs/setup/)
- [PHP](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/php/setup/)
- [Python](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/python/setup/)