﻿---
title: ECS formatted application logs
description: Logs formatted in Elastic Common Schema (ECS) don't require manual parsing, and the same ingest configuration can be reused across applications. ECS-formatted...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/observability/logs/ecs-formatted-application-logs
products:
  - Elastic Cloud Serverless
  - Elastic Observability
applies_to:
  - Elastic Cloud Serverless: Generally available
  - Elastic Stack: Generally available
---

# ECS formatted application logs
Logs formatted in Elastic Common Schema (ECS) don't require manual parsing, and the same ingest configuration can be reused across applications. ECS-formatted logs, when paired with an APM agent or [Elastic Distribution of OpenTelemetry SDKs](https://docs-v3-preview.elastic.dev/elastic/opentelemetry/tree/main/reference/edot-sdks), allow you to correlate logs to easily view logs that belong to a particular trace.
<tip>
  We recommend using the [Elastic Distribution of OpenTelemetry SDKs](https://docs-v3-preview.elastic.dev/elastic/opentelemetry/tree/main/reference/edot-sdks) to instrument your applications and collect ECS-formatted logs with automatic log correlation. You can also format your logs in ECS format.
</tip>

You can format your logs in ECS format the following ways:
- [ECS loggers](#ecs-loggers): plugins for your logging libraries that reformat your logs into ECS format.
- [APM agent ECS reformatting](#apm-agent-ecs-reformatting): Java, Ruby, and Python APM agents automatically reformat application logs to ECS format without a logger.


## ECS loggers

ECS loggers reformat your application logs into ECS-compatible JSON, removing the need for manual parsing. ECS loggers require Filebeat or Elastic Agent configured to monitor and capture application logs. In addition, pairing ECS loggers with your framework’s APM agent allows you to correlate logs to easily view logs that belong to a particular trace.

### Get started with ECS loggers

For more information on adding an ECS logger to your application, refer to the guide for your framework:
- [.NET](https://docs-v3-preview.elastic.dev/elastic/ecs-dotnet/tree/main/reference/setup)
- Go: [zap](https://docs-v3-preview.elastic.dev/elastic/ecs-logging-go-zap/tree/main/reference/setup)
- [Java](https://docs-v3-preview.elastic.dev/elastic/ecs-logging-java/tree/main/reference/setup)
- Node.js: [morgan](https://docs-v3-preview.elastic.dev/elastic/ecs-logging-nodejs/tree/main/reference/winston)
- [PHP](https://docs-v3-preview.elastic.dev/elastic/ecs-logging-php/tree/main/reference/setup)
- [Python](https://docs-v3-preview.elastic.dev/elastic/ecs-logging-python/tree/main/reference/installation)
- [Ruby](https://docs-v3-preview.elastic.dev/elastic/ecs-logging-ruby/tree/main/reference/setup)


## APM agent ECS reformatting

Java, Ruby, and Python APM agents can automatically reformat application logs to ECS format without an ECS logger or the need to modify your application. The APM agent also allows for log correlation so you can easily view logs that belong to a particular trace.
To set up log ECS reformatting:
1. [Enable APM agent reformatting](#enable-log-ecs-reformatting)
2. [Ingest logs with Filebeat or Elastic Agent](#ingest-ecs-logs)
3. [View logs in Discover](#view-ecs-logs)


### Enable log ECS reformatting

Log ECS reformatting is controlled by the `log_ecs_reformatting` configuration option, and is disabled by default. Refer to the guide for your framework for information on enabling:
- [Java](https://docs-v3-preview.elastic.dev/elastic/apm-agent-java/tree/main/reference/config-logging#config-log-ecs-reformatting)
- [Ruby](https://docs-v3-preview.elastic.dev/elastic/apm-agent-ruby/tree/main/reference/configuration#config-log-ecs-formatting)
- [Python](https://docs-v3-preview.elastic.dev/elastic/apm-agent-python/tree/main/reference/configuration#config-log_ecs_reformatting)


### Ingest logs

After enabling log ECS reformatting, send your application logs to your project using one of the following shipping tools:
- [Filebeat](#ingest-ecs-logs-with-filebeat): A lightweight data shipper that sends log data to your project.
- [Elastic Agent](#ingest-ecs-logs-with-agent): A single agent for logs, metrics, security data, and threat prevention. With Fleet, you can centrally manage Elastic Agent policies and lifecycles directly from your project.


#### Ingest logs with Filebeat

Follow these steps to ingest application logs with Filebeat.

#### Step 1: Install Filebeat

Install Filebeat on the server you want to monitor by running the commands that align with your system:
<tab-set>
  <tab-item title="DEB">
    ```sh
    curl -L -O https\://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.3.1-amd64.deb
    sudo dpkg -i filebeat-9.3.1-amd64.deb
    ```
  </tab-item>

  <tab-item title="RPM">
    ```sh
    curl -L -O https\://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.3.1-x86_64.rpm
    sudo rpm -vi filebeat-9.3.1-x86_64.rpm
    ```
  </tab-item>

  <tab-item title="macOS">
    ```sh
    curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.3.1-darwin-x86_64.tar.gz
    tar xzvf filebeat-9.3.1-darwin-x86_64.tar.gz
    ```
  </tab-item>

  <tab-item title="Linux">
    ```sh
    curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.3.1-linux-x86_64.tar.gz
    tar xzvf filebeat-9.3.1-linux-x86_64.tar.gz
    ```
  </tab-item>

  <tab-item title="Windows">
    1. Download the [Filebeat Windows zip file](https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.3.1-windows-x86_64.zip).
    2. Extract the contents of the zip file into `C:\Program Files`.
    3. Rename the _filebeat-9.3.1-windows-x86_64_ directory to _Filebeat_:
    4. Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select **Run As Administrator**).
    5. From the PowerShell prompt, run the following commands to install Filebeat as a Windows service:
       ```powershell
       PS > cd 'C:\Program Files\{filebeat}'
       PS C:\Program Files\{filebeat}> .\install-service-filebeat.ps1
       ```
    If script execution is disabled on your system, you need to set the execution policy for the current session to allow the script to run. For example: `PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-filebeat.ps1`.
  </tab-item>
</tab-set>


#### Step 2: Connect to your project

Connect to your project using an API key to set up Filebeat. Set the following information in the `filebeat.yml` file:
```yaml
output.elasticsearch:
  hosts: ["your-projects-elasticsearch-endpoint"]
  api_key: "id:api_key"
```

1. Set the `hosts` to your deployment’s Elasticsearch endpoint. Copy the Elasticsearch endpoint from **Help menu (![help icon](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-help-icon.svg)) → Connection details**. For example, `https://my-deployment.es.us-central1.gcp.cloud.es.io:443`.
2. From **Developer tools**, run the following command to create an API key that grants `manage` permissions for the `cluster` and the `filebeat-*` indices using:
   ```json

   {
     "name": "filebeat_host001",
     "role_descriptors": {
       "filebeat_writer": {
         "cluster": ["manage"],
         "index": [
           {
             "names": ["filebeat-*"],
             "privileges": ["manage"]
           }
         ]
       }
     }
   }
   ```
   Refer to [Grant access using API keys](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/filebeat/beats-api-keys) for more information.


#### Step 3: Configure Filebeat

Add the following configuration to your `filebeat.yaml` file to start collecting log data.
```yaml
filebeat.inputs:
- type: filestream  
  enabled: true
  paths: /path/to/logs.log  
```


#### Step 4: Set up and start Filebeat

From the Filebeat installation directory, set the [index template](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/manage-data/data-store/templates) by running the command that aligns with your system:
<tab-set>
  <tab-item title="DEB">
    ```sh
    ./filebeat setup -e
    ```
  </tab-item>

  <tab-item title="RPM">
    ```sh
    ./filebeat setup -e
    ```
  </tab-item>

  <tab-item title="MacOS">
    ```sh
    PS > .\filebeat.exe setup -e
    ```
  </tab-item>

  <tab-item title="Linux">
    ```sh
    filebeat setup -e
    ```
  </tab-item>

  <tab-item title="Windows">
    ```sh
    filebeat setup -e
    ```
  </tab-item>
</tab-set>

From the Filebeat installation directory, start filebeat by running the command that aligns with your system:
<tab-set>
  <tab-item title="DEB">
    ```sh
    sudo service filebeat start
    ```

    <note>
      If you use an `init.d` script to start Filebeat, you can’t specify command line flags (see [Command reference](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/filebeat/command-line-options)). To specify flags, start Filebeat in the foreground.
    </note>
    Also see [Filebeat and systemd](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/filebeat/running-with-systemd).
  </tab-item>

  <tab-item title="RPM">
    ```sh
    sudo service filebeat start
    ```

    <note>
      If you use an `init.d` script to start Filebeat, you can’t specify command line flags (see [Command reference](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/filebeat/command-line-options)). To specify flags, start Filebeat in the foreground.
    </note>
    Also see [Filebeat and systemd](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/filebeat/running-with-systemd).
  </tab-item>

  <tab-item title="MacOS">
    ```sh
    ./filebeat -e
    ```
  </tab-item>

  <tab-item title="Linux">
    ```sh
    ./filebeat -e
    ```
  </tab-item>

  <tab-item title="Windows">
    ```sh
    PS C:\Program Files\filebeat> Start-Service filebeat
    ```
    By default, Windows log files are stored in `C:\ProgramData\filebeat\Logs`.
  </tab-item>
</tab-set>


#### Ingest logs with Elastic Agent

Add the custom logs integration to ingest and centrally manage your logs using Elastic Agent and Fleet:

#### Add the custom logs integration to your project

To add the custom logs integration to your project:
1. Find **Integrations** in the main menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/explore-analyze/find-and-organize/find-apps-and-objects).
2. Type `custom` in the search bar and select **Custom Logs**.
3. Click **Install Elastic Agent** at the bottom of the page, and follow the instructions for your system to install the Elastic Agent.
4. After installing the Elastic Agent, click **Save and continue** to configure the integration from the **Add Custom Logs integration** page.
5. Give your integration a meaningful name and description.
6. Add the **Log file path**. For example, `/var/log/your-logs.log`.
7. Click **Advanced options**.
8. In the **Processors** text box, add the following YAML configuration to add processors that enhance your data. Refer to [processors](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/filebeat/filtering-enhancing-data) to learn more.
   ```yaml
   processors:
     - add_host_metadata: \~
     - add_cloud_metadata: \~
     - add_docker_metadata: \~
     - add_kubernetes_metadata: \~
   ```
9. Under **Custom configurations**, add the following YAML configuration to collect data.
   ```yaml
     json:
       overwrite_keys: true 
       add_error_key: true 
       expand_keys: true 
       keys_under_root: true 
     fields_under_root: true 
     fields:
       service.name: your_service_name 
       service.version: your_service_version 
       service.environment: your_service_environment 
   ```
10. Give your agent policy a name. The agent policy defines the data your Elastic Agent collects.
11. Save your integration to add it to your deployment.


## View logs

Refer to the [Filter and aggregate logs](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/observability/logs/filter-aggregate-logs) documentation for more information on viewing and filtering your logs in Kibana.