﻿---
title: Collecting log data with Filebeat
description: You can use Filebeat to monitor the Elasticsearch log files, collect log events, and ship them to the monitoring cluster. Your recent logs are visible...
url: https://www.elastic.co/elastic/docs-builder/docs/2996/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat
products:
  - Elasticsearch
applies_to:
  - Self-managed Elastic deployments: Generally available
---

# Collecting log data with Filebeat
You can use Filebeat to monitor the Elasticsearch log files, collect log events, and ship them to the monitoring cluster. Your recent logs are visible on the **Monitoring** page in Kibana.
<important>
  If you’re using Elastic Agent, do not deploy Filebeat for log collection. Instead, configure the Elasticsearch integration to collect logs.
</important>

1. Verify that Elasticsearch is running and that the monitoring cluster is ready to receive data from Filebeat.
   <tip>
   In production environments, we strongly recommend using a separate cluster (referred to as the *monitoring cluster*) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster.For more information, refer to [Monitoring in a production environment](https://www.elastic.co/elastic/docs-builder/docs/2996/deploy-manage/monitor/stack-monitoring/es-self-monitoring-prod).
   </tip>
2. Identify which logs you want to monitor.
   The Filebeat Elasticsearch module can handle [audit logs](https://www.elastic.co/elastic/docs-builder/docs/2996/deploy-manage/security/logging-configuration/logfile-audit-output), [deprecation logs](https://www.elastic.co/elastic/docs-builder/docs/2996/deploy-manage/monitor/logging-configuration/elasticsearch-deprecation-logs), [gc logs](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/2996/reference/elasticsearch/jvm-settings#gc-logging), [server logs](https://www.elastic.co/elastic/docs-builder/docs/2996/deploy-manage/monitor/logging-configuration/elasticsearch-log4j-configuration-self-managed), and [slow logs](https://www.elastic.co/elastic/docs-builder/docs/2996/deploy-manage/monitor/logging-configuration/slow-logs). For more information about the location of your Elasticsearch logs, refer to the [path.logs](/elastic/docs-builder/docs/2996/deploy-manage/deploy/self-managed/important-settings-configuration#path-settings) setting.
   <important>
   If there are both structured (`*.json`) and unstructured (plain text) versions of the logs, you must use the structured logs. Otherwise, they might not appear in the appropriate context in Kibana.
   </important>
3. [Install Filebeat](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/2996/reference/beats/filebeat/filebeat-installation-configuration) on the Elasticsearch nodes that contain logs that you want to monitor.
4. Identify where to send the log data.
   For example, specify Elasticsearch output information for your monitoring cluster in the Filebeat configuration file (`filebeat.yml`):
   ```yaml
   output.elasticsearch:
     # Array of hosts to connect to.
     hosts: ["<ES_MONITORING_HOST1_URL>:9200", "<ES_MONITORING_HOST2_URL>:9200"] 

     # Optional protocol and basic auth credentials.
     #protocol: "https"
     #username: "elastic"
     #password: "changeme"
   ```
   If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a `hosts` setting like `https://es-mon-1:9200`.
   <important>
   The Elasticsearch monitoring features use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one [ingest node](https://www.elastic.co/elastic/docs-builder/docs/2996/manage-data/ingest/transform-enrich/ingest-pipelines).
   </important>
   If Elasticsearch security features are enabled on the monitoring cluster, you must provide a valid user ID and password so that Filebeat can send metrics successfully.
   For more information about these configuration options, see [Configure the Elasticsearch output](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/2996/reference/beats/filebeat/elasticsearch-output).
5. Optional: Identify where to visualize the data.
   Filebeat provides example Kibana dashboards, visualizations and searches. To load the dashboards into the appropriate Kibana instance, specify the `setup.kibana` information in the Filebeat configuration file (`filebeat.yml`) on each node:
   ```yaml
   setup.kibana:
     host: "localhost:5601"
     #username: "my_kibana_user"
     #password: "YOUR_PASSWORD"
   ```
   <tip>
   In production environments, we strongly recommend using a dedicated Kibana instance for your monitoring cluster.
   </tip>
   If security features are enabled, you must provide a valid user ID and password so that Filebeat can connect to Kibana:
   1. Create a user on the monitoring cluster that has the [`kibana_admin` built-in role](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/2996/reference/elasticsearch/roles#built-in-roles-kibana-admin) or equivalent privileges.
2. Add the `username` and `password` settings to the Elasticsearch output information in the Filebeat configuration file. The example shows a hard-coded password, but you should store sensitive values in the [secrets keystore](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/2996/reference/beats/filebeat/keystore).
   See [Configure the Kibana endpoint](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/2996/reference/beats/filebeat/setup-kibana-endpoint).
6. Enable the Elasticsearch module and set up the initial Filebeat environment on each node.
   For example:
   ```sh
   filebeat modules enable elasticsearch
   filebeat setup -e
   ```
   For more information, see [Elasticsearch module](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/2996/reference/beats/filebeat/filebeat-module-elasticsearch).
7. Configure the Elasticsearch module in Filebeat on each node.
   If the logs that you want to monitor aren’t in the default location, set the appropriate path variables in the `modules.d/elasticsearch.yml` file. See [Configure the Elasticsearch module](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/2996/reference/beats/filebeat/filebeat-module-elasticsearch#configuring-elasticsearch-module).
   <important>
   If there are JSON logs, configure the `var.paths` settings to point to them instead of the plain text logs.
   </important>
8. [Start Filebeat](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/2996/reference/beats/filebeat/filebeat-starting) on each node.
   <note>
   Depending on how you’ve installed Filebeat, you might see errors related to file ownership or permissions when you try to run Filebeat modules. See [Config file ownership and permissions](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/2996/reference/beats/libbeat/config-file-permissions).
   </note>
9. Check whether the appropriate indices exist on the monitoring cluster.
   For example, use the [cat indices](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices) command to verify that there are new `filebeat-*` indices.
   <tip>
   If you want to use the **Monitoring** UI in Kibana, there must also be `.monitoring-*` indices. Those indices are generated when you collect metrics about Elastic Stack products. For example, see [Collecting monitoring data with Metricbeat](https://www.elastic.co/elastic/docs-builder/docs/2996/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat).
   </tip>
10. [View the monitoring data in Kibana](https://www.elastic.co/elastic/docs-builder/docs/2996/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data).