﻿---
title: Log rotation results in lost or duplicate events
description: Filebeat supports reading from rotating log files, including GZIP compressed files. However, some log rotation strategies can result in lost or duplicate...
url: https://www.elastic.co/elastic/docs-builder/docs/2620/reference/beats/filebeat/file-log-rotation
products:
  - Beats
  - Filebeat
applies_to:
  - Elastic Stack: Generally available in 9.0+
---

# Log rotation results in lost or duplicate events
Filebeat supports reading from rotating log files, [including GZIP compressed files](/elastic/docs-builder/docs/2620/reference/beats/filebeat/filebeat-input-filestream#reading-gzip-files). However, some log rotation strategies can result in lost or duplicate events when using Filebeat to forward messages. To avoid this issue:
- **Be careful when changing the default file identity of the `filestream` input**
  By default, the file identity of the `filestream` input is set to `fingerprint`, which identifies files using file fingerprints produced by the scanner component of the `filestream` input. Changing the file identity configuration may result in duplicated events in the output.
- **Make sure Filebeat is configured to read from all rotated logs**
  To avoid missing events from a rotated file, configure the input to read from the log file and all the rotated files.
  When an input log file is moved or renamed during log rotation, Filebeat is able to recognize that the file has already been read. After the file is rotated, a new log file is created, and the application continues logging. If the rotated log are included in configuration paths, Filebeat picks up the new file during the next scan.
  For examples, refer to [Example configurations](#log-rotate-example).
- **Avoid log rotation strategies that copy and truncate log files**
  Strategies that copy and truncate the input log file can result in lost events if lines are written to the log file after it’s copied, but before it’s truncated.
  <note>
  In Filebeat versions prior to 9.0.0, using such strategies can also result in the duplication of events. In those versions, Filebeat identifies files by inode and device ID because the [file identity configuration](/elastic/docs-builder/docs/2620/reference/beats/filebeat/filebeat-input-filestream#filebeat-input-filestream-file-identity) of the `filestream` input is set to `native` by default. During log rotation, lines that Filebeat has already processed are moved to a new file. If the file identity is set to `native`, when Filebeat encounters the new file, it starts reading it from the beginning because the previous state information (the offset and read timestamp) is associated with the inode and device ID of the old file.
  </note>

If you’re using Windows, also check [More about log rotation on Windows](#log-rotation-windows).

## Example configurations

This section shows a typical configuration for logrotate, a popular tool for doing log rotation on Linux, followed by a Filebeat configuration that reads all the rotated logs.

### logrotate.conf

In this example, Filebeat reads web server log. The logs are rotated every day, and the new file is created with the specified permissions.
```yaml
/var/log/my-server/my-server.log {
    daily
    missingok
    rotate 7
    notifempty
    create 0640 www-data www-data
}
```


### filebeat.yml

In this example, Filebeat is configured to read all log files to make sure it does not miss any events.
```yaml
filebeat.inputs:
- type: filestream
  id: my-server-filestream-id
  paths:
  - /var/log/my-server/my-server.log*
```


## More about log rotation on Windows

On Windows, log rotation schemes that delete old files and rename newer files to old filenames might get blocked if the old files are being processed by Filebeat. This happens because Windows does not delete files and file metadata until the last process has closed the file. Unlike most *nix filesystems, a Windows filename cannot be reused until all processes accessing the file have closed the deleted file.
To avoid this problem, use dates in rotated filenames. The file will never be renamed to an older filename, and the log writer and log rotator will always be able to open the file. This approach also highly reduces the chance of log writing, rotation, and collection interfering with each other.
Because log rotation is typically handled by the logging application, we are not providing an example configuration for Windows.
Also read [Open file handlers cause issues with Windows file rotation](https://www.elastic.co/elastic/docs-builder/docs/2620/reference/beats/filebeat/windows-file-rotation).