Queues and data resiliency

As data flows through the event processing pipeline, Logstash can encounter situations that prevent it from delivering events to the configured output, such as unexpected data types or an abnormal Logstash shutdown.

Memory queue (MQ)

By default, Logstash uses in-memory bounded queues between pipeline stages (inputs → pipeline workers) to buffer events. Memory queues have limitations, but also offer benefits that make them a good choice for many users. If memory queues don't offer the resiliency you need, Logstash provides more options.

To guard against data loss and ensure that events flow through the pipeline without interruption, Logstash provides additional data resiliency features. These features are turned off by default. To use them, you must explicitly turn them on in the Logstash settings file.

Persistent queues (PQ)

Persistent queues (PQ) protect against data loss by storing events in an internal queue on disk.

Dead letter queues (DLQ)

Dead letter queues (DLQ) provide on-disk storage for events that Logstash is unable to process so that you can evaluate them. You can easily reprocess events in the dead letter queue by using the dead_letter_queue input plugin.

When you use Logstash to send data streams to Elasticsearch, you have an additional option for data resiliency: the Elasticsearch failure store. The Elasticsearch failure store for data streams offers Logstash users another alternative for handling events that can't be processed.

A failure store is a secondary set of indices inside a data stream that is dedicated to storing failed documents. When a data stream's failure store is turned on, failures are captured in a separate index and persisted to be analyzed later. Logstash offers the Dead Letter Queue (DLQ), but the failure store is likely to be a more practical option for most Elasticsearch users.

Refer to Failure store for details.