Logstash output

The Logstash output uses an internal protocol to send events directly to Logstash over TCP. Logstash provides additional parsing, transformation, and routing of data collected by Elastic Agent.

Compatibility: This output works with all compatible versions of Logstash. Refer to the Elastic Support Matrix.

This example configures a Logstash output called default in the elastic-agent.yml file:

outputs:
  default:
    type: logstash
    hosts: ["127.0.0.1:5044"] 1

1. The Logstash server and the port (5044) where Logstash is configured to listen for incoming Elastic Agent connections.

To receive the events in Logstash, you also need to create a Logstash configuration pipeline. The Logstash configuration pipeline listens for incoming Elastic Agent connections, processes received events, and then sends the events to Elasticsearch.

The following Logstash pipeline definition example configures a pipeline that listens on port 5044 for incoming Elastic Agent connections and routes received events to Elasticsearch.

input {
  elastic_agent {
    port => 5044
    enrich => none 1
    ssl => true
    ssl_certificate_authorities => ["<ca_path>"]
    ssl_certificate => "<server_cert_path>"
    ssl_key => "<server_cert_key_in_pkcs8>"
    ssl_verify_mode => "force_peer"
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"] 2
    # cloud_id => "..."
    data_stream => "true"
    api_key => "<api_key>" 3
    data_stream => true
    ssl => true
    # cacert => "<elasticsearch_ca_path>"
  }
}

1. Do not modify the events' schema.
2. The Elasticsearch server and the port (9200) where Elasticsearch is running.
3. The API Key used by Logstash to ship data to the destination data streams.

For more information about configuring Logstash, refer to Configuring Logstash and Elastic Agent input plugin.

The logstash output supports the following settings, grouped by category. Many of these settings have sensible defaults that allow you to run Elastic Agent with minimal configuration.

• Commonly used settings
• Authentication settings
• Memory queue settings
• Performance tuning settings

When sending data to a secured cluster through the logstash output, Elastic Agent can use SSL/TLS. For a list of available settings, refer to SSL/TLS, specifically the settings under Table 7, Common configuration options and Table 8, Client configuration options.

For more information, refer to Configure SSL/TLS for the Logstash output.

The memory queue keeps all events in memory.

The memory queue waits for the output to acknowledge or drop events. If the queue is full, no new events can be inserted into the memory queue. Only after the signal from the output will the queue free up space for more events to be accepted.

The memory queue is controlled by the parameters flush.min_events and flush.timeout. flush.min_events gives a limit on the number of events that can be included in a single batch, and flush.timeout specifies how long the queue should wait to completely fill an event request. If the output supports a bulk_max_size parameter, the maximum batch size will be the smaller of bulk_max_size and flush.min_events.

flush.min_events is a legacy parameter, and new configurations should prefer to control batch size with bulk_max_size. As of 8.13, there is never a performance advantage to limiting batch size with flush.min_events instead of bulk_max_size.

In synchronous mode, an event request is always filled as soon as events are available, even if there are not enough events to fill the requested batch. This is useful when latency must be minimized. To use synchronous mode, set flush.timeout to 0.

For backwards compatibility, synchronous mode can also be activated by setting flush.min_events to 0 or 1. In this case, batch size will be capped at 1/2 the queue capacity.

In asynchronous mode, an event request will wait up to the specified timeout to try and fill the requested batch completely. If the timeout expires, the queue returns a partial batch with all available events. To use asynchronous mode, set flush.timeout to a positive duration, for example 5s.

This sample configuration forwards events to the output when there are enough events to fill the output’s request (usually controlled by bulk_max_size, and limited to at most 512 events by flush.min_events), or when events have been waiting for 5s without filling the requested size:f 512 events are available or the oldest available event has been waiting for 5s in the queue:

queue.mem.events: 4096
queue.mem.flush.min_events: 512
queue.mem.flush.timeout: 5s

Settings that may affect performance.