Configure the Elasticsearch output

The Elasticsearch output sends events directly to Elasticsearch by using the Elasticsearch HTTP API.

Compatibility: This output works with all compatible versions of Elasticsearch. See the Elastic Support Matrix.

This example configures an Elasticsearch output called default in the elastic-agent.yml file:

outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    username: elastic
    password: changeme

This example is similar to the previous one, except that it uses the recommended token-based (API key) authentication:

outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    api_key: "my_api_key"

The elasticsearch output type supports the following settings, grouped by category. Many of these settings have sensible defaults that allow you to run Elastic Agent with minimal configuration.

• Commonly used settings
• Authentication settings
• Compatibility setting
• Data parsing, filtering, and manipulation settings
• HTTP settings
• Memory queue settings
• Performance tuning settings

When sending data to a secured cluster through the elasticsearch output, Elastic Agent can use any of the following authentication methods:

• Basic authentication credentials
• Token-based (API key) authentication
• Public Key Infrastructure (PKI) certificates
• Kerberos

outputs:
  default:
    type: elasticsearch
    hosts: ["https://myEShost:9200"]
    username: "your-username"
    password: "your-password"

outputs:
  default:
    type: elasticsearch
    hosts: ["https://myEShost:9200"]
    api_key: "KnR6yE41RrSowb0kQ0HWoA"

outputs:
  default:
    type: elasticsearch
    hosts: ["https://myEShost:9200"]
    ssl.certificate: "/etc/pki/client/cert.pem"
    ssl.key: "/etc/pki/client/cert.key"

For a list of available settings, refer to SSL/TLS, specifically the settings under Table 7, Common configuration options and Table 8, Client configuration options.

The following encryption types are supported:

• aes128-cts-hmac-sha1-96
• aes128-cts-hmac-sha256-128
• aes256-cts-hmac-sha1-96
• aes256-cts-hmac-sha384-192
• des3-cbc-sha1-kd
• rc4-hmac

Example output config with Kerberos password-based authentication:

outputs:
  default:
    type: elasticsearch
    hosts: ["http://my-elasticsearch.elastic.co:9200"]
    kerberos.auth_type: password
    kerberos.username: "elastic"
    kerberos.password: "changeme"
    kerberos.config_path: "/etc/krb5.conf"
    kerberos.realm: "ELASTIC.CO"

The service principal name for the Elasticsearch instance is constructed from these options. Based on this configuration, the name would be:

HTTP/my-elasticsearch.elastic.co@ELASTIC.CO

Settings used to parse, filter, and transform data.

Settings that modify the HTTP requests sent to Elasticsearch.

The memory queue keeps all events in memory.

The memory queue waits for the output to acknowledge or drop events. If the queue is full, no new events can be inserted into the memory queue. Only after the signal from the output will the queue free up space for more events to be accepted.

The memory queue is controlled by the parameters flush.min_events and flush.timeout. flush.min_events gives a limit on the number of events that can be included in a single batch, and flush.timeout specifies how long the queue should wait to completely fill an event request. If the output supports a bulk_max_size parameter, the maximum batch size will be the smaller of bulk_max_size and flush.min_events.

flush.min_events is a legacy parameter, and new configurations should prefer to control batch size with bulk_max_size. As of 8.13, there is never a performance advantage to limiting batch size with flush.min_events instead of bulk_max_size.

In synchronous mode, an event request is always filled as soon as events are available, even if there are not enough events to fill the requested batch. This is useful when latency must be minimized. To use synchronous mode, set flush.timeout to 0.

For backwards compatibility, synchronous mode can also be activated by setting flush.min_events to 0 or 1. In this case, batch size will be capped at 1/2 the queue capacity.

In asynchronous mode, an event request will wait up to the specified timeout to try and fill the requested batch completely. If the timeout expires, the queue returns a partial batch with all available events. To use asynchronous mode, set flush.timeout to a positive duration, for example 5s.

This sample configuration forwards events to the output when there are enough events to fill the output’s request (usually controlled by bulk_max_size, and limited to at most 512 events by flush.min_events), or when events have been waiting for

queue.mem.events: 4096
queue.mem.flush.min_events: 512
queue.mem.flush.timeout: 5s

Settings that may affect performance when sending data through the Elasticsearch output.

Use the preset option to automatically configure the group of performance tuning settings to optimize for throughput, scale, latency, or you can select a balanced set of performance specifications.

The performance tuning preset values take precedence over any settings that may be defined separately. If you want to change any setting, set preset to custom and specify the performance tuning settings individually.