﻿---
title: Kafka output plugin v10.7.0
description: A component of the kafka integration plugin, Integration version: v10.7.0, Released on: 2020-12-03, Changelog. For other versions, see the overview list...
url: https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/vpr/v10-7-0-plugins-outputs-kafka
---

# Kafka output plugin v10.7.0
- A component of the [kafka integration plugin](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/vpr/integration-kafka-index)
- Integration version: v10.7.0
- Released on: 2020-12-03
- [Changelog](https://github.com/logstash-plugins/logstash-integration-kafka/blob/v10.7.0/CHANGELOG.md)

For other versions, see the [overview list](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/vpr/output-kafka-index).
To learn more about Logstash, see the [Logstash Reference](https://www.elastic.co/guide/en/logstash/current/index.html).

## Getting help

For questions about the plugin, open a topic in the [Discuss](http://discuss.elastic.co) forums. For bugs or feature requests, open an issue in [Github](https://github.com/logstash-plugins/logstash-integration-kafka). For the list of Elastic supported plugins, please consult the [Elastic Support Matrix](https://www.elastic.co/support/matrix#matrix_logstash_plugins).

## Description

Write events to a Kafka topic.
This plugin uses Kafka Client 2.4. For broker compatibility, see the official [Kafka compatibility reference](https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix). If the linked compatibility wiki is not up-to-date, please contact Kafka support/community to confirm compatibility.
If you require features not yet available in this plugin (including client version upgrades), please file an issue with details about what you need.
This output supports connecting to Kafka over:
- SSL (requires plugin version 3.0.0 or later)
- Kerberos SASL (requires plugin version 5.1.0 or later)

By default security is disabled but can be turned on as needed.
The only required configuration is the topic_id.
The default codec is plain. Logstash will encode your events with not only the message field but also with a timestamp and hostname.
If you want the full content of your events to be sent as json, you should set the codec in the output configuration like this:
```
    output {
      kafka {
        codec => json
        topic_id => "mytopic"
      }
    }
```

For more information see 
Kafka producer configuration: 

## Kafka Output Configuration Options

This plugin supports the following configuration options plus the [Common options](#v10.7.0-plugins-outputs-kafka-common-options) described later.
Some of these options map to a Kafka option. Defaults usually reflect the Kafka default setting, and might change if Kafka’s producer defaults change. See the  for more details.

| Setting                                                                                                                                                | Input type                                                                                                                              | Required |
|--------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|----------|
| [`acks`](#v10.7.0-plugins-outputs-kafka-acks)`acks`                                                                                                    | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string, one of `["0", "1", "all"]`                                  | No       |
| [`batch_size`](#v10.7.0-plugins-outputs-kafka-batch_size)`batch_size`                                                                                  | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`bootstrap_servers`](#v10.7.0-plugins-outputs-kafka-bootstrap_servers)`bootstrap_servers`                                                             | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`buffer_memory`](#v10.7.0-plugins-outputs-kafka-buffer_memory)`buffer_memory`                                                                         | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`client_dns_lookup`](#v10.7.0-plugins-outputs-kafka-client_dns_lookup)`client_dns_lookup`                                                             | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`client_id`](#v10.7.0-plugins-outputs-kafka-client_id)`client_id`                                                                                     | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`compression_type`](#v10.7.0-plugins-outputs-kafka-compression_type)`compression_type`                                                                | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string, one of `["none", "gzip", "snappy", "lz4"]`                  | No       |
| [`jaas_path`](#v10.7.0-plugins-outputs-kafka-jaas_path)`jaas_path`                                                                                     | a valid filesystem path                                                                                                                 | No       |
| [`kerberos_config`](#v10.7.0-plugins-outputs-kafka-kerberos_config)`kerberos_config`                                                                   | a valid filesystem path                                                                                                                 | No       |
| [`key_serializer`](#v10.7.0-plugins-outputs-kafka-key_serializer)`key_serializer`                                                                      | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`linger_ms`](#v10.7.0-plugins-outputs-kafka-linger_ms)`linger_ms`                                                                                     | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`max_request_size`](#v10.7.0-plugins-outputs-kafka-max_request_size)`max_request_size`                                                                | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`message_key`](#v10.7.0-plugins-outputs-kafka-message_key)`message_key`                                                                               | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`metadata_fetch_timeout_ms`](#v10.7.0-plugins-outputs-kafka-metadata_fetch_timeout_ms)`metadata_fetch_timeout_ms`                                     | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`metadata_max_age_ms`](#v10.7.0-plugins-outputs-kafka-metadata_max_age_ms)`metadata_max_age_ms`                                                       | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`partitioner`](#v10.7.0-plugins-outputs-kafka-partitioner)`partitioner`                                                                               | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`receive_buffer_bytes`](#v10.7.0-plugins-outputs-kafka-receive_buffer_bytes)`receive_buffer_bytes`                                                    | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`reconnect_backoff_ms`](#v10.7.0-plugins-outputs-kafka-reconnect_backoff_ms)`reconnect_backoff_ms`                                                    | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`request_timeout_ms`](#v10.7.0-plugins-outputs-kafka-request_timeout_ms)`request_timeout_ms`                                                          | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`retries`](#v10.7.0-plugins-outputs-kafka-retries)`retries`                                                                                           | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`retry_backoff_ms`](#v10.7.0-plugins-outputs-kafka-retry_backoff_ms)`retry_backoff_ms`                                                                | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`sasl_jaas_config`](#v10.7.0-plugins-outputs-kafka-sasl_jaas_config)`sasl_jaas_config`                                                                | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`sasl_kerberos_service_name`](#v10.7.0-plugins-outputs-kafka-sasl_kerberos_service_name)`sasl_kerberos_service_name`                                  | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`sasl_mechanism`](#v10.7.0-plugins-outputs-kafka-sasl_mechanism)`sasl_mechanism`                                                                      | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`security_protocol`](#v10.7.0-plugins-outputs-kafka-security_protocol)`security_protocol`                                                             | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string, one of `["PLAINTEXT", "SSL", "SASL_PLAINTEXT", "SASL_SSL"]` | No       |
| [`send_buffer_bytes`](#v10.7.0-plugins-outputs-kafka-send_buffer_bytes)`send_buffer_bytes`                                                             | [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)number                                                              | No       |
| [`ssl_endpoint_identification_algorithm`](#v10.7.0-plugins-outputs-kafka-ssl_endpoint_identification_algorithm)`ssl_endpoint_identification_algorithm` | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`ssl_key_password`](#v10.7.0-plugins-outputs-kafka-ssl_key_password)`ssl_key_password`                                                                | [password](/elastic/logstash-docs-md/tree/main/lsr/value-types#password)password                                                        | No       |
| [`ssl_keystore_location`](#v10.7.0-plugins-outputs-kafka-ssl_keystore_location)`ssl_keystore_location`                                                 | a valid filesystem path                                                                                                                 | No       |
| [`ssl_keystore_password`](#v10.7.0-plugins-outputs-kafka-ssl_keystore_password)`ssl_keystore_password`                                                 | [password](/elastic/logstash-docs-md/tree/main/lsr/value-types#password)password                                                        | No       |
| [`ssl_keystore_type`](#v10.7.0-plugins-outputs-kafka-ssl_keystore_type)`ssl_keystore_type`                                                             | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`ssl_truststore_location`](#v10.7.0-plugins-outputs-kafka-ssl_truststore_location)`ssl_truststore_location`                                           | a valid filesystem path                                                                                                                 | No       |
| [`ssl_truststore_password`](#v10.7.0-plugins-outputs-kafka-ssl_truststore_password)`ssl_truststore_password`                                           | [password](/elastic/logstash-docs-md/tree/main/lsr/value-types#password)password                                                        | No       |
| [`ssl_truststore_type`](#v10.7.0-plugins-outputs-kafka-ssl_truststore_type)`ssl_truststore_type`                                                       | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |
| [`topic_id`](#v10.7.0-plugins-outputs-kafka-topic_id)`topic_id`                                                                                        | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | Yes      |
| [`value_serializer`](#v10.7.0-plugins-outputs-kafka-value_serializer)`value_serializer`                                                                | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string                                                              | No       |

Also see [Common options](#v10.7.0-plugins-outputs-kafka-common-options) for a list of options supported by all output plugins.

### `acks`

- Value can be any of: `0`, `1`, `all`
- Default value is `"1"`

The number of acknowledgments the producer requires the leader to have received before considering a request complete.
`acks=0`. The producer will not wait for any acknowledgment from the server.
`acks=1`. The leader will write the record to its local log, but will respond without waiting for full acknowledgement from all followers.
`acks=all`. The leader will wait for the full set of in-sync replicas before acknowledging the record.

### `batch_size`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- Default value is `16384`.

The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes.

### `bootstrap_servers`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- Default value is `"localhost:9092"`

This is for bootstrapping and the producer will only use it for getting metadata (topics, partitions and replicas). The socket connections for sending the actual data will be established based on the broker information returned in the metadata. The format is `host1:port1,host2:port2`, and the list can be a subset of brokers or a VIP pointing to a subset of brokers.

### `buffer_memory`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- Default value is `33554432` (32MB).

The total bytes of memory the producer can use to buffer records waiting to be sent to the server.

### `client_dns_lookup`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- Valid options are `use_all_dns_ips`, `resolve_canonical_bootstrap_servers_only`, `default`
- Default value is `"default"`

Controls how DNS lookups are done. If set to `use_all_dns_ips`, Logstash tries all IP addresses returned for a hostname before failing the connection. If set to `resolve_canonical_bootstrap_servers_only`, each entry will be resolved and expanded into a list of canonical names.

### `client_id`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- There is no default value for this setting.

The id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included with the request

### `compression_type`

- Value can be any of: `none`, `gzip`, `snappy`, `lz4`
- Default value is `"none"`

The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, or lz4.

### `jaas_path`

- Value type is [path](/elastic/logstash-docs-md/tree/main/lsr/value-types#path)
- There is no default value for this setting.

The Java Authentication and Authorization Service (JAAS) API supplies user authentication and authorization services for Kafka. This setting provides the path to the JAAS file. Sample JAAS file for Kafka client:
```
KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useTicketCache=true
  renewTicket=true
  serviceName="kafka";
  };
```

Please note that specifying `jaas_path` and `kerberos_config` in the config file will add these to the global JVM system properties. This means if you have multiple Kafka inputs, all of them would be sharing the same `jaas_path` and `kerberos_config`. If this is not desirable, you would have to run separate instances of Logstash on different JVM instances.

### `kerberos_config`

- Value type is [path](/elastic/logstash-docs-md/tree/main/lsr/value-types#path)
- There is no default value for this setting.

Optional path to kerberos config file. This is krb5.conf style as detailed in 

### `key_serializer`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- Default value is `"org.apache.kafka.common.serialization.StringSerializer"`

Serializer class for the key of the message

### `linger_ms`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- Default value is `0`

The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay—that is, rather than immediately sending out a record the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together.

### `max_request_size`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- Default value is `1048576` (1MB).

The maximum size of a request

### `message_key`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- There is no default value for this setting.

The key for the message.

### `metadata_fetch_timeout_ms`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- Default value is `60000` milliseconds (60 seconds).

The timeout setting for initial metadata request to fetch topic metadata.

### `metadata_max_age_ms`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- Default value is `300000` milliseconds (5 minutes).

The max time in milliseconds before a metadata refresh is forced.

### `partitioner`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- There is no default value for this setting.

The default behavior is to hash the `message_key` of an event to get the partition. When no message key is present, the plugin picks a partition in a round-robin fashion.
Available options for choosing a partitioning strategy are as follows:
- `default` use the default partitioner as described above
- `round_robin` distributes writes to all partitions equally, regardless of `message_key`
- `uniform_sticky` sticks to a partition for the duration of a batch than randomly picks a new one


### `receive_buffer_bytes`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- Default value is `32768` (32KB).

The size of the TCP receive buffer to use when reading data

### `reconnect_backoff_ms`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- Default value is `50`.

The amount of time to wait before attempting to reconnect to a given host when a connection fails.

### `request_timeout_ms`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- Default value is `40000` milliseconds (40 seconds).

The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

### `retries`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- There is no default value for this setting.

The default retry behavior is to retry until successful. To prevent data loss, the use of this setting is discouraged.
If you choose to set `retries`, a value greater than zero will cause the client to only retry a fixed number of times. This will result in data loss if a transport fault exists for longer than your retry count (network outage, Kafka down, etc).
A value less than zero is a configuration error.
Starting with version 10.5.0, this plugin will only retry exceptions that are a subclass of [RetriableException](https://kafka.apache.org/24/javadoc/org/apache/kafka/common/errors/RetriableException.html) and [InterruptException](https://kafka.apache.org/24/javadoc/org/apache/kafka/common/errors/InterruptException.html). If producing a message throws any other exception, an error is logged and the message is dropped without retrying. This prevents the Logstash pipeline from hanging indefinitely.
In versions prior to 10.5.0, any exception is retried indefinitely unless the `retries` option is configured.

### `retry_backoff_ms`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- Default value is `100` milliseconds.

The amount of time to wait before attempting to retry a failed produce request to a given topic partition.

### `sasl_jaas_config`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- There is no default value for this setting.

JAAS configuration setting local to this plugin instance, as opposed to settings using config file configured using `jaas_path`, which are shared across the JVM. This allows each plugin instance to have its own configuration.
If both `sasl_jaas_config` and `jaas_path` configurations are set, the setting here takes precedence.
Example (setting for Azure Event Hub):
```
    output {
      kafka {
        sasl_jaas_config => "org.apache.kafka.common.security.plain.PlainLoginModule required username='auser'  password='apassword';"
      }
    }
```


### `sasl_kerberos_service_name`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- There is no default value for this setting.

The Kerberos principal name that Kafka broker runs as. This can be defined either in Kafka’s JAAS config or in Kafka’s config.

### `sasl_mechanism`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- Default value is `"GSSAPI"`

[SASL mechanism](http://kafka.apache.org/documentation.html#security_sasl) used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.

### `security_protocol`

- Value can be any of: `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, `SASL_SSL`
- Default value is `"PLAINTEXT"`

Security protocol to use, which can be either of PLAINTEXT,SSL,SASL_PLAINTEXT,SASL_SSL

### `send_buffer_bytes`

- Value type is [number](/elastic/logstash-docs-md/tree/main/lsr/value-types#number)
- Default value is `131072` (128KB).

The size of the TCP send buffer to use when sending data.

### `ssl_endpoint_identification_algorithm`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- Default value is `"https"`

The endpoint identification algorithm, defaults to `"https"`. Set to empty string `""` to disable

### `ssl_key_password`

- Value type is [password](/elastic/logstash-docs-md/tree/main/lsr/value-types#password)
- There is no default value for this setting.

The password of the private key in the key store file.

### `ssl_keystore_location`

- Value type is [path](/elastic/logstash-docs-md/tree/main/lsr/value-types#path)
- There is no default value for this setting.

If client authentication is required, this setting stores the keystore path.

### `ssl_keystore_password`

- Value type is [password](/elastic/logstash-docs-md/tree/main/lsr/value-types#password)
- There is no default value for this setting.

If client authentication is required, this setting stores the keystore password

### `ssl_keystore_type`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- There is no default value for this setting.

The keystore type.

### `ssl_truststore_location`

- Value type is [path](/elastic/logstash-docs-md/tree/main/lsr/value-types#path)
- There is no default value for this setting.

The JKS truststore path to validate the Kafka broker’s certificate.

### `ssl_truststore_password`

- Value type is [password](/elastic/logstash-docs-md/tree/main/lsr/value-types#password)
- There is no default value for this setting.

The truststore password

### `ssl_truststore_type`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- There is no default value for this setting.

The truststore type.

### `topic_id`

- This is a required setting.
- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- There is no default value for this setting.

The topic to produce messages to

### `value_serializer`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- Default value is `"org.apache.kafka.common.serialization.StringSerializer"`

Serializer class for the value of the message

## Common options

These configuration options are supported by all output plugins:

| Setting                                                                        | Input type                                                                    | Required |
|--------------------------------------------------------------------------------|-------------------------------------------------------------------------------|----------|
| [`codec`](#v10.7.0-plugins-outputs-kafka-codec)`codec`                         | [codec](/elastic/logstash-docs-md/tree/main/lsr/value-types#codec)codec       | No       |
| [`enable_metric`](#v10.7.0-plugins-outputs-kafka-enable_metric)`enable_metric` | [boolean](/elastic/logstash-docs-md/tree/main/lsr/value-types#boolean)boolean | No       |
| [`id`](#v10.7.0-plugins-outputs-kafka-id)`id`                                  | [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)string    | No       |


### `codec`

- Value type is [codec](/elastic/logstash-docs-md/tree/main/lsr/value-types#codec)
- Default value is `"plain"`

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output without needing a separate filter in your Logstash pipeline.

### `enable_metric`

- Value type is [boolean](/elastic/logstash-docs-md/tree/main/lsr/value-types#boolean)
- Default value is `true`

Disable or enable metric logging for this specific plugin instance. By default we record all the metrics we can, but you can disable metrics collection for a specific plugin.

### `id`

- Value type is [string](/elastic/logstash-docs-md/tree/main/lsr/value-types#string)
- There is no default value for this setting.

Add a unique `ID` to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type. For example, if you have 2 kafka outputs. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.
```
output {
  kafka {
    id => "my_plugin_id"
  }
}
```