---
title: Kafka module
description: This is the Kafka module. The default metricsets are consumergroup and partition. If authorization is configured in the Kafka cluster, the following ACLs...
url: https://www.elastic.co/elastic/docs-builder/docs/3016/reference/beats/metricbeat/metricbeat-module-kafka
products:
  - Beats
  - Metricbeat
applies_to:
  - Elastic Cloud Serverless: Generally available
  - Elastic Stack: Generally available
---

# Kafka module
<admonition title="Prefer to use Elastic Agent for this use case?">
  Refer to the [Elastic Integrations documentation](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/kafka).
  <dropdown title="Learn more">
    Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, forward data from remote services or hardware, and more. Refer to the documentation for a detailed [comparison of Beats and Elastic Agent](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/fleet).
  </dropdown>
</admonition>

This is the Kafka module.
The default metricsets are `consumergroup` and `partition`.
If authorization is configured in the Kafka cluster, the following ACLs are required for the Metricbeat user:
- READ Topic, for the topics to be monitored
- DESCRIBE Group, for the groups to be monitored

For example, if the `stats` user is being used for Metricbeat, to monitor all topics and all consumer groups, ACLS can be granted with the following commands:
```shell
kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:stats --operation Read --topic '*'
kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:stats --operation Describe --group '*'
```


## Compatibility

This module is tested with Kafka 0.10.2.1, 1.1.0, 2.1.1, 2.2.2, 3.6.0 and 4.0.0.
The Broker, Producer, Consumer metricsets require [Jolokia](https://www.elastic.co/elastic/docs-builder/docs/3016/reference/beats/metricbeat/metricbeat-module-jolokia) to fetch JMX metrics. Refer to the link for Jolokia’s compatibility notes.

## Usage

The Broker, Producer, Consumer metricsets require [Jolokia](https://www.elastic.co/elastic/docs-builder/docs/3016/reference/beats/metricbeat/metricbeat-module-jolokia) to fetch JMX metrics. Refer to those Metricsets' documentation about how to use Jolokia.

## Dashboard

The Kafka module comes with a predefined dashboard. For example:
![metricbeat kafka dashboard](https://www.elastic.co/elastic/docs-builder/docs/3016/reference/beats/metricbeat/images/metricbeat_kafka_dashboard.png)

## Example configuration

The Kafka module supports the standard configuration options that are described in [Modules](https://www.elastic.co/elastic/docs-builder/docs/3016/reference/beats/metricbeat/configuration-metricbeat). Here is an example configuration:
```yaml
metricbeat.modules:
# Kafka metrics collected using the Kafka protocol
- module: kafka
  #metricsets:
  #  - partition
  #  - consumergroup
  period: 10s
  hosts: ["localhost:9092"]

  #client_id: metricbeat
  #retries: 3
  #backoff: 250ms

  # List of Topics to query metadata for. If empty, all topics will be queried.
  #topics: []

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

  # Client Certificate Passphrase (in case your Client Certificate Key is encrypted)
  #ssl.key_passphrase: "yourKeyPassphrase"

  # SASL authentication
  #username: ""
  #password: ""

  # SASL authentication mechanism used. Can be one of PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512.
  # Defaults to PLAIN when `username` and `password` are configured.
  #sasl.mechanism: ''

# Metrics collected from a Kafka broker using Jolokia
#- module: kafka
#  metricsets:
#    - broker
#  period: 10s
#  hosts: ["localhost:8779"]

# Metrics collected from a Java Kafka consumer using Jolokia
#- module: kafka
#  metricsets:
#    - consumer
#  period: 10s
#  hosts: ["localhost:8774"]

# Metrics collected from a Java Kafka producer using Jolokia
#- module: kafka
#  metricsets:
#    - producer
#  period: 10s
#  hosts: ["localhost:8775"]
```


## Metricsets

The following metricsets are available:
- [broker](https://www.elastic.co/elastic/docs-builder/docs/3016/reference/beats/metricbeat/metricbeat-metricset-kafka-broker)  <applies-to>Elastic Stack: Beta</applies-to>
- [consumer](https://www.elastic.co/elastic/docs-builder/docs/3016/reference/beats/metricbeat/metricbeat-metricset-kafka-consumer)  <applies-to>Elastic Stack: Beta</applies-to>
- [consumergroup](https://www.elastic.co/elastic/docs-builder/docs/3016/reference/beats/metricbeat/metricbeat-metricset-kafka-consumergroup)
- [partition](https://www.elastic.co/elastic/docs-builder/docs/3016/reference/beats/metricbeat/metricbeat-metricset-kafka-partition)
- [producer](https://www.elastic.co/elastic/docs-builder/docs/3016/reference/beats/metricbeat/metricbeat-metricset-kafka-producer)  <applies-to>Elastic Stack: Beta</applies-to>