Elastic Agent to Logstash to Kafka to Kafka ES Sink to Elasticsearch: Kafka as middleware message queue

Ingest model

Control path: Elastic Agent to Fleet to Elasticsearch
Data path: Elastic Agent to Logstash to Kafka to Kafka ES Sink to Elasticsearch: Kafka as middleware message queue.

Kafka ES Sink connector reads from Kafka and writes to Elasticsearch.

Use when

You are standardizing on Kafka as middleware message queue between Elastic Agent and {es}

Notes

The transformation from raw data to Elastic Common Schema (ECS) and any other enrichment can be handled by Logstash as described in Elastic Agent to Logstash (for enrichment) to Elasticsearch.

Resources

Info on Elastic Agent and agent integrations:

• Fleet and Elastic Agent Guide
• Elastic Agent integrations

Info on Logstash and Logstash plugins:

• Logstash Reference
• Logstash {agent} input
• Logstash Kafka output

Info on Elasticsearch:

• Elasticsearch Guide
• ES sink [ToDo: Add link]