Ingest architectures

We offer a variety of ingest architectures to serve a wide range of use cases and network configurations.

To ingest data into Elasticsearch, use the simplest option that meets your needs and satisfies your use case. For many users and use cases, the simplest approach is ingesting data with Elastic Agent and sending it to Elasticsearch. Elastic Agent and Elastic Agent integrations are available for many popular platforms and services, and are a good place to start.

Tip

You can host Elasticsearch on your own hardware or send your data to Elasticsearch on Elastic Cloud. For most users, Elastic Agent writing directly to Elasticsearch on Elastic Cloud provides the easiest and fastest time to value. {ess-leadin-short}

Decision tree

Data ingestion pipeline with decision tree

Ingest architecture Use when
Elastic Agent to Elasticsearch

Image showing Elastic Agent collecting data and sending to Elasticsearch		 An Elastic Agent integration is available for your data source:

* Software components with Elastic Agent installed
* Software components using APIs for data collection
Elastic Agent to Logstash to Elasticsearch

Image showing Elastic Agent to Logstash to Elasticsearch		 You need additional capabilities offered by Logstash:

* enrichment between Elastic Agent and Elasticsearch
* persistent queue (PQ) buffering to accommodate network issues and downstream unavailability
* proxying in cases where Elastic Agents have network restrictions for connecting outside of the Elastic Agent network
* data needs to be routed to multiple Elasticsearch clusters and other destinations depending on the content
Elastic Agent to proxy to Elasticsearch

Image showing connections between Elastic Agent and Elasticsearch using a proxy		 Agents have network restrictions that prevent connecting outside of the Elastic Agent network Note that Logstash as proxy is one option.
Elastic Agent to Elasticsearch with Kafka as middleware message queue

Image showing Elastic Agent collecting data and using Kafka as a message queue enroute to Elasticsearch		 Kafka is your middleware message queue:

* Kafka ES sink connector to write from Kafka to Elasticsearch
* Logstash to read from Kafka and route to Elasticsearch
Logstash to Elasticsearch

Image showing Logstash collecting data and sending to Elasticsearch		 You need to collect data from a source that Elastic Agent can’t read (such as databases, AWS Kinesis). Check out the Logstash input plugins.
Elastic air-gapped architectures

Image showing Elastic Stack in an air-gapped environment		 You want to deploy Elastic Agent and Elastic Stack in an air-gapped environment (no access to outside networks)