Ingesting timeseries data

Elastic and others offer tools to help you get your data from the original data source into Elasticsearch. Some tools are designed for particular data sources, and others are multi-purpose.

In this section, we’ll help you determine which option is best for you.

Elastic Agent and Elastic integrations ¶

A single Elastic Agent can collect multiple types of data when it is installed on a host computer. You can use standalone Elastic Agents and manage them locally on the systems where they are installed, or you can manage all of your agents and policies with the Fleet UI in Kibana.

Use Elastic Agent with one of hundreds of Elastic integrations to simplify collecting, transforming, and visualizing data. Integrations include default ingestion rules, dashboards, and visualizations to help you start analyzing your data right away. Check out the Integration quick reference to search for available integrations that can reduce your time to value.

Elastic Agent is the best option for collecting timestamped data for most data sources and use cases. If your data requires additional processing before going to Elasticsearch, you can use Elastic Agent processors, Logstash, or additional processing features in Elasticsearch. Check out additional processing to see options.

Ready to try Elastic Agent? Check out the installation instructions.

Beats ¶

Beats are the original Elastic lightweight data shippers, and their capabilities live on in Elastic Agent. When you use Elastic Agent, you’re getting core Beats functionality, but with more added features.

Beats require that you install a separate Beat for each type of data you want to collect. A single Elastic Agent installed on a host can collect and transport multiple types of data.

Best practice: Use Elastic Agent whenever possible. If your data source is not yet supported by Elastic Agent, use Beats. Check out the Beats and Elastic Agent comparison for more info. When you are ready to upgrade, check out Migrate from Beats to Elastic Agent.

OpenTelemetry (OTel) collectors ¶

OpenTelemetry is a vendor-neutral observability framework for collecting, processing, and exporting telemetry data. Elastic is a member of the Cloud Native Computing Foundation (CNCF) and active contributor to the OpenTelemetry project.

In addition to supporting upstream OTel development, Elastic provides Elastic Distributions of OpenTelemetry, specifically designed to work with Elastic Observability. We’re also expanding Elastic Agent to use OTel collection.

Logstash ¶

Logstash is a versatile open source data ETL (extract, transform, load) engine that can expand your ingest capabilities. Logstash can collect data from a wide variety of data sources with Logstash input plugins, enrich and transform the data with Logstash filter plugins, and output the data to Elasticsearch and other destinations with the Logstash output plugins.

Many users never need to use Logstash, but it’s available if you need it for:

  • Data collection (if an Elastic integration isn’t available). Elastic Agent and Elastic integrations provide many features out-of-the-box, so be sure to search or browse integrations for your data source. If you don’t find an Elastic integration for your data source, check Logstash for an input plugin for your data source.
  • Additional processing. One of the most common Logstash use cases is extending Elastic integrations. You can take advantage of the extensive, built-in capabilities of Elastic Agent and Elastic Integrations, and then use Logstash for additional data processing before sending the data on to Elasticsearch.
  • Advanced use cases. Logstash can help with advanced use cases, such as when you need persistence or buffering, additional data enrichment, proxying as a way to bridge network connections, or the ability to route data to multiple destinations.