Loading

Get started with Streams

This hands-on guide will take you through to the core features and common use cases of Streams. Before using Streams, make sure you have the following in place:

  • Elasticsearch and Kibana: Streams is available from Elasticsearch 9.1 (API, preview), 9.2 (Wired streams, preview), and 9.2+ (GA for classic streams). For Elastic Cloud Serverless, Streams is generally available.
  • Log data ingestion: Logs can be sent to Streams via OpenTelemetry Collector, Fluentd, Fluentbit, or through Elastic one-click integrations. No agent deployment is required for agentless ingest via the /logs endpoint (Logs Streams, tech preview).
  • Required permissions:

Streams requires these Elastic Cloud Serverless roles:

  • Admin: Ability to manage all Streams
  • Editor/Viewer: Limited access, cannot perform all actions

To manage all streams, you need the following permissions:

  • Cluster permissions: manage_index_templates, manage_ingest_pipelines, manage_pipeline, read_pipeline
  • Data stream level permissions: read, write, create, manage, monitor, manage_data_stream_lifecycle, read_failure_store, manage_failure_store, manage_ilm.

To view streams, you need the following permissions:

  • Data stream level: read, view_index_metadata, monitor

For more information, refer to Cluster privileges and Granting privileges for data streams and aliases

To start using Streams:

  1. Ingest log data

    Send logs via OpenTelemetry, Fluentd, Fluentbit, or an Elastic integration. For agentless ingest, send directly to the /logs endpoint.

  2. Access Streams

    From Kibana
    • Select Streams from the navigation menu or use the global search field.
    • Open the data stream for a specific document from Discover. To do this, expand the details flyout for a document that's stored in a data stream, and select Stream or an action associated with the document's data stream. Streams then opens filtered to the selected data stream.
    Using the API

    You can also access Streams features using the Streams API. Refer to the Streams API documentation for more information.

  3. Review AI-suggested partitions

    Streams automatically organizes your logs by source and component. Accept, adjust, or add partitions manually.

  4. Set retention policies

    Use the Retention tab to define how long each stream stores data and to review ingestion volume.

  5. Configure processing

    Use the Processing tab to parse and extract fields from log messages. Accept AI-generated GROK rules or write your own.

  6. Manage data quality

    Use the Data quality column to filter your streams by data quality status.

  7. Configure advanced settings

    Use the Advanced tab to view the underlying Elasticsearch configuration and advanced settings for this stream.

  8. Investigate with Significant Events

    Review the Significant Events view to triage critical signals across your streams.

Streams is not the only way, consider these alternatives depending on your needs:

  • Elastic Agent integrations: Pre-built integrations with automatic parsing and dashboards for common data sources. Best when your sources are covered by the Elastic integration catalog.
  • Logstash pipelines: Highly customizable, code-first pipeline configuration. Best for complex transformations or when you need to fan out to multiple destinations.
  • Elasticsearch ingest pipelines: Low-level pipeline configuration via the ES API. Best for teams who already manage Elasticsearch directly and want fine-grained control without a UI.