Streams-new
Streams allows you to automatically parse, structure, and organize your log data so you can query it immediately, without writing Grok expressions or maintaining custom pipelines.
When an incident hits, Streams gets you to answers faster. AI-powered detection continuously scans your logs for critical signals and surfaces what matters. Instead of manually scanning thousands of log lines, you get a prioritized list of what matters.
- Organize logs automatically
- Streams uses AI to partition your log data by source and component, without manual regex rules or pipeline configuration. As new log formats arrive, Streams continues to learn and extend its partitioning automatically.
- Get meaning from logs
- The AI-powered processing pipeline detects log formats and generates parsing rules that extract structured fields from unstructured text. You get clean, queryable data without writing a single GROK expression.
- Solve incidents in minutes, not hours
- Significant Events detection continuously scans your streams for critical signals: out-of-memory errors, crash loops, certificate expirations, and anomalies.
- Reduce time spent on managing pipelines
- Streams uses AI to simplify parsing, enrichment, partitioning, and schema updates. You can start investigating issues within minutes, rather than spending weeks on pipeline setup and data engineering.
- Control storage costs
- By surfacing the most critical logs and automatically structuring data for efficient storage, Streams allows you to retain high-value data without discarding important information, reducing overall storage costs.
This is a quick overview of the main steps to get started with Streams in Kibana. It covers how to get data in, organize it into streams, parse and enrich your logs, set retention policies, and monitor data quality.
This tour is an ideal way to familiarize yourself with the Streams UI and its core workflows. You can follow along directly in your Elastic Cloud or self-managed Elasticsearch environment.
-
Get data in
Send logs via OpenTelemetry, Fluentd, Fluentbit, or an Elastic integration. For agentless ingest, send directly to the
/logsendpoint. -
Organize your data
From Kibana- Select Streams from the navigation menu or use the global search field.
- Open the data stream for a specific document from Discover. To do this, expand the details flyout for a document that's stored in a data stream, and select Stream or an action associated with the document's data stream. Streams then opens filtered to the selected data stream.
Using the APIYou can also access Streams features using the Streams API. Refer to the Streams API documentation for more information. -
Parse and process
Streams automatically organizes your logs by source and component. Accept, adjust, or add partitions manually. Use the Processing tab to parse and extract fields from log messages. Accept AI-generated GROK rules or write your own.
-
Configure retention
Use the Retention tab to define how long each stream stores data and to review ingestion volume.
-
Manage data quality
Use the Data quality column to filter your streams by data quality status.