Get data into Streams

This page shows you how to send new data to Streams or use Streams to work with existing data:

• Ingest new data: Use wired streams to send logs to a managed endpoint. This option suits new deployments, custom logs, and mixed-format sources.
• Work with existing data: Use classic streams to work with data already flowing into Elasticsearch. No migration or configuration changes are required.

Streams requires the following permissions:

Wired streams send documents to a managed endpoint, from which you can route data into child streams based on partitioning rules. Child streams inherit mappings, lifecycle settings, and processors from the parent.

Select the endpoint that matches your deployment:

• Use logs.otel for OTel-native data or for a consistent, normalized format. Streams translates ECS field names to OTel equivalents automatically. Use logs.ecs when your data already uses ECS field names and you want to preserve them without transformation.
• Use logs. This is the only wired-stream endpoint available in these versions. Replace logs.otel with logs in the following examples.

Configure your shipper to use the endpoint:

processors:
  transform/logs-streams:
    log_statements:
      - context: resource
        statements:
          - set(attributes["elasticsearch.index"], "logs.otel")
service:
  pipelines:
    logs:
      receivers: [myreceiver]
      processors: [transform/logs-streams]
      exporters: [elasticsearch, otlp]
		

filebeat.inputs:
  - type: filestream
    id: my-filestream-id
    index: logs.otel
    enabled: true
    paths:
      - /var/log/*.log

# No need to install templates for wired streams
setup:
  template:
    enabled: false

output.elasticsearch:
  hosts: ["<elasticsearch-host>"]
  api_key: "<your-api-key>"
		

output {
  elasticsearch {
    hosts => ["<elasticsearch-host>"]
    api_key => "<your-api-key>"
    index => "logs.otel"
    action => "create"
  }
}
		

				POST /logs.otel/_bulk
					{ "create": {} }
{ "@timestamp": "2025-05-05T12:12:12", "body": { "text": "Hello world!" }, "resource": { "attributes": { "host.name": "my-host-name" } } }
{ "create": {} }
{ "@timestamp": "2025-05-05T12:12:12", "message": "Hello world!", "host.name": "my-host-name" }
		

After configuring your data source, confirm data is appearing in Discover.

For wired streams, you first need to make the index pattern available:

1. Manually create a data view for the wired streams index pattern (logs,logs.*).
2. Add the wired streams index pattern (logs,logs.*) to the observability:logSources Kibana advanced setting, which you can open from the navigation menu or by using the global search field.

Once data appears in Discover, you're ready to start organizing, parsing, and configuring retention for your streams.

Wired streams can contain fields stored in _source that are not explicitly mapped. By default, ES|QL returns an error when a query references an unmapped field. To make unmapped fields queryable, add SET unmapped_fields = "LOAD"; at the start of your ES|QL query:

SET unmapped_fields = "LOAD";
FROM logs.otel
| WHERE my_custom_field == "value"
		

When LOAD is set, unmapped fields are loaded from _source as keyword fields, or treated as null if absent from _source.

When you query a wired stream and the ES|QL editor detects an unknown column error, a Load unmapped fields quick fix action is available. Select it to apply this setting automatically.

For a conceptual overview and use cases, refer to Unmapped fields. For Kibana editor behavior, refer to Handle unmapped fields with SET unmapped_fields.

Classic streams let you use the Streams UI to extract fields and configure data retention for data that's already being ingested into Elasticsearch without additional configuration.

Classic streams:

• Are based on existing data streams, index templates, and component templates.
• Can follow the data retention policy set in the existing index template.
• Do not support hierarchical inheritance or cascading configuration updates.

Open classic streams from the following places in Kibana:

• Select Streams from the navigation menu or use the global search field, and find the data stream in the Streams table.

• Open the data stream for a specific document from Discover. To do this:

  1. From the Documents table, select the icon for a document to expand the details flyout.
  2. Under Stream, select the link to the stream (for example, logs-generic.default). The Streams UI opens filtered to the stream that contains the document.

You can also access Streams features using the Streams API. Refer to the Streams API documentation for more information.

Once your data is flowing into Streams, you can start organizing and enriching it:

• Organize your data: Use partitioning to route data subsets into dedicated child streams with independent retention and processing rules. Partitioning is only available for wired streams.
• Parse and process: Build a processing pipeline to extract structured fields from raw log messages using AI-generated or manually configured processors.