Loading

Get data in

Streams supports two entry points depending on where your data is today:

  • Ingest new data: Send logs to a managed endpoint for new ingestion. Best for new deployments, custom logs, and mixed-format sources.
  • Work with existing data: Work with data already flowing into Elasticsearch. No migration or configuration changes required.

Streams requires the following permissions:

Streams requires these Elastic Cloud Serverless roles:

  • Admin: Ability to manage all Streams
  • Editor/Viewer: Limited access, cannot perform all actions

To manage all streams, you need the following permissions:

  • Cluster permissions: manage_index_templates, manage_ingest_pipelines, manage_pipeline, read_pipeline
  • Data stream level permissions: read, write, create, manage, monitor, manage_data_stream_lifecycle, read_failure_store, manage_failure_store, manage_ilm.

To view streams, you need the following permissions:

  • Data stream level: read, view_index_metadata, monitor

For more information, refer to Cluster privileges and Granting privileges for data streams and aliases

Wired streams send your documents to a managed endpoint, from which you can route data into child streams based on partitioning rules. Child streams automatically inherit mappings, lifecycle settings, and processors from the parent, and configuration changes propagate through the hierarchy.

To send data to a wired stream, configure your shipper to point to the appropriate endpoint:

Note

Set the index based on your Elastic Stack version:

  • Set the index to logs. Only the logs endpoint is available in these versions.
  • Set the index to logs.otel or logs.ecs, depending on which endpoint you want to use.
processors:
  transform/logs-streams:
    log_statements:
      - context: resource
        statements:
          - set(attributes["elasticsearch.index"], "logs.otel")
service:
  pipelines:
    logs:
      receivers: [myreceiver]
      processors: [transform/logs-streams]
      exporters: [elasticsearch, otlp]
  1. Set to logs.otel or logs.ecs (serverless and stack 9.4+), or logs (stack 9.2–9.3)
  2. works with any logs receiver
  3. works with either
Note

Set the index based on your Elastic Stack version:

  • Set the index to logs. Only the logs endpoint is available in these versions.
  • Set the index to logs.otel or logs.ecs, depending on which endpoint you want to use.
filebeat.inputs:
  - type: filestream
    id: my-filestream-id
    index: logs.otel
    enabled: true
    paths:
      - /var/log/*.log

# No need to install templates for wired streams
setup:
  template:
    enabled: false

output.elasticsearch:
  hosts: ["<elasticsearch-host>"]
  api_key: "<your-api-key>"
  1. Set to logs.otel or logs.ecs (serverless and stack 9.4+), or logs (stack 9.2–9.3)
Note

Set the index based on your Elastic Stack version:

  • Set the index to logs. Only the logs endpoint is available in these versions.
  • Set the index to logs.otel or logs.ecs, depending on which endpoint you want to use.
output {
  elasticsearch {
    hosts => ["<elasticsearch-host>"]
    api_key => "<your-api-key>"
    index => "logs.otel"
    action => "create"
  }
}
  1. Set to logs.otel or logs.ecs (serverless and stack 9.4+), or logs (stack 9.2–9.3)

Use the Custom Logs (Filestream) integration to send data to wired streams:

  1. Find Fleet in the navigation menu or use the global search field.
  2. Select the Settings tab.
  3. Under Outputs, find the output you want to use and select the icon.
  4. Turn on Write to logs streams.
  5. Add the Custom Logs (Filestream) integration to an agent policy.
  6. Enable the Use the "logs" data stream setting under Change defaults.
  7. Under Where to add this integration, select an agent policy that uses the output configured in step 4.
Note

Set the endpoint based on your Elastic Stack version:

  • Set the endpoint to logs. Only the logs endpoint is available in these versions.
  • Set the endpoint to logs.otel or logs.ecs, depending on which endpoint you want to use.

Send data to the endpoint using the Bulk API:

POST /logs.otel/_bulk
{ "create": {} }
{ "@timestamp": "2025-05-05T12:12:12", "body": { "text": "Hello world!" }, "resource": { "attributes": { "host.name": "my-host-name" } } }
{ "create": {} }
{ "@timestamp": "2025-05-05T12:12:12", "message": "Hello world!", "host.name": "my-host-name" }
  1. Set to logs.otel or logs.ecs (serverless or stack 9.4+), or logs (stack 9.2–9.3)

Use classic streams when you want the ease of extracting fields and configuring data retention while working with data that's already being ingested into Elasticsearch.

Classic streams:

  • Are based on existing data streams, index templates, and component templates.
  • Can follow the data retention policy set in the existing index template.
  • Do not support hierarchical inheritance or cascading configuration updates.

No additional configuration is required. Open Streams from Kibana and your existing data streams appear automatically.

  1. Open Streams

    Open Streams from the following places in Kibana:

    • Select Streams from the navigation menu or use the global search field.

    • Open the data stream for a specific document from Discover. To do this, expand the details flyout for a document stored in a data stream, and select Stream or an action associated with the document's data stream. Streams then opens filtered to the selected data stream.

    You can also access Streams features using the Streams API. Refer to the Streams API documentation for more information.

  2. Verify data is flowing

    After configuring your data source, confirm data is appearing in Discover.

    For wired streams, you first need to make the index pattern available:

    1. Manually create a data view for the wired streams index pattern (logs,logs.*).
    2. Add the wired streams index pattern (logs,logs.*) to the observability:logSources Kibana advanced setting, which you can open from the navigation menu or by using the global search field.

    Once data appears in Discover, you're ready to start organizing, parsing, and configuring retention for your streams.