Loading

Organize your data

When logs from multiple sources flow into a single wired stream, partitioning lets you route subsets of that data into dedicated child streams. Each child stream can then be managed independently, with its own retention policy, processing rules, and field mappings, while automatically inheriting the parent's defaults.

For example, you can route firewall logs to a logs.otel.firewall child stream with a 7-day retention, and application logs to a logs.otel.application child stream with a 30-day retention, without duplicating any shared configuration.

Note

Partitioning is only available on wired streams. If you're using classic streams or all your logs need identical treatment, skip this step.

Before creating partitions, keep the following in mind:

  • Partition by logical groupings, not by high-cardinality fields. Group logs by team, technology type, or environment (for example, web-servers, application, security) rather than by individual service names or host identifiers, which can generate too many streams to manage effectively.
  • Aim for tens of partitions, not hundreds. Each partition creates a dedicated data stream in Elasticsearch. There is a cost to each one, so keep the number manageable.
  • Only partition when you need different lifecycle policies. If all your logs can share the same retention and processing rules, a single stream is simpler to operate.

  1. Open the Partitioning tab

    1. Open Streams from the navigation menu or use the global search field.
    2. Select your wired stream from the list.
    3. Go to the Partitioning tab.

  2. Create a partition

    Choose how to define partitions: manually using field-based conditions, or by letting AI analyze your data and suggest groupings.

    Create partitions manually
    1. Select Create partition manually.
    2. In the Data preview, hover over a field and select:
      • to route data where the field matches the value.
      • to route data where the field does not match the value.
    3. Under Stream name, give the child stream a name that reflects the condition.
    4. Select Save to create the child stream.

    Under Condition, you can also set the field, comparator, and value directly. Turn on the Syntax editor to enter conditions in YAML. For more on conditions, refer to Streamlang conditions.

    Suggest partitions with AI

    Requires a Generative AI connector.

    1. Select Suggest partitions with AI. Streams analyzes your data and suggests groupings.
    2. Review the suggested partitions, then Accept or Reject each one.
    3. To refine the results, select Modify suggestions, provide guidance (for example, "Partition by service name and severity level"), and submit. Streams regenerates suggestions based on your input.
    4. Continue refining as needed, or select Try again to start over.
    5. After accepting, review the generated Stream name and Condition.
    6. Select Create stream.

    1. Select Suggest partitions with AI. Streams analyzes your data and suggests groupings.
    2. Accept or Reject the suggestions. After accepting, review the Stream name and Condition.
    3. Select Create stream.

  3. Review the stream hierarchy

    After saving, your stream list updates to show the parent-child relationship. For example:

    logs
├── logs.otel.application  [30d retention]
└── logs.otel.firewall     [7d retention]

    Child streams automatically inherit the parent's field mappings, lifecycle settings, and processors. You can override any inherited setting at the child level without affecting the parent or other children.

After partitioning, each child stream can be configured independently. You're ready to add processing rules to extract fields, set retention policies per stream, or monitor data quality for individual streams.