Split an event into multiple documents

Sometimes a single incoming event contains multiple records. For example, an API response or a batched TCP payload might hold an array of items, and you want to index each item as its own document in Elasticsearch.

Elastic Agent processors and Elasticsearch ingest pipelines process each event individually: they can transform or drop an event, but they can't generate multiple documents from a single event.

To split one event into multiple documents, you have two options:

• Split events at collection time, if your data source's input supports it.
• Split events with Logstash, for everything else.

Most data sources don't need splitting. Elastic Integrations are designed to deliver one document per record. Splitting mainly matters for custom and input-only integrations, such as a Custom TCP Logs integration that receives batched JSON payloads.

Splitting turns one event into many documents. Rerouting sends each document to a different destination based on its content. If your goal is to route documents to different data streams or indices, you don't need to split, and you don't need Logstash. Instead, use the Elasticsearch reroute processor, typically in a @custom ingest pipeline.

Some Elastic Agent and Filebeat inputs can split an incoming payload into separate events as the data is collected, before anything reaches Elasticsearch. If your data source uses one of these inputs, this is the easiest option as it requires no extra components and no changes to how your data flows.

For example:

• The Custom API integration and the underlying HTTP JSON input can split API responses into separate events with the response.split setting.
• The CEL Custom API input integration can emit multiple events per request when the CEL program returns a list of events.
• The AWS S3 input can create one event per element of a JSON array with the expand_event_list_from_field setting.

Check your input's reference documentation for similar settings.

When your input has no native split option, run a Logstash pipeline between Elastic Agent and Elasticsearch. The Logstash split filter creates a copy of the event for each element of an array field, so each element becomes its own document.

If your data flows through an Elastic integration, pair the split filter with the elastic_integration filter, which runs the integration's ingest pipeline inside Logstash. This keeps the integration's processing intact while letting you split the results. For more details about this pattern, refer to Using Logstash with Elastic integrations.

• A Logstash instance that your Elastic Agents can reach.
• Elastic Agents configured to output to Logstash.
• If you use the elastic_integration filter: the required subscription, and credentials with the monitor, read_pipeline, and manage_index_templates privileges on your Elasticsearch cluster. Refer to the plugin documentation for details.

This pipeline receives events from Elastic Agent, runs the integration's ingest pipeline, and then splits each event on the items array field:

input {
  elastic_agent {
    port => 5044
  }
}

filter {
  elastic_integration {
    cloud_id => "<cloud-id>"
    api_key => "<api-key>"
  }

  split {
    field => "items"
  }
}

output {
  elasticsearch {
    cloud_id => "<cloud-id>"
    api_key => "<api-key>"
    data_stream => true
    ecs_compatibility => "v8"
  }
}
		

1. The elastic_integration filter runs the event's integration ingest pipeline inside Logstash. It must be the first filter in the pipeline. If your data doesn't flow through an integration, omit this filter.
2. The split filter creates one copy of the event per element of the items array. Each copy becomes its own document. If the array arrives as an unparsed string, parse it first, for example with the json filter, before splitting.
3. Elastic integrations are designed to work with data streams and ECS-compatible output, so set data_stream => true and ecs_compatibility to v1 or v8 in the elasticsearch output.