Simulate data ingestion

  • POST also GET /_ingest/{index}/_simulate
  • POST also GET /_ingest/_simulate
Auth

Run ingest pipelines against a set of provided documents, optionally with substitute pipeline definitions, to simulate ingesting data into an index.

This API is meant to be used for troubleshooting or pipeline development, as it does not actually index any data into Elasticsearch.

The API runs the default and final pipeline for that index against a set of documents provided in the body of the request. If a pipeline contains a reroute processor, it follows that reroute processor to the new index, running that index's pipelines as well the same way that a non-simulated ingest would. No data is indexed into Elasticsearch. Instead, the transformed document is returned, along with the list of pipelines that have been run and the name of the index where the document would have been indexed if this were not a simulation. The transformed document is validated against the mappings that would apply to this index, and any validation error is reported in the result.

This API differs from the simulate pipeline API in that you specify a single pipeline for that API, and it runs only that one pipeline. The simulate pipeline API is more useful for developing a single pipeline, while the simulate ingest API is more useful for troubleshooting the interaction of the various pipelines that get applied when ingesting into an index.

By default, the pipeline definitions that are currently in the system are used. However, you can supply substitute pipeline definitions in the body of the request. These will be used in place of the pipeline definitions that are already in the system. This can be used to replace existing pipeline definitions or to create new ones. The pipeline substitutions are used only within this request.

Required authorization

Index privileges index

Path parameters

index string optional

The index to simulate ingesting into. This value can be overridden by specifying an index on each document. If you specify this parameter in the request path, it is used for any documents that do not explicitly specify an index argument.

Leave it out to use /_ingest/_simulate.

Query parameters

pipeline string

The pipeline to use as the default pipeline. This value can be used to override the default pipeline of the index.

merge_type enum · default: index

The mapping merge type if mapping overrides are being provided in mapping_addition. The allowed values are one of index or template. The index option merges mappings the way they would be merged into an existing index. The template option merges mappings the way they would be merged into a template.

Values: index template

Request body

docs object[] required

Sample documents to test in the pipeline.

_id string

Unique identifier for the document. This ID must be unique within the _index.

_index string

Name of the index containing the document.

_source object required

JSON body for the document.

component_template_substitutions map string to object

A map of component template names to substitute component template definition objects.

<string> object
template object required
_meta object
version number
settings map string to object
mappings object
all_field object
analyzer string required
enabled boolean required
omit_norms boolean required
search_analyzer string required
similarity string required
store boolean required
store_term_vector_offsets boolean required
store_term_vector_payloads boolean required
store_term_vector_positions boolean required
store_term_vectors boolean required
date_detection boolean
dynamic enum
Values: strict runtime true false
dynamic_date_formats string[]
dynamic_templates string to object[]
_field_names object
enabled boolean required
index_field object
enabled boolean required
_meta object
numeric_detection boolean
properties map string to oneOf
_routing object
required boolean required
_size object
enabled boolean required
_source object
enabled boolean
excludes string[]
includes string[]
mode enum

Supported values include:

  • disabled
  • stored
  • columnar_stored
  • synthetic: Instead of storing source documents on disk exactly as you send them, Elasticsearch can reconstruct source content on the fly upon retrieval.
Values: disabled stored columnar_stored synthetic
runtime map string to object
<string> object
fields map string to object

For type composite

<string> object
type enum required
Values: boolean composite date double geo_point geo_shape ip keyword long lookup
fetch_fields object[]

For type lookup

field string required
format string
format string

A custom format for date type runtime fields.

input_field string

For type lookup

target_field string

For type lookup

target_index string

For type lookup

script string

Painless script executed at query time.

on_script_error enum
Values: fail continue
type enum required

Field type, which can be: boolean, composite, date, double, geo_point, ip,keyword, long, or lookup.

Values: boolean composite date double geo_point geo_shape ip keyword long lookup
enabled boolean
subobjects enum
Values: true false
_data_stream_timestamp object
enabled boolean required
aliases map string to object
<string> object
filter {} Query

Query used to limit documents the alias can access.

index_routing string

Value used to route indexing operations to a specific shard. If specified, this overwrites the routing value for indexing operations.

is_write_index boolean · default: false

If true, the index is the write index for the alias.

routing string

Value used to route indexing and search operations to a specific shard.

search_routing string

Value used to route search operations to a specific shard. If specified, this overwrites the routing value for search operations.

is_hidden boolean · default: false

If true, the alias is hidden. All indices for the alias must have the same is_hidden value.

lifecycle object
data_retention string

If defined, every document added to this data stream will be stored at least for this time frame. Any time after this duration the document could be deleted. When empty, every document in this data stream will be stored indefinitely.

effective_retention string

The least amount of time data should be kept by elasticsearch.

retention_determined_by enum

Configuration source that can influence the retention of a data stream.

Values: data_stream_configuration default_global_retention max_global_retention default_failures_retention
downsampling object[]

The list of downsampling rounds to execute as part of this downsampling configuration

after string required

The duration since rollover when this downsampling round should execute

fixed_interval string required

The downsample interval.

downsampling_method enum

The method used to downsample the data. There are two options aggregate and last_value. It requires downsampling to be defined. Defaults to aggregate.

Values: aggregate last_value
enabled boolean · default: true

If defined, it turns data stream lifecycle on/off (true/false) for this data stream. A data stream lifecycle that's disabled (enabled: false) will have no effect on the data stream.

frozen_after string

The period after which data stream backing indices are automatically converted to partially mounted searchable snapshots. This field is valid only on main data stream lifecycles and cannot be set on failure-store lifecycles.

data_stream_options object
failure_store object

If defined, it specifies configuration for the failure store of this data stream.

enabled boolean · default: true

If defined, it turns the failure store on/off (true/false) for this data stream. A data stream failure store that's disabled (enabled: false) will redirect no new failed indices to the failure store; however, it will not remove any existing data from the failure store.

lifecycle object

If defined, it specifies the lifecycle configuration for the failure store of this data stream.

data_retention string

If defined, every document added to this data stream will be stored at least for this time frame. Any time after this duration the document could be deleted. When empty, every document in this data stream will be stored indefinitely.

enabled boolean · default: true

If defined, it turns data stream lifecycle on/off (true/false) for this data stream. A data stream lifecycle that's disabled (enabled: false) will have no effect on the data stream.

version number
_meta object
deprecated boolean
created_date union oneOf

Date and time when the component template was created. Only returned if the human query parameter is true.

One of: string or number
created_date_millis number

Date and time when the component template was created, in milliseconds since the epoch.

modified_date union oneOf

Date and time when the component template was last modified. Only returned if the human query parameter is true.

One of: string or number
modified_date_millis number

Date and time when the component template was last modified, in milliseconds since the epoch.

index_template_substitutions map string to object

A map of index template names to substitute index template definition objects.

<string> object
index_patterns union oneOf required

Array of wildcard (*) expressions used to match the names of data streams and indices during creation.

One of: string[] or string
composed_of string[] required

An ordered list of component template names. Component templates are merged in the order specified, meaning that the last component template specified has the highest precedence.

template object

Template to be applied. It may optionally include an aliases, mappings, or settings configuration.

aliases map string to object

Aliases to add. If the index template includes a data_stream object, these are data stream aliases. Otherwise, these are index aliases. Data stream aliases ignore the index_routing, routing, and search_routing options.

<string> object
filter {} Query

Query used to limit documents the alias can access.

index_routing string

Value used to route indexing operations to a specific shard. If specified, this overwrites the routing value for indexing operations.

is_hidden boolean · default: false

If true, the alias is hidden. All indices for the alias must have the same is_hidden value.

is_write_index boolean · default: false

If true, the index is the write index for the alias.

routing string

Value used to route indexing and search operations to a specific shard.

search_routing string

Value used to route search operations to a specific shard. If specified, this overwrites the routing value for search operations.

mappings object

Mapping for fields in the index. If specified, this mapping can include field names, field data types, and mapping parameters.

Same fields as mappings in template
settings object

Configuration options for the index.

lifecycle object
data_retention string

If defined, every document added to this data stream will be stored at least for this time frame. Any time after this duration the document could be deleted. When empty, every document in this data stream will be stored indefinitely.

effective_retention string

The least amount of time data should be kept by elasticsearch.

retention_determined_by enum

Configuration source that can influence the retention of a data stream.

Values: data_stream_configuration default_global_retention max_global_retention default_failures_retention
downsampling object[]

The list of downsampling rounds to execute as part of this downsampling configuration

after string required

The duration since rollover when this downsampling round should execute

fixed_interval string required

The downsample interval.

downsampling_method enum

The method used to downsample the data. There are two options aggregate and last_value. It requires downsampling to be defined. Defaults to aggregate.

Values: aggregate last_value
enabled boolean · default: true

If defined, it turns data stream lifecycle on/off (true/false) for this data stream. A data stream lifecycle that's disabled (enabled: false) will have no effect on the data stream.

frozen_after string

The period after which data stream backing indices are automatically converted to partially mounted searchable snapshots. This field is valid only on main data stream lifecycles and cannot be set on failure-store lifecycles.

data_stream_options object
failure_store object

If defined, it specifies configuration for the failure store of this data stream.

enabled boolean · default: true

If defined, it turns the failure store on/off (true/false) for this data stream. A data stream failure store that's disabled (enabled: false) will redirect no new failed indices to the failure store; however, it will not remove any existing data from the failure store.

lifecycle object

If defined, it specifies the lifecycle configuration for the failure store of this data stream.

data_retention string

If defined, every document added to this data stream will be stored at least for this time frame. Any time after this duration the document could be deleted. When empty, every document in this data stream will be stored indefinitely.

enabled boolean · default: true

If defined, it turns data stream lifecycle on/off (true/false) for this data stream. A data stream lifecycle that's disabled (enabled: false) will have no effect on the data stream.

version number

Version number used to manage index templates externally. This number is not automatically generated by Elasticsearch.

priority number

Priority to determine index template precedence when a new data stream or index is created. The index template with the highest priority is chosen. If no priority is specified the template is treated as though it is of priority 0 (lowest priority). This number is not automatically generated by Elasticsearch.

_meta object

Optional user metadata about the index template. May have any contents. This map is not automatically generated by Elasticsearch.

allow_auto_create boolean
data_stream object

If this object is included, the template is used to create data streams and their backing indices. Supports an empty object. Data streams require a matching index template with a data_stream object.

hidden boolean · default: false

If true, the data stream is hidden.

allow_custom_routing boolean · default: false

If true, the data stream supports custom routing.

deprecated boolean

Marks this index template as deprecated. When creating or updating a non-deprecated index template that uses deprecated components, Elasticsearch will emit a deprecation warning.

ignore_missing_component_templates union oneOf

A list of component template names that are allowed to be absent.

One of: string[] or string
created_date union oneOf

Date and time when the index template was created. Only returned if the human query parameter is true.

One of: string or number
created_date_millis number

Date and time when the index template was created, in milliseconds since the epoch.

modified_date union oneOf

Date and time when the index template was last modified. Only returned if the human query parameter is true.

One of: string or number
modified_date_millis number

Date and time when the index template was last modified, in milliseconds since the epoch.

mapping_addition object
Same fields as mappings in template
pipeline_substitutions map string to object

Pipelines to test. If you don’t specify the pipeline request path parameter, this parameter is required. If you specify both this and the request path parameter, the API only uses the request path parameter.

<string> object
description string

Description of the ingest pipeline.

on_failure {} Processor[]

Processors to run immediately after a processor failure.

processors {} Processor[]

Processors used to perform transformations on documents before indexing. Processors run sequentially in the order specified.

version number

Version number used by external systems to track ingest pipelines.

deprecated boolean · default: false

Marks this ingest pipeline as deprecated. When a deprecated ingest pipeline is referenced as the default or final pipeline when creating or updating a non-deprecated index template, Elasticsearch will emit a deprecation warning.

_meta object

Arbitrary metadata about the ingest pipeline. This map is not automatically generated by Elasticsearch.

created_date union oneOf

Date and time when the pipeline was created. Only returned if the human query parameter is true.

One of: string or number
created_date_millis number

Date and time when the pipeline was created, in milliseconds since the epoch.

modified_date union oneOf

Date and time when the pipeline was last modified. Only returned if the human query parameter is true.

One of: string or number
modified_date_millis number

Date and time when the pipeline was last modified, in milliseconds since the epoch.

field_access_pattern enum · default: classic

Controls how processors in this pipeline should read and write data on a document's source.

Values: classic flexible

Response

application/json
docs object[] required
doc object
_id string required

Identifier for the document.

_index string required

Name of the index that the document would be indexed into if this were not a simulation.

_source map string to object required

JSON body for the document.

_version union oneOf required
One of: number or string
executed_pipelines string[] required

A list of the names of the pipelines executed on this document.

ignored_fields string to string[]

A list of the fields that would be ignored at the indexing step. For example, a field whose value is larger than the allowed limit would make it through all of the pipelines, but would not be indexed into Elasticsearch.

error object

Any error resulting from simulatng ingest on this doc. This can be an error generated by executing a processor, or a mapping validation error when simulating indexing the resulting doc.

type string required

The type of error

reason union string nullable

A human-readable explanation of the error, in English.

One of: string
stack_trace string

The server stack trace. Present only if the error_trace=true parameter was sent with the request.

caused_by object
Same fields as error in doc
root_cause object[]
Same fields as error in doc
suppressed object[]
Same fields as error in doc
effective_mapping object
Same fields as mappings in template