Simulate data ingestion

POST also GET /_ingest/{index}/_simulate
POST also GET /_ingest/_simulate

Run ingest pipelines against a set of provided documents, optionally with substitute pipeline definitions, to simulate ingesting data into an index.

This API is meant to be used for troubleshooting or pipeline development, as it does not actually index any data into Elasticsearch.

The API runs the default and final pipeline for that index against a set of documents provided in the body of the request. If a pipeline contains a reroute processor, it follows that reroute processor to the new index, running that index's pipelines as well the same way that a non-simulated ingest would. No data is indexed into Elasticsearch. Instead, the transformed document is returned, along with the list of pipelines that have been run and the name of the index where the document would have been indexed if this were not a simulation. The transformed document is validated against the mappings that would apply to this index, and any validation error is reported in the result.

This API differs from the simulate pipeline API in that you specify a single pipeline for that API, and it runs only that one pipeline. The simulate pipeline API is more useful for developing a single pipeline, while the simulate ingest API is more useful for troubleshooting the interaction of the various pipelines that get applied when ingesting into an index.

By default, the pipeline definitions that are currently in the system are used. However, you can supply substitute pipeline definitions in the body of the request. These will be used in place of the pipeline definitions that are already in the system. This can be used to replace existing pipeline definitions or to create new ones. The pipeline substitutions are used only within this request.

Required authorization

Index privileges: index

Path Parameters

index string IndexName optional

The index to simulate ingesting into. This value can be overridden by specifying an index on each document. If you specify this parameter in the request path, it is used for any documents that do not explicitly specify an index argument.

Leave it out to use /_ingest/_simulate.

Query Parameters

Request

Response

application/json

Examples

In this example the index my-index has a default pipeline called my-pipeline and a final pipeline called my-final-pipeline. Since both documents are being ingested into my-index, both pipelines are run using the pipeline definitions that are already in the system.

Request Console 1 / 8 languages

Console Kibana Dev Tools

POST /_ingest/_simulate
{
  "docs": [
    {
      "_id": "123",
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": "456",
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ]
}

JavaScript Elasticsearch JavaScript Client

const response = await client.simulate.ingest({
  docs: [
    {
      _id: "123",
      _index: "my-index",
      _source: {
        foo: "bar",
      },
    },
    {
      _id: "456",
      _index: "my-index",
      _source: {
        foo: "rab",
      },
    },
  ],
});

Python Elasticsearch Python Client

resp = client.simulate.ingest(
    docs=[
        {
            "_id": "123",
            "_index": "my-index",
            "_source": {
                "foo": "bar"
            }
        },
        {
            "_id": "456",
            "_index": "my-index",
            "_source": {
                "foo": "rab"
            }
        }
    ],
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_ingest/_simulate" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "docs": [
      {
        "_id": "123",
        "_index": "my-index",
        "_source": {
          "foo": "bar"
        }
      },
      {
        "_id": "456",
        "_index": "my-index",
        "_source": {
          "foo": "rab"
        }
      }
    ]
  }'

Java Elasticsearch Java Client

client.simulate().ingest(i -> i
    .docs(List.of(Document.of(d -> d
            .id("123")
            .index("my-index")
            .source(JsonData.fromJson("""
{"foo":"bar"}
"""))
        ),Document.of(d -> d
            .id("456")
            .index("my-index")
            .source(JsonData.fromJson("""
{"foo":"rab"}
"""))
        )))
);

C# Elasticsearch .NET Client

var response = await client.Simulate
    .IngestAsync(d1 => d1
        .Docs(d2 => d2
            .Id("123")
            .Index("my-index")
            .Source(JsonSerializer.Deserialize<JsonElement>("""
            {
              "foo": "bar"
            }
            """)), d2 => d2
            .Id("456")
            .Index("my-index")
            .Source(JsonSerializer.Deserialize<JsonElement>("""
            {
              "foo": "rab"
            }
            """))
        )
    );

PHP Elasticsearch PHP Client

$resp = $client->simulate()->ingest([
    "body" => [
        "docs" => array(
            [
                "_id" => "123",
                "_index" => "my-index",
                "_source" => [
                    "foo" => "bar",
                ],
            ],
            [
                "_id" => "456",
                "_index" => "my-index",
                "_source" => [
                    "foo" => "rab",
                ],
            ],
        ),
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.simulate.ingest(
  body: {
    "docs": [
      {
        "_id": "123",
        "_index": "my-index",
        "_source": {
          "foo": "bar"
        }
      },
      {
        "_id": "456",
        "_index": "my-index",
        "_source": {
          "foo": "rab"
        }
      }
    ]
  }
)

Response

{
  "docs": [
    {
      "doc": null,
      "_id": 123,
      "_index": "my-index",
      "_version": -3,
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "bar"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "doc": null,
      "_id": 456,
      "_index": "my-index",
      "_version": "-3,",
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "rab"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}

In this example the index my-index has a default pipeline called my-pipeline and a final pipeline called my-final-pipeline. But a substitute definition of my-pipeline is provided in pipeline_substitutions. The substitute my-pipeline will be used in place of the my-pipeline that is in the system, and then the my-final-pipeline that is already defined in the system will run.

Request Console 1 / 2 languages

Console Kibana Dev Tools

POST /_ingest/_simulate
{
  "docs": [
    {
      "_index": "my-index",
      "_id": "123",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_index": "my-index",
      "_id": "456",
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "uppercase": {
            "field": "foo"
          }
        }
      ]
    }
  }
}

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_ingest/_simulate" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "docs": [
    {
      "_index": "my-index",
      "_id": "123",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_index": "my-index",
      "_id": "456",
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "uppercase": {
            "field": "foo"
          }
        }
      ]
    }
  }
}'

Response

{
  "docs": [
    {
      "doc": null,
      "_id": 123,
      "_index": "my-index",
      "_version": -3,
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "bar"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "doc": null,
      "_id": 456,
      "_index": "my-index",
      "_version": "-3,",
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "rab"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}

In this example, imagine that the index my-index has a strict mapping with only the foo keyword field defined. Say that field mapping came from a component template named my-mappings-template. You want to test adding a new field, bar. So a substitute definition of my-mappings-template is provided in component_template_substitutions. The substitute my-mappings-template will be used in place of the existing mapping for my-index and in place of the my-mappings-template that is in the system.

Request Console 1 / 2 languages

Console Kibana Dev Tools

POST /_ingest/_simulate
{
  "docs": [
    {
      "_index": "my-index",
      "_id": "123",
      "_source": {
        "foo": "foo"
      }
    },
    {
      "_index": "my-index",
      "_id": "456",
      "_source": {
        "bar": "rab"
      }
    }
  ],
  "component_template_substitutions": {
    "my-mappings_template": {
      "template": {
        "mappings": {
          "dynamic": "strict",
          "properties": {
            "foo": {
              "type": "keyword"
            },
            "bar": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_ingest/_simulate" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "docs": [
    {
      "_index": "my-index",
      "_id": "123",
      "_source": {
        "foo": "foo"
      }
    },
    {
      "_index": "my-index",
      "_id": "456",
      "_source": {
        "bar": "rab"
      }
    }
  ],
  "component_template_substitutions": {
    "my-mappings_template": {
      "template": {
        "mappings": {
          "dynamic": "strict",
          "properties": {
            "foo": {
              "type": "keyword"
            },
            "bar": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}'

Response

{
  "docs": [
    {
      "doc": null,
      "_id": 123,
      "_index": "my-index",
      "_version": -3,
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "bar"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "doc": null,
      "_id": 456,
      "_index": "my-index",
      "_version": "-3,",
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "rab"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}

The pipeline, component template, and index template substitutions replace the existing pipeline details for the duration of this request.

Request Console 1 / 2 languages

Console Kibana Dev Tools

POST /_ingest/_simulate
{
  "docs": [
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "set": {
            "field": "field3",
            "value": "value3"
          }
        }
      ]
    }
  },
  "component_template_substitutions": {
    "my-component-template": {
      "template": {
        "mappings": {
          "dynamic": true,
          "properties": {
            "field3": {
              "type": "keyword"
            }
          }
        },
        "settings": {
          "index": {
            "default_pipeline": "my-pipeline"
          }
        }
      }
    }
  },
  "index_template_substitutions": {
    "my-index-template": {
      "index_patterns": [
        "my-index-*"
      ],
      "composed_of": [
        "component_template_1",
        "component_template_2"
      ]
    }
  },
  "mapping_addition": {
    "dynamic": "strict",
    "properties": {
      "foo": {
        "type": "keyword"
      }
    }
  }
}

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_ingest/_simulate" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "docs": [
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "set": {
            "field": "field3",
            "value": "value3"
          }
        }
      ]
    }
  },
  "component_template_substitutions": {
    "my-component-template": {
      "template": {
        "mappings": {
          "dynamic": true,
          "properties": {
            "field3": {
              "type": "keyword"
            }
          }
        },
        "settings": {
          "index": {
            "default_pipeline": "my-pipeline"
          }
        }
      }
    }
  },
  "index_template_substitutions": {
    "my-index-template": {
      "index_patterns": [
        "my-index-*"
      ],
      "composed_of": [
        "component_template_1",
        "component_template_2"
      ]
    }
  },
  "mapping_addition": {
    "dynamic": "strict",
    "properties": {
      "foo": {
        "type": "keyword"
      }
    }
  }
}'

Response

{
  "docs": [
    {
      "doc": null,
      "_id": 123,
      "_index": "my-index",
      "_version": -3,
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "bar"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "doc": null,
      "_id": 456,
      "_index": "my-index",
      "_version": "-3,",
      "_source": {
        "field1": "value1",
        "field2": "value2",
        "foo": "rab"
      },
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}

Required authorization

Path Parameters

Query Parameters { pipeline , merge_type , }

Request { docs , component_template_substitutions , index_template_substitutions , mapping_addition , pipeline_substitutions , }

Response

Query Parameters

Request