Reindex documents

POST /_reindex

Auth

Copy documents from a source to a destination. You can copy all documents to the destination index or reindex a subset of the documents. The source can be any existing index, alias, or data stream. The destination must differ from the source. For example, you cannot reindex a data stream into itself.

Important

Reindex requires _source to be enabled for all documents in the source. The destination should be configured as wanted before calling the reindex API. Reindex does not copy the settings from the source or its associated template. Mappings, shard counts, and replicas, for example, must be configured ahead of time.

If the Elasticsearch security features are enabled, you must have the following security privileges:

The read index privilege for the source data stream, index, or alias.
The write index privilege for the destination data stream, index, or index alias.
To automatically create a data stream or index with a reindex API request, you must have the auto_configure, create_index, or manage index privilege for the destination data stream, index, or alias.
If reindexing from a remote cluster, the source.remote.user must have the monitor cluster privilege and the read index privilege for the source data stream, index, or alias.

If reindexing from a remote cluster into a cluster using Elastic Stack, you must explicitly allow the remote host using the reindex.remote.whitelist node setting on the destination cluster. If reindexing from a remote cluster into an Elastic Cloud Serverless project, only remote hosts from Elastic Cloud Hosted and Elastic Cloud Serverless are allowed. Automatic data stream creation requires a matching index template with data stream enabled.

The dest element can be configured like the index API to control optimistic concurrency control. Omitting version_type or setting it to internal causes Elasticsearch to blindly dump documents into the destination, overwriting any that happen to have the same ID.

Setting version_type to external causes Elasticsearch to preserve the version from the source, create any documents that are missing, and update any documents that have an older version in the destination than they do in the source.

Setting op_type to create causes the reindex API to create only missing documents in the destination. All existing documents will cause a version conflict.

Important

Because data streams are append-only, any reindex request to a destination data stream must have an op_type of create. A reindex can only add new documents to a destination data stream. It cannot update existing documents in a destination data stream.

By default, version conflicts abort the reindex process. To continue reindexing if there are conflicts, set the conflicts request body property to proceed. In this case, the response includes a count of the version conflicts that were encountered. Note that the handling of other error types is unaffected by the conflicts property. Additionally, if you opt to count version conflicts, the operation could attempt to reindex more documents from the source than max_docs until it has successfully indexed max_docs documents into the target or it has gone through every document in the source query.

It's recommended to reindex on indices with a green status. Reindexing can fail when a node shuts down or crashes.

When requested with wait_for_completion=true (default), the request fails if the node shuts down.
When requested with wait_for_completion=false, a task id is returned, for use with the task management APIs. The task may disappear or fail if the node shuts down. When retrying a failed reindex operation, it might be necessary to set conflicts=proceed or to first delete the partial destination index. Additionally, dry runs, checking disk space, and fetching index recovery information can help address the root cause.

Refer to the linked documentation for examples of how to reindex documents.

About reindexing indices examples

Required privileges

Index privileges read write

Query parameters

refresh boolean · default: false

If true, the request refreshes affected shards to make this operation visible to search.

requests_per_second number · default: -1

The maximum number of documents to index per second, across the entire reindex operation (including slices). It can be either -1 to turn off throttling or any decimal number like 1.7 or 12 to throttle to that level.

scroll string · default: 5m

The period of time that a consistent view of the index should be maintained for scrolled search. In serverless, and stack versions >= v9.5.0, we use PIT rather than scroll for pagination. We only use scroll for reindexing from remote clusters that are older than v7.10.0. Therefore, this parameter is ignored unless you are reindexing from a remote cluster that is older than v7.10.0.

slices union oneOf · default: 1

The number of slices this task should be divided into. It defaults to one slice, which means the task isn't sliced into subtasks.

Reindex supports sliced scroll to parallelize the reindexing process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.

Note

Reindexing from remote clusters does not support manual or automatic slicing.

If set to auto, Elasticsearch chooses the number of slices to use. This setting will use one slice per shard, up to a certain limit. If there are multiple sources, it will choose the number of slices based on the index or backing index with the smallest number of shards.

Values: auto

max_docs number

The maximum number of documents to reindex. By default, all documents are reindexed. If it is a value less then or equal to scroll_size, a scroll will not be used to retrieve the results for the operation.

If conflicts is set to proceed, the reindex operation could attempt to reindex more documents from the source than max_docs until it has successfully indexed max_docs documents into the target or it has gone through every document in the source query.

If slices is set, the max_docs limit is split evenly across the slices. If the number of documents in the source is equal to or slightly more than max_docs, this could result in slightly fewer than max_docs documents being reindexed, due to skew in the slicing.

timeout string · default: 1m

The period each indexing waits for automatic index creation, dynamic mapping updates, and waiting for active shards. By default, Elasticsearch waits for at least one minute before failing. The actual wait time could be longer, particularly when multiple waits occur.

wait_for_completion boolean · default: true

If true, the request blocks until the operation is complete. If your requested reindex operation is complex or time-consuming, it might timeout due to transport-layer limitations. While the reindex will continue to be processed by the cluster, your client will not receive updates on status automatically after timeout. Set this option true if you anticipate a long-running reindex.

require_alias boolean · default: false

If true, the destination must be an index alias.

Request body

conflicts enum · default: abort

Indicates whether to continue reindexing even when there are conflicts.

Supported values include:

abort: Stop reindexing if there are conflicts.
proceed: Continue reindexing even if there are conflicts.

Values: abort proceed

dest object required

The destination you are copying to.

index string required

The name of the data stream, index, or index alias you are copying to.

op_type enum · default: index

If it is create, the operation will only index documents that do not already exist (also known as "put if absent").

Important

To reindex to a data stream destination, this argument must be create.

Supported values include:

index: Overwrite any documents that already exist.
create: Only index documents that do not already exist.

Values: index create

pipeline string

The name of the pipeline to use.

routing string · default: keep

By default, a document's routing is preserved unless it's changed by the script. If it is keep, the routing on the bulk request sent for each match is set to the routing on the match. If it is discard, the routing on the bulk request sent for each match is set to null. If it is =value, the routing on the bulk request sent for each match is set to all value specified after the equals sign (=). Not allowed when index.slice.enabled is true for the destination index; use _slice instead.

version_type enum

The versioning to use for the indexing operation.

Supported values include:

internal: Use internal versioning that starts at 1 and increments with each update or delete.
external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.
external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document. NOTE: The external_gte version type is meant for special use cases and should be used with care. If used incorrectly, it can result in loss of data.

Values: internal external external_gte

max_docs number

script string

The script to run to update the document source or metadata when reindexing.

source object required

The source you are copying from.

index union oneOf required: The name of the data stream, index, or alias you are copying from. It accepts a comma-separated list to reindex from multiple sources.; One of: string[] or string
query {} Query: The documents to reindex, which is defined with Query DSL.
remote object: A remote instance of Elasticsearch that you want to index from.
size number · default: 1000: The number of documents to index per batch. Use it when you are indexing from remote to ensure that the batches fit within the on-heap buffer, which defaults to a maximum size of 100 MB.
slice object: Slice the reindex request manually using the provided slice ID and total number of slices.
sort union oneOf deprecated: A comma-separated list of <field>:<direction> pairs to sort by before indexing. Use it in conjunction with max_docs to control what documents are reindexed.

Warning

Sort in reindex is deprecated. Sorting in reindex was never guaranteed to index documents in order and prevents further development of reindex such as resilience and performance improvements. If used in combination with max_docs, consider using a query filter instead.
_source union oneOf · default: true: If true, reindex all source fields. Set it to a list to reindex select fields.; One of:
runtime_mappings object

Response

application/json

batches number: The number of scroll responses that were pulled back by the reindex.
created number: The number of documents that were successfully created.
deleted number: The number of documents that were successfully deleted.
failures union oneOf[]: If there were any unrecoverable errors during the process, it is an array of those failures. If this array is not empty, the request ended because of those failures. Reindex is implemented using batches and any failure causes the entire process to end but all failures in the current batch are collected into the array. You can use the conflicts option to prevent the reindex from ending on version conflicts.; One of:
noops number: The number of documents that were ignored because the script used for the reindex returned a noop value for ctx.op.
retries object: The number of retries attempted by reindex.
requests_per_second number: The number of requests per second effectively run during the reindex.
slice_id number
slices object[]: Status of each slice if the reindex was sliced
task string
throttled_millis number: The number of milliseconds the request slept to conform to requests_per_second.
throttled_until_millis number: This field should always be equal to zero in a reindex response. It has meaning only when using the task API, where it indicates the next time (in milliseconds since epoch) that a throttled request will be run again in order to conform to requests_per_second.
timed_out boolean: If any of the requests that ran during the reindex timed out, it is true.
took number: The total milliseconds the entire operation took.
total number: The number of documents that were successfully processed.
updated number: The number of documents that were successfully updated. That is to say, a document with the same ID already existed before the reindex updated it.
version_conflicts number: The number of version conflicts that occurred.

Required privileges

Query parameters

Request body { }

Response

Request body