Migrate Elasticsearch data using the reindex API

The reindex API offers a convenient way for you to copy your Elasticsearch documents from a source index, data stream, or alias in one deployment to another.

This guide gives the example of reindexing a full index from an Elastic Cloud Hosted deployment to an Elastic Cloud Serverless project using the remote host parameters as shown in the reindex from remote example. These steps can be adapted to other deployment types as well. When you copy your data to deployment types other than Serverless, there are additional considerations to make note of.

For more advanced use cases, including data modification using scripts or ingest pipelines, refer to the Reindex indices examples and the reindex API documentation.

• An Elastic Cloud Hosted deployment with data to migrate

• An Elastic Cloud Serverless project configured and running

• An API key for authentication with the Elastic Cloud Hosted deployment

  Basic authentication can be used in place of an API key, but an API key is recommended as a more secure option.

• The target deployment must be able to access your original source cluster to perform the reindex operation. Access is controlled by the Elasticsearch reindex.remote.whitelist user setting.

  Domains matching the patterns ["*.io:*", "*.com:*"] are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define reindex.remote.whitelist.

  Otherwise, if your remote endpoint is not covered by the default patterns, adjust the setting to add the remote Elasticsearch cluster as an allowed host:

  1. From your deployment menu, go to the Edit page.

  2. In the Elasticsearch section, select Manage user settings and extensions. For deployments with existing user settings, you might have to expand the Edit elasticsearch.yml caret for each node type instead.

  3. Add the following reindex.remote.whitelist: [REMOTE_HOST:PORT] user setting, where REMOTE_HOST is a pattern matching the URL for the remote Elasticsearch host that you are reindexing from, and PORT is the host port number. Do not include the https:// prefix.

     If you override the parameter, it replaces the defaults: ["*.io:*", "*.com:*"]. If you still want these patterns to be allowed, you need to specify them explicitly in the value.

     For example:

     reindex.remote.whitelist: ["*.us-east-1.aws.found.io:9243", "*.com:*"]

  4. Save your changes.

1. In your Elastic Cloud Hosted deployment:

   1. Navigate to the deployment home page and copy the Elasticsearch endpoint.

   2. Go to the Index Management page in the navigation menu or use the global search field.

   3. Use the search field to identify the indices that you want to migrate.

2. In your Serverless project:

   1. Open the Developer Tools Console.

   2. Call the reindex API to migrate your index. If you have multiiple indices to migrate, we recommend doing a separate call for each.

      Example: Reindex from an Elastic Cloud Hosted deployment to a {[serverless-short]} project using an API key:

      POST _reindex
      {
        "source": {
          "remote": {
            "host": "https://<SERVERLESS_HOST_URL>:443",
            "api_key": "<ECH_API_KEY>"
          },
          "index": "<SOURCE_INDEX>"
        },
        "dest": {
          "index": "<DESTINATION_INDEX>"
        }
      }
      		

      1. The URL for your Serverless project. This is the Elasticsearch endpoint that you copied in Step 1. If you're migrating to, for example, an Elastic Cloud Hosted cluster, you can modify the remote host address accordingly.
      2. The API key for authenticating the connection to your Elastic Cloud Hosted deployment.
      3. The source index to copy from your Elastic Cloud Hosted deployment.
      4. The destination index in your Serverless project.

   3. Verify that the new index is present:

      GET INDEX-NAME/_search?pretty
      		

   4. If you are not planning to reindex more data from the remote and you configured a reindex.remote.whitelist user setting, that setting can now be removed.

The page demonstrates copying data from an Elastic Cloud Hosted deployment to Serverless. When you use the reindex API to copy data across other deployment types, consider the following:

• If you're migrating from a self-managed cluster that uses non–publicly trusted TLS certificates, including self-signed certificates and certificates signed by a private certificate authority (CA), refer to our guide Reindex from a self-managed cluster using a private CA.

• The target deployment must be able to access your original source cluster to perform the reindex operation. When you migrate to Serverless, access to all Elastic Cloud Hosted endpoints is allowed automatically. For migrating to other deployment types, access is controlled by the Elasticsearch reindex.remote.whitelist user setting.

  Domains matching the patterns ["*.io:*", "*.com:*"] are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define reindex.remote.whitelist.

  Otherwise, if your remote endpoint is not covered by the default patterns, adjust the setting to add the remote Elasticsearch cluster as an allowed host:

  1. From your deployment menu, go to the Edit page.

  2. In the Elasticsearch section, select Manage user settings and extensions. For deployments with existing user settings, you might have to expand the Edit elasticsearch.yml caret for each node type instead.

  3. Add the following reindex.remote.whitelist: [REMOTE_HOST:PORT] user setting, where REMOTE_HOST is a pattern matching the URL for the remote Elasticsearch host that you are reindexing from, and PORT is the host port number. Do not include the https:// prefix.

     If you override the parameter, it replaces the defaults: ["*.io:*", "*.com:*"]. If you still want these patterns to be allowed, you need to specify them explicitly in the value.

     For example:

     reindex.remote.whitelist: ["*.us-east-1.aws.found.io:9243", "*.com:*"]

  4. Save your changes.