Migrate Elasticsearch data with minimal downtime using snapshots

When moving your data and services from one Elasticsearch cluster to another, such as to Elastic Cloud Hosted, Elastic Cloud Enterprise, or new on-premises hardware, you can migrate with minimal downtime using incremental snapshots.

Migrating with incremental snapshots is useful when you want to:

• Migrate all data in your indices and configurations, such as roles and Kibana dashboards, from the old cluster to a new cluster.
• Ensure data ingestion, such as Logstash or Beats, and data consumption, such as applications using Elasticsearch as a backend, seamlessly migrate to the new cluster.
• Maintain data consistency and minimize disruption.

Incremental snapshots save only the data that has changed since the last snapshot. The first snapshot is a full copy of the data. Each subsequent snapshot contains only the differences, which makes creating and restoring snapshots faster and more efficient over time.

When restoring, Elasticsearch copies only the missing data segments from the snapshot repository to the new cluster local storage. When the changes between snapshots are small, the restore process is significantly faster.

By taking and restoring incremental snapshots in sequence, you can keep a new cluster closely synchronized with the old cluster, allowing you to migrate most of your data ahead of time and minimize downtime during the final cutover.

For more information about migrating your data with snapshot and restore, check Snapshot and restore.

Before you migrate, review the prerequisites and requirements.

• Learn how to set up and manage snapshot repositories.
• If restoring to a different cluster, review Restore to a different cluster.
• As an alternative migration method, you can reindex from a remote cluster.

• Cluster size – The new cluster must be the same size or larger than the old cluster.
• Version compatibility – Both clusters must use compatible Elasticsearch versions. To check if your cluster versions are compatible, check Snapshot version compatibility.
• Storage requirements - Ensure sufficient repository storage. Usage grows with snapshot frequency and data volume.
• Network overhead – Transferring snapshots across networks, regions, or providers can be time consuming and incur costs.
• Resource usage – Snapshot and restore operations can be resource intensive and affect cluster performance.
• Custom integrations – Some integrations that use the Elasticsearch API directly, such as the Elasticsearch Java Client library, can require additional handling during cutover.

To complete the migration with minimal downtime, use incremental snapshots. While the exact sequence may differ depending on your infrastructure and operational requirements, you can use the recommended migration timeline as a reliable baseline that you can adapt. Adjust the steps and times to fit your own operational needs.

1. 09:00: Take the initial full snapshot of the old cluster. You can also take the initial full snapshot the day before.
2. 09:30: Restore the snapshot to the new cluster.
3. 09:55: Take another snapshot of the old cluster and restore it to the new cluster. Repeat this process until the snapshot and restore operations take only a few seconds or minutes. Remember that when restoring indices that already exist in the new cluster (for example, to pull in recently copied data), they first need to be closed. Also, remember that the restore operation automatically opens indices, so you will likely need to close the actively written ones after restoring them.
4. 10:15: Perform the final cutover.
   1. In the old cluster, pause indexing or set indices to read-only. For details on setting indices to read-only to safely pause indexing during migration, check Index lifecycle actions: Read-only.
   2. Take a final snapshot.
   3. Restore the snapshot to the new cluster. Again, remember that to restore indices that already exist, they first need to be closed.
   4. Change ingestion and querying to the new cluster.
   5. Open the indices in the new cluster.

Follow these steps to migrate your Elasticsearch data.

In this step, you’ll configure a read-only snapshot repository in the new cluster that points to the storage location used by the old cluster. This allows the new cluster to access and restore snapshots created in the original environment.

1. On your old Elasticsearch cluster, retrieve the snapshot repository configuration:

   GET /_snapshot/_all
   		

   Take note of the repository name and type (for example, s3, gcs, or azure), its base path, and any additional settings. Authentication credentials are often stored in the secure settings on each node. You’ll need to replicate all this configuration when registering the repository in the new ECH or ECE deployment.

   If your old cluster has multiple repositories configured, identify the repository with the snapshots containing the data that you want to migrate.

2. Add the snapshot repository on the new cluster.

   Considerations:

   • If you're migrating searchable snapshots, the repository name must be identical in the old and new clusters.
   • If the old cluster still has write access to the repository, register the repository as read-only to avoid data corruption. You can do this using the readonly: true option.

   To connect the existing snapshot repository to your new deployment, follow the steps for the storage provider where the repository is hosted:

   • Amazon Web Services (AWS) Storage
     • Store credentials in the keystore
     • Create the repository
   • Google Cloud Storage (GCS)
     • Store credentials in the keystore
     • Create the repository
   • Azure Blob Storage
     • Store credentials in the keystore.
     • Create the repository.
   Important

   Although these instructions focus on Elastic Cloud Hosted, you should follow the same steps for Elastic Cloud Enterprise by configuring the repository directly at the deployment level.

   Do not configure the repository as an ECE-managed repository, which is intended for automatic snapshots of deployments. In this case, you need to add a custom repository that already contains snapshots from another cluster.

After you have registered and verified the repository, you are ready to restore any data from any of its snapshots to your new cluster.

You can run a restore operation using the Kibana Management UI, or using the Elasticsearch API. Refer to Restore a snapshot for more details, including API-based examples.

For details about the contents of a snapshot, refer to Snapshot and restore > Snapshot contents.

To start the restore process:

1. Open Kibana and go to the Snapshot and Restore management page using the navigation menu or the global search field.

2. Under the Snapshots tab, you can find the available snapshots from your newly added snapshot repository. Select any snapshot to view its details, and from there you can choose to restore it.

3. Select Restore.

4. Select the index or indices you wish to restore.

5. Optionally, configure additional restore options, such as Restore aliases, Restore global state, or Restore feature state.

6. Select Restore snapshot to begin the process.

7. Verify that each restored index is available in your deployment. You can do this using Kibana Index Management UI, or by running this query:

   GET INDEX_NAME/_search?pretty
   		

   If you have restored many indices you can also run GET _cat/indices?s=index to list all indices for verification.

To get expert assistance for your Elasticsearch migrations, go to Elastic Professional Services.