Prepare to upgrade

Elastic Stack ECE ECK Elastic Cloud Hosted Self Managed

Before you upgrade, review and complete the necessary preparation steps, which vary by version.

Important

Upgrading from a release candidate build, such as 9.0.0-rc1, is unsupported. Use pre-releases only for testing in a temporary environment.

Prepare to upgrade from 8.x

To upgrade from 8.17.0 or earlier to 9.0.0, you must first upgrade to the latest 8.18 patch release. This allows you to use the Upgrade Assistant to identify and resolve issues, reindex indices created before 8.0.0, and perform a rolling upgrade. Upgrading to the latest 8.18 patch release is required even if you choose a full Elasticsearch cluster restart. If you're using 7.x and earlier, you may need to complete multiple upgrades or perform a full-cluster restart to reach the latest 8.18 patch release before upgrading to 9.0.0.

Alternatively, you can create a 9.0 deployment and reindex from remote. For more information, refer to Reindex to upgrade.

Note

For flexible upgrade scheduling, 8.18.0 Beats and Logstash are compatible with 9.0.0 Elasticsearch. By default, 8.x Elasticsearch clients are compatible with 9.0.0 and use REST API compatibility to maintain compatibility with the 9.0.0 Elasticsearch server.

Review the following best practices to upgrade your deployments.

Run the Upgrade Assistant, which identifies deprecated settings in your configuration and guides you through resolving issues that could prevent a successful upgrade. The Upgrade Assistant also helps resolve issues with older indices created before version 8.0.0, providing the option to reindex older indices or mark them as read-only. To prevent upgrade failures, we strongly recommend you do not skip this step.

Note

Depending on your setup, reindexing can change your indices, and you may need to update alerts, transforms, or other code targeting the old index.
Before you change configurations or reindex, ensure you have a current snapshot.

Tip

Tip: In 8.3.0 and later, snapshots are generally available as simple archives. Use the archive functionality to search snapshots from 5.0.0 and later without needing an old Elasticsearch cluster. This ensures that your Elasticsearch data remains accessible after upgrading, without requiring a reindex process.

To successfully upgrade, resolve all critical issues. If you make additional changes, create a snapshot to back up your data.
To identify if your applications use unsupported features or behave differently in 9.0.0, review the deprecation logs in the Upgrade Assistant.
Major version upgrades can include breaking changes that require additional steps to ensure your applications function as expected. Review the breaking changes for each product you use to learn more about potential impacts on your application. Ensure you test with the new version before upgrading existing deployments.
Make the recommended changes to ensure your clients continue operating as expected after the upgrade.

Note

As a temporary solution, use the 8.x syntax to submit requests to 9.0.0 with REST API compatibility mode. While this allows you to submit requests using the old syntax, it doesn’t guarantee the same behavior. REST API compatibility should serve as a bridge during the upgrade, not a long-term solution. For more details on how to effectively use REST API compatibility during an upgrade, refer to REST API compatibility.
If you use Elasticsearch plugins, ensure each plugin is compatible with the Elasticsearch version you're upgrading.
Before upgrading your production deployment, we recommend creating a 9.0.0 test deployment and testing the upgrade in an isolated environment. Ensure the test and production environments use the same settings.

Important

After you upgrade, you cannot downgrade Elasticsearch nodes. If you can't complete the upgrade process, you must restore from the snapshot.
If you use a separate monitoring cluster, upgrade the monitoring cluster before the production cluster. The monitoring cluster and the clusters being monitored should be running the same version of the Elastic Stack. Monitoring clusters cannot monitor production clusters running newer versions of the Elastic Stack. If necessary, the monitoring cluster can monitor production clusters running the latest release of the previous major version.

Note

If you use cross-cluster search, 9.0.0 and later can search only remote clusters running the previous minor version, the same version, or a newer minor version in the same major version. For more information, refer to Cross-cluster search.

If you use cross-cluster replication, a cluster that contains follower indices must run the same or newer (compatible) version as the remote cluster. For more information and to view the version compatibility matrix, refer to Cross-cluster replication. To view your remote clusters in Kibana, go to Stack Management > Remote Clusters.

In addition, if you have CCR data streams, refer to Upgrade uni-directional cross-cluster replication clusters with followed data streams for specific instructions on reindexing.
To reduce overhead on the cluster during the upgrade, close machine learning jobs. Although machine learning jobs can run during a rolling upgrade, doing so increases the cluster workload.
If you have .ml-anomalies-*anomaly detection result indices created in Elasticsearch 7.x, reindex them, mark them as read-only, or delete them before you upgrade to 9.0.0. For more information, refer to Migrate anomaly detection results.
If you have transform destination indices created in Elasticsearch 7.x, reset, reindex, or delete them before you upgrade to 9.0.0. For more information, refer to Migrate transform destination indices.

Reindex to upgrade

Optionally create a 9.0.0 deployment and reindex from remote:

Provision an additional deployment running 9.0.0.
To reindex your data into the new Elasticsearch cluster, use the reindex documents API and temporarily send new index requests to both clusters.
Verify the new cluster performs as expected, fix any problems, and then permanently swap in the new cluster.
Delete the old deployment. On Elastic Cloud, you are billed only for the time the new deployment runs in parallel with your old deployment. Usage is billed on an hourly basis.

Upgrade uni-directional cross-cluster replication clusters with followed data streams

When moving to a new major version of Elasticsearch, you must perform specific actions to ensure that indices — including those that back a data stream — are compatible with the latest Lucene version. With a CCR-enabled cluster, consider whether you want to keep your older data writable or read-only to ensure you make changes to the cluster in the correct order.

Note

CCR-replicated data streams only allow writing to the most recent backing index, as ILM automatically injects an unfollow event after every rollover. Therefore, you can't reindex CCR-followed data streams since older backing indices are no longer replicated by CCR.

Migrate read-only historical data

If you want to keep your older data as read-only:

Issue a rollover for all replicated data streams on the follower cluster to ensure the write index is compatible with the version you're upgrading to.
Run the Upgrade Assistant on the CCR follower cluster and resolve any data stream deprecation notices, selecting the option to not reindex and allow the backing indices to become read-only after upgrading.
Upgrade the CCR follower cluster to the appropriate version. Ensure you take a snapshot before starting the upgrade.
Run the Upgrade Assistant on the CCR leader cluster and repeat the same steps as the follower cluster, opting not to reindex.
Upgrade the leader cluster and ensure CCR replication is healthy.

Migrate read-write historical data

If you need to write directly to non-write backing indices of data streams in a CCR-replicated cluster pair:

Before upgrading, remove the data stream and all follower indices from the CCR follower.
Run the Upgrade Assistant and select the “Reindex” option.
Once the reindexing is complete and the leader cluster is upgraded, re-add the newly reindexed backing indexes as follower indices on the CCR follower.

Migrate anomaly detection results

Reindex, mark as read-only, or delete the .ml-anomalies-* anomaly detection result indices created in Elasticsearch 7.x.

Reindex: While anomaly detection results are being reindexed, jobs continue to run and process new data. You cannot delete an anomaly detection job that stores results in the index until the reindexing is complete.

Mark indices as read-only: This is useful for large indexes that contain the results of one or two anomaly detection jobs. If you delete these jobs later, you cannot create a new job with the same name.

Delete: Delete jobs that are no longer needed in the Machine Learning app in Kibana. The result index is deleted when all jobs that store results in it have been deleted.

To identify indices that require action, use the Deprecation info API:

GET /.ml-anomalies-*/_migration/deprecations

The response contains the list of critical deprecation warnings in the index_settings section:

		"index_settings": {
  ".ml-anomalies-shared": [
    {
      "level": "critical",
      "message": "Index created before 8.0",
      "url": "https://ela.st/es-deprecation-8-reindex",
      "details": "This index was created with version 7.8.23 and is not compatible with 9.0. Reindex or remove the index before upgrading.",
      "resolve_during_rolling_upgrade": false
    }
  ]
}

	

For an index with less than 10 GB that contains results from multiple jobs that are still required, we recommend reindexing into a new format using the UI. You can use the Get index information API to obtain the size of an index:

GET _cat/indices/.ml-anomalies-custom-example?v&h=index,store.size

Use the Upgrade Assistant to initiate reindexing.

If an index size is greater than 10 GB, we recommend using the Reindex API. Reindexing consists of the following steps:

Set the original index to read-only.

PUT .ml-anomalies-custom-example/_block/read_only

Create a new index from the legacy index.

		POST _create_from/.ml-anomalies-custom-example/.reindexed-v9-ml-anomalies-custom-example

	

Reindex documents. To accelerate the reindexing process, we recommend setting the number of replicas to 0 before reindexing, then set it back to the original number once completed.
1. Get the number of replicas.
```
GET /.reindexed-v9-ml-anomalies-custom-example/_settings
```
Note the number of replicas in the response. For example:
```
{
  ".reindexed-v9-ml-anomalies-custom-example": {
    "settings": {
      "index": {
        "number_of_replicas": "1",
        "number_of_shards": "1"
      }
    }
  }
}
```
1. Set the number of replicas to 0. You must also set the auto_expand_replicas parameter to false to change the number of replicas:
```
PUT /.reindexed-v9-ml-anomalies-custom-example/_settings
{
  "index": {
    "auto_expand_replicas": false,
    "number_of_replicas": 0
  }
}
```
1. Start the reindexing process in asynchronous mode.
```
POST _reindex?wait_for_completion=false
{
  "source": {
    "index": ".ml-anomalies-custom-example"
  },
  "dest": {
    "index": ".reindexed-v9-ml-anomalies-custom-example"
  }
}
```
The response will contain a task_id. You can check when the task is completed using the following command:
```
GET _tasks/<task_id>
```
1. Set the number of replicas to the original number when the reindexing is finished. Optionally, you can set the auto_expand_replicas parameter back to its default value (0-1) to allow the number of replicas to be automatically adjusted based on the number of data nodes in the cluster.
```
PUT /.reindexed-v9-ml-anomalies-custom-example/_settings
{
  "index": {
    "number_of_replicas": "<original_number_of_replicas>",
    "auto_expand_replicas": "0-1"
  }
}
```
Get the aliases the original index is pointing to.

GET .ml-anomalies-custom-example/_alias

The response may contain multiple aliases if the results of multiple jobs are stored in the same index.

		{
  ".ml-anomalies-custom-example": {
    "aliases": {
      ".ml-anomalies-example1": {
        "filter": {
          "term": {
            "job_id": {
              "value": "example1"
            }
          }
        },
        "is_hidden": true
      },
      ".ml-anomalies-example2": {
        "filter": {
          "term": {
            "job_id": {
              "value": "example2"
            }
          }
        },
        "is_hidden": true
      }
    }
  }
}

	

Now you can reassign the aliases to the new index and delete the original index in one step. Note that when adding the new index to the aliases, you must use the same filter and is_hidden parameters as the original index.

		POST _aliases
{
  "actions": [
    {
      "add": {
        "index": ".reindexed-v9-ml-anomalies-custom-example",
        "alias": ".ml-anomalies-example1",
        "filter": {
          "term": {
            "job_id": {
              "value": "example1"
            }
          }
        },
        "is_hidden": true
      }
    },
    {
      "add": {
        "index": ".reindexed-v9-ml-anomalies-custom-example",
        "alias": ".ml-anomalies-example2",
        "filter": {
          "term": {
            "job_id": {
              "value": "example2"
            }
          }
        },
        "is_hidden": true
      }
    },
    {
      "remove": {
        "index": ".ml-anomalies-custom-example",
        "aliases": ".ml-anomalies-*"
      }
    },
    {
      "remove_index": {
        "index": ".ml-anomalies-custom-example"
      }
    },
    {
      "add": {
        "index": ".reindexed-v9-ml-anomalies-custom-example",
        "alias": ".ml-anomalies-custom-example",
        "is_hidden": true
      }
    }
  ]
}

	

Legacy indices created in Elasticsearch 7.x can be made read-only and supported in Elasticsearch 9.x. Making an index with a large amount of historical results read-only allows for a quick migration to the next major release, since you don’t have to wait for the data to be reindexed into the new format. However, it has the limitation that even after deleting an anomaly detection job, the historical results associated with this job are not completely deleted. Therefore, the system will prevent you from creating a new job with the same name.

To set the index as read-only, add the write block to the index:

PUT .ml-anomalies-custom-example/_block/write

Indices created in Elasticsearch 7.x that have a write block will not raise a critical deprecation warning.

If an index contains results of the jobs that are no longer required. To list all jobs that stored results in an index, use the terms aggregation:

		GET .ml-anomalies-custom-example/_search
{
  "size": 0,
  "aggs": {
    "job_ids": {
      "terms": {
        "field": "job_id",
        "size": 100
      }
    }
  }
}

	

The jobs can be deleted in the UI. After the last job is deleted, the index will be deleted as well.

Migrate transform destination indices

The transform destination indices created in Elasticsearch 7.x must be either reset, reindexed, or deleted before upgrading to 9.x.

Resetting: You can reset the transform to delete all state, checkpoints, and the destination index (if it was created by the transform). The next time you start the transform, it will reprocess all data from the source index, creating a new destination index in Elasticsearch 8.x compatible with 9.x. However, if data had been deleted from the source index, you will lose all previously computed results that had been stored in the destination index.

Reindexing: You can reindex the destination index and then update the transform to write to the new destination index. This is useful if there are results that you want to retain that may not exist in the source index. To prevent the transform and reindex tasks from conflicting with one another, you can either pause the transform while the reindex runs, or write to the new destination index while the reindex backfills old results.

Deleting: You can delete any transform that's no longer being used. Once the transform is deleted, you can delete the destination index or make it read-only.

To identify indices that require action, use the Deprecation info API:

GET /_migration/deprecations

The response contains the list of critical deprecation warnings in the index_settings section:

		"index_settings": {
    "my-destination-index": [
      {
        "level": "critical",
        "message": "One or more Transforms write to this index with a compatibility version < 9.0",
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/migrating-9.0.html#breaking_90_transform_destination_index",
        "details": "Transforms [my-transform] write to this index with version [7.8.23].",
        "resolve_during_rolling_upgrade": false
      }
    ]
  }

	

If the index was created by the transform, you can use the Transform Reset API to delete the destination index and recreate it the next time the transform runs.

If the index was not created by the transform and you still want to reset it, you can manually delete and recreate the index, then call the Reset API.

POST _transform/my-transform/_reset

When the Upgrade Assistant reindexes the documents, Kibana will put a write block on the old destination index, copy the results to a new index, delete the old index, and create an alias to the new index. During this time, the transform will pause and wait for the destination to become writable again. If you do not want the transform to pause, continue to reindexing the transform’s destination index while the transform is running.

If an index size is less than 10 GB, we recommend using the Upgrade Assistant to automatically migrate the index. You can use the Get index information API to obtain the size of an index:

GET _cat/indices/.transform-destination-example?v&h=index,store.size

Use the Upgrade Assistant to initiate reindexing.

If an index size is greater than 10 GB, we recommend using the Reindex API. Reindexing consists of the following steps:

Set the original index to read-only.

PUT .transform-destination-example/_block/read_only

Create a new index from the legacy index.

		POST _create_from/.transform-destination-example/.reindexed-v9-transform-destination-example

	

Reindex documents. To accelerate the reindexing process, we recommend setting the number of replicas to 0 before reindexing, then set it back to the original number once completed.

Get the number of replicas.

GET /.reindexed-v9-transform-destination-example/_settings

Note the number of replicas in the response. For example:

		{
  ".reindexed-v9-transform-destination-example": {
    "settings": {
      "index": {
        "number_of_replicas": "1",
        "number_of_shards": "1"
      }
    }
  }
}

	

Set the number of replicas to 0.

		PUT /.reindexed-v9-transform-destination-example/_settings
{
  "index": {
    "number_of_replicas": 0
  }
}

	

Start the reindexing process in asynchronous mode.

		POST _reindex?wait_for_completion=false
{
  "source": {
    "index": ".transform-destination-example"
  },
  "dest": {
    "index": ".reindexed-v9-transform-destination-example"
  }
}

	

The response will contain a task_id. You can check when the task is completed using the following command:

GET _tasks/<task_id>

Set the number of replicas to the original number when the reindexing is finished.

		PUT /.reindexed-v9-transform-destination-example/_settings
{
  "index": {
    "number_of_replicas": "<original_number_of_replicas>"
  }
}

	

Get the aliases the original index is pointing to.

GET .transform-destination-example/_alias

The response may contain multiple aliases if the results of multiple jobs are stored in the same index.

		{
  ".transform-destination-example": {
    "aliases": {
      ".transform-destination-example1": {
        "filter": {
          "term": {
            "job_id": {
              "value": "example1"
            }
          }
        },
        "is_hidden": true
      },
      ".transform-destination-example2": {
        "filter": {
          "term": {
            "job_id": {
              "value": "example2"
            }
          }
        },
        "is_hidden": true
      }
    }
  }
}

	

Now you can reassign the aliases to the new index and delete the original index in one step. Note that when adding the new index to the aliases, you must use the same filter and is_hidden parameters as the original index.

		POST _aliases
{
  "actions": [
    {
      "add": {
        "index": ".reindexed-v9-transform-destination-example",
        "alias": ".transform-destination-example1",
        "filter": {
          "term": {
            "job_id": {
              "value": "example1"
            }
          }
        },
        "is_hidden": true
      }
    },
    {
      "add": {
        "index": ".reindexed-v9-transform-destination-example",
        "alias": ".transform-destination-example2",
        "filter": {
          "term": {
            "job_id": {
              "value": "example2"
            }
          }
        },
        "is_hidden": true
      }
    },
    {
      "remove": {
        "index": ".transform-destination-example",
        "aliases": ".transform-destination-*"
      }
    },
    {
      "remove_index": {
        "index": ".transform-destination-example"
      }
    },
    {
      "add": {
        "index": ".reindexed-v9-transform-destination-example",
        "alias": ".transform-destination-example",
        "is_hidden": true
      }
    }
  ]
}

	

If you want the transform and the reindex task to write documents to the new destination index at the same time:

Set the original index to read-only.

POST _create_from/my-destination-index/my-new-destination-index

Update the transform to write to the new destination index:

		POST _transform/my-transform/_update
{
 "dest": {
   "index": "my-new-destination-index"
 }
}

	

Reindex documents. To accelerate the reindexing process, we recommend setting the number of replicas to 0 before reindexing, then set it back to the original number once completed.
1. Get the number of replicas.
```
GET /my-destination-index/_settings
```
1. Note the number of replicas in the response. For example:
```
{
  "my-destination-index":: {
    "settings": {
      "index": {
        "number_of_replicas": "1",
        "number_of_shards": "1"
      }
    }
  }
}
```
1. Set the number of replicas to 0.
```
PUT /my-destination-index/_settings
{
  "index": {
    "number_of_replicas": 0
  }
}
```
1. Start the reindexing process in asynchronous mode. Set the op_type to create so the reindex does not overwrite work that the transform is doing.
```
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "index": "my-destination-index"
  },
  "dest": {
    "index": "my-new-destination-index",
    "op_type": "create"
  }
}
```
The response will contain a task_id. You can check when the task is completed using the following command:
```
GET _tasks/<task_id>
```
1. Set the number of replicas to the original number when the reindexing is finished.
```
PUT /my-new-destination-index/_settings
{
  "index": {
    "number_of_replicas": "<original_number_of_replicas>"
  }
}
```

Get the aliases the original index is pointing to.

		GET my-destination-index/_alias
{
  "my-destination-index": {
    "aliases": {
      "my-destination-alias": {},
    }
  }
}

	

Now you can reassign the aliases to the new index and delete the original index in one step. Note that when adding the new index to the aliases, you must use the same filter and is_hidden parameters as the original index.

		POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "my-new-destination-index",
        "alias": "my-destination-alias"
      }
    },
    {
      "remove": {
        "index": "my-destination-index",
        "aliases": "my-destination-alias"
      }
    },
    {
      "remove_index": {
        "index": "my-destination-index"
      }
    }
  ]
}

	

You can use the Transform Delete API to delete the transform and stop it from writing to the destination index.

DELETE _transform/my-transform

If the destination index is no longer needed, it can be deleted with the transform.

DELETE _transform/my-transform?delete_dest_index