Configure data tiers for self-managed and Elastic Cloud on Kubernetes deployments

Whether you operate Elasticsearch on your own infrastructure or on Kubernetes with Elastic Cloud on Kubernetes, data tiers are expressed through each node’s data role. You choose which tiers the cluster offers by assigning the corresponding data_* roles to nodes or to ECK node sets.

• Review Elasticsearch data tiers so you match tiers to your workload.
• Understand how node roles map to hardware and allocation for each tier.

1. For each node, decide which data tier or tiers it should participate in (for example data_hot, data_warm, data_cold, data_frozen, or data_content).
2. Set node.roles in that node’s elasticsearch.yml to include the corresponding data_* roles (and any other roles the node should have, such as ingest or master).
3. Restart the node or apply your configuration rollout process so the new roles take effect.

For example, the highest-performance nodes in a cluster might be assigned to both the hot and content tiers:

node.roles: ["data_hot", "data_content"]
		

Elasticsearch settings that you normally put in elasticsearch.yml are set for each nodeSet under spec.nodeSets[?].config in the Elasticsearch resource manifest. Assign node.roles there to define each group of pods’ tiers. For example, you can assign the following tiers:

spec:
  nodeSets:
  - name: hot-content
    count: 3
    config:
      node.roles: ["data_hot", "data_content", "ingest"]
		

Some settings are managed by Elastic Cloud on Kubernetes; avoid overriding those. For the full mapping between manifest structure and Elasticsearch configuration, see Node configuration.

Follow this section when you need to remove the warm, cold, or frozen tier from a self-managed or Elastic Cloud on Kubernetes deployment. The hot and content tiers are required and cannot be removed. If you remove nodes assigned the data_hot or data_content role, ensure that the corresponding role remains assigned to other nodes.

The steps differ depending on whether the tier holds regular indices or searchable snapshot indices (typical for cold or frozen when using index lifecycle management (ILM)).

1. Identify which nodes belong to the data tier you want to remove:

   GET /_nodes?filter_path=nodes.*.name,nodes.*.ip,nodes.*.roles
   		

   Note the names of the nodes with the corresponding data_* role.

   Tip

   For Elastic Cloud on Kubernetes, also identify every nodeSet in your Elasticsearch manifest that has the data_* role associated with the tier you want to remove.

2. Check whether the tier you are removing holds regular indices, searchable snapshots, or both. Use the guidance for the tier you are removing:

   • Warm tier: This tier typically holds regular indices. Follow Remove a tier with regular indices unless you have manually mounted searchable snapshots on the tier.

   • Cold tier: This tier can hold regular indices or fully mounted searchable snapshots. Check for standard ILM-managed searchable snapshot indices:

     GET /_cat/indices/restored-*?expand_wildcards=all
     		

     For each returned index, check its current data tier preference to determine whether it is on the tier you are removing.

     Exclude any fully mounted indices associated with the hot tier from the removal inventory. The hot tier is required and is not removed by this procedure.

   • Frozen tier: This tier only holds partially mounted searchable snapshots. Check for standard ILM-managed indices:

     GET /_cat/indices/partial-*?expand_wildcards=all
     		

   Note

   Manually mounted searchable snapshots might not use the standard restored-* or partial-* prefixes. If you mounted snapshots manually, adapt the index names or patterns in these requests to match your configuration.

   • If the tier does not contain any searchable snapshot indices, follow Remove a tier with regular indices.
   • If the tier contains searchable snapshot indices, review Remove a tier with searchable snapshots and select the appropriate procedure based on how the indices are mounted. If regular indices also remain, restore or move the searchable snapshot indices first, but do not remove the nodes. Then return to the regular indices procedure.

To learn more about ILM or shard allocation filtering, refer to Create your index lifecycle policy, Managing the index lifecycle, and Shard allocation filters.

This section covers the removal of a tier that holds regular indices. The goal is to ensure all shard allocation rules allow the data to move to other tiers, and then vacate and remove the nodes. You also need to temporarily stop ILM to prevent new indices from being routed to the tier while you work.

1. Stop ILM to prevent new indices from being routed to the tier while you work.

   POST /_ilm/stop
   GET /_ilm/status
   		

   Wait until operation_mode is STOPPED before proceeding.

2. Determine which shards are allocated to the nodes you want to remove.

   GET /_cat/shards?v&h=index,shard,prirep,state,node
   		

   Filter the output by the node names you identified in Before you remove a data tier.

3. Check and update index allocation rules.

   ILM and manual index configurations use different index-level shard allocation filters to control shard placement. For every index that has shards on the nodes you are removing, check its allocation settings and apply the relevant substeps:

   GET /my-index/_settings
   		

   1. _tier_preference based rules.

      Data tier-based ILM policies use index.routing.allocation.include._tier_preference to express shard placement as an ordered list of preferred tiers. Indices using this method have settings similar to the following example:

      {
      ...
          "routing": {
              "allocation": {
                  "include": {
                      "_tier_preference": "data_warm,data_hot"
                  }
              }
          }
      ...
      }
      		

      1. The example represents an index in the warm tier.

      Before manually vacating the nodes, update _tier_preference so that the tier where you want the data to move is the first available tier in the list. This allows Elasticsearch to begin relocating the shards to that tier before the nodes are removed.

      Update the setting based on where you want to move the data:

      • To move the data to an existing fallback tier, remove the tier being removed from the list. For example, when removing the warm tier, change data_warm,data_hot to data_hot.
      • To move the data to a later lifecycle tier, add that tier before the tier being removed. For example, when removing the warm tier, change data_warm,data_hot to data_cold,data_warm,data_hot.

      The following example moves data from warm to cold:

      PUT /my-index/_settings
      {
          "routing": {
            "allocation": {
              "include": {
                  "_tier_preference": "data_cold,data_warm,data_hot"
              }
            }
          }
      }
      		

      1. You can also use data_cold,data_hot. Both values move the data to cold, but omitting data_warm removes that tier from the fallback sequence.
      Note

      Do not use the frozen tier as a fallback for regular indices. It is reserved for partially mounted searchable snapshots.

   2. Update node attribute allocation requirement rules.

      Older ILM policies and some custom configurations use index.routing.allocation.require to pin shards to nodes with a specific attribute. Indices using this method have settings similar to the following example:

      {
      ...
          "routing": {
              "allocation": {
                  "require": {
                      "data": "warm"
                  }
              }
          }
      ...
      }
      		

      Unlike _tier_preference, a require rule is a hard constraint: if the required nodes are gone, the shard becomes unassigned and Elasticsearch cannot move it automatically. You must remove or redirect these rules before vacating the nodes. To remove the attribute requirement:

      PUT /my-index/_settings
      {
          "routing": {
            "allocation": {
              "require": {
                  "data": null
              }
            }
          }
      }
      		

      Alternatively, redirect the index to a different tier by setting require to the desired attribute value. For example, to move an index to nodes with data attribute of cold:

      PUT /my-index/_settings
      {
          "routing": {
            "allocation": {
              "require": {
                  "data": "cold"
              }
            }
          }
      }
      		

      Adjust the data value to match the custom node attributes and index-level shard allocation filters your indices already use. You cannot send regular indices to the frozen tier.

      If you remove the require rule, Elasticsearch does not re-allocate shards immediately. They move when you vacate the nodes in the next step. If you redirect require to a different attribute value, re-allocation starts immediately.

   3. Review other custom allocation rules.

      If indices on the nodes being removed use other index-level shard allocation filters, such as include, exclude, or require configurations not covered earlier, update or remove any rules that would prevent shards from moving to the intended nodes. You can preserve rules unrelated to the tier removal.

      The following example removes all _name-based custom allocation filters from an index:

      PUT /my-index/_settings
      {
        "index.routing.allocation.require._name": null,
        "index.routing.allocation.include._name": null,
        "index.routing.allocation.exclude._name": null
      }
      		

4. Vacate the nodes.

   Note

   On Elastic Cloud on Kubernetes, removing a nodeSet from the Elasticsearch manifest causes Elastic Cloud on Kubernetes to migrate data away from its nodes before removing the underlying StatefulSet, as described in Cluster upgrade patterns. If the allocation rules in the previous step are correctly updated, you can skip the manual vacate and proceed directly to removing the nodeSet. However, we recommend completing the manual vacate first because it gives you more control and visibility over the relocation process.

   To vacate the nodes manually, exclude them from shard allocation by name. Elasticsearch then relocates their remaining shards to other eligible nodes:

   PUT /_cluster/settings
   {
     "persistent": {
       "cluster.routing.allocation.exclude._name": "<node-name-1>,<node-name-2>"
     }
   }
   		

   1. If _name exclusions are already configured, include their existing values in the comma-separated list to preserve them.
   Important

   Wait until GET /_cat/allocation?v=true&s=node shows that no shards remain on those nodes before proceeding. Updating settings starts the relocation process, but you must wait until shard allocation and recovery finish. If shards stay on the original tier, use the cluster allocation explain API to determine the cause. Common causes include disk watermarks or index.routing.allocation.total_shards_per_node limit reached on the destination nodes.

5. Remove the nodes.

   After confirming that no shards remain on the nodes, remove them using the instructions for your deployment type.

   Stop the Elasticsearch service on each node to be removed and decommission the host. For step-by-step instructions, refer to Add or remove Elasticsearch nodes.

   If an ElasticsearchAutoscaler policy manages the nodeSet, remove the matching policy before removing the nodeSet or setting its count to 0. Otherwise, autoscaling might change the nodeSet count while you complete this procedure. Refer to Autoscaling in ECK.

   Remove the nodeSet from your Elasticsearch manifest, or set its count to 0. If you skipped the manual vacate, Elastic Cloud on Kubernetes migrates the remaining data before safely stopping the pods.

6. Wait until GET /_cat/nodes?v shows no nodes from the removed tier remaining in the cluster.

   If you ran the manual vacate, remove the deleted node names from the exclusion rule only after the nodes have left the cluster. Restore any _name exclusions that existed before the vacate. If none existed, clear the setting:

   PUT /_cluster/settings
   {
     "persistent": {
       "cluster.routing.allocation.exclude._name": null
     }
   }
   		

   Confirm that GET /_cluster/health reports green.

7. Review your ILM policies and consider removing references to the deleted tier to keep them consistent with the cluster topology. This is especially important in older clusters where ILM uses node-attribute-based allocation, as those policies cannot run phases that target nodes that no longer exist.

   For guidance on updating policies, refer to Configure a lifecycle policy.

8. Re-enable ILM:

   POST /_ilm/start
   		

9. Verify that ILM is running and that no indices report errors related to the removed tier:

   GET /_ilm/status
   GET /_all/_ilm/explain?human=true&expand_wildcards=all&only_errors=true
   		

   Confirm that operation_mode is RUNNING. Investigate any reported errors and verify that no policy still attempts to allocate data to the removed tier.

This section explains how to remove a data tier that contains searchable snapshot indices. Your options for preserving the data depend on how the indices are mounted:

• Partially mounted searchable snapshots on the frozen tier: The only way to keep the data available as indices when removing the frozen tier is to restore all partially mounted indices as regular indices on another tier. Follow the steps in this section to restore the indices and remove the original searchable snapshot indices and source snapshots.
• Fully mounted searchable snapshots on the cold tier: To keep the indices as searchable snapshots, move them to another tier by following Remove a tier with regular indices. This works because fully mounted indices follow the same shard allocation rules as regular indices. Alternatively, follow the steps in this section to restore them as regular indices on another tier.

If you do not need to preserve the data, delete the searchable snapshot indices before removing the tier.

The following procedure captures the snapshot metadata, restores the indices as regular indices, removes the original searchable snapshot indices and source snapshots, and then removes the tier's nodes.

1. Stop ILM to prevent data from migrating to the phase you intend to remove while you work.

   POST /_ilm/stop
   GET /_ilm/status
   		

   Wait until operation_mode is STOPPED before proceeding.

2. Using the searchable snapshot indices identified in Before you remove a data tier, create an inventory of the indices to restore. For standard ILM configurations, you can use restored-* for the cold tier and partial-* for the frozen tier. For custom configurations, use individual index names or a pattern that matches the relevant indices. For each index, record the index name, source snapshot name, and snapshot repository.

   GET /<searchable-snapshot-index-name-or-pattern>/_settings?filter_path=**.index.store.snapshot.snapshot_name,**.index.store.snapshot.repository_name&expand_wildcards=all
   		

3. For each index in the inventory, remove any aliases that were applied to the searchable snapshot index.

   POST /_aliases
   {
     "actions": [
       {
         "remove": {
           "index": "<searchable-snapshot-index-name>",
           "alias": "<alias-name>"
         }
       }
     ]
   }
   		

   Note

   If you use a data stream, you can skip this step.

4. Restore each index in the inventory from its source snapshot.

   The restore request creates a regular index on the remaining tiers and prevents it from inheriting the previous ILM policy and rollover alias:

   POST /_snapshot/<snapshot_repository_name>/<searchable_snapshot_name>/_restore
   {
     "indices": "*",
     "index_settings": {
       "index.routing.allocation.include._tier_preference": "<data_tiers>",
       "index.number_of_replicas": 1,
       "index.lifecycle.name": null,
       "index.lifecycle.rollover_alias": null
     }
   }
   		

   1. Use the corresponding snapshot repository and snapshot name from the inventory.
   2. The * value restores every index in the snapshot. Snapshots created by the ILM searchable_snapshot action contain only the managed index. For a manually created snapshot that contains multiple indices, replace * with the name of the original index you want to restore.
   3. Specify an ordered list of remaining tiers where the restored index can be allocated. Refer to Update _tier_preference-based rules.
   4. Adjust index.number_of_replicas to match your resiliency needs.

   To manage the restored index with a different ILM policy, apply the policy after the restore and configure its rollover alias if required. Refer to Switch lifecycle policies.

5. Once all snapshots are restored, use GET /_cat/indices/<index-pattern>?v=true to check that the restored indices are green and are correctly reflecting the expected docs.count and store.size values.

   If you are using a data stream, you might need to use GET /_data_stream/<data-stream-name> to get the list of the backing indices, and then specify them by using GET /_cat/indices/<backing-index-name>?v=true to check. When you restore the backing indices of a data stream, some considerations apply, and you might need to manually add the restored indices into your data stream or re-create your data stream.

6. After verifying each restored index, delete the corresponding original searchable snapshot index from the inventory.

   DELETE /<searchable-snapshot-index-name>
   		

7. Delete the source snapshots recorded in the inventory:

   Warning

   A snapshot created by the ILM searchable_snapshot action contains only the managed index, so deleting it after successfully restoring and verifying that index does not affect other indices. This guarantee does not apply to manually created snapshots, which can contain multiple indices or support other mounted searchable snapshot indices. Before deleting a manually created snapshot, verify that it contains no other data you need and supports no other mounted indices.

   DELETE /_snapshot/<snapshot_repository_name>/<searchable_snapshot_name>
   		

   Tip

   You can also delete multiple snapshots at once in Kibana. Find Snapshot and Restore in the navigation menu or use the global search field, select the Snapshots tab, select the snapshots you want to delete, and click Delete.

8. Confirm that no shards remain on the data nodes you wish to remove using GET /_cat/allocation?v=true&s=node.

9. Remove the nodes.

   After confirming that no shards remain on the nodes, remove them using the instructions for your deployment type.

   Stop the Elasticsearch service on each node to be removed and decommission the host. For step-by-step instructions, refer to Add or remove Elasticsearch nodes.

   If an ElasticsearchAutoscaler policy manages the nodeSet, remove the matching policy before removing the nodeSet or setting its count to 0. Otherwise, autoscaling might change the nodeSet count while you complete this procedure. Refer to Autoscaling in ECK.

   Remove the nodeSet from your Elasticsearch manifest, or set its count to 0. Elastic Cloud on Kubernetes safely drains and stops the pods.

10. Confirm that GET /_cluster/health reports green and that GET /_cat/nodes?v shows no nodes from the removed tier remaining in the cluster.

11. Review your ILM policies and update any phases that target the removed tier. For example, when removing the frozen tier, remove the frozen phase. If you want future indices to continue using searchable snapshots, configure the searchable_snapshot action in an appropriate remaining phase. Also update any allocation rules that reference the removed tier.

    For guidance on updating policies, refer to Configure a lifecycle policy.

12. Re-enable ILM:

    POST /_ilm/start
    		

13. Verify that ILM is running and that no indices report errors related to the removed tier:

    GET /_ilm/status
    GET /_all/_ilm/explain?human=true&expand_wildcards=all&only_errors=true
    		

    Confirm that operation_mode is RUNNING. Investigate any reported errors and verify that no policy still attempts to allocate data to the removed tier.

• Configure data tiers
• Data tier index allocation
• Add or remove Elasticsearch nodes