--- title: Warning: Not enough nodes to allocate all shard replicas description: Distributing copies of the data (index shard replicas) on different nodes can parallelize processing requests, speeding up search queries. To achieve... url: https://www.elastic.co/elastic/docs-builder/docs/3028/troubleshoot/elasticsearch/increase-tier-capacity products: - Elasticsearch applies_to: - Elastic Stack: Generally available --- # Warning: Not enough nodes to allocate all shard replicas Distributing copies of the data (index shard replicas) on different nodes can parallelize processing requests, speeding up search queries. To achieve this, increase the number of replica shards up to the maximum value (total number of nodes minus one), which also protects against hardware failure. If the index has a preferred tier, Elasticsearch will only place the copies of the data for that index on nodes in the target tier. If a warning is encountered with not enough nodes to allocate all shard replicas, you can influence this behavior by adding more nodes to the cluster (or tier), or by reducing the [`index.number_of_replicas`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/index-settings/index-modules#dynamic-index-number-of-replicas) index setting. To accomplish this, complete the following steps: 1. [Determine which data tier needs more capacity](#determine-data-tier) to identify the tier where shards need to be allocated. 2. [Resize your deployment](#resize-deployment) to add capacity and accommodate all shard replicas. 3. [Check and adjust the index replicas limit](#adjust-index-replica-limit) to determine the current value and reduce it if needed. ## Determine which data tier needs more capacity You can run the following step using either [API console](https://www.elastic.co/elastic/docs-builder/docs/3028/explore-analyze/query-filter/tools/console) or direct [Elasticsearch API](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/rest-apis) calls. Use the [get index settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) API to retrieve the configured value for the `index.routing.allocation.include._tier_preference` setting: ```json ``` The response looks like this: ```json { "my-index-000001": { "settings": { "index.routing.allocation.include._tier_preference": "data_warm,data_hot" } } } ``` 1. Represents a comma-separated list of data tier node roles this index is allowed to be allocated on. The first tier in the list has the highest priority and is the tier the index is targeting. In this example, the tier preference is `data_warm,data_hot`, so the index is targeting the `warm` tier. If the warm tier lacks capacity, the index will fall back to the `data_hot` tier. ## Resize your deployment After you've identified the tier that needs more capacity, you can resize your deployment to distribute the shard load and allow previously unassigned shards to be allocated. In ECE, resizing is limited by your [allocator capacity](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/deploy/cloud-enterprise/ece-manage-capacity). To resize your deployment and increase its capacity by expanding a data tier or adding a new one, use the following options:**Option 1: Configure Autoscaling** 1. Log in to the [Elastic Cloud console](https://cloud.elastic.co?page=docs&placement=docs-body) or ECE Cloud UI. 2. On the home page, find your deployment and select **Manage**. 3. Go to **Actions** > **Edit deployment** and check that autoscaling is enabled. Adjust the **Enable Autoscaling for** dropdown menu as needed and select **Save**. 4. If autoscaling is successful, the cluster returns to a `healthy` status. If the cluster is still out of disk, check if autoscaling has reached its set limits and [update your autoscaling settings](/elastic/docs-builder/docs/3028/deploy-manage/autoscaling/autoscaling-in-ece-and-ech#ec-autoscaling-update). **Option 2: Configure deployment size and tiers**You can increase the deployment capacity by editing the deployment and adjusting the size of the existing data tiers or adding new ones. 1. In Kibana, open your deployment’s navigation menu (placed under the Elastic logo in the upper left corner) and go to **Manage this deployment**. 2. From the right hand side, click to expand the **Manage** dropdown button and select **Edit deployment** from the list of options. 3. On the **Edit** page, increase capacity for the data tier you identified earlier by either adding a new tier with **+ Add capacity** or adjusting the size of an existing one. Choose the desired size and availability zones for that tier. 4. Navigate to the bottom of the page and click the **Save** button. **Option 3: Change the hardware profiles/deployment templates**You can change the [hardware profile](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/deploy/elastic-cloud/ec-change-hardware-profile) for Elastic Cloud Hosted deployments or [deployment template](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/deploy/cloud-enterprise/deployment-templates) of the Elastic Cloud Enterprise cluster to one with a higher disk-to-memory ratio.**Option 4: Elastic Cloud Enterprise: Generally available Override disk quota**Elastic Cloud Enterprise administrators can temporarily override the disk quota of Elasticsearch nodes in real time as explained in [Resource overrides](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/deploy/cloud-enterprise/resource-overrides). We strongly recommend making this change only under the guidance of Elastic Support, and only as a temporary measure or for troubleshooting purposes. To increase the data node capacity in your cluster, you can [add more nodes](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/maintenance/add-and-remove-elasticsearch-nodes) to the cluster and assign the index’s target tier [node role](/elastic/docs-builder/docs/3028/manage-data/lifecycle/data-tiers#configure-data-tiers-on-premise) to the new nodes, or increase the disk capacity of existing nodes. Disk expansion procedures depend on your operating system and storage infrastructure and are outside the scope of Elastic support. In practice, this is often achieved by [removing a node from the cluster](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/maintenance/add-and-remove-elasticsearch-nodes) and reinstalling it with a larger disk. To increase the capacity of the data nodes in your Elastic Cloud on Kubernetes cluster, you can either add more data nodes to the desired tier, or increase the storage size of existing nodes.**Option 1: Add more data nodes** 1. Update the `count` field in your data node `nodeSets` to add more nodes: ```yaml apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: quickstart spec: version: 9.3.2 nodeSets: - name: data-nodes count: 5 config: node.roles: ["data"] volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi ``` 2. Apply the changes: ```sh kubectl apply -f your-elasticsearch-manifest.yaml ``` ECK automatically creates the new nodes with a `data` [node role](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles) and Elasticsearch will relocate shards to balance the load. You can monitor the progress using: ```json ``` **Option 2: Increase storage size of existing nodes** 1. If your storage class supports [volume expansion](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims), you can increase the storage size in the `volumeClaimTemplates`: ```yaml apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: quickstart spec: version: 9.3.2 nodeSets: - name: data-nodes count: 3 config: node.roles: ["data"] volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 200Gi ``` 2. Apply the changes. If the volume driver supports `ExpandInUsePersistentVolumes`, the filesystem will be resized online without restarting Elasticsearch. Otherwise, you might need to manually delete the Pods after the resize so they can be recreated with the expanded filesystem. For more information, refer to [Update your deployments](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/deploy/cloud-on-k8s/update-deployments) and [Volume claim templates > Updating the volume claim settings](/elastic/docs-builder/docs/3028/deploy-manage/deploy/cloud-on-k8s/volume-claim-templates#k8s-volume-claim-templates-update). AutoOps is a [monitoring](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/monitor) tool that simplifies cluster management through performance recommendations, resource utilization visibility, and real-time issue detection with resolution paths. Learn more about [AutoOps](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/monitor/autoops). ## Check and adjust the index replicas limit If it is not possible to increase capacity by resizing your deployment, you can reduce the number of replicas of your index data. You achieve this by inspecting the [`index.number_of_replicas`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/index-settings/index-modules#dynamic-index-number-of-replicas) index setting index setting and decreasing the configured value. 1. Use the [get index settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) API to retrieve the configured value for the `index.number_of_replicas` index setting. ```json ``` The response looks like this: ```json { "my-index-000001" : { "settings" : { "index" : { "number_of_replicas" : "2" } } } } ``` 2. Use the [`_cat/nodes`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-nodes) API to find the number of nodes in the target tier: ```json ``` The response looks like this, containing one row per node: ```json himrst mv himrst ``` You can count the rows containing the letter representing the target tier to know how many nodes you have. See [Query parameters](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-nodes) for details. The example above has two rows containing `h`, so there are two nodes in the hot tier. 3. Use the [update index settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) API to decrease the value for the total number of replica shards required for this index. As replica shards cannot reside on the same node as primary shards for [high availability](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/production-guidance/availability-and-resilience), the new value needs to be less than or equal to the number of nodes found above minus one. Since the example above found 2 nodes in the hot tier, the maximum value for `index.number_of_replicas` is 1. ```json { "index" : { "number_of_replicas" : 1 <1> } } ``` Reduce the `index.number_of_replicas` index setting. AutoOps is a [monitoring](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/monitor) tool that simplifies cluster management through performance recommendations, resource utilization visibility, and real-time issue detection with resolution paths. Learn more about [AutoOps](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/monitor/autoops).