Total number of shards per node has been reached

To resize your deployment and increase its capacity by expanding a data tier or adding a new one, use the following options:

Option 1: Configure Autoscaling

1. Log in to the Elastic Cloud console or ECE Cloud UI.
2. On the home page, find your deployment and select Manage.
3. Go to Actions > Edit deployment and check that autoscaling is enabled. Adjust the Enable Autoscaling for dropdown menu as needed and select Save.
4. If autoscaling is successful, the cluster returns to a healthy status. If the cluster is still out of disk, check if autoscaling has reached its set limits and update your autoscaling settings.

Option 2: Configure deployment size and tiers

You can increase the deployment capacity by editing the deployment and adjusting the size of the existing data tiers or adding new ones.

1. In Kibana, open your deployment’s navigation menu (placed under the Elastic logo in the upper left corner) and go to Manage this deployment.
2. From the right hand side, click to expand the Manage dropdown button and select Edit deployment from the list of options.
3. On the Edit page, increase capacity for the data tier you identified earlier by either adding a new tier with + Add capacity or adjusting the size of an existing one. Choose the desired size and availability zones for that tier.
4. Navigate to the bottom of the page and click the Save button.

Option 3: Change the hardware profiles/deployment templates

You can change the hardware profile for Elastic Cloud Hosted deployments or deployment template of the Elastic Cloud Enterprise cluster to one with a higher disk-to-memory ratio.

Option 4: Override disk quota

Elastic Cloud Enterprise administrators can temporarily override the disk quota of Elasticsearch nodes in real time as explained in Resource overrides. We strongly recommend making this change only under the guidance of Elastic Support, and only as a temporary measure or for troubleshooting purposes.

To increase the data node capacity in your cluster, you can add more nodes to the cluster and assign the index’s target tier node role to the new nodes, or increase the disk capacity of existing nodes. Disk expansion procedures depend on your operating system and storage infrastructure and are outside the scope of Elastic support. In practice, this is often achieved by removing a node from the cluster and reinstalling it with a larger disk.

To increase the capacity of the data nodes in your Elastic Cloud on Kubernetes cluster, you can either add more data nodes to the desired tier, or increase the storage size of existing nodes.

Option 1: Add more data nodes

1. Update the count field in your data node nodeSets to add more nodes:

   apiVersion: elasticsearch.k8s.elastic.co/v1
   kind: Elasticsearch
   metadata:
     name: quickstart
   spec:
     version: 9.3.0
     nodeSets:
     - name: data-nodes
       count: 5
       config:
         node.roles: ["data"]
       volumeClaimTemplates:
       - metadata:
           name: elasticsearch-data
         spec:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 100Gi
   		

   1. Increase from previous count

2. Apply the changes:

   kubectl apply -f your-elasticsearch-manifest.yaml
   		

   ECK automatically creates the new nodes with a data node role and Elasticsearch will relocate shards to balance the load.

   You can monitor the progress using:

   				GET /_cat/shards?v&h=state,node&s=state
   		

Option 2: Increase storage size of existing nodes

1. If your storage class supports volume expansion, you can increase the storage size in the volumeClaimTemplates:

   apiVersion: elasticsearch.k8s.elastic.co/v1
   kind: Elasticsearch
   metadata:
     name: quickstart
   spec:
     version: 9.3.0
     nodeSets:
     - name: data-nodes
       count: 3
       config:
         node.roles: ["data"]
       volumeClaimTemplates:
       - metadata:
           name: elasticsearch-data
         spec:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 200Gi
   		

   1. Increased from previous size

2. Apply the changes. If the volume driver supports ExpandInUsePersistentVolumes, the filesystem will be resized online without restarting Elasticsearch. Otherwise, you might need to manually delete the Pods after the resize so they can be recreated with the expanded filesystem.

For more information, refer to Update your deployments and Volume claim templates > Updating the volume claim settings.