Shard count high on data node
One or more data nodes host more shards than the recommended limit. Oversharding strains master coordination and increases per-shard memory use, especially on older Elasticsearch versions.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | Medium |
| Scope | Node |
| Domains | shard-allocation, stability |
You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.
The default customization settings are:
| Setting | Type | Default |
|---|---|---|
| Early warning margin before the max shards per node limit | Percentage | 30 |
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
The current number of shards on es-data-01 and es-data-02 is 24 (limit: 1000).
- Index logs-prod-000045 has 12 total primary shards, of which 12 are on this node.
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Reduce shard count
Condition: Shown when node hosts indices with primary shards.
Use the shrink API to reduce shard count from 24 to 12. See the Shrink API for prerequisites.
Merge similar indices
Condition: Always shown for this insight.
Merge similar indices to reduce the number of small indices and improve cluster performance and stability.
Add data node
Condition: Always shown for this insight.
Add a data node to increase capacity and reduce pressure on the existing nodes.
Too many shards on a data node can destabilize the cluster. Master nodes must track every shard in the cluster state, and a high shard count on individual nodes increases coordination overhead and can slow down cluster state updates. High shard counts can also approach limits for virtual memory (mmap) and open file descriptors on the affected nodes. Both might need adjustment if shard counts increase further. In the version (8.15.0) you're running, the memory usage per shard is significantly higher than in more recent versions, where a pivotal change was introduced to optimize memory consumption. Consider upgrading to a more recent version. If that is not an option right now, lower the cluster.max_shards_per_node cluster setting to keep memory consumption in check.