Shard Too Large
One or more shards exceed recommended size limits (roughly 50 GB or very high document counts). Oversized shards slow recovery, relocation, and search, and in extreme cases can become hard to move or write.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | Medium |
| Scope | Index |
| Domains | performance, index-management, shard-allocation |
You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.
The default customization settings are:
| Setting | Type | Default |
|---|---|---|
| Maximum optimal shard size in GB | Integer | 0.0004 |
| Exclude indices based on this pattern | List of strings |
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
Total number of indices with large shards: 12 The index that contains the largest shard is: logs-prod-000045, and the shard size is: 42,000,000. See a sample list: logs-prod-000045, logs-prod-000046.
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
ILM policy for large shards
Condition: Always shown for this insight.
If logs-prod-000045 is time-based, set or tune its ILM policy so rollover targets roughly 20–30 GB shards.
Reindex into smaller indices
Condition: Shown when for non-system indexes.
Reindex into smaller indices to reduce shard sizes and improve search performance.
Split index into more shards
Condition: Shown when primary shard count is less than data node count and shard size warrants split.
Use the Split API to split logs-prod-000045 (current size logs-prod-000045) into 2 primary shards with the action below. Run during low cluster load and confirm you have enough disk space. See the Split index API for prerequisites.
POST logs-prod-000045/_split/logs-prod-000045-0001
{
"settings": {
"index.number_of_shards": 2
}
}
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Delete old data
Condition: Shown when !!!.
Delete unused or old indices to free disk space. Snapshot indices first if you might need them later. Use the action below to remove logs-archive-000001.
DELETE logs-archive-000001
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Impact: Large shards can cause slow searches and decrease cluster performance. A shard that greatly exceeds the recommended limit of 50GB or that contains more than 2B documents might become corrupted or unwritable. Index data is partitioned into shards, and the number of shards in an index depends on the index settings used when the index was originally created. It is recommended not to surpass 50GB for a single shard and to try not to store more than 200M documents per shard. Beyond this size your shard might become unwritable, or cluster might be cannot relocate or recover index shards.