Disk watermark high
Disk usage on one or more nodes has crossed the high watermark. Elasticsearch begins relocating shards away from those nodes.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | High |
| Scope | Node |
| Domains | cluster-health, disk-storage, stability |
You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.
The default customization settings are:
| Setting | Type | Default |
|---|---|---|
| Early warning margin before the high watermark | Percentage | 0 |
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
The high disk watermark has been exceeded by 12 GB.
Affected nodes: es-data-01 and es-data-02.
Affected tier(s): hot, warm.
Remaining space before the flood-stage watermark: 48 GB.
Largest indices on the affected node(s): logs-prod-000045.
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Increase disk size
Condition: Shown when always included when disk watermark is crossed on a data node; on hot nodes this is ingest risk, on warm/cold nodes this is retention risk.
Increase disk size on the affected node (current capacity: 500 GB).
Temporary disk watermark increase
Condition: Shown when available disk space > 100GB.
Temporarily raise the disk watermark so you can free space before the cluster restricts shard allocation. After you reclaim disk, return the watermark to its previous value. Use the action below to update cluster.routing.allocation.disk.watermark.high.
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.disk.watermark.high": "90%"
}
}
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Move shard to less loaded node
Condition: Shown when other data nodes (preferably in the same tier) have spare space.
Move shard 0 of logs-prod-000045 from es-data-01 to es-data-02 with the action below to rebalance disk use.
POST _cluster/reroute
{
"commands": [
{
"move": {
"index": "logs-prod-000045",
"shard": 0,
"from_node": "es-data-01",
"to_node": "es-data-02"
}
}
]
}
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Reduce shard replicas
Condition: Shown when number of replicas > 1.
Reduce replica count on logs-prod-000045 from 1 to 2 to free disk space. Start with indices that have more than two shard copies.
PUT logs-prod-000045/_settings
{
"index": {
"number_of_replicas": 2
}
}
Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Delete old data
Condition: Shown when "old" index exists; warm/cold tier: consider retention policies and ILM.
Delete unused or old indices to free disk space. Snapshot indices first if you might need them later. Use the action below to remove logs-archive-000001.
DELETE logs-archive-000001
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Force merge deleted documents
Condition: Shown when old read-only indices exist on affected data node (particularly warm/cold tier).
Run a force merge with only_expunge_deletes=true on logs-prod-000045, logs-prod-000046 to reclaim disk used by deleted documents. Use the action below.
POST logs-archive-000001/_forcemerge?only_expunge_deletes=true
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
When the high watermark is crossed, Elasticsearch actively relocates shards away from the affected nodes to relieve disk pressure. Writes to existing shards on those nodes continue until the flood-stage watermark is reached, at which point indices are made read-only. Act promptly. On hot nodes, high disk usage directly threatens ingest throughput. On warm or cold nodes, ILM transitions or data deletion are typically the right levers. On any tier, if relocation is not enough, expand disk capacity or apply index retention policies before the flood stage is reached.