---
title: Watermark errors
description: When a data node reaches critical disk space usage, its disk-based shard allocation watermark settings trigger to protect the node's disk. The default...
url: https://www.elastic.co/elastic/docs-builder/docs/3016/troubleshoot/elasticsearch/fix-watermark-errors
products:
  - Elasticsearch
applies_to:
  - Elastic Stack: Generally available
---

# Watermark errors
When a data node reaches critical disk space usage, its [disk-based shard allocation watermark settings](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings#disk-based-shard-allocation) trigger to protect the node's disk. The default watermark percentage thresholds, the summary of Elasticsearch's response, and their corresponding Elasticsearch log are:
- 85% [`low`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings#cluster-routing-watermark-low): Elasticsearch stops allocating replica shards and primary shards unless from newly-created indices to the affected node(s).
  ```
  low disk watermark [85%] exceeded on [NODE_ID][NODE_NAME] free: Xgb[X%], replicas will not be assigned to this node
  ```
- 90% [`high`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings#cluster-routing-watermark-high): Elasticsearch rebalances shards away from the affected node(s).
  ```
  high disk watermark [90%] exceeded on [NODE_ID][NODE_NAME] free: Xgb[X%], shards will be relocated away from this node
  ```
- 95% [`flood-stage`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings#cluster-routing-flood-stage): Elasticsearch sets all indices on the affected node(s) to read-only. The write block is automatically removed once disk usage on the affected node falls below the high watermark.
  ```
  flood-stage watermark [95%] exceeded on [NODE_ID][NODE_NAME], all indices on this node will be marked read-only
  ```

<note applies-to="Elastic Cloud Hosted: Generally available, Elastic Cloud Enterprise: Generally available">
  At 75% disk usage, the Elastic Cloud Console displays a red disk indicator for the node to signal elevated usage. This threshold is a visual indicator only and is not tied to any Elasticsearch watermark or disk-enforcement behavior. No Elasticsearch allocation or write restrictions are applied at this stage.
</note>

To prevent a full disk, when a node reaches `flood-stage` watermark, Elasticsearch [blocks writes](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/elasticsearch/index-settings/index-block) to any index with a shard on the affected node(s). If the block affects related system indices, Kibana and other Elastic Stack features can become unavailable. For example, `flood-stage` can induce errors like:
- Kibana's `Kibana Server is not Ready yet` [error message](https://www.elastic.co/elastic/docs-builder/docs/3016/troubleshoot/kibana/error-server-not-ready).
- Elasticsearch's ingest API's [reject the request](https://www.elastic.co/elastic/docs-builder/docs/3016/troubleshoot/elasticsearch/rejected-requests) with HTTP 429 error bodies like:
  ```json
  {
    "reason": "index [INDEX_NAME] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];",
    "type": "cluster_block_exception"
  }
  ```

The following are some common setup issues leading to watermark errors:
- Sudden ingestion of large volumes of data that consumes disk above peak load testing expectations. Refer to [Indexing performance considerations](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/production-guidance/optimize-performance/indexing-speed) for guidance.
- Inefficient index settings, unnecessary stored fields, and suboptimal document structures can increase disk consumption. Refer to [Tune for disk usage](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/production-guidance/optimize-performance/disk-usage) for guidance.
- A high number of replicas can quickly multiply storage requirements, as each replica consumes the same disk space as the primary shard. Refer to [Index settings](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/elasticsearch/index-settings/index-modules) for details.
- Oversized shards can make disk usage spikes more likely and slow down recovery and rebalancing. Refer to [Size your shards](/elastic/docs-builder/docs/3016/deploy-manage/production-guidance/optimize-performance/size-shards#shard-size-recommendation) for guidance.


## Monitor disk usage

<admonition title="Simplify monitoring with AutoOps">
  AutoOps is a [monitoring](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/monitor) tool that simplifies cluster management through performance recommendations, resource utilization visibility, and real-time issue detection with resolution paths. Learn more about [AutoOps](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/monitor/autoops).
</admonition>

To track disk usage over time, enable monitoring using one of the following options, depending on your deployment type:
<applies-switch>
  <applies-item title="{ ess:, ece: }" applies-to="Elastic Cloud Hosted: Generally available, Elastic Cloud Enterprise: Generally available">
    - (Recommended) Enable [AutoOps](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/monitor/autoops).
    - Enable [logs and metrics](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring). When logs and metrics are enabled, monitoring information is visible on Kibana's [Stack Monitoring](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data) page. You can also enable the [Disk usage threshold alert](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts) to be notified about potential issues.
    - From your deployment menu, view the [**Performance**](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud) page's disk usage chart.
  </applies-item>

  <applies-item title="{ self:, eck: }" applies-to="Elastic Cloud on Kubernetes: Generally available, Self-managed Elastic deployments: Generally available">
    - (Recommended) Enable [AutoOps](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/monitor/autoops).
    - Enable [Elasticsearch monitoring](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/monitor/stack-monitoring). When logs and metrics are enabled, monitoring information is visible on Kibana's [Stack Monitoring](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data) page. You can also enable the [Disk usage threshold alert](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts) to be notified about potential issues.
  </applies-item>
</applies-switch>


## Monitor rebalancing

To verify that shards are moving off the affected node until it falls below high watermark, use the following Elasticsearch APIs:
- [Cluster health status API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-health) to check `relocating_shards`.
  ```json
  ```
- [CAT recovery API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-recovery) to check the count of recovering shards and their migrated `bp` bytes percent of `tb` total bytes.
  ```json
  ```

If shards remain on the node keeping it above high watermark, use the following Elasticsearch APIs:
- [CAT shards API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-shards) to determine which shards are hosted on the node.
  ```json
  ```
- [Cluster allocation explanation API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-allocation-explain) to get an explanation for the chosen shard's allocation status.
  ```json

  {
    "index": "my-index-000001",
    "shard": 0,
    "primary": false
  }
  ```
  Refer to [Using the cluster allocation API for troubleshooting](https://www.elastic.co/elastic/docs-builder/docs/3016/troubleshoot/elasticsearch/cluster-allocation-api-examples) for guidance on interpreting this output.

You should normally wait for Elasticsearch to balance itself. If advanced users determine shards which should migrate off node faster, whether due to forecasted ingestion rate or existing disk usage, they might consider using the [Reroute the cluster API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-reroute) to push their chosen shard to immediately rebalance to their determined target node.

## Temporary relief

To immediately restore write operations, you can temporarily increase [disk watermarks](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings#disk-based-shard-allocation) and remove the [write block](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/elasticsearch/index-settings/index-block).
```json

{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.low.max_headroom": "100GB",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.high.max_headroom": "20GB",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%",
    "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom": "5GB",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen": "97%",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom": "5GB"
  }
}


{
  "index.blocks.read_only_allow_delete": null
}
```

When a long-term solution is in place, to reset or reconfigure the disk watermarks:
```json

{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": null,
    "cluster.routing.allocation.disk.watermark.low.max_headroom": null,
    "cluster.routing.allocation.disk.watermark.high": null,
    "cluster.routing.allocation.disk.watermark.high.max_headroom": null,
    "cluster.routing.allocation.disk.watermark.flood_stage": null,
    "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom": null,
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen": null,
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom": null
  }
}
```

<note>
  Elasticsearch recommends using default watermark settings. Advanced users can override [the watermark thresholds and headroom](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings) but risk not giving enough disk for background processes such as force merge, not being right-sized to data ingestion rates vs index lifecycle management settings, and possibly `disk is full` errors if 100% disk is reached.
</note>


## Resolve

To resolve watermark errors permanently, perform one of the following actions:
- Horizontally scale nodes of the affected [data tiers](https://www.elastic.co/elastic/docs-builder/docs/3016/manage-data/lifecycle/data-tiers).
- Vertically scale existing nodes to increase disk space. Ensure nodes within a [data tier](https://www.elastic.co/elastic/docs-builder/docs/3016/manage-data/lifecycle/data-tiers) are scaled to matching hardware profiles to avoid [hot spotting](https://www.elastic.co/elastic/docs-builder/docs/3016/troubleshoot/elasticsearch/hotspotting).
- Delete indices using the [delete index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete), either permanently if the index isn’t needed, or temporarily to later [restore from snapshot](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/tools/snapshot-and-restore/restore-snapshot).

<tip>
  On Elastic Cloud Hosted and Elastic Cloud Enterprise, you might need to temporarily delete indices using the [Elasticsearch API Console](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/cloud-hosted/ec-api-console). This can resolve a `status: red` [cluster health](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-health) status, which blocks [deployment changes](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/deploy/elastic-cloud/keep-track-of-deployment-activity). After resolving the issue, you can [restore](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/tools/snapshot-and-restore/restore-snapshot) the indices from a snapshot. If you experience issues with this resolution flow, reach out to [Elastic Support](/elastic/docs-builder/docs/3016/troubleshoot#troubleshoot-work-with-support) for assistance.
</tip>


## Preventing watermark errors

To reduce the likelihood of watermark errors:
- Enable [Autoscaling](https://www.elastic.co/elastic/docs-builder/docs/3016/deploy-manage/autoscaling) to automatically adjust resources based on storage and performance needs. <applies-to>Elastic Cloud Enterprise: Generally available</applies-to> <applies-to>Elastic Cloud Hosted: Generally available</applies-to> <applies-to>Elastic Cloud on Kubernetes: Generally available</applies-to>
- Implement more restrictive [index lifecycle management policies](https://www.elastic.co/elastic/docs-builder/docs/3016/manage-data/lifecycle/index-lifecycle-management) to move data through [data tiers](https://www.elastic.co/elastic/docs-builder/docs/3016/manage-data/lifecycle/data-tiers) sooner to help keep higher tiers' disk usage under control.
- Avoid a mix of overly large and small indices which can cause an [unbalanced cluster](https://www.elastic.co/elastic/docs-builder/docs/3016/troubleshoot/elasticsearch/troubleshooting-unbalanced-cluster). Refer to [Size your shards](/elastic/docs-builder/docs/3016/deploy-manage/production-guidance/optimize-performance/size-shards#shard-size-recommendation).