﻿---
title: Watermark errors
description: When a data node is critically low on disk space and has reached the flood-stage disk usage watermark, the following error is logged: Error: disk usage...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/3238/troubleshoot/elasticsearch/fix-watermark-errors
products:
  - Elasticsearch
---

# Watermark errors
When a data node is critically low on disk space and has reached the [flood-stage disk usage watermark](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings#cluster-routing-flood-stage), the following error is logged: `Error: disk usage exceeded flood-stage watermark, index has read-only-allow-delete block`.
To prevent a full disk, when a node reaches this watermark, Elasticsearch [blocks writes](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/index-settings/index-block) to any index with a shard on the node. If the block affects related system indices, Kibana and other Elastic Stack features may become unavailable. For example, this could induce Kibana's `Kibana Server is not Ready yet` [error message](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/3238/troubleshoot/kibana/error-server-not-ready).
Elasticsearch will automatically remove the write block when the affected node’s disk usage falls below the [high disk watermark](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings#cluster-routing-watermark-high). To achieve this, Elasticsearch attempts to rebalance some of the affected node’s shards to other nodes in the same data tier.
<admonition title="Simplify monitoring with AutoOps" applies-to="{ ess:, ece:, eck:, self:, "serverless": "ga" }">
  AutoOps is a monitoring tool that simplifies cluster management through performance recommendations, resource utilization visibility, and real-time issue detection with resolution paths. Learn more about [AutoOps](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/3238/deploy-manage/monitor/autoops).
</admonition>


## Monitor rebalancing

To verify that shards are moving off the affected node until it falls below high watermark., use the [cat shards API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-shards) and [cat recovery API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-recovery):
```json
```

If shards remain on the node keeping it about high watermark, use the [cluster allocation explanation API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-allocation-explain) to get an explanation for their allocation status.
```json

{
  "index": "my-index",
  "shard": 0,
  "primary": false
}
```


## Temporary Relief

To immediately restore write operations, you can temporarily increase [disk watermarks](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings#disk-based-shard-allocation) and remove the [write block](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/index-settings/index-block).
```json

{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.low.max_headroom": "100GB",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.high.max_headroom": "20GB",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%",
    "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom": "5GB",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen": "97%",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom": "5GB"
  }
}


{
  "index.blocks.read_only_allow_delete": null
}
```

When a long-term solution is in place, to reset or reconfigure the disk watermarks:
```json

{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": null,
    "cluster.routing.allocation.disk.watermark.low.max_headroom": null,
    "cluster.routing.allocation.disk.watermark.high": null,
    "cluster.routing.allocation.disk.watermark.high.max_headroom": null,
    "cluster.routing.allocation.disk.watermark.flood_stage": null,
    "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom": null,
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen": null,
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom": null
  }
}
```


## Resolve

To resolve watermark errors permanently, perform one of the following actions:
- Horizontally scale nodes of the affected [data tiers](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/3238/manage-data/lifecycle/data-tiers).
- Vertically scale existing nodes to increase disk space.
- Delete indices using the [delete index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete), either permanently if the index isn’t needed, or temporarily to later [restore](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/3238/deploy-manage/tools/snapshot-and-restore/restore-snapshot).
- update related [ILM policy](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/3238/manage-data/lifecycle/index-lifecycle-management) to push indices through to later [data tiers](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/3238/manage-data/lifecycle/data-tiers)

<tip>
  On Elastic Cloud Hosted and Elastic Cloud Enterprise, indices may need to be temporarily deleted via its [Elasticsearch API Console](https://docs-v3-preview.elastic.dev/elastic/cloud/tree/master/reference/cloud-hosted/ec-api-console) to later [snapshot restore](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/3238/deploy-manage/tools/snapshot-and-restore/restore-snapshot) in order to resolve [cluster health](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-health) `status:red` which will block [attempted changes](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/3238/deploy-manage/deploy/elastic-cloud/keep-track-of-deployment-activity). If you experience issues with this resolution flow on Elastic Cloud Hosted, kindly reach out to [Elastic Support](https://support.elastic.co) for assistance.
</tip>