Status Yellow
Cluster health is yellow because one or more replica shards are not allocated, while primaries are generally serving reads and writes. You still have service, but losing another node could leave indices without redundancy.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | Medium |
| Scope | Cluster |
| Domains | cluster-health, high-availability |
You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.
The default customization settings are:
| Setting | Type | Default |
|---|---|---|
| Minimum number of initializing shards | Integer | 1 |
| Minimum duration of the yellow status in seconds | Integer | 60 |
| Reasons excluded to raise a yellow status | List of strings | REPLICA_ADDED,DANGLING_INDEX_IMPORTED,CLUSTER_RECOVERED,INDEX_CLOSED,INDEX_REOPENED,EXISTING_INDEX_RESTORED,NEW_INDEX_RESTORED,REALLOCATED_REPLICA,MANUAL_ALLOCATION |
| Minimum number of data nodes required to raise the yellow status event | Integer | 2 |
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
The current number of unassigned replica shards is: 12.</br></br>The maximum number unassigned replica shards is: 12.</br></br>The number of initializing shards is: 12.</br></br>The shards are unassigned because: </br></br></br>See sample list: logs-prod-000045, logs-prod-000046.
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Wait for shard recovery to complete
Condition: Shown when only initializing shards.
Shard recovery is in progress after data nodes restarted or were removed: es-data-01. Wait until recovery finishes before making other cluster changes.
Reduce shard replicas
Condition: Shown when unassigned shards and number of replicas > 1.
Reduce replica count on logs-prod-000045 from 1 to 2 to free disk space. Start with indices that have more than two shard copies.
PUT logs-prod-000045/_settings
{
"index": {
"number_of_replicas": 2
}
}
Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Add data node
Condition: Shown when total replicas / total primaries > number of data nodes.
Add a data node to increase capacity and reduce pressure on the existing nodes.
Retry failed shard allocations
Condition: Shown when any shard failed to be allocated.
If shards failed to allocate and nothing is recovering, retry allocation with a cluster reroute. Use the action below (retry_failed=true).
POST _cluster/reroute?retry_failed=true
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Explore shard allocation in the cluster
Condition: Shown when no initializing shards.
Use cluster allocation explain to see why shards are unassigned for index logs-prod-000045, shard 0. Run the action below.
GET _cluster/allocation/explain
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action is read-only.
Check total shards per node limit
Condition: Shown when no initializing shards.
index.routing.allocation.total_shards_per_node on logs-prod-000045 might block replica assignment. Fetch index settings with the action below and adjust or remove the limit if it causes unassigned shards.
GET logs-prod-000045/_settings
Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action is read-only.
Impact: When cluster status is yellow, it means there is a higher risk of permanent or temporary loss of data. Yellow status indicates that one or more of the replica shards on the cluster are not allocated to a node. While the cluster status is yellow, all data is still accessible for read and write operations, but in the event of data corruption or loss of a node, one or more of your indices will no longer be available for write or search. If you only have one node and your index has replica shards, then a yellow status is normal, since no other node is available to receive the replica. A cluster might also become yellow temporarily following a node restart, until the node resynchronizes the replica shards. However, if the yellow status persists, you should act to find out why and recover your cluster to Green status. If the shard recovery process has been completed and the cluster status is still yellow, it's an indication that the allocation was not carried out successfully. There could be various reasons for this failure: - There is only 1 node in your cluster, and replicas will never be allocated on the same node as the primary shard. - One or more nodes have disk utilization that is above the cluster.routing.allocation.disk.watermark.low value. - The use of attribute-based allocation rules is preventing a shard from being allocated on any of the available nodes.