Data Node Disconnected
A data node is no longer connected to the cluster. Remaining nodes must handle its indexing, search, and shard work, which can slow operations and reduce availability.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | High |
| Scope | Cluster |
| Domains | node-lifecycle, stability |
You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.
The default customization settings are:
| Setting | Type | Default |
|---|---|---|
| Node disconnect timeout in seconds | Integer | 300 |
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
The disconnected node is: es-data-01 and es-data-02 Before the disconnection of the nodes, the system detected the following indications:
-
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Investigate disconnected node activity
Condition: Always shown for this insight.
Review tasks, slow search logs, and indexing logs from before the disconnect to find costly requests or errors that might have caused the node to drop.
Impact: Disconnected nodes can impact your general availability and slow down search and indexing operations due to an increased load on existing data nodes. Nodes can disconnect from a cluster for a number of possible reasons. It’s important to consider that unplanned node disconnection is often a symptom of some underlying problem which must be investigated and solved. When a node disconnects it causes cluster instability because of the strain on the other nodes that need to take on all the activity that was previously shared. Additionally, the cluster needs to dedicate resources to regenerating the replica shards that have been lost from the cluster. In extreme cases this can cause a domino effect bringing down other nodes.