Rejected Indexing
The cluster is rejecting indexing requests on one or more nodes. Rejections can come from thread pool backpressure, mapping errors, disk flood-stage read-only mode, or tripped memory circuit breakers.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | High |
| Scope | Node |
| Domains | performance, indexing, stability |
You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.
The default customization settings are:
| Setting | Type | Default |
|---|---|---|
| Rejected indexing count threshold | Integer | 1 |
| Successive samplings to trigger | Integer | 1 |
Raising these thresholds reduces noise but delays detection. Lowering them triggers the insight sooner but can increase alerts during minor blips.
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
The affected node/s are: es-data-01 and es-data-02 High indexing activity indices:logs-prod-000045.
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Rollover indices
Condition: Shown when high-indexing activity is detected and if index has less primary shards than available data nodes.
If logs-prod-000045 are time-based, roll them over and set 2 primary shards on the new write index so you can avoid a heavy segment merge on the current indices.
Rollover indices by shard size
Condition: Shown when high-indexing activity is detected and if primary shard size > 30GB.
If logs-prod-000045 are time-based, roll them over when average shard size reaches your target.
Review index templates and mappings
Condition: Shown when high-indexing activity is detected and for indexes with highest indexing latency.
Review templates and mappings for logs-prod-000045; they strongly affect indexing performance. Fix oversized mappings, unnecessary fields, and inefficient index settings.
Review indexing slow logs
Condition: Shown when indexing rejections occur on the node.
Enable indexing slow logs with the action below on logs-prod-000045, then review the logs to find slow index operations. On Elasticsearch 8.14+, set index.indexing.slowlog.include.user to true to record which user triggered a slow operation.
PUT logs-prod-000045/_settings
{
"index.indexing.slowlog.threshold.index.warn": "10s",
"index.indexing.slowlog.threshold.index.info": "5s",
"index.indexing.slowlog.threshold.index.debug": "2s",
"index.indexing.slowlog.threshold.index.trace": "500ms",
"index.indexing.slowlog.source": "1000"
}
Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Add data node
Condition: Shown when high-indexing activity is detected and if index has more primary shards as available data nodes.
Add a data node to increase capacity and reduce pressure on the existing nodes.
Review indexing slow logs
Condition: Always shown for this insight.
Check indexing slow logs on es-data-01 to find expensive indexing operations and address the root cause.
Impact: Data loss might occur because the cluster is cannot handle all indexing operations. Indexing requests can be rejected for multiple reasons. Client applications might be sending too much data at once or at too fast a pace for the cluster to handle in terms of memory and CPU resources. It is also possible that indexing requests are being rejected because the documents do not follow the index mapping. Another reason could be that the disks are flooded and the cluster has been set to read-only. Index requests could also be rejected because of circuit breakers being tripped, due to a lack of available memory to process requests. It is recommended to check monitoring data and logs.