Search rejection
The search thread pool queue is full and Elasticsearch is rejecting new search requests on the listed nodes. Clients see errors instead of slow responses, which breaks dashboards and applications immediately.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | Medium |
| Scope | Node |
| Domains | performance, search, stability |
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
The search thread pool queue on es-data-01 has reached its limit. Elasticsearch is rejecting new search requests on es-data-01.
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Increase replica count
Condition: Shown when index has no replica.
Set number_of_replicas to 2 on logs-prod-000045 (currently 1) using the action below.
PUT logs-prod-000045/_settings
{
"index": {
"number_of_replicas": 2
}
}
Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Review query logs
Condition: Shown when cluster version is 9.4 or above.
Review query logs on es-data-01 to find expensive searches that fill the queue. Query logs (Elasticsearch 9.4+) give structured per-query timing with less overhead than search slow logs. See Query logs.
Enable and review search slow logs
Condition: Shown when cluster version is below 9.4.
Enable search slow logs with the action below, then review the slow log to find expensive queries. See Slow logs for configuration details.
PUT logs-prod-000045/_settings
{
"index.search.slowlog.threshold.query.warn": "10s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "2s",
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.threshold.fetch.warn": "1s",
"index.search.slowlog.threshold.fetch.info": "800ms",
"index.search.slowlog.threshold.fetch.debug": "500ms",
"index.search.slowlog.threshold.fetch.trace": "200ms"
}
Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Add data node
Condition: Shown when index has more than twice as many pri/rep shards as available data nodes.
Add a data node to increase capacity and reduce pressure on the existing nodes.
When the search thread pool queue fills up, Elasticsearch rejects new search requests immediately; callers receive error responses rather than waiting. This is the hard-failure stage that occurs after the search queue starts backing up. User-visible effects include failed dashboard refreshes, API errors in applications querying this node, and alert noise from dependent monitoring systems. Both coordinating nodes and data nodes can be sources of queue saturation, depending on where search traffic is routed. If the load is sustained, check whether search traffic can be spread across more nodes or whether queries can be optimized to reduce thread pool pressure. Adding coordinating or data capacity is the longer-term resolution when search volume genuinely exceeds current cluster headroom.