Rejected Search
The cluster is rejecting search requests on one or more nodes. Causes include overload, incompatible queries after upgrades, and memory circuit breakers tripping under heavy queries.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | High |
| Scope | Node |
| Domains | performance, search, stability |
You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.
The default customization settings are:
| Setting | Type | Default |
|---|---|---|
| Rejected search count threshold | Integer | 1 |
| Successive samplings to trigger | Integer | 1 |
Raising these thresholds reduces noise but delays detection. Lowering them triggers the insight sooner but can increase alerts during minor blips.
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
The affected node/s are: es-data-01 and es-data-02 High search activity indices:logs-prod-000045.
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Increase replica count
Condition: Shown when high-searching activity is detected and if index has no replica.
Set number_of_replicas to 2 on logs-prod-000045 (currently 1) using the action below.
PUT logs-prod-000045/_settings
{
"index": {
"number_of_replicas": 2
}
}
Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Enable and review search slow logs
Condition: Shown when search rejections occur on the node.
Enable search slow logs with the action below, then review the slow log to find expensive queries. See Slow logs for configuration details.
PUT logs-prod-000045/_settings
{
"index.search.slowlog.threshold.query.warn": "10s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "2s",
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.threshold.fetch.warn": "1s",
"index.search.slowlog.threshold.fetch.info": "800ms",
"index.search.slowlog.threshold.fetch.debug": "500ms",
"index.search.slowlog.threshold.fetch.trace": "200ms"
}
Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Add data node
Condition: Shown when high-searching activity is detected and if index has more than twice as many pri/rep shards as available data nodes.
Add a data node to increase capacity and reduce pressure on the existing nodes.
Review high search queue load
Condition: Always shown for this insight.
Investigate what is filling the search thread pool queue on es-data-01: expensive queries, under-sized search capacity, or hotspot nodes. Enable and review search slow logs to identify costly queries, then tune queries or add search capacity as needed.
Impact: Search operations are throwing errors or timing out. This could lead to limited/partial data retrieval. Search requests can be rejected by the cluster for multiple reasons. Client applications might be sending too much data at once or at too fast a pace for the cluster to handle in terms of memory and CPU resources. It is also possible that search requests are rejected because the queries are not compatible with the current ES DSL (for example, after an upgrade). Search requests could also be rejected because of circuit breakers being tripped due to lack of available memory to process requests. It is recommended to check monitoring data, logs and possibly to consider optimization of queries.