Rejected requests
When Elasticsearch rejects a request, it stops the operation and returns an error with a 429
response code. Rejected requests are commonly caused by:
- A depleted thread pool. A depleted
search
orwrite
thread pool returns aTOO_MANY_REQUESTS
error message. - A circuit breaker error.
- High indexing pressure that exceeds the
indexing_pressure.memory.limit
.
Admonition
If you’re using Elastic Cloud Hosted, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. For more information, refer to Monitor with AutoOps.
Check rejected tasks ¶
To check the number of rejected tasks for each thread pool, use the cat thread pool API. A high ratio of rejected
to completed
tasks, particularly in the search
and write
thread pools, means Elasticsearch regularly rejects requests.
GET /_cat/thread_pool?v=true&h=id,name,queue,active,rejected,completed
write
thread pool rejections frequently appear in the erring API and correlating log as EsRejectedExecutionException
with either QueueResizingEsThreadPoolExecutor
or queue capacity
.
These errors are often related to backlogged tasks.
See this video for a walkthrough of troubleshooting threadpool rejections.
Check circuit breakers ¶
To check the number of tripped circuit breakers, use the node stats API.
GET /_nodes/stats/breaker
These statistics are cumulative from node startup. For more information, see circuit breaker errors.
See this video for a walkthrough of diagnosing circuit breaker errors.
Check indexing pressure ¶
To check the number of indexing pressure rejections, use the node stats API.
GET _nodes/stats?human&filter_path=nodes.*.indexing_pressure
These stats are cumulative from node startup.
Indexing pressure rejections appear as an EsRejectedExecutionException
, and indicate that they were rejected due to combined_coordinating_and_primary
, coordinating
, primary
, or replica
.
These errors are often related to backlogged tasks, bulk index sizing, or the ingest target’s refresh_interval
setting.
See this video for a walkthrough of diagnosing indexing pressure rejections.
Prevent rejected requests ¶
Fix high CPU and memory usage
If Elasticsearch regularly rejects requests and other tasks, your cluster likely has high CPU usage or high JVM memory pressure. For tips, see High CPU usage and High JVM memory pressure.