﻿---
title: Task queue backlog
description: A backlogged task queue can prevent tasks from completing and lead to an unhealthy cluster state. Contributing factors include resource constraints, a...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4384/troubleshoot/elasticsearch/task-queue-backlog
products:
  - Elasticsearch
---

# Task queue backlog
A backlogged task queue can prevent tasks from completing and lead to an unhealthy cluster state. Contributing factors include resource constraints, a large number of tasks triggered at once, and long-running tasks.

## Diagnose a backlogged task queue

To identify the cause of the backlog, try these diagnostic actions.
- [Check the thread pool status](#diagnose-task-queue-thread-pool)
- [Inspect hot threads on each node](#diagnose-task-queue-hot-thread)
- [Identify long-running node tasks](#diagnose-task-queue-long-running-node-tasks)
- [Look for long-running cluster tasks](#diagnose-task-queue-long-running-cluster-tasks)


### Check the thread pool status

A [depleted thread pool](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4384/troubleshoot/elasticsearch/high-cpu-usage) can result in [rejected requests](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4384/troubleshoot/elasticsearch/rejected-requests).
Use the [cat thread pool API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-thread-pool) to monitor active threads, queued tasks, rejections, and completed tasks:
```json
```

- Look for high `active` and `queue` metrics, which indicate potential bottlenecks and opportunities to [reduce CPU usage](/elastic/docs-content/pull/4384/troubleshoot/elasticsearch/high-cpu-usage#reduce-cpu-usage).
- Determine whether thread pool issues are specific to a [data tier](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4384/manage-data/lifecycle/data-tiers).
- Check whether a specific node’s thread pool is depleting faster than others. This might indicate [hot spotting](#resolve-task-queue-backlog-hotspotting).


### Inspect hot threads on each node

If a particular thread pool queue is backed up, periodically poll the [nodes hot threads API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-hot-threads) to gauge the thread’s progression and ensure it has sufficient resources:
```json
```

Although the hot threads API response does not list the specific tasks running on a thread, it provides a summary of the thread’s activities. You can correlate a hot threads response with a [task management API response](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-tasks) to identify any overlap with specific tasks. For example, if the hot threads response indicates the thread is `performing a search query`, you can [check for long-running search tasks](#diagnose-task-queue-long-running-node-tasks) using the task management API.

### Identify long-running node tasks

Long-running tasks can also cause a backlog. Use the [task management API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-tasks) to check for excessive `running_time_in_nanos` values:
```json
```

You can filter on a specific `action`, such as [bulk indexing](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) or search-related tasks. These tend to be long-running.
- Filter on [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) actions:
  ```json
  ```
- Filter on search actions:
  ```json
  ```

Long-running tasks might need to be [canceled](#resolve-task-queue-backlog-stuck-tasks).
See this [this video](https://www.youtube.com/watch?v=lzw6Wla92NY) for a walkthrough of troubleshooting the [task management API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-tasks) output.

### Look for long-running cluster tasks

Use the [cluster pending tasks API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-pending-tasks) to identify delays in cluster state synchronization:
```json
```

Tasks with a high `timeInQueue` value are likely contributing to the backlog and might need to be [canceled](#resolve-task-queue-backlog-stuck-tasks).

## Recommendations

After identifying problematic threads and tasks, resolve the issue by increasing resources or canceling tasks.

### Increase available resources

If tasks are progressing slowly, try [reducing CPU usage](/elastic/docs-content/pull/4384/troubleshoot/elasticsearch/high-cpu-usage#reduce-cpu-usage).
In some cases, you might need to increase the thread pool size. For example, the `force_merge` thread pool defaults to a single thread. Increasing the size to 2 might help reduce a backlog of force merge requests.

### Cancel stuck tasks

If an active task’s [hot thread](#diagnose-task-queue-hot-thread) shows no progress, consider [canceling the task](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-tasks#task-cancellation).

### Address hot spotting

If a specific node’s thread pool is depleting faster than others, try addressing uneven node resource utilization, also known as hot spotting. For details on actions you can take, such as rebalancing shards, see [Hot spotting](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4384/troubleshoot/elasticsearch/hotspotting).

## Resources

Related symptoms:
- [High CPU usage](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4384/troubleshoot/elasticsearch/high-cpu-usage)
- [Rejected requests](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4384/troubleshoot/elasticsearch/rejected-requests)
- [Hot spotting](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4384/troubleshoot/elasticsearch/hotspotting)
- [Troubeshooting overview](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4384/troubleshoot)