Heap Size
JVM heap usage on a node is at or near its configured limit. Near-cap heap leads to slower operations, more garbage collection, and a higher chance of node failure or disconnect.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | Medium |
| Scope | Node |
| Domains | memory, stability |
You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.
The default customization settings are:
| Setting | Type | Default |
|---|---|---|
| High Heap threshold percent | Percentage | 10 |
| Successive samplings to trigger | Integer | 3 |
| Search queue threshold | Integer | 5 |
| Search latency threshold (ms) | Integer | 250 |
| Write queue threshold | Integer | 1 |
| Index latency threshold (ms) | Integer | 80 |
Raising these thresholds reduces noise but delays detection. Lowering them triggers the insight sooner but can increase alerts during minor blips.
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
The memory usage is too high on node: es-data-01 and es-data-02 The percent of memory used is: 30.
Current max heap size is: 500 GBGB. High search activity indices:logs-prod-000045 High indexing activity indices:logs-prod-000045
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Increase heap on node
Condition: Shown when rAM available on node and still abiding heap size rules.
Current max heap is 30 GB. Set JVM heap to 31 GB (suitable for this machine type), then restart the node.
Enable and review search slow logs
Condition: Always shown for this insight.
Enable search slow logs with the action below, then review the slow log to find expensive queries. See Slow logs for configuration details.
PUT logs-prod-000045/_settings
{
"index.search.slowlog.threshold.query.warn": "10s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "2s",
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.threshold.fetch.warn": "1s",
"index.search.slowlog.threshold.fetch.info": "800ms",
"index.search.slowlog.threshold.fetch.debug": "500ms",
"index.search.slowlog.threshold.fetch.trace": "200ms"
}
Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Balance old and young GC
Condition: Always shown for this insight.
Set -XX:NewRatio=2 in your JVM options file to tune the ratio between old and young GC, then perform a rolling restart of affected nodes.
Tune circuit breakers
Condition: Always shown for this insight.
Tune circuit breaker limits to prevent out-of-memory errors while allowing legitimate query and aggregation load. Adjust parent, fielddata, and request breaker settings in elasticsearch.yml based on your heap size and workload.
Upgrade node hardware
Condition: Shown when configured heap exceeds recommended max on a data node.
This node is under memory pressure. Reduce heap size to less than 31 GB.
Add coordinating node
Condition: Shown when no coordinating node and slow searches/queues/rejections exist.
Add a coordinating-only node to handle client requests and reduce load on data nodes.
Impact: This could cause node disconnection, affect index and search operations, and cause service interruptions. Heap size is the amount of RAM allocation to the Java Virtual Machine of a node. There can be a number of reasons why heap memory usage can increase. It’s important to keep the heap size at the right balance for proper function of your nodes.