Vector memory pressure
This insight targets HNSW vector indices: dense vectors used for semantic and kNN search whose graphs are held primarily in physical memory (off the Java heap). A node can look healthy on heap charts while HNSW off-heap use still consumes a large share of RAM.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | High |
| Scope | Node |
| Domains | performance, search, memory, vector |
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
This insight applies to HNSW vector indices: the default in-memory dense-vector index type in Elasticsearch, where the approximate nearest-neighbor graph is kept off the Java heap in node RAM. In what we can measure, the cushion between available RAM and HNSW off-heap footprint is tight or shrinking, and there are other signs of load on the node. That usually means a growing risk of slower kNN search, stuck ingest, or a stressed node if load does not ease. Affected node(s): es-data-01 and es-data-02. 30 Java heap usage: JVM heap is also elevated on this node off-heap vector memory and field data
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Reduce vector footprint
Condition: Shown when vector footprint is growing, headroom is escaping, or compression regime is tight (vecGrowth ∨ headroomEsc ∨ compTight).
Reduce HNSW off-heap memory on this node where quality allows: use fewer dimensions, quantized or compressed vector mappings, delete or archive unneeded data, split indices, or reindex with a leaner mapping. See BBQ HNSW for options. Validate recall and latency against your SLOs after changes.
Switch to DiskBBQ vector index
Condition: Shown when heap usage is above the node's 24h median threshold (HEAP_HOT) and this event is active.
When HNSW indices keep the full graph in physical memory, switch to DiskBBQ to store most of the graph on disk instead of RAM. Expect different recall and latency versus HNSW; test before changing production mappings.
Upgrade node hardware
Condition: Shown when headroom is escaping or compression regime is tight (headroomEsc ∨ compTight).
This node is under memory pressure. Reduce heap size to less than 31 GB.
Tune kNN query load
Condition: Shown when search latency is hot relative to baseline without a matching rate increase (SRCH_LAT_HOT).
Reduce kNN load when search latency is high: lower num_candidates, reduce concurrent query rate or client fan-out, and narrow knn or script_score usage where possible. Re-check latency and error rates after changes.
Enable and review search slow logs
Condition: Shown when search or index latency stress is present (SRCH_LAT_HOT ∨ IDX_LAT_HOT).
Enable search slow logs with the action below, then review the slow log to find expensive queries. See Slow logs for configuration details.
PUT logs-prod-000045/_settings
{
"index.search.slowlog.threshold.query.warn": "10s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "2s",
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.threshold.fetch.warn": "1s",
"index.search.slowlog.threshold.fetch.info": "800ms",
"index.search.slowlog.threshold.fetch.debug": "500ms",
"index.search.slowlog.threshold.fetch.trace": "200ms"
}
Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Increase heap on node
Condition: Shown when heap usage is above the node's 24h median threshold (HEAP_HOT).
Current max heap is 30 GB. Set JVM heap to 31 GB (suitable for this machine type), then restart the node.
Tune circuit breakers
Condition: Shown when circuit breaker is hot and heap, search/index latency, or disk read stress is also present (BRK_HOT paired with HEAP_HOT, SRCH_LAT_HOT, IDX_LAT_HOT, or FS_RD_HOT).
Tune circuit breaker limits to prevent out-of-memory errors while allowing legitimate query and aggregation load. Adjust parent, fielddata, and request breaker settings in elasticsearch.yml based on your heap size and workload.
Impact: When HNSW off-heap footprint grows into a tight memory band, kNN and dense-vector workloads on in-memory graphs compete with indexing, search, and disk I/O for the same node. Latency can rise, thread pools can back up, and circuit breakers might trip if pressure continues. AutoOps uses dense-vector desired off-heap size as a proxy for HNSW graph pressure in RAM and requires corroborating subsystem stress before firing. Disk-read and breaker signals only count toward escape when paired with other stress so isolated spikes do not fire the insight alone.