Loading

Threshold queries in the experimental alerting system

A threshold query aggregates your data first, then applies an alert condition to the result. Use this pattern when a single matching event isn't enough to warrant an alert. You want to know whether a metric (count, rate, average) has crossed a limit over a time window.

This page covers two variants: a single-series threshold that produces one result for all your data, and a grouped threshold that tracks each subject (host, service, user) independently.

This query counts HTTP 5xx responses and error rates over the lookback window and alerts when the error rate for any service exceeds 10%.

FROM logs-*
| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
| STATS
    error_count = COUNT_IF(http.response.status_code >= 500),
    total_count = COUNT(*)
  BY service.name
| EVAL error_rate = error_count / total_count
| WHERE error_rate > 0.10
| KEEP service.name, error_rate, error_count, total_count
  1. Scope to the rule's configured lookback window
  2. Count only error responses
  3. Count all requests for the denominator
  4. Compute error rate as a fraction (0–1)
  5. Alert condition: services above 10% error rate are breaches

One window, one aggregate per service, one threshold check. The result is either a breach or no breach for each service.

Adding BY to the STATS command splits the result into one row per unique field value. Each row is evaluated independently against the alert condition, and each breach creates its own alert series.

This query tracks average CPU usage per host and alerts when any host exceeds 90%:

FROM metrics-*
| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend
| STATS avg_cpu = AVG(system.cpu.total.pct) BY host.name
| WHERE avg_cpu > 0.90
| KEEP host.name, avg_cpu
  1. One result row per host
  2. Each host above the threshold is a breach

Without the BY host.name clause, the query would produce a single average across all hosts combined. A cluster-wide spike might push the average above the threshold even if no single host is the problem. Grouping by host gives each host its own alert episode with its own lifecycle. If host A recovers but host B stays critical, those are tracked separately.

The BY fields you use here must match the grouping configuration in the rule. Refer to Rule grouping for how to align them.

If both of these patterns don't fit your use case, the next pages cover more specific detection problems: