Change point aggregation

A sibling pipeline that detects, spikes, dips, and change points in a metric. Given a distribution of values provided by the sibling multi-bucket aggregation, this aggregation indicates the bucket of any spike or dip and/or the bucket at which the largest change in the distribution of values, if they are statistically significant.

buckets_path

(Required, string) Path to the buckets that contain one set of values in which to detect a change point. There must be at least 22 bucketed values. Fewer than 1,000 is preferred. For syntax, see buckets_path Syntax.

gap_policy

(Optional, string) Determines what to do with buckets that contain no documents. Defaults to skip. For the full description of each policy, see Dealing with gaps in the data. Earlier versions accept this parameter but always behave as skip.

skip

Empty buckets are dropped before the change point is detected. Because a sparse series can fall below the 22-value minimum this way, a change point that is visible in the data may be reported as indeterminable.

keep_values

Empty buckets are kept whenever the metric still reports a finite value. A sum over an empty bucket reports 0, so those buckets are analyzed. A min, max or avg over an empty bucket has no value to keep, so those buckets are still dropped.

insert_zeros

Empty buckets are analyzed as 0, whatever the metric reports.

A buckets_path ending in _count is unaffected by this setting, because a document count is never treated as missing.

Warning

A date_histogram uses min_doc_count: 0 by default, so a sparse series can produce a long run of empty buckets. With insert_zeros every one of them is analyzed as 0, which can both dominate the result and push the series past the 1,000 buckets this aggregation is designed for.

A change_point aggregation looks like this in isolation:

{
  "change_point": {
    "buckets_path": "date_histogram>_count"
  }
}
		

1. The buckets containing the values to test against.

bucket

(Optional, object) Values of the bucket that indicates the discovered change point. Not returned if no change point was found. All the aggregations in the bucket are returned as well.

Properties of `bucket:

key

(value) The key of the bucket matched. Could be string or numeric.

doc_count

(number) The document count of the bucket.

type

(object) The found change point type and its related values. Possible types:

• dip: a significant dip occurs at this change point
• distribution_change: the overall distribution of the values has changed significantly
• non_stationary: there is no change point, but the values are not from a stationary distribution
• spike: a significant spike occurs at this point
• stationary: no change point found
• step_change: the change indicates a statistically significant step up or down in value distribution
• trend_change: there is an overall trend change occurring at this point

The following example uses the Kibana sample data logs data set.

GET kibana_sample_data_logs/_search
{
  "aggs": {
    "date":{
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "1d"
      },
      "aggs": {
        "avg": {
          "avg": {
            "field": "bytes"
          }
        }
      }
    },
    "change_points_avg": {
      "change_point": {
        "buckets_path": "date>avg"
      }
    }
  }
}
		

1. A date histogram aggregation that creates buckets with one day long interval.
2. A sibling aggregation of the date aggregation that calculates the average value of the bytes field within every bucket.
3. The change point detection aggregation configuration object.
4. The path of the aggregation values to detect change points. In this case, the input of the change point aggregation is the value of avg which is a sibling aggregation of date.

The request returns a response that is similar to the following:

"change_points_avg" : {
  "bucket" : {
    "key" : "2023-04-29T00:00:00.000Z",
    "doc_count" : 329,
    "avg" : {
      "value" : 4737.209726443769
    }
  },
  "type" : {
    "dip" : {
      "p_value" : 3.8999455212466465e-10,
      "change_point" : 41
    }
  }
}
		

1. The bucket key that is the change point.
2. The number of documents in that bucket.
3. Aggregated values in the bucket.
4. Type of change found.
5. The p_value indicates how extreme the change is; lower values indicate greater change.
6. The specific bucket where the change occurs (indexing starts at 0).