Create an anomaly detection job

PUT /_ml/anomaly_detectors/{job_id}

If you include a datafeed_config, you must have read index privileges on the source index. If you include a datafeed_config but do not provide a query, the datafeed uses {"match_all": {"boost": 1}}.

Required authorization

Authentication

Api key

Authorization: <value>

Elasticsearch APIs support key-based authentication. You must create an API key and use the encoded value in the request header. For example:

		curl -X GET "${ES_URL}/_cat/indices?v=true" \
  -H "Authorization: ApiKey ${API_KEY}"

To get API keys, use the /_security/api_key APIs.

Basic

Authorization: Basic <credentials>

Bearer

Authorization: Bearer <token>

Elasticsearch APIs support the use of bearer tokens in the Authorization HTTP header to authenticate with the API. For examples, refer to Token-based authentication services

Path parameters

Query parameters

Request body

allow_lazy_open boolean · default: false: Advanced configuration option. Specifies whether this job can open when there is insufficient machine learning node capacity for it to be immediately assigned to a node. By default, if a machine learning node with capacity to run the job cannot immediately be found, the open anomaly detection jobs API returns an error. However, this is also subject to the cluster-wide xpack.ml.max_lazy_ml_nodes setting. If this option is set to true, the open anomaly detection jobs API does not return an error and the job waits in the opening state until sufficient machine learning node capacity is available.
analysis_config object required: Specifies how to analyze the data. After you create a job, you cannot change the analysis configuration; all the properties are informational.
analysis_limits object: Limits can be applied for the resources required to hold the mathematical models in memory. These limits are approximate and can be set per job. They do not control the memory used by other processes, for example the Elasticsearch Java processes.
background_persist_interval string: Advanced configuration option. The time between each periodic persistence of the model. The default value is a randomized value between 3 to 4 hours, which avoids all jobs persisting at exactly the same time. The smallest allowed value is 1 hour. For very large models (several GB), persistence could take 10-20 minutes, so do not set the background_persist_interval value too low.
custom_settings object: Advanced configuration option. Contains custom meta data about the job.
daily_model_snapshot_retention_after_days number · default: 1: Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies a period of time (in days) after which only the first snapshot per day is retained. This period is relative to the timestamp of the most recent snapshot for this job. Valid values range from 0 to model_snapshot_retention_days.
data_description object required: Defines the format of the input data when you send data to the job by using the post data API. Note that when configure a datafeed, these properties are automatically set. When data is received via the post data API, it is not stored in Elasticsearch. Only the results for anomaly detection are retained.
datafeed_config object: Defines a datafeed for the anomaly detection job. If Elasticsearch security features are enabled, your datafeed remembers which roles the user who created it had at the time of creation and runs the query using those same roles. If you provide secondary authorization headers, those credentials are used instead.
description string: A description of the job.
job_id string: The identifier for the anomaly detection job. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.
groups string[]: A list of job groups. A job can belong to no groups or many.
model_plot_config object: This advanced configuration option stores model information along with the results. It provides a more detailed view into anomaly detection. If you enable model plot it can add considerable overhead to the performance of the system; it is not feasible for jobs with many entities. Model plot provides a simplified and indicative view of the model and its bounds. It does not display complex features such as multivariate correlations or multimodal data. As such, anomalies may occasionally be reported which cannot be seen in the model plot. Model plot config can be configured when the job is created or updated later. It must be disabled if performance issues are experienced.
model_snapshot_retention_days number · default: 10: Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies the maximum period of time (in days) that snapshots are retained. This period is relative to the timestamp of the most recent snapshot for this job. By default, snapshots ten days older than the newest snapshot are deleted.
renormalization_window_days number: Advanced configuration option. The period over which adjustments to the score are applied, as new data is seen. The default value is the longer of 30 days or 100 bucket spans.
results_index_name string · default: shared: A text string that affects the name of the machine learning results index. By default, the job generates an index named .ml-anomalies-shared.
results_retention_days number: Advanced configuration option. The period of time (in days) that results are retained. Age is calculated relative to the timestamp of the latest bucket result. If this property has a non-null value, once per day at 00:30 (server time), results that are the specified number of days older than the latest bucket result are deleted from Elasticsearch. The default value is null, which means all results are retained. Annotations generated by the system also count as results for retention purposes; they are deleted after the same number of days as results. Annotations added by users are retained forever.

Response

application/json

allow_lazy_open boolean required
analysis_config object required
analysis_limits object required
background_persist_interval string
create_time union required
custom_settings object
daily_model_snapshot_retention_after_days number required
data_description object required
datafeed_config object
description string
groups string[]
job_id string required
job_type string required
job_version string required
model_plot_config object
model_snapshot_id string
model_snapshot_retention_days number required
renormalization_window_days number
results_index_name string required
results_retention_days number

Examples

Request Console 1 / 8 languages

A request to create an anomaly detection job and datafeed.

Console Kibana Dev Tools

PUT /_ml/anomaly_detectors/job-01
{
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "detector_description": "Sum of bytes",
        "function": "sum",
        "field_name": "bytes"
      }
    ]
  },
  "data_description": {
    "time_field": "timestamp",
    "time_format": "epoch_ms"
  },
  "analysis_limits": {
    "model_memory_limit": "11MB"
  },
  "model_plot_config": {
    "enabled": true,
    "annotations_enabled": true
  },
  "results_index_name": "test-job1",
  "datafeed_config": {
    "indices": [
      "kibana_sample_data_logs"
    ],
    "query": {
      "bool": {
        "must": [
          {
            "match_all": {}
          }
        ]
      }
    },
    "runtime_mappings": {
      "hour_of_day": {
        "type": "long",
        "script": {
          "source": "emit(doc['timestamp'].value.getHour());"
        }
      }
    },
    "datafeed_id": "datafeed-test-job1"
  }
}

JavaScript Elasticsearch JavaScript Client

const response = await client.ml.putJob({
  job_id: "job-01",
  analysis_config: {
    bucket_span: "15m",
    detectors: [
      {
        detector_description: "Sum of bytes",
        function: "sum",
        field_name: "bytes",
      },
    ],
  },
  data_description: {
    time_field: "timestamp",
    time_format: "epoch_ms",
  },
  analysis_limits: {
    model_memory_limit: "11MB",
  },
  model_plot_config: {
    enabled: true,
    annotations_enabled: true,
  },
  results_index_name: "test-job1",
  datafeed_config: {
    indices: ["kibana_sample_data_logs"],
    query: {
      bool: {
        must: [
          {
            match_all: {},
          },
        ],
      },
    },
    runtime_mappings: {
      hour_of_day: {
        type: "long",
        script: {
          source: "emit(doc['timestamp'].value.getHour());",
        },
      },
    },
    datafeed_id: "datafeed-test-job1",
  },
});

Python Elasticsearch Python Client

resp = client.ml.put_job(
    job_id="job-01",
    analysis_config={
        "bucket_span": "15m",
        "detectors": [
            {
                "detector_description": "Sum of bytes",
                "function": "sum",
                "field_name": "bytes"
            }
        ]
    },
    data_description={
        "time_field": "timestamp",
        "time_format": "epoch_ms"
    },
    analysis_limits={
        "model_memory_limit": "11MB"
    },
    model_plot_config={
        "enabled": True,
        "annotations_enabled": True
    },
    results_index_name="test-job1",
    datafeed_config={
        "indices": [
            "kibana_sample_data_logs"
        ],
        "query": {
            "bool": {
                "must": [
                    {
                        "match_all": {}
                    }
                ]
            }
        },
        "runtime_mappings": {
            "hour_of_day": {
                "type": "long",
                "script": {
                    "source": "emit(doc['timestamp'].value.getHour());"
                }
            }
        },
        "datafeed_id": "datafeed-test-job1"
    },
)

curl Shell

curl -X PUT "$ELASTICSEARCH_URL/_ml/anomaly_detectors/job-01" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"analysis_config":{"bucket_span":"15m","detectors":[{"detector_description":"Sum of bytes","function":"sum","field_name":"bytes"}]},"data_description":{"time_field":"timestamp","time_format":"epoch_ms"},"analysis_limits":{"model_memory_limit":"11MB"},"model_plot_config":{"enabled":true,"annotations_enabled":true},"results_index_name":"test-job1","datafeed_config":{"indices":["kibana_sample_data_logs"],"query":{"bool":{"must":[{"match_all":{}}]}},"runtime_mappings":{"hour_of_day":{"type":"long","script":{"source":"emit(doc[' \
  "'" \
  'timestamp' \
  "'" \
  '].value.getHour());"}}},"datafeed_id":"datafeed-test-job1"}}'

Java Elasticsearch Java Client

client.ml().putJob(p -> p
    .analysisConfig(a -> a
        .bucketSpan(b -> b
            .time("15m")
        )
        .detectors(d -> d
            .detectorDescription("Sum of bytes")
            .fieldName("bytes")
            .function("sum")
        )
    )
    .analysisLimits(an -> an
        .modelMemoryLimit("11MB")
    )
    .dataDescription(d -> d
        .timeField("timestamp")
        .timeFormat("epoch_ms")
    )
    .datafeedConfig(d -> d
        .datafeedId("datafeed-test-job1")
        .indices("kibana_sample_data_logs")
        .query(q -> q
            .bool(b -> b
                .must(m -> m
                    .matchAll(ma -> ma)
                )
            )
        )
        .runtimeMappings("hour_of_day", r -> r
            .script(s -> s
                .source(so -> so
                    .scriptString("emit(doc['timestamp'].value.getHour());")
                )
            )
            .type(RuntimeFieldType.Long)
        )
    )
    .jobId("job-01")
    .modelPlotConfig(m -> m
        .annotationsEnabled(true)
        .enabled(true)
    )
    .resultsIndexName("test-job1")
);

C# Elasticsearch .NET Client

var response = await client.MachineLearning
    .PutJobAsync(jobId: "job-01", d1 => d1
        .AnalysisConfig(d2 => d2
            .BucketSpan("15m")
            .Detectors(d3 => d3
                .DetectorDescription("Sum of bytes")
                .FieldName(x => x.Bytes)
                .Function("sum")
            )
        )
        .AnalysisLimits(d2 => d2
            .ModelMemoryLimit(new ByteSize("11MB"))
        )
        .DataDescription(d2 => d2
            .TimeField(x => x.Timestamp)
            .TimeFormat("epoch_ms")
        )
        .DatafeedConfig(d2 => d2
            .DatafeedId("datafeed-test-job1")
            .Indices(new[] { "kibana_sample_data_logs" })
            .Query(d3 => d3
                .Bool(d4 => d4
                    .Must(d5 => d5
                        .MatchAll()
                    )
                )
            )
            .RuntimeMappings(d3 => d3
                .Add(global::Elastic.Clients.Elasticsearch.Infer.Field<MyDocument>(x => x.HourOfDay), d4 => d4
                    .Script(d5 => d5
                        .Source("emit(doc['timestamp'].value.getHour());")
                    )
                    .Type(RuntimeFieldType.Long)
                )
            )
        )
        .ModelPlotConfig(d2 => d2
            .AnnotationsEnabled(true)
            .Enabled(true)
        )
        .ResultsIndexName("test-job1")
    );

PHP Elasticsearch PHP Client

$resp = $client->ml()->putJob([
    "job_id" => "job-01",
    "body" => [
        "analysis_config" => [
            "bucket_span" => "15m",
            "detectors" => array(
                [
                    "detector_description" => "Sum of bytes",
                    "function" => "sum",
                    "field_name" => "bytes",
                ],
            ),
        ],
        "data_description" => [
            "time_field" => "timestamp",
            "time_format" => "epoch_ms",
        ],
        "analysis_limits" => [
            "model_memory_limit" => "11MB",
        ],
        "model_plot_config" => [
            "enabled" => true,
            "annotations_enabled" => true,
        ],
        "results_index_name" => "test-job1",
        "datafeed_config" => [
            "indices" => array(
                "kibana_sample_data_logs",
            ),
            "query" => [
                "bool" => [
                    "must" => array(
                        [
                            "match_all" => new ArrayObject([]),
                        ],
                    ),
                ],
            ],
            "runtime_mappings" => [
                "hour_of_day" => [
                    "type" => "long",
                    "script" => [
                        "source" => "emit(doc['timestamp'].value.getHour());",
                    ],
                ],
            ],
            "datafeed_id" => "datafeed-test-job1",
        ],
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.ml.put_job(
  job_id: "job-01",
  body: {
    "analysis_config": {
      "bucket_span": "15m",
      "detectors": [
        {
          "detector_description": "Sum of bytes",
          "function": "sum",
          "field_name": "bytes"
        }
      ]
    },
    "data_description": {
      "time_field": "timestamp",
      "time_format": "epoch_ms"
    },
    "analysis_limits": {
      "model_memory_limit": "11MB"
    },
    "model_plot_config": {
      "enabled": true,
      "annotations_enabled": true
    },
    "results_index_name": "test-job1",
    "datafeed_config": {
      "indices": [
        "kibana_sample_data_logs"
      ],
      "query": {
        "bool": {
          "must": [
            {
              "match_all": {}
            }
          ]
        }
      },
      "runtime_mappings": {
        "hour_of_day": {
          "type": "long",
          "script": {
            "source": "emit(doc['timestamp'].value.getHour());"
          }
        }
      },
      "datafeed_id": "datafeed-test-job1"
    }
  }
)

Response

A successful response when creating an anomaly detection job and datafeed.

{
  "job_id": "test-job1",
  "job_type": "anomaly_detector",
  "job_version": "8.4.0",
  "create_time": 1656087283340,
  "datafeed_config": {
    "datafeed_id": "datafeed-test-job1",
    "job_id": "test-job1",
    "authorization": {
      "roles": [
        "superuser"
      ]
    },
    "query_delay": "61499ms",
    "chunking_config": {
      "mode": "auto"
    },
    "indices_options": {
      "expand_wildcards": [
        "open"
      ],
      "ignore_unavailable": false,
      "allow_no_indices": true,
      "ignore_throttled": true
    },
    "query": {
      "bool": {
        "must": [
          {
            "match_all": {}
          }
        ]
      }
    },
    "indices": [
      "kibana_sample_data_logs"
    ],
    "scroll_size": 1000,
    "delayed_data_check_config": {
      "enabled": true
    },
    "runtime_mappings": {
      "hour_of_day": {
        "type": "long",
        "script": {
          "source": "emit(doc['timestamp'].value.getHour());"
        }
      }
    }
  },
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "detector_description": "Sum of bytes",
        "function": "sum",
        "field_name": "bytes",
        "detector_index": 0
      }
    ],
    "influencers": [],
    "model_prune_window": "30d"
  },
  "analysis_limits": {
    "model_memory_limit": "11mb",
    "categorization_examples_limit": 4
  },
  "data_description": {
    "time_field": "timestamp",
    "time_format": "epoch_ms"
  },
  "model_plot_config": {
    "enabled": true,
    "annotations_enabled": true
  },
  "model_snapshot_retention_days": 10,
  "daily_model_snapshot_retention_after_days": 1,
  "results_index_name": "custom-test-job1",
  "allow_lazy_open": false
}

Required authorization

Authentication

Path parameters

Query parameters

Request body { }

Response

Request body