Create a datafeed

PUT /_ml/datafeeds/{datafeed_id}

Datafeeds retrieve data from Elasticsearch for analysis by an anomaly detection job. You can associate only one datafeed with each anomaly detection job. The datafeed contains a query that runs at a defined interval (frequency). If you are concerned about delayed data, you can add a delay (query_delay') at each interval. By default, the datafeed uses the following query: {"match_all": {"boost": 1}}`.

When Elasticsearch security features are enabled, your datafeed remembers which roles the user who created it had at the time of creation and runs the query using those same roles. If you provide secondary authorization headers, those credentials are used instead. You must use Kibana, this API, or the create anomaly detection jobs API to create a datafeed. Do not add a datafeed directly to the .ml-config index. Do not give users write privileges on the .ml-config index.

Required authorization

Index privileges: read
Cluster privileges: manage_ml

Path Parameters

datafeed_id string Id required: A numerical character string that uniquely identifies the datafeed. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

Query Parameters

Request

aggregations map string to {} AggregationContainer: If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.
chunking_config {} ChunkingConfig: Datafeeds might be required to search over long time periods, for several months or years. This search is split into time chunks in order to ensure the load on Elasticsearch is managed. Chunking configuration controls how the size of these time chunks are calculated; it is an advanced configuration option.
delayed_data_check_config {} DelayedDataCheckConfig: Specifies whether the datafeed checks for missing data and the size of the window. The datafeed can optionally search over indices that have already been read in an effort to determine whether any data has subsequently been added to the index. If missing data is found, it is a good indication that the query_delay is set too low and the data is being indexed after the datafeed has passed that moment in time. This check runs only on real-time datafeeds.
frequency string Duration: The interval at which scheduled queries are made while the datafeed runs in real time. The default value is either the bucket span for short bucket spans, or, for longer bucket spans, a sensible fraction of the bucket span. When frequency is shorter than the bucket span, interim results for the last (partial) bucket are written then eventually overwritten by the full bucket results. If the datafeed uses aggregations, this value must be divisible by the interval of the date histogram aggregation.
indices string Indices: An array of index names. Wildcards are supported. If any of the indices are in remote clusters, the master nodes and the machine learning nodes must have the remote_cluster_client role.
indices_options {} IndicesOptions: Specifies index expansion options that are used during search
job_id string Id: Identifier for the anomaly detection job.
max_empty_searches number: If a real-time datafeed has never seen any data (including during any initial training period), it automatically stops and closes the associated job after this many real-time searches return no documents. In other words, it stops after frequency times max_empty_searches of real-time operation. If not set, a datafeed with no end time that sees no data remains started until it is explicitly stopped. By default, it is not set.
max_consecutive_extraction_failures number: The maximum number of consecutive real-time data extraction failures the datafeed tolerates before it automatically stops itself, leaving the associated job open. The consecutive-failure counter resets on any cycle that extracts successfully, including empty-data cycles. If not set, the threshold defaults to roughly one day's worth of searches based on the datafeed frequency (floored at 1). Set to -1 to disable auto-stop and retry indefinitely. Values of 0 or less than -1 are rejected.
query {} QueryContainer · default: {"match_all": {"boost": 1}}: The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this object is passed verbatim to Elasticsearch.
query_delay string Duration: The number of seconds behind real time that data is queried. For example, if data from 10:04 a.m. might not be searchable in Elasticsearch until 10:06 a.m., set this property to 120 seconds. The default value is randomly selected between 60s and 120s. This randomness improves the query performance when there are multiple jobs running on the same node.
runtime_mappings RuntimeFields: Specifies runtime fields for the datafeed search.
script_fields map string to ScriptField: Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.
scroll_size number · default: 1000: The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of index.max_result_window, which is 10,000 by default.
headers HttpHeaders

Response

application/json

Examples

An example body for a PUT _ml/datafeeds/datafeed-test-job?pretty request.

Request Console 1 / 8 languages

Console Kibana Dev Tools

PUT _ml/datafeeds/datafeed-test-job?pretty
{
  "indices": [
    "kibana_sample_data_logs"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        }
      ]
    }
  },
  "job_id": "test-job"
}

JavaScript Elasticsearch JavaScript Client

const response = await client.ml.putDatafeed({
  datafeed_id: "datafeed-test-job",
  pretty: "true",
  indices: ["kibana_sample_data_logs"],
  query: {
    bool: {
      must: [
        {
          match_all: {},
        },
      ],
    },
  },
  job_id: "test-job",
});

Python Elasticsearch Python Client

resp = client.ml.put_datafeed(
    datafeed_id="datafeed-test-job",
    pretty=True,
    indices=[
        "kibana_sample_data_logs"
    ],
    query={
        "bool": {
            "must": [
                {
                    "match_all": {}
                }
            ]
        }
    },
    job_id="test-job",
)

curl Shell

curl -X PUT "$ELASTICSEARCH_URL/_ml/datafeeds/datafeed-test-job?pretty" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "indices": [
      "kibana_sample_data_logs"
    ],
    "query": {
      "bool": {
        "must": [
          {
            "match_all": {}
          }
        ]
      }
    },
    "job_id": "test-job"
  }'

Java Elasticsearch Java Client

client.ml().putDatafeed(p -> p
    .datafeedId("datafeed-test-job")
    .indices("kibana_sample_data_logs")
    .jobId("test-job")
    .query(q -> q
        .bool(b -> b
            .must(m -> m
                .matchAll(ma -> ma)
            )
        )
    )
);

C# Elasticsearch .NET Client

var response = await client.MachineLearning
    .PutDatafeedAsync(datafeedId: "datafeed-test-job", d1 => d1
        .Indices(new[] { "kibana_sample_data_logs" })
        .JobId("test-job")
        .Query(d2 => d2
            .Bool(d3 => d3
                .Must(d4 => d4
                    .MatchAll()
                )
            )
        )
    );

PHP Elasticsearch PHP Client

$resp = $client->ml()->putDatafeed([
    "datafeed_id" => "datafeed-test-job",
    "pretty" => "true",
    "body" => [
        "indices" => array(
            "kibana_sample_data_logs",
        ),
        "query" => [
            "bool" => [
                "must" => array(
                    [
                        "match_all" => new ArrayObject([]),
                    ],
                ),
            ],
        ],
        "job_id" => "test-job",
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.ml.put_datafeed(
  datafeed_id: "datafeed-test-job",
  pretty: "true",
  body: {
    "indices": [
      "kibana_sample_data_logs"
    ],
    "query": {
      "bool": {
        "must": [
          {
            "match_all": {}
          }
        ]
      }
    },
    "job_id": "test-job"
  }
)

Response

No example

Required authorization

Path Parameters

Query Parameters { allow_no_indices , expand_wildcards , ignore_throttled , ignore_unavailable , }

Request { aggregations , chunking_config , delayed_data_check_config , frequency , indices , indices_options , job_id , max_empty_searches , max_consecutive_extraction_failures , query , query_delay , runtime_mappings , script_fields , scroll_size , headers , }

Response

Query Parameters

Request