---
type: api
title: Create a datafeed
description: Datafeeds retrieve data from Elasticsearch for analysis by an anomaly detection job.
url: https://www.elastic.co/elastic/docs-builder/docs/4371/api/doc/elasticsearch-serverless/operation/operation-ml-put-datafeed
resource: https://www.elastic.co/elastic/docs-builder/docs/4371/api/doc/elasticsearch-serverless/operation/operation-ml-put-datafeed
products:
  - Elasticsearch Serverless
---

# Create a datafeed

Availability:

## Paths

- `PUT` `/_ml/datafeeds/{datafeed_id}`

- [Api key auth](/elastic/docs-builder/docs/4371/api/doc/elasticsearch-serverless/authentication#apikeyauth)

#### Path Parameters

- `datafeed_id` (string Id)
  A numerical character string that uniquely identifies the datafeed.
This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores.
It must start and end with alphanumeric characters.

## Description

Datafeeds retrieve data from Elasticsearch for analysis by an anomaly detection job.
You can associate only one datafeed with each anomaly detection job.
The datafeed contains a query that runs at a defined interval (`frequency`).
If you are concerned about delayed data, you can add a delay (`query_delay') at each interval.
By default, the datafeed uses the following query: `{"match_all": {"boost": 1}}`.

When Elasticsearch security features are enabled, your datafeed remembers which roles the user who created it had
at the time of creation and runs the query using those same roles. If you provide secondary authorization headers,
those credentials are used instead.
You must use Kibana, this API, or the create anomaly detection jobs API to create a datafeed. Do not add a datafeed
directly to the `.ml-config` index. Do not give users `write` privileges on the `.ml-config` index.

## Required authorization

* Index privileges: `read`
* Cluster privileges: `manage_ml`

## Query String Parameters

- `allow_no_indices` (boolean · default: true)
  A setting that does two separate checks on the index expression.
If `false`, the request returns an error (1) if any wildcard expression
(including `_all` and `*`) resolves to zero matching indices or (2) if the
complete set of resolved indices, aliases or data streams is empty after all
expressions are evaluated. If `true`, index expressions that resolve to no
indices are allowed and the request returns an empty result.
- `expand_wildcards` (string ExpandWildcards · default: open)
  Type of index that wildcard patterns can match. If the request can target data streams, this argument determines
whether wildcard expressions match hidden data streams. Supports comma-separated values.

Supported values include:
  - `all`: Match any data stream or index, including hidden ones.
  - `open`: Match open, non-hidden indices. Also matches any non-hidden data stream.
  - `closed`: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
  - `hidden`: Match hidden data streams and hidden indices. Must be combined with `open`, `closed`, or `both`.
  - `none`: Wildcard expressions are not accepted.
  Values: `all`, `open`, `closed`, `hidden`, `none`
- `ignore_throttled` (boolean · default: true) — deprecated
  If true, concrete, expanded, or aliased indices are ignored when frozen.
- `ignore_unavailable` (boolean · default: false)
  If `false`, the request returns an error if it targets a concrete (non-wildcarded)
index, alias, or data stream that is missing, closed, or otherwise unavailable.
If `true`, unavailable concrete targets are silently ignored.

## Request Body

`application/json`

- `aggregations` (map string to {} AggregationContainer) — optional
  If set, the datafeed performs aggregation searches.
Support for aggregations is limited and should be used only with low cardinality data.
  See [AggregationContainer](https://www.elastic.co/elastic/docs-builder/docs/4371/api/doc/elasticsearch-serverless/types/_types-aggregations-aggregationcontainer)
- `chunking_config` ({} ChunkingConfig) — optional
  Datafeeds might be required to search over long time periods, for several months or years.
This search is split into time chunks in order to ensure the load on Elasticsearch is managed.
Chunking configuration controls how the size of these time chunks are calculated;
it is an advanced configuration option.
  - `mode` (enum ChunkingMode) — required
    If the mode is `auto`, the chunk size is dynamically calculated;
this is the recommended value when the datafeed does not use aggregations.
If the mode is `manual`, chunking is applied according to the specified `time_span`;
use this mode when the datafeed uses aggregations. If the mode is `off`, no chunking is applied.
    Values: `auto`, `manual`, `off`
  - `time_span` (string Duration · default: 3h) — optional
    The time span that each search will be querying. This setting is applicable only when the `mode` is set to `manual`.
- `delayed_data_check_config` ({} DelayedDataCheckConfig) — optional
  Specifies whether the datafeed checks for missing data and the size of the window.
The datafeed can optionally search over indices that have already been read in an effort to determine whether
any data has subsequently been added to the index. If missing data is found, it is a good indication that the
`query_delay` is set too low and the data is being indexed after the datafeed has passed that moment in time.
This check runs only on real-time datafeeds.
  - `check_window` (string Duration) — optional
    The window of time that is searched for late data. This window of time ends with the latest finalized bucket.
It defaults to null, which causes an appropriate `check_window` to be calculated when the real-time datafeed runs.
In particular, the default `check_window` span calculation is based on the maximum of `2h` or `8 * bucket_span`.
  - `enabled` (boolean) — required
    Specifies whether the datafeed periodically checks for delayed data.
- `frequency` (string Duration) — optional
  The interval at which scheduled queries are made while the datafeed runs in real time.
The default value is either the bucket span for short bucket spans, or, for longer bucket spans, a sensible
fraction of the bucket span. When `frequency` is shorter than the bucket span, interim results for the last
(partial) bucket are written then eventually overwritten by the full bucket results. If the datafeed uses
aggregations, this value must be divisible by the interval of the date histogram aggregation.
- `indices` (string Indices) — optional
  An array of index names. Wildcards are supported. If any of the indices are in remote clusters, the master
nodes and the machine learning nodes must have the `remote_cluster_client` role.
- `indices_options` ({} IndicesOptions) — optional
  Specifies index expansion options that are used during search
  - `allow_no_indices` (boolean) — optional
    A setting that does two separate checks on the index expression.
If `false`, the request returns an error (1) if any wildcard expression
(including `_all` and `*`) resolves to zero matching indices or (2) if the
complete set of resolved indices, aliases or data streams is empty after all
expressions are evaluated. If `true`, index expressions that resolve to no
indices are allowed and the request returns an empty result.
  - `expand_wildcards` (string ExpandWildcards) — optional
    Type of index that wildcard patterns can match. If the request can target data streams, this argument
determines whether wildcard expressions match hidden data streams. Supports comma-separated values,
such as `open,hidden`.

Supported values include:
  - `all`: Match any data stream or index, including hidden ones.
  - `open`: Match open, non-hidden indices. Also matches any non-hidden data stream.
  - `closed`: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
  - `hidden`: Match hidden data streams and hidden indices. Must be combined with `open`, `closed`, or `both`.
  - `none`: Wildcard expressions are not accepted.
    Values: `all`, `open`, `closed`, `hidden`, `none`
  - `ignore_unavailable` (boolean · default: false) — optional
    If `false`, the request returns an error if it targets a concrete (non-wildcarded)
index, alias, or data stream that is missing, closed, or otherwise unavailable.
If `true`, unavailable concrete targets are silently ignored.
  - `ignore_throttled` (boolean · default: true) — optional
    If true, concrete, expanded or aliased indices are ignored when frozen.
- `job_id` (string Id) — optional
  Identifier for the anomaly detection job.
- `max_empty_searches` (number) — optional
  If a real-time datafeed has never seen any data (including during any initial training period), it automatically
stops and closes the associated job after this many real-time searches return no documents. In other words,
it stops after `frequency` times `max_empty_searches` of real-time operation. If not set, a datafeed with no
end time that sees no data remains started until it is explicitly stopped. By default, it is not set.
- `max_consecutive_extraction_failures` (number) — optional
  The maximum number of consecutive real-time data extraction failures the datafeed tolerates before it
automatically stops itself, leaving the associated job open. The consecutive-failure counter resets on any
cycle that extracts successfully, including empty-data cycles. If not set, the threshold defaults to roughly
one day's worth of searches based on the datafeed `frequency` (floored at 1). Set to `-1` to disable auto-stop
and retry indefinitely. Values of `0` or less than `-1` are rejected.
- `query` ({} QueryContainer · default: {"match_all": {"boost": 1}}) — optional
  The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an
Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this
object is passed verbatim to Elasticsearch.
  See [QueryContainer](https://www.elastic.co/elastic/docs-builder/docs/4371/api/doc/elasticsearch-serverless/types/_types-query_dsl-querycontainer)
- `query_delay` (string Duration) — optional
  The number of seconds behind real time that data is queried. For example, if data from 10:04 a.m. might
not be searchable in Elasticsearch until 10:06 a.m., set this property to 120 seconds. The default
value is randomly selected between `60s` and `120s`. This randomness improves the query performance
when there are multiple jobs running on the same node.
- `runtime_mappings` (RuntimeFields) — optional
  Specifies runtime fields for the datafeed search.
- `script_fields` (map string to ScriptField) — optional
  Specifies scripts that evaluate custom expressions and returns script fields to the datafeed.
The detector configuration objects in a job can contain functions that use these script fields.
  Map values ({} ScriptField)
  - `script` (string Script) — required
  - `ignore_failure` (boolean) — optional
- `scroll_size` (number · default: 1000) — optional
  The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations.
The maximum value is the value of `index.max_result_window`, which is 10,000 by default.
- `headers` (HttpHeaders) — optional

## Response

`application/json`

- `aggregations` (map string to {} AggregationContainer)
  See [AggregationContainer](https://www.elastic.co/elastic/docs-builder/docs/4371/api/doc/elasticsearch-serverless/types/_types-aggregations-aggregationcontainer)
- `authorization` ({} DatafeedAuthorization)
  - `api_key` ({} ApiKeyAuthorization)
    If an API key was used for the most recent update to the datafeed, its name and identifier are listed in the response.
    - `id` (string)
      The identifier for the API key.
    - `name` (string)
      The name of the API key.
  - `roles` ([] string)
    If a user ID was used for the most recent update to the datafeed, its roles at the time of the update are listed in the response.
  - `service_account` (string)
    If a service account was used for the most recent update to the datafeed, the account name is listed in the response.
- `chunking_config` ({} ChunkingConfig)
  - `mode` (enum ChunkingMode)
    If the mode is `auto`, the chunk size is dynamically calculated;
this is the recommended value when the datafeed does not use aggregations.
If the mode is `manual`, chunking is applied according to the specified `time_span`;
use this mode when the datafeed uses aggregations. If the mode is `off`, no chunking is applied.
    Values: `auto`, `manual`, `off`
  - `time_span` (string Duration · default: 3h)
    The time span that each search will be querying. This setting is applicable only when the `mode` is set to `manual`.
- `delayed_data_check_config` ({} DelayedDataCheckConfig)
  - `check_window` (string Duration)
    The window of time that is searched for late data. This window of time ends with the latest finalized bucket.
It defaults to null, which causes an appropriate `check_window` to be calculated when the real-time datafeed runs.
In particular, the default `check_window` span calculation is based on the maximum of `2h` or `8 * bucket_span`.
  - `enabled` (boolean)
    Specifies whether the datafeed periodically checks for delayed data.
- `datafeed_id` (string Id)
- `frequency` (string Duration)
- `indices` ([] string)
- `job_id` (string Id)
- `indices_options` ({} IndicesOptions)
  - `allow_no_indices` (boolean)
    A setting that does two separate checks on the index expression.
If `false`, the request returns an error (1) if any wildcard expression
(including `_all` and `*`) resolves to zero matching indices or (2) if the
complete set of resolved indices, aliases or data streams is empty after all
expressions are evaluated. If `true`, index expressions that resolve to no
indices are allowed and the request returns an empty result.
  - `expand_wildcards` (string ExpandWildcards)
    Type of index that wildcard patterns can match. If the request can target data streams, this argument
determines whether wildcard expressions match hidden data streams. Supports comma-separated values,
such as `open,hidden`.

Supported values include:
  - `all`: Match any data stream or index, including hidden ones.
  - `open`: Match open, non-hidden indices. Also matches any non-hidden data stream.
  - `closed`: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
  - `hidden`: Match hidden data streams and hidden indices. Must be combined with `open`, `closed`, or `both`.
  - `none`: Wildcard expressions are not accepted.
    Values: `all`, `open`, `closed`, `hidden`, `none`
  - `ignore_unavailable` (boolean · default: false)
    If `false`, the request returns an error if it targets a concrete (non-wildcarded)
index, alias, or data stream that is missing, closed, or otherwise unavailable.
If `true`, unavailable concrete targets are silently ignored.
  - `ignore_throttled` (boolean · default: true)
    If true, concrete, expanded or aliased indices are ignored when frozen.
- `max_empty_searches` (number)
- `max_consecutive_extraction_failures` (number)
- `query` ({} QueryContainer)
  See [QueryContainer](https://www.elastic.co/elastic/docs-builder/docs/4371/api/doc/elasticsearch-serverless/types/_types-query_dsl-querycontainer)
- `query_delay` (string Duration)
- `runtime_mappings` (RuntimeFields)
- `script_fields` (map string to ScriptField)
  Map values ({} ScriptField)
  - `script` (string Script)
  - `ignore_failure` (boolean)
- `scroll_size` (number)
## Code Examples

#### Console

```console
PUT _ml/datafeeds/datafeed-test-job?pretty
{
  "indices": [
    "kibana_sample_data_logs"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        }
      ]
    }
  },
  "job_id": "test-job"
}
```

#### JavaScript

```javascript
const response = await client.ml.putDatafeed({
  datafeed_id: "datafeed-test-job",
  pretty: "true",
  indices: ["kibana_sample_data_logs"],
  query: {
    bool: {
      must: [
        {
          match_all: {},
        },
      ],
    },
  },
  job_id: "test-job",
});
```

#### Python

```python
resp = client.ml.put_datafeed(
    datafeed_id="datafeed-test-job",
    pretty=True,
    indices=[
        "kibana_sample_data_logs"
    ],
    query={
        "bool": {
            "must": [
                {
                    "match_all": {}
                }
            ]
        }
    },
    job_id="test-job",
)
```

#### curl

```curl
curl -X PUT "$ELASTICSEARCH_URL/_ml/datafeeds/datafeed-test-job?pretty" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "indices": [
      "kibana_sample_data_logs"
    ],
    "query": {
      "bool": {
        "must": [
          {
            "match_all": {}
          }
        ]
      }
    },
    "job_id": "test-job"
  }'
```

#### Java

```java
client.ml().putDatafeed(p -> p
    .datafeedId("datafeed-test-job")
    .indices("kibana_sample_data_logs")
    .jobId("test-job")
    .query(q -> q
        .bool(b -> b
            .must(m -> m
                .matchAll(ma -> ma)
            )
        )
    )
);
```

#### C#

```c#
var response = await client.MachineLearning
    .PutDatafeedAsync(datafeedId: "datafeed-test-job", d1 => d1
        .Indices(new[] { "kibana_sample_data_logs" })
        .JobId("test-job")
        .Query(d2 => d2
            .Bool(d3 => d3
                .Must(d4 => d4
                    .MatchAll()
                )
            )
        )
    );
```

#### PHP

```php
$resp = $client->ml()->putDatafeed([
    "datafeed_id" => "datafeed-test-job",
    "pretty" => "true",
    "body" => [
        "indices" => array(
            "kibana_sample_data_logs",
        ),
        "query" => [
            "bool" => [
                "must" => array(
                    [
                        "match_all" => new ArrayObject([]),
                    ],
                ),
            ],
        ],
        "job_id" => "test-job",
    ],
]);
```

#### Ruby

```ruby
response = client.ml.put_datafeed(
  datafeed_id: "datafeed-test-job",
  pretty: "true",
  body: {
    "indices": [
      "kibana_sample_data_logs"
    ],
    "query": {
      "bool": {
        "must": [
          {
            "match_all": {}
          }
        ]
      }
    },
    "job_id": "test-job"
  }
)
```