Create a transform

Auth

Creates a transform.

A transform copies data from source indices, transforms it, and persists it into an entity-centric destination index. You can also think of the destination index as a two-dimensional tabular data structure (known as a data frame). The ID for each document in the data frame is generated from a hash of the entity, so there is a unique row per entity.

You must choose either the latest or pivot method for your transform; you cannot use both in a single transform. If you choose to use the pivot method for your transform, the entities are defined by the set of group_by fields in the pivot object. If you choose to use the latest method, the entities are defined by the unique_key field values in the latest object.

You must have create_index, index, and read privileges on the destination index and read and view_index_metadata privileges on the source indices. When Elasticsearch security features are enabled, the transform remembers which roles the user that created it had at the time of creation and uses those same roles. If those roles do not have the required privileges on the source and destination indices, the transform fails when it attempts unauthorized operations.

Note

You must use Kibana or this API to create a transform. Do not add a transform directly into any .transform-internal* indices using the Elasticsearch index API. If Elasticsearch security features are enabled, do not give users any privileges on .transform-internal* indices. If you used transforms prior to 7.5, also do not give users any privileges on .data-frame-internal* indices.

Required privileges

Index privileges create_index read index view_index_metadata
Cluster privileges manage_transform

Path parameters

transform_id string required

Identifier for the transform. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It has a 64 character limit and must start and end with alphanumeric characters.

Query parameters

defer_validation boolean · default: false

When the transform is created, a series of validations occur to ensure its success. For example, there is a check for the existence of the source indices and a check that the destination index is not part of the source index pattern. You can use this parameter to skip the checks, for example when the source index does not exist until after the transform is created. The validations are always run when you start the transform, however, with the exception of privilege checks.

timeout string · default: 30s

Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

Request body

dest object required

The destination for the transform.

index string

The destination index for the transform. The mappings of the destination index are deduced based on the source fields when possible. If alternate mappings are required, use the create index API prior to starting the transform.

aliases object[]

The aliases that the destination index for the transform should have. Aliases are manipulated using the stored credentials of the transform, which means the secondary credentials supplied at creation time (if both primary and secondary credentials are specified).

The destination index is added to the aliases regardless of whether the destination index was created by the transform or pre-created by the user.

alias string required

The name of the alias.

move_on_creation boolean · default: false

Whether the destination index should be the only index in this alias. If true, all the other indices will be removed from this alias before adding the destination index to this alias. This does not delete the removed indices; it only removes them from the alias.

pipeline string

The unique identifier for an ingest pipeline.

description string

Free text description of the transform.

frequency string · default: 1m

The interval between checks for changes in the source indices when the transform is running continuously. Also determines the retry interval in the event of transient failures while the transform is searching or indexing. The minimum value is 1s and the maximum is 1h.

latest object

The latest method transforms the data by finding the latest document for each unique key.

sort string required

Specifies the date field that is used to identify the latest documents.

unique_key string[] required

Specifies an array of one or more fields that are used to group the data.

_meta object

Defines optional transform metadata.

pivot object

The pivot method transforms the data by aggregating and grouping it. These objects define the group by fields and the aggregation to reduce the data.

aggregations map string to {} Aggregation

Defines how to aggregate the grouped data. The following aggregations are currently supported: average, bucket script, bucket selector, cardinality, filter, geo bounds, geo centroid, geo line, max, median absolute deviation, min, missing, percentiles, rare terms, scripted metric, stats, sum, terms, top metrics, value count, weighted average.

group_by map string to object

Defines how to group the data. More than one grouping can be defined per pivot. The following groupings are currently supported: date histogram, geotile grid, histogram, terms.

<string> object · min properties: 1 · max properties: 1
date_histogram object
calendar_interval enum

Calendar-aware interval. Can be specified using the unit name, such as month, or as a single unit quantity, such as 1M.

Values: second 1s minute 1m hour 1h day 1d week 1w month 1M quarter 1q year 1y
extended_bounds object

Enables extending the bounds of the histogram beyond the data itself.

max union oneOf

Maximum value for the bound.

One of: string or number
min union oneOf

Minimum value for the bound.

One of: string or number
hard_bounds object

Limits the histogram to specified bounds.

max union oneOf

Maximum value for the bound.

One of: string or number
min union oneOf

Minimum value for the bound.

One of: string or number
field string

The date field whose values are use to build a histogram.

fixed_interval string

Fixed intervals: a fixed number of SI units and never deviate, regardless of where they fall on the calendar.

format string

The date format used to format key_as_string in the response. If no format is specified, the first date format specified in the field mapping is used.

interval string deprecated
min_doc_count number

Only returns buckets that have min_doc_count number of documents. By default, all buckets between the first bucket that matches documents and the last one are returned.

missing union oneOf

The value to apply to documents that do not have a value. By default, documents without a value are ignored.

One of: string or number
offset string

Changes the start value of each bucket by the specified positive (+) or negative offset (-) duration.

order union oneOf

The sort order of the returned buckets.

One of: string to enum[] or string to enum
params map string to object
script string
time_zone string

Time zone used for bucketing and rounding. Defaults to Coordinated Universal Time (UTC).

keyed boolean

Set to true to associate a unique string key with each bucket and return the ranges as a hash rather than an array.

geotile_grid object
field string

Field containing indexed geo_point or geo_shape values. If the field contains an array, geotile_grid aggregates all array values.

precision number · default: 7

Integer zoom of the key used to define cells/buckets in the results. Values outside of the range [0,29] will be rejected.

shard_size number

Allows for more accurate counting of the top cells returned in the final result the aggregation. Defaults to returning max(10,(size x number-of-shards)) buckets from each shard.

size number · default: 10000

The maximum number of buckets to return.

bounds union oneOf

A bounding box to filter the geo-points or geo-shapes in each bucket.

One of:
top number required
bottom number required
left number required
right number required
histogram object
extended_bounds object

Enables extending the bounds of the histogram beyond the data itself.

max number

Maximum value for the bound.

min number

Minimum value for the bound.

hard_bounds object

Limits the range of buckets in the histogram. It is particularly useful in the case of open data ranges that can result in a very large number of buckets.

max number

Maximum value for the bound.

min number

Minimum value for the bound.

field string

The name of the field to aggregate on.

interval number

The interval for the buckets. Must be a positive decimal.

min_doc_count number

Only returns buckets that have min_doc_count number of documents. By default, the response will fill gaps in the histogram with empty buckets.

missing number

The value to apply to documents that do not have a value. By default, documents without a value are ignored.

offset number

By default, the bucket keys start with 0 and then continue in even spaced steps of interval. The bucket boundaries can be shifted by using the offset option.

order union oneOf

The sort order of the returned buckets. By default, the returned buckets are sorted by their key ascending.

One of: string to enum[] or string to enum
script string
format string
keyed boolean · default: false

If true, returns buckets as a hash instead of an array, keyed by the bucket keys.

terms object
collect_mode enum

Determines how child aggregations should be calculated: breadth-first or depth-first.

Supported values include:

  • depth_first: Expands all branches of the aggregation tree in one depth-first pass, before any pruning occurs.
  • breadth_first: Caches the set of documents that fall into the uppermost buckets for subsequent replay.
Values: depth_first breadth_first
exclude union oneOf

Values to exclude. Accepts regular expressions and partitions.

One of: string[] or string
execution_hint enum

Determines whether the aggregation will use field values directly or global ordinals.

Values: map global_ordinals global_ordinals_hash global_ordinals_low_cardinality
field string

The field from which to return terms.

include union oneOf

Values to include. Accepts regular expressions and partitions.

One of:
string[]
min_doc_count number · default: 1

Only return values that are found in more than min_doc_count hits.

missing boolean | number | string

The value to apply to documents that do not have a value. By default, documents without a value are ignored.

missing_order enum
Values: first last default
missing_bucket boolean
value_type string

Coerced unmapped fields into the specified type.

order union oneOf

Specifies the sort order of the buckets. Defaults to sorting by descending document count.

One of: string to enum[] or string to enum
script string
shard_min_doc_count number

Regulates the certainty a shard has if the term should actually be added to the candidate list or not with respect to the min_doc_count. Terms will only be considered if their local shard frequency within the set is higher than the shard_min_doc_count.

shard_size number

The number of candidate terms produced by each shard. By default, shard_size will be automatically estimated based on the number of shards and the size parameter.

show_term_doc_count_error boolean

Set to true to return the doc_count_error_upper_bound, which is an upper bound to the error on the doc_count returned by each shard.

size number · default: 10

The number of buckets returned out of the overall terms list.

format string
retention_policy object

Defines a retention policy for the transform. Data that meets the defined criteria is deleted from the destination index.

time object

Specifies that the transform uses a time field to set the retention policy.

field string required

The date field that is used to calculate the age of the document.

max_age string required

Specifies the maximum age of a document in the destination index. Documents that are older than the configured value are removed from the destination index.

settings object

Defines optional transform settings.

align_checkpoints boolean · default: true

Specifies whether the transform checkpoint ranges should be optimized for performance. Such optimization can align checkpoint ranges with the date histogram interval when date histogram is specified as a group source in the transform config. As a result, less document updates in the destination index will be performed thus improving overall performance.

dates_as_epoch_millis boolean · default: false

Defines if dates in the output should be written as ISO formatted string or as millis since epoch. epoch_millis was the default for transforms created before version 7.11. For compatible output set this value to true.

deduce_mappings boolean · default: true

Specifies whether the transform should deduce the destination index mappings from the transform configuration.

docs_per_second number

Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.

max_page_search_size number

Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 65,536. The default value is 500 for pivot transforms and 5000 for latest transforms.

use_point_in_time boolean · default: true

Specifies whether the transform checkpoint will use the Point In Time API while searching over the source index. In general, Point In Time is an optimization that will reduce pressure on the source index by reducing the amount of refreshes and merges, but it can be expensive if a large number of Point In Times are opened and closed for a given index. The benefits and impact depend on the data being searched, the ingest rate into the source index, and the amount of other consumers searching the same source index.

Reference
num_failure_retries number

Defines the number of retries on a recoverable failure before the transform task is marked as failed. The minimum value is 0 and the maximum is 100, where -1 indicates that the transform retries indefinitely. If unset, the cluster-level setting num_transform_failure_retries is used.

This setting cannot be specified when unattended is true, because unattended transforms always retry indefinitely.

unattended boolean · default: false

If true, the transform runs in unattended mode. In unattended mode, the transform retries indefinitely in case of an error which means the transform never fails. Setting the number of retries other than infinite fails in validation.

indexer_request_timeout union string nullable

The initial timeout for the requests the transform sends to nodes to retrieve checkpoint information. Each consecutive failure doubles the timeout, up to a maximum of 12h. The minimum value is 1ms and the maximum is 12h. Defaults to 30s.

source object required

The source of the data for the transform.

index union oneOf required

The source indices for the transform. It can be a single index, an index pattern (for example, "my-index-*""), an array of indices (for example, ["my-index-000001", "my-index-000002"]), or an array of index patterns (for example, ["my-index-*", "my-other-index-*"]. For remote indices use the syntax "remote_name:index_name". If any indices are in remote clusters then the master node and at least one transform node must have the remote_cluster_client node role.

One of: string[] or string
runtime_mappings object

Definitions of search-time runtime fields that can be used by the transform. For search runtime fields all data nodes, including remote nodes, must be 7.12 or later.

query object

A query clause that retrieves a subset of data from the source index.

Query DSL
sync object

Defines the properties transforms require to run continuously.

time object

Specifies that the transform uses a time field to synchronize the source and destination indices.

delay string · default: 60s

The time delay between the current time and the latest input data time.

field string required

The date field that is used to identify new documents in the source. In general, it’s a good idea to use a field that contains the ingest timestamp. If you use a different field, you might need to set the delay such that it accounts for data transmission delays.

Response

application/json
acknowledged boolean required

For a successful response, this value is always true. On failure, an exception is returned instead.