Update a transform

Auth

Updates certain properties of a transform.

All updated properties except description do not take effect until after the transform starts the next checkpoint, thus there is data consistency in each checkpoint. To use this API, you must have read and view_index_metadata privileges for the source indices. You must also have index and read privileges for the destination index. When Elasticsearch security features are enabled, the transform remembers which roles the user who updated it had at the time of update and runs with those privileges.

Required authorization

Index privileges read index view_index_metadata
Cluster privileges manage_transform

Path parameters

transform_id string required

Identifier for the transform.

Query parameters

defer_validation boolean

When true, deferrable validations are not run. This behavior may be desired if the source index does not exist until after the transform is created.

timeout string · default: 30s

Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

Request body

dest object

The destination for the transform.

index string

The destination index for the transform. The mappings of the destination index are deduced based on the source fields when possible. If alternate mappings are required, use the create index API prior to starting the transform.

aliases object[]

The aliases that the destination index for the transform should have. Aliases are manipulated using the stored credentials of the transform, which means the secondary credentials supplied at creation time (if both primary and secondary credentials are specified).

The destination index is added to the aliases regardless of whether the destination index was created by the transform or pre-created by the user.

alias string required

The name of the alias.

move_on_creation boolean · default: false

Whether the destination index should be the only index in this alias. If true, all the other indices will be removed from this alias before adding the destination index to this alias. This does not delete the removed indices; it only removes them from the alias.

pipeline string

The unique identifier for an ingest pipeline.

description string

Free text description of the transform.

frequency string · default: 1m

The interval between checks for changes in the source indices when the transform is running continuously. Also determines the retry interval in the event of transient failures while the transform is searching or indexing. The minimum value is 1s and the maximum is 1h.

_meta object

Defines optional transform metadata.

source object

The source of the data for the transform.

index union oneOf required

The source indices for the transform. It can be a single index, an index pattern (for example, "my-index-*""), an array of indices (for example, ["my-index-000001", "my-index-000002"]), or an array of index patterns (for example, ["my-index-*", "my-other-index-*"]. For remote indices use the syntax "remote_name:index_name". If any indices are in remote clusters then the master node and at least one transform node must have the remote_cluster_client node role.

One of: string[] or string
runtime_mappings object

Definitions of search-time runtime fields that can be used by the transform. For search runtime fields all data nodes, including remote nodes, must be 7.12 or later.

query object

A query clause that retrieves a subset of data from the source index.

Query DSL
settings object

Defines optional transform settings.

align_checkpoints boolean · default: true

Specifies whether the transform checkpoint ranges should be optimized for performance. Such optimization can align checkpoint ranges with the date histogram interval when date histogram is specified as a group source in the transform config. As a result, less document updates in the destination index will be performed thus improving overall performance.

dates_as_epoch_millis boolean · default: false

Defines if dates in the output should be written as ISO formatted string or as millis since epoch. epoch_millis was the default for transforms created before version 7.11. For compatible output set this value to true.

deduce_mappings boolean · default: true

Specifies whether the transform should deduce the destination index mappings from the transform configuration.

docs_per_second number

Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.

max_page_search_size number

Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 65,536. The default value is 500 for pivot transforms and 5000 for latest transforms.

use_point_in_time boolean · default: true

Specifies whether the transform checkpoint will use the Point In Time API while searching over the source index. In general, Point In Time is an optimization that will reduce pressure on the source index by reducing the amount of refreshes and merges, but it can be expensive if a large number of Point In Times are opened and closed for a given index. The benefits and impact depend on the data being searched, the ingest rate into the source index, and the amount of other consumers searching the same source index.

Reference
num_failure_retries number

Defines the number of retries on a recoverable failure before the transform task is marked as failed. The minimum value is 0 and the maximum is 100, where -1 indicates that the transform retries indefinitely. If unset, the cluster-level setting num_transform_failure_retries is used.

This setting cannot be specified when unattended is true, because unattended transforms always retry indefinitely.

unattended boolean · default: false

If true, the transform runs in unattended mode. In unattended mode, the transform retries indefinitely in case of an error which means the transform never fails. Setting the number of retries other than infinite fails in validation.

indexer_request_timeout union string nullable

The initial timeout for the requests the transform sends to nodes to retrieve checkpoint information. Each consecutive failure doubles the timeout, up to a maximum of 12h. The minimum value is 1ms and the maximum is 12h. Defaults to 30s.

sync object

Defines the properties transforms require to run continuously.

time object

Specifies that the transform uses a time field to synchronize the source and destination indices.

delay string · default: 60s

The time delay between the current time and the latest input data time.

field string required

The date field that is used to identify new documents in the source. In general, it’s a good idea to use a field that contains the ingest timestamp. If you use a different field, you might need to set the delay such that it accounts for data transmission delays.

retention_policy union oneOf nullable

Defines a retention policy for the transform. Data that meets the defined criteria is deleted from the destination index.

One of:
time object

Specifies that the transform uses a time field to set the retention policy.

field string required

The date field that is used to calculate the age of the document.

max_age string required

Specifies the maximum age of a document in the destination index. Documents that are older than the configured value are removed from the destination index.

_force_rekeying boolean

When true, force reminting of the transform's internal cloud API key from the caller's cloud credential without requiring other configuration changes. Requires a cloud-authenticated caller and an environment that supports cross-project calls. Rejected with 400 otherwise.

Response

application/json
authorization object
api_key object

If an API key was used for the most recent update to the transform, its name and identifier are listed in the response.

id string required

The identifier for the API key.

name string required

The name of the API key.

roles string[]

If a user ID was used for the most recent update to the transform, its roles at the time of the update are listed in the response.

service_account string

If a service account was used for the most recent update to the transform, the account name is listed in the response.

create_time number required
description string
dest object required
index string

The destination index for the transform. The mappings of the destination index are deduced based on the source fields when possible. If alternate mappings are required, use the create index API prior to starting the transform.

aliases object[]

The aliases that the destination index for the transform should have. Aliases are manipulated using the stored credentials of the transform, which means the secondary credentials supplied at creation time (if both primary and secondary credentials are specified).

The destination index is added to the aliases regardless of whether the destination index was created by the transform or pre-created by the user.

alias string required

The name of the alias.

move_on_creation boolean · default: false

Whether the destination index should be the only index in this alias. If true, all the other indices will be removed from this alias before adding the destination index to this alias. This does not delete the removed indices; it only removes them from the alias.

pipeline string

The unique identifier for an ingest pipeline.

frequency string
id string required
latest object
sort string required

Specifies the date field that is used to identify the latest documents.

unique_key string[] required

Specifies an array of one or more fields that are used to group the data.

pivot object
aggregations map string to {} Aggregation

Defines how to aggregate the grouped data. The following aggregations are currently supported: average, bucket script, bucket selector, cardinality, filter, geo bounds, geo centroid, geo line, max, median absolute deviation, min, missing, percentiles, rare terms, scripted metric, stats, sum, terms, top metrics, value count, weighted average.

group_by map string to object

Defines how to group the data. More than one grouping can be defined per pivot. The following groupings are currently supported: date histogram, geotile grid, histogram, terms.

<string> object · min properties: 1 · max properties: 1
date_histogram object
calendar_interval enum

Calendar-aware interval. Can be specified using the unit name, such as month, or as a single unit quantity, such as 1M.

Values: second 1s minute 1m hour 1h day 1d week 1w month 1M quarter 1q year 1y
extended_bounds object

Enables extending the bounds of the histogram beyond the data itself.

max union oneOf

Maximum value for the bound.

One of: string or number
min union oneOf

Minimum value for the bound.

One of: string or number
hard_bounds object

Limits the histogram to specified bounds.

max union oneOf

Maximum value for the bound.

One of: string or number
min union oneOf

Minimum value for the bound.

One of: string or number
field string

The date field whose values are use to build a histogram.

fixed_interval string

Fixed intervals: a fixed number of SI units and never deviate, regardless of where they fall on the calendar.

format string

The date format used to format key_as_string in the response. If no format is specified, the first date format specified in the field mapping is used.

interval string deprecated
min_doc_count number

Only returns buckets that have min_doc_count number of documents. By default, all buckets between the first bucket that matches documents and the last one are returned.

missing union oneOf

The value to apply to documents that do not have a value. By default, documents without a value are ignored.

One of: string or number
offset string

Changes the start value of each bucket by the specified positive (+) or negative offset (-) duration.

order union oneOf

The sort order of the returned buckets.

One of: string to enum[] or string to enum
params map string to object
script string
time_zone string

Time zone used for bucketing and rounding. Defaults to Coordinated Universal Time (UTC).

keyed boolean

Set to true to associate a unique string key with each bucket and return the ranges as a hash rather than an array.

geotile_grid object
field string

Field containing indexed geo_point or geo_shape values. If the field contains an array, geotile_grid aggregates all array values.

precision number · default: 7

Integer zoom of the key used to define cells/buckets in the results. Values outside of the range [0,29] will be rejected.

shard_size number

Allows for more accurate counting of the top cells returned in the final result the aggregation. Defaults to returning max(10,(size x number-of-shards)) buckets from each shard.

size number · default: 10000

The maximum number of buckets to return.

bounds union oneOf

A bounding box to filter the geo-points or geo-shapes in each bucket.

One of:
top number required
bottom number required
left number required
right number required
histogram object
extended_bounds object

Enables extending the bounds of the histogram beyond the data itself.

max number

Maximum value for the bound.

min number

Minimum value for the bound.

hard_bounds object

Limits the range of buckets in the histogram. It is particularly useful in the case of open data ranges that can result in a very large number of buckets.

max number

Maximum value for the bound.

min number

Minimum value for the bound.

field string

The name of the field to aggregate on.

interval number

The interval for the buckets. Must be a positive decimal.

min_doc_count number

Only returns buckets that have min_doc_count number of documents. By default, the response will fill gaps in the histogram with empty buckets.

missing number

The value to apply to documents that do not have a value. By default, documents without a value are ignored.

offset number

By default, the bucket keys start with 0 and then continue in even spaced steps of interval. The bucket boundaries can be shifted by using the offset option.

order union oneOf

The sort order of the returned buckets. By default, the returned buckets are sorted by their key ascending.

One of: string to enum[] or string to enum
script string
format string
keyed boolean · default: false

If true, returns buckets as a hash instead of an array, keyed by the bucket keys.

terms object
collect_mode enum

Determines how child aggregations should be calculated: breadth-first or depth-first.

Supported values include:

  • depth_first: Expands all branches of the aggregation tree in one depth-first pass, before any pruning occurs.
  • breadth_first: Caches the set of documents that fall into the uppermost buckets for subsequent replay.
Values: depth_first breadth_first
exclude union oneOf

Values to exclude. Accepts regular expressions and partitions.

One of: string[] or string
execution_hint enum

Determines whether the aggregation will use field values directly or global ordinals.

Values: map global_ordinals global_ordinals_hash global_ordinals_low_cardinality
field string

The field from which to return terms.

include union oneOf

Values to include. Accepts regular expressions and partitions.

One of:
string[]
min_doc_count number · default: 1

Only return values that are found in more than min_doc_count hits.

missing boolean | number | string

The value to apply to documents that do not have a value. By default, documents without a value are ignored.

missing_order enum
Values: first last default
missing_bucket boolean
value_type string

Coerced unmapped fields into the specified type.

order union oneOf

Specifies the sort order of the buckets. Defaults to sorting by descending document count.

One of: string to enum[] or string to enum
script string
shard_min_doc_count number

Regulates the certainty a shard has if the term should actually be added to the candidate list or not with respect to the min_doc_count. Terms will only be considered if their local shard frequency within the set is higher than the shard_min_doc_count.

shard_size number

The number of candidate terms produced by each shard. By default, shard_size will be automatically estimated based on the number of shards and the size parameter.

show_term_doc_count_error boolean

Set to true to return the doc_count_error_upper_bound, which is an upper bound to the error on the doc_count returned by each shard.

size number · default: 10

The number of buckets returned out of the overall terms list.

format string
retention_policy object
time object

Specifies that the transform uses a time field to set the retention policy.

field string required

The date field that is used to calculate the age of the document.

max_age string required

Specifies the maximum age of a document in the destination index. Documents that are older than the configured value are removed from the destination index.

settings object required
align_checkpoints boolean · default: true

Specifies whether the transform checkpoint ranges should be optimized for performance. Such optimization can align checkpoint ranges with the date histogram interval when date histogram is specified as a group source in the transform config. As a result, less document updates in the destination index will be performed thus improving overall performance.

dates_as_epoch_millis boolean · default: false

Defines if dates in the output should be written as ISO formatted string or as millis since epoch. epoch_millis was the default for transforms created before version 7.11. For compatible output set this value to true.

deduce_mappings boolean · default: true

Specifies whether the transform should deduce the destination index mappings from the transform configuration.

docs_per_second number

Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.

max_page_search_size number

Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 65,536. The default value is 500 for pivot transforms and 5000 for latest transforms.

use_point_in_time boolean · default: true

Specifies whether the transform checkpoint will use the Point In Time API while searching over the source index. In general, Point In Time is an optimization that will reduce pressure on the source index by reducing the amount of refreshes and merges, but it can be expensive if a large number of Point In Times are opened and closed for a given index. The benefits and impact depend on the data being searched, the ingest rate into the source index, and the amount of other consumers searching the same source index.

Reference
num_failure_retries number

Defines the number of retries on a recoverable failure before the transform task is marked as failed. The minimum value is 0 and the maximum is 100, where -1 indicates that the transform retries indefinitely. If unset, the cluster-level setting num_transform_failure_retries is used.

This setting cannot be specified when unattended is true, because unattended transforms always retry indefinitely.

unattended boolean · default: false

If true, the transform runs in unattended mode. In unattended mode, the transform retries indefinitely in case of an error which means the transform never fails. Setting the number of retries other than infinite fails in validation.

indexer_request_timeout union string nullable

The initial timeout for the requests the transform sends to nodes to retrieve checkpoint information. Each consecutive failure doubles the timeout, up to a maximum of 12h. The minimum value is 1ms and the maximum is 12h. Defaults to 30s.

source object required
index union oneOf required

The name of the data stream, index, or alias you are copying from. It accepts a comma-separated list to reindex from multiple sources.

One of: string[] or string
query {} Query

The documents to reindex, which is defined with Query DSL.

remote object

A remote instance of Elasticsearch that you want to index from.

connect_timeout string · default: 30s

The remote connection timeout.

headers map string to string

An object containing the headers of the request.

host string required

The URL for the remote instance of Elasticsearch that you want to index from. This information is required when you're indexing from remote.

username string

The username to use for authentication with the remote host (required when using basic auth).

password string

The password to use for authentication with the remote host (required when using basic auth).

api_key string

The API key to use for authentication with the remote host (as an alternative to basic auth when the remote cluster is in Elastic Cloud). (It is not permitted to set this and also to set an Authorization header via headers.)

socket_timeout string · default: 30s

The remote socket read timeout.

size number · default: 1000

The number of documents to index per batch. Use it when you are indexing from remote to ensure that the batches fit within the on-heap buffer, which defaults to a maximum size of 100 MB.

slice object

Slice the reindex request manually using the provided slice ID and total number of slices.

field string
id string required
max number required
sort union oneOf deprecated

A comma-separated list of <field>:<direction> pairs to sort by before indexing. Use it in conjunction with max_docs to control what documents are reindexed.

Warning

Sort in reindex is deprecated. Sorting in reindex was never guaranteed to index documents in order and prevents further development of reindex such as resilience and performance improvements. If used in combination with max_docs, consider using a query filter instead.

A value or an array; each item is one of:

Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

_source union oneOf · default: true

If true, reindex all source fields. Set it to a list to reindex select fields.

One of:
runtime_mappings object
sync object
time object

Specifies that the transform uses a time field to synchronize the source and destination indices.

delay string · default: 60s

The time delay between the current time and the latest input data time.

field string required

The date field that is used to identify new documents in the source. In general, it’s a good idea to use a field that contains the ingest timestamp. If you use a different field, you might need to set the delay such that it accounts for data transmission delays.

version string required
_meta object