---
title: Elasticsearch ingest pipelines
description: Elasticsearch ingest pipelines let you perform common transformations on your data before indexing. For example, you can use pipelines to remove fields,...
url: https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/ingest/transform-enrich/ingest-pipelines
products:
  - Elasticsearch
applies_to:
  - Elastic Cloud Serverless: Generally available
  - Elastic Stack: Generally available
---

# Elasticsearch ingest pipelines
Elasticsearch ingest pipelines let you perform common transformations on your data before indexing. For example, you can use pipelines to remove fields, extract values from text, and enrich your data.
A pipeline consists of a series of configurable tasks called [processors](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/enrich-processor). Each processor runs sequentially, making specific changes to incoming documents. After the processors have run, Elasticsearch adds the transformed documents to your data stream or index.
![Ingest pipeline diagram](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/images/elasticsearch-reference-ingest-process.svg)

You can create and manage ingest pipelines using Kibana's **Ingest Pipelines** feature or the [ingest APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ingest). Elasticsearch stores pipelines in the [cluster state](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-state).

## Prerequisites

- Nodes with the [`ingest`](/elastic/docs-builder/docs/3028/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles#node-ingest-node) node role handle pipeline processing. To use ingest pipelines, your cluster must have at least one node with the `ingest` role. For heavy ingest loads, we recommend creating [dedicated ingest nodes](/elastic/docs-builder/docs/3028/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles#node-ingest-node).
- If the Elasticsearch security features are enabled, you must have the `manage_pipeline` [cluster privilege](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/security-privileges#privileges-list-cluster) to manage ingest pipelines. To use Kibana's **Ingest Pipelines** feature, you also need the `cluster:monitor/nodes/info` cluster privileges.
- Pipelines including the `enrich` processor require additional setup. See [*Enrich your data*](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/ingest/transform-enrich/data-enrichment).


## Create and manage pipelines

In Kibana, go to the **Ingest Pipelines** management page using the navigation menu or the [global search field](https://www.elastic.co/elastic/docs-builder/docs/3028/explore-analyze/find-and-organize/find-apps-and-objects).
From the list view, you can:
- View a list of your pipelines and drill down into details
- Edit or clone existing pipelines
- Delete pipelines

![Kibana's Ingest Pipelines list view](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/images/elasticsearch-reference-ingest-pipeline-list.png)

To create a pipeline, click **Create pipeline > New pipeline**. For an example tutorial, see [Example: Parse logs](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/ingest/transform-enrich/example-parse-logs).
<tip>
  The **New pipeline from CSV** option lets you use a CSV to create an ingest pipeline that maps custom data to the [Elastic Common Schema (ECS)](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/ecs). Mapping your custom data to ECS makes the data easier to search and lets you reuse visualizations from other datasets. To get started, check [Map custom data to ECS](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/ecs/ecs-converting).
</tip>

You can also use the [ingest APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ingest) to create and manage pipelines. The following [create pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) request creates a pipeline containing two [`set`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/enrich-processor/set-processor) processors followed by a [`lowercase`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/enrich-processor/lowercase-processor) processor. The processors run sequentially in the order specified.
```json

{
  "description": "My optional pipeline description",
  "processors": [
    {
      "set": {
        "description": "My optional processor description",
        "field": "my-long-field",
        "value": 10
      }
    },
    {
      "set": {
        "description": "Set 'my-boolean-field' to true",
        "field": "my-boolean-field",
        "value": true
      }
    },
    {
      "lowercase": {
        "field": "my-keyword-field"
      }
    }
  ]
}
```


## Manage pipeline versions

When you create or update a pipeline, you can specify an optional `version` integer. You can use this version number with the [`if_version`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline#put-pipeline-api-query-params) parameter to conditionally update the pipeline. When the `if_version` parameter is specified, a successful update increments the pipeline’s version.
```json

{
  "version": 1,
  "processors": [ ... ]
}
```

To unset the `version` number using the API, replace or update the pipeline without specifying the `version` parameter.

## Test a pipeline

Before using a pipeline in production, we recommend you test it using sample documents. When creating or editing a pipeline in Kibana, click **Add documents**. In the **Documents** tab, provide sample documents and click **Run the pipeline**.
![Test a pipeline in Kibana](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/images/elasticsearch-reference-test-a-pipeline.png)

You can also test pipelines using the [simulate pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-simulate). You can specify a configured pipeline in the request path. For example, the following request tests `my-pipeline`.
```json

{
  "docs": [
    {
      "_source": {
        "my-keyword-field": "FOO"
      }
    },
    {
      "_source": {
        "my-keyword-field": "BAR"
      }
    }
  ]
}
```

Alternatively, you can specify a pipeline and its processors in the request body.
```json

{
  "pipeline": {
    "processors": [
      {
        "lowercase": {
          "field": "my-keyword-field"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "my-keyword-field": "FOO"
      }
    },
    {
      "_source": {
        "my-keyword-field": "BAR"
      }
    }
  ]
}
```

The API returns transformed documents:
```json
{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "my-keyword-field": "foo"
        },
        "_ingest": {
          "timestamp": "2099-03-07T11:04:03.000Z"
        }
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "my-keyword-field": "bar"
        },
        "_ingest": {
          "timestamp": "2099-03-07T11:04:04.000Z"
        }
      }
    }
  ]
}
```


## Add a pipeline to an indexing request

Use the `pipeline` query parameter to apply a pipeline to documents in [individual](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create) or [bulk](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) indexing requests.
```json

{
  "@timestamp": "2099-03-07T11:04:05.000Z",
  "my-keyword-field": "foo"
}


{ "create":{ } }
{ "@timestamp": "2099-03-07T11:04:06.000Z", "my-keyword-field": "foo" }
{ "create":{ } }
{ "@timestamp": "2099-03-07T11:04:07.000Z", "my-keyword-field": "bar" }
```

You can also use the `pipeline` parameter with the [update by query](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update-by-query) or [reindex](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) APIs.
```json



{
  "source": {
    "index": "my-data-stream"
  },
  "dest": {
    "index": "my-new-data-stream",
    "op_type": "create",
    "pipeline": "my-pipeline"
  }
}
```


## Set a default pipeline

Use the [`index.default_pipeline`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/index-settings/index-modules#index-default-pipeline) index setting to set a default pipeline. Elasticsearch applies this pipeline to indexing requests if no `pipeline` parameter is specified.

## Set a final pipeline

Use the [`index.final_pipeline`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/index-settings/index-modules#index-final-pipeline) index setting to set a final pipeline. Elasticsearch applies this pipeline after the request or default pipeline, even if neither is specified.

## Pipelines for Beats

To add an ingest pipeline to an Elastic Beat, specify the `pipeline` parameter under `output.elasticsearch` in `<BEAT_NAME>.yml`. For example, for Filebeat, you’d specify `pipeline` in `filebeat.yml`.
```yaml
output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: my-pipeline
```


## Pipelines for Fleet and Elastic Agent

Elastic Agent integrations ship with default ingest pipelines that preprocess and enrich data before indexing. [Fleet](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/fleet) applies these pipelines using [index templates](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/data-store/templates) that include [pipeline index settings](#set-default-pipeline). Elasticsearch matches these templates to your Fleet data streams based on the [stream’s naming scheme](/elastic/docs-builder/docs/3028/reference/fleet/data-streams#data-streams-naming-scheme).
Each default integration pipeline calls a nonexistent, unversioned `*@custom` ingest pipeline. If unaltered, this pipeline call has no effect on your data. However, you can modify this call to create custom pipelines for integrations that persist across upgrades. Refer to [Tutorial: Transform data with custom ingest pipelines](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/fleet/data-streams-pipeline-tutorial) to learn more.
Fleet doesn’t provide a default ingest pipeline for the **Custom logs** integration, but you can specify a pipeline for this integration using an [index template](#pipeline-custom-logs-index-template) or a [custom configuration](#pipeline-custom-logs-configuration).

**Option 1: Index template**
1. [Create](#create-manage-ingest-pipelines) and [test](#test-pipeline) your ingest pipeline. Name your pipeline `logs-<dataset-name>-default`. This makes tracking the pipeline for your integration easier.
   For example, the following request creates a pipeline for the `my-app` dataset. The pipeline’s name is `logs-my_app-default`.
   ```json

   {
     "description": "Pipeline for `my_app` dataset",
     "processors": [ ... ]
   }
   ```
2. Create an [index template](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/data-store/templates) that includes your pipeline in the [`index.default_pipeline`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/index-settings/index-modules#index-default-pipeline) or [`index.final_pipeline`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/index-settings/index-modules#index-final-pipeline) index setting. Ensure the template is [data stream enabled](/elastic/docs-builder/docs/3028/manage-data/data-store/data-streams/set-up-data-stream#create-index-template). The template’s index pattern should match `logs-<dataset-name>-*`.
   You can create this template using Kibana's [**Index Management**](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/data-store/templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template).
   For example, the following request creates a template matching `logs-my_app-*`. The template uses a component template that contains the `index.default_pipeline` index setting.
   ```json
   # Creates a component template for index settings

   {
     "template": {
       "settings": {
         "index.default_pipeline": "logs-my_app-default",
         "index.lifecycle.name": "logs"
       }
     }
   }

   # Creates an index template matching `logs-my_app-*`

   {
     "index_patterns": ["logs-my_app-*"],
     "data_stream": { },
     "priority": 500,
     "composed_of": ["logs-my_app-settings", "logs-my_app-mappings"]
   }
   ```
3. When adding or editing your **Custom logs** integration in Fleet, click **Configure integration > Custom log file > Advanced options**.
4. In **Dataset name**, specify your dataset’s name. Fleet will add new data for the integration to the resulting `logs-<dataset-name>-default` data stream.
   For example, if your dataset’s name was `my_app`, Fleet adds new data to the `logs-my_app-default` data stream.
   ![Set up custom log integration in Fleet](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/images/elasticsearch-reference-custom-logs.png)
5. Use the [rollover API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover) to roll over your data stream. This ensures Elasticsearch applies the index template and its pipeline settings to any new data for the integration.
   ```json
   ```


**Option 2: Custom configuration**
1. [Create](#create-manage-ingest-pipelines) and [test](#test-pipeline) your ingest pipeline. Name your pipeline `logs-<dataset-name>-default`. This makes tracking the pipeline for your integration easier.
   For example, the following request creates a pipeline for the `my-app` dataset. The pipeline’s name is `logs-my_app-default`.
   ```json

   {
     "description": "Pipeline for `my_app` dataset",
     "processors": [ ... ]
   }
   ```
2. When adding or editing your **Custom logs** integration in Fleet, click **Configure integration > Custom log file > Advanced options**.
3. In **Dataset name**, specify your dataset’s name. Fleet will add new data for the integration to the resulting `logs-<dataset-name>-default` data stream.
   For example, if your dataset’s name was `my_app`, Fleet adds new data to the `logs-my_app-default` data stream.
4. In **Custom Configurations**, specify your pipeline in the `pipeline` policy setting.
   ![Custom pipeline configuration for custom log integration](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/images/elasticsearch-reference-custom-logs-pipeline.png)

**Elastic Agent standalone**
If you run Elastic Agent standalone, you can apply pipelines using an [index template](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/data-store/templates) that includes the [`index.default_pipeline`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/index-settings/index-modules#index-default-pipeline) or [`index.final_pipeline`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/index-settings/index-modules#index-final-pipeline) index setting. Alternatively, you can specify the `pipeline` policy setting in your `elastic-agent.yml` configuration. See [Install standalone Elastic Agents](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/fleet/install-standalone-elastic-agent).

## Pipelines for search indices

When you create Elasticsearch indices for search use cases, for example, using the [web crawler^](https://www.elastic.co/guide/en/enterprise-search/current/crawler.html) or [connectors](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/search-connectors), these indices are automatically set up with specific ingest pipelines. These processors help optimize your content for search. See [*Ingest pipelines in Search*](https://www.elastic.co/elastic/docs-builder/docs/3028/solutions/search/ingest-for-search) for more information.

## Access source fields in a processor

Processors have read and write access to an incoming document’s source fields. To access a field key in a processor, use its field name. The following `set` processor accesses `my-long-field`.
```json

{
  "processors": [
    {
      "set": {
        "field": "my-long-field",
        "value": 10
      }
    }
  ]
}
```

You can also prepend the `_source` prefix.
```json

{
  "processors": [
    {
      "set": {
        "field": "_source.my-long-field",
        "value": 10
      }
    }
  ]
}
```

Use dot notation to access object fields.
<important>
  If your document contains flattened objects, use the [`dot_expander`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/enrich-processor/dot-expand-processor) processor to expand them. If you wish to maintain your document structure, use the [`flexible`](#access-source-pattern-flexible) access pattern in your pipeline definition. Otherwise Ingest processors cannot access dotted field names.
</important>

```json

{
  "processors": [
    {
      "dot_expander": {
        "description": "Expand 'my-object-field.my-property'",
        "field": "my-object-field.my-property"
      }
    },
    {
      "set": {
        "description": "Set 'my-object-field.my-property' to 10",
        "field": "my-object-field.my-property",
        "value": 10
      }
    }
  ]
}
```


Several processor parameters support [Mustache](https://mustache.github.io) template snippets. To access field values in a template snippet, enclose the field name in triple curly brackets:`{{{field-name}}}`. You can use template snippets to dynamically set field names.
```json

{
  "processors": [
    {
      "set": {
        "description": "Set dynamic '<service>' field to 'code' value",
        "field": "{{{service}}}",
        "value": "{{{code}}}"
      }
    }
  ]
}
```


## Ingest field access pattern

<applies-to>
  - Elastic Cloud Serverless: Generally available
  - Elastic Stack: Generally available since 9.2
</applies-to>

The default ingest pipeline access pattern does not recognize dotted field names in documents. Retrieving flattened and dotted field names from an ingest document requires a different field retrieval algorithm that does not have this limitation. We know that some pipelines have come to rely on these dotted field name limitations in their logic. In order to continue supporting the original behavior while still adding support for dotted field names, ingest pipelines now support configuring an access pattern to use for all processors in the pipeline.
The `field_access_pattern` property on an ingest pipeline defines how ingest document fields are read and written for all processors in the current pipeline. It accepts two values: `classic` (which is the default) and `flexible`.
```json

{
  "field_access_pattern": "classic", <1>
  "processors": [
    {
      "set": {
        "description": "Set some searchable tags in our document's flattened field",
        "field": "event.tags.ingest.processed_by", <2>
        "value": "my-pipeline"
      }
    }
  ]
}
```


### Classic field access pattern

The `classic` access pattern is the default access pattern that has been around since ingest node first released. Field paths given to processors (for example, `event.tags.ingest.processed_by`) are split on the dot character (`.`). The processor then uses the resulting field names to traverse the document until a value is found. When writing a value to a document, if its parent fields do not exist in the source, the processor will create nested objects for the missing fields.
```json

{
  "pipeline" : {
    "description": "example pipeline",
    "field_access_pattern": "classic", <1>
    "processors": [
      {
        "set" : {
          "description" : "Copy the foo.bar field into the a.b.c.d field if it exists",
          "copy_from" : "foo.bar", <2>
          "field" : "a.b.c.d", <3>
          "ignore_empty_value": true
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": {
          "bar": "baz" <4>
        }
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo.bar": "baz" <5>
      }
    }
  ]
}
```

```json
{
   "docs": [
      {
         "doc": {
            "_id": "id",
            "_index": "index",
            "_version": "-3",
            "_source": {
              "foo": {
                "bar": "baz" 
              },
              "a": {
                "b": {
                  "c": {
                    "d": "baz" 
                  }
                }
              }
            },
            "_ingest": {
               "timestamp": "2017-05-04T22:30:03.187Z"
            }
         }
      },
      {
         "doc": {
            "_id": "id",
            "_index": "index",
            "_version": "-3",
            "_source": {
               "foo.bar": "baz" 
            },
            "_ingest": {
               "timestamp": "2017-05-04T22:30:03.188Z"
            }
         }
      }
   ]
}
```

If the documents you are ingesting contain dotted field names, to read them with the `classic` access pattern, you must use the [`dot_expander`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/enrich-processor/dot-expand-processor) processor. This approach is not always reasonable though. Consider the following document:
```json
{
  "event": {
    "tags": {
      "http.host": "localhost:9200",
      "http.host.name": "localhost",
      "http.host.port": 9200
    }
  }
}
```

If the `event.tags` field was processed with the [`dot_expander`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/enrich-processor/dot-expand-processor) processor, the field values would collide. The `http.host` field cannot be a text value and an object value at the same time.

### Flexible field access pattern

The `flexible` access pattern allows for ingest pipelines to access both nested and dotted field names without using the [`dot_expander`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/enrich-processor/dot-expand-processor) processor. Additionally, when writing a value to a field that does not exist, any parent fields that are missing are concatenated to the start of the new key. Use the `flexible` access pattern if your documents have dotted field names, and also if you prefer to write missing fields to the document with dotted names.
```json

{
  "pipeline" : {
    "description": "example pipeline",
    "field_access_pattern": "flexible", <1>
    "processors": [
      {
        "set" : {
          "description" : "Copy the foo.bar field into the a.b.c.d field if it exists",
          "copy_from" : "foo.bar", <2>
          "field" : "a.b.c.d", <3>
          "ignore_empty_value": true
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": {
          "bar": "baz" <4>
        },
        "a": {} <5>
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo.bar": "baz", <6>
      }
    }
  ]
}
```

```json
{
   "docs": [
      {
         "doc": {
            "_id": "id",
            "_index": "index",
            "_version": "-3",
            "_source": {
              "foo": {
                "bar": "baz" 
              },
              "a": {
                "b.c.d": "baz" 
              }
            },
            "_ingest": {
               "timestamp": "2017-05-04T22:30:03.187Z"
            }
         }
      },
      {
         "doc": {
            "_id": "id",
            "_index": "index",
            "_version": "-3",
            "_source": {
               "foo.bar": "baz", 
               "a.b.c.d": "baz" 
            },
            "_ingest": {
               "timestamp": "2017-05-04T22:30:03.188Z"
            }
         }
      }
   ]
}
```


## Access metadata fields in a processor

Processors can access the following metadata fields by name:
- `_index`
- `_id`
- `_routing`
- `_dynamic_templates`

```json

{
  "processors": [
    {
      "set": {
        "description": "Set '_routing' to 'geoip.country_iso_code' value",
        "field": "_routing",
        "value": "{{{geoip.country_iso_code}}}"
      }
    }
  ]
}
```

Use a Mustache template snippet to access metadata field values. For example, `{{{_routing}}}` retrieves a document’s routing value.
```json

{
  "processors": [
    {
      "set": {
        "description": "Use geo_point dynamic template for address field",
        "field": "_dynamic_templates",
        "value": {
          "address": "geo_point"
        }
      }
    }
  ]
}
```

The set processor above tells ES to use the dynamic template named `geo_point` for the field `address` if this field is not defined in the mapping of the index yet. This processor overrides the dynamic template for the field `address` if already defined in the bulk request, but has no effect on other dynamic templates defined in the bulk request.
<warning>
  If you [automatically generate](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create#create-document-ids-automatically) document IDs, you cannot use `{{{_id}}}` in a processor. Elasticsearch assigns auto-generated `_id` values after ingest.
</warning>


## Access ingest metadata in a processor

Ingest processors can add and access ingest metadata using the `_ingest` key.
Unlike source and metadata fields, Elasticsearch does not index ingest metadata fields by default. Elasticsearch also allows source fields that start with an `_ingest` key. If your data includes such source fields, use `_source._ingest` to access them.
Pipelines only create the `_ingest.timestamp` ingest metadata field by default. This field contains a timestamp of when Elasticsearch received the document’s indexing request. To index `_ingest.timestamp` or other ingest metadata fields, use the `set` processor.
```json

{
  "processors": [
    {
      "set": {
        "description": "Index the ingest timestamp as 'event.ingested'",
        "field": "event.ingested",
        "value": "{{{_ingest.timestamp}}}"
      }
    }
  ]
}
```


## Handling pipeline failures

A pipeline’s processors run sequentially. By default, pipeline processing stops when one of these processors fails or encounters an error.
To ignore a processor failure and run the pipeline’s remaining processors, set `ignore_failure` to `true`.
```json

{
  "processors": [
    {
      "rename": {
        "description": "Rename 'provider' to 'cloud.provider'",
        "field": "provider",
        "target_field": "cloud.provider",
        "ignore_failure": true
      }
    }
  ]
}
```

Use the `on_failure` parameter to specify a list of processors to run immediately after a processor failure. If `on_failure` is specified, Elasticsearch afterward runs the pipeline’s remaining processors, even if the `on_failure` configuration is empty.
```json

{
  "processors": [
    {
      "rename": {
        "description": "Rename 'provider' to 'cloud.provider'",
        "field": "provider",
        "target_field": "cloud.provider",
        "on_failure": [
          {
            "set": {
              "description": "Set 'error.message'",
              "field": "error.message",
              "value": "Field 'provider' does not exist. Cannot rename to 'cloud.provider'",
              "override": false
            }
          }
        ]
      }
    }
  ]
}
```

Nest a list of `on_failure` processors for nested error handling.
```json

{
  "processors": [
    {
      "rename": {
        "description": "Rename 'provider' to 'cloud.provider'",
        "field": "provider",
        "target_field": "cloud.provider",
        "on_failure": [
          {
            "set": {
              "description": "Set 'error.message'",
              "field": "error.message",
              "value": "Field 'provider' does not exist. Cannot rename to 'cloud.provider'",
              "override": false,
              "on_failure": [
                {
                  "set": {
                    "description": "Set 'error.message.multi'",
                    "field": "error.message.multi",
                    "value": "Document encountered multiple ingest errors",
                    "override": true
                  }
                }
              ]
            }
          }
        ]
      }
    }
  ]
}
```

You can also specify `on_failure` for a pipeline. If a processor without an `on_failure` value fails, Elasticsearch uses this pipeline-level parameter as a fallback. Elasticsearch will not attempt to run the pipeline’s remaining processors.
```json

{
  "processors": [ ... ],
  "on_failure": [
    {
      "set": {
        "description": "Index document to 'failed-<index>'",
        "field": "_index",
        "value": "failed-{{{ _index }}}"
      }
    }
  ]
}
```

Additional information about the pipeline failure may be available in the document metadata fields `on_failure_message`, `on_failure_processor_type`, `on_failure_processor_tag`, and `on_failure_pipeline`. These fields are accessible only from within an `on_failure` block.
The following example uses the metadata fields to include information about pipeline failures in documents.
```json

{
  "processors": [ ... ],
  "on_failure": [
    {
      "set": {
        "description": "Record error information",
        "field": "error_information",
        "value": "Processor {{ _ingest.on_failure_processor_type }} with tag {{ _ingest.on_failure_processor_tag }} in pipeline {{ _ingest.on_failure_pipeline }} failed with message {{ _ingest.on_failure_message }}"
      }
    }
  ]
}
```


## Conditionally run a processor

Each processor supports an optional `if` condition, written as a [Painless script](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/scripting-languages/painless/painless). If provided, the processor only runs when the `if` condition is `true`.
<important>
  `if` condition scripts run in Painless’s [ingest processor context](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/scripting-languages/painless/painless-ingest-processor-context). In `if` conditions, `ctx` values are read-only.
</important>

```json

{
  "processors": [
    {
      "drop": {
        "description": "Drop documents with 'network.name' of 'Guest'",
        "if": "ctx?.network?.name == 'Guest'"
      }
    }
  ]
}
```

If the [`script.painless.regex.enabled`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/circuit-breaker-settings#script-painless-regex-enabled) cluster setting is enabled, you can use regular expressions in your `if` condition scripts. For supported syntax, see [Painless regular expressions](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/scripting-languages/painless/painless-regexes).
<tip>
  If possible, avoid using regular expressions. Expensive regular expressions can slow indexing speeds.
</tip>

```json

{
  "processors": [
    {
      "set": {
        "description": "If 'url.scheme' is 'http', set 'url.insecure' to true",
        "if": "ctx.url?.scheme =~ /^http[^s]/",
        "field": "url.insecure",
        "value": true
      }
    }
  ]
}
```

You must specify `if` conditions as valid JSON on a single line. However, you can use the [Kibana console](/elastic/docs-builder/docs/3028/explore-analyze/query-filter/tools/console#configuring-console)'s triple quote syntax to write and debug larger scripts.
<tip>
  If possible, avoid using complex or expensive `if` condition scripts. Expensive condition scripts can slow indexing speeds.
</tip>

```json

{
  "processors": [
    {
      "drop": {
        "description": "Drop documents that don't contain 'prod' tag",
        "if": """
            Collection tags = ctx.tags;
            if(tags != null){
              for (String tag : tags) {
                if (tag.toLowerCase().contains('prod')) {
                  return false;
                }
              }
            }
            return true;
        """
      }
    }
  ]
}
```

You can also specify a [stored script](https://www.elastic.co/elastic/docs-builder/docs/3028/explore-analyze/scripting/modules-scripting-store-and-retrieve) as the `if` condition.
```json

{
  "script": {
    "lang": "painless",
    "source": """
      Collection tags = ctx.tags;
      if(tags != null){
        for (String tag : tags) {
          if (tag.toLowerCase().contains('prod')) {
            return false;
          }
        }
      }
      return true;
    """
  }
}


{
  "processors": [
    {
      "drop": {
        "description": "Drop documents that don't contain 'prod' tag",
        "if": { "id": "my-prod-tag-script" }
      }
    }
  ]
}
```

Incoming documents often contain object fields. If a processor script attempts to access a field whose parent object does not exist, Elasticsearch returns a `NullPointerException`. To avoid these exceptions, use [null safe operators](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/scripting-languages/painless/painless-operators-reference#null-safe-operator), such as `?.`, and write your scripts to be null safe.
For example, `ctx.network?.name.equalsIgnoreCase('Guest')` is not null safe. `ctx.network?.name` can return null. Rewrite the script as `'Guest'.equalsIgnoreCase(ctx.network?.name)`, which is null safe because `Guest` is always non-null.
If you can’t rewrite a script to be null safe, include an explicit null check.
```json

{
  "processors": [
    {
      "drop": {
        "description": "Drop documents that contain 'network.name' of 'Guest'",
        "if": "ctx.network?.name != null && ctx.network.name.contains('Guest')"
      }
    }
  ]
}
```


## Conditionally apply pipelines

Combine an `if` condition with the [`pipeline`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/enrich-processor/pipeline-processor) processor to apply other pipelines to documents based on your criteria. You can use this pipeline as the [default pipeline](#set-default-pipeline) in an [index template](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/data-store/templates) used to configure multiple data streams or indices.
```json

{
  "processors": [
    {
      "pipeline": {
        "description": "If 'service.name' is 'apache_httpd', use 'httpd_pipeline'",
        "if": "ctx.service?.name == 'apache_httpd'",
        "name": "httpd_pipeline"
      }
    },
    {
      "pipeline": {
        "description": "If 'service.name' is 'syslog', use 'syslog_pipeline'",
        "if": "ctx.service?.name == 'syslog'",
        "name": "syslog_pipeline"
      }
    },
    {
      "fail": {
        "description": "If 'service.name' is not 'apache_httpd' or 'syslog', return a failure message",
        "if": "ctx.service?.name != 'apache_httpd' && ctx.service?.name != 'syslog'",
        "message": "This pipeline requires service.name to be either `syslog` or `apache_httpd`"
      }
    }
  ]
}
```


## Get pipeline usage statistics

Use the [node stats](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-stats) API to get global and per-pipeline ingest statistics. Use these stats to determine which pipelines run most frequently or spend the most time processing.
```json
```