---
title: Delimited payload token filter
description: Separates a token stream into tokens and payloads based on a specified delimiter. For example, you can use the delimited_payload filter with a | delimiter...
url: https://www.elastic.co/elastic/docs-builder/docs/3028/reference/text-analysis/analysis-delimited-payload-tokenfilter
products:
  - Elasticsearch
---

# Delimited payload token filter
<warning>
  The older name `delimited_payload_filter` is deprecated and should not be used with new indices. Use `delimited_payload` instead.
</warning>

Separates a token stream into tokens and payloads based on a specified delimiter.
For example, you can use the `delimited_payload` filter with a `|` delimiter to split `the|1 quick|2 fox|3` into the tokens `the`, `quick`, and `fox` with respective payloads of `1`, `2`, and `3`.
This filter uses Lucene’s [DelimitedPayloadTokenFilter](https://lucene.apache.org/core/10_0_0/analysis/common/org/apache/lucene/analysis/payloads/DelimitedPayloadTokenFilter.md).
<admonition title="Payloads">
  A payload is user-defined binary data associated with a token position and stored as base64-encoded bytes.Elasticsearch does not store token payloads by default. To store payloads, you must:
  - Set the [`term_vector`](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/elasticsearch/mapping-reference/term-vector) mapping parameter to `with_positions_payloads` or `with_positions_offsets_payloads` for any field storing payloads.
  - Use an index analyzer that includes the `delimited_payload` filter
  You can view stored payloads using the [term vectors API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-termvectors).
</admonition>


## Example

The following [analyze API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-analyze) request uses the `delimited_payload` filter with the default `|` delimiter to split `the|0 brown|10 fox|5 is|0 quick|10` into tokens and payloads.
```json

{
  "tokenizer": "whitespace",
  "filter": ["delimited_payload"],
  "text": "the|0 brown|10 fox|5 is|0 quick|10"
}
```

The filter produces the following tokens:
```text
[ the, brown, fox, is, quick ]
```

Note that the analyze API does not return stored payloads. For an example that includes returned payloads, see [Return stored payloads](#analysis-delimited-payload-tokenfilter-return-stored-payloads).

## Add to an analyzer

The following [create index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create) request uses the `delimited-payload` filter to configure a new [custom analyzer](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/manage-data/data-store/text-analysis/create-custom-analyzer).
```json

{
  "settings": {
    "analysis": {
      "analyzer": {
        "whitespace_delimited_payload": {
          "tokenizer": "whitespace",
          "filter": [ "delimited_payload" ]
        }
      }
    }
  }
}
```


## Configurable parameters

<definitions>
  <definition term="delimiter">
    (Optional, string) Character used to separate tokens from payloads. Defaults to `|`.
  </definition>
  <definition term="encoding">
    (Optional, string) Data type for the stored payload. Valid values are:
  </definition>
  <definition term="float">
    (Default) Float
  </definition>
  <definition term="identity">
    Characters
  </definition>
  <definition term="int">
    Integer
  </definition>
</definitions>


## Customize and add to an analyzer

To customize the `delimited_payload` filter, duplicate it to create the basis for a new custom token filter. You can modify the filter using its configurable parameters.
For example, the following [create index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create) request uses a custom `delimited_payload` filter to configure a new [custom analyzer](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/manage-data/data-store/text-analysis/create-custom-analyzer). The custom `delimited_payload` filter uses the `+` delimiter to separate tokens from payloads. Payloads are encoded as integers.
```json

{
  "settings": {
    "analysis": {
      "analyzer": {
        "whitespace_plus_delimited": {
          "tokenizer": "whitespace",
          "filter": [ "plus_delimited" ]
        }
      },
      "filter": {
        "plus_delimited": {
          "type": "delimited_payload",
          "delimiter": "+",
          "encoding": "int"
        }
      }
    }
  }
}
```


## Return stored payloads

Use the [create index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create) to create an index that:
- Includes a field that stores term vectors with payloads.
- Uses a [custom index analyzer](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/manage-data/data-store/text-analysis/create-custom-analyzer) with the `delimited_payload` filter.

```json

{
  "mappings": {
    "properties": {
      "text": {
        "type": "text",
        "term_vector": "with_positions_payloads",
        "analyzer": "payload_delimiter"
      }
    }
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "payload_delimiter": {
          "tokenizer": "whitespace",
          "filter": [ "delimited_payload" ]
        }
      }
    }
  }
}
```

Add a document containing payloads to the index.
```json

{
  "text": "the|0 brown|3 fox|4 is|0 quick|10"
}
```

Use the [term vectors API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-termvectors) to return the document’s tokens and base64-encoded payloads.
```json

{
  "fields": [ "text" ],
  "payloads": true
}
```

The API returns the following response:
```json
{
  "_index": "text_payloads",
  "_id": "1",
  "_version": 1,
  "found": true,
  "took": 8,
  "term_vectors": {
    "text": {
      "field_statistics": {
        "sum_doc_freq": 5,
        "doc_count": 1,
        "sum_ttf": 5
      },
      "terms": {
        "brown": {
          "term_freq": 1,
          "tokens": [
            {
              "position": 1,
              "payload": "QEAAAA=="
            }
          ]
        },
        "fox": {
          "term_freq": 1,
          "tokens": [
            {
              "position": 2,
              "payload": "QIAAAA=="
            }
          ]
        },
        "is": {
          "term_freq": 1,
          "tokens": [
            {
              "position": 3,
              "payload": "AAAAAA=="
            }
          ]
        },
        "quick": {
          "term_freq": 1,
          "tokens": [
            {
              "position": 4,
              "payload": "QSAAAA=="
            }
          ]
        },
        "the": {
          "term_freq": 1,
          "tokens": [
            {
              "position": 0,
              "payload": "AAAAAA=="
            }
          ]
        }
      }
    }
  }
}
```