---
title: Unique token filter
description: Removes duplicate tokens from a stream. For example, you can use the unique filter to change the lazy lazy dog to the lazy dog. If the only_on_same_position...
url: https://www.elastic.co/elastic/docs-builder/docs/3028/reference/text-analysis/analysis-unique-tokenfilter
products:
  - Elasticsearch
---

# Unique token filter
Removes duplicate tokens from a stream. For example, you can use the `unique` filter to change `the lazy lazy dog` to `the lazy dog`.
If the `only_on_same_position` parameter is set to `true`, the `unique` filter removes only duplicate tokens *in the same position*.
<note>
  When `only_on_same_position` is `true`, the `unique` filter works the same as [`remove_duplicates`](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/text-analysis/analysis-remove-duplicates-tokenfilter) filter.
</note>


## Example

The following [analyze API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-analyze) request uses the `unique` filter to remove duplicate tokens from `the quick fox jumps the lazy fox`:
```json

{
  "tokenizer" : "whitespace",
  "filter" : ["unique"],
  "text" : "the quick fox jumps the lazy fox"
}
```

The filter removes duplicated tokens for `the` and `fox`, producing the following output:
```text
[ the, quick, fox, jumps, lazy ]
```


## Add to an analyzer

The following [create index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create) request uses the `unique` filter to configure a new [custom analyzer](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/manage-data/data-store/text-analysis/create-custom-analyzer).
```json

{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "standard_truncate" : {
        "tokenizer" : "standard",
        "filter" : ["unique"]
        }
      }
    }
  }
}
```


## Configurable parameters

<definitions>
  <definition term="only_on_same_position">
    (Optional, Boolean) If `true`, only remove duplicate tokens in the same position. Defaults to `false`.
  </definition>
</definitions>


## Customize

To customize the `unique` filter, duplicate it to create the basis for a new custom token filter. You can modify the filter using its configurable parameters.
For example, the following request creates a custom `unique` filter with `only_on_same_position` set to `true`.
```json

{
  "settings": {
    "analysis": {
      "analyzer": {
        "letter_unique_pos": {
          "tokenizer": "letter",
          "filter": [ "unique_pos" ]
        }
      },
      "filter": {
        "unique_pos": {
          "type": "unique",
          "only_on_same_position": true
        }
      }
    }
  }
}
```