Get tokens from text analysis

POST also GET /{index}/_analyze
POST also GET /_analyze

Auth

The analyze API performs analysis on a text string and returns the resulting tokens.

Generating excessive amount of tokens may cause a node to run out of memory. The index.analyze.max_token_count setting enables you to limit the number of tokens that can be produced. If more than this limit of tokens gets generated, an error occurs. The _analyze endpoint without a specified index will always use 10000 as its limit.

Learn more about text analysis

Required authorization

Index privileges index

Path parameters

index string optional

Index used to derive the analyzer. If specified, the analyzer or field parameter overrides this value. If no index is specified or the index does not have a default analyzer, the analyze API uses the standard analyzer.

Leave it out to use /_analyze.

Query parameters

index string: Index used to derive the analyzer. If specified, the analyzer or field parameter overrides this value. If no index is specified or the index does not have a default analyzer, the analyze API uses the standard analyzer.

Request body

analyzer string: The name of the analyzer that should be applied to the provided text. This could be a built-in analyzer, or an analyzer that’s been configured in the index.
attributes string[]: Array of token attributes used to filter the output of the explain parameter.
char_filter object[]: Array of character filters used to preprocess characters before the tokenizer.
explain boolean · default: false: If true, the response includes token attributes and additional details.
field string: Field used to derive the analyzer. To use this parameter, you must specify an index. If specified, the analyzer parameter overrides this value.
filter object[]: Array of token filters used to apply after the tokenizer.
normalizer string: Normalizer to use to convert text into a single token.
text union oneOf: Text to analyze. If an array of strings is provided, it is analyzed as a multi-value field.; One of: string[] or string
tokenizer object: Tokenizer to use to convert text into tokens.

Response

application/json

detail object
tokens object[]

Examples

Request Console 1 / 8 languages

You can apply any of the built-in analyzers to the text string without specifying an index.

Console Kibana Dev Tools

POST /_analyze
{
  "analyzer": "standard",
  "text": "this is a test"
}

JavaScript Elasticsearch JavaScript Client

const response = await client.indices.analyze({
  analyzer: "standard",
  text: "this is a test",
});

Python Elasticsearch Python Client

resp = client.indices.analyze(
    analyzer="standard",
    text="this is a test",
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_analyze" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "analyzer": "standard",
    "text": "this is a test"
  }'

Java Elasticsearch Java Client

client.indices().analyze(a -> a
    .analyzer("standard")
    .text("this is a test")
);

C# Elasticsearch .NET Client

var response = await client.Indices
    .AnalyzeAsync(d1 => d1
        .Analyzer("standard")
        .Text("this is a test")
    );

PHP Elasticsearch PHP Client

$resp = $client->indices()->analyze([
    "body" => [
        "analyzer" => "standard",
        "text" => "this is a test",
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.indices.analyze(
  body: {
    "analyzer": "standard",
    "text": "this is a test"
  }
)

Response

A successful response for an analysis with explain set to true.

{
  "detail": {
    "custom_analyzer": true,
    "charfilters": [],
    "tokenizer": {
      "name": "standard",
      "tokens": [
        {
          "token": "detailed",
          "start_offset": 0,
          "end_offset": 8,
          "type": "<ALPHANUM>",
          "position": 0
        },
        {
          "token": "output",
          "start_offset": 9,
          "end_offset": 15,
          "type": "<ALPHANUM>",
          "position": 1
        }
      ]
    },
    "tokenfilters": [
      {
        "name": "snowball",
        "tokens": [
          {
            "token": "detail",
            "start_offset": 0,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 0,
            "keyword": false
          },
          {
            "token": "output",
            "start_offset": 9,
            "end_offset": 15,
            "type": "<ALPHANUM>",
            "position": 1,
            "keyword": false
          }
        ]
      }
    ]
  }
}

Request Console 1 / 8 languages

If the text parameter is provided as array of strings, it is analyzed as a multi-value field.

Console Kibana Dev Tools

POST /_analyze
{
  "analyzer": "standard",
  "text": [
    "this is a test",
    "the second text"
  ]
}

JavaScript Elasticsearch JavaScript Client

const response = await client.indices.analyze({
  analyzer: "standard",
  text: ["this is a test", "the second text"],
});

Python Elasticsearch Python Client

resp = client.indices.analyze(
    analyzer="standard",
    text=[
        "this is a test",
        "the second text"
    ],
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_analyze" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "analyzer": "standard",
    "text": [
      "this is a test",
      "the second text"
    ]
  }'

Java Elasticsearch Java Client

client.indices().analyze(a -> a
    .analyzer("standard")
    .text(List.of("this is a test","the second text"))
);

C# Elasticsearch .NET Client

var response = await client.Indices
    .AnalyzeAsync(d1 => d1
        .Analyzer("standard")
        .Text("this is a test", "the second text")
    );

PHP Elasticsearch PHP Client

$resp = $client->indices()->analyze([
    "body" => [
        "analyzer" => "standard",
        "text" => array(
            "this is a test",
            "the second text",
        ),
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.indices.analyze(
  body: {
    "analyzer": "standard",
    "text": [
      "this is a test",
      "the second text"
    ]
  }
)

Response

A successful response for an analysis with explain set to true.

{
  "detail": {
    "custom_analyzer": true,
    "charfilters": [],
    "tokenizer": {
      "name": "standard",
      "tokens": [
        {
          "token": "detailed",
          "start_offset": 0,
          "end_offset": 8,
          "type": "<ALPHANUM>",
          "position": 0
        },
        {
          "token": "output",
          "start_offset": 9,
          "end_offset": 15,
          "type": "<ALPHANUM>",
          "position": 1
        }
      ]
    },
    "tokenfilters": [
      {
        "name": "snowball",
        "tokens": [
          {
            "token": "detail",
            "start_offset": 0,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 0,
            "keyword": false
          },
          {
            "token": "output",
            "start_offset": 9,
            "end_offset": 15,
            "type": "<ALPHANUM>",
            "position": 1,
            "keyword": false
          }
        ]
      }
    ]
  }
}

Request Console 1 / 7 languages

You can test a custom transient analyzer built from tokenizers, token filters, and char filters. Token filters use the filter parameter.

Console Kibana Dev Tools

POST /_analyze
{
  "tokenizer": "keyword",
  "filter": [
    "lowercase"
  ],
  "char_filter": [
    "html_strip"
  ],
  "text": "this is a <b>test</b>"
}

JavaScript Elasticsearch JavaScript Client

const response = await client.indices.analyze({
  tokenizer: "keyword",
  filter: ["lowercase"],
  char_filter: ["html_strip"],
  text: "this is a test</b>",
});

Python Elasticsearch Python Client

resp = client.indices.analyze(
    tokenizer="keyword",
    filter=[
        "lowercase"
    ],
    char_filter=[
        "html_strip"
    ],
    text="this is a test</b>",
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_analyze" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "tokenizer": "keyword",
    "filter": [
      "lowercase"
    ],
    "char_filter": [
      "html_strip"
    ],
    "text": "this is a test\u003C/b\u003E"
  }'

Java Elasticsearch Java Client

client.indices().analyze(a -> a
    .charFilter(c -> c
        .name("html_strip")
    )
    .filter(f -> f
        .name("lowercase")
    )
    .text("this is a test</b>")
    .tokenizer(t -> t
        .name("keyword")
    )
);

PHP Elasticsearch PHP Client

$resp = $client->indices()->analyze([
    "body" => [
        "tokenizer" => "keyword",
        "filter" => array(
            "lowercase",
        ),
        "char_filter" => array(
            "html_strip",
        ),
        "text" => "this is a test</b>",
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.indices.analyze(
  body: {
    "tokenizer": "keyword",
    "filter": [
      "lowercase"
    ],
    "char_filter": [
      "html_strip"
    ],
    "text": "this is a test</b>"
  }
)

Response

A successful response for an analysis with explain set to true.

{
  "detail": {
    "custom_analyzer": true,
    "charfilters": [],
    "tokenizer": {
      "name": "standard",
      "tokens": [
        {
          "token": "detailed",
          "start_offset": 0,
          "end_offset": 8,
          "type": "<ALPHANUM>",
          "position": 0
        },
        {
          "token": "output",
          "start_offset": 9,
          "end_offset": 15,
          "type": "<ALPHANUM>",
          "position": 1
        }
      ]
    },
    "tokenfilters": [
      {
        "name": "snowball",
        "tokens": [
          {
            "token": "detail",
            "start_offset": 0,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 0,
            "keyword": false
          },
          {
            "token": "output",
            "start_offset": 9,
            "end_offset": 15,
            "type": "<ALPHANUM>",
            "position": 1,
            "keyword": false
          }
        ]
      }
    ]
  }
}

Request Console 1 / 7 languages

Custom tokenizers, token filters, and character filters can be specified in the request body.

Console Kibana Dev Tools

POST /_analyze
{
  "tokenizer": "whitespace",
  "filter": [
    "lowercase",
    {
      "type": "stop",
      "stopwords": [
        "a",
        "is",
        "this"
      ]
    }
  ],
  "text": "this is a test"
}

JavaScript Elasticsearch JavaScript Client

const response = await client.indices.analyze({
  tokenizer: "whitespace",
  filter: [
    "lowercase",
    {
      type: "stop",
      stopwords: ["a", "is", "this"],
    },
  ],
  text: "this is a test",
});

Python Elasticsearch Python Client

resp = client.indices.analyze(
    tokenizer="whitespace",
    filter=[
        "lowercase",
        {
            "type": "stop",
            "stopwords": [
                "a",
                "is",
                "this"
            ]
        }
    ],
    text="this is a test",
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_analyze" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "tokenizer": "whitespace",
    "filter": [
      "lowercase",
      {
        "type": "stop",
        "stopwords": [
          "a",
          "is",
          "this"
        ]
      }
    ],
    "text": "this is a test"
  }'

Java Elasticsearch Java Client

client.indices().analyze(a -> a
    .filter(List.of(TokenFilter.of(t -> t
            .name("lowercase")
        ),TokenFilter.of(to -> to
            .definition(d -> d
                .stop(s -> s
                    .stopwords(List.of("a","is","this"))
                )
            )
        )))
    .text("this is a test")
    .tokenizer(tok -> tok
        .name("whitespace")
    )
);

PHP Elasticsearch PHP Client

$resp = $client->indices()->analyze([
    "body" => [
        "tokenizer" => "whitespace",
        "filter" => array(
            "lowercase",
            [
                "type" => "stop",
                "stopwords" => array(
                    "a",
                    "is",
                    "this",
                ),
            ],
        ),
        "text" => "this is a test",
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.indices.analyze(
  body: {
    "tokenizer": "whitespace",
    "filter": [
      "lowercase",
      {
        "type": "stop",
        "stopwords": [
          "a",
          "is",
          "this"
        ]
      }
    ],
    "text": "this is a test"
  }
)

Response

A successful response for an analysis with explain set to true.

{
  "detail": {
    "custom_analyzer": true,
    "charfilters": [],
    "tokenizer": {
      "name": "standard",
      "tokens": [
        {
          "token": "detailed",
          "start_offset": 0,
          "end_offset": 8,
          "type": "<ALPHANUM>",
          "position": 0
        },
        {
          "token": "output",
          "start_offset": 9,
          "end_offset": 15,
          "type": "<ALPHANUM>",
          "position": 1
        }
      ]
    },
    "tokenfilters": [
      {
        "name": "snowball",
        "tokens": [
          {
            "token": "detail",
            "start_offset": 0,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 0,
            "keyword": false
          },
          {
            "token": "output",
            "start_offset": 9,
            "end_offset": 15,
            "type": "<ALPHANUM>",
            "position": 1,
            "keyword": false
          }
        ]
      }
    ]
  }
}

Request Console 1 / 8 languages

Run POST /analyze_sample/_analyze to run an analysis on the text using the default index analyzer associated with the analyze_sample index. Alternatively, the analyzer can be derived based on a field mapping.

Console Kibana Dev Tools

POST /analyze_sample/_analyze
{
  "field": "obj1.field1",
  "text": "this is a test"
}

JavaScript Elasticsearch JavaScript Client

const response = await client.indices.analyze({
  index: "analyze_sample",
  field: "obj1.field1",
  text: "this is a test",
});

Python Elasticsearch Python Client

resp = client.indices.analyze(
    index="analyze_sample",
    field="obj1.field1",
    text="this is a test",
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/analyze_sample/_analyze" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "field": "obj1.field1",
    "text": "this is a test"
  }'

Java Elasticsearch Java Client

client.indices().analyze(a -> a
    .field("obj1.field1")
    .index("analyze_sample")
    .text("this is a test")
);

C# Elasticsearch .NET Client

var response = await client.Indices
    .AnalyzeAsync(d1 => d1
        .Index("analyze_sample")
        .Field(x => x.Obj1.Field1)
        .Text("this is a test")
    );

PHP Elasticsearch PHP Client

$resp = $client->indices()->analyze([
    "index" => "analyze_sample",
    "body" => [
        "field" => "obj1.field1",
        "text" => "this is a test",
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.indices.analyze(
  index: "analyze_sample",
  body: {
    "field": "obj1.field1",
    "text": "this is a test"
  }
)

Response

A successful response for an analysis with explain set to true.

{
  "detail": {
    "custom_analyzer": true,
    "charfilters": [],
    "tokenizer": {
      "name": "standard",
      "tokens": [
        {
          "token": "detailed",
          "start_offset": 0,
          "end_offset": 8,
          "type": "<ALPHANUM>",
          "position": 0
        },
        {
          "token": "output",
          "start_offset": 9,
          "end_offset": 15,
          "type": "<ALPHANUM>",
          "position": 1
        }
      ]
    },
    "tokenfilters": [
      {
        "name": "snowball",
        "tokens": [
          {
            "token": "detail",
            "start_offset": 0,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 0,
            "keyword": false
          },
          {
            "token": "output",
            "start_offset": 9,
            "end_offset": 15,
            "type": "<ALPHANUM>",
            "position": 1,
            "keyword": false
          }
        ]
      }
    ]
  }
}

Request Console 1 / 8 languages

Run POST /analyze_sample/_analyze and supply a normalizer for a keyword field if there is a normalizer associated with the specified index.

Console Kibana Dev Tools

POST /analyze_sample/_analyze
{
  "normalizer": "my_normalizer",
  "text": "BaR"
}

JavaScript Elasticsearch JavaScript Client

const response = await client.indices.analyze({
  index: "analyze_sample",
  normalizer: "my_normalizer",
  text: "BaR",
});

Python Elasticsearch Python Client

resp = client.indices.analyze(
    index="analyze_sample",
    normalizer="my_normalizer",
    text="BaR",
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/analyze_sample/_analyze" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "normalizer": "my_normalizer",
    "text": "BaR"
  }'

Java Elasticsearch Java Client

client.indices().analyze(a -> a
    .index("analyze_sample")
    .normalizer("my_normalizer")
    .text("BaR")
);

C# Elasticsearch .NET Client

var response = await client.Indices
    .AnalyzeAsync(d1 => d1
        .Index("analyze_sample")
        .Normalizer("my_normalizer")
        .Text("BaR")
    );

PHP Elasticsearch PHP Client

$resp = $client->indices()->analyze([
    "index" => "analyze_sample",
    "body" => [
        "normalizer" => "my_normalizer",
        "text" => "BaR",
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.indices.analyze(
  index: "analyze_sample",
  body: {
    "normalizer": "my_normalizer",
    "text": "BaR"
  }
)

Response

A successful response for an analysis with explain set to true.

{
  "detail": {
    "custom_analyzer": true,
    "charfilters": [],
    "tokenizer": {
      "name": "standard",
      "tokens": [
        {
          "token": "detailed",
          "start_offset": 0,
          "end_offset": 8,
          "type": "<ALPHANUM>",
          "position": 0
        },
        {
          "token": "output",
          "start_offset": 9,
          "end_offset": 15,
          "type": "<ALPHANUM>",
          "position": 1
        }
      ]
    },
    "tokenfilters": [
      {
        "name": "snowball",
        "tokens": [
          {
            "token": "detail",
            "start_offset": 0,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 0,
            "keyword": false
          },
          {
            "token": "output",
            "start_offset": 9,
            "end_offset": 15,
            "type": "<ALPHANUM>",
            "position": 1,
            "keyword": false
          }
        ]
      }
    ]
  }
}

Request Console 1 / 7 languages

If you want to get more advanced details, set explain to true. It will output all token attributes for each token. You can filter token attributes you want to output by setting the attributes option. NOTE: The format of the additional detail information is labelled as experimental in Lucene and it may change in the future.

Console Kibana Dev Tools

POST /_analyze
{
  "tokenizer": "standard",
  "filter": [
    "snowball"
  ],
  "text": "detailed output",
  "explain": true,
  "attributes": [
    "keyword"
  ]
}

JavaScript Elasticsearch JavaScript Client

const response = await client.indices.analyze({
  tokenizer: "standard",
  filter: ["snowball"],
  text: "detailed output",
  explain: true,
  attributes: ["keyword"],
});

Python Elasticsearch Python Client

resp = client.indices.analyze(
    tokenizer="standard",
    filter=[
        "snowball"
    ],
    text="detailed output",
    explain=True,
    attributes=[
        "keyword"
    ],
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_analyze" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "tokenizer": "standard",
    "filter": [
      "snowball"
    ],
    "text": "detailed output",
    "explain": true,
    "attributes": [
      "keyword"
    ]
  }'

Java Elasticsearch Java Client

client.indices().analyze(a -> a
    .attributes("keyword")
    .explain(true)
    .filter(f -> f
        .name("snowball")
    )
    .text("detailed output")
    .tokenizer(t -> t
        .name("standard")
    )
);

PHP Elasticsearch PHP Client

$resp = $client->indices()->analyze([
    "body" => [
        "tokenizer" => "standard",
        "filter" => array(
            "snowball",
        ),
        "text" => "detailed output",
        "explain" => true,
        "attributes" => array(
            "keyword",
        ),
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.indices.analyze(
  body: {
    "tokenizer": "standard",
    "filter": [
      "snowball"
    ],
    "text": "detailed output",
    "explain": true,
    "attributes": [
      "keyword"
    ]
  }
)

Response

A successful response for an analysis with explain set to true.

{
  "detail": {
    "custom_analyzer": true,
    "charfilters": [],
    "tokenizer": {
      "name": "standard",
      "tokens": [
        {
          "token": "detailed",
          "start_offset": 0,
          "end_offset": 8,
          "type": "<ALPHANUM>",
          "position": 0
        },
        {
          "token": "output",
          "start_offset": 9,
          "end_offset": 15,
          "type": "<ALPHANUM>",
          "position": 1
        }
      ]
    },
    "tokenfilters": [
      {
        "name": "snowball",
        "tokens": [
          {
            "token": "detail",
            "start_offset": 0,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 0,
            "keyword": false
          },
          {
            "token": "output",
            "start_offset": 9,
            "end_offset": 15,
            "type": "<ALPHANUM>",
            "position": 1,
            "keyword": false
          }
        ]
      }
    ]
  }
}