Pattern analyzer

The pattern analyzer uses a regular expression to split the text into terms. The regular expression should match the token separators not the tokens themselves. The regular expression defaults to \W+ (or all non-word characters).

 POST _analyze {
  "analyzer": "pattern",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

The above sentence would produce the following terms:

[ the, 2, quick, brown, foxes, jumped, over, the, lazy, dog, s, bone ]

The pattern analyzer accepts the following parameters:

pattern

A Java regular expression, defaults to \W+.

flags

Java regular expression flags. Flags should be pipe-separated, eg "CASE_INSENSITIVE|COMMENTS".

lowercase

Should terms be lowercased or not. Defaults to true.

stopwords

A pre-defined stop words list like _english_ or an array containing a list of stop words. Defaults to _none_.

stopwords_path

The path to a file containing stop words.

See the Stop Token Filter for more information about stop word configuration.

In this example, we configure the pattern analyzer to split email addresses on non-word characters or on underscores (\W|_), and to lower-case the result:

 PUT my-index-000001 {
  "settings": {
    "analysis": {
      "analyzer": {
        "my_email_analyzer": {
          "type":      "pattern",
          "pattern":   "\\W|_", 1
          "lowercase": true
        }
      }
    }
  }
}

POST my-index-000001/_analyze
{
  "analyzer": "my_email_analyzer",
  "text": "John_Smith@foo-bar.com"
}

1. The backslashes in the pattern need to be escaped when specifying the pattern as a JSON string.

The above example produces the following terms:

[ john, smith, foo, bar, com ]

The following more complicated example splits CamelCase text into tokens:

 PUT my-index-000001 {
  "settings": {
    "analysis": {
      "analyzer": {
        "camel": {
          "type": "pattern",
          "pattern": "([^\\p{L}\\d]+)|(?<=\\D)(?=\\d)|(?<=\\d)(?=\\D)|(?<=[\\p{L}&&[^\\p{Lu}]])(?=\\p{Lu})|(?<=\\p{Lu})(?=\\p{Lu}[\\p{L}&&[^\\p{Lu}]])"
        }
      }
    }
  }
}

GET my-index-000001/_analyze
{
  "analyzer": "camel",
  "text": "MooseX::FTPClass2_beta"
}

The above example produces the following terms:

[ moose, x, ftp, class, 2, beta ]

The regex above is easier to understand as:

  ([^\p{L}\d]+)                1
| (?<=\D)(?=\d)                2
| (?<=\d)(?=\D)                3
| (?<=[ \p{L} && [^\p{Lu}]])   4
  (?=\p{Lu})                   5
| (?<=\p{Lu})                  6
  (?=\p{Lu}                    7
    [\p{L}&&[^\p{Lu}]]         8
  )

1. swallow non letters and numbers,
2. or non-number followed by number,
3. or number followed by non-number,
4. or lower case
5. followed by upper case,
6. or upper case
7. followed by upper case
8. then lower case

The pattern analyzer consists of:

Tokenizer

• Pattern Tokenizer

Token Filters

• Lower Case Token Filter
• Stop Token Filter (disabled by default)

If you need to customize the pattern analyzer beyond the configuration parameters then you need to recreate it as a custom analyzer and modify it, usually by adding token filters. This would recreate the built-in pattern analyzer and you can use it as a starting point for further customization:

 PUT /pattern_example {
  "settings": {
    "analysis": {
      "tokenizer": {
        "split_on_non_word": {
          "type":       "pattern",
          "pattern":    "\\W+" 1
        }
      },
      "analyzer": {
        "rebuilt_pattern": {
          "tokenizer": "split_on_non_word",
          "filter": [
            "lowercase"       2
          ]
        }
      }
    }
  }
}

1. The default pattern is \W+ which splits on non-word characters and this is where you’d change it.
2. You’d add other token filters after lowercase.