Loading

Attachment processor

The attachment processor lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by using the Apache text extraction library Tika.

The source field must be a base64 encoded binary. If you do not want to incur the overhead of converting back and forth between base64, you can use the CBOR format instead of JSON and specify the field as a bytes array instead of a string representation. The processor will skip the base64 decoding then.

To limit the size of the raw source value, use the processor option max_field_bytes or the node setting ingest.attachment.max_field_size. For more details, refer to Limit raw attachment field size.

Name Required Default Description
field yes - The field to get the base64 encoded field from
target_field no attachment The field that will hold the attachment information
indexed_chars no 100000 The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.
indexed_chars_field no null Field name from which you can overwrite the number of chars being used for extraction. See indexed_chars.
max_field_bytes no -1 Maximum allowed size of the attachment field value in bytes: length of a string (if base64 in JSON, checked before base64 decoding) or byte[] length for binary (for example, CBOR). If set to -1, there is no per-processor limit. The node setting ingest.attachment.max_field_size also applies; refer to Limit raw attachment field size.
properties no all properties Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language
ignore_missing no false If true and field does not exist, the processor quietly exits without modifying the document
remove_binary encouraged false If true, the binary field will be removed from the document. This option is not required, but setting it explicitly is encouraged, and omitting it will result in a warning.
resource_name no Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

If attaching files to JSON documents, you must first encode the file as a base64 string. On Unix-like systems, you can do this using a base64 command:

base64 -in myfile.rtf

The command returns the base64-encoded string for the file. The following base64 string is for an .rtf file containing the text Lorem ipsum dolor sit amet: e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=.

Use an attachment processor to decode the string and extract the file’s properties:

				PUT _ingest/pipeline/attachment
					{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data",
        "remove_binary": true
      }
    }
  ]
}
				PUT my-index-000001/_doc/my_id?pipeline=attachment
					{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
}
				GET my-index-000001/_doc/my_id

The document’s attachment object contains extracted properties for the file:

{
  "found": true,
  "_index": "my-index-000001",
  "_id": "my_id",
  "_version": 1,
  "_seq_no": 22,
  "_primary_term": 1,
  "_source": {
    "attachment": {
      "content_type": "application/rtf",
      "language": "ro",
      "content": "Lorem ipsum dolor sit amet",
      "content_length": 28
    }
  }
}

In some cases, attachment sizes can result in significant memory overhead, as can any further processing (for example, by Apache Tika). To restrict the raw size of the attachment source field, use the node setting ingest.attachment.max_field_size or the processor parameter max_field_bytes. Limits are enforced early in the process (for example, before base64 decoding) to avoid allocating or decoding oversized payloads. If a document exceeds the limit, use an on_failure handler to address the resulting ingest failure.

The node setting defaults to -1, which means no node-wide limit. You can set either an absolute size using byte-size syntax (for example, 10mb, 1gb), or a relative value as a percentage or ratio of the JVM heap for the node (for example, 5% or 0.05).

Unlike the node setting, max_field_bytes is an attachment processor parameter that accepts only integer values. The default is -1, meaning no per-processor limit, but the node setting still applies.

The fields which might be extracted from a document are:

  • content,
  • title,
  • author,
  • keywords,
  • date,
  • content_type,
  • content_length,
  • language,
  • modified,
  • format,
  • identifier,
  • contributor,
  • coverage,
  • modifier,
  • creator_tool,
  • publisher,
  • relation,
  • rights,
  • source,
  • type,
  • description,
  • print_date,
  • metadata_date,
  • latitude,
  • longitude,
  • altitude,
  • rating,
  • comments

To extract only certain attachment fields, specify the properties array:

				PUT _ingest/pipeline/attachment
					{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data",
        "properties": [ "content", "title" ],
        "remove_binary": true
      }
    }
  ]
}
Note

Extracting contents from binary data is a resource intensive operation and consumes a lot of resources. It is highly recommended to run pipelines using this processor in a dedicated ingest node.

Keeping the binary as a field within the document might consume a lot of resources. It is highly recommended to remove that field from the document, by setting remove_binary to true to automatically remove the field, as in the other examples shown on this page. If you do want to keep the binary field, explicitly set remove_binary to false to avoid the warning you get from omitting it:

				PUT _ingest/pipeline/attachment
					{
  "description" : "Extract attachment information including original binary",
  "processors" : [
    {
      "attachment" : {
        "field" : "data",
        "remove_binary": false
      }
    }
  ]
}
				PUT my-index-000001/_doc/my_id?pipeline=attachment
					{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
}
				GET my-index-000001/_doc/my_id

The document’s _source object includes the original binary field:

{
  "found": true,
  "_index": "my-index-000001",
  "_id": "my_id",
  "_version": 1,
  "_seq_no": 22,
  "_primary_term": 1,
  "_source": {
    "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
    "attachment": {
      "content_type": "application/rtf",
      "language": "ro",
      "content": "Lorem ipsum dolor sit amet",
      "content_length": 28
    }
  }
}

To avoid encoding and decoding JSON to base64, you can instead pass CBOR data to the attachment processor. For example, the following request creates the cbor-attachment pipeline, which uses the attachment processor.

				PUT _ingest/pipeline/cbor-attachment
					{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data",
        "remove_binary": true
      }
    }
  ]
}

The following Python script passes CBOR data to an HTTP indexing request that includes the cbor-attachment pipeline. The HTTP request headers use a content-type of application/cbor.

Note

Not all Elasticsearch clients support custom HTTP request headers.

import cbor2
import requests

file = 'my-file'
headers = {'content-type': 'application/cbor'}

with open(file, 'rb') as f:
  doc = {
    'data': f.read()
  }
  requests.put(
    'http://localhost:9200/my-index-000001/_doc/my_id?pipeline=cbor-attachment',
    data=cbor2.dumps(doc),
    headers=headers
  )

To prevent extracting too many chars and overload the node memory, the number of chars being used for extraction is limited by default to 100000. You can change this value by setting indexed_chars. Use -1 for no limit but ensure when setting this that your node will have enough HEAP to extract the content of very big documents.

You can also define this limit per document by extracting from a given field the limit to set. If the document has that field, it will overwrite the indexed_chars setting. To set this field, define the indexed_chars_field setting.

For example:

				PUT _ingest/pipeline/attachment
					{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data",
        "indexed_chars" : 11,
        "indexed_chars_field" : "max_size",
        "remove_binary": true
      }
    }
  ]
}
				PUT my-index-000001/_doc/my_id?pipeline=attachment
					{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
}
				GET my-index-000001/_doc/my_id

Returns this:

{
  "found": true,
  "_index": "my-index-000001",
  "_id": "my_id",
  "_version": 1,
  "_seq_no": 35,
  "_primary_term": 1,
  "_source": {
    "attachment": {
      "content_type": "application/rtf",
      "language": "is",
      "content": "Lorem ipsum",
      "content_length": 11
    }
  }
}
		
				PUT _ingest/pipeline/attachment
					{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data",
        "indexed_chars" : 11,
        "indexed_chars_field" : "max_size",
        "remove_binary": true
      }
    }
  ]
}
				PUT my-index-000001/_doc/my_id_2?pipeline=attachment
					{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
  "max_size": 5
}
				GET my-index-000001/_doc/my_id_2

Returns this:

{
  "found": true,
  "_index": "my-index-000001",
  "_id": "my_id_2",
  "_version": 1,
  "_seq_no": 40,
  "_primary_term": 1,
  "_source": {
    "max_size": 5,
    "attachment": {
      "content_type": "application/rtf",
      "language": "sl",
      "content": "Lorem",
      "content_length": 5
    }
  }
}

To use the attachment processor within an array of attachments the foreach processor is required. This enables the attachment processor to be run on the individual elements of the array.

For example, given the following source:

{
  "attachments" : [
    {
      "filename" : "ipsum.txt",
      "data" : "dGhpcyBpcwpqdXN0IHNvbWUgdGV4dAo="
    },
    {
      "filename" : "test.txt",
      "data" : "VGhpcyBpcyBhIHRlc3QK"
    }
  ]
}

In this case, we want to process the data field in each element of the attachments field and insert the properties into the document so the following foreach processor is used:

				PUT _ingest/pipeline/attachment
					{
  "description" : "Extract attachment information from arrays",
  "processors" : [
    {
      "foreach": {
        "field": "attachments",
        "processor": {
          "attachment": {
            "target_field": "_ingest._value.attachment",
            "field": "_ingest._value.data",
            "remove_binary": true
          }
        }
      }
    }
  ]
}
				PUT my-index-000001/_doc/my_id?pipeline=attachment
					{
  "attachments" : [
    {
      "filename" : "ipsum.txt",
      "data" : "dGhpcyBpcwpqdXN0IHNvbWUgdGV4dAo="
    },
    {
      "filename" : "test.txt",
      "data" : "VGhpcyBpcyBhIHRlc3QK"
    }
  ]
}
				GET my-index-000001/_doc/my_id

Returns this:

{
  "_index" : "my-index-000001",
  "_id" : "my_id",
  "_version" : 1,
  "_seq_no" : 50,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "attachments" : [
      {
        "filename" : "ipsum.txt",
        "attachment" : {
          "content_type" : "text/plain; charset=ISO-8859-1",
          "language" : "en",
          "content" : "this is\njust some text",
          "content_length" : 24
        }
      },
      {
        "filename" : "test.txt",
        "attachment" : {
          "content_type" : "text/plain; charset=ISO-8859-1",
          "language" : "en",
          "content" : "This is a test",
          "content_length" : 16
        }
      }
    ]
  }
}

Note that the target_field needs to be set, otherwise the default value is used which is a top level field attachment. The properties on this top level field will contain the value of the first attachment only. However, by specifying the target_field on to a value on _ingest._value it will correctly associate the properties with the correct attachment.