--- title: Common Expression Language input description: Use the cel input to read messages from a file path or HTTP API with a variety of payloads using the Common Expression Language (CEL) and the mito CEL... url: https://www.elastic.co/elastic/docs-builder/docs/3028/reference/beats/filebeat/filebeat-input-cel products: - Beats - Filebeat applies_to: - Elastic Cloud Serverless: Generally available - Elastic Stack: Generally available since 8.6 --- # Common Expression Language input Use the `cel` input to read messages from a file path or HTTP API with a variety of payloads using the [Common Expression Language (CEL)](https://opensource.google.com/projects/cel) and the [mito](https://pkg.go.dev/github.com/elastic/mito/lib) CEL extension libraries. CEL is a non-Turing complete language that can perform evaluation of expression in inputs, which can include file and API endpoints using the mito extension library. The `cel` input periodically runs a CEL program that is given an execution environment that may be configured by the user, and publishes the set of events that result from the program evaluation. Optionally the CEL program may return cursor states that will be provided to the next execution of the CEL program. The cursor states may be used to control the behavior of the program. This input supports: - Auth - Basic - Digest - Elastic Stack: Generally available since 9.3 File - OAuth2 - Elastic Stack: Generally available since 9.3 AWS - Retrieval at a configurable interval - Pagination - Retries - Rate limiting - Proxying Example configurations: ```yaml filebeat.inputs: # Fetch your public IP every minute. - type: cel interval: 1m resource.url: https://api.ipify.org/?format=json program: | get(state.url).Body.as(body, { "events": [body.decode_json()] }) ``` or equivalently using the text format from ipify.org ```yaml filebeat.inputs: # Fetch your public IP every minute. - type: cel interval: 1m resource.url: https://api.ipify.org/?format=text program: | { "events": [{"ip": string(get(state.url).Body)}] } ``` ```yaml filebeat.inputs: - type: cel resource.url: http://localhost:9200/_search state: scroll: 5m program: | ( !has(state.cursor) || !has(state.cursor.scroll_id) ? post(state.url+"?scroll=5m", "", "") : post( state.url+"/scroll?"+{"scroll_id": [state.cursor.scroll_id]}.format_query(), "application/json", {"scroll": state.scroll}.encode_json() ) ).as(resp, resp.Body.decode_json().as(body, { "events": body.hits.hits, "cursor": {"scroll_id": body._scroll_id}, })) ``` ## Execution The execution environment provided for the input includes the functions, macros, and global variables provided by the mito library. A single JSON object is provided as an input accessible through a `state` variable. `state` contains a string `url` field and may contain arbitrary other fields configured via the input’s `state` configuration. If the CEL program saves cursor states between executions of the program, the configured `state.cursor` value will be replaced by the saved cursor prior to execution. On start the `state` is will be something like this: ```json { "url": , "cursor": { ... }, ... } ``` The `state.url` field will be present and may be an HTTP end-point or a file path. It is the responsibility of the CEL program to handle removing the scheme from a file URL if it is present. The `state.url` field may be mutated during execution of the program, but the mutated state will not be persisted between restarts The `state.url` field must be present in the returned value to ensure that it is available in the next evaluation unless the program has the resource address hard-coded in or it is available from the cursor. Additional fields may be present at the root of the object and if the program tolerates it, the cursor value may be absent. Only the cursor is persisted over restarts, but all fields in state are retained between iterations of the processing loop except for the produced events array, see below. If the cursor is present the program should perform and process requests based on its value. If cursor is not present the program must have alternative logic to determine what requests to make. After completion of a program’s execution it should return a single object with a structure looking like this: ```json { "events": [ {...}, ... ], "cursor": [ {...}, ... ], "url": , "status_code": , "header": , "rate_limit": , "want_more": false } ``` The `status_code`, `header` and `rate_limit` values may be omitted if the program is not interacting with an HTTP API end-point and so will not be needed to contribute to program control. ## Debug state logging The CEL input will log the complete state after evaluation when logging at the DEBUG level. This will include any sensitive or secret information kept in the `state` object, and so DEBUG level logging should not be used in production when sensitive information is retained in the `state` object. Values under `state.secret` are always redacted automatically (see [`secret_state`](#secret-state-cel)). See [`redact`](#cel-state-redact) configuration parameters for settings to exclude other sensitive fields from DEBUG logs. ## CEL extension libraries As noted above the `cel` input provides functions, macros, and global variables to extend the language. - [AWS v4 request signing](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#AWS) Elastic Stack: Generally available since 9.1 - [Sign AWS from env](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Sign_AWS_from_env-AWS) - [Sign AWS from shared credentials](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Sign_AWS_from_shared_credentials-AWS) - [Sign AWS from static credentials](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Sign_AWS_from_static_credentials-AWS) - [Collections](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#Collections) - [Collate](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Collate-Collections) - [Drop](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Drop-Collections) - [Drop Empty](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Drop_Empty-Collections) - [Flatten](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Flatten-Collections) - [Front](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Front-Collections) - [Keys](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Keys-Collections) - [Max](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Max-Collections) - list maximum - pair maximum - [Min](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Min-Collections) - list minimum - pair minimum - [Sum](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Sum-Collections) - [Tail](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Tail-Collections) - one parameter - two parameter - [Values](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Values-Collections) - [With](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-With-Collections) - [With Replace](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-With_Replace-Collections) - [With Update](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-With_Update-Collections) - [Zip](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Zip-Collections) - [Crypto](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#Crypto) - [Base64](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Base64-Crypto) - [Base64 Decode](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Base64_Decode-Crypto) - [Base64 Raw](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Base64_Raw-Crypto) - [Base64 Raw Decode](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Base64_Raw_Decode-Crypto) - [Hex](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Hex-Crypto) - [Hex Decode](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Hex_Decode-Crypto) - [MD5](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-MD5-Crypto) - [SHA-1](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-SHA_1-Crypto) - [SHA-256](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-SHA_256-Crypto) - [HMAC](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-HMAC-Crypto) - [UUID](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-UUID-Crypto) - [File](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#File) — the file extension is initialized with MIME handlers for "application/gzip", ["application/x-ndjson"](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#NDJSON), ["application/zip"](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#Zip), ["text/csv; header=absent"](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#CSVNoHeader), and ["text/csv; header=present"](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#CSVHeader). - [Dir](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Dir-File) - [File](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-File-File) - [HTTP](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#HTTP) - [HEAD](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-HEAD-HTTP) - [GET](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-GET-HTTP) - [GET Request](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-GET_Request-HTTP) - [POST](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-POST-HTTP) - [POST Request](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-POST_Request-HTTP) - [Request](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Request-HTTP) - [Basic Authentication](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Basic_Authentication-HTTP) - [Do Request](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Do_Request-HTTP) - [Parse URL](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Parse_URL-HTTP) - [Format URL](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Format_URL-HTTP) - [Parse Query](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Parse_Query-HTTP) - [Format Query](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Format_Query-HTTP) - [JSON](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#JSON) - [Encode JSON](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Encode_JSON-JSON) - [Decode JSON](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Decode_JSON-JSON) - [Decode JSON Stream](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Decode_JSON_Stream-JSON) - [XML](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#XML) — the XML extension is initialized with XML schema definitions provided via the `xsd` configuration option. - [Decode XML](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Decode_XML-XML) - Optional XSD definition in one-parameter form. - [Limit](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#Limit) — the rate limit extension is initialized with [Okta (as "okta")](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#OktaRateLimit) and the [Draft Rate Limit (as "draft")](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#DraftRateLimit) policies. - [Rate Limit](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Rate_Limit-Limit) - [MIME](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#MIME) — the MIME extension is initialized with MIME handlers for "application/gzip", ["application/x-ndjson"](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#NDJSON), ["application/zip"](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#Zip), ["text/csv; header=absent"](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#CSVNoHeader), and ["text/csv; header=present"](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#CSVHeader). - [MIME](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-MIME-MIME) - [Regexp](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#Regexp) — the regular expression extension is initialized with the patterns specified in the user input configuration via the `regexp` field. - [RE Match](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-RE_Match) - [RE Find](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-RE_Find) - [RE Find All](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-RE_Find_All) - [RE Find Submatch](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-RE_Find_Submatch) - [RE Find All Submatch](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-RE_Find_All_Submatch) - [RE Replace All](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-RE_Replace_All) - [Printf](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#Printf) - [Sprintf](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Sprintf-Printf) - [Strings](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#Strings) - [String Methods](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-String_Methods-Strings) - [String List Methods](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-String_List_Methods-Strings) - [Bytes Methods](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Bytes_Methods-Strings) - [Time](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#Time) - [Format](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Format-Time) - [Parse Time](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Parse_Time-Time) - [Round](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Round-Time) Elastic Stack: Generally available since 9.1 - [Truncate](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Truncate-Time) Elastic Stack: Generally available since 9.3 - [Global Variables](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Global_Variables-Time) - Support for [`DateOnly`](https://pkg.go.dev/time#DateOnly), [`DateTime`](https://pkg.go.dev/time#DateTime) and [`TimeOnly`](https://pkg.go.dev/time#TimeOnly) time formats. - [Try](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#Try) - [Try](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Try-Try) - [Is Error](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Is_Error-Try) - [Debug](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#Debug) — the debug handler registers a logger with the name extension `cel_debug` and calls to the CEL `debug` function are emitted to that logger. - [Debug](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Debug) In addition to the extensions provided in the packages listed above, a global variable `useragent` is also provided which gives the user CEL program access to the filebeat user-agent string. By default, this value is assigned to all requests' user-agent headers unless the CEL program has already set the user-agent header value. Programs wishing to not provide a user-agent, should set this header to the empty string, `""`. Host environment variables are made available via the global map `env`. Only environment variables that have been allow listed via the `allowed_environment` configuration list are visible to the CEL program. The CEL environment enables the [optional types](https://pkg.go.dev/github.com/google/cel-go/cel#OptionalTypes) library using the version defined [here](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#OptionalTypesVersion) and the [two-variable comprehensions extensions](https://pkg.go.dev/github.com/google/cel-go/ext#TwoVarComprehensions) library using the version defined [here](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#TwoVarComprehensionVersion). - Optional types - Two-variable comprehensions Elastic Stack: Generally available since 9.1 Additionally, it supports authentication via: - Basic Authentication - Digest Authentication - OAuth2 - file-based headers Elastic Stack: Generally available since 9.3 - token authentication Elastic Stack: Generally available since 9.1 - AWS Authentication Elastic Stack: Generally available since 9.3 As described in Mito's [HTTP](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#HTTP) documentation, configuration for Basic Authentication or token authentication will only affect direct HEAD, GET and POST method calls, not explicity constructed requests run with `.do_request()`. Configuration for Digest Authentication, file-based headers or OAuth2 will be used for all requests made from CEL. Example configurations with authentication: ```yaml filebeat.inputs: - type: cel auth.basic: user: user@domain.tld password: P@$$W0₹D resource.url: http://localhost ``` ```yaml filebeat.inputs: - type: cel auth.digest: user: user@domain.tld password: P@$$W0₹D resource.url: http://localhost ``` ```yaml filebeat.inputs: - type: cel auth.oauth2: client.id: 12345678901234567890abcdef client.secret: abcdef12345678901234567890 token_url: http://localhost/oauth2/token resource.url: http://localhost ``` ```yaml filebeat.inputs: - type: cel auth.oauth2: client.id: 12345678901234567890abcdef client.secret: abcdef12345678901234567890 token_url: http://localhost/oauth2/token user: user@domain.tld password: P@$$W0₹D resource.url: http://localhost ``` ```yaml filebeat.inputs: - type: cel auth.file: path: /etc/elastic/token prefix: "Bearer " refresh_interval: 10m resource.url: http://localhost ``` ```yaml filebeat.inputs: - type: cel auth.token: type: Bearer value: supersecret_bearer_token resource.url: http://localhost ``` ```yaml filebeat.inputs: - type: cel auth.token: type: Token value: supersecret_token resource.url: http://localhost ``` ```yaml filebeat.inputs: - type: cel auth.aws: access_key_id: "AKIAIOSFODNN7EXAMPLE" secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" request.url: https://guardduty.us-east-1.amazonaws.com/detector/abc123/findings ``` ```yaml filebeat.inputs: - type: cel auth.aws: credential_profile_name: fb-aws shared_credential_file: /etc/filebeat/aws_credentials request.url: https://guardduty.us-east-1.amazonaws.com/detector/abc123/findings ``` ## Input state The `cel` input keeps a runtime state between requests. This state can be accessed by the CEL program and may contain arbitrary objects. The state must contain a `url` string and may contain any object the user wishes to store in it. All objects are stored at runtime, except `cursor`, which has values that are persisted between restarts. ## HTTP rate limit handling CEL evaluations may return a `rate_limit` object as noted in the [Execution](#_execution) section. This can either be calculated explicitly in CEL code, or by using the [`rate_limit`](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#hdr-Rate_Limit-Limit) extension function. If the `rate_limit` extension is used, calculated rate limits are applied directly to the HTTP client used by the CEL input. This includes uses of `rate_limit` that do not return their results from the CEL context. Elastic Stack: Generally available since 9.3 ## CEL input and handling numbers Numeric values passed in to and out of a CEL evaluation environment are passed as floating point values. This can sometimes cause issues when the numbers in the input state are expected to be integers, and may result in unexpected field values being ingested into Elasticsearch documents when other parts of the ingest pipeline render floating point values with E-notation or add decimal points to numbers that are expected to be integers. This is most likely to happen when numbers are large (not within ±10^7^). Above the maximum exact integer representation threshold for double precision floating point values, within ±2^53^ (±9×10^15^), integer values will lose precision when they are returned from the CEL evaluation environment. The CEL input will automatically convert integers outside of the ±2^53^ range to strings in order to prevent loss of precision in these values, but potentially leading to a situation where some numbers received by the ingest pipeline are numbers and some are strings. To avoid these issues, when you have large integer values as part of an evaluation result, convert the field value to a string before returning it, and convert input numbers to integers explicitly at the start of a CEL program. ## Configuration options The `cel` input supports the following configuration options plus the [Common options](#filebeat-input-cel-common-options) described later. ### `interval` Duration between repeated requests. It may make additional pagination requests in response to the initial request if pagination is enabled. Default: `60s`. ### `program` The CEL program that is executed each polling period. This field is required. ### `max_executions` `max_executions` is the maximum number of times a CEL program can request to be re-run with a `want_more` field. This is used to ensure that accidental infinite loops do not halt processing. When the execution budget is exceeded, execution will be restarted at the next interval and a warning will be written into the logs. Default: 1000. The number of executions remaining in the execution budget after the completion of the current evaluation is available within the CEL program by referencing the `remaining_executions` global variable. Elastic Stack: Generally available since 9.2 ### `state` `state` is an optional object that is passed to the CEL program as the `state` variable on the first execution. Subsequent executions of the program during the life of the input will populate the `state` variable with the return value of the previous execution, but with the `state.events` field removed. Except for the `state.cursor` field, returned `state` data does not persist over restarts. ### `state.cursor` The cursor is an object available as `state.cursor` where arbitrary values may be stored. Cursor state is kept between input restarts and updated after each event of a request has been published. When a cursor is used the CEL program must either create a cursor state for each event that is returned by the program, or a single cursor that reflect the cursor for completion of the full set of events. ```yaml filebeat.inputs: # Fetch your public IP every minute and note when the last request was made. - type: cel interval: 1m resource.url: https://api.ipify.org/?format=json program: | get(state.url).Body.as(body, { "events": [body.decode_json().with({ "last_requested_at": has(state.cursor) && has(state.cursor.last_requested_at) ? state.cursor.last_requested_at : now })], "cursor": {"last_requested_at": now} }) ``` ### `secret_state` - Elastic Stack: Generally available since 9.2 `secret_state` is an optional object holding secret key-value pairs. When configured in a Fleet integration package with `secret: true`, the values are stored encrypted by Fleet and decrypted before being passed to the input. At runtime, the contents of `secret_state` are placed at `state.secret`, making them available to the CEL program as `state.secret.`. The key `secret` in the top-level `state` configuration is reserved — the input rejects any configuration where `state` contains a `secret` key, since values in `state` cannot be guaranteed to be encrypted in the stored configuration. `state.secret` is unconditionally added to the redaction list for debug logging, regardless of whether `secret_state` is configured. This means: - Secrets from `secret_state` are automatically redacted without any `redact` configuration. - CEL programs that stash sensitive runtime values (such as session tokens) into `state.secret` during execution will also have those values redacted in debug logs. - If `state.secret` is absent at log time, the redaction is a no-op. - Values explicitly emitted by CEL `debug()` may include secrets. ```yaml filebeat.inputs: - type: cel interval: 1m resource.url: https://api.example.com/data secret_state: api_key: "my-secret-api-key" program: | request("GET", state.url).with({ "Header": {"X-API-Key": [state.secret.api_key]} }).do_request().as(resp, { "events": [resp.Body.decode_json()], "secret": state.secret, }) ``` In this example, `state.secret.api_key` holds the API key used in request headers. The `"secret": state.secret` line in the return value preserves the secret state across evaluation loops; like any other state field, it must be included in the program's return value to persist between evaluations. Enabling `failure_dump` can write secrets (including `state.secret`) to disk, since the dump captures the full CEL evaluation state. This applies to any secrets in state, not only those from `secret_state`. ### `allowed_environment` - Elastic Stack: Generally available since 8.16 A list of host environment variable that will be made visible to the CEL execution environment. By default, no environment variables are visible. ```yaml filebeat.inputs: # Publish the list of files in $PATH every minute. - type: cel interval: 1m resource.url: "" allowed_environment: - PATH program: | { "events": { "message": env.?PATH.orValue("").split(":") .map(p, try(dir(p))) .filter(d, type(d) != type("")) .flatten() .collate("name") } } ``` ### `regexp` A set of named regular expressions that may be used during a CEL program’s execution using the `regexp` extension library. The syntax used for the regular expressions is [RE2](https://github.com/google/re2/wiki/Syntax). ```yaml filebeat.inputs: - type: cel # Define two regular expressions, 'products' and 'solutions' for use during CEL execution. regexp: products: '(?i)(Elasticsearch|Beats|Logstash|Kibana)' solutions: '(?i)(Search|Observability|Security)' ``` ### `xsd` - Elastic Stack: Generally available since 8.9 XML documents may require additional type information to enable correct parsing and ingestion. This information can be provided as an XML Schema Definitions (XSD) for XML documents using the `xsd` option. The key under which the XSD information is provided is accessed via the `decode_xml` CEL extension. ```yaml filebeat.inputs: - type: cel # Provide an XSD, 'order', for use during CEL execution (truncated for example). xsd: order: | ``` The `xsd` for an XML document structure may be omitted. ### `auth.basic.enabled` When set to `false`, disables the basic auth configuration. Default: `true`. Basic auth settings are disabled if either `enabled` is set to `false` or the `auth.basic` section is missing. Basic auth settings do not affect requests run with `.do_request()`, as explained in [HTTP](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#HTTP). ### `auth.basic.user` The user to authenticate with. ### `auth.basic.password` The password to use. ### `auth.digest.enabled` - Elastic Stack: Generally available since 8.12 When set to `false`, disables the digest auth configuration. Default: `true`. digest auth settings are disabled if either `enabled` is set to `false` or the `auth.digest` section is missing. ### `auth.digest.user` - Elastic Stack: Generally available since 8.12 The user to authenticate with. ### `auth.digest.password` - Elastic Stack: Generally available since 8.12 The password to use. ### `auth.digest.no_reuse` - Elastic Stack: Generally available since 8.12 When set to `true`, Digest Authentication challenges are not reused. ### `auth.file.enabled` - Elastic Stack: Generally available since 9.3 When set to `false`, disables the file auth configuration. Default: `true`. File auth settings are disabled if either `enabled` is set to `false` or the `auth.file` section is missing. ### `auth.file.path` - Elastic Stack: Generally available since 9.3 The path to the file containing the authentication value. The file contents are trimmed before use. This field is required when file auth is enabled. By default, Filebeat requires the file to have `0600` permissions (read/write for owner only) and will fail to start if the file is more permissive. This security measure helps prevent unauthorized access to credentials. To allow files with different permissions, set [`relaxed_permissions`](#_auth_file_relaxed_permissions) to `true`.On Windows, POSIX-style permission checking is not enforced. Ensure file security using NTFS file permissions or Access Control Lists (ACLs). ### `auth.file.header` - Elastic Stack: Generally available since 9.3 The request header that receives the value loaded from `path`. Defaults to `Authorization` when omitted or empty. ### `auth.file.prefix` - Elastic Stack: Generally available since 9.3 An optional prefix that is prepended to the trimmed value from `path` before it is set on the request header. This is commonly used for tokens that require a leading value such as `Bearer `. ### `auth.file.refresh_interval` - Elastic Stack: Generally available since 9.3 How frequently Filebeat rereads the file defined by `path` to pick up changes. Defaults to `1m`. The value must be greater than zero when set. ### `auth.file.relaxed_permissions` - Elastic Stack: Generally available since 9.3 When set to `true`, allows the authentication file to have permissions other than `0600`. By default (`false`), Filebeat requires the file to have `0600` permissions and will fail to start if the file is more permissive. This security measure helps prevent unauthorized access to credentials. Setting this to `true` reduces security. Only enable this option if you understand the security implications and cannot set the file to `0600` permissions. ### `auth.oauth2.enabled` When set to `false`, disables the oauth2 configuration. Default: `true`. OAuth2 settings are disabled if either `enabled` is set to `false` or the `auth.oauth2` section is missing. ### `auth.oauth2.provider` Used to configure supported oauth2 providers. Each supported provider will require specific settings. It is not set by default. Supported providers are: `azure`, `google`, `okta`. ### `auth.oauth2.client.id` The client ID used as part of the authentication flow. It is always required except if using `google` as provider. Required for providers: `default`, `azure`, `okta`. ### `auth.oauth2.client.secret` The client secret used as part of the authentication flow. It is always required except if using `google` or `okta` as provider. Required for providers: `default`, `azure`. ### `auth.oauth2.user` The user used as part of the authentication flow. It is required for authentication - grant type password. It is only available for provider `default`. ### `auth.oauth2.password` The password used as part of the authentication flow. It is required for authentication - grant type password. It is only available for provider `default`. user and password are required for grant_type password. If user and password is not used then it will automatically use the `token_url` and `client credential` method. ### `auth.oauth2.scopes` A list of scopes that will be requested during the oauth2 flow. It is optional for all providers. ### `auth.oauth2.token_url` The endpoint that will be used to generate the tokens during the oauth2 flow. It is required if no provider is specified. For `azure` provider either `token_url` or `azure.tenant_id` is required. ### `auth.oauth2.endpoint_params` Set of values that will be sent on each request to the `token_url`. Each param key can have multiple values. Can be set for all providers except `google`. ```yaml - type: cel auth.oauth2: endpoint_params: Param1: - ValueA - ValueB Param2: - Value ``` ### `auth.oauth2.azure.tenant_id` Used for authentication when using `azure` provider. Since it is used in the process to generate the `token_url`, it can’t be used in combination with it. It is not required. For information about where to find it, you can refer to [[https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal)](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal). ### `auth.oauth2.azure.resource` The accessed WebAPI resource when using `azure` provider. It is not required. ### `auth.oauth2.google.credentials_file` The credentials file for Google. Only one of the credentials settings can be set at once. If none is provided, loading default credentials from the environment will be attempted via ADC. For more information about how to provide Google credentials, please refer to [[https://cloud.google.com/docs/authentication](https://cloud.google.com/docs/authentication)](https://cloud.google.com/docs/authentication). ### `auth.oauth2.google.credentials_json` Your credentials information as raw JSON. Only one of the credentials settings can be set at once. If none is provided, loading default credentials from the environment will be attempted via ADC. For more information about how to provide Google credentials, please refer to [[https://cloud.google.com/docs/authentication](https://cloud.google.com/docs/authentication)](https://cloud.google.com/docs/authentication). ### `auth.oauth2.google.jwt_file` The JWT Account Key file for Google. Only one of the credentials settings can be set at once. If none is provided, loading default credentials from the environment will be attempted via ADC. For more information about how to provide Google credentials, please refer to [[https://cloud.google.com/docs/authentication](https://cloud.google.com/docs/authentication)](https://cloud.google.com/docs/authentication). ### `auth.oauth2.google.jwt_json` The JWT Account Key file as raw JSON. Only one of the credentials settings can be set at once. If none is provided, loading default credentials from the environment will be attempted via ADC. For more information about how to provide Google credentials, please refer to [[https://cloud.google.com/docs/authentication](https://cloud.google.com/docs/authentication)](https://cloud.google.com/docs/authentication). ### `auth.oauth2.google.delegated_account` Email of the delegated account used to create the credentials (usually an admin). Used in combination with `auth.oauth2.google.jwt_file` or `auth.oauth2.google.jwt_json`. ### `auth.oauth2.okta.jwk_file` The RSA JWK Private Key file for your Okta Service App which is used for interacting with Okta Org Auth Server to mint tokens with okta.* scopes. Only one of the credentials settings can be set at once. For more information please refer to [[https://developer.okta.com/docs/guides/implement-oauth-for-okta-serviceapp/main/](https://developer.okta.com/docs/guides/implement-oauth-for-okta-serviceapp/main/)](https://developer.okta.com/docs/guides/implement-oauth-for-okta-serviceapp/main/) ### `auth.oauth2.okta.jwk_json` The RSA JWK Private Key JSON for your Okta Service App which is used for interacting with Okta Org Auth Server to mint tokens with okta.* scopes. Only one of the credentials settings can be set at once. For more information please refer to [[https://developer.okta.com/docs/guides/implement-oauth-for-okta-serviceapp/main/](https://developer.okta.com/docs/guides/implement-oauth-for-okta-serviceapp/main/)](https://developer.okta.com/docs/guides/implement-oauth-for-okta-serviceapp/main/) ### `auth.oauth2.okta.jwk_pem` The RSA JWK private key PEM block for your Okta Service App which is used for interacting with Okta Org Auth Server to mint tokens with okta.* scopes. Only one of the credentials settings can be set at once. For more information please refer to [[https://developer.okta.com/docs/guides/implement-oauth-for-okta-serviceapp/main/](https://developer.okta.com/docs/guides/implement-oauth-for-okta-serviceapp/main/)](https://developer.okta.com/docs/guides/implement-oauth-for-okta-serviceapp/main/) ### `auth.oauth2.okta.dpop_key_pem` - Elastic Stack: Generally available since 9.2 The Demonstrating Proof-of-Possession private key PEM block for your Okta authentication token. When this key is provided, Okta authentication will make use of the [Okta DPoP authentication flow](https://www.okta.com/blog/product-innovation/a-leap-forward-in-token-security-okta-adds-support-for-dpop/). ### `auth.token.enabled` When set to `false`, disables the token authentication configuration. Default: `true`. Token authentication settings are disabled if either `enabled` is set to `false` or the `auth.token` section is missing. Token authentication settings do not affect requests run with `.do_request()`, as explained in [HTTP](https://pkg.go.dev/github.com/elastic/mito@v1.24.1/lib#HTTP). ### `auth.token.type` The type of token to authenticate with, for example "Token" or "Bearer". ### `auth.token.value` The token value to use. ### `auth.aws.enabled` - Elastic Stack: Generally available since 9.3 When set to `false`, disables the file AWS auth configuration. Default: `true`. AWS auth settings are disabled if either `enabled` is set to `false` or the `auth.aws` section is missing. ### `auth.aws.access_key_id` - Elastic Stack: Generally available since 9.3 The AWS access key ID. It should be used with [`auth.aws.secret_access_key`](#_auth_aws_secret_access_key). ### `auth.aws.secret_access_key` - Elastic Stack: Generally available since 9.3 The AWS secret access key. It should be used with [`auth.aws.access_key_id`](#_auth_aws_access_key_id). Use either direct keys ([`auth.aws.access_key_id`](#_auth_aws_access_key_id) and [`auth.aws.secret_access_key`](#_auth_aws_secret_access_key)) or a shared credentials file ([`auth.aws.shared_credential_file`](#_auth_aws_shared_credential_file)). If both are set, the direct keys take precedence.If neither the direct keys nor the shared credentials file is set, AWS SDK [LoadDefaultConfig](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/config#LoadDefaultConfig) will be used. ### `auth.aws.session_token` - Elastic Stack: Generally available since 9.3 The AWS session token that can be optionally set when direct keys ([`auth.aws.access_key_id`](#_auth_aws_access_key_id) and [`auth.aws.secret_access_key`](#_auth_aws_secret_access_key)) are used. ### `auth.aws.shared_credential_file` - Elastic Stack: Generally available since 9.3 The path of the AWS shared credentials file. Use either direct keys ([`auth.aws.access_key_id`](#_auth_aws_access_key_id) and [`auth.aws.secret_access_key`](#_auth_aws_secret_access_key)) or a shared credentials file ([`auth.aws.shared_credential_file`](#_auth_aws_shared_credential_file)). If both are set, the direct keys take precedence.If neither the direct keys nor the shared credentials file is set, AWS SDK [LoadDefaultConfig](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/config#LoadDefaultConfig) will be used. ### `auth.aws.credential_profile_name` - Elastic Stack: Generally available since 9.3 The profile name of the AWS shared credentials file. This is optional and can be used with [`auth.aws.shared_credential_file`](#_auth_aws_shared_credential_file). ### `auth.aws.role_arn` - Elastic Stack: Generally available since 9.3 The IAM Role ARN to assume. Assume-role authentication is layered on top of the base credentials, which may come from a direct access key, a shared credentials file, or the default SDK configuration. The assume-role request will use whichever credentials have already been established. ### `auth.aws.external_id` - Elastic Stack: Generally available since 9.3 Specifies the external ID to use for every IAM assume-role request. This is optional and can be used when [`auth.aws.role_arn`](#_auth_aws_role_arn) is configured. ### `auth.aws.assume_role.duration` - Elastic Stack: Generally available since 9.3 Specifies the duration of the credentials retrieved by the IAM assume-role. This is optional and can be used when [`auth.aws.role_arn`](#_auth_aws_role_arn) is configured. ### `auth.aws.assume_role.expiry_window` - Elastic Stack: Generally available since 9.3 Specifies the credentials retrieved by the IAM assume-role to trigger refreshing prior to the credentials actually expiring. This is optional and can be used when [`auth.aws.role_arn`](#_auth_aws_role_arn) is configured. ### `auth.aws.service_name` - Elastic Stack: Generally available since 9.3 Specifies the AWS service name that will be used in the v4 signing process. This is optional and if not set it will be inferred by the request URL. ### `auth.aws.default_region` - Elastic Stack: Generally available since 9.3 Specifies the AWS region that will be used in the v4 signing process. This is optional and if not set it will be inferred by the request URL. ### `resource.url` The URL of the HTTP API. Required. The API endpoint may be accessed via unix socket and Windows named pipes by adding `+unix` or `+npipe` to the URL scheme, for example, `http+unix:///var/socket/`. ### `resource.headers` - Elastic Stack: Generally available since 8.18 Headers to be added to all requests. Headers are added before authentication headers, so any collision between headers in this configuration and authentication headers will result in the colliding headers here not being included in requests. Header values must be provided as an array. ```yaml filebeat.inputs: # Fetch your public IP every minute. - type: cel interval: 1m resource.url: https://api.ipify.org/?format=text resource.headers: Custom-Header: - Value Other-Custom-Header: - Other value resource.proxy_url: http://proxy.example:8080 program: | { "events": [{"ip": string(get(state.url).Body)}] } ``` ### `resource.timeout` Duration before declaring that the HTTP client connection has timed out. Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`. Default: `30s`. ### `resource.ssl` This specifies SSL/TLS configuration. If the ssl section is missing, the host’s CAs are used for HTTPS connections. See [SSL](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/beats/filebeat/configuration-ssl) for more information. ### `resource.proxy_url` This specifies proxy configuration in the form of `http[s]://:@:`. Proxy headers may be configured using the `resource.proxy_headers` field which accepts a set of key/value pairs. ```yaml filebeat.inputs: # Fetch your public IP every minute. - type: cel interval: 1m resource.url: https://api.ipify.org/?format=text resource.proxy_url: http://proxy.example:8080 program: | { "events": [{"ip": string(get(state.url).Body)}] } ``` ### `resource.keep_alive.disable` This specifies whether to disable keep-alives for HTTP end-points. Default: `true`. ### `resource.keep_alive.max_idle_connections` The maximum number of idle connections across all hosts. Zero means no limit. Default: `0`. ### `resource.keep_alive.max_idle_connections_per_host` The maximum idle connections to keep per-host. If zero, defaults to two. Default: `0`. ### `resource.keep_alive.idle_connection_timeout` The maximum amount of time an idle connection will remain idle before closing itself. Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`. Zero means no limit. Default: `0s`. ### `resource.retry.max_attempts` The maximum number of retries for the HTTP client. Default: `5`. ### `resource.retry.wait_min` The minimum time to wait before a retry is attempted. Default: `1s`. ### `resource.retry.wait_max` The maximum time to wait before a retry is attempted. Default: `60s`. ### `resource.redirect.forward_headers` When set to `true` request headers are forwarded in case of a redirect. Default: `false`. ### `resource.redirect.headers_ban_list` When `redirect.forward_headers` is set to `true`, all headers *except* the ones defined in this list will be forwarded. Default: `[]`. ### `resource.redirect.max_redirects` The maximum number of redirects to follow for a request. Default: `10`. ### `resource.max_body_size` - Elastic Stack: Generally available since 8.18 The maximum size of a response body that will be accepted by the client if non-zero. Bodies that are too large will result in an error, "response body too big". Default: `0`. ### `resource.rate_limit.limit` The value of the response that specifies the maximum overall resource request rate. ### `resource.rate_limit.burst` The maximum burst size. Burst is the maximum number of resource requests that can be made above the overall rate limit. ### `resource.tracer.enable` It is possible to log HTTP requests and responses in a CEL program to a local file-system for debugging configurations. This option is enabled by setting `resource.tracer.enabled` to true and setting the `resource.tracer.filename` value. Additional options are available to tune log rotation behavior. To delete existing logs, set `resource.tracer.enabled` to false without unsetting the filename option. Enabling this option compromises security and should only be used for debugging. ### `resource.tracer.filename` To differentiate the trace files generated from different input instances, a placeholder `*` can be added to the filename and will be replaced with the input instance id. For Example, `http-request-trace-*.ndjson`. The path must point to a target in the cel directory in the [Filebeat logs directory](https://www.elastic.co/docs/reference/beats/filebeat/directory-layout). Setting `resource.tracer.filename` with `resource.tracer.enable` set to false will cause any existing trace logs matching the filename option to be deleted. ### `resource.tracer.maxsize` This value sets the maximum size, in megabytes, the log file will reach before it is rotated. By default logs are allowed to reach 1MB before rotation. Individual request/response bodies will be truncated to 10% of this size. ### `resource.tracer.maxage` This specifies the number days to retain rotated log files. If it is not set, log files are retained indefinitely. ### `resource.tracer.maxbackups` The number of old logs to retain. If it is not set all old logs are retained subject to the `resource.tracer.maxage` setting. ### `resource.tracer.localtime` Whether to use the host’s local time rather that UTC for timestamping rotated log file names. ### `resource.tracer.compress` This determines whether rotated logs should be gzip compressed. ### `redact` - Elastic Stack: Generally available since 8.7 During debug level logging, the `state` object and the resulting evaluation result are included in logs. This may result in leaking of secrets. In order to prevent this, fields may be redacted or deleted from the logged `state`. The `redact` configuration allows users to configure this field redaction behavior. For safety reasons if the `redact` configuration is missing a warning is logged. The `state.secret` field is always redacted automatically (see [`secret_state`](#secret-state-cel)). When `secret_state` is configured and `redact` is not, the missing-redact warning is suppressed. In the case of no-required redaction an empty `redact.fields` configuration should be used to silence the logged warning. ```yaml - type: cel redact: fields: ~ ``` As an example, if a user-constructed Basic Authentication request is used in a CEL program the password can be redacted like so ```yaml filebeat.inputs: - type: cel resource.url: http://localhost:9200/_search state: user: user@domain.tld password: P@$$W0₹D redact: fields: - password delete: true ``` Note that fields under the `auth` configuration hierarchy are not exposed to the `state` and so do not need to be redacted. For this reason it is preferable to use these for authentication over the request construction shown above where possible. ### `redact.fields` This specifies fields in the `state` to be redacted prior to debug logging. Fields listed in this array will be either replaced with a `*` or deleted entirely from messages sent to debug logs. ### `redact.delete` This specifies whether fields should be replaced with a `*` or deleted entirely from messages sent to debug logs. If delete is `true`, fields will be deleted rather than replaced. ### `failure_dump.enabled` - Elastic Stack: Generally available since 8.18 It is possible to log CEL program evaluation failures to a local file-system for debugging configurations. This option is enabled by setting `failure_dump.enabled` to true and setting the `failure_dump.filename` value. To delete existing failure dumps, set `failure_dump.enabled` to false without unsetting the filename option. Enabling this option compromises security and should only be used for debugging. ### `failure_dump.filename` - Elastic Stack: Generally available since 8.18 This specifies a directory path to write failure dumps to. If it is not empty and a CEL program evaluation fails, the complete set of states for the CEL program’s evaluation will be written as a JSON file, along with the error that was reported. This option should only be used when debugging a failure as it imposes a significant performance impact on the input and may potentially use large quantities of memory to hold the full set of states. If a failure dump is configured, it is recommended that data input sizes be reduced to avoid excessive memory consumption, and making dumps that are intractable to analysis. To delete existing failure dumps, set `failure_dump.enabled` to false without unsetting the filename option. ### `record_coverage` - Elastic Stack: Generally available since 8.18 This specifies that CEL code evaluation coverage should be recorded and logged in debug logs. This is a developer-only option. ## Metrics This input exposes metrics under the [HTTP monitoring endpoint](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/beats/filebeat/http-endpoint). These metrics are exposed under the `/inputs` path. They can be used to observe the activity of the input. | Metric | Description | |----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| | `resource` | URL or path of the input resource. | | `cel_executions` | Number times the CEL program has been executed. | | `batches_received_total` | Number of event arrays received. | | `events_received_total` | Number of events received. | | `batches_published_total` | Number of event arrays published. | | `events_published_total` | Number of events published. | | `cel_processing_time` | Histogram of the elapsed successful CEL program processing times in nanoseconds. | | `batch_processing_time` | Histogram of the elapsed successful batch processing times in nanoseconds (time of receipt to time of ACK for non-empty batches). | | `http_request_total` | Total number of processed requests. | | `http_request_errors_total` | Total number of request errors. | | `http_request_delete_total` | Total number of `DELETE` requests. | | `http_request_get_total` | Total number of `GET` requests. | | `http_request_head_total` | Total number of `HEAD` requests. | | `http_request_options_total` | Total number of `OPTIONS` requests. | | `http_request_patch_total` | Total number of `PATCH` requests. | | `http_request_post_total` | Total number of `POST` requests. | | `http_request_put_total` | Total number of `PUT` requests. | | `http_request_body_bytes_total` | Total of the requests body size. | | `http_request_body_bytes` | Histogram of the requests body size. | | `http_response_total` | Total number of responses received. | | `http_response_errors_total` | Total number of response errors. | | `http_response_1xx_total` | Total number of `1xx` responses. | | `http_response_2xx_total` | Total number of `2xx` responses. | | `http_response_3xx_total` | Total number of `3xx` responses. | | `http_response_4xx_total` | Total number of `4xx` responses. | | `http_response_5xx_total` | Total number of `5xx` responses. | | `http_response_body_bytes_total` | Total of the responses body size. | | `http_response_body_bytes` | Histogram of the responses body size. | | `http_round_trip_time` | Histogram of the round trip time. | ## Developer tools A stand-alone CEL environment that implements the majority of the CEL input’s Comment Expression Language functionality is available in the [Elastic Mito](https://github.com/elastic/mito) repository. This tool may be used to help develop CEL programs to be used by the input. Installation is available from source by running `go install github.com/elastic/mito/cmd/mito@latest` and requires a Go toolchain. ## Common options The following configuration options are supported by all inputs. #### `enabled` Use the `enabled` option to enable and disable inputs. By default, enabled is set to true. #### `tags` A list of tags that Filebeat includes in the `tags` field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration. Example: ```yaml filebeat.inputs: - type: cel . . . tags: ["json"] ``` #### `fields` Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a `fields` sub-dictionary in the output document. To store the custom fields as top-level fields, set the `fields_under_root` option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here. ```yaml filebeat.inputs: - type: cel . . . fields: app_id: query_engine_12 ``` #### `fields_under_root` If this option is set to true, the custom [fields](#filebeat-input-cel-fields) are stored as top-level fields in the output document instead of being grouped under a `fields` sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields. #### `processors` A list of processors to apply to the input data. See [Processors](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/beats/filebeat/filtering-enhancing-data) for information about specifying processors in your config. #### `pipeline` The ingest pipeline ID to set for the events generated by this input. The pipeline ID can also be configured in the Elasticsearch output, but this option usually results in simpler configuration files. If the pipeline is configured both in the input and output, the option from the input is used. The `pipeline` is always lowercased. If `pipeline: Foo-Bar`, then the pipeline name in Elasticsearch needs to be defined as `foo-bar`. #### `keep_null` If this option is set to true, fields with `null` values will be published in the output document. By default, `keep_null` is set to `false`. #### `index` If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the `raw_index` field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use `output.elasticsearch.index` or a processor. Example value: `"%{[agent.name]}-myindex-%{+yyyy.MM.dd}"` might expand to `"filebeat-myindex-2019.11.01"`. #### `publisher_pipeline.disable_host` By default, all events contain `host.name`. This option can be set to `true` to disable the addition of this field to all events. The default value is `false`.