--- title: HTTP Endpoint input description: The HTTP Endpoint input initializes a listening HTTP server that collects incoming HTTP POST requests containing a JSON body. The body must be either... url: https://www.elastic.co/elastic/docs-builder/docs/3028/reference/beats/filebeat/filebeat-input-http_endpoint products: - Beats - Filebeat applies_to: - Elastic Cloud Serverless: Generally available - Elastic Stack: Generally available --- # HTTP Endpoint input The HTTP Endpoint input initializes a listening HTTP server that collects incoming HTTP POST requests containing a JSON body. The body must be either an object or an array of objects, otherwise a Common Expression Language expression that converts the the JSON body to these types can be provided. Any other data types will result in an HTTP 400 (Bad Request) response. For arrays, one document is created for each object in the array. gzip encoded request bodies are supported if a `Content-Encoding: gzip` header is sent with the request. This input can for example be used to receive incoming webhooks from a third-party application or service. Multiple endpoints may be assigned to a single address and port, and the HTTP Endpoint input will resolve requests based on the URL pattern configuration. If multiple endpoints are configured on a single address they must all have the same TLS configuration, either all disabled or all enabled with identical configurations. These are the possible response codes from the server. | HTTP Response Code | Name | Reason | |--------------------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 200 | OK | Returned on success. | | 400 | Bad Request | Returned if JSON body decoding fails, if an OPTIONS request is made and `options_headers` has not been set in the config, or if `wait_for_completion_timeout` query validation fails. | | 401 | Unauthorized | Returned when basic auth, secret header, or HMAC validation fails. | | 405 | Method Not Allowed | Returned if methods other than POST are used. | | 406 | Not Acceptable | Returned if the POST request does not contain a body. | | 415 | Unsupported Media Type | Returned if the Content-Type is not application/json. Or if Content-Encoding is present and is not gzip. | | 500 | Internal Server Error | Returned if an I/O error occurs reading the request. | | 503 | Service Unavailable | Returned if the hard limit `max_in_flight_bytes` is exceeded during body reading, or if the total in-flight bytes are above the `high_water_in_flight_bytes` threshold. The response includes a `Retry-After` header. | | 504 | Gateway Timeout | Returned if a request publication cannot be ACKed within the required timeout. | The endpoint will enforce end-to-end ACK when a URL query parameter `wait_for_completion_timeout` with a duration is provided. For example `http://localhost:8080/?wait_for_completion_timeout=1m` will wait up to 1 minute for the event to be published to the cluster and then return the user-defined response message. In the case that the publication does not complete within the timeout duration, the HTTP response will have a 504 Gateway Timeout status code. The syntax for durations is a number followed by units which may be h, m and s. No other HTTP query is accepted. If another query parameter is provided or duration syntax is incorrect, the request will fail with an HTTP 400 "Bad Request" status. Example configurations: Basic example: ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 ``` Custom response example: ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 response_code: 200 response_body: '{"message": "success"}' url: "/" prefix: "json" ``` OPTIONS request-aware example: ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 response_code: 200 options_headers: Custom-Options-Header: [custom-options-header-value] url: "/" prefix: "json" ``` Map request to root of document example: ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 prefix: "." ``` Multiple endpoints example: ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 url: "/open/" tags: [open] - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 url: "/admin/" basic_auth: true username: adminuser password: somepassword tags: [admin] ``` Disable Content-Type checks ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 content_type: "" prefix: "json" ``` Basic auth and SSL example: ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 ssl.enabled: true ssl.certificate: "/home/user/server.pem" ssl.key: "/home/user/server.key" ssl.verification_mode: "none" ssl.certificate_authority: "/home/user/ca.pem" basic_auth: true username: someuser password: somepassword ``` Authentication or checking that a specific header includes a specific value ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 secret.header: someheadername secret.value: secretheadertoken ``` Validate webhook endpoint for a specific provider using CRC ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 secret.header: someheadername secret.value: secretheadertoken crc.provider: webhookProvider crc.secret: secretToken ``` Validate a HMAC signature from a specific header ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 hmac.header: "X-Hub-Signature-256" hmac.key: "password123" hmac.type: "sha256" hmac.prefix: "sha256=" ``` Preserving original event and including headers in document ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 preserve_original_event: true include_headers: ["TestHeader"] ``` Common Expression Language example: ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 program: | obj.records.map(r, { "requestId": obj.requestId, "timestamp": string(obj.timestamp), "event": r, }) ``` This example would allow handling of a JSON body that is an object containing more than one event that each should be ingested as separate documents with the common timestamp and request ID: ```json { "requestId": "ed4acda5-034f-9f42-bba1-f29aea6d7d8f", "timestamp": 1578090901599, "records": [ { "data": "event record 1" }, { "data": "event record 2" } ] } ``` In-flight byte limiting example: ```yaml filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 max_in_flight_bytes: 10485760 high_water_in_flight_bytes: 5242880 low_water_in_flight_bytes: 4194304 retry_after: 5 ``` This configuration limits memory usage by tracking in-flight request bytes. When total in-flight bytes exceed 5MB, new requests receive a 503 response with a `Retry-After: 5` header. Once in-flight bytes drop below 4MB, new requests are accepted again. If a single request causes in-flight bytes to exceed the 10MB hard limit during reading, that request is terminated with a 503 Service Unavailable response and a `Retry-After` header. ## Configuration options The `http_endpoint` input supports the following configuration options plus the [Common options](#filebeat-input-http_endpoint-common-options) described later. ### `basic_auth` Enables or disables HTTP basic auth for each incoming request. If enabled then `username` and `password` will also need to be configured. ### `username` If `basic_auth` is enabled, this is the username used for authentication against the HTTP listener. Requires `password` to also be set. ### `password` If `basic_auth` is enabled, this is the password used for authentication against the HTTP listener. Requires `username` to also be set. ### `secret.header` The header to check for a specific value specified by `secret.value`. Certain webhooks provide the possibility to include a special header and secret to identify the source. ### `secret.value` The secret stored in the header name specified by `secret.header`. Certain webhooks provide the possibility to include a special header and secret to identify the source. ### `hmac.header` The name of the header that contains the HMAC signature: `X-Dropbox-Signature`, `X-Hub-Signature-256`, etc. HMAC signatures may be encoded as hex or base64. ### `hmac.key` The secret key used to calculate the HMAC signature. Typically, the webhook sender provides this value. ### `hmac.type` The hash algorithm to use for the HMAC comparison. At this time the only valid values are `sha256` or `sha1`. ### `hmac.prefix` The prefix for the signature. Certain webhooks prefix the HMAC signature with a value, for example `sha256=`. ### `content_type` By default the input expects the incoming POST to include a Content-Type of `application/json` to try to enforce the incoming data to be valid JSON. In certain scenarios when the source of the request is not able to do that, it can be overwritten with another value or set to null. ### `max_in_flight_bytes` The hard limit on the total sum of request body bytes that are allowed to be in-flight at any given time. If this limit is exceeded during body reading, the request is terminated with a 503 Service Unavailable response and a `Retry-After` header. This serves as a safety valve to prevent memory exhaustion. The default value is zero, meaning no limit. In-flight bytes are tracked from the moment they are read from the request body until the event is acknowledged by the output (for requests with `wait_for_completion_timeout`) or until the request completes (for requests without the timeout parameter). Note that in-flight byte tracking uses the raw request body size as a heuristic proxy for memory consumption. For requests without `wait_for_completion_timeout`, the byte count is released when the HTTP request completes, even though published events may still be queued in the output pipeline. This means the in-flight count can underestimate actual memory use under sustained load from non-ACK requests. Using `wait_for_completion_timeout` provides tighter accounting because the byte count is held until the output acknowledges the events. This option works together with `high_water_in_flight_bytes` and `low_water_in_flight_bytes` to implement hysteresis-based admission control. See those options for details. ### `high_water_in_flight_bytes` Elastic Stack: Generally available since 9.2 Elastic Stack: Generally available since 8.19 The soft limit threshold for in-flight bytes. When in-flight bytes exceed this value, new requests are rejected with a 503 Service Unavailable response (with a `Retry-After` header). Once in-flight bytes drop below `low_water_in_flight_bytes`, new requests are accepted again. This hysteresis mechanism prevents rapid oscillation between accepting and rejecting requests when the system is near capacity. If not specified and `max_in_flight_bytes` is set, this defaults to 50% of `max_in_flight_bytes`. ### `low_water_in_flight_bytes` Elastic Stack: Generally available since 9.2 Elastic Stack: Generally available since 8.19 The threshold below which new requests are accepted again after being rejected due to exceeding `high_water_in_flight_bytes`. This creates a hysteresis band that prevents rapid state transitions. If not specified and `max_in_flight_bytes` is set, this defaults to the lesser of 80% of `high_water_in_flight_bytes` or `high_water_in_flight_bytes` minus 64kB. The relationship between the three limits must be: `low_water_in_flight_bytes` < `high_water_in_flight_bytes` < `max_in_flight_bytes`. ### `retry_after` Elastic Stack: Generally available since 9.2 Elastic Stack: Generally available since 8.19 When a request is rejected due to exceeding `high_water_in_flight_bytes` or `max_in_flight_bytes`, the response includes a `Retry-After` header specifying how many seconds the client should wait before retrying. For `high_water_in_flight_bytes` rejection the configured value is used directly; for `max_in_flight_bytes` rejection 2x the configured value is used. The default value is 10 seconds. ### `program` The normal operation of the input treats the body either as a single event when the body is an object, or as a set of events when the body is an array. If the body should be handled differently, for example a set of events in an array field of an object to be handled as a set of events, then a [Common Expression Language (CEL)](https://opensource.google.com/projects/cel) program can be provided through this configuration field. The name of the object in the CEL program is `obj`. No CEL extensions are provided beyond the function in the CEL [standard library](https://github.com/google/cel-spec/blob/master/doc/langdef.md#standard). CEL [optional types](https://pkg.go.dev/github.com/google/cel-go/cel#OptionalTypes) are supported. Note that during evaluation, numbers that are not representable exactly within a double floating point value will be converted to a string to avoid data corruption. ### `response_code` The HTTP response code returned upon success. Should be in the 2XX range. ### `response_body` The response body returned upon success. ### `options_headers` A set of response headers to add to the response for OPTIONS requests. Headers with the same canonical MIME header name will be replaced with the values in this configuration. ### `options_response_code` The HTTP response code to return for OPTIONS requests. Defaults to `200` (OK). ### `listen_address` If multiple interfaces is present the `listen_address` can be set to control which IP address the listener binds to. Defaults to `127.0.0.1`. ### `listen_port` Which port the listener binds to. Defaults to 8000. ### `url` This options specific which URL path to accept requests on. Defaults to `/` ### `prefix` This option specifies which prefix the incoming request will be mapped to. If `prefix` is "`.`", the request will be mapped to the root of the resulting document. ### `include_headers` This options specifies a list of HTTP headers that should be copied from the incoming request and included in the document. All configured headers will always be canonicalized to match the headers of the incoming request. For example, `["content-type"]` will become `["Content-Type"]` when the filebeat is running. ### `preserve_original_event` This option includes the JSON representation of the incoming request in the `event.original` field as a string before sending the event to Elasticsearch. The representation may not be a verbatim copy of the original message, but is guaranteed to be an [RFC7493](https://datatracker.ietf.org/doc/html/rfc7493) compliant message. ### `crc.provider` This option defines the provider of the webhook that uses CRC (Challenge-Response Check) for validating the endpoint. The HTTP endpoint input is responsible for ensuring the authenticity of incoming webhook requests by generating and verifying a unique token. By specifying the `crc.provider`, you ensure that the system correctly handles the specific CRC validation process required by the chosen provider. ### `crc.secret` The secret token provided by the webhook owner for the CRC validation. It is required when a `crc.provider` is set. ### `method` The HTTP method handled by the endpoint. If specified, `method` must be `POST`, `PUT` or `PATCH`. The default method is `POST`. If `PUT` or `PATCH` are specified, requests using those method types are accepted, but are treated as `POST` requests and are expected to have a request body containing the request data. ### `tracer.enabled` It is possible to log HTTP requests to a local file-system for debugging configurations. This option is enabled by setting `tracer.enabled` to true and setting the `tracer.filename` value. Additional options are available to tune log rotation behavior. To delete existing logs, set `tracer.enabled` to false without unsetting the filename option. Enabling this option compromises security and should only be used for debugging. ### `tracer.filename` To differentiate the trace files generated from different input instances, a placeholder `*` can be added to the filename and will be replaced with the input instance id. For Example, `http-request-trace-*.ndjson`. The path must point to a target in the http_endpoint directory in the [Filebeat logs directory](https://www.elastic.co/docs/reference/beats/filebeat/directory-layout). ### `tracer.maxsize` This value sets the maximum size, in megabytes, the log file will reach before it is rotated. By default logs are allowed to reach 1MB before rotation. Individual request bodies will be truncated to a maximum size of 10kiB. ### `tracer.maxage` This specifies the number days to retain rotated log files. If it is not set, log files are retained indefinitely. ### `tracer.maxbackups` The number of old logs to retain. If it is not set all old logs are retained subject to the `tracer.maxage` setting. ### `tracer.localtime` Whether to use the host’s local time rather that UTC for timestamping rotated log file names. ### `tracer.compress` This determines whether rotated logs should be gzip compressed. ## Metrics This input exposes metrics under the [HTTP monitoring endpoint](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/beats/filebeat/http-endpoint). These metrics are exposed under the `/inputs` path. They can be used to observe the activity of the input. | Metric | Description | |---------------------------|-------------------------------------------------------------------------------------------------------------------------------------| | `bind_address` | Bind address of input. | | `route` | HTTP request route of the input. | | `is_tls_connection` | Whether the input is listening on a TLS connection. | | `api_errors_total` | Number of API errors. | | `batches_received_total` | Number of event arrays received. | | `batches_published_total` | Number of event arrays published. | | `batches_acked_total` | Number of event arrays ACKed. | | `events_published_total` | Number of events published. | | `size` | Histogram of request content lengths. | | `batch_size` | Histogram of the received event array length. | | `batch_processing_time` | Histogram of the elapsed successful batch processing times in nanoseconds (time of receipt to time of ACK for non-empty batches). | | `batch_ack_time` | Histogram of the elapsed successful batch ACKing times in nanoseconds (time of handler start to time of ACK for non-empty batches). | ## Common options The following configuration options are supported by all inputs. #### `enabled` Use the `enabled` option to enable and disable inputs. By default, enabled is set to true. #### `tags` A list of tags that Filebeat includes in the `tags` field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration. Example: ```yaml filebeat.inputs: - type: http_endpoint . . . tags: ["json"] ``` #### `fields` Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a `fields` sub-dictionary in the output document. To store the custom fields as top-level fields, set the `fields_under_root` option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here. ```yaml filebeat.inputs: - type: http_endpoint . . . fields: app_id: query_engine_12 ``` #### `fields_under_root` If this option is set to true, the custom [fields](#filebeat-input-http_endpoint-fields) are stored as top-level fields in the output document instead of being grouped under a `fields` sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields. #### `processors` A list of processors to apply to the input data. See [Processors](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/beats/filebeat/filtering-enhancing-data) for information about specifying processors in your config. #### `pipeline` The ingest pipeline ID to set for the events generated by this input. The pipeline ID can also be configured in the Elasticsearch output, but this option usually results in simpler configuration files. If the pipeline is configured both in the input and output, the option from the input is used. The `pipeline` is always lowercased. If `pipeline: Foo-Bar`, then the pipeline name in Elasticsearch needs to be defined as `foo-bar`. #### `keep_null` If this option is set to true, fields with `null` values will be published in the output document. By default, `keep_null` is set to `false`. #### `index` If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the `raw_index` field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use `output.elasticsearch.index` or a processor. Example value: `"%{[agent.name]}-myindex-%{+yyyy.MM.dd}"` might expand to `"filebeat-myindex-2019.11.01"`. #### `publisher_pipeline.disable_host` By default, all events contain `host.name`. This option can be set to `true` to disable the addition of this field to all events. The default value is `false`.