Logstash breaking changes

The bundled Kafka integration plugin has been upgraded from 11.x to 12.x, which uses Apache Kafka client 4.x. This introduces the following breaking changes:

• The partitioner options default and uniform_sticky have been removed. Remove any explicit partitioner setting that uses these values; the Kafka client now uses its own internal default strategy.
• The linger_ms default value has changed from 0 to 5. If your output relies on zero-latency batching, explicitly set linger_ms => 0.

For more information, check #205.

Impact
Kafka output configurations that explicitly set partitioner => "default" or partitioner => "uniform_sticky" will fail to start. Configurations relying on the previous linger_ms default of 0 may observe slightly increased latency (up to 5ms) in exchange for better batching.

Action
Remove or update any partitioner setting that uses the removed values. If zero-latency sending is required, explicitly set linger_ms => 0 in your Kafka output configuration.

The deprecated concurrency :legacy output mode has been removed. This mode was superseded by concurrency :single and concurrency :shared in Logstash 6.x.

For more information, check #19003.

Impact
Any third-party output plugin that still declares concurrency :legacy will fail to load.

Action
Update affected plugins to declare concurrency :single or concurrency :shared.

The logstash-output-pipe plugin is no longer included in the default Logstash plugin bundle. Pipelines that use output { pipe { ... } } will fail to start after upgrading.

For more information, check #19159.

Impact
Any pipeline configuration using the pipe output will fail to start with a plugin-not-found error.

Action
Install the plugin manually before upgrading: bin/logstash-plugin install logstash-output-pipe.

JDK 21 is now the minimum version of the JDK required to run Logstash. This release upgrades JRuby from 9.x to 10.x, which requires JDK 21 or later. For the best experience, we still recommend running Logstash using the bundled-jdk. See Logstash JVM requirements for details #19010.

We’ve removed deprecated SSL settings in some Logstash plugins, and have replaced them with updated settings. If your plugin configuration contains any of these obsolete options, the plugin may fail to start.

We've improved memory configuration for certain Logstash plugins. Input plugins such as elastic_agent, beats, tcp, and http allocate buffers in Java memory to read events from the network. The default allocation method is direct memory rather than heap memory to simplify configuration, and to help facilitate debugging memory usage problems through the analysis of heap dumps. If you need to re-enable the previous behavior, change the pipeline.buffer.type setting in logstash.yml. Check out off-heap-buffers-allocation for details. #16500

We have removed the Logstash modules framework, and encourage users to try Elastic Integrations This includes the netflow, azure and arcsight modules, and the modules framework as a whole. #16794

We have removed support for previously deprecated configuration settings:

• http.* prefixed settings for the Logstash API. Settings prefixed by http.* have been replaced by the equivalent settings prefixed with api.*. #16552

• event_api.tags.illegal Any events that include field named tags automatically rename the field _tags to avoid any clash with the reserved Logstash tags field. Instead, Logstash generates _tagsparsefailure in the event tags and the illegal value is written to the _tags field. #16461

The ingest converter, which was used to convert ingest pipelines to Logstash pipelines, has been removed. #16453

The logstash-filter-elastic_integration plugin offers similar functionality, and can help you use Logstash to extend Elastic integrations.

JDK17 is the minimum version of the JDK required to run Logstash. For the best experience, we still recommend running Logstash using the bundled-jdk. See Logstash JVM requirements for details. #16443

The base image for Logstash on Docker has been changed from Ubuntu to UBI9. If you create a Docker image based on the Logstash image and rely on it being Ubuntu based, you need to change your derived image to take account of this change. For example, if your derived docker image installs additional packages using a package manager, UBI9 uses microdnf, rather than apt. #16599

We've changed the default behavior to prevent users from accidentally running Logstash as a superuser. If you try to run Logstash as a superuser, it logs an error and fails to start, ensuring that users cannot run Logstash with elevated privileges by accident.

You can change the value of the allow_superuser setting to true in logstash.yml if you want to restore the previous behavior and allow Logstash to run with superuser privileges. #16558

To continue using deprecated internal collection to monitor Logstash, set xpack.monitoring.allow_legacy_collection to true in logstash.yml. We encourage you to move to agent-driven monitoring, the latest, supported way to monitor Logstash #16586

We've improved the way we deal with duplicate message fields in json documents. Some code paths that log in json produce log events that include multiple instances of the message field. (The JSON codec plugin is one example.) While this approach produces JSON that is technically valid, many clients do not parse this data correctly, and either crash or discard one of the fields.

We recently introduced the option to fix duplicates, and made it the default behavior for 9.0 and later. To re-enable the previous behavior, change the log.format.json.fix_duplicate_message_fields setting in logstash.yml to false.

Check out Logging in json format can write duplicate message fields for more details about the issue. #16578

We’ve removed the Logstash Enterprise_search integration plugin, and its component App Search and Workplace Search plugins from the default Logstash install. These plugins will receive only security updates and critical fixes moving forward.

We recommend using our native Elasticsearch tools for your Search use cases. For more details, please visit the Search solution and use case documentation.