Elastic APM breaking changes

This will surface any agent misconfiguration causing data to be sent to / instead of the correct endpoint (for example, /v1/traces for OTLP/HTTP). For more information, check #15976.

Impact
Any methods other than GET and HEAD to / will return HTTP 405 Method Not Allowed.

Action
Update any existing usage, for example, update POST / to GET /.

The Elasticsearch apm_user role has been removed. For more information, check #14876.

Impact
If you are relying on the apm_user to provide access, users may lose access when upgrading to the next version.

Action
After this change if you are relying on apm_user you will need to specify its permissions manually.

The sampling.tail.storage_limit default value changed to 0. While 0 means unlimited local tail-sampling database size, it now enforces a max 80% disk usage on the disk where the data directory is located. Any tail sampling writes that occur after this threshold will be rejected, similar to what happens when tail-sampling database size exceeds a non-0 storage limit. Setting sampling.tail.storage_limit to non-0 maintains the existing behavior, which limits the tail-sampling database size to sampling.tail.storage_limit and does not have the new disk usage threshold check. For more information, check #15467 and #15524.

Impact
If sampling.tail.storage_limit is already set to a non-0 value, tail sampling will maintain the existing behavior. If you're using the new default, it will automatically scale with a larger disk.

Action
To continue using the existing behavior, set the sampling.tail.storage_limit to a non-0 value. To use the new disk usage threshold check, set the sampling.tail.storage_limit to 0 (the default value).

Due to a change in the underlying tail-based sampling database, the 8.x database files are ignored when running APM Server 9.0+.

Impact
There is a limited, temporary impact around the time when an upgrade happens. Any locally stored events awaiting tail sampling decision before upgrading to 9.0 will effectively be ignored, as if their traces are unsampled. If there are many APM Server making tail sampling decisions, it may result in broken traces.

Action
No action needed. There is no impact on traces ingested after upgrade.

As tail-based sampling database files from version 8.x are now ignored and consume unnecessary disk space, some files can be deleted to reclaim disk space. It does not affect Elastic Cloud Hosted, as storage is automatically cleaned up by design.

Impact
Unnecessary disk usage, except for Elastic Cloud Hosted users.

Action
To clean up the unnecessary files, first identify APM Server data path, configured via path.data, which is also usually printed with "Data path: " in APM Server logs on startup. Assuming the environment variable PATH_DATA is the data path for TBS we identified above, the files that should be deleted are KEYREGISTRY, LOCK, MANIFEST, *.vlog, *.sst under $PATH_DATA/tail_sampling/. Do not delete other files.

APM Server no longer supports the agent configuration fetching mechanism where Fleet injects agent configuration into the APM Server at regular intervals. The default behavior is now to fetch agent configuration direclty from Elasticsearch. Fetching the agent configuration from Kibana is still supported when APM Server is configured without an Elasticsearch output.

For more information, check #14539.

The apiKey subcommand has been removed from the apm-server binary.

For more information, check #14127.

Impact
Creating API keys for ingestion is not supported through the CLI.

Action
Use Kibana to create API keys for ingestion.

TLS version 1.1 protocol is not included anymore in the list of supported protocols in the default apm-server.yml configuration file. TLS 1.1 is considered insecure and is not recommended for use. The ssl.supported_protocols setting now defaults to [TLSv1.2, TLSv1.3].

For more information, check #14790.

Impact
The default configuration doesn't allow TLSv1.1 connections.

Action
Modify the ssl.supported_protocols setting to include TLSv1.1 if needed.

the following deprecated options in the RUM configuration have been removed:

• event_rate.limit
• event_rate.lru_size
• allow_service_names

Impact
The aforementioned configurations are ignored.

Action
Refer to the RUM section in APM documentation to use non-deprecated options.

The error.grouping_key field in events processed by APM Server is now calculated using xxhash instead of md5. Thus, the format of the error.grouping_key field has changed, making it different with documents ingested in previous versions.

For more information, check #15211.

Impact
The error.grouping_key field has a different format than previous versions.

Jaeger API support was deprecated in 8.16. APM Server no longer supports the Jaeger API for ingesting distributed traces.

For more information, check #14791.

Impact
Events cannot be ingested anymore with the Jaeger gRPC API.

Action
Migrate to APM or OTLP.