Loading

Elasticsearch breaking changes

Breaking changes can impact your Elastic applications, potentially disrupting normal operations. Before you upgrade, carefully review the Elasticsearch breaking changes and take the necessary steps to mitigate any issues.

If you are migrating from a version prior to version 9.0, you must first upgrade to the last 8.x version available. To learn how to upgrade, check out Upgrade.

ILM: * In earlier versions of {es} the downsampling API was force merging the downsampled index. In {es} 9.3.0 and later we moved this behaviour from the downsampling API to the ILM downsampling action and we enabled by default. This allowed users to disabled it if they wished. Seeing that the force merge is not inherently needed by the downsampling operation, in {es} 9.4.0, we choose to disable it by default. This change only affects the downsampling ILM actions; other actions that force merge an index by default, such as the searchable snapshot action, will remain unchanged. If a user was relying on this behaviour they will need to update their ILM policy, either by adding a force merge action in the same or a subsequent phase of the policy or by setting the force_merge_index parameter of the downsample action to true. #145400 (issue: #140811)

Mapping:

  • OTLP endpoint: use exponential_histograms by default #145065

TSDB: * Sequence numbers have significant storage footprint in metrics applications (up to 30% for OTLP). To alleviate this, we are disabling sequence numbers for indices in time-series mode, meaning that they get trimmed once they are no longer needed for replication. This helps with storage efficiency, as well as reducing the overhead for segment merging. The downside is that Optimistic Concurrency Control is no longer applicable to these indices. Index, update and delete operations using if_seq_no error out, and search calls with seq_no_primary_term set return sentinel values for sequence numbers. Moreover, update_by_query and delete_by_query operations proceed without conflict detection, so concurrent modifications may silently overwrite the affected docs without triggering version conflict errors. This is largely a favorable tradeoff as concurrent modifications are rather rare in metrics applications and the storage savings are substantial. Still, users requiring OCC support can restore the old behavior by setting index.disable_sequence_numbers to false in their time-series index templates. #145737 (issue: #136305)

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

Mapping:

  • Implement native synthetic source for normalized keywords #136915

Search:

  • Disable cross-cluster functionality for _fleet/_fleet_msearch #136703
  • Disable cross-cluster functionality for _fleet/_fleet_search #136039

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

Ingest Node:

  • Simulate API: Return 400 on invalid processor(s) #130325 (issue: #120731)

Mapping:

  • Don't enable norms for fields of type text when the index mode is LogsDB or TSDB #131317

Vector Search:

  • Enable exclude_source_vectors by default for new indices #131907

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

Discovery-Plugins:

Migrates `discovery-ec2` plugin to AWS SDK v2

The discovery-ec2 plugin now uses AWS SDK v2 instead of v1, as AWS plans to deprecate SDK v1 before the end of Elasticsearch 8.19’s support period. AWS SDK v2 introduces several behavior changes that affect configuration.

Impact: If you use the discovery-ec2 plugin, your existing settings may no longer be compatible. Notable changes include, but may not be limited to:

  • AWS SDK v2 does not support the EC2 IMDSv1 protocol.
  • AWS SDK v2 does not support the aws.secretKey or com.amazonaws.sdk.ec2MetadataServiceEndpointOverride system properties.
  • AWS SDK v2 does not permit specifying a choice between HTTP and HTTPS so the discovery.ec2.protocol setting is no longer effective.
  • AWS SDK v2 does not accept an access key without a secret key or vice versa.

Action: Test the upgrade in a non-production environment. Adapt your configuration to the new SDK functionality. This includes, but may not be limited to, the following items:

  • If you use IMDS to determine the availability zone of a node or to obtain credentials for accessing the EC2 API, ensure that it supports the IMDSv2 protocol.
  • If applicable, discontinue use of the aws.secretKey and com.amazonaws.sdk.ec2MetadataServiceEndpointOverride system properties.
  • If applicable, specify that you wish to use the insecure HTTP protocol to access the EC2 API by setting discovery.ec2.endpoint to a URL which starts with http://.
  • Either supply both an access key and a secret key using the discovery.ec2.access_key and discovery.ec2.secret_key keystore settings, or configure neither of these settings.

For more information, view #122062.

ES|QL:

ES|QL now returns partial results by default

In previous versions, ES|QL queries failed entirely when any error occurred. As of 8.19.0, ES|QL returns partial results instead.

Impact: Callers must check the is_partial flag in the response to determine whether the result is complete. Relying on full results without checking this flag may lead to incorrect assumptions about the response.

Action: If partial results are not acceptable for your use case, you can disable this behavior by:

  • Setting allow_partial_results=false in the query URL per request, or
  • Setting the esql.query.allow_partial_results cluster setting to false.

For more information, view #127351 (issue: #122802)

Disallows parentheses in unquoted index patterns in ES|QL

To avoid ambiguity with subquery syntax, ES|QL no longer allows the use of ( and ) in unquoted index patterns.

Impact: Queries that include parentheses in unquoted index names will now result in a parsing exception.

Action: Update affected queries to quote index names that contain parentheses. For example, use FROM "("foo")" instead of FROM (foo). For more information, view #130427 (issue: #130378)

Disallows mixing quoted and unquoted components in `FROM` index patterns

ES|QL no longer allows mixing quoted and unquoted parts in FROM index patterns (e.g. FROM remote:"index"). Previously, such patterns were parsed inconsistently and could result in misleading runtime errors.

Impact: Queries using partially quoted index patterns—such as quoting only the index or only the remote cluster—will now be rejected at parse time. This change simplifies grammar handling and avoids confusing validation failures.

Action: Ensure index patterns are either fully quoted or fully unquoted. For example:

  • Valid: FROM "remote:index" or FROM remote:index
  • Invalid: FROM remote:"index", FROM "remote":index

For more information, view #127636 (issue: #122651)

`skip_unavailable` now catches all remote cluster runtime errors in ES|QL

When skip_unavailable is set to true, ES|QL now treats all runtime errors from that cluster as non-fatal. Previously, this setting only applied to connectivity issues (i.e. when a cluster was unavailable).

Impact: Errors such as missing indices on a remote cluster will no longer cause the query to fail. Instead, the cluster will appear in the response metadata as skipped or partial.

Action: If your workflows rely on detecting remote cluster errors, review your use of skip_unavailable and adjust error handling as needed.

For more information, view #128163

Snapshot/Restore:

Upgrades `repository-s3` plugin to AWS SDK v2

The repository-s3 plugin now uses AWS SDK v2 instead of v1, as AWS will deprecate SDK v1 before the end of Elasticsearch 8.19’s support period. The two SDKs differ in behavior, which may require updates to your configuration.

Impact: Existing repository-s3 configurations may no longer be compatible. Notable differences in AWS SDK v2 include, but may not be limited to:

  • AWS SDK v2 requires users to specify the region to use for signing requests, or else to run in an environment in which it can determine the correct region automatically. The older SDK used to determine the region based on the endpoint URL as specified with the s3.client.${CLIENT_NAME}.endpoint setting, together with other data drawn from the operating environment, and fell back to us-east-1 if no better value was found.
  • AWS SDK v2 does not support the EC2 IMDSv1 protocol.
  • AWS SDK v2 does not support the com.amazonaws.sdk.ec2MetadataServiceEndpointOverride system property.
  • AWS SDK v2 does not permit specifying a choice between HTTP and HTTPS so the s3.client.${CLIENT_NAME}.protocol setting is deprecated and no longer has any effect.
  • AWS SDK v2 does not permit control over throttling for retries, so the s3.client.${CLIENT_NAME}.use_throttle_retries setting is deprecated and no longer has any effect.
  • AWS SDK v2 requires the use of the V4 signature algorithm, therefore, the s3.client.${CLIENT_NAME}.signer_override setting is deprecated and no longer has any effect.
  • AWS SDK v2 does not support the log-delivery-write canned ACL.
  • AWS SDK v2 counts 4xx responses differently in its metrics reporting.
  • AWS SDK v2 always uses the regional STS endpoint, whereas AWS SDK v2 could use either a regional endpoint or the global https://sts.amazonaws.com one.

Action: Test the upgrade in a non-production environment. Adapt your configuration to the new SDK functionality. This includes, but may not be limited to, the following items:

  • Specify the correct signing region using the s3.client.${CLIENT_NAME}.region setting on each node. {es} will try to determine the correct region based on the endpoint URL and other data drawn from the operating environment, but might not do so correctly in all cases.
  • If you use IMDS to determine the availability zone of a node or to obtain credentials for accessing the EC2 API, ensure that it supports the IMDSv2 protocol.
  • If applicable, discontinue use of the com.amazonaws.sdk.ec2MetadataServiceEndpointOverride system property.
  • If applicable, specify that you wish to use the insecure HTTP protocol to access the S3 API by setting s3.client.${CLIENT_NAME}.endpoint to a URL which starts with http://.
  • If applicable, discontinue use of the log-delivery-write canned ACL.

For more information, view #126843 (issue: #120993)

There are no breaking changes associated with this release.

There are no breaking changes associated with this release.

Snapshot/Restore:

  • Make S3 custom query parameter optional #128043

There are no breaking changes associated with this release.

Aggregations:

  • Remove date histogram boolean support #118484

Allocation:

  • Increase minimum threshold in shard balancer #115831
  • Remove cluster.routing.allocation.disk.watermark.enable_for_single_data_node setting #114207
  • Remove cluster state from /_cluster/reroute response #114231 (issue: #88978)

Analysis:

  • Snowball stemmers have been upgraded #114146
  • The 'german2' stemmer is now an alias for the 'german' snowball stemmer #113614
  • The 'persian' analyzer has stemmer by default #113482 (issue: #113050)
  • The Korean dictionary for Nori has been updated #114124

Authentication:

  • Configuring a bind DN in an LDAP or Active Directory (AD) realm without a corresponding bind password will prevent node from starting #118366

Cluster Coordination:

  • Remove unsupported legacy value for discovery.type #112903

EQL:

  • Set allow_partial_search_results=true by default #120267

Extract&Transform:

  • Restrict Connector APIs to manage/monitor_connector privileges #119863

Highlighting:

  • Remove support for deprecated force_source highlighting parameter #116943

Indices APIs:

  • Apply more strict parsing of actions in bulk API #115923
  • Remove deprecated local attribute from alias APIs #115393
  • Remove the ability to read frozen indices #120108
  • Remove unfreeze REST endpoint #119227

Infra/Core:

  • Change Elasticsearch timeouts to 429 response instead of 5xx #116026
  • Limit ByteSizeUnit to 2 decimals #120142
  • Remove client.type setting #118192 (issue: #104574)
  • Remove any references to org.elasticsearch.core.RestApiVersion#V_7 #118103

Infra/Logging:

  • Change deprecation.elasticsearch keyword to elasticsearch.deprecation #117933 (issue: #83251)
  • Rename deprecation index template #125606 (issue: #125445)

Infra/Metrics:

  • Deprecated tracing.apm.* settings got removed. #119926

Infra/REST API:

  • Output a consistent format when generating error json #90529 (issue: #89387)

Ingest Node:

  • Remove ecs option on user_agent processor #116077
  • Remove ignored fallback option on GeoIP processor #116112

Logs:

  • Conditionally enable logsdb by default for data streams matching with logs-- pattern. #121049 (issue: #106489)

Machine Learning:

  • Disable machine learning on macOS x86_64 #104125

Mapping:

  • Remove support for type, fields, copy_to and boost in metadata field definition #118825
  • Turn _source meta fieldmapper's mode attribute into a no-op #119072 (issue: #118596)

Search:

  • Adjust random_score default field to _seq_no field #118671
  • Change Semantic Text To Act Like A Normal Text Field #120813
  • Remove legacy params from range query #116970

Snapshot/Restore:

  • Remove deprecated xpack.searchable.snapshot.allocate_on_rolling_restart setting #114202

TLS:

  • Drop TLS_RSA cipher support for JDK 24 #123600
  • Remove TLSv1.1 from default protocols #121731

Transform:

  • Remove data_frame_transforms roles #117519

Vector Search:

  • Remove old _knn_search tech preview API in v9 #118104

Watcher:

  • Removing support for types field in watcher search #120748