Configure cluster settings for ES|QL Data Federation
The data sources feature adds the following cluster settings. For general guidance on how to apply cluster settings across deployment types, refer to configure Elasticsearch.
The object-count limits and authentication gates are operator-managed and take effect without a restart. Settings marked Dynamic can also be updated at runtime. The remaining settings require a node restart.
Some of these settings were renamed. Where a row lists two names, the badge on each shows which versions accept it. Unless a note says otherwise, the older name is no longer registered, and a node that still sets it in elasticsearch.yml fails to start.
These settings cap the number of data sources and datasets a cluster can hold.
| Setting | Default | Description |
|---|---|---|
esql.data_sources.max_count |
100 | Maximum number of data sources that can be defined. Range 0–1000. |
esql.datasets.max_count |
1000 | Maximum number of datasets that can be defined. Range 0–10,000. |
These settings control how many concurrent requests each node sends to external storage and how long it retries throttled requests.
| Setting | Default | Description |
|---|---|---|
esql.external.max_concurrent_requests |
allocated processors * 3, minimum 16 and maximum 100 |
Maximum concurrent cloud API requests per storage scheme, per node. 0 removes the limit. Range 0–500. |
esql.external.throttle_max_retry_duration |
30 | Maximum total time, in seconds, spent retrying throttled cloud API requests before failing the query. 0 removes the budget. Range 0–300 seconds. |
esql.external.max_concurrent_segmenters
esql.external.max_concurrent_segmentators
|
0 |
Maximum number of file segmentation tasks that run concurrently. 0 derives the value automatically. Range 0–4096. |
These settings bound file discovery for glob patterns. Exceeding the file cap aborts the query; exceeding the brace-expansion cap falls back to listing the storage.
| Setting | Default | Description |
|---|---|---|
esql.external.max_discovered_files |
10,000 | Hard cap on files collected by glob expansion before the query aborts. Protects against degenerate globs. Range 1–1,000,000. Dynamic. |
esql.external.max_glob_expansion |
100 | Cap on concrete paths generated by brace expansion before falling back to listing. Range 1–10,000. Dynamic. |
These settings control which authentication modes data sources can use.
| Setting | Default | Description |
|---|---|---|
esql.external.managed_identity.enabled
esql.datasource.managed_identity.enabled
|
false | Enables auth: "managed_identity" (the node's own cloud identity through the instance metadata service (IMDS)). Operator-only. Intended for single-cloud, single-tenant deployments. Never enable in serverless or multi-tenant clusters. Refer to the managed identity row in authentication models for guidance. |
esql.external.federated_identity.enabled
esql.datasource.federated_identity.enabled
|
false | Enables auth: "federated_identity" (OIDC-to-STS token exchange). Operator-only. Available on Elastic Cloud Hosted and Serverless; not available on self-managed, Elastic Cloud Enterprise, or Elastic Cloud on Kubernetes. For setup details, refer to connect with federated identity. |
These settings control the external-source cache, which stores inferred schemas and file listings.
| Setting | Default | Description |
|---|---|---|
esql.external.cache.enabled
esql.source.cache.enabled
|
true | Enables the external-source cache (inferred schemas and file listings). Dynamic. |
esql.external.cache.size
esql.source.cache.size
|
0.4% of heap | Memory budget for the cache. Applied at node startup only. |
esql.source.cache.schema.ttl |
— | Deprecated and ignored. Inferred schemas are invalidated by file identity and bounded by the cache memory budget, not by a TTL. |
esql.external.cache.listing.ttl
esql.source.cache.listing.ttl
|
30s | How long a file-listing result is cached. Applied at node startup only. |
The esql.source.cache.* keys are accepted as deprecated fallbacks and emit a deprecation warning. You can set them in elasticsearch.yml, but you can't update them through the cluster settings API. Use the esql.external.cache.* keys for new configuration.