---
title: Configure cluster settings for ES|QL Data Federation
description: Reference for ES|QL Data Federation cluster settings. Tune object limits, control request concurrency, and adjust file discovery and caching.
url: https://www.elastic.co/elastic/docs-builder/docs/4024/reference/query-languages/esql/esql-data-federation-cluster-settings
products:
- Elasticsearch
applies_to:
- Elastic Cloud Serverless: Unavailable
- Elastic Stack: Experimental since 9.5
---
# Configure cluster settings for ES|QL Data Federation
The data sources feature adds the following cluster settings. For general guidance on how to apply cluster settings across deployment types, refer to [configure Elasticsearch](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/4024/deploy-manage/stack-settings).
The object-count limits and authentication gates are operator-managed and take effect without a restart. Settings marked Dynamic can also be updated at runtime. The remaining settings require a node restart.
Some of these settings were renamed. Where a row lists two names, the badge on each shows which versions accept it. Unless a note says otherwise, the older name is no longer registered, and a node that still sets it in `elasticsearch.yml` fails to start.
## Object limits
These settings cap the number of data sources and datasets a cluster can hold.
| Setting | Default | Description |
|-------------------------------|---------|-------------------------------------------------------------------|
| `esql.data_sources.max_count` | 100 | Maximum number of data sources that can be defined. Range 0–1000. |
| `esql.datasets.max_count` | 1000 | Maximum number of datasets that can be defined. Range 0–10,000. |
## Request concurrency and throttling
These settings control how many concurrent requests each node sends to external storage and how long it retries throttled requests.
| Setting | Default | Description |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
| `esql.external.max_concurrent_requests` | `allocated processors * 3`, minimum 16 and maximum 100 | Maximum concurrent cloud API requests per storage scheme, per node. `0` removes the limit. Range 0–500. |
| `esql.external.throttle_max_retry_duration` | 30 | Maximum total time, in seconds, spent retrying throttled cloud API requests before failing the query. `0` removes the budget. Range 0–300 seconds. |
| `esql.external.max_concurrent_segmenters` Elastic Stack: Planned`esql.external.max_concurrent_segmentators` Elastic Stack: Experimental in 9.5 | `0` | Maximum number of file segmentation tasks that run concurrently. `0` derives the value automatically. Range 0–4096. |
## Glob and file-discovery limits
These settings bound file discovery for glob patterns. Exceeding the file cap aborts the query; exceeding the brace-expansion cap falls back to listing the storage.
| Setting | Default | Description |
|--------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------|
| `esql.external.max_discovered_files` | 10,000 | Hard cap on files collected by glob expansion before the query aborts. Protects against degenerate globs. Range 1–1,000,000. Dynamic. |
| `esql.external.max_glob_expansion` | 100 | Cap on concrete paths generated by brace expansion before falling back to listing. Range 1–10,000. Dynamic. |
## Authentication
These settings control which authentication modes data sources can use.
| Setting | Default | Description |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `esql.external.managed_identity.enabled` Elastic Stack: Planned`esql.datasource.managed_identity.enabled` Elastic Stack: Planned for deprecation, Elastic Stack: Experimental in 9.5 | false | Enables `auth: "managed_identity"` (the node's own cloud identity through the instance metadata service (IMDS)). Operator-only. Intended for single-cloud, single-tenant deployments. Never enable in serverless or multi-tenant clusters. Refer to the managed identity row in [authentication models](/elastic/docs-builder/docs/4024/reference/query-languages/esql/esql-data-federation-sources#authentication) for guidance. |
| `esql.external.federated_identity.enabled` Elastic Stack: Planned`esql.datasource.federated_identity.enabled` Elastic Stack: Planned for deprecation, Elastic Stack: Experimental in 9.5 | false | Enables `auth: "federated_identity"` (OIDC-to-STS token exchange). Operator-only. Available on Elastic Cloud Hosted and Serverless; not available on self-managed, Elastic Cloud Enterprise, or Elastic Cloud on Kubernetes. For setup details, refer to [connect with federated identity](https://www.elastic.co/elastic/docs-builder/docs/4024/reference/query-languages/esql/esql-data-federation-federated-identity). |
## Caching
These settings control the external-source cache, which stores inferred schemas, file listings, and the footers of columnar files.
| Setting | Default | Description |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `esql.external.cache.enabled` Elastic Stack: Planned`esql.source.cache.enabled` Elastic Stack: Planned for deprecation, Elastic Stack: Experimental in 9.5 | true | Enables the external-source cache (inferred schemas and file listings). [Dynamic](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/4024/deploy-manage/stack-settings#dynamic-cluster-setting). |
| `esql.external.cache.size` Elastic Stack: Planned`esql.source.cache.size` Elastic Stack: Planned for deprecation, Elastic Stack: Experimental in 9.5 | 0.4% of heap | Memory budget for the cache. Applied at node startup only. |
| `esql.source.cache.schema.ttl` | — | Deprecated and ignored. Inferred schemas are invalidated by file identity and bounded by the cache memory budget, not by a TTL. |
| `esql.external.cache.listing.ttl` Elastic Stack: Planned`esql.source.cache.listing.ttl` Elastic Stack: Planned for deprecation, Elastic Stack: Experimental in 9.5 | 30s | How long a file-listing result is cached. Applied at node startup only. |
| `esql.external.cache.footer.size` Elastic Stack: Planned | 0.5% of heap | Memory budget for cached raw footer bytes (for example, Parquet footers), which are reused across the resolution, split discovery, and execution phases of a query and across back-to-back queries. The budget applies per columnar format reader. Accepts a percentage of heap or an absolute size, and must be greater than zero. Applied at node startup only. |
| `esql.external.cache.footer.parsed.size` Elastic Stack: Planned | 1% of heap | Memory budget for cached deserialized footers, which avoid re-parsing a footer in every query phase. A parsed footer costs several times its serialized form and grows with column count rather than file size, so raise this when querying wide schemas across large file sets. Applies per columnar format reader, like `esql.external.cache.footer.size`. Applied at node startup only. |
| `esql.external.cache.footer.ttl` Elastic Stack: Planned | 5m | How long a cached footer survives without being accessed. Shared by the raw and parsed footer caches. Footer entries are keyed by path and file length rather than modification time, so a file overwritten in place at the same length can be served from the cache until its entry expires. Lower this if your data files are mutated in place. Applied at node startup only. |
The `esql.source.cache.*` keys are accepted as deprecated fallbacks and emit a deprecation warning. You can set them in `elasticsearch.yml`, but you can't update them through the cluster settings API. Use the `esql.external.cache.*` keys for new configuration.