Configure cluster settings for ES|QL Data Federation

The data sources feature adds the following cluster settings. For general guidance on how to apply cluster settings across deployment types, refer to configure Elasticsearch.

The object-count limits and authentication gates are operator-managed and take effect without a restart. Settings marked Dynamic can also be updated at runtime. The remaining settings require a node restart.

Note

Some of these settings were renamed. Where a row lists two names, the badge on each shows which versions accept it. Unless a note says otherwise, the older name is no longer registered, and a node that still sets it in elasticsearch.yml fails to start.

These settings cap the number of data sources and datasets a cluster can hold.

Setting Default Description
esql.data_sources.max_count 100 Maximum number of data sources that can be defined. Range 0–1000.
esql.datasets.max_count 1000 Maximum number of datasets that can be defined. Range 0–10,000.

These settings control how many concurrent requests each node sends to external storage and how long it retries throttled requests.

Setting Default Description
esql.external.max_concurrent_requests allocated processors * 3, minimum 16 and maximum 100 Maximum concurrent cloud API requests per storage scheme, per node. 0 removes the limit. Range 0–500.
esql.external.throttle_max_retry_duration 30 Maximum total time, in seconds, spent retrying throttled cloud API requests before failing the query. 0 removes the budget. Range 0–300 seconds.
esql.external.max_concurrent_segmenters
esql.external.max_concurrent_segmentators 		0 Maximum number of file segmentation tasks that run concurrently. 0 derives the value automatically. Range 0–4096.

These settings bound file discovery for glob patterns. Exceeding the file cap aborts the query; exceeding the brace-expansion cap falls back to listing the storage.

Setting Default Description
esql.external.max_discovered_files 10,000 Hard cap on files collected by glob expansion before the query aborts. Protects against degenerate globs. Range 1–1,000,000. Dynamic.
esql.external.max_glob_expansion 100 Cap on concrete paths generated by brace expansion before falling back to listing. Range 1–10,000. Dynamic.

These settings control which authentication modes data sources can use.

Setting Default Description
esql.external.managed_identity.enabled
esql.datasource.managed_identity.enabled 		false Enables auth: "managed_identity" (the node's own cloud identity through the instance metadata service (IMDS)). Operator-only. Intended for single-cloud, single-tenant deployments. Never enable in serverless or multi-tenant clusters. Refer to the managed identity row in authentication models for guidance.
esql.external.federated_identity.enabled
esql.datasource.federated_identity.enabled 		false Enables auth: "federated_identity" (OIDC-to-STS token exchange). Operator-only. Available on Elastic Cloud Hosted and Serverless; not available on self-managed, Elastic Cloud Enterprise, or Elastic Cloud on Kubernetes. For setup details, refer to connect with federated identity.

These settings control the external-source cache, which stores inferred schemas, file listings, and the footers of columnar files.

Setting Default Description
esql.external.cache.enabled
esql.source.cache.enabled 		true Enables the external-source cache (inferred schemas and file listings). Dynamic.
esql.external.cache.size
esql.source.cache.size 		0.4% of heap Memory budget for the cache. Applied at node startup only.
esql.source.cache.schema.ttl — Deprecated and ignored. Inferred schemas are invalidated by file identity and bounded by the cache memory budget, not by a TTL.
esql.external.cache.listing.ttl
esql.source.cache.listing.ttl 		30s How long a file-listing result is cached. Applied at node startup only.
esql.external.cache.footer.size 0.5% of heap Memory budget for cached raw footer bytes (for example, Parquet footers), which are reused across the resolution, split discovery, and execution phases of a query and across back-to-back queries. The budget applies per columnar format reader. Accepts a percentage of heap or an absolute size, and must be greater than zero. Applied at node startup only.
esql.external.cache.footer.parsed.size 1% of heap Memory budget for cached deserialized footers, which avoid re-parsing a footer in every query phase. A parsed footer costs several times its serialized form and grows with column count rather than file size, so raise this when querying wide schemas across large file sets. Applies per columnar format reader, like esql.external.cache.footer.size. Applied at node startup only.
esql.external.cache.footer.ttl 5m How long a cached footer survives without being accessed. Shared by the raw and parsed footer caches. Footer entries are keyed by path and file length rather than modification time, so a file overwritten in place at the same length can be served from the cache until its entry expires. Lower this if your data files are mutated in place. Applied at node startup only.
Note

The esql.source.cache.* keys are accepted as deprecated fallbacks and emit a deprecation warning. You can set them in elasticsearch.yml, but you can't update them through the cluster settings API. Use the esql.external.cache.* keys for new configuration.