Configure cluster settings for ES|QL Data Federation
The data sources feature adds the following cluster settings. For general guidance on how to apply cluster settings across deployment types, refer to configure Elasticsearch.
The object-count limits and authentication gates are operator-managed and take effect without a restart. The remaining settings require a node restart.
These settings cap the number of data sources and datasets a cluster can hold.
| Setting | Default | Description |
|---|---|---|
esql.data_sources.max_count |
100 | Maximum number of data sources that can be defined. Range 0–1000. |
esql.datasets.max_count |
1000 | Maximum number of datasets that can be defined. Range 0–10,000. |
These settings control how many concurrent requests each node sends to external storage and how long it retries throttled requests.
| Setting | Default | Description |
|---|---|---|
esql.external.max_concurrent_requests |
allocated processors * 3, minimum 16 and maximum 100 |
Maximum concurrent cloud API requests per storage scheme, per node. 0 removes the limit. Range 0–500. |
esql.external.throttle_max_retry_duration |
30 | Maximum total time, in seconds, spent retrying throttled cloud API requests before failing the query. 0 removes the budget. Range 0–300 seconds. |
esql.external.max_concurrent_segmentators |
0 |
Maximum number of file segmentation tasks that run concurrently. 0 derives the value automatically. Range 0–4096. |
These settings bound file discovery for glob patterns. Exceeding the file cap aborts the query; exceeding the brace-expansion cap falls back to listing the storage.
| Setting | Default | Description |
|---|---|---|
esql.external.max_discovered_files |
10,000 | Hard cap on files collected by glob expansion before the query aborts. Protects against degenerate globs. Range 1–1,000,000. Dynamic. |
esql.external.max_glob_expansion |
100 | Cap on concrete paths generated by brace expansion before falling back to listing. Range 1–10,000. Dynamic. |
These settings control which authentication modes data sources can use.
| Setting | Default | Description |
|---|---|---|
esql.datasource.managed_identity.enabled |
false | Enables auth: "managed_identity" (the node's own cloud identity through the instance metadata service (IMDS)). Operator-only. Intended for single-cloud, single-tenant deployments. Never enable in serverless or multi-tenant clusters. Refer to the managed identity row in authentication models for guidance. |
esql.datasource.federated_identity.enabled |
false | Enables auth: "federated_identity" (OIDC-to-STS token exchange). Operator-only. Available on Elastic Cloud Hosted and Serverless; not available on self-managed, Elastic Cloud Enterprise, or Elastic Cloud on Kubernetes. For setup details, refer to connect with federated identity. |
These settings control the external-source cache, which stores inferred schemas and file listings.
| Setting | Default | Description |
|---|---|---|
esql.source.cache.enabled |
true | Enables the external-source cache (inferred schemas and file listings). Dynamic. |
esql.source.cache.size |
0.4% of heap | Memory budget for the cache. Applied at node startup only. |
esql.source.cache.schema.ttl |
5m | How long an inferred schema is cached. Applied at node startup only. |
esql.source.cache.listing.ttl |
30s | How long a file-listing result is cached. Applied at node startup only. |