External data sources in ES|QL Data Federation
This feature is experimental. It is not intended for production use and there are no guarantees around performance, scale, or stability in this release.
A data source defines the connection to an external storage system. It stores the connection type, endpoint, and credentials. A data source defines how to connect, not what data to query. One data source can serve many datasets. When credentials rotate, you update the data source in one place without touching the datasets that reference it.
The following data source types are supported:
| Type | Storage system |
|---|---|
s3 |
Amazon S3 |
Other S3-compatible services have not been validated and are not supported.
You can connect, view, edit, and delete data sources in the Kibana UI or with the /_query/data_source API. Refer to Connect and manage data sources for ES|QL Data Federation.
Settings vary by data source type. For every setting each type accepts, refer to the data source settings reference.
A data source authenticates to its store with one of the following models. The models are mutually exclusive on a data source.
| Model | auth value |
Description |
|---|---|---|
| Static credentials | static_credentials |
A fixed access key and secret key. The common form for a service account. To set one up, refer to connect with static credentials. |
| Anonymous | anonymous |
For public data that needs no credentials. The quickstart walks through this method. |
| Federated identity | federated_identity |
Keyless. Elasticsearch exchanges a short-lived OIDC token for temporary AWS credentials via STS, so no static keys are stored. To set it up, refer to connect with federated identity. Available on Elastic Cloud Hosted on AWS, Azure, and Google Cloud. Not available on IBM Cloud, in special environments such as AWS GovCloud, or on self-managed, Elastic Cloud Enterprise, or Elastic Cloud on Kubernetes deployments. |
| Managed identity | managed_identity |
Keyless. Uses the Elasticsearch node's own cloud identity, for example an EC2 instance IAM role. Operator-only and API-only, and not available in serverless. Requires the esql.external.managed_identity.enabled cluster setting. |
Managed identity uses the cloud identity attached to each Elasticsearch node (for example, an IAM role on EC2 or a service account on GKE). Different nodes might have different identities, and the node that performs the connection is not guaranteed.
You are responsible for configuring cloud IAM so that every node's identity has the required permissions on the target bucket. This model is best suited for single-cloud, single-tenant deployments where node identities are uniform.
A data source reads from external storage as a single cloud identity, and your cloud provider controls what that identity can read. For Amazon S3, the identity is the IAM user that owns the access key, the IAM role that Elasticsearch assumes, or the node's own IAM role. Elasticsearch doesn't check these permissions when you create a data source or a dataset. A missing permission shows up as an access error the first time you query a dataset.
The identity needs permission to read the files in every dataset that uses the data source. If a dataset's resource is a prefix or a glob pattern, the identity also needs permission to list objects in that location. Because one data source can serve many datasets, check its permissions whenever you add a dataset that reads from a new location.
To grant read access for each data source type, refer to:
- Amazon S3: Allow
s3:GetObjecton the dataset's objects, ands3:ListBucketon the bucket when a dataset's resource is a prefix or a glob pattern. For a complete policy and troubleshooting, refer to grant read access in Amazon S3.
With audit logging enabled,
creating, updating, and deleting data sources generates security_config_change events. Refer to
put_data_source, and
delete_data_source.