---
title: Connect external data sources for ES|QL Data Federation
description: Connect Elasticsearch to external storage with ES|QL Data Federation. Create and manage data sources, choose an authentication model, and grant read access to your data.
url: https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-sources
products:
- Elasticsearch
applies_to:
- Elastic Cloud Serverless: Unavailable
- Elastic Stack: Experimental since 9.5
---
# Connect external data sources for ES|QL Data Federation
A data source defines the connection to an external storage system. It stores the connection type, endpoint, and credentials. A data source defines how to connect, not what data to query. One data source can serve many [datasets](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-datasets). When credentials rotate, you update the data source in one place without touching the datasets that reference it.
This feature is experimental. It is not intended for production use and there are no guarantees around performance, scale, or stability in this release.
## Supported data source types
The following data source types are supported:
| Type | Storage system |
|------|----------------|
| `s3` | Amazon S3 |
Other S3-compatible services have not been validated and are not supported.
## Manage data sources using the Kibana UI
In Kibana, you connect and manage data sources from the **Data sources** tab under **Data management** > **ES|QL Data Federation**.
The **Data sources** tab lists each registered data source including:
- its type
- its description
- the number of datasets that reference it
From this tab you can search your data sources, connect a new one, and edit or delete an existing one.
### Connect a new data source
Click **Connect data source** to open a flyout where you define the connection:
- **Data source type**: the storage system to connect to, such as **Amazon S3**.
- **Name**: a unique name for the data source. Names must be lowercase and cannot begin with `-`, `_`, or `+`.
- **Description**: an optional description (up to 1,000 characters).
- **Endpoint**: an optional Amazon S3 endpoint override. Leave it empty to have the endpoint resolved from the region. For accepted values, refer to the [`endpoint` setting](/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-data-source-settings#endpoint).
- **Authentication**: select an authentication model from the dropdown, then fill in the credentials it requires.
For the full set of authentication methods and what each one requires, refer to [authentication models](#authentication). For setup walkthroughs, refer to the [Amazon S3 authentication models](/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-s3#s3-auth-models).
## Manage data sources using the API
Data sources are managed under the `/_query/data_source` endpoint. All data source operations require the cluster `manage` privilege or a `global.data_source` privilege. Refer to [manage credentials and privileges](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-security) for details.
| Operation | Endpoint | API reference |
|---------------------------------------------------------------------------------------|-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|
| [Create or update](#create-or-update-a-data-source) | `PUT /_query/data_source/{name}` | [Create or update an ES|QL data source](https://www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-esql-put-data-source) |
| [Get](#get-a-data-source) | `GET /_query/data_source/{name}` | [Get ES|QL data sources](https://www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-esql-get-data-source) |
| [List all](#list-all-data-sources) | `GET /_query/data_source` | [Get ES|QL data sources](https://www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-esql-get-data-source) |
| [Delete](#delete-a-data-source) | `DELETE /_query/data_source/{name}` | [Delete ES|QL data sources](https://www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-esql-delete-data-source) |
| [Test connection](#test-a-connection) Elastic Stack: Planned | `POST /_query/data_source/_test` | [Test an ES|QL data source connection](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-esql-data-source-test-connection) |
### Create or update a data source
`PUT` creates a new data source or replaces an existing one entirely with one exception. Secrets you omit from the request are carried forward from the existing definition rather than cleared, so you can update non-secret settings without re-sending credentials.
The create request does not validate connectivity to the external system.
Elastic Stack: Planned To verify that credentials and endpoint are correct before saving, use the [test connection](#test-a-connection) endpoint.
Data source names follow the same naming rules as index names: lowercase only, at most 255 bytes, and they cannot begin with `-`, `_`, or `+`, contain spaces, or contain the characters `\ / * ? " < > |`.
A cluster holds at most 100 data sources by default. In Elastic Stack deployments, if you need more than 100 data sources, then you can raise the limit using the [`esql.data_sources.max_count`](/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-cluster-settings#object-limits) cluster setting.
The optional `description` can be at most 1,000 characters long.
```json
{
"type": "s3",
"description": "Production S3 logs bucket",
"settings": {
"auth": "static_credentials",
"access_key": "",
"secret_key": ""
}
}
```
```bash
curl -X PUT "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"type": "s3",
"description": "Production S3 logs bucket",
"settings": {
"auth": "static_credentials",
"access_key": "",
"secret_key": ""
}
}'
```
For step-by-step guides on setting up each authentication model in AWS, refer to the [Amazon S3 authentication models](/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-s3#s3-auth-models).
### Get a data source
Retrieves a data source by name. You can pass a comma-separated list of names and use `*` wildcards. A concrete name that does not exist returns a `404`; a wildcard that matches nothing returns an empty list. Credential values are replaced by `::es_redacted::` in the response.
```json
```
```bash
curl -X GET "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \
-H "Authorization: ApiKey ${API_KEY}"
```
### List all data sources
Returns all registered data sources.
```json
```
```bash
curl -X GET "${ELASTICSEARCH_URL}/_query/data_source" \
-H "Authorization: ApiKey ${API_KEY}"
```
### Delete a data source
Deletes one or more data sources by name. You can pass a comma-separated list. If any named data source does not exist, the request returns a `404` and nothing is deleted.
```json
```
```bash
curl -X DELETE "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \
-H "Authorization: ApiKey ${API_KEY}"
```
A data source cannot be deleted while datasets still reference it. Delete the dependent datasets first, or the request returns a `409 Conflict` error.
### Test a connection
- Elastic Stack: Planned
Use `POST /_query/data_source/_test` to verify that a configuration can reach its backend before saving it. The data source does not need to exist in cluster state — the endpoint accepts the same `type` and `settings` fields as the `PUT` request.
The response contains a `status` field with one of three values:
| Status | Meaning |
|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `success` | The probe ran on every eligible node and the backend is reachable with the given credentials. |
| `failure` | The probe ran but the backend was unreachable or rejected the credentials. An `error` field contains a human-readable reason. |
| `untestable` | The configuration cannot be verified at the data source level. An optional `message` field, when present, explains why. Common reasons: the backend type has no connectivity probe; credentials are scoped to a container or bucket rather than the account; authentication uses anonymous or managed-identity access that requires a dataset path to resolve the endpoint; or no eligible node could complete the probe. The configuration may still be correct — create a dataset and run a query to validate access. |
The probe runs on every eligible node in parallel, with a 30-second timeout per node. If any node cannot complete the probe, the result is `untestable` rather than `success`.
```json
{
"type": "s3",
"settings": {
"auth": "static_credentials",
"access_key": "",
"secret_key": ""
}
}
```
```bash
curl -X POST "${ELASTICSEARCH_URL}/_query/data_source/_test" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"type": "s3",
"settings": {
"auth": "static_credentials",
"access_key": "",
"secret_key": ""
}
}'
```
Example responses:
```json
{ "status": "success" }
```
```json
{ "status": "failure", "error": "The AWS Access Key Id you provided does not exist in our records." }
```
```json
{ "status": "untestable", "message": "Anonymous access targets public buckets; create a dataset to validate read access." }
```
An unknown `type` value returns a `400 Bad Request` rather than a `failure` status, because the type is not registered — it is a client error, not a connectivity problem.
## Data source settings
Settings vary by data source type. For every setting each type accepts, refer to the [data source settings reference](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-data-source-settings).
## Authentication
A data source authenticates to its store with one of the following models. The models are mutually exclusive on a data source.
| Model | `auth` value | Description |
|--------------------|----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Static credentials | `static_credentials` | A fixed access key and secret key. The common form for a service account. To set one up, refer to [connect with static credentials](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-static-credentials). |
| Anonymous | `anonymous` | For public data that needs no credentials. The [quickstart](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-quickstart) walks through this method. |
| Federated identity | `federated_identity` | Keyless. Elasticsearch exchanges a short-lived OIDC token for temporary AWS credentials via STS, so no static keys are stored. Available on Elastic Cloud Hosted and serverless only. Operator-gated by a [cluster setting](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-cluster-settings).Elastic Stack: Planned `esql.external.federated_identity.enabled`Elastic Stack: Planned for deprecation, Elastic Stack: Experimental in 9.5 `esql.datasource.federated_identity.enabled`To set it up, refer to [connect with federated identity](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-federated-identity). |
| Managed identity | `managed_identity` | Keyless. Uses the Elasticsearch node's own cloud identity, for example an EC2 instance IAM role. Operator-only and API-only, and not available in serverless. Requires a [cluster setting](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-cluster-settings).Elastic Stack: Planned `esql.external.managed_identity.enabled`Elastic Stack: Planned for deprecation, Elastic Stack: Experimental in 9.5 `esql.datasource.managed_identity.enabled` |
Managed identity uses the cloud identity attached to each Elasticsearch node (for example, an IAM role on EC2 or a service account on GKE). Different nodes might have different identities, and the node that performs the connection is not guaranteed. You are responsible for configuring cloud IAM so that every node's identity has the required permissions on the target bucket. This model is best suited for single-cloud, single-tenant deployments where node identities are uniform.
## Grant read access to your data
A data source reads from external storage as a single cloud identity, and your cloud provider controls what that identity can read. For Amazon S3, the identity is the IAM user that owns the access key, the IAM role that Elasticsearch assumes, or the node's own IAM role. Elasticsearch doesn't check these permissions when you create a data source or a dataset. A missing permission shows up as an access error the first time you query a dataset.
The identity needs permission to read the files in every dataset that uses the data source. If a dataset's resource is a prefix or a glob pattern, the identity also needs permission to list objects in that location. Because one data source can serve many datasets, check its permissions whenever you add a dataset that reads from a new location.
To grant read access for each data source type, refer to:
- **Amazon S3**: Allow `s3:GetObject` on the dataset's objects, and `s3:ListBucket` on the bucket when a dataset's resource is a prefix or a glob pattern. For a complete policy and troubleshooting, refer to [grant read access in Amazon S3](/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-s3#s3-permissions).
## Next steps
Continue with the following data-source tasks:
- [Create datasets](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-manage-datasets) that point at specific files in your data source, then review [file formats](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-file-formats) and [dataset settings](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-dataset-settings).
- [Query your datasets](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-querying) with `FROM` to learn how partition pruning, column selection, and filter pushdown reduce storage reads.
- [Manage credentials and privileges](https://www.elastic.co/elastic/docs-builder/docs/4492/reference/query-languages/esql/esql-data-federation-security) to control who can create data sources and read external data.