--- title: Connect and manage data sources for ES|QL Data Federation description: Connect, view, edit, and delete ES|QL Data Federation data sources in Kibana or with the Elasticsearch API. url: https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-manage-data-sources products: - Elasticsearch applies_to: - Elastic Cloud Serverless: Unavailable - Elastic Stack: Experimental in 9.5 --- # Connect and manage data sources for ES|QL Data Federation This feature is experimental. It is not intended for production use and there are no guarantees around performance, scale, or stability in this release. You can manage data sources in the Kibana UI or with the Elasticsearch API. ## Manage data sources in the Kibana UI In the Kibana UI, you connect and manage data sources from the **Data sources** tab under **Data management** > **ES|QL Data Federation**. Elastic Stack: Experimental in 9.5 To use the UI, [enable it in Kibana](/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation#enable-the-feature) first. ### View data sources The **Data sources** tab lists each data source with its type, its description, and the number of datasets that read from it. You can search your data sources and connect new ones. Each row has the following actions: - **Edit** (`pencil`): Opens the data source's connection settings. - **Delete** (`trash`): Available only when no datasets read from the data source. In the following example, `ookla_speedtest` can't be deleted because a dataset reads from it: ![The Data sources tab listing two Amazon S3 data sources, with the delete action unavailable for ookla_speedtest because one dataset reads from it](https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/images/data-federation/data-sources-tab.png) ### Connect a new data source To connect a data source: 1. Click **Connect data source** to open the **Connect external data source** flyout. 2. Enter a **Name** and an optional **Description**. 3. In **Preferred method**, choose how Elasticsearch [authenticates](/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-sources#authentication) to your storage: - Your bucket is public: select **Anonymous**. No credentials are stored. - Your bucket is private and your deployment runs on Elastic Cloud Hosted on AWS, Azure, or Google Cloud: select **Federated Identity**. It's the recommended method because no keys are stored, and it's the default when it's available. Refer to [connect with federated identity](https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-federated-identity). - Your bucket is private and federated identity isn't available: select **Access and Secret Keys**, then enter the key pair. Refer to [connect with static credentials](https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-static-credentials). 4. Click **Connect**. Some connection options are available only with the [API](#manage-data-sources-using-the-api): - A custom Amazon S3 [`endpoint`](/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-data-source-settings#endpoint) - [Managed identity](/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-sources#authentication) authentication #### Check the connection Connecting a data source doesn't test whether it can read your files. To check the connection, [create a dataset](https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-manage-datasets) that reads from it and query the dataset. ### Edit a data source To change a data source: 1. Click **Edit** (`pencil`) in the data source's row. 2. Update its settings. You can't change the data source type or name. 3. Click **Save**. Credential fields that you leave unchanged keep their stored values. When credentials rotate, update them here. Datasets that read from the data source use the new credentials without any changes. ### Delete a data source A data source can't be deleted while datasets read from it. To delete a data source: 1. [Delete the datasets](https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-manage-datasets) that read from it. 2. Click **Delete** (`trash`) in the data source's row. To delete several unused data sources at once, select them, then click **Delete**. ## Manage data sources with the Elasticsearch API Data sources are managed under the `/_query/data_source` endpoint. All data source operations require the cluster `manage` privilege or a `global.data_source` privilege. Refer to [manage credentials and privileges](https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-security) for details. | Operation | Endpoint | API reference | |-----------------------------------------------------|-------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------| | [Create or update](#create-or-update-a-data-source) | `PUT /_query/data_source/{name}` | [Create or update an ES|QL data source](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-esql-put-data-source) | | [Get](#get-a-data-source) | `GET /_query/data_source/{name}` | [Get ES|QL data sources](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-esql-get-data-source) | | [List all](#list-all-data-sources) | `GET /_query/data_source` | [Get ES|QL data sources](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-esql-get-data-source) | | [Delete](#delete-a-data-source) | `DELETE /_query/data_source/{name}` | [Delete ES|QL data sources](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-esql-delete-data-source) | This section is a quick reference for each operation. For the complete request and response details, follow the API reference links in the table. ### Create or update a data source `PUT` creates a new data source or replaces an existing one entirely with one exception. Secrets you omit from the request are carried forward from the existing definition rather than cleared, so you can update non-secret settings without re-sending credentials. The create request does not validate connectivity to the external system. ```json { "type": "s3", "description": "Production S3 logs bucket", "settings": { "auth": "static_credentials", "access_key": "", "secret_key": "" } } ``` ```bash curl -X PUT "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \ -H "Authorization: ApiKey ${API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "type": "s3", "description": "Production S3 logs bucket", "settings": { "auth": "static_credentials", "access_key": "", "secret_key": "" } }' ``` For step-by-step guides on setting up each authentication model in AWS, refer to the [Amazon S3 authentication models](/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-s3#s3-auth-models). #### Naming rules and limits A create or update request is rejected when it breaks one of these rules or limits: - **Name**: Follows the same rules as index names. Lowercase only and at most 255 bytes. Can't begin with `-`, `_`, or `+`, and can't contain spaces or the characters `\ / * ? " < > |`. - **Description**: At most 1,000 characters. - **Number of data sources**: At most 100 per cluster by default. In Elastic Stack deployments, you can raise the limit with the [`esql.data_sources.max_count`](/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-cluster-settings#object-limits) cluster setting. ### Get a data source Retrieves a data source by name. You can pass a comma-separated list of names and use `*` wildcards. A concrete name that does not exist returns a `404`; a wildcard that matches nothing returns an empty list. Credential values are replaced by `::es_redacted::` in the response. ```json ``` ```bash curl -X GET "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \ -H "Authorization: ApiKey ${API_KEY}" ``` ### List all data sources Returns all registered data sources. ```json ``` ```bash curl -X GET "${ELASTICSEARCH_URL}/_query/data_source" \ -H "Authorization: ApiKey ${API_KEY}" ``` ### Delete a data source Deletes one or more data sources by name. You can pass a comma-separated list. If any named data source does not exist, the request returns a `404` and nothing is deleted. ```json ``` ```bash curl -X DELETE "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \ -H "Authorization: ApiKey ${API_KEY}" ``` A data source cannot be deleted while datasets still reference it. Delete the dependent datasets first, or the request returns a `409 Conflict` error. ## Next steps Continue with the following data-source tasks: - [Create datasets](https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-manage-datasets) that point at specific files in your data source, then review [file formats](https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-file-formats) and [dataset settings](https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-dataset-settings). - [Query your datasets](https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-querying) with `FROM` to learn how partition pruning, column selection, and filter pushdown reduce storage reads. - [Manage credentials and privileges](https://www.elastic.co/elastic/docs-builder/docs/4641/reference/query-languages/esql/esql-data-federation-security) to control who can create data sources and read external data.