Connect external data sources for ES|QL Data Federation

A data source defines the connection to an external storage system. It stores the connection type, region, endpoint, and credentials. A data source defines how to connect, not what data to query. One data source can serve many datasets. When credentials rotate, you update the data source in one place without touching the datasets that reference it.

The following data source types are supported:

In Kibana, you connect and manage data sources from the Data sources tab under Data management > ES|QL Data Federation.

The Data sources tab lists each registered data source including:

• its type
• its description
• the number of datasets that reference it

From this tab you can search your data sources, connect a new one, and edit or delete an existing one.

×

Click Connect data source to open a flyout where you define the connection:

• Data source type: the storage system to connect to, such as Amazon S3.
• Name: a unique name for the data source. Names must be lowercase and cannot begin with -, _, or +.
• Description: an optional description.
• Region: the cloud region where your storage is located, such as us-east-1.
• Endpoint: an optional Amazon S3 endpoint override.
• Authentication: select an authentication model from the dropdown, then fill in the credentials it requires.

For the full set of authentication methods and what each one requires, refer to authentication models. For detailed setup walkthroughs, refer to connect with static credentials or connect with federated identity.

Show the Connect data source flyout

×

Data sources are managed under the /_query/data_source endpoint. All data source operations require the cluster manage privilege or a global.data_source privilege. Refer to manage credentials and privileges for details.

PUT creates a new data source or replaces an existing one entirely with one exception. Secrets you omit from the request are carried forward from the existing definition rather than cleared, so you can update non-secret settings without re-sending credentials.

The create request does not validate connectivity to the external system. To verify that credentials and endpoint are correct, create a dataset that references the data source and query it.

A cluster holds at most 100 data sources by default. In Elastic Stack deployments, if you need more than 100 data sources, then you can raise the limit using the esql.data_sources.max_count cluster setting.

				PUT /_query/data_source/prod_s3_logs
					{
  "type": "s3",
  "description": "Production S3 logs bucket, us-east-1",
  "settings": {
    "region": "us-east-1",
    "auth": "static_credentials",
    "access_key": "<AWS_ACCESS_KEY_ID>",
    "secret_key": "<AWS_SECRET_ACCESS_KEY>"
  }
}
		

curl -X PUT "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \
  -H "Authorization: ApiKey ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
  "type": "s3",
  "description": "Production S3 logs bucket, us-east-1",
  "settings": {
    "region": "us-east-1",
    "auth": "static_credentials",
    "access_key": "<AWS_ACCESS_KEY_ID>",
    "secret_key": "<AWS_SECRET_ACCESS_KEY>"
  }
}'
		

Retrieves a data source by name. You can pass a comma-separated list of names and use * wildcards. A concrete name that does not exist returns a 404; a wildcard that matches nothing returns an empty list. Credential values are replaced by ::es_redacted:: in the response.

				GET /_query/data_source/prod_s3_logs
		

curl -X GET "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \
  -H "Authorization: ApiKey ${API_KEY}"
		

Returns all registered data sources.

				GET /_query/data_source
		

curl -X GET "${ELASTICSEARCH_URL}/_query/data_source" \
  -H "Authorization: ApiKey ${API_KEY}"
		

Deletes one or more data sources by name. You can pass a comma-separated list. If any named data source does not exist, the request returns a 404 and nothing is deleted.

				DELETE /_query/data_source/prod_s3_logs
		

curl -X DELETE "${ELASTICSEARCH_URL}/_query/data_source/prod_s3_logs" \
  -H "Authorization: ApiKey ${API_KEY}"
		

Settings vary by data source type.

The following settings are available for s3 data sources:

Connection settings:

Authentication settings:

A data source authenticates to its store with one of the following models. The models are mutually exclusive on a data source.

• Create datasets that point at specific files in your data source, and configure file formats, schema inference, and parsing settings.
• Query your datasets with FROM to learn how partition pruning, column selection, and filter pushdown reduce storage reads.
• Manage credentials and privileges to control who can create data sources and read external data.