ES|QL Data Federation

You can query data stored in compatible external data sources, using the same syntax you use for native indices and other index abstractions, without any ingestion into Elasticsearch. You query the files in place: nothing is copied into Elasticsearch, and there is no mapping to define up front: the schema is discovered from the files.

Warning

This feature is experimental. It is not intended for production use and there are no guarantees around performance, scale, or stability in this release.

Requirements

This is an experimental feature and is not enabled by default. Refer to Enable the feature.
For Elastic Cloud Hosted, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes deployments or self-managed clusters, ES|QL Data Federation requires an Enterprise subscription.

Enable the feature

ES|QL Data Federation is not enabled by default. To use it, enable the feature in Elasticsearch and enable the UI in Kibana.

Elasticsearch

Add the following to your Elasticsearch configuration:

esql.federation.enabled: true

Kibana

Add the following to your Kibana configuration:

xpack.dataFederation.enabled: true

To learn how to configure these settings on your deployment, refer to Elastic Stack settings.

Why use federated data

Many organizations store large volumes of data in cloud object storage for cost and compliance reasons. Querying that data typically requires a separate tool like Apache Spark, Amazon Athena, or Trino, which means managing extra infrastructure and switching between query languages.

ES|QL federated data enables you to query this data directly from Elasticsearch, with several advantages:

No ingestion required. Query external data without copying or indexing it into Elasticsearch. The data stays in your storage.
One language for all your data. Use the same ES|QL syntax for both indexed data and external data. No context-switching, no second query engine.
No extra infrastructure. Query external data natively in Elasticsearch without deploying or managing additional compute services, catalogs, or connectors.
Progressive acceleration. Start by querying raw data directly in object storage. When specific datasets need faster performance, promote them into Elasticsearch for indexed search. Both tiers stay queryable with the same ES|QL syntax.

How it works

Federated data requires two objects: a data source, which defines the connection, and one or more datasets, which define what to read. These steps walk through the model. For the setup procedures, refer to Connect external data sources for ES|QL Data Federation and Select external datasets for ES|QL Data Federation.

Your data lives in cloud storage

You have Parquet files, CSVs, or NDJSON sitting in a bucket. The data is not ingested into Elasticsearch.
You create a data source (the connection)

A data source tells Elasticsearch where the storage is and how to authenticate. It stores the connection type, region, endpoint, and credentials. Any number of datasets can read through a single data source. When credentials rotate, you update the data source in one place without touching the datasets that reference it.
You create datasets (what to read)

Each dataset points at specific files in that storage and makes them queryable. One data source can serve many datasets.

Datasets are designed to work like indices for queries. They share the same namespace as indices, data streams, aliases, and ES|QL views, so a dataset cannot have the same name as any of them.
You query your dataset like any index

Once a dataset exists, you query it the same way you query any Elasticsearch index. There is no special syntax for federated data. Use FROM with the dataset name, and Elasticsearch handles file discovery, format detection, compression, and schema inference automatically. For example, to return the first 10 rows from a dataset named my_s3_bucket_logs:
```
FROM my_s3_bucket_logs
| LIMIT 10
		
```
Tip

For a hands-on example, refer to Get started with ES|QL Data Federation.

Supported data source types

The following data source types are supported:

Type	Storage system
`s3`	Amazon S3

Tip

Amazon S3 is the first supported data source type. Support for additional storage systems, including Google Cloud Storage and Azure Blob Storage, is planned.

Supported file formats

Federated data sources can read the following file formats:

Format	Schema source	Compression
Parquet	Read from file headers	Internal per column chunk: UNCOMPRESSED, SNAPPY, ZSTD, GZIP
NDJSON	Inferred by sampling rows	gzip, zstd
CSV and TSV	Inferred by sampling rows	gzip, zstd

The format is detected automatically from the file extension. You can override this in the dataset settings if needed.

For details on type-specific settings and format options, refer to Select external datasets for ES|QL Data Federation.

Capabilities and limitations

Datasets behave like indices. In most places where ES|QL accepts an index name, it accepts a dataset name too: FROM, WHERE, STATS, SORT, EVAL, KEEP, and the rest of the processing commands work the same way, on the same execution engine used for native indices. You can query a dataset on its own, or alongside indices, aliases, and views, in the same FROM.

The exceptions are operations that need structures only an Elasticsearch index has, such as the inverted index, doc values, or time series metadata. Relevance scoring returns _score as null, and KNN, LOOKUP JOIN with a dataset as the lookup target, and TS each fail with a clear error rather than returning wrong results. For the full list, refer to query limitations.

Get started

Quickstart: Register a data source, create a dataset, and run your first query against external data.
Connect data sources: Connect to external storage, configure S3 settings, and set up authentication.
Add datasets: Select which files to query, configure format settings, and control schema inference.
Query datasets: Learn how the engine reduces storage reads, query external and indexed data together, review current limitations, and troubleshoot common issues.
Manage access: Control access to data sources and datasets, encrypt credentials, and configure privileges.
Cluster settings: Tune object limits, control request concurrency, and adjust file discovery and caching behavior.