---
title: ES|QL Data Federation
description: Query data stored in external cloud storage using ES|QL without ingesting it into Elasticsearch.
url: https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation
products:
  - Elasticsearch
applies_to:
  - Elastic Cloud Serverless: Unavailable
  - Elastic Stack: Experimental in 9.5
---

# ES|QL Data Federation
You can query data stored in compatible external data sources, using the same syntax you use for native indices and other index abstractions, without any ingestion into Elasticsearch. You query the files in place: nothing is copied into Elasticsearch, and there is no mapping to define up front: the schema is discovered from the files.
<warning>
  This feature is experimental. It is not intended for production use and there are no guarantees around performance, scale, or stability in this release.
</warning>


## Requirements

- This is an experimental feature and is not enabled by default. Refer to [Enable the feature](#enable-the-feature).
- For Elastic Cloud Hosted, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes deployments or self-managed clusters, ES|QL Data Federation requires an [Enterprise subscription](https://www.elastic.co/subscriptions).


## Enable the feature

ES|QL Data Federation is not enabled by default. To use it, enable the feature in Elasticsearch and enable the UI in Kibana.
**Elasticsearch**
Add the following to your Elasticsearch configuration:
```yaml
esql.federation.enabled: true
```

**Kibana**
Add the following to your Kibana configuration:
```yaml
xpack.dataFederation.enabled: true
```

To learn how to configure these settings on your deployment, refer to [Elastic Stack settings](https://docs-v3-preview.elastic.dev/elastic/docs-content/tree/main/deploy-manage/stack-settings).

## Why use federated data

Many organizations store large volumes of data in cloud object storage for cost and compliance reasons. Querying that data typically requires a separate tool like Apache Spark, Amazon Athena, or Trino, which means managing extra infrastructure and switching between query languages.
ES|QL federated data enables you to query this data directly from Elasticsearch, with several advantages:
- **No ingestion required.** Query external data without copying or indexing it into Elasticsearch. The data stays in your storage.
- **One language for all your data.** Use the same ES|QL syntax for both indexed data and external data. No context-switching, no second query engine.
- **No extra infrastructure.** Query external data natively in Elasticsearch without deploying or managing additional compute services, catalogs, or connectors.
- **Progressive acceleration.** Start by querying raw data directly in object storage. When specific datasets need faster performance, promote them into Elasticsearch for indexed search. Both tiers stay queryable with the same ES|QL syntax.


## How it works

Federated data requires two objects: a data source, which defines the connection, and one or more datasets, which define what to read. These steps walk through the model. For the setup procedures, refer to [Connect external data sources for ES|QL Data Federation](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-sources) and [Select external datasets for ES|QL Data Federation](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-datasets).
<stepper>
  <step title="Your data lives in cloud storage">
    You have Parquet files, CSVs, or NDJSON sitting in a bucket. The data is not ingested into Elasticsearch.
  </step>

  <step title="You create a data source (the connection)">
    A [data source](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-sources) tells Elasticsearch where the storage is and how to authenticate. It stores the connection type, region, endpoint, and credentials. Any number of datasets can read through a single data source. When credentials rotate, you update the data source in one place without touching the datasets that reference it.
  </step>

  <step title="You create datasets (what to read)">
    Each [dataset](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-datasets) points at specific files in that storage and makes them queryable. One data source can serve many datasets.Datasets are designed to work like indices for queries. They share the same namespace as indices, data streams, aliases, and [ES|QL views](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-views), so a dataset cannot have the same name as any of them.
  </step>

  <step title="You query your dataset like any index">
    Once a dataset exists, you [query](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-querying) it the same way you query any Elasticsearch index. There is no special syntax for federated data. Use [`FROM`](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/commands/from) with the dataset name, and Elasticsearch handles file discovery, format detection, compression, and schema inference automatically. For example, to return the first 10 rows from a dataset named `my_s3_bucket_logs`:
    ```esql
    FROM my_s3_bucket_logs
    | LIMIT 10
    ```

    <tip>
      For a hands-on example, refer to [Get started with ES|QL Data Federation](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-quickstart).
    </tip>
  </step>
</stepper>


## Supported data source types

The following data source types are supported:

| Type | Storage system |
|------|----------------|
| `s3` | Amazon S3      |

<tip>
  Amazon S3 is the first supported data source type. Support for additional storage systems, including Google Cloud Storage and Azure Blob Storage, is planned.
</tip>


## Supported file formats

Federated data sources can read the following file formats:

| Format      | Schema source             | Compression                                                 |
|-------------|---------------------------|-------------------------------------------------------------|
| Parquet     | Read from file headers    | Internal per column chunk: UNCOMPRESSED, SNAPPY, ZSTD, GZIP |
| NDJSON      | Inferred by sampling rows | gzip, zstd                                                  |
| CSV and TSV | Inferred by sampling rows | gzip, zstd                                                  |

The format is detected automatically from the file extension. You can override this in the dataset settings if needed.
For details on type-specific settings and format options, refer to [Select external datasets for ES|QL Data Federation](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-datasets).

## Capabilities and limitations

Datasets behave like indices. In most places where ES|QL accepts an index name, it accepts a dataset name too: `FROM`, `WHERE`, `STATS`, `SORT`, `EVAL`, `KEEP`, and the rest of the processing commands work the same way, on the same execution engine used for native indices. You can query a dataset on its own, or alongside indices, aliases, and views, in the same `FROM`.
The exceptions are operations that need structures only an Elasticsearch index has, such as the inverted index, doc values, or time series metadata. Relevance scoring returns `_score` as null, and `KNN`, `LOOKUP JOIN` with a dataset as the lookup target, and `TS` each fail with a clear error rather than returning wrong results. For the full list, refer to [query limitations](/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-querying#limitations).

## Get started

<definitions>
  <definition term="Quickstart">
    Register a data source, create a dataset, and run your first query against external data.
  </definition>
  <definition term="Connect data sources">
    Connect to external storage, configure S3 settings, and set up authentication.
  </definition>
  <definition term="Add datasets">
    Select which files to query, configure format settings, and control schema inference.
  </definition>
  <definition term="Query datasets">
    Learn how the engine reduces storage reads, query external and indexed data together, review current limitations, and troubleshoot common issues.
  </definition>
  <definition term="Manage access">
    Control access to data sources and datasets, encrypt credentials, and configure privileges.
  </definition>
  <definition term="Cluster settings">
    Tune object limits, control request concurrency, and adjust file discovery and caching behavior.
  </definition>
</definitions>