---
title: Select external datasets for ES|QL Data Federation
description: Create ES|QL Data Federation datasets to query files in external storage. Choose file formats, adjust Parquet and CSV parsing, and control schema inference.
url: https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-datasets
products:
  - Elasticsearch
applies_to:
  - Elastic Cloud Serverless: Unavailable
  - Elastic Stack: Experimental in 9.5
---

# Select external datasets for ES|QL Data Federation
Datasets share the same namespace as indices, data streams, aliases, and [ES|QL views](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-views). A dataset cannot have the same name as any of them.
<warning>
  This feature is experimental. It is not intended for production use and there are no guarantees around performance, scale, or stability in this release.
</warning>


## Supported file formats

Federated data sources can read the following file formats:

| Format      | Schema source             | Compression                                                 |
|-------------|---------------------------|-------------------------------------------------------------|
| Parquet     | Read from file headers    | Internal per column chunk: UNCOMPRESSED, SNAPPY, ZSTD, GZIP |
| NDJSON      | Inferred by sampling rows | gzip, zstd                                                  |
| CSV and TSV | Inferred by sampling rows | gzip, zstd                                                  |

The format is detected automatically from the file extension. You can override this in the [dataset settings](#common-settings).

### Text formats

The following text formats are recognized by file extension:

| Format | Recognized extensions        |
|--------|------------------------------|
| CSV    | `.csv`                       |
| TSV    | `.tsv`                       |
| NDJSON | `.ndjson`, `.jsonl`, `.json` |


### Compression for text formats

A text resource is read uncompressed, or compressed with a codec identified from a trailing extension: `clicks.csv`, `clicks.csv.gz`, `clicks.csv.zst`.

| Codec        | Extensions      |
|--------------|-----------------|
| uncompressed | none            |
| gzip         | `.gz`, `.gzip`  |
| zstd         | `.zst`, `.zstd` |


### Parquet

Parquet declares its compression internally, per column chunk, so Parquet resources are not externally compressed. They are recognized by the `.parquet` and `.parq` extensions. The following internal codecs are supported:
- `UNCOMPRESSED`
- `SNAPPY`
- `ZSTD`
- `GZIP`
- `LZ4_RAW`
- `LZ4` (legacy Hadoop-framed format, supported for reading only)


## Manage datasets in the UI

In Kibana, you create and manage datasets from the **Datasets** tab under **Data management** > **ES|QL Data Federation**.
The **Datasets** tab lists each dataset including:
- its data source and data source type
- its resource
- its description

From this tab you can search your datasets, filter by data source, add a new one, and edit or delete an existing one.

### Add a new dataset

Click **Add dataset** to open a flyout where you define the dataset:
- **Data source**: the connected data source to read through.
- **Name**: a unique name for use in queries. Names must be lowercase and cannot begin with `-`, `_`, or `+`. A dataset cannot share a name with any existing index, data stream, alias, or view.
- **Description**: an optional description.
- **Resource**: the URI and glob pattern that selects the files to read.
- **Format**: the file format. This selection is required in the Kibana UI. The API can omit `settings.format` to auto-detect it from the file extension. Refer to [supported file formats](#supported-file-formats).

To configure how the format is read, expand **Advanced settings**. Refer to [dataset settings](#dataset-settings).
To customize the inferred schema, rename columns, or override field types, use the [dataset mappings API](#declare-a-dataset-mapping). Schema customization is not available in the UI.

## Manage datasets using the API

Datasets are managed under the `/_query/dataset` endpoint. All dataset operations require the index `manage` privilege on the dataset name, or a fine-grained dataset privilege. Refer to [manage credentials and privileges](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-security) for details.

| Operation                                       | Endpoint                        |
|-------------------------------------------------|---------------------------------|
| [Create or update](#create-or-update-a-dataset) | `PUT /_query/dataset/{name}`    |
| [Get](#get-a-dataset)                           | `GET /_query/dataset/{name}`    |
| [List all](#list-all-datasets)                  | `GET /_query/dataset`           |
| [Delete](#delete-a-dataset)                     | `DELETE /_query/dataset/{name}` |


### Create or update a dataset

`PUT` creates a new dataset or replaces an existing one entirely.
<important>
  A dataset cannot have the same name as an existing index, data stream, alias, or view, because dataset names share the same namespace. Dataset names must be lowercase and cannot begin with `-`, `_`, or `+`.
</important>

<tab-set>
  <tab-item title="Console">
    ```json

    {
      "data_source": "prod_s3_logs",
      "resource": "s3://logs-bucket/access/**/*.parquet",
      "description": "Production access logs",
      "settings": {
        "partition_detection": "hive"
      }
    }
    ```
  </tab-item>

  <tab-item title="curl">
    ```bash
    curl -X PUT "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
      -H "Authorization: ApiKey ${API_KEY}" \
      -H "Content-Type: application/json" \
      -d '{
      "data_source": "prod_s3_logs",
      "resource": "s3://logs-bucket/access/**/*.parquet",
      "description": "Production access logs",
      "settings": {
        "partition_detection": "hive"
      }
    }'
    ```
  </tab-item>
</tab-set>

<tip>
  After creating a dataset, you can check the field mappings that Elasticsearch inferred from your files. Refer to [check field mappings](/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-quickstart#check-field-mappings) in the quickstart for a hands-on example.
</tip>


### Declare a dataset mapping

By default, Elasticsearch infers a dataset's schema from its files. You can instead add an optional `mappings` block to the create or update request to control column names and types. Dataset mappings are currently available only through the API. The Kibana **Add dataset** flyout does not expose them.
The following example declares the complete schema, renames the physical `event_time` column to `@timestamp`, supplies its date format, and uses `request_id` as the row's `_id`:
```json

{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.csv",
  "mappings": {
    "dynamic": "false",
    "properties": {
      "@timestamp": {
        "type": "date",
        "path": "event_time",
        "format": "yyyy-MM-dd HH:mm:ss"
      },
      "request_id": { "type": "keyword" },
      "service": { "type": "keyword" },
      "status_code": { "type": "integer" }
    },
    "_id": {
      "path": "request_id"
    }
  }
}
```

The `mappings` block supports the following properties:
- `properties`: Columns keyed by their logical name. Each column requires a `type`.
  - `path`: Optional physical column name. Use it to expose a file column under a different logical name, including renaming a timestamp column to `@timestamp`.
- `format`: Optional date parsing pattern for a column with type `date`.
- `_id.path`: Optional source column whose value becomes the row's `_id`.
- `dynamic`: Controls undeclared columns. The default, `true`, overlays the declared columns on the inferred schema. Set it to `false` to treat the declaration as the complete schema, skip schema inference for text formats, and leave undeclared columns unavailable to queries.

<note>
  With `dynamic: false`, declared columns bind to file columns by name. In CSV and TSV files with a header row, each declared column binds to the header column of the same name (or the name given in `path`). A declared column absent from a file reads as null with a warning, not an error. Headerless files bind by position.For self-describing columnar formats such as Parquet, names bind to the file schema the same way, and a declared type is accepted when the file's type can be coerced to it. Only incompatible type pairs are rejected.
</note>


### Get a dataset

Retrieves a dataset by name.
<tab-set>
  <tab-item title="Console">
    ```json
    ```
  </tab-item>

  <tab-item title="curl">
    ```bash
    curl -X GET "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
      -H "Authorization: ApiKey ${API_KEY}"
    ```
  </tab-item>
</tab-set>


### List all datasets

Returns all registered datasets.
<tab-set>
  <tab-item title="Console">
    ```json
    ```
  </tab-item>

  <tab-item title="curl">
    ```bash
    curl -X GET "${ELASTICSEARCH_URL}/_query/dataset" \
      -H "Authorization: ApiKey ${API_KEY}"
    ```
  </tab-item>
</tab-set>


### Delete a dataset

Deletes a dataset by name.
<tab-set>
  <tab-item title="Console">
    ```json
    ```
  </tab-item>

  <tab-item title="curl">
    ```bash
    curl -X DELETE "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
      -H "Authorization: ApiKey ${API_KEY}"
    ```
  </tab-item>
</tab-set>


## Dataset settings

Dataset settings configure how a resource's format is read. They are specified in the `settings` object of a dataset definition. They divide into settings users commonly change and advanced settings with sensible defaults.

### Common settings

The following settings apply to all file-based data sources:

| Setting               | Default                    | Description                                                                                                                                                                             |
|-----------------------|----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `format`              | Auto-detect from extension | Override format detection. Valid values: `"parquet"`, `"csv"`, `"tsv"`, `"ndjson"`.                                                                                                     |
| `partition_detection` | `auto`                     | Partition detection mode. Valid values: `"auto"`, `"hive"`, `"none"`.                                                                                                                   |
| `schema_resolution`   | `union_by_name`            | How schemas are reconciled across multiple files. Valid values: `"first_file_wins"`, `"strict"`, `"union_by_name"`. Refer to [schema merge strategies](#schema-merge-strategies).       |
| `error_mode`          | `fail_fast`                | How malformed rows are handled. Valid values: `"fail_fast"`, `"skip_row"`, `"null_field"`. For Parquet, `skip_row` fills affected columns with null instead of skipping the entire row. |
| `max_errors`          | unbounded                  | Maximum malformed rows allowed before the query fails. Ignored when `error_mode` is `fail_fast`.                                                                                        |
| `max_error_ratio`     | `0.0`                      | Fraction of malformed rows allowed (0.0–1.0). Ignored when `error_mode` is `fail_fast`.                                                                                                 |


### CSV and TSV settings

**Commonly changed:**

| Setting      | Default (CSV / TSV) | Description                                                                                               |
|--------------|---------------------|-----------------------------------------------------------------------------------------------------------|
| `delimiter`  | `,` / `\t`          | The field separator.                                                                                      |
| `mode`       | `quoted` / `plain`  | A preset bundling quoting and escaping into one choice. Valid values: `"quoted"`, `"escaped"`, `"plain"`. |
| `header_row` | `true`              | Whether the first row names the columns.                                                                  |
| `null_value` | `""` (empty)        | The token read as null (for example `NULL`, `NA`, `\N`).                                                  |
| `encoding`   | `UTF-8`             | The file's character encoding.                                                                            |

**Advanced:**

| Setting              | Default (CSV / TSV) | Description                                                                                            |
|----------------------|---------------------|--------------------------------------------------------------------------------------------------------|
| `quote`              | `"` / none          | The quote character, or `"none"` to turn quoting off. An explicit value overrides the `mode` preset.   |
| `escape`             | `\` / none          | The escape character, or `"none"` to turn escaping off. An explicit value overrides the `mode` preset. |
| `comment`            | `//`                | Lines beginning with this prefix are skipped.                                                          |
| `column_prefix`      | `col`               | Prefix for generated column names when `header_row` is `false`.                                        |
| `datetime_format`    | ISO-8601            | The pattern used to parse date and time values.                                                        |
| `trim_spaces`        | `false`             | Whether to remove surrounding ASCII whitespace from string field values.                               |
| `multi_value_syntax` | `none`              | Whether bracketed multi-values are recognized. Valid values: `"none"`, `"brackets"`.                   |
| `max_field_size`     | `10485760` (10 MB)  | The maximum size of a single field. `0` is unlimited.                                                  |


### NDJSON settings


| Setting           | Default                     | Description                                                           |
|-------------------|-----------------------------|-----------------------------------------------------------------------|
| `segment_size`    | `4mb`                       | The unit a file is divided into for parallel reading. Minimum 64 KiB. |
| `datetime_format` | `strict_date_optional_time` | The pattern used to infer and parse date and time values.             |


### Parquet

Parquet is self-describing and is read with no settings in the common case. Its two settings are read-performance toggles, defaulted on.

| Setting                | Default | Description                                                                                                                                                                                                                                                                      |
|------------------------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `optimized_reader`     | `true`  | Uses vectorized decoding, page skipping, and I/O prefetch for the next row group. Leave enabled for normal scans. Disable it only to troubleshoot a suspected optimized-reader issue by using the baseline read path.                                                            |
| `late_materialization` | `true`  | When a filter can be pushed to the reader, reads predicate columns first and materializes other projected columns only for surviving rows. This is most useful for selective queries over wide files. Leave enabled unless you are troubleshooting filter or read-path behavior. |


## How schemas are inferred

Because federated data does not live in Elasticsearch, the system discovers schemas before queries can run. How this works depends on the file format:
- Parquet reads its schema from file metadata, which also provides column statistics and bloom filters that the engine uses to skip irrelevant data.
- For CSV, TSV, and NDJSON, schemas are inferred by sampling rows from the data files.


### Schema merge strategies

When a dataset spans multiple files, the files might have different schemas. Set `schema_resolution` in the dataset's `settings` object to choose a strategy:
- `union_by_name` (default): Merges schemas from all files by column name. Types are widened where possible: when two files disagree with no common type, the column falls back to `keyword` and the response carries a warning suggesting `strict` if you want the conflict to fail instead. `union_by_name` never fails on a type conflict. This is safer when files can vary, at the cost of reading and merging more file metadata.
- `first_file_wins`: Uses the first file alphabetically to define the schema and assumes later files match it. This is typically faster, but schema differences in later files can cause query errors or values to be read under the wrong assumptions.
- `strict`: Requires every file to have the same schema, apart from nullability, and returns an error when they differ. Use this when schema drift must fail explicitly.


## Next steps

- [Query your datasets](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-querying) to learn how partition pruning, filter pushdown, and column selection reduce the amount of data read from storage.
- If queries return unexpected types or missing values, check the [schema merge strategies](#schema-merge-strategies) or declare column types explicitly with [dataset mappings](#declare-a-dataset-mapping).
- [Tune cluster settings](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/esql/esql-data-federation-cluster-settings) to adjust file-discovery limits, caching TTLs, and request concurrency for your workload.