Select external datasets for ES|QL Data Federation

Datasets share the same namespace as indices, data streams, aliases, and ES|QL views. A dataset cannot have the same name as any of them.

Federated data sources can read the following file formats:

The format is detected automatically from the file extension. You can override this in the dataset settings.

The following text formats are recognized by file extension:

A text resource is read uncompressed, or compressed with a codec identified from a trailing extension: clicks.csv, clicks.csv.gz, clicks.csv.zst.

Parquet declares its compression internally, per column chunk, so Parquet resources are not externally compressed. They are recognized by the .parquet and .parq extensions. The following internal codecs are supported:

• UNCOMPRESSED
• SNAPPY
• ZSTD
• GZIP
• LZ4_RAW
• LZ4 (legacy Hadoop-framed format, supported for reading only)

In Kibana, you create and manage datasets from the Datasets tab under Data management > ES|QL Data Federation.

The Datasets tab lists each dataset including:

• its data source and data source type
• its resource
• its description

From this tab you can search your datasets, filter by data source, add a new one, and edit or delete an existing one.

Click Add dataset to open a flyout where you define the dataset:

• Data source: the connected data source to read through.
• Name: a unique name for use in queries. Names must be lowercase and cannot begin with -, _, or +. A dataset cannot share a name with any existing index, data stream, alias, or view.
• Description: an optional description.
• Resource: the URI and glob pattern that selects the files to read. Refer to resource patterns for the pattern language.
• Format: the file format. This selection is required in the Kibana UI. The API can omit settings.format to auto-detect it from the file extension. Refer to supported file formats.

To configure how the format is read, expand Advanced settings. Refer to dataset settings.

To customize the inferred schema, rename columns, or override field types, use the dataset mappings API. Schema customization is not available in the UI.

Datasets are managed under the /_query/dataset endpoint. All dataset operations require the index manage privilege on the dataset name, or a fine-grained dataset privilege. Refer to manage credentials and privileges for details.

PUT /_query/dataset/{name} creates a new dataset or replaces an existing one entirely.

				PUT /_query/dataset/access_logs
					{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.parquet",
  "description": "Production access logs",
  "settings": {
    "partition_detection": "hive"
  }
}
		

curl -X PUT "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.parquet",
  "description": "Production access logs",
  "settings": {
    "partition_detection": "hive"
  }
}'
		

By default, Elasticsearch infers a dataset's schema from its files. You can instead add an optional mappings block to the create or update request to control column names and types. Dataset mappings are currently available only through the API. The Kibana Add dataset flyout does not expose them.

The following example declares the complete schema, renames the physical event_time column to @timestamp, supplies its date format, and uses request_id as the row's _id:

				PUT /_query/dataset/access_logs
					{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.csv",
  "mappings": {
    "dynamic": "false",
    "properties": {
      "@timestamp": {
        "type": "date",
        "path": "event_time",
        "format": "yyyy-MM-dd HH:mm:ss"
      },
      "request_id": { "type": "keyword" },
      "service": { "type": "keyword" },
      "status_code": { "type": "integer" }
    },
    "_id": {
      "path": "request_id"
    }
  }
}
		

The mappings block supports the following properties:

• properties: Columns keyed by their logical name. Each column requires a type.
  • path: Optional physical column name. Use it to expose a file column under a different logical name, including renaming a timestamp column to @timestamp.
  • format: Optional date parsing pattern for a column with type date.
• _id.path: Optional source column whose value becomes the row's _id.
• dynamic: Controls undeclared columns. The default, true, overlays the declared columns on the inferred schema. Set it to false to treat the declaration as the complete schema, skip schema inference for text formats, and leave undeclared columns unavailable to queries.

GET /_query/dataset/{name} retrieves a dataset by name.

				GET /_query/dataset/access_logs
		

curl -X GET "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}"
		

GET /_query/dataset returns all registered datasets.

				GET /_query/dataset
		

curl -X GET "${ELASTICSEARCH_URL}/_query/dataset" \
  -H "Authorization: ApiKey ${API_KEY}"
		

DELETE /_query/dataset/{name} deletes a dataset by name.

				DELETE /_query/dataset/access_logs
		

curl -X DELETE "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}"
		

Dataset settings configure how a resource's format is read. They are specified in the settings object of a dataset definition. They divide into settings users commonly change and advanced settings with sensible defaults.

The following settings apply to all file-based data sources:

Object-store prefixes rarely hold only data. A Spark _SUCCESS marker, .crc sidecars, a _temporary/ or _delta_log/ subtree, or a folder placeholder the S3 console created all sit next to the files you want to read, and any object no reader can claim fails the whole query.

By default a dataset skips them. The default exclusion list is:

["**/_*", "**/.*", "**/_temporary/**", "**/_delta_log/**"]
		

The first two entries follow the convention Spark, Hive, and Trino use: a file whose name begins with _ or . is not data, at any depth. Because * never crosses a /, these entries match only the last path segment. They exclude files by name and cannot touch a directory, so partition directories are never affected: _dept=alpha/part-0.parquet and _foo/part-0.parquet are read under every partition detection mode. The last two entries name the two well-known directories whose contents look like data but are not: a failed Spark job's leftovers under _temporary, and a Delta Lake transaction log under _delta_log.

Each entry is an ordinary pattern in the same pattern language as resource, matched against the same string: the object's path relative to the resource's listing prefix. An object whose relative path matches any entry is dropped from the listing. Setting file_exclusions replaces the default list entirely.

Note that the resource pattern is your first filter: a dataset on **/*.parquet never sees a README.md at all, so there is nothing to exclude. This setting is for objects the resource pattern does match, most often data-shaped files in a directory you do not want read. A retired partition kept alongside the live ones is the common case:

access/year=2024/part-0.parquet      <- read
access/year=2025/part-0.parquet      <- read
access/backup_2024/part-0.parquet    <- also matches **/*.parquet, but is not current data
		

Restate the default and add the directory:

				PUT /_query/dataset/access_logs
					{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.parquet",
  "settings": {
    "file_exclusions": ["**/_*", "**/.*", "**/_temporary/**", "**/_delta_log/**", "backup_2024/**"]
  }
}
		

The added entry is matched against paths relative to the listing prefix s3://logs-bucket/access/, so backup_2024/** drops everything under that one directory. To drop directories of that name at any depth, write **/backup_2024/** instead.

Whenever exclusion drops something, the response carries a warning saying how many of the objects your resource selected were excluded, naming one of them and the entry that matched it:

2 of 4 objects matching the resource under [s3://logs-bucket/access/] were excluded by the
[file_exclusions] dataset setting, for example [_SUCCESS] which matched entry [**/_*]
		

The warning is emitted for the default list as well as for one you set, because a dataset that never configured exclusion is exactly the one where a missing file is hardest to explain. It is a single warning per listing however many objects were dropped, so it does not grow with the size of the prefix.

To turn exclusion off entirely, set "file_exclusions": []. Directory placeholder keys are still skipped (see below), so this reads every object the resource pattern matches except those.

Two rules are worth knowing. A malformed entry is rejected when you register the dataset, with an error naming the invalid pattern. And exclusion applies to wildcard discovery only. An object you name explicitly in resource is always read, because naming it is a request to read it, whether the resource carries no wildcard at all, names the object as one entry of a comma-separated resource, or names it through a finite brace pattern such as data/{a,b}.csv.

Objects whose key ends in / are directory placeholders rather than files, the empty markers an object-store console creates for a folder. They are skipped before any pattern is consulted, so no setting can bring them back.

Commonly changed:

Advanced:

Parquet is self-describing and has no format-specific dataset settings.

Because federated data does not live in Elasticsearch, the system discovers schemas before queries can run. How this works depends on the file format:

• Parquet reads its schema from file metadata, which also provides column statistics and bloom filters that the engine uses to skip irrelevant data.
• For CSV, TSV, and NDJSON, schemas are inferred by sampling rows from the data files.

When a dataset spans multiple files, the files might have different schemas. Set schema_resolution in the dataset's settings object to choose a strategy:

• union_by_name (default): Merges schemas from all files by column name. Types are widened where possible: when two files disagree with no common type, the column falls back to keyword and the response carries a warning suggesting strict if you want the conflict to fail instead. union_by_name never fails on a type conflict. This is safer when files can vary, at the cost of reading and merging more file metadata.
• first_file_wins: Uses the first file alphabetically to define the schema and assumes later files match it. This is typically faster, but schema differences in later files can cause query errors or values to be read under the wrong assumptions.
• strict: Requires every file to have the same schema, apart from nullability, and returns an error when they differ. Use this when schema drift must fail explicitly.

• Query your datasets to learn how partition pruning, filter pushdown, and column selection reduce the amount of data read from storage.
• If queries return unexpected types or missing values, check the schema merge strategies or declare column types explicitly with dataset mappings.
• Tune cluster settings to adjust file-discovery limits, caching TTLs, and request concurrency for your workload.