Select external datasets for ES|QL Data Federation

Datasets share the same namespace as indices, data streams, aliases, and ES|QL views. A dataset cannot have the same name as any of them.

Federated data sources can read the following file formats:

The format is detected automatically from the file extension. You can override this in the dataset settings.

The following text formats are recognized by file extension:

A text resource is read uncompressed, or compressed with a codec identified from a trailing extension: clicks.csv, clicks.csv.gz, clicks.csv.zst.

Parquet declares its compression internally, per column chunk, so Parquet resources are not externally compressed. They are recognized by the .parquet and .parq extensions. The following internal codecs are supported:

• UNCOMPRESSED
• SNAPPY
• ZSTD
• GZIP
• LZ4_RAW
• LZ4 (legacy Hadoop-framed format, supported for reading only)

In Kibana, you create and manage datasets from the Datasets tab under Data management > ES|QL Data Federation.

The Datasets tab lists each dataset including:

• its data source and data source type
• its resource
• its description

From this tab you can search your datasets, filter by data source, add a new one, and edit or delete an existing one.

Click Add dataset to open a flyout where you define the dataset:

• Data source: the connected data source to read through.
• Name: a unique name for use in queries. Names must be lowercase and cannot begin with -, _, or +. A dataset cannot share a name with any existing index, data stream, alias, or view.
• Description: an optional description.
• Resource: the URI and glob pattern that selects the files to read.
• Format: the file format. This selection is required in the Kibana UI. The API can omit settings.format to auto-detect it from the file extension. Refer to supported file formats.

To configure how the format is read, expand Advanced settings. Refer to dataset settings.

To customize the inferred schema, rename columns, or override field types, use the dataset mappings API. Schema customization is not available in the UI.

Datasets are managed under the /_query/dataset endpoint. All dataset operations require the index manage privilege on the dataset name, or a fine-grained dataset privilege. Refer to manage credentials and privileges for details.

PUT creates a new dataset or replaces an existing one entirely.

				PUT /_query/dataset/access_logs
					{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.parquet",
  "description": "Production access logs",
  "settings": {
    "partition_detection": "hive"
  }
}
		

curl -X PUT "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.parquet",
  "description": "Production access logs",
  "settings": {
    "partition_detection": "hive"
  }
}'
		

By default, Elasticsearch infers a dataset's schema from its files. You can instead add an optional mappings block to the create or update request to control column names and types. Dataset mappings are currently available only through the API. The Kibana Add dataset flyout does not expose them.

The following example declares the complete schema, renames the physical event_time column to @timestamp, supplies its date format, and uses request_id as the row's _id:

				PUT /_query/dataset/access_logs
					{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.csv",
  "mappings": {
    "dynamic": "false",
    "properties": {
      "@timestamp": {
        "type": "date",
        "path": "event_time",
        "format": "yyyy-MM-dd HH:mm:ss"
      },
      "request_id": { "type": "keyword" },
      "service": { "type": "keyword" },
      "status_code": { "type": "integer" }
    },
    "_id": {
      "path": "request_id"
    }
  }
}
		

The mappings block supports the following properties:

• properties: Columns keyed by their logical name. Each column requires a type.
  • path: Optional physical column name. Use it to expose a file column under a different logical name, including renaming a timestamp column to @timestamp.
  • format: Optional date parsing pattern for a column with type date.
• _id.path: Optional source column whose value becomes the row's _id.
• dynamic: Controls undeclared columns. The default, true, overlays the declared columns on the inferred schema. Set it to false to treat the declaration as the complete schema, skip schema inference for text formats, and leave undeclared columns unavailable to queries.

Retrieves a dataset by name.

				GET /_query/dataset/access_logs
		

curl -X GET "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}"
		

Returns all registered datasets.

				GET /_query/dataset
		

curl -X GET "${ELASTICSEARCH_URL}/_query/dataset" \
  -H "Authorization: ApiKey ${API_KEY}"
		

Deletes a dataset by name.

				DELETE /_query/dataset/access_logs
		

curl -X DELETE "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}"
		

Dataset settings configure how a resource's format is read. They are specified in the settings object of a dataset definition. They divide into settings users commonly change and advanced settings with sensible defaults.

The following settings apply to all file-based data sources:

Commonly changed:

Advanced:

Parquet is self-describing and is read with no settings in the common case. Its two settings are read-performance toggles, defaulted on.

Because federated data does not live in Elasticsearch, the system discovers schemas before queries can run. How this works depends on the file format:

• Parquet reads its schema from file metadata, which also provides column statistics and bloom filters that the engine uses to skip irrelevant data.
• For CSV, TSV, and NDJSON, schemas are inferred by sampling rows from the data files.

When a dataset spans multiple files, the files might have different schemas. Set schema_resolution in the dataset's settings object to choose a strategy:

• union_by_name (default): Merges schemas from all files by column name. Types are widened where possible: when two files disagree with no common type, the column falls back to keyword and the response carries a warning suggesting strict if you want the conflict to fail instead. union_by_name never fails on a type conflict. This is safer when files can vary, at the cost of reading and merging more file metadata.
• first_file_wins: Uses the first file alphabetically to define the schema and assumes later files match it. This is typically faster, but schema differences in later files can cause query errors or values to be read under the wrong assumptions.
• strict: Requires every file to have the same schema, apart from nullability, and returns an error when they differ. Use this when schema drift must fail explicitly.

• Query your datasets to learn how partition pruning, filter pushdown, and column selection reduce the amount of data read from storage.
• If queries return unexpected types or missing values, check the schema merge strategies or declare column types explicitly with dataset mappings.
• Tune cluster settings to adjust file-discovery limits, caching TTLs, and request concurrency for your workload.