Load historical data into a time series data stream

By default, a Time series data stream (TSDS) works well for continuous, near-real-time ingestion. Only documents with @timestamp values that fall inside the time range of existing backing indices are accepted.

To import historical data into an existing TSDS, enable the data_stream.past_tsdb_index_creation_enabled cluster setting. You can then use the same APIs you use for live data. Elasticsearch creates the past backing indices needed to store past documents as they arrive. The documents must fall within the eligible write window. This is the period of time between "current" and the data stream retention limit or the first occurrence of a lifecycle action that makes a backing index read-only, whichever occurs first. Write-time deduplication and TSDS storage optimizations apply to historical data the same way they apply to live data.

Note

Users who trigger past index creation need the auto_configure index privilege. For details, refer to Secure a TSDS.

For data that falls within the eligible write window, point your migration or replay pipeline at the live time series data stream. Elasticsearch creates past backing indices as needed.

This approach works well when you're backfilling recent history alongside live ingestion, such as late-arriving metrics or a short bootstrap period.

Go to Set up a time series data stream for an example of setting up and loading historical data into a new time series data stream.

You can't load data older than the eligible write window directly into a TSDS. For example, if downsampling makes indices read-only after seven days, you can't backfill eighteen months of history into that same data stream.

Instead, create a separate historical TSDS without a lifecycle, load the data, then add a data stream lifecycle when the load is complete.

  1. Create an index template for the historical data stream

    Use the same mappings as your live TSDS, but don't include a lifecycle policy in the template. For example, use the create index template API:

    				PUT _index_template/metrics-historical
					{
  "index_patterns": ["metrics-historical-*"],
  "data_stream": {},
  "template": {
    "settings": {
      "index.mode": "time_series"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "sensor_id": { "type": "keyword", "time_series_dimension": true },
        "temperature": { "type": "half_float", "time_series_metric": "gauge" }
      }
    }
  }
}

  2. Create the historical data stream

    Create a data stream with a name that matches the pattern in the index template. For example, use the create a data stream API:

    				PUT _data_stream/metrics-historical-2024

  3. Index historical data

    Index historical data into the historical data stream while current data continues flowing into the original TSDS.

    Important

    Historical data must fit on the target tier as a whole before you enable data stream lifecycle. If you're importing a large data set, split it into batches. Each batch should fit within available disk space at indexing time.

  4. Add data stream lifecycle

    When the load is complete, add a data stream lifecycle to the historical data stream. For example, use the update data stream lifecycles API:

    				PUT _data_stream/metrics-historical-2024/_lifecycle
					{
  "enabled": true,
  "data_retention": "365d",
  "downsampling": [
    {
      "after": "7d",
      "fixed_interval": "10m"
    }
  ]
}

    Processing begins immediately and creates a backlog of downsampling work. If you include data_retention settings, data stream lifecycle deletes expired backing indices but does not remove the data stream itself.

  5. Query across both data streams

    Query both streams with a wildcard pattern or a data stream alias. For example, use the search API:

    				GET metrics-*/_search
					{
  "size": 10,
  "sort": [{ "@timestamp": "desc" }]
}

Delete historical data streams manually when their data is no longer needed.

Loading months of historical data can trigger significant storage use, force merge activity, and lifecycle processing in parallel. Verify that your cluster has enough available resources before you start.

When you enable a lifecycle on a data stream with many indices that qualify for downsampling, data stream lifecycle can queue multiple downsampling operations at once. To limit concurrent downsampling per data stream, configure the data_streams.lifecycle.downsampling.max_indices_in_progress cluster setting. For details, refer to Downsample with a data stream lifecycle.

Backfill and creation of past indices have the following limitations:

  • System data streams are excluded.
  • Cross-cluster replication (CCR) follower data streams rely on the leader data stream, so you can't backfill follower streams directly.
Edit page