Loading

Configure data retention with Streams

Managing data lifecycle across multiple indexes typically requires configuring index lifecycle management (ILM), data stream lifecycle, index templates, and index settings, each in a different place. Streams replaces this with a single UI so you can control storage and meet regulatory or compliance requirements.

The Data lifecycle tab (Retention prior to Stack v9.5) provides a single place to manage lifecycle policies for your streams:

  • Set retention periods per stream: Configure how long each stream retains data without touching ILM policies, index templates, or index settings directly.
  • Parent retention cascades to child streams: For wired streams, parent stream retention policies automatically apply to child streams. Override at the child level when a specific child stream needs different retention settings.
  • Monitor storage in one view: See storage size, ingestion averages, and phase distribution so you can align retention periods with storage costs and compliance requirements.
  • Reduce storage with downsampling: For time series data, replace high-resolution metrics with statistical summaries as data ages. This reduces storage costs and lets you retain data longer, with some loss of temporal precision.

To edit data retention in Elastic Stack, you need the following data stream level privileges:

  • manage_data_stream_lifecycle
  • manage_ilm

For more information, refer to Granting privileges for data streams and aliases.

Follow these steps to review your stream's storage footprint, select a lifecycle method, and apply the policy.

  1. Open Streams from the navigation menu or use the global search field.
  2. Select your stream from the list.
  3. Go to the Data lifecycle tab (Retention in earlier versions).

Before setting a retention policy, review the following panels to understand your data's footprint:

  • Storage size: Total data volume and document count for the stream, including data across all phases.
  • Ingestion averages: Estimated ingestion per day and per month, based on total stream size divided by stream age.
  • Data lifecycle or ILM policy data tiers: The amount of data in each phase (Hot, Warm, Cold, Frozen) so you can see where data is accumulating.
  • Ingestion over time: A chart of estimated ingestion volume over time to help spot trends or spikes. For streams with an active frozen phase, volume is split by phase.

Use this information to decide how long you need to retain data and which retention method best fits your cost and compliance requirements.

For more information on data retention, refer to Data stream lifecycle.

Select Edit lifecycle method to open the configuration options, then select one of the following methods:

  • Inherit retention: Use retention settings from the stream's index template (classic streams) or parent stream (wired streams).
    • Classic streams: This preserves existing data streams' behavior while still benefiting from Streams' other features.
    • Wired streams: Child streams automatically inherit lifecycle settings and updates from their parent stream.
  • Set a retention period: Define a minimum number of days before data is deleted. Data stays in the hot phase for best performance. Recommended when retention is specific to a single stream.
  • Follow an ILM policy: Apply an existing ILM policy to automate how data moves through lifecycle phases as it ages. Recommended when you want to share a policy across multiple streams.

To use the retention settings from the stream's index template (classic streams) or parent stream (wired streams) without setting a custom period or policy:

  1. Select Edit lifecycle method.
  2. Turn on Inherit from index template or parent stream.

For wired streams, you can override retention for a specific child stream by opening that stream's Data lifecycle tab and configuring a different method. The child stream will then use its own policy instead of inheriting from the parent.

  1. Select Edit retention method.
  2. Turn on Inherit from index template or parent stream.

For wired streams, you can override retention for a specific child stream by opening that stream's Retention tab and configuring a different method. The child stream will then use its own policy instead of inheriting from the parent.

To set a specific retention period:

  1. Select Edit lifecycle method.
  2. Turn off Inherit from index template or parent stream if enabled.
  3. Select Data stream lifecycle.
  4. From the Data stream lifecycle panel, select Add data phase → Delete phase.
  5. Set the delete phase to the number of days you want to retain data and select Apply.

To define a global default retention policy for serverless projects, refer to project settings.

  1. Select Edit retention method.
  2. Turn off Inherit from index template or parent stream, if enabled.
  3. Select Custom period.
  4. Set the number of days you want to retain data.

Select an existing ILM policy to automate how data moves through phases (Hot, Warm, Cold, Frozen) as it ages. ILM policies let you standardize data retention across Streams and other data streams.

To follow an existing policy:

  1. Select Edit lifecycle method.
  2. Turn off Inherit from index template or parent stream, if enabled.
  3. Select ILM policy, then select a predefined policy from the list.

After selecting a policy, you can configure data lifecycle phases directly from the Data lifecycle tab.

  1. Select Edit retention method.
  2. Turn off Inherit from index template or parent stream, if enabled.
  3. Select ILM policy, then choose a predefined policy from the list.

After selecting a policy, you can configure data lifecycle phases directly from the Retention tab.

If the policy you need doesn't exist, refer to Configure a lifecycle policy to create one.

The Data lifecycle tab (Retention in earlier versions) shows your stream's phases as a visual timeline. From here, you can edit existing phases or add new ones:

  • To edit an existing phase, select the phase from the Data phases timeline and select Edit.
  • To add a phase, select Add data phase, then choose a phase.

This opens Edit data phases, where you can configure or update your phases. For streams using a data stream lifecycle, you can add Frozen and Delete phases. For streams using an ILM, you can add any of the following phases:

Hot
The index is actively updated and queried. This is the default phase for all data. Options include enabling read-only access and downsampling.
Warm
The index is updated infrequently but still queried. Set the minimum age for data to move into this phase. Options include enabling read-only access and downsampling.
Cold
The index is rarely updated or queried, and slower query performance is acceptable. Set the minimum age for data to move into this phase. Options include enabling read-only access, downsampling, and searchable snapshots.
Frozen

The index is no longer updated and is queried rarely. Optimized for long-term retention at the lowest possible cost. Set the minimum age for data to move into this phase and configure a snapshot repository.

For streams using a data stream lifecycle, you need to set a default snapshot repository and the appropriate subscription before adding a frozen phase. If no default repository is set, you'll be prompted to set one. After setting it, select to resume.

Delete
Remove the index after a specified period of time. Set how long data is stored before deletion and optionally delete any associated searchable snapshots.

For more information, refer to index lifecycle management phases and actions.

Searchable snapshots let you search infrequently accessed, read-only data directly from a snapshot repository without needing replica shards, significantly reducing storage costs. A local data cache speeds up repeat access. They are best suited for historical data that requires infrequent access or archival.

Searchable snapshots are available in the Cold and Frozen phases.

For more information, refer to Searchable snapshots.

Downsampling reduces storage for time series data by replacing original metrics with statistical summaries at a higher sampling interval. For example, metrics sampled every 10 seconds can be consolidated into hourly data points as the data ages, significantly reducing storage while keeping the data queryable.

Downsampling is available in the Hot, Warm, and Cold phases and only applies to time series data streams.

For more information, refer to Downsampling concepts.

When a document fails to be ingested because of a processor error or a mapping conflict, Streams writes it to the failure store instead of dropping it. This lets you inspect what went wrong and fix issues using the actual failing documents, rather than losing data silently.

You can enable and configure failure store retention directly from the Data lifecycle tab (Retention in earlier versions). Select Enable failure store to turn it on and set the retention period for failed documents.

To review and resolve ingestion failures, refer to Manage data quality.