---
title: Migrating ILM-managed data streams to data stream lifecycle
description: This tutorial describes how to migrate a data stream from Index Lifecycle Management (ILM) to the newer data stream lifecycle. It explains migration steps,...
url: https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle
products:
- Elasticsearch
applies_to:
- Elastic Cloud Serverless: Generally available
- Elastic Stack: Generally available
---
# Migrating ILM-managed data streams to data stream lifecycle
This tutorial describes how to migrate a data stream from [Index Lifecycle Management (ILM)](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/lifecycle/index-lifecycle-management) to the newer [data stream lifecycle](https://www.elastic.co/elastic/docs-builder/docs/3028/manage-data/lifecycle/data-stream). It explains migration steps, compatibility considerations, and validation best practices.
During the migration, existing ILM managed backing indices continue to be managed by ILM until they age out and are deleted by ILM. Newly created backing indices are managed by data stream lifecycle. This way, a data stream is gradually migrated from being managed by ILM to being managed by data stream lifecycle. ILM and data stream lifecycle can co-manage a data stream, however an index can be managed by only one system at a time.
Starting with Elastic Stack version 9.2, the [**Streams**](https://www.elastic.co/elastic/docs-builder/docs/3028/solutions/observability/streams/streams) page provides a centralized interface for common data management tasks in Kibana, including tasks such as configuring data retention policies. You can choose to retain your data indefinitely, for a custom period, or by following an existing ILM policy. For more information, refer to [Manage data retention in Streams](/elastic/docs-builder/docs/3028/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream#data-retention-streams).
To migrate a data stream from ILM to data stream lifecycle using APIs you need to run two steps:
1. Update the index template that’s backing the data stream to set [prefer_ilm](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings#index-lifecycle-prefer-ilm) to `false`, and to configure data stream lifecycle.
2. Configure the data stream lifecycle for the *existing* data stream using the [lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle).
For more details refer to [Migrate to data stream lifecycle](#migrate-from-ilm-to-dsl).
## Setup ILM managed data stream
Let’s first create a data stream with two backing indices managed by ILM. We first create an ILM policy:
```json
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_primary_shard_size": "50gb"
}
}
},
"delete": {
"min_age": "7d",
"actions": {
"delete": {}
}
}
}
}
}
```
And let’s create an index template that’ll back the data stream and configures ILM:
```json
{
"index_patterns": ["dsl-data-stream*"],
"data_stream": { },
"priority": 500,
"template": {
"settings": {
"index.lifecycle.name": "pre-dsl-ilm-policy"
}
}
}
```
We’ll now index a document targetting `dsl-data-stream` to create the data stream and we’ll also manually rollover the data stream to have another generation index created:
```json
{
"@timestamp": "2023-10-18T16:21:15.000Z",
"message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
}
```
```json
```
We’ll use the [GET _data_stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-stream) API to inspect the state of the data stream:
```json
```
Inspecting the response we’ll see that both backing indices are managed by ILM and that the next generation index will also be managed by ILM:
```json
{
"data_streams": [
{
"name": "dsl-data-stream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-dsl-data-stream-2023.10.19-000001",
"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
"prefer_ilm": true,
"ilm_policy": "pre-dsl-ilm-policy",
"managed_by": "Index Lifecycle Management"
},
{
"index_name": ".ds-dsl-data-stream-2023.10.19-000002",
"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
"prefer_ilm": true,
"ilm_policy": "pre-dsl-ilm-policy",
"managed_by": "Index Lifecycle Management"
}
],
"generation": 2,
"status": "GREEN",
"template": "dsl-data-stream-template",
"next_generation_managed_by": "Index Lifecycle Management",
"prefer_ilm": true,
"ilm_policy": "pre-dsl-ilm-policy",
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"rollover_on_write": false
}
]
}
```
## Migrate data stream to data stream lifecycle
To migrate the `dsl-data-stream` to data stream lifecycle we’ll have to execute two steps:
1. Update the index template that’s backing the data stream to set [prefer_ilm](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings#index-lifecycle-prefer-ilm) to `false`, and to configure data stream lifecycle.
2. Configure the data stream lifecycle for the *existing* `dsl-data-stream` using the [lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle).
The data stream lifecycle configuration that’s added to the index template, being a data stream configuration, will only apply to **new** data streams. Our data stream exists already, so even though we added a data stream lifecycle configuration in the index template it will not be applied to `dsl-data-stream`.
Let’s update the index template:
```json
{
"index_patterns": ["dsl-data-stream*"],
"data_stream": { },
"priority": 500,
"template": {
"settings": {
"index.lifecycle.name": "pre-dsl-ilm-policy",
"index.lifecycle.prefer_ilm": false <1>
},
"lifecycle": {
"data_retention": "7d" <2>
}
}
}
```
We’ve now made sure that new data streams will be managed by data stream lifecycle.
Let’s update our existing `dsl-data-stream` and configure data stream lifecycle:
```json
{
"data_retention": "7d"
}
```
We can inspect the data stream to check that the next generation will indeed be managed by data stream lifecycle:
```json
```
```json
{
"data_streams": [
{
"name": "dsl-data-stream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-dsl-data-stream-2023.10.19-000001",
"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
"prefer_ilm": true,
"ilm_policy": "pre-dsl-ilm-policy",
"managed_by": "Index Lifecycle Management"
},
{
"index_name": ".ds-dsl-data-stream-2023.10.19-000002",
"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
"prefer_ilm": true,
"ilm_policy": "pre-dsl-ilm-policy",
"managed_by": "Index Lifecycle Management"
}
],
"generation": 2,
"status": "GREEN",
"template": "dsl-data-stream-template",
"lifecycle": {
"enabled": true,
"data_retention": "7d",
"effective_retention": "7d",
"retention_determined_by": "data_stream_configuration"
},
"ilm_policy": "pre-dsl-ilm-policy",
"next_generation_managed_by": "Data stream lifecycle",
"prefer_ilm": false,
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"rollover_on_write": false
}
]
}
```
We’ll now rollover the data stream to see the new generation index being managed by data stream lifecycle:
```json
```
```json
```
```json
{
"data_streams": [
{
"name": "dsl-data-stream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-dsl-data-stream-2023.10.19-000001",
"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
"prefer_ilm": true,
"ilm_policy": "pre-dsl-ilm-policy",
"managed_by": "Index Lifecycle Management"
},
{
"index_name": ".ds-dsl-data-stream-2023.10.19-000002",
"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
"prefer_ilm": true,
"ilm_policy": "pre-dsl-ilm-policy",
"managed_by": "Index Lifecycle Management"
},
{
"index_name": ".ds-dsl-data-stream-2023.10.19-000003",
"index_uuid": "PA_JquKGSiKcAKBA8abcd1",
"prefer_ilm": false,
"ilm_policy": "pre-dsl-ilm-policy",
"managed_by": "Data stream lifecycle"
}
],
"generation": 3,
"status": "GREEN",
"template": "dsl-data-stream-template",
"lifecycle": {
"enabled": true,
"data_retention": "7d",
"effective_retention": "7d",
"retention_determined_by": "data_stream_configuration"
},
"ilm_policy": "pre-dsl-ilm-policy",
"next_generation_managed_by": "Data stream lifecycle",
"prefer_ilm": false,
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"rollover_on_write": false
}
]
}
```
## Migrate data stream back to ILM
We can easily change this data stream to be managed by ILM because we didn’t remove the ILM policy when we [updated the index template](#update-index-template-for-dsl).
We can achieve this in two ways:
1. [Delete the lifecycle](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-lifecycle) from the data streams
2. Disable data stream lifecycle by configuring the `enabled` flag to `false`.
Let’s implement option 2 and disable the data stream lifecycle:
```json
{
"data_retention": "7d",
"enabled": false <1>
}
```
```json
```
```json
{
"data_streams": [
{
"name": "dsl-data-stream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-dsl-data-stream-2023.10.19-000001",
"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
"prefer_ilm": true,
"ilm_policy": "pre-dsl-ilm-policy",
"managed_by": "Index Lifecycle Management"
},
{
"index_name": ".ds-dsl-data-stream-2023.10.19-000002",
"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
"prefer_ilm": true,
"ilm_policy": "pre-dsl-ilm-policy",
"managed_by": "Index Lifecycle Management"
},
{
"index_name": ".ds-dsl-data-stream-2023.10.19-000003",
"index_uuid": "PA_JquKGSiKcAKBA8abcd1",
"prefer_ilm": false,
"ilm_policy": "pre-dsl-ilm-policy",
"managed_by": "Index Lifecycle Management"
}
],
"generation": 3,
"status": "GREEN",
"template": "dsl-data-stream-template",
"lifecycle": {
"enabled": false,
"data_retention": "7d"
},
"ilm_policy": "pre-dsl-ilm-policy",
"next_generation_managed_by": "Index Lifecycle Management",
"prefer_ilm": false,
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"rollover_on_write": false
}
]
}
```
Had we removed the ILM policy from the index template when we [updated](#update-index-template-for-dsl) it, the write index of the data stream will now be `Unmanaged` because the index wouldn’t have the ILM policy configured to fallback onto.