﻿---
title: Tutorial: Migrate ILM managed data stream to data stream lifecycle
description: In this tutorial we’ll look at migrating an existing data stream from Index Lifecycle Management (ILM) to data stream lifecycle. The existing ILM managed...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4232/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle
products:
  - Elasticsearch
---

# Tutorial: Migrate ILM managed data stream to data stream lifecycle
In this tutorial we’ll look at migrating an existing data stream from [Index Lifecycle Management (ILM)](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4232/manage-data/lifecycle/index-lifecycle-management) to [data stream lifecycle](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4232/manage-data/lifecycle/data-stream). The existing ILM managed backing indices will continue to be managed by ILM until they age out and get deleted by ILM; however, the new backing indices will be managed by data stream lifecycle. This way, a data stream is gradually migrated away from being managed by ILM to being managed by data stream lifecycle. As we’ll see, ILM and data stream lifecycle can co-manage a data stream; however, an index can only be managed by one system at a time.
<admonition title="Configure data retention policies for Streams" applies-to="{"stack": "ga 9.2, preview 9.1", "serverless": "ga"}">
  Starting with Elastic Stack version 9.2, the [**Streams**](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4232/solutions/observability/streams/streams) page provides a centralized interface for common data management tasks in Kibana, including tasks such as configuring data retention policies. You can choose to retain your data indefinitely, for a custom period, or by following an existing ILM policy. For more information, refer to [Manage data retention in Streams](/elastic/docs-content/pull/4232/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream#data-retention-streams).
</admonition>

To migrate a data stream from ILM to data stream lifecycle using APIs you need to run two steps:
1. Update the index template that’s backing the data stream to set [prefer_ilm](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings#index-lifecycle-prefer-ilm) to `false`, and to configure data stream lifecycle.
2. Configure the data stream lifecycle for the *existing* data stream using the [lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle).

For more details see the [migrate to data stream lifecycle](#migrate-from-ilm-to-dsl) section.

## Setup ILM managed data stream

Let’s first create a data stream with two backing indices managed by ILM. We first create an ILM policy:
```json

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
```

And let’s create an index template that’ll back the data stream and configures ILM:
```json

{
  "index_patterns": ["dsl-data-stream*"],
  "data_stream": { },
  "priority": 500,
  "template": {
    "settings": {
      "index.lifecycle.name": "pre-dsl-ilm-policy"
    }
  }
}
```

We’ll now index a document targetting `dsl-data-stream` to create the data stream and we’ll also manually rollover the data stream to have another generation index created:
```json

{
  "@timestamp": "2023-10-18T16:21:15.000Z",
  "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
}
```

```json
```

We’ll use the [GET _data_stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-stream) API to inspect the state of the data stream:
```json
```

Inspecting the response we’ll see that both backing indices are managed by ILM and that the next generation index will also be managed by ILM:
```json
{
  "data_streams": [
    {
      "name": "dsl-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000001",    
          "index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
          "prefer_ilm": true,                                       
          "ilm_policy": "pre-dsl-ilm-policy",                       
          "managed_by": "Index Lifecycle Management"                
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000002",
          "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 2,
      "status": "GREEN",
      "template": "dsl-data-stream-template",
      "next_generation_managed_by": "Index Lifecycle Management",   
      "prefer_ilm": true,                                           
      "ilm_policy": "pre-dsl-ilm-policy",                           
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}
```


## Migrate data stream to data stream lifecycle

To migrate the `dsl-data-stream` to data stream lifecycle we’ll have to execute two steps:
1. Update the index template that’s backing the data stream to set [prefer_ilm](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings#index-lifecycle-prefer-ilm) to `false`, and to configure data stream lifecycle.
2. Configure the data stream lifecycle for the *existing* `dsl-data-stream` using the [lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle).

<important>
  The data stream lifecycle configuration that’s added to the index template, being a data stream configuration, will only apply to **new** data streams. Our data stream exists already, so even though we added a data stream lifecycle configuration in the index template it will not be applied to `dsl-data-stream`.
</important>


Let’s update the index template:
```json

{
  "index_patterns": ["dsl-data-stream*"],
  "data_stream": { },
  "priority": 500,
  "template": {
    "settings": {
      "index.lifecycle.name": "pre-dsl-ilm-policy",
      "index.lifecycle.prefer_ilm": false             <1>
    },
    "lifecycle": {
      "data_retention": "7d"                          <2>
    }
  }
}
```

We’ve now made sure that new data streams will be managed by data stream lifecycle.
Let’s update our existing `dsl-data-stream` and configure data stream lifecycle:
```json

{
    "data_retention": "7d"
}
```

We can inspect the data stream to check that the next generation will indeed be managed by data stream lifecycle:
```json
```

```json
{
  "data_streams": [
    {
      "name": "dsl-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000001",
          "index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"                
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000002",
          "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"                
        }
      ],
      "generation": 2,
      "status": "GREEN",
      "template": "dsl-data-stream-template",
      "lifecycle": {
        "enabled": true,
        "data_retention": "7d",
        "effective_retention": "7d",
        "retention_determined_by": "data_stream_configuration"
      },
      "ilm_policy": "pre-dsl-ilm-policy",
      "next_generation_managed_by": "Data stream lifecycle",         
      "prefer_ilm": false,                                           
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}
```

We’ll now rollover the data stream to see the new generation index being managed by data stream lifecycle:
```json
```

```json
```

```json
{
  "data_streams": [
    {
      "name": "dsl-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000001",
          "index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"                
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000002",
          "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"                
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000003",
          "index_uuid": "PA_JquKGSiKcAKBA8abcd1",
          "prefer_ilm": false,                                      
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Data stream lifecycle"                     
        }
      ],
      "generation": 3,
      "status": "GREEN",
      "template": "dsl-data-stream-template",
      "lifecycle": {
        "enabled": true,
        "data_retention": "7d",
        "effective_retention": "7d",
        "retention_determined_by": "data_stream_configuration"
      },
      "ilm_policy": "pre-dsl-ilm-policy",
      "next_generation_managed_by": "Data stream lifecycle",
      "prefer_ilm": false,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}
```


## Migrate data stream back to ILM

We can easily change this data stream to be managed by ILM because we didn’t remove the ILM policy when we [updated the index template](#update-index-template-for-dsl).
We can achieve this in two ways:
1. [Delete the lifecycle](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-lifecycle) from the data streams
2. Disable data stream lifecycle by configuring the `enabled` flag to `false`.

Let’s implement option 2 and disable the data stream lifecycle:
```json

{
    "data_retention": "7d",
    "enabled": false <1>
}
```

```json
```

```json
{
  "data_streams": [
    {
      "name": "dsl-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000001",
          "index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000002",
          "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000003",
          "index_uuid": "PA_JquKGSiKcAKBA8abcd1",
          "prefer_ilm": false,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"                
        }
      ],
      "generation": 3,
      "status": "GREEN",
      "template": "dsl-data-stream-template",
      "lifecycle": {
        "enabled": false,                                          
        "data_retention": "7d"
      },
      "ilm_policy": "pre-dsl-ilm-policy",
      "next_generation_managed_by": "Index Lifecycle Management",  
      "prefer_ilm": false,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}
```

Had we removed the ILM policy from the index template when we [updated](#update-index-template-for-dsl) it, the write index of the data stream will now be `Unmanaged` because the index wouldn’t have the ILM policy configured to fallback onto.