Set the retention period for logging and metrics indices

Elastic Cloud Enterprise sets up index lifecycle management (ILM) policies for the ECE platform monitoring data it collects inside the logging-and-metrics system deployment.

By default, metrics indices are retained for two days and logging indices for eight days, as defined in the ece_metrics and ece_logs ILM policies of the deployment. This accounts for daily rollover plus the additional retention periods of one day for metrics and seven days for logs. These default policies and their associated index templates are managed by Elastic Cloud Enterprise and should not be modified.

You might need to adjust the retention period for one of the following reasons:

• If your business requires you to retain logs and metrics for longer than the default period.
• If the volume of logs and metrics collected is high enough to require reducing the amount of storage space consumed.

To customize the retention period, create a new ILM policy with the required settings and apply it to the target data streams. On ECE 4.1.1 or later, apply it through a component template, which is the recommended approach. On earlier versions, you can clone the relevant index template and configure it to use your custom ILM policy, though this requires repeating the procedure after upgrades that change template names.

The following list contains the names of the most relevant index templates and data streams in the ECE logging-and-metrics system deployment. You can check the entire list directly in Kibana on the Index Management -> Index templates page:

Starting from ECE 4.1.1, each index template in the logging-and-metrics cluster includes a composed_of array that references a set of reserved component template names. If you create a component template with one of these names, Elasticsearch merges its settings into new backing indices automatically. Because these names do not include a version tag, the customization persists across ECE upgrades without any additional action.

The following component template names are reserved for customization. Define only the one that matches the scope you need:

If you define multiple @custom component templates, the most specific one takes precedence: an index-specific template (for example, cluster-logs@custom) overrides ece-logs@custom, which in turn overrides ece-all@custom.

To customize the retention period:

1. Create a new ILM policy with the required retention settings.

2. Create a component template using the name that matches your target scope, and set index.lifecycle.name to the ILM policy you created. You can do this from Kibana Index Management → Component templates, or from the Kibana Console. For example, to apply a custom ILM policy to all cluster-logs data streams:

   				PUT _component_template/cluster-logs@custom
   					{
     "template": {
       "settings": {
         "index.lifecycle.name": "<MY_CUSTOM_ILM_POLICY>"
       }
     }
   }
   		

3. If you want the changes to take effect immediately, you can manually roll over the associated data stream using the Kibana Console. For example:

   				POST /cluster-logs-<version>/_rollover/
   		

To customize the retention period for the different data streams, create a new ILM policy with the required settings, and apply it to the relevant data sets as follows:

1. In Kibana, go to Index Management → Index Templates and identify the template that applies to the data stream or indices whose retention you want to change. Refer to Available index templates for a list of the most common templates.

2. Open the template's contextual menu and select Clone to create a new template. When cloning the template:

   1. Assign a higher priority to the new template so it takes precedence over the default template.
   2. In the Index settings section, set index.lifecycle.name to the custom ILM policy that has the required retention settings.
   Note

   Cloning an existing index template is recommended over creating one from scratch, so as to ensure that all required mappings and settings are preserved.

3. Save the new template and verify that it differs from the default template only in the priority and index.lifecycle.name settings.

4. If you want the changes to take effect immediately, you can manually roll over the associated data stream using the Kibana Console. For example:

   				POST /cluster-logs-<version>/_rollover/
   		

   After the rollover completes, a new backing index is created using the new index template and is associated with the custom ILM policy. You can verify this by checking the data stream information:

   				GET _data_stream/cluster-logs-8.18.8
   		

   Response example

   {
     "data_streams": [
       {
         "name": "cluster-logs-<version>",
         ...
         "indices": [
           {
             "index_name": ".ds-cluster-logs-<version>-<date>-000001",
             "index_uuid": "6hPZZ0YbTdaflfBZ9UkVUw",
             "prefer_ilm": true,
             "ilm_policy": "ece_logs",
             "managed_by": "Index Lifecycle Management"
           },
           ...
           {
             "index_name": ".ds-cluster-logs-<version>-<date>-000002",
             "index_uuid": "rGrlk1_iR-as_ZM3iw6EFg",
             "prefer_ilm": true,
             "ilm_policy": "<NEW_ILM_POLICY>",
             "managed_by": "Index Lifecycle Management"
           }
         ],
         "template": "<CLONED_TEMPLATE>",
         "ilm_policy": "<NEW_ILM_POLICY>",
         ...
       }
     ]
   }
   		

   1. Previous backing indices remain associated with the original ILM policy.
   2. The ilm_policy for the new backing index and the data stream should match the custom ILM policy.
   3. The template value should match the name of the new index template.