Set the retention period for logging and metrics indices

Elastic Cloud Enterprise sets up index lifecycle management (ILM) policies for the ECE platform monitoring data it collects inside the logging-and-metrics system deployment.

By default, metrics indices are retained for two days and logging indices for eight days, as defined in the ece_metrics and ece_logs ILM policies of the deployment. This accounts for daily rollover plus the additional retention periods of one day for metrics and seven days for logs. These default policies and their associated index templates are managed by Elastic Cloud Enterprise and should not be modified.

You might need to adjust the retention period for one of the following reasons:

• If your business requires you to retain logs and metrics for longer than the default period.
• If the volume of logs and metrics collected is high enough to require reducing the amount of storage space consumed.

The following list contains the names of the most relevant index templates and data streams in the ECE logging-and-metrics system deployment. You can check the entire list directly in Kibana on the Index Management -> Index templates page:

To customize the retention period for the different data streams, create a new ILM policy with the required settings, and apply it to the relevant data sets as follows:

1. In Kibana, go to Index Management → Index Templates and identify the template that applies to the data stream or indices whose retention you want to change. Refer to Available index templates for a list of the most common templates.

2. Open the template’s contextual menu and select Clone to create a new template. When cloning the template:

   1. Assign a higher priority to the new template so it takes precedence over the default template.
   2. In the Index settings section, set index.lifecycle.name to the custom ILM policy that has the required retention settings.
   Note

   Cloning an existing index template is recommended over creating one from scratch, so as to ensure that all required mappings and settings are preserved.

3. Save the new template and verify that it differs from the default template only in the priority and index.lifecycle.name settings.

4. If you want the changes to take effect immediately, you can manually roll over the associated data stream using the Kibana Console. For example:

   				POST /cluster-logs-<version>/_rollover/
   		

   After the rollover completes, a new backing index is created using the new index template and is associated with the custom ILM policy. You can verify this by checking the data stream information:

   				GET _data_stream/cluster-logs-8.18.8
   		

   Response example

   {
     "data_streams": [
       {
         "name": "cluster-logs-<version>",
         ...
         "indices": [
           {
             "index_name": ".ds-cluster-logs-<version>-<date>-000001",
             "index_uuid": "6hPZZ0YbTdaflfBZ9UkVUw",
             "prefer_ilm": true,
             "ilm_policy": "ece_logs",
             "managed_by": "Index Lifecycle Management"
           },
           ...
           {
             "index_name": ".ds-cluster-logs-<version>-<date>-000002",
             "index_uuid": "rGrlk1_iR-as_ZM3iw6EFg",
             "prefer_ilm": true,
             "ilm_policy": "<NEW_ILM_POLICY>",
             "managed_by": "Index Lifecycle Management"
           }
         ],
         "template": "<CLONED_TEMPLATE>",
         "ilm_policy": "<NEW_ILM_POLICY>",
         ...
       }
     ]
   }
   		

   1. Previous backing indices remain associated with the original ILM policy.
   2. The ilm_policy for the new backing index and the data stream should match the custom ILM policy.
   3. The template value should match the name of the new index template.