﻿---
title: Set up a data stream
description: To set up a data stream, follow these steps: Create an index lifecycle policy, Create component templates, Create an index template, Create the data stream,...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4067/manage-data/data-store/data-streams/set-up-data-stream
products:
  - Elasticsearch
---

# Set up a data stream
To set up a data stream, follow these steps:
1. [Create an index lifecycle policy](#create-index-lifecycle-policy)
2. [Create component templates](#create-component-templates)
3. [Create an index template](#create-index-template)
4. [Create the data stream](#create-data-stream)
5. [Secure the data stream](#secure-data-stream)

You can also [convert an index alias to a data stream](#convert-index-alias-to-data-stream).
<important>
  If you use Fleet, Elastic Agent, or Logstash, skip this tutorial. They all set up data streams for you.For Fleet and Elastic Agent, refer to [Elastic Agent data streams for Fleet](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4067/reference/fleet/data-streams). For Logstash, refer to the [data streams settings](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-outputs-elasticsearch#plugins-outputs-elasticsearch-data_stream) for the `elasticsearch output` plugin.
</important>


## Create an index lifecycle policy

While optional, we recommend using ILM to automate the management of your data stream’s backing indices. ILM requires an index lifecycle policy.
To create an index lifecycle policy in Kibana:
1. Go to the **Index Lifecycle Policies** management page using the navigation menu or the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4067/explore-analyze/find-and-organize/find-apps-and-objects).
2. Click **Create policy**.

You can also use the [create lifecycle policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-put-lifecycle).
```json

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "60d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      },
      "delete": {
        "min_age": "735d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
```


## Create component templates

A data stream requires a matching index template. In most cases, you compose this index template using one or more component templates. You typically use separate component templates for mappings and index settings. This lets you reuse the component templates in multiple index templates.
When creating your component templates, include:
- A [`date`](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/mapping-reference/date) or [`date_nanos`](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/mapping-reference/date_nanos) mapping for the `@timestamp` field. If you don’t specify a mapping, Elasticsearch maps `@timestamp` as a `date` field with default options.
- Your lifecycle policy in the `index.lifecycle.name` index setting.

<tip>
  Use the [Elastic Common Schema (ECS)](https://docs-v3-preview.elastic.dev/elastic/ecs/tree/main/reference) when mapping your fields. ECS fields integrate with several Elastic Stack features by default.If you’re unsure how to map your fields, use [runtime fields](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4067/manage-data/data-store/mapping/define-runtime-fields-in-search-request) to extract fields from [unstructured content](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/mapping-reference/keyword#mapping-unstructured-content) at search time. For example, you can index a log message to a `wildcard` field and later extract IP addresses and other data from this field during a search.
</tip>

To create a component template in Kibana:
1. Go to the **Index Management** page using the navigation menu or the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4067/explore-analyze/find-and-organize/find-apps-and-objects).
2. In the **Index Templates** tab, click **Create component template**.

You can also use the [create component template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template).
```json
# Creates a component template for mappings

{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "date_optional_time||epoch_millis"
        },
        "message": {
          "type": "wildcard"
        }
      }
    }
  },
  "_meta": {
    "description": "Mappings for @timestamp and message fields",
    "my-custom-meta-field": "More arbitrary metadata"
  }
}

# Creates a component template for index settings

{
  "template": {
    "settings": {
      "index.lifecycle.name": "my-lifecycle-policy"
    }
  },
  "_meta": {
    "description": "Settings for ILM",
    "my-custom-meta-field": "More arbitrary metadata"
  }
}
```


## Create an index template

Use your component templates to create an index template. Specify:
- One or more index patterns that match the data stream’s name. We recommend using our [data stream naming scheme](/elastic/docs-content/pull/4067/reference/fleet/data-streams#data-streams-naming-scheme).
- That the template is data stream enabled.
- Any component templates that contain your mappings and index settings.
- A priority higher than `200` to avoid collisions with built-in templates. See [Avoid index pattern collisions](/elastic/docs-content/pull/4067/manage-data/data-store/templates#avoid-index-pattern-collisions).

To create an index template in Kibana:
1. Go to the **Index Management** page using the navigation menu or the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4067/explore-analyze/find-and-organize/find-apps-and-objects).
2. In the **Index Templates** tab, click **Create template**.

You can also use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template). Include the `data_stream` object to enable data streams.
```json

{
  "index_patterns": ["my-data-stream*"],
  "data_stream": { },
  "composed_of": [ "my-mappings", "my-settings" ],
  "priority": 500,
  "_meta": {
    "description": "Template for my time series data",
    "my-custom-meta-field": "More arbitrary metadata"
  }
}
```


## Create the data stream

[Indexing requests](/elastic/docs-content/pull/4067/manage-data/data-store/data-streams/use-data-stream#add-documents-to-a-data-stream)     [Indexing requests](/elastic/docs-content/pull/4067/manage-data/data-store/data-streams/use-data-stream#add-documents-to-a-data-stream)  add documents to a data stream. These requests must use an `op_type` of `create`. Documents must include a `@timestamp` field.
To automatically create your data stream, submit an indexing request that targets the stream’s name. This name must match one of your index template’s index patterns.
```json

{ "create":{ } }
{ "@timestamp": "2099-05-06T16:21:15.000Z", "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736" }
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:25:42.000Z", "message": "192.0.2.255 - - [06/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638" }


{
  "@timestamp": "2099-05-06T16:21:15.000Z",
  "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
}
```

You can also manually create the stream using the [create data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream). The stream’s name must still match one of your template’s index patterns.
```json
```

After it's been created, you can view and manage this and other data streams from the **Index Management** view. Refer to [Manage a data stream](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4067/manage-data/data-store/data-streams/manage-data-stream) for details.

## Secure the data stream

Use [index privileges](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/security-privileges#privileges-list-indices) to control access to a data stream. Granting privileges on a data stream grants the same privileges on its backing indices.
For an example, see [Data stream privileges](/elastic/docs-content/pull/4067/deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases#data-stream-privileges).

## Convert an index alias to a data stream

Prior to Elasticsearch 7.9, you’d typically use an [index alias with a write index](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4067/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams) to manage time series data. Data streams replace this functionality, require less maintenance, and automatically integrate with [data tiers](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4067/manage-data/lifecycle/data-tiers).
To convert an index alias with a write index to a data stream with the same name, use the [migrate to data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-migrate-to-data-stream). During conversion, the alias’s indices become hidden backing indices for the stream. The alias’s write index becomes the stream’s write index. The stream still requires a matching index template with data stream enabled.
```json
```


## Get information about a data stream

To get information about a data stream in Kibana:
1. Go to the **Index Management** page using the navigation menu or the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4067/explore-analyze/find-and-organize/find-apps-and-objects).
2. In the **Data Streams** tab, click the data stream’s name.

You can also use the [get data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-stream).
```json
```


## Delete a data stream

To delete a data stream and its backing indices in Kibana:
1. Go to the **Index Management** page using the navigation menu or the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4067/explore-analyze/find-and-organize/find-apps-and-objects).
2. In the **Data Streams** view, click the trash icon. The icon only displays if you have the `delete_index` [security privilege](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/security-privileges) for the data stream.

You can also use the [delete data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-stream).
```json
```