Upgrade

This section gives general recommendations for upgrading Beats shippers:

• Upgrade between minor versions
• Upgrade from 7.x to 8.x
• Troubleshoot Beats upgrade issues

If you’re upgrading other products in the stack, also read the Elastic Stack Installation and Upgrade Guide.

As a general rule, you can upgrade between minor versions (for example, 8.x to 8.y, where x < y) by simply installing the new release and restarting the Beat process. Beats typically maintain backwards compatibility for configuration settings and exported fields. Please review the release notes for potential exceptions.

Upgrading between non-consecutive major versions (e.g. 6.x to 8.x) is not supported.

Before upgrading your Beats, review the breaking changes and the Release notes.

If you’re upgrading other products in the stack, also read the Elastic Stack Installation and Upgrade Guide.

We recommend that you fully upgrade Elasticsearch and Kibana to version 8.0 before upgrading Beats. The Beats version must be lower than or equal to the Elasticsearch version. Beats cannot send data to older versions of Elasticsearch.

If you use the Uptime app in Kibana, make sure you add heartbeat-8* and synthetics-* to Uptime indices on the Settings page. The first index is used by newer versions of a Beat, while the latter is used by Fleet.

If you’re on Beats 7.0 through 7.16, upgrade the Elastic Stack and Beats to version 7.17 before proceeding with the 8.0 upgrade.

Upgrading between non-consecutive major versions (e.g. 6.x to 8.x) is not supported.

The upgrade procedure assumes that you have Beats 7.17 installed. If you’re on a previous 7.x version of Beats, upgrade to version 7.17 first. If you’re using other products in the Elastic Stack, upgrade Beats as part of the Elastic Stack upgrade process.

After upgrading to 7.17, go to Index Management in Kibana and verify that the 7.17 index template has been loaded into Elasticsearch.

If the 7.17 index template is not loaded, load it now.

If you created custom dashboards prior to version 7.17, you must upgrade them to 7.17 before proceeding. Otherwise, the dashboards will stop working because Kibana no longer provides the API used for dashboards in 7.x.

Before upgrading:

1. Stop the existing Beats process by using the appropriate command for your system.

2. Back up the data and config directories by copying them to another location.

   Tip

   The location of these directories depends on the installation method. To see the current paths, start the Beat from a terminal, and the data and config paths are printed at startup.

To upgrade using a Debian or RPM package:

• Use rpm or dpkg to install the new package. All files are installed in the appropriate location for the operating system and Beats config files are not overwritten.

To upgrade using a zip or compressed tarball:

1. Extract the zip or tarball to a new directory. This is critical if you are not using external config and data directories.

2. Set the following options in the Beats configuration file:

   • Set path.config to point to your external config directory. If you are not using an external config directory, copy your old configuration over to the new installation.
   • Set path.data to point to your external data directory. If you are not using an external data directory, copy your old data directory over to the new installation.
   • Set path.logs to point to the location where you want to store your logs. If you do not specify this setting, logs are stored in the directory you extracted the archive to.

Complete the upgrade tasks described in the following sections before restarting the Beats process.

Beats 8.0 comes with several backwards incompatible configuration changes. Before upgrading, review the 8.0 document. Also review the full list of breaking changes in the Beats version 8.0.0.

Where possible, we kept the old configuration options working, but deprecated them. However, deprecation was not always possible, so if you use any of the settings described under breaking changes, make sure you understand the alternatives that we provide.

Starting in version 8.0, the default Elasticsearch index templates configure data streams instead of traditional Elasticsearch indices. Data streams are optimized for storing append-only time series data. They are well-suited for logs, events, metrics, and other continuously generated data. However, unlike traditional Elasticsearch indices, data streams support create operations only; they do not support update and delete operations.

To use data streams, load the default index templates before ingesting any data into Elasticsearch.

beatname setup --index-management

beatname setup --index-management

./beatname setup --index-management

./beatname setup --index-management

docker run --rm --net="host" docker.elastic.co/beats/beatname:9.0.0-beta1 setup --index-management

PS > .\beatname.exe setup --index-management

If you’re not collecting time series data, you can continue to use Beats to send data to aliases and indices. To do this, create a custom index template and load it manually. To learn more about creating index templates, refer to Index templates.

We recommend that you load the 8.0 Kibana dashboards after upgrading Kibana and Beats. That way, you can take advantage of improvements added in 8.0. To load the dashboards, run:

beatname setup --dashboards

beatname setup --dashboards

./beatname setup --dashboards

./beatname setup --dashboards

docker run --rm --net="host" docker.elastic.co/beats/beatname:9.0.0-beta1 setup --dashboards

PS > .\beatname.exe setup --dashboards

All Elastic Beats send events with ECS-compliant field names. If you have any custom Kibana dashboards or visualizations that use old fields, adjust them now to use ECS field names.

To learn more about ECS, refer to the ECS overview.

After you’ve completed the migration, start the upgraded Beat. Use the command that works with your system.

Check the console and logs for errors.

In Kibana, go to Discover and verify that events are streaming into Elasticsearch.

This section describes common problems you might encounter when upgrading to Beats 8.x.

You can avoid some of these problems by reading Upgrade from 7.x to 8.x before upgrading Beats.

Data streams are designed for use cases where existing data is rarely, if ever, updated. You cannot send update or deletion requests for existing documents directly to a data stream.

If needed, you can update or delete documents by submitting requests directly to the document’s backing index. To learn how, refer to the docs about using a data stream.

Beats requires a timestamp field to send data to data streams. If the timestamp field added by Beats is inadvertently removed by a processor, Beats will be unable to index the event. To fix the problem, modify your processor configuration to avoid removing the timestamp field.

You may have run the Beat before loading the required index template. To clean up and start again:

1. Delete the index that was created when you ran the Beat. For example:

   DELETE metricbeat-9.0.0-beta1-2019.04.02*
   

   Warning

   Be careful when using wildcards to delete indices. Make sure the pattern matches only the indices you want to delete. The example shown here deletes all data indexed into the metricbeat-9.0.0-beta1 indices on 2019.04.02.

2. Delete the index template that was loaded earlier. For example:

   DELETE /_index_template/metricbeat-9.0.0-beta1
   

3. Load the correct index template. See Load the 8.0 Elasticsearch index templates.

4. Restart Beats.