Remote Elasticsearch output

Remote Elasticsearch outputs allow you to send Elastic Agent data to a remote Elasticsearch cluster. This is especially useful for data that you want to keep separate and independent from the deployment where you use Fleet to manage the agents.

A remote Elasticsearch cluster supports the same output settings as your main Elasticsearch cluster.

To configure a remote Elasticsearch cluster for your Elastic Agent data:

1. In Fleet, open the Settings tab.

2. In the Outputs section, select Add output.

3. In the Add new output flyout, provide a name for the output and select Remote Elasticsearch as the output type.

4. In the Hosts field, add the URL that agents should use to access the remote Elasticsearch cluster.

   1. To find the remote host address, in the remote cluster open Kibana and go to Management → Fleet → Settings.
   2. Copy the Hosts value for the default output.
   3. Back in your main cluster, paste the value you copied into the output Hosts field.

5. Create a service token to access the remote cluster.

   1. Below the Service Token field, copy the API request.

   2. In the remote cluster, open the Kibana menu and go to Management > Dev Tools.

   3. Run the API request.

   4. Copy the value for the generated token.

   5. Back in your main cluster, paste the value you copied into the output Service Token field.

      Note

      To prevent unauthorized access the Elasticsearch Service Token is stored as a secret value. While secret storage is recommended, you can choose to override this setting and store the password as plain text in the agent policy definition. Secret storage requires Fleet Server version 8.12 or higher. This setting can also be stored as a secret value or as plain text for preconfigured outputs. See Preconfiguration settings in the Kibana Guide to learn more.

6. Choose whether integrations should automatically be synchronized on the remote Elasticsearch cluster. Refer to Automatic integrations synchronization below to configure this feature.

7. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, Elastic Agents use this output to send data if no other output is set in the agent policy.

8. Select which performance tuning settings you’d prefer in order to optimize Elastic Agent for throughput, scale, or latency, or leave the default balanced setting.

9. Add any advanced YAML configuration settings that you’d like for the output.

10. Click Save and apply settings.

After the output is created, you can update an Elastic Agent policy to use the new output and send data to the remote Elasticsearch cluster:

1. In Fleet, open the Agent policies tab.
2. Click the agent policy to edit it, then click Settings.
3. To send integrations data, set the Output for integrations option to use the output that you configured in the previous steps.
4. To send Elastic Agent monitoring data, set the Output for agent monitoring option to use the output that you configured in the previous steps.
5. Click Save changes.

The remote Elasticsearch cluster is now configured.

If you have chosen not to automatically synchronize integrations, you need to make sure that for any integrations that have been added to your Elastic Agent policy, the integration assets have been installed on the remote Elasticsearch cluster. Refer to Install and uninstall Elastic Agent integration assets for the steps.

Elastic Stack 9.1.0

When enabled, this feature keeps integrations synced between your main Elasticsearch cluster and remote Elasticsearch clusters.

This feature requires setting up cross-cluster replication, which is available to Platinum and Enterprise subscriptions. Remote clusters must be running the same version of Elasticsearch as the main cluster or a newer version that is compatible with cross-cluster replication.

Remote clusters require access to the Elastic Package Registry to install integrations.

1. Configure cross-cluster replication on the remote cluster.

   1. In the remote cluster, open the Kibana menu and go to Stack Management > Remote Clusters.
   2. Refer to Remote clusters to add your main cluster (where the remote Elasticsearch output is configured) as a remote cluster.
   3. Go to Stack Management > Cross-Cluster Replication.
   4. Create a follower index named fleet-synced-integrations-ccr-<output name> that replicates the fleet-synced-integrations leader index on the main cluster.
   5. Resume replication once the follower index is created.

2. In the main cluster, in the Remote Kibana URL field, add the Kibana URL of the remote cluster.

3. Create an API key to access Kibana on the remote cluster.

   1. Below the Remote Kibana API Key field, copy the API request.
   2. In the remote cluster, open the Kibana menu and go to Management > Dev Tools.
   3. Run the API request.
   4. Copy the encoded value of the generated API key.
   5. Back in the main cluster, paste the value you copied into the Remote Kibana API Key field.