﻿---
title: Monitor Microsoft Azure with Beats
description: In this tutorial, you’ll learn how to monitor your Microsoft Azure deployments using Elastic Observability: Logs and Infrastructure metrics. You’ll learn...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/observability/cloud/monitor-microsoft-azure-with-beats
products:
  - Elastic Observability
applies_to:
  - Elastic Cloud Serverless: Generally available
  - Elastic Stack: Generally available
---

# Monitor Microsoft Azure with Beats
<note>
  **Are you sure you want to use Beats?**Elastic Agent is the recommended way to monitor Azure if you want to manage your agents centrally in Fleet. To learn how to use Elastic Agent, refer to [Monitor Microsoft Azure with Elastic Agent](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/observability/cloud/monitor-microsoft-azure-with-elastic-agent).
</note>

In this tutorial, you’ll learn how to monitor your Microsoft Azure deployments using Elastic Observability: Logs and Infrastructure metrics.

## What you’ll learn

You’ll learn how to:
- Create an Elasticsearch resource in the Azure portal.
- Ingest Azure platform logs using the native integration and view those logs in Kibana.
- Ingest logs and metrics from your virtual machines and view those logs and infrastructure metrics in Kibana.
- Ingest other metrics (such as billing) using the [Metricbeat Azure module](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/metricbeat/metricbeat-module-azure) and view those metrics in Kibana.


## Step 1: Create an Elasticsearch resource in the Azure portal

Microsoft Azure allows you to find, deploy, and manage Elasticsearch from within the Azure portal. The Microsoft Azure portal integration makes it faster and easier for you to experience the value of Elastic in your Azure environment. Behind the scenes, this process will provision a marketplace subscription with Elastic Cloud.

### Create an Elasticsearch resource

1. Log in to the [Azure portal](https://portal.azure.com/).
   <note>
   Ensure your Azure account is configured with **Owner** access on the subscription you want to deploy Elasticsearch. To learn more about Azure subscriptions, see the [Microsoft Azure documentation](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/add-change-subscription-administrator#assign-a-subscription-administrator).
   </note>
2. In the search bar, enter **Elasticsearch** and then select it.
3. Click **Create**.
4. Enter the **Subscription**, **Resource group**, and the **Resource name**.
5. Select a region and then click **Review + create**.
   ![Create Elastic resource](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-create-elastic-resource.png)
   <note>
   We will cover **logs** and **infrastructure metrics** later in this tutorial.
   </note>
6. To create the Elasticsearch deployment, click **Create**.
7. After your deployment is complete, click **Go to resource**. Here you can see and configure your deployment details. To access the cluster, click **Kibana**.
   ![Elastic resource](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-elastic-deployment.png)
8. To single sign-on directly into Elastic, select your Azure account.
9. To see if there is any available data, click **Observability**. There should be no data yet, but next, you will ingest logs.
   ![Kibana Observability page (no data)](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-kibana-observability-page-empty.png)


## Step 2: Ingest logs using the native integration

To ingest Azure subscription and resource logs into Elastic using the Microsoft Azure native integration is straightforward.
1. On to the Elasticsearch resource page in Azure, click **Ingest logs and metrics from Azure Services**.
   ![Click on Ingest logs and metrics from Azure Services](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-elastic-click-ingest-logs.png)
2. Check both checkboxes and click **Save**.
   ![Elastic configure logs and metrics](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-elastic-config-logs-metrics.png)
   <note>
   This configuration can also be applied during the Elastic resource creation. To make the concepts clearer, this tutorial separates the two steps.
   </note>
   <note>
   Native metrics collection is not fully supported yet and is discussed later.
   </note>
3. In Kibana, find the Observability **Overview** in the main menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/explore-analyze/find-and-organize/find-apps-and-objects). Refresh the page until you see some data. This may take a few minutes.
4. To analyze your subscription and resource logs, click **Show Logs**.


## Step 3: Ingest logs and metrics from your virtual machines.

1. Go to your Elastic resource and click **Virtual machines**.
   ![Elastic resource](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-elastic-deployment.png)
2. Select the VMs that you want to collect logs and metrics from, click **Install Extension**, and then click **OK**.
   ![Select VMs to collect logs and metrics from](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-elastic-vms.png)
3. Wait until it is installed and sending data (if the list does not update, click **Refresh** ). To see the logs from the VM, open **Discover** (find `Discover` in the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/explore-analyze/find-and-organize/find-apps-and-objects)).
   To view VM metrics, go to **Infrastructure inventory** and then select a VM. (To open **Infrastructure inventory**, find **Infrastructure** in the main menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/explore-analyze/find-and-organize/find-apps-and-objects).)

<note>
  Both logs and metrics are filtered by the VM name that you selected. To view the data for all the monitored VMs, delete the filter.
</note>


## Step 4: Ingest other Azure metrics using the Metricbeat Azure module

Some Azure metrics are not available via the native integration. If you want to collect those metrics, you need to use the [Azure Monitor REST API](https://docs.microsoft.com/en-us/rest/api/monitor/) and Metricbeat.
The Azure Monitor REST API allows you to get insights into your Azure resources using different operations. To access the Azure Monitor REST API you need to use the Azure Resource Manager authentication model. Therefore, you must authenticate all requests with Azure Active Directory (Azure AD). You can create the service principal using the [Azure portal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal) or [Azure PowerShell](https://docs.microsoft.com/en-us/powershell/azure/create-azure-service-principal-azureps?view=azps-2.7.0). Then, you need to grant access permission, which is detailed [here](https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles). This tutorial uses the Azure portal.

### Create an Azure service principal

1. Go to the [Azure Management Portal](https://portal.azure.com/). Search and click on **Azure Active Directory**.
   ![Search and click on Azure Active Directory](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-search-active-directory.png)
2. Click on **App registrations** in the navigation pane of the selected Active Directory and then click on **New registration**.
   ![Click on App registrations](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-click-app-registration.png)
3. Type the name of your application (this tutorial uses `monitor-azure`) and click on **Register** (leave all the other options with the default value).
   ![Register an application](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-register-app.png)
   Copy the **Application (client) ID**, and save it for future reference. This id is required to configure Metricbeat to connect to your Azure account.
4. Click on **Certificates & secrets**. Then, click on **New client secret** to create a new security key.
   ![Click on new client secret](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-click-client-secret.png)
5. Type a key description and select a key duration in the expire list. Click on **Add** to create a client secret. The next page will display the key value under the **Value** field. Copy the secret and save it (along with your Client ID) for future reference.
   <important>
   This is your only chance to copy this value. You can’t retrieve the key value after you leave the page.
   </important>


### Grant access permission for your service principal

After creating the Azure service principal you need to grant it the correct permission. You need `Reader` permission to configure Metricbeat to monitor your services.
1. On Azure Portal, search and click on **Subscriptions**.
   ![Search and click on Subscriptions](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-search-subscriptions.png)
2. In the Subscriptions page, click on your subscription.
3. Click on **Access control (IAM)** in the subscription navigation pane.
4. Click on **Add** and select **Add role assignment**.
5. Select the **Reader** role.
6. In the **Select** field, type the description name of the configured service principal (`monitor-azure`).
   ![Add role assignment](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-add-role-assignment.png)
7. Select the application and click on save to grant the service principal access to your subscription.


### Install and configure Metricbeat

To configure Metricbeat you need the Elasticsearch cluster details.
<applies-switch>
  <applies-item title="stack: ga" applies-to="Elastic Stack: Generally available">
    1. On the Elasticsearch resource page, click **Manage changes in Elastic Cloud**.
       ![Elastic resource](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-elastic-deployment.png)
    2. Copy the **Cloud ID** and keep it safe. You will use it later.
       ![Elastic Cloud deployment](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-kibana-deployment.png)
    3. Click **Security** and then click **Reset password**. Confirm, and copy the password. Keep it safe as you will use it later.
       ![Elastic Cloud security](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-kibana-security.png)
  </applies-item>

  <applies-item title="serverless: ga" applies-to="Elastic Cloud Serverless: Generally available">
    1. From your Serverless project, select the help icon, then select **Connection details**.
    2. Copy the **Elasticsearch endpoint** and keep it safe. You will use it later.
    3. Create an API key for authentication and keep it safe. You will use it later.
  </applies-item>
</applies-switch>

You can run Metricbeat on any machine. This tutorial uses a small Azure VM, **B2s** (2 vCPUs, 4 GB memory), with an Ubuntu distribution.

### Install Metricbeat

Download and install Metricbeat.
<tab-set>
  <tab-item title="DEB">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.3.1-amd64.deb
    sudo dpkg -i metricbeat-9.3.1-amd64.deb
    ```
  </tab-item>

  <tab-item title="RPM">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.3.1-x86_64.rpm
    sudo rpm -vi metricbeat-9.3.1-x86_64.rpm
    ```
  </tab-item>

  <tab-item title="MacOS">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.3.1-darwin-x86_64.tar.gz
    tar xzvf metricbeat-9.3.1-darwin-x86_64.tar.gz
    ```
  </tab-item>

  <tab-item title="Linux">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.3.1-linux-x86_64.tar.gz
    tar xzvf metricbeat-9.3.1-linux-x86_64.tar.gz
    ```
  </tab-item>

  <tab-item title="Windows">
    1. Download the [Metricbeat Windows zip file](https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.3.1-windows-x86_64.zip).
    2. Extract the contents of the zip file into `C:\Program Files`.
    3. Rename the _metricbeat-9.3.1-windows-x86_64_ directory to _Metricbeat_.
    4. Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select *Run As Administrator*).
    5. From the PowerShell prompt, run the following commands to install Metricbeat as a Windows service:

    ```shell
    PS > cd 'C:\Program Files\Metricbeat'
    PS C:\Program Files\Metricbeat> .\install-service-metricbeat.ps1
    ```

    <note>
      If script execution is disabled on your system, you need to set the execution policy for the current session to allow the script to run. For example: `PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-metricbeat.ps1`.
    </note>
  </tab-item>
</tab-set>


### Set up assets

<applies-switch>
  <applies-item title="stack: ga" applies-to="Elastic Stack: Generally available">
    Metricbeat comes with predefined assets for parsing, indexing, and visualizing your data. Run the following command to load these assets. It can take a few minutes.
    ```bash
    ./metricbeat setup -e -E 'cloud.id=YOUR_DEPLOYMENT_CLOUD_ID' -E 'cloud.auth=elastic:YOUR_SUPER_SECRET_PASS' 
    ```
  </applies-item>

  <applies-item title="serverless: ga" applies-to="Elastic Cloud Serverless: Generally available">
    Metricbeat comes with predefined assets for parsing, indexing, and visualizing your data. Run the following command to load these assets. It can take a few minutes.
    ```bash
    ./metricbeat setup -e -E 'output.elasticsearch.hosts=["https://hostname:port"]' -E 'output.elasticsearch.api_key=YOUR_API_KEY' 
    ```
  </applies-item>
</applies-switch>

<important>
  Setting up Metricbeat is an admin-level task that requires extra privileges. As a best practice, [use an administrator role to set up](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/metricbeat/privileges-to-setup-beats), and a more restrictive role for event publishing (which you will do next).
</important>


### Configure Metricbeat output

Next, you are going to configure Metricbeat output to Elasticsearch.
1. Use the Metricbeat keystore to store [secure settings](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/metricbeat/keystore). Store your connection details in the keystore.
   <applies-switch>
   <applies-item title="stack: ga" applies-to="Elastic Stack: Generally available">
   ```bash
   ./metricbeat keystore create
   echo -n "<Your Deployment Cloud ID>" | ./metricbeat keystore add CLOUD_ID --stdin
   ```
   </applies-item>

   <applies-item title="serverless: ga" applies-to="Elastic Cloud Serverless: Generally available">
   ```bash
   ./metricbeat keystore create
   echo -n "<Your Elasticsearch endpoint URL>" | ./metricbeat keystore add ES_HOST --stdin
   ```
   </applies-item>
   </applies-switch>
2. To store metrics in Elasticsearch with minimal permissions, create an API key to send data from Metricbeat to Elasticsearch. Log into Kibana and find `Dev Tools` in the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/explore-analyze/find-and-organize/find-apps-and-objects). From the **Console**, send the following request:
   ```json

   {
     "name": "metricbeat-monitor",
     "role_descriptors": {
       "metricbeat_writer": {
         "cluster": ["monitor", "read_ilm"],
         "index": [
           {
             "names": ["metricbeat-*"],
             "privileges": ["view_index_metadata", "create_doc"]
           }
         ]
       }
     }
   }
   ```
3. The response contains an `api_key` and an `id` field, which can be stored in the Metricbeat keystore in the following format: `id:api_key`.
   ```bash
   echo -n "IhrJJHMB4JmIUAPLuM35:1GbfxhkMT8COBB4JWY3pvQ" | ./metricbeat keystore add ES_API_KEY --stdin
   ```
   <note>
   Make sure you specify the `-n` parameter; otherwise, you will have painful debugging sessions due to adding a newline at the end of your API key.
   </note>
4. To see if both settings have been stored, run the following command:
   ```bash
   ./metricbeat keystore list
   ```
5. To configure Metricbeat to output to Elasticsearch, edit the `metricbeat.yml` configuration file. Add the following lines to the end of the file.
   <applies-switch>
   <applies-item title="stack: ga" applies-to="Elastic Stack: Generally available">
   ```yaml
   cloud.id: ${CLOUD_ID}
   output.elasticsearch:
   api_key: ${ES_API_KEY}
   ```
   </applies-item>

   <applies-item title="serverless: ga" applies-to="Elastic Cloud Serverless: Generally available">
   ```yaml
   output.elasticsearch:
   hosts: ["${ES_HOST}"]
   api_key: ${ES_API_KEY}
   ```
   </applies-item>
   </applies-switch>
6. Finally, test if the configuration is working. If it is not working, verify if you used the right credentials and add them again.
   ```bash
   ./metricbeat test output
   ```

Now that the output is working, you are going to set up the input (Azure).

### Configure Metricbeat Azure module

To collect metrics from Microsoft Azure, use the [Metricbeat Azure module](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/metricbeat/metricbeat-module-azure). This module periodically fetches monitoring metrics from Microsoft Azure using the [Azure Monitor REST API](https://docs.microsoft.com/en-us/rest/api/monitor/).
<warning>
  Extra Azure charges on metric queries my be generated by this module. See [additional notes about metrics and costs](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/metricbeat/metricbeat-module-azure#azure-api-cost) for more details.
</warning>

1. The azure module configuration needs three ids and one secret. Use the commands below to store each one of them in the keystore.
   ```bash
   echo -n "<client_id>" | ./metricbeat keystore add AZURE_CLIENT_ID --stdin
   echo -n "<client_secret>" | ./metricbeat keystore add AZURE_CLIENT_SECRET --stdin
   echo -n "<tenant_id>" | ./metricbeat keystore add AZURE_TENANT_ID --stdin
   echo -n "<subscription_id>" | ./metricbeat keystore add AZURE_SUBSCRIPTION_ID --stdin
   ```
   <note>
   You can find the `tenant_id` in the main Azure Active Directory page. You can find the `subscription_id` in the main Subscriptions page.
   </note>
2. Enable the Azure module.
   ```bash
   ./metricbeat modules enable azure
   ```
3. Edit the `modules.d/azure.yml` file to collect `billing` metrics.
   ```yaml
   - module: azure
     metricsets:
     - billing  
     enabled: true
     period: 24h  
     client_id: '${AZURE_CLIENT_ID:""}'
     client_secret: '${AZURE_CLIENT_SECRET:""}'
     tenant_id: '${AZURE_TENANT_ID:""}'
     subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
     refresh_list_interval: 600s
   ```
4. To check if Metricbeat can collect data, test the input by running the following command:
   ```bash
   ./metricbeat test modules azure
   ```
   Metricbeat will print `billing` metrics to the terminal, if the setup is correct.
   <note>
   If it returns a timeout error, try again. The `test modules` timeout is short.
   </note>
5. When the input and output are ready, start Metricbeat to collect the data.
   ```bash
   ./metricbeat -e
   ```
6. Finally, log into Kibana and open the **[Metricbeat Azure] Billing overview** dashboard. Keep in mind it collects data every 24 hours.
   ![Metricbeat azure billing overview dashboard](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5545/solutions/images/observability-monitor-azure-billing-overview-dashboard.png)