﻿---
title: Monitor Google Cloud Platform (GCP)
description: In this guide, you’ll learn how to monitor your Google Cloud Platform (GCP) deployments using Elastic Observability: Logs and Infrastructure metrics. 
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/observability/cloud/monitor-google-cloud-platform-gcp
products:
  - Elastic Observability
---

# Monitor Google Cloud Platform (GCP)
In this guide, you’ll learn how to monitor your Google Cloud Platform (GCP) deployments using Elastic Observability: Logs and Infrastructure metrics.
<note>
  If you don’t want to provision VM and install data shippers due to process and management overhead, you can skip this step and ingest logs directly from Pub/Sub in the Google Cloud Console to Elastic with [GCP Dataflow Templates](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/observability/cloud/gcp-dataflow-templates).
</note>


## What you’ll learn

You’ll learn how to:
- Set up a GCP Service Account.
- Ingest metrics using the [Metricbeat Google Cloud Platform module](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/metricbeat/metricbeat-module-gcp) and view those metrics in Kibana.
- Export GCP audit logs through Pub/Sub topics.
- Ingest logs using the [Filebeat Google Cloud module](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/filebeat/filebeat-module-gcp) and view those logs in Kibana.


## Before you begin

Create an [Elastic Cloud Hosted](https://cloud.elastic.co/registration?page=docs&placement=docs-body) deployment. The deployment includes an Elasticsearch cluster for storing and searching your data, and Kibana for visualizing and managing your data.

## Step 1: Setup a Service Account

Google Cloud Platform implements [service accounts](https://cloud.google.com/compute/docs/access/service-accounts) as a way to access APIs securely. To monitor GCP with Elastic, you will need a service account. The easiest way is to use a predefined service account that GCP [creates automatically](https://cloud.google.com/compute/docs/access/service-accounts?hl=en#default_service_account). Alternatively, you can create a new service account. This tutorial creates a new one.
First, to access the service account menu, click **Menu** → **IAM & Admin** → **Service Accounts**.
![Service account menu](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-service-account-menu.png)

Next, click **Create Service Account**. Define the new service account name (for example, "gcp-monitor") and the description (for example, "Service account to monitor GCP services using the Elastic Stack").
![Service account name](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-service-account-name.png)

<important>
  Make sure to select the correct roles.
</important>

To monitor GCP services, you need to add these roles to the service account:
**Compute Viewer**:
![Service account roles compute viewer](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-service-account-roles-compute-viewer.png)

**Monitoring Viewer**:
![Service account roles monitoring viewer](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-service-account-roles-monitoring-viewer.png)

**Pub/Sub Subscriber**:
![Service account roles pub/sub subscriber](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-service-account-roles-pubsub-subscriber.png)

The final result should be the following:
![Service account roles result](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-service-account-roles-final.png)

Click **Continue**, then skip granting users access to this service. Finally, click **Done**. The service account is now ready to be used.
Next, to use the service account, click **Manage keys**.
![Service account manage keys](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-service-account-manage-keys.png)

Then, add a new JSON key type by selecting **Create new key**.
![Service account create key](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-service-account-create-key.png)

After that, the credential file is downloaded. Keep this file in an accessible place to use later.

## Step 2: Install and configure Metricbeat

<note>
  This tutorial assumes the Elastic cluster is already running. Make sure you have your **cloud ID** and your **credentials** on hand.
</note>

To monitor GCP using the Elastic Stack, you need two main components: an Elastic deployment to store and analyze the data and an agent to collect and ship the data.
Two agents can be used to monitor GCP: Metricbeat is used to monitor metrics, and Filebeat to monitor logs. You can run the agents on any machine. This tutorial uses a small GCP instance, e2-small (2 vCPUs, 2 GB memory), with an Ubuntu distribution.

### Install Metricbeat

Download and install Metricbeat.
<tab-set>
  <tab-item title="DEB">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.2.2-amd64.deb
    sudo dpkg -i metricbeat-9.2.2-amd64.deb
    ```
  </tab-item>

  <tab-item title="RPM">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.2.2-x86_64.rpm
    sudo rpm -vi metricbeat-9.2.2-x86_64.rpm
    ```
  </tab-item>

  <tab-item title="MacOS">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.2.2-darwin-x86_64.tar.gz
    tar xzvf metricbeat-9.2.2-darwin-x86_64.tar.gz
    ```
  </tab-item>

  <tab-item title="Linux">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.2.2-linux-x86_64.tar.gz
    tar xzvf metricbeat-9.2.2-linux-x86_64.tar.gz
    ```
  </tab-item>

  <tab-item title="Windows">
    1. Download the [Metricbeat Windows zip file](https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.2.2-windows-x86_64.zip).
    2. Extract the contents of the zip file into `C:\Program Files`.
    3. Rename the `metricbeat-[version]-windows-x86_64` directory to `Metricbeat`.
    4. Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select *Run As Administrator*).
    5. From the PowerShell prompt, run the following commands to install Metricbeat as a Windows service:

    ```shell
    PS > cd 'C:\Program Files\Metricbeat'
    PS C:\Program Files\Metricbeat> .\install-service-metricbeat.ps1
    ```

    <note>
      If script execution is disabled on your system, you need to set the execution policy for the current session to allow the script to run. For example: `PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-metricbeat.ps1`.
    </note>
  </tab-item>
</tab-set>


### Set up assets

Metricbeat comes with predefined assets for parsing, indexing, and visualizing your data. Run the following command to load these assets. It may take a few minutes.
```bash
./metricbeat setup -e -E 'cloud.id=YOUR_DEPLOYMENT_CLOUD_ID' -E 'cloud.auth=elastic:YOUR_SUPER_SECRET_PASS' 
```

<important>
  Setting up Metricbeat is an admin-level task that requires extra privileges. As a best practice, [use an administrator role to set up](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/metricbeat/privileges-to-setup-beats), and a more restrictive role for event publishing (which you will do next).
</important>


### Configure Metricbeat output

Next, you are going to configure Metricbeat output to Elastic Cloud.
1. Use the Metricbeat keystore to store [secure settings](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/metricbeat/keystore). Store the Cloud ID in the keystore.
   ```bash
   ./metricbeat keystore create
   echo -n "<Your Deployment Cloud ID>" | ./metricbeat keystore add CLOUD_ID --stdin
   ```
2. To store metrics in Elasticsearch with minimal permissions, create an API key to send data from Metricbeat to Elastic Cloud. Log into Kibana (you can do so from the Cloud Console without typing in any permissions) and find `Dev Tools` in the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/explore-analyze/find-and-organize/find-apps-and-objects). From the **Console**, send the following request:
   ```json

   {
     "name": "metricbeat-monitor",
     "role_descriptors": {
       "metricbeat_writer": {
         "cluster": ["monitor", "read_ilm"],
         "index": [
           {
             "names": ["metricbeat-*"],
             "privileges": ["view_index_metadata", "create_doc"]
           }
         ]
       }
     }
   }
   ```
3. The response contains an `api_key` and an `id` field, which can be stored in the Metricbeat keystore in the following format: `id:api_key`.
   ```bash
   echo -n "IhrJJHMB4JmIUAPLuM35:1GbfxhkMT8COBB4JWY3pvQ" | ./metricbeat keystore add ES_API_KEY --stdin
   ```
   <note>
   Make sure you specify the `-n` parameter; otherwise, you will have painful debugging sessions due to adding a newline at the end of your API key.
   </note>
4. To see if both settings have been stored, run the following command:
   ```bash
   ./metricbeat keystore list
   ```
5. To configure Metricbeat to output to Elastic Cloud, edit the `metricbeat.yml` configuration file. Add the following lines to the end of the file.
   ```yaml
   cloud.id: ${CLOUD_ID}
   output.elasticsearch:
     api_key: ${ES_API_KEY}
   ```
6. Finally, test if the configuration is working. If it is not working, verify if you used the right credentials and add them again.
   ```bash
   ./metricbeat test output
   ```

Now that the output is working, you are going to set up the input (GCP).

## Step 3: Configure Metricbeat Google Cloud Platform module

To collect metrics from Google Cloud Platform, use the [Google Cloud Platform](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/metricbeat/metricbeat-module-gcp) module. This module periodically fetches monitoring metrics from Google Cloud Platform using [Stackdriver Monitoring API](https://cloud.google.com/monitoring/api/metrics_gcp) for Google Cloud Platform services.
<warning>
  Extra GCP charges on Stackdriver Monitoring API requests may be generated by this module. See [rough estimation of the number of API calls](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/metricbeat/metricbeat-module-gcp#gcp-api-requests) for more details.
</warning>

1. Enable the GCP module.
   ```bash
   ./metricbeat modules enable gcp
   ```
2. Edit the `modules.d/gcp.yml` file to configure which metrics to collect.
   ```yaml
   - module: gcp
     metricsets:
       - compute 
     zone: "" 
     project_id: "your-project-id" 
     period: 1m 
     credentials_file_path: "/home/ubuntu/credentials.json" 
   ```
3. To check if Metricbeat can collect data, test the input by running the following command:
   ```bash
   ./metricbeat test modules gcp
   ```
   Metricbeat will print GCP metrics to the terminal, if the setup is correct.
4. When the input and output are ready, start Metricbeat to collect the data.
   ```bash
   ./metricbeat -e
   ```
5. Finally, log into Kibana and open the **[Metricbeat GCP] Compute Overview** dashboard.
   ![Metricbeat compute overview dashboard](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-compute-overview-dashboard.png)


## Step 4: Install and configure Filebeat

Now that Metricbeat is up and running, configure Filebeat to collect Google Cloud logs.

#### Install Filebeat

Download and install Filebeat.
<tab-set>
  <tab-item title="DEB">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.2.2-amd64.deb
    sudo dpkg -i filebeat-9.2.2-amd64.deb
    ```
  </tab-item>

  <tab-item title="RPM">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.2.2-x86_64.rpm
    sudo rpm -vi filebeat-9.2.2-x86_64.rpm
    ```
  </tab-item>

  <tab-item title="MacOS">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.2.2-darwin-x86_64.tar.gz
    tar xzvf filebeat-9.2.2-darwin-x86_64.tar.gz
    ```
  </tab-item>

  <tab-item title="Linux">
    ```shell
    curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.2.2-linux-x86_64.tar.gz
    tar xzvf filebeat-9.2.2-linux-x86_64.tar.gz
    ```
  </tab-item>

  <tab-item title="Windows">
    1. Download the [Filebeat Windows zip file](https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.2.2-windows-x86_64.zip).
    2. Extract the contents of the zip file into `C:\Program Files`.
    3. Rename the `filebeat-[stack-version]-windows-x86_64` directory to `Filebeat`.
    4. Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select *Run As Administrator*).
    5. From the PowerShell prompt, run the following commands to install Filebeat as a Windows service:

    ```shell
    PS > cd 'C:\Program Files\Filebeat'
    PS C:\Program Files\Filebeat> .\install-service-filebeat.ps1
    ```

    <note>
      If script execution is disabled on your system, you need to set the execution policy for the current session to allow the script to run. For example: `PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-filebeat.ps1`.
    </note>
  </tab-item>
</tab-set>


#### Set up assets

Filebeat comes with predefined assets for parsing, indexing, and visualizing your data. Run the following command to load these assets. It may take a few minutes.
```bash
./filebeat setup -e -E 'cloud.id=YOUR_DEPLOYMENT_CLOUD_ID' -E 'cloud.auth=elastic:YOUR_SUPER_SECRET_PASS' 
```

<important>
  Setting up Filebeat is an admin-level task that requires extra privileges. As a best practice, [use an administrator role to set up](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/filebeat/privileges-to-setup-beats) and a more restrictive role for event publishing (which you will do next).
</important>


#### Configure Filebeat output

Next, you are going to configure Filebeat output to Elastic Cloud.
1. Use the Filebeat keystore to store [secure settings](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/filebeat/keystore). Store the Cloud ID in the keystore.
   ```bash
   ./filebeat keystore create
   echo -n "<Your Deployment Cloud ID>" | ./filebeat keystore add CLOUD_ID --stdin
   ```
2. To store logs in Elasticsearch with minimal permissions, create an API key to send data from Filebeat to Elastic Cloud. Log into Kibana (you can do so from the Cloud Console without typing in any permissions) and find `Dev Tools` in the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/explore-analyze/find-and-organize/find-apps-and-objects). Send the following request:
   ```json

   {
     "name": "filebeat-monitor-gcp",
     "role_descriptors": {
       "filebeat_writer": {
         "cluster": [
           "monitor",
           "read_ilm",
           "cluster:admin/ingest/pipeline/get", <1>
           "cluster:admin/ingest/pipeline/put" <1>
         ],
         "index": [
           {
             "names": ["filebeat-*"],
             "privileges": ["view_index_metadata", "create_doc"]
           }
         ]
       }
     }
   }
   ```
3. The response contains an `api_key` and an `id` field, which can be stored in the Filebeat keystore in the following format: `id:api_key`.
   ```bash
   echo -n "IhrJJHMB4JmIUAPLuM35:1GbfxhkMT8COBB4JWY3pvQ" | ./filebeat keystore add ES_API_KEY --stdin
   ```
   <note>
   Make sure you specify the `-n` parameter; otherwise, you will have painful debugging sessions due to adding a newline at the end of your API key.
   </note>
4. To see if both settings have been stored, run the following command:
   ```bash
   ./filebeat keystore list
   ```
5. To configure Filebeat to output to Elastic Cloud, edit the `filebeat.yml` configuration file. Add the following lines to the end of the file.
   ```yaml
   cloud.id: ${CLOUD_ID}
   output.elasticsearch:
     api_key: ${ES_API_KEY}
   ```
6. Finally, test if the configuration is working. If it is not working, verify that you used the right credentials and, if necessary, add them again.
   ```bash
   ./filebeat test output
   ```

Now that the output is working, you are going to set up the input (GCP).

## Step 5: Configure Filebeat Google Cloud module

To collect logs from Google Cloud Platform, use the [Google Cloud Platform](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/filebeat/filebeat-module-gcp) module. This module periodically fetches logs that have been exported from Stackdriver to a Google Pub/Sub topic sink. There are three available filesets: `audit`, `vpcflow`, `firewall`. This tutorial covers the `audit` fileset.
1. Go to the **Logs Router** page to configure GCP to export logs to a Pub/Sub topic. Use the search bar to find the page:
   ![Navigate to Logs Router page](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-navigate-logs-router.png)
   To set up the logs routing sink, click  **Create sink**. Set **sink name** as `monitor-gcp-audit-sink`. Select the **Cloud Pub/Sub topic** as the **sink service** and **Create new Cloud Pub/Sub topic** named `monitor-gcp-audit`:
   ![Create Pub/Sub topic](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-create-pubsub-topic.png)
   Finally, under **Choose logs to include in sink**, add `logName:"cloudaudit.googleapis.com"` (it includes all audit logs). Click **create sink**.  It will look something like the following:
   ![Create logs routing sink](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-create-sink.png)
2. Now go to the **Pub/Sub** page to add a subscription to the topic you just created. Use the search bar to find the page:
   ![GCP Pub/Sub](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-pub-sub.png)
   To add a subscription to the `monitor-gcp-audit` topic click **Create subscription**:
   ![Create GCP Pub/Sub Subscription](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-pub-sub-create-subscription.png)
   Set `monitor-gcp-audit-sub` as the **Subscription ID** and leave the **Delivery type** as pull:
   ![GCP Pub/Sub Subscription ID](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-pub-sub-subscription-id.png)
   Finally, scroll down and click **Create**.
3. Now that GCP is configured to export audit logs, enable Filebeat Google Cloud module.
   ```bash
   ./filebeat modules enable gcp
   ```
4. Edit the `modules.d/gcp.yml` file with the following configurations.
   ```yaml
   - module: gcp
     vpcflow:
       enabled: false 
     firewall:
       enabled: false 
     audit:
       enabled: true 
       var.project_id: "elastic-education" 
       var.topic: "monitor-gcp-audit" 
       var.subscription_name: "monitor-gcp-audit-sub" 
       var.credentials_file: "/home/ubuntu/credentials.json" 
   ```
5. Start Filebeat to collect the logs.
   ```bash
   ./filebeat -e
   ```
6. Finally, log into Kibana and open the **[Filebeat GCP] Audit** dashboard.
   ![Filebeat audit overview dashboard](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4391/solutions/images/observability-monitor-gcp-audit-overview-dashboard.png)