Billing

<div class="condensed-table">
| | |
| --- | --- |
| Version | 2.41.0 (View all) |
| Compatible Kibana version(s) | 8.13.0 or higher
9.0.0 or higher |
| Supported Serverless project types
What’s this? | Security
Observability |
| Subscription level
What’s this? | Basic |

</div>

The billing dataset collects Cloud Billing Reports information from Google Cloud BigQuery daily cost detail table. BigQuery is a fully-managed, serverless data warehouse. Cloud Billing export to BigQuery enables you to export detailed Google Cloud billing data (such as usage, cost estimates, and pricing data) automatically throughout the day to a BigQuery dataset that you specify. Then you can access your Cloud Billing data from BigQuery for detailed analysis.

Please see export cloud billing data to BigQuery for more details on how to export billing data.

In BigQuery dataset, detailed Google Cloud daily cost data is loaded into a data table named gcp_billing_export_v1_<BILLING_ACCOUNT_ID>. There is a defined schema for Google Cloud daily cost data that is exported to BigQuery. Please see daily cost detail data schema for more details.

For standard usage cost data, set the table pattern format to gcp_billing_export_v1. This table pattern is set as the default when no other is specified.

For detailed usage cost data, set the table pattern to gcp_billing_export_resource_v1. Detailed tables include the standard fields and additional fields, such as effective_price, enabling a more granular view of expenses.

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Before using any GCP integration you will need:

• GCP Credentials to connect with your GCP account.
• GCP Permissions to make sure the service account you’re using to connect has permission to share the relevant data.

To collect GCP Billing metrics, the following permissions are required to access the necessary data:

• roles/bigquery.dataViewer
• roles/bigquery.jobUser
• roles/billing.viewer

The dataset_id is the unique identifier of your BigQuery dataset where your billing data is stored. You can find this ID in your Google Cloud Console under the BigQuery section.

The table_pattern parameter allows you to specify which tables to retrieve from the specified dataset. This can be set to either gcp_billing_export_v1 for standard usage cost data or gcp_billing_export_resource_v1 for detailed usage cost data.

The cost_type parameter enables you to filter the cost data based on specific cost categories. You can select one of the following options:

• regular: This cost type includes all the regular costs associated with your usage of GCP services. This does not include any taxes, adjustments, or rounding errors.
• tax: This cost type includes all the taxes associated with your usage of GCP services. This does not include the regular costs, adjustments, or rounding errors.
• adjustment: This cost type includes any adjustments made to your billing data. Adjustments can include credits, discounts, refunds, or any other modifications to the original costs.
• rounding_error: This cost type includes any rounding errors that occurred when calculating your costs. These are typically very small amounts and are used to reconcile any discrepancies due to rounding.

Here’s an example of what your configuration might look like:

dataset_id: "my_billing_dataset"
table_pattern: "gcp_billing_export_resource_v1"
project_id: "my_project"
cost_type: "regular"

In this example, the Agent will pull data from all tables within the my_billing_dataset dataset that start with the pattern gcp_billing_export_resource_v1.

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "01475F-5B1080-1137E7"
        },
        "project": {
            "id": "elastic-bi",
            "name": "elastic-containerlib-prod"
        },
        "provider": "gcp"
    },
    "event": {
        "dataset": "gcp.billing",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "billing": {
            "billing_account_id": "01475F-5B1080-1137E7",
            "cost_type": "regular",
            "invoice_month": "202106",
            "project_id": "containerlib-prod-12763",
            "project_name": "elastic-containerlib-prod",
            "total": 4717.170681,
            "sku_id": "0D56-2F80-52A5",
            "service_id": "6F81-5844-456A",
            "sku_description": "Network Inter Region Ingress from Jakarta to Americas",
            "service_description": "Compute Engine",
            "effective_price": 0.00292353,
            "tags": [
                {
                    "key": "stage",
                    "value": "prod"
                },
                {
                    "key": "size",
                    "value": "standard"
                }
            ]
        }
    },
    "metricset": {
        "name": "billing",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.