﻿---
title: Monitor Microsoft Azure OpenAI
description: This tutorial shows you how to use the Elastic Azure OpenAI integration, the Azure portal, and Elastic Agent to collect and monitor Azure OpenAI logs...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/solutions/observability/cloud/monitor-microsoft-azure-openai
products:
  - Elastic Observability
---

# Monitor Microsoft Azure OpenAI
<note>
  **New to Elastic?** Follow the steps in our [getting started guide](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/solutions/observability/get-started/quickstart-monitor-hosts-with-elastic-agent) instead of the steps described here. Return to this tutorial after you’ve learned the basics.
</note>

This tutorial shows you how to use the Elastic Azure OpenAI integration, the Azure portal, and Elastic Agent to collect and monitor Azure OpenAI logs and metrics with Elastic Observability.

## What you’ll learn

You’ll learn how to:
- Set up your Azure instance to allow the Azure OpenAI integration to collect logs and metrics.
- Configure the Azure OpenAI integration to collect logs and metrics.
- Install Elastic Agent on your host.
- View your logs and metrics in Kibana using built-in dashboards and Discover.


## Step 1: Set up Azure to collect logs

The Elastic Azure OpenAI integration captures audit logs and request and response logs.
- Audit logs provide a range of information related to the use and management of Azure OpenAI services.
- Request and response logs provide information about each request made to the service and the corresponding response provided by the service.

For more on the fields ingested from audit and request and response logs, refer to the [Azure OpenAI integration](https://docs.elastic.co/en/integrations/azure_openai#settings) documentation.
Before Elastic Agent can collect your logs and send them to Kibana, complete the following steps in the [Azure portal](https://portal.azure.com/):
1. Create an event hub to receive logs exported from the Azure service and make them available to the Elastic Agent.
2. Configure diagnostic settings to send your logs to the event hub.
3. Create a storage account container where the Elastic Agent can store consumer group information.


### Create an event hub

[Azure Event Hubs](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about) is a data streaming platform and event ingestion service that you use to store in-flight Azure logs before sending them to Elasticsearch. For this tutorial, you create a single event hub because you are collecting logs from one service.
To create an Azure event hub:
1. Go to the [Azure portal](https://portal.azure.com/).
2. Search for and select **Event Hubs**.
3. Click **Create** and create a new Event Hubs namespace. You’ll need to create a new resource group, or choose an existing one.
4. Enter the required settings for the namespace and click **Review + create**.
5. Click **Create** to deploy the resource.
6. In the new namespace, click **+ Event Hub** and enter a name for the event hub.
7. Click **Review + create**, and then click **Create** to deploy the resource.
8. Make a note of the event hub name because you’ll use it to configure your integration settings in [Step 3: Configure the Azure OpenAI integration](#azure-openai-configure-integration).


### Configure diagnostic settings

Every Azure service that creates logs has diagnostic settings that allow you to export logs and metrics to an external destination. In this step, you’ll configure the Azure OpenAI service to export audit and request and response logs to the event hub you created in the previous step.
To configure diagnostic settings to export logs:
1. Go to the [Azure portal](https://portal.azure.com/) and open your OpenAI resource.
2. In the navigation pane, select **Diagnostic settings** → **Add diagnostic setting**.
3. Enter a name for the diagnostic setting.
4. In the list of log categories, select **Audit logs** and **Request and Response Logs**.
5. Under Destination details, select **Stream to an event hub** and select the namespace and event hub you created in [Create an event hub](#azure-openai-event-hub).
6. Save the diagnostic settings.


### Create a storage account container

The Elastic Agent stores the consumer group information (state, position, or offset) in a storage account container. Making this information available to Elastic Agents allows them to share the logs processing and resume from the last processed logs after a restart. The Elastic Agent can use one storage account container for all integrations. The agent uses the integration name and the event hub name to identify the blob to store the consumer group information uniquely.
To create the storage account:
1. Go to the [Azure portal](https://portal.azure.com/) and select **Storage accounts**.
2. Select **Create storage account**.
3. Under **Advanced**, make sure these settings are as follows:
   - **Hierarchical namespace**: disabled
- **Minimum TLS version**: Version 1.2
- **Access tier**: Hot
4. Under **Data protections**, make sure these settings are as follows:
   - **Enable soft delete for blobs**: disabled
- **Enable soft delete for containers**: disabled
5. Click **Review + create**, and then click **Create**.
6. Make note of the storage account name and the storage account access keys because you’ll use them later to authenticate your Elastic application’s requests to this storage account in [Step 3: Configure the Azure OpenAI integration](#azure-openai-configure-integration).


## Step 2: Set up Azure to collect metrics

The Azure OpenAI integration metric data stream collects the cognitive service metrics specific to the Azure OpenAI service. Before Elastic Agent can collect your metrics and send them to Kibana, it needs an app registration to access Azure on your behalf and collect data using the Azure APIs.
Complete the following steps in your Azure instance to register a new Azure app:
1. Create the app registration.
2. Add credentials to the app.
3. Add role assignment to your app.


### Create an app registration

To register your app:
1. Go to the [Azure portal](https://portal.azure.com/).
2. Search for and select **Microsoft Entra ID**.
3. Under **Manage**, select **App registrations** → **New registration**.
4. Enter a display name for your app (for example, `elastic-agent`).
5. Specify who can use the app.
6. The Elastic Agent doesn’t use a redirect URI, so you can leave this field blank.
7. Click **Register**.
8. Make note of the **Application (client) ID** because you’ll use it to specify the **Client ID** in the integration settings in [Step 3: Configure the Azure OpenAI integration](#azure-openai-configure-integration).


### Create credentials and add them to your app

Credentials allow your app to access Azure APIs and authenticate itself, so you won’t need to do anything at runtime. The Elastic Azure OpenAI integration uses client secrets to authenticate.
To create and add client secrets:
1. From the [Azure portal](https://portal.azure.com/), select the app you created in the previous section.
2. Select **Certificates & secrets** → **Client secrets** → **New client secret**.
3. Add a description (for example, "Elastic Agent client secrets").
4. Select an expiration or specify a custom lifetime.
5. Select **Add**.
6. Make note of the **Value** in the **Client secrets** table because you’ll use it to specify the **Client Secret** in [Step 3: Configure the Azure OpenAI integration](#azure-openai-configure-integration).
   <warning>
   The secret value is not viewable after you leave this page. Record the value in a safe place.
   </warning>


### Add role assignment to your app

To add a role assignment to your app:
1. From the [Azure portal](https://portal.azure.com/), search for and select **Subscriptions**.
2. Select the subscription to assign the app.
3. Select **Access control (IAM)**.
4. Select **Add** → **Add role assignment**.
5. In the **Role** tab, search for and select **Monitoring Reader**.
6. Click **Next** to open the **Members** tab.
7. Select **Assign access to** → **User, group, or service principal**, and select **Select members**.
8. Search for and select your app name (for example, "elastic-agent").
9. Click **Select**.
10. Click **Review + assign**.
11. Make note of the **Subscription ID** and **Tenant ID** from your Microsoft Entra because you’ll use these to specify settings in the integration.


## Step 3: Configure the Azure OpenAI integration

1. Find **Integrations** in the main menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/explore-analyze/find-and-organize/find-apps-and-objects).
2. In the query bar, search for **Azure OpenAI** and select the Azure OpenAI integration card.
3. Click **Add Azure OpenAI**.
4. Under Integration settings, configure the integration name and optionally add a description.
   <tip>
   If you don’t have options for configuring the integration, you’re probably in a workflow designed for new deployments. Follow the steps, then return to this tutorial when you’re ready to configure the integration.
   </tip>


### Configure logs collection

To collect Azure OpenAI logs, specify values for the following required fields:
<definitions>
  <definition term="Event hub">
    The name of the event hub you created earlier.
  </definition>
  <definition term="Connection String">
    The connection string primary key of the event hub namespace. To learn how to get the connection string, refer to [Get an Event Hubs connection string](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string) in the Azure documentation.
    <tip>
      Instead of copying the connection string from the RootManageSharedAccessKey policy, you should create a new shared access policy (with permission to listen) and copy the connection string from the new policy.
    </tip>
  </definition>
  <definition term="Storage account">
    The name of a blob storage account that you set up in [Create a storage account container](#azure-openai-storage-account-container). You can use the same storage account container for all integrations.
  </definition>
  <definition term="Storage account key">
    A valid access key defined for the storage account you created in [Create a storage account container](#azure-openai-storage-account-container).
  </definition>
</definitions>


### Configure metrics collection

To collect Azure OpenAI metrics, specify values for the following required fields:
<definitions>
  <definition term="Client ID">
    The Application (client) ID that you copied earlier when you created the service principal.
  </definition>
  <definition term="Client secret">
    The secret value that you copied earlier.
  </definition>
  <definition term="Tenant ID">
    The tenant ID listed on the main Azure Active Directory Page.
  </definition>
  <definition term="Subscription ID">
    The subscription ID listed on the main Subscriptions page.
  </definition>
</definitions>

After you’ve finished configuring your integration, click **Save and continue**. You’ll get a notification that your integration was added. Select **Add Elastic Agent to your hosts**.

## Step 4: Install Elastic Agent

<important>
  To get support for the latest API changes from Azure, we recommend that you use the latest in-service version of Elastic Agent compatible with your Elastic Stack. Otherwise, your integrations may not function as expected.
</important>

You can install Elastic Agent on any host that can access the Azure account and forward events to Elasticsearch.
1. In the popup, click **Add Elastic Agent to your hosts** to open the **Add agent** flyout.
   <tip>
   If you accidentally closed the popup, go to **Fleet** → **Agents**, then click **Add agent** to access the installation instructions.
   </tip>
   The **Add agent** flyout has two options: **Enroll in Fleet** and **Run standalone**. The default is to enroll the agents in Fleet, as this reduces the amount of work on the person managing the hosts by providing a centralized management tool in Kibana.
2. The enrollment token you need should already be selected.
   <note>
   The enrollment token is specific to the Elastic Agent policy that you just created. When you run the command to enroll the agent in Fleet, you will pass in the enrollment token.
   </note>
3. To download, install, and enroll the Elastic Agent, select your host operating system and copy the installation command shown in the instructions.
4. Run the command on the host where you want to install Elastic Agent.

It takes a few minutes for Elastic Agent to enroll in Fleet, download the configuration specified in the policy, and start collecting data. You can wait to confirm incoming data, or close the window.

## Step 5: View logs and metrics in Kibana

Now that your log and metric data is streaming to Elasticsearch, you can view them in Kibana. You have the following options for viewing your data:
- [View logs and metrics with the overview dashboard](#azure-openai-overview-dashboard): Use the built-in overview dashboard for insight into your Azure OpenAI service like total requests and token usage.
- [View logs and metrics with Discover](#azure-openai-discover): Use Discover to find and filter your log and metric data based on specific fields.


### View logs and metrics with the overview dashboard

The Elastic Azure OpenAI integration comes with a built-in overview dashboard to visualize your log and metric data. To view the integration dashboards:
1. Find **Dashboards** in the main menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/explore-analyze/find-and-organize/find-apps-and-objects).
2. Search for **Azure OpenAI**.
3. Select the `[Azure OpenAI] Overview` dashboard.

From here, you’ll find visualizations of important metrics for your Azure OpenAI service, like the request rate, error rate, token usage, and chat completion latency. To zoom in on your data, click and drag across the bars in a visualization.
![screenshot of the Azure OpenAI integration dashboard](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/solutions/images/observability-azure-openai-dashboard.png)
For more on dashboards and visualization, refer to the [Dashboards and visualizations](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/explore-analyze/dashboards) documentation.

### View logs and metrics with Discover

Find **Discover** in the main menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/explore-analyze/find-and-organize/find-apps-and-objects). From the data view drop-down, select either `logs-*` or `metrics-*` to view specific data. You can also create data views if, for example, you wanted to view both `logs-*` and `metrics-*` simultaneously.
![screenshot of the Discover data view dropdown](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/solutions/images/observability-discover-data-view-menu.png)
From here, filter your data and dive deeper into individual logs to find information and troubleshoot issues. For a list of Azure OpenAI fields you may want to filter by, refer to the [Azure OpenAI integration](https://docs.elastic.co/en/integrations/azure_openai#settings) docs.
![screenshot of the discover main page](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/solutions/images/observability-azure-openai-discover.png)

For more on using Discover and creating data views, refer to the [Discover](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/explore-analyze/discover) documentation.

## Step 6: Monitor Microsoft Azure OpenAI APM with OpenTelemetry

The Azure OpenAI API provides useful data to help monitor and understand your code. Using OpenTelemetry, you can ingest this data into Elastic Observability. From there, you can view and analyze your data to monitor the cost and performance of your applications.
For this tutorial, we’ll be using an [example Python application](https://github.com/mdbirnstiehl/AzureOpenAIAPMmonitoringOtel) and the Python OpenTelemetry libraries to instrument the application and send data to Observability.

### Set your environment variables

To start collecting APM data for your Azure OpenAI applications, gather the OpenTelemetry OTLP exporter endpoint and authentication header from your Elastic Cloud instance:
1. Find **Integrations** in the main menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/explore-analyze/find-and-organize/find-apps-and-objects).
2. Select the **APM** integration.
3. Scroll down to **APM Agents** and select the **OpenTelemetry** tab.
4. Make note of the configuration values for the following configuration settings:
   - `OTEL_EXPORTER_OTLP_ENDPOINT`
- `OTEL_EXPORTER_OTLP_HEADERS`

With the configuration values from the APM integration and your [Azure OpenAI API key and endpoint](https://learn.microsoft.com/en-us/azure/ai-services/openai/quickstart?tabs=command-line%2Cpython-new&pivots=programming-language-python#retrieve-key-and-endpoint), set the following environment variables using the export command on the command line:
```bash
export AZURE_OPENAI_API_KEY="your-Azure-OpenAI-API-key"
export AZURE_OPENAI_ENDPOINT="your-Azure-OpenAI-endpoint"
export OPENAI_API_VERSION="your_api_version"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20<your-otel-exporter-auth-header>"
export OTEL_EXPORTER_OTLP_ENDPOINT="your-otel-exporter-endpoint"
export OTEL_RESOURCE_ATTRIBUTES=service.name=your-service-name
```


### Download Python libraries

Install the necessary Python libraries using this command:
```bash
pip3 install openai flask opentelemetry-distro[otlp] opentelemetry-instrumentation
```


### Instrument the application

The following code is from the [example application](https://github.com/mdbirnstiehl/AzureOpenAIAPMmonitoringOtel). In a real use case, you would add the import statements to your code.
The app we’re using in this tutorial is a simple example that calls Azure OpenAI APIs with the following message: “How do I send my APM data to Elastic Observability?”:
```python
import os

from flask import Flask
from openai import AzureOpenAI
from opentelemetry import trace

from monitor import count_completion_requests_and_tokens


# Initialize Flask app
app = Flask(__name__)

# Set OpenAI API key
client = AzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    api_version=os.getenv("OPENAI_API_VERSION"),
    azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"),
)

# Monkey-patch the openai.Completion.create function
client.chat.completions.create = count_completion_requests_and_tokens(
    client.chat.completions.create
)

tracer = trace.get_tracer("counter")


@app.route("/completion")
@tracer.start_as_current_span("completion")
def completion():
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[
            {
                "role": "user",
                "content": "How do I send my APM data to Elastic Observability?",
            }
        ],
        max_tokens=20,
        temperature=0,
    )
    return response.choices[0].message.content.strip()
```

The code uses monkey patching, a technique in Python that dynamically modifies the behavior of a class or module at runtime by modifying its attributes or methods, to modify the behavior of the `chat.completions` call so we can add the response metrics to the OpenTelemetry spans.
The [`monitor.py` file](https://github.com/mdbirnstiehl/AzureOpenAIAPMmonitoringOtel/blob/main/monitor.py) in the example application instruments the application and can be used to instrument your own applications.
```python
def count_completion_requests_and_tokens(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        counters["completion_count"] += 1
        response = func(*args, **kwargs)

        token_count = response.usage.total_tokens
        prompt_tokens = response.usage.prompt_tokens
        completion_tokens = response.usage.completion_tokens
        cost = calculate_cost(response)
        strResponse = json.dumps(response, default=str)

        # Set OpenTelemetry attributes
        span = trace.get_current_span()
        if span:
            span.set_attribute("completion_count", counters["completion_count"])
            span.set_attribute("token_count", token_count)
            span.set_attribute("prompt_tokens", prompt_tokens)
            span.set_attribute("completion_tokens", completion_tokens)
            span.set_attribute("model", response.model)
            span.set_attribute("cost", cost)
            span.set_attribute("response", strResponse)
        return response

    return wrapper
```

Adding this data to our span lets us send it to our OTLP endpoint, so you can search for the data in Observability and build dashboards and visualizations.
Implementing the following function allows you to calculate the cost of a single request to the OpenAI APIs.
```python
def calculate_cost(response):
    if response.model in ["gpt-4", "gpt-4-0314"]:
        cost = (
            response.usage.prompt_tokens * 0.03
            + response.usage.completion_tokens * 0.06
        ) / 1000
    elif response.model in ["gpt-4-32k", "gpt-4-32k-0314"]:
        cost = (
            response.usage.prompt_tokens * 0.06
            + response.usage.completion_tokens * 0.12
        ) / 1000
    elif "gpt-3.5-turbo" in response.model:
        cost = response.usage.total_tokens * 0.002 / 1000
    elif "davinci" in response.model:
        cost = response.usage.total_tokens * 0.02 / 1000
    elif "curie" in response.model:
        cost = response.usage.total_tokens * 0.002 / 1000
    elif "babbage" in response.model:
        cost = response.usage.total_tokens * 0.0005 / 1000
    elif "ada" in response.model:
        cost = response.usage.total_tokens * 0.0004 / 1000
    else:
        cost = 0
    return cost
```

To download the example application and try it for yourself, go to the [GitHub repo](https://github.com/mdbirnstiehl/AzureOpenAIAPMmonitoringOtel).

### View APM data from OpenTelemetry in Kibana

After ingesting your data, you can filter and explore it using Discover in Kibana. Go to **Discover** from the Kibana menu under **Analytics**. You can then filter by the fields sent to Observability by OpenTelemetry, including:
- `numeric_labels.completion_count`
- `numeric_labels.completion_tokens`
- `numeric_labels.cost`
- `numeric_labels.prompt_tokens`
- `numeric_labels.token_count`

![screenshot of the discover main page](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/solutions/images/observability-azure-openai-apm-discover.png)

Then, use these fields to create visualizations and build dashboards. Refer to the [Dashboard and visualizations](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/explore-analyze/dashboards) documentation for more information.
![screenshot of the Azure OpenAI APM dashboard](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/solutions/images/observability-azure-openai-apm-dashboard.png)


## What’s next?

Now that you know how to find and visualize your Azure OpenAI logs and metrics, you’ll want to make sure you’re getting the most out of your data. Elastic has some useful tools to help you do that:
- **Alerts**: Create threshold rules to notify you when your metrics or logs reach or exceed a specified value: Refer to [Metric threshold](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/solutions/observability/incident-management/create-metric-threshold-rule) and [Log threshold](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/solutions/observability/incident-management/create-log-threshold-rule) for more on setting up alerts.
- **SLOs**: Set measurable targets for your Azure OpenAI service performance based on your metrics. Once defined, you can monitor your SLOs with dashboards and alerts and track their progress against your targets over time. Refer to [Service-level objectives (SLOs)](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/solutions/observability/incident-management/service-level-objectives-slos) for more on setting up and tracking SLOs.
- **Machine learning (ML) jobs**: Set up ML jobs to find anomalous events and patterns in your Azure OpenAI data. Refer to [Finding anomalies](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4433/explore-analyze/machine-learning/anomaly-detection/ml-ad-finding-anomalies) for more on setting up ML jobs.