Azure eventhub input

Stack

Users can make use of the azure-eventhub input in order to read messages from an azure eventhub. The azure-eventhub input implementation is based on the the event processor host (EPH is intended to be run across multiple processes and machines while load balancing message consumers more on this here https://github.com/Azure/azure-event-hubs-go#event-processor-host, https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-event-processor-host). State such as leases on partitions and checkpoints in the event stream are shared between receivers using an Azure Storage container. For this reason, as a prerequisite to using this input, users will have to create or use an existing storage account.

Users can enable internal logs tracing for this input by setting the environment variable BEATS_AZURE_EVENTHUB_INPUT_TRACING_ENABLED: true. When enabled, this input will log additional information to the logs. Additional information includes partition ownership, blob lease information, and other internal state.

Example configuration using connection string authentication with processor v1:

filebeat.inputs:
- type: azure-eventhub
  eventhub: "insights-operational-logs"
  consumer_group: "$Default"
  connection_string: "Endpoint=sb://your-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=your-shared-access-key"
  storage_account: "your-storage-account"
  storage_account_key: "your-storage-account-key"
  storage_account_container: "your-storage-container"
  processor_version: "v1"
		

Example configuration using connection string authentication with processor v2:

filebeat.inputs:
- type: azure-eventhub
  eventhub: "insights-operational-logs"
  consumer_group: "$Default"
  auth_type: "connection_string"
  connection_string: "Endpoint=sb://your-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=your-shared-access-key"
  storage_account: "your-storage-account"
  storage_account_connection_string: "DefaultEndpointsProtocol=https;AccountName=your-storage-account;AccountKey=your-storage-account-key;EndpointSuffix=core.windows.net"
  storage_account_container: "your-storage-container"
		

Stack Planned ### Client secret authentication (processor v2)

Example configuration using Azure Active Directory service principal authentication with processor v2:

filebeat.inputs:
- type: azure-eventhub
  eventhub: "insights-operational-logs"
  consumer_group: "$Default"
  auth_type: "client_secret"
  eventhub_namespace: "your-namespace.servicebus.windows.net"
  tenant_id: "your-tenant-id"
  client_id: "your-client-id"
  client_secret: "your-client-secret"
  storage_account: "your-storage-account"
  storage_account_container: "your-storage-container"
		

Note: When using client_secret authentication, the service principal must have the appropriate Azure RBAC permissions. See Required permissions for details.

The azure-eventhub input supports multiple authentication methods. The auth_type configuration option controls the authentication method used for both Event Hub and Storage Account.

The following authentication types are supported:

• connection_string (default if auth_type is not specified): Uses Azure Event Hubs and Storage Account connection strings.
• Stack Planned client_secret: Uses Azure Active Directory service principal with client secret credentials.

When using client_secret authentication, the service principal needs the following Azure RBAC permissions:

For Azure Event Hubs:

• Azure Event Hubs Data Receiver role on the Event Hubs namespace or Event Hub
• Alternatively, a custom role with the following permissions:
  • Microsoft.EventHub/namespaces/eventhubs/read
  • Microsoft.EventHub/namespaces/eventhubs/consumergroups/read

For Azure Storage Account:

• Storage Blob Data Contributor role on the Storage Account or container
• Alternatively, a custom role with the following permissions:
  • Microsoft.Storage/storageAccounts/blobServices/containers/read
  • Microsoft.Storage/storageAccounts/blobServices/containers/write
  • Microsoft.Storage/storageAccounts/blobServices/containers/delete
  • Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action

For detailed instructions on how to set up an Azure AD service principal and configure permissions, refer to the official Microsoft documentation:

• Create an Azure service principal with Azure CLI
• Create an Azure AD app registration using the Azure portal
• Assign Azure roles using Azure CLI
• Azure Event Hubs authentication and authorization
• Authorize access to blobs using Azure Active Directory

The azure-eventhub input supports the following configuration options:

The name of the eventhub users would like to read from, field required.

Optional, we recommend using a dedicated consumer group for the azure input. Reusing consumer groups among non-related consumers can cause unexpected behavior and possibly lost events.

Stack Planned

Specifies the authentication method to use for both Event Hub and Storage Account. If not specified, defaults to connection_string for backwards compatibility.

Valid values include:

• connection_string (default): Uses connection string authentication. You must provide a connection_string.
• client_secret: Uses Azure Active Directory service principal with client secret credentials.

The connection string required to communicate with Event Hubs when using connection_string authentication. For more information, refer to Get an Azure Event Hubs connection string.

This option is required if:

• auth_type is set to connection_string
• auth_type is not specified (in which case it defaults to connection_string for backwards compatibility)

A Blob Storage account is required to store, retrieve, or update the offset or state of the Event Hub messages. This means that after stopping Filebeat it can resume from where it stopped processing messages.

Stack Planned

The fully qualified namespace for the Event Hub. Required when using client_secret authentication (auth_type is set to client_secret). Format: your-eventhub-namespace.servicebus.windows.net

Stack Planned

The Azure Active Directory tenant ID. Required when using client_secret authentication for Event Hub or Storage Account.

Stack Planned

The Azure Active Directory application (client) ID. Required when using client_secret authentication for Event Hub or Storage Account.

Stack Planned

The Azure Active Directory application client secret. Required when using client_secret authentication for Event Hub or Storage Account.

Stack Planned

The Azure Active Directory authority host. Optional when using client_secret authentication. Defaults to Azure Public Cloud (https://login.microsoftonline.com).

Supported values:

• https://login.microsoftonline.com (Azure Public Cloud - default)
• https://login.microsoftonline.us (Azure Government)
• https://login.chinacloudapi.cn (Azure China)

The name of the storage account. Required.

The storage account key, this key will be used to authorize access to data in your storage account, option is required.

Optional, the name of the storage account container you would like to store the offset information in.

Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different azure environment. Ex: https://management.chinacloudapi.cn/ for azure ChinaCloud https://management.microsoftazure.de/ for azure GermanCloud https://management.azure.com/ for azure PublicCloud https://management.usgovcloudapi.net/ for azure USGovernmentCloud Users can also use this in case of a Hybrid Cloud model, where one may define their own endpoints.

This input exposes metrics under the HTTP monitoring endpoint. These metrics are exposed under the /inputs path. They can be used to observe the activity of the input.