Loading

Microsoft Office 365 Metrics Integration

Version 0.6.6 beta:[] (View all)
Compatible Kibana version(s) 8.16.0 or higher
9.0.0 or higher
Supported Serverless project types
What's this?
Security
Observability
Subscription level
What's this?
Basic
Level of support
What's this?
Elastic

This integration uses the Microsoft Graph API to collect essential metrics from Microsoft Office 365, offering detailed insights into user activity, application usage, and overall system performance.

Following Microsoft 365 Graph Reports can be collected by Microsoft Office 365 Metrics integration.

Report API Data-stream Name Aggregation Level
Microsoft 365 Active Users Service User Counts reportRoot: getOffice365ServicesUserCounts Microsoft 365 Active Users metrics Period-based
Microsoft 365 Groups Activity Group Detail reportRoot: getOffice365GroupsActivityDetail Microsoft 365 Groups Activity Group Detail Day-based
OneDrive Usage Account Detail reportRoot: getOneDriveUsageAccountDetail Microsoft 365 OneDrive Usage Account Detail Day-based
OneDrive Usage Account Counts reportRoot: getOneDriveUsageAccountCounts Microsoft 365 One Drive Usage metrics Period-based
OneDrive Usage File Counts reportRoot: getOneDriveUsageFileCounts Microsoft 365 One Drive Usage metrics Period-based
OneDrive Usage Storage reportRoot: getOneDriveUsageStorage Microsoft 365 One Drive Usage metrics Period-based
Outlook Activity Counts reportRoot: getEmailActivityCounts Microsoft 365 Outlook Activity metrics Period-based
Outlook App Usage Version Counts reportRoot: getEmailAppUsageVersionsUserCounts Microsoft 365 Outlook App Usage Version Counts metrics Period-based
Outlook Mailbox Usage Quota Status Mailbox Counts reportRoot: getMailboxUsageQuotaStatusMailboxCounts Microsoft 365 mailbox usage quota status metrics Period-based
Outlook Mailbox Usage Detail reportRoot: getMailboxUsageDetail Microsoft 365 mailbox usage detail metrics Period-based
SharePoint Site Usage Storage reportRoot: getSharePointSiteUsageStorage Microsoft 365 Sharepoint Site Usage metrics Period-based
SharePoint Site Usage Detail reportRoot: getSharePointSiteUsageDetail Microsoft 365 Sharepoint Site Usage metrics Period-based
Teams Device Usage User Counts reportRoot: getTeamsDeviceUsageUserCounts Microsoft 365 Teams Device Usage User Counts metrics Period-based
Teams User Activity User Counts reportRoot: getTeamsUserActivityUserCounts Microsoft 365 Teams User Activity User Counts metrics Period-based
Teams User Activity User Detail reportRoot: getTeamsUserActivityUserDetail Microsoft 365 Teams User Activity User Detail Day-based
Viva Engage Groups Activity Group Detail reportRoot: getYammerGroupsActivityDetail Microsoft 365 Viva Engage Groups Activity Day-based
Viva Engage Device Usage User Counts reportRoot: getYammerDeviceUsageUserCounts Microsoft 365 Viva Engage Device Usage User Counts metrics Period-based
Service Health reportRoot: getServiceHealth Microsoft 365 Service Health metrics No aggregation
Subscriptions reportRoot: subscribedSkus Microsoft 365 Subscriptions metrics No aggregation
Teamms Call Quality reportRoot: callRecords Microsoft 365 Teams Call Quality metrics No aggregation

To use this package you need to enable datastreams you want to collect metrics for and register an application in Microsoft Entra ID (formerly known as Azure Active Directory).

Once the application is registered, configure and/or note the following to setup O365 metrics Elastic integration:

  1. Note Application (client) ID and the Directory (tenant) ID in the registered application's Overview page.
  2. Create a new secret to configure the authentication of your application.
    • Navigate to Certificates & Secrets section.
    • Click New client secret and provide some description to create new secret.
    • Note the Value which is required for the integration setup.
  3. Add permissions to your registered application. Please check O365 Graph API permissions for more details.
    • Navigate to API permissions page and click Add a permission
    • Select Office 365 Management APIs tile from the listed tiles.
    • Click Application permissions.
    • If User.Read and Reports.Read.All permission under Microsoft.Graph tile is not added by default, add this permission. Additional permissions, such as ServiceHealth.Read.All for data streams like Service Health may be required. Refer to the API documentation under the Permissions section to determine the necessary permissions.
    • After the permissions are added, the admin has to grant consent for these permissions.

Once the secret is created and permissions are granted by admin, setup Elastic Agent's Microsoft O365 integration:

  • Click Add Microsoft Office 365.
  • Enable Collect Office 365 metrics via Graph API using CEL Input.
  • Add Directory (tenant) ID noted in Step 1 into Directory (tenant) ID parameter. This is required field.
  • Add Application (client) ID noted in Step 1 into Application (client) ID parameter. This is required field.
  • Add the secret Value noted in Step 2 into Client Secret parameter. This is required field.
  • Oauth2 Token URL can be added to generate the tokens during the oauth2 flow. If not provided, above Directory (tenant) ID will be used for oauth2 token generation.
  • Modify any other parameters as necessary.

Some data-streams listed earlier ingest data aggregated by a period, while other data-streams ingest data aggregated by day i.e., aggregated daily.

  • When configuring Period-based data-streams, the configuration option Period must be used during setup. The supported values are: D7, D30, D90, and D180.
  • As Day-based data-streams ingest aggregated data per day, the configuration option Initial Interval must be set which indicates how far back (in number of days) to fetch the data. Values between 1-28 are allowed.

Microsoft 365 reports are typically available within 48 hours, but may sometimes take several days. As per their documentation, data quality is ensured by performing daily validation checks to fill any gaps in data. During this process, users may notice differences in historical data in Microsoft 365 Reports in admin center.

To ensure these filled gaps and historical data-accuracy is also ingested into Elastic, the Microsoft Office 365 Metrics integration enables you to adjust Sync Days in the past parameter for Day-based data-streams. You can use this parameter to re-fetch the Microsoft 365 reports starting from N days in the past. Default value for this paramater is 3. You can gradually increase this value if you see any discrepancies between Microsoft Reports and Elastic data (maximum value allowed is 28).

Due to this re-fetching of data on same dates and the way Elastic data-streams work in append-only design, the ingested data may have duplicates. For example, you may see duplicate documents in Elastic on the source data-stream backed indices per resource (user/group/site) per report date. To maintain only the latest copy of document, the Microsoft Office 365 Metrics integration installs Latest Transforms, one per report. These latest transform periodically pulls the data from source data-stream backed indices into a destination non-data-stream backed index. Hence the destination indices only contains single (latest) document per resource (user/group/site) per report date. Inside the reports dataset, you can distinguish between source and destination indices using the field labels.is_transform_source. This is set to true for source data-stream backed indices and false for destination (latest) indices.

Thus when searching for data, you should use a filter labels.is_transform_source: false to avoid seeing any duplicates. The Microsoft Office 365 Metrics integration dashboards also has this filter to only show the latest datapoints.

As the latest data is available in destination indices, the source data-stream backed indices are purged based on ILM policy metrics-o365_metrics.<data_stream>-default_policy.

o365.metrics.report.name Source filter Source indices Destination filter Destination indices Destination alias
Microsoft 365 Groups Activity Group Detail labels.is_transform_source: true metrics-o365_metrics.groups_activity_group_detail-* labels.is_transform_source: false metrics-o365_metrics.groups_activity_group_detail_latest-* metrics-o365_metrics.groups_activity_group_detail_latest
OneDrive Usage Account Detail labels.is_transform_source: true metrics-o365_metrics.onedrive_usage_account_detail-* labels.is_transform_source: false metrics-o365_metrics.onedrive_usage_account_detail_latest-* metrics-o365_metrics.onedrive_usage_account_detail_latest
Teams User Activity User Detail labels.is_transform_source: true metrics-o365_metrics.teams_user_activity_user_detail-* labels.is_transform_source: false metrics-o365_metrics.teams_user_activity_user_detail_latest-* metrics-o365_metrics.teams_user_activity_user_detail_latest
Viva Engage Groups Activity Group Detail labels.is_transform_source: true metrics-o365_metrics.viva_engage_groups_activity_group_detail-* labels.is_transform_source: false metrics-o365_metrics.viva_engage_groups_activity_group_detail_latest-* metrics-o365_metrics.viva_engage_groups_activity_group_detail_latest

Note: Sync Days in the past and Latest Transforms are only used in Day-based data-streams, i.e., for data-streams aggregated per day.

By default for all Microsoft 365 usage reports, the user names, emails, group, or site information are anonymized by Microsoft using MD5 hashes. You can revert this change for a tenant and show identifiable user, group, and site information if your organization's privacy practices allow it. To do this, follow below steps:

  1. Login to Microsoft 365 admin center
  2. Navigate to Settings --> Org Settings --> Services page.
  3. Select Reports
  4. Uncheck the statement Display concealed user, group, and site names in all reports, and then save your changes.

Uses the Microsoft 365 Graph API to retrieve metrics from Microsoft 365.

Get details about Active Users Services User Count from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Mailbox Usage Quota Status from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Mailbox Usage Detail from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Microsoft 365 groups activity by group from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about OneDrive usage by account from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about OneDrive usage by account counts from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about OneDrive usage by file counts from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about OneDrive usage by storage from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Outlook Activity from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Microsoft Outlook App Usage Version Counts from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about SharePoint Site Usage Detail from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about SharePoint Site Usage Storage from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Teams User Activity User Counts from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Teams User Activity User Detail from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Yammer Groups Activity Group Detail by group from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Yammer Device Usage User Counts from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Teams Device Usage User Counts from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Service Health from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Subscriptions from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Teams Call Quality from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.