Loading

Microsoft Office 365 Metrics Integration

<div class="condensed-table">
| | |
| --- | --- |
| Version | 0.3.0 [beta] (View all) |
| Compatible Kibana version(s) | 8.16.0 or higher |
| Supported Serverless project types
What’s this? | Security
Observability |
| Subscription level
What’s this? | Basic |
| Level of support
What’s this? | Elastic |

</div>
This integration uses the Microsoft Graph API to collect essential metrics from Microsoft Office 365, offering detailed insights into user activity, application usage, and overall system performance.

Following Microsoft 365 Graph Reports can be collected by Microsoft Office 365 Metrics integration.

Report API Data-stream Name Aggregation Level
Microsoft 365 Active Users Service User Counts reportRoot: getOffice365ServicesUserCounts Office 365 Active Users metrics Period-based
Microsoft 365 Groups Activity Group Detail reportRoot: getOffice365GroupsActivityDetail Microsoft 365 Groups Activity Group Detail Day-based
OneDrive Usage Account Detail reportRoot: getOneDriveUsageAccountDetail Microsoft 365 OneDrive Usage Account Detail Day-based
OneDrive Usage Account Counts reportRoot: getOneDriveUsageAccountCounts Office 365 One Drive Usage metrics Period-based
OneDrive Usage File Counts reportRoot: getOneDriveUsageFileCounts Office 365 One Drive Usage metrics Period-based
OneDrive Usage Storage reportRoot: getOneDriveUsageStorage Office 365 One Drive Usage metrics Period-based
Outlook Activity Counts reportRoot: getEmailActivityCounts Office 365 Outlook Activity metrics Period-based
Outlook App Usage Version User Counts reportRoot: getEmailAppUsageVersionsUserCounts Office 365 Outlook App Usage metrics Period-based
Outlook Mailbox Usage Quota Status Mailbox Counts reportRoot: getMailboxUsageQuotaStatusMailboxCounts O365 mailbox quota status and mailbox usage detail metrics Period-based
Outlook Mailbox Usage Mailbox Detail reportRoot: getMailboxUsageDetail O365 mailbox quota status and mailbox usage detail metrics Period-based
SharePoint Site Usage Storage reportRoot: getSharePointSiteUsageStorage Office 365 Sharepoint Site Usage metrics Period-based
SharePoint Site Usage Site Detail reportRoot: getSharePointSiteUsageDetail Office 365 Sharepoint Site Usage metrics Period-based
Teams User Activity User Counts reportRoot: getTeamsUserActivityUserCounts Office 365 Teams User Activity User Counts metrics Period-based
Teams User Activity User Detail reportRoot: getTeamsUserActivityUserDetail Microsoft 365 Teams User Activity User Detail Day-based
Viva Engage Groups Activity Group Detail reportRoot: getYammerGroupsActivityDetail Microsoft 365 Viva Engage Groups Activity Day-based
Yammer Device Usage User Counts reportRoot: getYammerDeviceUsageUserCounts Office 365 Yammer Device Usage metrics Period-based

To use this package you need to enable datastreams you want to collect metrics for and register an application in Microsoft Entra ID (formerly known as Azure Active Directory).

Once the application is registered, configure and/or note the following to setup O365 metrics Elastic integration:

  1. Note Application (client) ID and the Directory (tenant) ID in the registered application’s Overview page.

  2. Create a new secret to configure the authentication of your application.

    • Navigate to Certificates & Secrets section.
    • Click New client secret and provide some description to create new secret.
    • Note the Value which is required for the integration setup.
  3. Add permissions to your registered application. Please check O365 Graph API permissions for more details.

    • Navigate to API permissions page and click Add a permission
    • Select Office 365 Management APIs tile from the listed tiles.
    • Click Application permissions.
    • If User.Read and Reports.Read.All permission under Microsoft.Graph tile is not added by default, add this permission.
    • After the permissions are added, the admin has to grant consent for these permissions.

Once the secret is created and permissions are granted by admin, setup Elastic Agent’s Microsoft O365 integration:

  • Click Add Microsoft Office 365.
  • Enable Collect Office 365 metrics via Graph API using CEL Input.
  • Add Directory (tenant) ID noted in Step 1 into Directory (tenant) ID parameter. This is required field.
  • Add Application (client) ID noted in Step 1 into Application (client) ID parameter. This is required field.
  • Add the secret Value noted in Step 2 into Client Secret parameter. This is required field.
  • Oauth2 Token URL can be added to generate the tokens during the oauth2 flow. If not provided, above Directory (tenant) ID will be used for oauth2 token generation.
  • Modify any other parameters as necessary.

Some data-streams listed earlier ingest data aggregated by a period, while other data-streams ingest data aggregated by day i.e., aggregated daily.

  • When configuring Period-based data-streams, the configuration option Period must be used during setup. The supported values are: D7, D30, D90, and D180.
  • As Day-based data-streams ingest aggregated data per day, the configuration option Initial Interval must be set which indicates how far back (in number of days) to fetch the data. Values between 1-28 are allowed.

Microsoft 365 reports are typically available within 48 hours, but may sometimes take several days. As per their documentation, data quality is ensured by performing daily validation checks to fill any gaps in data. During this process, users may notice differences in historical data in Microsoft 365 Reports in admin center.

To ensure these filled gaps and historical data-accuracy is also ingested into Elastic, the Microsoft Office 365 Metrics integration enables you to adjust Sync Days in the past parameter for Day-based data-streams. You can use this parameter to re-fetch the Microsoft 365 reports starting from N days in the past. Default value for this paramater is 3. You can gradually increase this value if you see any discrepancies between Microsoft Reports and Elastic data (maximum value allowed is 28).

Due to this re-fetching of data on same dates and the way Elastic data-streams work in append-only design, the ingested data may have duplicates. For example, you may see duplicate documents in Elastic on the source data-stream backed indices per resource (user/group/site) per report date. To maintain only the latest copy of document, the Microsoft Office 365 Metrics integration installs Latest Transforms, one per report. These latest transform periodically pulls the data from source data-stream backed indices into a destination non-data-stream backed index. Hence the destination indices only contains single (latest) document per resource (user/group/site) per report date. Inside the reports dataset, you can distinguish between source and destination indices using the field labels.is_transform_source. This is set to true for source data-stream backed indices and false for destination (latest) indices.

Thus when searching for data, you should use a filter labels.is_transform_source: false to avoid seeing any duplicates. The Microsoft Office 365 Metrics integration dashboards also has this filter to only show the latest datapoints.

As the latest data is available in destination indices, the source data-stream backed indices are purged based on ILM policy metrics-o365_metrics.<data_stream>-default_policy.

o365.metrics.report.name Source filter Source indices Destination filter Destination indices Destination alias
Microsoft 365 Groups Activity Group Detail labels.is_transform_source: true metrics-o365_metrics.groups_activity_group_detail-* labels.is_transform_source: false metrics-o365_metrics.groups_activity_group_detail_latest-* metrics-o365_metrics.groups_activity_group_detail_latest
OneDrive Usage Account Detail labels.is_transform_source: true metrics-o365_metrics.onedrive_usage_account_detail-* labels.is_transform_source: false metrics-o365_metrics.onedrive_usage_account_detail_latest-* metrics-o365_metrics.onedrive_usage_account_detail_latest
Teams User Activity User Detail labels.is_transform_source: true metrics-o365_metrics.teams_user_activity_user_detail-* labels.is_transform_source: false metrics-o365_metrics.teams_user_activity_user_detail_latest-* metrics-o365_metrics.teams_user_activity_user_detail_latest
Viva Engage Groups Activity Group Detail labels.is_transform_source: true metrics-o365_metrics.viva_engage_groups_activity_group_detail-* labels.is_transform_source: false metrics-o365_metrics.viva_engage_groups_activity_group_detail_latest-* metrics-o365_metrics.viva_engage_groups_activity_group_detail_latest
Note

Sync Days in the past and Latest Transforms are only used in Day-based data-streams, i.e., for data-streams aggregated per day.

By default for all Microsoft 365 usage reports, the user names, emails, group, or site information are anonymized by Microsoft using MD5 hashes. You can revert this change for a tenant and show identifiable user, group, and site information if your organization’s privacy practices allow it. To do this, follow below steps:

  1. Login to Microsoft 365 admin center
  2. Navigate to Settings -→ Org Settings -→ Services page.
  3. Select Reports
  4. Uncheck the statement Display concealed user, group, and site names in all reports, and then save your changes.

Uses the Office 365 Graph API to retrieve metrics from Office 365.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Microsoft 365 groups activity by group from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about OneDrive usage by account from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Microsoft Teams user activity by user from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Get details about Yammer groups activity by group from Microsoft Graph API.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.