Setting up machine learning

Serverless Stack

To use the Elastic Stack machine learning features, you must have:

• the appropriate subscription level or the free trial period activated

• xpack.ml.enabled set to its default value of true on every node in the cluster (refer to Machine learning settings in Elasticsearch)

• ml value defined in the list of node.roles on the machine learning nodes

• machine learning features visible in the Kibana space

• security privileges assigned to the user that:

  • grant use of machine learning features, and
  • grant access to source and destination indices.

Assigning security privileges affects how users access machine learning features. Consider the two main categories:

• Elasticsearch API user: uses an Elasticsearch client, cURL, or Kibana Dev Tools to access machine learning features via Elasticsearch APIs. It requires Elasticsearch security privileges.
• Kibana user: uses the machine learning features in Kibana and does not use Dev Tools. It requires either Kibana feature privileges or Elasticsearch security privileges and is granted the most permissive combination of both. Kibana feature privileges are recommended if you control job level visibility via Spaces. Machine learning features must be visible in the relevant space. Refer to Feature visibility in Spaces for configuration information.

You can configure these privileges

• under the Roles and Spaces management pages. Find these pages in the main menu or use the global search field.
• via the respective Elasticsearch security APIs.

If you use machine learning APIs, you must have the following cluster and index privileges:

For full access:

• machine_learning_admin built-in role or the equivalent cluster privileges
• read and view_index_metadata on source indices
• read, manage, and index on destination indices (for data frame analytics analytics jobs only)

For read-only access:

• machine_learning_user built-in role or the equivalent cluster privileges
• read index privileges on source indices
• read index privileges on destination indices (for data frame analytics analytics jobs only)

In Kibana, the machine learning features must be visible in your space. To manage which features are visible in your space, go to the Spaces management page using the navigation menu or the global search field.

×

In addition to index privileges, source data views must also exist in the same space as your machine learning jobs. You can configure these under Data Views. To open Data Views, find Stack Management > Kibana in the main menu, or use the global search field.

Each machine learning job and trained model can be assigned to all, one, or multiple spaces. This can be configured in Machine Learning. To open Machine Learning, find the page in the main menu, or use the global search field. You can edit the spaces that a job or model is assigned to by clicking the icons in the Spaces column.

×

Within a Kibana space, for full access to the machine learning features, you must have:

• Machine Learning: All Kibana privileges
• Data Views Management: All Kibana feature privileges
• read, and view_index_metadata index privileges on your source indices
• data views for your source indices
• data views, read, manage, and index index privileges on destination indices (for data frame analytics analytics jobs only)

Within a Kibana space, for read-only access to the machine learning features, you must have:

• Machine Learning: Read Kibana privileges
• data views for your source indices
• read index privilege on your source indices
• data views and read index privileges on destination indices (for data frame analytics analytics jobs only)

For access to use machine learning APIs via Dev Tools in Kibana, set the Elasticsearch security privileges and grant access to machine_learning_admin or machine_learning_user built-in roles.

Within a Kibana space, to upload and import files in the Data Visualizer, you must have:

• Machine Learning: Read or Discover: All Kibana feature privileges
• Data Views Management: All Kibana feature privileges
• ingest_admin built-in role, or manage_ingest_pipelines cluster privilege
• create, create_index, manage and read index privileges for destination indices

For more information, see Security privileges and Kibana privileges.

Export and import anomaly detection jobs and data frame analytics jobs to transfer them between clusters or environments, for example, from a test environment to production.

• If your anomaly detection jobs use custom rules with filter lists, the filter lists must exist; otherwise, the import fails. To create filter lists, use Kibana or the create filters API
• If your anomaly detection jobs were associated with calendars, you must create the calendar in the new environment and add your imported jobs to the calendar. Use Kibana or the create calendars, add events to calendar, and add jobs to calendar APIs.

1. To navigate to Anomaly detection jobs, use the global search field.
2. Click Export jobs.
3. Select the jobs, then click Export to download the job definition file.

1. To navigate to Anomaly detection jobs, use the global search field.
2. Click Import jobs.
3. Upload the file that defines the anomaly detection job.
4. Enter a job ID and click Import.

After importing an anomaly detection job, you must run it so that it can learn from your current data and build a model that reflects the new environment.

• Data frame analytics jobs require their source index to exist before they can be imported. If the source index is missing, the import fails.

1. To navigate to Data frame analytics, use the global search field.
2. Click Export jobs.
3. Select the jobs, then click Export to download the job definition file.

1. To navigate to Data frame analytics, use the global search field.
2. Click Import jobs.
3. Select the file that defines the data frame analytics job.
4. Enter a job ID and a destination index, then click Import.