Cloud credential problems
A cross-project search datafeed stores an internal cloud (UIAM) API key so periodic searches can read linked projects on your behalf. Elasticsearch mints that key only when a cloud-authenticated caller creates or updates the datafeed, not during a scheduled extraction cycle. Failures fall into four paths: the key could not be created, an existing key no longer authorizes search, the key was cleared by a non-cloud-authenticated update, or no key was ever minted because the datafeed was created or last updated without cloud authentication.
Where to look
anomaly detection job job messages in Kibana: Open Machine Learning → Anomaly Detection, select the job, and review the Job messages tab for audit entries and warnings about linked projects, credentials, or scope changes. On the Datafeed tab, View datafeed counts opens the datafeed chart flyout for extraction timing.
The same entries are stored in
.ml-notifications-*.GET _ml/datafeeds/{datafeed_id}: Shows the effectiveproject_routingvalue and, when an internal cloud credential exists,authorization.cloud_api_key.id.GET _ml/datafeeds/{datafeed_id}/_stats: While the datafeed runs, showsremote_cluster_statswithtotal_clusters,available_clusters,skipped_clusters,availability_ratio,stabilized_cluster_aliases, andper_cluster_consecutive_skips. The object is absent until the first search cycle establishes a baseline..ml-annotations-*: Scope-change annotations for the job. The annotationeventfield carriessearch_scope_changed(not the separatetypefield).GET /_project/tags: Lists linked projects and their tags so you can compare them with a routing expression.Elastic Cloud console: Review linked projects in Link and manage projects.
During active extraction problems, Job messages are authoritative. remote_cluster_stats from get datafeed stats reflects the last completed cycle.
Which failure is this?
Use observable behavior to pick a path:
- Create or update fails immediately: Error during validation before minting or during minting itself. See Creation and update failures below.
- Searches used to work and
GET _ml/datafeeds/{datafeed_id}containsauthorization.cloud_api_key.id: runtime auth or authz failure. See Runtime failures below. authorization.cloud_api_key.idis missing: two cases:- Credential cleared: Job messages record
Internal cloud API key cleared on datafeed update with non-cloud credentialsafter a non-cloud-authenticated update removed a stored key. See Credential cleared below. - Never minted: the datafeed was created or last updated without cloud authentication, so no key was ever stored and no CLEARED message appears. Job messages lack
Internal cloud API key minted for cross-project datafeed. Mint a key with a cloud-authenticated create or update.
- Credential cleared: Job messages record
Creation and update failures
Elasticsearch runs a validate-before-mint search probe with the caller's credential before storing a new internal key. Probe failures fail the create or update synchronously.
Missing index privileges in a linked project:
User lacks the required permissions to read datafeed indices on project [...].
On a specific index in the origin project or a qualified index pattern:
User lacks the required permissions to read datafeed index [...].
The bracketed project alias or index name varies. When no single index is named:
User lacks the required permissions to read from the datafeed indices.
Other probe failures surface as:
Datafeed search probe failed with status [FORBIDDEN]
The bracketed status is the RestStatus enum name (for example UNAUTHORIZED or FORBIDDEN), not an HTTP status code.
Routing that matches no linked project is reported separately. See Project scope problems.
Runtime failures
After minting succeeds, periodic searches can fail when the stored credential is invalid or lacks privileges. Job messages record:
Invalidated or expired key:
Internal cloud API key [abc123def456] failed authentication during datafeed search; it may have been revoked or expired. Re-key by issuing a cloud-authenticated POST _ml/datafeeds/_update on this datafeed
The bracketed value is the key identifier (the same authorization.cloud_api_key.id from GET _ml/datafeeds/{datafeed_id}), not the credential secret. Elasticsearch never logs or returns the internal key text.
Insufficient privileges:
Datafeed search was denied (forbidden) while using internal cloud API key [abc123def456]; the key's privileges or the requesting user's cross-project access may be insufficient. Verify the key and the datafeed owner's project privileges, then re-key with a cloud-authenticated update if the key is the cause
Credential cleared
When a caller that is not cloud-authenticated updates a datafeed that still has a stored internal key, Elasticsearch clears the credential instead of re-keying it. Job messages record:
Internal cloud API key cleared on datafeed update with non-cloud credentials
After clearing, GET _ml/datafeeds/{datafeed_id} omits authorization.cloud_api_key.id. Cross-project portions of the search fail until a cloud-authenticated update mints a new key.
Lifecycle messages for context
Success entries for timeline context (not failure signals):
Internal cloud API key minted for cross-project datafeed
Internal cloud API key re-keyed for cross-project datafeed update
Internal cloud API key revoked for cross-project datafeed
Skipping revocation of cloud API key [abc123def456] — revoke primitive not yet available
A mint or re-key followed by a runtime failure means the key worked at create or update time but later stopped authorizing search.
Stop the datafeed before updating it. If the datafeed is still running, the update fails with:
Cannot update datafeed [my-datafeed] while its status is started
The bracketed datafeed id and status vary with your configuration.
When a project_routing update changes the effective search scope, Elasticsearch retains the job's current model snapshot as a rollback point before applying the change. By default, that rollback gate requires the anomaly detection job to be closed. If the job is still open, the update fails with:
Cannot update project_routing for datafeed [my-datafeed] while job [my-job] is opened. Close the job so a rollback model snapshot can be retained.
The bracketed datafeed id, job id, and job state vary with your configuration.
The rollback gate also requires an existing model snapshot. If the job has never produced one, the update fails with:
Cannot update project_routing for datafeed [my-datafeed] because job [my-job] has no model snapshot to use as a rollback point. Open the job, ingest data, then close it before changing scope.
A job that has never run has no snapshot. Open it, let it process data, then close it before changing project scope.
Elasticsearch records the retained rollback snapshot in the job's Job messages (or .ml-notifications-*), for example:
Rollback model snapshot [1720000000] retained before project_routing scope change: Automatic rollback snapshot retained before project_routing scope change [] -> [_alias:prod-*]
When the datafeed already had a stored routing value, the bracketed pair shows the old and new expressions instead of an empty left side:
Rollback model snapshot [1720000000] retained before project_routing scope change: Automatic rollback snapshot retained before project_routing scope change [_alias:_origin] -> [_alias:prod-*]
You can revert to that snapshot if detection quality degrades after a scope change.
Exception: assigning _alias:_origin for the first time to a datafeed that had no project_routing preserves the existing local-only scope. That update bypasses the rollback gate entirely. Neither the closed-job check nor the snapshot requirement applies.
Re-create or re-key the credential
Resolve missing index privileges, routing that matches nothing, or other validate-before-mint errors before retrying create or update. Sign in to Kibana or call the API as a cloud-authenticated user: a Elastic Cloud session or cloud-managed credential, not a stack API key alone.
When a stored key still exists but no longer works, stop the datafeed and issue a force re-key:
POST _ml/datafeeds/my-datafeed/_update
{
"_force_rekeying": true
}
Replace my-datafeed with your datafeed id. Expect Internal cloud API key re-keyed for cross-project datafeed update in Job messages and a new authorization.cloud_api_key.id on GET _ml/datafeeds/{datafeed_id}.
Elasticsearch also re-keys automatically when a cloud-authenticated update changes the cross-project search surface: project_routing, indices, or indices_options. A no-op update without _force_rekeying does not replace an existing key.
When the credential was cleared, a cloud-authenticated update mints a new key even if the configuration is unchanged.
Avoid clearing the credential
Use Kibana signed in to Elastic Cloud or an API client with cloud-managed authentication. Non-cloud-authenticated callers clear the internal key instead of minting or re-keying it. _force_rekeying requires cloud authentication; operator basic-auth updates return a validation error.
Verify recovery
GET _ml/datafeeds/{datafeed_id}
GET _ml/datafeeds/{datafeed_id}/_stats
Expect a new authorization.cloud_api_key.id, a mint or re-key entry in Job messages, and successful extraction cycles in _stats once linked projects authorize search again.
Escalation
If a cloud-authenticated force re-key or surface-changing update completes but cross-project search still fails, contact Elastic support with:
- Origin project id
- anomaly detection job and datafeed ids
authorization.cloud_api_key.idbefore and after the update- Relevant Job messages excerpts (probe errors, runtime failures, or lifecycle entries)
Superseded key revocation
After re-keying, Elasticsearch best-effort revokes the old key. If revocation fails:
Failed to revoke internal cloud API key [abc123def456]
Search uses the new key. The message is informational and does not block recovery.