Cloud token runtime failure

After cross-project search credentials were minted successfully, the stored internal cloud API key can later fail authorization during periodic datafeeds searches. Cross-project portions of the search may be skipped or degraded while the datafeeds keeps running on whatever scope still succeeds.

If create or update fails immediately with a mint error, see Cloud token mint failure instead.

When it happens

Runtime failure occurs after a successful create or update. The datafeeds holds a persisted internal credential but that key no longer authorizes cross-project search.

Check the stored key id

				GET _ml/datafeeds/{datafeed_id}
		

When an internal credential exists, the response includes authorization.cloud_api_key.id (never the raw credential envelope).

Verify key status:

				GET _security/api_key?id={authorization.cloud_api_key.id}
		

Look for invalidated: true, expiration, or a missing key.

Review job Messages

Open ML → Anomaly Detection → your job → Messages (or query .ml-notifications-*) for credential lifecycle entries such as:

Internal cloud API key minted for cross-project datafeed
		

Internal cloud API key re-keyed for cross-project datafeed update
		

A mint or re-key message followed by new extraction auth errors suggests the stored key is no longer valid at runtime.

Rule out linked-project skips

Auth failures can co-occur with project skips. Check remote_cluster_stats on stats and read the full extraction error — pure connectivity skips use a different root message:

[N] remote clusters out of [M] were skipped when performing datafeed search
		

See Linked project skipped when skips dominate and the key is still valid.

Force re-key with a cloud-authenticated update

Issue an update signed in as a valid Elastic Cloud user (Kibana session or cloud-managed API credential). Even a no-op config change triggers a fresh internal key:

				POST _ml/datafeeds/{datafeed_id}/_update
					{
  "indices": ["<unchanged>"],
  "project_routing": "<unchanged or explicit current value>"
}
		

Verify recovery

				GET _ml/datafeeds/{datafeed_id}
				GET _security/api_key?id={new authorization.cloud_api_key.id}
				GET _ml/datafeeds/{datafeed_id}/_stats
		

Expect a new authorization.cloud_api_key.id, invalidated: false, and restored linked-project search scope in remote_cluster_stats. Messages should record a re-key event.

Escalate if re-key does not help

If a cloud-authenticated re-key completes but cross-project search still fails authorization, contact Elastic support with:

• Origin project id
• datafeeds and job id
• authorization.cloud_api_key.id before and after re-key
• Relevant Messages excerpts