---
title: Cloud token runtime failure
description: After cross-project search credentials were minted successfully, the stored internal cloud API key can later fail authorization during periodic datafeeds...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7403/troubleshoot/elasticsearch/machine-learning/cps-datafeed-cloud-token-runtime-failure
products:
  - Elastic Cloud Serverless
  - Elasticsearch
  - Machine Learning
applies_to:
  - Elastic Cloud Serverless: Preview
  - Elastic Stack: Unavailable
---

# Cloud token runtime failure
After cross-project search credentials were minted successfully, the stored internal cloud API key can **later** fail authorization during periodic datafeeds searches. Cross-project portions of the search may be skipped or degraded while the datafeeds keeps running on whatever scope still succeeds.
If create or update fails immediately with a mint error, see [Cloud token mint failure](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7403/troubleshoot/elasticsearch/machine-learning/cps-datafeed-cloud-token-mint-failure) instead.
<tip>
  If you can't find your issue here, explore the other [troubleshooting topics](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7403/troubleshoot) or [contact us](/elastic/docs-content/pull/7403/troubleshoot#contact-us).
</tip>


## Diagnose cloud token runtime failure

**When it happens**
Runtime failure occurs **after** a successful create or update. The datafeeds holds a persisted internal credential but that key no longer authorizes cross-project search.
**Check the stored key id**
```json
```

When an internal credential exists, the response includes `authorization.cloud_api_key.id` (never the raw credential envelope).
Verify key status:
```json
```

Look for `invalidated: true`, expiration, or a missing key.
**Review job Messages**
Open **ML → Anomaly Detection → your job → Messages** (or query `.ml-notifications-*`) for credential lifecycle entries such as:
```txt
Internal cloud API key minted for cross-project datafeed
```

```txt
Internal cloud API key re-keyed for cross-project datafeed update
```

A mint or re-key message followed by new extraction auth errors suggests the stored key is no longer valid at runtime.
**Rule out linked-project skips**
Auth failures can co-occur with project skips. Check `remote_cluster_stats` on stats and read the full extraction error — pure connectivity skips use a different root message:
```txt
[N] remote clusters out of [M] were skipped when performing datafeed search
```

See [Linked project skipped](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7403/troubleshoot/elasticsearch/machine-learning/cps-datafeed-linked-project-skipped) when skips dominate and the key is still valid.

## Resolve cloud token runtime failure

**Force re-key with a cloud-authenticated update**
Issue an update signed in as a valid Elastic Cloud user (Kibana session or cloud-managed API credential). Even a no-op config change triggers a fresh internal key:
```json

{
  "indices": ["<unchanged>"],
  "project_routing": "<unchanged or explicit current value>"
}
```

<important>
  Use a cloud-authenticated caller. Stack API keys alone may clear the internal credential instead of re-keying it.
</important>

**Verify recovery**
```json
```

Expect a new `authorization.cloud_api_key.id`, `invalidated: false`, and restored linked-project search scope in `remote_cluster_stats`. **Messages** should record a re-key event.
**Escalate if re-key does not help**
If a cloud-authenticated re-key completes but cross-project search still fails authorization, contact Elastic support with:
- Origin project id
- datafeeds and job id
- `authorization.cloud_api_key.id` before and after re-key
- Relevant **Messages** excerpts