---
title: Cloud credential problems
description: A cross-project search datafeed stores an internal cloud (UIAM) API key so periodic searches can read linked projects on your behalf. Elasticsearch mints...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7403/troubleshoot/elasticsearch/machine-learning/cps-datafeed-credentials
products:
  - Elastic Cloud Serverless
  - Elasticsearch
  - Machine Learning
applies_to:
  - Elastic Cloud Serverless: Preview
  - Elastic Stack: Unavailable
---

# Cloud credential problems
A cross-project search datafeed stores an internal cloud (UIAM) API key so periodic searches can read linked projects on your behalf. Elasticsearch mints that key only when a cloud-authenticated caller creates or updates the datafeed, not during a scheduled extraction cycle. Failures fall into four paths: the key could not be created, an existing key no longer authorizes search, the key was cleared by a non-cloud-authenticated update, or no key was ever minted because the datafeed was created or last updated without cloud authentication.

## Diagnose cloud credential problems

**Where to look**
- **anomaly detection job job messages in Kibana**: Open **Machine Learning → Anomaly Detection**, select the job, and review the **Job messages** tab for audit entries and warnings about linked projects, credentials, or scope changes. On the **Datafeed** tab, **View datafeed counts** opens the datafeed chart flyout for extraction timing.
  The same entries are stored in `.ml-notifications-*`.
- **`GET _ml/datafeeds/{datafeed_id}`**: Shows the effective `project_routing` value and, when an internal cloud credential exists, `authorization.cloud_api_key.id`.
- **`GET _ml/datafeeds/{datafeed_id}/_stats`**: While the datafeed runs, shows `remote_cluster_stats` with `total_clusters`, `available_clusters`, `skipped_clusters`, `availability_ratio`, `stabilized_cluster_aliases`, and `per_cluster_consecutive_skips`. The object is absent until the first search cycle establishes a baseline.
- **`.ml-annotations-*`**: Scope-change annotations for the job. The annotation `event` field carries `search_scope_changed` (not the separate `type` field).
- **`GET /_project/tags`**: Lists linked projects and their tags so you can compare them with a routing expression.
- **Elastic Cloud console**: Review linked projects in [Link and manage projects](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7403/deploy-manage/cross-project-search-config/cps-config-link-and-manage).

During active extraction problems, **Job messages** are authoritative. `remote_cluster_stats` from [get datafeed stats](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-datafeed-stats) reflects the last **completed** cycle.
**Which failure is this?**
Use observable behavior to pick a path:
- **Create or update fails immediately**: Error during validation before minting or during minting itself. See **Creation and update failures** below.
- **Searches used to work and** `GET _ml/datafeeds/{datafeed_id}` **contains** `authorization.cloud_api_key.id`: runtime auth or authz failure. See **Runtime failures** below.
- `authorization.cloud_api_key.id` **is missing**: two cases:
  - **Credential cleared**: **Job messages** record `Internal cloud API key cleared on datafeed update with non-cloud credentials` after a non-cloud-authenticated update removed a stored key. See **Credential cleared** below.
- **Never minted**: the datafeed was created or last updated without cloud authentication, so no key was ever stored and no CLEARED message appears. **Job messages** lack `Internal cloud API key minted for cross-project datafeed`. Mint a key with a cloud-authenticated create or update.

**Creation and update failures**
Elasticsearch runs a validate-before-mint search probe with the caller's credential before storing a new internal key. Probe failures fail the create or update synchronously.
Missing index privileges in a linked project:
```txt
User lacks the required permissions to read datafeed indices on project [...].
```

On a specific index in the origin project or a qualified index pattern:
```txt
User lacks the required permissions to read datafeed index [...].
```

The bracketed project alias or index name varies. When no single index is named:
```txt
User lacks the required permissions to read from the datafeed indices.
```

Other probe failures surface as:
```txt
Datafeed search probe failed with status [FORBIDDEN]
```

The bracketed status is the `RestStatus` enum name (for example `UNAUTHORIZED` or `FORBIDDEN`), not an HTTP status code.
Routing that matches no linked project is reported separately. See [Project scope problems](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/7403/troubleshoot/elasticsearch/machine-learning/cps-datafeed-project-scope).
**Runtime failures**
After minting succeeds, periodic searches can fail when the stored credential is invalid or lacks privileges. **Job messages** record:
Invalidated or expired key:
```txt
Internal cloud API key [abc123def456] failed authentication during datafeed search; it may have been revoked or expired. Re-key by issuing a cloud-authenticated POST _ml/datafeeds/_update on this datafeed
```

The bracketed value is the key **identifier** (the same `authorization.cloud_api_key.id` from `GET _ml/datafeeds/{datafeed_id}`), not the credential secret. Elasticsearch never logs or returns the internal key text.
Insufficient privileges:
```txt
Datafeed search was denied (forbidden) while using internal cloud API key [abc123def456]; the key's privileges or the requesting user's cross-project access may be insufficient. Verify the key and the datafeed owner's project privileges, then re-key with a cloud-authenticated update if the key is the cause
```

**Credential cleared**
When a caller that is **not** cloud-authenticated updates a datafeed that still has a stored internal key, Elasticsearch clears the credential instead of re-keying it. **Job messages** record:
```txt
Internal cloud API key cleared on datafeed update with non-cloud credentials
```

After clearing, `GET _ml/datafeeds/{datafeed_id}` omits `authorization.cloud_api_key.id`. Cross-project portions of the search fail until a cloud-authenticated update mints a new key.
**Lifecycle messages for context**
Success entries for timeline context (not failure signals):
```txt
Internal cloud API key minted for cross-project datafeed
```

```txt
Internal cloud API key re-keyed for cross-project datafeed update
```

```txt
Internal cloud API key revoked for cross-project datafeed
```

```txt
Skipping revocation of cloud API key [abc123def456] — revoke primitive not yet available
```

A mint or re-key followed by a runtime failure means the key worked at create or update time but later stopped authorizing search.

## Resolve cloud credential problems

Stop the datafeed before updating it. If the datafeed is still running, the update fails with:
```txt
Cannot update datafeed [my-datafeed] while its status is started
```

The bracketed datafeed id and status vary with your configuration.
When a `project_routing` update changes the effective search scope, Elasticsearch retains the job's current model snapshot as a rollback point before applying the change. By default, that rollback gate requires the anomaly detection job to be closed. If the job is still open, the update fails with:
```txt
Cannot update project_routing for datafeed [my-datafeed] while job [my-job] is opened. Close the job so a rollback model snapshot can be retained.
```

The bracketed datafeed id, job id, and job state vary with your configuration.
The rollback gate also requires an existing model snapshot. If the job has never produced one, the update fails with:
```txt
Cannot update project_routing for datafeed [my-datafeed] because job [my-job] has no model snapshot to use as a rollback point. Open the job, ingest data, then close it before changing scope.
```

A job that has never run has no snapshot. Open it, let it process data, then close it before changing project scope.
Elasticsearch records the retained rollback snapshot in the job's **Job messages** (or `.ml-notifications-*`), for example:
```txt
Rollback model snapshot [1720000000] retained before project_routing scope change: Automatic rollback snapshot retained before project_routing scope change [] -> [_alias:prod-*]
```

When the datafeed already had a stored routing value, the bracketed pair shows the old and new expressions instead of an empty left side:
```txt
Rollback model snapshot [1720000000] retained before project_routing scope change: Automatic rollback snapshot retained before project_routing scope change [_alias:_origin] -> [_alias:prod-*]
```

You can revert to that snapshot if detection quality degrades after a scope change.
Exception: assigning `_alias:_origin` for the first time to a datafeed that had no `project_routing` preserves the existing local-only scope. That update bypasses the rollback gate entirely. Neither the closed-job check nor the snapshot requirement applies.
**Re-create or re-key the credential**
Resolve missing index privileges, routing that matches nothing, or other validate-before-mint errors before retrying create or update. Sign in to Kibana or call the API as a cloud-authenticated user: a Elastic Cloud session or cloud-managed credential, not a stack API key alone.
When a stored key still exists but no longer works, stop the datafeed and issue a force re-key:
```json

{
  "_force_rekeying": true
}
```

Replace `my-datafeed` with your datafeed id. Expect `Internal cloud API key re-keyed for cross-project datafeed update` in **Job messages** and a new `authorization.cloud_api_key.id` on `GET _ml/datafeeds/{datafeed_id}`.
Elasticsearch also re-keys automatically when a cloud-authenticated update changes the cross-project search surface: `project_routing`, `indices`, or `indices_options`. A no-op update without `_force_rekeying` does not replace an existing key.
When the credential was cleared, a cloud-authenticated update mints a new key even if the configuration is unchanged.
**Avoid clearing the credential**
Use Kibana signed in to Elastic Cloud or an API client with cloud-managed authentication. Non-cloud-authenticated callers clear the internal key instead of minting or re-keying it. `_force_rekeying` requires cloud authentication; operator basic-auth updates return a validation error.
**Verify recovery**
```json
```

Expect a new `authorization.cloud_api_key.id`, a mint or re-key entry in **Job messages**, and successful extraction cycles in `_stats` once linked projects authorize search again.
**Escalation**
If a cloud-authenticated force re-key or surface-changing update completes but cross-project search still fails, contact [Elastic support](/elastic/docs-content/pull/7403/troubleshoot#contact-us) with:
- Origin project id
- anomaly detection job and datafeed ids
- `authorization.cloud_api_key.id` before and after the update
- Relevant **Job messages** excerpts (probe errors, runtime failures, or lifecycle entries)

**Superseded key revocation**
After re-keying, Elasticsearch best-effort revokes the old key. If revocation fails:
```txt
Failed to revoke internal cloud API key [abc123def456]
```

Search uses the new key. The message is informational and does not block recovery.