---
title: Troubleshoot cloud credential problems
description: A cross-project search datafeed stores an internal cloud API key so periodic searches can read linked projects on your behalf. Elasticsearch mints that...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/8277/troubleshoot/elasticsearch/machine-learning/cps-datafeed-credentials
products:
  - Elastic Cloud Serverless
  - Elasticsearch
  - Machine Learning
applies_to:
  - Elastic Cloud Serverless: Preview
  - Elastic Stack: Unavailable
---

# Troubleshoot cloud credential problems
A cross-project search datafeed stores an internal cloud API key so periodic searches can read linked projects on your behalf. Elasticsearch mints that key only when you create or update the datafeed in Kibana, or through the API with an [Elastic Cloud API key](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/8277/deploy-manage/api-keys/elastic-cloud-api-keys), not during a scheduled extraction cycle. Failures fall into three paths covered on this page: the key could not be created, an existing key no longer authorizes search, or the key was cleared or never minted.

## Where to look

Use these sources to gather diagnostic information:
- **Anomaly detection job job messages in Kibana**: Open **Machine Learning → Anomaly Detection**, select the job, and review the **Job messages** tab for audit entries and warnings about linked projects, credentials, or scope changes. On the **Datafeed** tab, **View datafeed counts** opens the datafeed chart flyout for extraction timing.
  The same entries are stored in `.ml-notifications-*`.
- **`GET _ml/datafeeds/{datafeed_id}`**: Shows the effective `project_routing` value and, when an internal cloud API key exists, `authorization.cloud_api_key.id`.
- **`GET _ml/datafeeds/{datafeed_id}/_stats`**: While the datafeed runs, shows `remote_cluster_stats` with `total_clusters`, `available_clusters`, `skipped_clusters`, `availability_ratio`, `stabilized_cluster_aliases`, and `per_cluster_consecutive_skips`. The object is absent until the first search cycle establishes a baseline.
- **`.ml-annotations-read`**: Scope-change annotations for the job. The annotation `event` field carries `search_scope_changed` (not the separate `type` field).
- **`GET /_project/tags`**: Lists linked projects and their tags so you can compare them with a routing expression.
- **Elastic Cloud console**: Review linked projects in [Link and manage projects](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/8277/deploy-manage/cross-project-search-config/cps-config-link-and-manage).

If extraction failures are ongoing, check **Job messages** first. `remote_cluster_stats` from [get datafeed stats](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-datafeed-stats) only updates after a cycle completes.

## Creation or update failures


### What you see

Elasticsearch probes the search with the caller's credential before storing a new internal key. Probe failures fail the create or update synchronously.
Missing index privileges in a linked project:
```txt
User lacks the required permissions to read datafeed indices on project [...].
```

On a specific index in the origin project or a qualified index pattern:
```txt
User lacks the required permissions to read datafeed index [...].
```

The bracketed project alias or index name varies. When no single index is named:
```txt
User lacks the required permissions to read from the datafeed indices.
```

Other probe failures surface as:
```txt
Datafeed search probe failed with status [FORBIDDEN]
```

The bracket shows a status name such as `UNAUTHORIZED` or `FORBIDDEN`, not an HTTP status code.
Routing that matches no linked project is reported separately. See [Project scope problems](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/8277/troubleshoot/elasticsearch/machine-learning/cps-datafeed-project-scope).

### Fix

Resolve missing index privileges, routing that matches nothing, or other probe errors before retrying create or update. Retry in Kibana, or through the API with an [Elastic Cloud API key](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/8277/deploy-manage/api-keys/elastic-cloud-api-keys). [Elasticsearch API keys](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/8277/deploy-manage/api-keys/serverless-project-api-keys) are scoped to a single project. They cannot mint or re-key the internal credential, and they clear it instead.

### Verify

The create or update succeeds. Job messages show `Internal cloud API key minted for cross-project datafeed`. `GET _ml/datafeeds/{datafeed_id}` includes `authorization.cloud_api_key.id`.

## Runtime authorization failures


### What you see

After minting succeeds, periodic searches can fail when the stored credential is invalid or lacks privileges. Job messages record:
Invalidated or expired key:
```txt
Internal cloud API key [abc123def456] failed authentication during datafeed search; it may have been revoked or expired. Re-key by issuing a cloud-authenticated POST _ml/datafeeds/_update on this datafeed
```

The bracketed value is the key **identifier** (the same `authorization.cloud_api_key.id` from `GET _ml/datafeeds/{datafeed_id}`), not the credential secret. Elasticsearch never logs or returns the internal key text.
Insufficient privileges:
```txt
Datafeed search was denied (forbidden) while using internal cloud API key [abc123def456]; the key's privileges or the requesting user's cross-project access may be insufficient. Verify the key and the datafeed owner's project privileges, then re-key with a cloud-authenticated update if the key is the cause
```

A mint or re-key followed by a runtime failure means the key worked at create or update time but later stopped authorizing search.

### Fix

Stop the datafeed and issue a force re-key:
```json

{
  "_force_rekeying": true
}
```

Replace `my-datafeed` with your datafeed id. Call `_force_rekeying` from Console while signed in to Kibana, or through the API with an Elastic Cloud API key. Elasticsearch API keys return a validation error.
Elasticsearch also re-keys automatically when you change the cross-project search surface from Kibana, or through the API with an Elastic Cloud API key: `project_routing`, `indices`, or `indices_options`. A no-op update without `_force_rekeying` does not replace an existing key.

### Verify

```json
```

Expect a new `authorization.cloud_api_key.id`, a re-key entry in job messages, and successful extraction cycles in `_stats` once linked projects authorize search again.
After re-keying, Elasticsearch best-effort revokes the old key. If revocation fails, job messages might show `Failed to revoke internal cloud API key [abc123def456]`. Search uses the new key. The message is informational and does not block recovery.

## Cleared or never-minted credential


### What you see

When `authorization.cloud_api_key.id` is missing, one of two things happened:
- **Credential cleared**: job messages record `Internal cloud API key cleared on datafeed update with non-cloud credentials` after an update that used an Elasticsearch API key or other non-Elastic Cloud credential removed a stored key.
- **Never minted**: the datafeed was created or last updated outside Kibana without an Elastic Cloud API key, so no key was ever stored and the cleared-key message never appears. Job messages lack `Internal cloud API key minted for cross-project datafeed`.

After clearing, `GET _ml/datafeeds/{datafeed_id}` omits `authorization.cloud_api_key.id`. Cross-project portions of the search fail until you update the datafeed in Kibana, or through the API with an Elastic Cloud API key.

### Fix

Update the datafeed in Kibana, or through the API with an Elastic Cloud API key. When the credential was cleared, that update mints a new key even if the configuration is unchanged.

### Verify

`GET _ml/datafeeds/{datafeed_id}` includes a new `authorization.cloud_api_key.id`. Job messages show `Internal cloud API key minted for cross-project datafeed` or `Internal cloud API key re-keyed for cross-project datafeed update`.

### When to contact support

If a force re-key or surface-changing update in Kibana, or through the API with an Elastic Cloud API key, completes but cross-project search still fails, contact [Elastic support](/elastic/docs-content/pull/8277/troubleshoot#contact-us) with:
- Origin project id
- anomaly detection job and datafeed ids
- `authorization.cloud_api_key.id` before and after the update
- Relevant job message excerpts (probe errors, runtime failures, or lifecycle entries)