Loading

Troubleshoot cross-project search datafeeds

Anomaly detection datafeeds on Elastic Cloud Serverless can search data across linked projects when cross-project search is configured. These topics help you diagnose and resolve problems with Project scope (project_routing), internal cloud credentials, linked-project availability, and field mappings.

Before you troubleshoot, confirm that projects are linked and that users have access. See Link and manage projects and the Cross-project search overview.

Tip

If you can't find your issue here, explore the other troubleshooting topics or contact us.

Where to look

  • anomaly detection job job messages in Kibana: Open Machine Learning → Anomaly Detection, select the job, and review the Job messages tab for audit entries and warnings about linked projects, credentials, or scope changes. On the Datafeed tab, View datafeed counts opens the datafeed chart flyout for extraction timing.

    The same entries are stored in .ml-notifications-*.

  • GET _ml/datafeeds/{datafeed_id}: Shows the effective project_routing value and, when an internal cloud credential exists, authorization.cloud_api_key.id.

  • GET _ml/datafeeds/{datafeed_id}/_stats: While the datafeed runs, shows remote_cluster_stats with total_clusters, available_clusters, skipped_clusters, availability_ratio, stabilized_cluster_aliases, and per_cluster_consecutive_skips. The object is absent until the first search cycle establishes a baseline.

  • .ml-annotations-*: Scope-change annotations for the job. The annotation event field carries search_scope_changed (not the separate type field).

  • GET /_project/tags: Lists linked projects and their tags so you can compare them with a routing expression.

  • Elastic Cloud console: Review linked projects in Link and manage projects.

During active extraction problems, Job messages are authoritative. remote_cluster_stats from get datafeed stats reflects the last completed cycle.

Start with the symptom that best matches what you see:

Symptom Start here When to use a different page
The datafeed returns no results Project scope problems —
Results come only from the origin project Project scope problems When project_routing is _alias:_origin or the job has no stored routing (legacy default). When authorization.cloud_api_key.id is missing or Job messages report a cleared or never-minted key, use Cloud credential problems instead
Extraction cycles are suddenly slower after you linked projects Project scope problems —
The datafeed keeps failing with extraction errors Linked project unavailable When Job messages report a skipped linked project. For authorization failures, use Cloud credential problems. For field type conflicts, use Field mapping conflicts
Elasticsearch or Kibana rejected a project scope change Project scope changes —
Some jobs failed during a bulk Change project scope update Project scope changes —
Anomaly scores spiked after a scope change Project scope changes —
A field is missing, a project is excluded from a run, or mappings conflict across projects Field mapping conflicts —
Authorization errors after the datafeed had been working Cloud credential problems —

If you already have an error or audit string from Job messages, the API, or Kibana, use this index:

Message (substring match) Page
matched no linked project / cannot search any project Project scope problems
remote clusters out of / were skipped when performing datafeed search Linked project unavailable
Cannot update project_routing / no model snapshot to use as a rollback point / while its status is started Project scope changes
Rollback model snapshot retained before project_routing scope change Project scope changes
CPS migration: project_routing defaulted Project scope changes
Datafeed search scope changed / Elevated anomaly scores detected after search scope change Project scope changes
Internal cloud API key / Datafeed search probe failed / User lacks the required permissions Cloud credential problems
Failed to revoke internal cloud API key Cloud credential problems
Cross-project field conflict / conflicting types across projects / excluded project from this run Field mapping conflicts
Cannot run datafeed + required time field Field mapping conflicts
Datafeed has recovered data extraction / started retrieving data again Linked project unavailable

Every valid project_routing value starts with _alias:. Bare _origin is not a routing value. Each value is a single routing expression; comma-separated lists (for example _alias:a,b) are not supported.

Alias matching is case-insensitive. Prefix, suffix, and contains wildcards are supported. Internal wildcards (for example _alias:prod*eu) and multiple sequential wildcards are not.

project_routing value Effective search scope
Empty or omitted (null) Flat world: searches the origin project and all linked projects
_alias:_origin Origin only: searches the origin project
_alias:* Same as flat world: equivalent to an empty or omitted value. Searches the origin project and all linked projects
_alias:production-* (prefix wildcard) Subset: linked projects whose alias starts with production-, plus the origin project when its alias matches
_alias:*-prod (suffix wildcard) Subset: linked projects whose alias ends with -prod, plus the origin project when its alias matches
_alias:*staging* (contains wildcard) Subset: linked projects whose alias contains staging, plus the origin project when its alias matches
_alias:production-us (exact alias) Single linked project: only the named alias, plus the origin project when its alias matches

Index qualifier, not routing. In indices, prefix a pattern with _origin: to target the origin project (for example _origin:logs-*). That qualifier is separate from project_routing.

For syntax details and examples, refer to Project routing in CPS.

Contact Elastic support when:

  • A linked project or region appears unavailable across multiple jobs and you have confirmed project linking in Elastic Cloud.
  • The same datafeed fails repeatedly after you apply the fixes in these topics.
  • The origin project reports memory pressure or out-of-memory errors while CPS datafeeds are running, and narrowing project_routing does not relieve the symptoms.