Loading

Troubleshoot Automatic Migration

The .kibana-siem-rule-migrations-integrations index contains a semantic_text field named elser_embedding that is intended to use Elastic's ELSER sparse embedding model for semantic search. However, on some deployments the field might be bound to a different embedding model (for example, a Jina dense embedding model with inference_id: .jina-embeddings-v5-text-small) instead.

This can cause the following symptom:

  • No integrations found for translated rules — after completing an automatic migration, the Integrations column on the Translated rules page shows no integrations even when they should exist, because integration matching relies on semantic search against this index.

The root cause is that when the index was created, the ELSER inference endpoint was not available, so Elasticsearch fell back to whichever inference endpoint was configured. The field name elser_embedding is a label, but what matters is the inference_id baked into the index mapping at creation time. To confirm whether the field is bound to the wrong inference endpoint, refer to Check the elser_embedding inference ID.

The fix requires re-creating the index with the correct ELSER inference endpoint and re-indexing all documents so their embeddings are regenerated using ELSER.

  • ELSER ML model must be available on the cluster.
  • You must have index admin privileges.
  • Expect downtime or degraded search on this index during the reindex process.

Run the following API requests in Kibana Dev Tools Console.

1a. Check what inference endpoints are currently configured on the cluster:

GET _inference

Confirm whether an ELSER endpoint exists. If you only see Jina or other non-ELSER endpoints, ELSER has not been deployed on this cluster and must be created in Step 2.

GET .kibana-siem-rule-migrations-integrations/_mapping/field/elser_embedding

Look for the inference_id inside the field mapping. If it shows .jina-embeddings-v5-text-small (or anything other than an ELSER endpoint), the field is misconfigured and the remediation steps in this guide are required. For example:

"elser_embedding": {
  "type": "semantic_text",
  "inference_id": ".jina-embeddings-v5-text-small"
}

The corrected mapping should reference an ELSER inference endpoint, such as:

"elser_embedding": {
  "type": "semantic_text",
  "inference_id": "elser-2-elasticsearch"
}

You can create the ELSER inference endpoint with the API request below, or deploy ELSER from the Kibana Model Management > Trained Models page. For more information, refer to ELSER.

PUT _inference/sparse_embedding/elser-2-elasticsearch
{
  "service": "elasticsearch",
  "service_settings": {
    "adaptive_allocations": {
      "enabled": true,
      "min_number_of_allocations": 1,
      "max_number_of_allocations": 10
    },
    "num_threads": 1,
    "model_id": ".elser_model_2"
  }
}

This registers a new ELSER sparse embedding inference endpoint with the ID elser-2-elasticsearch. This ID will be referenced in the index mapping. If you deploy ELSER from Kibana instead, use that endpoint ID in the mapping examples below.

Note

The inference ID cannot start with a dot (.) — that prefix is reserved for system-provisioned endpoints on Elastic Cloud. Use a plain alphanumeric name.

Adjust the adaptive allocation settings and num_threads for better throughput during reindex if your cluster has spare ML capacity.

PUT kibana-siem-rule-migrations-integrations-backup
{
  "settings": {
    "index.mapping.total_fields.limit": 2000
  },
  "mappings": {
    "dynamic": false,
    "properties": {
      "id": { "type": "keyword" },
      "title": { "type": "text" },
      "description": { "type": "text" },
      "data_streams": {
        "type": "object",
        "properties": {
          "title": { "type": "text" },
          "dataset": { "type": "keyword" },
          "index_pattern": { "type": "keyword" }
        }
      },
      "elser_embedding": {
        "type": "semantic_text",
        "inference_id": "elser-2-elasticsearch"
      }
    }
  }
}
Note

Why dynamic: false: ELSER sparse vectors store thousands of unique NLP tokens as sub-fields at index time. With dynamic: true (the default), each token becomes a new mapped field, quickly exhausting the total_fields.limit. Setting dynamic: false prevents these token fields from being added to the mapping while still allowing them to be indexed and searched correctly.

Why a backup index first: The original index cannot have its inference_id changed in-place — mappings are immutable for semantic_text fields. A backup index lets you validate the reindex succeeded before destroying the original, and it remains available after final verification.

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": ".kibana-siem-rule-migrations-integrations",
    "_source": {
      "excludes": ["elser_embedding"]
    }
  },
  "dest": {
    "index": "kibana-siem-rule-migrations-integrations-backup"
  },
  "script": {
    "source": """
      def embedding = ctx._source.remove('elser_embedding');
      if (embedding != null) {
        if (embedding instanceof Map && embedding.containsKey('text')) {
          ctx._source['elser_embedding'] = embedding['text'];
        } else if (embedding instanceof String) {
          ctx._source['elser_embedding'] = embedding;
        }
      }
    """,
    "lang": "painless"
  }
}
Note

Why excludes: ["elser_embedding"]: The source documents contain vector data stored inside the elser_embedding field structure. Excluding it from _source prevents the raw vector payload from being copied. The Painless script then re-injects only the plain text back into elser_embedding, which triggers the new ELSER inference endpoint to generate fresh sparse embeddings on ingest.

wait_for_completion=false: Returns a task ID immediately so the reindex runs in the background. Use the task ID in Step 5 to monitor progress.

GET _tasks/<task_id>

Replace <task_id> with the one returned in Step 4.

Check for:

  • "completed": true — reindex finished.
  • "failures": [] — no documents failed.
  • response.created should equal response.total.
GET .kibana-siem-rule-migrations-integrations/_count
GET kibana-siem-rule-migrations-integrations-backup/_count

Both counts must be equal before proceeding. If the backup index has fewer documents, check the task failures from Step 5 before continuing.

DELETE .kibana-siem-rule-migrations-integrations
Warning

This is irreversible. Only proceed if Step 6 confirmed counts match and Step 5 showed zero failures.

Kibana SIEM rule migration features that depend on this index will be unavailable until Step 9 completes.

PUT .kibana-siem-rule-migrations-integrations
{
  "settings": {
    "index.mapping.total_fields.limit": 2000
  },
  "mappings": {
    "dynamic": false,
    "properties": {
      "id": { "type": "keyword" },
      "title": { "type": "text" },
      "description": { "type": "text" },
      "data_streams": {
        "type": "object",
        "properties": {
          "title": { "type": "text" },
          "dataset": { "type": "keyword" },
          "index_pattern": { "type": "keyword" }
        }
      },
      "elser_embedding": {
        "type": "semantic_text",
        "inference_id": "elser-2-elasticsearch"
      }
    }
  }
}

The original index name is hardcoded in Kibana's SIEM rule migration code. The index must exist under its original name. This step recreates it with the correct mapping used for the backup index.

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "kibana-siem-rule-migrations-integrations-backup"
  },
  "dest": {
    "index": ".kibana-siem-rule-migrations-integrations"
  }
}

The backup index already contains correct ELSER embeddings, so no script or exclusions are needed — this is a straight document copy.

GET _tasks/<task_id>

Replace <task_id> with the one returned in Step 9. Apply the same checks as Step 5.

GET .kibana-siem-rule-migrations-integrations/_count
GET kibana-siem-rule-migrations-integrations-backup/_count

Confirm that both document counts match.

GET .kibana-siem-rule-migrations-integrations/_mapping/field/elser_embedding

The inference_id should now show elser-2-elasticsearch. If it does, the field is correctly bound to ELSER. Keep kibana-siem-rule-migrations-integrations-backup available as a backup after final verification.