Multimodal search

Multimodal search finds results by meaning across more than one content type. Common use cases include text-to-image product search, finding similar images, searching media libraries by natural language, and retrieving screenshots or document pages with text or image queries.

Multimodal search builds on vector search. A multimodal embedding model maps each supported input into a dense vector so content with similar meaning is nearby in vector space, even when the media types differ.

Multimodal embedding models map different media types into a shared vector space. Text, images, and other supported modalities (such as audio, video, or PDF, depending on the model) become dense vectors that live in the same space. That shared space is what makes cross-modal retrieval possible: a text query can match an image embedding, and an image query can match text or other images.

Use the same model (and compatible endpoint settings) at ingest and at search time. Mixing models breaks similarity comparisons because the vector spaces are not interchangeable.

In Elasticsearch, the model is exposed through an inference endpoint that uses the embedding task type. The endpoint determines which modalities you can index and query.

Common multimodal search use cases include:

Product and catalog visual search

Shoppers describe an item in natural language ("leather crossbody bag with gold zipper") and retrieve matching product photos, even when titles and tags are incomplete. Combine with filters for price, brand, or availability.

Similar-image and reverse image search

A user uploads a photo or selects an existing asset and finds visually similar products, duplicates, or near-duplicates in a catalog or media library.

Digital asset management

Search large image and media collections by meaning instead of filenames or manual tags. For example, find campaign assets that match a brief written in plain language.

Multilingual text-to-image search

Query an image index in one language and retrieve the same visual results you would get in another, when the embedding model is trained for multilingual text-image matching.

PDF and document-page search

Embed document pages (or PDFs) so queries can match layout and visual content (diagrams, tables, scanned pages), not only extracted plain text.

Multimodal retrieval for RAG

Retrieve images, charts, or document pages alongside text passages so a downstream generative model can ground answers in visual evidence as well as prose.

Compare the multimodal embedding models available with Elasticsearch:

For the full Jina catalog, deployment matrix, and input examples, refer to Jina models. For EIS availability by stack version, refer to Supported models on EIS.

You can deploy or access Jina multimodal models in these ways:

• Elastic Inference Service (EIS): Elastic hosts the model. Use this when you want managed inference without provisioning ML nodes.
• Jina API: The model runs on the hosted Jina platform. Use this when you want Jina-hosted inference outside EIS.
• On-prem: You run the model in Docker on your own infrastructure. Use this for air-gapped, offline, or compliance scenarios.

The semantic field type is the simplest way to run multimodal search in Elasticsearch. For field parameters, defaults, supported input types, and limitations, refer to the semantic field documentation.

The semantic field type simplifies semantic and multimodal search across text, images, audio, video, and PDF files. With a compatible multimodal embedding model, you can search from any supported input type to any other supported input type. The field automatically:

• Generates embeddings when you index field values, without an ingest pipeline or inference processor.
• Splits long text into smaller passages, called chunks.
• Indexes the generated embeddings using default index options that optimize for common use cases.
• Searches the embeddings generated for each value or text chunk.

Here's an example using the Jina Embeddings v5 Omni Small endpoint:

PUT my-multimodal-index
{
  "mappings": {
    "properties": {
      "content": {
        "type": "semantic",
        "inference_id": ".jina-embeddings-v5-omni-small"
      }
    }
  }
}
		

Multiple semantic fields can share one inference endpoint. For example, one field for images and another for descriptions; you can then search either field or both.

• Build multimodal search with a semantic field: Index images and search them with text, image, and PDF input
• semantic field type: Mapping overview and comparison guidance
• Jina models: Model catalog, deployment options, and input formats
• Elastic Inference Service: Hosted inference without managing ML nodes
• inference API: Create and manage inference endpoints