Create a Llama inference endpoint

PUT /_inference/{task_type}/{llama_inference_id}

Api key auth

Create an inference endpoint to perform an inference task with the llama service.

Required authorization

Cluster privileges: manage_inference

Path Parameters

Query Parameters

timeout string Duration · default: 30s: Specifies the amount of time to wait for the inference endpoint to be created.

Request

Response

application/json

Examples

Run PUT _inference/text_embedding/llama-text-embedding to create a Llama inference endpoint that performs a text_embedding task.

Request Console 1 / 8 languages

Console Kibana Dev Tools

PUT _inference/text_embedding/llama-text-embedding
{
  "service": "llama",
  "service_settings": {
    "url": "http://localhost:8321/v1/inference/embeddings",
    "dimensions": 384,
    "model_id": "all-MiniLM-L6-v2" 
  }
}

JavaScript Elasticsearch JavaScript Client

const response = await client.inference.put({
  task_type: "text_embedding",
  inference_id: "llama-text-embedding",
  inference_config: {
    service: "llama",
    service_settings: {
      url: "http://localhost:8321/v1/inference/embeddings",
      dimensions: 384,
      model_id: "all-MiniLM-L6-v2",
    },
  },
});

Python Elasticsearch Python Client

resp = client.inference.put(
    task_type="text_embedding",
    inference_id="llama-text-embedding",
    inference_config={
        "service": "llama",
        "service_settings": {
            "url": "http://localhost:8321/v1/inference/embeddings",
            "dimensions": 384,
            "model_id": "all-MiniLM-L6-v2"
        }
    },
)

curl Shell

curl -X PUT "$ELASTICSEARCH_URL/_inference/text_embedding/llama-text-embedding" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "service": "llama",
    "service_settings": {
      "url": "http://localhost:8321/v1/inference/embeddings",
      "dimensions": 384,
      "model_id": "all-MiniLM-L6-v2"
    }
  }'

Java Elasticsearch Java Client

client.inference().put(p -> p
    .inferenceId("llama-text-embedding")
    .taskType(TaskType.TextEmbedding)
    .inferenceConfig(i -> i
        .service("llama")
        .serviceSettings(JsonData.fromJson("""
{"url":"http://localhost:8321/v1/inference/embeddings","dimensions":384,"model_id":"all-MiniLM-L6-v2"}
"""))
    )
);

C# Elasticsearch .NET Client

var response = await client.Inference
    .PutAsync(new PutInferenceRequest()
    {
        InferenceId = "llama-text-embedding",
        TaskType = TaskType.TextEmbedding,
        InferenceConfig = new()
        {
            Service = "llama",
            ServiceSettings = JsonSerializer.Deserialize<JsonElement>("""
            {
              "url": "http://localhost:8321/v1/inference/embeddings",
              "dimensions": 384,
              "model_id": "all-MiniLM-L6-v2"
            }
            """)
        }
    });

PHP Elasticsearch PHP Client

$resp = $client->inference()->put([
    "task_type" => "text_embedding",
    "inference_id" => "llama-text-embedding",
    "body" => [
        "service" => "llama",
        "service_settings" => [
            "url" => "http://localhost:8321/v1/inference/embeddings",
            "dimensions" => 384,
            "model_id" => "all-MiniLM-L6-v2",
        ],
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.inference.put(
  task_type: "text_embedding",
  inference_id: "llama-text-embedding",
  body: {
    "service": "llama",
    "service_settings": {
      "url": "http://localhost:8321/v1/inference/embeddings",
      "dimensions": 384,
      "model_id": "all-MiniLM-L6-v2"
    }
  }
)

Response

No example

Run PUT _inference/completion/llama-completion to create a Llama inference endpoint that performs a completion task.

Request Console 1 / 2 languages

Console Kibana Dev Tools

PUT _inference/completion/llama-completion
{
  "service": "llama",
  "service_settings": {
    "url": "http://localhost:8321/v1/openai/v1/chat/completions",
    "model_id": "llama3.2:3b" 
  }
}

curl Shell

curl -X PUT "$ELASTICSEARCH_URL/_inference/completion/llama-completion" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "service": "llama",
  "service_settings": {
    "url": "http://localhost:8321/v1/openai/v1/chat/completions",
    "model_id": "llama3.2:3b" 
  }
}'

Response

No example

Run PUT _inference/chat_completion/llama-chat-completion to create a Llama inference endpoint that performs a chat_completion task.

Request Console 1 / 2 languages

Console Kibana Dev Tools

PUT _inference/chat_completion/llama-chat-completion
{
  "service": "llama",
  "service_settings": {
    "url": "http://localhost:8321/v1/openai/v1/chat/completions",
    "model_id": "llama3.2:3b" 
  }
}

curl Shell

curl -X PUT "$ELASTICSEARCH_URL/_inference/chat_completion/llama-chat-completion" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "service": "llama",
  "service_settings": {
    "url": "http://localhost:8321/v1/openai/v1/chat/completions",
    "model_id": "llama3.2:3b" 
  }
}'

Response

No example

Required authorization

Path Parameters { task_type , llama_inference_id , }

Query Parameters

Request { chunking_settings , service , service_settings , }

Response

Path Parameters

Request