Create a custom inference endpoint
- PUT /_inference/{task_type}/{custom_inference_id}
The custom service gives more control over how to interact with external inference services that aren't explicitly supported through dedicated integrations.
The custom service gives you the ability to define the headers, url, query parameters, request body, and secrets.
The custom service supports the template replacement functionality, which enables you to define a template that can be replaced with the value associated with that key.
Templates are portions of a string that start with ${ and end with }.
The parameters secret_parameters and task_settings are checked for keys for template replacement. Template replacement is supported in the request, headers, url, and query_parameters.
If the definition (key) is not found for a template, an error message is returned.
In case of an endpoint definition like the following:
PUT _inference/text_embedding/test-text-embedding
{
"service": "custom",
"service_settings": {
"secret_parameters": {
"api_key": "<some api key>"
},
"url": "...endpoints.huggingface.cloud/v1/embeddings",
"headers": {
"Authorization": "Bearer ${api_key}",
"Content-Type": "application/json"
},
"request": "{\"input\": ${input}}",
"response": {
"json_parser": {
"text_embeddings":"$.data[*].embedding[*]"
}
}
}
}
To replace ${api_key} the secret_parameters and task_settings are checked for a key named api_key.
info Templates should not be surrounded by quotes.
Pre-defined templates:
${input}refers to the array of input strings that comes from theinputfield of the subsequent inference requests.${input_type}refers to the input type translation values.${query}refers to the query field used specifically for reranking tasks.${top_n}refers to thetop_nfield available when performing rerank requests.${return_documents}refers to thereturn_documentsfield available when performing rerank requests.
- Cluster privileges:
manage_inference
Path Parameters
-
task_typeCustomTaskTyperequired -
The type of the inference task that the model will perform.
-
Values:
text_embeddingsparse_embeddingrerankcompletion -
custom_inference_idstring Idrequired -
The unique identifier of the inference endpoint.
Request
-
chunking_settingsInferenceChunkingSettings -
The chunking configuration object.
- Reference
-
max_chunk_sizenumber · default: 250 -
The maximum size of a chunk in words. This value cannot be higher than
300or lower than20(forsentencestrategy) or10(forwordstrategy). -
overlapnumber · default: 100 -
The number of overlapping words for chunks. It is applicable only to a
wordchunking strategy. This value cannot be higher than half themax_chunk_sizevalue. -
sentence_overlapnumber · default: 1 -
The number of overlapping sentences for chunks. It is applicable only for a
sentencechunking strategy. It can be either1or0. -
strategystring · default: sentence -
The chunking strategy:
sentenceorword. -
serviceCustomServiceTyperequired -
The type of service supported for the specified task type. In this case,
custom. -
Values:
custom -
service_settingsCustomServiceSettingsrequired -
Settings used to install the inference model. These settings are specific to the
customservice. -
batch_sizenumber -
Specifies the batch size used for the semantic_text field. If the field is not provided, the default is 10. The batch size is the maximum number of inputs in a single request to the upstream service. The chunk within the batch are controlled by the selected chunking strategy for the semantic_text field.
- Reference
-
headersobject -
Specifies the HTTP header parameters – such as
AuthenticationorContent-Type– that are required to access the custom service. For example:"headers":{ "Authorization": "Bearer ${api_key}", "Content-Type": "application/json;charset=utf-8" } -
input_typeobject -
Specifies the input type translation values that are used to replace the
${input_type}template in the request body. For example:"input_type": { "translation": { "ingest": "do_ingest", "search": "do_search" }, "default": "a_default" },If the subsequent inference requests come from a search context, the
searchkey will be used and the template will be replaced withdo_search. If it comes from the ingest contextdo_ingestis used. If it's a different context that is not specified, the default value will be used. If no default is specified an empty string is used.translationcan be:classificationclusteringingestsearch
-
query_parametersobject -
Specifies the query parameters as a list of tuples. The arrays inside the
query_parametersmust have two items, a key and a value. For example:"query_parameters":[ ["param_key", "some_value"], ["param_key", "another_value"], ["other_key", "other_value"] ]If the base url is
https://www.elastic.coit results in:https://www.elastic.co?param_key=some_value¶m_key=another_value&other_key=other_value. -
requestCustomRequestParamsrequired -
The request configuration object.
-
contentstringrequired -
The body structure of the request. It requires passing in the string-escaped result of the JSON format HTTP request body. For example:
"request": "{\"input\":${input}}"info The content string needs to be a single line except when using the Kibana console.
-
responseCustomResponseParamsrequired -
The response configuration object.
-
json_parserobjectrequired -
Specifies the JSON parser that is used to parse the response from the custom service. Different task types require different json_parser parameters. For example:
# text_embedding # For a response like this: { "object": "list", "data": [ { "object": "embedding", "index": 0, "embedding": [ 0.014539449, -0.015288644 ] } ], "model": "text-embedding-ada-002-v2", "usage": { "prompt_tokens": 8, "total_tokens": 8 } } # the json_parser definition should look like this: "response":{ "json_parser":{ "text_embeddings":"$.data[*].embedding[*]" } } # sparse_embedding # For a response like this: { "request_id": "75C50B5B-E79E-4930-****-F48DBB392231", "latency": 22, "usage": { "token_count": 11 }, "result": { "sparse_embeddings": [ { "index": 0, "embedding": [ { "token_id": 6, "weight": 0.101 }, { "token_id": 163040, "weight": 0.28417 } ] } ] } } # the json_parser definition should look like this: "response":{ "json_parser":{ "token_path":"$.result.sparse_embeddings[*].embedding[*].token_id", "weight_path":"$.result.sparse_embeddings[*].embedding[*].weight" } } # rerank # For a response like this: { "results": [ { "index": 3, "relevance_score": 0.999071, "document": "abc" }, { "index": 4, "relevance_score": 0.7867867, "document": "123" }, { "index": 0, "relevance_score": 0.32713068, "document": "super" } ], } # the json_parser definition should look like this: "response":{ "json_parser":{ "reranked_index":"$.result.scores[*].index", "relevance_score":"$.result.scores[*].score", "document_text":"xxx" } } # completion # For a response like this: { "id": "chatcmpl-B9MBs8CjcvOU2jLn4n570S5qMJKcT", "object": "chat.completion", "created": 1741569952, "model": "gpt-4.1-2025-04-14", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Hello! How can I assist you today?", "refusal": null, "annotations": [] }, "logprobs": null, "finish_reason": "stop" } ] } # the json_parser definition should look like this: "response":{ "json_parser":{ "completion_result":"$.choices[*].message.content" } }- optional
- optional
-
secret_parametersobjectrequired -
Specifies secret parameters, like
api_keyorapi_token, that are required to access the custom service. For example:"secret_parameters":{ "api_key":"<api_key>" } -
urlstring -
The URL endpoint to use for the requests.
-
task_settingsCustomTaskSettings -
Settings to configure the inference task. These settings are specific to the task type you specified.
-
parametersobject -
Specifies parameters that are required to run the custom service. The parameters depend on the model your custom service uses. For example:
"task_settings":{ "parameters":{ "input_type":"query", "return_token":true } }warn The
task_settings.parameterscannot contain the same keys assecret_parameters. If they do, an error will be returned. This applies to PUT requests and POST requests.
Response
-
chunking_settingsInferenceChunkingSettings -
Chunking configuration object
-
max_chunk_sizenumber · default: 250 -
The maximum size of a chunk in words. This value cannot be higher than
300or lower than20(forsentencestrategy) or10(forwordstrategy). -
overlapnumber · default: 100 -
The number of overlapping words for chunks. It is applicable only to a
wordchunking strategy. This value cannot be higher than half themax_chunk_sizevalue. -
sentence_overlapnumber · default: 1 -
The number of overlapping sentences for chunks. It is applicable only for a
sentencechunking strategy. It can be either1or0. -
strategystring · default: sentence -
The chunking strategy:
sentenceorword. -
servicestringrequired -
The service type
-
service_settingsServiceSettingsrequired -
Settings specific to the service
-
task_settingsTaskSettings -
Task settings specific to the service and task type
-
inference_idstringrequired -
The inference Id
-
task_typeTaskTypeCustomrequired -
The task type
-
Values:
text_embeddingsparse_embeddingrerankcompletion