Create a Google Vertex AI inference endpoint
- PUT /_inference/{task_type}/{googlevertexai_inference_id}
Create an inference endpoint to perform an inference task with the googlevertexai service.
- Cluster privileges:
manage_inference
Path Parameters
-
task_typeGoogleVertexAITaskTyperequired -
The type of the inference task that the model will perform.
-
Values:
reranktext_embeddingcompletionchat_completion -
googlevertexai_inference_idstring Idrequired -
The unique identifier of the inference endpoint.
Query Parameters
-
timeoutstring Duration · default: 30s Specifies the amount of time to wait for the inference endpoint to be created.
Request
-
chunking_settingsInferenceChunkingSettings -
The chunking configuration object. Applies only to the
text_embeddingtask type. Not applicable to thererank,completion, orchat_completiontask types. - Reference
-
max_chunk_sizenumber · default: 250 -
The maximum size of a chunk in words. This value cannot be lower than
20(forsentencestrategy) or10(forwordstrategy). This value should not exceed the window size for the associated model. -
overlapnumber · default: 100 -
The number of overlapping words for chunks. It is applicable only to a
wordchunking strategy. This value cannot be higher than half themax_chunk_sizevalue. -
sentence_overlapnumber · default: 1 -
The number of overlapping sentences for chunks. It is applicable only for a
sentencechunking strategy. It can be either1or0. -
separator_groupstring -
Only applicable to the
recursivestrategy and required when using it.Sets a predefined list of separators in the saved chunking settings based on the selected text type. Values can be
markdownorplaintext.Using this parameter is an alternative to manually specifying a custom
separatorslist. -
separatorsstring -
Only applicable to the
recursivestrategy and required when using it.A list of strings used as possible split points when chunking text.
Each string can be a plain string or a regular expression (regex) pattern. The system tries each separator in order to split the text, starting from the first item in the list.
After splitting, it attempts to recombine smaller pieces into larger chunks that stay within the
max_chunk_sizelimit, to reduce the total number of chunks generated. -
serviceGoogleVertexAIServiceTyperequired -
The type of service supported for the specified task type. In this case,
googlevertexai. -
Values:
googlevertexai -
service_settingsGoogleVertexAIServiceSettingsrequired -
Settings used to install the inference model. These settings are specific to the
googlevertexaiservice. -
providerGoogleModelGardenProvider -
The name of the Google Model Garden Provider for
completionandchat_completiontasks. In order for a Google Model Garden endpoint to be usedprovidermust be defined and be other thangoogle. Modes:- Google Model Garden (third-party models): set
providerto a supported non-googlevalue and provideurland/orstreaming_url. - Google Vertex AI: omit
provideror set it togoogle. In this mode, do not seturlorstreaming_urland Elastic will construct the endpoint url fromlocation,model_id, andproject_idparameters.
- Google Model Garden (third-party models): set
-
Values:
googleanthropicmetahugging_facemistralai21 -
urlstring -
The URL for non-streaming
completionrequests to a Google Model Garden provider endpoint. If bothurlandstreaming_urlare provided, each is used for its respective mode. Ifstreaming_urlis not provided,urlis also used for streamingcompletionandchat_completion. Ifprovideris not provided or set togoogle(Google Vertex AI), do not seturl(orstreaming_url). At least one ofurlorstreaming_urlmust be provided for Google Model Garden endpoint usage. Certain providers require separate URLs for streaming and non-streaming operations (e.g., Anthropic, Mistral, AI21). Others support both operation types through a single URL (e.g., Meta, Hugging Face). Information on constructing the URL for various providers can be found in the Google Model Garden documentation for the model, or on the endpoint’sSample requestpage. The request examples also illustrate the proper formatting for theurl. -
streaming_urlstring -
The URL for streaming
completionandchat_completionrequests to a Google Model Garden provider endpoint. If bothstreaming_urlandurlare provided, each is used for its respective mode. Ifurlis not provided,streaming_urlis also used for non-streamingcompletionrequests. Ifprovideris not provided or set togoogle(Google Vertex AI), do not setstreaming_url(orurl). At least one ofstreaming_urlorurlmust be provided for Google Model Garden endpoint usage. Certain providers require separate URLs for streaming and non-streaming operations (e.g., Anthropic, Mistral, AI21). Others support both operation types through a single URL (e.g., Meta, Hugging Face). Information on constructing the URL for various providers can be found in the Google Model Garden documentation for the model, or on the endpoint’sSample requestpage. The request examples also illustrate the proper formatting for thestreaming_url. -
locationstring -
The name of the location to use for the inference task for the Google Vertex AI inference task. For Google Vertex AI, when
provideris omitted orgoogle,locationis optional. Whenlocationis omitted, the request targets the Google Vertex AI global endpoint (https://aiplatform.googleapis.com); when it is provided, the corresponding regional endpoint (https://<location>-aiplatform.googleapis.com) is used. The field must be omitted to select the global endpoint; an empty string is not a validlocation. For Google Model Garden'scompletionandchat_completiontasks, whenprovideris a supported non-googlevalue -locationis ignored. Refer to the Google documentation for the list of supported locations. - External docs
-
model_idstring -
The name of the model to use for the inference task. For Google Vertex AI
model_idis mandatory. For Google Model Garden'scompletionandchat_completiontasks, whenprovideris a supported non-googlevalue -model_idwill be used for some providers that require it, otherwise - ignored. Refer to the Google documentation for the list of supported models for Google Vertex AI. - External docs
-
project_idstring -
The name of the project to use for the Google Vertex AI inference task. For Google Vertex AI
project_idis mandatory. For Google Model Garden'scompletionandchat_completiontasks, whenprovideris a supported non-googlevalue -project_idis ignored. -
rate_limitRateLimitSetting -
This setting helps to minimize the number of rate limit errors returned from Google Vertex AI. By default, the
googlevertexaiservice sets the number of requests allowed per minute to 30.000. -
requests_per_minutenumber -
The number of requests allowed per minute. By default, the number of requests allowed per minute is set by each service as follows:
alibabacloud-ai-searchservice:1000amazonbedrockservice:240anthropicservice:50azureaistudioservice:240azureopenaiservice and task typetext_embedding:1440azureopenaiservice and task typescompletionorchat_completion:120cohereservice:10000contextualaiservice:1000elasticservice and task typechat_completion:240fireworksaiservice:6000googleaistudioservice:360googlevertexaiservice:30000hugging_faceservice:3000jinaaiservice:2000llamaservice:3000mistralservice:240openaiservice and task typetext_embedding:3000openaiservice and task typecompletion:500openshift_aiservice:3000voyageaiservice:2000watsonxaiservice:120
-
service_account_jsonstringrequired -
A valid service account in JSON format for the Google Vertex AI API.
-
dimensionsnumber -
For a
text_embeddingtask, the number of dimensions the resulting output embeddings should have. By default, the model's standard output dimension is used. Refer to the Google documentation for more information. - External docs
-
max_batch_sizenumber · default: 250 -
Only applicable for the
text_embeddingtask type. Controls the batch size of chunked inference requests sent to Google Vertex AI.Setting this parameter lower reduces the risk of exceeding token limits but may result in more API calls. Setting it higher increases throughput but may risk hitting token limits.
To estimate a safe
max_batch_sizevalue, you can use it together with themax_chunk_sizeparameter using the following formula:max_batch_size ≈ max_chunk_size × 1.3 × 512 ÷ 20000Where:
1.3is an approximate tokens-per-word ratio512is the maximum number of chunks that can be generated per document20000is the Google Vertex AI token limit per request
This estimate assumes the worst-case scenario with a document generating the maximum 512 chunks.
-
task_settingsGoogleVertexAITaskSettings -
Settings to configure the inference task. These settings are specific to the task type you specified.
-
auto_truncateboolean -
For a
text_embeddingtask, truncate inputs longer than the maximum token length automatically. -
top_nnumber -
For a
reranktask, the number of the top N documents that should be returned. -
thinking_configThinkingConfig -
For a
completionorchat_completiontask, allows configuration of the thinking features for the model. Refer to the Google documentation for the allowable configurations for each model type. - External docs
-
thinking_budgetnumber -
Indicates the desired thinking budget in tokens.
-
max_tokensnumber -
For
completionandchat_completiontasks, specifies themax_tokensvalue for requests sent to the Google Model Gardenanthropicprovider. Ifprovideris not set toanthropic, this field is ignored. Ifmax_tokensis specified - it must be a positive integer. If not specified, the default value of 1024 is used. Anthropic models requiremax_tokensto be set for each request. Please refer to the Anthropic documentation for more information. - External docs
Response
-
chunking_settingsInferenceChunkingSettings -
The chunking configuration object. Applies only to the
embedding,sparse_embeddingandtext_embeddingtask types. Not applicable to thererank,completion, orchat_completiontask types. -
max_chunk_sizenumber · default: 250 -
The maximum size of a chunk in words. This value cannot be lower than
20(forsentencestrategy) or10(forwordstrategy). This value should not exceed the window size for the associated model. -
overlapnumber · default: 100 -
The number of overlapping words for chunks. It is applicable only to a
wordchunking strategy. This value cannot be higher than half themax_chunk_sizevalue. -
sentence_overlapnumber · default: 1 -
The number of overlapping sentences for chunks. It is applicable only for a
sentencechunking strategy. It can be either1or0. -
separator_groupstring -
Only applicable to the
recursivestrategy and required when using it.Sets a predefined list of separators in the saved chunking settings based on the selected text type. Values can be
markdownorplaintext.Using this parameter is an alternative to manually specifying a custom
separatorslist. -
separatorsstring -
Only applicable to the
recursivestrategy and required when using it.A list of strings used as possible split points when chunking text.
Each string can be a plain string or a regular expression (regex) pattern. The system tries each separator in order to split the text, starting from the first item in the list.
After splitting, it attempts to recombine smaller pieces into larger chunks that stay within the
max_chunk_sizelimit, to reduce the total number of chunks generated. -
servicestringrequired -
The service type
-
service_settingsServiceSettingsrequired -
Settings specific to the service
-
task_settingsTaskSettings -
Task settings specific to the service and task type
-
inference_idstringrequired -
The inference Id
-
task_typeTaskTypeGoogleVertexAIrequired -
The task type
-
Values:
chat_completioncompletiontext_embeddingrerank