---
type: api
title: Create a Google Vertex AI inference endpoint
description: Create an inference endpoint to perform an inference task with the `googlevertexai` service.
url: https://www.elastic.co/elastic/docs-builder/docs/4371/api/doc/elasticsearch/v9/operation/operation-inference-put-googlevertexai
resource: https://www.elastic.co/elastic/docs-builder/docs/4371/api/doc/elasticsearch/v9/operation/operation-inference-put-googlevertexai
products:
  - Elasticsearch
---

# Create a Google Vertex AI inference endpoint

Availability: 8.15+

## Paths

- `PUT` `/_inference/{task_type}/{googlevertexai_inference_id}`

- [Api key auth](/elastic/docs-builder/docs/4371/api/doc/elasticsearch/v9/authentication#apikeyauth)
- [Basic auth](/elastic/docs-builder/docs/4371/api/doc/elasticsearch/v9/authentication#basicauth)
- [Bearer auth](/elastic/docs-builder/docs/4371/api/doc/elasticsearch/v9/authentication#bearerauth)

#### Path Parameters

- `task_type` (enum GoogleVertexAITaskType)
  The type of the inference task that the model will perform.
  Values: `rerank`, `text_embedding`, `completion`, `chat_completion`
- `googlevertexai_inference_id` (string Id)
  The unique identifier of the inference endpoint.

## Description

Create an inference endpoint to perform an inference task with the `googlevertexai` service.

## Required authorization

* Cluster privileges: `manage_inference`

## Query String Parameters

- `timeout` (string Duration · default: 30s)
  Specifies the amount of time to wait for the inference endpoint to be created.

## Request Body

`application/json`

- `chunking_settings` ({} InferenceChunkingSettings) — optional
  The chunking configuration object.
Applies only to the `text_embedding` task type.
Not applicable to the `rerank`, `completion`, or `chat_completion` task types.
  - `max_chunk_size` (number · default: 250) — optional
    The maximum size of a chunk in words.
This value cannot be lower than `20` (for `sentence` strategy) or `10` (for `word` strategy).
This value should not exceed the window size for the associated model.
  - `overlap` (number · default: 100) — optional
    The number of overlapping words for chunks.
It is applicable only to a `word` chunking strategy.
This value cannot be higher than half the `max_chunk_size` value.
  - `sentence_overlap` (number · default: 1) — optional
    The number of overlapping sentences for chunks.
It is applicable only for a `sentence` chunking strategy.
It can be either `1` or `0`.
  - `separator_group` (string) — optional
    Only applicable to the `recursive` strategy and required when using it.

Sets a predefined list of separators in the saved chunking settings based on the selected text type.
Values can be `markdown` or `plaintext`.

Using this parameter is an alternative to manually specifying a custom `separators` list.
  - `separators` ([] string) — optional
    Only applicable to the `recursive` strategy and required when using it.

A list of strings used as possible split points when chunking text.

Each string can be a plain string or a regular expression (regex) pattern.
The system tries each separator in order to split the text, starting from the first item in the list.

After splitting, it attempts to recombine smaller pieces into larger chunks that stay within
the `max_chunk_size` limit, to reduce the total number of chunks generated.
  - `strategy` (string · default: sentence) — optional
    The chunking strategy: `sentence`, `word`, `none` or `recursive`.

 * If `strategy` is set to `recursive`, you must also specify:

- `max_chunk_size`
- either `separators` or`separator_group`

Learn more about different chunking strategies in the linked documentation.
- `service` (enum GoogleVertexAIServiceType) — required
  The type of service supported for the specified task type. In this case, `googlevertexai`.
  Values: `googlevertexai`
- `service_settings` ({} GoogleVertexAIServiceSettings) — required
  Settings used to install the inference model. These settings are specific to the `googlevertexai` service.
  - `provider` (enum GoogleModelGardenProvider) — optional
    The name of the Google Model Garden Provider for `completion` and `chat_completion` tasks.
In order for a Google Model Garden endpoint to be used `provider` must be defined and be other than `google`.
Modes:
- Google Model Garden (third-party models): set `provider` to a supported non-`google` value and provide `url` and/or `streaming_url`.
- Google Vertex AI: omit `provider` or set it to `google`. In this mode, do not set `url` or `streaming_url` and Elastic will construct the endpoint url from `location`, `model_id`, and `project_id` parameters.
    Values: `google`, `anthropic`, `meta`, `hugging_face`, `mistral`, `ai21`
  - `url` (string) — optional
    The URL for non-streaming `completion` requests to a Google Model Garden provider endpoint.
If both `url` and `streaming_url` are provided, each is used for its respective mode.
If `streaming_url` is not provided, `url` is also used for streaming `completion` and `chat_completion`.
If `provider` is not provided or set to `google` (Google Vertex AI), do not set `url` (or `streaming_url`).
At least one of `url` or `streaming_url` must be provided for Google Model Garden endpoint usage.
Certain providers require separate URLs for streaming and non-streaming operations (e.g., Anthropic, Mistral, AI21). Others support both operation types through a single URL (e.g., Meta, Hugging Face).
Information on constructing the URL for various providers can be found in the Google Model Garden documentation for the model, or on the endpoint’s `Sample request` page. The request examples also illustrate the proper formatting for the `url`.
  - `streaming_url` (string) — optional
    The URL for streaming `completion` and `chat_completion` requests to a Google Model Garden provider endpoint.
If both `streaming_url` and `url` are provided, each is used for its respective mode.
If `url` is not provided, `streaming_url` is also used for non-streaming `completion` requests.
If `provider` is not provided or set to `google` (Google Vertex AI), do not set `streaming_url` (or `url`).
At least one of `streaming_url` or `url` must be provided for Google Model Garden endpoint usage.
Certain providers require separate URLs for streaming and non-streaming operations (e.g., Anthropic, Mistral, AI21). Others support both operation types through a single URL (e.g., Meta, Hugging Face).
Information on constructing the URL for various providers can be found in the Google Model Garden documentation for the model, or on the endpoint’s `Sample request` page. The request examples also illustrate the proper formatting for the `streaming_url`.
  - `location` (string) — optional
    The name of the location to use for the inference task for the Google Vertex AI inference task.
For Google Vertex AI, when `provider` is omitted or `google`, `location` is optional.
When `location` is omitted, the request targets the Google Vertex AI global endpoint (`https://aiplatform.googleapis.com`); when it is provided, the corresponding regional endpoint (`https://<location>-aiplatform.googleapis.com`) is used.
The field must be omitted to select the global endpoint; an empty string is not a valid `location`.
For Google Model Garden's `completion` and `chat_completion` tasks, when `provider` is a supported non-`google` value - `location` is ignored.
Refer to the Google documentation for the list of supported locations.
  - `model_id` (string) — optional
    The name of the model to use for the inference task.
For Google Vertex AI `model_id` is mandatory.
For Google Model Garden's `completion` and `chat_completion` tasks, when `provider` is a supported non-`google` value - `model_id` will be used for some providers that require it, otherwise - ignored.
Refer to the Google documentation for the list of supported models for Google Vertex AI.
  - `project_id` (string) — optional
    The name of the project to use for the Google Vertex AI inference task.
For Google Vertex AI `project_id` is mandatory.
For Google Model Garden's `completion` and `chat_completion` tasks, when `provider` is a supported non-`google` value - `project_id` is ignored.
  - `rate_limit` ({} RateLimitSetting) — optional
    This setting helps to minimize the number of rate limit errors returned from Google Vertex AI.
By default, the `googlevertexai` service sets the number of requests allowed per minute to 30.000.
    - `requests_per_minute` (number) — optional
      The number of requests allowed per minute.
By default, the number of requests allowed per minute is set by each service as follows:

* `alibabacloud-ai-search` service: `1000`
* `amazonbedrock` service: `240`
* `anthropic` service: `50`
* `azureaistudio` service: `240`
* `azureopenai` service and task type `text_embedding`: `1440`
* `azureopenai` service and task types `completion` or `chat_completion`: `120`
* `cohere` service: `10000`
* `contextualai` service: `1000`
* `elastic` service and task type `chat_completion`: `240`
* `fireworksai` service: `6000`
* `googleaistudio` service: `360`
* `googlevertexai` service: `30000`
* `hugging_face` service: `3000`
* `jinaai` service: `2000`
* `llama` service: `3000`
* `mistral` service: `240`
* `openai` service and task type `text_embedding`: `3000`
* `openai` service and task type `completion`: `500`
* `openshift_ai` service: `3000`
* `voyageai` service: `2000`
* `watsonxai` service: `120`
  - `service_account_json` (string) — required
    A valid service account in JSON format for the Google Vertex AI API.
  - `dimensions` (number) — optional
    For a `text_embedding` task, the number of dimensions the resulting output embeddings should have.
By default, the model's standard output dimension is used.
Refer to the Google documentation for more information.
  - `max_batch_size` (number · default: 250) — optional
    Only applicable for the `text_embedding` task type.
Controls the batch size of chunked inference requests sent to Google Vertex AI.

Setting this parameter lower reduces the risk of exceeding token limits but may result in more API calls. Setting it higher increases throughput but may risk hitting token limits.

To estimate a safe `max_batch_size` value, you can use it together with the `max_chunk_size` parameter using the following formula:
`max_batch_size ≈ max_chunk_size × 1.3 × 512 ÷ 20000`

Where:
- `1.3` is an approximate tokens-per-word ratio
- `512` is the maximum number of chunks that can be generated per document
- `20000` is the Google Vertex AI token limit per request

This estimate assumes the worst-case scenario with a document generating the maximum 512 chunks.
- `task_settings` ({} GoogleVertexAITaskSettings) — optional
  Settings to configure the inference task.
These settings are specific to the task type you specified.
  - `auto_truncate` (boolean) — optional
    For a `text_embedding` task, truncate inputs longer than the maximum token length automatically.
  - `top_n` (number) — optional
    For a `rerank` task, the number of the top N documents that should be returned.
  - `thinking_config` ({} ThinkingConfig) — optional
    For a `completion` or `chat_completion` task, allows configuration of the thinking features for the model.
Refer to the Google documentation for the allowable configurations for each model type.
    - `thinking_budget` (number) — optional
      Indicates the desired thinking budget in tokens.
  - `max_tokens` (number) — optional
    For `completion` and `chat_completion` tasks, specifies the `max_tokens` value for requests sent to the Google Model Garden `anthropic` provider.
If `provider` is not set to `anthropic`, this field is ignored.
If `max_tokens` is specified - it must be a positive integer. If not specified, the default value of 1024 is used.
Anthropic models require `max_tokens` to be set for each request. Please refer to the Anthropic documentation for more information.

## Response

`application/json`

- `chunking_settings` ({} InferenceChunkingSettings)
  The chunking configuration object.
Applies only to the `embedding`, `sparse_embedding` and `text_embedding` task types.
Not applicable to the `rerank`, `completion`, or `chat_completion` task types.
  - `max_chunk_size` (number · default: 250)
    The maximum size of a chunk in words.
This value cannot be lower than `20` (for `sentence` strategy) or `10` (for `word` strategy).
This value should not exceed the window size for the associated model.
  - `overlap` (number · default: 100)
    The number of overlapping words for chunks.
It is applicable only to a `word` chunking strategy.
This value cannot be higher than half the `max_chunk_size` value.
  - `sentence_overlap` (number · default: 1)
    The number of overlapping sentences for chunks.
It is applicable only for a `sentence` chunking strategy.
It can be either `1` or `0`.
  - `separator_group` (string)
    Only applicable to the `recursive` strategy and required when using it.

Sets a predefined list of separators in the saved chunking settings based on the selected text type.
Values can be `markdown` or `plaintext`.

Using this parameter is an alternative to manually specifying a custom `separators` list.
  - `separators` ([] string)
    Only applicable to the `recursive` strategy and required when using it.

A list of strings used as possible split points when chunking text.

Each string can be a plain string or a regular expression (regex) pattern.
The system tries each separator in order to split the text, starting from the first item in the list.

After splitting, it attempts to recombine smaller pieces into larger chunks that stay within
the `max_chunk_size` limit, to reduce the total number of chunks generated.
  - `strategy` (string · default: sentence)
    The chunking strategy: `sentence`, `word`, `none` or `recursive`.

 * If `strategy` is set to `recursive`, you must also specify:

- `max_chunk_size`
- either `separators` or`separator_group`

Learn more about different chunking strategies in the linked documentation.
- `service` (string)
  The service type
- `service_settings` (ServiceSettings)
  Settings specific to the service
- `task_settings` (TaskSettings)
  Task settings specific to the service and task type
- `inference_id` (string)
  The inference Id
- `task_type` (enum TaskTypeGoogleVertexAI)
  The task type
  Values: `chat_completion`, `completion`, `text_embedding`, `rerank`
## Code Examples

#### Console

```console
PUT _inference/text_embedding/google_vertex_ai_embeddingss
{
    "service": "googlevertexai",
    "service_settings": {
        "service_account_json": "service-account-json",
        "model_id": "model-id",
        "location": "location",
        "project_id": "project-id"
    }
}
```

#### JavaScript

```javascript
const response = await client.inference.put({
  task_type: "text_embedding",
  inference_id: "google_vertex_ai_embeddingss",
  inference_config: {
    service: "googlevertexai",
    service_settings: {
      service_account_json: "service-account-json",
      model_id: "model-id",
      location: "location",
      project_id: "project-id",
    },
  },
});
```

#### Python

```python
resp = client.inference.put(
    task_type="text_embedding",
    inference_id="google_vertex_ai_embeddingss",
    inference_config={
        "service": "googlevertexai",
        "service_settings": {
            "service_account_json": "service-account-json",
            "model_id": "model-id",
            "location": "location",
            "project_id": "project-id"
        }
    },
)
```

#### curl

```curl
curl -X PUT "$ELASTICSEARCH_URL/_inference/text_embedding/google_vertex_ai_embeddingss" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "service": "googlevertexai",
    "service_settings": {
      "service_account_json": "service-account-json",
      "model_id": "model-id",
      "location": "location",
      "project_id": "project-id"
    }
  }'
```

#### Java

```java
client.inference().put(p -> p
    .inferenceId("google_vertex_ai_embeddingss")
    .taskType(TaskType.TextEmbedding)
    .inferenceConfig(i -> i
        .service("googlevertexai")
        .serviceSettings(JsonData.fromJson("""
{"service_account_json":"service-account-json","model_id":"model-id","location":"location","project_id":"project-id"}
"""))
    )
);
```

#### C#

```c#
var response = await client.Inference
    .PutAsync(new PutInferenceRequest()
    {
        InferenceId = "google_vertex_ai_embeddingss",
        TaskType = TaskType.TextEmbedding,
        InferenceConfig = new()
        {
            Service = "googlevertexai",
            ServiceSettings = JsonSerializer.Deserialize<JsonElement>("""
            {
              "service_account_json": "service-account-json",
              "model_id": "model-id",
              "location": "location",
              "project_id": "project-id"
            }
            """)
        }
    });
```

#### PHP

```php
$resp = $client->inference()->put([
    "task_type" => "text_embedding",
    "inference_id" => "google_vertex_ai_embeddingss",
    "body" => [
        "service" => "googlevertexai",
        "service_settings" => [
            "service_account_json" => "service-account-json",
            "model_id" => "model-id",
            "location" => "location",
            "project_id" => "project-id",
        ],
    ],
]);
```

#### Ruby

```ruby
response = client.inference.put(
  task_type: "text_embedding",
  inference_id: "google_vertex_ai_embeddingss",
  body: {
    "service": "googlevertexai",
    "service_settings": {
      "service_account_json": "service-account-json",
      "model_id": "model-id",
      "location": "location",
      "project_id": "project-id"
    }
  }
)
```