Perform streaming chat completion inference on the service

POST /_inference/chat_completion/{inference_id}/_stream

The chat completion inference API enables real-time responses for chat completion tasks by delivering answers incrementally, reducing response times during computation. It only works with the chat_completion task type.

NOTE: The chat_completion task type supports both streaming and non-streaming. The Chat completion inference API and the Stream inference API differ in their response structure and capabilities. The Chat completion inference API provides more comprehensive customization options through more fields and function calling support. To determine whether a given inference service supports this task type, please see the page for that service.

Path Parameters

inference_id string Id required: The inference Id

Query Parameters

timeout string Duration · default: 120s: Specifies the amount of time to wait for the inference request to complete.

Request

messages [] Message required

A list of objects representing the conversation. Requests should generally only add new messages from the user (role user). The other message roles (assistant, system, or tool) should generally only be copied from the response to a previous completion request, such that the messages array is built up throughout a conversation.

content MessageContent

The content of the message.

String example:

		{
   "content": "Some string"
}
		
	

Text example:

		{
  "content": [
      {
       "text": "Some text",
       "type": "text"
      }
   ]
}
		
	

Image example:

		{
  "content": [
      {
       "image_url": {
         "url": "data:image/jpeg;base64,..."
       },
       "type": "image_url"
      }
   ]
}
		
	

File example:

		{
  "content": [
      {
       "file": {
         "file_data": "data:application/pdf;base64,...",
         "filename": "somePDF"
       },
       "type": "file"
      }
   ]
}
		
	

role string required

The role of the message author. Valid values are user, assistant, system, and tool.

tool_call_id string Id

Only for tool role messages. The tool call that this message is responding to.

tool_calls [] ToolCall

Only for assistant role messages. The tool calls generated by the model. If it's specified, the content field is optional. Example:

		{
  "tool_calls": [
      {
          "id": "call_KcAjWtAww20AihPHphUh46Gd",
          "type": "function",
          "function": {
              "name": "get_current_weather",
              "arguments": "{\"location\":\"Boston, MA\"}"
          }
      }
  ]
}
		
	

reasoning string

Only for assistant role messages. The reasoning details generated by the model as plaintext. Currently supported only for elastic provider.

reasoning_details [] union ReasoningDetail

Only for assistant role messages. The reasoning details generated by the model as structured data. Currently supported only for elastic provider.

One of:

The reasoning detail with encrypted reasoning data that may be redacted or protected.

format string: The format of the reasoning detail.
id string: The identifier of the reasoning detail.
index number: The index of the reasoning detail, which indicates its position in the sequence of reasoning details generated by the model.
type enum required: Values: reasoning.encrypted
data string required: The encrypted reasoning data generated by the model, which may be redacted or protected based on the model's configuration and the nature of the reasoning information.

model string

The ID of the model to use. By default, the model ID is set to the value included when creating the inference endpoint.

max_completion_tokens number

The upper bound limit for the number of tokens that can be generated for a completion request.

reasoning {} Reasoning

The reasoning configuration for the completion request. This controls the model's reasoning process in one of two ways:

By specifying the model’s reasoning effort level with the effort field.
By enabling reasoning with default settings by setting enabled field to true.

It also includes optional settings to control:

The level of detail in the summary returned in the response with the summary field.
Whether reasoning details are included in the response at all with the exclude field.

Example (effort):

		{
   "reasoning": {
       "effort": "high",
       "summary": "concise",
       "exclude": false
   }
}
		
	

Example (enabled):

		{
   "reasoning": {
       "enabled": true,
       "summary": "concise",
       "exclude": false
   }
}
		
	

Currently supported only for elastic provider.

stop [] string

A sequence of strings to control when the model should stop generating additional tokens.

temperature number

The sampling temperature to use.

tool_choice CompletionToolType

Controls which tool is called by the model. String representation: One of auto, none, or requrired. auto allows the model to choose between calling tools and generating a message. none causes the model to not call any tools. required forces the model to call one or more tools. Example (object representation):

		{
  "tool_choice": {
      "type": "function",
      "function": {
          "name": "get_current_weather"
      }
  }
}
		
	

tools [] CompletionTool

A list of tools that the model can call. Example:

		{
  "tools": [
      {
          "type": "function",
          "function": {
              "name": "get_price_of_item",
              "description": "Get the current price of an item",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "item": {
                          "id": "12345"
                      },
                      "unit": {
                          "type": "currency"
                      }
                  }
              }
          }
      }
  ]
}
		
	

top_p number

Nucleus sampling, an alternative to sampling with temperature.

Response

application/json

Examples

Run POST _inference/chat_completion/openai-completion/_stream to perform a chat completion on the example question with streaming.

Request Console 1 / 8 languages

Console Kibana Dev Tools

POST _inference/chat_completion/openai-completion/_stream
{
  "model": "gpt-4o",
  "messages": [
      {
          "role": "user",
          "content": "What is Elastic?"
      }
  ]
}

JavaScript Elasticsearch JavaScript Client

const response = await client.inference.chatCompletionUnified({
  inference_id: "openai-completion",
  chat_completion_request: {
    model: "gpt-4o",
    messages: [
      {
        role: "user",
        content: "What is Elastic?",
      },
    ],
  },
});

Python Elasticsearch Python Client

resp = client.inference.chat_completion_unified(
    inference_id="openai-completion",
    chat_completion_request={
        "model": "gpt-4o",
        "messages": [
            {
                "role": "user",
                "content": "What is Elastic?"
            }
        ]
    },
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_inference/chat_completion/openai-completion/_stream" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "What is Elastic?"
      }
    ]
  }'

Java Elasticsearch Java Client

client.inference().chatCompletionUnified(c -> c
    .inferenceId("openai-completion")
    .chatCompletionRequest(ch -> ch
        .messages(m -> m
            .content(co -> co
                .string("What is Elastic?")
            )
            .role("user")
        )
        .model("gpt-4o")
    )
);

C# Elasticsearch .NET Client

var response = await client.Inference
    .ChatCompletionUnifiedAsync(inferenceId: "openai-completion", d1 => d1
        .ChatCompletionRequest(d2 => d2
            .Messages(d3 => d3
                .Content("What is Elastic?")
                .Role("user")
            )
            .Model("gpt-4o")
        )
    );

PHP Elasticsearch PHP Client

$resp = $client->inference()->chatCompletionUnified([
    "inference_id" => "openai-completion",
    "body" => [
        "model" => "gpt-4o",
        "messages" => array(
            [
                "role" => "user",
                "content" => "What is Elastic?",
            ],
        ),
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.inference.chat_completion_unified(
  inference_id: "openai-completion",
  body: {
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "What is Elastic?"
      }
    ]
  }
)

Response

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":"","role":"assistant"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":Elastic"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":" is"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

(...)

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","usage":{"completion_tokens":28,"prompt_tokens":16,"total_tokens":44}}} 

event: message
data: [DONE]

Run POST _inference/chat_completion/openai-completion/_stream to perform a chat completion using an Assistant message with tool_calls.

Request Console 1 / 2 languages

Console Kibana Dev Tools

POST _inference/chat_completion/openai-completion/_stream
{
  "messages": [
      {
          "role": "assistant",
          "content": "Let's find out what the weather is",
          "tool_calls": [ 
              {
                  "id": "call_KcAjWtAww20AihPHphUh46Gd",
                  "type": "function",
                  "function": {
                      "name": "get_current_weather",
                      "arguments": "{\"location\":\"Boston, MA\"}"
                  }
              }
          ]
      },
      { 
          "role": "tool",
          "content": "The weather is cold",
          "tool_call_id": "call_KcAjWtAww20AihPHphUh46Gd"
      }
  ]
}

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_inference/chat_completion/openai-completion/_stream" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "messages": [
      {
          "role": "assistant",
          "content": "Let'\''s find out what the weather is",
          "tool_calls": [ 
              {
                  "id": "call_KcAjWtAww20AihPHphUh46Gd",
                  "type": "function",
                  "function": {
                      "name": "get_current_weather",
                      "arguments": "{\"location\":\"Boston, MA\"}"
                  }
              }
          ]
      },
      { 
          "role": "tool",
          "content": "The weather is cold",
          "tool_call_id": "call_KcAjWtAww20AihPHphUh46Gd"
      }
  ]
}'

Response

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":"","role":"assistant"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":Elastic"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":" is"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

(...)

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","usage":{"completion_tokens":28,"prompt_tokens":16,"total_tokens":44}}} 

event: message
data: [DONE]

Run POST _inference/chat_completion/openai-completion/_stream to perform a chat completion using a User message with tools and tool_choice.

Request Console 1 / 2 languages

Console Kibana Dev Tools

POST _inference/chat_completion/openai-completion/_stream
{
  "messages": [
      {
          "role": "user",
          "content": [
              {
                  "type": "text",
                  "text": "What's the price of a scarf?"
              }
          ]
      }
  ],
  "tools": [
      {
          "type": "function",
          "function": {
              "name": "get_current_price",
              "description": "Get the current price of a item",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "item": {
                          "id": "123"
                      }
                  }
              }
          }
      }
  ],
  "tool_choice": {
      "type": "function",
      "function": {
          "name": "get_current_price"
      }
  }
}

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_inference/chat_completion/openai-completion/_stream" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "messages": [
      {
          "role": "user",
          "content": [
              {
                  "type": "text",
                  "text": "What'\''s the price of a scarf?"
              }
          ]
      }
  ],
  "tools": [
      {
          "type": "function",
          "function": {
              "name": "get_current_price",
              "description": "Get the current price of a item",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "item": {
                          "id": "123"
                      }
                  }
              }
          }
      }
  ],
  "tool_choice": {
      "type": "function",
      "function": {
          "name": "get_current_price"
      }
  }
}'

Response

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":"","role":"assistant"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":Elastic"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":" is"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

(...)

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","usage":{"completion_tokens":28,"prompt_tokens":16,"total_tokens":44}}} 

event: message
data: [DONE]

Run POST _inference/chat_completion/reasoning-chat-completion/_stream to perform a chat completion task, using both effort parameter based reasoning configuration and including reasoning generated by the model on previous step.

Request Console 1 / 2 languages

Console Kibana Dev Tools

POST _inference/chat_completion/reasoning-chat-completion/_stream
{
  "messages": [{
      "role": "user",
      "content": [{
          "type": "text",
          "text": "Barber shaves all those, who do not shave themselves. Who shaves the barber?"
        }
      ]
    }, {
      "role": "assistant",
      "content": [{
          "type": "text",
          "text": "This is the barber paradox. Such a barber cannot logically exist."
        }
      ],
      "reasoning": "If the barber shaves himself, he should not; if he does not, he should.",
      "reasoning_details": [{
          "type": "reasoning.encrypted",
          "data": "[REDACTED]"
        }, {
          "type": "reasoning.summary",
          "summary": "Barber shaving himself creates contradiction"
        }, {
          "type": "reasoning.text",
          "text": "If the barber shaves himself, he should not; if he does not, he should.",
          "signature": "sig_123"
        }
      ]
    }, {
      "role": "user",
      "content": [{
          "type": "text",
          "text": "What if there are 2 barbers?"
        }
      ]
    }
  ],
  "reasoning": {
    "effort": "high",
    "summary": "detailed",
    "exclude": false
  }
}

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_inference/chat_completion/reasoning-chat-completion/_stream" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "messages": [{
      "role": "user",
      "content": [{
          "type": "text",
          "text": "Barber shaves all those, who do not shave themselves. Who shaves the barber?"
        }
      ]
    }, {
      "role": "assistant",
      "content": [{
          "type": "text",
          "text": "This is the barber paradox. Such a barber cannot logically exist."
        }
      ],
      "reasoning": "If the barber shaves himself, he should not; if he does not, he should.",
      "reasoning_details": [{
          "type": "reasoning.encrypted",
          "data": "[REDACTED]"
        }, {
          "type": "reasoning.summary",
          "summary": "Barber shaving himself creates contradiction"
        }, {
          "type": "reasoning.text",
          "text": "If the barber shaves himself, he should not; if he does not, he should.",
          "signature": "sig_123"
        }
      ]
    }, {
      "role": "user",
      "content": [{
          "type": "text",
          "text": "What if there are 2 barbers?"
        }
      ]
    }
  ],
  "reasoning": {
    "effort": "high",
    "summary": "detailed",
    "exclude": false
  }
}'

Response

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":"","role":"assistant"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":Elastic"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":" is"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

(...)

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","usage":{"completion_tokens":28,"prompt_tokens":16,"total_tokens":44}}} 

event: message
data: [DONE]

Run POST _inference/chat_completion/reasoning-chat-completion/_stream to perform a chat completion task, using both enabled parameter based reasoning configuration and including reasoning generated by the model on previous step.

Request Console 1 / 2 languages

Console Kibana Dev Tools

POST _inference/chat_completion/reasoning-chat-completion/_stream
{
  "messages": [{
      "role": "user",
      "content": [{
          "type": "text",
          "text": "Barber shaves all those, who do not shave themselves. Who shaves the barber?"
        }
      ]
    }, {
      "role": "assistant",
      "content": [{
          "type": "text",
          "text": "This is the barber paradox. Such a barber cannot logically exist."
        }
      ],
      "reasoning": "If the barber shaves himself, he should not; if he does not, he should.",
      "reasoning_details": [{
          "type": "reasoning.encrypted",
          "data": "[REDACTED]"
        }, {
          "type": "reasoning.summary",
          "summary": "Barber shaving himself creates contradiction"
        }, {
          "type": "reasoning.text",
          "text": "If the barber shaves himself, he should not; if he does not, he should.",
          "signature": "sig_123"
        }
      ]
    }, {
      "role": "user",
      "content": [{
          "type": "text",
          "text": "What if there are 2 barbers?"
        }
      ]
    }
  ],
  "reasoning": {
    "enabled": true,
    "summary": "detailed",
    "exclude": false
  }
}

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_inference/chat_completion/reasoning-chat-completion/_stream" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "messages": [{
      "role": "user",
      "content": [{
          "type": "text",
          "text": "Barber shaves all those, who do not shave themselves. Who shaves the barber?"
        }
      ]
    }, {
      "role": "assistant",
      "content": [{
          "type": "text",
          "text": "This is the barber paradox. Such a barber cannot logically exist."
        }
      ],
      "reasoning": "If the barber shaves himself, he should not; if he does not, he should.",
      "reasoning_details": [{
          "type": "reasoning.encrypted",
          "data": "[REDACTED]"
        }, {
          "type": "reasoning.summary",
          "summary": "Barber shaving himself creates contradiction"
        }, {
          "type": "reasoning.text",
          "text": "If the barber shaves himself, he should not; if he does not, he should.",
          "signature": "sig_123"
        }
      ]
    }, {
      "role": "user",
      "content": [{
          "type": "text",
          "text": "What if there are 2 barbers?"
        }
      ]
    }
  ],
  "reasoning": {
    "enabled": true,
    "summary": "detailed",
    "exclude": false
  }
}'

Response

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":"","role":"assistant"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":Elastic"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":" is"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}

(...)

event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","usage":{"completion_tokens":28,"prompt_tokens":16,"total_tokens":44}}} 

event: message
data: [DONE]

Path Parameters

Query Parameters

Request { messages , model , max_completion_tokens , reasoning , stop , temperature , tool_choice , tools , top_p , }

Response

Request