Perform streaming chat completion inference on the service

Auth

The chat completion inference API enables real-time responses for chat completion tasks by delivering answers incrementally, reducing response times during computation. It only works with the chat_completion task type.

Note

The chat_completion task type supports both streaming and non-streaming. The Chat completion inference API and the Stream inference API differ in their response structure and capabilities. The Chat completion inference API provides more comprehensive customization options through more fields and function calling support. To determine whether a given inference service supports this task type, please see the page for that service.

Path parameters

inference_id string required

The inference Id

Query parameters

timeout string · default: 120s

Specifies the amount of time to wait for the inference request to complete.

Request body

messages object[] required

A list of objects representing the conversation. Requests should generally only add new messages from the user (role user). The other message roles (assistant, system, or tool) should generally only be copied from the response to a previous completion request, such that the messages array is built up throughout a conversation.

content union oneOf

The content of the message.

String example:

{
   "content": "Some string"
}

Text example:

{
  "content": [
      {
       "text": "Some text",
       "type": "text"
      }
   ]
}

Image example:

{
  "content": [
      {
       "image_url": {
         "url": "data:image/jpeg;base64,..."
       },
       "type": "image_url"
      }
   ]
}

File example:

{
  "content": [
      {
       "file": {
         "file_data": "data:application/pdf;base64,...",
         "filename": "somePDF"
       },
       "type": "file"
      }
   ]
}
One of:
role string required

The role of the message author. Valid values are user, assistant, system, and tool.

tool_call_id string

Only for tool role messages. The tool call that this message is responding to.

tool_calls object[]

Only for assistant role messages. The tool calls generated by the model. If it's specified, the content field is optional. Example:

{
  "tool_calls": [
      {
          "id": "call_KcAjWtAww20AihPHphUh46Gd",
          "type": "function",
          "function": {
              "name": "get_current_weather",
              "arguments": "{\"location\":\"Boston, MA\"}"
          }
      }
  ]
}
id string required

The identifier of the tool call.

function object required

The function that the model called.

arguments string required

The arguments to call the function with in JSON format.

name string required

The name of the function to call.

type string required

The type of the tool call.

reasoning string

Only for assistant role messages. The reasoning details generated by the model as plaintext. Currently supported only for elastic provider.

reasoning_details union oneOf[]

Only for assistant role messages. The reasoning details generated by the model as structured data. Currently supported only for elastic provider.

One of:

The reasoning detail with encrypted reasoning data that may be redacted or protected.

type enum required
Values: reasoning.encrypted
format string

The format of the reasoning detail.

id string

The identifier of the reasoning detail.

index number

The index of the reasoning detail, which indicates its position in the sequence of reasoning details generated by the model.

data string required

The encrypted reasoning data generated by the model, which may be redacted or protected based on the model's configuration and the nature of the reasoning information.

model string

The ID of the model to use. By default, the model ID is set to the value included when creating the inference endpoint.

max_completion_tokens number

The upper bound limit for the number of tokens that can be generated for a completion request.

reasoning object

The reasoning configuration for the completion request. This controls the model's reasoning process in one of two ways:

  • By specifying the model’s reasoning effort level with the effort field.
  • By enabling reasoning with default settings by setting enabled field to true.

It also includes optional settings to control:

  • The level of detail in the summary returned in the response with the summary field.
  • Whether reasoning details are included in the response at all with the exclude field.

Example (effort):

{
   "reasoning": {
       "effort": "high",
       "summary": "concise",
       "exclude": false
   }
}

Example (enabled):

{
   "reasoning": {
       "enabled": true,
       "summary": "concise",
       "exclude": false
   }
}

Currently supported only for elastic provider.

effort enum

The level of effort the model should put into reasoning. This is a hint that guides the model in how much effort to put into reasoning, with xhigh being the most effort and none being no effort.

Values: xhigh high medium low minimal none
enabled boolean

Whether to enable reasoning with default settings. This is a shortcut for enabling reasoning without having to specify the other parameters. If enabled is set to true, then reasoning at the medium effort level is enabled. Ignored if effort is specified, in which case that parameter will control the reasoning process instead.

exclude boolean

Whether to exclude reasoning information from the response. If true, the response will not include any reasoning details.

summary enum

The level of detail included in the reasoning summary returned in the response. This is a hint on how much detail to include in the summary of the reasoning that is returned in the response, with auto being the default level of detail, concise being less detail, and detailed being more detail.

Values: auto concise detailed
stop string[]

A sequence of strings to control when the model should stop generating additional tokens.

temperature number

The sampling temperature to use.

tool_choice union oneOf

Controls which tool is called by the model. String representation: One of auto, none, or requrired. auto allows the model to choose between calling tools and generating a message. none causes the model to not call any tools. required forces the model to call one or more tools. Example (object representation):

{
  "tool_choice": {
      "type": "function",
      "function": {
          "name": "get_current_weather"
      }
  }
}
One of:
tools object[]

A list of tools that the model can call. Example:

{
  "tools": [
      {
          "type": "function",
          "function": {
              "name": "get_price_of_item",
              "description": "Get the current price of an item",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "item": {
                          "id": "12345"
                      },
                      "unit": {
                          "type": "currency"
                      }
                  }
              }
          }
      }
  ]
}
type string required

The type of tool.

function object required

The function definition.

description string

A description of what the function does. This is used by the model to choose when and how to call the function.

name string required

The name of the function.

parameters object

The parameters the functional accepts. This should be formatted as a JSON object.

strict boolean

Whether to enable schema adherence when generating the function call.

top_p number

Nucleus sampling, an alternative to sampling with temperature.

Response

application/json

Response Type: object