> ## Documentation Index
> Fetch the complete documentation index at: https://sgl-project-mini-sglang.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Chat Completions

> Generate text completions using the /v1/chat/completions endpoint

## Endpoint

```http theme={null}
POST /v1/chat/completions
```

The chat completions endpoint supports both chat-style messages and raw text prompts, compatible with the OpenAI API format.

## Request Body

**model** `string` *required*

The model ID to use for generation. You can retrieve available models from the `/v1/models` endpoint.

**messages** `array`

An array of message objects for chat-style completions. Each message has:

* `role` (string): One of "system", "user", or "assistant"
* `content` (string): The message content

Either `messages` or `prompt` must be provided.

**prompt** `string`

A raw text prompt for completion. Either `messages` or `prompt` must be provided.

**max\_tokens** `integer` (default: 16)

The maximum number of tokens to generate.

**temperature** `float` (default: 1.0)

Sampling temperature. Use 0.0 for greedy decoding.

**top\_k** `integer` (default: -1)

Top-k sampling parameter. -1 disables top-k filtering.

**top\_p** `float` (default: 1.0)

Top-p (nucleus) sampling parameter.

**n** `integer` (default: 1)

Number of completions to generate (currently only n=1 is supported).

**stream** `boolean` (default: false)

Whether to stream the response using Server-Sent Events (SSE).

**stop** `array` (default: \[])

List of stop sequences (currently not implemented).

**presence\_penalty** `float` (default: 0.0)

Presence penalty (currently not implemented).

**frequency\_penalty** `float` (default: 0.0)

Frequency penalty (currently not implemented).

**ignore\_eos** `boolean` (default: false)

Whether to ignore the end-of-sequence token and continue generation.

## Response Format

### Streaming Response

When `stream=true`, the endpoint returns Server-Sent Events (SSE) with the following format:

**id** `string`

Unique completion ID (format: `cmpl-{uid}`).

**object** `string`

Always "text\_completion.chunk" for streaming responses.

**choices** `array`

Array of completion choices. Each choice contains:

* `delta` (object): Incremental content update
  * `role` (string): "assistant" (only in first chunk)
  * `content` (string): Generated text fragment
* `index` (integer): Choice index (always 0)
* `finish_reason` (string): null during generation, "stop" when complete

The stream ends with a `data: [DONE]` message.

## Examples

### Chat Completion with Messages

```bash theme={null}
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 100,
    "temperature": 0.7
  }'
```

### Streaming Response

```bash theme={null}
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Write a haiku about programming."}
    ],
    "max_tokens": 50,
    "stream": true
  }'
```

Example streaming output:

```
data: {"id":"cmpl-1","object":"text_completion.chunk","choices":[{"delta":{"role":"assistant"},"index":0,"finish_reason":null}]}

data: {"id":"cmpl-1","object":"text_completion.chunk","choices":[{"delta":{"content":"Code"},"index":0,"finish_reason":null}]}

data: {"id":"cmpl-1","object":"text_completion.chunk","choices":[{"delta":{"content":" flows"},"index":0,"finish_reason":null}]}

data: {"id":"cmpl-1","object":"text_completion.chunk","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}

data: [DONE]
```

### Using Python OpenAI Client

```python theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"  # API key not required
)

# Non-streaming
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing in simple terms."}
    ],
    max_tokens=150,
    temperature=0.7
)

print(response.choices[0].message.content)
```

### Streaming with Python OpenAI Client

```python theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"
)

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Count to 10."}],
    max_tokens=50,
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
```

### Raw Prompt (Non-Chat)

```bash theme={null}
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "prompt": "Once upon a time",
    "max_tokens": 100,
    "temperature": 0.8
  }'
```

### Advanced Sampling Parameters

```python theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Generate creative text."}],
    max_tokens=200,
    temperature=0.9,
    top_p=0.95,
    top_k=50
)
```

## Notes

* The endpoint always returns streaming responses (Server-Sent Events format)
* Non-streaming responses are not yet implemented
* The `stop`, `presence_penalty`, and `frequency_penalty` parameters are accepted but not yet implemented
* Greedy decoding is automatically used when `temperature=0.0` or `top_k=1`
* Client disconnection triggers automatic request cancellation to free resources


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.