> ## Documentation Index
> Fetch the complete documentation index at: https://sgl-project-mini-sglang.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Docker Deployment

> Deploy Mini-SGLang in containerized environments

Mini-SGLang provides official Docker support for easy deployment in containerized environments. This guide covers building images, running containers, and best practices for production deployments.

## Prerequisites

**1. Install Docker**

Follow the [official Docker installation guide](https://docs.docker.com/get-docker/) for your platform.

**2. Install NVIDIA Container Toolkit**

Required for GPU access in containers:

```bash theme={null}
# Ubuntu/Debian
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker
```

Verify installation:

```bash theme={null}
docker run --rm --gpus all nvidia/cuda:12.8.1-base-ubuntu24.04 nvidia-smi
```

## Building the Docker Image

**1. Clone the repository**

```bash theme={null}
git clone https://github.com/sgl-project/mini-sglang.git
cd mini-sglang
```

**2. Build the image**

Build with default settings:

```bash theme={null}
docker build -t minisgl .
```

Or customize build arguments:

```bash theme={null}
docker build -t minisgl \
  --build-arg CUDA_VERSION=12.8.1 \
  --build-arg PYTHON_VERSION=3.12 \
  --build-arg UBUNTU_VERSION=24.04 \
  .
```

**3. Verify the build**

Check that the image was created:

```bash theme={null}
docker images minisgl
```

Expected output:

```
REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
minisgl      latest    abc123def456   2 minutes ago   8.5GB
```

## Running the Container

### Basic Server Deployment

Launch an API server with GPU access:

```bash theme={null}
docker run --gpus all -p 1919:1919 \
  minisgl --model Qwen/Qwen3-0.6B --host 0.0.0.0
```

> [!NOTE]
>
> The `--host 0.0.0.0` flag is required for the server to be accessible outside the container.

### Interactive Shell Mode

Run in interactive shell mode:

```bash theme={null}
docker run -it --gpus all \
  minisgl --model Qwen/Qwen3-0.6B --shell
```

### Custom Port Mapping

Map to a different host port:

```bash theme={null}
docker run --gpus all -p 8000:1919 \
  minisgl --model Qwen/Qwen3-0.6B --host 0.0.0.0
```

Access at `http://localhost:8000`

## Using Volume Mounts

### Persistent Cache Directories

Use Docker volumes to cache downloaded models and compiled kernels:

```bash theme={null}
docker run --gpus all -p 1919:1919 \
  -v huggingface_cache:/app/.cache/huggingface \
  -v tvm_cache:/app/.cache/tvm-ffi \
  -v flashinfer_cache:/app/.cache/flashinfer \
  minisgl --model Qwen/Qwen3-0.6B --host 0.0.0.0
```

> [!TIP]
>
> Using volume mounts significantly speeds up subsequent container starts by avoiding re-downloading models and re-compiling kernels.

### Using Host Directories

Alternatively, mount specific host directories:

```bash theme={null}
mkdir -p ~/.cache/minisgl/{huggingface,tvm-ffi,flashinfer}

docker run --gpus all -p 1919:1919 \
  -v ~/.cache/minisgl/huggingface:/app/.cache/huggingface \
  -v ~/.cache/minisgl/tvm-ffi:/app/.cache/tvm-ffi \
  -v ~/.cache/minisgl/flashinfer:/app/.cache/flashinfer \
  minisgl --model Qwen/Qwen3-0.6B --host 0.0.0.0
```

## Multi-GPU Deployment

### All Available GPUs

```bash theme={null}
docker run --gpus all -p 1919:1919 \
  -v huggingface_cache:/app/.cache/huggingface \
  minisgl --model "meta-llama/Llama-3.1-70B-Instruct" --tp 4 --host 0.0.0.0
```

### Specific GPUs

Select specific GPU devices:

```bash theme={null}
docker run --gpus '"device=0,1,2,3"' -p 1919:1919 \
  minisgl --model "meta-llama/Llama-3.1-70B-Instruct" --tp 4 --host 0.0.0.0
```

## Production Deployment

### Docker Compose

Create a `docker-compose.yml`:

```yaml theme={null}
version: '3.8'

services:
  minisgl:
    image: minisgl:latest
    command: --model Qwen/Qwen3-0.6B --host 0.0.0.0
    ports:
      - "1919:1919"
    volumes:
      - huggingface_cache:/app/.cache/huggingface
      - tvm_cache:/app/.cache/tvm-ffi
      - flashinfer_cache:/app/.cache/flashinfer
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  huggingface_cache:
  tvm_cache:
  flashinfer_cache:
```

Launch with:

```bash theme={null}
docker compose up -d
```

### Health Checks

Add a health check to the Dockerfile or docker-compose.yml:

```yaml theme={null}
services:
  minisgl:
    # ... other configuration ...
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:1919/v1/models"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 120s
```

### Resource Limits

Set memory and CPU limits:

```bash theme={null}
docker run --gpus all -p 1919:1919 \
  --memory="32g" \
  --cpus="8" \
  -v huggingface_cache:/app/.cache/huggingface \
  minisgl --model Qwen/Qwen3-0.6B --host 0.0.0.0
```

## Environment Variables

Pass environment variables to configure the container:

```bash theme={null}
docker run --gpus all -p 1919:1919 \
  -e CUDA_VISIBLE_DEVICES=0,1 \
  -e HF_TOKEN=your_huggingface_token \
  minisgl --model meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0
```

### Available Environment Variables

**CUDA\_VISIBLE\_DEVICES** `string`

Comma-separated GPU indices to use

**HF\_TOKEN** `string`

HuggingFace authentication token for gated models

**MINISGL\_DISABLE\_OVERLAP\_SCHEDULING** `boolean`

Set to `1` to disable overlap scheduling

## Troubleshooting

> **GPU Not Detected**
>
> Verify NVIDIA Container Toolkit is installed:
>
> ```bash theme={null}
> docker run --rm --gpus all nvidia/cuda:12.8.1-base-ubuntu24.04 nvidia-smi
> ```
>
> If this fails, reinstall the NVIDIA Container Toolkit.

> **Permission Denied**
>
> The container runs as a non-root user (UID 1001). Ensure mounted volumes have correct permissions:
>
> ```bash theme={null}
> sudo chown -R 1001:1001 ~/.cache/minisgl
> ```

> **Out of Memory**
>
> Increase Docker's memory limit:
>
> ```bash theme={null}
> # In Docker Desktop: Settings → Resources → Memory
> # Or use --memory flag
> docker run --gpus all --memory="64g" -p 1919:1919 minisgl --model ...
> ```

> **Model Download Fails**
>
> * Check network connectivity
> * Try using `--model-source modelscope`
> * For gated models, provide `HF_TOKEN` environment variable

## Windows (WSL2) Deployment

For Windows users with WSL2:

**1. Install WSL2 and Docker Desktop**

* Install [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install)
* Install [Docker Desktop](https://www.docker.com/products/docker-desktop/) with WSL2 backend

**2. Build and run in WSL2**

Open WSL2 terminal and follow the standard Linux instructions:

```bash theme={null}
docker build -t minisgl .
docker run --gpus all -p 1919:1919 minisgl --model Qwen/Qwen3-0.6B --host 0.0.0.0
```

**3. Access from Windows**

The server will be accessible at `http://localhost:1919` from Windows browsers and applications.

> [!TIP]
>
> For production deployments with load balancing and orchestration, consider using Kubernetes with NVIDIA GPU Operator.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.