Deploying Qwen3.6-27B-FP8 with vLLM

Deploying Qwen3.6-27B-FP8 with vLLM

This guide walks through serving Qwen/Qwen3.6-27B-FP8 on a Linux GPU server using vLLM, with an OpenAI-compatible HTTP API. It is based on hands-on deployment and benchmarking: memory limits on consumer/datacenter GPUs, the performance impact of CUDA graphs and MTP speculative decoding, context-length tradeoffs, and verification of JSON mode, function calling, and reasoning toggles.

The official vLLM recipe is the starting point. This article documents what works on a single ~48 GB GPU in practice, where the recipe assumptions break down, and how to recover both throughput and context length.

---

Table of contents

1. Overview
2. Prerequisites
3. Hardware recommendations
4. Understanding GPU memory usage
5. Server setup
6. Install vLLM
7. Obtain model weights
8. Run the server
9. Performance tuning
10. Context length tuning
11. OpenAI API compatibility
12. Performance reference
13. Run as a persistent service
14. Access the API
15. Tuning and operations
16. Troubleshooting
17. Optional: expose externally

---

Overview

Architecture:

Client ──► vLLM (OpenAI API) ──► Qwen3.6-27B-FP8 ──► GPU
                                      │
                                      └── MTP draft head (optional)

For remote development:

Your laptop ──SSH tunnel──► server:127.0.0.1:8000 ──► vLLM

Qwen3.6-27B is the flagship dense model in the Qwen3.6 family. The FP8 checkpoint is the practical choice for a single 40–48 GB GPU. Unlike a pure transformer, it uses gated delta network hybrid attention (Mamba-style state + attention blocks), which affects KV cache sizing and decode throughput.

---

Prerequisites

• A Linux machine with an NVIDIA GPU and a working driver (nvidia-smi succeeds).
• Python 3.10+ (3.11 or 3.12 recommended).
• ≥ 40 GB GPU VRAM for FP8 inference (see Hardware recommendations).
• ~30 GB disk for model weights (~29 GB download), plus 8–12 GB for the vLLM/PyTorch environment and executable compile caches.
• Network access to Hugging Face or ModelScope for the first model download.
• Optional: systemd, Supervisor, or Docker for production-style deployment.

Before downloading, check the writable filesystem. The checkpoint, Python environment, and compile caches can exceed 40 GB combined, and some containerized environments expose a smaller writable overlay than the host disk size suggests.

df -hT / /var /workspace /dev/shm 2>/dev/null || true
mount | grep ' /dev/shm '

Large tmpfs mounts such as /dev/shm can be useful for temporary model weights when disk is constrained, but they are RAM-backed and often mounted noexec. If you use tmpfs for weights, keep Python environments and Triton/torch compile caches on an executable filesystem such as /opt, /var, or a persistent volume.

Verify the GPU:

nvidia-smi
python3 -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

---

Hardware recommendations

Reference hardware (benchmarks in this guide)

On this GPU with the tuned configuration below:

• Model weights: ~28 GiB
• KV cache (65K context, FP8): ~314K tokens total pool capacity
• Steady-state VRAM: ~46 GB

High-VRAM Blackwell reference

On a single NVIDIA RTX PRO 6000 Blackwell with ~98 GB VRAM, the same model can run the native 262K context window with the tuned MTP-3 configuration:

• Model weights: ~28 GiB
• KV cache (262K context, FP8): ~1.53M tokens total pool capacity
• Maximum full-context concurrency: ~5.8x at 262,144 tokens/request
• Steady-state VRAM: ~93 GB with --gpu-memory-utilization 0.94

This leaves little free VRAM by design. Stop other GPU processes before startup and expect nvidia-smi to show most VRAM allocated once vLLM is ready.

If you need vision input on the same class of GPU, remove --language-model-only and budget for the multimodal encoder. A validated high-VRAM profile is 200K context with vision enabled, which reported ~1.46M KV-cache tokens and ~7.3x full-context concurrency at 200,000 tokens/request.

CUDA compatibility notes:

• NVIDIA GPUs with compute capability ≥ 10.0 (Blackwell) need PyTorch/CUDA wheels built for CUDA ≥ 12.8.
• Install vLLM via pip/uv and let it pull a matching PyTorch build — do not mix arbitrary torch versions.
• The FP8 checkpoint uses block-scaled W8A8 kernels; vLLM may log warnings about missing per-GPU tuning files on uncommon cards (e.g. RTX 5880 Ada). This affects kernel selection, not correctness.

---

Understanding GPU memory usage

A 27B FP8 model weighs ~29 GB on disk but vLLM reserves additional VRAM for KV cache, CUDA graphs, MTP draft state, and compilation artifacts. On a 49 GB GPU you cannot run the recipe's default --max-model-len 262144 without running out of memory.

What consumes VRAM

Observed OOM boundary (RTX 5880 Ada, MTP-3, CUDA graphs on)

Observed high-VRAM boundary (RTX PRO 6000 Blackwell, MTP-3, CUDA graphs on)

If a 262K launch fails with a message like Free memory on device cuda:0 ... is less than desired GPU memory utilization, verify no orphaned vLLM worker is still holding VRAM. See Free GPU memory and the matching troubleshooting entry below.

The recipe's single-GPU FP8 example assumes an H100/L40S-class 40 GB card and does not enable MTP or tool calling. Real-world headroom is tighter once you add serving features.

Check actual usage

nvidia-smi
# After server is ready
curl -s http://127.0.0.1:8000/v1/models | python3 -m json.tool

Look for max_model_len in the response and Available KV cache memory / GPU KV cache size in vLLM logs.

---

Server setup

SSH into your server:

ssh user@your-server.example.com

Create an isolated Python environment:

uv venv ~/.venvs/qwen3-6-27b
source ~/.venvs/qwen3-6-27b/bin/activate

Or with standard venv:

python3 -m venv ~/.venvs/qwen3-6-27b
source ~/.venvs/qwen3-6-27b/bin/activate
pip install -U pip

Set a model cache directory:

export HF_HOME=/var/lib/huggingface
mkdir -p "$HF_HOME"

If model storage and executable cache storage need to live in different places, split them by execution requirements:

# Model weights and HF metadata can live on any large readable filesystem.
export MODEL_DIR=/models/qwen3.6-27b-fp8
export HF_HOME=/models/.hf_home

# vLLM, Triton, and torch.compile create executable shared objects.
# Keep these on an executable filesystem.
export VLLM_CACHE_ROOT=/var/cache/qwen3-6-27b/vllm
export TORCHINDUCTOR_CACHE_DIR=/var/cache/qwen3-6-27b/torchinductor
export TRITON_CACHE_DIR=/var/cache/qwen3-6-27b/triton
export XDG_CACHE_HOME=/var/cache/qwen3-6-27b/xdg
export TMPDIR=/var/cache/qwen3-6-27b/tmp

mkdir -p "$MODEL_DIR" "$HF_HOME" "$VLLM_CACHE_ROOT" "$TORCHINDUCTOR_CACHE_DIR" "$TRITON_CACHE_DIR" "$XDG_CACHE_HOME" "$TMPDIR"

This layout avoids putting executable caches on noexec filesystems. If MODEL_DIR points to tmpfs, remember that the weights are volatile and must be downloaded again after restart.

Optional: reduce fragmentation on long runs:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

---

Install vLLM

With the virtual environment activated:

uv pip install -U vllm --torch-backend=auto

On Blackwell or other very new GPUs, prefer letting uv/pip resolve the matching PyTorch/CUDA wheel. Avoid pinning an older torch build before installing vLLM.

If the executable filesystem is small, put resolver/download caches somewhere larger while installing, but keep the final venv on an executable filesystem:

export UV_CACHE_DIR=/dev/shm/qwen3-6-27b/uv-cache
export PIP_CACHE_DIR=/dev/shm/qwen3-6-27b/pip-cache
export TMPDIR=/var/cache/qwen3-6-27b/tmp

uv venv /opt/qwen3-6-27b/venv --python python3.12
source /opt/qwen3-6-27b/venv/bin/activate
uv pip install -U --link-mode=copy "vllm>=0.17.0"

Do not place the venv itself on /dev/shm if it is mounted noexec; console scripts and compiled extensions will fail with permission or shared-object mapping errors.

Or:

pip install "vllm>=0.17.0"

Confirm:

vllm --version
# 0.23.0 or newer recommended

---

Obtain model weights

By default, vllm serve Qwen/Qwen3.6-27B-FP8 downloads from Hugging Face on first start (~29 GB, ~30–60 seconds on a fast link).

To download explicitly before startup:

export HF_HOME=/models/.hf_home
export HF_XET_HIGH_PERFORMANCE=1

hf download Qwen/Qwen3.6-27B-FP8 \
  --local-dir /models/qwen3.6-27b-fp8

Use HF_TOKEN for higher Hugging Face rate limits if you have one, but do not paste long-lived tokens into logs, shell history, or issue trackers. Rotate the token after ad hoc use on shared machines.

ModelScope (when Hugging Face is slow or blocked)

pip install modelscope

export MODEL_DIR=/var/lib/models/qwen3.6-27b-fp8
modelscope download --model Qwen/Qwen3.6-27B-FP8 --local_dir "${MODEL_DIR}"

Serve from the local path:

vllm serve "${MODEL_DIR}" ...

Offline / air-gapped

1. Download on a connected machine.
2. rsync the full directory to the target server.
3. Point vllm serve at the local path.

rsync -avz /var/lib/models/qwen3.6-27b-fp8/ deploy@server:/var/lib/models/qwen3.6-27b-fp8/

---

Run the server

Recipe baseline (reference only)

From the vLLM recipe:

vllm serve Qwen/Qwen3.6-27B-FP8 \
  --max-model-len 262144 \
  --reasoning-parser qwen3

This targets a 40 GB datacenter GPU with minimal extras. On a 48 GB Ada card with tool calling and MTP, it OOMs during KV cache initialization.

Recommended production command (single GPU, tuned)

This configuration was validated end-to-end: stable startup, ~50 tok/s single-stream decode, 65K context, JSON mode, function calling, and thinking toggle.

source ~/.venvs/qwen3-6-27b/bin/activate

vllm serve Qwen/Qwen3.6-27B-FP8 \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 65536 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --language-model-only \
  --gpu-memory-utilization 0.94 \
  --max-cudagraph-capture-size 256 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking": false}'

First startup downloads weights (if needed), runs torch.compile, and captures CUDA graphs. Expect 3–5 minutes before /v1/models responds. Subsequent restarts are faster when compilation caches exist under ~/.cache/vllm/.

Verify:

curl -s http://127.0.0.1:8000/v1/models | python3 -m json.tool

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.6-27B-FP8",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "max_tokens": 64
  }' | python3 -m json.tool

Useful CLI flags

High-VRAM 262K variant

On a clean ~98 GB GPU, the tuned text-only command can keep the model's native 262K context:

source /opt/qwen3-6-27b/venv/bin/activate

export HF_HOME=/models/.hf_home
export VLLM_CACHE_ROOT=/var/cache/qwen3-6-27b/vllm
export TORCHINDUCTOR_CACHE_DIR=/var/cache/qwen3-6-27b/torchinductor
export TRITON_CACHE_DIR=/var/cache/qwen3-6-27b/triton
export XDG_CACHE_HOME=/var/cache/qwen3-6-27b/xdg
export TMPDIR=/var/cache/qwen3-6-27b/tmp

vllm serve /models/qwen3.6-27b-fp8 \
  --served-model-name Qwen/Qwen3.6-27B-FP8 \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --language-model-only \
  --gpu-memory-utilization 0.94 \
  --max-cudagraph-capture-size 256 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking": false}'

The split cache layout is intentional: model weights can live on large tmpfs, but vLLM/Triton compile caches must be on an executable filesystem. If /dev/shm is noexec, putting VLLM_CACHE_ROOT or TRITON_CACHE_DIR there can crash the first real request with failed to map segment from shared object.

High-VRAM vision variant

To enable image input, remove --language-model-only. On a ~98 GB GPU, a validated configuration keeps a 200K text context while enabling the multimodal encoder:

vllm serve /models/qwen3.6-27b-fp8 \
  --served-model-name Qwen/Qwen3.6-27B-FP8 \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 200000 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --gpu-memory-utilization 0.94 \
  --max-cudagraph-capture-size 256 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking": false}'

When vision is enabled, startup logs should include encoder-cache initialization instead of All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.

Prefix caching

Enable prefix caching when requests often share the same system prompt, retrieved context prefix, long document prefix, or tool instruction prefix:

vllm serve Qwen/Qwen3.6-27B-FP8 \
  --reasoning-parser qwen3 \
  --enable-prefix-caching

Prefix caching works for text-only and vision-enabled deployments. It consumes KV-cache capacity for reusable prefixes, so watch GPU KV cache size, Maximum concurrency, and cache hit metrics after enabling it.

Verify with Prometheus metrics:

curl -s http://127.0.0.1:8000/metrics | grep -E 'prefix_cache|prompt_tokens_cached'

Useful counters:

After enabling prefix caching on the 200K vision profile, the observed KV cache changed from ~1.46M to ~1.43M tokens, with maximum full-context concurrency moving from ~7.32x to ~7.16x. The small reduction is expected; the tradeoff is faster repeated-prefix workloads.

---

Performance tuning

Throughput on this model is highly sensitive to vLLM compilation settings. The biggest pitfall encountered in deployment: using --enforce-eager to work around startup OOM.

The --enforce-eager trap

When KV cache initialization fails at high context lengths, it is tempting to add:

--enforce-eager
--max-cudagraph-capture-size 128

This disables torch.compile and CUDA graphs entirely. vLLM logs:

Enforce eager set, disabling torch.compile and CUDAGraphs.
Cudagraph is disabled under eager mode

Impact on RTX 5880 Ada (512-token decode, thinking off):

Fix: Instead of --enforce-eager, reduce --max-model-len, add --kv-cache-dtype fp8, and optionally --language-model-only. Keep CUDA graphs enabled.

MTP speculative decoding

Qwen3.6 ships with a built-in MTP (multi-token prediction) draft head. Enable via:

--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

Higher speculative token counts increase draft work; 3 was the sweet spot on the test hardware. vLLM may warn that max_num_scheduled_tokens is suboptimal — consider tuning --max-num-batched-tokens if you push this further.

CUDA graph capture size

The recipe troubleshooting section recommends lowering --max-cudagraph-capture-size (default 512) if you hit Mamba cache / CUDA graph errors (vLLM PR #34571). On RTX 5880 Ada, 256 worked reliably with MTP-3 and 65K context.

---

Context length tuning

The model natively supports 262K tokens, but that requires YaRN scaling and multi-GPU setups per the model card. On a single 48 GB GPU, use a tiered approach:

With the balanced tier, logs reported:

GPU KV cache size: 314,078 tokens
Maximum concurrency for 65,536 tokens per request: 4.79x

On a 98 GB RTX PRO 6000 Blackwell at 262K context, logs reported:

Available KV cache memory: 53.27 GiB
GPU KV cache size: 1,528,180 tokens
Maximum concurrency for 262,144 tokens per request: 5.83x

That concurrency number is for full-length 262K requests. Short prompts can still batch far beyond 5 concurrent clients because they consume far fewer KV tokens.

To push beyond 65K on one GPU, try in order:

1. --kv-cache-dtype fp8 (if not already set)
2. Lower --speculative-config to num_speculative_tokens: 1 (frees VRAM)
3. Reduce --max-cudagraph-capture-size to 128
4. Lower --gpu-memory-utilization slightly (counterintuitive, but can help graph profiling fit)

Ultra-long context (multi-GPU)

For 524K–1M tokens, the recipe uses tensor parallelism and YaRN overrides:

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-27B-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 1010000 \
  --reasoning-parser qwen3 \
  --hf-overrides '{"text_config": {"rope_parameters": {"rope_type": "yarn", "factor": 4.0, ...}}}'

That is out of scope for single-GPU deploy but is the path to the model's full context window.

---

OpenAI API compatibility

The server exposes a standard OpenAI-compatible surface. The following were verified against vLLM 0.23.0.

Chat completions

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="Qwen/Qwen3.6-27B-FP8",
    messages=[{"role": "user", "content": "Write a haiku about neural networks."}],
    max_tokens=256,
)
print(resp.choices[0].message.content)

Vision input

Vision input works when the server is launched without --language-model-only. Use OpenAI-style multimodal message content:

import base64
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")

with open("zoo.webp", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode("utf-8")

resp = client.chat.completions.create(
    model="Qwen/Qwen3.6-27B-FP8",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Recognize the animal in this image. Answer briefly."},
            {"type": "image_url", "image_url": {"url": f"data:image/webp;base64,{image_b64}"}},
        ],
    }],
    max_tokens=128,
)

print(resp.choices[0].message.content)
# Example: Giraffe; the long neck and spotted coat make it clear.

If you see All limits of multimodal modalities supported by the model are set to 0, running in text-only mode, the server is still running with --language-model-only or equivalent multimodal limits.

JSON mode (response_format: json_object)

resp = client.chat.completions.create(
    model="Qwen/Qwen3.6-27B-FP8",
    messages=[{"role": "user", "content": "Return JSON with keys city, population, country. Use Tokyo."}],
    max_tokens=100,
    response_format={"type": "json_object"},
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
# content: {"city": "Tokyo", "population": 13960000, "country": "Japan"}

JSON schema (structured outputs)

resp = client.chat.completions.create(
    model="Qwen/Qwen3.6-27B-FP8",
    messages=[{"role": "user", "content": "What is 17 + 25?"}],
    max_tokens=100,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "math_answer",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "answer": {"type": "integer"},
                    "explanation": {"type": "string"},
                },
                "required": ["answer", "explanation"],
                "additionalProperties": False,
            },
        },
    },
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

Function calling

Requires --enable-auto-tool-choice and --tool-call-parser qwen3_xml at server startup.

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
        },
    },
}]

resp = client.chat.completions.create(
    model="Qwen/Qwen3.6-27B-FP8",
    messages=[{"role": "user", "content": "What is the weather in Paris?"}],
    tools=tools,
    tool_choice="auto",
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
# finish_reason: "tool_calls"
# tool_calls[0].function.name == "get_weather"
# tool_calls[0].function.arguments == '{"location": "Paris, France"}'

Full roundtrip (tool result → final answer) works with standard OpenAI message shapes.

Thinking mode (enable / disable)

Qwen3.6 supports chain-of-thought via the chat template. Control per request:

# Disabled — direct answer in content, reasoning is null
extra_body={"chat_template_kwargs": {"enable_thinking": False}}

# Enabled — reasoning in message.reasoning, answer in content
extra_body={"chat_template_kwargs": {"enable_thinking": True}}

Or set a server default:

--default-chat-template-kwargs '{"enable_thinking": false}'

Use --reasoning-parser qwen3 so vLLM surfaces the reasoning field separately from content.

---

Performance reference

All benchmarks: Qwen/Qwen3.6-27B-FP8, vLLM 0.23.0, NVIDIA RTX 5880 Ada (49 GB), thinking disabled, --language-model-only, tuned production config with MTP-3 and 65K context.

Single-stream decode (streaming, 512 completion tokens)

Concurrent throughput (256 max_tokens per request)

Batching scales well: per-request latency stays ~6–7 s while aggregate throughput exceeds 600 tok/s at 16 concurrent clients.

High-VRAM Blackwell reference (262K context)

Additional measurements on NVIDIA RTX PRO 6000 Blackwell (~98 GB), vLLM 0.23.0, thinking disabled, --language-model-only, MTP-3, FP8 KV cache, and --max-model-len 262144:

Compared with the RTX 5880 Ada 65K-context reference, the Blackwell system was about 1.5× faster single-stream and 1.3× faster at 16 concurrent short requests while also serving the larger 262K context window. The ×32 result used short prompts; full 262K prompts are bounded by the KV-cache concurrency reported above.

With vision enabled on the same GPU class, a 200K-context profile was validated with text and image requests:

Available KV cache memory: 52.05 GiB
GPU KV cache size: 1,464,233 tokens
Maximum concurrency for 200,000 tokens per request: 7.32x

The vision profile trades some context and a small amount of model memory for multimodal input. Use the text-only 262K profile when maximum context is more important than image support.

Shorter generation benchmark (128 max_tokens, non-streaming)

Note: The 128-token non-streaming figures were measured with the earlier MTP-1 / 32K config. With MTP-3 / 65K, single-stream decode is ~5× faster for long outputs. Short-prompt latency is dominated by prefill and sampling overhead.

Run your own benchmark

# Streaming decode TPS (Python)
python3 benchmark_decode_tps.py

# Concurrent batch test
python3 benchmark_vllm.py

Example streaming measurement logic:

import json, time, urllib.request

payload = {
    "model": "Qwen/Qwen3.6-27B-FP8",
    "messages": [{"role": "user", "content": "Write a 400-word essay on transformers."}],
    "max_tokens": 512,
    "temperature": 0,
    "stream": True,
    "stream_options": {"include_usage": True},
    "chat_template_kwargs": {"enable_thinking": False},
}
start = time.perf_counter()
# ... consume SSE chunks, record TTFT and completion_tokens from final usage ...
# decode_tps = completion_tokens / (end - ttft)

---

Run as a persistent service

Do not rely on a foreground shell in production.

Option A: systemd (recommended)

Create /usr/local/bin/qwen3-6-27b.sh:

#!/bin/bash
set -euo pipefail

source /home/deploy/.venvs/qwen3-6-27b/bin/activate

export HF_HOME="${HF_HOME:-/var/lib/huggingface}"
export PYTORCH_CUDA_ALLOC_CONF="${PYTORCH_CUDA_ALLOC_CONF:-expandable_segments:True}"
export VLLM_LOGGING_LEVEL="${VLLM_LOGGING_LEVEL:-INFO}"

MODEL="${QWEN36_MODEL:-Qwen/Qwen3.6-27B-FP8}"
PORT="${QWEN36_PORT:-8000}"
MAX_LEN="${QWEN36_MAX_MODEL_LEN:-65536}"
GPU_UTIL="${QWEN36_GPU_UTIL:-0.94}"
MTP_TOKENS="${QWEN36_MTP_TOKENS:-3}"

exec vllm serve "${MODEL}" \
  --host 127.0.0.1 \
  --port "${PORT}" \
  --max-model-len "${MAX_LEN}" \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --language-model-only \
  --gpu-memory-utilization "${GPU_UTIL}" \
  --max-cudagraph-capture-size 256 \
  --speculative-config "{\"method\": \"mtp\", \"num_speculative_tokens\": ${MTP_TOKENS}}" \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking": false}'

sudo chmod +x /usr/local/bin/qwen3-6-27b.sh

Create /etc/systemd/system/qwen3-6-27b.service:

[Unit]
Description=Qwen3.6-27B-FP8 Server (vLLM)
After=network.target

[Service]
Type=simple
User=deploy
Group=deploy
Environment=HF_HOME=/var/lib/huggingface
ExecStart=/usr/local/bin/qwen3-6-27b.sh
Restart=on-failure
RestartSec=15
KillMode=mixed
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now qwen3-6-27b
journalctl -u qwen3-6-27b -f

Note: Set TimeoutStartSec=600 in the unit if your orchestrator treats slow first-boot compilation as a failure. Cold starts with CUDA graph capture can exceed 3 minutes.

Option B: Supervisor

[program:qwen3_6_27b]
command=/usr/local/bin/qwen3-6-27b.sh
directory=/home/deploy
user=deploy
autostart=true
autorestart=true
startsecs=300
stopasgroup=true
killasgroup=true
stdout_logfile=/var/log/qwen3-6-27b.log
redirect_stderr=true

startsecs=300 avoids false FATAL states during compilation warmup.

For containerized or non-systemd environments, run vLLM as a supervisor-managed foreground process rather than a loose background command. Keep stopasgroup=true and killasgroup=true when possible so engine workers do not survive a failed restart and keep VRAM allocated.

Example wrapper with model weights and executable caches on separate configurable paths:

#!/bin/bash
set -eo pipefail

export MODEL_DIR="${MODEL_DIR:-/models/qwen3.6-27b-fp8}"
export HF_HOME="${HF_HOME:-/models/.hf_home}"
export HF_HUB_CACHE="${HF_HUB_CACHE:-${HF_HOME}/hub}"
export TRANSFORMERS_CACHE="${TRANSFORMERS_CACHE:-${HF_HOME}/transformers}"
export VLLM_CACHE_ROOT="${VLLM_CACHE_ROOT:-/var/cache/qwen3-6-27b/vllm}"
export TORCHINDUCTOR_CACHE_DIR="${TORCHINDUCTOR_CACHE_DIR:-/var/cache/qwen3-6-27b/torchinductor}"
export TRITON_CACHE_DIR="${TRITON_CACHE_DIR:-/var/cache/qwen3-6-27b/triton}"
export XDG_CACHE_HOME="${XDG_CACHE_HOME:-/var/cache/qwen3-6-27b/xdg}"
export TMPDIR="${TMPDIR:-/var/cache/qwen3-6-27b/tmp}"
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

source /opt/qwen3-6-27b/venv/bin/activate

exec vllm serve "${MODEL_DIR}" \
  --served-model-name Qwen/Qwen3.6-27B-FP8 \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len "${QWEN36_MAX_MODEL_LEN:-262144}" \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --gpu-memory-utilization "${QWEN36_GPU_UTIL:-0.94}" \
  --max-cudagraph-capture-size 256 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking": false}'

Add --language-model-only to the wrapper when you want maximum text context and do not need image input. If your supervisor environment provides its own logging helper, avoid set -u unless those helper scripts are written for unset variables.

Option C: Docker

docker run -d --name qwen3-6-27b --gpus all --restart unless-stopped \
  -p 8000:8000 \
  -v hf-cache:/data/huggingface \
  -e HF_HOME=/data/huggingface \
  vllm/vllm-openai:v0.23.0 \
  vllm serve Qwen/Qwen3.6-27B-FP8 \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 65536 \
    --kv-cache-dtype fp8 \
    --reasoning-parser qwen3 \
    --language-model-only \
    --gpu-memory-utilization 0.94 \
    --max-cudagraph-capture-size 256 \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
    --enable-prefix-caching \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --default-chat-template-kwargs '{"enable_thinking": false}'

Pin the image tag in production. Requires NVIDIA Container Toolkit.

---

Access the API

Same machine

curl http://127.0.0.1:8000/v1/models

Remote via SSH tunnel

ssh -L 8000:127.0.0.1:8000 user@your-server.example.com

Then on your laptop:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3.6-27B-FP8","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

---

Tuning and operations

Adjust context length

# 32K without FP8 KV (more headroom for other features)
QWEN36_MAX_MODEL_LEN=32768
# remove --kv-cache-dtype fp8 from wrapper

# 65K with FP8 KV (balanced)
QWEN36_MAX_MODEL_LEN=65536
# keep --kv-cache-dtype fp8

# 262K text-only with FP8 KV (requires roughly 98 GB VRAM for MTP-3)
QWEN36_MAX_MODEL_LEN=262144
# add --language-model-only

# 200K vision-enabled with FP8 KV (same high-VRAM class)
QWEN36_MAX_MODEL_LEN=200000
# remove --language-model-only

Restart the service after changes. Watch logs for GPU KV cache size and OOM errors.

Adjust throughput vs VRAM

Free GPU memory

sudo systemctl stop qwen3-6-27b
nvidia-smi

For Supervisor deployments:

supervisorctl stop qwen-vllm
nvidia-smi

If VRAM is still allocated after the service stops, look for orphaned vLLM workers:

ps -eo pid,ppid,stat,args | grep -E 'VLLM::EngineCore|vllm serve|APIServer' | grep -v grep

---

Troubleshooting

CUDA out of memory during startup

Usually KV cache allocation at high --max-model-len. Fix in order:

1. Add --kv-cache-dtype fp8
2. Lower --max-model-len (try 32768)
3. Add --language-model-only
4. Reduce --speculative-config MTP tokens
5. Lower --max-cudagraph-capture-size to 128

Do not reach for --enforce-eager unless you accept a ~5× decode slowdown.

Free memory on device ... is less than desired GPU memory utilization

This can mean the requested --gpu-memory-utilization is too high, but first check for stale GPU processes. A failed vLLM startup can leave an orphaned VLLM::EngineCore process holding tens of GB of VRAM even after the API server exits.

nvidia-smi
ps -eo pid,ppid,stat,args | grep -E 'VLLM::EngineCore|vllm serve|APIServer' | grep -v grep
kill <pid>
sleep 5
nvidia-smi

Only lower --gpu-memory-utilization or context length after confirming the GPU is actually clean. On a 98 GB Blackwell card, --max-model-len 262144 --kv-cache-dtype fp8 --gpu-memory-utilization 0.94 can start successfully from a clean GPU, but it will reserve almost all VRAM.

CUDA graph / Mamba cache size error

Lower capture size per the recipe:

--max-cudagraph-capture-size 128

"auto" tool choice requires --enable-auto-tool-choice

Start the server with:

--enable-auto-tool-choice --tool-call-parser qwen3_xml

Throughput unexpectedly low (~10 tok/s)

Check logs for Enforce eager set or Cudagraph is disabled. Remove --enforce-eager and ensure CUDA graphs are enabled.

Address already in use

ss -tlnp | grep 8000
# stop conflicting process or use --port 8001

Slow first request after restart

Expected. vLLM warms CUDA graphs on first real inference. Run a warmup request before load testing.

failed to map segment from shared object

If the path in the error is under /dev/shm, your compile cache is on a noexec mount. Triton and torch.compile generate .so files that must be mapped executable.

Move executable caches to /opt, /var, or a persistent executable volume:

export VLLM_CACHE_ROOT=/var/cache/qwen3-6-27b/vllm
export TORCHINDUCTOR_CACHE_DIR=/var/cache/qwen3-6-27b/torchinductor
export TRITON_CACHE_DIR=/var/cache/qwen3-6-27b/triton
export XDG_CACHE_HOME=/var/cache/qwen3-6-27b/xdg
export TMPDIR=/var/cache/qwen3-6-27b/tmp

It is still fine to keep model weights on /dev/shm if you accept that they are volatile.

Model download fails

Use ModelScope (see Obtain model weights) or pre-download with hf download Qwen/Qwen3.6-27B-FP8 --local-dir /path/to/model.

Older images may have a deprecated huggingface-cli that prints a warning and exits. Use the newer hf command from huggingface_hub instead.

no kernel image is available

PyTorch wheel does not support your GPU architecture. Reinstall vLLM with --torch-backend=auto or pick a wheel targeting your compute capability.

---

Optional: expose externally

Binding to 127.0.0.1 is the safest default.

SSH tunnel (recommended for dev)

ssh -L 8000:127.0.0.1:8000 user@your-server

Reverse proxy with authentication

Keep vLLM on localhost; terminate TLS and auth in Caddy or nginx:

llm.example.com {
    basicauth {
        deploy <bcrypt-hash>
    }
    reverse_proxy 127.0.0.1:8000
}

Some hosting platforms provide their own authenticated reverse proxy or mapped public port. Prefer the provider-reported service URL when available, and keep vLLM bound to 127.0.0.1 behind that proxy:

curl -H "Authorization: Bearer $SERVICE_TOKEN" \
  https://llm.example.com/v1/models

Direct bind (not recommended without auth)

vllm serve Qwen/Qwen3.6-27B-FP8 --host 0.0.0.0 --port 8000 ...

Restrict with firewall rules. vLLM does not ship production-grade API authentication.

---

Quick reference

# Install
uv venv ~/.venvs/qwen3-6-27b && source ~/.venvs/qwen3-6-27b/bin/activate
uv pip install -U vllm --torch-backend=auto

# Run (tuned single-GPU production config)
vllm serve Qwen/Qwen3.6-27B-FP8 \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 65536 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --language-model-only \
  --gpu-memory-utilization 0.94 --max-cudagraph-capture-size 256 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking": false}'

# Health check
curl -s http://127.0.0.1:8000/v1/models

# Chat
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3.6-27B-FP8","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

# SSH tunnel
ssh -L 8000:127.0.0.1:8000 user@your-server

---

Further reading

• vLLM recipe: Qwen3.6-27B
• Qwen3.6-27B model card
• Qwen3.6-27B-FP8 checkpoint
• vLLM OpenAI-compatible server
• Qwen3.8-27B on RTX PRO 6000 — SGLang follow-up for the next dense 27B, with NVFP4, MTP, and DSpark on 96 GB
• Qwen3.8-27B on a Single RTX 5090 — the 32 GB NVFP4 companion
• Deploying Qwen3 Embedding — companion guide for RAG embedding step
• Deploying Qwen3 Reranker — companion guide for RAG reranking step