Qwen3.8-27B on RTX PRO 6000: NVFP4, MTP, and DSpark with SGLang

Qwen3.8-27B on RTX PRO 6000: NVFP4, MTP, and DSpark with SGLang

This is the high-VRAM follow-up to Deploying Qwen3.6-27B-FP8 with vLLM and Qwen3.8-27B on a Single RTX 5090. The model family is the same hybrid Gated Delta Network 27B, but the serving story changes once you leave a 32–49 GB card.

On a single NVIDIA RTX PRO 6000 Blackwell (~96 GB) you can run the official NVFP4 checkpoint with SGLang, keep the model's native 262,144-token context, leave the vision tower on, and actually choose a speculative decoder. The 5090 guide had to drop DSpark and cap context around 200K. This article is the configuration that uses the extra VRAM.

It is based on a real deployment: official RadixArk/Qwen3.8-27B-NVFP4 plus the in-checkpoint MTP head, a side-by-side DSpark run, vision OCR, tool calling, and 8K/16K/32K needle retrieval. After first publish we stayed on the same card and added official Qwen/Qwen3.8-27B-FP8 with explicit FP8 KV, full 262K fills, a 2×262K overlap test, prefill timing, and radix-cache probes. The SGLang cookbook cell hw=rtx6000 / quant=nvfp4 / strategy=balanced is still the starting point.

---

Table of contents

1. Overview
2. What changed since Qwen3.6
3. Prerequisites
4. Hardware and checkpoint choices
5. Understanding the memory model
6. Install SGLang
7. Obtain the model weights
8. Run the server
9. Speculative decoding: MTP vs DSpark
10. Context, vision, and tools
11. Concurrency, prefill, and prompt cache
12. Performance reference
13. OpenAI API compatibility
14. Run as a persistent service
15. Access the API
16. Troubleshooting
17. Quick reference
18. Further reading

---

Overview

Architecture:

Client ──► SGLang (OpenAI API) ──► Qwen3.8-27B-NVFP4 ──► RTX PRO 6000 (96 GB)
                                      │
                                      ├── MTP draft head (recommended)
                                      └── DSpark draft (optional, extra checkpoint)

For remote development:

Your laptop ──SSH tunnel──► server:127.0.0.1:18000 ──► SGLang

Qwen3.8-27B is the dense follow-on to Qwen3.6-27B. The serving-relevant layout is the same hybrid GDN stack, now with an official SGLang cookbook, an NVFP4 checkpoint aimed at Blackwell tensor cores, and two speculative paths: the in-checkpoint MTP head, and a separate DSpark draft.

---

What changed since Qwen3.6

If you already followed the Qwen3.6 vLLM guide, the differences that matter on this card are:

The 3.6 article already recorded ~76 tok/s single-stream decode for FP8 + MTP-3 on this same GPU class under vLLM. Qwen3.8 on SGLang is a different stack (chunked hybrid GDN prefill, FlashInfer, radix cache), so do not treat the two TPS numbers as a like-for-like upgrade score. What you gain here is the official NVFP4 path, a much larger KV pool, and a working DSpark option.

---

Prerequisites

• A Linux machine with an NVIDIA GPU and a working driver (nvidia-smi succeeds).
• Python 3.12 and a Blackwell CUDA stack (CUDA ≥ 12.8; we used CUDA 13.0).
• ~96 GB GPU VRAM — this article is about using that headroom, not squeezing into 32 GB.
• ~50 GB disk if you keep NVFP4 + DSpark + the FP8 checkpoint for comparison; ~25 GB for NVFP4 alone.
• Network access to Hugging Face for the first download.

Verify the GPU before anything else:

nvidia-smi
python3 -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"
# torch 2.11.0+cu130 True NVIDIA RTX PRO 6000 Blackwell Workstation Edition

The test box was an unprivileged container: no kernel modules, no Docker-in-Docker. Long-running services were managed by a supervisor. The same commands work under systemd or a foreground process.

---

Hardware and checkpoint choices

Do not substitute a third-party NVFP4 export (for example Unsloth compressed-tensors) on SGLang 0.5.16. That format either fails to load or produces garbled output. Use the official ModelOpt mixed-precision checkpoint.

Reference hardware (benchmarks in this guide)

SM120 note: use --attention-backend flashinfer. trtllm_mha is SM100-only. The cookbook warns that MTP + FlashInfer wants a FlashInfer newer than 0.6.15.post1; on this 0.5.16 / 0.6.14 stack MTP still ran cleanly (accept length ~3.1 of 4). If your build errors on uniform_q_len, switch spec to --attention-backend triton.

How this card compares to the 5090 guide

If you are on 32 GB, stop here and use the 5090 article.

---

Understanding the memory model

Qwen3.8-27B is not a plain transformer. It is 64 layers laid out as 16 repeats of 3 × (Gated DeltaNet → FFN) then 1 × (Gated Attention → FFN) — 48 linear-attention layers and 16 full-attention layers, hidden size 5120. After weights, SGLang keeps two independent pools:

• A worst-case-reserved GDN state pool — one fixed-size state per running request slot. This sets the concurrency ceiling.
• A paged attention KV pool — sized per token.

--mamba-full-memory-ratio divides post-weight memory between them. The 5090 article has to spend several pages on this flag because the default 0.9 under-provisions the state pool and can clamp max_num_reqs to zero. On 96 GB the same default does the opposite of what most API traffic wants: it over-provisions KV and leaves the GDN pool at 85 slots. With extra_buffer (S=5) that is 85 ÷ 5 = 17 running requests — the log line max_running_requests was reduced from 48 to 17 is this cap, not a 96 GB hardware limit.

You do not need the 5090's 4.59 to boot. You do need a higher ratio, --max-mamba-cache-size, and/or --mamba-ssm-dtype bfloat16 if you want more than ~17 short requests in flight. MTP also reserves about 10 GB of intermediate GDN state (D=4 draft slots). That tax is why high admission and MTP fight each other.

What actually used VRAM

Measured on this card with --mem-fraction-static 0.85:

The NVFP4 checkpoint's kv_cache_quant_algo: FP8 is why --kv-cache-dtype auto already gives a 960K pool. Official FP8 does not declare that; auto stays on BF16 and a second 262K request queues. Add --kv-cache-dtype fp8_e4m3 and the same 24.5 GB of KV holds twice the tokens.

Radix cache (prefix cache for both KV and GDN state) is on by default. Do not add --disable-radix-cache unless you are deliberately trading prefix hits for S=1 state slots. Hybrid GDN only inserts prefix nodes every mamba_track_interval tokens (256 by default) — see Prompt cache.

---

Install SGLang

The cookbook image for this card is lmsysorg/sglang:qwen38-27b. If your runtime already bundles SGLang 0.5.16, verify the stack and stay there for MTP + vision + tools:

python3 -c "import sglang; print(sglang.__version__)"
python3 -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.get_device_name(0))"
# 0.5.16
# 2.11.0+cu130 13.0 NVIDIA RTX PRO 6000 Blackwell Workstation Edition

From a clean Python environment:

pip install --upgrade pip
pip install uv
uv pip install sglang

Set a cache directory that survives restarts:

export HF_HOME=/workspace/.hf_home
mkdir -p "$HF_HOME"

Optional: reduce fragmentation on long runs:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

---

Obtain the model weights

SGLang will pull from Hugging Face on first --model-path use. Pre-downloading keeps startup predictable:

export HF_HOME=/workspace/.hf_home

hf download RadixArk/Qwen3.8-27B-NVFP4 \
  --local-dir /workspace/models/Qwen3.8-27B-NVFP4

The NVFP4 tree is about 21 GB. If you also want the DSpark draft or the FP8 comparison checkpoint:

hf download RadixArk/Qwen3.8-27B-DSpark \
  --local-dir /workspace/models/Qwen3.8-27B-DSpark

hf download Qwen/Qwen3.8-27B-FP8 \
  --local-dir /workspace/models/Qwen3.8-27B-FP8

hf cannot take --local-dir and --cache-dir together. Use one or the other.

If Hugging Face is slow or blocked, the same IDs are on ModelScope. After the download, confirm the files landed:

du -sh /workspace/models/Qwen3.8-27B-*

The DSpark draft's config.json may declare architectures: ["DSparkDraftModel"] and auto_map that import specforge. SGLang's native class is Qwen3DSparkModel. If the server tries to load remote draft code and dies on specforge, rewrite the local architecture entry (see Troubleshooting).

---

Run the server

Official balanced recipe (no spec)

This is the cookbook cell hw=rtx6000 / quant=nvfp4 / strategy=balanced:

sglang serve \
  --trust-remote-code \
  --model-path /workspace/models/Qwen3.8-27B-NVFP4 \
  --mem-fraction-static 0.85 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --host 127.0.0.1 \
  --port 18000

Use this if you want the smallest moving surface. On this card it is leaving decode throughput on the table.

Recommended production command (NVFP4 + MTP)

The in-checkpoint MTP head is already in the NVFP4 tree. Turning it on is four flags:

sglang serve \
  --trust-remote-code \
  --model-path /workspace/models/Qwen3.8-27B-NVFP4 \
  --mem-fraction-static 0.85 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --host 127.0.0.1 \
  --port 18000

EAGLE here is the cookbook name for the in-checkpoint MTP head (NEXTN is an alias). 3 / 1 / 4 is the documented step/topk/draft-token triple.

Official FP8 + MTP + FP8 KV (sibling recipe)

Use this when you want the official Qwen weights and three native-262K requests in the KV pool. NVFP4 still decodes faster; this is the path that does not depend on the RadixArk tree.

sglang serve \
  --trust-remote-code \
  --model-path /workspace/models/Qwen3.8-27B-FP8 \
  --mem-fraction-static 0.85 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --kv-cache-dtype fp8_e4m3 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --host 127.0.0.1 \
  --port 18000

Boot should say KV Cache is allocated. dtype: torch.float8_e4m3fn, #tokens: 802375. Without --kv-cache-dtype fp8_e4m3 the same command allocates 401K BF16 tokens and cannot run two 262K requests at once.

First startup loads weights and captures CUDA graphs. Expect a couple of minutes before /v1/models responds.

Startup log, decoded

On this card the MTP boot sequence looks like:

Load weight end. type=Qwen3_5ForConditionalGeneration, quant=modelopt_mixed,
  quant_algo=MIXED_PRECISION, avail mem=74.12 GB, mem usage=20.14 GB.
Load weight end. type=Qwen3_5ForCausalLMMTP, quant=modelopt_mixed,
  avail mem=68.55 GB, mem usage=5.53 GB.
max_total_num_tokens=960523, chunked_prefill_size=2048, max_prefill_tokens=16384,
  max_running_requests=19, context_len=262144, available_gpu_mem=13.33 GB
Init Unified RadixTree with components (FULL, MAMBA)
The server is fired up and ready to roll!

Read those lines as:

• 20.14 GB + 5.53 GB — target + MTP head. This is still less than the FP8 target alone.
• 960,523 tokens — FP8 KV pool, about 3.6× the native 262K context. You are not fighting the 5090's 38K-token wall.
• context_len=262144 — the model native max is live; no --max-total-tokens shrink is required.
• Unified RadixTree — prefix cache is on for both attention KV and GDN state.

Verify:

curl -s http://127.0.0.1:18000/v1/models | python3 -m json.tool

curl -s http://127.0.0.1:18000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/workspace/models/Qwen3.8-27B-NVFP4",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "max_tokens": 64,
    "chat_template_kwargs": {"enable_thinking": false}
  }' | python3 -m json.tool

Useful CLI flags

---

Speculative decoding: MTP vs DSpark

On 96 GB you can run either path. They are not interchangeable.

MTP (recommended)

MTP uses the head already inside the NVFP4 checkpoint. No extra download, no architecture remapping, no lm_head patch. Accept length stayed ~3.1 of 4 draft tokens at concurrency 1/2/4, so the spec decoder is doing real work rather than rejecting almost every draft.

This is the default you should ship.

DSpark (optional)

DSpark is a separate 1.36B DFlash + Markov + confidence draft (gamma=7, so verify_num_draft_tokens=8):

sglang serve \
  --trust-remote-code \
  --model-path /workspace/models/Qwen3.8-27B-NVFP4 \
  --mem-fraction-static 0.85 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path /workspace/models/Qwen3.8-27B-DSpark \
  --speculative-draft-model-quantization unquant \
  --host 127.0.0.1 \
  --port 18000

A healthy boot prints:

Load weight end. type=Qwen3DSparkModel, avail mem=71.37 GB, mem usage=2.71 GB.
Initialized DSpark draft runner. attention_backend=flashinfer,
  model=Qwen3DSparkModel, gamma=7, verify_num_draft_tokens=8,
  mask_token_id=248077, markov_head=VanillaMarkov

--mamba-radix-cache-strategy extra_buffer_lazy is incompatible with spec on this SGLang build. Leave the default extra_buffer.

DSpark and the NVFP4 lm_head

SGLang 0.5.16's dense DSpark path computes draft-verify logits as hidden @ lm_head.weight.T. The official NVFP4 target uses a packed, bottlenecked lm_head (5120-dim hidden, 2560-dim packed weight). The first verify step then dies with:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (7x5120 and 2560x248320)

--disable-decode-cuda-graph does not fix it — the same matmul runs during warmup.

Two ways out:

1. Use the cookbook image lmsysorg/sglang:qwen38-27b, which is the stack the official 200 tok/s DSpark numbers were taken on.
2. On 0.5.16, patch sglang/srt/models/dspark.py compute_base_logits so a quantized lm_head goes through quant_method.apply(...) (the same branch LogitsProcessor already uses) instead of a raw matmul against .weight. Keep a backup of the file.

Until one of those is in place, DSpark + this NVFP4 checkpoint will not serve. The 5090 article hit the same lm_head mismatch and ran out of VRAM; here only the mismatch remains.

Which one is faster?

Same card, same NVFP4 target, same sglang.bench_serving workload (random, isl=1024 / osl=128, 32 prompts, --request-rate inf, --flush-cache):

MTP wins decode throughput at every concurrency we measured. DSpark wins time-to-first-token. If your traffic is long completions, ship MTP. If it is short, interactive turns where TTFT dominates, DSpark is the experiment worth keeping.

The cookbook's ~200 tok/s DSpark figure was taken on lmsysorg/sglang:qwen38-27b with a real decode workload, not on 0.5.16 with random 1024/128 tokens. Do not treat 134 vs 200 as a regression of this card.

---

Context, vision, and tools

These were verified against the NVFP4 + MTP server above. Thinking was disabled for the functional tests.

Native 262K context

/v1/models reports max_model_len: 262144. Needle retrieval (NVFP4 + MTP):

A later official-FP8 + MTP run filled the native 262,128-token window (max_new_tokens=4, unique prefixes so radix could not share KV) and finished HTTP 200 in 87–91 s. The cap is the model's 262K, not free VRAM — unless you leave official FP8 on BF16 KV, in which case the second 262K request is what does not fit. See Concurrency.

If you ever need to raise context past 262K, that is a YaRN / multi-GPU path (see the 3.6 article), not a single-GPU flag.

Vision

Leave multimodal enabled (do not pass a language-only flag). A 1122×1402 PNG of a Chinese retail poster produced 1,540 image tokens and a correct OCR of NEW! / 新货 / 上架 / 国企经营 in 1.66 s.

import base64
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:18000/v1", api_key="not-needed")

with open("poster.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode("utf-8")

resp = client.chat.completions.create(
    model="/workspace/models/Qwen3.8-27B-NVFP4",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Transcribe all text in this image."},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
        ],
    }],
    max_tokens=256,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.usage.prompt_tokens_details)
print(resp.choices[0].message.content)

Tool calling

--tool-call-parser qwen3_coder is what turns the checkpoint's <function=…> block into a structured tool_calls array. Without it, a harness sees raw text.

A get_weather schema with "What is the weather in Tokyo in celsius?" returned finish_reason: "tool_calls" and {"city": "Tokyo", "unit": "celsius"} in 0.25 s.

---

Concurrency, prefill, and prompt cache

Continuous batching is already on (2048-token prefill chunks, overlapping decode). Whether a second request runs or queues is only a memory-pool question.

Two ceilings

KV already allows 3 × 262K once the cache is FP8. The 17-request wall is the state pool. That is the official cookbook warning: default --mamba-full-memory-ratio 0.9 over-provisions KV and silently clamps concurrency.

State tax per running request with MTP (S=5, D=4): about 1.39 GB at FP32, 0.71 GB at BF16. --mamba-ssm-dtype bfloat16 is the first lever if you want ~30–35 running instead of 17. Raising the ratio (or pinning --max-mamba-cache-size) for 4–8K traffic can push admission into the 50s; drop MTP if you care about admission count more than single-stream tok/s (the intermediate GDN scratch is ~10 GB). CUDA graphs are only captured up to the current max_running_requests — raise the cap and restart so graphs recapture.

Short-request compute will saturate well before 50 streams. Two simultaneous 262K prefills already share the SMs.

Queued vs parallel at 262K

Unique-prefix /generate requests, max_new_tokens=4, official FP8 + MTP:

So: max running at native 262K is 1 on BF16 KV, 3 on FP8 KV. A second 262K job on BF16 KV is accepted and finishes only after the first releases tokens. After --kv-cache-dtype fp8_e4m3, both stay resident and prefill together.

That is not a 90 s 2× speedup. Prefill is compute-bound; one 262K fill is ~90 s, two sharing the GPU take ~200 s. Makespan is similar to the serialized 181 s. What you buy is mixed traffic: one request can decode while another prefills.

Prefill rates

--chunked-prefill-size 2048 is the official SM120 recipe. 262K is 128 chunks. Instantaneous input throughput on a 262K fill started near 3,800 tok/s and fell to ~2,700 tok/s as the 16 attention layers grew.

Those long-context numbers match other Qwen3.8 / 3.6 PRO 6000 reports (vLLM BF16 prefill of 2048 tokens at 248K context is ~2,440 tok/s). They are not a hang. NVFP4 + checkpoint-declared FP8 KV is faster at 100–200K than official FP8 on BF16 KV.

Prompt cache

Radix cache is on (UnifiedRadixCache, disable_radix_cache=False). Hybrid GDN only records prefix nodes every mamba_track_interval=256 tokens. A ~265-token shared chat system prompt misses. A ~1k–2.4k-token shared /generate prefix hits.

/v1/chat/completions omits usage.prompt_tokens_details.cached_tokens unless you start with --enable-cache-report. The native /generate meta_info.cached_tokens field is populated without that flag. Scheduler logs (#cached-token) are the other source of truth.

Measured /generate probe (same long prefix twice, then an unrelated prefix):

Use a prefix well above 256 tokens. Prometheus /metrics is 404 unless you pass --enable-metrics.

---

Performance reference

All rows: SGLang 0.5.16, RTX PRO 6000 (96 GB), random dataset, isl=1024 / osl=128, 32 prompts, --request-rate inf, --flush-cache, thinking off. Output tok/s is sglang.bench_serving's generated-token throughput.

NVFP4 + MTP (recommended)

NVFP4 + DSpark (after lm_head patch)

FP8 + MTP (same card, auto KV = BF16)

NVFP4 is not just smaller weights. It is also the FP8 KV declaration, which is why the same --mem-fraction-static 0.85 yields a 960K-token pool instead of 401K, and why decode is ~30% faster at every concurrency we measured. Official FP8 reaches 802K tokens (3 × 262K) if you add --kv-cache-dtype fp8_e4m3; that change is about admission at long context, not the 1024/128 decode table above.

Against the 5090 no-spec baseline

The 5090 article measured ~52 tok/s single-stream on the same NVFP4 checkpoint with no speculative decoding. MTP on this 96 GB card is about 2.6× that single-stream number. The comparison is not clean (different GPU, different spec setting), but it is the reason this follow-up exists: the extra VRAM is how you turn spec back on.

Run your own benchmark

python3 -m sglang.bench_serving \
  --backend sglang-oai \
  --host 127.0.0.1 --port 18000 \
  --model /workspace/models/Qwen3.8-27B-NVFP4 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 128 \
  --random-range-ratio 1 \
  --num-prompts 32 \
  --max-concurrency 4 \
  --request-rate inf \
  --flush-cache

Newer SGLang builds prefer python3 -m sglang.benchmark.serving with the same flags. Run a short warmup chat request first so CUDA graphs are captured before you time anything.

---

OpenAI API compatibility

The server exposes a standard OpenAI-compatible surface. Verified against the NVFP4 + MTP deployment above.

Chat completions

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:18000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="/workspace/models/Qwen3.8-27B-NVFP4",
    messages=[{"role": "user", "content": "Write a haiku about neural networks."}],
    max_tokens=256,
)
print(resp.choices[0].message.content)

If you served from a local directory, the model string is that path (or whatever --served-model-name you set). /v1/models is the source of truth.

Reasoning / thinking

Thinking is on by default. Disable per request:

resp = client.chat.completions.create(
    model="/workspace/models/Qwen3.8-27B-NVFP4",
    messages=[{"role": "user", "content": "What is 17 + 25?"}],
    max_tokens=100,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

With --reasoning-parser qwen3, the trace lands in message.reasoning / reasoning_content and the answer in message.content.

Tool calling

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["city"],
        },
    },
}]

resp = client.chat.completions.create(
    model="/workspace/models/Qwen3.8-27B-NVFP4",
    messages=[{"role": "user", "content": "What is the weather in Tokyo in celsius?"}],
    tools=tools,
    tool_choice="auto",
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
# finish_reason: "tool_calls"
# tool_calls[0].function.name == "get_weather"

---

Run as a persistent service

Do not rely on a foreground shell. The test environment used Supervisor; the same wrapper works under systemd.

Supervisor

/usr/local/bin/qwen3-8-27b-pro6000.sh:

#!/bin/bash
set -euo pipefail

export HF_HOME="${HF_HOME:-/workspace/.hf_home}"
export PYTORCH_CUDA_ALLOC_CONF="${PYTORCH_CUDA_ALLOC_CONF:-expandable_segments:True}"

MODEL="${QWEN38_MODEL:-/workspace/models/Qwen3.8-27B-NVFP4}"
PORT="${QWEN38_PORT:-18000}"

exec sglang serve \
  --trust-remote-code \
  --model-path "${MODEL}" \
  --mem-fraction-static 0.85 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --host 127.0.0.1 \
  --port "${PORT}"

/etc/supervisor/conf.d/qwen38-27b.conf:

[program:qwen38_27b]
command=/usr/local/bin/qwen3-8-27b-pro6000.sh
directory=/workspace
autostart=true
autorestart=true
startsecs=300
stopasgroup=true
killasgroup=true
stdout_logfile=/var/log/qwen3-8-27b.log
redirect_stderr=true

startsecs=300 avoids a false FATAL while weights load and CUDA graphs capture.

systemd equivalent

[Unit]
Description=Qwen3.8-27B-NVFP4 Server (SGLang, RTX PRO 6000)
After=network.target

[Service]
Type=simple
Environment=HF_HOME=/workspace/.hf_home
ExecStart=/usr/local/bin/qwen3-8-27b-pro6000.sh
Restart=on-failure
RestartSec=15
KillMode=mixed
TimeoutStopSec=60
TimeoutStartSec=600

[Install]
WantedBy=multi-user.target

---

Access the API

Same machine

curl http://127.0.0.1:18000/v1/models

Remote via SSH tunnel

ssh -L 18000:127.0.0.1:18000 user@your-server.example.com

Then on your laptop:

curl http://127.0.0.1:18000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"/workspace/models/Qwen3.8-27B-NVFP4","messages":[{"role":"user","content":"Hello"}],"max_tokens":64,"chat_template_kwargs":{"enable_thinking":false}}'

Keep the server on 127.0.0.1. Terminate TLS and auth in Caddy/nginx if you need a public URL — SGLang does not ship production-grade API authentication.

Public mapped port (Vast-style Caddy)

On the test instance the platform Caddy already listened on the mapped container port (:8000) and reverse-proxied to 127.0.0.1:18000. Do not bind SGLang to 0.0.0.0 and do not move it to an unmapped port such as 18000.

Caddy required a Bearer token (the same Open / Jupyter token the portal prints). Without it the proxy returns 401; SGLang itself is unchanged.

export TOKEN='<portal-token>'
export BASE='http://<public-ip>:<mapped-port-for-8000>'

curl -sS "$BASE/v1/models" -H "Authorization: Bearer $TOKEN"

curl -sS "$BASE/v1/chat/completions" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/workspace/models/Qwen3.8-27B-FP8",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 64,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Use whichever --served-model-name / local path the live /v1/models lists. A Cloudflare quick tunnel in front of the same Caddy port, if the portal provides one, is the TLS variant of this path.

---

Troubleshooting

DSpark mat1 and mat2 shapes cannot be multiplied (7x5120 and 2560x248320)

The NVFP4 lm_head is packed. SGLang 0.5.16's dense DSpark path still does hidden @ weight.T. Use lmsysorg/sglang:qwen38-27b, or patch compute_base_logits to call quant_method.apply when the target head is quantized. --disable-decode-cuda-graph is not a workaround.

The DSpark draft falls back to transformers remote code

Two local edits on the draft tree:

1. architectures must be ["Qwen3DSparkModel"], not DSparkDraftModel.
2. If auto_map imports specforge and that package is missing, either pip install specforge --no-deps (enough to pass the import gate) or drop the remote auto_map after the architecture remap.

extra_buffer_lazy unsupported with spec

Do not combine --mamba-radix-cache-strategy extra_buffer_lazy with MTP or DSpark on this release. Use the default extra_buffer.

Hybrid (mamba/linear-attention) state cache is too small to serve any requests

Rare on 96 GB with the flags in this article. If you copied the 5090 long-context knobs (--max-mamba-cache-size 20) onto this card you can recreate it. Raise the state pool, or add --mamba-ssm-dtype bfloat16. Do not disable radix cache just to lower S unless you also want to lose prefix hits.

max_running_requests was reduced from 48 to 17

Not an OOM. Default mamba-full-memory-ratio 0.9 left 85 GDN slots; S=5 → 17 running. KV can still hold three 262K requests. To admit more short requests: --mamba-ssm-dtype bfloat16, then raise --mamba-full-memory-ratio or pin --max-mamba-cache-size. Restart so CUDA graphs recapture the new bs list.

Second 262K request is slow / looks stuck

On official FP8 with auto KV the second job is queued (401K BF16 tokens < 2×262K). Prefill of one 262K window is ~90 s of 2048-token chunks at ~3k tok/s — GPU at 100%, not a hang. Add --kv-cache-dtype fp8_e4m3 for real overlap (both ~200 s together).

Shared system prompt shows #cached-token: 0

Expected if the shared prefix is under 256 tokens (mamba_track_interval). Probe with /generate and a 1k+ prefix; check meta_info.cached_tokens. Chat usage stays empty without --enable-cache-report.

speculative_num_draft_tokens must equal gamma + 1

DSpark derives draft tokens from the checkpoint gamma (7 → 8). You cannot pass --speculative-num-draft-tokens 4 to save VRAM.

Garbled generations after loading an "NVFP4" tree

You are probably on a compressed-tensors / Unsloth export, not RadixArk/Qwen3.8-27B-NVFP4. Delete it and download the official ModelOpt checkpoint.

Input length exceeds the maximum allowed length

A clean rejection at the KV / context cap, not an OOM. On this card the cap should be 262,144 unless you set --max-total-tokens yourself.

Slow first request after restart

Expected — CUDA graphs warm on first inference. Run a warmup before load testing.

Orphaned process holding VRAM

ps -eo pid,args | grep -E 'sglang serve|sglang::scheduler' | grep -v grep
kill <pid>
nvidia-smi

---

Quick reference

# Download
export HF_HOME=/workspace/.hf_home
hf download RadixArk/Qwen3.8-27B-NVFP4 \
  --local-dir /workspace/models/Qwen3.8-27B-NVFP4

# Run (recommended: NVFP4 + MTP)
sglang serve \
  --trust-remote-code \
  --model-path /workspace/models/Qwen3.8-27B-NVFP4 \
  --mem-fraction-static 0.85 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --host 127.0.0.1 \
  --port 18000

# Official FP8 + MTP + FP8 KV (3× native 262K in the KV pool)
sglang serve \
  --trust-remote-code \
  --model-path /workspace/models/Qwen3.8-27B-FP8 \
  --mem-fraction-static 0.85 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --kv-cache-dtype fp8_e4m3 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --host 127.0.0.1 \
  --port 18000

# Health check
curl -s http://127.0.0.1:18000/v1/models

# Chat
curl http://127.0.0.1:18000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"/workspace/models/Qwen3.8-27B-NVFP4","messages":[{"role":"user","content":"Hello"}],"max_tokens":64,"chat_template_kwargs":{"enable_thinking":false}}'

# Benchmark
python3 -m sglang.bench_serving \
  --backend sglang-oai \
  --host 127.0.0.1 --port 18000 \
  --model /workspace/models/Qwen3.8-27B-NVFP4 \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 128 \
  --random-range-ratio 1 \
  --num-prompts 32 --max-concurrency 4 --request-rate inf --flush-cache

---

Further reading

• SGLang cookbook: Qwen3.8-27B (RTX PRO 6000 / NVFP4 / balanced)
• Qwen3.8-27B-NVFP4 checkpoint
• Qwen3.8-27B-DSpark draft checkpoint
• Qwen3.8-27B model card
• Qwen3.8-27B on a Single RTX 5090 — the 32 GB companion: 200K context, no DSpark
• Deploying Qwen3.6-27B-FP8 with vLLM — previous-generation dense 27B on vLLM, including a 262K profile on this same GPU class
• Deploying Qwen3 Embedding — companion guide for the RAG embedding step
• Deploying Qwen3 Reranker — companion guide for the RAG reranking step