Qwen3.8-27B on a Single RTX 5090: Full-Context SGLang Deployment

Qwen3.8-27B on a Single RTX 5090: Full-Context SGLang Deployment

This guide walks through serving RadixArk/Qwen3.8-27B-NVFP4 with SGLang on a single NVIDIA RTX 5090 (32 GB). It is based on a real hands-on deployment: the memory math behind the model's hybrid Gated Delta Network architecture, the one flag that decides throughput vs. context, pushing the context window out to ~200K tokens, benchmarked throughput at 1/2/4 concurrent requests, and a reality check on why DSpark speculative decoding does not fit on a 32 GB card with this SGLang release.

The model is dense, 27B, text + vision capable, and ships an NVFP4 (W4A4) + FP8 checkpoint that was designed for exactly this class of consumer Blackwell card. If you have ever been confused by --mamba-full-memory-ratio or why a hybrid Mamba/attention model "won't serve any requests" despite free VRAM, this article explains it from first principles and shows the numbers.

---

Table of contents

1. Overview
2. Prerequisites
3. Hardware and checkpoint choices
4. Understanding the memory model
5. Install SGLang
6. Obtain the model weights
7. Run the server (official recipe)
8. Context length tuning to 200K
9. Performance reference
10. Speculative decoding: DSpark reality check
11. OpenAI API compatibility
12. Run as a persistent service
13. Access the API
14. Troubleshooting
15. Quick reference
16. Further reading

---

Overview

Architecture:

Client ──► SGLang (OpenAI API) ──► Qwen3.8-27B-NVFP4 ──► RTX 5090 (32 GB)

For remote development:

Your laptop ──SSH tunnel──► server:127.0.0.1:30000 ──► SGLang

The checkpoint's declared kv_cache_quant_algo: FP8 matters more than usual: with the default --kv-cache-dtype auto, SGLang runs the attention KV pool in fp8_e4m3, which is what makes the long-context numbers in this article possible on a 32 GB card.

---

Prerequisites

• A Linux machine with an NVIDIA GPU and a working driver (nvidia-smi succeeds).
• Python 3.12 and a recent PyTorch/CUDA stack for Blackwell (CUDA ≥ 12.8; we used CUDA 13.0).
• 32 GB GPU VRAM — this whole article is about making the 27B fit on exactly that.
• ~30 GB disk for the checkpoint download (~21.4 GB) plus SGLang/PyTorch compile caches.
• Network access to Hugging Face for the first download.

Verify the GPU before anything else:

nvidia-smi
python3 -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"
# torch 2.11.0+cu130 True NVIDIA GeForce RTX 5090

Note that this is an unprivileged container-like environment: no kernel modules, no Docker-in-Docker. Long-running services are managed by a supervisor, which is how the persistent-service section below is framed. The techniques apply unchanged to systemd or a plain foreground process.

---

Hardware and checkpoint choices

The NVFP4 checkpoint is the one designed for this card. Loaded on device it reports:

Reference hardware (benchmarks in this guide)

SM120/SM121 note: this Blackwell consumer card needs the flashinfer attention backend (--attention-backend flashinfer). trtllm_mha is SM100-only. MTP with FlashInfer requires a FlashInfer build newer than 0.6.15.post1; otherwise spec decoding falls back to --attention-backend triton.

---

Understanding the memory model

The single most important thing to understand about this model on a 32 GB card is that it is not a plain transformer. Qwen3.8-27B is a hybrid Gated Delta Network: 64 layers laid out as 16 repeats of 3 × (Gated DeltaNet → FFN) then 1 × (Gated Attention → FFN) — 48 linear-attention layers and 16 full-attention layers, hidden size 5120.

This split creates two independent memory pools after weights:

• A worst-case-reserved GDN state pool — one fixed-size state per running request slot. This pool sets the concurrency ceiling.
• A paged attention KV pool — sized per token, like a normal transformer.

--mamba-full-memory-ratio divides post-weight memory between them.

The per-request cost formula

SGLang's cookbook computes the balanced ratio as a per-request cost ratio:

ratio = (S + D) × state_bytes / (L × kv_bytes_per_token)

• S — state slots per running request: extra_buffer (default) =5, extra_buffer_lazy =4, no_buffer =3, radix cache disabled =1.
• D — extra state slots for speculative decoding intermediate states (0 without spec decoding).
• state_bytes — one GDN state slot: 153.9 MB at FP32, 78.4 MB at BF16.
• kv_bytes_per_token — 16 attention layers × GQA 4 × 256 × K+V: 32.8 KB at FP8, 65.5 KB at BF16.
• L — average request length in tokens (input + output).

The official recipe value 4.59 corresponds to L=5120, S=5, D=0, FP32 state, FP8 KV:

5 × 153.9 MB / (5120 × 32.8 KB) ≈ 4.58

--mamba-full-memory-ratio 4.59 routes ~82% of post-weight memory to the state pool and ~18% to KV. That is the balanced point where both pools exhaust at roughly the same concurrency.

Why the default can silently clamp concurrency to zero

The flag's default is 0.9, which under-provisions the state pool for short-to-medium requests. On a 32 GB card with the 27B loaded, that produced:

RuntimeError: Hybrid (mamba/linear-attention) state cache is too small to serve
any requests. max_mamba_cache_size=4, mamba_ratio=5, resulting max_num_reqs=0.

Only 4 state slots were available while each request needs 5 (S=5). The error message's remedies are exactly the right levers, in the right order:

1. Reduce --max-running-requests.
2. Increase --mem-fraction-static (give the pools more of the card).
3. Or, best for small VRAM: lower S — --mamba-radix-cache-strategy extra_buffer_lazy (S=4) or --disable-radix-cache (S=1).

The FP32 state is the memory hog

The default GDN state dtype is FP32 (153.9 MB/slot). Halving it is the cheapest single change on a small card:

--mamba-ssm-dtype bfloat16

This halves state_bytes to 78.4 MB/slot and nearly doubles the concurrency you can get from the same state budget. The official 5090 recipe omits it because at the balanced ratio it does not matter for throughput; it matters when you want more slots or longer context (below).

What actually uses VRAM

---

Install SGLang

With the environment activated:

pip install --upgrade pip
pip install uv
uv pip install sglang

Or, if your runtime already bundles SGLang (as the test environment did), verify the version and CUDA stack:

python3 -c "import sglang; print(sglang.__version__)"
python3 -c "import torch; print(torch.__version__, torch.version.cuda)"
# 0.5.16
# 2.11.0+cu130 cuda 13.0

Set a cache directory that survives restarts and can hold ~25 GB:

export HF_HOME=/workspace/.hf_home
mkdir -p "$HF_HOME"

---

Obtain the model weights

SGLang pulls from Hugging Face on first --model-path use, but pre-downloading keeps startup predictable and lets you watch the transfer speed:

export HF_HOME=/workspace/.hf_home
hf download RadixArk/Qwen3.8-27B-NVFP4 --max-workers 8

The NVFP4 checkpoint is ~21.4 GB across 22 files. On the test link the transfer ran at ~64 MB/s initially and ~137 MB/s at peak, completing in about five minutes. If you have a token, set HF_TOKEN for higher rate limits, but never paste it into logs or history.

The draft model used in the DSpark section is a separate, much smaller checkpoint:

hf download RadixArk/Qwen3.8-27B-DSpark --max-workers 4
# ~1.04 GB, single safetensors

Verify both landed:

du -sh "$HF_HOME/hub"/models--RadixArk--Qwen3.8-27B-*

---

Run the server (official recipe)

The official recommended RTX 5090 configuration is short and surprisingly boring — no speculative decoding, no exotic flags. This is the sweet spot:

sglang serve \
  --trust-remote-code \
  --model-path RadixArk/Qwen3.8-27B-NVFP4 \
  --mem-fraction-static 0.85 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --mamba-full-memory-ratio 4.59 \
  --host 0.0.0.0 \
  --port 30000

What each flag is doing:

Startup log, decoded

On the 32 GB card the tuned boot sequence looks like this:

Load weight end. ... type=Qwen3_5ForConditionalGeneration, quant=modelopt_mixed,
  avail mem=10.57 GB, mem usage=20.14 GB.
Mamba Cache is allocated. max_mamba_cache_size: 33, conv_state size: 0.09GB,
  ssm_state size: 4.78GB
KV Cache is allocated. dtype: torch.float8_e4m3fn, #tokens: 38688,
  K size: 0.59 GB, V size: 0.59 GB
Memory pool end. avail mem=4.47 GB
Capture target decode CUDA graph ... end. mem usage=0.10 GB, avail mem=3.86 GB.
The server is fired up and ready to roll!

Note dtype: torch.float8_e4m3fn — the FP8 KV pool is picked up automatically from the checkpoint's declared kv_cache_quant_algo, no flag needed. And max_mamba_cache_size: 33 means ~6 concurrent requests (33 // 5).

First startup downloads weights (if needed) and captures CUDA graphs; expect a few minutes before /v1/models responds.

---

Context length tuning to 200K

The model's native context is 262K, but on a 32 GB card the KV pool is the wall. With the official recipe the pool is only 38,688 tokens — a single request over ~38.6K returns a clean 400:

Input length (39152 tokens) exceeds the maximum allowed length (38682 tokens).

The failure is clean, not an OOM — the card stays healthy. To go bigger, you have to move memory from the state pool to the KV pool:

sglang serve \
  --trust-remote-code \
  --model-path RadixArk/Qwen3.8-27B-NVFP4 \
  --mem-fraction-static 0.95 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --max-total-tokens 200000 \
  --max-mamba-cache-size 20 \
  --mamba-ssm-dtype bfloat16 \
  --host 0.0.0.0 \
  --port 30000

The three changes that unlock 200K:

With this config the boot log shows the KV pool at the cap:

Mamba Cache is allocated. max_mamba_cache_size: 20, ... ssm_state size: 1.48GB
KV Cache is allocated. dtype: torch.float8_e4m3fn, #tokens: 200000,
  K size: 3.05 GB, V size: 3.05 GB
Memory pool end. avail mem=2.87 GB
The server is fired up and ready to roll!

Verified context sweep

Single-request context test, max_tokens=4:

~200K context works, no OOM, and the server stays stable at ~31.1/32.6 GB. The trade-off is concurrency: the state pool drops to ~4 concurrent requests (20 slots ÷ 5). A single long-context request also occupies the whole KV pool.

Push past 200K by raising --max-total-tokens and trimming the state pool further, but 200K is close to the practical limit on this card.

---

Performance reference

All benchmarks: RadixArk/Qwen3.8-27B-NVFP4, SGLang 0.5.16, RTX 5090 (32 GB), official recipe parameters, FP8 KV cache, no speculative decoding. Workload: random dataset, isl=1024 / osl=128, 100 prompts, --request-rate inf, --flush-cache.

Throughput at 1/2/4 concurrency

Latency at 1/2/4 concurrency

Key observations:

• Throughput scales almost linearly with concurrency (470 → 885 → 1,386 tok/s). Single-stream decode runs at ~52–55 tok/s (~18.5 ms/token).
• TTFT degrades at concurrency 4 (mean 796 ms, P99 ~3.1 s) — the first sign of the state pool becoming the bottleneck. This matches the model: the GDN state pool caps you around 4–6 concurrent requests.
• All 100/100 requests succeeded in every run.

Run your own benchmark

python3 -m sglang.bench_serving \
  --backend sglang-oai \
  --base-url http://127.0.0.1:30000 \
  --model RadixArk/Qwen3.8-27B-NVFP4 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 128 \
  --random-range-ratio 1 \
  --num-prompts 100 \
  --max-concurrency 4 \
  --request-rate inf \
  --flush-cache

---

Speculative decoding: DSpark reality check

SGLang's cookbook offers DSpark as an alternative to the in-checkpoint MTP head: a trained draft model shipped as a separate checkpoint.

--speculative-algorithm DSPARK \
--speculative-draft-model-path RadixArk/Qwen3.8-27B-DSpark

We attempted this on the 32 GB card. It does not work there. This section documents why, so you do not burn the same few hours.

What worked

The draft model loads fine. It is a 1.36B-parameter DFlash+Markov+confidence network (gamma=7, so verify_num_draft_tokens=8, markov_head=VanillaMarkov), ~2.71 GB in BF16. With two small environment fixes it loads inside SGLang's native DSpark worker:

Initialized DSpark draft runner. attention_backend=flashinfer,
  model=Qwen3DSparkModel, gamma=7, verify_num_draft_tokens=8,
  mask_token_id=248077, markov_head=VanillaMarkov

Two gotchas surfaced along the way:

1. The draft checkpoint's remote code imports specforge (from specforge.modeling.draft.dflash import DFlashDraftModel), which is not installed. Installing the pure-Python specforge wheel with pip install specforge --no-deps satisfies transformers' check_imports gate. (The model's own dflash.py is a vendored copy of the same class, so no heavy dependency tree is needed.)
2. The draft config declares architectures: ["DSparkDraftModel"], which is not a key in SGLang's model registry (it registers Qwen3DSparkModel), so SGLang falls back to transformers remote code and fails. Renaming the architecture entry to Qwen3DSparkModel in the local config.json makes SGLang resolve its native class.

The memory wall

DSPARK fixes verify_num_draft_tokens = gamma + 1 = 8. SGLang enforces this:

ValueError: DSpark speculative_num_draft_tokens must equal gamma + 1 (= 8 for
gamma=7), but got speculative_num_draft_tokens=4.

Those 8 draft positions each hold an intermediate GDN state. At BF16 (78.4 MB) that is ~627 MB of intermediate state per running request, on top of the draft's 2.71 GB of weights. The accounting on a 32 GB card:

The server boots and even captures graphs, then the first chat request dies:

Internal server error: CUDA error: out of memory

The final blocker: target lm_head shape mismatch

Even with memory trimmed, CUDA-graph capture of the draft verify path fails with:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (28x5120 and 2560x248320)

compute_base_logits in SGLang's dense DSpark path computes hidden @ lm_head.weight.T. The target's lm_head expects a 2560-dim input (this checkpoint uses a bottlenecked LM head), but the draft feeds the model's 5120-dim hidden states. This is a genuine incompatibility between the dense DSpark draft path in SGLang 0.5.16 and this target checkpoint's head structure — it needs a newer SGLang that understands the target's LM head pipeline.

Verdict for this hardware

• DSpark on 32 GB with Qwen3.8-27B-NVFP4 and SGLang 0.5.16: not usable. The draft is too expensive in VRAM and the LM-head mismatch blocks the draft verify path.
• DSpark is a plausible option on RTX PRO 6000 / H200 / DGX Spark-class memory budgets with a matching newer SGLang build — the draft model itself loads and runs correctly. The follow-up Qwen3.8-27B on RTX PRO 6000 is that deployment: native 262K context, MTP at ~134 tok/s, and DSpark after the NVFP4 lm_head patch.
• On this card, the official non-spec recipe (with MTP via the in-checkpoint head if you have the newer FlashInfer) is the right operating point.

---

OpenAI API compatibility

The server exposes a standard OpenAI-compatible surface. Verified against the deployment above.

Chat completions

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="RadixArk/Qwen3.8-27B-NVFP4",
    messages=[{"role": "user", "content": "Write a haiku about neural networks."}],
    max_tokens=256,
)
print(resp.choices[0].message.content)

Reasoning / thinking

Thinking is on by default for this checkpoint. Disable per request with the chat template:

resp = client.chat.completions.create(
    model="RadixArk/Qwen3.8-27B-NVFP4",
    messages=[{"role": "user", "content": "What is 17 + 25?"}],
    max_tokens=100,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

With --reasoning-parser qwen3, the thinking trace lands in message.reasoning and the answer in message.content.

Tool calling

The --tool-call-parser qwen3_coder flag on the command line is what makes tool calls come back structured. Without it, a harness receives the raw <function=…> text instead of a parsed tool_calls array.

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
        },
    },
}]

resp = client.chat.completions.create(
    model="RadixArk/Qwen3.8-27B-NVFP4",
    messages=[{"role": "user", "content": "What is the weather in Paris?"}],
    tools=tools,
    tool_choice="auto",
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
# finish_reason: "tool_calls"
# tool_calls[0].function.name == "get_weather"

---

Run as a persistent service

Do not rely on a foreground shell. The test environment used Supervisor; the same wrapper works under systemd.

Supervisor

/usr/local/bin/qwen3-8-27b.sh:

#!/bin/bash
set -euo pipefail

export HF_HOME="${HF_HOME:-/workspace/.hf_home}"
export PYTORCH_CUDA_ALLOC_CONF="${PYTORCH_CUDA_ALLOC_CONF:-expandable_segments:True}"

MODEL="${QWEN38_MODEL:-RadixArk/Qwen3.8-27B-NVFP4}"
PORT="${QWEN38_PORT:-18000}"
MAX_TOKENS="${QWEN38_MAX_TOKENS:-200000}"
MAMBA_SLOTS="${QWEN38_MAMBA_SLOTS:-20}"

exec sglang serve "${MODEL}" \
  --trust-remote-code \
  --host 127.0.0.1 \
  --port "${PORT}" \
  --mem-fraction-static 0.95 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --max-total-tokens "${MAX_TOKENS}" \
  --max-mamba-cache-size "${MAMBA_SLOTS}" \
  --mamba-ssm-dtype bfloat16

/etc/supervisor/conf.d/qwen38-27b.conf:

[program:qwen38_27b]
command=/usr/local/bin/qwen3-8-27b.sh
directory=/workspace
autostart=true
autorestart=true
startsecs=300
stopasgroup=true
killasgroup=true
stdout_logfile=/var/log/qwen3-8-27b.log
redirect_stderr=true

startsecs=300 avoids false FATAL states while the model loads and CUDA graphs capture.

systemd equivalent

[Unit]
Description=Qwen3.8-27B-NVFP4 Server (SGLang)
After=network.target

[Service]
Type=simple
Environment=HF_HOME=/workspace/.hf_home
ExecStart=/usr/local/bin/qwen3-8-27b.sh
Restart=on-failure
RestartSec=15
KillMode=mixed
TimeoutStopSec=60
TimeoutStartSec=600

[Install]
WantedBy=multi-user.target

---

Access the API

Same machine

curl http://127.0.0.1:18000/v1/models

Remote via SSH tunnel

ssh -L 18000:127.0.0.1:18000 user@your-server.example.com

Then on your laptop:

curl http://127.0.0.1:18000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"RadixArk/Qwen3.8-27B-NVFP4","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

---

Troubleshooting

Hybrid (mamba/linear-attention) state cache is too small to serve any requests

The state pool has fewer slots than each request needs (max_mamba_cache_size=4, mamba_ratio=5, max_num_reqs=0). Fix in order:

1. Add --mamba-ssm-dtype bfloat16 (halves slot size).
2. Raise --mem-fraction-static (0.95).
3. Lower S: --mamba-radix-cache-strategy extra_buffer_lazy (S=4) or --disable-radix-cache (S=1).
4. Lower --max-running-requests to match what the state pool can serve.

Do not add --disable-radix-cache if you need prefix caching; it kills both KV and state radix caching.

extra_buffer_lazy unsupported with spec

--mamba-radix-cache-strategy extra_buffer_lazy is incompatible with speculative decoding in this SGLang build. Use the default extra_buffer or --disable-radix-cache instead.

CUDA out of memory on the first request

The pools are sized to fill the card and leave ~0.2–0.9 GB free, which is not enough for forward-pass transients. Reduce pool pressure: lower --max-total-tokens, shrink --max-mamba-cache-size, or (counterintuitively) lower --mem-fraction-static so the pools leave explicit headroom.

speculative_num_draft_tokens must equal gamma + 1

For DSpark the draft token count is derived from the checkpoint's gamma (block_size) and cannot be lowered. Do not try to pass a smaller --speculative-num-draft-tokens to save VRAM; it fails validation.

DSpark mat1 and mat2 shapes cannot be multiplied (28x5120 and 2560x248320)

The target's lm_head expects 2560-dim input; SGLang 0.5.16's dense DSpark path feeds 5120-dim hidden states. This is a version/checkpoint incompatibility — try a newer SGLang or a bigger GPU with a matching build.

Input length exceeds the maximum allowed length

A clean rejection at the KV pool cap, not an OOM. Raise --max-total-tokens (and free memory from the state pool) or enable --allow-auto-truncate.

Slow first request after restart

Expected — SGLang warms CUDA graphs on first inference. Run a warmup request before load testing.

Orphaned process holding VRAM

ps -eo pid,args | grep -E 'sglang serve|sglang::scheduler' | grep -v grep
kill <pid>
nvidia-smi

---

Quick reference

# Install
pip install --upgrade pip && pip install uv
uv pip install sglang

# Download weights
export HF_HOME=/workspace/.hf_home
hf download RadixArk/Qwen3.8-27B-NVFP4 --max-workers 8

# Run (official RTX 5090 recipe)
sglang serve \
  --trust-remote-code \
  --model-path RadixArk/Qwen3.8-27B-NVFP4 \
  --mem-fraction-static 0.85 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --mamba-full-memory-ratio 4.59 \
  --host 0.0.0.0 \
  --port 30000

# 200K-context variant
sglang serve \
  --trust-remote-code \
  --model-path RadixArk/Qwen3.8-27B-NVFP4 \
  --mem-fraction-static 0.95 \
  --attention-backend flashinfer \
  --chunked-prefill-size 2048 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --max-total-tokens 200000 \
  --max-mamba-cache-size 20 \
  --mamba-ssm-dtype bfloat16 \
  --host 0.0.0.0 \
  --port 30000

# Health check
curl -s http://127.0.0.1:30000/v1/models

# Chat
curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"RadixArk/Qwen3.8-27B-NVFP4","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

# Benchmark
python3 -m sglang.bench_serving \
  --backend sglang-oai \
  --base-url http://127.0.0.1:30000 \
  --model RadixArk/Qwen3.8-27B-NVFP4 \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 128 \
  --random-range-ratio 1 \
  --num-prompts 100 --max-concurrency 4 --request-rate inf --flush-cache

---

Further reading

• SGLang cookbook: Qwen3.8-27B
• Qwen3.8-27B-NVFP4 checkpoint
• Qwen3.8-27B-DSpark draft checkpoint
• Qwen3.8-27B model card
• Qwen3.8-27B on RTX PRO 6000 — the 96 GB follow-up: native 262K context, MTP vs DSpark
• Deploying Qwen3.6-27B-FP8 with vLLM — the vLLM companion guide for the same GDN model family
• Deploying Qwen3 Embedding — companion guide for the RAG embedding step
• Deploying Qwen3 Reranker — companion guide for the RAG reranking step