Deploying Qwen3 Reranker with vLLM

Deploying Qwen3 Reranker with vLLM

This guide walks through serving Qwen3-Reranker on a Linux GPU server using vLLM, with score and rerank HTTP APIs. It covers the required model configuration, local development, production persistence, client integration, co-hosting with an embedding server, and troubleshooting.

The default setup binds to localhost only. Access from other machines is via SSH tunnel or a reverse proxy you control.

For the embedding step in a RAG pipeline, see Deploying Qwen3 Embedding.

---

Table of contents

1. Overview
2. Prerequisites
3. Hardware recommendations
4. Server setup
5. Install vLLM
6. Obtain model weights
7. Run the reranker server
8. Run as a persistent service
9. Access the API
10. Using the API correctly
11. Co-hosting with embedding on one GPU
12. Tuning and operations
13. Troubleshooting
14. Optional: expose externally
15. Docker: two-container RAG stack

---

Overview

Architecture (RAG pipeline):

Documents ──► Embedding server ──► vector search ──► top-K candidates
                                                          │
Query ────────────────────────────────────────────────────┘
                                                          ▼
                                              Reranker server ──► reordered results

Single reranker service:

Client ──► vLLM (score/rerank API) ──► Qwen3-Reranker ──► GPU

---

Prerequisites

• A Linux machine with an NVIDIA GPU and a working driver (nvidia-smi succeeds).
• Python 3.10+ (3.12 recommended).
• Enough GPU VRAM for the chosen model (see Hardware recommendations).
• Network access to Hugging Face or ModelScope for the first model download. See Obtain model weights if Hugging Face is unreachable.
• Optional: curl, systemd, or Supervisor for production-style deployment.

---

Hardware recommendations

Like embedding models, reranker checkpoints are small but vLLM reserves most of the GPU for KV cache and serving overhead. See Understanding GPU memory usage in the embedding guide for a full breakdown.

Verify the GPU:

nvidia-smi
python3 -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

---

Server setup

SSH into your server (adjust user, host, and port as needed):

ssh user@your-server.example.com

Create an isolated Python environment (can be shared with the embedding server):

python3 -m venv ~/.venvs/qwen3-rerank
source ~/.venvs/qwen3-rerank/bin/activate
pip install -U pip

Or with uv:

uv venv ~/.venvs/qwen3-rerank
source ~/.venvs/qwen3-rerank/bin/activate

Set a Hugging Face cache directory if you want weights stored in a specific path:

export HF_HOME=/var/lib/huggingface
mkdir -p "$HF_HOME"

---

Install vLLM

With the virtual environment activated:

pip install "vllm>=0.8.5"

Or with uv:

uv pip install "vllm>=0.8.5"

Confirm the CLI:

vllm --version

Version note: vLLM 0.23+ uses --convert classify for cross-encoder / reranker models.

---

Obtain model weights

By default, vllm serve Qwen/Qwen3-Reranker-0.6B pulls weights from Hugging Face on first start. If huggingface.co is slow or blocked in your region, download the same checkpoints from ModelScope and point vLLM at the local directory.

ModelScope uses the same model IDs as Hugging Face. The checkpoints are interchangeable. The --hf-overrides in Run the reranker server are still required for the official Qwen/Qwen3-Reranker-* weights regardless of download source.

Option A: ModelScope CLI (recommended when HF is blocked)

source ~/.venvs/qwen3-rerank/bin/activate
pip install modelscope

export MODEL_DIR=/var/lib/models/qwen3-reranker-0.6b
mkdir -p "${MODEL_DIR}"

modelscope download --model Qwen/Qwen3-Reranker-0.6B --local_dir "${MODEL_DIR}"

Then serve from the local path:

vllm serve "${MODEL_DIR}" \
  --convert classify \
  --hf-overrides '{"is_original_qwen3_reranker": true, "classifier_from_token": ["no", "yes"], "method": "from_2_way_softmax"}' \
  --host 127.0.0.1 \
  --port 8001 \
  --dtype auto \
  --max-model-len 8192 \
  --served-model-name qwen3-reranker

Option B: ModelScope Python API

pip install modelscope

from modelscope import snapshot_download

model_dir = snapshot_download(
    "Qwen/Qwen3-Reranker-0.6B",
    cache_dir="/var/lib/models",
)
print(model_dir)  # e.g. /var/lib/models/Qwen/Qwen3-Reranker-0.6B

Use the printed path as the vllm serve model argument (with the same --hf-overrides as above).

Option C: Hugging Face (default)

If your server can reach Hugging Face, skip pre-downloading and pass the HF model ID directly:

vllm serve Qwen/Qwen3-Reranker-0.6B --convert classify --hf-overrides '...' ...

Optional mirrors: set HF_ENDPOINT to a Hugging Face mirror, or pre-download with huggingface-cli download Qwen/Qwen3-Reranker-0.6B.

Offline / air-gapped servers

1. On a machine with network access, download via ModelScope (Option A or B).
2. Copy the full model directory to the target server (rsync, scp, etc.).
3. Start vLLM with the local path on the server.

rsync -avz /var/lib/models/qwen3-reranker-0.6b/ deploy@server:/var/lib/models/qwen3-reranker-0.6b/

For persistent services, set QWEN3_RERANKER_MODEL=/var/lib/models/qwen3-reranker-0.6b in your systemd unit or wrapper script.

---

Run the reranker server

Critical: hf-overrides for the official model

Qwen/Qwen3-Reranker-0.6B on Hugging Face is a generative checkpoint that scores relevance via the "no" / "yes" token logits. vLLM converts it to a sequence-classification head at load time, but you must pass explicit overrides or:

• /v1/score and /v1/rerank are not registered
• Scores are wrong (e.g. stuck at 0.5)

Required overrides:

{
  "is_original_qwen3_reranker": true,
  "classifier_from_token": ["no", "yes"],
  "method": "from_2_way_softmax"
}

Alternative: use a pre-converted checkpoint such as tomaarsen/Qwen3-Reranker-0.6B-seq-cls, which does not need these overrides. This guide focuses on the official Qwen/Qwen3-Reranker-0.6B model.

Quick test (foreground)

source ~/.venvs/qwen3-rerank/bin/activate

vllm serve Qwen/Qwen3-Reranker-0.6B \
  --convert classify \
  --hf-overrides '{"is_original_qwen3_reranker": true, "classifier_from_token": ["no", "yes"], "method": "from_2_way_softmax"}' \
  --host 127.0.0.1 \
  --port 8001 \
  --dtype auto \
  --max-model-len 8192 \
  --served-model-name qwen3-reranker

First startup downloads weights (~1–2 GB for 0.6B) if not already cached, then compiles CUDA kernels. Expect 60–120 seconds before the API is ready (reranker cold start is slower than embedding). Pre-download via ModelScope to avoid Hugging Face at runtime.

Verify the score and rerank routes are registered:

# Should list /v1/score and /v1/rerank in server logs
grep -E "Route:.*(score|rerank)" <your-log-file>

curl -s http://127.0.0.1:8001/v1/models | python3 -m json.tool

Test scoring:

curl -s http://127.0.0.1:8001/v1/score \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-reranker",
    "text_1": "What is the capital of China?",
    "text_2": "The capital of China is Beijing."
  }' | python3 -m json.tool

Expected: a score near 0.95–0.99 for a clearly relevant pair.

Test reranking:

curl -s http://127.0.0.1:8001/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-reranker",
    "query": "What is the capital of China?",
    "documents": [
      "The capital of China is Beijing.",
      "Gravity is a force that attracts two bodies towards each other."
    ]
  }' | python3 -m json.tool

Expected: Beijing document ranked first with a much higher relevance_score than the gravity document.

Useful CLI flags

Environment variables

---

Run as a persistent service

Do not rely on a foreground shell or nohup in production.

Option A: systemd (recommended)

Create a wrapper script at /usr/local/bin/qwen3-reranker.sh:

#!/bin/bash
set -euo pipefail

source /home/deploy/.venvs/qwen3-rerank/bin/activate

export HF_HOME="${HF_HOME:-/var/lib/huggingface}"
export VLLM_LOGGING_LEVEL="${VLLM_LOGGING_LEVEL:-INFO}"

MODEL="${QWEN3_RERANKER_MODEL:-Qwen/Qwen3-Reranker-0.6B}"
PORT="${QWEN3_RERANKER_PORT:-8001}"
GPU_UTIL="${QWEN3_RERANKER_GPU_UTIL:-0.9}"
HF_OVERRIDES='{"is_original_qwen3_reranker": true, "classifier_from_token": ["no", "yes"], "method": "from_2_way_softmax"}'

exec vllm serve "${MODEL}" \
  --convert classify \
  --hf-overrides "${HF_OVERRIDES}" \
  --host 127.0.0.1 \
  --port "${PORT}" \
  --dtype auto \
  --max-model-len 8192 \
  --gpu-memory-utilization "${GPU_UTIL}" \
  --served-model-name qwen3-reranker

sudo chmod +x /usr/local/bin/qwen3-reranker.sh

Create /etc/systemd/system/qwen3-reranker.service:

[Unit]
Description=Qwen3 Reranker Server (vLLM)
After=network.target

[Service]
Type=simple
User=deploy
Group=deploy
Environment=HF_HOME=/var/lib/huggingface
ExecStart=/usr/local/bin/qwen3-reranker.sh
Restart=on-failure
RestartSec=10
KillMode=mixed
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now qwen3-reranker
sudo systemctl status qwen3-reranker
journalctl -u qwen3-reranker -f

Option B: Supervisor

If your environment already uses supervisor:

[program:qwen3_reranker]
command=/usr/local/bin/qwen3-reranker.sh
directory=/home/deploy
user=deploy
autostart=true
autorestart=true
startsecs=120
stopasgroup=true
killasgroup=true
stdout_logfile=/var/log/qwen3-reranker.log
redirect_stderr=true

Note: Set startsecs=120 (not 30). Reranker cold starts routinely take 60–90 seconds; shorter values cause Supervisor to report BACKOFF / FATAL even when the process is still loading.

supervisorctl reread && supervisorctl update
supervisorctl start qwen3_reranker

Option C: Docker (reranker only)

Run a single reranker container with the official vLLM image:

docker run -d --name qwen3-reranker --gpus all --restart unless-stopped \
  -p 8001:8001 \
  -v hf-cache:/data/huggingface \
  -e HF_HOME=/data/huggingface \
  vllm/vllm-openai:v0.23.0 \
  vllm serve Qwen/Qwen3-Reranker-0.6B \
    --convert classify \
    --hf-overrides '{"is_original_qwen3_reranker": true, "classifier_from_token": ["no", "yes"], "method": "from_2_way_softmax"}' \
    --host 0.0.0.0 --port 8001 \
    --dtype auto --max-model-len 8192 \
    --served-model-name qwen3-reranker

Pin the image tag in production (e.g. v0.23.0). The --hf-overrides argument is required for the official Qwen3-Reranker weights.

Option D: Docker Compose (embedding + reranker, recommended)

To run both services on one GPU as two containers, use docs/docker/:

cd docs/docker
cp .env.example .env
docker compose up -d

See Docker: two-container RAG stack below.

Confirm the service is listening

ss -tlnp | grep 8001
# LISTEN 127.0.0.1:8001  users:(("vllm",pid=...,fd=...))

---

Access the API

Same machine

curl http://127.0.0.1:8001/v1/models

Remote machine via SSH tunnel

On your laptop:

ssh -L 8001:127.0.0.1:8001 user@your-server.example.com

Then locally:

curl http://127.0.0.1:8001/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-reranker","query":"hello","documents":["hello world"]}'

---

Using the API correctly

Qwen3-Reranker is instruction-aware, like the embedding model. For best results, format inputs with a task instruction and the Qwen chat template.

Prompt templates

PREFIX = (
    '<|im_start|>system\n'
    'Judge whether the Document meets the requirements based on the Query and the Instruct provided. '
    'Note that the answer can only be "yes" or "no".<|im_end|>\n'
    '<|im_start|>user\n'
)
SUFFIX = '<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'

INSTRUCTION = 'Given a web search query, retrieve relevant passages that answer the query'

def format_query(query: str) -> str:
    return f'{PREFIX}<Instruct>: {INSTRUCTION}\n<Query>: {query}\n'

def format_document(doc: str) -> str:
    return f'<Document>: {doc}{SUFFIX}'

/v1/score — single pair

Scores one query against one document. Use text_1 (query side) and text_2 (document side):

curl -s http://127.0.0.1:8001/v1/score \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-reranker",
    "text_1": "What is the capital of China?",
    "text_2": "The capital of China is Beijing."
  }'

Response shape:

{
  "object": "list",
  "data": [{ "index": 0, "score": 0.97 }],
  "model": "qwen3-reranker"
}

For batch scoring, pass arrays:

{
  "model": "qwen3-reranker",
  "text_1": ["query A", "query B"],
  "text_2": ["doc A", "doc B"]
}

/v1/rerank — one query, many documents

Cohere-compatible API. Ranks all documents against a single query:

curl -s http://127.0.0.1:8001/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-reranker",
    "query": "Explain gravity",
    "documents": [
      "The capital of China is Beijing.",
      "Gravity is a force that attracts two bodies towards each other."
    ],
    "top_n": 2
  }'

Response shape:

{
  "results": [
    { "index": 1, "relevance_score": 0.79 },
    { "index": 0, "relevance_score": 0.12 }
  ]
}

Python example (end-to-end RAG rerank step)

import requests

RERANK_URL = 'http://127.0.0.1:8001/v1/rerank'

query = 'What is the capital of China?'
candidates = [
    'Gravity is a force that attracts two bodies.',
    'The capital of China is Beijing.',
    'Paris is known for the Eiffel Tower.',
]

resp = requests.post(RERANK_URL, json={
    'model': 'qwen3-reranker',
    'query': query,
    'documents': candidates,
    'top_n': 3,
})
results = resp.json()['results']

for r in results:
    doc = candidates[r['index']]
    print(f"{r['relevance_score']:.3f}  {doc}")

Instruction formatting for maximum accuracy

The simple /v1/rerank and /v1/score endpoints accept plain text. For production RAG, pre-format text_1 / text_2 (or query / documents) with the templates above before sending. Skipping instructions typically costs 1–5% ranking accuracy. Write instructions in English even for multilingual workloads.

---

Co-hosting with embedding on one GPU

On a 16 GB GPU, run both services with reduced memory caps:

Embedding wrapper (qwen3-embedding.sh):

GPU_UTIL="${QWEN3_EMBEDDING_GPU_UTIL:-0.48}"
# ...
exec vllm serve "${MODEL}" \
  --convert embed \
  --gpu-memory-utilization "${GPU_UTIL}" \
  ...

Reranker wrapper (qwen3-reranker.sh):

GPU_UTIL="${QWEN3_RERANKER_GPU_UTIL:-0.48}"
# ...
exec vllm serve "${MODEL}" \
  --convert classify \
  --hf-overrides "${HF_OVERRIDES}" \
  --gpu-memory-utilization "${GPU_UTIL}" \
  ...

Start embedding first, wait until healthy, then start reranker:

curl -sf http://127.0.0.1:8000/v1/models && echo "embedding ok"
curl -sf http://127.0.0.1:8001/v1/models && echo "reranker ok"
nvidia-smi

If the reranker fails with OOM, the embedding process likely claimed too much VRAM. Lower both utilizations equally (e.g. 0.45).

For Docker, set GPU_MEMORY_UTILIZATION=0.45 in docs/docker/.env and run docker compose up -d --force-recreate.

---

Tuning and operations

Change model size

1. Stop the service.
2. Set QWEN3_RERANKER_MODEL=Qwen/Qwen3-Reranker-4B.
3. Restart and verify VRAM with nvidia-smi.

Adjust context length

--max-model-len 8192 is a practical default. Longer contexts use more KV cache and increase latency.

Free GPU memory

sudo systemctl stop qwen3-reranker
nvidia-smi

Logs

# systemd
journalctl -u qwen3-reranker -f

# supervisor
supervisorctl tail -f qwen3_reranker

---

Troubleshooting

/v1/score or /v1/rerank not found (only /classify registered)

The hf-overrides are missing or incorrect. Restart with:

--hf-overrides '{"is_original_qwen3_reranker": true, "classifier_from_token": ["no", "yes"], "method": "from_2_way_softmax"}'

Confirm in logs:

grep -E "Supported tasks|Route:" /var/log/qwen3-reranker.log
# Supported tasks should include classify; routes should include /v1/score and /v1/rerank

Scores always ~0.5

Same root cause: overrides not applied. The model falls back to an uninitialized classifier and outputs near-random scores.

CUDA out of memory when starting second model

Another vLLM process already reserved most of the GPU. Lower --gpu-memory-utilization on both services. See Co-hosting with embedding on one GPU.

Supervisor reports FATAL / BACKOFF during startup

Reranker cold start takes 60–120 seconds. Increase startsecs to 120 in the supervisor config.

Address already in use

Another process owns the port:

ss -tlnp | grep 8001
vllm serve ... --port 8002

Wrong ranking quality

• Add instruction formatting (see Using the API correctly).
• Ensure candidate documents from retrieval are not truncated mid-sentence.
• Rerank only the top 20–50 candidates from embedding search; reranking hundreds of long documents is slow.

no kernel image is available / architecture mismatch

Install a newer vLLM release or a PyTorch build targeting your GPU architecture (Blackwell needs CUDA ≥ 12.8 wheels).

Model download fails

If Hugging Face times out or is blocked, use ModelScope instead — see Obtain model weights.

pip install modelscope
modelscope download --model Qwen/Qwen3-Reranker-0.6B --local_dir /var/lib/models/qwen3-reranker-0.6b

Then set QWEN3_RERANKER_MODEL=/var/lib/models/qwen3-reranker-0.6b and restart the service.

Other fixes: set HF_ENDPOINT to a Hugging Face mirror, or pre-download with huggingface-cli download.

---

Optional: expose externally

Binding to 127.0.0.1 is the safest default. If other systems must call the API over the network, put a controlled layer in front of vLLM — same patterns as the embedding guide: SSH tunnel (recommended for dev), reverse proxy with auth, or direct bind with firewall rules (not recommended without auth).

Example SSH tunnel for both RAG services:

ssh -L 8000:127.0.0.1:8000 -L 8001:127.0.0.1:8001 user@your-server

---

Quick reference

# Install
python3 -m venv ~/.venvs/qwen3-rerank && source ~/.venvs/qwen3-rerank/bin/activate
pip install "vllm>=0.8.5"

# Run (foreground) — note required hf-overrides
vllm serve Qwen/Qwen3-Reranker-0.6B \
  --convert classify \
  --hf-overrides '{"is_original_qwen3_reranker": true, "classifier_from_token": ["no", "yes"], "method": "from_2_way_softmax"}' \
  --host 127.0.0.1 --port 8001 \
  --max-model-len 8192 --served-model-name qwen3-reranker

# Health check
curl -s http://127.0.0.1:8001/v1/models

# Score one pair
curl http://127.0.0.1:8001/v1/score \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-reranker","text_1":"your query","text_2":"candidate doc"}'

# Rerank candidates
curl http://127.0.0.1:8001/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-reranker","query":"your query","documents":["doc1","doc2"]}'

# Co-host on 16 GB GPU: add to both embedding and reranker
--gpu-memory-utilization 0.48

---

Docker: two-container RAG stack

The recommended Docker layout runs embedding and reranker as separate containers sharing one GPU. Full instructions, compose file, and troubleshooting are in docs/docker/README.md.

cd docs/docker
cp .env.example .env
docker compose up -d

Both containers use GPU_MEMORY_UTILIZATION=0.48 by default (~14 GB combined on a 16 GB card). The reranker starts after the embedding health check passes.

See also the embedding guide: Docker: two-container RAG stack.

---

Further reading

• Docker two-container stack — compose file and operations
• Deploying Qwen3 Embedding — companion guide for the retrieval step
• Qwen3-Reranker on ModelScope
• Qwen3-Reranker model card
• HF discussion: converting official reranker weights
• vLLM scoring / rerank docs
• vLLM Qwen3 reranker offline example