Deploying Qwen3 Embedding with vLLM

Deploying Qwen3 Embedding with vLLM

This guide walks through serving Qwen3-Embedding on a Linux GPU server using vLLM, with an OpenAI-compatible HTTP API. It covers local development, production persistence, client integration, benchmarking, and optional external exposure.

The default setup binds to localhost only. Access from other machines is via SSH tunnel or a reverse proxy you control.

---

Table of contents

1. Overview
2. Prerequisites
3. Hardware recommendations
4. Understanding GPU memory usage
5. Server setup
6. Install vLLM
7. Obtain model weights
8. Run the embedding server
9. Run as a persistent service
10. Access the API
11. Using the API correctly
12. Performance reference
13. Tuning and operations
14. Troubleshooting
15. Optional: expose externally
16. Docker: two-container RAG stack

---

Overview

Architecture:

Client ──► vLLM (OpenAI API) ──► Qwen3-Embedding ──► GPU

For remote development:

Your laptop ──SSH tunnel──► server:127.0.0.1:8000 ──► vLLM

---

Prerequisites

• A Linux machine with an NVIDIA GPU and a working driver (nvidia-smi succeeds).
• Python 3.10+ (3.12 recommended).
• Enough GPU VRAM for the chosen model (see Hardware recommendations).
• Network access to Hugging Face or ModelScope for the first model download (public Qwen models do not require a token). See Obtain model weights if Hugging Face is unreachable.
• Optional: curl, systemd, or Docker if you want production-style deployment.

---

Hardware recommendations

The weight size column is the Hugging Face checkpoint on disk. The serving VRAM column is what nvidia-smi shows after vLLM starts with default settings. See Understanding GPU memory usage for why these differ so much.

CUDA compatibility notes:

• NVIDIA GPUs with compute capability ≥ 10.0 (e.g. Blackwell) need PyTorch/CUDA wheels built for CUDA ≥ 12.8.
• The CUDA toolkit version in your environment does not need to exactly match the driver minor version, but the driver must support the wheel's CUDA major version.
• vLLM bundles compatible PyTorch builds when installed via pip — prefer that over mixing manual torch installs.

Verify the GPU:

nvidia-smi
python3 -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

---

Understanding GPU memory usage

A common surprise: Qwen/Qwen3-Embedding-0.6B weighs about 1.2 GB, yet a single vLLM process on a 16 GB GPU can report ~15 GB used. That is expected — vLLM is a full inference engine, not a minimal weight loader.

What consumes VRAM

vLLM reserves memory up front so it can serve concurrent requests without hitting OOM mid-request. The default --gpu-memory-utilization is 0.9, meaning vLLM claims ~90% of total VRAM (≈14.4 GB on a 16 GB card) regardless of how small the model is.

Check actual usage

# After the server starts
nvidia-smi

# Or continuously
watch -n1 nvidia-smi

Reduce memory footprint

Example for a single model on a 16 GB GPU when you need headroom for another service:

vllm serve Qwen/Qwen3-Embedding-0.6B \
  --convert embed \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.48 \
  --served-model-name qwen3-embedding

At 0.48 on a 16 GB card, one process reserves ~7.7 GB. This is enough for the 0.6B model with normal batch sizes.

Co-hosting embedding + reranker on one GPU

Both Qwen3-Embedding-0.6B and Qwen3-Reranker-0.6B can share a 16 GB GPU when each process uses --gpu-memory-utilization 0.48 (~7.7 GB each, ~14.3 GB total). Run them on separate ports (e.g. embedding on 8000, reranker on 8001) as two independent vLLM processes.

See Deploying Qwen3 Reranker for the reranker side of this setup, or Docker: two-container RAG stack for a compose-based deployment.

Why not one vLLM process for both? vLLM serves one model per process. Two models means two processes, each with its own memory reservation.

---

Server setup

SSH into your server (adjust user, host, and port as needed):

ssh user@your-server.example.com

Create an isolated Python environment:

python3 -m venv ~/.venvs/qwen3-embed
source ~/.venvs/qwen3-embed/bin/activate
pip install -U pip

Or with uv:

uv venv ~/.venvs/qwen3-embed
source ~/.venvs/qwen3-embed/bin/activate

Set a Hugging Face cache directory if you want weights stored in a specific path:

export HF_HOME=/var/lib/huggingface
mkdir -p "$HF_HOME"

---

Install vLLM

With the virtual environment activated:

pip install "vllm>=0.8.5"

Or with uv:

uv pip install "vllm>=0.8.5"

Confirm the CLI:

vllm --version

Version note: vLLM 0.23+ uses --convert embed for embedding models. Older docs may reference --task embed, which is no longer accepted by vllm serve.

---

Obtain model weights

By default, vllm serve Qwen/Qwen3-Embedding-0.6B pulls weights from Hugging Face on first start. If huggingface.co is slow or blocked in your region, download the same checkpoints from ModelScope (maintained by the Qwen team) and point vLLM at the local directory.

ModelScope uses the same model IDs as Hugging Face (Qwen/Qwen3-Embedding-0.6B, etc.). The checkpoints are interchangeable.

Option A: ModelScope CLI (recommended when HF is blocked)

source ~/.venvs/qwen3-embed/bin/activate
pip install modelscope

export MODEL_DIR=/var/lib/models/qwen3-embedding-0.6b
mkdir -p "${MODEL_DIR}"

modelscope download --model Qwen/Qwen3-Embedding-0.6B --local_dir "${MODEL_DIR}"

Then serve from the local path:

vllm serve "${MODEL_DIR}" \
  --convert embed \
  --host 127.0.0.1 \
  --port 8000 \
  --dtype auto \
  --max-model-len 8192 \
  --served-model-name qwen3-embedding

Option B: ModelScope Python API

pip install modelscope

from modelscope import snapshot_download

model_dir = snapshot_download(
    "Qwen/Qwen3-Embedding-0.6B",
    cache_dir="/var/lib/models",
)
print(model_dir)  # e.g. /var/lib/models/Qwen/Qwen3-Embedding-0.6B

Use the printed path as the vllm serve model argument.

Option C: Hugging Face (default)

If your server can reach Hugging Face, skip pre-downloading and pass the HF model ID directly:

vllm serve Qwen/Qwen3-Embedding-0.6B --convert embed ...

Optional mirrors: set HF_ENDPOINT to a Hugging Face mirror, or pre-download with huggingface-cli download Qwen/Qwen3-Embedding-0.6B.

Offline / air-gapped servers

1. On a machine with network access, download via ModelScope (Option A or B).
2. Copy the full model directory to the target server (rsync, scp, etc.).
3. Start vLLM with the local path on the server.

rsync -avz /var/lib/models/qwen3-embedding-0.6b/ deploy@server:/var/lib/models/qwen3-embedding-0.6b/

For persistent services, set QWEN3_EMBEDDING_MODEL=/var/lib/models/qwen3-embedding-0.6b in your systemd unit or wrapper script.

---

Run the embedding server

Quick test (foreground)

Run in the foreground first to confirm the model loads:

source ~/.venvs/qwen3-embed/bin/activate

vllm serve Qwen/Qwen3-Embedding-0.6B \
  --convert embed \
  --host 127.0.0.1 \
  --port 8000 \
  --dtype auto \
  --max-model-len 8192 \
  --served-model-name qwen3-embedding

First startup downloads weights (~1–2 GB for 0.6B) if not already cached, then compiles CUDA kernels. Expect 30–90 seconds before the API is ready. Pre-download via ModelScope to avoid Hugging Face at runtime.

Verify in another shell:

curl -s http://127.0.0.1:8000/v1/models | python3 -m json.tool

curl -s http://127.0.0.1:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-embedding",
    "input": ["What is the capital of China?", "The capital of China is Beijing."]
  }' | python3 -m json.tool

Expected: two 1024-dimensional vectors with L2 norm ≈ 1.0.

Useful CLI flags

Environment variables

To use the 4B model, change the model argument:

vllm serve Qwen/Qwen3-Embedding-4B --convert embed ...

---

Run as a persistent service

Do not rely on a foreground shell or nohup in production. Pick one of the patterns below.

Option A: systemd (recommended)

Create a wrapper script at /usr/local/bin/qwen3-embedding.sh:

#!/bin/bash
set -euo pipefail

source /home/deploy/.venvs/qwen3-embed/bin/activate

export HF_HOME="${HF_HOME:-/var/lib/huggingface}"
export VLLM_LOGGING_LEVEL="${VLLM_LOGGING_LEVEL:-INFO}"

MODEL="${QWEN3_EMBEDDING_MODEL:-Qwen/Qwen3-Embedding-0.6B}"
PORT="${QWEN3_EMBEDDING_PORT:-8000}"

exec vllm serve "${MODEL}" \
  --convert embed \
  --host 127.0.0.1 \
  --port "${PORT}" \
  --dtype auto \
  --max-model-len 8192 \
  --served-model-name qwen3-embedding

sudo chmod +x /usr/local/bin/qwen3-embedding.sh

Create /etc/systemd/system/qwen3-embedding.service:

[Unit]
Description=Qwen3 Embedding Server (vLLM)
After=network.target

[Service]
Type=simple
User=deploy
Group=deploy
Environment=HF_HOME=/var/lib/huggingface
ExecStart=/usr/local/bin/qwen3-embedding.sh
Restart=on-failure
RestartSec=10
KillMode=mixed
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now qwen3-embedding
sudo systemctl status qwen3-embedding
journalctl -u qwen3-embedding -f

Option B: Supervisor

If your environment already uses supervisor, add a program entry:

[program:qwen3_embedding]
command=/usr/local/bin/qwen3-embedding.sh
directory=/home/deploy
user=deploy
autostart=true
autorestart=true
startsecs=30
stopasgroup=true
killasgroup=true
stdout_logfile=/var/log/qwen3-embedding.log
redirect_stderr=true

supervisorctl reread && supervisorctl update
supervisorctl start qwen3_embedding

Option C: Docker (embedding only)

Run a single embedding container with the official vLLM image:

docker run -d --name qwen3-embedding --gpus all --restart unless-stopped \
  -p 8000:8000 \
  -v hf-cache:/data/huggingface \
  -e HF_HOME=/data/huggingface \
  vllm/vllm-openai:v0.23.0 \
  vllm serve Qwen/Qwen3-Embedding-0.6B \
    --convert embed --host 0.0.0.0 --port 8000 \
    --dtype auto --max-model-len 8192 \
    --served-model-name qwen3-embedding

Pin the image tag in production (e.g. v0.23.0), not latest. For a local ModelScope path, replace the model argument with /models/... and add -v /path/to/models:/models:ro.

Option D: Docker Compose (embedding + reranker, recommended)

To run both embedding and reranker on one GPU as two containers, use the compose stack in docs/docker/:

cd docs/docker
cp .env.example .env
docker compose up -d

See Docker: two-container RAG stack below for an overview.

Confirm the service is listening

ss -tlnp | grep 8000
# LISTEN 127.0.0.1:8000  users:(("vllm",pid=...,fd=...))

---

Access the API

Same machine

curl http://127.0.0.1:8000/v1/models

Remote machine via SSH tunnel

On your laptop:

ssh -L 8000:127.0.0.1:8000 user@your-server.example.com

Then locally:

curl http://127.0.0.1:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-embedding","input":["hello world"]}'

OpenAI Python client

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed",  # vLLM does not require a key unless you add auth
)

response = client.embeddings.create(
    model="qwen3-embedding",
    input=["What is the capital of China?"],
)

vector = response.data[0].embedding
print(len(vector), vector[:3])

JavaScript / fetch

const res = await fetch('http://127.0.0.1:8000/v1/embeddings', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    model: 'qwen3-embedding',
    input: ['hello world'],
  }),
});
const { data } = await res.json();
console.log(data[0].embedding.length);

---

Using the API correctly

Qwen3-Embedding is instruction-aware. Queries benefit from a task-specific instruction; documents typically do not.

Recommended query format

Instruct: Given a web search query, retrieve relevant passages that answer the query
Query:What is the capital of China?

Example: query + document embeddings

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")

task = "Given a web search query, retrieve relevant passages that answer the query"

def instruct_query(q: str) -> str:
    return f"Instruct: {task}\nQuery:{q}"

queries = [
    instruct_query("What is the capital of China?"),
    instruct_query("Explain gravity"),
]
documents = [
    "The capital of China is Beijing.",
    "Gravity is a force that attracts two bodies towards each other.",
]

q_embs = client.embeddings.create(model="qwen3-embedding", input=queries).data
d_embs = client.embeddings.create(model="qwen3-embedding", input=documents).data

def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))  # vectors are already L2-normalized

for i, q in enumerate(q_embs):
    scores = [cosine(q.embedding, d.embedding) for d in d_embs]
    print(f"Query {i}: {scores}")

Skipping instructions on queries typically costs 1–5% retrieval accuracy. Write instructions in English even for multilingual workloads.

Batch requests

Send multiple strings in one input array to amortize overhead:

curl -s http://127.0.0.1:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-embedding",
    "input": ["doc 1 text", "doc 2 text", "doc 3 text"]
  }'

Batching reduces per-text latency significantly (see Performance reference).

---

Performance reference

Benchmarked with Qwen/Qwen3-Embedding-0.6B and vLLM 0.23.0 on an NVIDIA RTX 5080 (16 GB). Your numbers will vary by GPU, driver, and input length.

Single-request latency (~15 tokens)

Batching

Input length (single request)

Sustained load (50 req/s, 128 tokens, 32 concurrency)

Burst throughput (vLLM bench)

Sweet spot for concurrent clients: ~16 workers (~322 req/s before latency climbs).

Run your own benchmark

source ~/.venvs/qwen3-embed/bin/activate

# Burst throughput
vllm bench serve \
  --backend openai-embeddings \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/embeddings \
  --model qwen3-embedding \
  --tokenizer Qwen/Qwen3-Embedding-0.6B \
  --dataset-name random \
  --random-input-len 128 \
  --num-prompts 300 \
  --request-rate inf \
  --percentile-metrics e2el

# Sustained load
vllm bench serve \
  --backend openai-embeddings \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/embeddings \
  --model qwen3-embedding \
  --tokenizer Qwen/Qwen3-Embedding-0.6B \
  --dataset-name random \
  --random-input-len 128 \
  --num-prompts 500 \
  --request-rate 50 \
  --max-concurrency 32 \
  --percentile-metrics e2el

Pass the Hugging Face model ID to --tokenizer, not the served model alias.

---

Tuning and operations

Change model size

1. Stop the service (systemctl stop qwen3-embedding or equivalent).
2. Set QWEN3_EMBEDDING_MODEL=Qwen/Qwen3-Embedding-4B.
3. Restart and verify VRAM: nvidia-smi.

Adjust context length

--max-model-len 8192 is a practical default. Qwen3-Embedding supports up to 32K, but longer contexts use more VRAM and increase latency:

--max-model-len 16384

Free GPU memory

sudo systemctl stop qwen3-embedding
nvidia-smi

Logs

# systemd
journalctl -u qwen3-embedding -f

# supervisor
supervisorctl tail -f qwen3_embedding

Persistence across reboots

• Install weights into HF_HOME so cold starts skip re-download.
• Enable the service unit (systemctl enable) or supervisor autostart.
• Pin vLLM to a specific version in your environment/requirements file.

---

Troubleshooting

unrecognized arguments: --task embed

vLLM 0.23+ renamed the flag. Use:

--convert embed

Address already in use

Another process owns the port. Find and stop it, or pick a different port:

ss -tlnp | grep 8000
vllm serve ... --port 8001

CUDA out of memory

• If another vLLM process is on the same GPU, lower --gpu-memory-utilization on both (e.g. 0.48 each on a 16 GB card). See Understanding GPU memory usage.
• Use a smaller model (0.6B instead of 8B).
• Reduce --max-model-len.
• Stop competing GPU processes: nvidia-smi then kill stale vLLM PIDs.

no kernel image is available / architecture mismatch

The installed PyTorch build does not include kernels for your GPU architecture. Install a newer vLLM release or a PyTorch build targeting your compute capability (Blackwell needs CUDA ≥ 12.8 wheels).

OSError: qwen3-embedding is not a valid model identifier (vllm bench)

Pass the Hugging Face ID to --tokenizer:

--tokenizer Qwen/Qwen3-Embedding-0.6B

Model download fails

If Hugging Face times out or is blocked, use ModelScope instead — see Obtain model weights.

Otherwise check disk space and cache permissions:

df -h
ls -la "${HF_HOME:-$HOME/.cache/huggingface}/hub/"

Other fixes:

• Pre-download with modelscope download --model Qwen/Qwen3-Embedding-0.6B --local_dir /var/lib/models/qwen3-embedding-0.6b
• Set HF_ENDPOINT to a Hugging Face mirror
• Pre-download with huggingface-cli download Qwen/Qwen3-Embedding-0.6B

High latency under load

• Batch documents in a single request when indexing.
• Cap client concurrency around 16 for a good throughput/latency balance.
• Use a shorter max-model-len if you do not need long contexts.

SSH tunnel not working

Confirm the service is up on the server first:

ssh user@your-server 'curl -sf http://127.0.0.1:8000/v1/models'

Then forward:

ssh -L 8000:127.0.0.1:8000 user@your-server

---

Optional: expose externally

Binding to 127.0.0.1 is the safest default. If other systems must call the API over the network, put a controlled layer in front of vLLM.

Option A: SSH tunnel (recommended for dev / single user)

No open ports on the server firewall. See Access the API.

Option B: Reverse proxy with authentication

Keep vLLM on localhost and terminate TLS + auth in nginx, Caddy, or Traefik.

Example Caddy snippet:

embeddings.example.com {
    basicauth {
        deploy <bcrypt-hash>
    }
    reverse_proxy 127.0.0.1:8000
}

Example nginx snippet:

server {
    listen 443 ssl;
    server_name embeddings.example.com;

    auth_basic "Embeddings";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
    }
}

Option C: Bind directly (not recommended without auth)

vllm serve Qwen/Qwen3-Embedding-0.6B \
  --convert embed \
  --host 0.0.0.0 \
  --port 8000 \
  ...

Restrict access with firewall rules (ufw, security groups, etc.) and add API-key middleware if the endpoint is reachable from untrusted networks. vLLM does not ship production-grade authentication.

---

Quick reference

# Install
python3 -m venv ~/.venvs/qwen3-embed && source ~/.venvs/qwen3-embed/bin/activate
pip install "vllm>=0.8.5"

# Run (foreground)
vllm serve Qwen/Qwen3-Embedding-0.6B \
  --convert embed --host 127.0.0.1 --port 8000 \
  --max-model-len 8192 --served-model-name qwen3-embedding

# Health check
curl -s http://127.0.0.1:8000/v1/models

# Embed
curl http://127.0.0.1:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-embedding","input":["your text"]}'

# SSH tunnel from laptop
ssh -L 8000:127.0.0.1:8000 user@your-server

---

Docker: two-container RAG stack

For a full embed → retrieve → rerank setup, run embedding and reranker as separate containers that share one GPU. vLLM loads one model per process, so two models means two containers (not one).

┌─ docker host ─────────────────────────────────────────────┐
│  GPU                                                     │
│   ┌─────────────────────┐   ┌─────────────────────┐     │
│   │ qwen3-embedding     │   │ qwen3-reranker      │     │
│   │ :8000 /v1/embeddings│   │ :8001 /v1/rerank    │     │
│   │ gpu-util 0.48       │   │ gpu-util 0.48       │     │
│   └─────────────────────┘   └─────────────────────┘     │
│            shared hf-cache volume                        │
└──────────────────────────────────────────────────────────┘

Files live in docs/docker/:

Quick start

cd docs/docker
cp .env.example .env
docker compose up -d
docker compose ps

Requires NVIDIA Container Toolkit. Default settings target a 16 GB GPU with both 0.6B models at GPU_MEMORY_UTILIZATION=0.48.

The reranker container waits for the embedding health check before starting, which avoids both processes competing for VRAM during weight download.

Client URLs

From another container on the same Compose network, use http://embedding:8000 and http://reranker:8001.

ModelScope / offline weights

Pre-download weights, mount into both containers, and set paths in .env — see docs/docker/README.md.

Why two containers (not one)?

---

Further reading

• Docker two-container stack — compose file and operations
• Deploying Qwen3 Reranker — companion guide for the reranking step in a RAG pipeline
• Qwen3-Embedding on ModelScope
• Qwen3-Embedding model card
• Qwen3 Embedding blog
• vLLM pooling / embedding docs
• vLLM OpenAI-compatible server