Hosting Ornith-1.0-35B-FP8 for Field Coding Agents

Hosting Ornith-1.0-35B-FP8 for Field Coding Agents

This tutorial shows how to host Ornith-1.0-35B-FP8 on your own GPU server and use it as a coding model from a laptop. It is written for beginners and for FDEs, or field deployed engineers, who often need to work close to a customer environment, private repository, lab machine, air-gapped cluster, or temporary GPU box.

The goal is not to build a perfect AI platform. The goal is to get a capable coding model running behind a simple local API so you can point coding tools at it and start doing vibe coding: fast interactive programming with an agent, terminal tools, large context, and local control over where prompts and code go.

This guide is based on a real deployment of Ornith-1.0-35B-FP8 on a high-VRAM NVIDIA GPU (RTX PRO 6000 Blackwell class, ~98 GB VRAM) using a recent vLLM release.

---

Table of contents

1. Why this model matters
2. What you are building
3. Hardware and storage checklist
4. Install vLLM
5. Download the FP8 checkpoint
6. Start the server
7. Access it from your laptop
8. Test the APIs
9. Use it with Claude Code-style clients
10. Performance notes
11. Charts from the deployment
12. Troubleshooting
13. Quick reference

---

Why this model matters

Ornith-1.0-35B-FP8 is a compact agentic coding model from DeepReinforce. The larger Ornith family targets coding-agent workloads such as repository editing, terminal use, tool calling, and benchmark suites like SWE-bench and Terminal-Bench.

For an FDE, the interesting part is not just the benchmark score. It is the operating model:

• You can host the model on a GPU box near the work.
• You can keep customer code, logs, stack traces, and internal docs inside your own network path.
• You can expose the model to OpenAI-compatible and Anthropic-compatible clients.
• You can use SSH tunneling instead of opening a public LLM endpoint.
• You can control context length, request concurrency, thinking mode, and tool parsing.

That makes it useful for temporary field setups: debugging a customer deployment, modernizing a monorepo, writing migration scripts, explaining unfamiliar code, or running coding agents where external SaaS access is not appropriate.

---

What you are building

The setup is intentionally simple:

Laptop coding tool
      │
      │ SSH tunnel
      ▼
127.0.0.1:8000 on your laptop
      │
      ▼
GPU server: vLLM on 127.0.0.1:8000
      │
      ▼
Ornith-1.0-35B-FP8 on NVIDIA GPU

The model server exposes several useful API shapes:

In this deployment, the model is served as:

Ornith-1.0-35B-FP8

---

Hardware and storage checklist

Before installing anything, verify the machine.

nvidia-smi
python3 --version
df -hT / /var /opt /models /workspace /dev/shm 2>/dev/null || true
mount | grep ' /dev/shm '

For the FP8 checkpoint, budget roughly:

For a comfortable single-GPU deployment, a high-VRAM GPU (~80–98 GB class) works well. Smaller GPUs can still work, but you will likely reduce --max-model-len to 65K, 128K, or 200K depending on available memory.

Storage warning for rented containers

On some rented GPU containers, the writable root filesystem can be much smaller than the GPU memory. For example, you may encounter a small overlay root (~64 GB) and a large /dev/shm tmpfs that is mounted with the noexec flag.

That means model weights may fit, but there is not much room for repeated downloads, environments, and caches. Keep executable caches on normal disk, not on noexec tmpfs:

# Example cache locations (customize for your server)
export VLLM_CACHE_ROOT=/var/cache/ornith/vllm
export TORCHINDUCTOR_CACHE_DIR=/var/cache/ornith/torchinductor
export TRITON_CACHE_DIR=/var/cache/ornith/triton
export XDG_CACHE_HOME=/var/cache/ornith/xdg
export TMPDIR=/var/cache/ornith/tmp

Using /dev/shm for temporary model downloads is fine, but do not put Triton or TorchInductor executable caches there if it is mounted noexec.

Tip: Throughout this guide we use /opt/ornith, /models/ornith-1.0-35b-fp8, and /var/cache/ornith/* as example paths. Replace them with locations appropriate for your environment.

---

Install vLLM

Create an isolated environment. uv is convenient, but standard venv also works.

# Example install location (customize as needed)
mkdir -p /opt/ornith
uv venv /opt/ornith/venv --python python3.12
source /opt/ornith/venv/bin/activate

Install recent vLLM and Hugging Face tooling:

export UV_CACHE_DIR=/dev/shm/ornith-uv-cache
export PIP_CACHE_DIR=/dev/shm/ornith-pip-cache
export TMPDIR=/var/cache/ornith/tmp

uv pip install -U --link-mode=copy "vllm>=0.19.1" huggingface_hub

On Blackwell (or newer) GPUs, avoid pinning older CUDA wheels. Let uv or pip resolve a PyTorch build that supports your GPU architecture.

Verify:

vllm --version
python3 - <<'PY'
import torch
print(torch.__version__)
print(torch.cuda.is_available())
print(torch.cuda.get_device_name(0))
PY

---

Download the FP8 checkpoint

Use the FP8 checkpoint, not the full BF16 model. For text-only coding, the official DeepReinforce FP8 checkpoint can serve. For multimodal use, prefer the protoLabs FP8 build because it preserves the vision tower and avoids the official FP8 multimodal issue described in Troubleshooting.

mkdir -p /models/ornith-1.0-35b-fp8

export HF_HOME=/models/.hf_home
export HF_XET_HIGH_PERFORMANCE=1

hf download protoLabsAI/Ornith-1.0-35B-FP8 \
  --local-dir /models/ornith-1.0-35b-fp8

If the disk is too small, download first to /dev/shm, then copy to persistent storage:

hf download protoLabsAI/Ornith-1.0-35B-FP8 \
  --local-dir /dev/shm/ornith-1.0-35b-fp8-tmp

mkdir -p /models
cp -a /dev/shm/ornith-1.0-35b-fp8-tmp /models/ornith-1.0-35b-fp8
rm -rf /dev/shm/ornith-1.0-35b-fp8-tmp

Only use HF_TOKEN if Hugging Face rate limits or authentication require it. Do not paste long-lived tokens into logs, scripts, or shared terminals.

The upstream model card for deepreinforce-ai/Ornith-1.0-35B lists the same core vLLM flags, but in deployment the official deepreinforce-ai/Ornith-1.0-35B-FP8 checkpoint produced broken image responses: image requests were accepted, but the model returned repeated ! tokens. Switching to protoLabsAI/Ornith-1.0-35B-FP8 fixed the issue; a simple red-square image test returned Red, and a local zoo.webp test returned Giraffe.

---

Start the server

This command serves the model locally on the GPU server. It binds to 127.0.0.1, which is safer than exposing the API to the internet.

source /opt/ornith/venv/bin/activate

# Cache locations (match what you used earlier)
export VLLM_CACHE_ROOT=/var/cache/ornith/vllm
export TORCHINDUCTOR_CACHE_DIR=/var/cache/ornith/torchinductor
export TRITON_CACHE_DIR=/var/cache/ornith/triton
export XDG_CACHE_HOME=/var/cache/ornith/xdg
export TMPDIR=/var/cache/ornith/tmp
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

vllm serve /models/ornith-1.0-35b-fp8 \
  --served-model-name Ornith-1.0-35B-FP8 \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.90 \
  --max-cudagraph-capture-size 256 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --trust-remote-code

The first startup can take several minutes. vLLM loads weights, compiles kernels, profiles memory, and captures CUDA graphs.

Healthy logs should include lines like:

Using max model len 262144
Chunked prefill is enabled
GPU KV cache size: ... tokens
Starting vLLM server on http://127.0.0.1:8000

Why these flags matter

For text-only service profiles, --kv-cache-dtype fp8 can reduce KV cache memory and leave more room for context or concurrency. For the multimodal protoLabs profile described here, the tested configuration left KV cache dtype on vLLM's default auto path.

Should you enable MTP speculative decoding?

Test it, but do not assume it helps.

In testing with Ornith-1.0-35B-FP8 on high-VRAM hardware, vLLM detected an MTP draft model, but throughput got worse. The no-MTP setup decoded around 201 tokens/s, while MTP-1 and MTP-3 were much slower. The service was left with MTP disabled.

---

Access it from your laptop

Keep the model server bound to localhost on the GPU machine. From your laptop, create an SSH tunnel:

# Forward port 8000 from the GPU server to your laptop
ssh -L 8000:127.0.0.1:8000 user@your-gpu-server-hostname

# If your SSH server uses a non-standard port:
# ssh -p 2222 -L 8000:127.0.0.1:8000 user@your-gpu-server-hostname

Then, on your laptop:

curl http://127.0.0.1:8000/v1/models

This is often the best FDE access pattern: no public LLM endpoint, no firewall changes, no accidental unauthenticated API exposed to the internet.

---

Test the APIs

OpenAI chat completions

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Ornith-1.0-35B-FP8",
    "messages": [
      {"role": "user", "content": "Think briefly, then answer exactly: chat-ok"}
    ],
    "max_tokens": 1024,
    "temperature": 0.6
  }'

Expected behavior:

• HTTP 200
• final answer appears in choices[0].message.content
• reasoning appears separately when the client surfaces it

OpenAI responses

curl http://127.0.0.1:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Ornith-1.0-35B-FP8",
    "input": "Think briefly, then answer exactly: responses-ok",
    "max_output_tokens": 1024,
    "temperature": 0.6
  }'

In the tested vLLM compatibility layer, this endpoint returned HTTP 200. One caveat: reasoning text may be included in the output text depending on the client and parser path.

Anthropic messages

curl http://127.0.0.1:8000/v1/messages \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "Ornith-1.0-35B-FP8",
    "messages": [
      {"role": "user", "content": "Think briefly, then answer exactly: anthropic-ok"}
    ],
    "max_tokens": 1024,
    "temperature": 0.6
  }'

Expected behavior:

• HTTP 200
• final text appears in content[].text
• stop_reason is usually end_turn

---

Use it with Claude Code-style clients

Because vLLM exposes an Anthropic-compatible /v1/messages endpoint, Claude Code-style clients can point at the local tunnel.

Example shell function:

function claude-ornith {
  unset CLAUDE_CODE_MAX_OUTPUT_TOKENS
  unset ANTHROPIC_API_KEY
  unset ANTHROPIC_AUTH_TOKEN
  unset ANTHROPIC_BASE_URL
  unset ANTHROPIC_MODEL
  unset ANTHROPIC_SMALL_FAST_MODEL
  unset ANTHROPIC_DEFAULT_HAIKU_MODEL
  unset ANTHROPIC_DEFAULT_SONNET_MODEL
  unset ANTHROPIC_DEFAULT_OPUS_MODEL

  # Requires SSH tunnel (example):
  # ssh -L 8000:127.0.0.1:8000 user@your-gpu-server-hostname
  export ANTHROPIC_BASE_URL=http://127.0.0.1:8000
  export ANTHROPIC_AUTH_TOKEN=not-needed
  export ANTHROPIC_API_KEY=not-needed
  export ANTHROPIC_MODEL=Ornith-1.0-35B-FP8
  export ANTHROPIC_SMALL_FAST_MODEL=Ornith-1.0-35B-FP8

  claude "$@"
}

For FDE work, keep a project-specific shell profile with this function and the SSH tunnel command. That makes it easy to switch between hosted SaaS models and the field-local model.

---

Performance notes

These are example measurements from one deployment on a high-VRAM NVIDIA GPU (~98 GB VRAM). Your numbers will vary with hardware, context length, batch size, and workload.

The important beginner lesson: benchmark your actual workload. Speculative decoding sounds like a free speedup, but for this model and runtime combination it hurt throughput.

Thinking mode also changes perceived latency. With thinking enabled, the first reasoning token arrives quickly, but the final answer may not appear until the model finishes its reasoning block.

ToolCall-15 smoke benchmark

For a quick coding-agent-adjacent tool-use check, the deployment also ran stevibe/ToolCall-15 against the local vLLM endpoint. This is not a SWE-bench replacement, but it is a useful deterministic smoke test for tool selection, argument precision, multi-step chains, restraint, and error recovery.

The CLI was configured against the OpenAI-compatible vLLM endpoint:

LLM_MODELS=llamacpp:Ornith-1.0-35B-FP8
LLAMACPP_HOST=http://127.0.0.1:8000
MODEL_REQUEST_TIMEOUT_SECONDS=120

npm run cli -- \
  --model llamacpp:Ornith-1.0-35B-FP8 \
  --temperature 0 \
  --top-p 1 \
  --timeout 120

Result on the protoLabs FP8 checkpoint:

Final score: 100/100
Points: 30/30
Scenarios: TC-01 through TC-15 all passed

Treat this as a simple service validation, not as proof of SWE-bench or Terminal-Bench parity. It does confirm that vLLM tool parsing with --tool-call-parser qwen3_xml works well enough for a deterministic local tool loop.

---

Charts from the deployment

The following charts are rendered as inline SVG so they work in a plain Markdown article without extra JavaScript.

Decode throughput

GPU memory shape

API compatibility smoke test

---

Troubleshooting

The server starts, but short answers return no final content

Ornith is a reasoning model. If thinking is enabled and max_tokens is too small, the model may spend the full budget inside reasoning.

Fixes:

• increase max_tokens or max_output_tokens
• disable thinking per request if you only need a short answer
• set server default with --default-chat-template-kwargs '{"enable_thinking": false}' for direct-answer workflows

CUDA out of memory during startup

Lower memory pressure in this order:

1. Lower --max-model-len from 262144 to 200000, 131072, or 65536.
2. Lower --max-num-seqs from 16 to 8 or 4.
3. Lower --max-cudagraph-capture-size from 256 to 128.
4. For text-only profiles, consider --kv-cache-dtype fp8.
5. Avoid --enforce-eager unless you accept much lower throughput.

The API is exposed publicly

Avoid binding vLLM directly to 0.0.0.0 unless you have authentication in front of it. For field work, prefer:

vllm serve ... --host 127.0.0.1 --port 8000
ssh -L 8000:127.0.0.1:8000 user@your-gpu-server

failed to map segment from shared object

Your executable compile cache may be on a noexec filesystem such as /dev/shm.

Move these to an executable filesystem:

export VLLM_CACHE_ROOT=/var/cache/ornith/vllm
export TORCHINDUCTOR_CACHE_DIR=/var/cache/ornith/torchinductor
export TRITON_CACHE_DIR=/var/cache/ornith/triton
export XDG_CACHE_HOME=/var/cache/ornith/xdg
export TMPDIR=/var/cache/ornith/tmp

Vision / image input does not work

Ornith-1.0-35B is built on a Qwen 3.5 family base with multimodal configuration. The issue in this deployment was not that vLLM rejected images. With the official deepreinforce-ai/Ornith-1.0-35B-FP8 checkpoint, image requests were accepted, but vision outputs degenerated into repeated ! tokens.

The practical fix was to switch to protoLabsAI/Ornith-1.0-35B-FP8. Its model card says the upstream FP8 release was reported broken and that the protoLabs quantization keeps the vision tower, linear-attention / SSM path, router gates, norms, embeddings, and lm_head in BF16 where needed. After switching, image tests worked:

224x224 red square -> Red
zoo.webp animal recognition -> Giraffe

If you only need text and want lower memory pressure, add --language-model-only. If you need image input, do not use --language-model-only, and use the protoLabs FP8 checkpoint rather than the official FP8 checkpoint.

---

Quick reference

Example paths used in this guide (customize for your server):

/opt/ornith                 # Python environment
/models/ornith-1.0-35b-fp8  # Model weights
/var/cache/ornith/*         # vLLM, Torch, Triton caches

Start the server:

source /opt/ornith/venv/bin/activate

export VLLM_CACHE_ROOT=/var/cache/ornith/vllm
export TORCHINDUCTOR_CACHE_DIR=/var/cache/ornith/torchinductor
export TRITON_CACHE_DIR=/var/cache/ornith/triton
export XDG_CACHE_HOME=/var/cache/ornith/xdg
export TMPDIR=/var/cache/ornith/tmp
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

vllm serve /models/ornith-1.0-35b-fp8 \
  --served-model-name Ornith-1.0-35B-FP8 \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 262144 --max-num-seqs 16 \
  --gpu-memory-utilization 0.90 \
  --max-cudagraph-capture-size 256 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --trust-remote-code

Tunnel from your laptop:

ssh -L 8000:127.0.0.1:8000 user@your-gpu-server-hostname

Check health:

curl http://127.0.0.1:8000/v1/models

Use with Anthropic-compatible clients:

export ANTHROPIC_BASE_URL=http://127.0.0.1:8000
export ANTHROPIC_AUTH_TOKEN=not-needed
export ANTHROPIC_API_KEY=not-needed
export ANTHROPIC_MODEL=Ornith-1.0-35B-FP8
export ANTHROPIC_SMALL_FAST_MODEL=Ornith-1.0-35B-FP8

For FDE work, this pattern is powerful because it keeps the loop short: code and context stay near the deployment, the laptop gets a normal local API, and your coding agent can still use tool calling and long context.