Qwen/Qwen3.5-35B-A3B
Compact Qwen3.5 multimodal MoE (35B total / 3B active) with gated delta networks, 256 experts, and 262K context
Compact Qwen3.5 MoE — single-GPU FP8, 2x GPU or 2x Xeon 6 NUMA nodes or 4x Intel Arc Pro B60/B70 BF16 serving
export VLLM_USE_RUST_FRONTEND=1 vllm serve Qwen/Qwen3.5-35B-A3B \ --trust-remote-code \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 \ --mm-encoder-tp-mode data
The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /Qwen/Qwen3.5-35B-A3B.md.
Overview
Qwen3.5-35B-A3B is the smallest MoE in the Qwen3.5 family, sharing the gated delta networks architecture with 35B total parameters and 3B activated per token (256 experts). With FP8 weights it fits on a single 80 GB GPU and supports the full 262K context.
Prerequisites
- vLLM version: >= 0.17.0
- Hardware (BF16): 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, 4x Intel Arc Pro B60 / B70
- Hardware (FP8): single H100/H200
- Hardware (Int4): single 24 GB GPU
Client Usage
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
# The server's model name is the checkpoint it runs: the selected variant's.
model = client.models.list().data[0].id
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
max_tokens=512,
)
print(resp.choices[0].message.content)
Qwen3.5 thinks before it answers. To turn thinking off for one request, pass
extra_body={"chat_template_kwargs": {"enable_thinking": False}}; for every
request, start the server with
--default-chat-template-kwargs '{"enable_thinking": false}'.
References
Engine startup times out on CPUCpu
A cold weight download followed by CPU compilation can exceed the default
600-second engine readiness timeout. The CPU command sets
VLLM_ENGINE_READY_TIMEOUT_S=1800; set any external health-check deadline
higher (for example, 2100 seconds). Increasing only the external deadline
does not prevent the frontend from timing out.
assert num_cache_lines >= batchNVIDIAAMD
The CUDA graph capture size is larger than the Mamba cache. Reduce
--max-cudagraph-capture-size (default 512); see
vllm#34571.
Prefix caching is experimental
Prefix caching for the Mamba cache runs in the experimental "align" mode.