# Qwen/Qwen3.5-397B-A17B — vLLM serving recipe

> Multimodal MoE model with gated delta networks architecture, 397B total / 17B active parameters, up to 262K context

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`, `encoder_parallel` · frontend `rust`

- Interactive page: https://vllm-recipes-git-fork-1195343015-modelscope-1a13f7-inferact-inc.vercel.app/Qwen/Qwen3.5-397B-A17B
- This configuration as JSON: https://vllm-recipes-git-fork-1195343015-modelscope-1a13f7-inferact-inc.vercel.app/Qwen/Qwen3.5-397B-A17B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `ascend_950dt`
- `variant`: `default`, `nvfp4`, `fp8`, `gptq_int4`, `amd_mxfp4`, `ascend_w8a8_mtp`, `ascend_w4a4_mxfp4`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only`, `encoder_parallel`, `long_context` (default: `tool_calling`, `reasoning`, `encoder_parallel`). Use `features=none` to turn them all off.
- `prefill_nodes`, `decode_nodes`: dedicated node counts for each pool under `strategy=pd_cluster`.
- `prefill_mode`, `decode_mode`: `tp`, `tep`, `dep`.
- `prefill_rank`, `decode_rank`: zero-based node indices (`0` to the pool's node count minus one). These select the singular command; `node_commands[]` always includes every node.
- `pd_router`: `vllm`, `dynamo`, `smg`, `ascend`. Hardware and pool compatibility applies.
- `pd_connector`: `nixl`, `mooncake`, `mooncake_ascend`, `moriio`. Select independently of the router; catalog hardware/router compatibility still applies.
- `preset`: a tuned, evidence-backed configuration (`baseline` opts out of a default one). Each sets its own hardware and strategy:
  - `b200_tp2_kv_offload` — TP2 + KV offload (multi-turn agentic) (b200 · TP · Multi-turn agentic coding, ~64k input, KV-reuse-bound (mean ITL under 20 ms) · opt-in): `hardware=b200&variant=nvfp4&strategy=single_node_tp&preset=b200_tp2_kv_offload`

## Install

Requires vLLM >= 0.17.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_DEEP_GEMM_WARMUP=skip
export VLLM_USE_DEEP_GEMM=0
export VLLM_USE_RUST_FRONTEND=1
vllm serve Qwen/Qwen3.5-397B-A17B \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_DEEP_GEMM_WARMUP=skip \
  -e VLLM_USE_DEEP_GEMM=0 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest Qwen/Qwen3.5-397B-A17B \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

## Known issues

### assert num_cache_lines >= batch

The CUDA graph capture size is larger than the Mamba cache. Reduce
`--max-cudagraph-capture-size` (default 512); see
[vllm#34571](https://github.com/vllm-project/vllm/pull/34571).

### Prefix caching is experimental

Prefix caching for the Mamba cache runs in the experimental "align" mode.

### Out of memory with Encoder Parallel

`--mm-encoder-tp-mode data` places a copy of the vision encoder on every
GPU, which takes extra memory. If the server runs out of memory, lower
`--max-num-seqs` or `--max-num-batched-tokens`, or drop
`--mm-encoder-tp-mode data`.

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) is a multimodal mixture-of-experts model featuring a gated delta networks architecture with 397B total parameters and 17B active parameters. This guide covers how to efficiently deploy and serve the model across different hardware configurations and workload profiles using vLLM.

### Verified configurations

These configurations were verified on 8x H200, 8x MI300X/MI355X and GB200. On
H200 and MI300X/MI355X they used the official FP8 checkpoint
[Qwen/Qwen3.5-397B-A17B-FP8](https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8).

| Workload | Hardware | Checkpoint | Parallelism | Other settings |
|---|---|---|---|---|
| Throughput, text only | 8x H200, 8x MI300X/MI355X | FP8 | DP8 + EP | `--language-model-only`, `--enable-prefix-caching` |
| Throughput, multimodal | 8x H200, 8x MI300X/MI355X | FP8 | DP8 + EP | `--mm-encoder-tp-mode data`, `--mm-processor-cache-type shm`, `--enable-prefix-caching` |
| Latency | 8x H200, 8x MI300X/MI355X | FP8 | TP8 | MTP-1, prefix caching off |
| Throughput, text only | GB200 (4 GPUs) | [NVFP4](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4) | DP4 + EP | `--language-model-only`, `--enable-prefix-caching` |
| Throughput, text only | MI355X | FP8 | TP2 + EP | `--language-model-only`, `--enable-prefix-caching` |

`--language-model-only` skips loading the vision encoder and frees memory for KV
cache. For multimodal workloads, `--mm-encoder-tp-mode data` runs the vision
encoder data-parallel and `--mm-processor-cache-type shm` caches preprocessed
multimodal inputs in shared memory. MTP-1 lowers time per output token with a
high acceptance rate, but speculative tokens take KV cache capacity, so
throughput drops under high concurrency; tune `num_speculative_tokens` (1-5) to
the workload.

### Single MI350X / MI355X (MXFP4)

AMD's MXFP4 checkpoint [amd/Qwen3.5-397B-A17B-MXFP4](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4) (239 GB)
fits one 288 GB MI350X or MI355X: pick the **MXFP4 (W4A4)** variant above. It sets three things vLLM does not
pick on its own: `VLLM_ROCM_USE_AITER=1` turns on the AITER kernels; `--attention-backend ROCM_AITER_FA` replaces
the auto-selected `ROCM_ATTN`, whose decode slows down as the context grows on this hybrid model; and
`--max-num-seqs 64`, because the default (1024) is more than the ~296 Mamba cache blocks a single GPU leaves,
and the server would refuse to start.

Single-stream decode in tokens/s at ~200 / ~2k / ~12k / ~30k prompt tokens (128 output tokens), one GPU,
`vllm/vllm-openai-rocm:v0.31.0`:

| | ~200 | ~2k | ~12k | ~30k |
|---|---|---|---|---|
| MXFP4, this variant (MI355X) | 119 | 119 | 119 | 117 |
| + MTP (3 tokens) + `--kv-cache-dtype fp8` (MI355X) | 219 | 201 | 209 | 188 |
| + MTP (3 tokens), default KV cache (MI350X) | 198 | 164 | 125 | 80 |
| auto-selected `ROCM_ATTN` instead of `ROCM_AITER_FA` (MI355X) | 116 | 89 | 39 | 19 |
| GPTQ-Int4, AITER + `ROCM_AITER_FA` (MI350X) | 75 | 74 | 74 | 74 |

With MTP, add `--kv-cache-dtype fp8`: on ROCm, MTP's gain otherwise fades as the prompt grows. The MXFP4
checkpoint quantizes activations as well as weights (W4A4): on the same wikitext-2 tokens vLLM scored PPL 3.43
with it and 3.39 with GPTQ-Int4 (W4A16), so GPTQ-Int4 is the higher-fidelity option at about two thirds of the speed.

### Processing Ultra-Long Texts

Qwen3.5-397B-A17B natively supports `262,144` tokens. To serve longer inputs, turn on the
**Long Context** feature above: it applies the model card's YaRN RoPE
configuration via `--hf-overrides` and raises `--max-model-len` to ~1,010,000 tokens.
Static YaRN holds the scaling factor constant regardless of input length, so
leave it off unless you actually serve long prompts; for a ~524K workload,
halve the card's `factor` to `2.0`.

See the [model card](https://huggingface.co/Qwen/Qwen3.5-397B-A17B#processing-ultra-long-texts) for the full parameter reference.

### Client Usage

```python
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)
# The server's model name is the checkpoint it runs: the selected variant's.
model = client.models.list().data[0].id

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model=model,
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

Qwen3.5 thinks before it answers. To turn thinking off for one request, pass
`extra_body={"chat_template_kwargs": {"enable_thinking": False}}`; for every
request, start the server with
`--default-chat-template-kwargs '{"enable_thinking": false}'`.

### References

- Model card: https://huggingface.co/Qwen/Qwen3.5-397B-A17B
- FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8
- NVFP4 checkpoint: https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4
- GPTQ-Int4 checkpoint: https://huggingface.co/Qwen/Qwen3.5-397B-A17B-GPTQ-Int4
- AMD MXFP4 checkpoint: https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4
- Ascend W8A8 checkpoint: https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp
- Ascend W4A4 checkpoint: https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a4-mxfp4
- vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.5-397B-A17B.html
- vLLM documentation: https://docs.vllm.ai/
