vLLM/Recipes
Search
Qwen

Qwen/Qwen3.5-397B-A17B

Multimodal MoE model with gated delta networks architecture, 397B total / 17B active parameters, up to 262K context

Verified on 8x H200, 8x MI300X/MI355X, a single MI350X/MI355X (MXFP4), GB200 nodes, and Ascend 950DT

moe397B / 17B262,144 ctxvLLM 0.17.0+Multimodal
Default configuration: 8× H200 · Tensor Parallel · BF16
export VLLM_DEEP_GEMM_WARMUP=skip
export VLLM_USE_DEEP_GEMM=0
export VLLM_USE_RUST_FRONTEND=1
vllm serve Qwen/Qwen3.5-397B-A17B \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data

The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /Qwen/Qwen3.5-397B-A17B.md.

Overview

Qwen3.5 is a multimodal mixture-of-experts model featuring a gated delta networks architecture with 397B total parameters and 17B active parameters. This guide covers how to efficiently deploy and serve the model across different hardware configurations and workload profiles using vLLM.

Verified configurations

These configurations were verified on 8x H200, 8x MI300X/MI355X and GB200. On H200 and MI300X/MI355X they used the official FP8 checkpoint Qwen/Qwen3.5-397B-A17B-FP8.

WorkloadHardwareCheckpointParallelismOther settings
Throughput, text only8x H200, 8x MI300X/MI355XFP8DP8 + EP--language-model-only, --enable-prefix-caching
Throughput, multimodal8x H200, 8x MI300X/MI355XFP8DP8 + EP--mm-encoder-tp-mode data, --mm-processor-cache-type shm, --enable-prefix-caching
Latency8x H200, 8x MI300X/MI355XFP8TP8MTP-1, prefix caching off
Throughput, text onlyGB200 (4 GPUs)NVFP4DP4 + EP--language-model-only, --enable-prefix-caching
Throughput, text onlyMI355XFP8TP2 + EP--language-model-only, --enable-prefix-caching

--language-model-only skips loading the vision encoder and frees memory for KV cache. For multimodal workloads, --mm-encoder-tp-mode data runs the vision encoder data-parallel and --mm-processor-cache-type shm caches preprocessed multimodal inputs in shared memory. MTP-1 lowers time per output token with a high acceptance rate, but speculative tokens take KV cache capacity, so throughput drops under high concurrency; tune num_speculative_tokens (1-5) to the workload.

Single MI350X / MI355X (MXFP4)

AMD's MXFP4 checkpoint amd/Qwen3.5-397B-A17B-MXFP4 (239 GB) fits one 288 GB MI350X or MI355X: pick the MXFP4 (W4A4) variant above. It sets three things vLLM does not pick on its own: VLLM_ROCM_USE_AITER=1 turns on the AITER kernels; --attention-backend ROCM_AITER_FA replaces the auto-selected ROCM_ATTN, whose decode slows down as the context grows on this hybrid model; and --max-num-seqs 64, because the default (1024) is more than the ~296 Mamba cache blocks a single GPU leaves, and the server would refuse to start.

Single-stream decode in tokens/s at ~200 / ~2k / ~12k / ~30k prompt tokens (128 output tokens), one GPU, vllm/vllm-openai-rocm:v0.31.0:

~200~2k~12k~30k
MXFP4, this variant (MI355X)119119119117
+ MTP (3 tokens) + --kv-cache-dtype fp8 (MI355X)219201209188
+ MTP (3 tokens), default KV cache (MI350X)19816412580
auto-selected ROCM_ATTN instead of ROCM_AITER_FA (MI355X)116893919
GPTQ-Int4, AITER + ROCM_AITER_FA (MI350X)75747474

With MTP, add --kv-cache-dtype fp8: on ROCm, MTP's gain otherwise fades as the prompt grows. The MXFP4 checkpoint quantizes activations as well as weights (W4A4): on the same wikitext-2 tokens vLLM scored PPL 3.43 with it and 3.39 with GPTQ-Int4 (W4A16), so GPTQ-Int4 is the higher-fidelity option at about two thirds of the speed.

Processing Ultra-Long Texts

Qwen3.5-397B-A17B natively supports 262,144 tokens. To serve longer inputs, turn on the Long Context feature above: it applies the model card's YaRN RoPE configuration via --hf-overrides and raises --max-model-len to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave it off unless you actually serve long prompts; for a ~524K workload, halve the card's factor to 2.0.

See the model card for the full parameter reference.

Client Usage

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)
# The server's model name is the checkpoint it runs: the selected variant's.
model = client.models.list().data[0].id

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model=model,
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

Qwen3.5 thinks before it answers. To turn thinking off for one request, pass extra_body={"chat_template_kwargs": {"enable_thinking": False}}; for every request, start the server with --default-chat-template-kwargs '{"enable_thinking": false}'.

References