MiniMaxAI/MiniMax-M3.1-Flash
MiniMax M3.1-Flash vision-language MoE (~428B total / 23B active) with MSA sparse attention throughout and a built-in DSpark draft; quantization-aware trained, MXFP8 dense layers and NVFP4 experts. Served TP4 on NVIDIA Blackwell or AMD MI355X, prefill/decode disaggregated as TP4 + TP4 on MI355X or DEP2 + DEP4 on GB200/GB300, and TP8 on NVIDIA Hopper.
MSA sparse attention, 1M context, DSpark speculative decoding built into the checkpoint
export VLLM_FLOAT32_MATMUL_PRECISION=high
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
--block-size 128 \
--disable-hybrid-kv-cache-manager \
--tensor-parallel-size 8 \
--kv-cache-dtype nvfp4 \
--attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \
--compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
--tool-call-parser minimax_m3 \
--enable-auto-tool-choice \
--reasoning-parser minimax_m3 \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /MiniMaxAI/MiniMax-M3.1-Flash.md.
Overview
MiniMax-M3.1-Flash is a vision-language MoE model from MiniMax that keeps the MoE architecture of MiniMax-M3: about 428B total / 23B active parameters and a 1M-token context, with MSA sparse attention in every layer.
- Quantization-aware training — the checkpoint stores MXFP8 dense layers and NVFP4 experts.
- Built-in DSpark draft — the
dspark/directory of the checkpoint;--speculative-configneeds no draft model path. - Native multimodal — image and video inputs.
Prerequisites
- AMD: MI355X (gfx950), ROCm 7.2+, TP4.
- NVIDIA Blackwell: GB200, GB300, B200 or B300, TP4; NVFP4 MoE runs natively on the FP4 tensor cores.
- NVIDIA Rubin (preview): VR200, TP4, with the
vllm/vllm-openai:minimax-m31-flash-cu134image (CUDA 13.4) and no pip path. It uses the Blackwell flags and hasn't been tuned on Rubin; the GB200/GB300 PD and AgentX configurations don't apply. - NVIDIA Hopper: H100 or H200, TP8 on a full node, with the FP8 variant: FP8 KV and indexer caches, NVFP4 MoE on Marlin.
--block-size 128is mandatory (MSA sparse/index cache).
Launching the Server
Single node (TP4, AMD MI355X)
VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
-tp 4 \
--reasoning-parser minimax_m3 \
--block-size 128 \
--kv-cache-dtype nvfp4 \
--attention-config '{"indexer_kv_dtype": "nvfp4", "indexer_decode_cp": true}' \
--quantization-config.moe mxfp8 \
--speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \
--disable-hybrid-kv-cache-manager
--kv-cache-dtype nvfp4andindexer_kv_dtype: nvfp4store the KV cache and the sparse-attention indexer's K cache in NVFP4.indexer_decode_cpsplits the indexer's decode scoring across the TP ranks by context instead of by index head.--quantization-config.moe mxfp8re-quantizes the NVFP4 experts to MXFP8 at load for the AITER MoE.- With a seven-token DSpark draft (
num_speculative_tokens: 7),VLLM_ROCM_MINIMAX_M3_MONO_DRAFTER=1runs draft steps of up to 10 requests as fused FlyDSL kernels.
Parallel index top-k (AMD MI355X)
parallel_topk enables parallel decode top-16 selection for the NVFP4
indexer. It is disabled by default. Use a build containing the selector and
set VLLM_ROCM_PARALLEL_INDEX_TOPK=1, or enable the recipe feature.
FP8 indexers, including mono decode, bypass this path. Benchmark and CI
lanes must explicitly enable it in an image containing the patch; adding
the kernel or running correctness CI alone does not enable it for serving.
DBO and SP prefill (TP4, AMD MI355X)
Two opt-in switches make long prefill steps faster on the single-node MI355X engine (the DBO / SP prefill feature in the command builder):
- DBO (dual-batch overlap):
--enable-dbosplits a prefill step into two microbatches and runs one microbatch's all-reduce under the other's compute.--dbo-decode-token-threshold 32768keeps decode steps whole. DBO needsVLLM_USE_V2_MODEL_RUNNER=1: without it,--enable-dboselects the V1 runner, which refuses the DSpark draft. - SP (sequence-parallel prefill):
VLLM_ROCM_MINIMAX_M3_SP_PREFILL=1keeps the residual stream sharded by tokens on steps of 4096+ tokens, so each layer's two all-reduces become an all-to-all and a reduce-scatter. It keeps a fullo_projcopy per GPU, about 3.1 GB less KV cache. - SP + DBO: set both, with
--dbo-prefill-token-threshold 16384, so each DBO microbatch has 8192+ tokens and stays on the SP path.
On the MXFP8 MoE, one 32K-token prefill takes 1035 ms plain, 655 ms with DBO
and 484 ms with SP + DBO. The server log confirms SP with
MiniMax-M3 sequence-parallel prefill (from 4096 tokens, DBO microbatches from 8192): on.
VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ROCM_MINIMAX_M3_SP_PREFILL=1 \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
-tp 4 \
--reasoning-parser minimax_m3 \
--block-size 128 \
--kv-cache-dtype nvfp4 \
--attention-config '{"indexer_kv_dtype": "nvfp4", "indexer_decode_cp": true}' \
--quantization-config.moe mxfp8 \
--speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \
--enable-dbo \
--dbo-prefill-token-threshold 16384 \
--dbo-decode-token-threshold 32768 \
--disable-hybrid-kv-cache-manager
Single node (TP4, NVIDIA GB200)
VLLM_FLOAT32_MATMUL_PRECISION=high \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
-tp 4 \
--reasoning-parser minimax_m3 \
--block-size 128 \
--kv-cache-dtype nvfp4 \
--attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \
--compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \
--max-num-seqs 256 \
--disable-hybrid-kv-cache-manager
- Blackwell runs the NVFP4 experts natively, so no
--quantization-config.moeoverride is needed. use_prefill_query_quantizationquantizes the prefill query on the MSA indexer path (Q8KV4).
Single node (TP8, NVIDIA H100 / H200)
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
--tensor-parallel-size 8 \
--reasoning-parser minimax_m3 \
--block-size 128 \
--kv-cache-dtype fp8 \
--attention-config '{"indexer_kv_dtype": "fp8"}' \
--compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
--max-model-len 262144 \
--max-num-seqs 80 \
--limit-mm-per-prompt '{"video": 0}' \
--no-enable-log-requests \
--disable-hybrid-kv-cache-manager \
--moe-backend marlin \
--enable-auto-tool-choice \
--tool-call-parser minimax_m3
- Hopper has no FP4 tensor cores: the KV cache and the indexer K cache use FP8, and the NVFP4 experts run on the Marlin MoE kernel.
--limit-mm-per-prompt '{"video": 0}'turns off video inputs; images still work.- DSpark speculative decoding is not enabled on Hopper.
Prefill/decode disaggregation (1P1D, TP4 + TP4, AMD MI355X)
Prefill node:
VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
VLLM_KV_CACHE_LAYOUT=BLHNC VLLM_NIXL_SIDE_CHANNEL_HOST=<P_IP> \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
-tp 4 \
--reasoning-parser minimax_m3 \
--block-size 128 \
--kv-cache-dtype nvfp4 \
--attention-config '{"indexer_kv_dtype": "nvfp4"}' \
--quantization-config.moe mxfp8 \
--speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \
--disable-hybrid-kv-cache-manager \
--max-num-batched-tokens 32768 \
--port 8100 \
--kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[
{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_connector_extra_config":{"num_threads":4}},
{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":601295421440,"cpu_hugepage_block_size":"1GB","lazy_offload":false}}]}}'
Decode node:
VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
VLLM_KV_CACHE_LAYOUT=BLHNC VLLM_NIXL_SIDE_CHANNEL_HOST=<D_IP> \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
-tp 4 \
--reasoning-parser minimax_m3 \
--block-size 128 \
--kv-cache-dtype nvfp4 \
--attention-config '{"indexer_kv_dtype": "nvfp4", "indexer_decode_cp": true}' \
--quantization-config.moe mxfp8 \
--speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \
--disable-hybrid-kv-cache-manager \
--port 8200 \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"num_threads":4}}'
-
<P_IP>and<D_IP>are each node's own address, reachable from the other node (the NIXL side channel). -
VLLM_KV_CACHE_LAYOUT=BLHNCgoes on both nodes. It keeps all layers of a block contiguous, so NIXL reads one region per block instead of one per layer. -
The prefill node never decodes, so it leaves
indexer_decode_cpoff. -
The prefill node also offloads KV to host memory (
SimpleCPUOffloadConnector): 560 GiB per rank in 1 GiB HugeTLB pages, so 2240 pages for the four ranks. Reserve them on the host before the start (and allow them in the container's hugetlb limit), or the server fails at startup withCannot map 560.00 GiB of 1024 MiB HugeTLB pages:echo 2240 | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
Prefill/decode disaggregation (1P1D, DEP2 + DEP4, NVIDIA GB200 / GB300)
A DEP2 prefill on one 4-GPU tray feeds a DEP4 decode on a second tray over
NIXL. Both roles run attention data parallel with the NVFP4 experts
on the DeepGEMM MegaMoE kernel over expert parallelism, the V2 model runner,
NUMA binding, and a seven-token DSpark draft on the FlashInfer PrimiTS
(FLASHINFER_TS) backend. The settings were tuned on GB300; GB200 uses the
same commands.
Prefill node:
VLLM_FLOAT32_MATMUL_PRECISION=high VLLM_USE_V2_MODEL_RUNNER=1 \
VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_NIXL_SIDE_CHANNEL_HOST=<P_IP> \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
--data-parallel-size 2 \
--enable-expert-parallel \
--moe-backend deep_gemm_mega_moe \
--numa-bind \
--reasoning-parser minimax_m3 \
--block-size 128 \
--kv-cache-dtype nvfp4 \
--attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \
--compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \
--max-num-seqs 64 \
--max-num-batched-tokens 32768 \
--long-prefill-token-threshold 8192 \
--gpu-memory-utilization 0.9 \
--enable-prefix-caching \
--disable-hybrid-kv-cache-manager \
--port 8100 \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail","kv_connector_extra_config":{"engine_ttl":0}}'
Decode node:
VLLM_FLOAT32_MATMUL_PRECISION=high VLLM_USE_V2_MODEL_RUNNER=1 \
VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_NIXL_SIDE_CHANNEL_HOST=<D_IP> \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
--data-parallel-size 4 \
--enable-expert-parallel \
--moe-backend deep_gemm_mega_moe \
--numa-bind \
--reasoning-parser minimax_m3 \
--block-size 128 \
--kv-cache-dtype nvfp4 \
--attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "max_cudagraph_capture_size": 224}' \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \
--max-num-seqs 28 \
--max-num-batched-tokens 8192 \
--gpu-memory-utilization 0.95 \
--watermark 0.05 \
--no-enable-prefix-caching \
--disable-hybrid-kv-cache-manager \
--port 8200 \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_connector_extra_config":{"engine_ttl":0}}'
VLLM_ENGINE_READY_TIMEOUT_S=3600: with data parallel, each API server waits this long for its engines to start (default 600 s). A cold prefill start (JIT builds, CUDA graph capture, FlashInfer autotune) takes longer than 10 minutes.--max-num-seqsis per DP rank. The decode CUDA graphs cover 28 requests of 1 + 7 tokens (224).--watermark 0.05keeps 5% of the decode KV cache free when admitting requests, so extra requests wait in the queue instead of preempting running ones. On GB300 (about 4.5M KV tokens per decode rank) at a 200K-token mean context, expect about 19-21 running requests per rank.- Keep the decode
--max-num-batched-tokensat 8192: a preempted request recomputes its context in chunks of that size, and every DP rank runs eager steps until it finishes. engine_ttl: 0turns off NIXL's cleanup of idle remote engines; with the default one-hour TTL, a decode crashed an hour after its prefill went away.- Prefix caching stays on the prefill node only. For multi-turn traffic, the
prefill node can also offload KV to host memory, 180 GiB per rank:
--kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_connector_extra_config":{"engine_ttl":0}},{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":193273528320,"lazy_offload":true}}]}}'.
For a quick test, put the toy NIXL proxy from the vLLM repository in front of the two nodes and send requests to port 8000:
python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py --port 8000 \
--prefiller-host <P_IP> --prefiller-port 8100 \
--decoder-host <D_IP> --decoder-port 8200
Client Usage
Recommended sampling parameters (from the model card): temperature = 1.0,
top_p = 0.95. Set the reasoning effort per request with
chat_template_kwargs: {"reasoning_effort": "<level>"}: low, medium,
high, xhigh or max.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "MiniMaxAI/MiniMax-M3.1-Flash",
"temperature": 1.0,
"top_p": 0.95,
"messages": [
{"role": "user", "content": "Explain MSA sparse attention in 3 bullets."}
]
}'
Benchmarking
vllm bench serve \
--backend vllm \
--model MiniMaxAI/MiniMax-M3.1-Flash \
--endpoint /v1/completions \
--dataset-name random \
--random-input 8192 \
--random-output 1024 \
--max-concurrency 32 \
--num-prompts 320
Troubleshooting
--block-sizemismatch. MSA's sparse block size is 128; the vLLM KV cache block size must match (otherwiseNo common block size for 16).- Parsers.
--tool-call-parserand--reasoning-parserboth useminimax_m3. - HugeTLB pages (PD prefill node). See the AMD PD section above.
References
Configurations qualified on specific hardware, layered on the baseline command. Unless marked Opt-in, a preset applies automatically on its hardware and strategy; pick Baseline in the builder to drop it.
GB200 NVL42 presetsAgentX TP4 (Dynamo) · AgentX PD (Dynamo)
Aggregated TP4 vLLM workers, one per 4-GPU GB200 tray, behind the Dynamo KV router with session affinity. Seven-token DSpark draft on FLASH_ATTN and prefill CUDA graphs up to 8192 tokens. KV stays in HBM, or spills to host DRAM through SimpleCPUOffloadConnector.
2 points: GPU-resident KV, Host-DRAM KV offload · image vllm/vllm-openai:minimax-m31-flash-arm64-cu134 · dynamo 1.6.0.de · pip ai-dynamo-runtime==1.6.0.dev20261023 · pip ai-dynamo==1.6.0.dev20261023
Runbook
The GB200 aggregated AgentX lane in SemiAnalysis InferenceX (minimaxm31flash-fp4-gb200-dynamo-vllm-agentic-dspark-agg, SemiAnalysisAI/InferenceX#4632) serves the default checkpoint text-only, with one TP4 dynamo.vllm worker per 4-GPU tray behind the Dynamo frontend. Add trays to scale out: the KV router spreads conversations across them. At 16 sessions per tray (GPU-resident KV) the lane measured 29.3k tok/s/GPU at 402 tok/s/user.
Image and versions. The lane used an arm64 CUDA 13 build of the vllm/vllm-openai:minimax-m31-flash image (tag minimax-m31-flash-arm64-cu134); any build of that image family for GB200 will do. Dynamo 1.6.0.dev20261023 was installed on top: pip install --extra-index-url https://pypi.nvidia.com ai-dynamo-runtime==1.6.0.dev20261023 ai-dynamo==1.6.0.dev20261023.
Tuning rules.
--max-num-seqsis about twice the sessions one worker serves: 32 up to 16 sessions, 64 at 32-40.- GPU-resident KV (
--gpu-memory-utilization 0.88) is enough up to about 16 sessions per tray. Beyond that, multi-turn contexts outgrow HBM and evicted prefixes are recomputed. - Host-DRAM KV offload adds
SimpleCPUOffloadConnectorwith lazy offload.cpu_bytes_to_useis the budget for the whole worker, split across its four ranks: 360 GB pins 90 GB per rank. Size it from the host DRAM that is free after the engine processes are up. Lower--gpu-memory-utilizationto 0.82 to leave room for the transfer buffers.--kv-cache-metricsexports KV residency metrics, which help with that sizing; it can be left out. - The CUDA graph sizes go up to 8192 tokens, so prefill chunks of up to 8192 tokens also replay graphs.
--max-num-batched-tokens 32768still allows larger mixed steps, which run piecewise.
Running under Dynamo. Start etcd and NATS (32 MB max payload) once:
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<infra-host>:2379
printf 'max_payload: 33554432\n' > nats.conf && nats-server -c nats.conf
On each tray, export ETCD_ENDPOINTS=http://<infra-host>:2379 and NATS_SERVER=nats://<infra-host>:4222 together with the generated environment and DYN_TCP_CONNECT_TIMEOUT=120, DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT=1024, DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT=1024 and DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT=2048. Then run the generated flags with python3 -m dynamo.vllm --model MiniMaxAI/MiniMax-M3.1-Flash --served-model-name MiniMaxAI/MiniMax-M3.1-Flash --trust-remote-code in place of vllm serve MiniMaxAI/MiniMax-M3.1-Flash. dynamo.vllm rejects vLLM's API-server flags, so leave out --tool-call-parser minimax_m3 --enable-auto-tool-choice and add:
--dyn-tool-call-parser minimax_m3 --dyn-reasoning-parser minimax_m3 \
--kv-events-config '{"publisher":"zmq","topic":"kv-events","enable_kv_cache_events":true,"endpoint":"tcp://*:5200"}'
Start the frontend:
export ETCD_ENDPOINTS=http://<infra-host>:2379 NATS_SERVER=nats://<infra-host>:4222
export DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TCP_CONNECT_TIMEOUT=120 DYN_TOKENIZER=fastokens
python3 -m dynamo.frontend --http-port 8000 --router-mode kv --router-session-affinity-ttl-secs 14400
DYN_TOKENIZER=fastokens switches the frontend to the fastokens tokenizer backend that ships with Dynamo. With 100K+ token agentic prompts, the default tokenizer spends about 150 ms per request on the frontend (up to 600 ms at p90), all of it added to TTFT; fastokens takes about 1 ms.
The KV router uses the workers' KV events to send each request to the worker that holds its prefix. Session affinity keeps a conversation on that worker for up to four hours; send a stable conversation ID in the X-Dynamo-Session-ID request header.
The InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04), so every point is measured at the same acceptance rate. Production serving keeps the default rejection sampler (rejection_sample_method: block), as in the command above.
DEP4 prefill workers, one per GB200 tray, feed one DEP8 decode worker (throughput) or DEP16 (lower latency) over NIXL. Both roles share a Mooncake KV store in host DRAM. The Dynamo KV router routes with session affinity. Seven-token DSpark draft on FLASHINFER_TS, DeepGEMM MegaMoE on both roles.
2 points: DEP8 decode (throughput), DEP16 decode (lower latency) · image vllm/vllm-openai:minimax-m31-flash-arm64-cu134 · dynamo 1.6.0.de · pip ai-dynamo-runtime==1.6.0.dev20261023 · pip ai-dynamo==1.6.0.dev20261023 · pip mooncake-transfer-engine-cuda13==0.3.13.post1
RunbookTuning rules · Services · Workers · Frontend
The GB200 disaggregated AgentX lane in SemiAnalysis InferenceX (minimaxm31flash-fp4-gb200-dynamo-vllm-agentic-dspark-disagg, SemiAnalysisAI/InferenceX#4632) serves the default checkpoint text-only with Dynamo and vLLM. Several prefill workers, each DEP4 on one 4-GPU tray, feed one decode worker that is DEP8 across two trays or DEP16 across four. KV moves from prefill to decode over NIXL (UCX over RDMA), and every rank contributes host DRAM to a shared Mooncake KV store, so a returning conversation's prefix is loaded from the store instead of recomputed. Both roles use expert parallelism with DeepGEMM MegaMoE, attention data parallelism, the V2 model runner and the seven-token DSpark draft on FLASHINFER_TS.
| Prefill workers (DEP4) | Decode | GPUs | Sessions | tok/s/GPU | tok/s/user | TTFT p90 | E2E latency p90 |
|---|---|---|---|---|---|---|---|
| 2 | DEP8 | 16 | 224 | 94.2k | 250 | 3.0 s | 12.2 s |
| 2 | DEP8 | 16 | 256 | 98.8k | 244 | 5.3 s | 15.7 s |
| 3 | DEP8 | 20 | 256 | 86.4k | 240 | 2.4 s | 12.1 s |
| 3 | DEP8 | 20 | 288 | 93.2k | 230 | 3.5 s | 13.3 s |
| 3 | DEP16 | 28 | 280 | 70.4k | 302 | 1.9 s | 9.4 s |
| 3 | DEP16 | 28 | 320 | 77.3k | 291 | 2.2 s | 10.0 s |
| 3 | DEP16 | 28 | 360 | 85.4k | 284 | 2.5 s | 10.6 s |
| 4 | DEP16 | 32 | 352 | 74.5k | 284 | 1.8 s | 9.9 s |
The preset renders one prefill worker and the decode worker. Run more prefill workers with the same command, one per tray.
Image and versions. The lane used an arm64 CUDA 13 build of the vllm/vllm-openai:minimax-m31-flash image (tag minimax-m31-flash-arm64-cu134), with Dynamo 1.6.0.dev20261023 and Mooncake mooncake-transfer-engine-cuda13==0.3.13.post1 installed on every worker and on the Mooncake master host: pip install --extra-index-url https://pypi.nvidia.com ai-dynamo-runtime==1.6.0.dev20261023 ai-dynamo==1.6.0.dev20261023 mooncake-transfer-engine-cuda13==0.3.13.post1.
Tuning rules
- Prefill pool. The prefill pool sets TTFT, so scale prefill workers with load. The lane's points kept TTFT p90 between 1.8 and 5.3 s with one DEP4 prefill tray per 85-128 concurrent agentic sessions (2 trays at 224-256 sessions, 3 at 256-360, 4 at 352). Two prefill trays per DEP8 decode give the most tokens per GPU; a third keeps TTFT p90 near 3.5 s up to 288 sessions. Add a prefill tray when TTFT p90 climbs while decode still has room.
- DEP8 or DEP16 decode. DEP8 gives the most tokens per GPU: 28-36 sessions per decode GPU at 230-250 tok/s/user. DEP16 halves the batch per rank and gives about 25% more tok/s/user (302 at 280 sessions) and holds E2E latency p90 near 10 s up to 360 sessions, at about 10-20% fewer tokens per GPU.
--max-num-seqs 64per DP rank is a cap, not the operating point. The decode graph cap is--max-num-seqsx (1 + 7 draft tokens) = 512.--watermark 0.05keeps 5% of the decode KV cache free, so extra requests wait in the queue instead of preempting running ones. Keep decode--max-num-batched-tokensat 8192: a preempted request recomputes its context in chunks of that size. - Prefill batching.
--long-prefill-token-threshold 4096chunks long prompts into 4096-token pieces. Short prompts then share each step with them instead of waiting behind a 100K-token context, which lowers the TTFT tail. The recipe's 1P1D section above uses 8192.--max-num-batched-tokens 16384bounds a prefill step. Prefill captures CUDA graphs up to 2048 tokens. Prefill runs at--gpu-memory-utilization 0.92, decode at 0.88. - Mooncake pool size.
global_segment_sizeis registered by every DP rank, prefill and decode alike, so a tray pins four times that value. The lane used 190 GB per rank (760 GB per tray), which lets three DEP4 prefill trays serve up to 288 sessions with DEP8 and 360 with DEP16; 210 GB per rank ran the prefill trays out of host memory. Size it as (host DRAM free per tray after the engines are up - headroom for NIXL and the 4 GB local buffers) / ranks per tray. The decode ranks only add capacity to the pool (kv_consumerwithenable_lookup: falseandsave_decode_cache: false): the prefill workers write and read every block. - Routing. The KV router (
--router-mode kv) reads the prefill workers' KV events and sends each request to the worker that already holds its prefix. Session affinity (--router-session-affinity-ttl-secs 14400) keeps a conversation on that worker for up to four hours. Send a stable conversation ID in theX-Dynamo-Session-IDheader. The Mooncake store catches the prefixes that miss after a reroute. - Transport. Cross-tray NIXL needs
UCX_TLS=rc,cuda_copywithUCX_NET_DEVICESset to the tray's RDMA ports. Without them, UCX can fall back to TCP and transfers stall. PointUCX_MODULE_DIRat the UCX bundled with the NIXL wheel (nixl_cu13.libs/ucxin the image'sdist-packages).kv_lease_duration: 120holds a prefill's blocks for up to 120 s until decode reads them (the default is 30 s). - NUMA.
--numa-bindpins each rank to its GPU's NUMA node. Decode passes the tray's map,--numa-bind-nodes 0 0 1 1, explicitly. --moe-backend deep_gemm_mega_moeis the same as the lane's--kernel-config '{"moe_backend":"deep_gemm_mega_moe"}'.--prefill-schedule-interval 1, also set by the lane, is vLLM's default.
Services
Start etcd, NATS and the Mooncake master once, on a host that runs no engine workers, together with the frontend. The engine trays' host DRAM then goes to the Mooncake pool.
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<infra-host>:2379
printf 'max_payload: 33554432\n' > nats.conf && nats-server -c nats.conf
mooncake_master --rpc_port=50051 \
--default_kv_lease_ttl=60000 --eviction_high_watermark_ratio=0.95 --eviction_ratio=0.10
Every worker reads the store configuration from the JSON file that MOONCAKE_CONFIG_PATH points to (export it with the rest of the environment):
{"master_server_address": "<infra-host>:50051", "metadata_server": "P2PHANDSHAKE",
"global_segment_size": "190GB", "local_buffer_size": "4GB", "protocol": "rdma",
"device_name": "mlx5_0,mlx5_1,mlx5_2,mlx5_3", "enable_offload": false}
Workers
Export ETCD_ENDPOINTS=http://<infra-host>:2379, NATS_SERVER=nats://<infra-host>:4222 and the generated environment on every tray, then run the generated python3 -m dynamo.vllm commands. Add --served-model-name MiniMaxAI/MiniMax-M3.1-Flash --trust-remote-code --dyn-tool-call-parser minimax_m3 --dyn-reasoning-parser minimax_m3 to both roles. dynamo.vllm rejects vLLM's API-server flags --tool-call-parser and --enable-auto-tool-choice. Multi-node DEP decode runs one process per tray with --data-parallel-start-rank set to 0, 4, 8 and so on, and a shared --data-parallel-address and --data-parallel-rpc-port. --data-parallel-hybrid-lb lets every tray's Dynamo runtime register its own DP ranks with the frontend.
Frontend
export ETCD_ENDPOINTS=http://<infra-host>:2379 NATS_SERVER=nats://<infra-host>:4222
export DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TCP_CONNECT_TIMEOUT=120 DYN_TOKENIZER=fastokens
python3 -m dynamo.frontend --http-port 8000 --router-mode kv --router-session-affinity-ttl-secs 14400
DYN_TOKENIZER=fastokens switches the frontend to the fastokens tokenizer backend that ships with Dynamo. With 100K+ token agentic prompts, the default tokenizer spends about 150 ms per request on the frontend (up to 600 ms at p90), all of it added to TTFT; fastokens takes about 1 ms.
The InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04), so every point is measured at the same acceptance rate. Production serving keeps the default rejection sampler (rejection_sample_method: block), as in the command above.
B3002 presetsAgentX TP4 · AgentX TP2 + KV offload
One TP4 engine on four of the node's eight B300 GPUs with the seven-token DSpark draft on FLASH_ATTN; GPU-resident KV up to concurrency 32, host-DRAM KV offload (about 1.5 TB per engine) at 24 and 40-56.
9 points · concurrency 1–56 · image vllm/vllm-openai:minimax-m31-flash@sha256:86f582dae25e…
Runbook
The B300 AgentX lane in SemiAnalysis InferenceX (minimaxm31flash-fp4-b300-vllm-agentic-dspark, SemiAnalysisAI/InferenceX#4467, run 40340506758) serves the default checkpoint (MXFP8 dense layers, NVFP4 experts, NVFP4 KV and indexer K caches) as one aggregated text-only vLLM engine on a single 8-GPU B300 node. Its TP4 points run the seven-token DSpark draft with FLASH_ATTN draft attention; the recipe's NVIDIA default, FLASHINFER_TS, is what the AgentX TP2 points run.
- GPU-resident points (concurrency 1-16 and 32) keep the whole KV cache in HBM, with
--max-num-seqsat twice the concurrency (64 at 32). - KV offload points (24 and 40-56) add a host-DRAM KV tier through
SimpleCPUOffloadConnectorwith lazy offload.cpu_bytes_to_use(1,499 GB) is the budget for the whole engine: each of the four ranks pins 349 GiB of host memory, so the engine needs about 1.5 TB of free host RAM. Pinning takes a couple of minutes at startup. - The CUDA graph capture sizes go up to 8192 tokens, so prefill chunks of up to 8192 tokens also replay graphs.
The InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04) so every point is measured at the same acceptance rate. The command above keeps block verification.
One TP2 engine on two of the node's eight B300 GPUs with the seven-token DSpark draft on FLASHINFER_TS and host-DRAM KV offload (about 750 GB per engine).
4 points · concurrency 24–48 · image vllm/vllm-openai:minimax-m31-flash@sha256:86f582dae25e…
Runbook
The TP2 points of the B300 AgentX lane in SemiAnalysis InferenceX (minimaxm31flash-fp4-b300-vllm-agentic-dspark, SemiAnalysisAI/InferenceX#4467, run 40340506758) serve the default checkpoint as one aggregated text-only vLLM engine on two GPUs of an 8-GPU B300 node, with the recipe's seven-token DSpark draft on FLASHINFER_TS (the AgentX TP4 points use FLASH_ATTN).
Every point adds a host-DRAM KV tier through SimpleCPUOffloadConnector with lazy offload. cpu_bytes_to_use (749 GB) is the budget for the whole engine: each of the two ranks pins about 349 GiB of host memory. The lane ran one engine per node; several TP2 engines on one node each need their own share of host RAM.
The InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04) so every point is measured at the same acceptance rate. The command above keeps block verification.
GB300 NVL45 presetsAgentX TP4 (Dynamo) · AgentX PD (Dynamo) · AgentX TP4 · AgentX DEP4 · AgentX TP2 + KV offload
Aggregated TP4 vLLM engines, one per 4-GPU GB300 tray, behind Dynamo sidecars and the Dynamo KV router with session affinity. Seven-token DSpark draft on FLASH_ATTN, prefill CUDA graphs up to 8192 tokens, KV in HBM.
2 points: Lower latency, Throughput · image vllm/vllm-openai:cu134-arm64-1107-mdev@sha256:1b4a9b46c217… · dynamo 1.6.0.de · pip ai-dynamo-runtime==1.6.0.dev20261023 · pip ai-dynamo==1.6.0.dev20261023
Runbook
The GB300 aggregated AgentX lane in SemiAnalysis InferenceX (minimaxm31flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg, SemiAnalysisAI/InferenceX#4998, run 42955087162) serves the default checkpoint text-only, with one TP4 vLLM engine per 4-GPU tray behind a Dynamo sidecar and the Dynamo frontend. Add trays to scale out: the KV router spreads conversations across them.
| Sessions per tray | --max-num-seqs | tok/s/GPU | tok/s/user | TTFT p90 | E2E latency p90 |
|---|---|---|---|---|---|
| 1 | 2 | 7.4k | 580 | 0.3 s | 9.3 s |
| 4 | 8 | 9.6k | 576 | 0.3 s | 7.0 s |
| 8 | 16 | 16.4k | 512 | 0.3 s | 6.4 s |
| 16 | 32 | 29.2k | 426 | 0.4 s | 6.2 s |
| 32 | 64 | 61.8k | 301 | 0.6 s | 9.4 s |
Image and versions. vllm/vllm-openai:cu134-arm64-1107-mdev (arm64, CUDA 13) with Dynamo 1.6.0.dev20261023 installed on top: pip install --extra-index-url https://pypi.nvidia.com ai-dynamo-runtime==1.6.0.dev20261023 ai-dynamo==1.6.0.dev20261023.
Tuning rules.
--max-num-seqsis about twice the sessions one worker serves: 32 up to 16 sessions, 64 at 32. The KV cache stays in HBM throughout.--stream-interval 20sends output in 20-token chunks; AgentX requires 20 to 60, and vLLM's default of 1 sends every token.--prefill-schedule-interval 24admits prefill on one engine step in 24 and lets the other 23 decode alone, which keeps the per-user decode rate up while long agentic prompts arrive.- The CUDA graph sizes go up to 8192 tokens, so prefill chunks of up to 8192 tokens also replay graphs.
--max-num-batched-tokens 32768still allows larger mixed steps, which run piecewise.
Running under Dynamo. Start etcd once (etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<infra-host>:2379); the request plane is TCP and the event plane ZMQ, so no NATS is needed. On each tray, export ETCD_ENDPOINTS=http://<infra-host>:2379 DYN_REQUEST_PLANE=tcp DYN_EVENT_PLANE=zmq together with the generated environment and DYN_TCP_CONNECT_TIMEOUT=120, DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT=1024, DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT=1024 and DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT=2048. Run the generated flags with vllm-rs serve MiniMaxAI/MiniMax-M3.1-Flash --served-model-name MiniMaxAI/MiniMax-M3.1-Flash --trust-remote-code --grpc-port 19000 and --kv-events-config '{"publisher":"zmq","topic":"kv-events","enable_kv_cache_events":true,"endpoint":"tcp://*:5200"}', leaving out the engine's --tool-call-parser, --enable-auto-tool-choice and --reasoning-parser. Then connect a sidecar, which parses tool calls and reasoning:
DYN_VLLM_KV_EVENT_PORT=5200 python3 -m dynamo.vllm.sidecar --grpc-endpoint 127.0.0.1:19000 \
--grpc-startup-deadline-secs 3600 --dyn-tool-call-parser minimax_m3 --dyn-reasoning-parser minimax_m3
Start the frontend:
export ETCD_ENDPOINTS=http://<infra-host>:2379 DYN_REQUEST_PLANE=tcp DYN_EVENT_PLANE=zmq
export DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TCP_CONNECT_TIMEOUT=120 DYN_TOKENIZER=fastokens
python3 -m dynamo.frontend --http-port 8000 --router-mode kv --router-session-affinity-ttl-secs 14400
DYN_TOKENIZER=fastokens switches the frontend to the fastokens tokenizer backend that ships with Dynamo. With 100K+ token agentic prompts, the default tokenizer spends about 150 ms per request on the frontend (up to 600 ms at p90), all of it added to TTFT; fastokens takes about 1 ms.
The KV router uses the workers' KV events to send each request to the worker that holds its prefix. Session affinity keeps a conversation on that worker for up to four hours; send a stable conversation ID in the X-Dynamo-Session-ID request header.
The InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04), so every point is measured at the same acceptance rate. Production serving keeps the default rejection sampler (rejection_sample_method: block), as in the command above.
DEP4 prefill workers, one per GB300 tray, feed one DEP16 decode worker across four trays over NIXL. A Mooncake KV store holds prefill KV for returning sessions. The Dynamo KV router keeps each session on one decode rank. Seven-token DSpark draft on FLASHINFER_TS, DeepGEMM MegaMoE on both roles, EPLB on decode.
2 points: Throughput, Lower latency · image vllm/vllm-openai:cu134-arm64-1107-mdev@sha256:1b4a9b46c217… · dynamo 1.6.0.de · pip ai-dynamo-runtime==1.6.0.dev20261023 · pip ai-dynamo==1.6.0.dev20261023
RunbookTuning rules · Services · Workers · Frontend
The GB300 disaggregated AgentX lane in SemiAnalysis InferenceX (minimaxm31flash-fp4-gb300-dynamo-vllm-agentic-dspark-disagg, SemiAnalysisAI/InferenceX#4998, run 42955087162) serves the default checkpoint text-only with Dynamo and vLLM. One to six prefill workers, each DEP4 on one 4-GPU tray, feed one decode worker that is DEP16 across four trays. KV moves from prefill to decode over NIXL (UCX over RDMA), and the prefill workers share a Mooncake KV store, so a returning conversation's prefix is loaded from the store instead of recomputed. Both roles use expert parallelism with DeepGEMM MegaMoE, attention data parallelism, the V2 model runner and the seven-token DSpark draft on FLASHINFER_TS.
| Prefill workers (DEP4) | Decode | GPUs | Sessions | Decode --max-num-seqs | tok/s/GPU | tok/s/user | TTFT p90 | E2E latency p90 |
|---|---|---|---|---|---|---|---|---|
| 1 | DEP16 | 20 | 200 | 5 | 66.7k | 351 | 4.2 s | 11.2 s |
| 3 | DEP16 | 28 | 500 | 10 | 111.8k | 283 | 4.1 s | 12.9 s |
| 4 | DEP16 | 32 | 640 | 16 | 118.5k | 257 | 6.1 s | 16.0 s |
| 4 | DEP16 | 32 | 680 | 16 | 116.1k | 255 | 10.3 s | 19.6 s |
| 5 | DEP16 | 36 | 850 | 22 | 118.3k | 233 | 13.4 s | 23.7 s |
| 6 | DEP16 | 40 | 900 | 20 | 126.8k | 215 | 6.2 s | 18.1 s |
| 6 | DEP16 | 40 | 950 | 22 | 125.2k | 212 | 9.7 s | 21.5 s |
| 6 | DEP16 | 40 | 1000 | 24 | 116.2k | 215 | 17.5 s | 28.3 s |
The preset renders one prefill worker and the decode worker. Run more prefill workers with the same command, one per tray.
Image and versions. vllm/vllm-openai:cu134-arm64-1107-mdev (arm64, CUDA 13) ships Mooncake 0.3.13.post1 with the GPU-memory store segment (hbm_pool) and the decode KV-load limits (--max-num-inflight-kv-loads, --max-async-kv-loads-per-step); neither is in a vLLM release yet. Install Dynamo 1.6.0.dev20261023 on top on every worker and on the frontend host: pip install --extra-index-url https://pypi.nvidia.com ai-dynamo-runtime==1.6.0.dev20261023 ai-dynamo==1.6.0.dev20261023.
Tuning rules
- Prefill pool. The prefill pool sets TTFT, so scale prefill workers with load: about 150-170 sessions per DEP4 prefill tray kept TTFT p90 near 4-6 s (3 workers at 500, 4 at 640, 6 at 900-950). At 680 sessions on 4 workers and 850 on 5, TTFT p90 rose past 10 s; add a prefill worker before that.
- Decode.
--max-num-seqsper DP rank is about a third of the sessions per rank in the lane (10 at 500 sessions, 20 at 900, 24 at 1000); raise it with the load. The decode graph cap is--max-num-seqsx (1 + 7 draft tokens).--watermark 0.05keeps 5% of the decode KV cache free, so extra requests wait in the queue instead of preempting running ones. Keep decode--max-num-batched-tokensat 8192: a preempted request recomputes its context in chunks of that size. EPLB keeps 16 redundant experts across the 16 ranks. - KV loads on decode.
--max-num-inflight-kv-loads 64caps the NIXL loads a decode rank has in flight, and--max-async-kv-loads-per-step 4caps how many requests with finished KV loads decode admits per step, so a burst of arrivals does not stall running requests. - Prefill batching.
--long-prefill-token-threshold 4096chunks long prompts into 4096-token pieces. Short prompts then share each step with them instead of waiting behind a 100K-token context, which lowers the TTFT tail.--max-num-batched-tokens 16384bounds a prefill step. Prefill captures CUDA graphs up to 2048 tokens. - Mooncake store. The prefill ranks write their KV into the store and read returning prefixes from it. With
"hbm_pool": {"segment_size": "auto"}, each prefill rank also registers the GPU memory it does not need for its own KV cache as a store segment, so hot prefixes are copied GPU to GPU. The decode connector is a consumer withsave_decode_cache: falseandenable_lookup: false: decode only receives KV over NIXL. - Transport. Cross-tray NIXL needs
UCX_TLS=rc,cuda_copywithUCX_NET_DEVICESset to the tray's RDMA ports. Without them, UCX can fall back to TCP and transfers stall.UCX_MODULE_DIRpoints at the UCX bundled with the NIXL wheel.kv_lease_duration: 120holds a prefill's blocks for up to 120 s until decode reads them (the default is 30 s). - NUMA.
--numa-bindpins each rank to its GPU's NUMA node. Decode passes the tray's map,--numa-bind-nodes 0 0 1 1, explicitly. --moe-backend deep_gemm_mega_moeis the same as the lane's--kernel-config '{"moe_backend":"deep_gemm_mega_moe"}'.--prefill-schedule-interval 1, also set by the lane, is vLLM's default.
Services
Start etcd and the Mooncake master once, on a host that runs no engine workers, together with the frontend. The request plane is TCP and the event plane ZMQ, so no NATS is needed.
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<infra-host>:2379
mooncake_master --rpc_port=50051 \
--default_kv_lease_ttl=60000 --eviction_high_watermark_ratio=0.95 --eviction_ratio=0.10
Every worker reads the store configuration from the JSON file that MOONCAKE_CONFIG_PATH points to (export it with the rest of the environment):
{"master_server_address": "<infra-host>:50051", "metadata_server": "P2PHANDSHAKE",
"global_segment_size": "180GB", "local_buffer_size": "4GB", "protocol": "rdma",
"device_name": "mlx5_0,mlx5_1,mlx5_2,mlx5_3", "enable_offload": false}
Workers
Export ETCD_ENDPOINTS=http://<infra-host>:2379 DYN_REQUEST_PLANE=tcp DYN_EVENT_PLANE=zmq and the generated environment on every tray. Run each generated engine command as vllm-rs serve MiniMaxAI/MiniMax-M3.1-Flash --served-model-name MiniMaxAI/MiniMax-M3.1-Flash --trust-remote-code --grpc-port 19000, without the generated --reasoning-parser and --dyn-reasoning-parser, and connect one sidecar per engine, which parses tool calls and reasoning:
python3 -m dynamo.vllm.sidecar --grpc-endpoint 127.0.0.1:19000 --grpc-startup-deadline-secs 3600 \
--dyn-tool-call-parser minimax_m3 --dyn-reasoning-parser minimax_m3 \
--disaggregation-mode prefill --component prefill # decode: --disaggregation-mode decode
The engines publish no KV events. Multi-node DEP decode runs one process per tray with --data-parallel-start-rank set to 0, 4, 8 and 12, and a shared --data-parallel-address and --data-parallel-rpc-port.
Frontend
export ETCD_ENDPOINTS=http://<infra-host>:2379 DYN_REQUEST_PLANE=tcp DYN_EVENT_PLANE=zmq
export DYN_ROUTER_SESSION_AFFINITY_MODE=soft DYN_ROUTER_TEMPERATURE=0 DYN_ROUTER_QUEUE_THRESHOLD=None
export DYN_ROUTER_TTL_SECS=3600 DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TCP_CONNECT_TIMEOUT=120
export DYN_TCP_CHANNEL_BUFFER=128 DYN_TCP_REQUEST_TIMEOUT=60 DYN_TCP_POOL_SIZE=8192
export DYN_TOKENIZER=fastokens DYN_TOKENIZER_CACHE=1 DYN_TOKENIZER_CACHE_BYTES=8000000000
python3 -m dynamo.frontend --http-port 8000 --router-mode kv --router-session-affinity-ttl-secs 14400 \
--no-router-kv-events --active-decode-blocks-threshold None \
--active-prefill-tokens-threshold None --active-prefill-tokens-threshold-frac None
Without KV events the router predicts each worker's prefix cache from its own routing decisions. Soft session affinity keeps each conversation on one decode rank for up to four hours; send a stable conversation ID in the X-Dynamo-Session-ID header. The lane sent each prefill to the least-loaded prefill worker, ignoring session affinity (InferenceX's dynamo-two-tier-cost-fn worker selection with cache_threshold: 1.0); the Mooncake store catches the prefixes that then miss.
DYN_TOKENIZER=fastokens switches the frontend to the fastokens tokenizer backend that ships with Dynamo. With 100K+ token agentic prompts, the default tokenizer spends about 150 ms per request on the frontend (up to 600 ms at p90), all of it added to TTFT; fastokens takes about 1 ms.
The InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04), so every point is measured at the same acceptance rate. Production serving keeps the default rejection sampler (rejection_sample_method: block), as in the command above.
One TP4 engine on a 4-GPU GB300 tray with the seven-token DSpark draft on FLASHINFER_TS; GPU-resident KV, with per-concurrency scheduling and CUDA graph settings.
7 points · concurrency 1–40 · image vllm/vllm-openai:minimax-m31-flash-arm64-cu134@sha256:5d7634762897…
Runbook
The GB300 TP4 points of the AgentX lane in SemiAnalysis InferenceX (minimaxm31flash-fp4-gb300-vllm-agentic-dspark, SemiAnalysisAI/InferenceX#4662) serve the default checkpoint text-only on one TP4 engine per 4-GPU tray, with --max-num-seqs set per concurrency.
- Concurrency 1, 4 and 32 use the default CUDA graph capture sizes.
- Concurrency 8 and 16 capture CUDA graphs for every size up to 8192 tokens.
- Concurrency 24 raises
--max-num-batched-tokensto 32768 and schedules prefill every 16 steps; concurrency 40 keeps 8192 and raises--gpu-memory-utilizationto 0.95. Both cap prefill chunks at 8192 tokens and capture graphs to 8192.
One data-parallel, expert-parallel engine across the four GPUs of a GB300 tray behind vllm-router with session affinity. Three-token DSpark draft on FLASHINFER_TS.
1 point: 64 · image vllm/vllm-openai:minimax-m31-flash-arm64-cu134@sha256:5d7634762897…
Runbook
The DEP4 point of the GB300 AgentX lane in SemiAnalysis InferenceX (minimaxm31flash-fp4-gb300-vllm-agentic-dspark, SemiAnalysisAI/InferenceX#4662) runs one engine with --data-parallel-size 4 and --enable-expert-parallel; --max-num-seqs is the per-rank limit.
The router uses the consistent_hash policy so a conversation stays on the rank that holds its prefix. The lane also runs it with --request-timeout-secs 14400 and --disable-retries.
One TP2 engine on two of the tray's four GB300 GPUs with the seven-token DSpark draft on FLASHINFER_TS and host-DRAM KV offload (482 GB per engine).
3 points · concurrency 28–36 · image vllm/vllm-openai:minimax-m31-flash-arm64-cu134@sha256:5d7634762897…
Runbook
The TP2 points of the GB300 AgentX lane in SemiAnalysis InferenceX (minimaxm31flash-fp4-gb300-vllm-agentic-dspark, SemiAnalysisAI/InferenceX#4662) add a host-DRAM KV tier through SimpleCPUOffloadConnector with lazy offload. cpu_bytes_to_use (482 GB) is the budget for the whole engine, about half of the tray's host DRAM, so the engine needs that much free host RAM. --max-num-seqs is 1.5 times the concurrency.
MI355X2 presetsAgentX TP4 · AgentX 1P2D (MoRI-IO)
One TP4 engine on four of the node's eight MI355X GPUs, serving the Quark MXFP4 checkpoint on the minimax-m31-flash-75bc8cf1 image: NVFP4 KV cache, DBO and breakable CUDA graphs at every point, seven-token DSpark up to concurrency 32 and three-token DSpark at 64, with a host-DRAM KV tier (LRU eviction) at 32 and 64.
7 points · concurrency 1–64 · image vllm/vllm-openai-rocm:minimax-m31-flash-75bc8cf1@sha256:f8e8ab239e5e…
Runbook
The MI355X AgentX lane in SemiAnalysis InferenceX
(minimaxm31flash-fp4-mi355x-vllm-agentic-dspark,
SemiAnalysisAI/InferenceX#5035),
on the Quark MXFP4 checkpoint with --max-num-seqs at twice the concurrency.
Throughput runs replaced block verification with synthetic rejection at the
committed golden DSpark acceptance length (4.04 for seven tokens, 2.88 for three);
the commands above keep block verification.
The image minimax-m31-flash-75bc8cf1 is vLLM 0.33.0rc1.dev910+g4fab5a074 with
the MiniMax-M3.1 ROCm serving stack, the amd_kernels FlyDSL library and FlyDSL
0.5.0. It routes the dense GEMMs, MoE, qkv, sparse attention, indexer, norms, DSpark
draft, sampling, top-p and sequence-parallel prefill to amd_kernels by default
(the server logs amd_kernels routes: gemm, moe, qkv, attention, indexer, norm, draft, sampling, top_p, sp_prefill), so no VLLM_ROCM_* opt-in switch is set.
- Every point runs the NVFP4 KV and indexer K caches, DBO on prefill steps of 16384+ tokens, breakable CUDA graphs up to 2048 tokens on the V2 model runner, and 8 renderer workers, so short turns are not queued behind long prompt renders.
- Batch-sharded sampling splits the sampler across the TP ranks from
concurrency 2; vLLM refuses it at concurrency 1, where
--max-num-seqs2 is below TP 4. - Concurrency 32 and 64 add a host-DRAM KV tier through
SimpleCPUOffloadConnector(cpu_bytes_to_use1,349 GB for the whole engine) withcpu_lru_eviction, which evicts the least recently used prefixes from their tail. Concurrency 64 runs three-token DSpark at 0.90 memory utilization.
One-hour runs of these commands on one MI355X node each (agentx-harness 1.0.12, golden acceptance length, single run per point):
| Concurrency | Interactivity p50 / p90 (tokens/s/user) | Total tokens/s per GPU | TTFT p50 |
|---|---|---|---|
| 1 | 659 / 612 | 7,713 | 0.21 s |
| 4 | 684 / 556 | 9,954 | 0.12 s |
| 8 | 610 / 478 | 17,014 | 0.11 s |
| 16 | 529 / 367 | 29,734 | 0.12 s |
| 32 | 395 / 245 | 63,707 | 0.16 s |
| 64 | 166 / 114 | 113,237 | 0.19 s |
One TP4 prefill engine with a host-DRAM KV tier feeds two TP4 decode engines on one node over MoRI-IO RDMA WRITE, behind vllm-router with consistent-hash routing (12 GPUs). Seven-token DSpark, NVFP4 KV and indexer K caches, and the nightly's opt-in MiniMax-M3.1 ROCm switches on both roles.
6 points · concurrency 40–128 · image vllm/vllm-openai-rocm:nightly-4fab5a07…@sha256:6f126f27a3b5…
Runbook
A 1P2D AgentX lane on 12 MI355X GPUs, on the Quark MXFP4 checkpoint: one TP4 prefill
engine (GPUs 0-3 of its node) and two TP4 decode engines (GPUs 0-3 and 4-7 of a second
node), with KV moved from prefill to decode by MoRI-IO RDMA WRITE. Measured in
ROCm/ATOM
(m31-flash-vllm-moriio-rdma-1p2d-s65822-*, and -n4fab-c96-dcap16-cpu19-nodbo for
concurrency 96) with agentx-harness 1.0.12 on the
semianalysis_cc_traces_weka_062126 traces, 3600 s per point. Throughput runs replaced
block verification with synthetic rejection at the golden DSpark acceptance length
(4.04); the commands above keep block verification.
Image and vLLM. The vLLM nightly 4fab5a074 with the opt-in switches above, plus
vllm-project/vllm#65822: with
prefix caching on the decode engine, MoRI-IO WRITE skips the blocks the decoder already
holds instead of rewriting them. The lane overlaid that PR's Python files on the image;
use a nightly that contains it.
Running it. The preset renders the prefill engine and one decode engine. Start the
second decode engine on the same node with the same command and environment, with
CUDA_VISIBLE_DEVICES=4,5,6,7, --port 8003, and handshake_port 7401 and
notify_port 62105 in its --kv-transfer-config (each TP rank listens on base + rank).
Pin each engine to its GPUs' NUMA node (GPUs 0-3 on node 0, 4-7 on node 1). Run
vllm-router 0.1.16 (MoRI-IO WRITE needs 0.1.15 or later) with
--policy consistent_hash --prefill-policy consistent_hash --decode-policy consistent_hash
in place of round_robin, so every turn of a conversation reaches the engines that
cache its prefix.
Engines find the router through proxy_ip, not a list of endpoints: set ROUTER_HOST
to an address of the router's node that both nodes can reach; the default localhost
only reaches engines on the router's node. Each engine registers the address
get_ip() returns; on hosts with several NICs, set VLLM_HOST_IP to the node's
routable address.
Tuning rules.
- Prefill
--max-num-seqsis twice the concurrency. Each decode engine caps its batch (--max-num-seqs10-24), and its CUDA graphs cover that batch x (1 + 7 draft tokens). A capped engine queues extra requests instead of running a larger, slower batch. - The prefill host-DRAM KV tier keeps the conversations' prefixes once they outgrow HBM (about 53 KB per token): 1.349 TB up to concurrency 64, 1.9 TB (about 36M tokens) at 96 and 112, 2.2 TB at 128. The prefill prefix hit rate stayed at 95-97%.
- Decode engines run vLLM's default
FULL_AND_PIECEWISEgraphs.
| Concurrency | Decode cap | Total tokens/s per GPU | Interactivity p50 / p90 (tokens/s/user) | TTFT p90 | E2E latency p90 |
|---|---|---|---|---|---|
| 40 | 10 | 24,884 | 312.7 / 241.0 | 0.8 s | 9.8 s |
| 48 | 12 | 29,553 | 277.3 / 229.9 | 1.1 s | 9.8 s |
| 64 | 16 | 39,616 | 231.5 / 200.4 | 2.4 s | 11.6 s |
| 96 | 16 | 46,295 | 200.7 / 189.0 | 7.1 s | 16.4 s |
| 112 | 20 | 50,561 | 187.0 / 168.4 | 9.6 s | 18.4 s |
| 128 | 24 | 52,889 | 178.4 / 159.7 | 11.3 s | 21.5 s |
Single runs of the rendered commands. Dual-batch overlap on prefill
(--enable-dbo --dbo-prefill-token-threshold 16384 --dbo-decode-token-threshold 32768,
vllm-project/vllm#63483) gave the
same throughput and interactivity at every point and lowered TTFT p90 by 1-2 s at 112
and 128, so the preset leaves it off.
A worker can fail at startup with mori's tcp.cpp "Address already in use": MoRI-IO
picks its RDMA port with a probe-and-release, and another process on a busy node can
take it first. Restart the engine.