moonshotai/Kimi-K3.1-Pro
2.8T-parameter native multimodal MoE with Kimi Delta Attention, sparse MLA, and a 1M-token context window
TP8 on two GB300 trays or one B300 node, or one MI355X node, for the MXFP4 checkpoint; TP32 on a 4-host TPU slice
export GLOO_SOCKET_IFNAME=$IFACE_NAME export NCCL_SOCKET_IFNAME=$IFACE_NAME export VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve moonshotai/Kimi-K3.1-Pro \ --trust-remote-code \ --tensor-parallel-size 8 \ --nnodes 2 \ --node-rank 0 \ --master-addr $HEAD_IP \ --load-format fastsafetensors \ --enable-auto-tool-choice \ --tool-call-parser kimi_k3 \ --reasoning-parser kimi_k3
The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /moonshotai/Kimi-K3.1-Pro.md.
Overview
Kimi-K3.1 keeps the Kimi-K3 layout (Kimi Delta Attention, 896 routed experts with 16 active per token, a latent MoE projection and a vision encoder) and adds sparse attention (DSA, top-2048 indexer) to 24 of its MLA layers. The checkpoint ships MXFP4 routed-expert weights; everything else is BF16.
The baseline is the simplest working setup on NVIDIA: one tensor-parallel group of 8 GPUs, either two GB300 trays or one B300 node. Throughput was not tuned.
VR200 (preview) takes the same TP8 layout on two trays, in the vllm/vllm-openai:kimi-k31-cu134
image (CUDA 13.4). It uses the Blackwell flags and hasn't been tuned or verified on Rubin.
On one B300 node, three AgentX presets follow InferenceX's B300 vLLM AgentX lane (SemiAnalysisAI/InferenceX#4871), all with TP8 + DCP8 attention, the K3.1 DSpark draft, the low-latency decode backend and host KV offload: AgentX TP8 + DCP8 at concurrency 1 and 6, concurrency 12 and 24 with CUDA graphs for prefill and mixed steps, and, with routed experts EP8 on MegaMoE, AgentX TP8 + DCP8 + EP8 at concurrency 48 and 64.
On MI355X, one node takes the same TP8 layout with the AMD profile's AITER environment, in the
vllm/vllm-openai-rocm:kimi-k31-5c14fe8 image. The AgentX TP8 preset
follows InferenceX's MI355X vLLM AgentX lane (SemiAnalysisAI/InferenceX#5089) with one point per
concurrency, all with the K3.1 DSpark draft: plain TP8 with GPU KV and the low-latency decode
backend at concurrency 1, DCP8 with GPU KV and the low-latency decode backend at 4, and DCP8
with host KV offload from 16 to 48. It needs the image the preset pins.
On Zhenwu PPUs, the default MXFP4 checkpoint runs without speculative decoding on
two 16-PPU 810E nodes with TP32 / EP32, or four 8-PPU M890 nodes with TP8 / DP4 / EP32.
On 810E, ppu_deep_gemm_w4a16 loads the checkpoint with a W4A16 compute path.
Use the PPU image specified for each hardware configuration.
On TPU, the model is served by vLLM torch-TPU, text-only: the vision tower is not loaded. The validated layout is TP32 with expert parallelism on a 2x2x4 TPU v7x slice (16 chips, 32 devices, on 4 hosts). See TPU serving.
Tensor + expert parallelism
On one B300 node, the Tensor + Expert Parallel strategy keeps attention at TP8 and runs the routed experts with expert parallelism on the MegaMoE kernel. It has the lowest long-prompt time to first token of the single-node layouts: a single 340K-token prompt prefills in about 10 s, against about 15 s at plain TP8 (8x B300, one request at a time, pre-release checkpoint).
Long context: decode context parallelism
The Decode Context Parallelism row shards the MLA latent and indexer-key caches across the 8 GPUs of the TP group instead of replicating them, so DCP8 holds about 7x the KV cache tokens of TP8 (about 18.5M vs 2.6M tokens with FP8 KV on 8x B300). Prefill is as fast as at TP8. Decode steps are 7–25% slower than TP8 at the same batch, so use DCP when KV capacity is the limit: very long contexts or many long-lived agentic sessions.
Low-latency decode
The Low-latency decode feature switches decode steps to a fused CuTe DSL path tuned for
small batches; prefill and mixed steps still run the default path. On 8x B300 at TP8 it cuts
the decode step by 1.6–1.7x at batch 1 and 1.2–1.3x at batch 32–64. Under DCP8 it brings decode
to within 6% of TP8. It requires the V2 model runner and FULL decode CUDA graphs (the
defaults), and TP8 or TP8 + DCP8. The feature toggle is offered only without expert
parallelism; the B300 AgentX preset with routed experts EP8 on MegaMoE
(concurrency 48 and 64) sets
the same decode_backend in its own args.
On MI355X the same switch selects fused FlyDSL decode kernels. They cover only pure decode
steps of up to 16 rows, so among other checks they also need speculative decoding,
--max-num-seqs 1, 2, 4 or 8 with --max-num-seqs × (1 + draft tokens) ≤ 16, BF16 KDA
projections, an FP32 KDA recurrent state (--mamba-ssm-cache-dtype float32) and no KV
offload. Without DCP they serve only 3 or 7 draft tokens at --max-num-seqs 1 or 2, so the
MI355X default of 5 draft tokens needs DCP8. The concurrency-1 (no DCP) and 4 (DCP8) points
of the AgentX TP8 preset show working setups, on the image it pins;
current vLLM main does not yet run K3.1 on this path without DCP. Its points from
concurrency 16 do not qualify, so leave the feature off there.
NVFP4 KV cache
The NVFP4 KV cache feature stores the MLA latent in NVFP4 (352 bytes per token per layer instead of 576) and holds about 1.5x the KV cache tokens of FP8. The indexer keys stay FP8, so the top-2048 token selection is unchanged. GSM8K matched FP8 on 8x GB300 (96.1% vs 96.2%); long-context accuracy has not been measured yet.
TPU serving
TPU serving runs through vLLM torch-TPU and
is text-only: the vision tower is not loaded. K3.1 support is under review as
vllm-torchtpu#2775; until it merges,
the Install tab on TPU builds from the kimi-k31pro staging branch (Python 3.12, uv). No
published image or wheel carries it yet.
The generated command uses vllm-torchtpu's default multiprocessing multi-host backend: run the
head command on the slice's first host and the worker commands on the remaining hosts (the
builder renders both). The Multi-Node TEP strategy at 4 nodes matches the validated flags of
vllm-torchtpu#2775 with two
differences: #2775 ran one driver over a Ray cluster (TPU_MULTIHOST_BACKEND=ray), and its
GSM8K validation used the completions API, so the default-on chat and tool-call parsers were not
exercised on TPU. Validated on a 2x2x4 TPU v7x slice with an FP8 latent KV cache, the unified
block pool and prefix caching: the complete GSM8K split scored 97.50% (TP32) and 97.42%
(TP32 + DCP8), prefix-cache hits returned identical tokens and logprobs, and the BEAM-1M
long-context benchmark matched the GPU reference. The run-specific sizing of that validation
(10K context, 16 sequences, compile sizes) is not the TPU default — it lives in the two
Validated presets: TEP32 and
TEP32 + DCP8 for KV-capacity-bound workloads.
Spec decoding is gated off on TPU for now: DSpark with --async-scheduling (always on in
this configuration) needs vllm-torchtpu#2770,
still open. #2775 validated DSpark with 7 draft tokens once that fix is in.
Prefill/decode disaggregation
The Prefill/Decode Disaggregation strategy runs one TP8 prefill pool (with expert
parallelism) and one DEP16 decode pool (TP1 x DP16 with expert parallelism), and decode
pulls each request's KV from prefill over NIXL. Every PD worker sets
VLLM_SSM_CONV_STATE_LAYOUT=DS: NIXL transfers the Kimi Delta Attention conv state in three
reads, which needs that layout, and the NIXL connector refuses to start without it.
KV cache offloading
The Simple option on the KV Offload row adds SimpleCPUOffloadConnector, a host-DRAM
prefix tier for each GPU. Under Prefill/Decode Disaggregation it is added to the prefill pool
only, next to NIXL in a MultiConnector; decode keeps plain NIXL. The tier pins
cpu_bytes_to_use_per_rank of host memory for every GPU, so size it from the host DRAM that
is free after the engines are up. The tested GB300 prefill workers used 128 GiB per GPU
(512 GiB per tray).
Client usage
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "moonshotai/Kimi-K3.1-Pro",
"messages": [{"role": "user", "content": "Explain sparse attention in two sentences."}],
"temperature": 0.7,
"top_p": 0.95
}'
References
Configurations qualified on specific hardware, layered on the baseline command. Unless marked Opt-in, a preset applies automatically on its hardware and strategy; pick Baseline in the builder to drop it.
B3003 presetsAgentX TP8 + DCP8 · AgentX TP8 + DCP8, mixed-step graphs · AgentX TP8 + DCP8 + EP8
Concurrency 1 and 6 on one B300 node: TP8 with DCP8, the low-latency decode backend on FULL decode CUDA graphs, the K3.1 DSpark draft with 4 tokens on TOKENSPEED_MLA, and host KV offload.
2 points · concurrency 1–6 · image vllm/vllm-openai:kimi-k31-mc@sha256:f81998764e85…
RunbookThroughput measurement only (synthetic acceptance)
The single-node B300 TP8 + DCP8 AgentX configuration at concurrency 1 and 6. It follows the agentic.yaml points of InferenceX's B300 vLLM AgentX lane (SemiAnalysisAI/InferenceX#4871) and has been measured only on the image digest the preset pins. It serves text with tool-call and reasoning parsers.
Serving config.
- TP8 with DCP8. DCP8 shards the MLA latent and indexer-key caches across the TP group, with the A2A DCP backend,
--cp-kv-cache-interleave-size 1and the direct DCP all-to-all and gather paths (VLLM_USE_DIRECT_DCP_*). The MLA KV cache is FP8 and the indexer key cache MXFP4 (indexer_kv_dtype). - Low-latency decode (the Low-latency decode feature, on in this preset) runs every pure decode step on the fused CuTe DSL path, on the V2 model runner, in FULL_DECODE_ONLY CUDA graphs. Each point captures 5, 10, 15, … rows (1 + 4 draft tokens per request) up to its largest planned decode batch; a decode step above the largest captured size runs eager.
- Spec decoding (on in this preset) uses the K3.1 DSpark draft
Inferact/Kimi-K3.1-Pro-DSpark-previewat revision6a2ef20a, with 4 tokens, TOKENSPEED_MLA attention for the draft layers, probabilistic drafting and block rejection.tokenspeed_mla_min_split_kv: 32in--attention-configsets the minimum number of KV splits of that decode attention (larger values can take more workspace memory), andTOKENSPEED_MLA_FP16_PARTIALS=0turns off its FP16 partials. --use-replayssm. With speculative decoding it turns on the KDA layers' RecoverSSM path: verification replays the accepted prefix instead of keeping one recurrent state per draft token. The KDA recurrent state is FP32 (--mamba-ssm-cache-dtype float32), as the K3.1 precision policy requires.- Collectives. FlashInfer all-reduce stays on (
VLLM_ALLREDUCE_USE_FLASHINFER=1); NCCL symmetric memory, the fused GEMM + reduce-scatter and the sharded sequence-parallel shared expert stay off (VLLM_USE_NCCL_SYMM_MEM=0,VLLM_ENABLE_GEMM_RS=0,VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT=0). - Points.
--max-num-seqsis 2 at concurrency 1 and 6 at concurrency 6. The capture sizes cover 5 rows per admitted request. Size both to your own concurrency. - The Simple KV-offload option gets 281,125,000,000 bytes of host DRAM per rank. Size
cpu_bytes_to_use_per_rankto your host. --no-enable-flashinfer-autotune,--max-num-batched-tokens 16384,--gpu-memory-utilization 0.9,--stream-interval 10and--disable-uvicorn-access-logas in the lane.VLLM_ENGINE_READY_TIMEOUT_S=7200leaves time for the weight load and the specialized decode's JIT;VLLM_HTTP_TIMEOUT_KEEP_ALIVE=900keeps pooled client connections open between agent turns.
Benchmark-only settings left out. The lane also sets PYTHONHASHSEED=42, so that identical prefixes hash to the same block keys from run to run, and PYTHONUNBUFFERED=1 for its logs; neither changes serving. Its --served-model-name equals the model ID.
Throughput measurement only (synthetic acceptance)
To measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 4 draft tokens, 3.19. Only --speculative-config changes:
--speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"method":"dspark","draft_sample_method":"probabilistic","revision":"6a2ef20a05efe88aabbf9ecaa12d020c3b0edf94","attention_backend":"TOKENSPEED_MLA","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.19}'
Synthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.
Concurrency 12 and 24 on one B300 node: TP8 with DCP8, CUDA graphs for decode and for prefill and mixed steps up to 4096 tokens, prefill admitted every 8 engine steps, the low-latency decode backend, the K3.1 DSpark draft with 3 tokens, and host KV offload.
2 points · concurrency 12–24 · image vllm/vllm-openai:kimi-k31-mc@sha256:f81998764e85…
RunbookThroughput measurement only (synthetic acceptance)
The concurrency-12 and -24 points of InferenceX's B300 vLLM AgentX lane (SemiAnalysisAI/InferenceX#4871, agentic-mixed-graphs.yaml), measured only on the image digest the preset pins. Both points run the same server config. It keeps the TP8 + DCP8 layout, the low-latency decode backend, FP8 KV with the MXFP4 indexer cache, --use-replayssm, the FP32 KDA recurrent state and the Simple KV offload of AgentX TP8 + DCP8, and changes the following:
- CUDA graphs also cover prefill and mixed steps up to 4096 tokens. The default CUDA graph mode with
max_cudagraph_capture_size4096 captures 1, 2, then 4 to 384 in steps of 4 for decode, then 512 to 4096. InferenceX passes these as--cudagraph-capture-sizesand--max-cudagraph-capture-size, which set the same compilation-config fields. --prefill-schedule-interval 8. Prefill is admitted once every 8 engine steps, so the specialized decode serves the pure decode steps in between. A long prompt takes up to 8 times as many engine steps to finish prefilling.- Spec decoding drafts 3 tokens on the draft's default attention backend, with probabilistic drafting and block rejection.
--max-num-seqs 96and--gpu-memory-utilization 0.95. This config leaves FlashInfer autotuning on and--cp-kv-cache-interleave-sizeat its default, and its env sets only the direct DCP paths,VLLM_ENGINE_READY_TIMEOUT_S=7200andVLLM_HTTP_TIMEOUT_KEEP_ALIVE=900; everything else stays at the image defaults.
As in the other B300 presets, the lane's PYTHONHASHSEED=42 and PYTHONUNBUFFERED=1 are benchmark-only and left out.
Throughput measurement only (synthetic acceptance)
To measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 3 draft tokens, 2.88. Only --speculative-config changes:
--speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"method":"dspark","draft_sample_method":"probabilistic","revision":"6a2ef20a05efe88aabbf9ecaa12d020c3b0edf94","rejection_sample_method":"synthetic","synthetic_acceptance_length":2.88}'
Synthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.
Concurrency 48 and 64 on one B300 node: TP8 + DCP8 attention with routed experts EP8 on MegaMoE, the low-latency decode backend at every decode batch on FULL decode CUDA graphs, prefill admitted every 8 engine steps, the K3.1 DSpark draft with 4 tokens on TOKENSPEED_MLA, and host KV offload.
2 points · concurrency 48–64 · image vllm/vllm-openai:kimi-k31-mc@sha256:f81998764e85…
RunbookThroughput measurement only (synthetic acceptance)
The concurrency 48 and 64 points of InferenceX's B300 vLLM AgentX lane (SemiAnalysisAI/InferenceX#4871, agentic-ep.yaml), measured only on the image digest the preset pins. It is AgentX TP8 + DCP8 with routed experts on expert parallelism, and changes the following:
- Routed experts EP8 on MegaMoE. Attention stays TP8 + DCP8; the Tensor + Expert Parallel strategy adds
--enable-expert-paralleland--moe-backend deep_gemm_mega_moe. The lane keeps the default all-to-all backend, so the preset drops the strategy's--all2all-backend. - Low-latency decode at every decode batch. The specialized decode also runs this hybrid layout (TP8 attention, routed experts EP over every rank) on MegaMoE, with
VLLM_ENABLE_GEMM_RS=0. The Low-latency decode toggle is offered only without expert parallelism, so the preset sets--additional-config '{"decode_backend":"latency_specialized"}'itself. --prefill-schedule-interval 8. Prefill is admitted once every 8 engine steps, so the specialized decode serves the pure decode steps in between.- Points.
--max-num-seqsequals the concurrency, and FULL_DECODE_ONLY graphs capture 5 rows (1 + 4 draft tokens) per request up to it.--gpu-memory-utilizationis 0.97 at both points. tokenspeed_mla_min_split_kv(the minimum number of KV splits of the draft's TOKENSPEED_MLA decode attention) is 16. WithVLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT=0, sequence-parallel MoE replicates the shared expert on every rank, which costs weight memory on each GPU.- The Simple KV-offload option gets 374,875,000,000 bytes of host DRAM per rank. Size
cpu_bytes_to_use_per_rankto your host.
As in the other B300 presets, the lane's PYTHONHASHSEED=42 and PYTHONUNBUFFERED=1 are benchmark-only and left out.
Throughput measurement only (synthetic acceptance)
To measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 4 draft tokens, 3.19. Only --speculative-config changes:
--speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"method":"dspark","draft_sample_method":"probabilistic","revision":"6a2ef20a05efe88aabbf9ecaa12d020c3b0edf94","attention_backend":"TOKENSPEED_MLA","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.19}'
Synthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.
GB300 NVL44 presetsAgentX 1P3D TP8 + DCP8 · AgentX xP1D, DP4 x TP8 decode · AgentX 1P1D DEP16 · AgentX TP16 + DCP16, concurrency 1
Concurrency 20 to 56 on 32 GPUs: one TP8 + DCP8 prefill worker with EP8 on MegaMoE and three TP8 + DCP8 decode workers on the low-latency decode backend, two GB300 trays each, served by Dynamo with the K3.1 DSpark draft (4 tokens) and a shared Mooncake KV store in host DRAM.
3 points · concurrency 20–56 · image vllm/vllm-openai:kimi-k31-mc@sha256:f29efae15f9c… · dynamo 3dd0744e
RunbookServices · Frontend · Benchmark-only settings left out · Throughput measurement only (synthetic acceptance)
The concurrency 20, 32 and 56 points of the GB300 AgentX disaggregated lane in SemiAnalysis InferenceX (kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg, SemiAnalysisAI/InferenceX#4907). Dynamo + vLLM run one prefill worker and three decode workers, each TP8 + DCP8 across two 4-GPU GB300 trays (32 GPUs). The prefill worker also runs the routed experts EP8 on MegaMoE; the decode workers use no expert parallelism and run the low-latency decode backend. KV moves from prefill to decode over NIXL, and all workers share an RDMA Mooncake store in host DRAM. The lane ran the image digest the preset pins, Dynamo built from 3dd0744eb72c707d76d4c520ae4dea9e286a4ab6 (ai-dynamo/dynamo#18124, the first Dynamo revision that registers the K3.1 model type) and Mooncake 0.3.14; the preset has been measured only on that combination.
| Clients | Prefill workers | Decode workers | GPUs | Decode --max-num-seqs | Decode capture sizes | CPU binding |
|---|---|---|---|---|---|---|
| 20, 32 | 1 | 3 | 32 | 64 | 5 to 320 in steps of 5 | --numa-bind |
| 56 | 1 | 3 | 32 | 256 | 5 to 320 in steps of 5, then 340 to 1280 in steps of 20 | VLLM_WORKER_MULTIPROC_METHOD=spawn |
Each point renders one prefill worker and one decode worker; start the table's number of workers per role with the same commands, two trays each. Run every worker command on both of its trays with the same --master-addr (the first tray's address), --nnodes 2 and --node-rank 0 or 1.
Serving config.
- Prefill: TP8 + DCP8 with
--enable-expert-paralleland--moe-backend deep_gemm_mega_moe,--max-num-seqs 32, andFULL_AND_PIECEWISECUDA graphs for 4 to 8192 tokens.VLLM_ENABLE_GEMM_RS=1turns on the fused GEMM + reduce-scatter.DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMITandDYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMITset Dynamo's admission limits for the prefill engine to 1024 requests each. - Decode: TP8 + DCP8 without expert parallelism, with
VLLM_ENABLE_GEMM_RS=0.--additional-config '{"decode_backend":"latency_specialized"}'runs pure decode steps on the fused low-latency path; the Low-latency decode toggle is not offered under PD, so the preset sets the flag in the decode args.--use-replayssmmakes KDA verification replay the accepted prefix instead of keeping one recurrent state per draft token. The decode--compilation-configsets only the capture sizes (multiples of 5 rows: 1 + 4 draft tokens per request) and replaces the PD strategy'sFULL_DECODE_ONLY, so the CUDA graph mode is vLLM's default, as in the lane. - Both roles: FP8 MLA KV cache with an MXFP4 indexer key cache (
indexer_kv_dtype); FP32 KDA conv and recurrent state (--mamba-cache-dtype float32 --mamba-ssm-cache-dtype float32), as the K3.1 precision policy requires; the A2A DCP backend and the direct DCP all-to-all and gather paths (VLLM_USE_DIRECT_DCP_*); prefix caching with--prefix-match-unit 128;--gpu-memory-utilization 0.95,--enable-cumem-allocator,--no-enable-flashinfer-autotuneand--stream-interval 10. Both roles pin--cp-kv-cache-interleave-size 1: left unset, NIXL P/D raises the DCP KV interleave after the specialized decode has read it. - Spec decoding (on in this preset) uses the K3.1 DSpark draft
Inferact/Kimi-K3.1-Pro-DSpark-previewwith 4 tokens, probabilistic drafting and block rejection. - Collectives and kernels: FlashInfer all-reduce stays on (
VLLM_ALLREDUCE_USE_FLASHINFER=1); NCCL symmetric memory and the sharded sequence-parallel shared expert stay off (VLLM_USE_NCCL_SYMM_MEM=0,VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT=0), and so do TOKENSPEED_MLA's FP16 partials (TOKENSPEED_MLA_FP16_PARTIALS=0). NCCL runs over the NVLink domain (NCCL_MNNVL_ENABLE=1,NCCL_NVLS_ENABLE=1,NCCL_CUMEM_ENABLE=1,NCCL_P2P_LEVEL=NVL) without a network plugin (NCCL_NET_PLUGIN=none). - KV transfer: NIXL uses
UCX_MEMTYPE_REG_WHOLE=nandUCX_TCP_AF_PRIO=inet. The Mooncake transfer engine usesMC_SLICE_SIZE=1048576,MC_MAX_MR_SIZE=4294967296andMC_ENABLE_DEST_DEVICE_AFFINITY=1, andWITH_NVIDIA_PEERMEM=0for hosts without the nvidia-peermem module. vLLM loads from the store with 4 receive threads (VLLM_MOONCAKE_LOAD_RECV_THREADS=4), andVLLM_MOONCAKE_STORE_SELECTIVE_LEASE=1makes a store lookup lease only the keys that a hit loads. - Timeouts and housekeeping:
VLLM_ENGINE_READY_TIMEOUT_S=3600(from the recipe's Blackwell settings),VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800,VLLM_RPC_TIMEOUT=600000,DYN_TCP_REQUEST_TIMEOUT=120andETCD_LEASE_TTL=600.TILELANG_CLEANUP_TEMP_FILES=1removes TileLang's temporary build files. - CPU binding: at 20 and 32 clients,
--numa-bind --numa-bind-nodes 0 0 1 1binds each worker process to its GPU's NUMA node (GPUs 0 and 1 on node 0, GPUs 2 and 3 on node 1 of a tray). At 56 clients, workers start withVLLM_WORKER_MULTIPROC_METHOD=spawnand each tray's GPUs are pinned withCUDA_VISIBLE_DEVICES, as the rendered commands do.
Tool calling and reasoning are on in this preset. Under Dynamo the builder renders --dyn-tool-call-parser kimi_k3, --dyn-reasoning-parser kimi_k3 and vLLM's --reasoning-parser kimi_k3, and the preset adds --dyn-enable-structural-tag: the flags of the lane's accuracy runs.
The preset selects the distributed (embedded) Mooncake store on the KV Offload row, which wraps NIXL and the MooncakeStoreConnector in one MultiConnector on both roles; the preset sets the lane's connector options on it (recompute on a failed store load, the NIXL KV buffer on the GPU, cross-layer blocks and store offload off). Documented here only: the worker counts, the lane's Mooncake master options and store sizing (the builder's Mooncake option shows its own defaults), and the frontend's router mode and session affinity (the builder renders round-robin).
Services
Start etcd, NATS and a Mooncake master once:
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<infra-host>:2379
printf 'max_payload: 33554432\n' > nats.conf && nats-server -c nats.conf
mooncake_master --default_kv_lease_ttl=60000 --eviction_high_watermark_ratio=0.95 --eviction_ratio=0.10
Every worker reads the Mooncake store configuration from the JSON file that MOONCAKE_CONFIG_PATH points to. The lane registered 160GB of host DRAM per GPU:
{"master_server_address": "<infra-host>:<master-port>", "metadata_server": "P2PHANDSHAKE",
"global_segment_size": "160GB", "local_buffer_size": "4GB", "protocol": "rdma",
"device_name": "<the tray's RDMA devices>", "mode": "embedded", "enable_offload": false}
Frontend
export DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TOKENIZER_CACHE_BYTES=8589934592 DYN_TCP_REQUEST_TIMEOUT=120
export DYN_DISABLE_OPENAI_STRICT_SCHEMA_VALIDATION=1
python3 -m dynamo.frontend --router-mode least-loaded --router-session-affinity-ttl-secs 1
DYN_DISABLE_OPENAI_STRICT_SCHEMA_VALIDATION=1 skips the frontend's preflight validation of tool parameter schemas marked strict: true. Session affinity keys on the X-Dynamo-Session-Id request header, so send one ID per conversation.
Benchmark-only settings left out
The lane also passes --served-model-name moonshotai/Kimi-K3.1-Pro, which equals the model ID here, and sets VLLM_SERVER_DEV_MODE=1, which turns on vLLM's development-only endpoints; serving needs neither. Its throughput runs leave out the tool-call and reasoning parser flags. Its job settings (model cache path, health-check polling, telemetry and the benchmark client's own environment) are not server settings.
Throughput measurement only (synthetic acceptance)
To measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 4 draft tokens, 3.19. Only --speculative-config changes, on both roles:
--speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.19}'
Synthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.
Concurrency 192 and 384: two or three TP8 + DCP8 prefill workers with EP8 on MegaMoE, and one decode worker across eight GB300 trays that runs four TP8 + DCP8 data-parallel ranks with EP32 on MegaMoE and the low-latency decode backend, served by Dynamo with the K3.1 DSpark draft and a shared Mooncake KV store in host DRAM.
2 points · concurrency 192–384 · image vllm/vllm-openai:kimi-k31-mc@sha256:f29efae15f9c… · dynamo 3dd0744e
RunbookThroughput measurement only (synthetic acceptance)
The concurrency 192 and 384 points of the GB300 AgentX disaggregated lane in SemiAnalysis InferenceX (kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg, SemiAnalysisAI/InferenceX#4907), on the image, Dynamo revision and Mooncake version of the AgentX 1P3D TP8 + DCP8 preset. The prefill workers are that preset's: TP8 + DCP8 with EP8 on MegaMoE across two trays each. The single decode worker spans eight trays (32 GPUs) and runs four data-parallel ranks, each TP8 + DCP8 across two trays, with the routed experts EP32 on MegaMoE across all 32 GPUs and the low-latency decode backend. KV moves over NIXL, and all workers share the RDMA Mooncake store in host DRAM.
| Clients | Prefill workers | Decode workers | GPUs | Draft tokens | Decode --max-num-batched-tokens and largest capture size | Prefill chunking |
|---|---|---|---|---|---|---|
| 192 | 2 | 1 | 48 | 6 | 1792 | --long-prefill-token-threshold 4096, adaptive |
| 384 | 3 | 1 | 56 | 4 | 1280 | default |
Each point renders one prefill worker and the decode worker; start the table's number of prefill workers with the same command, two trays each, with the same --master-addr (the first tray's address), --nnodes 2 and --node-rank 0 or 1. Run the decode command on each of its eight trays with --nnodes 8, --node-rank 0 to 7, and the first tray's address as both --master-addr and --data-parallel-address; ranks 1 to 7 add --headless, as rendered.
Changes from the 1P3D preset.
- Decode layout. The builder renders the decode pool as one TEP engine across eight trays (
--tensor-parallel-size 32 --enable-expert-parallel); the decode args--tensor-parallel-size 8 --data-parallel-size 4replace its TP size, so the engine runs four TP8 data-parallel ranks with DCP8 inside each, and expert parallelism spans all 32 GPUs.--data-parallel-rpc-port 13345is the lane's choice; any free port works. - Decode batch. MegaMoE (
--moe-backend deep_gemm_mega_moe) on decode as well,--max-num-seqs 256per data-parallel rank,--gpu-memory-utilization 0.92, and--max-num-batched-tokensand CUDA graph sizes up to 256 × (1 + draft tokens): multiples of 7 up to 448 and then of 28 up to 1792 with 6 draft tokens, multiples of 5 up to 320 and then of 20 up to 1280 with 4. - Prefill.
--max-num-batched-tokens 16384. At 192 clients the prefill workers also chunk long prompts with--long-prefill-token-threshold 4096 --long-prefill-token-threshold-adaptive, and both roles draft 6 tokens; at 384 clients both roles draft 4. - Worker start. Every worker starts with
VLLM_WORKER_MULTIPROC_METHOD=spawn, and each tray's GPUs are pinned withCUDA_VISIBLE_DEVICES, as the rendered commands do; no--numa-bind.
Everything else (env, connector options, tool-call and reasoning parsers, services and frontend) is as in the 1P3D preset, including its benchmark-only settings left out.
Throughput measurement only (synthetic acceptance)
To measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for the draft length: 3.57 for 6 draft tokens (192 clients) and 3.19 for 4 (384 clients). Only --speculative-config changes, on both roles:
# 192 clients
--speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":6,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.57}'
# 384 clients
--speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.19}'
Synthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.
Concurrency 352 on 32 GPUs: one DEP16 prefill worker and one DEP16 decode worker (TP1 x DP16 with EP16 on MegaMoE), four GB300 trays each, with the low-latency decode backend, served by Dynamo with the K3.1 DSpark draft (4 tokens) and a shared Mooncake KV store in host DRAM.
1 point: 352 (1P1D) · image vllm/vllm-openai:kimi-k31-mc@sha256:5c243ce00d2d… · dynamo 3dd0744e
RunbookThroughput measurement only (synthetic acceptance)
The concurrency-352 point of the GB300 AgentX DEP16 lane in SemiAnalysis InferenceX (kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dep16-disagg, SemiAnalysisAI/InferenceX#4907). Dynamo + vLLM run one prefill worker and one decode worker, each DEP16 (--tensor-parallel-size 1 --data-parallel-size 16 with expert parallelism) across four 4-GPU GB300 trays, 32 GPUs in all, with MegaMoE on both roles and no DCP. KV moves over NIXL, and all workers share an RDMA Mooncake store in host DRAM. The lane ran the image digest the preset pins, with the Dynamo revision and Mooncake version of the AgentX 1P3D TP8 + DCP8 preset; the preset has been measured only on that combination.
Run each worker command on each of its four trays with --data-parallel-start-rank 0, 4, 8 and 12 and the same --data-parallel-address (the first tray's address), as rendered.
Serving config.
- Both roles:
--max-num-seqs 32per data-parallel rank and--block-size 16384: the image does not pick a power-of-two attention block size at TP1 by itself, so the lane pins the same size on both roles.--kv-cache-memory-bytessizes the KV cache per rank explicitly (61926160384 bytes on prefill, 64475268096 on decode): the size vLLM reported it could use on this image, minus 3 GiB. Size it to your own image and memory budget; the explicit size replaces the KV sizing from--gpu-memory-utilization 0.95. - Prefill:
--max-num-batched-tokens 2048with--long-prefill-token-threshold 512 --long-prefill-token-threshold-adaptive, so one long prompt does not fill a rank's step, andFULL_AND_PIECEWISECUDA graphs for 4 to 2048 tokens.VLLM_ENABLE_GEMM_RS=1and Dynamo's admission limits (1024) as in the 1P3D preset. - Decode: the low-latency decode backend (
--additional-config '{"decode_backend":"latency_specialized"}') with--use-replayssm,--max-num-batched-tokens 160, and CUDA graphs for 5 to 160 rows in steps of 5 (32 requests × (1 + 4 draft tokens)) in vLLM's default graph mode, which the decode--compilation-configkeeps by replacing the PD strategy'sFULL_DECODE_ONLY.VLLM_ENABLE_GEMM_RS=0. - Everything else (precision, kernels, collectives, KV transfer, timeouts, tool-call and reasoning parsers, connector options) is as in the 1P3D preset. The lane's env also keeps the direct DCP paths (
VLLM_USE_DIRECT_DCP_*=1), which have no effect without DCP. All workers start withVLLM_WORKER_MULTIPROC_METHOD=spawn, and each tray's GPUs are pinned withCUDA_VISIBLE_DEVICES, as the rendered commands do. The data-parallel RPC ports are the builder's placeholders; the lane used 13355 for prefill and 13345 for decode.
Start the services and the frontend as for the 1P3D preset; the same benchmark-only settings are left out.
Throughput measurement only (synthetic acceptance)
To measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 4 draft tokens, 3.19. Only --speculative-config changes, on both roles:
--speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.19}'
Synthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.
Concurrency 1 on four GB300 trays: one aggregated TP16 + DCP16 engine on the low-latency decode backend, with the K3.1 DSpark draft (7 tokens) and the Mooncake KV store in host DRAM.
1 point: 1 · image vllm/vllm-openai:kimi-k31-mc@sha256:5c243ce00d2d… · dynamo 3dd0744e
RunbookThroughput measurement only (synthetic acceptance)
The concurrency-1 point of the GB300 AgentX aggregated lane in SemiAnalysis InferenceX (kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-agg, SemiAnalysisAI/InferenceX#4907): one engine serves prefill and decode with TP16 + DCP16 across four 4-GPU GB300 trays, without expert parallelism, on the low-latency decode backend, with the Mooncake store in host DRAM as its KV tier (no NIXL). The lane ran the image digest the preset pins; the preset has been measured only on that image.
Serving config.
- TP16 + DCP16. DCP16 shards the MLA latent and indexer-key caches across the whole TP group, with the A2A DCP backend,
--cp-kv-cache-interleave-size 1and the direct DCP paths (VLLM_USE_DIRECT_DCP_*). FP8 MLA KV cache with an MXFP4 indexer key cache; FP32 KDA conv and recurrent state. - Low-latency decode (the Low-latency decode feature, on in this preset) runs pure decode steps on the fused path, with
--use-replayssm. A decode step is 8 rows (1 + 7 draft tokens);FULL_AND_PIECEWISECUDA graphs cover 7 to 2048 tokens. - Spec decoding (on in this preset) uses the K3.1 DSpark draft with 7 tokens, probabilistic drafting and block rejection.
--max-num-seqs 1,--gpu-memory-utilization 0.90, and--max-num-batched-tokens 16384with--long-prefill-token-threshold 4096: a lone prompt prefills in chunks of up to 16384 tokens, and next to other requests each prompt's chunk stops at 4096 tokens.- The env follows the 1P3D preset (collectives, kernels, Mooncake transfer settings, timeouts) with
VLLM_ENABLE_GEMM_RS=0andVLLM_SSM_CONV_STATE_LAYOUT=DS. It also keeps the lane'sUCX_*settings, which have no effect without NIXL. - The preset selects the distributed (embedded) Mooncake store on the KV Offload row and sets the lane's options on its
MooncakeStoreConnector: recompute on a failed load, asynchronous load and lookup, and cross-layer blocks and store offload off. Start the Mooncake master and size the store as for the 1P3D preset.
Served through Dynamo in the lane. InferenceX ran this engine as one aggregated Dynamo worker (python3 -m dynamo.vllm on each tray with the same engine flags) behind the Dynamo frontend of the 1P3D preset, with DYN_REQUEST_PLANE=tcp, DYN_TCP_REQUEST_TIMEOUT=120 and ETCD_LEASE_TTL=600. The builder renders the same engine flags for vllm serve on the Multi-Node TP strategy, with vLLM's Rust frontend. Under Dynamo, tool calling and reasoning take --dyn-tool-call-parser kimi_k3 --dyn-reasoning-parser kimi_k3 --dyn-enable-structural-tag and --reasoning-parser kimi_k3 in place of the rendered parser flags, as in the lane's accuracy runs.
Benchmark-only settings left out. As in the 1P3D preset: --served-model-name (the model ID), VLLM_SERVER_DEV_MODE=1, and the job and benchmark-client settings. The throughput runs leave out the parser flags.
Throughput measurement only (synthetic acceptance)
To measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 7 draft tokens, 3.69. Only --speculative-config changes:
--speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":7,"method":"dspark","draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.69}'
Synthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.
MI355X1 presetAgentX TP8
One MI355X node at TP8: plain TP8 with the low-latency decode backend and 7 draft tokens at concurrency 1, DCP8 with the low-latency decode backend and 5 draft tokens at 4, DCP8 with host KV offload and prefill every 32 engine steps at 16 to 32 (4, 4 and 3 draft tokens), and DCP8 with host KV offload and 3 draft tokens at 48, with linear layers, MoE combine and all-reduce in BF16.
6 points · concurrency 1–48 · image vllm/vllm-openai-rocm:kimi-k31-5c14fe8@sha256:a61996b4c270…
RunbookThroughput measurement only (synthetic acceptance)
The MI355X AgentX lane in SemiAnalysis InferenceX (kimik3.1-fp4-mi355x-vllm-agentic-dspark, SemiAnalysisAI/InferenceX#5089), one TP8 engine on the node's eight GPUs. It keeps linear layers, MoE combine and all-reduce in BF16 and has been measured only on the image digest the preset pins.
Points.
- Concurrency 1: plain TP8, no DCP, KV cache on the GPU. At one request DCP's extra KV capacity isn't needed: the server log reports 2.8M KV tokens, 2.7x a 1M-token request. The DSpark draft runs 7 tokens,
--max-num-seqsis 1, and the capture sizes 2 to 16 cover the 8-row decode step (1 + 7).VLLM_KIMI_SD_DSA_PROLOGUE=1runs each DSA layer's decode prologue as one launch; it needs DCP off. - Concurrency 4: DCP8, KV cache on the GPU. 5 draft tokens at
--max-num-seqs 2, so a decode step is at most 12 rows (2 × (1 + 5)) and the capture sizes are 6 and 12. DCP8 holds 19.1M KV tokens on the GPUs. - Concurrency 16, 24 and 32: DCP8 with host KV offload, prefill every 32 engine steps.
--max-num-seqsis 8, 13 and 20, below the concurrency, with 4, 4 and 3 draft tokens.--prefill-schedule-interval 32admits prefill only once every 32 engine steps, so decode runs in long uninterrupted stretches. With fewer slots than clients, requests always wait, and vLLM would otherwise drop the hold on any step that leaves requests waiting;VLLM_PREFILL_CADENCE_NO_ESCAPE=1keeps it. Concurrency 32 also runs--max-num-batched-tokens 10240at 0.91 memory utilization. - Concurrency 48: DCP8 with host KV offload, prefill on every step. 3 draft tokens at
--max-num-seqs 48. Decode batches are at most 48 × 4 rows, so the sizes 4 to 192 in steps of 4 cover every decode step, and the list continues to 10240, the value of--max-num-batched-tokens.
Serving config.
- Low-latency decode (the Low-latency decode feature, on at concurrency 1 and 4) sets
decode_backend: latency_specialized. On MI355X it runs pure decode steps of up to 16 rows on fused FlyDSL kernels; prefill and mixed steps run the default path. Among other startup checks, it needs: TP8 with no expert, data or pipeline parallelism; AITER on, theROCM_AITER_MLAattention backend and an fp8 (E4M3) KV cache; speculative decoding on; no KV offload connector;--max-num-seqs1, 2, 4 or 8 with--max-num-seqs× (1 + draft tokens) ≤ 16; a BF16 KDAin_proj(no--quantization-config); and an FP32 KDA recurrent state. Without DCP it serves only 3 or 7 draft tokens with--max-num-seqs1 or 2. At concurrency 1 the server log shows the fused path running the sparse MLA attention itself (the one-launch prologue, the indexer's scoring and DSA split-K, 16 splits); under DCP8 at concurrency 4 the MLA layers run vLLM's attention ops and the rest of the layer is fused. The pinned image also turns on this path's low-concurrency speedups by default: theVLLM_K31_BEAT_*variables andVLLM_K3_INDEXER_SAMPLED_TOPK, plusVLLM_KIMI_SD_DSA_PROLOGUE_INDEXERwhereVLLM_KIMI_SD_DSA_PROLOGUEis on (concurrency 1). All of them default to 1, so the preset sets none. Some act only at one request: at concurrency 4 the server log reports the fused Markov draft step off with more than one request. The decode backend used from concurrency 16 ignores them. - CUDA graphs. K3.1 has no
torch.compilesubmodule, so vLLM resolvesFULL_AND_PIECEWISEtoFULL_DECODE_ONLYand captures decode only. Breakable graphs (VLLM_USE_BREAKABLE_CUDAGRAPH=1, from the AMD profile) are what let the points from concurrency 16 capture the prefill sizes too. They cost graph memory, which comes out of the KV cache, so keep the capture list to the sizes the workload uses. Concurrency 1 and 4 capture decode sizes only and set it to 0. --max-num-batched-tokens16384 (10240 at concurrency 32 and 48). With prefix caching, the hybrid KV manager ends every non-final prefill chunk on a block boundary; the server log reports a KV block of 8192 tokens under DCP8 and 1024 without DCP (kv lcm block sizes).--gpu-memory-utilization0.90 (0.91 at concurrency 32 and 48) andVLLM_ROCM_MEM_HEADROOM_GIB=8. Kernel scratch is allocated outside this budget, so leave headroom. The memory profile also misses some DCP prefill transients, so the headroom variable caps torch's allocator 8 GiB below free memory (Allocator headroom: ... headroom 8.0 GiBin the server log).VLLM_K3_SP_PREFILL=1(set on every point, active under DCP8). In batches of at least 4096 tokens (prefill and large mixed steps), the residual stream stays token-sharded across the TP ranks, so each MoE layer's three all-reduces (attention output, routed latent and MoE output) become reduce-scatters and all-gathers over symmetric memory. These sum the TP partials in FP32 in a fixed order. Decode steps and smaller batches are unchanged. The server log reportssequence-parallel prefill enabled (batches of >= 4096 tokens)at concurrency 4 to 48; at concurrency 1 (no DCP) it reports the configuration unsupported and the default all-reduce path runs, and the variable is kept there as InferenceX ran it.- DCP8 shards the MLA latent and indexer-key caches across the TP group, with the A2A DCP backend and
--cp-kv-cache-interleave-size 1.VLLM_ROCM_DCP_VUP_RS=1appliesv_upfirst and then sums the ranks with AITER's LSE all-gather and reduce-scatter, for the target's MLA layers and the draft's; it needsVLLM_ROCM_DCP_AITER_ALLGATHER=1.VLLM_ROCM_DSA_MULTISTREAM=1runs the DSA indexer, the DCP score all-gather and merge, and the gate on a second stream in decode. Concurrency 1 keeps these variables as InferenceX ran it; with DCP off they are expected to have no effect. - The indexer key cache is MXFP4 (
indexer_kv_dtype). AITER needs padded FP4 indexer pages for it. Stock AITER asserts on the page stride at startup.kda_prefill_backend: fusedprefills the KDA layers with the fused gfx950 kernel. - Spec decoding (the Spec decoding feature, on in this preset) uses the K3.1 DSpark draft
Inferact/Kimi-K3.1-Pro-DSpark-preview(5 layers, not the K3 draft) withROCM_AITER_MLAattention, FP8 KV, probabilistic drafting and block rejection; each point sets its draft token count. - The Simple KV-offload option (concurrency 16 to 48) gets 224,875,000,000 bytes of host DRAM per rank, InferenceX's 1799 GB split over 8 ranks. Size
cpu_bytes_to_use_per_rankto your host.
Precision. As the K3.1 precision policy requires: BF16 all-reduce (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE), BF16 MoE combine (AITER_FLYDSL_STAGE2_FP8=0), BF16 KDA input projections (no --quantization-config) and an FP32 KDA recurrent state (--mamba-ssm-cache-dtype float32).
Image. The pinned image carries kernels and switches that are not in vLLM main yet. On main, the DCP and DSA variables (VLLM_ROCM_AITER_MLA_DCP_VERIFY, VLLM_ROCM_DCP_*, VLLM_ROCM_DSA_*), VLLM_ROCM_MEM_HEADROOM_GIB, VLLM_K3_SP_PREFILL, VLLM_KIMI_SD_DSA_PROLOGUE, VLLM_PREFILL_CADENCE_NO_ESCAPE and the low-latency speedup variables above are not read, and the low-latency decode backend does not serve K3.1 without DCP: there it also requires VLLM_ROCM_USE_AITER_FP4BMM=1 and still refuses K3.1's sparse (indexer) MLA layers, so the concurrency-1 point fails at startup on main or nightly images.
InferenceX run 43717568009 passed all six performance points and GSM8K at concurrency 1, 4, 32 and 48 at 75d74552 (single run per point, synthetic acceptance as below):
| Concurrency | Interactivity p50 / p90 (tokens/s/user) | Total tokens/s per GPU |
|---|---|---|
| 1 | 263.6 / 255.1 | 2,164 |
| 4 | 189.1 / 169.7 | 3,137 |
| 16 | 103.0 / 85.3 | 9,089 |
| 24 | 83.2 / 69.6 | 13,387 |
| 32 | 68.1 / 59.5 | 15,850 |
| 48 | 40.6 / 26.9 | 20,319 |
GSM8K exact match (strict) over 1319 questions, with block verification, was 0.980 at concurrency 1, 0.982 at 4, 0.980 at 32 and 0.979 at 48.
Throughput measurement only (synthetic acceptance)
To measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for the point's draft token count: 3.69 for 7 tokens, 3.41 for 5, 3.19 for 4 and 2.88 for 3. For example, at concurrency 48:
--speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"method":"dspark","attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":2.88}'
InferenceX also turns the tool-call and reasoning parsers off for these runs, because the Kimi parsers stop streaming on the unverified draft tokens. Synthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.
TPU7x2 presetsValidated TEP32 (v7x) · Validated TEP32 + DCP8 (v7x)
The #2775 validation config on a 2x2x4 v7x slice: complete GSM8K at 97.50%, prefix-cache hits returning identical tokens and logprobs, BEAM-1M matching the GPU reference.
Runbook
The configuration vllm-torchtpu#2775 validated on the real checkpoint on a 2x2x4 TPU v7x slice: complete GSM8K at 97.50% (1,286/1,319), prefix-cache probes returning identical tokens and logprobs, and BEAM-1M long context matching the GPU reference.
#2775 ran one vllm serve driver over a Ray cluster (TPU_MULTIHOST_BACKEND=ray); the builder renders torchtpu's default multiprocessing backend (per-host --nnodes / --node-rank / --headless). The flags match; the multi-host transport differs. GSM8K ran through the completions API, so the default-on chat and tool-call parsers were not exercised on TPU.
GSM8K capped outputs at 1,024 tokens with 8 concurrent requests; --max-model-len 10240 and --max-num-seqs 16 come from that run, not from a serving-quality target.
TEP32 + DCP8 on the same 2x2x4 v7x slice: complete GSM8K at 97.42%. Pick it when KV capacity, not decode speed, is the limit. DSpark does not support DCP on TPU, so this preset keeps Spec decoding off.
Runbook
The TP32 + DCP8 arm of the vllm-torchtpu#2775 validation on the same slice: complete GSM8K at 97.42% (1,285/1,319), with prefix-cache and long-context checks matching the TP32 arm. Compile sizes drop the 1-batch capture (DCP decode batches start at 8).