IFM/PAN2
29B foundation world model for text-to-video and image-to-video: a dual-stream diffusion transformer that simulates how a scene evolves under a time-grounded plan
29B video DiT — 13 s of 704x1248 video in ~119 s on 4×B300 (CFG parallel × Ulysses 2)
vllm serve IFM/PAN2 \ --num-gpus 4 \ --cfg-parallel-size 2 \ --usp 2 \ --vae-patch-parallel-size 4 \ --vae-parallel-mode tile \ --omni
The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /IFM/PAN2.md.
Overview
PAN 2 is a foundation world model that simulates how a scene evolves under a time-grounded plan of actions and camera movements. Its 29B generative simulator is a dual-stream multimodal diffusion transformer (MMDiT): the prompt is encoded into a continuous state stream, and the state and video streams, each with its own projections and feed-forward networks, are joined by self-attention at every layer. The plan guides video denoising, while the developing video refines the state. One server handles both tasks: a request with a first-frame image runs image-to-video (I2V), a request without one runs text-to-video (T2V).
| Item | Contract |
|---|---|
| Tasks | T2V; I2V from one first-frame image |
| Prompt | One structured-caption prompt per request, plus an optional negative prompt |
| Default output | 704x1248, 313 frames (13 s) at 24 fps |
| Longest output | 481 frames (20 s) at 24 fps |
| Size | height and width multiples of 32 |
| Frame count | num_frames of the form 4k+1 (duration_sec * 24 + 1 for whole seconds) |
| Sampling | 50 steps and CFG 2.5 by default; PAN 2's own flow-matching schedule |
Prompt format
PAN 2 needs a structured prompt. It is trained on the captions written by its
prompt rewriter,
and expects the prompt as that caption serialized to compact one-line
JSON (json.dumps(caption, ensure_ascii=False, separators=(",", ":"))).
A plain one-line prompt produces unusable video. The caption describes the
style, lighting, camera, subjects, the first frame, and a timeline of events:
{
"video_style": "Serene aerial travel footage during blue hour, captured in one uninterrupted continuous shot.",
"lighting": "Soft blue skylight after sunset, a pale violet horizon and small pools of amber light ...",
"perspective_view": "Wide drone view looking obliquely down along Santorini's caldera rim ...",
"subjects_lifecycle": [],
"init_frame_visual_description": "Two cobalt-blue church domes rise among tightly stepped white Cycladic buildings ...",
"events": [
{
"event_index": 0,
"time_range_sec": "0 to 4",
"event_description": "The drone glides gently right along the cliff ...",
"camera_motion": [{"time_range_sec": "0 to 4", "camera_motion": ["track_right"]}],
"contained_actions": []
}
],
"quality": {"video_quality": "Resolved architectural edges ...", "text_logos_overlays": null, "artifacts": null}
}
The negative prompt uses the same format. Without one, or with an empty one,
PAN 2 uses the negative prompt it was released with, its time ranges set to
the clip's duration. The request examples read the prompt and negative
prompt from prompt.json and negative_prompt.json
(i2v_prompt.json and i2v_negative_prompt.json for image-to-video).
Guardrails
PAN 2 runs guardrails on the prompt and the generated video, and they may
block a request with HTTP 400. They need the pan2-guardrail package.
Hardware profiles
Pick the hardware above and the command updates. Every profile is sized for the longest 481-frame, 704x1248 output. Peaks are per GPU.
| GPU memory | GPUs | Key arguments | Peak memory |
|---|---|---|---|
| ≥141 GB (H200, B200, B300, GB200, GB300) | 1 | (none) | 107.4 GiB |
| 4 | --cfg-parallel-size 2 --usp 2 | 110.7 GiB | |
| 8 | --cfg-parallel-size 2 --usp 4 | 110.4 GiB | |
| 96 GB (RTX PRO 6000) | 1 | --enable-distributed-layerwise-offload --dlo-resident-layers 48, offload_guardrail_models | 84.4 GiB |
| 2 | --tensor-parallel-size 2 | 84.5 GiB |
offload_guardrail_models is set with
--stage-overrides '{"0": {"model_config": {"offload_guardrail_models": true}}}'.
With --enable-distributed-layerwise-offload, the DiT blocks stay in pinned
host memory and stream to the GPU each step, with --dlo-resident-layers
leading blocks kept on the GPU. The offloaded weights live in pinned system
RAM, so these profiles need a large amount of it, and each CFG-parallel
branch adds its own copy.
Request options
-F "size=1248x704" -F "num_frames=481" \
-F "num_inference_steps=50" -F "guidance_scale=2.5"
Use num_frames to set the length; seconds is not used. Requests with a
size that is not a multiple of 32, a frame count that is not 4k+1, more
than one image, video input, or custom sigmas/timesteps are rejected
with HTTP 400 before they reach the GPUs.
Request-scoped quality
PAN 2 can switch Cache-DiT per request without a startup option:
-F 'quality=lossless' runs the reference path, -F 'quality=high' PAN 2's
conservative Cache-DiT profile. high is about 1.3× faster than lossless;
the speedup depends on the prompt and the parallel configuration.