vLLM/Recipes
Search
AutoTrust

autotrust/JEV-27B-VL

Dense 27B vision-language decision model on a Qwen3.8-27B backbone, with a rank-32 LoRA (jev-decision) for calibrated choices over text and images.

Qwen3.8-27B VL backbone; System 1 decisions through the jev-decision LoRA

dense27B262,144 ctxvLLM 0.32.0+Multimodal
Default configuration: 8× MI355X · Tensor Parallel · BF16
export VLLM_ROCM_USE_AITER=1
vllm serve autotrust/JEV-27B-VL \
  --served-model-name autotrust/JEV-27B-VL \
  --enable-lora \
  --max-lora-rank 32 \
  --lora-modules jev-decision=JEV-27B-VL/adapter_vllm \
  --logprobs-mode processed_logprobs \
  --max-model-len 32768 \
  --enable-prefix-caching \
  --mamba-cache-mode align \
  --limit-mm-per-prompt '{"image": 8}' \
  --max-num-seqs 8 \
  --trust-request-chat-template \
  --tensor-parallel-size 1 \
  --attention-backend ROCM_AITER_FA

The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /autotrust/JEV-27B-VL.md.

Guide

Overview

JEV-27B-VL is AutoTrust's vision-language decision model. The checkpoint is a dense BF16 Qwen3_5ForConditionalGeneration model (about 27.8B parameters, vision tower included) continued from Qwen/Qwen3.8-27B. System 2 is the served checkpoint, named autotrust/JEV-27B-VL. System 1 is the rank-32 LoRA jev-decision (adapter_vllm/, r 32, lora_alpha 64), used for calibrated choice probabilities over text or images.

MI355X is verified; see the AMD section below.

Prerequisites

  • vLLM: >= 0.32.0. Earlier releases are not supported for this checkpoint.
  • Weights: hf download autotrust/JEV-27B-VL --local-dir JEV-27B-VL so the LoRA path below exists. The same adapter_vllm directory is in the ModelScope checkout; if that folder is not named JEV-27B-VL, point the module at <checkout>/adapter_vllm (the HF snapshot path ends in snapshots/<rev>/adapter_vllm).
  • Docker: the image only bind-mounts the Hugging Face cache. Mount the checkout into the container and keep jev-decision pointed at that adapter_vllm directory.
  • Context: text_config.max_position_embeddings is 262144. The validated command uses --max-model-len 32768 (the serve.sh default). The card says a full 256K prompt needs about 17 GB of KV cache on top of the weights, and suggests --max-model-len 131072 on an 80 GB GPU.

Decision LoRA and serve flags

These flags are the model card's validated vllm serve line. They are base arguments, not optional toggles.

  • jev-decision — --enable-lora, --max-lora-rank 32, and --lora-modules jev-decision=JEV-27B-VL/adapter_vllm. The module name stays jev-decision. The path is the local checkout JEV-27B-VL/adapter_vllm, or the adapter_vllm directory inside an HF or ModelScope checkout.
  • --mamba-cache-mode align — the Qwen3.8 hybrid backbone mixes full attention with Mamba-style layers; the card serves it in align mode.
  • --limit-mm-per-prompt '{"image": 8}' — at most 8 images per request.
  • --max-num-seqs 8 — required. The card says that with more than 8 sequences in one batch, vLLM's LoRA path for this multimodal class returns wrong System 1 probabilities. Extra requests queue.
  • --logprobs-mode processed_logprobs — System 1 reads post-processor logprobs. On /v1/chat/completions or /v1/completions for jev-decision, the card also says to send top_k: 0 and top_p: 1.0: generation_config.json sets top_k 20 and top_p 0.95, and those defaults would truncate the returned distribution.
  • --trust-request-chat-template — image decisions send a raw decision chat template on the request. System 2 keeps the checkpoint's own template.
  • --served-model-name autotrust/JEV-27B-VL — public name for System 2. --enable-prefix-caching matches the card's serve line.

POST /v1/decide is not part of stock vllm serve. The checkpoint ships serve_decide.py, which starts the OpenAI server with these same flags and adds that route. Stock vllm serve still exposes System 2 as autotrust/JEV-27B-VL and System 1 as the LoRA model jev-decision.

AMD (1x MI355X, BF16)

One MI355X at TP1. VLLM_ROCM_USE_AITER=1 and --attention-backend ROCM_AITER_FA select AITER FlyDSL for the 48 gated-delta-net layers and ROCM_AITER_FA for the 16 full-attention layers.

docker run --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host --shm-size 16g \
  -e VLLM_ROCM_USE_AITER=1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /path/to/JEV-27B-VL:/workspace/JEV-27B-VL \
  -p 8000:8000 \
  vllm/vllm-openai-rocm:latest \
  autotrust/JEV-27B-VL \
  --served-model-name autotrust/JEV-27B-VL \
  --tensor-parallel-size 1 \
  --attention-backend ROCM_AITER_FA \
  --enable-lora --max-lora-rank 32 \
  --lora-modules jev-decision=/workspace/JEV-27B-VL/adapter_vllm \
  --logprobs-mode processed_logprobs \
  --max-model-len 32768 \
  --enable-prefix-caching \
  --mamba-cache-mode align \
  --limit-mm-per-prompt '{"image": 8}' \
  --max-num-seqs 8 \
  --trust-request-chat-template

Verified on 1x MI355X with :latest (vLLM 0.33.0); :nightly gives the same results:

  • System 1 on the card's example (billing / shipping / tech support) returns 0.9970 / 0.0000 / 0.0030 with 49 prompt tokens; the card reports 0.9969 / 0.0000 / 0.0031 and 49.
  • Images work on System 2 and System 1. text_only rejects image requests; encoder_parallel serves them.
  • At --max-model-len 262144, a needle was retrieved from a 250,391-token prompt.
  • vllm bench serve, 1000 in / 256 out, concurrency 32: about 550 output tok/s and 2,700 total tok/s. Median TTFT is about 11.4 s because --max-num-seqs 8 keeps only 8 requests running.

Client usage

System 2 (the base checkpoint), with an image:

client.chat.completions.create(
    model="autotrust/JEV-27B-VL",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": image_url}},
            {"type": "text", "text": "What does this image show?"},
        ],
    }],
)

System 1 through the checkpoint's /v1/decide route (serve_decide.py):

curl localhost:8000/v1/decide -H 'Content-Type: application/json' -d '{
  "kind": "choice",
  "state": "Customer: my card was charged twice for one coffee.",
  "question": "Which team should handle this?",
  "options": ["billing", "shipping", "tech support"]}'

Text-only System 1 calls the LoRA directly (model jev-decision). The card says those text decisions match autotrust/JEV-27B.

References