Latency

Up to 60% faster for several questions about one state. The same decisions.

Each answer is one output token, so latency is prompt processing: the state, its images and every question. We timed each stage and removed the redundant work.

One question

	fast 0.8B	balanced 4B	quality 35B-A3B
Support ticket	70 ms −13%	278 ms −4%	332 ms −6%
1.2k-token policy	205 ms −7%	996 ms −1%	1.09 s −2%
448×672 screenshot	136 ms −7%	735 ms −1%	1.04 s −3%
2048×1536 photo	256 ms −18%	911 ms −6%	1.42 s −1%
Repeated request	14 ms −41%	35 ms −20%	34 ms −20%

Four questions about the same state

	fast 0.8B	balanced 4B	quality 35B-A3B
Support ticket	165 ms −35%	623 ms −42%	859 ms −35%
1.2k-token policy	311 ms −41%	1.43 s −45%	1.68 s −39%
448×672 screenshot	235 ms −53%	1.00 s −60%	2.09 s −57%

Median of 7 interleaved trials, measured at the client, with the change from the previous version. Method

Server time on every response

"timing": {
  "processing_ms": 131.4,
  "parse_ms": 0.9,
  "prepare_ms": 2.1,
  "queue_ms": 0.0,
  "inference_ms": 128.2
}

processing_ms runs from the moment the request arrives to the moment the response is ready; the stages add up to it. The x-openjev-processing-ms and Server-Timing headers carry the same values, on errors too. model, answers and usage keep the Jev shape, and the official typesafe-sdk ignores the extra field. OPENJEV_RESPONSE_TIMING=false removes it. Field reference

Where the time went

Previous version, balanced profile:

Stage	Cost	Paid
Chat template (/apply-template)	9 ms at any content size	per request
Token count (/tokenize)	0.6 ms for 150 tokens, 2.7 ms for 1,250	per question, in turn
Shared state and images	read and encoded again	per additional question
Photo preparation in the API	65 ms for 2048×1536, 119 ms for 4032×3024, then resized again by llama.cpp	per image
Prompt processing	0.29 s for a 276-token ticket, 1.0 s for a 1.2k-token policy, 0.75 s for a 448×672 screenshot	per token and image

Qwen3.5 and Qwen3.6 are hybrid models. Their recurrent layers cannot roll back to an arbitrary position, and llama.cpp checkpoints only near the end of each prompt, so every further question re-read the whole state: forcing full prompt re-processing due to lack of cache data.

What changed

Prefix priming. A request with several questions evaluates the shared prefix once. Each question resumes from that checkpoint and reads only its own text. Usage still counts every question's full prompt.

Template skeleton. The Qwen template wraps each user or system message in fixed text. The API renders that frame once, checks its first results against llama.cpp and then fills it locally, saving a backend call on every request. Assistant and tool messages, and content with unusual edge whitespace, still go to llama.cpp.

Parallel token counts. The shared prefix is counted once, the questions in parallel.

One resize per image. An oversized image goes straight to the vision encoder's size on its 32 px grid, instead of being resized twice. Large JPEGs decode at reduced scale: API-side preparation drops to 31 ms for 2048×1536 and 55 ms for 4032×3024.

Repeated state

When consecutive requests repeat a state with new questions, the API keeps a checkpoint right after the state, and each request reads only its question. OpenJev's llama.cpp patch goes one step further: it places that checkpoint exactly at the end of the state, and it lets a request skip the extra pass llama.cpp otherwise spends to checkpoint the end of every prompt.

Qwen3.8-27B, compact Tetris prompt	Median per decision
Stock llama.cpp	1.93 s
With the repeated-state cache	0.83 s
With the cache and the patch	0.66 s

Sixteen decisions per setup, one server at a time (receipts). All 32 answers of a fixed check set matched stock llama.cpp. OPENJEV_PRIME_REPEATED_STATE=false turns the cache off.

Several states can take turns. The llama.cpp that scripts/build-llama.sh builds keeps earlier prompts in memory with their checkpoints, so four 800-token states used in rotation each answered in 0.11 s on Qwen3.5-4B, against 0.69 s with llama.cpp b9670, which re-read every state.

Accuracy

Every trial sends the identical request to both versions.

• Decisions: the same in all 168 pairs (3 profiles × 8 requests × 7 trials).
• One question, no large image: identical probabilities.
• Several questions: probabilities within 0.001 on the 0.8B and 4B models, within 0.067 on 35B-A3B. The prefix now runs as its own batch, which changes the floating-point summation order.
• Photos: within 0.018, from one resampling instead of two.

Not shipped: an MLX engine

An in-process MLX engine was faster with images on the smaller models: a screenshot with four questions took 0.11 s on 0.8B and 0.57 s on 4B, against 0.24 s and 1.00 s with llama.cpp. It is not included.

• Different answers. Probabilities moved by up to 0.52. The 0.8B model changed decisions in 5 of 8 request types; the larger models, on screenshots.
• Slower on 35B-A3B. 1.01 s for one question about the ticket, against 0.33 s.
• Unstable. A vision run ended in a macOS kernel panic in the GPU driver. A retry needs MLX memory and cache limits, a cache flush after every request and no concurrent GPU work.

The receipts keep these measurements.

Next

• Batch the questions of one request from the shared checkpoint.
• Label-prior calibration and multi-token option scoring, evaluated on their own because they change probabilities.
• Continuous batching across requests, for throughput.
• A trained readout, such as LoRA or distillation from the quality profile, once labeled decisions exist.

Method

scripts/latency.py sends each new request to both versions in rotating order. Each version runs its own llama.cpp process, so neither answers from the other's cache. A fresh identifier opens every state, so every request starts cold; the repeat case times the second of two identical requests. Seven trials per request after two warm-ups, on an M3 Max (128 GB) with llama.cpp b9670, September 21, 2026. No sample overlapped other inference.

uv run python scripts/latency.py --quiet --output benchmarks/performance/balanced.json \
  before=http://127.0.0.1:8101 after=http://127.0.0.1:8100
uv run python scripts/render_performance.py --chart

Receipts: fast · balanced · quality