Speed test matrix
Qwen3.8-Flash-Next-W4A16 on Intel Arc Pro B65 32GB ×4
vllm · 0.30.0 · INT4_W4A166 contexts (0–200K) × 3 concurrency levels (1–4) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.30.0
- Quantization
- INT4_W4A16
Run
- Served model
- qwen38-flash-next
- Revision
- 40b8f18df4d4a32cb6e687a51c78207e5e438522
- Benchmark tool
- llm-inference-bench v0.7.7
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-pf-mtp3-mbt8192 --net=host --shm-size=32g --ipc=host --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/models:ro -v /models/bench/fnpf-20261020/session-p3/pf-mtp3-mbt8192/session/cache:/cache --env-file /models/bench/fnpf-20261020/session-p3/pf-mtp3-mbt8192/session/server-configs/pf-mtp3-mbt8192.env kasa-lab-vllm-xpu:lumnus-b70.2-b103e0f vllm serve /models/Qwen3.8-Flash-Next-W4A16 --served-model-name qwen38-flash-next --host 0.0.0.0 --port 8000 --dtype bfloat16 --tensor-parallel-size 4 --enable-expert-parallel --trust-remote-code --gpu-memory-utilization 0.88 --max-model-len 262144 --max-num-batched-tokens 8192 --max-num-seqs 4 --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cudagraph-metrics --reasoning-parser qwen3 --override-generation-config \{\"temperature\":\ 0.7\} --enable-prompt-tokens-details --compilation-config \{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"\,\"cudagraph_capture_sizes\":\[1\,2\,4\,8\,12\,16\,20\,24\,28\,32\,36\,40\,44\,48\,52\,56\,60\,64\,256\,512\,1024\]\} --speculative-config \{\"method\":\"mtp\"\,\"num_speculative_tokens\":3\} --no-async-schedulingPrefill
prompt tok/s by context length8K context5,1921,579 ms TTFT
16K context5,2063,107 ms TTFT
32K context4,8426,641 ms TTFT
64K context4,29514,929 ms TTFT
128K context3,42037,440 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 |
|---|---|---|---|
| 0 | 84.4119 ms TTFT | 147.3173 ms TTFT | 246.3182 ms TTFT |
| 16K | 75.5371 ms TTFT | 121.2Requests queued420 ms TTFT | 183.5Requests queued428 ms TTFT |
| 32K | 78.3475 ms TTFT | 117.4Requests queued546 ms TTFT | 167.9Requests queued588 ms TTFT |
| 64K | 64.9583 ms TTFT | 100.0Requests queued681 ms TTFT | 141.3Requests queued718 ms TTFT |
| 128K | 54.61,182 ms TTFT | 85.0Requests queued1,240 ms TTFT | Over capacity |
| 200K | 51.41,400 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,192.0 | — | 1,579 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,206.0 | — | 3,107 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,842.0 | — | 6,641 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,295.0 | — | 14,929 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,420.0 | — | 37,440 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 84.4 | 87.8 | 119 ms | 11.4 ms | 0.9 | 11 | 2,530 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 147.3 | 76.9 | 173 ms | 13.0 ms | 1.9 | 20 | 4,419 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 246.3 | 63.1 | 182 ms | 15.8 ms | 3.8 | 36 | 7,387 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 75.5 | 84.9 | 371 ms | 11.8 ms | 0.9 | 9 | 2,265 | 30.0 s |
| Sustained decode | 16K | 2 | Requests queued | 121.2 | 66.5 | 420 ms | 15.0 ms | 1.9 | 18 | 3,631 | 30.0 s |
| Sustained decode | 16K | 4 | Requests queued | 183.5 | 51.0 | 428 ms | 19.6 ms | 3.7 | 24 | 5,474 | 29.8 s |
| Sustained decode | 32K | 1 | Valid | 78.3 | 95.0 | 475 ms | 10.5 ms | 0.8 | 10 | 2,350 | 30.0 s |
| Sustained decode | 32K | 2 | Requests queued | 117.4 | 67.6 | 546 ms | 14.8 ms | 1.7 | 16 | 3,521 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 167.9 | 48.7 | 588 ms | 20.5 ms | 3.6 | 21 | 5,032 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 64.9 | 74.3 | 583 ms | 13.5 ms | 0.8 | 8 | 1,947 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 100.0 | 57.4 | 681 ms | 17.4 ms | 1.7 | 12 | 3,000 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 141.3 | 41.1 | 718 ms | 24.3 ms | 3.5 | 17 | 4,236 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 54.6 | 75.4 | 1,182 ms | 13.3 ms | 0.7 | 7 | 1,637 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 85.0 | 53.3 | 1,240 ms | 18.7 ms | 1.5 | 11 | 2,548 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 51.4 | 72.7 | 1,400 ms | 13.8 ms | 0.7 | 6 | 1,541 | 30.0 s |
| Sustained decode | 200K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 4 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.