Speed test matrix
Qwen3.8-Flash-Next-W4A16 on Intel Arc Pro B65 32GB ×4
vllm · 0.30.0 · INT4_W4A166 contexts (0–200K) × 3 concurrency levels (1–4) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.30.0
- Quantization
- INT4_W4A16
Run
- Served model
- qwen38-flash-next
- Revision
- 40b8f18df4d4a32cb6e687a51c78207e5e438522
- Benchmark tool
- llm-inference-bench v0.7.7
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-lb-g2s16-mtp3-262k --net=host --shm-size=32g --ipc=host --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/models:ro -v /models/bench/lcc-20261021/session-matrix/lb-g2s16-mtp3-262k-matrix/session/cache:/cache --env-file /models/bench/lcc-20261021/session-matrix/lb-g2s16-mtp3-262k-matrix/session/server-configs/lb-g2s16-mtp3-262k.env kasa-lab-vllm-xpu:lumnus-b70.2-b103e0f vllm serve /models/Qwen3.8-Flash-Next-W4A16 --served-model-name qwen38-flash-next --host 0.0.0.0 --port 8000 --dtype bfloat16 --tensor-parallel-size 4 --enable-expert-parallel --trust-remote-code --max-model-len 262144 --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cudagraph-metrics --reasoning-parser qwen3 --override-generation-config \{\"temperature\":\ 0.7\} --enable-prompt-tokens-details --compilation-config \{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"\,\"cudagraph_capture_sizes\":\[1\,2\,4\,8\,12\,16\,20\,24\,28\,32\,36\,40\,44\,48\,52\,56\,60\,64\,256\,512\,1024\]\} --speculative-config \{\"method\":\"mtp\"\,\"num_speculative_tokens\":3\} --no-async-scheduling --max-num-batched-tokens 2048 --gpu-memory-utilization 0.92 --max-num-seqs 16Prefill
prompt tok/s by context length8K context3,9612,069 ms TTFT
16K context3,6974,375 ms TTFT
32K context3,4889,220 ms TTFT
64K context3,05820,969 ms TTFT
128K context2,45452,181 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 |
|---|---|---|---|
| 0 | 84.5119 ms TTFT | 143.8174 ms TTFT | 242.6Requests queued182 ms TTFT |
| 16K | 73.4364 ms TTFT | 122.0Requests queued419 ms TTFT | 184.9Requests queued456 ms TTFT |
| 32K | 77.9477 ms TTFT | 122.8Requests queued546 ms TTFT | 173.1Requests queued592 ms TTFT |
| 64K | 69.6581 ms TTFT | 100.9Requests queued682 ms TTFT | 145.9Requests queued720 ms TTFT |
| 128K | 57.31,176 ms TTFT | 86.5Requests queued1,236 ms TTFT | Over capacity |
| 200K | 49.51,418 ms TTFT | 73.7Requests queued1,464 ms TTFT | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 3,961.0 | — | 2,069 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 3,697.0 | — | 4,375 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,488.0 | — | 9,220 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,058.0 | — | 20,969 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,454.0 | — | 52,181 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 84.5 | 87.8 | 119 ms | 11.4 ms | 0.8 | 11 | 2,534 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 143.8 | 75.6 | 174 ms | 13.2 ms | 1.9 | 19 | 4,310 | 30.0 s |
| Sustained decode | 0 | 4 | Requests queued | 242.6 | 63.0 | 182 ms | 15.9 ms | 3.8 | 35 | 7,278 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 73.4 | 81.8 | 364 ms | 12.2 ms | 0.9 | 9 | 2,202 | 30.0 s |
| Sustained decode | 16K | 2 | Requests queued | 122.0 | 66.6 | 419 ms | 15.0 ms | 1.9 | 16 | 3,655 | 29.9 s |
| Sustained decode | 16K | 4 | Requests queued | 184.9 | 51.1 | 456 ms | 19.6 ms | 3.7 | 22 | 5,548 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 77.9 | 90.6 | 477 ms | 11.1 ms | 0.9 | 10 | 2,304 | 29.6 s |
| Sustained decode | 32K | 2 | Requests queued | 122.8 | 72.4 | 546 ms | 13.8 ms | 1.6 | 16 | 3,681 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 173.1 | 50.7 | 592 ms | 19.7 ms | 3.7 | 24 | 5,193 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 69.6 | 87.3 | 581 ms | 11.5 ms | 0.9 | 9 | 2,086 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 100.9 | 56.8 | 682 ms | 17.6 ms | 1.8 | 12 | 3,026 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 145.9 | 41.2 | 720 ms | 24.3 ms | 3.6 | 18 | 4,376 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 57.3 | 78.0 | 1,176 ms | 12.8 ms | 0.7 | 7 | 1,717 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 86.5 | 54.8 | 1,236 ms | 18.2 ms | 1.7 | 11 | 2,584 | 29.9 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 49.5 | 70.2 | 1,418 ms | 14.2 ms | 0.8 | 6 | 1,485 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 73.7 | 52.4 | 1,464 ms | 19.1 ms | 1.4 | 10 | 2,181 | 29.6 s |
| Sustained decode | 200K | 4 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.