Speed test matrix
Qwen3.8-Flash-Next-W4A16 on Intel Arc Pro B65 32GB ×4
vllm · 0.30.0 · INT4_W4A166 contexts (0–200K) × 3 concurrency levels (1–4) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.30.0
- Quantization
- INT4_W4A16
Run
- Served model
- qwen38-flash-next
- Revision
- 40b8f18df4d4a32cb6e687a51c78207e5e438522
- Benchmark tool
- llm-inference-bench v0.7.7
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Prefill
prompt tok/s by context length8K context3,5132,333 ms TTFT
16K context3,3484,830 ms TTFT
32K context3,10410,359 ms TTFT
64K context2,65024,190 ms TTFT
128K context2,04162,709 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 |
|---|---|---|---|
| 0 | 42.7117 ms TTFT | 85.1173 ms TTFT | 159.2246 ms TTFT |
| 16K | 40.8242 ms TTFT | 79.4343 ms TTFT | 141.9Requests queued510 ms TTFT |
| 32K | 39.3363 ms TTFT | 75.2Requests queued366 ms TTFT | 127.3Requests queued783 ms TTFT |
| 64K | 38.6352 ms TTFT | 73.5455 ms TTFT | 127.6594 ms TTFT |
| 128K | 34.5751 ms TTFT | 61.2Requests queued993 ms TTFT | 101.7Requests queued1,477 ms TTFT |
| 200K | 31.5953 ms TTFT | 57.0Requests queued1,215 ms TTFT | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 3,513.0 | — | 2,333 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 3,348.0 | — | 4,830 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,104.0 | — | 10,359 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 2,650.0 | — | 24,190 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,041.0 | — | 62,709 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 42.7 | 43.4 | 117 ms | 23.1 ms | 1.0 | 5 | 1,281 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 85.1 | 43.6 | 173 ms | 22.9 ms | 2.0 | 10 | 2,551 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 159.2 | 41.2 | 246 ms | 24.3 ms | 3.9 | 20 | 4,772 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 40.8 | 42.3 | 242 ms | 23.6 ms | 1.0 | 5 | 1,224 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 79.4 | 41.9 | 343 ms | 23.9 ms | 1.9 | 10 | 2,381 | 30.0 s |
| Sustained decode | 16K | 4 | Requests queued | 141.9 | 37.8 | 510 ms | 26.4 ms | 3.9 | 16 | 4,256 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 39.3 | 41.6 | 363 ms | 24.0 ms | 0.9 | 5 | 1,178 | 30.0 s |
| Sustained decode | 32K | 2 | Requests queued | 75.2 | 41.3 | 366 ms | 24.2 ms | 1.9 | 10 | 2,238 | 29.8 s |
| Sustained decode | 32K | 4 | Requests queued | 127.3 | 34.9 | 783 ms | 28.7 ms | 3.9 | 16 | 3,816 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 38.6 | 40.4 | 352 ms | 24.8 ms | 0.9 | 5 | 1,151 | 29.8 s |
| Sustained decode | 64K | 2 | Valid | 73.5 | 38.9 | 455 ms | 25.7 ms | 2.0 | 8 | 2,202 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 127.6 | 34.5 | 594 ms | 29.0 ms | 4.0 | 16 | 3,824 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 34.5 | 38.1 | 751 ms | 26.2 ms | 0.9 | 4 | 1,034 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 61.2 | 35.2 | 993 ms | 28.4 ms | 1.8 | 8 | 1,836 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 101.7 | 28.9 | 1,477 ms | 34.6 ms | 3.5 | 12 | 3,048 | 30.0 s |
| Sustained decode | 200K | 1 | Valid | 31.5 | 35.9 | 953 ms | 27.8 ms | 0.9 | 4 | 944 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 57.0 | 32.3 | 1,215 ms | 31.0 ms | 1.8 | 6 | 1,710 | 30.0 s |
| Sustained decode | 200K | 4 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.