Speed test matrix
Qwen3.8-Flash-Next on H200 SXM 141GB ×4
vllm · 0.1.1.dev5+ge77daef89 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.87 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --enable-expert-parallel --engram-config '{"cpu_offload":false}'Prefill
prompt tok/s by context length8K context33.7K243 ms TTFT
16K context36.7K440 ms TTFT
32K context33.1K971 ms TTFT
64K context34.8K1,841 ms TTFT
128K context36.4K3,519 ms TTFT
251K context34.5K7,273 ms TTFT
256K context33.8K7,576 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 163.165 ms TTFT | 295.787 ms TTFT | 522.5222 ms TTFT | 908.8246 ms TTFT | 1,520.5498 ms TTFT | 2,477.6519 ms TTFT | 4,059.3911 ms TTFT | Looped |
| 16K | 162.483 ms TTFT | 295.0124 ms TTFT | 517.3180 ms TTFT | 904.3415 ms TTFT | 1,523.2457 ms TTFT | 2,479.5759 ms TTFT | 4,071.51,277 ms TTFT | 6,511.02,302 ms TTFT |
| 32K | 161.6122 ms TTFT | 292.8140 ms TTFT | 518.6195 ms TTFT | 901.7322 ms TTFT | 1,472.3554 ms TTFT | 2,459.81,012 ms TTFT | 3,993.21,904 ms TTFT | Over capacity |
| 64K | 160.1287 ms TTFT | 289.4381 ms TTFT | 505.7498 ms TTFT | 903.2802 ms TTFT | 1,470.51,217 ms TTFT | 2,405.72,235 ms TTFT | Over capacity | Over capacity |
| 128K | 159.1374 ms TTFT | 287.3436 ms TTFT | 504.0430 ms TTFT | 891.81,210 ms TTFT | 1,434.32,191 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 155.9865 ms TTFT | 281.6911 ms TTFT | 487.7909 ms TTFT | 855.63,041 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33.7K | — | 243 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 36.7K | — | 440 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33.1K | — | 971 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 34.8K | — | 1,841 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 36.4K | — | 3,519 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 34.5K | — | 7,273 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 33.8K | — | 7,576 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 163.1 | 163.4 | 65 ms | 6.1 ms | 1.0 | 1 | 4,893 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 295.7 | 147.9 | 87 ms | 6.8 ms | 2.0 | 2 | 8,872 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 522.5 | 131.1 | 222 ms | 7.6 ms | 4.0 | 4 | 15,675 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 908.8 | 113.2 | 246 ms | 8.8 ms | 8.0 | 0 | 27,264 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,520.5 | 95.0 | 498 ms | 10.5 ms | 16.0 | 0 | 45,616 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,477.6 | 77.1 | 519 ms | 13.0 ms | 32.0 | 0 | 74,304 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,059.3 | 63.1 | 911 ms | 15.8 ms | 64.0 | 0 | 121,792 | 30.0 s |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 162.4 | 162.9 | 83 ms | 6.1 ms | 1.0 | 1 | 4,872 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 295.0 | 147.9 | 124 ms | 6.8 ms | 2.0 | 2 | 8,850 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 517.3 | 129.7 | 180 ms | 7.7 ms | 3.9 | 4 | 15,519 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 904.3 | 113.1 | 415 ms | 8.8 ms | 8.0 | 0 | 27,128 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,523.2 | 94.7 | 457 ms | 10.6 ms | 16.0 | 0 | 45,696 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,479.5 | 77.3 | 759 ms | 12.9 ms | 32.0 | 0 | 74,368 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,071.5 | 62.7 | 1,277 ms | 15.9 ms | 64.0 | 0 | 122,112 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 6,511.0 | 49.5 | 2,302 ms | 20.2 ms | 128.0 | 0 | 195,328 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 161.6 | 162.4 | 122 ms | 6.2 ms | 1.0 | 1 | 4,849 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 292.8 | 147.1 | 140 ms | 6.8 ms | 2.0 | 2 | 8,782 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 518.6 | 130.3 | 195 ms | 7.7 ms | 4.0 | 4 | 15,557 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 901.7 | 112.5 | 322 ms | 8.9 ms | 8.0 | 0 | 27,048 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,472.3 | 91.5 | 554 ms | 10.9 ms | 16.0 | 0 | 44,160 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,459.8 | 76.1 | 1,012 ms | 13.1 ms | 32.0 | 0 | 73,795 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,993.2 | 61.2 | 1,904 ms | 16.3 ms | 64.0 | 0 | 119,806 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 160.1 | 161.7 | 287 ms | 6.2 ms | 1.0 | 1 | 4,802 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 289.4 | 146.4 | 381 ms | 6.8 ms | 2.0 | 2 | 8,682 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 505.7 | 127.9 | 498 ms | 7.8 ms | 4.0 | 4 | 15,168 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 903.2 | 112.0 | 802 ms | 8.9 ms | 8.0 | 0 | 27,096 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,470.5 | 90.0 | 1,217 ms | 11.1 ms | 16.0 | 0 | 44,112 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,405.7 | 72.4 | 2,235 ms | 13.8 ms | 32.0 | 0 | 72,160 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 159.1 | 161.4 | 374 ms | 6.2 ms | 1.0 | 1 | 4,774 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 287.3 | 145.7 | 436 ms | 6.9 ms | 1.9 | 2 | 8,619 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 504.0 | 127.8 | 430 ms | 7.8 ms | 4.0 | 4 | 15,119 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 891.8 | 110.7 | 1,210 ms | 9.0 ms | 8.0 | 0 | 26,752 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,434.3 | 88.6 | 2,191 ms | 11.3 ms | 16.0 | 0 | 43,024 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 155.9 | 161.1 | 865 ms | 6.2 ms | 1.0 | 1 | 4,677 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 281.6 | 145.2 | 911 ms | 6.9 ms | 1.9 | 2 | 8,445 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 487.7 | 125.9 | 909 ms | 7.9 ms | 3.9 | 4 | 14,631 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 855.6 | 106.7 | 3,041 ms | 9.4 ms | 8.0 | 0 | 25,664 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.