Speed test matrix
Qwen3.8-Flash-Next on H200 SXM 141GB ×4
vllm · 0.1.dev1+g50a061f79 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g50a061f79
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
50a061f792f3
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLE_MMAP=1 "$HOME/env_h200-qwen-flash-next-vllm-pr-54129/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel
Prefill
prompt tok/s by context length8K context25.2K325 ms TTFT
16K context29.5K547 ms TTFT
32K context30.2K1,065 ms TTFT
64K context30.9K2,075 ms TTFT
128K context31K4,136 ms TTFT
251K context29.6K8,481 ms TTFT
256K context28.4K9,021 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 137.1208 ms TTFT | 234.9151 ms TTFT | 382.4295 ms TTFT | 603.6310 ms TTFT | 929.7815 ms TTFT | 1,544.2674 ms TTFT | 2,557.51,212 ms TTFT | 4,301.21,959 ms TTFT |
| 16K | 138.1365 ms TTFT | 239.3200 ms TTFT | 391.8373 ms TTFT | 626.61,221 ms TTFT | 953.41,065 ms TTFT | 1,587.91,155 ms TTFT | 2,660.31,641 ms TTFT | 4,373.42,593 ms TTFT |
| 32K | 137.8662 ms TTFT | 239.8244 ms TTFT | 396.1636 ms TTFT | 622.8710 ms TTFT | 956.9790 ms TTFT | 1,581.91,211 ms TTFT | 2,688.62,008 ms TTFT | 4,333.33,695 ms TTFT |
| 64K | 136.51,300 ms TTFT | 241.7517 ms TTFT | 393.2866 ms TTFT | 623.01,251 ms TTFT | 975.11,707 ms TTFT | 1,542.12,645 ms TTFT | 2,621.84,427 ms TTFT | 4,274.08,062 ms TTFT |
| 128K | 133.82,337 ms TTFT | 240.7718 ms TTFT | 389.3918 ms TTFT | 619.71,492 ms TTFT | 953.52,678 ms TTFT | 1,540.84,724 ms TTFT | 2,563.79,129 ms TTFT | 4,117.318,522 ms TTFT |
| 251K | 131.74,745 ms TTFT | 239.91,378 ms TTFT | 390.72,026 ms TTFT | 607.09,916 ms TTFT | 907.55,231 ms TTFT | 1,468.99,579 ms TTFT | 2,468.318,372 ms TTFT | 3,864.036,752 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 25.2K | — | 325 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.5K | — | 547 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.2K | — | 1,065 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30.9K | — | 2,075 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 31K | — | 4,136 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 29.6K | — | 8,481 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 28.4K | — | 9,021 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 137.1 | 138.6 | 208 ms | 7.2 ms | 1.0 | 1 | 4,112 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 234.9 | 117.5 | 151 ms | 8.5 ms | 2.0 | 0 | 7,046 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 382.4 | 96.0 | 295 ms | 10.4 ms | 4.0 | 0 | 11,468 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 603.6 | 75.6 | 310 ms | 13.2 ms | 8.0 | 0 | 18,104 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 929.7 | 57.8 | 815 ms | 17.3 ms | 16.0 | 0 | 27,888 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,544.2 | 48.2 | 674 ms | 20.7 ms | 32.0 | 0 | 46,304 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,557.5 | 39.4 | 1,212 ms | 25.4 ms | 64.0 | 0 | 76,736 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,301.2 | 33.1 | 1,959 ms | 30.2 ms | 128.0 | 0 | 129,024 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 138.1 | 139.8 | 365 ms | 7.2 ms | 1.0 | 1 | 4,144 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 239.3 | 120.5 | 200 ms | 8.3 ms | 2.0 | 2 | 7,178 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 391.8 | 98.2 | 373 ms | 10.2 ms | 4.0 | 0 | 11,752 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 626.6 | 78.3 | 1,221 ms | 12.8 ms | 8.0 | 0 | 18,792 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 953.4 | 60.2 | 1,065 ms | 16.6 ms | 16.0 | 0 | 28,592 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,587.9 | 49.8 | 1,155 ms | 20.1 ms | 32.0 | 0 | 47,616 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,660.3 | 41.5 | 1,641 ms | 24.1 ms | 64.0 | 0 | 79,808 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,373.4 | 33.6 | 2,593 ms | 29.8 ms | 128.0 | 0 | 131,200 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 137.8 | 140.1 | 662 ms | 7.1 ms | 1.0 | 1 | 4,135 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 239.8 | 120.6 | 244 ms | 8.3 ms | 2.0 | 2 | 7,181 | 29.9 s |
| Sustained decode | 32K | 4 | Valid | 396.1 | 98.1 | 636 ms | 10.2 ms | 4.0 | 0 | 11,882 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 622.8 | 77.6 | 710 ms | 12.9 ms | 8.0 | 0 | 18,680 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 956.9 | 59.0 | 790 ms | 17.0 ms | 16.0 | 0 | 28,704 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,581.9 | 47.9 | 1,211 ms | 20.9 ms | 32.0 | 0 | 47,456 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,688.6 | 40.2 | 2,008 ms | 24.9 ms | 64.0 | 0 | 80,640 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,333.3 | 32.1 | 3,695 ms | 31.1 ms | 128.0 | 0 | 130,048 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 136.5 | 139.3 | 1,300 ms | 7.2 ms | 1.0 | 1 | 4,093 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 241.7 | 120.6 | 517 ms | 8.3 ms | 2.0 | 2 | 7,224 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 393.2 | 97.9 | 866 ms | 10.2 ms | 4.0 | 0 | 11,796 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 623.0 | 77.5 | 1,251 ms | 12.9 ms | 8.0 | 0 | 18,688 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 975.1 | 60.1 | 1,707 ms | 16.6 ms | 16.0 | 0 | 29,248 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,542.1 | 46.6 | 2,645 ms | 21.5 ms | 32.0 | 0 | 46,240 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,621.8 | 38.0 | 4,427 ms | 26.3 ms | 64.0 | 0 | 78,656 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,274.0 | 28.6 | 8,062 ms | 34.9 ms | 128.0 | 0 | 128,256 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 133.8 | 137.9 | 2,337 ms | 7.3 ms | 1.0 | 1 | 4,014 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 240.7 | 120.6 | 718 ms | 8.3 ms | 2.0 | 1 | 7,221 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 389.3 | 96.7 | 918 ms | 10.3 ms | 4.0 | 0 | 11,680 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 619.7 | 76.2 | 1,492 ms | 13.1 ms | 8.0 | 0 | 18,584 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 953.5 | 57.7 | 2,678 ms | 17.3 ms | 16.0 | 0 | 28,592 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,540.8 | 45.3 | 4,724 ms | 22.1 ms | 32.0 | 0 | 46,208 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,563.7 | 36.1 | 9,129 ms | 27.7 ms | 64.0 | 0 | 76,864 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 4,117.3 | 27.3 | 18,522 ms | 36.7 ms | 128.0 | 0 | 123,520 | 30.0 s |
| Sustained decode | 251K | 1 | Valid | 131.7 | 139.1 | 4,745 ms | 7.2 ms | 1.0 | 1 | 3,951 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 239.9 | 119.9 | 1,378 ms | 8.3 ms | 2.0 | 0 | 7,196 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 390.7 | 97.1 | 2,026 ms | 10.3 ms | 4.0 | 0 | 11,720 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 607.0 | 77.6 | 9,916 ms | 12.9 ms | 8.0 | 0 | 18,210 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 907.5 | 55.9 | 5,231 ms | 17.9 ms | 16.0 | 0 | 27,216 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,468.9 | 44.4 | 9,579 ms | 22.5 ms | 32.0 | 0 | 44,064 | 30.0 s |
| Sustained decode | 251K | 64 | Valid | 2,468.3 | 36.4 | 18,372 ms | 27.5 ms | 64.0 | 0 | 74,048 | 30.0 s |
| Sustained decode | 251K | 128 | Valid | 3,864.0 | 28.2 | 36,752 ms | 35.5 ms | 128.0 | 0 | 115,839 | 30.0 s |