Speed test matrix
Qwen3.8-Flash-Next on H200 SXM 141GB ×4
vllm · 0.1.1.dev5+ge77daef89 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --enable-expert-parallel --engram-config '{"cpu_offload":true}' --speculative-config '{"method":"mtp","num_speculative_tokens":1}'Prefill
prompt tok/s by context length8K context28.1K292 ms TTFT
16K context29.8K542 ms TTFT
32K context33.2K970 ms TTFT
64K context35.5K1,806 ms TTFT
128K context34.6K3,700 ms TTFT
251K context32.4K7,743 ms TTFT
256K context31.8K8,059 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 224.2154 ms TTFT | 404.6173 ms TTFT | 694.2179 ms TTFT | 1,145.1307 ms TTFT | 1,923.5422 ms TTFT | 3,257.5819 ms TTFT | 5,281.01,270 ms TTFT | 7,950.52,198 ms TTFT |
| 16K | 223.1170 ms TTFT | 403.7184 ms TTFT | 688.5186 ms TTFT | 1,145.6314 ms TTFT | 1,936.3401 ms TTFT | 3,278.2946 ms TTFT | 5,267.31,404 ms TTFT | 7,875.42,259 ms TTFT |
| 32K | 222.5208 ms TTFT | 400.9234 ms TTFT | 693.7266 ms TTFT | 1,139.9356 ms TTFT | 1,935.7845 ms TTFT | 3,210.21,246 ms TTFT | Looped | Over capacity |
| 64K | 222.2288 ms TTFT | 402.6367 ms TTFT | 686.8380 ms TTFT | 1,132.6529 ms TTFT | 1,917.71,174 ms TTFT | 3,185.42,340 ms TTFT | Over capacity | Over capacity |
| 128K | 238.6463 ms TTFT | 395.3505 ms TTFT | 692.8539 ms TTFT | 1,088.6614 ms TTFT | 1,888.12,005 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 225.2923 ms TTFT | 401.41,010 ms TTFT | 676.71,213 ms TTFT | 1,057.31,547 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 28.1K | — | 292 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.8K | — | 542 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33.2K | — | 970 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 35.5K | — | 1,806 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 34.6K | — | 3,700 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 32.4K | — | 7,743 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 31.8K | — | 8,059 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 224.2 | 225.7 | 154 ms | 4.4 ms | 1.0 | 1 | 6,726 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 404.6 | 202.0 | 173 ms | 5.0 ms | 2.0 | 2 | 12,136 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 694.2 | 175.5 | 179 ms | 5.7 ms | 4.0 | 4 | 20,825 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,145.1 | 143.1 | 307 ms | 7.0 ms | 8.0 | 8 | 34,354 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,923.5 | 120.0 | 422 ms | 8.3 ms | 15.9 | 7 | 57,525 | 29.9 s |
| Sustained decode | 8K | 32 | Valid | 3,257.5 | 101.2 | 819 ms | 9.9 ms | 32.0 | 0 | 97,704 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,281.0 | 81.1 | 1,270 ms | 12.3 ms | 64.0 | 0 | 158,397 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,950.5 | 60.0 | 2,198 ms | 16.7 ms | 128.0 | 0 | 238,487 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 223.1 | 225.1 | 170 ms | 4.4 ms | 1.0 | 1 | 6,692 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 403.7 | 202.3 | 184 ms | 4.9 ms | 2.0 | 2 | 12,109 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 688.5 | 173.0 | 186 ms | 5.8 ms | 4.0 | 4 | 20,653 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,145.6 | 142.9 | 314 ms | 7.0 ms | 8.0 | 8 | 34,365 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,936.3 | 120.4 | 401 ms | 8.3 ms | 16.0 | 6 | 57,938 | 29.9 s |
| Sustained decode | 16K | 32 | Valid | 3,278.2 | 101.9 | 946 ms | 9.8 ms | 32.0 | 0 | 98,347 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,267.3 | 81.3 | 1,404 ms | 12.3 ms | 64.0 | 0 | 157,959 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 7,875.4 | 59.7 | 2,259 ms | 16.7 ms | 128.0 | 0 | 236,294 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 222.5 | 225.1 | 208 ms | 4.4 ms | 1.0 | 1 | 6,675 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 400.9 | 202.1 | 234 ms | 4.9 ms | 2.0 | 2 | 12,025 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 693.7 | 173.1 | 266 ms | 5.8 ms | 4.0 | 4 | 20,805 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,139.9 | 143.4 | 356 ms | 7.0 ms | 7.9 | 8 | 34,187 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,935.7 | 119.7 | 845 ms | 8.4 ms | 16.0 | 2 | 58,059 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,210.2 | 98.2 | 1,246 ms | 10.2 ms | 32.0 | 0 | 96,299 | 30.0 s |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 222.2 | 225.1 | 288 ms | 4.4 ms | 1.0 | 1 | 6,665 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 402.6 | 201.2 | 367 ms | 5.0 ms | 2.0 | 2 | 12,077 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 686.8 | 173.2 | 380 ms | 5.8 ms | 4.0 | 4 | 20,601 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,132.6 | 142.4 | 529 ms | 7.0 ms | 7.9 | 8 | 33,979 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,917.7 | 116.8 | 1,174 ms | 8.6 ms | 16.0 | 3 | 57,511 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,185.4 | 95.9 | 2,340 ms | 10.4 ms | 32.0 | 0 | 95,529 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 238.6 | 241.4 | 463 ms | 4.1 ms | 1.0 | 1 | 7,159 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 395.3 | 202.2 | 505 ms | 4.9 ms | 1.9 | 2 | 11,859 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 692.8 | 174.8 | 539 ms | 5.7 ms | 4.0 | 4 | 20,780 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,088.6 | 136.7 | 614 ms | 7.3 ms | 8.0 | 8 | 32,648 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,888.1 | 114.8 | 2,005 ms | 8.7 ms | 16.0 | 4 | 56,302 | 29.8 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 225.2 | 232.9 | 923 ms | 4.3 ms | 1.0 | 1 | 6,756 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 401.4 | 205.9 | 1,010 ms | 4.9 ms | 2.0 | 2 | 12,041 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 676.7 | 175.6 | 1,213 ms | 5.7 ms | 3.9 | 4 | 20,301 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,057.3 | 134.4 | 1,547 ms | 7.4 ms | 7.7 | 8 | 31,710 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.