Speed test matrix
Qwen3.8-Flash-Next on H200 SXM 141GB ×4
vllm · 0.1.1.dev5+ge77daef89 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --enable-expert-parallel --engram-config '{"cpu_offload":false}' --speculative-config '{"method":"mtp","num_speculative_tokens":1}'Prefill
prompt tok/s by context length8K context24.9K329 ms TTFT
16K context29.9K540 ms TTFT
32K context33.5K960 ms TTFT
64K context34.8K1,841 ms TTFT
128K context34.8K3,679 ms TTFT
251K context32.7K7,679 ms TTFT
256K context32.1K7,969 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 224.5158 ms TTFT | 399.2173 ms TTFT | 689.1833 ms TTFT | 1,140.4294 ms TTFT | 1,946.4423 ms TTFT | 3,250.02,420 ms TTFT | 5,273.71,291 ms TTFT | 7,931.52,188 ms TTFT |
| 16K | 228.4169 ms TTFT | 405.8194 ms TTFT | 692.1311 ms TTFT | 1,142.7341 ms TTFT | 1,935.0467 ms TTFT | 3,252.7947 ms TTFT | 5,276.91,425 ms TTFT | Over capacity |
| 32K | Looped | 399.6224 ms TTFT | 691.5224 ms TTFT | 1,136.0361 ms TTFT | 1,936.6845 ms TTFT | 3,221.91,326 ms TTFT | Over capacity | Over capacity |
| 64K | 226.6282 ms TTFT | 395.7306 ms TTFT | 689.5395 ms TTFT | 1,136.7506 ms TTFT | 1,914.41,237 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 217.3451 ms TTFT | 400.7544 ms TTFT | 688.2527 ms TTFT | 1,119.31,080 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 219.7907 ms TTFT | 405.11,030 ms TTFT | 678.3966 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 24.9K | — | 329 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.9K | — | 540 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33.5K | — | 960 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 34.8K | — | 1,841 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 34.8K | — | 3,679 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 32.7K | — | 7,679 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 32.1K | — | 7,969 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 224.5 | 226.6 | 158 ms | 4.4 ms | 1.0 | 1 | 6,734 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 399.2 | 198.9 | 173 ms | 5.0 ms | 2.0 | 2 | 11,976 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 689.1 | 176.4 | 833 ms | 5.7 ms | 4.0 | 4 | 20,674 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,140.4 | 131.2 | 294 ms | 7.6 ms | 8.0 | 8 | 34,208 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,946.4 | 117.0 | 423 ms | 8.6 ms | 16.0 | 3 | 58,379 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,250.0 | 100.3 | 2,420 ms | 10.0 ms | 32.0 | 0 | 97,498 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,273.7 | 74.3 | 1,291 ms | 13.5 ms | 64.0 | 0 | 158,174 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,931.5 | 60.0 | 2,188 ms | 16.7 ms | 128.0 | 0 | 238,085 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 228.4 | 229.1 | 169 ms | 4.4 ms | 1.0 | 1 | 6,850 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 405.8 | 203.0 | 194 ms | 4.9 ms | 2.0 | 2 | 12,173 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 692.1 | 175.2 | 311 ms | 5.7 ms | 4.0 | 4 | 20,763 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,142.7 | 142.0 | 341 ms | 7.0 ms | 7.9 | 8 | 34,279 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,935.0 | 120.7 | 467 ms | 8.3 ms | 16.0 | 6 | 58,034 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,252.7 | 101.2 | 947 ms | 9.9 ms | 32.0 | 0 | 97,540 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,276.9 | 81.2 | 1,425 ms | 12.3 ms | 64.0 | 0 | 158,263 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Valid | 399.6 | 199.9 | 224 ms | 5.0 ms | 2.0 | 2 | 11,986 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 691.5 | 173.0 | 224 ms | 5.8 ms | 4.0 | 4 | 20,745 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,136.0 | 141.5 | 361 ms | 7.1 ms | 8.0 | 8 | 34,070 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,936.6 | 119.8 | 845 ms | 8.3 ms | 16.0 | 4 | 57,858 | 29.9 s |
| Sustained decode | 32K | 32 | Valid | 3,221.9 | 99.2 | 1,326 ms | 10.1 ms | 32.0 | 0 | 96,621 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 226.6 | 229.4 | 282 ms | 4.4 ms | 1.0 | 1 | 6,797 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 395.7 | 200.5 | 306 ms | 5.0 ms | 2.0 | 2 | 11,870 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 689.5 | 173.1 | 395 ms | 5.8 ms | 4.0 | 4 | 20,682 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,136.7 | 142.0 | 506 ms | 7.0 ms | 7.9 | 8 | 34,094 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,914.4 | 117.8 | 1,237 ms | 8.5 ms | 16.0 | 4 | 57,224 | 29.9 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 217.3 | 221.1 | 451 ms | 4.5 ms | 1.0 | 1 | 6,518 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 400.7 | 204.4 | 544 ms | 4.9 ms | 1.9 | 2 | 12,020 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 688.2 | 174.5 | 527 ms | 5.7 ms | 3.9 | 4 | 20,644 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,119.3 | 141.6 | 1,080 ms | 7.1 ms | 7.9 | 8 | 33,570 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 219.7 | 227.1 | 907 ms | 4.4 ms | 1.0 | 1 | 6,589 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 405.1 | 208.9 | 1,030 ms | 4.8 ms | 2.0 | 2 | 12,150 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 678.3 | 175.9 | 966 ms | 5.7 ms | 3.9 | 4 | 20,346 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.