Speed test matrix
Qwen3.8-Flash-Next on H200 SXM 141GB ×4
vllm · 0.1.1.dev5+ge77daef89 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.87 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --enable-expert-parallel --engram-config '{"cpu_offload":true}'Prefill
prompt tok/s by context length8K context33K248 ms TTFT
16K context36.6K442 ms TTFT
32K context33K975 ms TTFT
64K context34.6K1,854 ms TTFT
128K context36.2K3,540 ms TTFT
251K context34.4K7,295 ms TTFT
256K context33.5K7,632 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 163.765 ms TTFT | 297.087 ms TTFT | 519.5220 ms TTFT | 911.2242 ms TTFT | 1,507.2489 ms TTFT | 2,479.6559 ms TTFT | 4,057.8901 ms TTFT | 6,547.21,478 ms TTFT |
| 16K | 163.083 ms TTFT | 295.9115 ms TTFT | 516.5243 ms TTFT | 911.5337 ms TTFT | 1,509.7458 ms TTFT | 2,454.6762 ms TTFT | 4,052.21,246 ms TTFT | 6,506.02,185 ms TTFT |
| 32K | 162.3118 ms TTFT | 294.2141 ms TTFT | 514.5199 ms TTFT | 895.6323 ms TTFT | 1,498.5564 ms TTFT | 2,449.41,028 ms TTFT | 3,999.51,926 ms TTFT | Over capacity |
| 64K | 160.7283 ms TTFT | 289.7380 ms TTFT | 506.6501 ms TTFT | 898.6820 ms TTFT | 1,477.01,210 ms TTFT | 2,415.72,163 ms TTFT | 3,940.33,955 ms TTFT | Over capacity |
| 128K | 159.6373 ms TTFT | 288.3450 ms TTFT | 503.5449 ms TTFT | 880.11,224 ms TTFT | 1,422.52,192 ms TTFT | 2,355.54,170 ms TTFT | Over capacity | Over capacity |
| 251K | 156.7803 ms TTFT | 283.3912 ms TTFT | 488.7865 ms TTFT | 872.92,705 ms TTFT | 1,414.04,877 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33K | — | 248 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 36.6K | — | 442 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33K | — | 975 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 34.6K | — | 1,854 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 36.2K | — | 3,540 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 34.4K | — | 7,295 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 33.5K | — | 7,632 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 163.7 | 164.1 | 65 ms | 6.1 ms | 1.0 | 1 | 4,911 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 297.0 | 148.7 | 87 ms | 6.7 ms | 2.0 | 2 | 8,910 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 519.5 | 130.5 | 220 ms | 7.7 ms | 4.0 | 4 | 15,583 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 911.2 | 113.3 | 242 ms | 8.8 ms | 8.0 | 0 | 27,336 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,507.2 | 94.3 | 489 ms | 10.6 ms | 16.0 | 0 | 45,216 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,479.6 | 77.2 | 559 ms | 12.9 ms | 32.0 | 0 | 74,368 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,057.8 | 63.1 | 901 ms | 15.9 ms | 64.0 | 0 | 121,728 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,547.2 | 50.7 | 1,478 ms | 19.7 ms | 128.0 | 0 | 196,352 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 163.0 | 163.5 | 83 ms | 6.1 ms | 1.0 | 1 | 4,890 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 295.9 | 148.3 | 115 ms | 6.7 ms | 1.9 | 2 | 8,876 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 516.5 | 130.1 | 243 ms | 7.7 ms | 3.9 | 4 | 15,494 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 911.5 | 113.4 | 337 ms | 8.8 ms | 8.0 | 0 | 27,344 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,509.7 | 94.2 | 458 ms | 10.6 ms | 16.0 | 0 | 45,280 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,454.6 | 76.6 | 762 ms | 13.1 ms | 32.0 | 0 | 73,631 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,052.2 | 62.4 | 1,246 ms | 16.0 ms | 64.0 | 0 | 121,536 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 6,506.0 | 49.4 | 2,185 ms | 20.3 ms | 128.0 | 0 | 195,200 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 162.3 | 163.1 | 118 ms | 6.1 ms | 1.0 | 1 | 4,870 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 294.2 | 147.6 | 141 ms | 6.8 ms | 2.0 | 2 | 8,826 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 514.5 | 129.4 | 199 ms | 7.7 ms | 3.9 | 4 | 15,434 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 895.6 | 111.6 | 323 ms | 9.0 ms | 8.0 | 0 | 26,864 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,498.5 | 93.0 | 564 ms | 10.8 ms | 16.0 | 0 | 44,944 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,449.4 | 75.4 | 1,028 ms | 13.3 ms | 32.0 | 0 | 73,472 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,999.5 | 61.1 | 1,926 ms | 16.4 ms | 64.0 | 0 | 120,000 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 160.7 | 162.4 | 283 ms | 6.2 ms | 1.0 | 1 | 4,822 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 289.7 | 146.6 | 380 ms | 6.8 ms | 2.0 | 2 | 8,690 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 506.6 | 128.2 | 501 ms | 7.8 ms | 4.0 | 4 | 15,196 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 898.6 | 111.4 | 820 ms | 9.0 ms | 8.0 | 0 | 26,952 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,477.0 | 90.7 | 1,210 ms | 11.0 ms | 16.0 | 0 | 44,304 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,415.7 | 72.7 | 2,163 ms | 13.8 ms | 32.0 | 0 | 72,448 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,940.3 | 56.9 | 3,955 ms | 17.6 ms | 64.0 | 0 | 118,208 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 159.6 | 161.9 | 373 ms | 6.2 ms | 1.0 | 1 | 4,788 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 288.3 | 146.3 | 450 ms | 6.8 ms | 2.0 | 2 | 8,648 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 503.5 | 127.6 | 449 ms | 7.8 ms | 4.0 | 4 | 15,103 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 880.1 | 109.4 | 1,224 ms | 9.1 ms | 8.0 | 0 | 26,400 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,422.5 | 87.9 | 2,192 ms | 11.4 ms | 16.0 | 0 | 42,672 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,355.5 | 72.0 | 4,170 ms | 13.9 ms | 32.0 | 0 | 70,656 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 156.7 | 161.5 | 803 ms | 6.2 ms | 1.0 | 1 | 4,701 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 283.3 | 146.1 | 912 ms | 6.8 ms | 2.0 | 2 | 8,498 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 488.7 | 125.6 | 865 ms | 8.0 ms | 3.9 | 4 | 14,659 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 872.9 | 108.3 | 2,705 ms | 9.2 ms | 8.0 | 0 | 26,184 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,414.0 | 87.2 | 4,877 ms | 11.5 ms | 16.0 | 0 | 42,416 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.