Speed test matrix
Qwen3.8-Flash-Next-FP8 on H200 NVL 141GB ×2
vllm · 0.1.1.dev5+ge77daef89 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLE_CPU_OFFLOAD=1 "$HOME/env_h200-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next-FP8 --served-model-name qwen --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.846472 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 256 --enable-prefix-caching --no-enable-flashinfer-autotune --enable-expert-parallel --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":3}'Prefill
prompt tok/s by context length8K context18.6K443 ms TTFT
16K context20K813 ms TTFT
32K context20.7K1,559 ms TTFT
64K context20.4K3,157 ms TTFT
128K context19.4K6,648 ms TTFT
251K context17.3K14,579 ms TTFT
256K context17K15,163 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 148.4326 ms TTFT | 291.8217 ms TTFT | 445.5506 ms TTFT | 767.4821 ms TTFT | 1,390.01,279 ms TTFT | 2,129.62,355 ms TTFT | 3,005.14,182 ms TTFT | 2,927.1Requests queued7,790 ms TTFT |
| 16K | 150.1340 ms TTFT | 291.4395 ms TTFT | 428.0548 ms TTFT | 819.7879 ms TTFT | 1,397.91,377 ms TTFT | 2,205.52,414 ms TTFT | 2,993.74,519 ms TTFT | 2,789.6Requests queued8,126 ms TTFT |
| 32K | 145.8378 ms TTFT | 280.0437 ms TTFT | 443.3650 ms TTFT | 780.3733 ms TTFT | 1,442.31,525 ms TTFT | 2,156.12,668 ms TTFT | 2,843.74,868 ms TTFT | 3,324.4Requests queued8,588 ms TTFT |
| 64K | 151.6468 ms TTFT | 283.5561 ms TTFT | 420.4670 ms TTFT | 791.31,060 ms TTFT | 1,424.42,065 ms TTFT | 2,174.83,471 ms TTFT | 2,821.15,714 ms TTFT | 2,652.6Requests queued9,745 ms TTFT |
| 128K | 145.7576 ms TTFT | 298.9684 ms TTFT | 446.1714 ms TTFT | 773.31,529 ms TTFT | 1,448.52,542 ms TTFT | 2,130.94,498 ms TTFT | 2,850.47,974 ms TTFT | 1,345.8Requests queued13,344 ms TTFT |
| 251K | 145.6976 ms TTFT | 280.5981 ms TTFT | 441.11,586 ms TTFT | 829.82,251 ms TTFT | 1,373.83,746 ms TTFT | 2,000.37,131 ms TTFT | 2,675.013,340 ms TTFT | 692.9Requests queued24,499 ms TTFT |
Requests queued: the server could not run all requests at once, so some waited in a queue.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 18.6K | — | 443 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 20K | — | 813 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 20.7K | — | 1,559 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.4K | — | 3,157 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.4K | — | 6,648 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.3K | — | 14,579 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17K | — | 15,163 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 148.4 | 157.4 | 326 ms | 6.4 ms | 1.0 | 1 | 4,451 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 291.8 | 147.4 | 217 ms | 6.8 ms | 2.0 | 2 | 8,753 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 445.5 | 114.3 | 506 ms | 8.8 ms | 4.0 | 0 | 13,367 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 767.4 | 96.3 | 821 ms | 10.4 ms | 8.0 | 0 | 23,022 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,390.0 | 86.5 | 1,279 ms | 11.6 ms | 16.0 | 0 | 41,686 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,129.6 | 65.0 | 2,355 ms | 15.4 ms | 32.0 | 0 | 63,884 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,005.1 | 43.5 | 4,182 ms | 23.0 ms | 64.0 | 0 | 90,137 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 2,927.1 | 25.5 | 7,790 ms | 39.2 ms | 118.2 | 0 | 87,711 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 150.1 | 151.8 | 340 ms | 6.6 ms | 1.0 | 1 | 4,501 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 291.4 | 146.8 | 395 ms | 6.8 ms | 2.0 | 2 | 8,739 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 428.0 | 107.2 | 548 ms | 9.3 ms | 4.0 | 0 | 12,842 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 819.7 | 101.5 | 879 ms | 9.9 ms | 8.0 | 0 | 24,586 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,397.9 | 87.0 | 1,377 ms | 11.5 ms | 16.0 | 0 | 41,912 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,205.5 | 66.6 | 2,414 ms | 15.0 ms | 32.0 | 0 | 66,169 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,993.7 | 43.5 | 4,519 ms | 23.0 ms | 64.0 | 0 | 89,774 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 2,789.6 | 24.8 | 8,126 ms | 40.2 ms | 117.3 | 0 | 83,523 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 145.8 | 147.5 | 378 ms | 6.8 ms | 1.0 | 1 | 4,373 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 280.0 | 140.3 | 437 ms | 7.1 ms | 2.0 | 2 | 8,397 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 443.3 | 110.2 | 650 ms | 9.1 ms | 4.0 | 0 | 13,295 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 780.3 | 97.3 | 733 ms | 10.3 ms | 8.0 | 0 | 23,410 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,442.3 | 88.3 | 1,525 ms | 11.3 ms | 16.0 | 0 | 43,250 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,156.1 | 65.7 | 2,668 ms | 15.2 ms | 32.0 | 0 | 64,665 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,843.7 | 41.6 | 4,868 ms | 24.0 ms | 64.0 | 0 | 85,214 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 3,324.4 | 26.3 | 8,588 ms | 38.1 ms | 117.0 | 0 | 99,561 | 29.9 s |
| Sustained decode | 64K | 1 | Valid | 151.6 | 157.9 | 468 ms | 6.3 ms | 1.0 | 1 | 4,546 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 283.5 | 139.9 | 561 ms | 7.1 ms | 2.0 | 2 | 8,504 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 420.4 | 105.6 | 670 ms | 9.5 ms | 4.0 | 0 | 12,620 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 791.3 | 96.0 | 1,060 ms | 10.4 ms | 8.0 | 0 | 23,725 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,424.4 | 87.7 | 2,065 ms | 11.4 ms | 16.0 | 0 | 42,722 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,174.8 | 64.6 | 3,471 ms | 15.5 ms | 32.0 | 0 | 65,222 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,821.1 | 40.7 | 5,714 ms | 24.6 ms | 64.0 | 0 | 84,533 | 30.0 s |
| Sustained decode | 64K | 128 | Requests queued | 2,652.6 | 25.8 | 9,745 ms | 38.8 ms | 109.4 | 5 | 79,181 | 29.9 s |
| Sustained decode | 128K | 1 | Valid | 145.7 | 152.6 | 576 ms | 6.6 ms | 1.0 | 1 | 4,371 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 298.9 | 152.9 | 684 ms | 6.5 ms | 2.0 | 2 | 8,964 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 446.1 | 107.8 | 714 ms | 9.3 ms | 4.0 | 1 | 13,376 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 773.3 | 97.1 | 1,529 ms | 10.3 ms | 8.0 | 0 | 23,197 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,448.5 | 87.5 | 2,542 ms | 11.4 ms | 16.0 | 0 | 43,431 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,130.9 | 62.2 | 4,498 ms | 16.1 ms | 32.0 | 0 | 63,902 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,850.4 | 40.2 | 7,974 ms | 24.9 ms | 64.0 | 0 | 85,384 | 30.0 s |
| Sustained decode | 128K | 128 | Requests queued | 1,345.8 | 20.6 | 13,344 ms | 48.6 ms | 106.5 | 90 | 40,358 | 30.0 s |
| Sustained decode | 251K | 1 | Valid | 145.6 | 148.9 | 976 ms | 6.7 ms | 1.0 | 1 | 4,366 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 280.5 | 150.9 | 981 ms | 6.6 ms | 2.0 | 2 | 8,413 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 441.1 | 109.0 | 1,586 ms | 9.2 ms | 4.0 | 0 | 13,231 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 829.8 | 103.9 | 2,251 ms | 9.6 ms | 8.0 | 0 | 24,888 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,373.8 | 81.0 | 3,746 ms | 12.3 ms | 16.0 | 0 | 41,200 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 2,000.3 | 55.6 | 7,131 ms | 18.0 ms | 32.0 | 0 | 60,014 | 30.0 s |
| Sustained decode | 251K | 64 | Valid | 2,675.0 | 33.5 | 13,340 ms | 29.9 ms | 64.0 | 0 | 80,179 | 30.0 s |
| Sustained decode | 251K | 128 | Requests queued | 692.9 | 20.6 | 24,499 ms | 48.5 ms | 95.5 | 26 | 20,726 | 29.9 s |
Requests queued: the server could not run all requests at once, so some waited in a queue.