Speed test matrix
Qwen3.8-Flash-Next-FP8 on H200 NVL 141GB ×2
vllm · 0.1.1.dev5+ge77daef89 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLE_CPU_OFFLOAD=1 "$HOME/env_h200-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next-FP8 --served-model-name qwen --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.858381 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 256 --enable-prefix-caching --no-enable-flashinfer-autotune --enable-expert-parallel --moe-backend triton
Prefill
prompt tok/s by context length8K context19.6K420 ms TTFT
16K context21K773 ms TTFT
32K context21.3K1,516 ms TTFT
64K context21.4K3,007 ms TTFT
128K context20.2K6,384 ms TTFT
251K context18.1K13,908 ms TTFT
256K context17.9K14,384 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 111.9456 ms TTFT | 209.8165 ms TTFT | 390.2270 ms TTFT | 705.9339 ms TTFT | 944.5611 ms TTFT | 1,554.4961 ms TTFT | 2,767.31,504 ms TTFT | 4,103.22,363 ms TTFT |
| 16K | 111.7476 ms TTFT | 209.3234 ms TTFT | 388.8354 ms TTFT | 707.5481 ms TTFT | 925.4834 ms TTFT | 1,550.01,221 ms TTFT | 2,646.41,816 ms TTFT | 4,197.83,285 ms TTFT |
| 32K | 111.2482 ms TTFT | 211.0544 ms TTFT | 388.3321 ms TTFT | 710.5551 ms TTFT | 922.1993 ms TTFT | 1,554.71,608 ms TTFT | 2,595.22,443 ms TTFT | 4,077.24,565 ms TTFT |
| 64K | 112.6502 ms TTFT | 208.4561 ms TTFT | 397.4332 ms TTFT | 705.1467 ms TTFT | 902.0800 ms TTFT | 1,546.41,392 ms TTFT | 2,918.12,794 ms TTFT | 4,229.45,317 ms TTFT |
| 128K | 110.4608 ms TTFT | 206.9692 ms TTFT | 380.8641 ms TTFT | 697.81,116 ms TTFT | 938.21,857 ms TTFT | 1,524.13,427 ms TTFT | 2,677.56,794 ms TTFT | 3,948.212,362 ms TTFT |
| 251K | 108.91,084 ms TTFT | 207.61,208 ms TTFT | 386.61,541 ms TTFT | 694.42,443 ms TTFT | 882.53,837 ms TTFT | 1,444.37,170 ms TTFT | 2,455.414,780 ms TTFT | 3,762.131,628 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19.6K | — | 420 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21K | — | 773 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 21.3K | — | 1,516 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.4K | — | 3,007 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20.2K | — | 6,384 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 18.1K | — | 13,908 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.9K | — | 14,384 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 111.9 | 112.0 | 456 ms | 8.9 ms | 1.0 | 0 | 3,359 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 209.8 | 104.8 | 165 ms | 9.5 ms | 2.0 | 0 | 6,294 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 390.2 | 97.6 | 270 ms | 10.2 ms | 4.0 | 0 | 11,704 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 705.9 | 88.2 | 339 ms | 11.3 ms | 8.0 | 0 | 21,185 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 944.5 | 58.9 | 611 ms | 17.0 ms | 16.0 | 0 | 28,336 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,554.4 | 49.0 | 961 ms | 20.4 ms | 32.0 | 0 | 46,624 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,767.3 | 43.8 | 1,504 ms | 22.9 ms | 64.0 | 0 | 83,008 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,103.2 | 31.7 | 2,363 ms | 31.6 ms | 128.0 | 0 | 123,136 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 111.7 | 111.7 | 476 ms | 8.9 ms | 1.0 | 0 | 3,350 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 209.3 | 104.7 | 234 ms | 9.6 ms | 2.0 | 0 | 6,280 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 388.8 | 97.5 | 354 ms | 10.3 ms | 4.0 | 0 | 11,665 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 707.5 | 88.4 | 481 ms | 11.3 ms | 8.0 | 0 | 21,224 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 925.4 | 58.1 | 834 ms | 17.2 ms | 16.0 | 0 | 27,760 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,550.0 | 48.6 | 1,221 ms | 20.6 ms | 32.0 | 0 | 46,496 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,646.4 | 41.2 | 1,816 ms | 24.3 ms | 64.0 | 0 | 79,360 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,197.8 | 31.8 | 3,285 ms | 31.5 ms | 128.0 | 0 | 125,952 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 111.2 | 111.3 | 482 ms | 9.0 ms | 1.0 | 0 | 3,335 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 211.0 | 105.6 | 544 ms | 9.5 ms | 2.0 | 0 | 6,330 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 388.3 | 97.1 | 321 ms | 10.3 ms | 4.0 | 0 | 11,647 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 710.5 | 88.5 | 551 ms | 11.3 ms | 8.0 | 0 | 21,312 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 922.1 | 57.4 | 993 ms | 17.4 ms | 16.0 | 0 | 27,664 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,554.7 | 48.4 | 1,608 ms | 20.7 ms | 32.0 | 0 | 46,624 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,595.2 | 39.1 | 2,443 ms | 25.6 ms | 64.0 | 0 | 77,824 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,077.2 | 29.2 | 4,565 ms | 34.2 ms | 128.0 | 0 | 122,367 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 112.6 | 113.1 | 502 ms | 8.8 ms | 1.0 | 0 | 3,378 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 208.4 | 104.3 | 561 ms | 9.6 ms | 2.0 | 0 | 6,252 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 397.4 | 99.2 | 332 ms | 10.1 ms | 4.0 | 0 | 11,920 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 705.1 | 87.9 | 467 ms | 11.4 ms | 8.0 | 0 | 21,152 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 902.0 | 56.1 | 800 ms | 17.8 ms | 16.0 | 0 | 27,056 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,546.4 | 48.1 | 1,392 ms | 20.8 ms | 32.0 | 0 | 46,368 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,918.1 | 43.8 | 2,794 ms | 22.8 ms | 64.0 | 0 | 87,552 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,229.4 | 30.5 | 5,317 ms | 32.8 ms | 128.0 | 0 | 126,848 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 110.4 | 111.5 | 608 ms | 9.0 ms | 1.0 | 0 | 3,315 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 206.9 | 104.5 | 692 ms | 9.6 ms | 2.0 | 0 | 6,206 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 380.8 | 95.2 | 641 ms | 10.5 ms | 4.0 | 0 | 11,425 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 697.8 | 86.7 | 1,116 ms | 11.5 ms | 8.0 | 0 | 20,928 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 938.2 | 57.8 | 1,857 ms | 17.3 ms | 16.0 | 0 | 28,144 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,524.1 | 46.9 | 3,427 ms | 21.3 ms | 32.0 | 0 | 45,728 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,677.5 | 40.4 | 6,794 ms | 24.8 ms | 64.0 | 0 | 80,320 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 3,948.2 | 29.2 | 12,362 ms | 34.2 ms | 128.0 | 0 | 118,400 | 30.0 s |
| Sustained decode | 251K | 1 | Valid | 108.9 | 109.3 | 1,084 ms | 9.2 ms | 1.0 | 0 | 3,267 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 207.6 | 103.8 | 1,208 ms | 9.6 ms | 2.0 | 0 | 6,226 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 386.6 | 95.9 | 1,541 ms | 10.4 ms | 4.0 | 0 | 11,596 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 694.4 | 85.5 | 2,443 ms | 11.7 ms | 8.0 | 0 | 20,832 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 882.5 | 53.4 | 3,837 ms | 18.7 ms | 16.0 | 0 | 26,464 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,444.3 | 41.4 | 7,170 ms | 24.2 ms | 32.0 | 0 | 43,328 | 30.0 s |
| Sustained decode | 251K | 64 | Valid | 2,455.4 | 30.9 | 14,780 ms | 32.3 ms | 64.0 | 0 | 73,665 | 30.0 s |
| Sustained decode | 251K | 128 | Valid | 3,762.1 | 21.0 | 31,628 ms | 47.6 ms | 128.0 | 0 | 112,896 | 30.0 s |