Speed test matrix
Qwen3.8-27B on H200 NVL 141GB
vllm · 0.26.1rc1.dev914+gc3dabcddc · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-vllm/bin/vllm" serve Qwen/Qwen3.8-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.908214 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context10.1K818 ms TTFT
16K context9,9001,643 ms TTFT
32K context9,5313,393 ms TTFT
64K context8,7507,368 ms TTFT
128K context7,38717,428 ms TTFT
251K context5,66644,521 ms TTFT
256K context5,62745,717 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 102.6796 ms TTFT | 216.0982 ms TTFT | 423.9672 ms TTFT | 767.11,307 ms TTFT | 1,112.52,260 ms TTFT | 1,521.14,801 ms TTFT | 1,367.3Requests queued5,772 ms TTFT | 1,229.5Requests queued5,819 ms TTFT |
| 16K | 109.41,612 ms TTFT | 213.81,732 ms TTFT | 411.0630 ms TTFT | 698.11,097 ms TTFT | 1,085.72,253 ms TTFT | 1,408.44,366 ms TTFT | 1,258.2Requests queued5,119 ms TTFT | 1,233.1Requests queued4,970 ms TTFT |
| 32K | 110.23,361 ms TTFT | 208.43,534 ms TTFT | 397.2830 ms TTFT | 682.71,594 ms TTFT | 1,003.22,995 ms TTFT | 1,264.25,753 ms TTFT | 1,051.8Requests queued7,377 ms TTFT | Over capacity |
| 64K | 103.57,356 ms TTFT | 198.27,517 ms TTFT | 367.3823 ms TTFT | 603.21,415 ms TTFT | 845.32,587 ms TTFT | 1,031.34,929 ms TTFT | Over capacity | Over capacity |
| 128K | 99.917,364 ms TTFT | 184.117,649 ms TTFT | 316.11,338 ms TTFT | 495.82,252 ms TTFT | 632.24,184 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 96.744,463 ms TTFT | 153.344,678 ms TTFT | 258.61,453 ms TTFT | 372.32,283 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.1K | — | 818 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,900.0 | — | 1,643 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,531.0 | — | 3,393 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,750.0 | — | 7,368 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,387.0 | — | 17,428 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,666.0 | — | 44,521 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,627.0 | — | 45,717 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 102.6 | 105.8 | 796 ms | 9.5 ms | 1.0 | 0 | 3,076 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 216.0 | 111.0 | 982 ms | 9.0 ms | 2.0 | 0 | 6,480 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 423.9 | 106.8 | 672 ms | 9.4 ms | 4.0 | 0 | 12,717 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 767.1 | 96.8 | 1,307 ms | 10.3 ms | 8.0 | 0 | 23,013 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,112.5 | 71.4 | 2,260 ms | 14.0 ms | 16.0 | 0 | 33,358 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,521.1 | 48.9 | 4,801 ms | 20.4 ms | 32.0 | 0 | 45,588 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,367.3 | 34.5 | 5,772 ms | 29.0 ms | 37.4 | 0 | 40,940 | 29.9 s |
| Sustained decode | 8K | 128 | Requests queued | 1,229.5 | 33.8 | 5,819 ms | 29.6 ms | 37.1 | 0 | 36,878 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 109.4 | 112.1 | 1,612 ms | 8.9 ms | 1.0 | 0 | 3,281 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 213.8 | 108.7 | 1,732 ms | 9.2 ms | 2.0 | 0 | 6,415 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 411.0 | 104.3 | 630 ms | 9.6 ms | 4.0 | 0 | 12,321 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 698.1 | 89.1 | 1,097 ms | 11.2 ms | 8.0 | 0 | 20,937 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,085.7 | 69.5 | 2,253 ms | 14.4 ms | 16.0 | 0 | 32,573 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,408.4 | 45.1 | 4,366 ms | 22.2 ms | 32.0 | 0 | 42,190 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,258.2 | 33.3 | 5,119 ms | 30.0 ms | 37.5 | 0 | 37,697 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,233.1 | 33.5 | 4,970 ms | 29.8 ms | 37.2 | 0 | 36,925 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 110.2 | 113.6 | 3,361 ms | 8.8 ms | 1.0 | 0 | 3,306 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 208.4 | 107.6 | 3,534 ms | 9.3 ms | 2.0 | 0 | 6,251 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 397.2 | 101.3 | 830 ms | 9.9 ms | 4.0 | 0 | 11,913 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 682.7 | 87.6 | 1,594 ms | 11.4 ms | 8.0 | 0 | 20,488 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,003.2 | 64.6 | 2,995 ms | 15.5 ms | 16.0 | 0 | 30,091 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,264.2 | 40.2 | 5,753 ms | 24.9 ms | 32.0 | 0 | 37,912 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,051.8 | 28.3 | 7,377 ms | 35.3 ms | 36.4 | 0 | 31,512 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 103.5 | 104.4 | 7,356 ms | 9.6 ms | 1.0 | 0 | 3,105 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 198.2 | 101.0 | 7,517 ms | 9.9 ms | 2.0 | 0 | 5,943 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 367.3 | 90.7 | 823 ms | 11.0 ms | 4.0 | 0 | 11,019 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 603.2 | 78.7 | 1,415 ms | 12.7 ms | 8.0 | 0 | 18,090 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 845.3 | 53.9 | 2,587 ms | 18.6 ms | 16.0 | 0 | 25,356 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,031.3 | 33.5 | 4,929 ms | 29.8 ms | 32.0 | 0 | 30,906 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 99.9 | 103.3 | 17,364 ms | 9.7 ms | 1.0 | 0 | 2,995 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 184.1 | 95.0 | 17,649 ms | 10.5 ms | 2.0 | 0 | 5,520 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 316.1 | 81.7 | 1,338 ms | 12.2 ms | 4.0 | 0 | 9,481 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 495.8 | 61.5 | 2,252 ms | 16.3 ms | 8.0 | 0 | 14,863 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 632.2 | 39.5 | 4,184 ms | 25.3 ms | 16.0 | 0 | 18,965 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 96.7 | 97.9 | 44,463 ms | 10.2 ms | 1.0 | 0 | 2,899 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 153.3 | 79.7 | 44,678 ms | 12.5 ms | 2.0 | 0 | 4,596 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 258.6 | 66.1 | 1,453 ms | 15.1 ms | 4.0 | 0 | 7,756 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 372.3 | 47.0 | 2,283 ms | 21.3 ms | 8.0 | 0 | 11,161 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.