Speed test matrix
Qwen3.8-27B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.8-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.885500 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":3}'Prefill
prompt tok/s by context length8K context10.2K807 ms TTFT
16K context9,8921,644 ms TTFT
32K context9,4533,421 ms TTFT
64K context8,6817,426 ms TTFT
128K context7,34117,538 ms TTFT
251K context5,64444,694 ms TTFT
256K context5,62345,754 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 99.8141 ms TTFT | 191.1625 ms TTFT | 385.9495 ms TTFT | 723.2930 ms TTFT | 1,217.31,739 ms TTFT | 1,891.02,939 ms TTFT | 2,381.24,625 ms TTFT | Over capacity |
| 16K | 104.2183 ms TTFT | 205.9308 ms TTFT | 370.9550 ms TTFT | 680.9916 ms TTFT | 1,139.61,688 ms TTFT | 1,671.82,705 ms TTFT | Over capacity | Over capacity |
| 32K | 97.6225 ms TTFT | 193.0384 ms TTFT | 372.6635 ms TTFT | 628.61,239 ms TTFT | 1,010.21,945 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 93.5323 ms TTFT | 176.7480 ms TTFT | 335.0827 ms TTFT | 556.91,420 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 94.9534 ms TTFT | 162.8716 ms TTFT | 282.41,313 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 83.8808 ms TTFT | 141.31,076 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.2K | — | 807 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,892.0 | — | 1,644 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,453.0 | — | 3,421 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,681.0 | — | 7,426 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,341.0 | — | 17,538 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,644.0 | — | 44,694 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,623.0 | — | 45,754 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 99.8 | 99.7 | 141 ms | 10.0 ms | 1.0 | 0 | 2,992 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 191.1 | 96.6 | 625 ms | 10.4 ms | 2.0 | 0 | 5,731 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 385.9 | 96.6 | 495 ms | 10.4 ms | 4.0 | 0 | 11,570 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 723.2 | 90.7 | 930 ms | 11.0 ms | 8.0 | 0 | 21,694 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,217.3 | 76.7 | 1,739 ms | 13.0 ms | 16.0 | 0 | 36,497 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,891.0 | 59.2 | 2,939 ms | 16.9 ms | 32.0 | 0 | 56,671 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,381.2 | 36.1 | 4,625 ms | 27.7 ms | 64.0 | 0 | 71,424 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 104.2 | 104.1 | 183 ms | 9.6 ms | 1.0 | 0 | 3,125 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 205.9 | 102.9 | 308 ms | 9.7 ms | 2.0 | 0 | 6,176 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 370.9 | 92.6 | 550 ms | 10.8 ms | 4.0 | 0 | 11,125 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 680.9 | 86.0 | 916 ms | 11.6 ms | 8.0 | 0 | 20,424 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,139.6 | 71.3 | 1,688 ms | 14.0 ms | 16.0 | 0 | 34,185 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,671.8 | 51.8 | 2,705 ms | 19.3 ms | 32.0 | 0 | 50,157 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 97.6 | 97.9 | 225 ms | 10.2 ms | 1.0 | 0 | 2,928 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 193.0 | 97.4 | 384 ms | 10.3 ms | 2.0 | 0 | 5,787 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 372.6 | 91.6 | 635 ms | 10.9 ms | 4.0 | 0 | 11,171 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 628.6 | 79.4 | 1,239 ms | 12.6 ms | 8.0 | 0 | 18,853 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,010.2 | 63.3 | 1,945 ms | 15.8 ms | 16.0 | 0 | 30,298 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 93.5 | 94.3 | 323 ms | 10.6 ms | 1.0 | 0 | 2,805 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 176.7 | 89.6 | 480 ms | 11.2 ms | 2.0 | 0 | 5,298 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 335.0 | 83.2 | 827 ms | 12.0 ms | 4.0 | 0 | 10,050 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 556.9 | 70.7 | 1,420 ms | 14.1 ms | 8.0 | 0 | 16,701 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 94.9 | 95.0 | 534 ms | 10.5 ms | 1.0 | 0 | 2,848 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 162.8 | 81.9 | 716 ms | 12.2 ms | 2.0 | 0 | 4,882 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 282.4 | 70.7 | 1,313 ms | 14.2 ms | 4.0 | 0 | 8,472 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 83.8 | 83.7 | 808 ms | 12.0 ms | 1.0 | 0 | 2,515 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 141.3 | 71.8 | 1,076 ms | 13.9 ms | 2.0 | 0 | 4,234 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.