Speed test matrix
Qwen3.8-27B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.8-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883421 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context10.4K795 ms TTFT
16K context10.2K1,599 ms TTFT
32K context9,6683,345 ms TTFT
64K context8,8457,288 ms TTFT
128K context7,48017,212 ms TTFT
251K context5,66544,528 ms TTFT
256K context5,63245,674 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 59.771 ms TTFT | 118.1134 ms TTFT | 229.31,818 ms TTFT | 432.7361 ms TTFT | 789.5619 ms TTFT | 1,369.11,353 ms TTFT | 2,063.92,620 ms TTFT | Over capacity |
| 16K | 59.3135 ms TTFT | 116.6192 ms TTFT | 224.4348 ms TTFT | 414.9517 ms TTFT | 734.01,025 ms TTFT | 1,201.81,887 ms TTFT | Over capacity | Over capacity |
| 32K | 58.4119 ms TTFT | 113.6184 ms TTFT | 214.4258 ms TTFT | 384.0333 ms TTFT | 641.5450 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 56.7162 ms TTFT | 108.1254 ms TTFT | 197.5325 ms TTFT | 332.5483 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 53.7277 ms TTFT | 98.5404 ms TTFT | 169.2565 ms TTFT | 256.5878 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 48.7683 ms TTFT | 83.2869 ms TTFT | 129.11,263 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.4K | — | 795 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.2K | — | 1,599 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,668.0 | — | 3,345 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,845.0 | — | 7,288 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,480.0 | — | 17,212 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,665.0 | — | 44,528 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,632.0 | — | 45,674 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 59.7 | 59.8 | 71 ms | 16.7 ms | 1.0 | 0 | 1,792 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 118.1 | 59.1 | 134 ms | 16.9 ms | 2.0 | 0 | 3,542 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 229.3 | 57.4 | 1,818 ms | 17.4 ms | 4.0 | 0 | 6,880 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 432.7 | 54.1 | 361 ms | 18.5 ms | 8.0 | 0 | 12,984 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 789.5 | 49.3 | 619 ms | 20.3 ms | 16.0 | 0 | 23,696 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,369.1 | 43.0 | 1,353 ms | 23.3 ms | 32.0 | 0 | 41,056 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,063.9 | 32.4 | 2,620 ms | 30.9 ms | 64.0 | 0 | 61,888 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 59.3 | 59.3 | 135 ms | 16.9 ms | 1.0 | 0 | 1,778 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 116.6 | 58.3 | 192 ms | 17.1 ms | 2.0 | 0 | 3,498 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 224.4 | 56.2 | 348 ms | 17.8 ms | 4.0 | 0 | 6,732 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 414.9 | 51.8 | 517 ms | 19.3 ms | 8.0 | 0 | 12,440 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 734.0 | 45.9 | 1,025 ms | 21.8 ms | 16.0 | 0 | 22,016 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,201.8 | 37.5 | 1,887 ms | 26.7 ms | 32.0 | 0 | 36,032 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 58.4 | 58.5 | 119 ms | 17.1 ms | 1.0 | 0 | 1,753 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 113.6 | 56.9 | 184 ms | 17.6 ms | 2.0 | 0 | 3,408 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 214.4 | 53.6 | 258 ms | 18.6 ms | 4.0 | 0 | 6,432 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 384.0 | 48.0 | 333 ms | 20.8 ms | 8.0 | 0 | 11,520 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 641.5 | 40.0 | 450 ms | 25.0 ms | 16.0 | 0 | 19,232 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 56.7 | 56.8 | 162 ms | 17.6 ms | 1.0 | 0 | 1,702 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 108.1 | 54.1 | 254 ms | 18.5 ms | 2.0 | 0 | 3,242 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 197.5 | 49.4 | 325 ms | 20.2 ms | 4.0 | 0 | 5,924 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 332.5 | 41.5 | 483 ms | 24.1 ms | 8.0 | 0 | 9,976 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 53.7 | 53.8 | 277 ms | 18.6 ms | 1.0 | 0 | 1,611 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 98.5 | 49.3 | 404 ms | 20.3 ms | 2.0 | 0 | 2,956 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 169.2 | 42.4 | 565 ms | 23.6 ms | 4.0 | 0 | 5,076 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 256.5 | 32.1 | 878 ms | 31.2 ms | 8.0 | 0 | 7,696 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 48.7 | 48.8 | 683 ms | 20.5 ms | 1.0 | 0 | 1,461 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 83.2 | 41.7 | 869 ms | 24.0 ms | 2.0 | 0 | 2,496 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 129.1 | 32.2 | 1,263 ms | 31.0 ms | 4.0 | 0 | 3,872 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.