Speed test matrix
Qwen3.8-27B on H200 NVL 141GB
vllm · 0.26.1rc1.dev914+gc3dabcddc · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-incoai-vllm/bin/vllm" serve Qwen/Qwen3.8-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.908214 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context10K821 ms TTFT
16K context9,8771,647 ms TTFT
32K context9,4863,409 ms TTFT
64K context8,7307,384 ms TTFT
128K context7,38217,439 ms TTFT
251K context5,65744,592 ms TTFT
256K context5,61945,782 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 109.6798 ms TTFT | 234.1969 ms TTFT | 427.3669 ms TTFT | 740.31,307 ms TTFT | 1,174.32,259 ms TTFT | 1,499.24,798 ms TTFT | 1,226.2Requests queued5,913 ms TTFT | 1,381.1Requests queued5,881 ms TTFT |
| 16K | 112.11,616 ms TTFT | 217.81,741 ms TTFT | 410.3626 ms TTFT | 717.01,091 ms TTFT | 1,119.42,255 ms TTFT | 1,414.34,389 ms TTFT | 1,273.0Requests queued5,229 ms TTFT | 1,250.3Requests queued5,273 ms TTFT |
| 32K | 114.53,361 ms TTFT | 212.73,550 ms TTFT | 410.3779 ms TTFT | 653.41,590 ms TTFT | 1,010.22,989 ms TTFT | 1,223.95,764 ms TTFT | 1,077.7Requests queued7,063 ms TTFT | Over capacity |
| 64K | 102.47,335 ms TTFT | 229.17,554 ms TTFT | 377.8824 ms TTFT | 595.51,427 ms TTFT | 847.92,600 ms TTFT | 1,010.75,065 ms TTFT | Over capacity | Over capacity |
| 128K | 106.717,430 ms TTFT | 188.817,730 ms TTFT | 344.41,318 ms TTFT | 477.82,260 ms TTFT | 645.14,277 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 97.744,564 ms TTFT | 169.244,893 ms TTFT | 268.51,454 ms TTFT | 360.32,347 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10K | — | 821 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,877.0 | — | 1,647 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,486.0 | — | 3,409 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,730.0 | — | 7,384 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,382.0 | — | 17,439 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,657.0 | — | 44,592 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,619.0 | — | 45,782 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 109.6 | 111.8 | 798 ms | 8.9 ms | 1.0 | 0 | 3,287 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 234.1 | 118.8 | 969 ms | 8.4 ms | 2.0 | 0 | 7,021 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 427.3 | 109.1 | 669 ms | 9.2 ms | 4.0 | 0 | 12,816 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 740.3 | 94.9 | 1,307 ms | 10.5 ms | 8.0 | 0 | 22,191 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,174.3 | 74.2 | 2,259 ms | 13.5 ms | 16.0 | 0 | 35,202 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,499.2 | 47.7 | 4,798 ms | 21.0 ms | 32.0 | 0 | 44,955 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,226.2 | 33.2 | 5,913 ms | 30.2 ms | 37.1 | 0 | 36,744 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,381.1 | 35.7 | 5,881 ms | 28.0 ms | 37.2 | 0 | 41,355 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 112.1 | 116.6 | 1,616 ms | 8.6 ms | 1.0 | 0 | 3,362 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 217.8 | 110.7 | 1,741 ms | 9.0 ms | 2.0 | 0 | 6,534 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 410.3 | 101.2 | 626 ms | 9.9 ms | 4.0 | 0 | 12,303 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 717.0 | 91.3 | 1,091 ms | 11.0 ms | 8.0 | 0 | 21,501 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,119.4 | 70.8 | 2,255 ms | 14.1 ms | 16.0 | 0 | 33,563 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,414.3 | 45.1 | 4,389 ms | 22.2 ms | 32.0 | 0 | 42,431 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,273.0 | 33.7 | 5,229 ms | 29.7 ms | 37.2 | 0 | 38,165 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,250.3 | 33.8 | 5,273 ms | 29.6 ms | 37.2 | 0 | 37,483 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 114.5 | 116.5 | 3,361 ms | 8.6 ms | 1.0 | 0 | 3,434 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 212.7 | 108.7 | 3,550 ms | 9.2 ms | 2.0 | 0 | 6,381 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 410.3 | 105.2 | 779 ms | 9.5 ms | 4.0 | 0 | 12,304 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 653.4 | 83.2 | 1,590 ms | 12.0 ms | 8.0 | 0 | 19,587 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,010.2 | 64.9 | 2,989 ms | 15.4 ms | 16.0 | 0 | 30,304 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,223.9 | 39.3 | 5,764 ms | 25.4 ms | 32.0 | 0 | 36,670 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,077.7 | 28.7 | 7,063 ms | 34.9 ms | 36.5 | 0 | 32,291 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 102.4 | 104.2 | 7,335 ms | 9.6 ms | 1.0 | 0 | 3,073 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 229.1 | 116.2 | 7,554 ms | 8.6 ms | 2.0 | 0 | 6,874 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 377.8 | 96.4 | 824 ms | 10.4 ms | 4.0 | 0 | 11,337 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 595.5 | 76.5 | 1,427 ms | 13.1 ms | 8.0 | 0 | 17,857 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 847.9 | 53.8 | 2,600 ms | 18.6 ms | 16.0 | 0 | 25,429 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,010.7 | 32.5 | 5,065 ms | 30.8 ms | 32.0 | 0 | 30,249 | 29.9 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 106.7 | 108.9 | 17,430 ms | 9.2 ms | 1.0 | 0 | 3,201 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 188.8 | 97.4 | 17,730 ms | 10.3 ms | 2.0 | 0 | 5,663 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 344.4 | 86.1 | 1,318 ms | 11.6 ms | 4.0 | 0 | 10,330 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 477.8 | 60.6 | 2,260 ms | 16.5 ms | 8.0 | 0 | 14,329 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 645.1 | 40.4 | 4,277 ms | 24.7 ms | 16.0 | 0 | 19,326 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 97.7 | 100.4 | 44,564 ms | 10.0 ms | 1.0 | 0 | 2,932 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 169.2 | 86.1 | 44,893 ms | 11.6 ms | 2.0 | 0 | 5,077 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 268.5 | 67.4 | 1,454 ms | 14.8 ms | 4.0 | 0 | 8,054 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 360.3 | 44.5 | 2,347 ms | 22.5 ms | 8.0 | 0 | 10,810 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.