Speed test matrix
gpt-oss-20b on H200 NVL 141GB
vllm · 0.26.1rc1.dev914+gc3dabcddc · MXFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- MXFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- gpt-oss
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 "$HOME/env_h200-z-lab-vllm/bin/vllm" serve openai/gpt-oss-20b --served-model-name gpt-oss --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser openai --max-model-len 131072 --gpu-memory-utilization 0.827527 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/gpt-oss-20b-DFlash","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context19.9K411 ms TTFT
16K context26.2K618 ms TTFT
32K context31.1K1,032 ms TTFT
64K context29.5K2,173 ms TTFT
123K context23.5K5,228 ms TTFT
128K context21.6K5,923 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 302.648 ms TTFT | 557.653 ms TTFT | 946.157 ms TTFT | 1,583.068 ms TTFT | 2,346.687 ms TTFT | 3,582.9104 ms TTFT | 4,828.5158 ms TTFT | 5,651.7241 ms TTFT |
| 16K | 280.074 ms TTFT | 485.776 ms TTFT | 815.582 ms TTFT | 1,306.794 ms TTFT | 1,935.3115 ms TTFT | 2,789.3151 ms TTFT | 3,565.5215 ms TTFT | 3,982.9367 ms TTFT |
| 32K | 221.1109 ms TTFT | 384.3112 ms TTFT | 627.2116 ms TTFT | 983.6120 ms TTFT | 1,376.3148 ms TTFT | 1,849.8189 ms TTFT | 2,211.6291 ms TTFT | Over capacity |
| 64K | 182.41,053 ms TTFT | 288.1167 ms TTFT | 458.7170 ms TTFT | 662.1199 ms TTFT | 849.1214 ms TTFT | 1,082.7277 ms TTFT | Over capacity | Over capacity |
| 123K | 153.1238 ms TTFT | 238.6252 ms TTFT | 350.7260 ms TTFT | 451.8262 ms TTFT | 561.5303 ms TTFT | 658.1429 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19.9K | — | 411 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 26.2K | — | 618 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.1K | — | 1,032 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29.5K | — | 2,173 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 23.5K | — | 5,228 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 21.6K | — | 5,923 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 302.6 | 338.4 | 48 ms | 3.0 ms | 1.0 | 8 | 9,076 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 557.6 | 305.4 | 53 ms | 3.3 ms | 2.0 | 15 | 16,729 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 946.1 | 261.0 | 57 ms | 3.8 ms | 3.9 | 42 | 28,378 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,583.0 | 216.3 | 68 ms | 4.6 ms | 8.0 | 77 | 47,479 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,346.6 | 165.3 | 87 ms | 6.0 ms | 16.0 | 73 | 70,399 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,582.9 | 122.2 | 104 ms | 8.2 ms | 31.9 | 142 | 107,584 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,828.5 | 81.6 | 158 ms | 12.2 ms | 63.8 | 235 | 144,790 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,651.7 | 46.1 | 241 ms | 21.7 ms | 127.8 | 481 | 169,488 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 280.0 | 292.5 | 74 ms | 3.4 ms | 1.0 | 9 | 8,400 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 485.7 | 272.6 | 76 ms | 3.7 ms | 2.0 | 18 | 14,570 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 815.5 | 220.6 | 82 ms | 4.5 ms | 4.0 | 23 | 24,463 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,306.7 | 181.9 | 94 ms | 5.5 ms | 7.9 | 47 | 39,192 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,935.3 | 130.8 | 115 ms | 7.6 ms | 16.0 | 61 | 58,055 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,789.3 | 94.5 | 151 ms | 10.6 ms | 31.8 | 95 | 83,685 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,565.5 | 60.0 | 215 ms | 16.7 ms | 63.8 | 209 | 106,957 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,982.9 | 33.6 | 367 ms | 29.8 ms | 127.6 | 298 | 119,349 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 221.1 | 226.6 | 109 ms | 4.4 ms | 1.0 | 2 | 6,633 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 384.3 | 212.1 | 112 ms | 4.7 ms | 1.9 | 14 | 11,529 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 627.2 | 163.6 | 116 ms | 6.1 ms | 3.9 | 17 | 18,810 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 983.6 | 132.0 | 120 ms | 7.6 ms | 8.0 | 22 | 29,509 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,376.3 | 90.3 | 148 ms | 11.1 ms | 15.9 | 29 | 41,288 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,849.8 | 60.8 | 189 ms | 16.5 ms | 31.8 | 71 | 55,484 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,211.6 | 36.7 | 291 ms | 27.2 ms | 63.9 | 95 | 66,223 | 29.9 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 182.4 | 183.0 | 1,053 ms | 5.5 ms | 1.0 | 1 | 5,470 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 288.1 | 150.6 | 167 ms | 6.6 ms | 1.9 | 10 | 8,641 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 458.7 | 119.2 | 170 ms | 8.4 ms | 4.0 | 17 | 13,757 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 662.1 | 84.3 | 199 ms | 11.9 ms | 8.0 | 12 | 19,861 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 849.1 | 54.5 | 214 ms | 18.4 ms | 15.9 | 36 | 25,467 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,082.7 | 35.0 | 277 ms | 28.6 ms | 31.8 | 39 | 32,466 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 153.1 | 160.8 | 238 ms | 6.2 ms | 1.0 | 2 | 4,594 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 238.6 | 127.1 | 252 ms | 7.9 ms | 1.9 | 13 | 7,156 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 350.7 | 91.1 | 260 ms | 11.0 ms | 3.9 | 13 | 10,516 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 451.8 | 59.5 | 262 ms | 16.8 ms | 7.9 | 28 | 13,552 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 561.5 | 36.3 | 303 ms | 27.5 ms | 15.9 | 61 | 16,833 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 658.1 | 21.4 | 429 ms | 46.7 ms | 31.6 | 57 | 19,706 | 29.9 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.