Speed test matrix
gpt-oss-120b on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · MXFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- MXFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- gpt-oss
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-gpt-oss-vllm/bin/vllm" serve openai/gpt-oss-120b --served-model-name gpt-oss --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser openai --max-model-len 131072 --gpu-memory-utilization 0.877719 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context9,982820 ms TTFT
16K context14.5K1,115 ms TTFT
32K context18K1,781 ms TTFT
64K context17.9K3,571 ms TTFT
123K context15.2K8,066 ms TTFT
128K context14.8K8,621 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 190.953 ms TTFT | 336.656 ms TTFT | 532.163 ms TTFT | 771.986 ms TTFT | 1,188.1101 ms TTFT | 1,831.5129 ms TTFT | 2,598.8172 ms TTFT | 3,648.8242 ms TTFT |
| 16K | 189.770 ms TTFT | 329.472 ms TTFT | 503.081 ms TTFT | 748.4100 ms TTFT | 1,104.8119 ms TTFT | 1,622.5154 ms TTFT | 2,270.0201 ms TTFT | Over capacity |
| 32K | 175.994 ms TTFT | 306.5104 ms TTFT | 467.9106 ms TTFT | 691.3117 ms TTFT | 986.2144 ms TTFT | 1,385.9178 ms TTFT | Over capacity | Over capacity |
| 64K | 162.9138 ms TTFT | 274.2Requests queued153 ms TTFT | 406.5158 ms TTFT | 568.0166 ms TTFT | 770.5199 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 150.7208 ms TTFT | 243.3229 ms TTFT | 335.1230 ms TTFT | 440.5239 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,982.0 | — | 820 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.5K | — | 1,115 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 18K | — | 1,781 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 17.9K | — | 3,571 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 15.2K | — | 8,066 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.8K | — | 8,621 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 190.9 | 198.0 | 53 ms | 5.1 ms | 1.0 | 19 | 5,726 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 336.6 | 174.2 | 56 ms | 5.7 ms | 1.9 | 37 | 10,097 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 532.1 | 138.4 | 63 ms | 7.2 ms | 3.9 | 57 | 15,962 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 771.9 | 102.3 | 86 ms | 9.8 ms | 8.0 | 68 | 23,152 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,188.1 | 78.7 | 101 ms | 12.7 ms | 15.9 | 113 | 35,638 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,831.5 | 59.4 | 129 ms | 16.8 ms | 31.7 | 168 | 54,947 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,598.8 | 42.6 | 172 ms | 23.5 ms | 63.8 | 377 | 77,946 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,648.8 | 29.5 | 242 ms | 33.9 ms | 127.1 | 380 | 109,433 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 189.7 | 194.5 | 70 ms | 5.1 ms | 1.0 | 14 | 5,690 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 329.4 | 170.9 | 72 ms | 5.8 ms | 2.0 | 28 | 9,881 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 503.0 | 129.0 | 81 ms | 7.8 ms | 4.0 | 24 | 15,090 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 748.4 | 97.2 | 100 ms | 10.3 ms | 7.9 | 62 | 22,445 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,104.8 | 72.4 | 119 ms | 13.8 ms | 15.9 | 93 | 33,144 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,622.5 | 52.7 | 154 ms | 19.0 ms | 31.8 | 159 | 48,634 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,270.0 | 36.7 | 201 ms | 27.2 ms | 63.7 | 209 | 68,086 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 175.9 | 189.4 | 94 ms | 5.3 ms | 1.0 | 22 | 5,278 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 306.5 | 163.2 | 104 ms | 6.1 ms | 1.9 | 34 | 9,194 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 467.9 | 123.9 | 106 ms | 8.1 ms | 3.9 | 37 | 14,036 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 691.3 | 90.6 | 117 ms | 11.0 ms | 7.9 | 67 | 20,735 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 986.2 | 63.9 | 144 ms | 15.6 ms | 15.8 | 90 | 29,573 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,385.9 | 45.0 | 178 ms | 22.2 ms | 31.8 | 140 | 41,576 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 162.9 | 180.2 | 138 ms | 5.6 ms | 1.0 | 21 | 4,887 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 274.2 | 149.6 | 153 ms | 6.7 ms | 1.8 | 32 | 8,226 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 406.5 | 108.1 | 158 ms | 9.3 ms | 3.8 | 41 | 12,191 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 568.0 | 74.4 | 166 ms | 13.4 ms | 7.9 | 73 | 17,038 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 770.5 | 50.2 | 199 ms | 19.9 ms | 15.8 | 81 | 23,075 | 29.9 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 150.7 | 165.5 | 208 ms | 6.0 ms | 0.9 | 15 | 4,521 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 243.3 | 131.3 | 229 ms | 7.6 ms | 1.9 | 20 | 7,298 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 335.1 | 89.3 | 230 ms | 11.2 ms | 3.8 | 28 | 10,052 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 440.5 | 57.8 | 239 ms | 17.3 ms | 7.7 | 37 | 13,209 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.