Speed test matrix
gpt-oss-safeguard-20b on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · MXFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- MXFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- gpt-oss
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-gpt-oss-vllm/bin/vllm" serve openai/gpt-oss-safeguard-20b --served-model-name gpt-oss --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser openai --max-model-len 131072 --gpu-memory-utilization 0.877674 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context43K190 ms TTFT
16K context45.2K358 ms TTFT
32K context40.7K789 ms TTFT
64K context29.2K2,195 ms TTFT
123K context23.4K5,253 ms TTFT
128K context22K5,813 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 280.849 ms TTFT | 495.644 ms TTFT | 853.598 ms TTFT | 1,371.472 ms TTFT | 2,306.5140 ms TTFT | 3,711.5394 ms TTFT | 5,577.4679 ms TTFT | 7,201.81,407 ms TTFT |
| 16K | 275.671 ms TTFT | 486.070 ms TTFT | 814.670 ms TTFT | 1,281.690 ms TTFT | 2,085.1124 ms TTFT | 3,192.5609 ms TTFT | 4,536.61,018 ms TTFT | 5,503.31,731 ms TTFT |
| 32K | 267.988 ms TTFT | 467.3105 ms TTFT | 761.6106 ms TTFT | 1,164.4179 ms TTFT | 1,794.2572 ms TTFT | 2,543.51,111 ms TTFT | 3,304.91,744 ms TTFT | Over capacity |
| 64K | 253.8139 ms TTFT | 424.0148 ms TTFT | 665.2152 ms TTFT | 954.6177 ms TTFT | 1,372.3970 ms TTFT | 1,776.11,989 ms TTFT | 2,082.83,437 ms TTFT | Over capacity |
| 123K | 230.2209 ms TTFT | 368.5231 ms TTFT | 539.7226 ms TTFT | 741.1246 ms TTFT | 945.9237 ms TTFT | 1,125.0277 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 43K | — | 190 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 45.2K | — | 358 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 40.7K | — | 789 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29.2K | — | 2,195 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 23.4K | — | 5,253 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22K | — | 5,813 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 280.8 | 281.7 | 49 ms | 3.5 ms | 1.0 | 2 | 8,425 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 495.6 | 249.7 | 44 ms | 4.0 ms | 2.0 | 7 | 14,868 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 853.5 | 215.6 | 98 ms | 4.6 ms | 4.0 | 5 | 25,601 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,371.4 | 171.0 | 72 ms | 5.8 ms | 8.0 | 14 | 41,141 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,306.5 | 144.1 | 140 ms | 6.9 ms | 16.0 | 31 | 69,199 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,711.5 | 118.0 | 394 ms | 8.5 ms | 32.0 | 30 | 111,324 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,577.4 | 88.5 | 679 ms | 11.3 ms | 63.9 | 54 | 167,290 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,201.8 | 57.3 | 1,407 ms | 17.4 ms | 127.9 | 73 | 216,048 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 275.6 | 278.7 | 71 ms | 3.6 ms | 1.0 | 3 | 8,266 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 486.0 | 246.3 | 70 ms | 4.1 ms | 2.0 | 6 | 14,581 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 814.6 | 206.3 | 70 ms | 4.8 ms | 4.0 | 11 | 24,439 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,281.6 | 163.8 | 90 ms | 6.1 ms | 7.9 | 18 | 38,448 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,085.1 | 132.4 | 124 ms | 7.6 ms | 16.0 | 24 | 62,558 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,192.5 | 101.5 | 609 ms | 9.9 ms | 32.0 | 17 | 95,782 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,536.6 | 72.2 | 1,018 ms | 13.9 ms | 63.9 | 46 | 136,077 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,503.3 | 44.1 | 1,731 ms | 22.7 ms | 127.9 | 59 | 165,074 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 267.9 | 271.5 | 88 ms | 3.7 ms | 1.0 | 3 | 8,038 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 467.3 | 234.6 | 105 ms | 4.3 ms | 2.0 | 3 | 14,018 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 761.6 | 193.3 | 106 ms | 5.2 ms | 4.0 | 9 | 22,847 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,164.4 | 147.0 | 179 ms | 6.8 ms | 8.0 | 12 | 34,928 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,794.2 | 113.6 | 572 ms | 8.8 ms | 16.0 | 7 | 53,822 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,543.5 | 81.0 | 1,111 ms | 12.4 ms | 31.9 | 20 | 76,303 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,304.9 | 52.8 | 1,744 ms | 18.9 ms | 63.9 | 28 | 99,098 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 253.8 | 255.4 | 139 ms | 3.9 ms | 1.0 | 2 | 7,614 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 424.0 | 220.0 | 148 ms | 4.5 ms | 2.0 | 9 | 12,719 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 665.2 | 169.4 | 152 ms | 5.9 ms | 4.0 | 13 | 19,947 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 954.6 | 121.1 | 177 ms | 8.3 ms | 8.0 | 14 | 28,633 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,372.3 | 86.9 | 970 ms | 11.5 ms | 16.0 | 11 | 41,161 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,776.1 | 56.9 | 1,989 ms | 17.6 ms | 32.0 | 12 | 53,289 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,082.8 | 34.0 | 3,437 ms | 29.4 ms | 63.9 | 22 | 62,462 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 230.2 | 242.6 | 209 ms | 4.1 ms | 1.0 | 5 | 6,906 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 368.5 | 192.6 | 231 ms | 5.2 ms | 1.9 | 7 | 11,056 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 539.7 | 140.7 | 226 ms | 7.1 ms | 3.9 | 12 | 16,192 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 741.1 | 94.5 | 246 ms | 10.6 ms | 8.0 | 9 | 22,228 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 945.9 | 60.7 | 237 ms | 16.5 ms | 15.8 | 23 | 28,376 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,125.0 | 36.4 | 277 ms | 27.5 ms | 31.7 | 54 | 33,741 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.