Speed test matrix
gpt-oss-safeguard-20b on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · MXFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- MXFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- gpt-oss
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-openai-vllm-73a5831127a9/bin/vllm" serve openai/gpt-oss-safeguard-20b --served-model-name gpt-oss --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser openai --max-model-len 131072 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --moe-backend marlin
Prefill
prompt tok/s by context length8K context35.6K230 ms TTFT
16K context34.6K468 ms TTFT
32K context30K1,072 ms TTFT
64K context23.5K2,726 ms TTFT
123K context16.7K7,370 ms TTFT
128K context16.3K7,847 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 248.333 ms TTFT | 386.641 ms TTFT | 645.458 ms TTFT | 1,052.492 ms TTFT | 1,652.1172 ms TTFT | 2,830.0336 ms TTFT | 4,655.4707 ms TTFT | 7,397.81,040 ms TTFT |
| 16K | 239.839 ms TTFT | 372.355 ms TTFT | 634.288 ms TTFT | 984.6163 ms TTFT | 1,504.3308 ms TTFT | 2,472.4625 ms TTFT | 4,172.21,027 ms TTFT | Over capacity |
| 32K | 236.649 ms TTFT | 375.975 ms TTFT | 602.8145 ms TTFT | 899.3280 ms TTFT | 1,414.7551 ms TTFT | 2,208.91,120 ms TTFT | 3,543.81,913 ms TTFT | Over capacity |
| 64K | 201.184 ms TTFT | 326.5142 ms TTFT | 552.6249 ms TTFT | 803.1514 ms TTFT | 1,131.21,094 ms TTFT | 1,595.42,196 ms TTFT | Over capacity | Over capacity |
| 123K | 166.1144 ms TTFT | 285.2263 ms TTFT | 507.4521 ms TTFT | 598.6876 ms TTFT | 825.72,238 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 35.6K | — | 230 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.6K | — | 468 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30K | — | 1,072 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 23.5K | — | 2,726 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 16.7K | — | 7,370 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16.3K | — | 7,847 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 248.3 | 249.0 | 33 ms | 4.0 ms | 1.0 | 2 | 7,449 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 386.6 | 193.2 | 41 ms | 5.2 ms | 2.0 | 2 | 11,596 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 645.4 | 166.0 | 58 ms | 6.0 ms | 4.0 | 4 | 19,359 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,052.4 | 140.5 | 92 ms | 7.1 ms | 8.0 | 8 | 31,571 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,652.1 | 107.2 | 172 ms | 9.3 ms | 16.0 | 0 | 49,551 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,830.0 | 90.5 | 336 ms | 11.0 ms | 32.0 | 0 | 84,896 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,655.4 | 78.6 | 707 ms | 12.7 ms | 64.0 | 0 | 139,657 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,397.8 | 62.7 | 1,040 ms | 16.0 ms | 128.0 | 0 | 221,906 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 239.8 | 240.4 | 39 ms | 4.2 ms | 1.0 | 1 | 7,193 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 372.3 | 187.4 | 55 ms | 5.3 ms | 2.0 | 2 | 11,169 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 634.2 | 159.5 | 88 ms | 6.3 ms | 4.0 | 4 | 19,024 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 984.6 | 138.4 | 163 ms | 7.3 ms | 8.0 | 8 | 29,534 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,504.3 | 101.2 | 308 ms | 9.9 ms | 16.0 | 0 | 45,122 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,472.4 | 84.1 | 625 ms | 11.9 ms | 32.0 | 0 | 74,171 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,172.2 | 70.8 | 1,027 ms | 14.1 ms | 64.0 | 0 | 125,156 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 236.6 | 225.8 | 49 ms | 4.4 ms | 1.0 | 1 | 7,097 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 375.9 | 180.7 | 75 ms | 5.5 ms | 2.0 | 2 | 11,274 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 602.8 | 154.4 | 145 ms | 6.5 ms | 4.0 | 4 | 18,081 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 899.3 | 115.2 | 280 ms | 8.7 ms | 8.0 | 0 | 26,979 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,414.7 | 91.9 | 551 ms | 10.9 ms | 16.0 | 0 | 42,434 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,208.9 | 71.6 | 1,120 ms | 14.0 ms | 32.0 | 0 | 66,245 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,543.8 | 58.2 | 1,913 ms | 17.2 ms | 64.0 | 0 | 106,298 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 201.1 | 202.0 | 84 ms | 5.0 ms | 1.0 | 1 | 6,033 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 326.5 | 164.7 | 142 ms | 6.1 ms | 2.0 | 2 | 9,794 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 552.6 | 144.2 | 249 ms | 6.9 ms | 4.0 | 4 | 16,578 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 803.1 | 103.0 | 514 ms | 9.7 ms | 8.0 | 0 | 24,089 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,131.2 | 76.4 | 1,094 ms | 13.1 ms | 16.0 | 0 | 33,938 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,595.4 | 55.0 | 2,196 ms | 18.2 ms | 32.0 | 0 | 47,840 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 166.1 | 167.5 | 144 ms | 6.0 ms | 1.0 | 1 | 4,984 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 285.2 | 145.5 | 263 ms | 6.9 ms | 2.0 | 2 | 7,793 | 27.3 s |
| Sustained decode | 123K | 4 | Valid | 507.4 | 139.3 | 521 ms | 7.2 ms | 4.0 | 4 | 15,220 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 598.6 | 84.9 | 876 ms | 11.8 ms | 8.0 | 0 | 17,954 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 825.7 | 58.8 | 2,238 ms | 17.0 ms | 16.0 | 0 | 24,771 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.