Speed test matrix
Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 --served-model-name nemotron --trust-remote-code --tp 1 --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.875117 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context25.9K316 ms TTFT
16K context31K524 ms TTFT
32K context30.3K1,066 ms TTFT
64K context28.5K2,264 ms TTFT
128K context24.4K5,284 ms TTFT
251K context19.1K13,220 ms TTFT
256K context19K13,539 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 378.598 ms TTFT | 727.0145 ms TTFT | 1,084.4191 ms TTFT | 2,047.6Requests queued214 ms TTFT | 2,274.8327 ms TTFT | 4,374.8610 ms TTFT | 1,707.91,159 ms TTFT | 2,995.72,156 ms TTFT |
| 16K | 368.9302 ms TTFT | 745.9493 ms TTFT | 898.7694 ms TTFT | 1,607.3632 ms TTFT | 2,450.4657 ms TTFT | 2,376.11,238 ms TTFT | 2,749.42,417 ms TTFT | 4,053.94,772 ms TTFT |
| 32K | 380.4359 ms TTFT | 563.4436 ms TTFT | 681.6646 ms TTFT | 1,377.5752 ms TTFT | 2,364.9945 ms TTFT | 2,432.82,990 ms TTFT | 2,730.15,892 ms TTFT | 4,544.911,619 ms TTFT |
| 64K | 355.7498 ms TTFT | 375.4669 ms TTFT | 970.9847 ms TTFT | 1,227.91,629 ms TTFT | 2,161.72,236 ms TTFT | 2,408.86,388 ms TTFT | 3,369.912,840 ms TTFT | 3,941.525,418 ms TTFT |
| 128K | 337.6725 ms TTFT | 116.61,018 ms TTFT | 923.81,752 ms TTFT | 1,255.62,462 ms TTFT | 1,878.86,481 ms TTFT | 1,853.112,918 ms TTFT | 3,236.126,084 ms TTFT | Over capacity |
| 251K | 303.31,292 ms TTFT | 517.21,609 ms TTFT | 785.5Requests queued2,797 ms TTFT | 1,044.0Requests queued4,998 ms TTFT | 852.813,036 ms TTFT | 1,979.326,958 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 25.9K | — | 316 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31K | — | 524 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.3K | — | 1,066 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.5K | — | 2,264 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 24.4K | — | 5,284 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 19.1K | — | 13,220 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19K | — | 13,539 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 378.5 | 383.1 | 98 ms | 2.6 ms | 1.0 | 3 | 11,356 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 727.0 | 1291.8 | 145 ms | 0.8 ms | 2.0 | 4 | 21,808 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,084.4 | 768.2 | 191 ms | 1.3 ms | 4.0 | 8 | 32,531 | 30.0 s |
| Sustained decode | 8K | 8 | Requests queued | 2,047.6 | 252.7 | 214 ms | 4.0 ms | 7.5 | 16 | 61,424 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,274.8 | 173.3 | 327 ms | 5.8 ms | 21.1 | 16 | 68,241 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,374.8 | 134.3 | 610 ms | 7.5 ms | 30.9 | 32 | 131,249 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,707.9 | 92.2 | 1,159 ms | 10.8 ms | 64.0 | 0 | 51,237 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,995.7 | 59.4 | 2,156 ms | 16.8 ms | 128.0 | 0 | 89,849 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 368.9 | 1198.2 | 302 ms | 1.5 ms | 1.0 | 3 | 11,066 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 745.9 | 2018.7 | 493 ms | 0.6 ms | 2.0 | 6 | 22,377 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 898.7 | 263.4 | 694 ms | 3.8 ms | 4.6 | 4 | 26,960 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,607.3 | 233.1 | 632 ms | 4.3 ms | 7.7 | 8 | 48,219 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,450.4 | 170.3 | 657 ms | 5.9 ms | 16.0 | 16 | 73,506 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,376.1 | 121.8 | 1,238 ms | 8.2 ms | 32.0 | 0 | 71,280 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,749.4 | 86.1 | 2,417 ms | 11.6 ms | 64.0 | 0 | 82,483 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,053.9 | 56.0 | 4,772 ms | 17.8 ms | 128.0 | 0 | 121,608 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 380.4 | 1822.9 | 359 ms | 0.5 ms | 1.0 | 2 | 8,618 | 22.7 s |
| Sustained decode | 32K | 2 | Valid | 563.4 | 321.3 | 436 ms | 3.1 ms | 19.0 | 4 | 16,900 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 681.6 | 245.1 | 646 ms | 4.1 ms | 4.7 | 4 | 20,448 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,377.5 | 207.0 | 752 ms | 4.8 ms | 8.0 | 8 | 41,326 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,364.9 | 158.6 | 945 ms | 6.3 ms | 15.9 | 16 | 70,942 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,432.8 | 120.2 | 2,990 ms | 8.3 ms | 32.0 | 3 | 72,906 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,730.1 | 71.0 | 5,892 ms | 14.1 ms | 67.0 | 0 | 81,907 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,544.9 | 55.5 | 11,619 ms | 18.0 ms | 128.0 | 0 | 136,350 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 355.7 | 367.9 | 498 ms | 2.7 ms | 1.0 | 2 | 10,669 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 375.4 | 243.4 | 669 ms | 4.4 ms | 32.6 | 2 | 11,263 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 970.9 | 257.9 | 847 ms | 3.9 ms | 3.8 | 8 | 28,999 | 29.9 s |
| Sustained decode | 64K | 8 | Valid | 1,227.9 | 197.3 | 1,629 ms | 5.1 ms | 10.1 | 8 | 36,833 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,161.7 | 139.3 | 2,236 ms | 7.2 ms | 15.7 | 16 | 64,798 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,408.8 | 107.3 | 6,388 ms | 9.3 ms | 32.0 | 0 | 72,262 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,369.9 | 72.7 | 12,840 ms | 13.8 ms | 64.0 | 0 | 101,084 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,941.5 | 46.5 | 25,418 ms | 21.5 ms | 128.0 | 0 | 118,251 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 337.6 | 354.8 | 725 ms | 2.8 ms | 0.9 | 2 | 10,128 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 116.6 | 64.1 | 1,018 ms | 18.3 ms | 96.1 | 0 | 3,498 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 923.8 | 240.1 | 1,752 ms | 4.2 ms | 3.8 | 4 | 27,716 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,255.6 | 166.6 | 2,462 ms | 6.0 ms | 7.8 | 8 | 37,666 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,878.8 | 119.9 | 6,481 ms | 8.3 ms | 15.8 | 8 | 56,016 | 29.8 s |
| Sustained decode | 128K | 32 | Valid | 1,853.1 | 64.7 | 12,918 ms | 15.5 ms | 40.0 | 0 | 55,596 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 3,236.1 | 60.6 | 26,084 ms | 16.5 ms | 64.0 | 0 | 97,088 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 303.3 | 331.9 | 1,292 ms | 3.0 ms | 0.9 | 2 | 9,098 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 517.2 | 248.5 | 1,609 ms | 4.1 ms | 2.5 | 2 | 15,514 | 30.0 s |
| Sustained decode | 251K | 4 | Requests queued | 785.5 | 213.4 | 2,797 ms | 4.7 ms | 3.8 | 4 | 23,563 | 30.0 s |
| Sustained decode | 251K | 8 | Requests queued | 1,044.0 | 147.8 | 4,998 ms | 6.8 ms | 7.4 | 8 | 31,320 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 852.8 | 63.7 | 13,036 ms | 15.7 ms | 24.0 | 0 | 25,572 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,979.3 | 77.4 | 26,958 ms | 12.9 ms | 32.0 | 0 | 59,379 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.