Speed test matrix
NVIDIA-Nemotron-3-Super-120B-A12B-FP8 on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev638+g52be12cfa · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 "$HOME/env_h200-nvidia-nemotron/bin/vllm" serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 --served-model-name nemotron --trust-remote-code --tensor-parallel-size 2 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 1048576 --gpu-memory-utilization 0.880548 --host 0.0.0.0 --port 8000 --api-key <redacted> --async-scheduling --dtype auto --kv-cache-dtype fp8 --max-cudagraph-capture-size 128 --enable-chunked-prefill --mamba-ssm-cache-dtype float32
Prefill
prompt tok/s by context length8K context30.3K270 ms TTFT
16K context30.5K531 ms TTFT
32K context31.5K1,024 ms TTFT
64K context29.6K2,177 ms TTFT
128K context27K4,760 ms TTFT
256K context22.8K11,260 ms TTFT
512K context17.2K29,899 ms TTFT
1019K context11.5K88,645 ms TTFT
1024K context11.5K89,383 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 140.9252 ms TTFT | 267.2379 ms TTFT | 475.9596 ms TTFT | 767.71,041 ms TTFT | 1,170.51,938 ms TTFT | 1,775.43,972 ms TTFT | 2,854.07,568 ms TTFT | 4,357.414,899 ms TTFT |
| 16K | 140.3274 ms TTFT | 266.9400 ms TTFT | 478.7638 ms TTFT | 764.51,110 ms TTFT | 1,155.42,048 ms TTFT | 1,786.53,887 ms TTFT | 2,779.87,356 ms TTFT | 4,222.514,607 ms TTFT |
| 32K | 139.5311 ms TTFT | 265.3424 ms TTFT | 472.5693 ms TTFT | 761.51,197 ms TTFT | 1,151.12,195 ms TTFT | 1,761.03,922 ms TTFT | 2,726.17,402 ms TTFT | 4,063.614,668 ms TTFT |
| 64K | 137.6360 ms TTFT | 261.5463 ms TTFT | 469.9798 ms TTFT | 767.41,360 ms TTFT | 1,123.62,235 ms TTFT | 1,698.53,847 ms TTFT | 2,583.47,153 ms TTFT | 3,757.614,001 ms TTFT |
| 128K | 134.2427 ms TTFT | 255.7508 ms TTFT | 462.2741 ms TTFT | 735.51,176 ms TTFT | 1,069.81,904 ms TTFT | 1,586.03,230 ms TTFT | 2,336.55,766 ms TTFT | 3,239.411,120 ms TTFT |
| 256K | 126.2897 ms TTFT | 249.51,121 ms TTFT | 446.91,586 ms TTFT | 682.32,532 ms TTFT | 989.34,126 ms TTFT | 1,395.97,813 ms TTFT | 1,925.514,544 ms TTFT | Over capacity |
| 512K | 118.91,650 ms TTFT | 229.52,050 ms TTFT | 410.12,945 ms TTFT | 600.34,686 ms TTFT | 854.78,129 ms TTFT | 1,116.015,084 ms TTFT | Over capacity | Over capacity |
| 1019K | 102.03,585 ms TTFT | 197.44,136 ms TTFT | 358.15,668 ms TTFT | 498.38,991 ms TTFT | 655.015,533 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.3K | — | 270 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 30.5K | — | 531 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.5K | — | 1,024 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29.6K | — | 2,177 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 27K | — | 4,760 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 22.8K | — | 11,260 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 17.2K | — | 29,899 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 11.5K | — | 88,645 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 11.5K | — | 89,383 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 140.9 | 142.3 | 252 ms | 7.0 ms | 1.0 | 1 | 4,228 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 267.2 | 134.9 | 379 ms | 7.4 ms | 1.9 | 2 | 8,014 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 475.9 | 118.2 | 596 ms | 8.5 ms | 4.0 | 0 | 14,277 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 767.7 | 94.3 | 1,041 ms | 10.6 ms | 8.0 | 0 | 23,023 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,170.5 | 70.4 | 1,938 ms | 14.2 ms | 16.0 | 0 | 35,110 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,775.4 | 51.9 | 3,972 ms | 19.3 ms | 32.0 | 0 | 53,250 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,854.0 | 37.9 | 7,568 ms | 26.4 ms | 64.0 | 0 | 85,633 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,357.4 | 25.1 | 14,899 ms | 39.8 ms | 128.0 | 0 | 130,825 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 140.3 | 141.9 | 274 ms | 7.0 ms | 1.0 | 1 | 4,210 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 266.9 | 135.4 | 400 ms | 7.4 ms | 2.0 | 2 | 8,007 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 478.7 | 118.8 | 638 ms | 8.4 ms | 4.0 | 0 | 14,359 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 764.5 | 94.1 | 1,110 ms | 10.6 ms | 8.0 | 0 | 22,929 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,155.4 | 69.8 | 2,048 ms | 14.3 ms | 16.0 | 0 | 34,657 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,786.5 | 51.6 | 3,887 ms | 19.4 ms | 32.0 | 0 | 53,599 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,779.8 | 37.2 | 7,356 ms | 26.9 ms | 64.0 | 0 | 83,398 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,222.5 | 24.4 | 14,607 ms | 40.9 ms | 128.0 | 0 | 126,635 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 139.5 | 141.4 | 311 ms | 7.1 ms | 1.0 | 1 | 4,183 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 265.3 | 134.7 | 424 ms | 7.4 ms | 1.9 | 2 | 7,964 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 472.5 | 117.5 | 693 ms | 8.5 ms | 4.0 | 0 | 14,180 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 761.5 | 93.9 | 1,197 ms | 10.7 ms | 8.0 | 0 | 22,845 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,151.1 | 69.7 | 2,195 ms | 14.3 ms | 16.0 | 0 | 34,528 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,761.0 | 51.0 | 3,922 ms | 19.6 ms | 32.0 | 0 | 52,832 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,726.1 | 36.2 | 7,402 ms | 27.7 ms | 64.0 | 0 | 81,788 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,063.6 | 23.6 | 14,668 ms | 42.5 ms | 128.0 | 0 | 122,107 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 137.6 | 140.0 | 360 ms | 7.1 ms | 1.0 | 1 | 4,127 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 261.5 | 132.8 | 463 ms | 7.5 ms | 2.0 | 2 | 7,842 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 469.9 | 117.0 | 798 ms | 8.5 ms | 4.0 | 0 | 14,096 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 767.4 | 94.9 | 1,360 ms | 10.5 ms | 8.0 | 0 | 23,015 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,123.6 | 68.3 | 2,235 ms | 14.6 ms | 16.0 | 0 | 33,695 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,698.5 | 49.6 | 3,847 ms | 20.2 ms | 32.0 | 0 | 50,943 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,583.4 | 34.6 | 7,153 ms | 28.9 ms | 64.0 | 0 | 77,500 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,757.6 | 21.9 | 14,001 ms | 45.6 ms | 128.0 | 0 | 112,772 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 134.2 | 137.7 | 427 ms | 7.3 ms | 1.0 | 1 | 4,025 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 255.7 | 130.0 | 508 ms | 7.7 ms | 1.9 | 2 | 7,670 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 462.2 | 115.0 | 741 ms | 8.7 ms | 4.0 | 0 | 13,863 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 735.5 | 91.0 | 1,176 ms | 11.0 ms | 8.0 | 0 | 22,061 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,069.8 | 65.5 | 1,904 ms | 15.3 ms | 16.0 | 0 | 32,096 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,586.0 | 47.0 | 3,230 ms | 21.3 ms | 32.0 | 0 | 47,581 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,336.5 | 32.5 | 5,766 ms | 30.8 ms | 64.0 | 0 | 70,080 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 3,239.4 | 20.2 | 11,120 ms | 49.4 ms | 128.0 | 0 | 97,153 | 30.0 s |
| Sustained decode | 256K | 1 | Valid | 126.2 | 135.1 | 897 ms | 7.4 ms | 1.0 | 1 | 3,787 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 249.5 | 124.3 | 1,121 ms | 8.0 ms | 1.9 | 2 | 7,316 | 29.3 s |
| Sustained decode | 256K | 4 | Valid | 446.9 | 110.0 | 1,586 ms | 9.1 ms | 4.0 | 0 | 13,403 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 682.3 | 82.3 | 2,532 ms | 12.1 ms | 8.0 | 0 | 20,463 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 989.3 | 57.3 | 4,126 ms | 17.5 ms | 16.0 | 0 | 29,677 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,395.9 | 37.5 | 7,813 ms | 26.7 ms | 32.0 | 0 | 41,855 | 30.0 s |
| Sustained decode | 256K | 64 | Valid | 1,925.5 | 22.5 | 14,544 ms | 44.4 ms | 64.0 | 0 | 57,767 | 30.0 s |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 118.9 | 119.6 | 1,650 ms | 8.4 ms | 1.0 | 0 | 3,567 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 229.5 | 114.4 | 2,050 ms | 8.7 ms | 2.0 | 0 | 6,886 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 410.1 | 100.6 | 2,945 ms | 9.9 ms | 4.0 | 0 | 12,304 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 600.3 | 71.7 | 4,686 ms | 13.9 ms | 8.0 | 0 | 18,007 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 854.7 | 48.4 | 8,129 ms | 20.7 ms | 16.0 | 0 | 25,640 | 30.0 s |
| Sustained decode | 512K | 32 | Valid | 1,116.0 | 28.9 | 15,084 ms | 34.6 ms | 32.0 | 0 | 33,469 | 30.0 s |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 102.0 | 103.0 | 3,585 ms | 9.7 ms | 1.0 | 0 | 3,060 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 197.4 | 97.9 | 4,136 ms | 10.2 ms | 2.0 | 0 | 5,924 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 358.1 | 86.0 | 5,668 ms | 11.6 ms | 4.0 | 0 | 10,740 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 498.3 | 56.8 | 8,991 ms | 17.6 ms | 8.0 | 0 | 14,944 | 30.0 s |
| Sustained decode | 1019K | 16 | Valid | 655.0 | 34.1 | 15,533 ms | 29.3 ms | 16.0 | 0 | 19,649 | 30.0 s |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.