Speed test matrix
NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 on H200 SXM 141GB ×4
vllm · 0.28.1rc1.dev562+g9c2d21046 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev562+g9c2d21046
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
9c2d21046bb3
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nemotron-ultra-vllm-9c2d21046/bin/vllm" serve nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 --served-model-name nemotron --trust-remote-code --tensor-parallel-size 4 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype fp8 --max-num-seqs 16 --max-num-batched-tokens 32768 --enable-chunked-prefill --enable-prefix-caching --mamba-backend triton --mamba-ssm-cache-dtype float32 --attention-backend FLASH_ATTN --moe-backend emulation --speculative-config '{"method":"nemotron_h_mtp","num_speculative_tokens":5,"moe_backend":"auto"}'Prefill
prompt tok/s by context length8K context9,394872 ms TTFT
16K context10.1K1,603 ms TTFT
32K context10K3,222 ms TTFT
64K context10K6,443 ms TTFT
128K context9,53213,502 ms TTFT
251K context8,69629,003 ms TTFT
256K context8,65029,739 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 247.8849 ms TTFT | 410.0884 ms TTFT | 550.21,628 ms TTFT | 1,056.03,162 ms TTFT | 1,328.67,713 ms TTFT | 539.7Requests queued12,296 ms TTFT | 624.7Requests queued12,376 ms TTFT | 556.2Requests queued12,468 ms TTFT |
| 16K | 235.81,585 ms TTFT | 280.01,616 ms TTFT | 473.34,607 ms TTFT | 1,024.57,609 ms TTFT | 896.813,626 ms TTFT | 817.8Requests queued22,633 ms TTFT | 1,021.4Requests queued19,710 ms TTFT | 1,071.6Requests queued19,823 ms TTFT |
| 32K | 234.51,590 ms TTFT | 326.01,609 ms TTFT | 527.63,156 ms TTFT | 1,030.97,478 ms TTFT | 1,221.913,350 ms TTFT | 863.8Requests queued22,157 ms TTFT | 1,028.6Requests queued17,835 ms TTFT | 977.0Requests queued20,872 ms TTFT |
| 64K | 222.01,581 ms TTFT | 310.21,589 ms TTFT | 534.72,321 ms TTFT | 1,003.27,067 ms TTFT | 1,329.612,562 ms TTFT | 1,325.7Requests queued48,104 ms TTFT | 1,058.2Requests queued52,404 ms TTFT | Over capacity |
| 128K | 220.31,486 ms TTFT | 333.71,952 ms TTFT | 538.62,533 ms TTFT | 676.36,574 ms TTFT | 960.511,619 ms TTFT | 1,099.5Requests queued79,431 ms TTFT | Over capacity | Over capacity |
| 251K | 218.12,610 ms TTFT | 293.9Requests queued3,603 ms TTFT | 541.5Requests queued6,481 ms TTFT | 937.510,401 ms TTFT | 921.218,204 ms TTFT | 223.0Requests queued92,062 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,394.0 | — | 872 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.1K | — | 1,603 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10K | — | 3,222 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10K | — | 6,443 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,532.0 | — | 13,502 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,696.0 | — | 29,003 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,650.0 | — | 29,739 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 247.8 | 244.0 | 849 ms | 4.1 ms | 1.0 | 1 | 7,433 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 410.0 | 189.2 | 884 ms | 5.3 ms | 2.0 | 2 | 12,293 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 550.2 | 106.3 | 1,628 ms | 9.9 ms | 4.0 | 4 | 16,497 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,056.0 | 113.3 | 3,162 ms | 8.8 ms | 7.9 | 5 | 30,791 | 29.2 s |
| Sustained decode | 8K | 16 | Valid | 1,328.6 | 65.8 | 7,713 ms | 15.2 ms | 16.0 | 0 | 39,841 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 539.7 | 39.3 | 12,296 ms | 25.5 ms | 15.8 | 15 | 16,152 | 29.9 s |
| Sustained decode | 8K | 64 | Requests queued | 624.7 | 50.5 | 12,376 ms | 19.8 ms | 16.0 | 16 | 18,479 | 29.6 s |
| Sustained decode | 8K | 128 | Requests queued | 556.2 | 40.1 | 12,468 ms | 24.9 ms | 15.8 | 15 | 16,651 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 235.8 | 239.0 | 1,585 ms | 4.2 ms | 1.0 | 1 | 7,068 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 280.0 | 136.4 | 1,616 ms | 7.3 ms | 2.0 | 2 | 8,275 | 29.6 s |
| Sustained decode | 16K | 4 | Valid | 473.3 | 118.0 | 4,607 ms | 8.5 ms | 4.0 | 1 | 13,981 | 29.5 s |
| Sustained decode | 16K | 8 | Valid | 1,024.5 | 104.0 | 7,609 ms | 9.6 ms | 8.0 | 1 | 30,708 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 896.8 | 41.7 | 13,626 ms | 24.0 ms | 16.0 | 0 | 26,895 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 817.8 | 50.8 | 22,633 ms | 19.7 ms | 15.8 | 12 | 23,648 | 28.9 s |
| Sustained decode | 16K | 64 | Requests queued | 1,021.4 | 53.2 | 19,710 ms | 18.8 ms | 15.4 | 12 | 29,413 | 28.8 s |
| Sustained decode | 16K | 128 | Requests queued | 1,071.6 | 56.3 | 19,823 ms | 17.8 ms | 15.8 | 10 | 29,775 | 27.8 s |
| Sustained decode | 32K | 1 | Valid | 234.5 | 237.0 | 1,590 ms | 4.2 ms | 1.0 | 1 | 7,033 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 326.0 | 155.6 | 1,609 ms | 6.5 ms | 2.0 | 2 | 9,772 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 527.6 | 115.6 | 3,156 ms | 8.6 ms | 4.0 | 3 | 15,819 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,030.9 | 107.0 | 7,478 ms | 9.4 ms | 8.0 | 1 | 30,912 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,221.9 | 55.7 | 13,350 ms | 18.0 ms | 16.0 | 0 | 36,611 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 863.8 | 52.8 | 22,157 ms | 19.0 ms | 15.7 | 12 | 25,619 | 29.7 s |
| Sustained decode | 32K | 64 | Requests queued | 1,028.6 | 54.2 | 17,835 ms | 18.5 ms | 15.9 | 8 | 30,128 | 29.3 s |
| Sustained decode | 32K | 128 | Requests queued | 977.0 | 54.7 | 20,872 ms | 18.3 ms | 15.7 | 12 | 29,032 | 29.7 s |
| Sustained decode | 64K | 1 | Valid | 222.0 | 224.7 | 1,581 ms | 4.5 ms | 1.0 | 1 | 6,660 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 310.2 | 134.7 | 1,589 ms | 8.2 ms | 2.0 | 2 | 9,299 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 534.7 | 100.9 | 2,321 ms | 10.2 ms | 3.9 | 4 | 16,027 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,003.2 | 106.5 | 7,067 ms | 9.4 ms | 8.0 | 0 | 30,094 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,329.6 | 63.7 | 12,562 ms | 15.7 ms | 16.0 | 0 | 39,855 | 30.0 s |
| Sustained decode | 64K | 32 | Requests queued | 1,325.7 | 57.9 | 48,104 ms | 17.3 ms | 16.0 | 16 | 39,773 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 1,058.2 | 49.2 | 52,404 ms | 20.3 ms | 16.0 | 16 | 31,699 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 220.3 | 221.7 | 1,486 ms | 4.5 ms | 1.0 | 1 | 6,604 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 333.7 | 154.7 | 1,952 ms | 6.7 ms | 2.0 | 2 | 10,006 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 538.6 | 100.0 | 2,533 ms | 10.9 ms | 3.9 | 4 | 16,134 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 676.3 | 76.0 | 6,574 ms | 13.2 ms | 8.0 | 0 | 20,277 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 960.5 | 45.6 | 11,619 ms | 21.9 ms | 16.0 | 0 | 28,787 | 30.0 s |
| Sustained decode | 128K | 32 | Requests queued | 1,099.5 | 56.3 | 79,431 ms | 17.7 ms | 16.0 | 32 | 32,970 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 218.1 | 226.2 | 2,610 ms | 4.4 ms | 0.9 | 1 | 6,540 | 30.0 s |
| Sustained decode | 251K | 2 | Requests queued | 293.9 | 135.9 | 3,603 ms | 7.8 ms | 1.9 | 2 | 8,815 | 30.0 s |
| Sustained decode | 251K | 4 | Requests queued | 541.5 | 132.7 | 6,481 ms | 7.5 ms | 3.8 | 3 | 15,176 | 28.0 s |
| Sustained decode | 251K | 8 | Valid | 937.5 | 94.5 | 10,401 ms | 10.6 ms | 8.0 | 0 | 28,112 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 921.2 | 41.0 | 18,204 ms | 24.4 ms | 16.0 | 0 | 27,580 | 29.9 s |
| Sustained decode | 251K | 32 | Requests queued | 223.0 | 45.4 | 92,062 ms | 22.0 ms | 15.9 | 29 | 6,623 | 29.7 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.