Speed test matrix
Nex-N2-mini on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.1.dev1+ge0317b28c · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+ge0317b28c
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e0317b28ce3b
Run
- Served model
- nex-n2-mini
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nex-agi-vllm-e0317b28ce3b/bin/vllm" serve nex-agi/Nex-N2-mini --served-model-name nex-n2-mini --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-ssm-cache-dtype float32 --attention-backend FLASH_ATTN --moe-backend flashinfer_cutlass
Prefill
prompt tok/s by context length8K context30.9K265 ms TTFT
16K context30.6K531 ms TTFT
32K context30.2K1,068 ms TTFT
64K context27.8K2,320 ms TTFT
128K context22.7K5,676 ms TTFT
251K context16.8K14,972 ms TTFT
256K context16.5K15,561 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 215.266 ms TTFT | 326.792 ms TTFT | 570.3135 ms TTFT | 898.7255 ms TTFT | 1,374.7482 ms TTFT | 2,049.5974 ms TTFT | 3,100.22,051 ms TTFT | 4,420.14,433 ms TTFT |
| 16K | 212.873 ms TTFT | 323.0103 ms TTFT | 561.6129 ms TTFT | 881.8241 ms TTFT | 1,324.4445 ms TTFT | 1,985.8875 ms TTFT | 2,943.01,874 ms TTFT | 4,170.34,206 ms TTFT |
| 32K | 207.6108 ms TTFT | 312.3135 ms TTFT | 542.8192 ms TTFT | 851.6335 ms TTFT | 1,282.1595 ms TTFT | 1,848.31,191 ms TTFT | 2,603.32,516 ms TTFT | Over capacity |
| 64K | 198.5144 ms TTFT | 295.1159 ms TTFT | 521.5235 ms TTFT | 809.6447 ms TTFT | 1,152.1740 ms TTFT | 1,612.51,350 ms TTFT | Over capacity | Over capacity |
| 128K | 181.0317 ms TTFT | 267.6396 ms TTFT | 469.2594 ms TTFT | 706.0951 ms TTFT | 985.21,683 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 155.3617 ms TTFT | 231.2802 ms TTFT | 377.11,173 ms TTFT | 584.71,882 ms TTFT | 759.93,323 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.9K | — | 265 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 30.6K | — | 531 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.2K | — | 1,068 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 27.8K | — | 2,320 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.7K | — | 5,676 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 16.8K | — | 14,972 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.5K | — | 15,561 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 215.2 | 215.7 | 66 ms | 4.6 ms | 1.0 | 1 | 6,455 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 326.7 | 164.3 | 92 ms | 6.1 ms | 2.0 | 2 | 9,800 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 570.3 | 143.3 | 135 ms | 7.0 ms | 4.0 | 4 | 17,108 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 898.7 | 112.0 | 255 ms | 8.9 ms | 8.0 | 0 | 26,961 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,374.7 | 85.4 | 482 ms | 11.7 ms | 16.0 | 0 | 41,232 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,049.5 | 63.5 | 974 ms | 15.7 ms | 32.0 | 0 | 61,472 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,100.2 | 47.3 | 2,051 ms | 21.1 ms | 64.0 | 0 | 92,992 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,420.1 | 32.9 | 4,433 ms | 30.4 ms | 128.0 | 0 | 132,608 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 212.8 | 213.4 | 73 ms | 4.7 ms | 1.0 | 1 | 6,385 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 323.0 | 162.1 | 103 ms | 6.2 ms | 2.0 | 2 | 9,691 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 561.6 | 141.0 | 129 ms | 7.1 ms | 4.0 | 4 | 16,848 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 881.8 | 110.2 | 241 ms | 9.1 ms | 8.0 | 0 | 26,446 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,324.4 | 82.8 | 445 ms | 12.1 ms | 16.0 | 0 | 39,728 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,985.8 | 61.8 | 875 ms | 16.2 ms | 32.0 | 0 | 59,552 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,943.0 | 45.1 | 1,874 ms | 22.2 ms | 64.0 | 0 | 88,256 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,170.3 | 30.9 | 4,206 ms | 32.4 ms | 128.0 | 0 | 125,056 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 207.6 | 208.5 | 108 ms | 4.8 ms | 1.0 | 1 | 6,229 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 312.3 | 156.9 | 135 ms | 6.4 ms | 2.0 | 2 | 9,368 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 542.8 | 136.4 | 192 ms | 7.3 ms | 4.0 | 4 | 16,280 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 851.6 | 106.2 | 335 ms | 9.4 ms | 8.0 | 0 | 25,544 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,282.1 | 79.6 | 595 ms | 12.6 ms | 16.0 | 0 | 38,465 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,848.3 | 56.9 | 1,191 ms | 17.6 ms | 32.0 | 0 | 55,450 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,603.3 | 39.3 | 2,516 ms | 25.5 ms | 64.0 | 0 | 78,080 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 198.5 | 199.6 | 144 ms | 5.0 ms | 1.0 | 1 | 5,954 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 295.1 | 148.4 | 159 ms | 6.7 ms | 2.0 | 2 | 8,853 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 521.5 | 131.2 | 235 ms | 7.6 ms | 4.0 | 4 | 15,644 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 809.6 | 100.9 | 447 ms | 9.9 ms | 8.0 | 0 | 24,289 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,152.1 | 71.9 | 740 ms | 13.9 ms | 16.0 | 0 | 34,560 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,612.5 | 50.1 | 1,350 ms | 20.0 ms | 32.0 | 0 | 48,352 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 181.0 | 183.1 | 317 ms | 5.5 ms | 1.0 | 1 | 5,429 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 267.6 | 135.5 | 396 ms | 7.4 ms | 1.9 | 2 | 8,027 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 469.2 | 117.0 | 594 ms | 8.5 ms | 4.0 | 0 | 14,075 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 706.0 | 87.5 | 951 ms | 11.4 ms | 8.0 | 0 | 21,176 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 985.2 | 60.0 | 1,683 ms | 16.7 ms | 16.0 | 0 | 29,552 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 155.3 | 159.0 | 617 ms | 6.3 ms | 1.0 | 1 | 4,660 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 231.2 | 115.5 | 802 ms | 8.7 ms | 2.0 | 0 | 6,934 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 377.1 | 93.9 | 1,173 ms | 10.7 ms | 4.0 | 0 | 11,312 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 584.7 | 72.2 | 1,882 ms | 13.9 ms | 8.0 | 0 | 17,536 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 759.9 | 45.6 | 3,323 ms | 21.9 ms | 16.0 | 0 | 22,784 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.