Speed test matrix
North-Mini-Code-1.0 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF167 contexts (8K–494,880) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 494,880, 500,000
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- north-mini-code
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-coherelabs-vllm-73a5831127a9/bin/vllm" serve CohereLabs/North-Mini-Code-1.0 --served-model-name north-mini-code --tensor-parallel-size 1 --reasoning-parser cohere_command4 --enable-auto-tool-choice --tool-call-parser cohere_command4 --max-model-len 500000 --gpu-memory-utilization 0.80 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --linear-backend torch --moe-backend triton
Prefill
prompt tok/s by context length8K context30.2K271 ms TTFT
16K context28.6K564 ms TTFT
32K context25.3K1,268 ms TTFT
64K context20.5K3,111 ms TTFT
128K context14.8K8,578 ms TTFT
256K context9,55626,623 ms TTFT
494,880 context5,85581,986 ms TTFT
500,000 context5,80083,621 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 142.236 ms TTFT | 214.551 ms TTFT | 338.263 ms TTFT | 513.286 ms TTFT | 811.5136 ms TTFT | 1,217.3235 ms TTFT | Over capacity | Over capacity |
| 16K | 139.255 ms TTFT | 214.466 ms TTFT | 297.094 ms TTFT | 459.9145 ms TTFT | 803.6226 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 133.984 ms TTFT | 113.5100 ms TTFT | 330.5133 ms TTFT | 521.1200 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 123.7136 ms TTFT | 195.8188 ms TTFT | 307.8242 ms TTFT | 484.1392 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 108.5265 ms TTFT | 176.0360 ms TTFT | 291.3521 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 86.2554 ms TTFT | 140.2770 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 494,880 | 62.71,143 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.2K | — | 271 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 28.6K | — | 564 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 25.3K | — | 1,268 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.5K | — | 3,111 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.8K | — | 8,578 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,556.0 | — | 26,623 ms | — | — | 1 | — | — |
| Prefill | 494,880 | 1 | Valid | 5,855.0 | — | 81,986 ms | — | — | 1 | — | — |
| Prefill | 500,000 | 1 | Valid | 5,800.0 | — | 83,621 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 142.2 | 142.6 | 36 ms | 7.0 ms | 1.0 | 1 | 4,266 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 214.5 | 108.2 | 51 ms | 9.2 ms | 2.0 | 0 | 6,430 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 338.2 | 85.9 | 63 ms | 11.6 ms | 4.0 | 0 | 10,146 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 513.2 | 67.5 | 86 ms | 14.8 ms | 8.0 | 0 | 15,389 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 811.5 | 52.5 | 136 ms | 19.0 ms | 16.0 | 0 | 24,330 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,217.3 | 40.6 | 235 ms | 24.6 ms | 32.0 | 0 | 36,501 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 139.2 | 139.8 | 55 ms | 7.2 ms | 1.0 | 1 | 4,176 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 214.4 | 107.5 | 66 ms | 9.3 ms | 2.0 | 0 | 6,432 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 297.0 | 83.1 | 94 ms | 12.0 ms | 4.0 | 0 | 8,909 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 459.9 | 66.1 | 145 ms | 15.1 ms | 8.0 | 0 | 13,798 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 803.6 | 51.5 | 226 ms | 19.4 ms | 16.0 | 0 | 24,095 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 133.9 | 134.9 | 84 ms | 7.4 ms | 1.0 | 1 | 4,016 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 113.5 | 104.8 | 100 ms | 9.5 ms | 2.0 | 0 | 3,403 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 330.5 | 83.2 | 133 ms | 12.0 ms | 4.0 | 0 | 9,912 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 521.1 | 65.9 | 200 ms | 15.2 ms | 8.0 | 0 | 15,632 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 123.7 | 130.2 | 136 ms | 7.7 ms | 1.0 | 1 | 3,711 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 195.8 | 98.3 | 188 ms | 10.2 ms | 2.0 | 0 | 5,875 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 307.8 | 77.7 | 242 ms | 12.9 ms | 4.0 | 0 | 9,232 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 484.1 | 61.2 | 392 ms | 16.3 ms | 8.0 | 0 | 14,519 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 108.5 | 108.7 | 265 ms | 9.2 ms | 1.0 | 0 | 3,256 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 176.0 | 88.3 | 360 ms | 11.3 ms | 2.0 | 0 | 5,280 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 291.3 | 73.2 | 521 ms | 13.7 ms | 4.0 | 0 | 8,736 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 86.2 | 86.5 | 554 ms | 11.6 ms | 1.0 | 0 | 2,586 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 140.2 | 70.4 | 770 ms | 14.2 ms | 2.0 | 0 | 4,204 | 30.0 s |
| Sustained decode | 256K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 494,880 | 1 | Valid | 62.7 | 63.0 | 1,143 ms | 15.9 ms | 1.0 | 0 | 1,879 | 30.0 s |
| Sustained decode | 494,880 | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 494,880 | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 494,880 | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 494,880 | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 494,880 | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 494,880 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 494,880 | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.