Speed test matrix
North-Mini-Code-1.0 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF167 contexts (8K–314,880) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 314,880, 320,000
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- north-mini-code
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-cohere-vllm/bin/vllm" serve CohereLabs/North-Mini-Code-1.0 --served-model-name north-mini-code --trust-remote-code --tensor-parallel-size 1 --reasoning-parser cohere_command4 --enable-auto-tool-choice --tool-call-parser cohere_command4 --max-model-len 320000 --gpu-memory-utilization 0.885868 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context39.4K207 ms TTFT
16K context36K447 ms TTFT
32K context33.4K957 ms TTFT
64K context28.4K2,248 ms TTFT
128K context21.1K6,021 ms TTFT
256K context13.5K18,805 ms TTFT
314,880 context11.7K26,016 ms TTFT
320,000 context11.7K26,594 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 198.253 ms TTFT | 338.967 ms TTFT | 563.698 ms TTFT | 867.3181 ms TTFT | 1,347.6228 ms TTFT | 2,054.3354 ms TTFT | 2,921.7523 ms TTFT | 4,032.8937 ms TTFT |
| 16K | 195.680 ms TTFT | 334.6107 ms TTFT | 550.2143 ms TTFT | 854.3213 ms TTFT | 1,317.3296 ms TTFT | 1,918.3576 ms TTFT | 2,703.2771 ms TTFT | Over capacity |
| 32K | 191.3123 ms TTFT | 327.0118 ms TTFT | 532.8225 ms TTFT | 809.2342 ms TTFT | 1,210.2543 ms TTFT | 1,741.9694 ms TTFT | 2,344.01,359 ms TTFT | Over capacity |
| 64K | 183.9192 ms TTFT | 308.5235 ms TTFT | 494.5243 ms TTFT | 736.7582 ms TTFT | 1,062.01,009 ms TTFT | 1,438.91,782 ms TTFT | Over capacity | Over capacity |
| 128K | 169.9323 ms TTFT | 279.6383 ms TTFT | 442.2571 ms TTFT | 618.2960 ms TTFT | 851.41,552 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 147.5644 ms TTFT | 237.1881 ms TTFT | 356.81,214 ms TTFT | 468.82,051 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 314,880 | 139.8797 ms TTFT | 222.81,150 ms TTFT | 334.41,672 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 39.4K | — | 207 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 36K | — | 447 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33.4K | — | 957 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.4K | — | 2,248 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 21.1K | — | 6,021 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.5K | — | 18,805 ms | — | — | 1 | — | — |
| Prefill | 314,880 | 1 | Valid | 11.7K | — | 26,016 ms | — | — | 1 | — | — |
| Prefill | 320,000 | 1 | Valid | 11.7K | — | 26,594 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 198.2 | 198.8 | 53 ms | 5.0 ms | 1.0 | 1 | 5,946 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 338.9 | 170.8 | 67 ms | 5.9 ms | 2.0 | 2 | 10,167 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 563.6 | 142.7 | 98 ms | 7.0 ms | 4.0 | 4 | 16,907 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 867.3 | 109.9 | 181 ms | 9.1 ms | 8.0 | 0 | 26,016 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,347.6 | 85.5 | 228 ms | 11.7 ms | 16.0 | 0 | 40,417 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,054.3 | 65.5 | 354 ms | 15.3 ms | 32.0 | 0 | 61,629 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,921.7 | 46.3 | 523 ms | 21.6 ms | 64.0 | 0 | 87,614 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,032.8 | 31.8 | 937 ms | 31.4 ms | 128.0 | 0 | 120,958 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 195.6 | 196.4 | 80 ms | 5.1 ms | 1.0 | 1 | 5,869 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 334.6 | 168.5 | 107 ms | 5.9 ms | 2.0 | 2 | 10,038 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 550.2 | 140.7 | 143 ms | 7.1 ms | 4.0 | 4 | 16,503 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 854.3 | 108.0 | 213 ms | 9.3 ms | 8.0 | 0 | 25,624 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,317.3 | 83.1 | 296 ms | 12.0 ms | 16.0 | 0 | 39,520 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,918.3 | 60.9 | 576 ms | 16.4 ms | 32.0 | 0 | 57,536 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,703.2 | 42.7 | 771 ms | 23.4 ms | 64.0 | 0 | 81,082 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 191.3 | 192.4 | 123 ms | 5.2 ms | 1.0 | 1 | 5,737 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 327.0 | 164.6 | 118 ms | 6.1 ms | 2.0 | 2 | 9,811 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 532.8 | 136.3 | 225 ms | 7.3 ms | 3.9 | 4 | 15,987 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 809.2 | 102.0 | 342 ms | 9.8 ms | 8.0 | 0 | 24,280 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,210.2 | 76.3 | 543 ms | 13.1 ms | 16.0 | 0 | 36,304 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,741.9 | 55.0 | 694 ms | 18.2 ms | 32.0 | 0 | 52,257 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,344.0 | 36.8 | 1,359 ms | 27.2 ms | 64.0 | 0 | 70,335 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 183.9 | 185.6 | 192 ms | 5.4 ms | 1.0 | 1 | 5,517 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 308.5 | 156.4 | 235 ms | 6.4 ms | 2.0 | 2 | 9,255 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 494.5 | 125.4 | 243 ms | 8.0 ms | 4.0 | 4 | 14,833 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 736.7 | 92.6 | 582 ms | 10.8 ms | 8.0 | 0 | 22,112 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,062.0 | 66.7 | 1,009 ms | 15.0 ms | 16.0 | 0 | 31,856 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,438.9 | 45.3 | 1,782 ms | 22.1 ms | 32.0 | 0 | 43,171 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 169.9 | 172.4 | 323 ms | 5.8 ms | 1.0 | 1 | 5,096 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 279.6 | 142.8 | 383 ms | 7.0 ms | 2.0 | 2 | 8,387 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 442.2 | 110.7 | 571 ms | 9.0 ms | 4.0 | 0 | 13,264 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 618.2 | 77.5 | 960 ms | 12.9 ms | 8.0 | 0 | 18,552 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 851.4 | 53.1 | 1,552 ms | 18.8 ms | 16.0 | 0 | 25,552 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 147.5 | 152.5 | 644 ms | 6.6 ms | 1.0 | 1 | 4,426 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 237.1 | 119.2 | 881 ms | 8.4 ms | 2.0 | 0 | 7,110 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 356.8 | 89.7 | 1,214 ms | 11.2 ms | 4.0 | 0 | 10,709 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 468.8 | 59.1 | 2,051 ms | 16.9 ms | 8.0 | 0 | 14,063 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 314,880 | 1 | Valid | 139.8 | 145.7 | 797 ms | 6.9 ms | 1.0 | 1 | 4,192 | 30.0 s |
| Sustained decode | 314,880 | 2 | Valid | 222.8 | 112.2 | 1,150 ms | 8.9 ms | 2.0 | 0 | 6,684 | 30.0 s |
| Sustained decode | 314,880 | 4 | Valid | 334.4 | 84.0 | 1,672 ms | 11.9 ms | 4.0 | 0 | 10,034 | 30.0 s |
| Sustained decode | 314,880 | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 314,880 | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 314,880 | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 314,880 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 314,880 | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.