Speed test matrix
command-a-plus-05-2026-w4a4 on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev638+g52be12cfa · NVFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- command-a-plus
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-cohere-vllm/bin/vllm" serve CohereLabs/command-a-plus-05-2026-w4a4 --served-model-name command-a-plus --tensor-parallel-size 2 --reasoning-parser cohere_command4 --enable-auto-tool-choice --tool-call-parser cohere_command4 --max-model-len 131072 --gpu-memory-utilization 0.862996 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context9,940823 ms TTFT
16K context9,6291,675 ms TTFT
32K context9,3583,422 ms TTFT
64K context8,7627,282 ms TTFT
123K context7,65815,985 ms TTFT
128K context7,60216,756 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 145.058 ms TTFT | 243.980 ms TTFT | 387.587 ms TTFT | 598.2159 ms TTFT | 926.1197 ms TTFT | 1,384.9329 ms TTFT | 1,950.9569 ms TTFT | 2,814.9865 ms TTFT |
| 16K | 144.290 ms TTFT | 242.5103 ms TTFT | 386.5154 ms TTFT | 593.0222 ms TTFT | 913.1300 ms TTFT | 1,357.1514 ms TTFT | 1,873.0775 ms TTFT | 2,660.21,540 ms TTFT |
| 32K | 142.8102 ms TTFT | 239.3120 ms TTFT | 379.5216 ms TTFT | 576.6310 ms TTFT | 879.3465 ms TTFT | 1,305.3759 ms TTFT | 1,737.31,390 ms TTFT | Over capacity |
| 64K | 139.8163 ms TTFT | 234.4277 ms TTFT | 369.7371 ms TTFT | 554.3624 ms TTFT | 829.71,007 ms TTFT | 1,163.21,533 ms TTFT | Over capacity | Over capacity |
| 123K | 134.4292 ms TTFT | 224.7387 ms TTFT | 349.2620 ms TTFT | 517.6975 ms TTFT | 735.01,599 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,940.0 | — | 823 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,629.0 | — | 1,675 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,358.0 | — | 3,422 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,762.0 | — | 7,282 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 7,658.0 | — | 15,985 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,602.0 | — | 16,756 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 145.0 | 145.6 | 58 ms | 6.9 ms | 1.0 | 1 | 4,348 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 243.9 | 124.7 | 80 ms | 8.0 ms | 2.0 | 2 | 7,315 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 387.5 | 97.6 | 87 ms | 10.2 ms | 4.0 | 0 | 11,632 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 598.2 | 76.1 | 159 ms | 13.1 ms | 8.0 | 0 | 17,952 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 926.1 | 58.8 | 197 ms | 17.0 ms | 16.0 | 0 | 27,774 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,384.9 | 44.0 | 329 ms | 22.7 ms | 32.0 | 0 | 41,535 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,950.9 | 30.9 | 569 ms | 32.4 ms | 64.0 | 0 | 58,494 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,814.9 | 22.3 | 865 ms | 44.9 ms | 128.0 | 0 | 84,478 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 144.2 | 145.2 | 90 ms | 6.9 ms | 1.0 | 1 | 4,325 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 242.5 | 124.4 | 103 ms | 8.0 ms | 2.0 | 2 | 7,274 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 386.5 | 97.1 | 154 ms | 10.3 ms | 4.0 | 0 | 11,592 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 593.0 | 75.2 | 222 ms | 13.3 ms | 8.0 | 0 | 17,784 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 913.1 | 58.1 | 300 ms | 17.2 ms | 16.0 | 0 | 27,390 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,357.1 | 43.0 | 514 ms | 23.2 ms | 32.0 | 0 | 40,704 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,873.0 | 29.6 | 775 ms | 33.8 ms | 64.0 | 0 | 56,191 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,660.2 | 21.1 | 1,540 ms | 47.5 ms | 128.0 | 0 | 79,743 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 142.8 | 143.6 | 102 ms | 7.0 ms | 1.0 | 1 | 4,282 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 239.3 | 128.7 | 120 ms | 7.8 ms | 2.0 | 2 | 7,180 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 379.5 | 95.4 | 216 ms | 10.5 ms | 4.0 | 0 | 11,388 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 576.6 | 73.1 | 310 ms | 13.7 ms | 8.0 | 0 | 17,296 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 879.3 | 55.8 | 465 ms | 17.9 ms | 16.0 | 0 | 26,367 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,305.3 | 41.2 | 759 ms | 24.3 ms | 32.0 | 0 | 39,135 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,737.3 | 27.4 | 1,390 ms | 36.5 ms | 64.0 | 0 | 52,094 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 139.8 | 141.1 | 163 ms | 7.1 ms | 1.0 | 1 | 4,193 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 234.4 | 117.7 | 277 ms | 8.5 ms | 2.0 | 0 | 7,036 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 369.7 | 93.0 | 371 ms | 10.8 ms | 4.0 | 0 | 11,092 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 554.3 | 70.2 | 624 ms | 14.3 ms | 8.0 | 0 | 16,624 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 829.7 | 52.5 | 1,007 ms | 19.1 ms | 16.0 | 0 | 24,880 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,163.2 | 36.8 | 1,533 ms | 27.2 ms | 32.0 | 0 | 34,880 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 134.4 | 137.3 | 292 ms | 7.3 ms | 1.0 | 1 | 4,034 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 224.7 | 112.8 | 387 ms | 8.9 ms | 2.0 | 0 | 6,740 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 349.2 | 87.7 | 620 ms | 11.4 ms | 4.0 | 0 | 10,476 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 517.6 | 65.3 | 975 ms | 15.3 ms | 8.0 | 0 | 15,536 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 735.0 | 46.4 | 1,599 ms | 21.5 ms | 16.0 | 0 | 22,048 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.