Speed test matrix
command-a-plus-05-2026-bf16 on H200 SXM 141GB ×4
vllm · 0.28.1rc1.dev455+g22f931fe2 · BF166 contexts (8K–194,880) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 194,880, 200,000
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev455+g22f931fe2
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
22f931fe2177
Run
- Served model
- command-a-plus
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-cohere-vllm-pr-54479/bin/vllm" serve CohereLabs/command-a-plus-05-2026-bf16 --served-model-name command-a-plus --tensor-parallel-size 4 --reasoning-parser cohere_command4 --enable-auto-tool-choice --tool-call-parser cohere_command4 --max-model-len 200000 --gpu-memory-utilization 0.877427 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16
Prefill
prompt tok/s by context length8K context23.2K353 ms TTFT
16K context24.6K656 ms TTFT
32K context23.6K1,359 ms TTFT
64K context22.2K2,876 ms TTFT
128K context18.9K6,734 ms TTFT
194,880 context16.5K11,440 ms TTFT
200,000 context16.5K11,745 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | Looped | Looped | Looped | 692.0197 ms TTFT | 1,057.2245 ms TTFT | Looped | 2,837.1648 ms TTFT | 4,354.61,145 ms TTFT |
| 16K | 189.3103 ms TTFT | 307.4120 ms TTFT | 454.0159 ms TTFT | 688.6223 ms TTFT | Looped | Looped | Looped | Over capacity |
| 32K | 187.5144 ms TTFT | 305.1168 ms TTFT | 450.0256 ms TTFT | 686.4400 ms TTFT | Looped | 1,665.2992 ms TTFT | Over capacity | Over capacity |
| 64K | 183.7245 ms TTFT | 298.7321 ms TTFT | 440.8462 ms TTFT | Looped | 991.41,227 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 176.1456 ms TTFT | 287.2450 ms TTFT | 424.2802 ms TTFT | 637.31,276 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 194,880 | 169.4669 ms TTFT | 276.0724 ms TTFT | 412.81,245 ms TTFT | 615.42,194 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 23.2K | — | 353 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 24.6K | — | 656 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 23.6K | — | 1,359 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22.2K | — | 2,876 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.9K | — | 6,734 ms | — | — | 1 | — | — |
| Prefill | 194,880 | 1 | Valid | 16.5K | — | 11,440 ms | — | — | 1 | — | — |
| Prefill | 200,000 | 1 | Valid | 16.5K | — | 11,745 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Valid | 692.0 | 87.6 | 197 ms | 11.4 ms | 8.0 | 0 | 20,760 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,057.2 | 67.3 | 245 ms | 14.9 ms | 16.0 | 0 | 31,711 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 2,837.1 | 45.4 | 648 ms | 22.0 ms | 64.0 | 0 | 85,246 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,354.6 | 34.6 | 1,145 ms | 28.9 ms | 128.0 | 0 | 130,684 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 189.3 | 190.1 | 103 ms | 5.3 ms | 1.0 | 1 | 5,680 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 307.4 | 155.0 | 120 ms | 6.5 ms | 2.0 | 2 | 9,220 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 454.0 | 113.8 | 159 ms | 8.8 ms | 4.0 | 0 | 13,616 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 688.6 | 87.1 | 223 ms | 11.5 ms | 8.0 | 0 | 20,655 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 187.5 | 188.7 | 144 ms | 5.3 ms | 1.0 | 1 | 5,626 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 305.1 | 153.8 | 168 ms | 6.5 ms | 2.0 | 2 | 9,152 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 450.0 | 113.0 | 256 ms | 8.9 ms | 4.0 | 0 | 13,500 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 686.4 | 86.6 | 400 ms | 11.5 ms | 8.0 | 0 | 20,592 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 1,665.2 | 52.9 | 992 ms | 18.9 ms | 32.0 | 0 | 49,952 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 183.7 | 185.6 | 245 ms | 5.4 ms | 1.0 | 1 | 5,512 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 298.7 | 151.7 | 321 ms | 6.6 ms | 2.0 | 2 | 8,960 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 440.8 | 110.7 | 462 ms | 9.0 ms | 4.0 | 0 | 13,224 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 991.4 | 62.7 | 1,227 ms | 16.0 ms | 16.0 | 0 | 29,743 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 176.1 | 179.5 | 456 ms | 5.6 ms | 1.0 | 1 | 5,284 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 287.2 | 146.9 | 450 ms | 6.8 ms | 2.0 | 2 | 8,618 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 424.2 | 106.7 | 802 ms | 9.4 ms | 4.0 | 0 | 12,726 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 637.3 | 80.5 | 1,276 ms | 12.4 ms | 8.0 | 0 | 19,114 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 194,880 | 1 | Valid | 169.4 | 173.9 | 669 ms | 5.8 ms | 1.0 | 1 | 5,081 | 30.0 s |
| Sustained decode | 194,880 | 2 | Valid | 276.0 | 143.8 | 724 ms | 7.0 ms | 2.0 | 2 | 8,278 | 30.0 s |
| Sustained decode | 194,880 | 4 | Valid | 412.8 | 103.8 | 1,245 ms | 9.6 ms | 4.0 | 0 | 12,384 | 30.0 s |
| Sustained decode | 194,880 | 8 | Valid | 615.4 | 77.7 | 2,194 ms | 12.9 ms | 8.0 | 0 | 18,456 | 30.0 s |
| Sustained decode | 194,880 | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 194,880 | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 194,880 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 194,880 | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.