Speed test matrix
Qwen3.5-122B-A10B-NVFP4 on CMP 170HX 64GB ×2
vllm · 0.32.0rc1.dev249+g18305948a · NVFP46 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- GPU count
- 2
- VRAM
- 64 GB (2 × 32 GB)
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead. Layout: PP2 (stock vLLM flag)
# 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve nvidia/Qwen3.5-122B-A10B-NVFP4 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only --hf-overrides {"text_config":{"tie_word_embeddings":false}} --speculative-config {"method":"mtp","num_speculative_tokens":2}Prefill
prompt tok/s by context length8K context4,4201,842 ms TTFT
16K context4,4083,656 ms TTFT
32K context3,8588,316 ms TTFT
64K context3,10520,622 ms TTFT
128K context2,18758,478 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 86.2156 ms TTFT | 166.7151 ms TTFT | 233.5305 ms TTFT | 410.0398 ms TTFT | 650.7574 ms TTFT | 956.3797 ms TTFT | 1,066.4Requests queued11,795 ms TTFT | Failed |
| 16K | 88.31,995 ms TTFT | 164.22,758 ms TTFT | 268.24,280 ms TTFT | 394.16,979 ms TTFT | 636.813,068 ms TTFT | 961.224,972 ms TTFT | Looped | Over capacity |
| 32K | 86.32,437 ms TTFT | 161.13,542 ms TTFT | 225.25,371 ms TTFT | 379.78,519 ms TTFT | 595.915,234 ms TTFT | 879.928,196 ms TTFT | Over capacity | Over capacity |
| 64K | 86.62,726 ms TTFT | 124.54,133 ms TTFT | 215.96,262 ms TTFT | 355.89,719 ms TTFT | 544.116,387 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 85.04,866 ms TTFT | 121.87,031 ms TTFT | 207.310,723 ms TTFT | 318.216,150 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 81.210,654 ms TTFT | 123.514,912 ms TTFT | 178.822,020 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Failed: requests to the server failed during this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,420.0 | — | 1,842 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 4,408.0 | — | 3,656 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,858.0 | — | 8,316 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,105.0 | — | 20,622 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,187.0 | — | 58,478 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 86.2 | 86.4 | 156 ms | 11.6 ms | 1.0 | 0 | 2,584 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 166.7 | 83.2 | 151 ms | 12.0 ms | 2.0 | 0 | 5,000 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 233.5 | 58.5 | 305 ms | 17.1 ms | 4.0 | 0 | 7,001 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 410.0 | 51.8 | 398 ms | 19.3 ms | 8.0 | 0 | 12,298 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 650.7 | 41.5 | 574 ms | 24.1 ms | 16.0 | 0 | 19,521 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 956.3 | 30.9 | 797 ms | 32.4 ms | 32.0 | 0 | 28,642 | 29.9 s |
| Sustained decode | 0 | 64 | Requests queued | 1,066.4 | 24.7 | 11,795 ms | 40.4 ms | 43.8 | 473 | 31,974 | 30.0 s |
| Sustained decode | 0 | 128 | Failed | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 88.3 | 88.6 | 1,995 ms | 11.3 ms | 1.0 | 0 | 2,648 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 164.2 | 81.2 | 2,758 ms | 12.3 ms | 2.0 | 0 | 4,925 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 268.2 | 63.3 | 4,280 ms | 15.8 ms | 4.0 | 0 | 8,041 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 394.1 | 44.2 | 6,979 ms | 22.6 ms | 8.0 | 0 | 11,818 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 636.8 | 31.2 | 13,068 ms | 32.1 ms | 16.0 | 0 | 19,088 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 961.2 | 19.4 | 24,972 ms | 51.7 ms | 32.0 | 0 | 28,811 | 30.0 s |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 86.3 | 87.1 | 2,437 ms | 11.5 ms | 1.0 | 0 | 2,586 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 161.1 | 79.3 | 3,542 ms | 12.6 ms | 2.0 | 0 | 4,827 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 225.2 | 54.6 | 5,371 ms | 18.3 ms | 4.0 | 0 | 6,753 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 379.7 | 42.0 | 8,519 ms | 23.8 ms | 8.0 | 0 | 11,388 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 595.9 | 28.4 | 15,234 ms | 35.2 ms | 16.0 | 0 | 17,866 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 879.9 | 17.0 | 28,196 ms | 58.9 ms | 32.0 | 0 | 26,376 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 86.6 | 87.1 | 2,726 ms | 11.5 ms | 1.0 | 0 | 2,598 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 124.5 | 61.7 | 4,133 ms | 16.2 ms | 2.0 | 0 | 3,733 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 215.9 | 51.5 | 6,262 ms | 19.4 ms | 4.0 | 0 | 6,475 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 355.8 | 39.8 | 9,719 ms | 25.1 ms | 8.0 | 0 | 10,669 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 544.1 | 26.1 | 16,387 ms | 38.4 ms | 16.0 | 0 | 16,307 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 85.0 | 85.6 | 4,866 ms | 11.7 ms | 1.0 | 0 | 2,548 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 121.8 | 59.4 | 7,031 ms | 16.9 ms | 2.0 | 0 | 3,652 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 207.3 | 48.2 | 10,723 ms | 20.8 ms | 4.0 | 0 | 6,217 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 318.2 | 31.7 | 16,150 ms | 31.5 ms | 8.0 | 0 | 9,543 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 81.2 | 82.2 | 10,654 ms | 12.2 ms | 1.0 | 0 | 2,436 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 123.5 | 57.9 | 14,912 ms | 17.3 ms | 2.0 | 0 | 3,704 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 178.8 | 36.2 | 22,020 ms | 27.8 ms | 4.0 | 0 | 5,362 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Failed: requests to the server failed during this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.