Speed test matrix
ZAYA1-74B-preview-legacy on H200 NVL 141GB
vllm · 0.1.dev17064+g8df704c52 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev17064+g8df704c52
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/Zyphra/vllm
- Build
- cuda-13.2
- Commit
8df704c52588
Run
- Served model
- zaya
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-zyphra-legacy-vllm/bin/vllm" serve Zyphra/ZAYA1-74B-preview-legacy --served-model-name zaya --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser zaya_xml --max-model-len 262144 --gpu-memory-utilization 0.862309 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-dtype float32 --dtype bfloat16 --data-parallel-size 2 --enable-expert-parallel
Prefill
prompt tok/s by context length8K context15.4K534 ms TTFT
16K context15.3K1,060 ms TTFT
32K context20.4K1,583 ms TTFT
64K context17.9K3,604 ms TTFT
128K context15.2K8,456 ms TTFT
251K context10.7K23,612 ms TTFT
256K context10.4K24,762 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 46.9347 ms TTFT | 92.4Requests queued503 ms TTFT | 170.6Requests queued936 ms TTFT | 304.5Requests queued1,395 ms TTFT | 542.8Requests queued2,674 ms TTFT | 950.8Requests queued4,440 ms TTFT | 1,582.1Requests queued7,786 ms TTFT | Over capacity |
| 16K | 46.8687 ms TTFT | 93.1Requests queued979 ms TTFT | 168.0Requests queued1,605 ms TTFT | 299.3Requests queued2,754 ms TTFT | 522.0Requests queued4,333 ms TTFT | 911.6Requests queued7,313 ms TTFT | Over capacity | Over capacity |
| 32K | 46.31,483 ms TTFT | 92.5Requests queued1,828 ms TTFT | 166.9Requests queued2,770 ms TTFT | 295.4Requests queued4,610 ms TTFT | 510.5Requests queued8,622 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 45.73,262 ms TTFT | 90.7Requests queued4,150 ms TTFT | 162.9Requests queued6,333 ms TTFT | 150.3Requests queued8,104 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 44.58,325 ms TTFT | 88.2Requests queued9,997 ms TTFT | 156.8Requests queued15,407 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 42.323,673 ms TTFT | 83.8Requests queued27,165 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.4K | — | 534 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.3K | — | 1,060 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 20.4K | — | 1,583 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 17.9K | — | 3,604 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15.2K | — | 8,456 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.7K | — | 23,612 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.4K | — | 24,762 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 46.9 | 47.0 | 347 ms | 21.3 ms | 1.0 | 0 | 1,408 | 30.0 s |
| Sustained decode | 8K | 2 | Requests queued | 92.4 | 47.1 | 503 ms | 21.2 ms | 1.0 | 2 | 2,772 | 30.0 s |
| Sustained decode | 8K | 4 | Requests queued | 170.6 | 42.6 | 936 ms | 23.5 ms | 2.0 | 0 | 5,116 | 30.0 s |
| Sustained decode | 8K | 8 | Requests queued | 304.5 | 37.9 | 1,395 ms | 26.4 ms | 4.0 | 0 | 9,128 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 542.8 | 33.6 | 2,674 ms | 29.7 ms | 8.0 | 0 | 16,288 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 950.8 | 28.9 | 4,440 ms | 34.6 ms | 16.0 | 0 | 28,512 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,582.1 | 23.4 | 7,786 ms | 42.8 ms | 32.0 | 0 | 47,468 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 46.8 | 46.8 | 687 ms | 21.4 ms | 1.0 | 0 | 1,403 | 30.0 s |
| Sustained decode | 16K | 2 | Requests queued | 93.1 | 46.5 | 979 ms | 21.5 ms | 1.0 | 2 | 2,740 | 29.4 s |
| Sustained decode | 16K | 4 | Requests queued | 168.0 | 41.9 | 1,605 ms | 23.9 ms | 2.0 | 0 | 5,044 | 30.0 s |
| Sustained decode | 16K | 8 | Requests queued | 299.3 | 37.2 | 2,754 ms | 26.9 ms | 4.0 | 0 | 8,984 | 30.0 s |
| Sustained decode | 16K | 16 | Requests queued | 522.0 | 31.9 | 4,333 ms | 31.4 ms | 8.0 | 0 | 15,648 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 911.6 | 26.7 | 7,313 ms | 37.5 ms | 16.0 | 0 | 27,328 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 46.3 | 46.4 | 1,483 ms | 21.5 ms | 1.0 | 0 | 1,389 | 30.0 s |
| Sustained decode | 32K | 2 | Requests queued | 92.5 | 46.3 | 1,828 ms | 21.6 ms | 1.0 | 0 | 2,773 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 166.9 | 41.4 | 2,770 ms | 24.2 ms | 2.0 | 0 | 5,006 | 30.0 s |
| Sustained decode | 32K | 8 | Requests queued | 295.4 | 36.0 | 4,610 ms | 27.7 ms | 4.0 | 0 | 8,864 | 30.0 s |
| Sustained decode | 32K | 16 | Requests queued | 510.5 | 29.7 | 8,622 ms | 33.7 ms | 8.0 | 0 | 15,312 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 45.7 | 45.7 | 3,262 ms | 21.9 ms | 1.0 | 1 | 1,372 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 90.7 | 45.5 | 4,150 ms | 22.0 ms | 1.0 | 2 | 2,720 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 162.9 | 39.7 | 6,333 ms | 25.2 ms | 2.0 | 4 | 4,889 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 150.3 | 32.3 | 8,104 ms | 31.0 ms | 3.6 | 8 | 4,466 | 29.7 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 44.5 | 44.6 | 8,325 ms | 22.4 ms | 1.0 | 0 | 1,334 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 88.2 | 44.1 | 9,997 ms | 22.7 ms | 1.0 | 2 | 2,646 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 156.8 | 37.2 | 15,407 ms | 26.9 ms | 2.0 | 4 | 4,704 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 42.3 | 42.5 | 23,673 ms | 23.5 ms | 1.0 | 0 | 1,267 | 30.0 s |
| Sustained decode | 251K | 2 | Requests queued | 83.8 | 41.7 | 27,165 ms | 24.0 ms | 1.0 | 2 | 2,514 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.