Speed test matrix
ZAYA1-8B-legacy on H200 NVL 141GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- zaya
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-zyphra-sglang/bin/sglang" serve --model-path Zyphra/ZAYA1-8B-legacy --served-model-name zaya --tp 1 --reasoning-parser qwen3 --context-length 131072 --mem-fraction-static 0.859496 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context34.1K240 ms TTFT
16K context45.6K356 ms TTFT
32K context43.5K743 ms TTFT
64K context37.3K1,728 ms TTFT
123K context28.2K4,392 ms TTFT
128K context27.4K4,700 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 75.689 ms TTFT | Failed | Failed | Failed | Failed | Failed | Failed | Over capacity |
| 16K | 74.6208 ms TTFT | Failed | Failed | Failed | Failed | Failed | Failed | Over capacity |
| 32K | 73.3266 ms TTFT | Failed | Failed | Failed | Failed | Failed | Over capacity | Over capacity |
| 64K | 71.4384 ms TTFT | Failed | Failed | Failed | Failed | Over capacity | Over capacity | Over capacity |
| 123K | 68.5339 ms TTFT | Failed | Failed | Failed | Over capacity | Over capacity | Over capacity | Over capacity |
Failed: requests to the server failed during this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34.1K | — | 240 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 45.6K | — | 356 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 43.5K | — | 743 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 37.3K | — | 1,728 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 28.2K | — | 4,392 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 27.4K | — | 4,700 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 75.6 | 75.6 | 89 ms | 13.2 ms | 1.0 | 0 | 2,267 | 30.0 s |
| Sustained decode | 8K | 2 | Failed | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 8K | 8 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 8K | 16 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 8K | 32 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 8K | 64 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 74.6 | 74.6 | 208 ms | 13.4 ms | 1.0 | 0 | 2,237 | 30.0 s |
| Sustained decode | 16K | 2 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 16K | 4 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 16K | 8 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 16K | 16 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 16K | 32 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 16K | 64 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 73.3 | 73.3 | 266 ms | 13.6 ms | 1.0 | 0 | 2,200 | 30.0 s |
| Sustained decode | 32K | 2 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 32K | 4 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 32K | 8 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 32K | 16 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 32K | 32 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 71.4 | 71.4 | 384 ms | 14.0 ms | 1.0 | 0 | 2,142 | 30.0 s |
| Sustained decode | 64K | 2 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 64K | 4 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 64K | 8 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 64K | 16 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 68.5 | 68.5 | 339 ms | 14.6 ms | 1.0 | 0 | 2,054 | 30.0 s |
| Sustained decode | 123K | 2 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 123K | 4 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 123K | 8 | Failed | — | — | — | — | — | 0 | 0 | 1.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Failed: requests to the server failed during this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.