Speed test matrix
Trinity-Large-Preview on H200 SXM 141GB ×8
vllm · 0.29.0 · BF167 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- arcee-ai/Trinity-Large-Preview
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve arcee-ai/Trinity-Large-Preview --served-model-name arcee-ai/Trinity-Large-Preview --trust-remote-code --tensor-parallel-size 8 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 524288 --gpu-memory-utilization 0.885586 --host 0.0.0.0 --port 8000 --api-key <redacted> --hf-overrides '{"max_position_embeddings":524288}' --kernel-config.enable_flashinfer_autotune=FalsePrefill
prompt tok/s by context length8K context42.9K191 ms TTFT
16K context46.6K349 ms TTFT
32K context46.2K700 ms TTFT
64K context43.5K1,484 ms TTFT
128K context38.8K3,322 ms TTFT
256K context32.2K8,024 ms TTFT
507K context24K21,319 ms TTFT
512K context23.1K22,285 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 174.348 ms TTFT | 315.553 ms TTFT | 558.654 ms TTFT | 898.5118 ms TTFT | Looped | Looped | Looped | Looped |
| 16K | 173.174 ms TTFT | 313.881 ms TTFT | Looped | 892.2219 ms TTFT | Looped | Looped | 3,152.11,181 ms TTFT | Looped |
| 32K | 170.7124 ms TTFT | 309.4139 ms TTFT | Looped | 871.7362 ms TTFT | Looped | Looped | 2,961.52,320 ms TTFT | Over capacity |
| 64K | 166.3234 ms TTFT | 302.2252 ms TTFT | 534.7252 ms TTFT | 853.7739 ms TTFT | 1,247.41,326 ms TTFT | 1,929.72,497 ms TTFT | Over capacity | Over capacity |
| 128K | 158.0486 ms TTFT | 287.6507 ms TTFT | 509.6498 ms TTFT | 798.71,623 ms TTFT | 1,149.42,912 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 143.01,020 ms TTFT | 262.41,082 ms TTFT | Looped | 624.83,640 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 127.52,182 ms TTFT | 232.93,106 ms TTFT | 418.94,690 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 42.9K | — | 191 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 46.6K | — | 349 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 46.2K | — | 700 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 43.5K | — | 1,484 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 38.8K | — | 3,322 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 32.2K | — | 8,024 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 24K | — | 21,319 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 23.1K | — | 22,285 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 174.3 | 174.6 | 48 ms | 5.7 ms | 1.0 | 1 | 5,230 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 315.5 | 158.1 | 53 ms | 6.3 ms | 2.0 | 2 | 9,463 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 558.6 | 140.1 | 54 ms | 7.1 ms | 4.0 | 4 | 16,757 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 898.5 | 112.3 | 118 ms | 8.9 ms | 8.0 | 0 | 26,949 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 173.1 | 173.6 | 74 ms | 5.8 ms | 1.0 | 1 | 5,192 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 313.8 | 157.5 | 81 ms | 6.4 ms | 2.0 | 2 | 9,414 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 892.2 | 111.6 | 219 ms | 9.0 ms | 8.0 | 0 | 26,766 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Valid | 3,152.1 | 49.2 | 1,181 ms | 20.3 ms | 64.0 | 0 | 94,524 | 30.0 s |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 170.7 | 171.5 | 124 ms | 5.8 ms | 1.0 | 1 | 5,120 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 309.4 | 155.6 | 139 ms | 6.4 ms | 2.0 | 2 | 9,282 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 871.7 | 109.1 | 362 ms | 9.2 ms | 8.0 | 0 | 26,150 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Valid | 2,961.5 | 46.5 | 2,320 ms | 21.5 ms | 64.0 | 0 | 88,830 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 166.3 | 167.9 | 234 ms | 6.0 ms | 1.0 | 1 | 4,988 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 302.2 | 152.7 | 252 ms | 6.5 ms | 2.0 | 2 | 9,063 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 534.7 | 134.9 | 252 ms | 7.4 ms | 4.0 | 4 | 16,040 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 853.7 | 106.8 | 739 ms | 9.4 ms | 8.0 | 0 | 25,610 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,247.4 | 78.1 | 1,326 ms | 12.8 ms | 16.0 | 0 | 37,421 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,929.7 | 60.4 | 2,497 ms | 16.5 ms | 32.0 | 0 | 57,889 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 158.0 | 161.2 | 486 ms | 6.2 ms | 1.0 | 1 | 4,738 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 287.6 | 146.9 | 507 ms | 6.8 ms | 1.9 | 2 | 8,627 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 509.6 | 129.7 | 498 ms | 7.7 ms | 4.0 | 4 | 15,287 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 798.7 | 100.2 | 1,623 ms | 10.0 ms | 8.0 | 0 | 23,962 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,149.4 | 72.1 | 2,912 ms | 13.9 ms | 16.0 | 0 | 34,482 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 143.0 | 149.9 | 1,020 ms | 6.7 ms | 1.0 | 1 | 4,291 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 262.4 | 137.8 | 1,082 ms | 7.3 ms | 1.9 | 2 | 7,873 | 30.0 s |
| Sustained decode | 256K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 8 | Valid | 624.8 | 90.1 | 3,640 ms | 11.1 ms | 8.0 | 0 | 18,739 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 127.5 | 128.5 | 2,182 ms | 7.8 ms | 1.0 | 1 | 3,632 | 28.5 s |
| Sustained decode | 507K | 2 | Valid | 232.9 | 119.0 | 3,106 ms | 8.4 ms | 2.0 | 1 | 6,987 | 30.0 s |
| Sustained decode | 507K | 4 | Valid | 418.9 | 105.5 | 4,690 ms | 9.5 ms | 4.0 | 0 | 12,564 | 30.0 s |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.