Speed test matrix
Trinity-Large-Thinking on H200 SXM 141GB ×8
vllm · 0.29.0 · BF167 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- arcee-ai/Trinity-Large-Thinking
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve arcee-ai/Trinity-Large-Thinking --served-model-name arcee-ai/Trinity-Large-Thinking --trust-remote-code --tensor-parallel-size 8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 524288 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --hf-overrides '{"max_position_embeddings":524288}' --kernel-config.enable_flashinfer_autotune=FalsePrefill
prompt tok/s by context length8K context43.1K190 ms TTFT
16K context46.7K348 ms TTFT
32K context46.8K692 ms TTFT
64K context43.6K1,480 ms TTFT
128K context39.3K3,287 ms TTFT
256K context32.3K7,974 ms TTFT
507K context23.9K21,394 ms TTFT
512K context23.4K21,999 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 174.348 ms TTFT | 316.162 ms TTFT | 560.359 ms TTFT | 900.1117 ms TTFT | 1,346.0207 ms TTFT | 2,222.5377 ms TTFT | 3,202.0666 ms TTFT | 4,631.01,330 ms TTFT |
| 16K | 173.173 ms TTFT | 314.087 ms TTFT | 556.787 ms TTFT | 895.0216 ms TTFT | 1,333.7383 ms TTFT | 2,191.5658 ms TTFT | 3,120.61,244 ms TTFT | 4,491.92,360 ms TTFT |
| 32K | 170.8123 ms TTFT | 310.2141 ms TTFT | 549.1135 ms TTFT | 878.8364 ms TTFT | 1,303.7641 ms TTFT | 2,114.91,202 ms TTFT | 2,954.82,383 ms TTFT | Over capacity |
| 64K | 166.4231 ms TTFT | 302.4244 ms TTFT | 535.3252 ms TTFT | 848.8772 ms TTFT | 1,250.81,402 ms TTFT | 1,969.42,599 ms TTFT | Over capacity | Over capacity |
| 128K | 158.1457 ms TTFT | 287.7509 ms TTFT | 509.7492 ms TTFT | Looped | 1,153.02,985 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 143.1967 ms TTFT | 262.41,060 ms TTFT | 472.91,730 ms TTFT | 714.93,646 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 127.52,356 ms TTFT | Looped | 418.05,224 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 43.1K | — | 190 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 46.7K | — | 348 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 46.8K | — | 692 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 43.6K | — | 1,480 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 39.3K | — | 3,287 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 32.3K | — | 7,974 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 23.9K | — | 21,394 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 23.4K | — | 21,999 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 174.3 | 174.6 | 48 ms | 5.7 ms | 1.0 | 1 | 5,227 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 316.1 | 158.3 | 62 ms | 6.3 ms | 2.0 | 2 | 9,482 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 560.3 | 140.4 | 59 ms | 7.1 ms | 4.0 | 4 | 16,808 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 900.1 | 112.5 | 117 ms | 8.9 ms | 8.0 | 0 | 27,000 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,346.0 | 84.2 | 207 ms | 11.9 ms | 16.0 | 0 | 40,368 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,222.5 | 69.6 | 377 ms | 14.4 ms | 32.0 | 0 | 66,656 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,202.0 | 50.2 | 666 ms | 19.9 ms | 64.0 | 0 | 96,064 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,631.0 | 36.6 | 1,330 ms | 27.3 ms | 128.0 | 0 | 138,880 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 173.1 | 173.6 | 73 ms | 5.8 ms | 1.0 | 1 | 5,191 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 314.0 | 157.5 | 87 ms | 6.3 ms | 2.0 | 2 | 9,420 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 556.7 | 139.5 | 87 ms | 7.2 ms | 4.0 | 4 | 16,701 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 895.0 | 111.8 | 216 ms | 8.9 ms | 8.0 | 0 | 26,848 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,333.7 | 83.4 | 383 ms | 12.0 ms | 16.0 | 0 | 40,000 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,191.5 | 68.5 | 658 ms | 14.6 ms | 32.0 | 0 | 65,728 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,120.6 | 48.9 | 1,244 ms | 20.5 ms | 64.0 | 0 | 93,569 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,491.9 | 35.3 | 2,360 ms | 28.4 ms | 128.0 | 0 | 134,784 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 170.8 | 171.6 | 123 ms | 5.8 ms | 1.0 | 1 | 5,122 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 310.2 | 156.0 | 141 ms | 6.4 ms | 2.0 | 2 | 9,304 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 549.1 | 137.9 | 135 ms | 7.2 ms | 4.0 | 4 | 16,471 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 878.8 | 109.8 | 364 ms | 9.1 ms | 8.0 | 0 | 26,360 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,303.7 | 81.5 | 641 ms | 12.3 ms | 16.0 | 0 | 39,104 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,114.9 | 66.1 | 1,202 ms | 15.1 ms | 32.0 | 0 | 63,424 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,954.8 | 46.3 | 2,383 ms | 21.6 ms | 64.0 | 0 | 88,640 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 166.4 | 167.9 | 231 ms | 6.0 ms | 1.0 | 1 | 4,991 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 302.4 | 152.7 | 244 ms | 6.5 ms | 2.0 | 2 | 9,072 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 535.3 | 135.0 | 252 ms | 7.4 ms | 4.0 | 4 | 16,060 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 848.8 | 106.2 | 772 ms | 9.4 ms | 8.0 | 0 | 25,464 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,250.8 | 78.2 | 1,402 ms | 12.8 ms | 16.0 | 0 | 37,520 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,969.4 | 61.5 | 2,599 ms | 16.3 ms | 32.0 | 0 | 59,072 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 158.1 | 161.2 | 457 ms | 6.2 ms | 1.0 | 1 | 4,743 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 287.7 | 147.0 | 509 ms | 6.8 ms | 1.9 | 2 | 8,632 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 509.7 | 129.6 | 492 ms | 7.7 ms | 4.0 | 4 | 15,291 | 30.0 s |
| Sustained decode | 128K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 16 | Valid | 1,153.0 | 72.2 | 2,985 ms | 13.9 ms | 16.0 | 0 | 34,592 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 143.1 | 149.4 | 967 ms | 6.7 ms | 1.0 | 1 | 4,294 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 262.4 | 137.3 | 1,060 ms | 7.3 ms | 1.9 | 2 | 7,869 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 472.9 | 120.2 | 1,730 ms | 8.3 ms | 3.9 | 2 | 14,186 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 714.9 | 89.8 | 3,646 ms | 11.1 ms | 8.0 | 0 | 21,440 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 127.5 | 128.4 | 2,356 ms | 7.8 ms | 1.0 | 1 | 3,658 | 28.7 s |
| Sustained decode | 507K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 507K | 4 | Valid | 418.0 | 105.5 | 5,224 ms | 9.5 ms | 4.0 | 1 | 12,538 | 30.0 s |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.