Speed test matrix
Step-3.7-Flash-NVFP4 on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev638+g52be12cfa · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- step
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-stepfun-vllm/bin/vllm" serve stepfun-ai/Step-3.7-Flash-NVFP4 --served-model-name step --trust-remote-code --tensor-parallel-size 2 --reasoning-parser step3p5 --enable-auto-tool-choice --tool-call-parser step3p5 --max-model-len 262144 --gpu-memory-utilization 0.868586 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --disable-cascade-attn --kv-cache-dtype fp8 --async-scheduling
Prefill
prompt tok/s by context length8K context18.3K446 ms TTFT
16K context18.1K897 ms TTFT
32K context17.7K1,827 ms TTFT
64K context16.5K3,909 ms TTFT
128K context14.1K9,122 ms TTFT
251K context10.9K23,041 ms TTFT
256K context10.8K23,773 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 123.754 ms TTFT | 212.377 ms TTFT | 382.3109 ms TTFT | 657.298 ms TTFT | 1,084.8224 ms TTFT | 1,705.8327 ms TTFT | 2,569.2482 ms TTFT | 3,572.1903 ms TTFT |
| 16K | 122.989 ms TTFT | 211.8107 ms TTFT | 376.4142 ms TTFT | 642.3209 ms TTFT | 1,051.0266 ms TTFT | 1,605.0455 ms TTFT | 2,354.0746 ms TTFT | 3,183.01,358 ms TTFT |
| 32K | 122.0111 ms TTFT | 208.6134 ms TTFT | 367.5187 ms TTFT | 617.0270 ms TTFT | 974.6442 ms TTFT | 1,449.5729 ms TTFT | 2,036.51,336 ms TTFT | 2,616.32,473 ms TTFT |
| 64K | 119.9149 ms TTFT | 201.8235 ms TTFT | 349.8339 ms TTFT | 571.1543 ms TTFT | 863.6961 ms TTFT | 1,215.71,556 ms TTFT | 1,598.22,766 ms TTFT | Over capacity |
| 128K | 115.6285 ms TTFT | 190.3414 ms TTFT | 319.7644 ms TTFT | 495.4936 ms TTFT | 700.41,549 ms TTFT | 914.82,986 ms TTFT | Over capacity | Over capacity |
| 251K | 108.2601 ms TTFT | 171.8835 ms TTFT | 274.11,175 ms TTFT | 393.01,888 ms TTFT | 515.43,415 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 18.3K | — | 446 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18.1K | — | 897 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 17.7K | — | 1,827 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16.5K | — | 3,909 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.1K | — | 9,122 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.9K | — | 23,041 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.8K | — | 23,773 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 123.7 | 125.1 | 54 ms | 8.0 ms | 1.0 | 1 | 3,712 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 212.3 | 106.2 | 77 ms | 9.4 ms | 2.0 | 0 | 6,368 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 382.3 | 95.7 | 109 ms | 10.4 ms | 4.0 | 0 | 11,468 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 657.2 | 82.4 | 98 ms | 12.1 ms | 8.0 | 0 | 19,712 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,084.8 | 68.1 | 224 ms | 14.7 ms | 16.0 | 0 | 32,544 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,705.8 | 53.7 | 327 ms | 18.6 ms | 32.0 | 0 | 51,168 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,569.2 | 40.5 | 482 ms | 24.7 ms | 64.0 | 0 | 77,056 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,572.1 | 28.1 | 903 ms | 35.6 ms | 128.0 | 0 | 107,136 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 122.9 | 129.6 | 89 ms | 7.7 ms | 1.0 | 1 | 3,685 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 211.8 | 105.9 | 107 ms | 9.4 ms | 2.0 | 0 | 6,352 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 376.4 | 94.3 | 142 ms | 10.6 ms | 4.0 | 0 | 11,292 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 642.3 | 80.5 | 209 ms | 12.4 ms | 8.0 | 0 | 19,264 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,051.0 | 66.0 | 266 ms | 15.1 ms | 16.0 | 0 | 31,520 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,605.0 | 50.5 | 455 ms | 19.8 ms | 32.0 | 0 | 48,128 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,354.0 | 37.1 | 746 ms | 27.0 ms | 64.0 | 0 | 70,592 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,183.0 | 25.1 | 1,358 ms | 39.9 ms | 128.0 | 0 | 95,488 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 122.0 | 122.1 | 111 ms | 8.2 ms | 1.0 | 0 | 3,662 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 208.6 | 104.4 | 134 ms | 9.6 ms | 2.0 | 0 | 6,258 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 367.5 | 91.9 | 187 ms | 10.9 ms | 4.0 | 0 | 11,024 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 617.0 | 77.2 | 270 ms | 13.0 ms | 8.0 | 0 | 18,504 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 974.6 | 61.1 | 442 ms | 16.4 ms | 16.0 | 0 | 29,232 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,449.5 | 45.5 | 729 ms | 22.0 ms | 32.0 | 0 | 43,488 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,036.5 | 31.9 | 1,336 ms | 31.3 ms | 64.0 | 0 | 61,056 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,616.3 | 20.5 | 2,473 ms | 48.7 ms | 128.0 | 0 | 78,464 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 119.9 | 120.1 | 149 ms | 8.3 ms | 1.0 | 0 | 3,599 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 201.8 | 101.0 | 235 ms | 9.9 ms | 2.0 | 0 | 6,055 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 349.8 | 87.6 | 339 ms | 11.4 ms | 4.0 | 0 | 10,493 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 571.1 | 71.5 | 543 ms | 14.0 ms | 8.0 | 0 | 17,128 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 863.6 | 54.2 | 961 ms | 18.4 ms | 16.0 | 0 | 25,904 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,215.7 | 38.2 | 1,556 ms | 26.2 ms | 32.0 | 0 | 36,448 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,598.2 | 25.1 | 2,766 ms | 39.8 ms | 64.0 | 0 | 47,935 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 115.6 | 115.9 | 285 ms | 8.6 ms | 1.0 | 0 | 3,468 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 190.3 | 95.3 | 414 ms | 10.5 ms | 2.0 | 0 | 5,708 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 319.7 | 80.1 | 644 ms | 12.5 ms | 4.0 | 0 | 9,588 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 495.4 | 62.1 | 936 ms | 16.1 ms | 8.0 | 0 | 14,872 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 700.4 | 44.0 | 1,549 ms | 22.7 ms | 16.0 | 0 | 21,008 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 914.8 | 28.7 | 2,986 ms | 34.8 ms | 32.0 | 0 | 27,424 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 108.2 | 108.6 | 601 ms | 9.2 ms | 1.0 | 0 | 3,245 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 171.8 | 86.2 | 835 ms | 11.6 ms | 2.0 | 0 | 5,152 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 274.1 | 68.7 | 1,175 ms | 14.5 ms | 4.0 | 0 | 8,220 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 393.0 | 49.3 | 1,888 ms | 20.3 ms | 8.0 | 0 | 11,792 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 515.4 | 32.4 | 3,415 ms | 30.9 ms | 16.0 | 0 | 15,456 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.