入門指南排行榜Decode calculator模型OrganizationsReportsMatrix runs硬體基準評測市集租用ProAPI文件
語言
Actual Computer — Every computer, one endpoint
Community field report

One endpoint for an RTX 3090 and a Tenstorrent P150: how we run EndpointMaxxing

We run seven inference engines across an RTX 3090 and a Tenstorrent P150 behind one OpenAI-compatible address with EndpointMaxxing (emx). How it is set up, what it does day to day (memory placement, shared-chip locking, metering, verified LocalMaxxing runs), who it helps, and what it does not do yet. Includes our verified Qwen3.8-Flash-Next run on the P150 through Strata: 30.4 tok/s.

We have one machine (Ryzen 9 9950X, 96 GB RAM, Ubuntu) with two very different accelerators in it: an NVIDIA RTX 3090 and a Tenstorrent P150. Between them they run seven inference engines. Until recently that meant seven ways to start a model, seven ports, and a folder of launch scripts nobody wanted to touch.

EndpointMaxxing (the command is emx) replaced that with one command line and one address. This is how we use it, what it fixed, and who we think it is for.

The problem it solves

Every engine starts differently. llama.cpp wants a GGUF file and its own flags. vLLM wants a checkpoint folder and a memory fraction. The Tenstorrent engines run in Docker containers with a dozen mounts and environment variables each. Each one has its own way of saying "I'm ready" and its own way of shutting down.

On top of that, the two cards have different rules. The 3090 can hold several models at once if they fit in 24 GB. The P150 is one chip that one job owns at a time, and other work on the machine uses it too.

So the real cost was never one model. It was switching: stop this, free that, remember which port, check nothing else is on the chip.

What our setup looks like

HardwareEngines we run on itWhat we use it for
RTX 3090 (24 GB)llama.cpp, buun-llama-cpp, vLLM, SGLangsmall and mid-size models, quick tests, anything GGUF
Tenstorrent P150Strata, tt-metal vLLM (Docker), a multi-model hostQwen3.8-Flash-Next through Strata, LFM models, several small Qwen2.5 models kept loaded together

Twelve models are registered. Each was registered once:

sh
emx model add qwen3-0.6b ~/models/Qwen3-0.6B            # emx picks vLLM
emx model add qwen35-4b ~/models/Qwen3.5-4B-GGUF        # emx picks llama.cpp
emx model add flash-next-iq3s ~/strata/strata-fnx-iq3s.json   # emx picks Strata

After that, nobody needs to know which engine or which card a model lives on. emx serve gives one OpenAI- and Anthropic-compatible address, http://127.0.0.1:7777/v1, and a request names the model it wants. If that model isn't running, emx starts it on the right hardware and then answers.

What it does for us day to day

It places models by memory on the 3090. When we ran a vLLM model and a llama.cpp model side by side, emx planned about 10 GB for the first and 1.5 to 2 GB for the second and put both on the card. When something doesn't fit, it stops a model the gateway started itself and nobody is using, never one we started by hand.

It treats the P150 as what it is: one chip, shared. emx takes the same lock file our other Tenstorrent jobs use, so it never starts a model on a chip another job holds, and emx doctor says whether the chip is free. If an engine dies badly, the chip is reset only once nothing else is on it.

It keeps several small models warm on one chip. The multi-model host holds a few Qwen2.5 models on the P150 at once. Measured on our card: about 7 seconds for a warm host start plus the first load, 1 to 3 seconds for each further load, and no added time when requests switch between loaded models.

It records what happened. Every request is metered, so emx usage --by model answers "what did we actually run this month". Ours shows Flash-Next on Strata decoding at about 30 tokens per second on the P150, and the 0.6B Qwen on the 3090 averaging 379.

It feeds LocalMaxxing. emx bench run flash-next-iq3s --submit starts the model, runs lmx against it and uploads the result with the engine, quantization and hardware filled in. Our Flash-Next run on the P150 went up that way and is attached to this article: 30.4 tokens per second out, 158 prefill, 2.2 seconds to first token, 1,024 tokens generated, marked verified.

Agents can drive it. emx mcp exposes the same commands as tools. An agent can start a model, ask it something, and check usage. By default it cannot do anything that chooses what the machine runs or reads; that needs an explicit flag.

What we have actually run through it

Every figure in this section is from the gateway's own usage database on this machine. Between 13 October and 5 November it handled 345 requests, 314 of them answered successfully, for 15 different models on seven engines.

EngineCardRequests answeredTokens generated
llama.cppRTX 309013926,579
vLLMRTX 30909524,379
our Qwen3.8-27B runtimeP15032783
StrataP150199,606
multi-model hostP15017264
tt-metal vLLMP15011339
SGLangRTX 3090116

Most of the llama.cpp requests (124 of 139) went to a tiny test model we use to exercise the gateway itself; the rest were Qwen3.5-4B and an LFM2.5 GGUF.

The use cases behind those rows:

A vision model on one P150, added without touching emx. Our own Qwen3.8-27B runtime (mixed BFP4/BFP8, speculative decoding, image input) joined through a manifest we wrote, the same way any outside engine would. It answered 32 requests carrying about 11,000 prompt tokens. A second manifest runs the same model for 4, 8 or 16 users per decode step.

A model too slow to start by hand. That 27B model took between 7 and 11 minutes to become ready, every time. Nobody wants to babysit that. With emx it is one registered name: the first request starts it and waits, and later requests just use it.

Qwen3.8-Flash-Next through Strata. 15 requests and 9,558 generated tokens at about 30 tokens per second, plus the verified benchmark attached to this article.

Three small models sharing one chip. The multi-model host loaded models seven times and answered requests for Qwen2.5 0.5B-Coder, 1.5B and 3B from the same P150. Requests to the 1.5B and 3B averaged 1.4 and 1.7 seconds.

The same model on two engines. On the 3090 the 0.6B Qwen ran through vLLM for 95 requests and, registered a second time under another name, through SGLang as a check. Switching engine is the --engine flag; the client code did not change.

Starting on demand. 30 requests arrived for a model that was not running. emx started it and answered; the caller only saw a slower first response.

Recovering from crashes. Engines exited unexpectedly 13 times. Each exit was recorded with the last lines of the engine's log, and six times the gateway restarted the model on its own.

More than one client style. Chat completions, plain completions, the Anthropic messages format and the Responses format were all used against the same address, and 125 of the 345 requests were streamed.

Feeding the tools around it. lmx (from v0.1.62) uses the same gateway: lmx gateway start runs it, lmx model load registers a model with it, and lmx benchmarks go through it when it is up.

How long each model takes to become ready, as the gateway recorded it:

ModelEngineCardTime to ready
Qwen3.5-4B (GGUF)llama.cppRTX 30902 to 4 s
a model on the multi-model hostmulti-model hostP150about 4 s
Qwen3-0.6BvLLMRTX 309015 to 18 s
Qwen3.8-Flash-Next IQ3_SStrataP15027 to 35 s
LFM2.5-24Btt-metal vLLMP15030 to 39 s
Qwen3.6-35Btt-metal vLLMP150105 s
Qwen3.8-27Bour runtimeP1507 to 11 minutes

These are small numbers. This is one machine, and the software had its first public release this week. They are the real ones, though, and they are why we stopped using launch scripts.

A ten-minute load test on the 3090

On 13 November we pointed twelve concurrent clients at the gateway for ten minutes: eight at Qwen3-0.6B (Q8) and four at Qwen3.8-27B (Q4_K_XL GGUF), both through llama.cpp. Neither model was running beforehand.

ModelRequestsFailedTokens generatedCombined speedAverage time to first token
Qwen3-0.6B Q8, 8 clients6880273,373439 tok/s150 ms
Qwen3.8-27B Q4, 4 clients82039,26063 tok/s1.2 s

What that run showed:

  • Two models on one card. emx started both on the first requests (1 second for the small one, 10 seconds for the 27B) and kept both loaded: about 18.2 GB for the 27B and 2.7 GB for the 0.6B, 21 of the card's 24 GB.
  • No failed requests out of 770, with 312,633 tokens generated.
  • The metering is exact. We counted requests and tokens on the client side and compared them with the gateway's records: the same 770 requests, the same prompt tokens and the same completion tokens, to the token.
  • Mixed client styles under load. 220 of the requests were streamed and 110 used the Anthropic messages format.

The P150 was not part of this run, and that is a use case of its own. Another job was using the chip. We asked the gateway for the Flash-Next model anyway: it started nothing, left the other job alone, and the request simply waited. That is the behaviour we want from a shared accelerator.

Using it from a coding agent

We pointed the OMP coding agent at the gateway. This is the whole configuration:

yaml
providers:
  emx:
    baseUrl: http://127.0.0.1:7777/v1
    api: openai-completions
    auth: none
    discovery:
      type: openai-models-list

The agent then listed all 14 models registered in emx, on both cards, running or not, and any of them can be selected by name. We gave it a small task on the 27B: read a file and say what is wrong with a function. It made a tool call, read the file and answered correctly. The model was not running when the task began; the first request started it.

Two things we learned doing this:

  • Agents need room. The agent's opening prompt alone was about 7,500 tokens. Our first attempt failed because the 27B had been loaded for four clients with 4,096 tokens each. Re-registered with a 32K context for one client, it worked.
  • The gateway does not yet tell a client how much context each model has, so the agent assumed 128K for all of them. It also lists models whose files have since been moved. Both are on our list to fix.

Who benefits

  • People with mixed hardware. If you have an NVIDIA card and anything else, you already know the scripts problem. One registry and one address removes it.
  • Anyone who benchmarks. Comparing engines on the same model is one flag (--engine) instead of a new setup, and results go to LocalMaxxing with the metadata already right.
  • Small teams sharing a box. API keys, per-key model lists, rate limits, token budgets and an audit trail are built in, so "who used the GPU all night" has an answer.
  • People building agents. One stable endpoint that starts models on demand is much easier to point a tool at than a set of ports that come and go.
  • Engine authors. An engine joins through one YAML manifest. There is no engine-specific code in emx; Strata and the Tenstorrent engines were added exactly that way.

What it is not, yet

It is version 0.1.0. Linux is where we run it with real engines and real hardware. The macOS and Windows builds pass their tests and a start/serve/stop check on every change, but nobody has run a real engine through them yet. It manages one machine; routing across several is not there. The model list it gives to clients does not yet include each model's context size.

Try it

sh
go install github.com/LottoLottoLotto/endpointmaxxing/cmd/emx@latest
emx doctor            # what engines and hardware it sees
emx model add my-model /path/to/model
emx serve             # http://127.0.0.1:7777/v1

Binaries for Linux, macOS and Windows are on the releases page. emx serves the models; lmx measures them.

Attached evidenceLocalMaxxing runs referenced by this report — open any run to see the full submission
Discussion

0 comments

Questions, reproduction notes, and follow-up results.

No comments yet. Start the technical discussion.