Splitting prefill and decode across different silicon is one of the most sensible ideas in inference right now.
@general_compute is running prompt reads on GPUs and handing token-by-token generation to @positron_ai’s memory-first systems, then selling both as one endpoint.
i.e. run prefill on GPUs and decode on Positron hardware, giving customers one endpoint for a request that crosses 2 chip types.
These chips are built to write AI answers faster and cheaper by focusing on memory speed (the real bottleneck when AI generates text) instead of raw processing power.
The split follows the physics. Prefill is a large parallel compute job. Decode is mostly the chip waiting to pull weights and cache out of memory for every new token. A single chip has to compromise on at least one of those.
Positron builds for the second job, and builds it without relying on HBM, the memory GPUs are all competing for.
From the customer’s side it stays one contract and one set of SLAs. Deciding which chip handles which phase is General Compute’s problem.