When you send a prompt to a model, a couple of things happen:
1. The model processes the prompt and generates the first token (prefill stage).
2. Then, the model continuously processes the prompt + new tokens until it produces the complete answer (decode stage)
These guys are doing something really cool:
They are using different hardware for each stage. They run prefill on GPUs and decode on Positron's specialized inference systems.
This results in faster generation.
Our heterogeneous cloud just got better. We're announcing a cluster scale deployment of @positron_ai as our next partner. We believe demand for fast inference will outstrip supply for the next decade. Our bet on Positron is another step towards getting fast inference for opensource models into builders' hands. Hardware goes live Q1 2027. Thanks to @mitesh711, @trsohmers and the whole Positron team! We're excited for this partnership. LFG!
Santiago · x.comPublished · added here
Original title: When you send a prompt to a model, a couple of things happen:
