How does PhysisServe serve multiple concurrent interactive tasks under limited compute?
Dynamic Placement of Virtual Ranks (Multi-Stream Scaling)
Encapsulates the compute tasks and runtime states of each stream across stages intovirtual ranks, which are dynamically mapped onto physical GPUs.
When a new stream arrives, the scheduler places it based on remaining GPU capacity.
Admission control accepts a new stream only if all of its stages can be accommodated, without migrating already-running streams—ensuring service stability while balancing cross-GPU loads and utilizing fragmented idle capacity.