When extreme minimum latency isn't required, how can a single stream meet target SLOs using fewer GPUs?
Time-Division Multiplexing (Compute Sharing)
Hardware Decoupling: Decouples stages from physical GPUs so a single GPU can execute multiple stage tasks across different time slices, rather than dedicating one GPU per stage.
Batch Merging: Merges tasks from different chunks on the same Transformer block into a single batch for execution, amortizing weight loading and kernel launch overhead while filling idle GPU cycles.