KV transfer: reducing copy overhead on NVLink At first, KV transfer on NVLink is slower than InfiniBand due to KV layout fragementation: one 200K-token request turned into 32K tiny copies.
Original titleKV transfer: reducing copy overhead on NVLink
AISummary
vLLM's block-major BLHNC layout cut transfer descriptors ~10x and cut mean transfer time by half.
Source: Prime Intellect · x.com