Speculators v0.9.0 adds a fused loss kernel that makes forward training passes up to 2x faster, according to vLLM. An FP8 hidden-state backend cuts storage 50% with no quality loss, and the speedup applies to small models of about 1B parameters. The trained speculators can be served in vLLM.
Speculators v0.9.0 is out — faster, lighter speculative decoding training 📷
Up to 2x faster forward training passes with a fused loss kernel, plus an FP8 hidden-state backend that cuts storage 50% with no quality loss.
Train better speculators, serve them in vLLM. Great work @RedHat_AI 📷
Speculators v0.9.0 is out. Training got faster and lighter: a fused loss kernel gives up to 2x faster forward training passes on small (~1B) models, and an FP8 hidden-state backend cuts storage 50% while keeping speculator quality. Full notes: https://github.com/vllm-project/speculators/releases/tag/v0.9.0
Source: vLLM · x.comPublished
