Skip to content
Trending storyDeveloping

vLLM releases Speculators v0.9.0 with faster, lower-storage speculative decoding training

1 article1 sourcesince Oct 19Last article 1h ago ·

Overview

AISummary of 1 article

vLLM releases Speculators v0.9.0, which it says makes forward training passes up to 2x faster with a fused loss kernel.

A new FP8 hidden-state backend cuts storage 50% with no quality loss, according to vLLM. vLLM says the speedup applies to small models of about 1B parameters, and the trained speculators can be served in vLLM.

Written by AI from the articles below · updated Oct 19, 12:27 PM ET

Check the sources:

Latest developmentOct 19, 12:23 PM ET
Speculators v0.9.0 speeds up speculative decoding training and cuts storage 50%

Article timeline

The articles in this story. Times are ET.

Oct 19
  1. vLLMOfficial
    Speculators v0.9.0 speeds up speculative decoding training and cuts storage 50%

    AISpeculators v0.9.0 adds a fused loss kernel that makes forward training passes up to 2x faster, according to vLLM. An FP8 hidden-state backend cuts storage 50% with no quality loss, and the speedup applies to small models of about 1B parameters. The trained speculators can be served in vLLM.

Heat Trend

Not enough continuous observations to show a trend yet.