Amazon Web Services introduces priming to convert Transformers into hybrid state space models
Overview
Amazon Web Services introduces priming, a method that builds hybrid state space models from pretrained Transformer weights.
The team says the approach cuts pretraining cost by more than 99%. It reports that a 32-billion-parameter GKA-based hybrid matches its source Transformer on reasoning benchmarks and decodes up to twice as fast at 128,000 tokens of context.
The paper, models, training code, and a vLLM plugin are released under the Apache 2.0 License. The released models include primed hybrids at 8B and 32B parameters for long-context reasoning and instruction following, available through the Hybrid Model Factory on GitHub and a HuggingFace model zoo.
Written by AI from the articles below · updated Oct 12, 1:48 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Amazon ScienceOfficialPickAmazon Web Services introduces priming to build hybrid state space models from pretrained Transformers
AIAmazon Web Services introduces priming, a method that starts hybrid architectures from pretrained Transformer weights and cuts their pretraining cost by more than 99%. The team reports that a 32-billion-parameter GKA-based hybrid matches its source Transformer on reasoning benchmarks while decoding up to twice as quickly at 128,000 tokens of context. The paper and released models, training code, and vLLM plugin are available under the Apache 2.0 License.
Heat trend
- Current comparable heat
- 9
- Peak comparable heat
- 10Oct 12, 2:00 PM ET
- Change in past 24 hours
- –
The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.