Amazon Web Services introduces priming to build hybrid state space models from pretrained Transformers
Original titleCan we change an AI model’s architecture without retraining it from scratch?
Amazon Web Services introduces priming, a method that starts hybrid architectures from pretrained Transformer weights and cuts their pretraining cost by more than 99%.
The team reports that a 32-billion-parameter GKA-based hybrid matches its source Transformer on reasoning benchmarks while decoding up to twice as quickly at 128,000 tokens of context.
The paper and released models, training code, and vLLM plugin are available under the Apache 2.0 License.
The post explains how reusing pretrained Transformer weights cuts hybrid architecture pretraining cost, a method that makes controlled architecture comparisons affordable.
Source: Amazon Science · amazon.sciencePublished · added here