Amazon and Booking.com pair distillation with Medusa to speed up travel LLMs
Overview
Amazon Science reports that combining knowledge distillation with Medusa decoding let a 1.1-billion-parameter TinyLlama student match its 70-billion-parameter Llama-3.1 teacher while running 10 to 20 times as fast.
The authors say the approach kept quality at less than one-third the cost. The work was tested with Booking.com on the AI Trip Planner, Smart Filters, and accommodation description generation.
The approach was presented at ACL and won the Industry Track best-paper award.
Written by AI from the articles below · updated Oct 29, 1:22 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Amazon ScienceOfficialPickAmazon and Booking.com combine distillation and Medusa to speed up travel LLMs
AIAmazon Science reports that combining knowledge distillation with Medusa decoding let a 1.1-billion-parameter TinyLlama student match its 70-billion-parameter Llama-3.1 teacher while running 10 to 20 times as fast. The work was tested with Booking.com on the AI Trip Planner, Smart Filters, and accommodation description generation, and the authors say it maintained quality at less than one-third the cost. The approach was presented at ACL and received the Industry Track best-paper award.
Heat Trend
Not enough continuous observations to show a trend yet.