i14 runs its Laplace 3B linear attention model on AWS Trainium2 with native PyTorch
Overview
i14 and AWS trained and served Laplace 3B, a 3B-parameter linear attention model, on one trn2.48xlarge instance using native PyTorch through TorchNeuron Native, according to an AWS Machine Learning Blog post published by AWS. The post reports training throughput of about 120,000 tokens per second and serving throughput of 11,544 output tokens per second across 1,280 concurrent sequences, using an unmodified vLLM engine. It says the model keeps a 29.5 MB recurrent state per sequence regardless of context length. These figures come from the AWS post and have not been independently verified.
Written by AI from the articles below · updated Oct 20, 5:57 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- AWS Machine Learning BlogOfficialPicki14 runs its Laplace 3B linear attention model on AWS Trainium2 with native PyTorch
AIi14 and AWS trained and served Laplace 3B, a 3B-parameter linear attention model, on one trn2.48xlarge instance using native PyTorch through TorchNeuron Native. The model keeps a 29.5 MB recurrent state per sequence regardless of context length, and training reached about 120,000 tokens per second. Serving reached 11,544 output tokens per second across 1,280 concurrent sequences with an unmodified vLLM engine.
Heat Trend
Not enough continuous observations to show a trend yet.