AWS shows llama.cpp serving SemIf decision models on SageMaker AI GPU and CPU
Overview
AWS Machine Learning Blog publishes a guide to deploying decision models with llama.cpp on Amazon SageMaker AI. The setup uses AWS's llama.cpp Deep Learning Container for OpenAI-compatible chat completions and a SemIf child image that returns probabilities over declared options.
In a serial test of 231 JevBench tasks on Qwen3.5-4B Q4_K_M, AWS reports the GPU endpoint on ml.g6e.xlarge ran 4.9 times faster at p50 (0.352 seconds versus 1.715 seconds) than the CPU endpoint on ml.c6i.8xlarge. Accuracy was 77.9% on GPU and 77.5% on CPU.
Written by AI from the articles below · updated Oct 12, 12:51 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- AWS Machine Learning BlogOfficialAWS shows how to serve SemIf decision models with llama.cpp on SageMaker AI
AIAWS published a guide to deploying decision models with llama.cpp on Amazon SageMaker AI, using its llama.cpp Deep Learning Container for OpenAI-compatible chat completions and a SemIf child image for probabilities over declared options. In a serial test of 231 JevBench tasks on Qwen3.5-4B Q4_K_M, the GPU endpoint on ml.g6e.xlarge ran 4.9 times faster at p50 (0.352 seconds versus 1.715 seconds) than the CPU endpoint on ml.c6i.8xlarge. Accuracy was 77.9% on GPU and 77.5% on CPU.
Heat trend
Not enough continuous observations to show a trend yet.