AWS shows how to serve SemIf decision models with llama.cpp on SageMaker AI

AWS published a guide to deploying decision models with llama.cpp on Amazon SageMaker AI, using its llama.cpp Deep Learning Container for OpenAI-compatible chat completions and a SemIf child image for probabilities over declared options.

In a serial test of 231 JevBench tasks on Qwen3.5-4B Q4_K_M, the GPU endpoint on ml.g6e.xlarge ran 4.9 times faster at p50 (0.352 seconds versus 1.715 seconds) than the CPU endpoint on ml.c6i.8xlarge. Accuracy was 77.9% on GPU and 77.5% on CPU.