Post from NVIDIA AI linking to x.com

We are very excited to serve Nemotron 3.5 Super VL 120B-A12B by @NVIDIAAI with unmatched performance on the Litespark Inference Engine. The results speak for themselves: running on just two NVIDIA H200 GPUs, Litespark delivers 962 tokens per second per user and 947 tokens per second of output throughput in our warm-start benchmark. The recent emergence of state-of-the-art open source models has gained a lot of traction in both academia and industry. To meet the power and complexity of these models, the demand for efficient inference has also skyrocketed. Thus creating an equally important need for high performance inference engines. From maximizing throughput and hardware utilization to minimizing time-to-first-token and inter-token-latency, inference engines play a critical role in bringing the power of these models to users.   Litespark Inference Engine supports a wide range of tunable engine parameters, enabling the user to tune their deployment performance based on specific workload requirements. Whether optimizing for throughput, latency or hardware utilization, Litespark Inference provides the flexibility to adapt to a diverse range of production scenarios for open-source model deployments. Learn more: https://mindbeam.ai/research/litespark-inference-nemotron-3-5-super-vl