public benchmarks are useful for building a model shortlist
the real test is replaying the workloads you actually serve and seeing which models pass your own evals
leaderboards help you shortlist. your traffic tells you what to ship
Together AI says to test models on your own workloads, not leaderboards alone
AITogether AI says public benchmarks help build a model shortlist, but the real test is replaying the workloads you serve and checking which models pass your own evals. It advises using leaderboards to shortlist and live traffic to decide what to ship.
Source details
Together AI · x.comPublished · added here
Original title: public benchmarks are useful for building a model shortlist