TypeSafe AI (@typesafeai) says a good eval snapshots performance at one moment, measures results on the tasks and data a user cares about, and targets desired outcomes rather than proxies.
The post is a short principle list, with no benchmark figures or model names. A reply from @TimNeumann84165 notes that Parallel and co-author @p0 evaluated eight decision models on audited company training datasets, with Jev ranked first.
who benchmarks the benchmarkers?
a good eval is:
take notes!
Typesafe's CEO @CompleteSkeptic does not trust public benchmarks. Neither do I, Diogo. So @p0 and I wrote a blog evaluating 8 decision models on real company training datasets we have personally audited and iterated on. For what it's worth though, Jev is still on top 👑 http://parallel.ai/blog/are-all-system-one-models-created-equalView quoted post on X
Source: TypeSafe AI · x.comPublished · added here
