Private benchmarks often show open models are further behind, but the problem with private benchmarks is you don’t have the community to squash the inevitable bugs for you. I'm 50/50 on if I think private or open-source is the best approach.
Private benchmarks may overstate gaps between open and closed models
AINathan Lambert says private benchmarks often show open models trailing further behind. He argues they lack the community that finds and fixes bugs, and says he is split 50/50 on whether private or open benchmarks are the better approach.
Source details
Nathan Lambert · x.comPublished · added here
Original title: Private benchmarks often show open models are further behind, but the problem with private benchmarks is you don’t have the community to ...