Learn to build evals, folks!
You are not RSI-pilled enough if you don't know how to properly measure the thing you want to improve.
It doesn't matter if it's a model, complex codebase, or a business function. This is extremely hard in the real world.
You ever wonder why most of what AI is great at today is around verifiable problems?
The open-ended world requires a completely different approach to improving and measuring models.
We need to go back to hardcore science and evals. Enough of this short-term benchmaxxing.
This space offers so much opportunity. We are doing research and building evals in this space @dair_ai. I am currently chatting with folks building in this space too. Feel free to reach out for potential collab opportunities or just share perspectives.