We previously highlighted @lightningrodai's data recipe for training forecasters. Their new work with @PTetlock and @VSatopaa adds another key piece: choosing the rule for scoring predictions, and how each one trades off accuracy and error.
Lightning Rod's new work shows scoring rules reshape LLM forecaster profiles
AISummary
Lightning Rod, working with Philip Tetlock and Ville Satopää, post-trained five versions of the same LLM that differed only in the scoring rule used as the RL reward. The versions reached similar aggregate scores but had very different bias, information, and noise (BIN) profiles, so a good Brier score alone does not show whether a forecaster can distinguish likely from unlikely events.
Post on XView on X
@tinkerapi
New preprint from @lightningrodai, this time with Philip Tetlock and Ville Satopää! We post-train 5 versions of the same LLM, changing only the scoring rule used as the RL reward. Similar aggregate scores, but very different BIN profiles. A good Brier score alone doesn't tell you if a forecaster can distinguish likely from unlikely events. One might discern well but lose if its probabilities run systematically too high. A less discerning one might score better by hugging the base rate. BIN splits forecast performance into bias, information, and noise. Bias is a systematic shift in the probabilities. Noise is random scatter. Information is real signal about which outcomes are more likely — the part you need a powerful LLM for. Different uses call for different profiles. Reward choice is one lever shaping which forecaster you get. Congrats to co-authors @indiequant @KSkotheim64001 @VSatopaa @PTetlock 🙌 Full paper: https://arxiv.org/abs/2608.28482
Source: Tinker · x.comPublished · added here