Skip to content
Trending storyDeveloping

Dongxi NLP explains how reward models are trained from preference pairs

2 articles1 sourcesince Oct 31Last article 1h ago ·

Overview

AISummary of 2 articles

Dongxi NLP, in part 5 of its Post Training series, says a reward model is the inverse of the Bradley-Terry model.

The post says the model learns from pairs of preferred and rejected answers rather than explicit scores, and uses the loss L = −log σ(r(chosen) − r(rejected)) to turn score differences into choice probabilities.

According to the post, the reward model keeps the base LLM's body and replaces the LM head with a single-row score head that outputs one scalar. An illustrative two-weight example shows gradient descent raising the chosen answer's score. The series plans to cover the loss, network architecture, and what the reward model learns during training.

Written by AI from the articles below · updated Oct 31, 4:57 AM ET

Check the sources:

Latest developmentOct 31, 4:28 AM ET
Dongxi NLP explains reward models as the inverse of Bradley-Terry

Developments

2 developments

  1. Oct 31, 4:28 AM ET · 1 article
    Post-training series part 5: Reward Model as the mirror of Bradley-Terry
    Dongxi NLP: Dongxi NLP explains reward models as the inverse of Bradley-Terry
  2. Oct 31, 4:25 AM ET · 1 article
    Dongxi NLP explains how reward models turn human choices into scores
    Dongxi NLP: Dongxi NLP explains how reward models turn human choices into scores

Article timeline

The articles in this story. Times are ET.

Oct 31
  1. Dongxi NLPX
    Dongxi NLP explains reward models as the inverse of Bradley-Terry

    AIDongxi NLP says reward models are the inverse of the Bradley-Terry model, in the fifth part of its Post Training series. The post plans to cover how preference choices become a loss, how that loss updates the network, the network's architecture, and what the reward model learns during training.

    Video from @dongxi_nlp's post
  2. Dongxi NLPX
    Dongxi NLP explains how reward models turn human choices into scores

    AIDongxi NLP explains that a reward model learns from pairs of preferred and rejected answers rather than explicit scores. The post says the Bradley–Terry loss, L = −log σ(r(chosen) − r(rejected)), turns score differences into choice probabilities, and an illustrative two-weight example shows gradient descent raising the chosen answer's score. It adds that the reward model keeps the base LLM's body and replaces the LM head with a single-row score head that outputs one scalar.

Heat Trend

Not enough continuous observations to show a trend yet.