Ai2 releases FairStress, a benchmark for hidden demographic asymmetry in LLM decisions
AIAi2 introduces FairStress, a benchmark of 300 two-candidate decision scenarios across 29 domains that tests how stable LLM decisions are under adversarial prefixes. The team reports that models switched from stronger candidates to minority-coded ones more often than the reverse, and that counterfactual chain-of-thought reduced this gap while largely preserving accuracy.
AIWhy it matters.
The report shows that models can pick the same candidate yet treat demographic groups asymmetrically under pressure, a gap that answer-level fairness tests miss.
Loading source posts…
Source details
Hugging Face Blog · huggingface.coPublished · added here
Original title: FairStress: A deeper evaluation of fairness in LLM decision-making