Ai2 releases FairStress benchmark testing demographic asymmetry in LLM decisions
Article timeline
The articles in this story. Times are ET.
Ai2OfficialFairStress benchmark tests demographic fairness beyond final answersAIAi2 introduces FairStress, a benchmark that tests whether AI models show demographic fairness beyond their final answers. It measures how readily a model changes a well-supported decision and how it explains its choice. The code is available on GitHub at
- Hugging Face BlogOfficialPickAi2 releases FairStress, a benchmark for hidden demographic asymmetry in LLM decisions
AIAi2 introduces FairStress, a benchmark of 300 two-candidate decision scenarios across 29 domains that tests how stable LLM decisions are under adversarial prefixes. The team reports that models switched from stronger candidates to minority-coded ones more often than the reverse, and that counterfactual chain-of-thought reduced this gap while largely preserving accuracy. The benchmark is available for download and is being added to Olmo-eval.
- Ai2 (Allen Institute for AI)OfficialPickAi2 introduces FairStress, a benchmark for hidden demographic asymmetries in LLM decisions
AIAi2 releases FairStress, a benchmark of 300 two-candidate decision scenarios across 29 domains that measures whether models change decisions asymmetrically under adversarial prefixes depending on demographic group. Across 15 models on the full benchmark, the average gap in abandoning correct answers was 6.9 points with implied identity and 9.3 points with explicit labels. The benchmark is available for download and is being built into Olmo-eval.