Ai2 introduces FairStress, a benchmark for hidden demographic asymmetries in LLM decisions
AIAi2 releases FairStress, a benchmark of 300 two-candidate decision scenarios across 29 domains that measures whether models change decisions asymmetrically under adversarial prefixes depending on demographic group. Across 15 models on the full benchmark, the average gap in abandoning correct answers was 6.9 points with implied identity and 9.3 points with explicit labels.
Loading source posts…
This story is featured in top picks: “Ai2 releases FairStress, a benchmark for hidden demographic asymmetry in LLM decisions”
Source details
Ai2 (Allen Institute for AI) · allenai.orgPublished · added here
Original title: FairStress: A deeper evaluation of fairness in LLM decision-making