Study finds LLM and human annotators unreliable for machine translation quality scoring

Apple Machine Learning Research evaluates large language models against human annotators on two machine translation quality schemes, Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA), using a 70-language-pair test set and WMT23 and WMT25 data.

The authors, Hala Almaghout, Christian Federmann and Qin Gao, report that LLM agreement with human annotators sometimes exceeds agreement between human annotators on some tasks. Both LLM and human agreement vary substantially across annotation schemes, language pairs and domains, and remain unreliable for most settings.

The authors say the results point to room for improvement in both human and LLM annotation, possibly through human-LLM collaborative annotation pipelines.