大语言模型在机器翻译质量标注中的表现：人类与模型均面临挑战
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
研究评估了 LLM 在多维质量指标（MQM）和错误跨度标注（ESA）任务中的表现，通过对比 70 个语言对及 WMT23 和 WMT25 数据集，发现 LLM 与人类标注者的一致性在不同任务中差异巨大，在多数场景下仍不可靠。研究指出人类在细粒度标注和低资源语言上表现受限，而 LLM 则难以处理细微错误、错误语言变体及错误跨度标注。该结果强调了通过人机协作标注流程提升评估可靠性的潜力。
AuthorsHala Almaghout, Christian Federmann, Qin Gao
Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.
Related readings and updates.
Safety policies define what constitutes safe and unsafe AI outputs, guiding data annotation and model development. However, annotation disagreement is pervasive and can stem from multiple sources such as operational failures (annotators misunderstand or misexecute the task), policy ambiguity (policy wording leaves room for interpretation), or value pluralism (different annotators hold different perspectives on safety). Distinguishing these…
Machine learning (ML) and artificial intelligence (AI) systems rely heavily on human-annotated data for training and evaluation. A major challenge in this context is the occurrence of annotation errors, as their effects can degrade model performance. This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks for three industry-scale ML applications (music streaming, video streaming, and…
来源：Apple Machine Learning Research · machinelearning.apple.com