Kei Moriyama
2026
Task Assignment meets Annotator Modeling: Human-LLM Collaborative Annotation with Constraints
Kei Moriyama | Kouta Nakayama | Yukino Baba
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)
Kei Moriyama | Kouta Nakayama | Yukino Baba
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)
Crowdsourced annotators and Large Language Models (LLMs) offer complementary, cost-effective ways to obtain labeled data, yet ensuring high label quality remains challenging.We observe that task features influence the accuracy of humans and LLMs, while real-world constraints, such as per-annotator assignment limits, further complicate allocation.Prior work typically addresses either task features or constraints, but not both.We present an integrated framework that (i) estimates per-task accuracy from task features using a learning from crowds model and (ii) incorporates these estimations into a linear programming formulation that assigns tasks under practical constraints. Experimental results demonstrate that the proposed method achieves accuracy comparable to that of baseline methods while satisfying given constraints.