Abstract
We describe SemEval-2023 Task 11 on behavioral segregation of annotations to find the similarities and contextual thinking of a group of annotators. We have utilized a behavioral segmentation analysis on the annotators to model them independently and combine the results to yield soft and hard scores. Our team focused on experimenting with hierarchical clustering with various distance metrics for similarity, dissimilarity, and reliability. We modeled the clusters and assigned weightage to find the soft and hard scores. Our team was able to find out hidden behavioral patterns among the judgments of annotators after rigorous experiments. The proposed system is made available.- Anthology ID:
- 2023.semeval-1.295
- Volume:
- Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Atul Kr. Ojha, A. Seza Doğruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, Elisa Sartori
- Venue:
- SemEval
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2137–2142
- Language:
- URL:
- https://aclanthology.org/2023.semeval-1.295
- DOI:
- 10.18653/v1/2023.semeval-1.295
- Cite (ACL):
- Guneet Singh Kohli and Vinayak Tiwari. 2023. Arguably at SemEval-2023 Task 11: Learning the disagreements using unsupervised behavioral clustering and language models. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2137–2142, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Arguably at SemEval-2023 Task 11: Learning the disagreements using unsupervised behavioral clustering and language models (Kohli & Tiwari, SemEval 2023)
- PDF:
- https://preview.aclanthology.org/ingest-bitext-workshop/2023.semeval-1.295.pdf