MoodMetric at SemEval-2026 Task 4:Narrative Story Similarity and Narrative Representation Learning

Samanvitha Bolisetty, Shreya Ashar, Nishchay Mittal, Pruthwik Mishra


Abstract
This paper presents our system for narrative similarity modeling in SemEval Task 4, focusing on transformer-based dense embedding approaches. Modeling similarity between long-form narratives is particularly challenging due to the need to capture event progression, causal structure, character dynamics, and thematic coherence beyond surface-level lexical overlap.We evaluate multiple pretrained encoder-only architectures, including DeBERTa-v3, BGE-Base, BGE-Large, and E5-Large, fine-tuned using triplet margin and contrastive objectives. In addition, we implement a hybrid lexical–semantic baseline combining TF-IDF and SBERT features. Our experiments analyze the impact of model scale, pooling strategies, layer freezing, training duration, and embedding-level ensembling under low-resource conditions (approximately 1,900 training triplets, with additional synthetic augmentation).Results show that larger contrastively pretrained embedding models consistently outperform smaller variants, with BGE-Large achieving the strongest standalone performance. However, performance saturates quickly, and moderate fine-tuning (4–5 epochs) yields optimal validation accuracy, while extended training leads to overfitting. Instruction-tuned embeddings do not demonstrate significant advantages over contrastively aligned alternatives for this task. Finally, arithmetic averaging of embeddings from diverse models produces the most robust representations, achieving approximately 65% validation accuracy.
Anthology ID:
2026.semeval-1.336
Volume:
Proceedings of the 20th International Workshop on Semantic Evaluation (2026)
Month:
July
Year:
2026
Address:
San Diego, California, USA
Editors:
Ekaterina Kochmar, Debanjan Ghosh, Kai North, Mamoru Komachi
Venues:
SemEval | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2664–2672
Language:
URL:
https://preview.aclanthology.org/ingest-acl-workshops/2026.semeval-1.336/
DOI:
Bibkey:
Cite (ACL):
Samanvitha Bolisetty, Shreya Ashar, Nishchay Mittal, and Pruthwik Mishra. 2026. MoodMetric at SemEval-2026 Task 4:Narrative Story Similarity and Narrative Representation Learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (2026), pages 2664–2672, San Diego, California, USA. Association for Computational Linguistics.
Cite (Informal):
MoodMetric at SemEval-2026 Task 4:Narrative Story Similarity and Narrative Representation Learning (Bolisetty et al., SemEval 2026)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-workshops/2026.semeval-1.336.pdf
Supplementarymaterial:
 2026.semeval-1.336.SupplementaryMaterial.tex