# SemEval-2026 Task 4 — Narrative Story Similarity (Track A)

**Team HausaNLP** · Faisal Muhammad Adam, Lukman Jibril Aliyu, Sani Aji

📄 **Paper:** [hausanlp_semeval2026_task4.tex on Prism](https://prism.openai.com/?u=87895436-b09d-4821-8254-87184eb2e155&pg=1&m=hausanlp_semeval2026_task4.tex&d=7)

---

## Task Overview

[SemEval-2026 Task 4](https://narrative-similarity-task.github.io/) is a comparative judgment task: given an **anchor** story and two candidate stories, a system must identify which candidate is more narratively similar to the anchor. Narrative similarity is defined along three dimensions:

- **Abstract Theme** — the underlying ideas and motives
- **Course of Action** — the sequence of central events and turning points
- **Outcomes** — the results and resolutions of the story

All stories are Wikipedia plot summaries. The dataset contains over 1,000 annotated triples; we evaluate on the 200-instance development split.

---

## Our Approach

We frame Track A as a **semantic similarity ranking** problem and evaluate five systems in an ablation study:

| # | System | Model | Type |
|---|--------|-------|------|
| 1 | TF-IDF | — | Lexical baseline |
| 2 | SBERT MiniLM | `all-MiniLM-L6-v2` | Bi-encoder (official task baseline) |
| 3 | **SBERT MPNet** | `all-mpnet-base-v2` | **Bi-encoder (our best system)** |
| 4 | Paraphrase MPNet | `paraphrase-mpnet-base-v2` | Bi-encoder |
| 5 | Cross-Encoder | `cross-encoder/stsb-roberta-large` | Cross-encoder |

For bi-encoders, each text is encoded independently into a dense vector; the candidate with the higher cosine similarity to the anchor is selected. The cross-encoder jointly encodes each anchor–candidate pair and selects the higher-scoring candidate.

---

## Results

### Development Set (n = 200)

| System | Accuracy | Macro-F1 | Precision | Recall |
|--------|----------|----------|-----------|--------|
| TF-IDF (Lexical) | 54.50 | 54.44 | 54.49 | 54.47 |
| SBERT MiniLM (Official Baseline) | 55.00 | 54.93 | 54.99 | 54.97 |
| Paraphrase MPNet | 59.00 | 58.85 | 59.05 | 58.95 |
| Cross-Encoder RoBERTa | 55.50 | 55.49 | 55.49 | 55.49 |
| **SBERT MPNet (Ours)** | **61.50** | **61.48** | **61.50** | **61.48** |

### Official Test Set

Our primary submission used `all-MiniLM-L6-v2` and achieved **61.50% accuracy**, placing **33rd of 44 teams**. The top system (COGNAC) achieved 78.00% using LLM-based voting ensembles.

**Key finding:** The cross-encoder underperforms despite its stronger architecture. We attribute this to the long-document nature of Wikipedia summaries — `cross-encoder/stsb-roberta-large` was fine-tuned on short sentence pairs (STS-B) and truncates these longer inputs, eliminating the cross-attention advantage. `all-mpnet-base-v2`, which encodes each text independently, is not subject to this constraint.

---

## Error Analysis

We categorise all 200 development instances into five error types:

| Category | Count | % |
|----------|-------|---|
| All Correct / Partial | 85 | 42.5% |
| Proposed-Only Error (MPNet wrong, others right) | 26 | 13.0% |
| Proposed Recovers (TF-IDF + SBERT fail, MPNet correct) | 24 | 12.0% |
| Hard Case (all systems fail) | 23 | 11.5% |
| Lexical Trap (TF-IDF fails, neural correct) | 23 | 11.5% |
| Neural Failure (TF-IDF correct, neural wrong) | 19 | 9.5% |

**Lexical Traps** (11.5%): Surface distractors — shared dates, character types, setting terms — inflate TF-IDF similarity for the wrong candidate, but semantic models correctly identify the deeper thematic match.

**Proposed Recoveries** (12.0%): Instances where `all-mpnet-base-v2` succeeds where both TF-IDF and the official baseline fail — the direct gain from using a higher-capacity bi-encoder.

**Hard Cases** (11.5%): Genuinely ambiguous instances that no system resolves, consistent with the dataset's low inter-annotator agreement (α = 0.33).

**Proposed-Only Errors** (13.0%): Instances where MPNet predicts incorrectly while simpler systems are correct, suggesting higher-capacity models introduce new failure modes through overconfident semantic generalisation.

---

## Reproducing the Experiments

```bash
pip install sentence-transformers scikit-learn pandas

# Run all 5 systems (add --skip_crossencoder if RAM is limited)
python semeval_experiments.py --data_path data/dev_track_a.jsonl

# Skip the cross-encoder
python semeval_experiments.py --data_path data/dev_track_a.jsonl --skip_crossencoder
```

Outputs written to the working directory:

- `results_summary.csv` — accuracy, macro-F1, precision, recall per system
- `error_analysis.csv` — per-instance predictions and error category labels

---

## Citation

If you use this work, please cite the shared task:

```bibtex
@inproceedings{hatzel2026semeval,
  author    = {Hans Ole Hatzel and Ekaterina Artemova and Haimo Stiemer and
               Natalia Fedorova and Evelyn Gius and Chris Biemann},
  title     = {{SemEval}-2026 Task 4: Narrative Story Similarity and
               Narrative Representation Learning},
  booktitle = {Proceedings of the 20th International Workshop on
               Semantic Evaluation (SemEval-2026)},
  address   = {San Diego, CA, USA},
  year      = {2026}
}
```
