# SemEval-2026 Task 1: MWAHAHA — Paper repository

This repository holds the LaTeX source and reproducibility scripts for the
SemEval-2026 Task 1 (MWAHAHA) task description paper.

## Repository layout

```
main.tex                  Paper source (XeLaTeX)
custom.bib                Hand-authored citations
systems.bib               Citations for the participating-system papers
img/                      Figures (incl. img/elo_by_category.pdf, generated)
scripts/                  Analysis + figure-generation + verification scripts
output/                   Build artefacts (gitignored)
.venv/                    Local Python virtualenv (gitignored)
requirements.txt          Pinned Python dependencies for the analysis scripts
```

## Building the paper

```bash
latexmk main.tex
```

`latexmk` reads `.latexmkrc`, runs XeLaTeX + BibTeX as needed, and writes
`output/main.pdf`. **Don't** use `--paper_type long` against this venue —
SemEval-2026 task description papers have a 9-page submission / 10-page
camera-ready limit (see https://semeval.github.io/paper-requirements.html), so
ACL's 8-page default does not apply.

## Python environment

Create a local virtualenv and install the pinned deps:

```bash
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
```

The `.venv/` is gitignored. From here on, run scripts via
`.venv/bin/python scripts/...` (or `source .venv/bin/activate` first).

## Verifying everything

```bash
bash scripts/verify.sh
```

Runs three checks in order:

1. `latexmk main.tex` — clean LaTeX build.
2. `aclpubcheck --paper_type other output/main.pdf` — margins/font/etc.
   (`other` skips the spurious 8-page limit; the body-page check below
   enforces the SemEval-specific 10-page limit instead).
3. `scripts/check_body_pages.py` — body content (everything before
   References / Acknowledgments / Limitations / Ethical Considerations)
   fits inside the 10-page camera-ready budget.

The script exits non-zero on any failure.

## Analysis scripts

All scripts live in `scripts/` and assume the venv above is set up.

### `iaa.py` — annotation-reliability analysis

Computes the per-subtask skip/tie rates, Fleiss' κ + Krippendorff's α on
overlapping joke pairs, BT-vs-majority agreement, split-half ranking
reliability, and split-by-annotator ranking reliability. Numbers from this
script feed directly into §6.3 of the paper.

```bash
.venv/bin/python scripts/iaa.py
```

Default input is `~/Dropbox/mwahaha/votes.csv` (override with `--votes`).
Expected columns: `session_id`, `task`, `prompt_id`, `system_id_a`,
`system_id_b`, `vote ∈ {a, b, t, n}`, `date`. Outputs a text report at
`scripts/iaa_report.txt`.

### `test_iaa.py` — cross-validation tests

Validates `iaa.py`'s statistical helpers against `scipy.stats.spearmanr`,
`sklearn.metrics.cohen_kappa_score`, and the `krippendorff` PyPI package, plus
hand-crafted edge cases. Run after any change to the analysis code.

```bash
.venv/bin/pytest scripts/test_iaa.py -v
```

### `plot_elo_by_category.py` — Figure 2

Pulls the live leaderboard JSONs from
`https://pln-fing-udelar.github.io/semeval-2026-humor-gen/scores/` and renders
a per-subtask box+strip plot of system Elo grouped by approach category.

```bash
.venv/bin/python scripts/plot_elo_by_category.py
```

Writes `img/elo_by_category.pdf`.

### `check_body_pages.py` — body page count

```bash
.venv/bin/python scripts/check_body_pages.py [--limit N] [--pdf PATH]
```

Defaults to `output/main.pdf` and `--limit 10`. Exits non-zero if the body
exceeds the limit.

## Memory aids

- `bash scripts/verify.sh` is the canonical "is the paper still healthy?"
  check; run it after any non-trivial edit.
- aclpubcheck's `--paper_type long` will spuriously flag a page-limit
  violation for this paper. Use `--paper_type other`.
- New LaTeX prose follows the convention of one sentence per source line so
  diffs stay tight.
