# SemEval-2026 Task 6: Political Evasion Detection - Team ttda704

This repository contains the official implementation for **SemEval-2026 Task 6: Detecting Evasion in Political Discourse**. Our approach combines parameter-efficient fine-tuning of open-source models (Qwen3) with advanced reasoning via API-based models (DeepSeek-V3.2, Grok-4-Fast).

**Paper:** System Description for SemEval-2026 Task 6: Structured Chain-of-Thought Prompting for Political Evasion Detection

**Team:** ttda704

---

## 📋 Table of Contents

- [Overview](#overview)
- [Quick Start](#quick-start)
- [Project Structure](#project-structure)
- [Installation](#installation)
- [Configuration](#configuration)
- [Usage](#usage)
  - [Fine-Tuning](#fine-tuning)
  - [API Inference](#api-inference)
  - [Evaluation](#evaluation)
- [Reproducing Paper Results](#reproducing-paper-results)
- [Methodology](#methodology)

---

## Overview

Our system addresses two subtasks:

- **Subtask 1 (Clarity):** 3-class classification (Clear Reply / Ambivalent Reply / Clear Non-Reply)
- **Subtask 2 (Evasion):** 9-class classification of specific evasion strategies

We demonstrate that **reasoning-capable API models with few-shot hierarchical CoT prompting** outperform fine-tuned models without explicit reasoning support by 15+ points in Macro F1.

**Best Result (Evaluation on Test Set):**

- Grok-4-Fast Reasoning + Few-shot CoT (Hierarchical)
- Subtask 1 F1: **0.7979**
- Subtask 2 F1: **0.5147**

---

## Quick Start

> Note for supplementary package users:
> In some distributions, the training set is provided as `data/train_jsonl.zip` to reduce package size.
> Please extract it to `data/train.jsonl` before running training/evaluation commands.

```bash
# 1. Install dependencies
pip install -r requirements.txt

# 2. Set API keys (if using API models)
export DEEPSEEK_API_KEY="your_key"
export GROK_API_KEY="your_key"

# 3. Run best configuration
python api_inference/grok_inference.py \
    --api_key $GROK_API_KEY \
    --model_name grok-4-fast-reasoning \
    --prompt_type fewshot_cot \
    --hierarchical \
    --evaluate

# Or fine-tune a model
python finetune/train.py \
    --model_size 14B \
    --prompt_type fewshot_cot \
    --num_epochs 1
```

---

## Project Structure

```
SemEval-2026-Task6/
├── config.py                    # ⭐ Central configuration (all settings)
├── requirements.txt             # Python dependencies
├── README.md                    # This file
│
├── data/                        # Dataset
│   ├── train.jsonl             # Training set (N=3448, extracted)
│   ├── train_jsonl.zip         # Compressed training set (for supplementary upload)
│   ├── valid.jsonl             # Validation split (10%)
│   ├── test_labeled.jsonl      # Test set with labels (for evaluation)
│   └── test_no_labels.jsonl    # Blind test set (no labels)
│
├── prompts/                     # Prompt templates
│   ├── cot/                    # Chain-of-Thought prompts
│   │   ├── fewshot_cot.txt    # Few-shot + CoT (hierarchical)
│   │   ├── fewshot_flat.txt   # Few-shot + CoT (flat)
│   │   ├── zeroshot_cot.txt   # Zero-shot + CoT (hierarchical)
│   │   └── zeroshot_flat.txt  # Zero-shot + CoT (flat)
│   └── standard/               # Standard prompts
│       ├── fewshot.txt        # Few-shot (no CoT)
│       └── zeroshot.txt       # Zero-shot (no CoT)
│
├── utils/                       # ⭐ Utility modules
│   ├── __init__.py
│   ├── data_loader.py          # Data loading, preprocessing, prompting
│   └── metrics.py              # Evaluation, metrics, visualization
│
├── notebooks/                  # Reference notebooks (thinking follow-up)
│   └── qwen3_32b_thinking_followup.ipynb
├── finetune/                    # Fine-tuning scripts
│   ├── train.py                # ⭐ Unified training script
│   ├── inference.py            # ⭐ Unified inference script
│   ├── qwen_chat_format.py    # Qwen ChatML + optional native thinking
│   ├── qwen_4b.py             # (Legacy) Qwen3-4B specific
│   ├── qwen_14b.py            # (Legacy) Qwen3-14B specific
│   ├── qwen_32b.py            # (Legacy) Qwen3-32B specific
│   └── llama_70b.py           # (Legacy) Llama-3.1-70B specific
│
├── api_inference/               # API inference scripts
│   ├── deepseek_inference.py  # ⭐ Unified DeepSeek script
│   ├── grok_inference.py      # ⭐ Unified Grok script
│   ├── deepseek_chat.py       # (Legacy) DeepSeek non-reasoning
│   ├── deepseek_thinking.py   # (Legacy) DeepSeek reasoning
│   └── grok_non_reasoning.py  # (Legacy) Grok non-reasoning
│
├── results/                     # Outputs (auto-created)
│   └── [experiment_name]/
│       ├── predictions.jsonl
│       ├── metrics.json
│       └── confusion_matrix.png
│
└── outputs/                     # Model checkpoints (auto-created)
    └── [model_name]/
```

⭐ = New unified scripts (recommended)

---

## Installation

### Requirements

- Python 3.8+
- CUDA 11.8+ (for fine-tuning)
- **GPU:** H100 80GB VRAM (recommended for optimal speed)
  - ⚠️ **Note:** Hyperparameters in `config.py` are optimized for H100 80GB to support the largest models (32B). Using lower-spec GPUs (A100, V100, RTX 4090, etc.) will be significantly slower and may require reducing batch size or model size.
  - For 32B models: Minimum 80GB VRAM required (with current hyperparameters)
  - For 14B models: Minimum 40GB VRAM required
  - For 4B models: 16GB+ VRAM sufficient

### Install Dependencies

```bash
pip install -r requirements.txt

# For fine-tuning, also install Unsloth
pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
```

### Data Format

If `data/train.jsonl` is missing but `data/train_jsonl.zip` exists, extract it first:

```bash
# Linux/macOS
unzip data/train_jsonl.zip -d data

# Windows PowerShell
Expand-Archive -Path data/train_jsonl.zip -DestinationPath data -Force
```

JSONL files with:

```json
{
  "id": "unique_id",
  "interview_question": "Full interview context...",
  "question": "Specific question...",
  "answer": "Response...",
  "clarity_label": "Ambivalent",
  "evasion_label": "Dodging"
}
```

**Note:**

- `train.jsonl`, `valid.jsonl`, `test_labeled.jsonl` contain both `clarity_label` and `evasion_label` fields
- `test_no_labels.jsonl` has empty `evasion_label` fields (blind test set for submission)

---

## Configuration

All settings are centralized in **`config.py`**. Key configuration classes:

- **`DataConfig`**: Data paths, label mappings
- **`FineTuningConfig`**: Training hyperparameters  
- **`DeepSeekConfig`**: DeepSeek API settings
- **`GrokConfig`**: Grok API settings

**Edit `config.py` or use command-line arguments:**

```python
# config.py
ft_config.num_train_epochs = 3
ft_config.learning_rate = 2e-5
```

Or:

```bash
python finetune/train.py --num_epochs 3 --model_size 14B
```

---

## Usage

### Fine-Tuning

#### Training

```bash
# Train Qwen3-14B with few-shot CoT (recommended)
python finetune/train.py \
    --model_size 14B \
    --model_variant base \
    --prompt_type fewshot_cot \
    --output_dir ./outputs/qwen14b_fewshot_cot \
    --num_epochs 1

# Available options:
# --model_size: 4B, 14B, 32B
# --model_variant: base, instruct
# --prompt_type: zeroshot, fewshot, zeroshot_cot, fewshot_cot
```

#### Inference

⚠️ **Important:** For inference, use datasets **with labels** (e.g., `data/valid.jsonl`, `data/test_labeled.jsonl`, `data/train.jsonl`). The blind test set (`data/test_no_labels.jsonl`) has no labels and cannot be evaluated automatically.

```bash
python finetune/inference.py \
    --model_path ./outputs/qwen14b_fewshot_cot/final_model \
    --test_file data/valid.jsonl \
    --prompt_type fewshot_cot \
    --output predictions.jsonl \
    --evaluate
```

#### Qwen3 native thinking (paper diagnostic follow-up)

The main paper keeps **training without** Qwen’s native thinking chat-template flag for comparability with the reported PEFT baseline. For the **thinking-mode follow-up** (Appendix), training and inference use the same tokenizer behavior as in `notebooks/qwen3_32b_thinking_followup.ipynb`: `apply_chat_template(..., enable_thinking=True)` when supported, with a `/think` or `/no_think` suffix fallback.

Unified scripts (`finetune/qwen_chat_format.py`, wired from `train.py` / `inference.py`):

```bash
# Train with thinking-enabled chat template (optional preset: conservative LR/dropout)
python finetune/train.py --model_size 32B --prompt_type fewshot_cot --thinking_followup

# Or only flip the thinking flag on top of your usual hyperparameters
python finetune/train.py --model_size 32B --prompt_type fewshot_cot --enable_thinking

# Inference with native thinking at decode time (stratified / sampled evaluation as in the paper)
python finetune/inference.py \
    --model_path ./outputs/qwen3-32b-thinking-followup/final_model \
    --test_file data/test_labeled.jsonl \
    --prompt_type fewshot_cot \
    --enable_thinking \
    --max_new_tokens 1024 \
    --output predictions_thinking.jsonl \
    --evaluate
```

The notebook under `notebooks/` retains the full end-to-end script (metrics, hard-fallback logging, trace dumps) used during experiments.

### API Inference

⚠️ **Important:** When using `--evaluate` flag, ensure your `--test_file` contains labels (use `data/valid.jsonl`, `data/test_labeled.jsonl`, or `data/train.jsonl`). The blind test set (`data/test_no_labels.jsonl`) has no labels for evaluation.

#### DeepSeek-V3.2

```bash
# Reasoning mode (best)
python api_inference/deepseek_inference.py \
    --api_key YOUR_API_KEY \
    --model_name deepseek-reasoner \
    --prompt_type fewshot_cot \
    --hierarchical \
    --test_file data/valid.jsonl \
    --output predictions_deepseek.jsonl \
    --evaluate

# Non-reasoning mode
python api_inference/deepseek_inference.py \
    --model_name deepseek-chat \
    --prompt_type fewshot_cot \
    --evaluate
```

#### Grok-4-Fast

```bash
# Reasoning mode (best overall)
python api_inference/grok_inference.py \
    --api_key YOUR_API_KEY \
    --model_name grok-4-fast-reasoning \
    --prompt_type fewshot_cot \
    --hierarchical \
    --test_file data/valid.jsonl \
    --batch_size 50 \
    --output predictions_grok.jsonl \
    --evaluate
```

### Evaluation

With `--evaluate` flag, you get:

- ✅ Subtask 1 (Clarity) + Subtask 2 (Evasion) metrics
- ✅ Per-class precision, recall, F1
- ✅ Confusion matrix visualization
- ✅ Detailed classification report

Results saved to: `results/[experiment_name]/`

---

## Reproducing Paper Results

### Best Configuration

**Grok-4-Fast Reasoning + Few-shot CoT (Hierarchical)**

```bash
python api_inference/grok_inference.py \
    --api_key $GROK_API_KEY \
    --model_name grok-4-fast-reasoning \
    --prompt_type fewshot_cot \
    --hierarchical \
    --evaluate

# Results:
# Subtask 1 F1: 0.7979
# Subtask 2 F1: 0.5147
```

### Best Fine-Tuned

**Qwen3-32B + Few-shot CoT**

```bash
python finetune/train.py --model_size 32B --prompt_type fewshot_cot
python finetune/inference.py \
    --model_path ./outputs/final_model \
    --prompt_type fewshot_cot \
    --evaluate

# Results:
# Subtask 1 F1: 0.5654
# Subtask 2 F1: 0.3630
```

### Best DeepSeek

**DeepSeek-V3.2 Reasoner + Few-shot CoT (Hierarchical)**

```bash
python api_inference/deepseek_inference.py \
    --api_key $DEEPSEEK_API_KEY \
    --model_name deepseek-reasoner \
    --prompt_type fewshot_cot \
    --hierarchical \
    --evaluate

# Results:
# Subtask 1 F1: 0.7735
# Subtask 2 F1: 0.5115
```

### Ablation Studies

**Effect of Reasoning Mode:**

```bash
# With reasoning
python api_inference/deepseek_inference.py --model_name deepseek-reasoner --prompt_type fewshot_cot

# Without reasoning
python api_inference/deepseek_inference.py --model_name deepseek-chat --prompt_type fewshot_cot
```

**Effect of Taxonomy Structure:**

```bash
# Hierarchical (better for reasoning models)
python api_inference/grok_inference.py --prompt_type fewshot_cot --hierarchical

# Flat (simpler structure)
python api_inference/grok_inference.py --prompt_type fewshot_cot --flat
```

**Effect of Prompting Strategy:**

```bash
# Zero-shot
python finetune/inference.py --prompt_type zeroshot

# Few-shot (best)
python finetune/inference.py --prompt_type fewshot_cot
```

---

## Methodology

### Hierarchical Classification

1. **Subtask 1 (Clarity):**
   - Clear Reply: Direct answer to the question
   - Ambivalent Reply: Evasive, vague, or partial response
   - Clear Non-Reply: Explicit refusal or request for clarification

2. **Subtask 2 (Evasion):** 9 fine-grained strategies
   - Clear Reply: `Explicit`
   - Ambivalent Reply: `Implicit`, `General`, `Dodging`, `Deflection`, `Partial/half-answer`
   - Clear Non-Reply: `Declining to answer`, `Claims ignorance`, `Clarification`

### Key Findings

✅ **Reasoning Matters**: Extended thinking (DeepSeek Reasoner, Grok Reasoning) improves F1 by 5-10 points  
✅ **Hierarchical Taxonomy**: Improves performance for reasoning models, providing cognitive scaffolding  
✅ **Few-shot > Zero-shot**: In-context exemplars are crucial (+10-15 points F1)  
✅ **API > Fine-tuning**: For this task, reasoning-capable APIs outperform even 32B fine-tuned models  

### Main Takeaway

> For evasion detection, allocating budget to reasoning-capable API models with few-shot hierarchical CoT is more effective than scaling local fine-tuned models without explicit reasoning support.
