# MaskedVerbalizer (MV)

**Automatic Verbalizer Construction for Few-Shot Text Classification**

MaskedVerbalizer (MV) is a lightweight and effective method for automatically constructing verbalizers in prompt-based few-shot text classification, particularly for low-resource and right-to-left languages.

In prompt-based learning, classification is reformulated as a masked language modeling task. A template equivalent to:

    This sentence is [MASK].

is used across all datasets (translated into the respective dataset language: Arabic, Urdu, Pashto, Sindhi, English).

A verbalizer maps predicted tokens at the `[MASK]` position to class labels. Instead of manually designing label words or relying on heuristic search, MaskedVerbalizer automatically constructs task-specific verbalizers using the model’s own correct masked predictions.

The method:

1. Starts with 2 seed words per class (manual or GPT-assisted).
2. Fine-tunes a masked language model using only 4 examples per class (4-shot).
3. Evaluates on held-out training data.
4. Collects predicted tokens at `[MASK]` for correctly classified samples.
5. Selects the most frequent predicted tokens per class.
6. Uses them as the final verbalizer for training and evaluation.

This makes the verbalizer:

- Model-aware  
- Dataset-specific  
- Self-supervised  
- Adapted to morphologically rich and low-resource languages  

---

## Paper

This work has been accepted at:

**LREC-COLING 2026**  
MaskedVerbalizer: Automatic Verbalizer Construction for Few-Shot Text Classification in Low-Resource Right-to-Left Languages

(Full citation will be added after camera-ready publication.)

---

## 4-Shot Experimental Setup

- Dataset split: 70/30 (train/test)

- The 70% training portion is further divided into:
  - 20% of the full dataset → used to select 4 examples per class (4-shot) and initialize seed verbalizers
  - 50% of the full dataset → used to collect masked predictions for automatic verbalizer construction

- Final evaluation is performed on the 30% test split

Training Configuration:
- Learning rate: 2e-5  
- Epochs: 5  
- Optimizer: AdamW  
- Loss: CrossEntropy  
---

## Models Evaluated

- mBERT  
- XLM-RoBERTa-large  
- LLaMA 3.2 (1B) (for generative comparison)

MaskedVerbalizer is designed primarily for masked language models (mBERT, XLM-R) and demonstrates strong few-shot performance across multilingual settings.

---

## Datasets

Evaluated across multiple languages:

- Arabic Spam–Ham Tweets  
- Urdu (iNLTK News)  
- Urdu Sentiment Corpus v1  
- Pashto Sentiment Dataset  
- Sindhi News Sentiment  
- SST-2 (English)

Focus: low-resource and right-to-left languages.

---

## Repository Structure

```
MV/
│
├── dataset_split_files/
│   └── Scripts for dataset splitting and 4-shot selection
│
├── verbalizer_generation_files/
│   └── MaskedVerbalizer construction and token extraction
│
├── testing_files/
│   └── Final training and evaluation scripts
```

---

## How to Use

### 1️⃣ Split Dataset and Create 4-Shot Setup

Run the scripts inside:

```
dataset_split_files/
```

This prepares:
- 70/30 train-test split  
- 4-shot samples per class  

---

### 2️⃣ Generate MaskedVerbalizer

Run the scripts inside:

```
verbalizer_generation_files/
```

This step:
- Fine-tunes the model on 4-shot data  
- Collects correct masked predictions  
- Builds the final verbalizer automatically  

---

### 3️⃣ Final Training and Testing

Run the scripts inside:

```
testing_files/
```

This performs:
- Final 4-shot training using generated verbalizer  
- Evaluation on the 30% test split  

---

## Summary

MaskedVerbalizer demonstrates that **verbalizer quality is the key bottleneck in prompt-based learning**.  
By leveraging the model’s own masked predictions, we achieve strong few-shot performance in low-resource multilingual classification without manual verbalizer engineering.
