@inproceedings{engelmann-etal-2024-arts,
    title = "{ARTS}: Assessing Readability {\&} Text Simplicity",
    author = {Engelmann, Bj{\"o}rn  and
      Kreutz, Christin Katharina  and
      Haak, Fabian  and
      Schaer, Philipp},
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2024.findings-emnlp.877/",
    doi = "10.18653/v1/2024.findings-emnlp.877",
    pages = "14925--14942",
    abstract = "Automatic text simplification aims to reduce a text`s complexity. Its evaluation should quantify how easy it is to understand a text. Datasets with simplicity labels on text level are a prerequisite for developing such evaluation approaches. However, current publicly available datasets do not align with this, as they mainly treat text simplification as a relational concept ({\textquotedblleft}How much simpler has this text gotten compared to the original version?{\textquotedblright}) or assign discrete readability levels.This work alleviates the problem of Assessing Readability {\&} Text Simplicity. We present ARTS, a method for language-independent construction of datasets for simplicity assessment. We propose using pairwise comparisons of texts in conjunction with an Elo algorithm to produce a simplicity ranking and simplicity scores. Additionally, we provide a high-quality human-labeled and three GPT-labeled simplicity datasets. Our results show a high correlation between human and LLM-based labels, allowing for an effective and cost-efficient way to construct large synthetic datasets."
}