Daan Kolkman


2024

pdf
Clearer Governmental Communication: Text Simplification with ChatGPT Evaluated by Quantitative and Qualitative Research
Nadine Beks van Raaij | Daan Kolkman | Ksenia Podoynitsyna
Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING 2024

This research investigates the application of ChatGPT for the simplification of Dutch government letters, aiming to enhance their comprehensibility without compromising legal accuracy. We use a three-stage mixed method evaluation procedure to compare the performance of a naive approach, RoBERTA, and ChatGPT. We select the six most complicated letters from a corpus of 200 letters and use the three approaches to simplify them. First, we compare their scores on four evaluation metrics (ROUGE, BLEU, BLEURT, and LiNT), then we assess the simplifications with a legal and linguistic expert. Finally we investigate the performance of ChatGPT in a randomized controlled trial with 72 participants. Our findings reveal that ChatGPT significantly improves the readability of government letters, demonstrating over a 20% increase in comprehensibility scores and a 19% increase in correct question answering among participants. We also demonstrate the importance of a robust evaluation procedure.

2022

pdf
The Elementary Scenario Component Metric for Summarization Evaluation
Martin Kirilov | Daan Kolkman | Bert-Jan Butijn
Proceedings of the 5th International Conference on Natural Language and Speech Processing (ICNLSP 2022)