@inproceedings{huang-etal-2024-self,
    title = "Self-Evaluation of Large Language Model based on Glass-box Features",
    author = "Huang, Hui  and
      Qu, Yingqi  and
      Liu, Jing  and
      Yang, Muyun  and
      Xu, Bing  and
      Zhao, Tiejun  and
      Lu, Wenpeng",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Author-page-Marten-During-lu/2024.findings-emnlp.333/",
    doi = "10.18653/v1/2024.findings-emnlp.333",
    pages = "5813--5820",
    abstract = "The proliferation of open-source Large Language Models (LLMs) underscores the pressing need for evaluation methods. Existing works primarily rely on external evaluators, focusing on training and prompting strategies. However, a crucial aspect {--} model-aware glass-box features {--} is overlooked. In this study, we explore the utility of glass-box features under the scenario of self-evaluation, namely applying an LLM to evaluate its own output. We investigate various glass-box feature groups and discovered that the softmax distribution serves as a reliable quality indicator for self-evaluation. Experimental results on public benchmarks validate the feasibility of self-evaluation of LLMs using glass-box features."
}