@inproceedings{barale-etal-2023-language,
    title = "Do Language Models Learn about Legal Entity Types during Pretraining?",
    author = "Barale, Claire  and
      Rovatsos, Michael  and
      Bhuta, Nehal",
    editor = "Preoțiuc-Pietro, Daniel  and
      Goanta, Catalina  and
      Chalkidis, Ilias  and
      Barrett, Leslie  and
      Spanakis, Gerasimos  and
      Aletras, Nikolaos",
    booktitle = "Proceedings of the Natural Legal Language Processing Workshop 2023",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2023.nllp-1.4/",
    doi = "10.18653/v1/2023.nllp-1.4",
    pages = "25--37",
    abstract = "Language Models (LMs) have proven their ability to acquire diverse linguistic knowledge during the pretraining phase, potentially serving as a valuable source of incidental supervision for downstream tasks. However, there has been limited research conducted on the retrieval of domain-specific knowledge, and specifically legal knowledge. We propose to explore the task of Entity Typing, serving as a proxy for evaluating legal knowledge as an essential aspect of text comprehension, and a foundational task to numerous downstream legal NLP applications. Through systematic evaluation and analysis and two types of prompting (cloze sentences and QA-based templates) and to clarify the nature of these acquired cues, we compare diverse types and lengths of entities both general and domain-specific entities, semantics or syntax signals, and different LM pretraining corpus (generic and legal-oriented) and architectures (encoder BERT-based and decoder-only with Llama2). We show that (1) Llama2 performs well on certain entities and exhibits potential for substantial improvement with optimized prompt templates, (2) law-oriented LMs show inconsistent performance, possibly due to variations in their training corpus, (3) LMs demonstrate the ability to type entities even in the case of multi-token entities, (4) all models struggle with entities belonging to sub-domains of the law (5) Llama2 appears to frequently overlook syntactic cues, a shortcoming less present in BERT-based architectures."
}