<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="theodoropoulos-etal-2021-imposing">
    <titleInfo>
        <title>Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Christos</namePart>
        <namePart type="family">Theodoropoulos</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">James</namePart>
        <namePart type="family">Henderson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Andrei</namePart>
        <namePart type="given">Catalin</namePart>
        <namePart type="family">Coman</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marie-Francine</namePart>
        <namePart type="family">Moens</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-nov</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 25th Conference on Computational Natural Language Learning</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Though language model text embeddings have revolutionized NLP research, their ability to capture high-level semantic information, such as relations between entities in text, is limited. In this paper, we propose a novel contrastive learning framework that trains sentence embeddings to encode the relations in a graph structure. Given a sentence (unstructured text) and its graph, we use contrastive learning to impose relation-related structure on the token level representations of the sentence obtained with a CharacterBERT (El Boukkouri et al., 2020) model. The resulting relation-aware sentence embeddings achieve state-of-the-art results on the relation extraction task using only a simple KNN classifier, thereby demonstrating the success of the proposed method. Additional visualization by a tSNE analysis shows the effectiveness of the learned representation space compared to baselines. Furthermore, we show that we can learn a different space for named entity recognition, again using a contrastive learning objective, and demonstrate how to successfully combine both representation spaces in an entity-relation task.</abstract>
    <identifier type="citekey">theodoropoulos-etal-2021-imposing</identifier>
    <identifier type="doi">10.18653/v1/2021.conll-1.27</identifier>
    <location>
        <url>https://aclanthology.org/2021.conll-1.27</url>
    </location>
    <part>
        <date>2021-nov</date>
        <extent unit="page">
            <start>337</start>
            <end>348</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning
%A Theodoropoulos, Christos
%A Henderson, James
%A Coman, Andrei Catalin
%A Moens, Marie-Francine
%S Proceedings of the 25th Conference on Computational Natural Language Learning
%D 2021
%8 nov
%I Association for Computational Linguistics
%C Online
%F theodoropoulos-etal-2021-imposing
%X Though language model text embeddings have revolutionized NLP research, their ability to capture high-level semantic information, such as relations between entities in text, is limited. In this paper, we propose a novel contrastive learning framework that trains sentence embeddings to encode the relations in a graph structure. Given a sentence (unstructured text) and its graph, we use contrastive learning to impose relation-related structure on the token level representations of the sentence obtained with a CharacterBERT (El Boukkouri et al., 2020) model. The resulting relation-aware sentence embeddings achieve state-of-the-art results on the relation extraction task using only a simple KNN classifier, thereby demonstrating the success of the proposed method. Additional visualization by a tSNE analysis shows the effectiveness of the learned representation space compared to baselines. Furthermore, we show that we can learn a different space for named entity recognition, again using a contrastive learning objective, and demonstrate how to successfully combine both representation spaces in an entity-relation task.
%R 10.18653/v1/2021.conll-1.27
%U https://aclanthology.org/2021.conll-1.27
%U https://doi.org/10.18653/v1/2021.conll-1.27
%P 337-348

Markdown (Informal)

[Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning](https://aclanthology.org/2021.conll-1.27) (Theodoropoulos et al., CoNLL 2021)

• Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning (Theodoropoulos et al., CoNLL 2021)

ACL

• Christos Theodoropoulos, James Henderson, Andrei Catalin Coman, and Marie-Francine Moens. 2021. Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 337–348, Online. Association for Computational Linguistics.