<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="geva-etal-2020-injecting">
    <titleInfo>
        <title>Injecting Numerical Reasoning Skills into Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Mor</namePart>
        <namePart type="family">Geva</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ankit</namePart>
        <namePart type="family">Gupta</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jonathan</namePart>
        <namePart type="family">Berant</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-jul</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GenBERT, on this data, dramatically improves performance on DROP (49.3 –\textgreater 72.3 F1), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GenBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation.</abstract>
    <identifier type="citekey">geva-etal-2020-injecting</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-main.89</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-main.89</url>
    </location>
    <part>
        <date>2020-jul</date>
        <extent unit="page">
            <start>946</start>
            <end>958</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Injecting Numerical Reasoning Skills into Language Models
%A Geva, Mor
%A Gupta, Ankit
%A Berant, Jonathan
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
%D 2020
%8 jul
%I Association for Computational Linguistics
%C Online
%F geva-etal-2020-injecting
%X Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GenBERT, on this data, dramatically improves performance on DROP (49.3 –\textgreater 72.3 F1), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GenBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation.
%R 10.18653/v1/2020.acl-main.89
%U https://aclanthology.org/2020.acl-main.89
%U https://doi.org/10.18653/v1/2020.acl-main.89
%P 946-958

Markdown (Informal)

[Injecting Numerical Reasoning Skills into Language Models](https://aclanthology.org/2020.acl-main.89) (Geva et al., ACL 2020)

• Injecting Numerical Reasoning Skills into Language Models (Geva et al., ACL 2020)

ACL

• Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. Injecting Numerical Reasoning Skills into Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 946–958, Online. Association for Computational Linguistics.