<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="haffari-etal-2017-efficient">
    <titleInfo>
        <title>Efficient Benchmarking of NLP APIs using Multi-armed Bandits</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Gholamreza</namePart>
        <namePart type="family">Haffari</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tuan</namePart>
        <namePart type="given">Dung</namePart>
        <namePart type="family">Tran</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mark</namePart>
        <namePart type="family">Carman</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2017-apr</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Valencia, Spain</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Comparing NLP systems to select the best one for a task of interest, such as named entity recognition, is critical for practitioners and researchers. A rigorous approach involves setting up a hypothesis testing scenario using the performance of the systems on query documents. However, often the hypothesis testing approach needs to send a lot of document queries to the systems, which can be problematic. In this paper, we present an effective alternative based on the multi-armed bandit (MAB). We propose a hierarchical generative model to represent the uncertainty in the performance measures of the competing systems, to be used by Thompson Sampling to solve the resulting MAB. Experimental results on both synthetic and real data show that our approach requires significantly fewer queries compared to the standard benchmarking technique to identify the best system according to F-measure.</abstract>
    <identifier type="citekey">haffari-etal-2017-efficient</identifier>
    <location>
        <url>https://aclanthology.org/E17-1039</url>
    </location>
    <part>
        <date>2017-apr</date>
        <extent unit="page">
            <start>408</start>
            <end>416</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Efficient Benchmarking of NLP APIs using Multi-armed Bandits
%A Haffari, Gholamreza
%A Tran, Tuan Dung
%A Carman, Mark
%S Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers
%D 2017
%8 apr
%I Association for Computational Linguistics
%C Valencia, Spain
%F haffari-etal-2017-efficient
%X Comparing NLP systems to select the best one for a task of interest, such as named entity recognition, is critical for practitioners and researchers. A rigorous approach involves setting up a hypothesis testing scenario using the performance of the systems on query documents. However, often the hypothesis testing approach needs to send a lot of document queries to the systems, which can be problematic. In this paper, we present an effective alternative based on the multi-armed bandit (MAB). We propose a hierarchical generative model to represent the uncertainty in the performance measures of the competing systems, to be used by Thompson Sampling to solve the resulting MAB. Experimental results on both synthetic and real data show that our approach requires significantly fewer queries compared to the standard benchmarking technique to identify the best system according to F-measure.
%U https://aclanthology.org/E17-1039
%P 408-416

Markdown (Informal)

[Efficient Benchmarking of NLP APIs using Multi-armed Bandits](https://aclanthology.org/E17-1039) (Haffari et al., EACL 2017)

• Efficient Benchmarking of NLP APIs using Multi-armed Bandits (Haffari et al., EACL 2017)

ACL

• Gholamreza Haffari, Tuan Dung Tran, and Mark Carman. 2017. Efficient Benchmarking of NLP APIs using Multi-armed Bandits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 408–416, Valencia, Spain. Association for Computational Linguistics.