<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="robinson-etal-2010-dialogues">
    <titleInfo>
        <title>Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Susan</namePart>
        <namePart type="family">Robinson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Antonio</namePart>
        <namePart type="family">Roque</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">David</namePart>
        <namePart type="family">Traum</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2010-may</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)</title>
        </titleInfo>
        <originInfo>
            <publisher>European Language Resources Association (ELRA)</publisher>
            <place>
                <placeTerm type="text">Valletta, Malta</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>As conversational agents are now being developed to encounter more complex dialogue situations it is increasingly difficult to find satisfactory methods for evaluating these agents. Task-based measures are insufficient where there is no clearly defined task. While user-based evaluation methods may give a general sense of the quality of an agent’s performance, they shed little light on the relative quality or success of specific features of dialogue that are necessary for system improvement. This paper examines current dialogue agent evaluation practices and motivates the need for a more detailed approach for defining and measuring the quality of dialogues between agent and user. We present a framework for evaluating the dialogue competence of artificial agents involved in complex and underspecified tasks when conversing with people. A multi-part coding scheme is proposed that provides a qualitative analysis of human utterances, and rates the appropriateness of the agent’s responses to these utterances. The scheme is outlined, and then used to evaluate Staff Duty Officer Moleno, a virtual guide in Second Life.</abstract>
    <identifier type="citekey">robinson-etal-2010-dialogues</identifier>
    <location>
        <url>http://www.lrec-conf.org/proceedings/lrec2010/pdf/674_Paper.pdf</url>
    </location>
    <part>
        <date>2010-may</date>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue
%A Robinson, Susan
%A Roque, Antonio
%A Traum, David
%S Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)
%D 2010
%8 may
%I European Language Resources Association (ELRA)
%C Valletta, Malta
%F robinson-etal-2010-dialogues
%X As conversational agents are now being developed to encounter more complex dialogue situations it is increasingly difficult to find satisfactory methods for evaluating these agents. Task-based measures are insufficient where there is no clearly defined task. While user-based evaluation methods may give a general sense of the quality of an agent’s performance, they shed little light on the relative quality or success of specific features of dialogue that are necessary for system improvement. This paper examines current dialogue agent evaluation practices and motivates the need for a more detailed approach for defining and measuring the quality of dialogues between agent and user. We present a framework for evaluating the dialogue competence of artificial agents involved in complex and underspecified tasks when conversing with people. A multi-part coding scheme is proposed that provides a qualitative analysis of human utterances, and rates the appropriateness of the agent’s responses to these utterances. The scheme is outlined, and then used to evaluate Staff Duty Officer Moleno, a virtual guide in Second Life.
%U http://www.lrec-conf.org/proceedings/lrec2010/pdf/674_Paper.pdf

Markdown (Informal)

[Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue](http://www.lrec-conf.org/proceedings/lrec2010/pdf/674_Paper.pdf) (Robinson et al., LREC 2010)

• Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue (Robinson et al., LREC 2010)

ACL

• Susan Robinson, Antonio Roque, and David Traum. 2010. Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta. European Language Resources Association (ELRA).