Evaluating Agent Interactions Through Episodic Knowledge Graphs

Selene Baez Santamaria, Piek Vossen, Thomas Baier


Abstract
We present a new method based on episodic Knowledge Graphs (eKGs) for evaluating (multimodal) conversational agents in open domains. This graph is generated by interpreting raw signals during conversation and is able to capture the accumulation of knowledge over time. We apply structural and semantic analysis of the resulting graphs and translate the properties into qualitative measures. We compare these measures with existing automatic and manual evaluation metrics commonly used for conversational agents. Our results show that our Knowledge-Graph-based evaluation provides more qualitative insights into interaction and the agent’s behavior.
Anthology ID:
2022.ccgpk-1.3
Volume:
Proceedings of the 1st Workshop on Customized Chat Grounding Persona and Knowledge
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Editors:
Heuiseok Lim, Seungryong Kim, Yeonsoo Lee, Steve Lin, Paul Hongsuck Seo, Yumin Suh, Yoonna Jang, Jungwoo Lim, Yuna Hur, Suhyune Son
Venue:
CCGPK
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
15–28
Language:
URL:
https://aclanthology.org/2022.ccgpk-1.3
DOI:
Bibkey:
Cite (ACL):
Selene Baez Santamaria, Piek Vossen, and Thomas Baier. 2022. Evaluating Agent Interactions Through Episodic Knowledge Graphs. In Proceedings of the 1st Workshop on Customized Chat Grounding Persona and Knowledge, pages 15–28, Gyeongju, Republic of Korea. Association for Computational Linguistics.
Cite (Informal):
Evaluating Agent Interactions Through Episodic Knowledge Graphs (Baez Santamaria et al., CCGPK 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl-24-ws-corrections/2022.ccgpk-1.3.pdf
Code
 selbaez/evaluating-conversations-as-ekg
Data
Topical-Chat