Please use this identifier to cite or link to this item: https://repositorio.uti.edu.ec//handle/123456789/3034
Title: Experimentation for Chatbot Usability Evaluation: A Secondary Study
Authors: Ren, Ranci
Zapata, Mireya
Castro, John
Dieste, Oscar
Acuna, Silvia
Issue Date: 2022
Publisher: IEEE Access. Volume 10, Pages 12430 - 12464
Abstract: Interest in chatbot development is on the rise. As a usability evaluation is an essential step in chatbot development, the number of experimental studies on chatbot usability has grown as well. As a result, we think a systematic mapping study is opportune. We analyzed more than 700 sources and retrieved 28 primary studies. By aggregating the research questions and examining the characteristics and metrics used to evaluate the usability of chatbots in experiments, it is possible to identify the state of the art in chatbot usability experimentation. We conducted a systematic mapping study to identify the research questions, characteristics, and metrics used to evaluate the usability of chatbots in experiments. Most experiments adopted a within-subjects design. On the other hand, few experiments provided raw data, and only one of the identified papers was part of a family of experiments. Effectiveness, efficiency, and satisfaction are usability characteristics used to identify how well users can learn and use chatbots to achieve their goals and how satisfied users are during the interaction. Generally, the experimental results revealed that chatbots have several advantages (e.g., they provide a real-time response and they improve ease of use) and some shortcomings (e.g., natural language processing, which is rated as the weakness most in need of improvement). This research offers an overview of chatbot usability experimentation. The increasing interest in this area is very recent, as works did not start to be published until 2018. Chatbot usability experiments should be more replicable to improve the reliability and transparency of the experimental results.
URI: https://ieeexplore.ieee.org/document/9687577
http://repositorio.uti.edu.ec//handle/123456789/3034
Appears in Collections:Artículos Científicos Indexados

Files in This Item:
There are no files associated with this item.


This item is licensed under a Creative Commons License Creative Commons