Chatbots’ Perceived Usability in Information Retrieval Tasks: An Exploratory Analysis.

Despite many benefits, a substantial number of chatbots struggle to satisfy users. To understand what factors influence interaction with a chatbot and how to assess those factors, the present study aimed to; (a) explore the factors that are essential for user satisfaction, (b) investigate UMUX-Lite scale as a usability measure of a chatbot, and to (c) propose a design approach for a potential usability questionnaire. The research consisted of three phases: a systematic literature review, an online survey, and an interaction test. A comprehensive literature review identified 27 factors that influence interaction with a chatbot. Further, the initial list was filtered via the online survey in which 11 experts and 9 end-users participated. Next, the interaction test with 15 participants was conducted to get the perspective of users who just had experience with chatbots and to assess UMUX-Lite as a usability measure for chatbots. The online survey and interaction test distinguished 18 factors as important for satisfactory chatbot interactions. Furthermore, the current study found that UMUX-Lite is reasonably sensitive and a reliable measure of usability. However, as UMUX-lite does not include all the aspects important for measuring the perceived usability of the chatbot there is a concern about the extent of the sensitivity and validity of the scale. These findings suggest a need for a post-test questionnaire that will be able to capture more aspects of interaction with chatbots. Therefore, the present study recommends the development of a new questionnaire that will incorporate UMUX-Lite and proceed further on the basis of the key factors established in this study.