University of Twente Student Theses


Development of a Preliminary Measurement Tool of User Satisfaction for Information-Retrieval Chatbots

Waldera, L. (2019) Development of a Preliminary Measurement Tool of User Satisfaction for Information-Retrieval Chatbots.

[img] PDF
Abstract:Based on the findings of Tariverdiyeva and Borsci (2019), it can be concluded that the implementation of 18 key features of information-retrieval chatbots are directly leading to a change in perceived quality and user satisfaction. Standard measurement tools for the usability of software, like UMUX-Lite (Lewis et al., 2013), are found to be failing at capturing these specific requirements for chatbots. This research aims to reassess and extend the list of relevant key features found by Tariverdiyeva and Borsci (2019). Moreover, this research aims to develop a preliminary list of questionnaire items that can be used in a future measurement tool of user satisfaction for information-retrieval chatbots from an end user’s perspective. This is achieved by conducting a systematic literature review and discussions with possible end users in focus groups in the first experimental phase and conducting a usability testing with different chatbots in the second experimental phase. After the first phase, a total of 14 features were found to be relevant to measure user satisfaction. Questionnaire items for these features were generated and used for the development of a preliminary questionnaire – UMT/C. After the second experimental phase, the UMT/C was found to be correlating with the standard measurement tool UMUX-Lite. A factor analysis was applied to the questionnaire data which showed a total of 9 factors being measured by 25 items. While some of the factors correlated highly with the results of the UMUX-Lite scale, others were found to capture additional information related to user satisfaction. Therefore, the present study recommends to test the quality of this preliminary questionnaire and to refine the pool of items in order to develop a valid and reliable tool for assessing the quality of chatbots from an end user’s perspective. Keywords: information-retrieval chatbots, usability testing, UMUX-Lite, questionnaire development, factor analysis
Item Type:Essay (Bachelor)
Faculty:BMS: Behavioural, Management and Social Sciences
Subject:77 psychology
Programme:Psychology BSc (56604)
Link to this item:
Export this item as:BibTeX
HTML Citation
Reference Manager


Repository Staff Only: item control page