University of Twente Student Theses
The chatbot usability scale : an evaluation of the Dutch version of the BUS-11
Huijsmans, Maarten (2022) The chatbot usability scale : an evaluation of the Dutch version of the BUS-11.
|
PDF
1MB |
| Abstract: | The chatbot market has grown considerably in recent years. One domain in which the demand for chatbots has increased is customer service. Despite this development, there is a lack of standardized methods for evaluating the perceived quality of these systems. The current study evaluated the BUS-11, a questionnaire used to evaluate user satisfaction with chatbots, from both a psychometric and a designometric perspective. This questionnaire is a shorter version of the BUS-15 (Borsci et al., 2021). The five-factor model, originally developed for the BUS-15 by Borsci et al., was evaluated using a confirmatory factor analysis and resulted in an 11-item scale (BUS 11, see: Borsci et al., 2021). In the present study, we aimed to replicate the confirmatory validation of the Dutch version of the BUS-11, the reliability of the questionnaire was evaluated by examining its internal consistency. The BUS-11’s concurrent validity was examined by comparing it to the UMUX-Lite and RSME. Finally, the current study explored the effect of previous experiences as declared by participants in using chatbots on user satisfaction as measured by the BUS-11. Results of the confirmatory factor analysis indicated a good fit of the five-factor model for the psychometric perspective, while a potential problem associated with the first item of the scale was identified. From the designometric point of view, due to limited data, it was hard to establish the quality of the factorial model. One of the 11 questionnaire items showed low reliability, a finding which is in line with previous research. This item may be too general. Overall, the BUS-11 was found to have good internal consistency. Furthermore, a Spearman’s rank-order test indicated a strong positive correlation of the BUS-11 with both the UMUX-Lite and RSME. No effect of previous experience on user satisfaction as measured by the BUS-11 was found. The findings of the current study suggest that the BUS-11 is a reliable and valid tool for measuring user satisfaction with chatbots at least from the psychometric point of view. Further research is needed to confirm or reject the five-factor model from the designometric perspective. |
| Item Type: | Essay (Master) |
| Faculty: | BMS: Behavioural, Management and Social Sciences |
| Subject: | 05 communication studies, 50 technical science in general, 54 computer science |
| Programme: | Psychology MSc (66604) |
| Link to this item: | https://purl.utwente.nl/essays/90654 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page