Evaluating the Chatbot Usability Scale: A Psychometric and Designometric Perspective

The present study was conducted to further validate the five-factor structure of the BUS-11, as well as its reliability and validity. Moreover, the BUS-11 was assessed from a psychometric and designometric perspective. Finally, other factors possibly affecting satisfaction with chatbots were examined. A Structural Equation Model (SEM) was used to conduct a Confirmatory Factor Analysis (CFA) as well as examine relationships between satisfaction and cognitive workload, previous experience, and a disposition to trust. The psychometric sample consisted of 137 participants and the designometric sample consisted of 22 chatbots. Results confirm the five-factor structure from a psychometric, but not from a designometric perspective. Moreover, the BUS-11 was shown to be a reliable and valid instrument. A negative relationship between satisfaction and cognitive workload was found. This research further validated the BUS-11 from a designometric point of view and showed that it can be used to assess user satisfaction after chatbot interaction. Further research is needed into examining the factorial structure from a designometric perspective as the factorial structure is likely to be different from the psychometric perspective.