Testing a scale for perceived usability and user satisfaction in chatbots: Testing the BotScale

The present study aimed to replicate the study by Balaji and Borsci (2019), who have developed a 42-item chatbot scale (BotScale) with four factors. Fifty volunteers participated in the assessment of nine chatbots by using an English and Dutch version of the BotScale. Additionally, an already existing scale for voice interfaces, the Speech User Interface Service Quality (SUISQ-R) (Lewis & Hardzinski, 2015), was used to check for external validity. A principal component analysis on a chatbot x item dataset was performed to reduce the original scale from 42 to 14 items by looking at factor loadings, item-total correlations and reliability. Results show that a four-factor structure for the BotScale captures 85% present of variance and the 14-item BotScale had a reliability of α = 0.93. A correlational analysis between the Dutch and English version of the BotScale showed that the Dutch translation was as reliable (α = 0.89) as the original scale (α = 0.93). Furthermore, the Dutch version had a significant strong positive correlation to the English version of the BotScale. Finally, a correlation between the BotScale and the SUISQ-R was performed, and results suggested a significant positive moderate correlation.