University of Twente Student Theses
As of Friday, 8 August 2025, the current Student Theses repository is no longer available for thesis uploads. A new Student Theses repository will be available starting Friday, 15 August 2025.
Usability testing of BuddyGPT: A proprietary and integrated conversational agent to support the learning of university students
Li, Jiawei (2025) Usability testing of BuddyGPT: A proprietary and integrated conversational agent to support the learning of university students.
|
PDF
2MB |
| Abstract: | With the advancement of Education 5.0 and the increasing integration of digital tools in learning environments, AI-powered chatbots have gained growing importance in the field of education. Rooted in the principles of Industry 5.0, the student-centered design philosophy has become a core guideline for developing AI-driven educational tools. In line with this approach, ensuring the usability of AI-powered educational chatbots is an essential precondition to the delivery of high-quality tools to assist students in education. The University of Twente supported the development of BuddyGPT, an AI-powered chatbot designed to assist students in retrieving course-related information from the university educational platform, i.e., Canvas. The present work presents and discusses the results of the usability assessment of BuddyGPT in order to provide actionable design recommendations to support its future development. This study used a within-subjects design. A total of 53 participants were invited to use BuddyGPT to achieve three tasks associated with a course called Software Systems at the University of Twente. During the experience, four main variables were collected, with effectiveness (task success rate and query reformulation rate), efficiency (task completion time), satisfaction (BUS-11 scale), and recommendation intent (individual-level Net Promoter Score). Overall, BuddyGPT enabled participants to complete 77.7% of the tasks. Participants yielded an average satisfaction score of 3.49, indicating the participants were satisfied with BuddyGPT. A group-level Net Promoter Score of 14.3 reflects strong user loyalty and a high likelihood of recommending BuddyGPT. Moreover, we measured relationships among the main factors in the evaluation. For instance, results confirmed the correlation between individual-level Net Promoter Score and satisfaction; nevertheless, they did not find the expected correlation between the effectiveness and efficiency of people's performance in the tasks and their declared satisfaction. These findings offered developers fresh insights for boosting satisfaction with BuddyGPT and demonstrated its value in supporting students’ learning. Finally, we presented practical design recommendations to guide the future development of BuddyGPT and other AI-driven educational chatbots, accelerating their adoption and enhancing their impact in the education sector. |
| Item Type: | Essay (Bachelor) |
| Faculty: | BMS: Behavioural, Management and Social Sciences |
| Subject: | 77 psychology |
| Programme: | Psychology BSc (56604) |
| Link to this item: | https://purl.utwente.nl/essays/107142 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page