University of Twente Student Theses
Evaluating ChatGPT’s Ability for Automated Dialogue Analysis
Sedaghat, Andisheh (2024) Evaluating ChatGPT’s Ability for Automated Dialogue Analysis.
|
PDF
738kB |
| Abstract: | This study examines the potential of ChatGPT in analyzing student dialogue to assess learning, particularly within online learning environments where both social and individual contributions matter. The research is grounded in the Academically Productive Talk framework by Michaels & O’Connor (2012), which emphasizes four goals: sharing knowledge, attentive listening, deepening reasoning, and collaborative thinking. To evaluate ChatGPT’s effectiveness, the study compared its performance in deductive coding, which uses predefined categories to analyze dialogue, with that of a human researcher. Findings revealed that ChatGPT effectively identified key utterances that contribute to productive dialogue, showing its potential to support deductive data analysis when the limitation in specifying each individual goal is not a primary concern. Additionally, the results indicated that the presence of certain indicative vocabulary impacts ChatGPT’s performance. The inter-rater reliability analysis, taking into account the characteristics of dialogues, showed that ChatGPT was particularly effective in coding instances where students shared knowledge during verbose dialogues, listened attentively to each other in short dialogues, and engaged in longer dialogues with high word counts and balanced word distribution among students. Furthermore, higher agreement levels were observed in short dialogues when students deepened their reasoning and engaged in collaborative thinking. These insights first inform researchers and educators for better handling of the prompts and also customizing AI tools in designing educational, supportive tools. Second, it calls for further research to build on these initial findings and enhance the integration of AI in educational contexts. |
| Item Type: | Essay (Master) |
| Faculty: | BMS: Behavioural, Management and Social Sciences |
| Subject: | 81 education, teaching |
| Programme: | Educational Science and Technology MSc (60023) |
| Link to this item: | https://purl.utwente.nl/essays/104686 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page