University of Twente Student Theses
Applying Text Mining and Machine Learning to Build Methods for Automated Grading
Psalmerosi, Febriya Hotriati (2019) Applying Text Mining and Machine Learning to Build Methods for Automated Grading.
|
PDF
3MB |
| Abstract: | Nowadays, machine learning and text mining have become an interesting topic for both research and practice. The impact of machine learning and text mining technologies is significant in any area of the business or the public sector, including education. Specifically, in education, one of the most interesting applications of these technologies is in the evaluation of students' tests' results that come out of open-question-based examination processes. This thesis responds to the trend to employ machine learning and text mining techniques in evaluation of students' responses to open questions. The present research is focused on the identification of the best approach to automate the grading of students' answers in open-question-based examination. To this end, we conducted a comparative study of a set of alternative methods. In open-question-based examination, there are several types of open questions, however previous researches have been done for the essay and short answer only. This study explores the grading process as supported by machine learning and text mining techniques, regarding two types of open questions: (1) "mention and explain a couple of examples for different categories," and (2) "give a concise and valid argument about a given statement." Additionally, the present study focuses on finding better approaches for the small dataset (less than 50) in contrast to previously published literature which tends to investigate their method in a big dataset. Therefore, current research provides several contributions to the theory. This study examines other open question types that have not been explored in previous works. This research also proposes techniques for automated grading system using combinations of text mining and machine learning for an automated grading system for the small dataset. Then, this study demonstrates the use of RapidMiner for automated grading implementation. This study uses text mining and machine learning techniques to assess each question type. Unlike related works in this area, the present study does not aim to give a score to a student's answer, but to examine those characteristics of an open question that can be advantageous for automated grading. Therefore, this research provides several suggestions for lecturers about how to create a question that can be easily graded by an automated system and to determine the performance of the implemented technique for two types of question. Current research evaluates the proposed method in two ways: (1) by doing an experiment, and (2) by conducting an evaluation survey from three lecturers in the University of Twente. The first type of open questions is examined by counting the number of examples mentioned in the answer and by employing a classification technique using Support Vector Machine. The related experiment findings show the acceptable result to identify the number of examples within a category, with the accuracy of more than 70%. Moreover, the produced classifier model identifies the examples to its category with the accuracy of more than 85% and correlation value more than 0.700. These values signify high likelihood that students' answers are similar and related to each other. For the second type, this study implements sentiment analysis and clustering with X-Means algorithm. The Davies-Bouldin (DB) index and Silhouette index are applied to measure the performance of the clustering. The optimal number of clusters is 7, using Manhattan Distance with DB index, which is 0.334, and Silhouette index which is 1.332. Our analysis found that answer length is the most dominant factor in determining the clusters, and Term Document Matrix influences the results of the clustering. This master thesis project used RapidMiner for the purpose of experimentation. All answer files are written in text files. In addition to the experiment results, an evaluation survey based on the Unified Theory of Acceptance and Use of Technology (UTAUT) by Venkatesh et al. (2003) was performed. From the evaluation results, performance expectancy becomes the strong determinant of the behavioral intention to use the proposed method. The most negative feedback is self-efficacy construct as there is a possibility that all participants think introduction session is important before using the proposed method. |
| Item Type: | Essay (Master) |
| Faculty: | EEMCS: Electrical Engineering, Mathematics and Computer Science |
| Subject: | 54 computer science |
| Programme: | Business Information Technology MSc (60025) |
| Link to this item: | https://purl.utwente.nl/essays/77190 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page