University of Twente Student Theses

Login

Automatic Short Answer Grading using Text-to-Text Transfer Transformer Model

Haller, Stefan (2020) Automatic Short Answer Grading using Text-to-Text Transfer Transformer Model.

[img] PDF
3MB
Abstract:We explore in this study the effects of multi-task training and domain adaptation on Automatic Short Answer Grading (ASAG) using the text-to-text transfer transformer model (T5). Within this study, we design an ASAG model and evaluate its applicability to a practice dataset. We fine-tuned a multi-task model that is trained on a profound selection of related tasks and an extensively pre-trained model. We evaluate the performance of the models on the SciEntsBank dataset and achieved new-state-of-the-art results. With the best performing model, we showed that domain-independent fine-tuning is preferable to domain-specific fine-tuning for data-sparse cases. The optimized model was used and demonstrated in the university context. The prediction behavior of the model was explained with different model-agnostic methods which resulted in several hypotheses. The reported results reveal that the model is biased towards correct answers and has problems with partially correct answers. Through the knowledge about the decision behavior, the model's robustness was evaluated and tested. Within a validation study, we asked students to generate manipulation answers. Our findings emphasize the susceptibility of the model towards manipulations and difficulties with handling imbalanced and sparse data. We observe that for a functional ASAG model balanced and extensive data are necessary.
Item Type:Essay (Master)
Faculty:EEMCS: Electrical Engineering, Mathematics and Computer Science
Subject:54 computer science, 81 education, teaching
Programme:Business Information Technology MSc (60025)
Link to this item:https://purl.utwente.nl/essays/83879
Export this item as:BibTeX
EndNote
HTML Citation
Reference Manager

 

Repository Staff Only: item control page