University of Twente Student Theses
Assessing Rater Reliability and Assessment Accessibility of a Newly-Designed, High-Stakes, Documented Writing Assessment in The Netherlands
Nijland, Victoire (2024) Assessing Rater Reliability and Assessment Accessibility of a Newly-Designed, High-Stakes, Documented Writing Assessment in The Netherlands.
|
PDF
5MB |
| Abstract: | This study explored how a documented writing assessment was perceived by students and raters before, during and after administration. Six raters assigning ratings to nine subdomains based on 26 performances presented a G-coefficient of .81 for agreement without training or exemplars. A G-coefficient of .68 was found for two raters. Further research should determine if instructions improve agreement levels. Qualitative results showed that raters rely on their beliefs about the concept and its domains, and rating experience in scoring stages to assign ratings that discriminate between students’ writing abilities. Raters could identify differences between performances, and are comfortable applying the criteria. It also means that this sample of raters have a mutual understanding of the concept documented writing and its domains. Assessment accessibility factors, effectiveness of instructed writing strategies, difficulty level of sources in documentation file and assessment administration setting significantly correlated on obtained writing scores. Hence, when making decisions on students’ relative standing in writing ability using this writing task, the impact of these factors should be considered when determining pass/ fail grades. Also, the design and support materials of a standardized writing assessment should be aligned with the construct based on language abstraction, language level and genre. |
| Item Type: | Essay (Master) |
| Faculty: | BMS: Behavioural, Management and Social Sciences |
| Subject: | 81 education, teaching |
| Programme: | Educational Science and Technology MSc (60023) |
| Link to this item: | https://purl.utwente.nl/essays/99934 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page