Exploring automatic scoring of an observation-based multimedia-based performance assessment

Performance-based assessment (PBA) is the most used assessment method in vocational education. However, the reliability and validity of PBA are questioned, mainly because of assessor, occasion, and task induced errors. Simulation-based assessment (SBA), especially with the use of technology, is becoming an increasingly popular assessment method for vocational education because it can strengthen and broaden this domain of assessment. This is especially the case for multimedia-based performance assessment (MBPA) which is a fully virtual type of SBA. Because of this, MBPA is more standardizable among other advantages. This study explored how to validly standardize a PBA by creating an observation-based MBPA using VR-technology for technical vocational education. The study consisted of two parts. First, an observation-based MBPA and an automated algorithm were designed and developed using the MBPA framework of de Klerk, Veldkamp, and Eggen (2018), and a focus group of subject matter experts. During the second part of the study, data in the form of video captures were gathered from participants who performed in the observation-based MBPA. The scoring of this data by two subject matter experts was compared with the scores of the automated algorithm. A high correlation was found between the experts and the algorithm. Furthermore, variation between the experts showed assessor induced errors, which as the results also showed, could even cause unjustifiable scoring of performance on high-risk tasks. This study laid a basis to further explore the automatization of observation-based MBPA for more complex assessments.