Content area
Plný text
Abstract: Context: In science, technology, engineering, and mathematics (STEM) subjects, project-based assessment has been a key component of the assessment landscape, due to its authenticity and effectiveness as a learning approach. Problem: Large classes require marking of student assignments to be distributed among several markers. This raises the problem of consistency among markers, who may interpret assignment requirements and marking scales differently. Approach: We tried two approaches to rubric design, in attempt to provide markers with clear guidelines for assigning marks, and thereby yield consistent results among markers. The first approach, employed to mark final reports submitted by engineering students in a capstone project class, used a 10-point qualitative scale that markers used to mark components of an assignment. The second, employed in marking a series of deliverables submitted by students as a group project in a Master's level computer science project class, used "Yes/No" questions to assessed components of each deliverable. These were then aggregated, and converted into scores on the same 10-point qualitative scale. Results: We found that both approaches facilitated speedy but thorough marks: the 10-point rubric allowed markers to mark an entire final report in 60 minutes or less, while the "Yes/No" rubrics required between 10 and 20 minutes per deliverable, depending on size. Consistency was good for both approaches, with the "Yes/No" approach producing higher consistency at the expense of greater up-front effort. Also, while markers appreciated the speed of marking enabled by "Yes/No" rubrics, some were frustrated that there was no middle mark that they could apply to certain components. The 10-point rubric was easier to create, but produced lower initial consistency, which then had to be resolved in post-marking discussions between markers. Conclusion: Both approaches enable acceptable consistency when different markers mark the same subset of student assignments. The 10-point rubric takes little time to prepare and deploy, but requires more effort on the part of markers to produce acceptable consistency. The "Yes/No" rubric is easier for the markers to use, but requires more initial effort to create. Thus, the choice between one or the other is a trade-off between when effort is available to expend on consistency.
Keywords: rubrics, reliability, e-assessment, STEM, assessing large cohorts
1.Introduction
Engineering and computer science courses include a substantial amount of...