Effect of Characteristics of Peer Rater on Validity of Peer Assessment in Massive Open Online Courses

Open Access
Guo, Xiuyan
Graduate Program:
Educational Psychology
Master of Science
Document Type:
Master Thesis
Date of Defense:
May 01, 2015
Committee Members:
  • Hoi Kin Suen, Thesis Advisor
  • Peer assessment
  • MOOC
  • validity
  • characteristics of peer rater
The present study investigated the effect of the following two characteristics of peer rater on validity of peer assessment in MOOCs: knowledge of course content measured by the assessment and seriousness in making responsible judgments. The extent of participation of MOOC requirements and the course performance are two indicators being used to identify high quality peer raters who have good knowledge of course content; verified certificate registration and rating time are two indicators being used to identify high quality peer raters who demonstrate seriousness judgments. The author attempts to determine whether high quality peer raters can provide more accurate scores than their counterparts in MOOCs. Our findings yielded clear evidence that a peer rater with higher extent of articipation and better course performance, who spend sufficient time in rating peers’ assignment, or who pay the verified certificate fee tend to provide a more accurate score than those with lower extent of participation, poor course performance, who do not spend the necessary amount of time, or who do not pay the verified certificate fee. However, none of them had statistically significant result, and all the effect sizes were very small, showing little practical significance. The limitations of this study due to use of secondary data analyses are discussed. For future research, the author recommends researchers to begin with measuring and investigating some controllable independent variables in a traditional online peer assessment, and then apply the confirmed effective variable in a MOOC environment to collect more accurate and meaningful data.