EFFECT OF QUALITY CHARACTERISTICS OF PEER RATER ON THE RELIABILITY AND VALIDITY OF PEER ASSESSMENT
Open Access
- Author:
- Guo, Xiuyan
- Graduate Program:
- Educational Psychology
- Degree:
- Doctor of Philosophy
- Document Type:
- Dissertation
- Date of Defense:
- July 07, 2017
- Committee Members:
- Hoi Kin Suen, Dissertation Advisor/Co-Advisor
Hoi Kin Suen, Committee Chair/Co-Chair
Pui-Wa Lei, Committee Member
Mosuk Chow, Committee Member
Mindy L Kornhaber, Outside Member - Keywords:
- Peer assessment
Rater error
Reliability
Validity
Rating motivation
Content knowledge
Training
Rater characteristics - Abstract:
- There is a large body of research on influential factors of the reliability and validity of peer assessment scores. Far less research has been done on the effect of peer raters’ quality characteristics. This study investigated the effects of the following three characteristics of peer rater on the reliability and validity of peer assessment: knowledge of course content measured by the assessment, rater training experience, and rater motivation in making responsible judgments. A quasi-experimental, three-way completely crossed factorial design was used to analyze data from 838 students in 24 classes in a typical Chinese high school. The students rated four well-selected essays written by their peers using the standard Gaokao (National College Entrance Examination) Chinese Wring Rubric. In this study, knowledge of course content is measured by an exam score in the assessed domain. Training experience is measured by (1) the number of times a student has previously participated in a peer assessment, and (2) whether the student received self-paced training in this study. Motivation of the peer rater is measured by (1) whether they were offered monetary incentives, and (2) whether they have been told about the general benefits of completing peer assessment seriously before conducting the peer assessment in this study. It was hypothesized that peer raters with high quality characteristics would provide more reliable and valid scores than their counterparts. All the data were be analyzed using a series of two-level, hierarchical linear models. The findings confirmed the hypothesis, and yielded clear evidence that peer raters with more knowledge in the assessed domain, more peer rating experience, or higher rating motivation can provide more reliable and more valid scores than those with less knowledge in assessed domain, less peer rating experience, or lower rating motivation. This study also found significant interaction effects between peer raters’ content knowledge and rating motivation, and rating experience and rating motivation. Implications for in person and online peer assessments are discussed, as well as suggestions for further research.