Response process validation of equivalent test forms: How qualitative data can support the construct validity of multiple test forms
Open Access
Author:
Zappe, Sarah Elizabeth
Graduate Program:
Educational Psychology
Degree:
Doctor of Philosophy
Document Type:
Dissertation
Date of Defense:
April 25, 2007
Committee Members:
Hoi Kin Suen, Committee Chair/Co-Chair Dorothy H Evensen, Committee Member Pui Wa Lei, Committee Member James Landis Rosenberger, Committee Member
Keywords:
construct equivalence think-aloud validity test development
Abstract:
When developing multiple test forms, test developers need to be concerned with whether each form is measuring the intended construct in the same manner. This study examined the agreement between three methods of examining construct equivalence on a test of legal case reading and reasoning. The three methods utilized included response process information from think-aloud procedures, expert judgments of similarity, and statistical differential item functioning (DIF) methods. The study showed that the methods did not agree about which item pairs were considered construct equivalent. The think-aloud protocols identified instances of individual items functioning in a different manner than intended, due to construct irrelevant influences on student response such as item wording, ambiguities, item-writer oversight, and failure of the items to conform to item writing guidelines. The study concludes that all three methods provide some unique value in test development. In addition, the benefits of the think-aloud procedure outweigh the costs in terms of time and money.
