A Comparison of Selected Methods for Assessing the Dimensionality of Polytomous Items

Open Access
Author:
Cheng, Weiyi
Graduate Program:
Educational Psychology
Degree:
Doctor of Philosophy
Document Type:
Dissertation
Date of Defense:
April 21, 2017
Committee Members:
  • Pui-Wa Lei, Dissertation Advisor
  • Pui-Wa Lei, Committee Chair
  • Hoi Kin Suen, Committee Member
  • Jonna Marie Kulikowich, Committee Member
  • Mosuk Chow, Outside Member
Keywords:
  • polytomous item
  • dimensionality analysis
  • Monte Carlo simulation
Abstract:
The assessment of test dimensionality of test data is of crucial importance in test development and quality control. An exploratory approach to understand the factor structure can be particularly useful in validation of newly developed instruments. The primary purpose of this study was to investigate the performance of selected methods in determining test dimensionality for polytomous items, the type of assessments that are becoming more popular in current educational and psychological settings. The current study examined the performance of the dimensionality assessment methods through Monte Carlo simulation. Specifically, this study examined the performance of four currently popular methods, weighted least squares estimation with Mplus, Parallel Analysis, the method rooted in conditional covariance theory implemented in PolyDETECT, and Mokken scale analysis. In the simulation, a variety of conditions were manipulated, including sample size, test length, correlations between latent traits, and complexity of the dimensional structure. An understanding of those methods’ differences is important to inform selection and interpretation of their results under varying conditions. The performance of the methods were compared with regard to Type I error rate when one dimension is present, the power to detect the presence of multiple dimensions, and accuracy of assigning items to proper dimensions for multidimensional data sets. Results suggested that the parametric methods are recommended for determining the number of dimensions especially when large sample size, long test length, and/or (approximate) simple structure are present. MSA-AISP is recommended for determining only when inter-trait correlation is extremely high. PolyDETECT is recommended for item classification when test length is short or when the clusters need to be mutually exclusive. The factor analytic methods are recommended for item classification when items are allowed to associate with more than one trait.