Leveraging a Human-Machine Collaboration to Understand the Structure of Short-Answer Student Response Data

restricted_to_institution
Restricted (Penn State Only)
Author:
Lloyd, Susan
Graduate Program:
Statistics
Degree:
Doctor of Philosophy
Document Type:
Dissertation
Date of Defense:
June 06, 2025
Committee Members:
Runze Li, Professor in Charge/Director of Graduate Studies
Neil Hatfield, Major Field Member
Dennis Pearl, Major Field Member
Matthew Beckman, Chair & Co-Dissertation Advisr
Rebecca Passonneau, Outside Unit & Field Member
Nicole Lazar, Dissertation Co-Advisor
Keywords:
Human-Machine Collaboration
Short-Answer Student Response Data
Formative Assessment
Clustering
Statistical Literacy Surrounding Confidence Intervals
Educational Assessment
Topological Data Analysis
Abstract:
This work leverages a human-machine collaboration to understand the structure of short-answer student response data. All three projects in this dissertation share a common theme of scoring, clustering, and providing feedback to students on the basis of their short-answer responses to statistical tasks. As a result, each of these projects would directly, or indirectly, benefit from the use of a human-machine collaboration to improve the scalability of quality formative assessment. In the first project, we study extant data consisting of responses to short-answer tasks about statistical inference. We conduct a thorough scoring reliability analysis among trained human raters, and find evidence of substantial agreement among the raters. Using the reliability estimates from the human raters as a baseline, we establish that an automated (i.e., algorithmic) rater achieves similarly strong reliability measures. We propose, and test, a human-machine collaboration to work towards improving the feasibility of scalable formative assessment. We pilot a manual cluster analysis of feedback to these short-answer responses, and find strong evidence human-generated clustering can serve as a guided process for algorithmic clustering. In the second project, we pilot an innovative approach to educational assessment. We use a factorial design to develop an instrument to measure students’ statistical literacy surrounding confidence intervals upon the conclusion of an introductory statistics course with simulation-based inference methods. This design allows for the opportunity to study interaction effects, both statistically and cognitively. To obtain a more complete perspective of students’ understanding, we collect both quantitative and qualitative data on the instrument. We find preliminary evidence that students coordinate their ways of thinking about related attributes of confidence intervals, justifying the use of this screening design for applications to educational assessment. In the third project, we draw connections between Betti curves in topological data analysis and survival curves in survival analysis, to leverage the techniques of survival analysis to permit a new perspective on the topological structure of the short-answer response datasets from our instrument. We also perform a manual clustering exercise on select open-ended items from our instrument. Collectively, these efforts allow us to develop a meaningful interpretation of the topological features that emerge in short-answer response datasets and to relate these findings to students’ understanding of important statistical concepts.