Adaptation, Application and Analysis of a Content Analysis Tool for Essay Feedback in Middle School Science Classrooms
Open Access
- Author:
- Sheikhi Karizaki, Mahsa
- Graduate Program:
- Computer Science and Engineering
- Degree:
- Master of Science
- Document Type:
- Master Thesis
- Date of Defense:
- July 03, 2024
- Committee Members:
- Rebecca Jane Passonneau, Thesis Advisor/Co-Advisor
Wenpeng Yin, Committee Member
Chitaranjan Das, Program Head/Chair
Chan Min Kim, Committee Member - Keywords:
- Automated Essay Analysis
Formative Feedback
Student Writing Clarity - Abstract:
- This thesis investigates methods to apply a pre-existing software tool, PyrEval, in middle school classrooms for formative assessment of science essays, analyzes characteristics of writing that affect its performance, and compares PyrEval with two other approaches. These are a recently developed end-to-end neural network for assessment of lab reports, and large language models (LLMs). PyrEval has been under continuous development over the past decade. In recent years, it has been applied to formative assessment of science explanation essays written by middle school students in Wisconsin public schools. PyrEval performs well in identifying whether an essay expresses important target ideas from a curriculum where students learn about energy, mass and speed through simulated roller coaster experiments. As a result, PyrEval supports feedback to students and teachers, and this feedback has been shown to lead to improved understanding of science concepts in students’ revised essays. This thesis reports our method to align PyrEval with essay rubrics, comparison of multiple semantic vector methods for use in PyrEval, in-depth analyses of several aspects of student writing quality and the impact on PyrEval performance, and finally, comparison of PyrEval with two other automated assessment methods. Previous work by team members compared two vector dictionary methods, meaning methods that compute a fixed vector for each word string, and found that a matrix factorization method for words and phrases yielded the best performance. My work investigates a variety of methods to assess whether performance can be improved. These methods include modification of the training corpus to be more specific to the student writing, modification of the vector space for the fixed word and phrase vectors, and use of contextualized vectors.