An Eyes and Hands Model: Extending Visual and Motor Modules for Cognitive Architectures

open_access
Open Access
Author:
Tehranchi, Farnaz
Graduate Program:
Computer Science and Engineering
Degree:
Doctor of Philosophy
Document Type:
Dissertation
Date of Defense:
September 24, 2020
Committee Members:
Rebecca Jane Passonneau, Dissertation Advisor/Co-Advisor
Rebecca Jane Passonneau, Committee Chair/Co-Chair
Reginald Adams, Jr., Outside Member
Robert Collins, Committee Member
Jesse Louis Barlow, Committee Member
Frank Edward Ritter, Dissertation Advisor/Co-Advisor
Chitaranjan Das, Program Head/Chair
Frank Edward Ritter, Committee Chair/Co-Chair
Keywords:
User modeling
Visual attention
Motor skills
Cognitive architectures
Cognitive computing
Human-computer interaction
Artificial intelligence
Error detection
Error correction
human errors
Eyes and Hands model
Abstract:
A form of Artificial Intelligence simulates human intelligence and behavior. These simulations are not always complete and not always interactive. Adding a new type of memory and extending the visual and motor modules to existing cognitive architecture offers a motivating approach for simulating human behavior. This dissertation presents an Eyes and Hands model, a new approach to facilitate cognitive models to interact with the world. For this approach, the Java Segmentation and Manipulation (JSegMan) tool is built. JSegMan builds upon Java packages to segment and manipulate the screen. JSegMan also generates operating system commands to implement actions with interfaces. Cognitive architectures provide a unified theory of cognition for developing and simulating cognition and human behavior. The Eyes and Hands model extends two cognitive architecture modules, along with JSegMan, to facilitate interaction. Eyes and hands models can be used to explore the role of interaction in human behavior. In this dissertation, three Eyes and Hands models were developed: (a) the Dismal model that completed a spreadsheet task in the Dismal mode of Emacs, (b) the Biased-coin model based on an existing two-choice experiment, and (c) the Excel model that completed the spreadsheet task in the Excel task environment. I conducted two studies to investigate the model’s visual attention and response time. In the first study, learners’ eye movements data were recorded to predict learning. The results showed that with eye movement data, the learners’ performance could be predicted correctly 76% of the time. Therefore, where users are looking is important and should be considered in the simulation. In the second study, participants’ response time and eye movements were recorded. The Excel model was built upon this study. A simple Eyes and Hands Error model was built to demonstrate how the model’s time is allocated to error detection, error correction, and different types of knowledge. The results suggested that further analysis is required to investigate human errors.