From Gaze to Insight: Leveraging Eye Tracking for Structural Inspection
Restricted (Penn State Only)
- Author:
- Saleem, Muhammad Rakeh
- Graduate Program:
- Architectural Engineering
- Degree:
- Doctor of Philosophy
- Document Type:
- Dissertation
- Date of Defense:
- October 30, 2024
- Committee Members:
- Alan Wagner, Outside Unit Member
Simon Miller, Outside Field Member
Rebecca Napolitano, Chair & Dissertation Advisor
Thomas Boothby, Major Field Member
James Freihaut, Program Head/Chair - Keywords:
- eye tracking
damage assessment
fixation maps
visual attention
computer vision
building inspection
human-building interaction
artificial intelligence
saliency maps
disaster reconnaissance - Abstract:
- The safety and integrity of the built environment depend on routine inspections for preventive monitoring and reactive inspections following natural disasters. These inspections are crucial for identifying and mitigating potential hazards and ensuring the longevity and safety of structures. Currently, there are three primary modalities for visual inspections: 1) human-driven visual inspections, which involve inspectors walking around structures and taking notes. This method, while requiring minimal technology, is cost-inefficient and poses significant safety risks, particularly when entering the vicinity of damaged structures post-disaster; 2) UAV-based inspections that follow pre-programmed flight paths to capture images of structures. Although safer and less technologically demanding, this method lacks the decision-making capabilities required for dynamic imaging, such as zooming in on areas of interest, and 3) remote piloting of drones by inspectors. This approach, while safer for inspectors, demands higher levels of drone piloting expertise and lacks the critical thinking and sense-making integration between the inspector and the UAV. Given these challenges, there is a critical need for rapid, efficient, and safe structural inspection methods. This dissertation aims to address this need by capturing expert interactions with structures during inspections to develop more detailed and situationally-aware automatic inspection methods. By leveraging eye tracking technology, this research collects and analyzes human gaze data to understand the reasoning abilities and decision-making processes of inspectors during structural assessments. Eye tracking technology can accurately map where inspectors are looking and what they are focusing on, thereby recognizing and inferring human implicit attention. The gathered eye tracking data will be used to develop advanced saliency maps and attention mechanisms, which help to identify and prioritize areas of interest during structural inspections, simulating human visual attention. This research incorporates a comprehensive disaster case study to evaluate human expert knowledge in post-disaster damage assessments using eye tracking. By understanding how experts assess damage in disaster scenarios, the findings will inform the development of autonomous inspection systems that mimic expert behaviors, significantly improving safety, efficiency, and decision-making in both routine and reactive inspections. Moreover, the study evaluates the differences in visual attention and gaze patterns between expert and novice inspectors, providing valuable insights for training novice inspectors and improving human-robot collaboration. The integration of human expertise with automated systems promises to significantly reduce the time and cost associated with disaster reconnaissance missions while ensuring thorough and reliable structural assessments. This dissertation aims to bridge the gap between human expertise and advanced artificial intelligence and robotic technologies, creating a synergy that enhances structural inspection methodologies. The ultimate goal is to develop an inspection framework that combines the strengths of human judgment and advanced technological capabilities, paving the way for safer, more efficient, and cost-effective structural inspections. This research contributes to the broader field of automated structural inspection and disaster response by highlighting the potential and current limitations of applying computer vision techniques to real-world inspection tasks.