Deep Reinforcement Learning Based Decision-Making Frameworks for Infrastructure Management Under Uncertainty
Restricted (Penn State Only)
- Author:
- Saifullah, Mohammad
- Graduate Program:
- Civil Engineering
- Degree:
- Doctor of Philosophy
- Document Type:
- Dissertation
- Date of Defense:
- June 05, 2024
- Committee Members:
- Farshad Rajabipour, Program Head/Chair
Shelley Stoffels, Major Field Member
Hui Yang, Outside Field Member
Hadi Hosseini, Outside Unit Member
Gordon Warn, Major Field Member
Kostas Papakonstantinou, Chair & Dissertation Advisor - Keywords:
- Deep Reinforcement Learning (DRL)
Multi-Agent Systems
POMDPs
Decision-Making Frameworks
Infrastructure Management
Value of Information
Structural Health Monitoring
Life-Cycle Costs Optimization
Stochastic Modeling
Traffic
Transportation networks - Abstract:
- The reliability and efficiency of infrastructure systems, particularly transportation networks, are critical for societal well-being and economic stability. As these systems age and demands increase, the complexity of managing their maintenance and optimization grows. This dissertation addresses the pressing need for advanced decision-support frameworks capable of handling the dynamic and uncertain nature of large-scale infrastructure systems. Traditional methods often fall short, leading to suboptimal decisions and increased risks of failure. Recent incidents, such as the bridge collapses in Baltimore and Pittsburgh, underscore the urgent need for innovative solutions in infrastructure management. The primary objective of this research is to develop and implement novel decision-support frameworks that integrate advanced machine learning techniques with optimization strategies to improve the management of large-scale infrastructure systems. This dissertation focuses on creating models that can handle high-dimensional state and action spaces, optimize long-term life-cycle costs, and incorporate real-time data for dynamic decision-making. The specific goals include: developing multi-agent deep reinforcement learning (DRL) models, integrating traffic adaptation with infrastructure maintenance, quantifying the value of information, and validating the proposed frameworks through real-world applications. The dissertation first explores the challenges posed by the curse of dimensionality and the curse of history in infrastructure management. The curse of dimensionality refers to the exponential increase in computational resources required to process and analyze data as the number of dimensions increases. The curse of history involves the difficulties associated with making decisions based on historical data, where past decisions and actions influence future states, creating complex dependencies. Traditional approaches, such as genetic algorithms, threshold-based methods, and decision-trees, often suffer from suboptimality and scalability issues in these contexts. To address these challenges, the research leverages advances in partially observable Markov decision processes (POMDPs) and deep reinforcement learning (DRL). POMDPs allow for decision-making under partial observability and incorporate noisy real-time data, but they still face challenges related to the curse of dimensionality. DRL frameworks, which integrate deep neural networks with reinforcement learning techniques, have shown remarkable success in various domains by providing scalable solutions to high-dimensional problems and enabling agents to learn optimal policies through interaction with the environment. The dissertation introduces a novel Deep Decentralized Multi-agent Actor-Critic with Centralized Training and Decentralized Execution (DDMAC-CTDE) framework. This approach allows for scalable and efficient policy learning in large transportation networks, demonstrated through a detailed case study involving a network in Virginia, USA. The DDMAC-CTDE framework addresses the limitations of traditional single-agents or centralized DRL methods by enabling coordinated decision-making among multiple agents, each responsible for different components of the infrastructure system. The research also explores the integration of traffic adaptation actions into the maintenance planning process. By adjusting maintenance schedules based on real-time traffic conditions, the proposed method notably reduces total life-cycle costs compared to traditional approaches. This holistic approach considers both the physical and operational aspects of the system, enhancing overall efficiency. Furthermore, the dissertation provides a comprehensive methodology to quantify the Value of Information (VoI) and utilizes it in estimating gradients for DRL training and interpretation of the resultant policies. By using POMDP formulations, the research evaluates the benefits of inspection, i.e., how much value they are providing at each time step, offering insights into the optimal use of resources for infrastructure maintenance. The VoI metric helps in making informed decisions about where to allocate inspection and monitoring efforts to achieve the greatest improvements in system reliability and performance. Extensive numerical experiments and case studies are conducted to validate the proposed methods. These real-world applications demonstrate substantial improvements in performance and cost savings compared to existing policies, highlighting the potential for transformative impacts in infrastructure management. The dissertation provides practical examples of how the methodologies can be implemented and the benefits they can deliver, offering valuable insights for policymakers and practitioners. In summary, this dissertation presents innovative solutions for the management of large-scale infrastructure systems. By integrating advanced DRL techniques with optimization strategies, the research addresses critical challenges in infrastructure management, providing scalable, efficient, and effective decision-support frameworks. The findings have significant implications for enhancing the resilience and efficiency of infrastructure systems, ultimately contributing to societal well-being and economic stability.