Towards computationally efficient reinforcement learning frameworks for the design of cyber-physical systems
Open Access
- Author:
- Agrawal, Akash
- Graduate Program:
- Engineering Design
- Degree:
- Master of Science
- Document Type:
- Master Thesis
- Date of Defense:
- June 29, 2021
- Committee Members:
- Scarlett Miller, Program Head/Chair
Chris Mc Comb, Thesis Advisor/Co-Advisor
Matthew B Parkinson, Committee Member - Keywords:
- computational efficiency
engineering design
reinforcement learning
design automation
cyber-physical systems - Abstract:
- The design of cyber-physical systems, such as autonomous underwater vehicles, is challenging due to their large-scale, hybrid, and heterogeneous nature. Moreover, the design space of cyber-physical systems is enormous and involves computationally-expensive objective functions and constraints, leading to long development timelines. Further, the disciplinary domains in such systems are intrinsically coupled, and design decisions made in one domain affect other domains. As cyber-physical systems are typically deployed in environments which evolve continually and can involve unforeseen changes, the systems also need to have a dynamic and adaptive nature. Further, cyber physical systems are often physically separated and networked. It is challenging to design them for achieving collaborative behavior, especially in presence of adversarial environments. These unique design challenges can be overcome by (1) design methodologies based on simulation techniques for cyber-physical systems and their environment, (2) design representations and simulation models of varying fidelities for domains that involve computationally expensive objective functions and constraints, and (3) frameworks that automatedly and efficiently explore the design space based on these models to search for high-performance designs. However, decision making involved in the design of cyber-physical systems is sequential in nature. The outcome of past decisions will influence future decisions, and evaluating this influence will have some degree of uncertainty. For this reason, it is not feasible to make all design decisions at once and reach a high-performance design. Deep Reinforcement Learning (RL) algorithms can autonomously learn effective strategies for sequential decision-making tasks by simulating the task and updating the policy towards strategies that maximize the long-term rewards of these decisions. Thereby it is possible to leverage deep RL for exploring high-dimensional design spaces of cyber-physical systems. The ability of deep RL to handle stochastic decision-making can also enable cyber-physical systems to respond in an adaptive manner. However, the computational expense of deep RL frameworks could be unreasonable in terms of sustainability and deployment times of large-scale cyber-physical systems. This work proposes two distinct deep RL frameworks for the design of cyber-physical systems. The first framework address the computational challenges associated with (1) distributed nature of cyber-physical systems, and (2) adaptivity to environment. The second framework address the computational challenges associated with (1) the enormity of the design space, and (2) the computational expense of the high-fidelity simulations often necessary to evaluate objective functions in design space exploration In Chapter 2, I propose a multi-agent RL framework that consists of multiple interacting intelligent agents in a distributed system who share their experience to reduce computational cost. This chapter focusses on learning a control policy in an autonomous mobile robot-driven shop floor, a state-of-the-art cyber-physical system in intelligent manufacturing. A low-fidelity simulator-based case study is conducted to showcase that the proposed framework is computationally more efficient than a single-agent framework. Moreover, such a decentralized framework is shown to be tolerant to failures and unforeseen changes as the agents are adaptive to the environment and can compensate for each other’s failures. While this multi-agent RL framework addresses the challenges of the distributed nature of cyber-physical systems and adaptivity to environment using a low-fidelity simulator, many cyber-physical systems demand the use of high-fidelity representations and simulations (based on state-of-the-art physics-based modelling) to satisfy strict margins of error for objectives like human safety. Moreover, they can involve combinatorically large design spaces. In Chapter 3, I propose a multi-fidelity RL framework for design space exploration that involves a single agent leveraging models of varying fidelities across domains to converge to high-performance designs. This framework offers the flexibility to incorporate predefined constant or variable schedules for exploration using models of different fidelities across domains. I showcase the same in a case study that involves the design of the physical components of a vehicle. The RL agent converges to high-performance regions of the design space using objective evaluations at two fidelity levels. A parametric study with different training schedules for exploration at these fidelity levels demonstrates the trade-off between computational expense and solution quality. Lastly, I identify that the proposed framework yields a diverse set of solutions.