DESIGN AS A MARKOV DECISION PROCESS: A METHOD FOR BROAD AND EFFICIENT TRADESPACE EXPLORATION

Open Access
- Author:
- Chhabra, Jaskanwal Preet
- Graduate Program:
- Civil Engineering
- Degree:
- Doctor of Philosophy
- Document Type:
- Dissertation
- Date of Defense:
- May 21, 2018
- Committee Members:
- Gordon P. Warn, Dissertation Advisor/Co-Advisor
Gordon P. Warn, Committee Chair/Co-Chair
Konstantinos Papakonstantinou, Committee Member
Venkataraman Shankar, Committee Member
Timothy Simpson, Outside Member - Keywords:
- Reinforcement learning
Design exploration
Multi-fidelity
Sequential Decision Process
Surrogate modeling
Gaussian process - Abstract:
- Engineers rely heavily upon highly detailed and computationally intensive models to verify the performance of engineered systems on the basis of various criteria like life-cycle cost, reliability, among others. This high computational complexity of detailed models has traditionally precluded a broad comparison of design alternatives based upon these criteria early in the design process. Recently it has been suggested that this conundrum can be overcome by viewing design as a sequential decision process (SDP), in which carefully constructed, low-fidelity models are used to first filter out infeasible/dominated design(s) by performing cheap model executions, and then higher-fidelity evaluations are performed only on the candidate design(s) that appear to be promising after the lower-fidelity evaluations. In line with the idea of set-based design, viewing design as a SDP facilitates design freedom in the early phases of the design process, and then systematically eliminates inefficient design alternatives as the design process continues. Although SDP has been shown to be efficient in localizing to a set of non-dominated designs by sequencing models of increasing fidelity, some sequences of modeling efforts can be significantly efficient than others, and the optimal modeling sequencing is generally not known in the beginning of the design process. Furthermore, in large and continuous design spaces, it might not be possible to evaluate all the design alternatives under consideration, in a reasonable amount of time, even with the lowest-fidelity/cheapest modeling effort. This dissertation is motivated by these two restrictions of the SDP. First the multi-fidelity model selection problem is viewed as a finite Markov Decision Process (MDP) and an online reinforcement learning algorithm, namely Q-learning, is used to obtain and follow an approximately optimal modeling policy. The key motivation for this perspective is to dynamically identify efficient models without exhaustively executing them on a large set of design alternatives. The outcome is a new design methodology referred to as Reinforcement Learning based Design (RL-D), able to learn efficient sequencing of multi-fidelity models by estimating their computational cost and discriminatory power while analyzing randomly sampled design alternatives from the design space throughout the design process. The methodology is formally introduced, and the key theoretical aspects, the underlying assumptions, and similarities and departures from the existing SDP methodology are discussed. Various design examples are presented to demonstrate the application of RL-D methodology, and to study: 1) its efficiency in identifying the approximately optimal modeling policy, 2) its efficiency in converging upon the set of non-dominated design alternatives, and 3) its limitations. Then with an aim of efficiently exploring large and continuous design spaces, it is suggested that data generated by the reinforcement learning agent be used to inform the construction of surrogate models using Gaussian process regression. The key idea here is to use the developed RL-D methodology to efficiently evaluate a set of discrete designs, and then use all the generated training data to fit a surrogate model that is highly accurate in the non-dominated region(s) of the design domain, and captures the trend of the underlying objective function over the complete design domain. The utility of the surrogate modeling framework is illustrated through two numerical applications.