Reinforcement learning for well location optimization
Open Access
- Author:
- Dawar, Kshitij
- Graduate Program:
- Energy and Mineral Engineering
- Degree:
- Doctor of Philosophy
- Document Type:
- Dissertation
- Date of Defense:
- February 26, 2021
- Committee Members:
- Sanjay Srinivasan, Dissertation Advisor/Co-Advisor
Sanjay Srinivasan, Committee Chair/Co-Chair
Eugene C Morgan, Committee Member
Jeremy Michael Gernand, Committee Member
Sridhar Anandakrishnan, Outside Member
Mort D Webster, Dissertation Advisor/Co-Advisor
Mort D Webster, Committee Chair/Co-Chair
Mort D Webster, Program Head/Chair - Keywords:
- Reinforcement learning
Well location optimization
Well location
Dynamic programming
Multi-stage optimization
Deep learning
Geostatistics
Geostatistical simulation - Abstract:
- The strategic placement of exploratory wells during the process of hydrocarbon production is critical for both the determination of the reservoir properties and the eventual profitability of the reservoir. Conventional procedures for selecting oil and gas well locations typically collect seismic and other data, conduct extensive simulations and analysis, and then choose the next location for one or more wells. The criteria for selection emphasize maximizing future production. This approach is effectively a single-stage optimization of well location, and it ignores the value of information from initial well locations in reducing the uncertainty about the subsurface properties of the reservoir. In addition to the extraction of hydrocarbon resources, wells provide insights regarding the petrophysical and geological properties of the reservoir and are the primary source of information to update reservoir models leading to the better characterization of the reservoir. In the initial stages of reservoir development, the spatial characterization of geological properties have significant uncertainty because of the scarcity of direct subsurface measurements and the poor granularity of auxiliary information such as seismic data. Each new well location, in addition to its production, provides information that can reduce the uncertainty about reservoir properties at other locations and improve the selection of subsequent well locations. In essence, the placement of initial wells entails a tradeoff between exploration, enabling better subsequent well locations, and exploitation, maximizing profitability of the next well location. In general, single-stage optimization approaches that optimize each decision sequentially often do not lead to the maximum cumulative benefit. A better approach is to select well locations within a multi-stage framework that accounts for the information value from well placement on subsequent reservoir development decisions. In this dissertation, I frame the well location selection problem as a multi-stage sequential decision under uncertainty. My contribution is to apply reinforcement learning techniques and to formulate sequential well location selection as a dynamic programming problem. Within the reinforcement learning framework, I apply geostatistical simulation techniques to characterize the uncertainty in reservoir models and to determine the updates to the beliefs regarding reservoir rock properties that would result from alternative well location decisions and hypothetical observations at those locations. Because of the number of feasible sequences of well locations, the possible observations within each sequence, and the computational demands of simulating updated reservoir properties, traditional dynamic programming solution methods are not tractable. I demonstrate the application of reinforcement learning, or approximate dynamic programming, a class of algorithms that combine Monte Carlo sampling methods with functional approximations of the objective function. These methods provide a computationally tractable approach to exploring sequential well location selection with explicit consideration of the information value of initial well locations. Using this approach, I test the hypothesis that a well location selection strategy is more robust to uncertainty in the initial data than single-stage optimization approaches. To test this hypothesis, I apply a novel reinforcement learning framework to select well locations accounting for information value. I develop several proxy geostatistical models to reduce the computational time and effort and explore the effects of the choice of proxy model on the suggested well placement strategy. I demonstrate the solution of the well location problem using one reinforcement learning algorithm, Q-learning, and discuss its limitations. I first demonstrate the framework on a simpler two-dimensional well location problem, using both tabular approaches and artificial neural networks as functional approximators. In the 2D case, I explore the sensitivity of the policy developed using reinforcement learning to the hyperparameters that control the exploration of the state-action space, and I demonstrate convergence to the optimal policy. I then apply the deep reinforcement learning algorithm to the Stanford V and SPE comparative solutions project model 2 reservoirs, which are larger three-dimensional reservoir cases. The results of the larger reservoir cases demonstrate the benefits of sequential well location selection relative to single-stage optimization approaches.