Deep Reinforcement Learning Approaches to Perimeter Metering Control Problems
Open Access
- Author:
- Zhou, Dongqin
- Graduate Program:
- Civil Engineering
- Degree:
- Doctor of Philosophy
- Document Type:
- Dissertation
- Date of Defense:
- February 09, 2024
- Committee Members:
- Farshad Rajabipour, Program Head/Chair
S. Ilgin Guler, Major Field Member
Satadru Dey, Outside Unit & Field Member
Vikash Gayah, Chair, Minor Member & Dissertation Advisor
Kostas Papakonstantinou, Major Field Member - Keywords:
- Macroscopic fundamental diagram
Perimeter metering control
Deep reinforcement learning - Abstract:
- Perimeter metering control has long been an active research topic since well-defined relationships between network productivity and usage, i.e., network macroscopic fundamental diagrams (MFD), were shown capable of describing regional traffic dynamics. This control scheme provides a convenient way to mitigate urban congestion by manipulating vehicular movements across homogeneous regions without modeling the detailed behaviors and interactions involved with individual vehicle presence. Numerous methods have been proposed to solve perimeter metering control problems, but most existing strategies require accurate modeling of traffic dynamics with full knowledge of the network MFD and dynamic equations to describe how vehicles move across regions of the network. However, such information is generally difficult to obtain and subject to significant estimation errors. For these limitations, this dissertation investigates the applicability of (model-free) deep reinforcement learning approaches to perimeter metering control problems. For two-region urban networks, a model free deep reinforcement learning perimeter control scheme is proposed that features agents with either continuous or discrete action spaces. The proposed agents learn to select control actions through a reinforcement learning process without assuming any information about environment dynamics. This scheme has shown comparable performances to the model predictive control method, particularly when uncertainty exists. To further improve its learning and control performances as well as scalability to slightly larger urban networks, domain control knowledge of congestion dynamics is integrated into the agent designs that provides knowledge-guided exploration strategies for the agents such that they can explore around the most rewarding part of the action spaces. Building upon these successes, the multi-region perimeter metering control problem, which holds promise for efficient traffic management in large-scale urban networks, is studied. A novel scalable model-free scheme based on model-free multi-agent deep reinforcement learning is proposed, which features value function decomposition in the paradigm of centralized training with decentralized execution, coupled with critical advances of single-agent deep reinforcement learning and problem reformulation guided by domain expertise. Further, for realistic evaluation, a microsimulation environment is utilized to gauge the effectiveness and transferability of a reinforcement learning-based perimeter controller, noting that the ability of such methods to transfer the learned knowledge and quickly adapt control policies to a new setting is critical, particularly in real-life situations where training a method from scratch is intractable. Last but not least, this dissertation presents a joint framework for inter-regional perimeter metering control and intra-regional traffic signal control, which yields effective and implementable city-level traffic management policies.