Beyond Words: Evaluating Large Language Models in Transport Planning

Open Access
- Author:
- Ying, Shaowei
- Graduate Program:
- Spatial Data Science
- Degree:
- Master of Science
- Document Type:
- Master Thesis
- Date of Defense:
- June 13, 2024
- Committee Members:
- Li Zhenlong, Thesis Advisor/Co-Advisor
Yu Manzhu, Committee Member
Anthony Robinson, Program Head/Chair - Keywords:
- Generative Artificial Intelligence in Transport Planning
Evaluation of Large Language Models
GPT-4 and Phi-3-mini - Abstract:
- The resurgence and rapid advancement of Generative Artificial Intelligence (GenAI) in 2023 has catalyzed transformative shifts across numerous industry sectors, including urban transportation and logistics. This thesis investigates the evaluation of Large Language Models (LLMs), specifically GPT-4 and Phi-3-mini, within the domain of Geospatial Artificial Intelligence (GeoAI) to enhance transport planning. The study aims to assess the performance and spatial comprehension of these models through a transport-informed evaluation framework that includes general geospatial skills, general transport domain skills, and real-world transport problem-solving. Utilizing a mixed-methods approach, the research encompasses an evaluation of the LLMs’ general GIS skills, general transport domain knowledge as well as abilities to support human decision-making in the real-world transport planning scenarios of congestion pricing. Results indicate that GPT-4 demonstrates superior accuracy and reliability across various GIS and transport-specific tasks compared to Phi-3-mini, highlighting its potential as a robust tool for transport planners. Nonetheless, Phi-3-mini exhibits competence in specific analytical scenarios, suggesting its utility in resource-constrained environments. The findings underscore the transformative potential of GenAI technologies in urban transportation planning, paving the way for more efficient, sustainable, and informed decision-making processes. Future work will explore the application of newer LLMs and the impact of Retrieval-Augmented Generation techniques, on a broader set of real-world transport planning and operations challenges, so as to deepen the integration of advanced AI models in transport management practices.