Lexicographic Hyperparameter Optimization and its Applications
Restricted (Penn State Only)
- Author:
- Zhang, Shaokun
- Graduate Program:
- Informatics
- Degree:
- Master of Science
- Document Type:
- Master Thesis
- Date of Defense:
- March 14, 2024
- Committee Members:
- Qingyun Wu, Thesis Advisor/Co-Advisor
Jinghui Chen, Committee Member
Jeffrey Bardzell, Program Head/Chair
Lu Lin, Committee Member - Keywords:
- Hyperparameter optimization
temporal distribution shifts
AutoML - Abstract:
- Hyperparameter optimization (HPO) is a crucial component of automated machine learning (AutoML). It involves finding an optimal set of hyperparameters that maximizes the model’s performance. In practical machine learning systems, there are typically multiple metrics to evaluate the model’s performance, which makes the HPO problem more complex. In this work, we focus on a targeted HPO scenario where practitioners have a priority order over the objectives, enabling a total ordering of all the configurations. We formalize a general notion of priority order as a lexicographic preference over multiple objectives in HPO tasks. We propose an algorithm named LexiFlow as a general solution for lexicographic hyperparameter optimizations and perform extensive empirical evaluations to verify the proposed tuning algorithm. Building upon the hyperparameter optimization algorithm, we address the challenge of temporal distribution shifts in machine learning. Such shifts can significantly impact the performance of deployed models due to data distribution discrepancies between test and training phases. To mitigate this, we present HyperTime - a hyperparameter optimization method that leverages LexiFlow as its core. HyperTime could identify hyperparameters robust to potential temporal distribution shifts in unseen test data. Our method’s effectiveness is confirmed through experiments involving gradient-boosting trees and neural networks across diverse datasets exhibiting temporal distribution shifts.