Collaborative Learning From Heterogeneous Sources with Provable Guarantees

Open Access
- Author:
- Deng, Yuyang
- Graduate Program:
- Computer Science and Engineering
- Degree:
- Doctor of Philosophy
- Document Type:
- Dissertation
- Date of Defense:
- November 01, 2024
- Committee Members:
- Chitaranjan Das, Program Head/Chair
Rui Zhang, Major Field Member
Necdet Aybat, Outside Unit & Field Member
Mehrdad Mahdavi, Chair & Dissertation Advisor
Jing Yang, Major Field Member - Keywords:
- Federated learning
Collaborative learning
optimization - Abstract:
- In today's data-rich landscape, valuable information is generated from diverse sources, including personal cellphones, wearable devices, and autonomous vehicles. The challenge of effectively harnessing this wealth of data to build powerful machine learning models has given rise to the learning scenario known as collaborative learning from multiple data sources. However, deploying collaborative learning among millions of heterogeneous data sources presents multiple challenges, including statistical heterogeneity, privacy concerns, and communicational and computational burdens. In this thesis, we present our work addressing these challenges. First, to guarantee the worst-case performance of the learned model at each data source, we propose algorithms that distributionally robustify the celebrated FedAvg algorithm while maintaining communication efficiency. Second, we advocate personalizing the single global model for each user, significantly outperforming the traditional "single model for all" collaborative learning paradigm. Notably, when users' data sources exhibit high heterogeneity, it may be impossible to learn a good global model. Therefore, we propose abandoning the "learn global model then personalize" approach. For the first time, we solve personalized objectives for each user in parallel, introducing Personalized Empirical Risk Minimization (PERM), which significantly outperforms all existing state-of-the-art collaborative/federated learning algorithms. Finally, we propose algorithms that can design suitable objectives for each user and efficiently solve them. Our work makes significant progress toward effective collaborative learning among heterogeneous sources. We believe the efficiency and efficacy of our algorithms will have practical applications in the era of large models, such as the collaborative learning of large language models (LLMs) on edge devices.