Fed-RHLP: Enhancing Federated Learning with Random High-Local Performance Client Selection for Improved Convergence and Accuracy
Abstract
:1. Introduction
2. Concept and Federated Averaging (FedAvg) Algorithm
2.1. FL Concept
2.2. Federated Averaging (FedAvg) Algorithm
Algorithm 1: FedAvg algorithm | |
1: | ServerExecute: |
2: | W0 = Initial ML model’s parameters |
3: | N = Number of all clients |
4: | C = Fraction of selected clients |
5: | For t = 1,2,… to T do |
6: | m = N.C |
7: | St = {random set of m clients} |
8: | For k ∈ St in Parallel do |
9: | = ClientUpdate(k, wt) |
10: | End for |
11: | |
12: | End for |
13: | ClientUpdate(k, w): |
14: | B = (Split dk into batches of size B) |
15: | For i = 1,2,… to E do |
16: | For b ∈ B do |
17: | w ← w − n∇(w; b) |
18: | End for |
19: | End for |
20: | Return w to the server |
3. Proposed Federated Random High Local Performance Client Selection (Fed-RHLP)
3.1. Random High-Local Performance Client Selection on the Roulette Wheel
3.2. Insight of Fed-RHLP Algorithm
Algorithm 2: Strategy of using a roulette wheel on Fed-RHLP | |
1: | ServerExecute: |
2: | W0 = Initial ML model’s parameters |
3: | N = Number of all clients |
4: | C = Fraction of selected clients |
5: | K = Set of all clients |
6: | For t = 1,2… to T do |
7: | m = N.C |
8: | St = {Roulettewheel_Selection(K, Wt)} |
9: | For k ∈ St in Parallel do |
10: | = ClientUpdate(k,Wt) |
11: | End for |
12: | |
13: | End for |
14: | Roulettewheel_Selection(K, W) : |
15: | For k ∈ K do |
16: | |
17: | TotalLP+= LPk |
18: | End for |
19: | For k ∈ K do |
20: | |
21: | End for |
22: | St = random(K, m, ,replace = False) |
23: | Return St to the server |
24: | ClientUpdate(k, w): |
25: | B = (Split dk into batches of size B) |
26: | For i = 1,2,… to E do |
27: | For b ∈ B do |
28: | w ← w − n∇(w; b) |
29: | End for |
30: | End for |
31: | Return w to the server |
4. Experimental Setup and Results
4.1. Experimental Setup
4.1.1. Data Preparation
- The MNIST dataset contained 70,000 handwriting images of numbers 0–9 which were classified into 60,000 training samples and 10,000 test samples. Each example was a 28 × 28 grayscale image [29].
- The FMNIST was a dataset created by Zalando that consists of images of various fashion items. It includes a training set of 60,000 images and a test set of 10,000 images. Each image is 28 × 28 pixels and in grayscale, with each one labeled into one of 10 different classes [30].
- The QMNIST was a dataset derived from the NIST Special Database 19 which were classified into 60,000 training samples and 10,000 test samples. The QMNIST dataset is designed to closely follow the preprocessing steps of the original MNIST dataset. It is distributed under a BSD-style license [31].
- The CIFAR was a dataset of 60,000 tiny images collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. It consisted of a training set of 50,000 examples and a test set of 10,000 examples. Each example was a 32 × 32 color image in 10 classes [32].
4.1.2. Setting FL Algorithm and CNN Model’s Parameters
4.2. Experimental Results
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Zhang, C.; Xie, Y.; Bai, H.; Yu, B.; Li, W.; Gao, Y. A survey on federated learning. Knowl.-Based Syst. 2021, 216, 106775. [Google Scholar] [CrossRef]
- Bebortta, S.; Tripathy, S.S.; Basheer, S.; Chowdhary, C.L. FedEHR: A Federated Learning Approach towards the Prediction of Heart Diseases in IoT-Based Electronic Health Records. Diagnostics 2023, 13, 3166. [Google Scholar] [CrossRef] [PubMed]
- Moshawrab, M.; Adda, M.; Bouzouane, A.; Ibrahim, H.; Raad, A. Reviewing Federated Learning Aggregation Algorithms; Strategies, Contributions, Limitations and Future Perspectives. Electronics 2023, 12, 2287. [Google Scholar] [CrossRef]
- Rafi, T.H.; Noor, F.A.; Hussain, T.; Chae, D.K.; Yang, Z. A Generalized Look at Federated Learning: Survey and Perspectives. arXiv 2023, arXiv:2303.14787. [Google Scholar]
- Wang, R.; Lai, J.; Zhang, Z.; Li, X.; Vijayakumar, P.; Karuppiah, M. Privacy-preserving federated learning for internet of medical things under edge computing. IEEE J. Biomed. Health Inform. 2022, 27, 854–865. [Google Scholar] [CrossRef]
- Jung, J.P.; Ko, Y.B.; Lim, S.H. Federated Learning with Pareto Optimality for Resource Efficiency and Fast Model Convergence in Mobile Environments. Sensors 2024, 24, 2476. [Google Scholar] [CrossRef]
- Nilsson, A.; Smith, S.; Ulm, G.; Gustavsson, E.; Jirstrand, M. A performance evaluation of federated learning algorithms. In Proceedings of the Second Workshop on Distributed Infrastructures for Deep Learning, Rennes, France, 10–11 December 2018; pp. 1–8. [Google Scholar]
- Zhu, H.; Jin, Y. Multi-objective evolutionary federated learning. IEEE Trans. Neural Netw. Learn. Syst. 2019, 31, 1310–1322. [Google Scholar] [CrossRef]
- Putra, M.A.; Putri, A.R.; Zainudin, A.; Kim, D.S.; Lee, J.M. Acs: Accuracy-based client selection mechanism for federated industrial iot. Internet Things 2023, 21, 100657. [Google Scholar] [CrossRef]
- Khajehali, N.; Yan, J.; Chow, Y.W.; Fahmideh, M. A Comprehensive Overview of IoT-Based Federated Learning: Focusing on Client Selection Methods. Sensors 2023, 23, 7235. [Google Scholar] [CrossRef]
- Zhang, S.Q.; Lin, J.; Zhang, Q. A multi-agentreinforcement learning approach for efficient client selection in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Virtually, 22 February–1 March 2022; Volume 36, pp. 9091–9099. [Google Scholar]
- Zhou, H.; Lan, T.; Venkataramani, G.; Ding, W. On the Convergence of Heterogeneous Federated Learning with Arbitrary Adaptive Online Model Pruning. arXiv 2022, arXiv:2201.11803. [Google Scholar]
- Mu, X.; Shen, Y.; Cheng, K.; Geng, X.; Fu, J.; Zhang, T.; Zhang, Z. Fedproc: Prototypical contrastive federated learning on non-iid data. Future Gener. Comput. Syst. 2023, 143, 93–104. [Google Scholar] [CrossRef]
- Cho, Y.J.; Wang, J.; Joshi, G. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv 2020, arXiv:2010.01243. [Google Scholar]
- Zeng, Y.; Teng, S.; Xiang, T.; Zhang, J.; Mu, Y.; Ren, Y.; Wan, J. A Client Selection Method Based on Loss Function Optimization for Federated Learning. Comput. Model. Eng. Sci. 2023, 137, 1047–1064. [Google Scholar] [CrossRef]
- Khalil, A.; Wainakh, A.; Zimmer, E.; Parra-Arnau, J.; Anta, A.F.; Meuser, T.; Steinmetz, R. Label-Aware Aggregation for Improved Federated Learning. In Proceedings of the 2023 Eighth International Conference on Fog and Mobile Edge Computing (FMEC), Tartu, Estonia, 18–20 September 2023; IEEE: New York, NY, USA, 2023; pp. 216–223. [Google Scholar]
- Fu, L.; Zhang, H.; Gao, G.; Zhang, M.; Liu, X. Client selection in federated learning: Principles, challenges, and opportunities. IEEE Internet Things J. 2023, 10, 21811–21819. [Google Scholar] [CrossRef]
- Rai, S.; Kumari, A.; Prasad, D.K. Client Selection in Federated Learning under Imperfections in Environment. AI 2022, 3, 124–145. [Google Scholar] [CrossRef]
- Yaqoob, M.M.; Alsulami, M.; Khan, M.A.; Alsadie, D.; Saudagar, A.K.; AlKhathami, M.; Khattak, U.F. Symmetry in privacy-based healthcare: A review of skin cancer detection and classification using federated learning. Symmetry 2023, 15, 1369. [Google Scholar] [CrossRef]
- Ye, M.; Fang, X.; Du, B.; Yuen, P.C.; Tao, D. Heterogeneous federated learning: State-of-the-art and research challenges. ACM Comput. Surv. 2023, 56, 1–44. [Google Scholar] [CrossRef]
- Huang, T.; Lin, W.; Wu, W.; He, L.; Li, K.; Zomaya, A.Y. An Efficiency-Boosting Client Selection Scheme for Federated Learning With Fairness Guarantee. IEEE Trans. Parallel Distrib. Syst. 2021, 32, 1552–1564. [Google Scholar] [CrossRef]
- Ma, X.; Liao, L.; Li, Z.; Lai, R.X.; Zhang, M. Applying federated learning in software-defined networks: A survey. Symmetry 2022, 14, 195. [Google Scholar] [CrossRef]
- McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; Arcas, B.A. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA, 20–22 April 2017; PMLR: Birmingham, UK, 2017; pp. 1273–1282. [Google Scholar]
- Rahman, N.A.B.A.; Hamid, Z.B.A.; Musirin, I.B.; Salim, N.A.B. Hybrid Loss Sensitivity Factor And Mutated Ant Lion Optimizer For Optimal Distributed Generation Placement With Multiple Loadings. J. Theor. Appl. Inf. Technol. 2023, 101, 6703–6715. [Google Scholar]
- Creevey, F.M.; Hill, C.D.; Hollenberg, L.C. GASP: A genetic algorithm for state preparation on quantum computers. Sci. Rep. 2023, 13, 11956. [Google Scholar] [CrossRef] [PubMed]
- Almotairi, K.H. Gene selection for high-dimensional imbalanced biomedical data based on marine predators algorithm and evolutionary population dynamics. Arab. J. Sci. Eng. 2024, 49, 3935–3961. [Google Scholar] [CrossRef]
- Hosseinzadeh, M.; Hudson, N.; Heshmati, S.; Khamfroush, H. Communication-Loss Trade-Off in Federated Learning: A Distributed Client Selection Algorithm. In Proceedings of the 2022 IEEE 19th Annual Consumer Communications & Networking Conference (CCNC), Las Vegas, NV, USA, 8–11 January 2022; IEEE: New York, NY, USA, 2022; pp. 1–6. [Google Scholar]
- Mohammed, I.; Tabatabai, S.; Al-Fuqaha, A.; El Bouanani, F.; Qadir, J.; Qolomany, B.; Guizani, M. Budgeted online selection of candidate IoT clients to participate in federated learning. IEEE Internet Things J. 2020, 8, 5938–5952. [Google Scholar] [CrossRef]
- LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 1998, 86, 2278–2324. [Google Scholar] [CrossRef]
- Xiao, H.; Rasul, K.; Vollgraf, R. Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv 2017, arXiv:1708.07747. [Google Scholar]
- Yadav, C.; Bottou, L. Cold case: The lost mnist digits. Adv. Neural Inf. Process. Syst. 2019, 32. [Google Scholar] [CrossRef]
- Krizhevsky, A. Learning Multiple Layers of Features from Tiny Images. Master’s Thesis, Department of Computer Science, University of Toronto, Toronto, ON, Canada, 2009. [Google Scholar]
- Xiao, J.; Du, C.; Duan, Z.; Guo, W. A novel server-side aggregation strategy for federated learning in non-iid situations. In Proceedings of the 2021 20th International Symposium on Parallel and Distributed Computing (ISPDC), Cluj-Napoca, Romania, 28–30 July 2021; IEEE: New York, NY, USA, 2021; pp. 17–24. [Google Scholar]
- Duan, J.H.; Li, W.; Lu, S. FedDNA: Federated learning with decoupled normalization-layer aggregation for non-iid data. In Machine Learning and Knowledge Discovery in Databases, Proceedings of the Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, 13–17 September 2021; Proceedings, Part I 21 2021; Springer International Publishing: Berlin/Heidelberg, Germany, 2021; pp. 722–737. [Google Scholar]
- Li, Q.; Diao, Y.; Chen, Q.; He, B. Federated learning on non-iid data silos: An experimental study. In Proceedings of the 2022 IEEE 38th International Conference on Data Engineering (ICDE), Kuala Lumpur, Malaysia, 9–12 May 2022; IEEE: New York, NY, USA, 2022; pp. 965–978. [Google Scholar]
- Zhong, J.; Wu, Y.; Ma, W.; Deng, S.; Zhou, H. Optimizing multi-objective federated learning on non-iid data with improved nsga-iii and hierarchical clustering. Symmetry 2022, 14, 1070. [Google Scholar] [CrossRef]
- Li, T.; Sahu, A.K.; Zaheer, M.; Sanjabi, M.; Talwalkar, A.; Smith, V. Federated optimization in heterogeneous networks. Proc. Mach. Learn. Syst. 2020, 2, 429–450. [Google Scholar]
- Frid-Adar, M.; Diamant, I.; Klang, E.; Amitai, M.; Goldberger, J.; Greenspan, H. GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing 2018, 321, 321–331. [Google Scholar] [CrossRef]
- Ren, H.; Deng, J.; Xie, X. GRNN: Generative regression neural network—A data leakage attack for federated learning. ACM Trans. Intell. Syst. Technol. (TIST) 2022, 13, 65. [Google Scholar] [CrossRef]
- Wu, Y.; Liu, L.; Bae, J.; Chow, K.H.; Iyengar, A.; Pu, C.; Wei, W.; Yu, L.; Zhang, Q. Demystifying learning rate policies for high accuracy training of deep neural networks. In Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, 9–12 December 2019; IEEE: New York, NY, USA, 2019; pp. 1971–1980. [Google Scholar]
Dataset | Layer | Input Channel | Output Channel | Kernel | Stride | Padding |
---|---|---|---|---|---|---|
MNIST | Convolutional layer 1 | 1 | 10 | 5 × 5 | - | - |
Convolutional layer 2 | 10 | 20 | 5 × 5 | - | - | |
Convolutional layer 2 dropout = 0.5 | - | - | - | - | - | |
Fully connected layer 1 | 320 | 50 | - | - | - | |
Fully connected layer 2 | 50 | 10 Classes | - | - | - | |
FMNIST | Convolutional layer 1 | 1 | 32 | 3 × 3 | - | 1 |
Max pooling layer | - | - | 2 × 2 | 2 | - | |
Convolutional layer 2 | 32 | 64 | 3 × 3 | - | - | |
Max pooling layer | - | - | 2 × 2 | 2 | - | |
Fully connected layer 1 | 2304 | 600 | - | - | - | |
Dropout layer = 0.25 | - | - | - | - | - | |
Fully connected layer 2 | 600 | 120 | - | - | - | |
Fully connected layer 3 | 120 | 10 Classes | - | - | - | |
QMNIST | Convolutional layer 1 | 1 | 16 | 3 × 3 | - | 1 |
Convolutional layer 2 | 16 | 32 | 3 × 3 | - | 1 | |
Convolutional layer 3 | 32 | 64 | 3 × 3 | - | 1 | |
Max pooling layer | - | - | 2 × 2 | 2 | - | |
Fully connected layer 1 | 576 | 120 | - | - | - | |
Fully connected layer 2 | 120 | 84 | - | - | - | |
Fully connected layer 3 | 84 | 47 Classes | - | - | - | |
CIFAR | Convolutional layer 1 | 3 | 16 | 3 × 3 | 1 | 1 |
Convolutional layer 2 | 16 | 32 | 3 × 3 | 1 | 1 | |
Convolutional layer 3 | 32 | 64 | 3 × 3 | 1 | 1 | |
Max pooling layer | - | - | 2 × 2 | 2 | - | |
Fully connected layer 1 | 1024 | 512 | - | - | - | |
Fully connected layer 2 | 512 | 64 | - | - | - | |
Fully connected layer 3 | 64 | 10 Classes | - | - | - | |
Dropout layer = 0.5 | - | - | - | - | - |
Component Name | Description |
---|---|
Processor | 12th Gen Intel(R) Core(TM) i7-12650H 2.30 GHz |
Graphic cards | Nvidia Geforce RTX 4060 Laptop GPU |
Random access memory | DDR5, 40.0 GB |
Operating system | Windows 11 |
Environments | Python 3.7.2 with the Pytorch framework |
Algorithm | Accuracy of Global Model | Dataset | Convergence Rounds and Time (Seconds) | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Accuracy: 50% | Accuracy: 60% | Accuracy: 70% | Accuracy: 80% | Accuracy: 90% | ||||||||
Round | Time | Round | Time | Round | Time | Round | Time | Round | Time | |||
Proposed Fed-RHLP | 99.08% | MNIST IID | 1 | 139.8 | 1 | 139.8 | 1 | 139.8 | 1 | 139.8 | 2 | 282 |
98.71% | MNIST Non-IID | 4 | 526.8 | 5 | 648.6 | 5 | 648.6 | 9 | 1140.6 | 11 | 1385.4 | |
Fed-Avg | 98.55% | MNIST IID | 2 | 130.8 | 3 | 196.8 | 3 | 196.8 | 4 | 261 | 9 | 597.6 |
95.36% | MNIST Non-IID | 8 | 412.2 | 10 | 515.4 | 12 | 621 | 20 | 1018.8 | 46 | 2365.2 | |
Power-of-Choice | 98.53% | MNIST IID | 2 | 183 | 3 | 259.2 | 3 | 259.2 | 4 | 331.2 | 8 | 620.4 |
96.36% | MNIST Non-IID | 8 | 560.4 | 12 | 822.6 | 13 | 887.4 | 17 | 1152.6 | 47 | 3294.6 |
Algorithm | Accuracy of Global Model | Dataset | Convergence Rounds and Time (Seconds) | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Accuracy: 50% | Accuracy: 60% | Accuracy: 70% | Accuracy: 80% | Accuracy: 90% | ||||||||
Round | Time | Round | Time | Round | Time | Round | Time | Round | Time | |||
Proposed Fed-RHLP | 91.56% | FMNIST IID | 1 | 243 | 1 | 243 | 1 | 243 | 1 | 243 | 12 | 2916 |
88.96% | FMNIST Non-IID | 3 | 1123.2 | 5 | 1872 | 9 | 3369.6 | 18 | 6739.2 | - | - | |
Fed-Avg | 91.04% | FMNIST IID | 1 | 166.2 | 1 | 166.2 | 1 | 166.2 | 2 | 332.4 | 54 | 8974.8 |
87.55% | FMNIST Non-IID | 4 | 1212.8 | 6 | 1819.2 | 8 | 2425.6 | 25 | 7580 | - | - | |
Power-of-Choice | 91.2% | FMNIST IID | 1 | 243 | 1 | 243 | 1 | 243 | 3 | 729 | 41 | 9963 |
88.02% | FMNIST Non-IID | 6 | 2049 | 6 | 2049 | 12 | 4098 | 20 | 6830 | - | - |
Algorithm | Accuracy of Global Model | Dataset | Convergence Rounds and Time (Seconds) | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Accuracy: 50% | Accuracy: 60% | Accuracy: 70% | Accuracy: 80% | Accuracy: 90% | ||||||||
Round | Time | Round | Time | Round | Time | Round | Time | Round | Time | |||
Proposed Fed-RHLP | 99.32% | QMNIST IID | 1 | 198 | 1 | 198 | 1 | 198 | 1 | 198 | 1 | 198 |
98.90% | QMNIST Non-IID | 3 | 675 | 3 | 675 | 4 | 900 | 6 | 1350 | 10 | 2250 | |
Fed-Avg | 99.07% | QMNIST IID | 1 | 133.8 | 1 | 133.8 | 2 | 267.6 | 2 | 267.6 | 3 | 401.4 |
98.63% | QMNIST Non-IID | 5 | 840 | 6 | 1008 | 6 | 1008 | 14 | 2352 | 15 | 2520 | |
Power-of-Choice | 99.04% | QMNIST IID | 1 | 198 | 1 | 198 | 2 | 396 | 2 | 396 | 3 | 594 |
98.69% | QMNIST Non-IID | 7 | 1570.8 | 7 | 1570.8 | 7 | 1570.8 | 9 | 2019.6 | 13 | 2917.2 |
Algorithm | Accuracy of Global Model | Dataset | Convergence Rounds and Time (Seconds) | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Accuracy: 30% | Accuracy: 40% | Accuracy: 50% | Accuracy: 60% | Accuracy: 70% | ||||||||
Round | Time | Round | Time | Round | Time | Round | Time | Round | Time | |||
Proposed Fed-RHLP | 74.21% | CIFAR IID | 7 | 634.2 | 11 | 996.6 | 20 | 1812 | 36 | 3261.6 | 83 | 7519.8 |
66.11% | CIFAR Non-IID | 29 | 6925.2 | 36 | 8596.8 | 99 | 23,641.2 | 121 | 28,894.8 | - | - | |
Fed-Avg | 62.0% | CIFAR IID | 42 | 3288 | 61 | 4783.8 | 101 | 7967.4 | 178 | 14,135.4 | - | - |
56.74% | CIFAR Non-IID | 77 | 12,427.8 | 115 | 18,561 | 159 | 25,662.6 | - | - | - | - | |
Power-of-Choice | 61.97% | CIFAR IID | 39 | 4609.8 | 59 | 6973.8 | 103 | 12,174.6 | 181 | 21,394.2 | - | - |
59.74% | CIFAR Non-IID | 72 | 17,107.2 | 102 | 24,235.2 | 146 | 34,689.6 | - | - | - | - |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Sittijuk, P.; Tamee, K. Fed-RHLP: Enhancing Federated Learning with Random High-Local Performance Client Selection for Improved Convergence and Accuracy. Symmetry 2024, 16, 1181. https://doi.org/10.3390/sym16091181
Sittijuk P, Tamee K. Fed-RHLP: Enhancing Federated Learning with Random High-Local Performance Client Selection for Improved Convergence and Accuracy. Symmetry. 2024; 16(9):1181. https://doi.org/10.3390/sym16091181
Chicago/Turabian StyleSittijuk, Pramote, and Kreangsak Tamee. 2024. "Fed-RHLP: Enhancing Federated Learning with Random High-Local Performance Client Selection for Improved Convergence and Accuracy" Symmetry 16, no. 9: 1181. https://doi.org/10.3390/sym16091181