Skip to Content
MDPIMDPI

Advanced

by
  • Dasaradharami Reddy Kandati and
  • Thippa Reddy Gadekallu *

Reviewer 1: Mennan Selimi Reviewer 2: Mohd Adli B. Md Ali

Round 1

Reviewer 1 Report

This paper proposes a hybrid algorithm that groups edge devices based on the training hyperparameters and modifies the parameters cluster wise genetically. This approach is based on Federated Learning. 

 

Strengths:

The structure of the paper is clear and it has all the elements needed (introduction, literature review,methodology, initial measurements, proposed work that move forward the state of the art, evaluation, and conclusions). The authors carefully wrote some background sections that helps to understand better they work. Overall the paper is trying to solve a practical and relevant problem. The goal of the paper is quite clear. It is well written.

 

Weaknesses:

  • Since the authors are considering the edge devices (low capacity devices) in the scenario, the training process must not only target for highest accuracy, but also on reducing the training time and the resource consumption (CPU and memory consumption). For instance, identifying what is the best number of epochs per training round in federated machine learning model training taking into account the CPU consumption of the devices is very critical. 
  • The experimental results should be further discussed including more insightful comments. More graphs showing the benefits of Genetic CFL algorithms are needed.
  • Some of the paper in literature review are not from the problem domain that the authors are trying to solve: 
  • Some very relevant papers that needs to be cited: 
    • FedTune: Automatic Tuning of Federated Learning Hyper-Parameters from System Perspective - https://arxiv.org/abs/2110.03061
    • BePOCH: Improving Federated Learning Performance in Resource-Constrained Computing Devices - IEEE Globecom 2021 - https://ieeexplore.ieee.org/document/9685095 
    • Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing  - https://openreview.net/forum?id=p99rWde9fVJ 

 

Overall, the reviewer thinks the topic is relevant and the proposed approach valid for the edge use-cases, however the paper should be improved for what concerns the explanation of the algorithm and its evaluation (more results).

Author Response

The authors would like to thank the area editor and the reviewers for their precious time and invaluable comments. We have carefully addressed all the comments. The corresponding changes and refinements made in the revised paper are summarized in our response below.

Reviewer: 1

  1. Since the authors are considering the edge devices (low capacity devices) in the scenario, the training process must not only target for highest accuracy, but also on reducing the training time and the resource consumption (CPU and memory consumption). For instance, identifying what is the best number of epochs per training round in federated machine learning model training taking into account the CPU consumption of the devices is very critical.

 

Response:  Thank you very much for your comment. As per the above-mentioned query, based on the experimental results it is observed that the utilization of CPU is optimal till 10 epochs.

  1. The experimental results should be further discussed including more insightful comments. More graphs showing the benefits of Genetic CFL algorithms are needed.

 

Response:  Thank you very much for your comment. As per the above-mentioned query precision, recall, and F1-Score results has been added.

  1. Some of the paper in literature review are not from the problem domain that the authors are trying to solve:

 

Some very relevant papers that needs to be cited: 

 

FedTune: Automatic Tuning of Federated Learning Hyper-Parameters from System Perspective - https://arxiv.org/abs/2110.03061

 

BePOCH: Improving Federated Learning Performance in Resource-Constrained Computing Devices - IEEE Globecom 2021 - https://ieeexplore.ieee.org/document/9685095 

 

Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing -  https://openreview.net/forum?id=p99rWde9fVJ 

 

Response:  Thank you very much for your comment. As per the above-mentioned query, the above-mentioned papers has been added.

 

Reviewer 2 Report

Firstly I would like to applaud the author for a well-written paper. The content is presented clearly, and the reader can readily appreciate the author's point without much confusion. 

 

I do not find any error in the methodology and its validations. Nevertheless, there are some points to improve this paper. 

 

1) There's no discussion on the dataset, the data source, feature, size etc. Is it binary or multiclass, or multilabel? Without knowing the dataset's characteristics, it is hard to appreciate the result of the training. 

 

2) There's no mention of the ANN architecture it uses. Is it shallow ANN, deep ANN? Thus the model used are homogenous or heterogeneous. Will this method work in a model-agnostic environment? 

 

3) Does the algorithm only optimize the learning rate? How about other hyperparameters such as activation function, regularizer, dropout rate etc.?

 

4) The result should be expanded to not just focus on accuracy. Sensitivity, recall and F1 should also be shown. Accuracy alone is not enough to prove that GFL is better than FL. 

 

5) This method closely resembles federated gossip learning and other FL that uses non-global learning. While it is great if authors can compare their results with different types of cluster base FL, it is sufficient if the author also expands the literature review to include another cluster base FL. 

 

Thank you. I am looking forward to here the command from the authors. 

Author Response

Reviewer: 2

 

  1. There's no discussion on the dataset, the data source, feature, size, etc. Is it binary or multiclass, or multilabel? Without knowing the dataset's characteristics, it is hard to appreciate the result of the training.   

Response:  Thank you very much for your comment. As per the above-mentioned query the dataset description table has been provided in the revised manuscript.

      Updates are made in Table (2).

  1. There's no mention of the ANN architecture it uses. Is it shallow ANN, deep ANN? Thus, the model used are homogenous or heterogeneous. Will this method work in a model-agnostic environment? 

Response:  Thank you very much for your comment.  As per the above-mentioned query, the ANN architecture and its uses has been added.

  1. Does the algorithm only optimize the learning rate? How about other hyperparameters such as activation function, regularizer, dropout rate, etc.?

 Response:  Thank you very much for your comment.  As per the above-mentioned query, currently, our paper deals only with optimizing the learning rate. We will consider other hyper-parameter optimizations in our future work.

 

  1. The result should be expanded to not just focus on accuracy. Sensitivity, recall, and F1 should also be shown. Accuracy alone is not enough to prove that GFL is better than FL. 

Response:  Thank you very much for your comment. As per the above-mentioned query precision, recall, and F1-Score results has been added.

 

  1. This method closely resembles federated gossip learning and other FL that uses non-global learning. While it is great if authors can compare their results with different types of cluster base FL, it is sufficient if the author also expands the literature review to include another cluster base FL. 

Response:  Thank you very much for your comment. As per the above-mentioned query, in the literature review, a few papers related to cluster base FL has been added.

Updates are made in Table (1).

Round 2

Reviewer 1 Report

The requested changes has been applied. 

Reviewer 2 Report

I recommend this paper to be publish