Federated Learning: Centralized and P2P for a Siamese Deep Learning Model for Diabetes Foot Ulcer Classification

This manuscript is very interesting and contributed to the knowledge. The significant contribution of this paper is the exploration of Peer-to-Peer (P2P) federated learning in the context of classifying Diabetes Foot Ulcers (DFU) using a Siamese deep learning model. The authors extend the FedAVG algorithm, a leading centralized federated learning algorithm, to operate in a P2P environment. They refer to these extended algorithms as FedAVGP2P and FedSGDP2P. The study compares the performance of FedAVGP2P and FedSGDP2P with their centralized counterparts in terms of model convergence behaviors and communication costs. The results demonstrate that using FedAVGP2P, the average model convergence behavior is comparable to models trained with centralized federated versions. This research provides insights into the feasibility of using P2P federated learning for collaborative model training without sharing sensitive data, specifically in the context of DFU classification. However, the paper should be improved in some regards by giving more explanation about:

Reviewer’s comment

Authors’ response

What is federated learning and how does it relate to classifying Diabetes Foot Ulcers?

In Section 2, Background and Preliminaries, we give an introduction of Federated Learning Architectures which mainly concerns confidentiality of sensible data.

In Section 1. Paragraph 2, we explain problem of training machine learning models when medical data is concerned.

In Section 1. Paragraph 3 we explain how Federated Learning ensure raw training data is not shared by clients and how DFU is concerned.

We have added the following paragraph to make the link between FL and classification of DFUs.

Can the authors provide more information about the Siamese deep learning model mentioned in the manuscript?

We have added a paragraph to explain same in in section 2. Background and Preliminaries, sub section 2.1. Siamese Neural Network (SNN).

Further in section 5.2 Application of FL P2P for DFU classification we have explained the architecture of the Siamese network we propose to experiment which is based on a CNN and ViT combination.

How does the use of data sharing between medical institutions benefit the classification of Diabetes Foot Ulcers?

In section 5.2 Application of FL P2P for DFU classification, we have added a paragraph justifying the experimentation with FL.

Could the authors acknowledge any limitations of the study and suggest avenues for future research?

This demonstrates a critical understanding of the research and helps to guide further investigations in the field.

Kindly note that we have sub-section 5.7 limitations which explains the limitation we faced in terms of processing power.

Concerning avenues for future research, in section 6. we point towards experimenting with more other Heuristics and more importantly being able to experiment with more clients which was impossible because of lack of computation resources.

:

Reviewer’s comment

Authors’ response

Keywords should be sorted alphabetically.

We have made appropriate corrections and it is now listed as follows:

Centralised and Peer-2-Peer Architecture, Confidentiality of data, Diabetes Foot Ulcer and Deep Learning, Federated Learning, Siamese Network

References should be updated, only 62% are from the last 5 years.

Kindy note that there are limited references in the field of federated learning while we worked on this article. We have nevertheless added more references from latest publications.

Fig 3 and 5 are not clear

·        We have resized Figure 3 to its original size making it more readable.

·        We have also increased the size of the tow images in Figure 5 making it more interpretable for readers.

You jumped from Fig 5 to figure 11, I think you miss writing the captions of the figures in between.

The captions were cropped for some reason. We have corrected same and now all the figure from fig 5, fig6 …to figure 11 appears with appropriate caption.

In the introduction, you need to emphasize why your work is distinguished, and where the novelty is

We have amended the following paragraph: Section 1. Introduction

The work seems promising; however, the output is little discussed, you need to explain the results especially from Fig 5- Fig 11.  

We have added a paragraph after each figure to explain the results obtained from figure 5 to figure 11. These are highlighted in revised article.

The overall impression of the technical contribution of the current study is reasonable. However, the Authors may consider making necessary amendments to the manuscript for better comprehensibility of the study.

Reviewer’s comment

Authors’ response

The method names should not be capitalized. Moreover, it is not the best practice to employ abbreviations in the abstract, they should be used when the term is introduced for the first time.

We have reviewed the abstract as follows:

It is known fact that AI models need massive data for training. In medical field the data are not necessarily available at a single site but distributed over several sites. In the field of medical data sharing, particularly among healthcare institutions, the need to maintain the confidentiality of sensitive information often restricts the comprehensive utilization of real-world data in machine learning. To address this challenge, our study experiments an innovative approach using federated learning to enable collaborative model training without compromising data privacy. We present an adaptation of the Federated Averaging algorithm, a predominant centralized learning algorithm, to a Peer-to-Peer federated learning environment. This adaptation led to the development of two extended algorithms: Federated Averaging Peer-to-Peer and Federated Stochastic Gradient Descent Peer-to-Peer. These algorithms were applied to train deep neural network models for the detection and monitoring of diabetic foot ulcers, a critical health condition among diabetic patients. Our work compares the performance of Federated Averaging Peer-to-Peer and Federated Stochastic Gradient Descent Peer-to-Peer with their centralized counterparts in terms of model convergence and communication costs. Additionally, we explored enhancements to these algorithms using targeted heuristics based on client identities and F1 scores for each class. The findings demonstrate that models trained with Federated Averaging Peer-to-Peer exhibit convergence behavior comparable to those trained using traditional centralized federated learning methods, marking a significant advancement in the domain of privacy-preserving medical data analysis and machine learning.

The contribution of the current study must be briefly discussed as bullet points in the introduction. And motivation must also be discussed in the manuscript.

The motivation of the is part of the modified Section 1. Introduction:

We have nevertheless added a new paragraph:

Introduction section must discuss the technical gaps associated with the current problem.

We have modified Section 1. Introduction to better explain the context, objective and novelty.

The literature section is missing. Authors are recommended to incorporate the same for better comprehensibility of the study.

We have Section 3. Related Work, where we present some work done in the field of Federated Learning. It should be noted that the are limited literature which are available for this field.

“Centralised Federated Learning Architecture” what is the significance of the sub-section, it a known fact to everyone about federated learning?

This paper deals with Deep learning, Medical Data and data confidentiality. It is the intersection of several fields. Hence, the need to have a brief introduction on centralised federated learning to improve the readability of the article coming from multiple domain.

Figure 2 can be further enlarged and the image seems to be snipped from different online source.

We have amended it with a better quality version.

Where is the base model discussed and how are the weights updated in the ferderated model. For better idea refer the studies like “A Software Framework for Intelligent Security Measures Regarding Sensor Data in the Context of Ambient Assisted Technology“

We have now included 2.1. Siamese Neural Network (SNN) where we explain the weight sharing mechanism and how training take place and the loss function normally used in SNN.

We have carefully studied the proposed article to gain insights. The article also comfort us to pursue the work in this field.

More explanation of the proposed model is desired on technical grounds.

While we have not concentrated on working on the deep learning model directly we have however expanded Section 5.2. Application of FL P2P for DFU classification by explaining the models of the ensemble architecture which is used.

We have also amended Sub Section 5.4. Experimental parameters and added a table with additional parameters we are using. We further intend to make available via GITHUB in open access once the article is published.

What is the size of the input image that is considered for processing and the size of the kernels?

We have amended Sub Section 5.4. Experimental parameters and added a table with additional parameters. Kindly note we intend to make the codes available via GITHUB once paper is published.

The important details, like the input/tensor/kernel size, must be discussed, and whether authors have used Stride 1 or Stride, 2 must be presented. What type of activation function is being used in the current study.

We have amended Sub Section 5.4. Experimental parameters and added a table with additional parameters.

Kindly note we intend to make the codes available via GITHUB once paper is published.

For how many epochs does the proposed model execute. what is the initial learning rate, and after how many epochs does the model's learning rate saturated.

We have amended Sub Section 5.4. Experimental parameters and added a table with additional parameters.

What are those features that are being exchanged in federated environment.

The features shared are gradient, weights and learning rate. We intend to make the codes available via GITHUB once paper is published.

Please discuss more on the implementation platform and the dataset details as two sub-sections in the manuscript.

We Sub Section 5.3. Dataset  which describe the dataset used and also the class distribution and points to its limitation.

 What are the cases assumed as TP, TN, FP, FN (confusion matrix) in the current study.

Despite the fact that our objective is not specifically to get best accuracy but to investigate the performance of P2PFL and CFL we still use the F1 score as a threshold. Hence this is how the TP, TN, FP and FN are assumed:

In our study, the confusion matrix is constructed to evaluate the performance of the multi-class classification model across four distinct classes: Both, Infection, Ischaemia, and None. Here is how we define each term within our confusion matrix for each class:

True Positives (TP): These are cases where the model correctly identifies the presence of a condition. For instance, if a case is actually 'Both' (meaning it has both Infection and Ischaemia), and our model also predicts 'Both', it is counted as a true positive for that class. Similarly, we count TPs for each of the other classes (Infection, Ischaemia, None) when the model's prediction matches the actual label.

True Negatives (TN): These are the cases where the model correctly identifies that a condition is not present. For example, for the class 'Infection', a true negative is when the model predicts any class other than 'Infection' (be it Ischaemia, Both, or None), and the actual class is indeed not 'Infection'. This logic applies similarly for the other classes.

False Positives (FP): These occur when the model incorrectly predicts the presence of a condition. Taking 'Ischaemia' as an example, if the model predicts 'Ischaemia' when the actual class is 'None', 'Infection', or 'Both', it would be considered a false positive for 'Ischaemia'.

False Negatives (FN): These occur when the model incorrectly predicts the absence of a condition. Using the class 'None' as an example, a false negative would be when the model predicts either 'Infection', 'Ischaemia', or 'Both', but the actual condition is 'None'.

A Sub-Section 5.5. Metrics have been added to include the above.

More comparative analysis with state-of-art models is desired.

The base model is the centralised model. This work is the first realised on DFU dataset with objective to compare it to FL P2P approach. Hence, no comparison is done with existing model.

By considering the current form of the conclusion section, it is hard to understand by MDPI Journal readers. It should be extended with new sentences about the necessity and contributions of the study by considering the authors' opinions about the experimental results derived from some other well-known objective evaluation values if it is possible.

We have amended the conclusion and tried to make more insightful.

Authors should use more alternative models as the benchmarking models, authors should also conduct some statistical tests to ensure the superiority of the proposed approach, i.e., how could authors ensure that their results are superior to others? Meanwhile, the authors also have to provide some insightful discussion of the results.

Discussions have been amended to take into consideration the review. However, the objective of the article is to confront Centralised Federated learning and Federated Learning in a Peer to Peer environment using four different heuristics. We do propose to test several other heuristics with varying parameters as  future work.

 Are the conclusions substantiated by the results?

English proofreading is strongly recommended for a better understanding of the study. Few sentences are written in passive voice and it is also observed that few sentences stopped abruptly. 

We have reviewed the article and correction English related errors.

I have read the article and I have found that the topic is interesting and instructive in machine learning-based medical applications. However, the article requires revision as there are many concern points. I have highlighted many comments and I recommend major corrections to be strictly made by the authors.

Reviewer’s comment

Authors’ response

The abstract has to highlight clearly the contribution and novelty.

We have modified the abstract as follows:

It is known fact that AI models need massive data for training. In medical field the data are not necessarily available at a single site but distributed over several sites. In the field of medical data sharing, particularly among healthcare institutions, the need to maintain the confidentiality of sensitive information often restricts the comprehensive utilization of real-world data in machine learning. To address this challenge, our study experiments an innovative approach using federated learning to enable collaborative model training without compromising data privacy. We present an adaptation of the Federated Averaging algorithm, a predominant centralized learning algorithm, to a Peer-to-Peer federated learning environment. This adaptation led to the development of two extended algorithms: Federated Averaging Peer-to-Peer and Federated Stochastic Gradient Descent Peer-to-Peer. These algorithms were applied to train deep neural network models for the detection and monitoring of diabetic foot ulcers, a critical health condition among diabetic patients. Our work compares the performance of Federated Averaging Peer-to-Peer and Federated Stochastic Gradient Descent Peer-to-Peer with their centralized counterparts in terms of model convergence and communication costs. Additionally, we explored enhancements to these algorithms using targeted heuristics based on client identities and F1 scores for each class. The findings demonstrate that models trained with Federated Averaging Peer-to-Peer exhibit convergence behavior comparable to those trained using traditional centralized federated learning methods, marking a significant advancement in the domain of privacy-preserving medical data analysis and machine learning.

Confidentiality of data has not been assessed in this study. Can it be involved in Keywords?

This work tackles the confidentiality and data privacy as concerning the training of data for deep learning model. It does not access confidentiality but proposes solution to tackle confidentiality of medical data for training deep learning model.

I suggest the following study to be included in the introduction part:

(Iot and cloud computing in health-care: A new wearable device and cloud-based deep learning algorithm for monitoring of diabetes)

This study was promptly considered and included in the introduction part.

  The authors mentioned "The main objective of this work…". Please, try to conduct contributions instead of objectives.

Section1. Introduction has been modified and made clearer to point out novelty of work.

The authors have to explain why the conducted this study towards centralized in spite of their weakness at some aspects.

This work access the possibility using a peer to peer federated learning approach as compared to Centralised federated learning approach. Thus it was important to experiment with Centralised FL and use the output as a basis of comparison for the Peer-to-Peer counterparts.

There are missing sentences like "on an intermediary and this…"

We have conducted a complete proof reading of the whole article and made corrections.

Figure 2 has low resolution.

 We have amended it with a better quality version.

The authors used previously studied algorithms and they developed their algorithm. What is the new!!!

We have extended Federated Averaging algorithm in a Peer to Peer approach and also propose Federated Stochastic Gradient Descent Peer-to-Peer. We experiment same with 4 heuristics and we present the results comparing it to the centralised architecture using a Siamese neural network on a real diabetic ulcer dataset meticulously annotated by medical experts.

The numbering of terms is not suitable. (Section 3.2).

While we do not have Section 3.2. We checked and corrected misinterpretation of Section 2.3 where we have a list of items.

The weakness of related works has to be addressed and the motivation has to be highlighted accordingly.

We have added a paragraph in Section 3 to tackle the need for additional research.

The results have to be numerically qualified; numeric Tables have be listed for comparison.

Kindly note that we have interpreted the numerical values in each graph presented and we summarised it in Table 2. Summary of Communication rounds needed to cross f1-score threshold.

The authors have mentioned that "the performance has been assessed in terms of model convergence behaviors and communication costs". The authors have not evaluated both indices in results.

We have set the F1-score at 0.9 and set it as a benchmark to evaluate the Algorithms and the number of communication rounds it takes to cross the threshold. We further evaluate the model convergence in terms of Heuristics and Fraction of Client.

The study lacks mathematical analysis.  

The aim of the study was to experiments the behaviour of DFU  

The conclusion is descriptive. It is void of quantitative and numerical improvement and comparison.

We discussed the numerical values in discussion and added a further paragraph in discussion section.

The future work has to be added.

Conclusion have been re-organised and future works included .