Detection of Bovine Mastitis in Raw Milk, Using a Low-Cost NIR Spectrometer and k-NN Algorithm

Round 1

Reviewer 1 Report

Authors provided the interesting method for bovine mastitis detection using smartphone-connected NIR spectrometer. The method was optimized using k-Nearest Neighbors models. The simulated results showed reasonable accuracy and classification data. However, authors need to check the reference format. In addition, authors need to provide some valuable comments for the results. Authors need to check the journal format guidelines. Therefore, the manuscript could be minor revision if authors follow the guidelines as below.

1. Authors need to follow the journal format guidelines.
2. Authors need to provide the funding information and data availability sections.
3. Authors need to provide the date, city, and country information for conference papers.
4. Authors need to reduce line spacing so please check format.
5. In Figure 1, the labels located in the top section needs to be enlarged.
6. In Figure 1, information of x- and y-axis need to be provided such as wavelength.
7. Figure 5 labels are too small.
8. In Table 2, authors need to provide how 5 different conditions are divided.
9. Authors had better provide the reference (When a Machine Learning model is developed, its performance has to be evaluated on a 164 different dataset than the one used for training) with (https://www.mdpi.com/1424-8220/21/15/4968).
10. Authors mentioned that "the differences among weak positive and strong positive are statistically significant". I am wondering how to obtain p value (<0.05).
11. I am still curious how to collect the data for the experiment and how to determine the classification. Please provide clearer description.
12. Authors mentioned that "l is the length of the light through the sample". How to determine the length among the samples ?

Author Response

Author's Reply to the Review Report (Reviewer 1)

 

Thank you for your time, we really appreciate your accurate comments. They have been carefully read, and the current version is the result of those modifications according to them. We have checked the reference format, in order to adjust our manuscript, also we noticed that numbers in text cites changed, due to an issue with the reference manager, we have carefully double checked and corrected, when needed, the reviewed version.

 

In the following paragraphs, we have put in a lighter colour your original comment while the specific response or modifications on the paper has been added in a darker colour: 

 

Authors need to follow the journal format guidelines.

We have adjusted the document to the journal format guidelines according to the template submitted by the editor.

 

Authors need to provide the funding information and data availability sections.

A specific section to provide the funding information is at the end of the manuscript while a link to the data is available on the new section as well as the relationship with the funding.

 

Authors need to provide the date, city, and country information for conference papers.

According to Applied Science citation style that information should not be included while the proceedings are referenced. However, we have included this information in the reviewed version unless the editorial said that we have to remove it.

 

Authors need to reduce line spacing so please check format.

The spacing has been changed in the whole text in order to match the style guide of the editorial.

 

In Figure 1, the labels located in the top section needs to be enlarged. In Figure 1, information of x- and y-axis need to be provided such as wavelength. Figure 5 labels are too small. In Table 2, authors need to provide how 5 different conditions are divided.

Thank you for your feedback on this particular issue. We have reviewed all figures and tables to improve the readability and they have also been reordered.

 

Authors had better provide the reference (When a Machine Learning model is developed, its performance has to be evaluated on a 164 different dataset than the one used for training) with (https://www.mdpi.com/1424-8220/21/15/4968).

Thank you for the recommendation, the reference has been included.

 

Authors mentioned that "the differences among weak positive and strong positive are statistically significant". I am wondering how to obtain p value (<0.05).

We used a simple t-test to determine the statistical significance, in the revised manuscript, now we mention this.

 

I am still curious how to collect the data for the experiment and how to determine the classification. Please provide clearer description.

In section entitled Materials and Methods, a subsection “Data Acquisition” can be found describing the process. For each sample, the data was collected in the farms and, right after the acquisition, the sample was scanned with a portable low-cost NIR device. Once the  spectrometer was obtained, the sample was tested using the gold standard technique,CMT. The result of the CMT determines the sample as positive or negative and as that the spectrometer was labelled and added to the dataset. The whole dataset can be accessed in the following link which has also been added to the text.

https://data.mendeley.com/datasets/mj8wcb9fsd/draft?a=f457ba3f-3005-4153-97b3-048edfb4027f

 

Authors mentioned that "l is the length of the light through the sample". How to determine the length among the samples ?

Fort this particular propose, we have used the method described in https://doi.org/10.1016/j.trac.2009.07.007 which has been also included in the text and clarified

 

Author Response File: Author Response.docx

Reviewer 2 Report

The premise of the paper - a low cost, rapid method for on-farm detection of mastitis is potentially interesting but the paper in its current form has many significant weaknesses. It is quite poorly referenced and the novelty is questionable. Well known algorithms are applied to a problem which has seemingly been addressed before.

Specific feedback:

line 11 ‘Among the bovine diseases, mastitis is the one causing the highest economic losses in any dairy production system’. While references are given in the Introduction relating to economic loss due to Mastitis, no reference is given to specifically support this statement (that is higher than any other bovine disease). Also the references given are about ten years old – please use more recent sources.

line 48: ‘However, this method requires a minimal expertise to interpret the results, and also the handling of reagents in the farm[12].’ Do you mean a minimal level of expertise? minimal expertise suggests anyone can do it. Please also clarify the negative aspects of handling reagents on the farm. It is not stated how long the test takes.

line 59 – place the appropriate reference where Russell is cited.

References 18 and 19 were published over a decade ago – it is not clear how the current work improves on these?

The referencing of literature on Machine Learning algorithms applied to NIR is poor – recent references are not included. The justification for the k-NN approach is based on a paper from 2008. NIR is widely used in food, pharmaceutical, chemical and other processes and there has been intensive research on the effectiveness of various algorithms in recent years.

line 119 who are the ‘experts’ and how many were there? Please elaborate and add a reference if possible to strengthen the assertion that the test is quite subjective.

 

line 154: Describes only filter-based methods of feature selection. Embedded and Wrapper methods are ignored.

Figure 5 is too small. Also make it clear in which loop the hyperparameters are selected

Equations (4)-(9) These are incorrect. In general the equations throughout are poorly presented and the equation numbering is in the wrong order.

line 247 what is meant by ‘calculation of descriptive features’?

Figure 2 there is no legend for the symbols a) to i) representing different preprocessing techniques. Parts a) and b) are not labelled.

line 302 spelling mistake on ‘nearest’

The comparison of results in the discussion section is unclear – what methods and datasets have been used in these alternative works? Particular what validation method is used? Leave-One-Out CV is a weak validation method. This point should be made in the discussion.

Author Response

Author's Reply to the Review Report (Reviewer 2)

 

First and foremost, we want to acknowledge that you take your time in order to review this paper. Your comments have been very valuable to improve the paper according to the issues that you have pointed out.

 

On the following paragraphs your comments can be found in a lighter colour while the response to them or the modifications on the text can be found in a darker font.

 

The premise of the paper - a low cost, rapid method for on-farm detection of mastitis is potentially interesting but the paper in its current form has many significant weaknesses. It is quite poorly referenced and the novelty is questionable. Well known algorithms are applied to a problem which has seemingly been addressed before.

Thank you for your words about the interest of our work. According to the reference, this point has been reviewed and improved significantly, while, about the novelty of the work, the key element in this development is the use of a low-cost and portable NIR instead of a laboratory grade one as previous works have made. In order to work with the reduced quality of the signals that the device provided we have explored the use of well-known Machine Learning approaches. It is true that the algorithms are not new but the challenge is to capture information with the described device and make a prediction in the wild with a similar grade of confidence that the gold-standard is there.

Specific feedback:

line 11 ‘Among the bovine diseases, mastitis is the one causing the highest economic losses in any dairy production system’. While references are given in the Introduction relating to economic loss due to Mastitis, no reference is given to specifically support this statement (that is higher than any other bovine disease). Also the references given are about ten years old – please use more recent sources.

Thank you for your observation, we have changed the text in Abstract section: now it reads:

“Among the bovine diseases, mastitis is causes high economic losses in dairy production system.”

line 48: ‘However, this method requires a minimal expertise to interpret the results, and also the handling of reagents in the farm[12].’ Do you mean a minimal level of expertise? minimal expertise suggests anyone can do it. Please also clarify the negative aspects of handling reagents on the farm. It is not stated how long the test takes.

Thank you for your comment, in the revised document, the text has been changed for:

However, this method requires a minimal expertise to interpret the results, and also the handling of reagents in the farm is a challenge for small farmers[12].

line 59 – place the appropriate reference where Russell is cited.

Thank you for pointing to this mistake. Now the reference has been correct and it can be read correctly

References 18 and 19 were published over a decade ago – it is not clear how the current work improves on these?

Indeed other works used a laboratory spectrometer, however, there is no reference of a similar work in which a low-cost portable spectrometer has been used. In our work the aim was to develop a low-cost, real-time, field-applicable method to determine the presence and the severity of bovine mastitis

The referencing of literature on Machine Learning algorithms applied to NIR is poor – recent references are not included. The justification for the k-NN approach is based on a paper from 2008. NIR is widely used in food, pharmaceutical, chemical and other processes and there has been intensive research on the effectiveness of various algorithms in recent years.

An improved version of the bibliography has been included in the manuscript, where several updated references have been included and adequately referenced in the text.

line 119 who are the ‘experts’ and how many were there? Please elaborate and add a reference if possible to strengthen the assertion that the test is quite subjective.

Dr. Lenin Aguilar (co-author of this manuscript) is an expert in that field. Previously to this one, other works has stated that CMT is a subjective screening test. Here are some examples:

McDougall, S., Murdough, P., Pankey, W., Delaney, C., Barlow, J., & Scruton, D. (2001). Relationships among somatic cell count, California mastitis test, impedance and bacteriological status of milk in goats and sheep in early lactation. Small Ruminant Research, 40(3), 245-254.

Perrin, G. G., Mallereau, M. P., Lenfant, D., & Baudry, C. (1997). Relationships between California mastitis test (CMT) and somatic cell counts in dairy goats. Small Ruminant Research, 26(1-2), 167-170.

line 154: Describes only filter-based methods of feature selection. Embedded and Wrapper methods are ignored.

Thank you for pointing out this issue which can confuse or mislead some readers. We have change that paragraph according to:

“Generally speaking, feature selection techniques try to reduce the number of input variables while keeping the maximum variability. This reduction can be performed in many different ways, but they can be broadly cframed in: filter methods, wrapped methods and embedded methods[34] . Among all possibilities, filtered methods are the most frequent ones. In general, those methods are based on the use of a threshold with a filter to choose a subset of the best wavelengths to develop a model[35,36]. In this work, a Correlation-based Feature Selection (CFS) filter proposed by Hall[37] was applied. This method measures the correlation of each feature with the output variable and among each other features. To evaluate CFS, it is important to follow the hypothesis that a good feature is highly correlated with the desired output variable and, oppositely, it is lowly correlated to other features. The threshold used by the CFS in this work has been tested with values between 1 and 100.”

Additionally the folllowing reference has also been included.

Chandrashekar, Girish, and Ferat Sahin. "A survey on feature selection methods." Computers & Electrical Engineering 40.1 (2014): 16-28.

 

Figure 5 is too small. Also make it clear in which loop the hyperparameters are selected

Thank you for your comment, this issue has been addressed and all figures in the text have been reviewed and improved to increase the clarity and made them more informative.

Equations (4)-(9) These are incorrect. In general the equations throughout are poorly presented and the equation numbering is in the wrong order.

All equations in the text have been reviewed and improved, the equation numbering has been also reordered correctly.

line 247 what is meant by ‘calculation of descriptive features’?

In the revised manuscript it now reads: Data preprocessing is required to enhance the quality of the collected data. This improvement could lead to a reduction of white noise or to the calculation of descriptive features that can be used as inputs to the ML system instead of raw signals which can be inputs for a machine learning model.

Figure 2 there is no legend for the symbols a) to i) representing different preprocessing techniques. Parts a) and b) are not labelled.

In Figure 2 caption the different preprocessing techniques are mentioned according to the letter on top of each subfigure.

line 302 spelling mistake on ‘nearest’

Thank you for your comment, it has been corrected in the revised version of manuscript

The comparison of results in the discussion section is unclear – what methods and datasets have been used in these alternative works? Particular what validation method is used? Leave-One-Out CV is a weak validation method. This point should be made in the discussion.

Thank you for your comment, but in this point let us politely disagree with the reviewer. Leave-one-out-cross-validation is one of the best established and robust methods to determine the suitability of a particular setup. Based on a solid statistical theory, the method is particularly indicated  with small datasets which are expensive to capture or contain very rare events. This method has been used and proved its relevance an on several publications that used this method such as:

 

Vehtari, Aki, Andrew Gelman, and Jonah Gabry. "Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC." Statistics and computing 27.5 (2017): 1413-1432.

 

Wong, Tzu-Tsung. "Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation." Pattern Recognition 48.9 (2015): 2839-2846.

 

Rushing, Christel, et al. "A leave-one-out cross-validation SAS macro for the identification of markers associated with survival." Computers in biology and medicine 57 (2015): 123-129.

 

Therefore, we could not agree with that point. On the other hand, we agree that including the dataset and the validation methods of the previous works is interesting. In Table 2, we have included the details of the dataset. Unfortunately, the mentioned works do not report any validation method due the fact that they used the whole dataset to adjust their methods due to the fact is a direct calculation with hard thresholds. Therefore we have included a table (Table II) for easy reference with that information on the previous works and the discussion has been adapted. Now it can be read as: 

 

“

High accuracies were achieved for both models as shown in Table 1. According to Hamann & Krömker[13], these models detect changes in milk constituents which are related to mastitis. For example, levels of protein, chloride and sodium tend to increase along with SCC while the levels of potassium and lactose tends to decrease.

Table 1. Performance metrics of models on test set, 10-fold nested cross-validation

Performance Metric	Model 1: Positive - Negative	Model 2: Clinical - Subclinical
Accuracy	x̅=0.912 σ=0.051	x̅=0.951 σ=0.08
Sensitivity	x̅=0.858 σ=0.129	x̅=0.95 σ=0.158
Specificity	x̅=0.94 σ=0.064	x̅=0.957 σ=0.096
Positive Predictive Value	x̅=0.89 σ=0.11	x̅=0.917 σ=0.18
Negative Predictive Value	x̅=0.925 σ=0.072	x̅=0.978 σ=0.07
F1 Score	x̅=0.867 σ=0.089	x̅=0.913 σ=0.144

In Jaeger et al.[56], the authors reported a SEN of 90.3% and a SPC of 71.8% in the detection of SCC with laboratory manual test as best results. Comparing these results with the one reported in this work, we can see that the sensitivity has no significant difference considering the intervals reported on the work, while the Specificity has a significant bump up . Meilina et. al.[18] reported another direct calculation method based on NIR spectra by simply using a double threshold to determine the presence of SCC. This work reported 77.78% and 80.56% for SEN and SPC by using a lab grade NIR spectrometer and the comparison with SCC. These results are summarized on table 2 together with comparative results of the proposed method and the details of the different datasets. It may be highlighted that some of the relevant data are not available on the sources.

 

Table 2: Comparison of datasets used 

Work	Dataset	Positive	Public	Validation Method	Sensitivity	Specificity
Jaeger et al.[56]	433	275	No	-	90.3%	71.8%
Meilina et. al.[18]	666	-	No	-	77.8%	80.6%
This work	210	80	Yes	LOOCV	85.8%	94.0 %

 

Although Aernouts et al.[25] argue that short range spectrometers (400–1,000) are not good enough for accurate monitoring of composition on raw milk, this study has proved that udder health can be estimated using a low-cost spectrometer which operates in a range close to the aforementioned with similar results to those obtained using laboratory methods.

Author Response File: Author Response.docx