A Scalable Open-Source Framework for Machine Learning-Based Image Collection, Annotation and Classification: A Case Study for Automatic Fish Species Identification

Round 1

Reviewer 1 Report

In this paper, the authors proposed an application of deep learning for automatic fish species identification. Overall, there is no novelty in the work as the authors relied on the pool of existing known models. In addition, the amount of data used for training is very small which questions the generalization of the work.

 1. In the abstract, What do fish observations stand for and how is it different from fish images?

2. These statements in the abstract highlight the essence of image collection with ML models:

a. Given that large quantities of fish observations and images are being collected by fishers every day, artificial intelligence (AI) and computer vision applications offer a great opportunity “to improve data collection”, automate analyses and inform management.

b. The framework presented here is highly customisable for further advancement and “community based image collection” and annotation

 How can data collection be improved with artificial intelligence (AI) and computer vision?

 3. Why AI based models for the community need to be developed when commerical ones are successful?

 4. Do methods, tools, and scripts represent the different information in the context of the problem? If that is the case, authors should provide an explanation.

 5. Data management does not mean image storage and its pre-processing. Need correction or provide strong evidence of using this word in AI domain for data acquisition, cleaning and pre-processing. If the former case is true, the text and figure need revision.

 6. Computer vision and related definitions are widely known to the research community. No need to put a separate paragraph (Page 2, lines 99-109)

 7. The image storage subsection should mainly discuss image characteristics rather than comparing cloud-based storage services. A one-sentence mention will suffice if authors have used this service during model training and evaluating test images.

 8. The content provided in the "Module 2: Image annotation" subsection is generic and can be removed.

 9. Who were the experts in manual annotations of images?

10. Are tools mentioned in Table 2 used for annotations? If not, there is no relevance in keeping them. This table should be removed.

 11. The main focus of the paper, i.e., deep learning, was explored little. The article should compare the characteristics, parameters and novelty of these frameworks and relate the outcomes of fish image classification with other problems using these models. It would be helpful to assess the generalization of the frameworks to solve classification problems.

 12. Given the small dataset size, high performance is expected with transfer learning. More efforts could be made to increase the dataset size and develop a robust model. The author themselves raised this concern.

13. The impact of data augmentation is missing in the paper. Why did the authors raise the concern about data augmentation in the discussion? Do they have doubts about their own work? If so, why have they not explored the current options like GANs?

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

The organization of rest of the manuscript should be provided at the end of section 1

Description of metadata should be specified, i.e. which metadata are stored along with the image

 

Too much of discussion on cloud in section Module 1, it shall be reduced. The information is not much useful in the context of work.

 

It would be nice if the author can share the results of intermediate steps of the algorithm, such as images with faces, images with detected faces, bounding boxes around detected objects, etc.

 

 

Authors have used things readily available almost for the entire work. The research component is missing in the article. It is expected that the author should come up with real challenges and propose remedies for that

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

1. Clarity in the objectives missing.

2. Proposed methodology is clear.

3. More recent articles should be analyzed.

4. More focus should be given to Result analysis section.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 4 Report

This report (ID: sustainability-1813372) is entitled “A scalable open-source framework for machine learning based image collection, annotation, and classification: a case study for automatic fish species identification” by research group Asta Audzijonyte. This article is fascinating and can make a great contribution to the scientific literature. The reviewer suggests revision before publication in a peer-reviewed journal. Some specific comments on the manuscript:

The article is innovative and exciting, Provide graphical abstract.

In introductions: Support statements with appropriate citations. Line 73: To help address, use either help or address

Suggest the possibility of practical applications, Insights on future steps after the pilot study;  provide sufficient data sets.

The suggestion is to improve flow and English. The overall presentation is descriptive and challenging to follow, improving readability.

Conclusions: Focus on the scope (such as future implications, sustainability, cost-effectiveness, improvement in performance)

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

 

The authors have addressed all the comments and made the required changes to improve manuscript quality, but still there are some minor issues.

P3, L12-16 needs to be revised.

Since the described framework is standard, P3, L121-12 can be stated briefly.

Code availability in Figure 1 is not required.

The authors should remove the background definitions as this material is readily available nowadays. So, these definitions would not be much helpful to the readers.

Figure numbering is wrong "Figure 32:" and "Figure 43:"

Image quality of the confusion matrix is poor

Since the authors agree that their study is a pilot one, they should not draw solid and lengthy conclusions as given in "Lessons learned, challenges and future applications" section. Specifically, ensemble learning, data augmentation, and help from citizens should be either removed or stated briefly. Often, findings at small scale do not generalize to a large scale. So, the authors should give general findings only.

The authors must remove the statement that your pipelines can be extended to other research areas that require image annotation, processing and classification owing to the pilot study (P3, L134-137).

Author Response

Comments and Suggestions for Authors

The authors have addressed all the comments and made the required changes to improve manuscript quality, but still there are some minor issues.

P3, L12-16 needs to be revised.

Response: We have corrected the citation box in page 1 to address this comment but we are not sure what section of the manuscript the reviewer is referring to as lines 12-16 are in page 1.

Since the described framework is standard, P3, L121-12 can be stated briefly.

Response: We have summarized this statement, line 121: “The framework includes steps for 1) data pre-processing, 2) image processing and 3) machine learning model development.”

Code availability in Figure 1 is not required.

Response: We have addressed this comment and removed the link to the code of this framework in figure 1.

The authors should remove the background definitions as this material is readily available nowadays. So, these definitions would not be much helpful to the readers.

Response: We are not sure what background definitions the reviewer is referring to. We agree that some definitions such as different types of annotations (bounding boxes, images segmentation) are available and widely known by the computer science community but there is still enough confusion about these definitions in other research communities. For example, some co-authors of this manuscript were initially themselves not aware of the difference between classification and object detection techniques. As we are trying to make this framework usable to the general audience and since this is an online only publication and we are not constrained by the word limit, we believe this information is relevant and should be kept in the manuscript. In the end, it is only one paragraph of text, which for many readers will improve the clarity.

Figure numbering is wrong "Figure 32:" and "Figure 43:"

Response: We have revised and corrected the figure numbering.

Image quality of the confusion matrix is poor

Response: We have uploaded a new figure with higher resolution.

Since the authors agree that their study is a pilot one, they should not draw solid and lengthy conclusions as given in "Lessons learned, challenges and future applications" section. Specifically, ensemble learning, data augmentation, and help from citizens should be either removed or stated briefly. Often, findings at small scale do not generalize to a large scale. So, the authors should give general findings only.

Response: We are agree that findings from small scale studies cannot be generalized to a large scale and have modified the paragraph starting on line 601 to highlight this: “Like other researchers (e.g. Lekunberri et al. 2022), we found that image augmentation through rotation and flips improved the image classification model performance (overall accuracy increased by 10%). As the dataset used in the pilot study was small and this difference in model accuracy can vary with dataset size, these results need to be interpreted with caution.”

However, most of the “lessons learned” we discuss in this section relate to our experience of applying the described framework, and focus on topics such as framework development (data preprocessing, annotations, etc..) and future applications (such as ensemble learning). The scale of the study are not as relevant for these questions (unlike model development and optimization). We believe these general findings are important, as we found that many researchers are interested in knowing the potential time and costs of developing AI models and those are the lessons learned we are sharing.

We have changed the title of this section to “Potential framework applications and challenges” to clarify this.

The authors must remove the statement that your pipelines can be extended to other research areas that require image annotation, processing and classification owing to the pilot study (P3, L134-137).

Response:  We have removed this sentence.

 

 

Reviewer 2 Report

The authors have incorporated comments in a proper way and the quality of the paper is better than the first version.

 

Author Response

Comments and Suggestions for Authors

The authors have incorporated comments in a proper way and the quality of the paper is better than the first version.

Response: Thank you for the feedback.

Round 3

Reviewer 1 Report

The authors have addressed the comments completely.