Predicting Dog Emotions Based on Posture Analysis Using DeepLabCut

Round 1

Reviewer 1 Report

This paper describes an emotion recognition system for dogs automatically identifying four major canine emotions. The paper has some methodological and technical presentation issues, which must be resolved before the paper could be considered for publication.

Comments:

1. The article needs a better structural organization. The parts of the paper which explain the methodology (including mapping emotions to stimuli and landmark point extraction) should be moved from the Introduction section to the Materials and Methods section.
2. The discussion on canine emotion recognition in the Introduction section is not comprehensive. Beyond dog posture, dog barking sounds previously have been used for dog emotion recognition. Perhaps, the authors would be interested to check and discuss “Recognition of emotional vocalizations of canine” published in Acta Acustica united with Acustica. The authors also should check the recent paper of prof. Raman et al., „Markerless dog pose recognition in the wild using resnet deep learning model“, which also used DeepLabCut.
3. Motivate the selection of languages and search websites described in Lines 128-131.
4. Add a workflow diagram explaining the proposed approach.
5. Figure 1: explain the legend. How the points assigned to both datasets have a different (third) color.
6. The quality of mathematical equations should be increased. Use professional math typing.
7. Figures 2-8: explain (in the caption or legend of the figure) the mathematical notations used in the figures.
8. Why do you stop training at 240 epochs? Can you motivate your stoppage criterion? How do you prevent/avoid model overfitting during training?
9. Figure 9: add a colorbar to explain the meaning of colored cells.
10. Figure 10 does not provide any explanatory meaning, because there are no decision probabilities (or weights) given.
11. In the case of multi-part figures (such as Figure) present the description of each subfigure in the caption of the figure.
12. It is not clear to me, how a neural network can deal with missing data.
13. The accuracy of the proposed model (see Table 6) is low. Does it have any practical value in a real-world setting?

Author Response

see attached rebuttal letter

Author Response File: Author Response.pdf

Reviewer 2 Report

This manuscript tested and presented a nice method used in predicting dog emotions based on posture analysis using DeepLabCut. In general, this is a novel research compared with existing models on dog emotion recognition, and the process and results are well organized and clearly presented, I did not see obvious issues, thereful I am happy to recommond this ms to be considered by Future Internet, one thing I did not see is the condition about the availability about the method, is the software available as soon as the publication is online? How could this method be used? I hope the author provides soem statement about this.

Author Response

see attached rebuttal letter

Author Response File: Author Response.pdf

Reviewer 3 Report

The paper addresses an important aspect of understanding dog behaviour and is generally well written. However, it is very difficult to assess because of the lack of detail on how the modelling was done and the methods in general are not detailed enough to allow assessment or reproduction. While I feel this can be remedied, it’s a significant issue with the paper.

I liked the breadth of introduction though I’d suggest it could be restructured as the first two paragraphs are perhaps somewhat too generalised for what is a specific question about emotions.

As I stated above, I have strong concerns about the methods and the dataset of images used.

There are no details on the breeds, types, or numbers of dogs used and this is a significant challenge when it comes to assessing the manuscript. Breeds with floppy ears will simply not be capable of raising their ears in the way that prick eared breeds can – thus Labradors may have very different “raised” ear positions to German shepherds.

I would like a more thorough explanation of how the videos were assessed.  While I’m happy with the search terms, it’s not clear that all dogs will show fear vs anger in all situations and I would like to know who coded the videos initially.  You are using this to validate the results of your code, but I am unclear on how the original assessments were made or what was used.

It would also be good if the current mechanisms of marking dog emotion were better explained, e.g. issues with the dog’s position in relationship to the camera and correctly identifying heads.

Please make it clearer which program the statistical models were run in and add more detailed descriptions of how they were performed. The logistic regression is first mentioned in results and I have no idea how you performed it as it was not mentioned in the methods.  Ditto the SVM.

The discussion is far too short and doesn’t really contextualise the results in regards to what is known from other automated emotion detectors.

Overall, the paper needs considerably more detail, but the work itself seems solid and I believe all of the issues above can be remedied by a revised draft with more extensive methods, results, and discussion sections. It's a very valuable and interesting tool with great potential for the future so I believe it it is worth revising the presentation.

Minor issues by line number:

13 - Citations required in the first paragraph.  A review here would be fine.

33 – While an admirable goal, I’m not sure how this is going to prevent attacks unless it’s built into an early warning system that seems unlikely.

90 – I’d suggest it’s less the puppy is not emotionally mature so much as their physical markers may not be mature, e.g. ears that are not yet erect and tails that are not under full control.

93 – Is there a reference for the reduction in emotional expression in cropped & docked dogs? I am happy with the exclusion of them and puppies, and think it’s the right choice, I’m just curious as to whether the statement has been investigated previous to this.

180 – How?

Author Response

see attached rebuttal letter

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

All revisions have been done correctly. The quality has been improved. The manuscript can be accepted for publicatiion.

Author Response

please see attached rebuttal letter

Author Response File: Author Response.pdf

Reviewer 3 Report

Thanks to the authors for their careful and considered response to all reviewers' comments in the rebuttal letter. They answered my concerns but some of the answers do not appear to have been added to the text. Please revise this before publication, e.g.:

1. Add the data on how many individual dogs and breeds were included in the final dog emotion data set, even as ESM. While I understand more than 120 breeds are present in the training dataset, but there is no information on the dog emotion data set.

2. Ditto the stats program that was used.

3. Add the detail of how the images were assessed. You explain this well in the rebuttal letter but did not add it to the paper.

A simple sentence is enough to answer each of these and would help the reader.

I love the diagrams and new figures. These are both charming and informative.

The paper is otherwise fine and just needs these final details to be ready for publication.

Author Response

please see attached rebuttal letter

Author Response File: Author Response.pdf