An Artificial Visual System for Motion Direction Detection Based on the Hassenstein–Reichardt Correlator Model
Round 1
Reviewer 1 Report
Review：
(1) In your paper, you cited only ten papers published after 2019, and most of these can not represent the top methods in your field, you need some new references.
(2) I checked another paper you published in ICCIA2021 called "The Mechanism of Motion Direction Detection Based on Hassenstein-Reichardt Model", some details in these two papers are similar, you must point out the difference between these two papers in detail.
(3) In the experiment section, it is necessary to make the comparision with another state-of-art works.
(4) Please check the name of Table1, I haven't see the result of time-considered CNN in this table.
(5) In your last experiment, it is best to introduce another mature structure from published reference for comparision, otherwise you cannot verify the proposed CNN is advanced and validity.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The authors argued to have developed a model deriving the motion direction of a moving object on the retina. They compared their implementation with a time-considered convolution neural network and highlighted that their model outperformed the neural network. However, the approach lacks fundamental support and justifications at multiple levels.
CNN comparison.
The authors have used a time-considered CNN without justifying its structure or highlighting when and by whom such a network has been used. Finding the direction of motion is similar to finding orientation from an array of elementary motion detectors, and perhaps a network suitable for detecting orientation in images might be suitable. For example, the work of Daniel Saez exposes such a network (https://github.com/d4nst/RotNet). Without grounding the use of CNN in previous work, it sounds like the authors are building a benchmark, perhaps not optimal and to the standard of the literature, and then test the AVS system.
Naturalistic scenario
The fly's motion system has been heavily studied in a variety of conditions and can serve as an inspiration for developing systems. However, the flies use their computation to navigate in their environments ([Kern 2012, Bertrand 2015, Lecoeur 2019]). They have to cope with naturalistic situations where the background in the images is not encoded by 0 and the objects in front by 1. How does the model behave in realistic situations, for example, exposed by Schwegmann [Schwegmann 2012]. The review of Lindemann may provide information about the effect of texture in naturalistic environments [Lindemann 2012].
Comparison to other systems
Indeed the fly's visual system has served as an inspiration for various systems. However, I feel many references are missing in this context. For example, Serres and Ruffier works is one of the pillars of motion detection and robotic implementation [e.g. Serres 2012]. Leaving out the classical architecture of frame-based calculation (which is not justified for biological systems because there is no evidence for a clock at the several Ghz in such animals), comparisons with the neuromorphic literature should be considered (e.g. Milde 2018)
Comparison with self motion estimation
When the authors refer to global motion, they seem to refer to an object's motion. They considered the camera static and objects moving in front of the camera. One could consider the world static and the agent or animal moving through the environment. The global motion is then the self-motion. It will be interesting to discuss how the network compares to the algorithm and hypothesis for self motion estimation (Krapp 1996, Krapp 2000).
(1) Kern, R.; Boeddeker, N.; Dittmar, L.; Egelhaaf, M. Blowfly Flight Characteristics Are Shaped by Environmental Features and Controlled by Optic Flow Information. The Journal of experimental biology 2012, 215 (Pt 14), 2501–2514. https://doi.org/10.1242/jeb.061713.
(2) Bertrand, O. J. N.; Lindemann, J. P.; Egelhaaf, M. A Bio-Inspired Collision Avoidance Model Based on Spatial Information Derived from Motion Detectors Leads to Common Routes. PLOS Computational Biology 2015, 11 (11), e1004339. https://doi.org/10.1371/journal.pcbi.1004339.
(3) Lecoeur, J.; Dacke, M.; Floreano, D.; Baird, E. The Role of Optic Flow Pooling in Insect Flight Control in Cluttered Environments. Scientific Reports 2019, 9 (1), 7707. https://doi.org/10.1038/s41598-019-44187-2.
(4) Schwegmann, A.; Lindemann, J. P.; Egelhaaf, M. Temporal Statistics of Natural Image Sequences Generated by Movements with Insect Flight Characteristics. PLoS ONE 2014, 9 (10), e110386. https://doi.org/10.1371/journal.pone.0110386.
(5) Martin Egelhaaf; Jens Peter Lindemann. Texture Dependance of Motion Sensing and Free Flight Behavior in Blowflies. Front. Behav. Neurosci., 2013 | https://doi.org/10.3389/fnbeh.2012.00092
(6) Serres, J. R.; Ruffier, F. Optic Flow-Based Collision-Free Strategies: From Insects to Robots. Arthropod Structure & Development 2017. https://doi.org/10.1016/j.asd.2017.06.003.
(7) Milde, M. B.; Bertrand, O. J. N.; Ramachandran, H.; Egelhaaf, M.; Chicca, E. Spiking Elementary Motion Detector in Neuromorphic Systems. Neural Computation 2018, 30 (9), 2384–2417. https://doi.org/10.1162/neco_a_01112.
(8) Krapp, H. G.; Hengstenberg, R. Estimation of Self-Motion by Optic Flow Processing in Single Visual Interneurons. Nature 1996, 384 (6608), 463–466. https://doi.org/10.1038/384463a0.
(9) Krapp, H. G. Neuronal Matched Filters for Optic Flow Processing in Flying Insects,
International Review of Neurobiology, 2000, https://doi.org/10.1016/S0074-7742(08)60739-4.
Specific comments:
- L31-32 references 4-6 sounds arbitrary for this type of mention.
- L46-47 1989 review is good but maybe a more thorough review is: Beauchemin, S. S.; Barron, J. L. The Computation of Optical Flow. ACM Computing Surveys 2011, 27 (3), 21–21. https://doi.org/10.1201/b10586-20.
- L53 “it is evidence to believe” phrasing sounds odd
- L57: What is meant by global motion direction detection?
- L102: What is meant by excitatory scheme?
- L139-141: How does it relates to the article: Borst, A.; Haag, J.; Mauss, A. S. How Fly Neurons Compute the Direction of Visual Motion. Journal of Comparative Physiology A: Neuroethology, Sensory, Neural, and Behavioral Physiology. Springer Verlag November 2019, pp 1–16. https://doi.org/10.1007/s00359-019-01375-9.
- L279: CNN are also inspired by biological mechanisms. They have been inspired by the V1 area in mammals.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
Based on my review,
- Please explain Equation 2-8 using diagrams to make the reader understand the mathematical functions.
- same question 1 applied for Equation 9-16.
- Any reasons table 2 achieved 100%? Provide some discussion.
- The authors need to explain more details on Figure 4 to relate the proposed method.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
Dear authors,
I am extremely disappointed by your revised manuscript. You have replied to my major comments only in response to the reviewer and not in the text. Your manuscript is misleading the readership and the scientific community at several levels. You justify in the answer that CNN is a suitable comparison because they are excellent for image recognition. Why image motion pertains to the field of image recognition?
Indeed the fly visual elementary motion detector seems to take an achromatic input, but such input does not mean 0 for the background and 1 for the foreground. Likewise, high contrast does not mean 0 for the background and 1 for the foreground. Please at least apply your neural network to real dataset that could find application in the real world!
The approach lacks a meaningful benchmark, lacks real case scenario, lacks justification of why 0 and 1 and misunderstand high contrast images vs foreground/background segmentation, does not compare to the existing literature. At least you should apply to real data set, to see whether your approach could be applied to real world situations. You so far tested the model with artificial data, that you created yourselves.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 3
Reviewer 2 Report
Dear Authors,
Thank you for revising the manuscript. When reading your answer, it became more apparent that two scopes are intertwined in the article. On the one hand, the article proposes an algorithm to detect the direction of motion on a camera, i.e. a technical or applied scope. On the other, the article proposes this algorithm as a potential explanation for detecting the direction of motion in the fly visual system, i.e. a biological scope. Both scopes are valid but require different elements to generate application or be considered by biologists as a model (i.e. a quantitative hypothesis) to be tested.
Regarding the first scope, proposing a novel algorithm requires comparing it with established benchmarks. The CNN does not convince me as a benchmark, but since the work you are referring to has not been published, I am afraid we cannot resolve this issue. Although computationally expensive, the Lukas Kannade algorithm might be a suitable benchmark. In addition to benchmarks, an algorithm used in an application needs to work in real-world situations, i.e. away from the 0 and 1 coding of foreground and background scenarios. I have suggested using natural images because datasets of such images exist. Thus the cost to apply an algorithm and the benchmark on an existing set is not tremendous. Finally, an algorithm also benefits from being cheap computationally and with low power consumption to find applications. Therefore, I suggested considering the literature about neuromorphic engineering, which moves away from a frame by frame processing but uses computation based on spike (or event). Several algorithms have notably been proposed to detect motion from event-based cameras.
Regarding the second scope, the use of benchmarks is perhaps more a piece of additional information than a need, except when the benchmark is one of the current or potential explanations to explain the underlying mechanisms of the biological system. CNNs don't have strong support in the insect literature, mostly because they tend to contain a lot of neurons for 'narrow' purposes relative to an insect brain. An algorithm proposed to explain the underlying mechanisms of global motion direction is a hypothesis for biologists. It shares similarities to hypothesis testing, but they challenge the proposed model instead of challenging the null hypothesis. The biologists need to be able to test, verify, and falsify the proposed algorithm. If the algorithm is not falsifiable, it is no longer a hypothesis but a tautological explanation of the underlying mechanisms. In your article, you show the impressive capabilities of your algorithm to detect the direction of motion. Still, you don't refer to when the algorithm might fail, under which conditions one could test it, etc. An algorithm designed in the lab with artificial stimuli often shows several limitations in real-world situations. In contrast to an applied system, animals might not require a 100% accurate system to detect the global direction of motion because other mechanisms might correct potential mistakes or because the situations triggering failures rarely occur in nature. The limit of the proposed algorithms in real-world situations thus provides key information on designing behavioural or neurophysiological experiments for biologists. Second, proposing an algorithm for abscure neural pathways and for behavioural tasks that are also unclear makes it difficult for biologists to study the algorithm within their systems. I, therefore, suggested making parallel to a much better-studied system, the estimation of self motions.
The minor revisions of your manuscript do not fulfil the elements for either of the scopes, but the manuscript remains between two chairs. Without these elements, the algorithm may fail to impact the scientific community, miss the potential to be used as a potential algorithm for applications, or be considered a hypothesis for biological systems.
Minor point:
In your answer to justify using 0 and 1 for background and foreground, respectively, you refer to this approach having been used for over 60 years. First, this is not entirely correct because other techniques without tethered flies have also been used to study the motion pathway of flies (see Schnell 2010, 2014, for example). Second, why did biologists have studied such a paradigm? They used such a paradigm because they had to simplify the system to make sense of it. The simplification of the input was perhaps the only chance and only way to study the optomotor responses of tethered flies. Such argumentation is not valid for a technical system.
Schnell B, Joesch M, Forstner F, Raghu SV, Otsuna H, Ito K, Borst A, Reiff DF (2010) Processing of horizontal optic flow in three visual interneurons of the Drosophila brain. J Neurophysiol 103:1646–1657
Schnell, B.; Weir, P. T.; Roth, E.; Fairhall, A. L.; Dickinson, M. H. (2014) Cellular Mechanisms for Integral Feedback in Visually Guided Behavior. Proceedings of the National Academy of Sciences, 111 (15), 5700–5705.
Author Response
Please see the attachment
Author Response File: Author Response.pdf
