Thermal Sensor Calibration for Unmanned Aerial Systems Using an External Heated Shutter

Round 1

Reviewer 1 Report

The presented work in this paper provided a comparison between an uncooled thermal camera and a modified version of the uncooled thermal camera (an external heated shutter as a calibration source, installed on the camera). They conducted an experiment demonstrating the effect of introduced modification on the camera in indoor, simulated environments and in the case of using Unmanned Aerial Vehicles.

1. While the paper presented interesting findings, some points should be considered:
   The structure of the paper can be improved to avoid confusion.
2. It would be more clear to have a "related work" section separated from the "introduction" section.
3. Figure 3,4 should be aligned and resized. Also, it would be better if the quality of the figures improves.
4. Regarding the sections of Figure 3, they need to be fixed to align with the journal format (Also add ")" for section a).
5. Reference 3, 20 should be corrected (the format and there is an extra line break).
6. For the experiment related to the use of a drone, it mentioned that the flights were carried out at a fix 45 meters distance. Since the measurements provided by a thermal camera depends on the environment, illumination, atmosphere, and other factors as well as the distance of the thermal camera from the inspected object. So, the reliability of acquired thermal values needs to be discussed in the scope of this paper. Since the distance, itself can influence the actual measurements.
7. Also, the thermal camera configuration needs to be mentioned, does the authors used auto-ranging in the camera? if yes, the relative temperature differences in the collected frames from the drone-based survey can also be affected by this factor.
8. Since, FLIR Vue Pro R supports the video recording only in 8bit format. Do the authors used tiff 14bit image sequences (which capture 2 frames per second) or they used periodical image capturing? Also due to the frame rate and the calibration process (every 20 seconds), the author should mention how they addressed avoiding motion blurriness because it may affect the stitching process.
9. Does the external heated shutter is in addition to the internal shutter of the camera?
10. In the case of "Simulating Operational Wind Conditions", the authors should elaborate more on the provided setup. Also, does the simulated air wind suppose to simulate the air turbulence generated by the DJI M600 drone? if yes, it would be better to explain how the authors decided on the 4 m/s air wind and what was the size of propellers used on the brushless motor? Also, since the M600 has 6 very powerful rotors, does use one rotor sufficient enough?

Author Response

Please see the attachment and updated manuscript.

Thank you for you input and advise.

Kind Regards

Jacob Virtue

Author Response File: Author Response.pdf

Reviewer 2 Report

The manuscript investigates a novel calibration methodology for UAV-mounted thermal cameras by using an external heated shutter. The integration of uncooled thermal sensors on UAV is useful for several agricultural and forestry applications. The assessment of accuracies of the thermal response is a current relevant research topic. Temperature fluctuations, due to environmental parameters and the sensor itself can all affect the accuracy of the acquired thermal data. The manuscript adequately considers other adopted approaches for calibrations (e.g. internal shutter) and post-processing solutions and proposes an external heated shutter to compensate for thermal fluctuations.   

The main concepts of the paper (evaluate and intercompare the stability and accuracy of thermal imaging sensors with and without an external heated shutter that can also be deployed for in situ calibration), and the terminology used is well presented and introduced. The objectives for work are laid out clearly and convincingly in the introduction.  
The adopted approaches and methodologies for laboratory and in-field tests are detailed, and sound and the supporting visual material is useful to better understand the exposed concepts. The analysis of results is appropriate in content, length and helpful in terms of overall insight. 
Some minor suggestions will be given by the reviewer to further improve the quality of the paper:
- Please, specify if the used thermal sensor (FVPR) is a radiometric sensor in section 2.1.1. It would be also of interest to further detail why you did not assess absolute temperature in the field trials in the introduction section. 
- The authors may consider also adding a picture of the UAS and of the thermal sensors (FVPR and FVPR TCC) overall configuration. 
- please specify in section 2.2.1 which type of temperature probe sensors has been used. 
- section 2.3.3 line 274 please check the "?"
- The authors correctly specified that no absolute temperature has been evaluated, but it is suggested to better specify how the comparison between the relative values of the two sensors has been made. How were set sensor parameters such as ambient temperature, emissivity, reflected temperature, etc.? Were they the same in the two sensors? 

Finally, please double-check typos and grammar errors over the entire manuscript. 

The reviewer's opinion is to try improving the clarity of some concepts of the paper, with a minor revision before publishing it. 

Author Response

Please see the attachment and updated manuscript.

Thank you for you input and advise.

Kind Regards

Jacob Virtue

Author Response File: Author Response.pdf

Reviewer 3 Report

The manuscript is addressing the well-known issue of inconsistent temperature readings of miniature thermal cameras. It aims to demonstrate how an external heated shutter can improve the overall performance and decrease sensitivity to changing environmental conditions. Whilst the topic is timely and of interest to the scientific community, I believe further work and clarifications are needed to support the validity of the findings. The authors should also ensure they do not make unsubstantiated claims. E.g. “The temperature stabilising benefits of a heated shutter limits the need to use calibration targets” (lines 560-561) – no absolute accuracy assessment of temperature readings was actually performed in the field. Please ensure the statements/conclusions are supported by the work presented here or by others.

My main reservation is that the testing was performed with two separate cameras (albeit of the same model). It is, therefore, uncertain to what degree the improvement was only a result of utilising the ThermalCapture calibrator. Whilst the authors aim to overcome the potential issue of unaccounted distortions by only analysing few central pixels, I would still want to see evidence that the two sensors do indeed exhibit comparable behaviour under same conditions.

In their field experiment, the authors showed the distribution of standard deviations for same locations obtained with the two instruments. Unfortunately, here the question on the similarity of the two cameras comes back. Miniature thermal cameras suffer from strong vignetting effect - the patterns and magnitudes will be different for each camera. Therefore, how can one decouple effects of different vignetting patterns to the use of the ThermalCapture calibrator? One option to alleviate this issue would be to apply a vignetting correction to the ‘unmodified’ datasets. It would arguably be a fairer comparison to currently used approaches, helping justify the main conclusion that the calibrator enables “more uniform thermal imaging performance”. Another valuable addition would be a laboratory investigation of how the ThermalCapture calibrator affects vignetting in the modified camera, i.e. by comparing the responses with ThermalCapture deactivated and activated.

A number of clarifications would be needed throughout the manuscript. Crucially, information on whether the camera’s non-uniformity correction was enabled and at what intervals it was performed. Was it also 20 seconds (as with the ThermalCapture) or was it performed at longer intervals? The methodology section also mentions that camera temperatures were logged throughout the laboratory experiments. Were there any significant differences in those between the two cameras throughout the experiment? Can you display that information on Figures 1 and 2 to allow comparison against their performances?

I would also suggest making changes to the structure and text. The introduction section, in particular, would require significant attention. It is currently full of repetitions, lacks tangible structure and flow. The data and methodology section would be easier to follow if it was split into two main sections on (1) laboratory tests and (2) field tests.

Author Response

Please see the attachment and updated manuscript.

Thank you for you input and advise.

Kind Regards

Jacob Virtue

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Thank you for replies and revisions of your work.

Just few suggestions:

1. It would be better to improve the figures (especially Figure 4), also it seems Figure 4 is out of page margins.
2. There is a page break on page 4.
3. There is a linebreak in the middle of a url in reference 19

Best regards