Real-Time Registration of Unmanned Aerial Vehicle Hyperspectral Remote Sensing Images Using an Acousto-Optic Tunable Filter Spectrometer

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper is mostly devoted to the examination of methods for computer processing and correction of images obtained by an acousto-optical spectrometer. Overall, the work is well written and will certainly be of interest to readers. However, it is necessary to note a number of shortcomings regarding the examined spectrometer design and acousto-optical devices in general.

1. The text contains a number of errors and inaccuracies regarding acousto-optical devices. For example, AOTF cannot be called a novel device, since it was first proposed about 60 years ago. It is also not clear what the authors mean when they talk about the large aperture of AO filters, since by the standards of optical instruments, neither the linear nor the angular aperture of AO filters are large. 

2. The article does not contain a detailed description of the exaomined spectrometer - there is neither its optical design scheme, nor the characteristics of the spectrometer in general and the AO filter in particular.

3. In the second paragraph of section 2.1 it is stated that AO diffraction is realized in +1 and -1 diffraction orders simultaneously, which, in fact, is a special case that may be observed in a special geometry of AO interaction. 

4. The article discusses compensation for image distortion, but does not mention or consider image distortion that occurs directly due to AO interaction.

5. The selection of cited literature concerning AO interaction is absolutely unacceptable, and the list of references is incomplete. The work does not mention the main articles on spectral filtering of optical radiation, the development of wide-angle AO filters, the development and practical application of AO imaging spectrometers, and the analysis of image distortions arising from AO diffraction and optical beam propagation in the AO crystal.

Comments on the Quality of English Language

In general, the paper is written well and minor editing of English language is required.

Author Response

Response to Reviewer 1

 

1. The text contains a number of errors and inaccuracies regarding acousto-optical devices. For example, AOTF cannot be called a novel device, since it was first proposed about 60 years ago. It is also not clear what the authors mean when they talk about the large aperture of AO filters, since by the standards of optical instruments, neither the linear nor the angular aperture of AO filters are large.
2. The article does not contain a detailed description of the examined spectrometer - there is neither its optical design scheme, nor the characteristics of the spectrometer in general and the AO filter in particular.

Answer: Thank you very much for your comments and suggestions. In section 2.1 of the revised manuscript, we have added a core optical path structure diagram based on a zoom lens AOTF spectrometer and provided detailed prototype design materials. In addition, section 2.2 has been added to introduce the spectral characteristics of AOTF and some issues with the designed spectrometer.

3. In the second paragraph of section 2.1 it is stated that AO diffraction is realized in +1 and -1 diffraction orders simultaneously, which, in fact, is a special case that may be observed in a special geometry of AO interaction. 

Answer: Thank you very much for pointing out this issue. Since this information was mentioned when describing the core optical path, we have deleted this part in the revised draft.

4. The article discusses compensation for image distortion, but does not mention or consider image distortion that occurs directly due to AO interaction.

Answer: Thank you very much for your comment. In section 2.2, we explained the relationship between wavelength and diffraction angle, analyzed the reasons for image distortion caused by AOTF interactions, and provided a solution to this problem.

5. The selection of cited literature concerning AO interaction is absolutely unacceptable, and the list of references is incomplete. The work does not mention the main articles on spectral filtering of optical radiation, the development of wide-angle AO filters, the development and practical application of AO imaging spectrometers, and the analysis of image distortions arising from AO diffraction and optical beam propagation in the AO crystal.

Answer: Thank you very much for your comments and constructive suggestions. We have added information on AOTF filters and referenced studies focused on the development and practical applications of AOTF spectrometers in the revised manuscript. We have also added an analysis of image distortion caused by AOTF diffraction and beam propagation in AOTF crystals. In the Introduction, we have cited studies that have provided solutions to image distortion caused by AOTF.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This study created a novel coarse-to-fine remote sensing image registration framework based on feature and optical flow theory. It was verified through real-time registration of spectral data using an acousto-optic tunable filter (AOTF) spectral imager mounted on an unmanned aerial vehicle, utilizing hyperspectral data.

The following comments need to be amended in the manuscript.

Abstract

State objectives clearly in the abstract following the background information.

Introduction

The objectives are primarily focused on technical development and validation without addressing potential broader impacts or applications of the proposed methods. Therefore, it is useful to consider wider implications or practical applications of the research.

Line 115:  Liu et al. [20]

Line 118:  Zhou et al. [21]

Line 127:  Zeng et al. [27]

Follow Authors et al. [Reference number] rule throughout

Results and Discussion

L433 – 450: Dataset section relevant to the methods and nothing to do with results and discussion.

L614: Figure 8 is missing.

Line 579 – 581: How did you determine these thresholds? Were those determined through trials or based on literature?

Conclusions

The conclusion gives a good overview on the study's contributions, its implications for practical applications, and a roadmap for future research directions.

Comments for author File: Comments.pdf

Author Response

Response to Reviewer 2

Abstract

State objectives clearly in the abstract following the background information.

Answer: Thank you very much for your comment. In the revised manuscript, we have rewritten the abstract section, which includes clarifying the research problem and reasons underlying the problem, providing our solution, and describing the final results of the study.

Introduction

The objectives are primarily focused on technical development and validation without addressing potential broader impacts or applications of the proposed methods. Therefore, it is useful to consider wider implications or practical applications of the research.

Answer: Thank you very much for your comments and suggestions. We have added information on previously proposed methods of addressing image distortion in AOTF spectrometers in the Introduction. Moreover, we described the development and practical applications of AOTF spectrometers in recent years. Please see the revised Introduction.

Line 115: Liu et al. [20]

Line 118: Zhou et al. [21]

Line 127: Zeng et al. [27]

Follow Authors et al. [Reference number] rule throughout

Answer: Thank you very much for pointing out this issue and for highlighting the citations in the original manuscript. We have reviewed all references and made corrections throughout the manuscript.

Results and Discussion

L433 – 450: Dataset section relevant to the methods and nothing to do with results and discussion.

Answer: Thank you very much for this valuable comment. In the revised draft, we have checked and adjusted the overall structure. Section 3 has been revised to the Methodology, and Chapter 4 has been revised to the Experiments and Discussion.

L614: Figure 8 is missing.

Answer: Thank you very much for calling out this issue. We have added Figure 8 to the revised manuscript and checked all the images.

Line 579 – 581: How did you determine these thresholds? Were those determined through trials or based on literature?

Answer: Thank you very much for this question. We have comprehensively considered the registration effect and computation time and determined these thresholds through multiple experiments.

Conclusions

The conclusion gives a good overview on the study's contributions, its implications for practical applications, and a roadmap for future research directions.

Answer: Thank you very much for your positive appraisal of the Conclusion section.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

Real-time hyperspectral imaging finds enormous applications in defence, industry and public safety. This work involves the deployment of acousto-optic tunable filter spectrometer for surveillance using drones. The article is written in a very comprehensive form,

Here are my suggestions to improve the presentation of this manuscript.

 1. The length of this article is very long and should be reduced so that the reader can focus on the main findings of this work.

2. The abstract is poorly presented. Therefore, it must be properly rewritten with removal of unnecessary long and ambiguous sentences.

3. The introduction is fine, however some recent trends are missing

DOI: 10.1007/978-3-030-71711-7_1

DOI: 10.3390/rs12162659

4. The details of optics and respective parameters involved in this project should be included in section-2.1

5. The datasets are mentioned in section-4.1. Include the quantitative details of your own dataset. Since you have developed your own imaging system and processing method. Therefore, All the images and the final results acquired in this work must be separately presented. Details included in raw image acquisition and detector calibration must be included.

6. Table-1 to Table-3, How the entries are arranged in specific order?

7. Organize the data based on year of publication and mention it in the text

8. Figures 4-6 , Clearly mention that the images are taken from online source or using your spectrometer setup.

9. Figure captions must narrate what is the difference between the images as under what conditions the data have been selected? Labels are missing

10. What is the dissimilarity in Figure-4 and Figure-5. Is there anything visual outstanding feature?

11. Figure-6 caption is confusing. Mention all the critical details involved in it.

12. Figure-8 is missing. Why?

13. Section-4 is missing critical issues involved in this work. Quantitatively discuss the accuracy of this algorithm, the efficiency in processing time, frame rate, environmental effects, maximum achievable resolution

14. What is the effect of geometric distortions and detector noise ?

15. One of the main purposes of this technique is to identify the objects remotely. The system must be able to recognize the features in the input scene as an output. Clearly it is missing in this work, why?

Comments on the Quality of English Language

Grammatical corrections and rephrasing is required.

Author Response

Response to Reviewer 3

 

1. The length of this article is very long and should be reduced so that the reader can focus on the main findings of this work.

Answer: Thank you very much for your comments and suggestions. Although we attempted to shorten the length of the manuscript as much as possible, we could not shorten the text without eliminating important information. Nevertheless, we have adjustments to the overall structure of this article to improve the organization.

2. The abstract is poorly presented. Therefore, it must be properly rewritten with removal of unnecessary long and ambiguous sentences.

Answer: Thank you very much for your suggestion. Based on your suggestion, we have rewritten the abstract and removed unnecessary long and ambiguous sentences. The abstract includes a clear description of the problem, its causes, identified solutions, and final results of the study.

3. The introduction is fine, however some recent trends are missing

DOI: 10.1007/978-3-030-71711-7_1

DOI: 10.3390/rs12162659

Answer: Thank you for providing the latest research trends. We have added the literature you mentioned and relevant recent trends in the revised manuscript.

4. The details of optics and respective parameters involved in this project should be included in section-2.1

Answer: Thank you very much for your valuable feedback. In section 2.1 (line 236), we have added relevant optical details and corresponding parameters.

5. The datasets are mentioned in section-4.1. Include the quantitative details of your own dataset. Since you have developed your own imaging system and processing method. Therefore, All the images and the final results acquired in this work must be separately presented. Details included in raw image acquisition and detector calibration must be included.

Answer: Thank you very much for your comments and suggestions. The dataset in section 4.1 (line 514) utilizes existing datasets and previously collected data from various scenarios. The main purpose is to compare the processing performance of the proposed algorithm and existing algorithms through different types of data. In section 4.3, the algorithm proposed in this article was deployed on a MINI-PC with GPU, and after real-time processing, all images and final results were presented.

6. Table-1 to Table-3, How the entries are arranged in specific order?

Answer: Thank you for your question. The information in Tables 1 to 3 is not provided in a specific order but rather ordered alphabetically according to the algorithm name.

7. Organize the data based on year of publication and mention it in the text

Answer: Thank you very much for your suggestion. In the revised draft, we have organized and reproduced the data and provided corresponding explanations.

8. Figures 4-6 , Clearly mention that the images are taken from online source or using your spectrometer setup.

Answer: Thank you very much for your suggestion. During the revision process, we removed the online dataset Cars and the ground-based dataset Buildings because the Cars dataset was not captured by an AOTF spectrometer and the Buildings dataset was not captured by a drone. We have ensured that all datasets include remote sensing image data captured by an AOTF spectrometer on a drone.

9. Figure captions must narrate what is the difference between the images as under what conditions the data have been selected? Labels are missing

Answer: Thank you very much for your comments and questions. In the revised draft, we have added appropriate titles and labels and provided detailed explanations in the titles of the figures. The selection criterion for detailed maps, as shown in Figure 8 using chessboard details, included features with continuity on the ground. If the registration was correct, then the features in the detail map will be continuous. If the registration was incorrect, then the scenery on the ground will be misaligned.

10. What is the dissimilarity in Figure-4 and Figure-5. Is there anything visual outstanding feature?

Answer: Thank you very much for your comments and questions. Figure 4 shows the original images in two spectral bands (580 nm and 620 nm). Figure 5 shows the image overlay differentiation display. The left side of Figure 5 shows the overlay display of the original image without registration, and the right side of Figure 5 shows the overlay display of the images of the two spectral bands after registration. Unregistered images have a more prominent differentiated display, while registered images have a less prominent differentiated display.

11. Figure-6 caption is confusing. Mention all the critical details involved in it.

Answer: We apologize for not clearly expressing the title of Figure 6 in the original manuscript. Figure 6 visualizes the details of the image registration results using a chessboard grid. Associated modifications to the text have been made in the manuscript (lines 515 to 518).

12. Figure-8 is missing. Why?

Answer: Thank you for your comment. This image was accidentally deleted during the final draft. The figure has been added in the revised draft, and the manuscript was checked for similar issues.

13. Section-4 is missing critical issues involved in this work. Quantitatively discuss the accuracy of this algorithm, the efficiency in processing time, frame rate, environmental effects, maximum achievable resolution.

Answer: Thank you very much for your comments and suggestions. The accuracy of the algorithm and the processing time efficiency are discussed in Figure 9 and Table 5. In addition, the frame rate for shooting at a waypoint was 2 Hz (including real-time registration processing). The maximum spatial resolution (flight altitude of 100 m, focal length of 16 mm) was 3 cm/pixel.

14. What is the effect of geometric distortions and detector noise?

Answer: Thank you very much for this meaningful question. First, this paper is focused on using image registration to solve the problem of field of view differences in images from different spectral bands obtained via unmanned aerial remote sensing data acquisition using an AOTF spectrometer based on zoom lenses. Geometric distortion is also one of the reasons for the field of view differences in images with different spectral bands. For example, spectral drift caused by the characteristics of AOTF crystals can cause geometric distortion in images. Fortunately, geometric distortion can be solved through image registration. Although the noise of the detector was not studied in this paper, if the noise of the detector is too large, then the signal-to-noise ratio of the image will be relatively small. Therefore, this issue will affect the feature extraction stage of image registration. For example, the number of features may decrease, which would ultimately lead to a decrease in the accuracy of registration. The impact of geometric distortion and detector noise will be the focus of future research.

15. One of the main purposes of this technique is to identify the objects remotely. The system must be able to recognize the features in the input scene as an output. Clearly it is missing in this work, why?

Answer: Thank you very much for your question. One of the main purposes of this technology is indeed remote object recognition, which requires extracting their features. In Figure 5, the framework of the proposed registration method is presented. The coarse process stage is based on feature-based image registration. One of the algorithms proposed in this article extracts the ORB features of the image and compare it with registration algorithms that extract features, such as KAZE, AKAZE, SIFT, SURF, etc.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

See the attached file

Comments for author File: Comments.pdf

Comments on the Quality of English Language

The quality of english is generaly fine

Author Response

Response to Reviewer 4

 

The manuscript titled "Real-time Registration of Unmanned Aerial Vehicle Hyperspectral Remote Sensing Images Using an Acousto-Optic Tunable Filter Spectrometer" presents a study focused on improving the registration process of hyperspectral images captured by an Unmanned Aerial Vehicle (UAV) equipped with an Acousto-Optic Tunable Filter (AOTF) spectrometer. The authors propose a coarse-to-fine image registration framework that leverages feature and optical flow theory to address image size deformation, image drift, and platform jitter issues. The following is my comments and advices for this manuscript:

 

1. The introduction section should give readers’ a general image of background and justify the necessity and importance of this research. However, the current introduction can’t provide a clear background and didn’t outline the importance of image registration technology in various fields. The introduction section is too lengthy. As authors are working on a topic that’s not very hot, it’s advised to reorganize the introduction section.

 

Answer: Thank you very much for your comments and suggestions. We have reorganized the introduction section. First, we have provided the research background and highlighted the necessity and importance of this study. Second, the characteristics of the AOTF spectrometer itself and the problems it faces in remote sensing applications were described to further demonstrate the need to research image registration technology. Then, a review of previous work on hyperspectral remote sensing image registration was presented. Finally, the problems to be solved and the main contributions of this article were clarified.

 

2. In related works section, it could be improved by directly relating the discussed methods to the specific limitations of AOTF spectrometers mentioned earlier.

 

Answer: Thank you for this suggestion. In the Related Work section, the design and related parameters of an airborne AOTF spectrometer based on a zoom lens were further introduced. Starting from the spectral characteristics of AOTF, the specific limitations of AOTF spectrometers were explained in Section 2.2. By using an electric focusing lens, the problem of blurring in AOTF imaging can be solved, although this process may cause image size deformation in different spectral bands. However, image registration can precisely solve the problem of image size deformation. In addition, image registration can also solve the problem of image drift caused by AOTF wavelength switching and field of view differences caused by drone platform jitter. Therefore, research on image registration presented in this article is very meaningful. Please review section 2 in the manuscript.

 

3. In material and methods, it would be helpful to include a flowchart or diagram illustrating the proposed algorithm's workflow for better clarity. Moreover, the algorithms process is better in pseudocode blocks for readers, like the section 3.1 and 3.2.

 

Answer: Thank you very much for this comment. In section 3, we have provided a flowchart of the algorithm's work, which is also depicted in Figure 5. The framework is a remote sensing image registration framework based on feature and optical flow theory, from coarse to fine. The registration algorithm framework is divided into two stages: a coarse registration stage based on feature methods and a fine registration stage based on optical flow theory. Moreover, a specific algorithm was provided to describe how to turn the actual program into pseudocode.

 

4. In results and discussion, including visual examples of the registration results (e.g., before and after images) would strengthen the presentation of results. The discussion should delve deeper into the implications of the results, particularly how the proposed method compares to existing solutions in terms of practical applications and limitations.

Answer: In terms of displaying the registration results, the presentation of the effects has been strengthened in the revised manuscript. Figure 6 shows the selected dataset display; Figure 7 displays the registration results (the left image shows the overlay of unregistered images, and the right image shows the overlay of registered images); Figure 8 shows the detailed information after registration using different algorithms using a chessboard pattern; and Figure 9 compares the results of real-time onboard processing. Data cubes 1 to 5 are unregistered raw data cubes, and data cubes 6 to 10 are registered using the method proposed in this article. We compared the registration results with some existing methods, as shown in Figure 10 and Table 5. In terms of practical application and limitations, verification has been completed on five waypoints in one flight experiment. However, the registration spectrum to be completed includes 120 bands, which will result in a longer processing time at a waypoint. As mentioned in the previous introduction, the AOTF spectral scanning imaging spectrometer has a unique advantage in selecting spectral bands. In the later stage, selecting the bands of interest and reducing the number of bands for data processing will facilitate the completion of certain hyperspectral real-time remote sensing processing applications for a specific application.

 

5. Ensure that all figures and tables are clear and understandable, with appropriate captions and labels. Verify that the resolution of images in the manuscript is high enough for publication.

 

Answer: Thank you very much for your suggestion. In the revised draft, we have checked all the icons and added appropriate titles and labels. In addition, we have optimized all the images to ensure that the resolution is sufficient.

 

6. The reproducibility of the proposed methods is one of my major concern. The manuscript should include information on how the data was collected and processed, allowing for the reproducibility of the study. It would be beneficial to mention if the code or algorithm will be made available for public use.

 

Answer: This article proposes a new coarse-to-fine remote sensing image registration framework based on feature and optical flow theory, and implements a code-based method. The effectiveness of the algorithm has been demonstrated through experiments. Based on your suggestion, during the finalization stage of the manuscript, the core part of the code will be uploaded as an attachment on GitHub.

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

Although the manuscript is in much better form in its present condition, I would suggest minor grammatical corrections and addition of technical information  before its publication.

1.  abstract: line 16-18 is too long sentence. It should be disintegrated into two parts. Correction required in Line 22 (Root mean squared  error is reduced)

2. abstract: line 24 requires correction, run a spell check for all the words.

3.  line 258 and Fig.2 caption is almost similar?

4. Recheck all the unnecessary words starting with a capital letter. one such example is line 280 ( the word Using).

5. Table-1 the frequency stability should be presented in frequency units

6. Mention the full form of TeO2 in line 283. All abbreviations must be explicitly shown in full form in the text

7. Fig. 3 why is it worth writing crystal outside diffraction angle along yaxis  . label it with appropriate name

 8. Fig.8 caption should be in correct form. Mention the major point of interest in the the figure

9. Fig. 9 (cube 1-5 and cue 6-10)  any extraordinary feature that make difference, must be highlighted with a marker  

Comments on the Quality of English Language

Grammatical Correction are still required throughout the manuscript. Repetition of text is abundant.  Please check the similarity report for this manuscript.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

The authors have made considerable improvement. I don't have further questions. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf