A Review of Image Inpainting Methods Based on Deep Learning

Round 1

Reviewer 1 Report

Dear authors,

I wish to extend my appreciation for the effort you've invested in crafting a comprehensive manuscript on image Inpainting. Your adept synthesis of existing literature and its logical organisation are commendable. However, I've identified a significant issue that appears to be lacking in your manuscript – the voice of the authors. The genuine interpretation of the Inpainting technique's development, challenges, and potential improvements is noticeably absent. A comprehensive review not only summarises existing literature but also provides valuable author insights into the domain. Additionally, your manuscript lacks an exploration of the ethical concerns associated with image Inpainting techniques.

Please find a more detailed review attached to the annotated manuscript.

Thanks and all the best. 

Comments for author File: Comments.pdf

The overall quality is satisfactory. There are instances of incomplete sentences and some redundancy is also evident. For a more details, please refer to the annotated manuscript.

Author Response

Dear Reviewer,

 

First and foremost, I would like to express my profound gratitude for your comprehensive review of our manuscript. Your invaluable feedback has been instrumental in enhancing our research work.

 

Regarding the author's voice:

 

You highlighted the absence of the author's voice in our manuscript, particularly the genuine interpretation of the development, challenges, and potential improvements in image inpainting techniques. To delve deeper into the evolution, challenges, and potential advancements of image inpainting, we have thoroughly rewritten the "Outlook and Challenges" section. We have detailed the various developmental phases image inpainting techniques have undergone since their inception, the challenges they currently face, and the directions for future improvements.

 

On ethical concerns:

 

To address the ethical implications of image inpainting techniques, we have added a new section titled "Ethical Considerations of Image Inpainting Techniques." In this section, we have explored in detail the series of ethical dilemmas brought about by the widespread application of this technology and the recommendations proposed to ensure its ethical deployment.

 

Regarding the attached detailed review:

 

We have carefully considered the other detailed modifications you mentioned in the attached review and have made revisions in line with your suggestions. This includes refining descriptions in the text, providing more references to support claims, and offering a more accurate description of the model's inherent assumptions, among others.

 

For the convenience of the reviewer, we have highlighted all the revised content in the paper in red. The updated version, with the red-marked sections, has been included in the attached PDF file

 

Thank you once again for your invaluable feedback and suggestions. We hope that this revision meets your expectations.

 

Warmest regards,

 

Wei Chen

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments - 

1) Provide a comparative analysis of different state of the art techniques inpainting techniques. Conduct experiments to investigate and analysis different bench mark techniques .   2) Since the authors are more focused on deep learning techniques for inpainting but brief categorization of traditional inpainting techniques must be depicted as these techniques are performing well .    3) The representation and structure of the entire article is scattered in the current form . Kindly refine the content.    4) Explain the basic inpainting process in mathematical form .    Similarly many aspects are missing from the article . 

Comments - 

1) Provide a comparative analysis of different state of the art techniques inpainting techniques. Conduct experiments to investigate and analysis different bench mark techniques .   2) Since the authors are more focused on deep learning techniques for inpainting but brief categorization of traditional inpainting techniques must be depicted as these techniques are performing well .    3) The representation and structure of the entire article is scattered in the current form . Kindly refine the content.    4) Explain the basic inpainting process in mathematical form .    Similarly many aspects are missing from the article . 

Author Response

Dear Reviewer,

 

First and foremost, I would like to express my gratitude for your meticulous review of our manuscript and for providing invaluable feedback.

 

1 Regarding the comparative analysis of the latest techniques:

We have incorporated a new section in the article titled "Comparative Analysis of Recent Techniques." In this section, we have detailed a comparison of 19 cutting-edge Inpainting techniques from post-2022. For each technique, we have briefly described its main features, advantages, and limitations, aiming to offer readers a comprehensive technical comparison.

 

2 On the description of traditional Inpainting techniques:

We have introduced a new section at the beginning of the review titled "Traditional Inpainting Techniques." Here, we provide a brief overview of five conventional Inpainting methods, such as sample-based methods and those based on partial differential equations, showcasing their performance and significance.

 

3 Concerning the organization and structure of the article:

We have reorganized the sequence of sections based on the theme and logical flow of the article. We ensured that each section has clear headings and subheadings to assist readers in better understanding the structure and content of the manuscript.

 

4 On explaining the basic restoration process in mathematical terms:

We have added a new section in the manuscript that specifically elucidates the fundamental restoration process using mathematical formulations, hoping to provide readers with a deeper understanding.

 

For the convenience of the reviewer, we have highlighted all the revised content in the paper in red. The updated version, with the red-marked sections, has been included in the attached PDF file

 

Once again, thank you for your invaluable suggestions and insights. We hope that the revisions meet your expectations, and we look forward to any further feedback you might have.

 

Warm regards,

 

Wei Chen

Author Response File: Author Response.pdf

Reviewer 3 Report

Image inpainting has been a long-standing challenge in the realm of image processing, with individuals throughout different periods attempting to tackle it through various means. Traditional algorithms for image inpainting have proven effective in addressing minor issues like scratches and wear. However, recent advancements in deep learning within the field of computer vision, along with the availability of abundant computational resources, have brought to the forefront the advantages of deep learning-based methods in areas such as semantic feature extraction, image transformation, and image generation. Consequently, deep learning-based image inpainting algorithms have now taken center stage in this domain.

 

This article aims to provide a comprehensive examination of some classic methods for image inpainting that rely on deep learning techniques. It attempts to organize these methods into categories based on optimizations related to component selection, network structure design, and training methodology, providing insights into the strengths and weaknesses of each approach. Additionally, the article briefly discusses commonly employed datasets and evaluation criteria for image inpainting.

 

The objectives of this article are intriguing, and the authors have adequately supplied the necessary materials. Nevertheless, there is room for improvement in terms of readability before the manuscript can be deemed suitable for publication. Below, you will find major and specific comments.

 

 

Major comments:

 

(1) Following the Introduction, it would be beneficial to present the sections on Datasets and Evaluation Metrics ahead of the review of the methods. This adjustment is suggested because Table 1 on page 6 has already conducted a performance comparison using the evaluation metrics.

 

(2) In the final section titled "Outlook and Challenges," it is advisable to begin with a summary of the methods mentioned earlier. Given that the article categorizes the practical applications of contemporary image inpainting into three primary scenarios – object removal, general image restoration, and facial inpainting – a summary of the strengths and weaknesses of these methods in relation to these scenarios should be provided.

 

(3) We recommend considering the creation of a table that defines important concepts such as CNN, attention, transformer, convolution, gated convolution, and so forth. This would serve as a valuable reference for readers seeking clarity on these key terms.

 

 

Specific comments:

 

(1)  In tables, it's best to avoid repeating messages to maintain clarity and conciseness. For example, you can avoid repetition by providing a note once that explains the meaning of arrows (↑ and ↓) next to metrics like PSNR, SSIM, FID, and L1. This way, readers can refer to the note for clarification without the need for redundant explanations.

 

(2) In the tables, it would be beneficial to explain why the names of some datasets were not specified.

 

(3) In Figure 2, some words are split into two lines (e.g., "based"). Please check and ensure proper formatting.

 

(4) Following "in Figure 5", descriptions such as "Here, X..." should be provided immediately, rather than being separated by Figure 5.

 

(5) In the captions of Figure 6, please specify which layer is the attention layer.

 

(6) Equation (5) does not appear to resemble cosine similarity. Please review the equation for accuracy.

 

(7) Please include a mathematical expression for the softmax function in Equation (6).

Moderate editing of English language required. 

Author Response

Dear Reviewer,

 

First and foremost, we would like to express our sincere gratitude for your detailed review and invaluable suggestions on our manuscript. Herein, we address each of your comments:

 

1 Regarding the section on datasets and evaluation metrics: As per your recommendation, we have introduced the datasets and evaluation metrics immediately after the introduction. This indeed provides clarity and coherence, especially in relation to Table 1 on page 6.

 

2 On the "Outlook and Challenges" section: We have now summarized the methods discussed earlier in this section and, in line with the three main scenarios of object removal, general image restoration, and facial repair, provided a summary of the advantages and disadvantages associated with the respective methods.

 

3 On creating a table defining key concepts: We have incorporated a table that outlines key concepts such as CNN, attention, transformer, convolution, and gated convolution for the readers' reference.

 

4 On repetitive information in the table: We have eliminated repetitive information in the table and provided a singular note explaining the significance of arrows (↑ and ↓) next to metrics like PSNR, SSIM, FID, and L1.

 

5 On unspecified dataset names in the table: We have clarified in the table's footnote the reason for certain dataset names not being specified.

 

6 On the formatting issue in Figure 2: We have reviewed and rectified the formatting in Figure 2 to ensure proper presentation.

 

7 On the description post Figure 5: As you suggested, we have moved the description to precede Figure 5 for better continuity.

 

8 On the title of Figure 6: We have specified which layer represents the attention mechanism in the title of Figure 6.

 

9 On the accuracy of Equation (5): We have revisited Equation (5) and ensured its accuracy.

 

10 On including the mathematical expression for the softmax function in Equation (6): We have now incorporated the mathematical expression for the softmax function in Equation (6).

 

For the convenience of the reviewer, we have highlighted all the revised content in the paper in red. The updated version, with the red-marked sections, has been included in the attached PDF file

 

 

Once again, we deeply appreciate your insightful comments and suggestions, which have been instrumental in enhancing our manuscript.

 

Warm regards,

 

Wei Chen

Author Response File: Author Response.pdf

Reviewer 4 Report

Following comments must be addressed in the revised version:

1. Main heading of introduction is missing after abstract. I suggest authors to correct it.

2.  Section 1 is stretched unnecessarily and its stretching makes it to loose importance. I suggest authors to split introduction into two sections with second section starting from Image Inpainting Classification. In your current ways, it is difficult to track sub-sections.

3. Section 2 covers applications of Image Inpainting. I suggest authors to add one table where you can help new readers to understand these applications supported by possible deep learning approaches. You can categorize applications based on methods used by the state-of-the-art.  

Author Response

Dear Reviewer,

 

Thank you for your detailed feedback and constructive suggestions on our manuscript. We highly value your insights and have taken steps to address each of the points you raised.

 

1 Missing Introduction Heading:

We have rectified the oversight by adding a primary heading titled "Introduction" immediately after the abstract, ensuring a clear demarcation and flow for the readers.

 

2 Lengthy Section 1:

Acknowledging your feedback on the extended nature of Section 1, we have restructured the content. The introduction has been divided into two parts, with the second part commencing from the classification of image inpainting. This restructuring aims to make the content more digestible and to provide a clearer roadmap for the readers.

 

3 Applications of Image Inpainting：

As per your suggestion, we have incorporated a table in this section to provide a concise overview of the applications of image inpainting, supported by potential deep learning methods. We believe this table will offer readers, especially those new to the field, a quick snapshot of the applications and the corresponding deep learning techniques.

 

For the convenience of the reviewer, we have highlighted all the revised content in the paper in red. The updated version, with the red-marked sections, has been included in the attached PDF file

 

We hope that these revisions address your concerns and enhance the overall quality and clarity of the manuscript. We are grateful for your time and expertise in reviewing our work and look forward to any further feedback you might have.

 

Warm regards,

 

Weichen

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

I have gone through the revised version of the manuscript .

It is suggested to authors to kindly experiment, investigate and provide analysis of  the various state  of the art techniques. Provide visual as well as objective metrics computations.

The English of the manuscript is still weak in a few instances . Kindly consult a native English speaker.

I have gone through the revised version of the manuscript .

It is suggested to authors to kindly experiment, investigate and provide analysis of  the various state  of the art techniques. Provide visual as well as objective metrics computations.

The English of the manuscript is still weak in a few instances . Kindly consult a native English speaker.

Author Response

Dear Editor and Reviewers,

First and foremost, we would like to express our gratitude for the valuable comments and suggestions you provided on our manuscript.

Regarding your feedback on the quality of the English language, we have consulted a native English speaker to polish and improve the language of the article. In the revised version, all modifications and changes have been highlighted in yellow for your convenience. The content highlighted in red pertains to our first major revision.

We made the following modifications:

1. "Traditional image Inpainting algorithms have effectively repaired minor damages such as scratches and wear." was revised to "Traditional image Inpainting algorithms have the ability to repair minor damages such as scratches and wear." We also removed the word "However".
2. "A comparison is also made based on commonly used datasets and evaluation metrics in image Inpainting." was changed to "A comparison is also made based on public datasets and evaluation metrics in image Inpainting."
3. "During our image processing, factors like environment, noise, shooting conditions, and network communications often result in image blurring and loss." was revised to "Image Inpainting can be utilized to fill in and repair the missing and damaged images, and to remove and replace unwanted objects in the images awaiting processing."
4. In the sentence "With the evolution of computer digital imaging technology, some early traditional algorithms effectively repaired minor damages like scratches using methods based on partial differential equations [1], sample-based image Inpainting models [2], variational Inpainting based on geometric image models [3], texture synthesis [4], and data-driven [5] methods.", the phrase "using methods based on" was removed.
5. "Image Inpainting is one of the hot research directions in deep learning" was changed to "Image Inpainting is one of the prominent research directions in deep learning".
6. "Deep learning-based image inpainting techniques, as discussed in [9,56,71,89], take the image to be repaired as input" was adjusted to "Deep learning-based image inpainting techniques take the image to be repaired as input\cite{ref9,ref56,ref71,ref89}."
7. In the statement "By "trained model", we refer to a neural network model that has been previously trained on a large dataset to understand and generate image patterns.", the word "we" was removed and the subject was changed.

In response to your suggestion to "kindly experiment, investigate, and provide analysis of the various state-of-the-art techniques, and provide visual as well as objective metrics computations", we would like to elaborate:

Table 2: Presents quantitative evaluations of convolutional operation improvement methods on commonly used datasets, including metrics such as PSNR, SSIM, FID, and L1. The datasets include Paris Street View and Places2.

Table 3: Showcases quantitative evaluations of image inpainting methods based on attention mechanisms on standard datasets. These methods obtain information from background areas distant from defects and propagate it to the defect areas.

Table 4: "Quantitative Evaluations of Transformer-Based Image Inpainting Methods" provides a quantitative assessment of Transformer-Based methods.

Table 5: Offers quantitative evaluations of multi-stage image inpainting algorithms on standard datasets, including Places2, Celeb A, Paris Street View, and Cityscapes. These methods primarily focus on edge structure prediction.

Table 6: Displays quantitative evaluations of boundary-center inpainting methods on standard datasets, with metrics such as PSNR, SSIM, FID, and L1. The datasets include Places2 and Paris Street View.

Table 7: Describes research methodologies of single-stage image inpainting networks and their quantitative evaluations on standard datasets, including Places2, Paris Street View, and Celeb A.

All these tables provide a detailed analysis and comparison of the methods mentioned in the review article, considering datasets, PSNR, SSIM, FID, and L1 among other metrics.

We hope this information addresses your concerns, and we once again appreciate the invaluable feedback you provided for our paper.

Thank you again for your review, and we look forward to your further feedback.

Sincerely, Wei Chen

Author Response File: Author Response.pdf

Reviewer 4 Report

All of my comments are addressed in the revised version. I am satisfied with the revision.

Author Response

Dear Reviewer,

Thank you very much for your positive feedback on our revised manuscript. We are pleased to have met your expectations and addressed all the comments you previously raised.

Your expert insights have been invaluable to our research, and we are honored to have your support. We will continue to strive to ensure the quality and integrity of our paper.

Thank you once again for your time and effort.

Best regards,

Wei Chen