Applications of Virtual Data in Subsea Inspections

Abstract

This paper investigates the role that virtual environments can play in assisting engineers and divers when performing subsea inspections. We outline the current state of research and technology that is relevant to the development of effective virtual environments. Three case studies are presented demonstrating how the inspection process can be enhanced through the use of virtual data. The first case study looks at how immersive virtual underwater scenes can be created to help divers and inspectors plan and implement real-world inspections. The second case study shows an example where deep learning-based computer vision methods are trained on datasets comprised of instances of virtual damage, specifically instances of barnacle fouling on the surface of a ship hull. The trained deep models are then applied to detect real-world instances of biofouling with promising results. The final case study shows how image-based damage detection methods can be calibrated using virtual images of damage captured under various simulated levels of underwater visibility. The work emphasizes the value of virtual data in creating a more efficient, safe and informed underwater inspection campaign for a wide range of built infrastructure, potentially leading to better monitoring, inspection and lifetime performance of such underwater structures.

1. Introduction

Assessing the submerged part of marine structures introduces new challenges for inspectors. Poor underwater visibility conditions [1] make damage assessment particularly difficult. Additionally, divers must contend with cold, uncomfortable, and often hazardous conditions, and they must often carry out the inspection within a narrow time window. These factors contribute to the increased variability and reduced accuracy of the inspection results [2,3]. Given the extensive effort and expense associated with undertaking such inspections, there is a strong need to develop tools that can improve the condition of monitoring and, ultimately, enhance the quality of the inspection results. In this vein, this paper describes an approach for developing carefully controlled underwater virtual scenes that: (i) validate methodologies of assessment, (ii) facilitate the design of better image-based damage detection tools, and (iii) enable the performance of image-based non-destructive testing (NDT) methods to be evaluated prior to real-world implementation. A virtual-based approach can be a substitute to real-implementation for many situations and also provide both qualitative and quantitative ideas around the best implementation methods for underwater inspections. This is particularly relevant for complex underwater inspection campaigns, or where the time window available or the variability of environmental parameters potentially can reduce the data obtained, unless an assessment using virtual simulations is carried out. Several recent projects have focused on virtual reality (VR) or simulated augmented reality (AR) providing insight into archeological robotics [4,5] underwater cultural heritage [6], and underwater mining [7]. The application of such approaches attempts to fill an existing and much needed gap in terms of engineering use-cases relevant to commercial and safety or serviceability aspects. This paper builds on the abovementioned need of this nascent and rapidly evolving sector.

Three distinct case studies are used to illustrate applications of virtual data in underwater inspections in this paper to establish this idea. The first case study looks at how the creation of a virtual environment to enable virtual reality (VR) can play a role in developing inspection methodologies to meet the needs of fast-emerging technologies like fish farms, floating wind turbines, wave devices, ocean-bed cables, risers and long umbilicals. In particular, this case study features a virtual aquaculture site. The aquaculture sector is rapidly growing, and new fish cage designs are continually being produced online. VR will be very important at lower technological readiness levels to integrate the suitability of an offshore design or intervention solution. The capacity to test inspection methodologies and NDT tools without having to deploy full campaigns or large-scale experiments is of high practical value for inspectors [8,9].

The second case study shows how virtual imagery can be used to help develop new algorithms by generating vast amounts of labelled photorealistic imagery which can then be used to train deep neural networks. Deep learning techniques have experienced a surge in popularity in recent years, and these techniques have demonstrated strong performances across a range of computer vision applications [10]. However, these techniques typically require large quantities of training data which are generally unavailable for underwater inspection applications. Other works have resorted to using internet-based search approaches as a means of assembling large datasets [11] but these approaches require manual verification (since, for instance, searching for images using a keyword “crack” returns many images that do not relate to structural cracks), and there may be copyright issues. Furthermore, labelled datasets generated from internet-based searches only provide a class label for each image and they do not indicate the location of the object of interest within in the image. For the most part, this limits their usefulness to classification problems, and not segmentation and localization problems. To address this limitation, this case study creates a virtual scene of barnacle-fouled ship hull, and from this, an extensive dataset of synthetic imagery is generated with accurate ground-truth information that reveals the exact location of barnacles in the scene. This large dataset is then used to train a convolutional neural network (CNN) which is then applied to detect barnacles in a video that was captured as part of a real-world ship hull inspection campaign.

The third case study demonstrates how virtual scenes can be used to evaluate the performance of image-processing algorithms. This case study investigates how the performance of corrosion and crack detection algorithms vary as the turbidity levels are varied. A key point here is that the synthetically generated turbidity levels can be calibrated against known and physically meaningful turbidity levels from the publicly accessible Underwater Lighting and Turbidity Image Repository (ULTIR) [1] so that realistic conditions can be accurately represented in virtual environments. The value of this case study is that it gives inspectors an insight into the relationship between underwater visibility and the performance of image-based damage assessment techniques. Furthermore, it enables inspectors to assess the viability of adopting image processing approaches prior to an inspection and helps them to identify the limits at which image-based methods begin to produce unacceptably poor results (i.e., high probability of false alarms and low probability or detection).

2. Background

Exploiting virtual data is an established practice in many fields, such as investigating manufacturing production processes in the automotive industry [12] and training pilots in the aerospace sector where flight simulators are routinely used for replicating a host of scenarios [13]. These simulators are useful for practising and refining common tasks, such take-off and landing and coordinating with air traffic control, as well as for handling emergency events, where they help pilots respond to these situations in a safe environment. Additionally, VR (Virtual Reality) is increasingly gaining a foothold in the consumer market, where VR headsets are becoming more affordable and immersive, such as the HTC Vive Pro Eye enabled with foveated rendering, hands-free interaction and precision eye tracking [14].

Adopting VR technology for underwater inspections shares many of the same benefits that flight simulators bring to the aerospace sector. VR-based inspection simulations can help divers gain a better understanding of what to expect during inspections in a risk-free setting and can serve as an effective route-planning tool by giving inspectors a chance to identify parts of a structure that require special attention (e.g., critical joints) and easily relaying this information to the dive team. Communicating this information with the help of VR is more visually compelling compared with traditional approaches, which usually consist of presenting the dive team with a written brief outlining the task at hand. Depending on the type of structure, divers may be asked to assess:

(1)

Corrosion or indicators of corrosion

(2)

Consumption of cathodic protection

(3)

Presence and appearance of cracks

(4)

Exposed rebar, missing bolts and signs of damage to coatings, sealings, joints etc.

(5)

Deformation of the structure

(6)

Presence of scour and erosion

(7)

Upstream and downstream blockages

(8)

Presence and extent of marine growth colonisation

These tasks are carried out for traditional built infrastructure, like underwater sheet piles and for bourgeoning sectors like monopiles for offshore wind turbines, where limited weather windows are available for inspection. In practice, it is often challenging for the diver to determine what is noteworthy. Having the ability to virtually “walk-through” an underwater infrastructure scene with the dive team, and being able to show site-specific visual examples of important structural components and damage forms that should be documented, is a major asset, especially when language, technical and interpretive barriers exist between the engineers and the dive team and/or when the divers have limited experience and are not experts in either structural assessments or marine biology/chemistry [15,16].

Numerous researchers have looked at exploiting virtual environments for underwater application—mainly for underwater archaeological and gaming applications where VR allows users to explore underwater sites of cultural significance such as shipwrecks [17,18,19]. A virtual and augmented reality system for exploration of underwater archaeological sites was demonstrated in [20]. The system offers archaeologists and the general public a way to explore a realistic reconstruction of the sites and glean new insights on underwater archaeological sites. Similarly, in [21], virtual reality technologies were proposed as tools that could help increase exploration time in an underwater archaeological site.

In [22], a virtual-environment-based testbed was developed to act as an alternative to difficult, costly, and possibly hazardous real-time testing and evaluation of control algorithms for autonomous underwater vehicles (AUVs). Other examples of graphical simulators used for AUVs are outlined in [23]. In another study, [24], the authors explored the process of automatically creating virtual tours from footage captured using an omnidirectional underwater camera in AUVs that can cover large marine areas with precise navigation.

Virtual underwater scenes have also been developed to facilitate SCUBA training and to provide virtual SCUBA diving experiences. In [25], a highly immersive, multi-sensory VR simulation is demonstrated whereby users are attached to a motion platform with their outstretched arms and legs placed in a suspended harness. They receive visual and aural feedback through the Oculus Rift head-mounted display and a pair of headphones for added realism. Additionally, buoyancy, drag, and temperature changes are simulated through various sensors.

Virtual reality technology has been employed to aid the inspection process in a number of cases. Virtual reality systems have been developed for aircraft inspection and maintenance training [26] and for fire safety inspections [27]. In [28], the authors discuss the use of virtual tours, augmented reality, and informational modelling for visual inspection and structural health monitoring. A methodology for integrating existing data about a structure and meta-data into a combination of virtual tour (VT), augmented reality (AR) and informational modeling (IM) environment was presented. The objective of their method was to enable on- and off-site presentation of engineering assessment data in an organized, intuitive, and interactive manner, and additionally foster communication between different parties involved with a structure. However, to the authors’ knowledge, there have been no works that consider the use of virtual environments for overcoming problems that are specific to underwater damage detection as part of structural inspections. This research is especially geared towards exploring application areas where VR can support inspection coordination and improve the quality of collected inspection data. Thus, an important contribution of this research is the identification of applications where VR can contribute to underwater monitoring, and the description of how virtual environments can be simulated for these applications.

3. Case Studies

This section presents the three case studies which show various benefits of using virtual environment as a tool for enhancing underwater inspections. The first case study focusses on how virtual environment can play a role in developing inspection methodologies, the second case study shows how virtual data can be used to develop performant deep learning computer vision techniques, while the third case study demonstrates how virtual scenes can be employed to gauge the performance levels of image-processing based damage assessment algorithms. The overall motivation behind the choice of the cases is to extend the applicability of VR approaches to a range of traditional and bourgeoning infrastructure sectors related to the marine environment. Typically, such sectors have a strong commercial and social relevance and their inspections are limited due to lack of access and constraints in archiving, analysing and synthesizing the obtained information. To this effect, three sectors are chosen. This first sector considered is the fisheries sector which is currently undergoing significant modernization through automation, analyses and monitoring [29]. The second case study focuses on the growth of barnacles on ship hulls and addresses an often-overlooked area of fluid-structure interaction due to marine growth. For static or mobile structures, marine growth can alter the hydrodynamics on the structure and fundamentally, such variation in force is related to the extent and the surface roughness of the growth, eventually impacting the parameters in Morison’s equation [30]. A VR approach-driven extensive simulation can thus integrate existing knowledge about such marine growth and eventually lead to the standardization of design process around this topic, linked to specific types of growths. The third sector considered in this paper is the degrading-built infrastructure in the marine environment, where significant information from diving or remotely operated vehicle (ROV) driven campaigns is often available but there is inadequate guidelines around the right environment to carry out such inspections mainly due to variations in lighting and turbidity conditions. The impact of such inspections is directly related to the safety and performance of such structures. While the choice of the three sectors considered are strongly guided by ideas on safety, the impact of each is presented in

Table 1. Alignment of Case Studies to UN Sustainable Development Goals.

Case Study	Alignment to UN Sustainable Development Goals
Fisheries	14. Life Below Water
Marine Growth	9. Industry, Innovation and Infrastructure
Structural Inspections	11. Sustainable Cities and Communities

, mapped to the sustainable development goals outlined by the United Nations (UN).

3.1. Virtual Scene Simulation to Investigate Assessment Methodologies: A Case Study on Fisheries

This case study demonstrates four distinct areas where virtual scenes can assist inspectors in the rapidly emerging sector of fisheries: (a) exploration, (b) interaction, (c) route planning, and (d) developing safety procedures. To create such a tool, the virtual underwater environment needs to be developed first, as presented in Figure 1. This step can be complex and related to site-specific conditions.

A virtual base scene, specific to the site under consideration, can be developed using cross-platform game engines. This paper uses Unity [31] in this regard. The base scene consists of the landscape, water plane, outdoor lighting model and the atmosphere involving the sky and sun features. In this paper, an external module (Aquas) was added to the project to extend the functionality of the water-related features for the base scene.

The structure within the base scene incorporated by importing their 3D models (including marine growth shapes, colours and textures if required) based on existing information from drawings and/or previous inspections. Realistic marine life and bubbles within such scenes can make the virtual experience richer and it is important for moving objects or structures within water to respect fundamental physics as much as possible.

To virtually explore the scene created in all directions (including jumping and dashing), camera controls are added next, operated by the user typically with mouse and arrow keys. Optical effects underwater due to varied lighting and turbidity are important factors influencing the quality of information from images obtained and such effects are simulated by adding a low contrast blue fog, representing scattering and varied absorption of light rays underwater, depending on their wavelength. The density of the added blue fog effect is representative of turbidity in water. Ambient above-water and underwater sounds (e.g., breaking waves, marine life, diving splash) are added next to create an immersive experience for users.

Virtual information panels are placed throughout the scene at critical points around the infrastructure to be monitored to guide users and provide site-specific information. Such information provides a virtual experience of the type of damage or other features of interest that divers should expect, and the related monitoring tasks that are subsequently required to be carried out. Such panels can also help engineers to communicate with the dive team more effectively to obtain maximized information. An immersive experience like this should avoid any visual lag for the end users and it may be necessary to optimize certain elements of the scene for this purpose. This is particularly relevant for hand-held devices, when dealing with particularly complex scenes and the design of scenes should balance realism with computational efficiency based on the device where a virtual-scene driven training or implementation will be carried out. The deployed virtual diving scenario should support a wide range of platforms (e.g., Windows, Max OS X and Linux, Android, WebGL) to allow preliminary virtual dives on-site.

Figure 2a presents a virtual salmon fisher representing an actual site (Figure 2b) in Ireland. It starts from above the water surface, so that the diver has a good sense of the site overall. To make the experience more attractive, a realistic terrestrial environment (e.g., coastline of inspection site, buoy) for the actual site was added.

The user can look around the above-water scene and the underwater effects are active (Figure 3) once the user dives below the surface. For realistic effects, continuous discharge of air bubbles, moving aquatic vegetation on seabed and 3D models of salmon inside the fish cage are animated using artificial intelligence techniques. An exploration of the scenes can thus engage the user into active specific to the site in use.

Interactive information panels are positioned at points of interest around the net structure for the fish farm, as shown in Figure 4. The panels advise divers on relevant damage types for inspection and specific tasks associated with inspections. This minimizes human error and has the potential to obtain data with better specification and control for comparison in future.

The use of numbered information panels also allows to create a pre-planned route for inspection, reducing time in water, minimizing diver risks and maximizing information obtained from dives. It can help test the testing contingency safety procedures, and help relay critical information underwater in a uniform manner with significantly lower chance of errors from mismatch of interpretation of visual information.

3.2. Utilizing Virtual Data to Develop Image-Processing Based Damage Detection Techniques: A Case Study on Marine Growth on Structures

The second case study demonstrates how virtual scenes can be useful for training deep neural networks. Marine growth on structures has been considered as an example in this regard since it can creates significant changes in hydrodynamic effects on the structures and such changes are often critically dependent on the correct description of the extent and surface roughness characteristics of marine growth.

Deep learning techniques across a wide range of computer vision tasks offer numerous advantages over conventional techniques by learning rich feature representations, which can be significantly more robust than traditional hand-crafted features based on the size and variety of the training set. However, currently, there is a paucity of large annotated training datasets for underwater applications and this case study attempts to demonstrate how large datasets of synthetic imagery can be generated and used to train deep neural networks for such applications. A CNN is trained in this regard to automatically detect barnacles from video footage recorded as a part of a ship hull inspection. Barnacles increase the surface roughness of the ship hull which causes more drag and thus increased fuel costs/lower speeds. Knowing the extent of barnacle colonization and being able to track its growth in time is useful to engineers as this information can be used as inputs to more detailed computational fluid dynamic (CFD) models. The information is also useful for ship owners/operators to decide when to carry out expensive cleaning regimes or maintenance activities as well as optimizing the frequency of cleaning operations.

A virtual scene was created using E-on VUE^® software featuring a barnacle-covered surface (Figure 5). Four prototype 3D models of fundamentally barnacles were created. From these four master barnacles, thousands of instances of new barnacles were developed by applying non-linear scaling operations to the original barnacles and applying unique material properties with controlled and random variations.

This allows for training on a large and diverse dataset featuring a broad range of types of barnacles, varying lighting and visibility conditions and viewing perspectives. The CNN can then perform successfully when presented with new scenes. Figure 6 shows the ship and the hull surface under some of the representative lighting and visibility conditions that were considered.

A total of 16,000 images of barnacles were rendered, some of which are shown in Figure 7. Also shown are some “background” images (i.e., images without any barnacles), required for the CNN to learn the characteristic features that enable the distinction between barnacle and non-barnacle classes.

Two CNN models with the same network architecture (Figure 8) were trained from scratch for 40 epochs using synthetic data generated from the virtual scene. One model was trained using 4000 images (48 min training time) of barnacles and another using 16,000 images (189 min training time) to test if and how their relative performance was related to the training set size, when tested with a 6GB graphics card (model: NVIDIA GeForce GTX 1060). The trained models were applied to detect barnacles in images extracted from a real-world ship hull inspection video.

Features in the images, some of which may be barnacles, were first detected using a circular filter. This was preferred over the slower approach of directly detecting barnacles at each pixel using a sliding window. Four sample frames for the filtered images are shown in Figure 9. Visually, it appears that there is a good correlation of feature detection with regions of barnacle fouling. Classification was performed only at locations where features are detected and this very significantly reduced the detection runtime without adversely affecting its accuracy. The two CNN models were applied to 10 frames of ship hull inspection video and a representative sample of the detection results is shown in Figure 9.

The corresponding average sensitivity, specificity, accuracy and precision metrics for the method [32], are shown in Figure 10. Sensitivity is defined as the proportion of barnacle pixels in an image which are identified by the algorithms as representing barnacles. Specificity is defined as the proportion of background pixels in the image which are identified as representing barnacles. Accuracy measures the overall effectiveness of the algorithm to differentiate the barnacle and background pixels correctly rather than its effectiveness on a class-by-class basis while precision is the fraction of pixels correctly identified as representing barnacles among the total number of pixels classified as representing barnacles. These metrics can be expressed as:

$$\mathit{Sensitivity}=\frac{TP}{TP+FN}$$

(1)

$$\mathit{Specificity}=\frac{TN}{TN+FP}$$

(2)

$$\mathit{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$$

(3)

$$\mathit{Precision}=\frac{TP}{TP+FP}$$

(4)

where TP (true positives) is the number of pixels correctly identified as representing a barnacle, FP (false positives) is the number of pixels incorrectly identified as representing a barnacle, TN (true negative) is the number of pixels correctly identified as representing the background/uncolonized surface, and FN (false negative) is the number of pixels incorrectly identified as representing the background. The ground truth was determined by a human operator who manually identified the barnacle regions in each image. The visually segmented images act as the control and are assumed to show the true composition of the scene.

The detection results in Figure 9 and the average sensitivity values shown in Figure 10 indicate that the CNN model trained on 16,000 images outperforms the model trained on 4,000 images, in terms of the model’s ability to correctly detect barnacles (sensitivity) whilst having fewer false positives (specificity). High specificity is generally achieved at the expense of sensitivity, but in the case of the model trained with 16,000 images, a better balance is achieved between the two metrics. The accuracy and precision are comparable for both models.

3.3. Performance Evaluation of Image-Processing based Damage Assessment algorithms: A Case Study on Detecting Cracks and Corrosion in an Underwater Structure

The third case study investigates how virtual scenes can be used to gauge the performance levels of image-processing based damage assessment algorithms. Chloride-induced corrosion and cracks in concrete structures are typical in corrosive marine environment. Visual inspections and vision-based systems can often detect their presence conveniently, but the efficacy of vision-based systems in practice is heavily reliant on the optical qualities of the underwater environment. Reduced visibility conditions diminish the ability of a camera, and related image-processing algorithms to effectively identify instances of damage. An understanding of the relationship between visibility conditions and the performance of image-processing techniques can thus be important to rationalise the use of image-processing as part of an underwater inspection campaign [33,34]. The performance metrics typically include of Probability of Detection (PoD), Probability of False Alarms (PFA), and Receiver Operating Characteristic (ROC) Curves or planes [35,36]. Virtual underwater scenes can be helpful in extracting meaningful information about the ROC curves related to the on-site performance of image-based damage detection techniques under a range of visibility conditions, providing an early estimate on the reliability of damage detection information obtained from such image-based methods. This case study implements this approach in identifying corrosion and cracks on marine structures. A virtual wharf was created (Figure 11), and crack and corrosion damage (derived from real-world photographs) were introduced to the virtual scene.

Image quality is mainly affected by luminosity, sharpness (focus accuracy), contrast and noise [37] due to variations in on-site operating conditions. Turbidity, defined as the cloudiness in water caused by the presence of suspended organic (e.g., decomposed plant and animal matter) or inorganic (e.g., silt, clay) solids scattering and absorbing light and reducing visibility, is one of the most influential factors in this regard [38]. Three levels of turbidity are considered here: clear water, medium turbidity, and high turbidity. Damaged surfaces under these turbidity levels are shown in Figure 12 and Figure 13 for cracks and corrosion respectively, at a distance of 1m from the virtual camera. The damages are clearly visible in clear water, and the visibility falls with increased turbidity, until they are barely visible in high turbidity conditions.

Crack detection is a well-studied topic and several image-based detection algorithms exist. Here, a percolation-based method [37] was applied to the virtual images under varying turbidity levels, (see in Figure 12). The detected cracks are shown in Figure 14 and the performance of this method under each turbidity condition is depicted via the ROC curve in Figure 15. The closer the curve can be to the best performance point with coordinates (0,1) and representing 100% sensitivity (no false negatives) and 100% specificity (no false positives), the better is the detection performance. The best performance point is the closest point on the curve to the best performance point [20].

Figure 15 shows that the performance of crack detection gradually declines with increasing turbidity, but the adopted image-processing method is still capable of producing reasonable results even in high turbidity conditions. Findings of this nature can be useful for inspectors as it allows them to choose a technique appropriate to their needs and one that is sufficiently robust to the onsite operating conditions. Virtual scenes can aid this selection process.

For corrosion damage, a good method should identify and accurately define all corroded regions whilst minimizing the inclusion of extraneous regions in an image. In reality, perfect damage detection is impossible to achieve given the inherent chromatic and luminous complexities encountered in natural scenes. Some corrosion surfaces are more distinguishable by comparing texture attributes while some are distinguishable by color. Color intensity and texture analysis-based methods are typically applied for corrosion detection and naturally, techniques in each group are suited to different applications. As an example, the performance of an established texture analysis technique [39] is applied corrosion stain photographs captured under three turbidity levels (Figure 13). The detected corroded regions are shown in Figure 16 and the performance of this method under each turbidity condition is depicted by the ROC curve in Figure 17.

For corrosion detection, the overall performance of the texture analysis-based technique was moderate in clear water conditions while the technique did not produce compelling results in high turbidity conditions. Under such circumstances, the choice of such a technique for implementation in field conditions can be questioned or scrutinized. Predicting the success of image-processing based methods when applied in field conditions is a challenging prospect and virtual scenes can help inspectors get a sense of the expected performance under realistic operating conditions prior to carrying out an actual inspection. This not only is economical but can avoid exposing divers to unnecessary risk. When compared with recent works [40] it is observed that the obtained performance estimates from virtual scenes are of a similar level to field investigations.

4. Discussion and Conclusions

The use of virtual technology has developed considerably over the past few years. This paper examines various aspects of virtual environments and virtual data that can help improve the condition of underwater inspections. Immersive virtual underwater scenes can assist divers and inspectors when planning and conducting real-world inspections. Divers can experience a virtual and unrestricted tour of real underwater inspection sites prior to carrying out, from which they can develop a rich understanding of the scene and are better positioned to develop effective inspection strategies. These learning benefits of virtual environment are illustrated in the first case study, which features a fish farm site.

The second study highlights the value that virtual scenes can have for training deep learning techniques. Deep learning techniques have already attracted significant interest in other fields owing to the high performances that they can achieve; however, their impact in the domain of underwater imaging is restricted, largely due to the lack of appropriate training data. Curating a dataset takes time and domain-specific knowledge of where and how to gather relevant information, and it often involves a human operator having to manually identify and delineate objects of interest from real-world images. This is a tedious and time-consuming task considering that datasets of up to thousands—or even tens of thousands—of training images are typically required to build a robust and effective deep network. The high cost of assembling datasets puts state-of-the-art deep learning techniques beyond the reach of many practitioners. This case study presents a framework for generating large datasets of synthetic images, from which deep neural networks can be trained and applied to tackle real-world problems. A virtual scene of an underwater ship hull surface is created, and from this, a large dataset of synthetic imagery is generated with accurate ground-truth information that reveals the exact composition of the scene. This large dataset is used to train a convolutional neural network which is then applied to a real video of an underwater ship hull inspection where the model can successfully detect barnacles on the surface of the ship.

The third study demonstrates how virtual scenes can be useful for the purpose of evaluating the performance of image-processing based algorithms. The quality of subsea inspections largely depends on the ability of inspectors to detect and objectively record details of defects. The type of damage present and the on-site operating conditions are crucial factors that dictate the effectiveness of image processing algorithms for detecting damage. The reduced visibility conditions that are often associated with underwater scenes diminish the ability of the camera, and subsequent image-processing algorithms, to successively identify instances of damage. It is therefore important that inspectors can develop an understanding of the relationship between visibility conditions and the performance of image-processing techniques so that they can rationalize the use of image-processing methods as part of an underwater inspection campaign. The presented case study shows how virtual scenes can be useful in this regard.

While the use of virtual data in the inspection process has numerous advantages, there are also some limitations. In terms of practical limitations, the production of a simulated environment and the rendering of synthetic data can be a time-consuming and highly involved process, especially for complex structures where 3D models are not already available and are difficult to model from scratch. Additionally, rendering underwater scenes is often a computationally intensive process because of the highly reflective and refractive materials in the scene.

Inspectors/machine-learning practitioners may also question how much they can ‘trust’ the validity of the virtual data since there will be a gap between real world imagery and synthetic data. For this reason, inspectors should view virtual data as ancillary tools that can aid in the inspection planning process and they should not rely too heavily on it. Similarly, machine learning practitioners should be mindful that some details may not be captured when creating the simulated environment and, as a consequence, damage-assessment algorithms trained with synthetic data may not generalize well when applied to some real-world examples. Therefore, it is prudent to collect some degree of “real” data in order to validate the machine learning model.

In terms of future research directions, the concept of “digital twin” is attracting growing interest and looks like it will be a promising research direction for marine structures also. Having a close pairing between the virtual model and physical structure allows analysis of data (such as fluid-structure simulations can be performed on updated virtual models to get a sense of the actual forces on the real-world structure) and monitoring of systems in order to guard against problems before they arise, thereby preventing downtime, and allowing researchers to explore ‘what-if’ scenarios and plans for future actions by using simulations.