1. Introduction
Routine dry dock maintenance on the outer hull of ships is essential for the efficient and sustainable operation of shipping [
1,
2]. Improper maintenance of ship hulls may lead to an increase of the fuel consumption due to the surface roughness [
2,
3]. In addition to that, safety issues can also be aroused due to the improper hull maintenance. Typically, maintenance work on ship hulls needs to be carried out every 4–5 years [
4]. One of the major maintenance work on ship hull is the blasting work carried out for removing rust and adherence [
2]. Hydro blasting or abrasive blasting is commonly used for this purpose. In most of the typical cases, these blasting works are carried out by human workers with the aid of semi-automatic devices [
2]. Furthermore, inspections of hull surfaces are done by humans to identify the areas that need to be blasted as well as to evaluate the quality of a blasting work [
2]. Thereby, the conventional ship hull maintenance work suffers from concerns in accuracy, efficiency, cost, and safety.
Many robotic solutions have been emerged for resolving the issues associated with conventional maintenance work that requires extensive human labor in diverse domains [
5,
6,
7,
8]. Similarly, many robots and devices have been developed for inspection and maintenance work of ship hulls to resolve the shortcoming of the conventional methods discussed above [
9]. In this regard, many ground-based robots and robotic systems have been introduced with the intention of automating dry dock maintenance work of ships hulls [
10,
11,
12,
13]. These robots require the support of external structures such as rail mechanisms, gantry cranes, and cherry pickers for facilitating the movements along a ship hull. A system that uses a set of stationary lidars and cameras for detecting corrosion spots in a ship hull has been proposed in [
14]. However, these ground robots have limited access to ship hulls due to hindrances caused by geometric constraints and occultations. Furthermore, it is required to deploy either a set of systems or facilitates the movement of the structures around a ship for covering an entire ship hull. Possibility of using Unmanned Area Vehicles (UAV) such as quadcopters and hexacopters for inspection of ship hulls has also been investigated since UAVs have proven the ability to inspect confined and limited access areas in other application domains [
9,
15]. Nevertheless, the usability of UAVs is limited in inspection of ship hulls due to the lack of operation for a longer time and distance as well as low accuracies in positioning.
Robots that can climb on ship hulls are preferred for inspection and maintenance work of ship hulls due to their performance benefits [
16,
17]. A climbing robot based on magnetic adhesion has been developed for inspecting long welding lines on ship hulls [
18]. MIRA [
19] is another climbing robot developed for the inspection of ship hulls. The wheels of this robot are integrated with permanent magnets fixed on a flexible strip to provide the climbing ability. A mobile robot with a novel design of magnetic wheels based on Halbach array has been introduced in [
20] for dry dock inspections. A tracked mobile robot that is capable of adapting to the uneven surfaces has been developed for ship hull inspection [
21]. The self-adaptability of the robot allows it to move through uneven surfaces. Since magnetic wheels are often used for inspection robots intended for ship hulls, in depth analysis on magnetic flux distribution of wheels for climbing robot has also been conducted [
22]. Furthermore, optimization of a magnetic wheel for a grit blasting robot has been investigated [
23]. Apart from the climbing robot with magnetic adhesion, robots based on negative pressure adhesion have also been developed [
17,
24]. The work [
24] analyzed the variation of required negative pressure in different conditions such as in wet conditions. Nevertheless, the scope of the work cited above is limited to the design and development of climbing robots or their adhesion mechanisms for inspection of ship hulls; the methods for automatic inspection and maintenance such as vision-based corrosion detection, are not discussed within the scope of the work.
A magnetic crawler robot with a monocular-camera based vision processing method for inspecting ship hulls has been developed [
25]. The vision processing method is capable of constructing a mosaic image, which can produce a metric representation of the areas inspected by the robot, from the multiple images captured from different positions of the robot. Many computer-vision based methods for detecting corrosion in metal structures, including ship hulls and steel bridges, have been developed [
26]. According to the outcomes of the survey [
26], the color is the principal representative feature for identifying the corroded areas. Furthermore, the cited survey concludes that learning-based methods could perform better than the non-learning methods in detecting corrosion through vision. A method for a surface corrosion grade classification of metals has been proposed using a deep convolution neural network [
27]. However, the method was developed to detect the correction grade based on microscopic images of the surface. A microscopic image could cover only a minimal area of the environment to be inspected due to the constraints of magnification. If microscopic images were taken to cover a ship hull, a considerably large number of images would need to be taken for covering the whole surface. The vision system has to be directed toward very near proximity areas many times in this regard. Therefore, the inspection process would be prolonged, hindering the feasibility of adopting the method proposed in [
27] for a robot intended for inspecting the corrosion grade of ship hulls. A vision-based defect detection method based on statistical approaches built upon circular histograms entropy analysis has been introduced to identify the rusted regions of ship hulls [
28]. Nevertheless, the proposed method has been verified merely using offline images and have not been validated in the real-world application context.
The ability to detect corrode areas of ship hulls from a set of stationary cameras by using histogram-based background detection and adaptive thresholding methods have been investigated in [
14]. According to the outcomes, these methods are effective only when the inspected zones have a majority of non-corroded areas. The work [
29] proposed an aerial robot equipped with a feedforward neural network trained for corrosion detection of ship hulls. The proposed neural network is capable of detecting the areas of corrosions/coating breakdowns of ship hulls. An automated visual inspection method that can detect defected areas for automated spot blasting has been proposed in [
30]. The cited work proposes a wavelet transformation combined with an entropy-based method for detecting the corrode spots. The proposed method has been implemented on a crane-based ground robotic system and verified the abilities.
Nevertheless, all the vision-based methods discussed above are intended for detecting corroded areas on a surface. The blasting quality needs to be classified into a few categories for benchmarking a hydro blasting work. Furthermore, the surface appearance becomes different after a blasting process. Hence, the methods discussed above could not be adopted for benchmarking the performance of a hydro blasting process. In addition to that, much of the work is limited to the development of vision-based detection mechanisms for robots, and the ways for utilizing the detection outcomes as a fully integrated system for synthesizing a benchmarking map for an already blasted ship hull are not considered within the scopes. Therefore, this paper proposes a novel vision-based benchmarking method for hydro blasting. The proposed benchmarking method can classify the hydro blasting quality into three categories, good, medium, and bad. A Self-Organizing Fuzzy logic (SOF) classifier is proposed to realize the classification into the benchmarking categories. In addition to that, a hydro blasting robotic system consists of a hydro blasting robot and a benchmarking robot has been designed and developed. An overview of the proposed robotic system is given in
Section 2.
Section 3 presents the theoretical backgrounds of the proposed SOF classifier. Particulars on experimental validation are discussed in
Section 4. Concluding remarks are given in
Section 5.
3. Self-Organizing Fuzzy Logic (SOF) Classifier
A Self-Organizing Fuzzy logic (SOF) classifier [
31] is proposed for the benchmarking classifier. The architecture of the SOF classifier was originally proposed by Gu and Angelov [
31] in 2018. The underlying theoretical rationales and mathematical proving of the architecture, such as analysis of convergence, have been analyzed in the cited work. Furthermore, the performance of the SOF classifier has been compared against the other method using well-known offline testing data such as data set for optical character recognition. Nevertheless, a SOF classifier has not been proposed for benchmarking the quality of hydro blasting. A SOF classifier was selected for this application based on the following reasons.
The benchmarking statuses are defined based on human expert knowledge, and the benchmarking categorization is performed based on the three fuzzy linguistic descriptors, good, medium, and bad. Moreover, the benchmarking classifier should emulate the human expert knowledge in the classification process. Fuzzy logic has been proven to be well suited for replicating the human expert knowledge that can be represented through linguistic expressions [
32,
33,
34]. Furthermore, fuzzy logic has the ability to cope with imprecise sensor information [
35,
36,
37]. Therefore, a method based on fuzzy concepts would be expected to perform well in this specific application.
A human interpretable and explainable set of rules is generated after the training of a SOF classifier. Explainable intelligent techniques are preferred for ensuring transparency and trust of safety in this sort of industrial application, which might become hazardous from undesired control actions that might be performed by a robot [
38]. Furthermore, the set of rules can be tailored based on expert knowledge.
A SOF classifier is a highly efficient model with high classification accuracy [
31,
39]. Therefore, it requires lower computational power with respect to the other existing models. In addition to that, a SOF classifier does not require dedicated optimized hardware such as GPU cores for the computation. Moreover, a SOF model is comparatively lightweight.
Many existing classification models rely heavily on prior assumptions on data generation models and user-defined trial and error parameters such as learning rate and the size of the network. In most of the practical cases, the assumptions on data generation are often too tough to be sustained, and user-defined parameters are often troublesome to define due to the insufficient prior knowledge of the problem. In contrast, a SOF classifier is nonparametric, and it does not require an assumption on data generation models and parameter knowledge about the problem of interest [
31].
The architecture of the SOF classifier is depicted in
Figure 8. It is designed to assign a class label to a given data sample,
x based on trained knowledge. It should be noted that
x can be a row vector with any dimension. The trained knowledge is stored as a set of zeroth-order AnYa type fuzzy rules [
40]. An AnYa type fuzzy rule has the form given in (
1). The antecedent of an AnYa type fuzzy rule is in a nonparametric vector form, and it does not require membership functions. Here, the symbol, ∼ denotes the similarity between a data sample and a data cloud. This similarity is similar to the concept of degree of membership in the Mamdani type or Sugeno type fuzzy inference system.
is the set of prototypes belonging to
class, and
is the number of prototype in
. These prototypes are the centers of the data clouds. The data clouds and the corresponding fuzzy rules are formed during the training phase of the SOF classifier.
To identify the class corresponding to an input data sample (i.e.,
x), the firing strength of
fuzzy rule,
is evaluated as in (
2) by the local decision-maker where
denotes distance/dissimilarity between the two data points. Common distance/dissimilarity measures such as Euclidean, Mahalanobis, and Cosine distance can be used in this purpose. Then, the class label of the data sample is assigned by the overall decision-maker using the winner-takes-all method as in (
3). For
.
The classifier identifies the prototypes from each class and forms the data clouds. A zeroth-order AnYa type fuzzy rule for each class is then formulated. The training process is independent for different classes, and there is no influence from the training of one class to another class. Thereby, the training process can be explained based on class such that . Suppose data sample set of class is denoted by such that , where is the number of data samples belonging to class and K is the total number of data samples in the dataset. All the data in a training data sample may not be unique and the same data sample may be repeated. The corresponding unique data sample set and their frequency of appearance are considered as and respectively, where is the number of unique data sample belonging to class, and is the total number of unique samples in the dataset. Moreover, this definitions lead to and . The main steps of the training of SOF classifier are given below.
Step 1: The multimodal density [
41] of
unique data sample of
class,
is calculated as in (
4) where
and
denotes distance/dissimilarity between the two data points.
Step 2: The sample
is sorted according to the multimodal density and mutual distances calculated in step 1. The sorted sample set is denoted by
, where
is given in (
5) and the rest are obtained as in (
6). It should be notated that
corresponding to
is excluded in each run of (
6), and the process is repeated for all the data in the sample.
Step 3: The multimodal density set after the sorting in step 2 is taken as
. The initial set of prototypes,
is generated by considering the condition given in (
7). Moreover, the local maxima of
are taken for
.
Step 4: The nearby data samples are attracted to form data clouds resembling Voronoi tesselation [
42]. The assignment of a data sample to a cloud is determined based on the winning prototype obtained as in (
8).
Step 5: The set of centers of the formed data clouds,
are identified.
is equivalent to
. The multimodal density at the center of
cloud is calculated as in (
9) where
is the number of members in
cloud, and
n is the number of clouds formed.
Step 6: The set of centers of neighboring data clouds of
data cloud,
is identified for each
i based on the condition given in (
10) such that
and
. Here,
defines the average radius of local influential area around each data sample. This parameter is calculated iteratively as given in (
11) based on the granularity level
defined by user. Here,
is the number of pairs of data samples where the distance between a pair is less than
for
. When
,
is the number of pairs of data samples where the distance between a pair is less than the average distance,
.
Step 7: The set of representative prototypes of
class,
are identified by evaluating the condition given in (
12).
Step 8: A zeroth order AnYa type fuzzy rule is created for
class in the format given in (
1), where
is the number of representative prototypes.
4. Results and Discussion
4.1. Data Collection and Training, and Classification Performance
The data set required for the training and testing was prepared by capturing images of blasted ship hulls through the robot’s camera. The captured images were manually labeled to benchmarking categories with support of expert knowledge. For each benchmarking class, 1850 images were gathered, yielding the total size of the data set to 5550 images. Geometrical transformations, such as rotation, were applied to the data set to prevent the possible overfitting. The data set was randomly divided into two subsets for training and testing in the ratio of 80:20.
The variation of the classification accuracy of the proposed Self-Organizing Fuzzy logic (SOF) classifier for the testing data was obtained by varying the granularity level (i.e.,
L) and the type of distance/dissimilarity measure. Furthermore, the variation of the execution time (i.e.,
t) of the classifier was also examined. The variation of the mean accuracy and the mean execution time with granularity level and distance type is given in
Table 1. The mean accuracy was calculated by considering 10 different trained classifiers (10 randomly selected data set for training) for each case. The mean execution time was obtained by running the classifier 100 times for each case in a laptop with the Intel Core i7-9750H processor and 16 GB memory. It should be noted that the execution time represents only the time taken by the classifier when assigning the class of a given input, and it does not include the time taken by the vision processing module for extracting the inputs for the classifier.
Improvement in the accuracy with increasing L could be observed when the cosine distance was used. In contrast, a reduction in accuracy could be observed when L changed from 8 to 12 when Euclidean distance was used. The overfitting of the classifier was the reason for this behavior since L controls the generalization of the classifier. An increase in execution time could be observed with increasing L irrespective of the distance type. Nevertheless, the execution times are in the order of microseconds (highest 23.8 ms), implying a trivial computational overhead for the realtime operation of the robot.
The highest mean accuracy of 0.9942 was observed when Euclidean distance and granularity level of 8 were considered for the SOF classifier. The mean execution time for this configuration was 8.67 µs. Therefore, a trained case of Euclidean distance and granularity level of 8 was considered for the real-time experimenting with the robotic system. The confusion matrix corresponding to this trained SOF classifier is given in
Table 2 for conveying insights of the classification performance.
4.2. Realtime Operation on the Robot
The proposed overall system has been implemented, and a benchmarking map generation has been tested on a ship hull. The benchmarking robot was placed in a ship hull, as shown in
Figure 9. The ship hull had been blasted by the blasting robot previous to this experiment. The blasting robot would work usually covering the ship hull with more or less even blasting quality in most situations. However, areas with different blasting qualities could be observed due to the operational issues of the robot, such as interruptions of pressurized water supply and sudden variations in navigation speed, during the real operation for a long time. For the sake of better demonstration of the benchmarking ability, such interruptions had been intentionally created during the blasting. Intentionally interrupting the blasting robot for making different blasting performances would not detract the experimental condition from the spontaneity of typical operating conditions. After placing the benchmarking robot on the ship hull, the robot was moved in a horizontal path. The SOF classifier analyzed the visual feedback of the robot in realtime. A video (
Supplementary Materials: Video S1) of this segment of the experiment is attached as a supplementary multimedia attachment.
The robot could achieve a frame rate of 14.66 frames per second (fps). However, the frame rate was intentionally limited to 7 fps to avoid unnecessarily higher overlapping of the areas inspected by the robot in each frame. The corresponding benchmarking map generated by the system is overlaid on top of the ship hull, as shown in
Figure 10. Here, the areas in green represent the good blasting quality, while the areas in yellow and red represent the areas with medium and bad blasting qualities. Altogether 195 captured frames were analyzed by the robot during this run, and the corresponding frame number is annotated in this overlaid map. The captured frames from the robot’s camera in intervals of 20 frames are given in Fig as samples for conveying the corresponding appearance and condition of the surface.
The area represented from frame 1 can be benchmarked as bad quality blasting based on expert knowledge. The corresponding color of the benchmarking map for the area represented by this frame is red, which indicates that the benchmarking method correctly identifies this frame. Similarly, the frames 20 and 40 were benchmarked as bad by the system as expected. The frames 60, 80, and 100 were benchmarked as good by the system as similar to human expert knowledge. However, frame 120 was benchmarked as good by the system, where the frame should not have been tagged as good per the human expert. The possible cause for this miss classification is that this image represents a combination of a good segment and a bad segment. The frames 140, 160, and 180 were tagged as bad in the benchmark map similar to the expert recommendation. Overall, the benchmarking robot was capable of correctly benchmarking the blasting quality to a greater extent. Nevertheless, few failure situations could be observed when the captured frames involve segments of different blasting qualities. Based on these observations, it can be concluded that the proposed benchmarking robotic system is capable of benchmarking the hydro blasting with adequate accuracy in realtime. Furthermore, it is capable of synthesizing a benchmarking map for a previously conducted hydro blasting.
The benchmarking map generated by the robot can be used to ensure the quality of a blasted ship hull before starting the painting. If the paint were applied to a ship hull that was not adequately blasted, the new paint would not last long, which degrades the quality of the maintenance work. Thereby, the proposed method for synthesizing a benchmarking map for hydro blasting would be useful to identify the areas that were not adequately blasted, and subsequently, selective spot blasting could be carried out on those areas. The benchmarking map would be useful in planning an efficient path for the blasting robot for the reblasting process in such cases. Furthermore, the representation of blasting quality categories in the map could be used to decide the amount of pressure required for the reblasting of the corresponding area. For example, if the benchmarking category is medium, a medium level of pressure can be used. In contrast, if the benchmarking category of an area is bad, then a high level of pressure can be applied for the corresponding area. Moreover, the categorical representation of the blasting quality in the benchmarking map would help to improve the overall efficiency of the blasting work. Therefore, the ability to synthesize the benchmarking map for hydro blasting work on a ship hull is widely useful for improving the efficiency and quality of ship hull maintenance.
A few miss classification was observed during the realtime operation with the robotic system. However, the accuracy of the proposed classifier is 99%, which is highly adequate for the application context. In addition to miss classification errors, the navigation errors of the robot due to poor localization cause errors to a synthesizing of a benchmarking map. Moreover, the accuracy of navigation and localization of the benchmarking robot should be maintained at a higher level to minimize the induce of errors. Nevertheless, solely relying on the beacon-based localization system is challenging in the working environment due to the high-frequency noises generated from the surrounding heavy-duty machinery. Therefore, the position estimations from the beacon system, inertial measurement unit, and wheel encoders are fused using the Kalman filter to improve the accuracy of the localization to an acceptable level.
The scope of this paper is limited to the design and development of a robotic benchmarking system for a robot-aided ship hull hydro blasting process. The proposed system is capable of synthesizing a benchmarking map for a previously blasted ship hull. Therefore, the work presented in this paper could contribute to improving the dry dock maintenance industry. Investigations on the development of methods for efficient path planning for a blasting robot and determination of optimum pressure for a selective blasting based on the information of a synthesized benchmarking map are proposed for future work.
In the current practice, the blasting quality is inspected by human experts for the decision making process. The blasting quality is determined based on qualitative factors of a blasted surface. This paper proposed a method to automate the inspection process. Nevertheless, the expert knowledge has to be captured for labeling the data set since a labeled data set is required for the training of the proposed classifier. Manual labeling of a large data set is a time-consuming task and would compromise the accuracy of the classifier if the expert knowledge was not captured accurately. During the labeling process, two human experts were asked to label the images independently. The label of an image is determined based on the unanimous decision of the human experts. Thus, this strategy ensures the correctness of the captured expert knowledge through manual labeling of the data set. The usage of unsupervised learning methods for the classification would alleviate the requirement of a labeled data set. The investigation on the possibility of using unsupervised learning methods for the classification is proposed for future work.
5. Conclusions
Routine dry dock maintenance work on ship hulls is essential for the efficient and sustainable operation of the shipping industry. In this regard, hydro blasting is one of the major maintenance work on ship hulls. Robots and systems have been developed for the dry dock maintenance industry to overcome the shortcoming of conventional methods.
This paper proposed a novel robotic system for benchmarking the robot-aided hydro blasting on ship hulls. The proposed robotic system is capable of synthesizing a benchmarking map that indicates the quality of a previously conducted hydro blasting in area wise. A novel Self-Organizing Fuzzy logic (SOF) classifier has been developed to realize the classification of blasting quality similar to human expert knowledge. A multipurpose inspection and maintenance robot called Hornbill, which can perform hydro blasting, benchmarking, and painting, has also been developed to facilitate a fully integrated system.
The SOF classifier has been trained and tested with a set of image data collected from the robot’s camera and labeled based on expert knowledge. The variation of the classification accuracy and the execution time of the SOF classifier with granularity level and distance/dissimilarity measure type has been studied to evaluate the classification performance. The highest achieved mean classification accuracy was 0.9942. The mean execution time of the classifier in the corresponding case was 8.67 µs, which indicates a trivial computational overhead to the system.
Real-time experimenting with the proposed robotic system has been conducted on a ship hull by navigating the benchmarking robot on a given path. According to the results, the proposed robotic system is capable of synthesizing a benchmarking map for a previously conducted hydro blasting process. A synthesized benchmarking map can represent the blasting quality of different areas in multilevel as similar to human expert categorization.
This benchmarking map can be used for planning an efficient navigation path for a blasting robot (for reblasting if a surface was not properly blasted) and by facilitating the spot blasting with different parameter settings based on identified blasting quality of the corresponding area. In addition to that, it can be used to verify whether the area of interest is adequality blasted before starting the painting. Therefore, the proposed robotic system would be highly beneficial in improving the quality and efficiency of hydro blasting work on the ship hull maintenance industry. Developments of methods for optimizing a reblasting process (robot-aided) based on a synthesized benchmarking map are proposed for future work.