Machine Learning-Based Image Processing for Ice Concentration during Chukchi and Beaufort Sea Trials

Abstract

Growing interest in finding the optimal route through the arctic ocean, and sea ice concentration is also emerging as a factor to be considered. In this paper, an algorithm to calculate the sea ice concentration was developed based on the images taken during the Arctic voyage of the Korean icebreaker ARAON in July 2019. A sea ice concentration calculation program was developed using the image processing functions in open-source image processing library, called OpenCV. To develop the algorithm, parameter studies were conducted on red, green, blue (RGB) color space and hue, saturation, value (HSV) color space, and k-means clustering. To verify the algorithm for sea ice concentration calculation, it was applied to images taken during Araon’s Arctic voyages. Lens curvature and view point were corrected through camera calibration. To improve the accuracy of sea ice concentration calculation, a binarization model based on random forest was proposed. A parameter study for training image numbers and tree numbers was conducted to establish the random forest model. The calculated sea ice concentrations by random forest and k-means clustering were compared and discussed.

1. Introduction

Sea ice is the most active factor in Arctic climate and its impacts. Recently, due to the reduction of Arctic sea ice due to global warming, studies for resource development and commercialization of the Arctic Sea Route are being actively conducted. Routes through the North Sea route can reduce fuel and pollutant emissions due to reduced travel distance [1,2]. In particular, from a resource development perspective, the Arctic contains more than 13% of the world’s undiscovered oil, 30% of gas, and several times more hydrates than gas [3]. Since various ice conditions affect the working performance of offshore structures [4,5,6], it is considered important to develop an accurate ice concentration model for the development of the Arctic route and ensure safe and efficient navigation in the Arctic Ocean. Silva et al. [7] predicted the sea ice concentration using a mesoscale eddy-resolving ice ocean coupled model to reduce an uncertainty in the narrow region prediction of a general numerical model. Liu et al. [8] implemented a dynamic system through satellites for sea ice concentration observation and sea ice concentration prediction. Kim et al. [9] developed a sea ice change prediction model using artificial satellites and convolutional neural networks (CNN), a deep learning neural network class.

In addition to sea ice concentration models, a variety of studies related to sea ice concentration have been conducted. These studies employ various methods, ranging from image transformation to machine learning and artificial neural networks, with the aim of accurately extracting and predicting information. Muramoto et al. [10] proposed a method to analyze sea ice concentration and floe size by transforming images captured in polar regions into orthographic projections using nearest-neighbor and bilinear interpolation procedures. Lu et al. [11] developed a method to correct geometric distortion in tilted camera images to accurately calculate ice concentration and floe size. Zhang et al. [12] measured the ice concentration using the hazard Otsu method, k-means clustering, and the gradient vector flow (GVF) snake algorithm. The GVF Snake algorithm was used to detect individual glaciers and measure the floe size distribution. Kalke et al. [13] developed an automated image processing method that could accurately calculate the ice concentration using a machine learning algorithm, support vector machines (SVM). Choi et al. [14] applied an artificial neural network (ANN) for short-term prediction of the Arctic sea ice concentration. Han et al. [15] used a random forest to retrieve sea ice concentrations in the Pacific Arctic Ocean. Singh and Bose [16,17] suggested state-of-the-art image techniques for early COVID-19 pneumonia. Zhang et al. [18] utilized a convolutional network and combined it with shallow layer features for sea ice semantic segmentation. Previous studies have analyzed sea ice concentration using satellite images or high-resolution images. However, the objective of the present study is to calculate real-time ice concentration during Arctic navigation by capturing high-resolution videos from the bow camera. As a result, the aim is to choose a binarization method that balances computational efficiency and accuracy.

Accurate prediction of sea ice concentration is crucial for Arctic research and industries. Therefore, the objective of the present study were (1) to propose a novel algorithm for the calculation of sea ice concentration, (2) compare the ice concentration algorithms and select an optimal one, and (3) calculate the concentration of sea ice based on data from the 2019 Arctic Ocean voyage in the Chuckchi Sea and Beaufort Sea of the ARAON, Korea’s first icebreaker research vessel. OpenCV, an open-source image processing library developed by Intel, was utilized to develop the algorithm. The algorithm developed can be applied to future Arctic footage to collect annual data on ice concentration in the region. Additionally, it can find utility in the design of icebreaking vessels. The machine learning algorithm used for implementation was random forest, and it was executed using the Scikit-learn (sklearn) library in Python.

2. Algorithm for Ice Concentration Calculation

Figure 1 shows the algorithm for the sea ice concentration. To process the image in real time, a 4k-size image was reduced to a full HD-size image using a bilinear interpolation method. The vessel was excluded from the image and a region of interest (ROI) was arbitrarily set around the vessel. The image was converted into a single-channel image considering the characteristics of seawater and then binarized using a thresholding method. An optical lens with a wide field of view (FOV) was used to see a wide range. Camera calibration was needed to correct the resulting image distortion. Camera correction was performed after image binarization, as errors in blurred boundary pixels could be magnified during image correction and could affect the sea ice concentration calculation [12]. Geometric orthorectification was performed to obtain a vertical view of seawater. After that, the sea ice concentration was calculated by the ratio between the binarized pixels of seawater and sea ice.

2.1. ROI

Figure 2 is an image taken by a camera mounted on the bow of the icebreaker ARAON. Seawater and sea ice can be distinguished by color, saturation, and brightness. Seawater has a blue color with high saturation and low brightness, and the surface of sea ice has low saturation and high brightness. The image ROI was designated as the lower left part of the red border area shown in Figure 2.

2.2. Single Channel Image Conversion & Binarization

2.2.1. RGB Color Space

The red, green, and blue (RGB) color space means expressing an image with red, green, and blue colors, and each pixel in the Cartesian coordinate system is expressed by the three-color values. The biggest characteristic that distinguishes seawater from the sea ice is blue color of seawater. The boundary was divided by extracting color values similar to the seawater region from the image. Only the blue color of the RGB color space was extracted from the image and binarized into seawater and sea ice using the image binarization method. To accurately distinguish the boundary between sea ice and seawater, where sea ice is floating or submerged in seawater, various binarization methods were compared to obtain an optimal value for sea ice boundary extraction. The thresholding method uses the intensity of image pixels to extract image features. Since the ice surface has a higher intensity value than that of seawater, it can be extracted through the thresholding method [19].

2.2.2. HSV Color Space

The hue, saturation, and value (HSV) color space divides pixels into color spaces for hue, saturation, and value. It is a region extraction method that can express color information using a uniform color space within the region. At the sea level, the colors are vivid and the saturation is high. On the other hand, the surface of the sea ice is bright due to light reflection and has high brightness. The area is divided based on the value of 240, because seawater is blue. In OpenCV, the value divided by 2 is used as the color value due to the limitation of the data type size. Therefore, the image is binarized based on the blue value of 120.

2.2.3. K-Means Clustering

Color-based methods have a problem in that a feature detection result changes depending on lighting. To divide the data into several groups, the k-means clustering was used to form clusters of each group. The k-means clustering algorithm [20] is a type of an unsupervised learning algorithm and does not require a separate training process. In general, the initial number of clusters is set empirically, but in image binarization, only sea ice and seawater need to be distinguished, so initially k is fixed at 2. The overall variance $V$ of the k-means clustering algorithm is calculated as follows.

$$V={{\displaystyle \sum }}_{i=1}^k{{\displaystyle \sum }}_{X_j\in S_i}{\left|X_j-{\mu }_i\right|}^2$$

(1)

$$S_i^{\left(t\right)}=\left\{x_p:{\left|x_p-{\mu }_i^{\left(t\right)}\right|}^2\le {\left|x_p-{\mu }_j^{\left(t\right)}\right|}^2\forall j,1\le j\le k\right\}$$

(2)

$${\mu }_i^{\left(t+1\right)}=\frac{1}{\left|S_i^{\left(t\right)}\right|}{{\displaystyle \sum }}_{x_j\in S_i^{\left(t\right)}}{x}_j$$

(3)

Here, the center of the $i$-th cluster is ${\mu }_i$, the set of points belonging to the cluster is $S_i$, and the goal of the algorithm is to find the $S_i$ value that minimizes the $V$ value. The algorithm sets the initial cluster center, calculates the distance from each data to the center of each cluster, finds the closest cluster in the data, and repeats the process of re-centering. If the cluster does not change, the repetition is stopped. The Euclidean distance from each data point to the centroids $({\mu }_i)$ of respective clusters is calculated. Subsequently, the data points are assigned to the cluster with the closest centroid. Following this assignment, each ${\mu }_i$ is redefined as the weighted centroid of the data points within the corresponding cluster.

2.2.4. Random Forest

Random forest is an ensemble model that combines multiple decision trees to make the final prediction, and it is mainly used for regression problems. Figure 3 shows the flow chart of sea ice concentration using random forest. To train a selected model, the original images and the binarized images, which accurately distinguished between seawater and sea ice, were required. To obtain high-accuracy binarized images, k-means clustering with k = 3 or bigger number was used. Finally, only the training images in which seawater and sea ice were clearly distinguished by the eyes of the authors were selected. Figure 4 shows the supervised learning-based random forest. By averaging the results of multiple decision trees, it could avoid the problem of overfitting to the training images, which was a drawback of individual decision trees, and the algorithm was generated by extracting decision trees using bootstrap sampling [21].

2.3. Camera Calibration

The sea ice images were obtained using an optical camera. The camera’s angular field of view (FOV) is H: 100.3° (Wide)~45.6° (Tele) at 4096 × 2160. To accurately obtain sea ice concentration, it is necessary to correct the distortion caused by the camera lens. Camera calibration plays an important role in the process of clearly linking the mapping relationship between the 3D real world and the 2D image. The geometric interpretation of the image is based on a basic pinhole camera model that is projected as an image through an external disparity pinhole.

The captured images were converted into images viewed vertically from the sea level through geometric orthorectification. The camera located at the bow was mounted to take pictures at an angle.

3. Sea Ice Concentration

Figure 5 shows sea ice and seawater in the ROI set in Figure 2. The image was converted into a single-channel blue image. Sea ice overlaps each other and some of the sea ice is covered with seawater.

In the single channel blue image, sea ice and seawater were expressed as pixel intensity. To find the optimal threshold value, binarized transformation was performed for different threshold values of 50, 100 and 150, and OTSU method [19] as shown in Figure 6. If the threshold value was low, sea ice and seawater could not be distinguished. Conversely, if the threshold value was high, the part of submerged sea ice in seawater was judged as seawater. Therefore, the threshold value was selected as 100, which most accurately divided the sea ice and seawater boundary. The image binarization with the optimal threshold value of 100 contained the submerged part of sea ice better than the OTSU method. Although sea ice and seawater could be distinguished even with single channel blue image, a region similar to blue appeared as noise in some sea ice or sea ice and seawater boundaries.

The image was converted to the single channel image by HSV color space and binarization was performed according to the different threshold ranges.

Table 1. Threshold ranges for binarization of HSV color space.

	Case 1	Case 2	Case 3	Case 4
Hue	100~140	95~145	90~150	85~155
Saturation	10~255	20~255	30~255	40~255
Value	10~255	20~255	30~255	40~255

shows the threshold ranges.

Figure 7 compares the binarization results according to the hue range. Because sea ice showed blue light when submerged in seawater, the hue value was tested by widening the range around 120, which represents blue light in OpenCV. The difference between the hue value ranges was not significant. The wide range of hue value was selected because blue light may look different when the weather becomes worse.

Figure 8 compares the binarization results according to the saturation range. There was no change in the saturation above 40, thus the maximum was selected as 40. When the minimum saturation was low, the boundary between sea ice and seawater was not well expressed. The saturation range was selected from 40 to 255.

Figure 9 compares the binarization results according to the value range. There was no change in the value below 20. If the minimum value exceeded 40, an error occurred in determining seawater as sea ice. Therefore, the value range was selected from 10 to 255.

Figure 10 compares the binarized images with optimal values of RGB color space and HSV color space. Both methods captured the boundary line between sea ice and seawater well. The results of binarization with HSV color space captured the sea ice surface better. However, such a method might be affected by illuminance depending on the time and the weather.

Figure 11 shows the binarized image using k-means clustering. The binarized image through the k-means clustering distinguished the sea ice and the seawater boundary well. However, if k-mean clustering was calculated with k = 2, sea ice submerged in seawater was treated as seawater not sea ice. Therefore, the clustering with k = 3 was divided into three categories: sea ice, sea ice submerged in seawater, and seawater, and then sea ice and the submerged sea ice region were combined. When calculated as the clustering with k = 3, sea ice and seawater boundaries were accurately separated and the sea ice area was extracted similarly to the actual area.

Figure 12 compares binarized images with optimal value of HSV color space and k-means clustering. The HSV color space captured the sea ice area well, but sometimes the sea ice and seawater boundaries were ambiguous. In k-means clustering, the sea ice and seawater boundaries were also clearly distinguished.

Camera calibration was performed on the binarized image. The curvature of the lens and the angle of the camera were corrected. Then, geometric orthorectification was carried out. Figure 13 compares the images before and after camera calibration. After completing camera calibration, the image was transformed into a vertical view of the sea surface, indicated by the red box. The boundary between sea ice and seawater was well distinguished and the sea ice concentration could be calculated.

4. Sea Ice Concentration Using Random Forest Machine Learning

The k-means clustering with k = 3 showed high accuracy in distinguishing the ice and water regions. However, in areas where it was difficult to visually identify sea ice and seawater boundary, the accuracy of extracting sea ice concentration decreased. Figure 14 shows an image where submerged sea ice exhibited almost an identical color to seawater. In cases where the color of submerged sea ice was similar to those of seawater, making it difficult to distinguish visually, even the k-means clustering with k = 3 could not accurately determine the sea ice concentration.

To address these issues, random forest [22], one of the machine learning algorithms, was selected. To set high-accuracy training images, k-means clustering with k = 3 or bigger number was applied and then the authors selected the training images. Figure 15 shows the binarized images for different decision trees. To evaluate the similarity between the binarized images, the structural similarity index measure (SSIM) was employed [23]. The binarized images showed a similarity of 99.51% using 20 and 100 decision trees, indicating that the number of decision trees had almost no effect on improving the accuracy of sea ice concentration calculation. Finally, 20 decision trees were used because they had a minimal impact on measurement accuracy.

Figure 16 compares the binarized images for different training images. When the number of training images was insufficient, the darkened areas near the frame of the image due to shadows were recognized as seawater, while when the number of training images was sufficient, the sea ice area was accurately distinguished. Therefore, it was confirmed that the quality and quantity of training images play an important role in model creation. One hundred images were used for training and 20 images were used for validation. In the end, 120 images were used to develop the random forest model. Figure 17 shows the comparison the SSIMs for the binarized images by random forest, alongside the x = y line. The red dashed line depicted in Figure 17 signifies the x = y line, while the black dots represent the SSIM values corresponding to each data point. The blue dashed line separates the training set from the validation set. Data from 1 to 100 represent the training set, while data from 101 to 120 represent the validation set. All data exhibit a high image similarity, with an SSIM of 97.14% or higher.

In Figure 18, the binarized images by k-means clustering with k = 3 and random forest were compared. The binarized images by k-means clustering had difficulty accurately distinguishing between sea ice and submerged sea ice. On the other hand, the binarized images produced by the random forest algorithm showed better results than those produced by k-means clustering, due to the model’s ability to distinguish between seawater and sea ice. In addition, the calculation of sea ice concentration using random forest is advantageous for real-time image processing on a ship because its computation time is significantly shorter than that of k-means clustering with k = 3. Figure 19 shows the calculated ice concentration by random forest. The ice concentration was calculated by pixel-based classification.

5. Concluding Remarks

To calculate the sea ice concentration during polar operation on the Korean icebreaker ARAON, an algorithm was proposed. The algorithm was developed as a program using OpenCV. The algorithm for sea ice concentration was applied to the images taken from the ARAON sea trials. To distinguish sea ice and seawater, images was converted into single-channel and binarized image using RGB color space, HSV color space, and k-means clustering. When the k-means clustering with k = 3 was used, the sea ice and seawater boundary were extracted well. To achieve precise calculation of sea ice concentration, camera calibration was performed to ensure that the images captured appear vertically aligned with the sea surface. k-means clustering provided the best results. It was observed that the accuracy decreased for images where the submerged sea ice had similar color. Random forest, a machine learning model, was used to create an image binarization model, which was trained using high-accuracy images obtained through k-means clustering. The number of trees did not significantly affect the results, but the accuracy of the model was influenced by the quality and quantity of the training images. Random forest outperformed k-means clustering in calculating sea ice concentration, while also reducing computation time. The accuracy of predicting sea ice concentration could be decreased due to weather effects or shadows from clouds. However, suggested ice concentration algorithm could improve the integration calculation results through the random forest method that learned enough input images. Therefore, random forest was shown to be effective for sea ice concentration binarization. In the future, ice concentration will be used to estimate the external forces acting on icebreakers and Arctic navigation. Future work will include models that use deep learning algorithms to calculate ice concentration.