Advancing Arctic Sea Ice Remote Sensing with AI and Deep Learning: Opportunities and Challenges

Abstract

Revolutionary advances in artificial intelligence (AI) in the past decade have brought transformative innovation across science and engineering disciplines. In the field of Arctic science, we have witnessed an increasing trend in the adoption of AI, especially deep learning, to support the analysis of Arctic big data and facilitate new discoveries. In this paper, we provide a comprehensive review of the applications of deep learning in sea ice remote sensing domains, focusing on problems such as sea ice lead detection, thickness estimation, sea ice concentration and extent forecasting, motion detection, and sea ice type classification. In addition to discussing these applications, we also summarize technological advances that provide customized deep learning solutions, including new loss functions and learning strategies to better understand sea ice dynamics. To promote the growth of this exciting interdisciplinary field, we further explore several research areas where the Arctic sea ice community can benefit from cutting-edge AI technology. These areas include improving multimodal deep learning capabilities, enhancing model accuracy in measuring prediction uncertainty, better leveraging AI foundation models, and deepening integration with physics-based models. We hope that this paper can serve as a cornerstone in the progress of Arctic sea ice research using AI and inspire further advances in this field.

1. Introduction

Arctic sea ice is integral to the global ecosystems of the Northern hemisphere as it serves as a vital climate regulator, helping to maintain the Earth’s surface temperature by reflecting sunlight back into space. Sea ice also provides an essential habitat for numerous wildlife species and plays a key role in global ocean circulation, influencing ocean currents and weather patterns on a global scale. In recent decades, the extent of Arctic sea ice has declined at an unprecedented rate of nearly 13% per decade, particularly by the end of the Arctic summer season [1]. Arctic sea ice has also thinned considerably and shifted towards being younger season after season. According to the IPCC (Intergovernmental Panel on Climate Change), since 1979, the area covered by 5-year-old ice has decreased by approximately 90% [2]. The loss of Arctic sea ice has resulted in concerning consequences, including increases in coastal disasters [3,4], altered weather patterns [5], disruptions in ecosystems [6], and impacts on indigenous communities [7]. Therefore, it is crucial to develop new monitoring and analytical capabilities to advance our understanding of sea ice dynamics, the changes resulting from recent warming, and their impact on the global environment and climate.

Remote sensing has been used as an important vehicle for collecting spatiotemporal information about sea ice properties and conditions in the polar regions, e.g., [8,9]. Traditional methods for remote sensing image analysis in Arctic sea ice research have relied on statistical approaches [10]. While popular, these methods also suffer from notable limitations. For example, statistical approaches are sensitive to data quality. Errors and noise in satellite imagery and data value inconsistency among different data sources may lead to inaccuracy in the analytical results [11]. Meanwhile, statistical approaches often involve point-based analysis, and they cannot handle big spatial data which have high spatial and temporal resolutions effectively. In addition, while statistical approaches (for example, ordinary least square regression and Bayesian statistics) are good at understanding cause-and-effect relationships, their prediction capability for unseen datasets is limited. These approaches often assume certain relationships between the independent and dependent variables, such as linear or logistic relations, and they therefore cannot capture complex relationships within the data. In comparison, as a new kind of statistical method, shallow machine learning, such as random forests and support vector machines, have overcome some of these issues to better handle big and complex datasets and achieve good prediction accuracy and efficiency [12]. However, these methods often require feature engineering to manually extract important features that will affect the outcome from the raw data [13]. The manual process is time-consuming and an incomplete set of selected features will result in reduced accuracy in the analytical results.

Object-based image analysis (OBIA) is also an important technique in sea ice remote sensing research [14]. It operates by automatically segmenting images into image regions and then conducting further classification based on predefined rules. This method also involves manual work, which suffers from limited generalizability when applied to diverse study areas and datasets. Fortunately, advances in artificial intelligence (AI), especially deep learning, have brought about fundamental changes by enabling effective data-driven analysis [15]. In particular, the revolutionized development of convolutional neural network (CNN) models allows for the automated extraction of both low-level and high-level image features critical for detecting new phenomena and events, as well as classifying different types of objects within an image [16]. The local, convolutional operations of CNN models also allow for parallel processing, making it possible to handle extensive geospatial data to extract generalizable patterns and trends. In recent years, Arctic researchers have increasingly embraced deep learning technology in sea ice research. Figure 1 provides an overview of the integration and customization of deep learning models to support Arctic sea ice research.

Excitingly, pioneering research has been pursued in multiple sea ice applications, such as sea ice lead detection, sea ice thickness estimation, sea ice concentration estimation and forecasting, sea ice motion detection, and sea ice type classification. In addition to CNN models, several deep learning feature extraction backbone models have been developed (Figure 1, Module named “Deep learning (backbone models)”). This includes the multi-layer perceptron [17] model, which estimates the classes (e.g., sea ice type) of an image pixel or an image patch based on extracted feature. The recurrent neural network [18] is designed for processing time-sequence input for forecasting tasks (e.g., predicting sea ice motions). A transformer-based model uses a self-attention mechanism to capture spatial and temporal data dependencies over a long range, achieving even more powerful pattern analysis and extraction than many CNN models [19]. Transformer-based models were originally designed for processing time-sequence data and have recently been adopted in the image analysis and computer vision domain through the development of vision transformers (ViT) and their variants. This new architecture, while computationally intensive, can be adapted for both image segmentation and forecasting tasks. Image segmentation, such as in the estimation of sea ice concentration, can also be achieved through an interesting deep learning model architecture called a generative adversarial network (GAN) [20]. Instead of aiming for information extraction and pattern recognition, the GAN is a type of generative AI. It introduces a generative network to generate image results as close to the expected results as possible and uses a discriminative network to evaluate them. In addition to the visual models, graph convolutional neural networks (GCN) [21] have also been employed to use graph theory to guide the topological analysis of key sea ice features, such as sea ice leads, to enhance the detection of ice ridges and their changes.

Based on these backbone models, various deep learning architectures (e.g., U-Net) have been applied to support sea ice research based on the problem formulation (Figure 1, Module named “Deep learning applications”). For example, certain research (e.g., sea ice concentration estimation) can be formulated as a spatial problem aiming to segment remotely sensed images at a pixel or patch level. Other research can be formulated as a spatial and temporal problem for forecasting based on time-sequence image data. The same research can also be formulated differently and solved using various architectures.

This paper will organize the review according to the application of these deep learning models in different sea ice research areas (Section 2). Differing from other reviews, our paper focuses on AI, particularly the applications of deep learning in various subareas of sea ice research, offering a detailed analysis of the integration of these two fields. Section 3 further discusses how new learning strategies are customized in the sea ice-deep learning solution framework (Figure 1, Module on “New techniques integrated”). Section 4 presents our perspective on the future advances that can be achieved through a deep integration of AI and sea ice research. Section 5 concludes this paper and discusses action items for continuing to foster growth in this exciting interdisciplinary field.

2. Deep Learning Applied to Sea Ice Remote Sensing

In this section, we will introduce various applications of deep learning to advance Arctic sea ice research. We categorize the papers by types of deep learning problems, techniques, input, and output. Each subsection includes a table summarizing the available research in each application area.

2.1. Sea Ice Lead Detection

Sea ice leads, which are distinct features of the polar seascape, are narrow, linear openings in sea ice that expose the ocean beneath [22]. They vary in size, ranging from meters to several kilometers in width and can extend up to hundreds of kilometers in length. Their detection and monitoring are important due to their multifaceted role in both ecological and climatological contexts. From an ecological perspective, these leads serve as vital habitats and migration paths for numerous Arctic species such as seals and polar bears, playing an essential role in sustaining polar biodiversity. From a climatological viewpoint, leads contribute significantly to the exchange of heat, moisture, and momentum between the ocean and atmosphere, influencing regional weather, sea ice dynamics, and global climate patterns. Additionally, accurate lead detection also aids in polar marine navigation, offering safer and more effective shipping routes.

The application of deep learning in sea ice lead detection primarily leverages satellite-based remote sensing data due to its comprehensive temporal and spatial coverage. Researchers employ a range of data types, from optical satellite imagery and synthetic aperture radar (SAR) data. In optical satellite imagery, thermal emissive bands are commonly used for lead detection, as leads exhibit higher temperatures relative to the surrounding ice due to their reduced albedo and increased solar absorption. Widely utilized instruments include MODIS and VIIRS [23,24,25,26]. However, their relatively moderate spatial resolution (∼500 m) makes it difficult to use in identifying narrow leads. Recent advancements in satellite technology have yielded optical image products such as Landsat-8 [27,28], Sentinel-2 [25], and SDGSAT-1 [25], providing the spatial resolution (∼10–30 m) needed for narrow leads detection.

While optical satellite images offer extensive coverage and ease of access, data collection can only occur in clear sky conditions and in the presence of solar illumination. In comparison, SAR data can be obtained at fine spatial resolution and can collect data in cloudy or dark conditions, yet being sensitive to surface roughness and susceptible to speckle noise. Notable examples of SAR data instruments encompass Sentinel-1 [25,28,29,30,31], RADARSAT-2 [32], and HY-2B [30]. To enhance analytical capabilities, additional data types are generally also included in the analysis, including sea ice concentration (SIC) products from the National Snow and Ice Data Center (NSIDC) for the ice pack mask [23], ERA5 meteorological data to provide local atmospheric forcing [25,26,28], sea ice velocity and deformation rate calculated from the buoy measurements to inform sea ice conditions [28], and CryoSat-2 to offer sea ice thickness data for additional context [32].

The primary outputs of deep learning applications in sea ice lead detection are lead maps or masks that identify and delineate sea ice leads from other ice types. Such maps or masks are usually generated at the same spatial resolution as the input data, providing a pixel-wise classification where each pixel is labeled according to its identified class: ‘lead’, ‘non-lead’, or various other ice types in the case of multi-class classification [23,29,32]. In applications seeking to increase the spatial resolution of certain optical satellite data, the outputs can include high-resolution reconstructions of these datasets. Deep learning techniques can be applied to upscale low-resolution images while preserving important details and enhancing the visibility of fine-scale features such as sea ice leads [26]. Another interesting application of deep learning in sea ice lead detection involves network analysis [33]. By modeling the lead network as a graph, various network analysis methods can be applied to understand its temporal change (e.g., shrinking or expanding).

In terms of deep learning methods, semantic segmentation is the method of choice in many sea ice lead detection studies because of its ability to classify each pixel in an image, thereby providing a detailed and comprehensive representation of leads amidst large ice cover. In terms of model selection, U-Net has become a popular choice due to its encoder-decoder structure [23,27,34]. The encoder part of U-Net captures the spatial context in the image, while the decoder part enables precise localization for delineating the boundaries of leads. This architecture also includes skip connections, which help in recovering the fine-grained details that might be lost during the downsampling in the encoder part. Dulam et al. [35] applied the U-Net architecture, which takes multiple modality data (e.g., visible-band WorldView and C-band Sentinel) as input, and performed accurate lead predictions with a small amount of training data. Liang et al. [29] developed an entropy-weighted network (EW-Net) by improving the baseline U-Net with additional modules, such as dense blocks. Dense blocks, originating from DenseNet [36], strengthen feature propagation and increase feature reuse, leading to improved gradient flow and model compactness. With these improvements, EW-Net has demonstrated better model predictive performance than the baseline U-Net model.

In addition to U-Net, researchers have explored other techniques to separate leads from ice floes, including both traditional machine learning approaches such as thresholding [25,30], random forest [28], the waveform mixture algorithm [30], and deep learning models such as multilayer perceptron [31,37] and DeepLabv3 [38]. Dawson et al. [32] formulated lead detection as a classification problem and employed 1D CNN to classify leads at the sampled data points. The authors compared their approach with a decision-tree method and found that the CNN-based approach led to fewer false positives and, therefore, higher classification accuracy. For data preprocessing, such as super resolution, a model may include a dedicated branch for upscaling the input data [26].

Table 1. Deep learning solution techniques for sea ice lead detection.

Sea Ice Application	Deep Learning Problem Formulated	Deep Learning Techniques (Models)	Output	References
Sea ice lead detection	Multi-class, pixel-wise classification	Semantic segmentation (U-Net, DeepLab, EW-Net)	Lead maps or masks that delineate sea ice leads from non-lead area	[23,27,29,34,35,38]
	Network analysis	Graph convolutional network (EvolveGCN)	Statistics to reveal the change (e.g., expansion or shrinking) in the sea ice lead network	[33]
	Point-based classification	Convolutional neural network (1D CNN), Multilayer perceptron (MLP)	Point type classification (such as lead or ice floe) at a certain sampled location	[31,32,37]

summarizes the main deep learning techniques and methods to support sea ice lead detection.

2.2. Sea Ice Thickness (SIT) Estimation

Sea ice thickness modulates the exchange of heat, moisture, and gases between the atmosphere and ocean, significantly impacting ocean circulation, sea level, and both marine and terrestrial ecosystems. The depth of sea ice is also an important climate change indicator. In the context of global warming, both Arctic and Antarctic sea ice have been undergoing significant changes, with sea ice thickness generally decreasing over the years [39,40]. Therefore, accurate and efficient methods for estimating sea ice thickness are essential to monitor and predict these changes, enabling scientists to better understand the ongoing alterations in our climate system and their potential impacts.

Within the realm of sea ice thickness estimation using deep learning, various types of data can be used. The brightness temperature data sourced from AMSR2 [41,42] provide a primary dataset, measuring the radiation naturally emitted by the sea ice within the microwave spectrum and exhibiting a strong correlation with sea ice thickness. The advantage of AMSR2 data is its ability to capture information under all weather conditions and during both day and night, offering continuous and reliable coverage. Additional input data encompass satellite products, including optical [32] and SAR [32,43], capturing surface features of sea ice and indirectly inferring its thickness. Furthermore, thermodynamic parameters such as air temperature, sea surface temperature, wind speed, and snow depth, derived from reanalysis data such as ERA5 [44,45], can suggest sea ice thickness indirectly, reflecting the conditions conducive to ice formation and evolution. The strength of reanalysis data lies in their global coverage and continuity, but they are model-derived and not direct measurements, so they may contain biases or uncertainties.

To train deep learning models effectively, ground truth data is indispensable for learning the correlation between input data and sea ice thickness. Ground truth data can be obtained from in situ measurements such as AWI Moored Upward Looking Sonar (ULS) data [43] or calculated from CryoSat-2 data [32,41,44], which are explicitly designed to monitor polar ice thickness fluctuations and variations in sea ice. Numerical simulation models like Pan-Arctic Ice Ocean Modeling and Assimilation System [46] provide another source of sea ice thickness data, despite potential limitations and uncertainties [45]. In terms of output, deep learning models typically generate a continuous map of sea ice thickness, usually in the form of gridded datasets representing the estimated sea ice thickness for specific geographic locations. The resolution of these datasets is contingent on the input satellite data and deep learning model specifics. These datasets then serve as foundations for subsequent analysis and climate model input.

Sea ice thickness estimation is typically addressed as a regression problem, given the output’s continuous nature. The goal is to predict a continuous value, the sea ice thickness, using the input data. Traditional machine learning models have been widely utilized for this purpose [43,47,48,49,50]. For instance, Shamshiri et al. [43] applied random forest regression to estimate sea ice thickness, leveraging input data features such as backscatter intensity, texture, and polarization from Sentinel-1. Similarly, CryoSat-2’s radar altimeter readings can be converted to ice thickness, aided by knowledge of ice and snow densities [47]. Random forest regression was applied in this study as well.

In recent years, there has been a shift towards the use of deep learning models for sea ice thickness estimation because of their enhanced capacity to model intricate relationships, handle extensive high-dimensional data, and learn useful features autonomously [51]. A typical solution is to formulate the problem as regression analysis. Herbert et al. [48] applied a regression-based fully connected neural network to estimate SIT from multiple input features, such as brightness temperature, surface temperature, and sea ice concentration. Yan and Huang [49] developed a CNN model to fuse image features and create sea ice thickness products from multi-source satellite data, such as AMSR2 and the Soil Moisture Ocean Salinity (SMOS) dataset. Similarly, Chi and Kim [41] advanced the model by developing an ensembled 1D CNN for SIT estimation, and the results were shown to achieve high consistency with reference data. Liang et al. [44] developed a self-attention CNN to incorporate thermodynamic parameters to estimate the daily SIT in the Arctic winter. The self-attention module was employed to explore the hidden connections among the extracted features to achieve a more accurate estimation.

SIT estimation has also been addressed as a clustering problem. Moreau et al. [52] integrated deep learning with Bayesian inference to estimate sea ice thickness by analyzing icequake waveforms. A deep scattering network was applied to convert the original data space to an abstract space where feature similarity can be better represented. These abstracted features were then clustered to create sub-families of icequake waveforms. Based on this result, Bayesian inversion was further applied to infer the ice thickness. SIT estimation can also be cast as a forecasting task [53], using temporal models such as PredRNN++ to predict future sea ice thickness [45]. By introducing a spatiotemporal memory flow, PredRNN [54] is capable of capturing the decoupled long- and short-term dynamics and the spatiotemporal variations in sequence data.

Table 2. Deep learning solution techniques for sea ice thickness estimation.

Sea Ice Application	Deep Learning Problem Formulated	Deep Learning Techniques (Models)	Output	References
Sea ice thickness estimation	Regression	Regression fully connected neural network	Thematic maps showing sea ice thickness	[48]
		Convolutional neural network (1D CNN, attention-based CNN)		[41,44,49]
	Clustering	Deep scattering network		[52]
	Time-series forecasting	Recurrent neural network (PredRNN++), hybrid deep learning model	Near-term forecast of sea ice thickness, creating continuous sea ice thickness maps in future timestamps	[45,53]

summarizes the main deep learning techniques and methods to support sea ice thickness estimation.

2.3. Sea Ice Concentration (SIC) Estimation

Sea ice concentration (SIC) is a measure of the proportion of sea area covered by ice, usually expressed as a percentage. For climate scientists, SIC estimation is a key metric for understanding the impacts of global warming, as it directly affects the Earth’s energy balance. For the shipping industry, accurate predictions of SIC can enhance navigation safety and efficiency in ice-infested waters. Thus, timely and accurate SIC estimation is of great importance.

Datasets from AMSR2, RADARSAT2, and Sentinel-1 are frequently leveraged for SIC estimation via deep learning algorithms, given their unique capabilities and strengths. AMSR2 offers brightness temperature data, which is highly relevant for analyzing ice concentration [55,56,57,58]. As already mentioned, the advantage of AMSR2 lies in its cloud penetration capacity, which ensures continuous data acquisition. Additionally, AMSR2’s multi-frequency operation facilitates the recording of varying ice properties. However, its low spatial resolution limits the detail of ice structure that can be captured. RADARSAT2, on the other hand, delivers SAR data, distinguished by its fine spatial resolution, which allows for capturing detailed ice feature. In addition, its operational independence from sunlight or atmospheric conditions makes it suitable for polar regions monitoring [59,60,61]. Nevertheless, RADARSAT2 data processing is complicated due to the complex nature of SAR signals, including speckle noise. In comparison, the Sentinel-1 mission offers C-band SAR data, which incorporates the strengths of SAR data and some unique benefits. C-band SAR data is sensitive to sea ice changes and is effective in detecting thin ice and open water areas [58,62,63,64,65]. The Sentinel-1 satellites also provide a high revisit frequency, ensuring updated and timely data. Like RADARSAT2, it also requires advanced processing and interpretation of SAR signals.

The ground truth data utilized for training deep learning models for SIC estimation stem from various sources, each bearing unique strengths and weaknesses. For instance, expert-crafted ice charts, such as those from the Greenland Ice Service, serve as invaluable data repositories. These charts incorporate a wealth of human experience and understanding of sea ice, providing context often absent in raw satellite data. However, they are subject to human interpretation and have limited frequency and coverage. Alternatively, satellite-derived products from the MODIS or AMSR-E are beneficial due to their expansive coverage, high temporal frequency, and objectivity. Nonetheless, they can be affected by cloud cover, and their accuracy can be contingent on the ice concentration calculation algorithms. The choice of ground truth data is guided by the specific study requirements, and a combination of different sources may be employed to optimize their strengths and mitigate their weaknesses.

Deep learning-based SIC estimation from satellite data can be treated as either a regression or a classification problem, depending on the task formulation. When viewed as a regression problem, the aim is to predict a continuous variable that indicates the sea ice concentration [55,59]. Despite providing granular sea ice concentration estimates, this method can be more challenging due to the complexity of modeling continuous variables. Alternatively, SIC estimation as a classification problem entails predicting discrete classes like “no ice”, “partial ice”, and “full ice” [60,61], simplifying the problem and potentially offering a more robust model, particularly with limited or noisy data. Semantic segmentation models are commonly used to achieve this goal. For example, U-Net, which is designed specifically for image segmentation, is frequently employed [62,63,64]. Another segmentation model, DeepLab, which uses dilated convolutions to enable multi-scale contextual information aggregation without resolution loss, also shows good performance for the SIC estimation task [56,58]. Super-resolution techniques applied to satellite data enhance SIC classification [56,57] by improving the spatial resolution of data, allowing for the capture of finer ice structure details, thereby potentially improving the SIC classification accuracy.

SIC concentration can also be formulated as a regression model predicting a continuous SIC value [55,59,60]. Chi et al. [55] developed a multilayer perceptron (MLP) deep learning-based regression model to predict the SIC from AMSR2 passive microwave data. The spectral unmixing algorithm was first applied to finer-resolution MODIS imagery to better discern first-year ice from melt ponds, as their spectral signals are often mixed within a single image pixel. This sub-pixel level analysis yields an accurate ice/water classification fractional abundance map. This class information was further mapped to a lower-resolution AMSR2 image to retrieve the percentage of sea ice cover within the image pixel. Based on this data annotation mechanism, the MLP was further trained to predict the SIC values. Results showed better regional and pan-Arctic accuracy than other SIC products. Cooke and Scott [59] applied SAR images and CNN-based regression to derive SIC. Instead of using ice charts as the ground truth, the authors adopted passive microwave data for the estimation of SIC using the ARTIST sea ice algorithm. In addition, patch-based analysis was applied, allowing the model to incorporate more spatial contextual information in its learning process. Other similar works include Nagi et al. [60]. In these works, commonly used CNN architectures include fewer-layer AlexNet and VGG and deeper models such as DenseNet and ResNet [36]. In addition to using CNN models, hybrid image segmentation and CNN-based SIC estimation methods have been developed. For instance, in [65], iterated conditional modes segmentation was first applied to SAR data to generate sea ice segments and reduce random variation caused by speckle. The segmented results were then connected to a CNN model to classify ice from open water and further perform SIC estimation.

Table 3. Deep learning solution techniques for sea ice concentration estimation.

Sea Ice Application	Deep Learning Problem Formulated	Deep Learning Techniques (Models)	Output	References
Sea ice concentration (SIC) estimation	Multi-class, pixel-wise classification	Semantic segmentation (U-Net, DeepLab)	Thematic map indicating SIC types, e.g., “no ice”, “partial ice”, and “full ice” for each image patch	[56,58,62,63,64]
	Regression	MLP, CNN (AlexNet, VGG, ResNet, DenseNet)	Predict a continuous variable indicating SIC	[55,59,60]
	Hybrid image segmentation and CNN approach	Iterated conditional modes (ICMs) segmentation, CNN	Predict a category as “ice” or “open water”, and/or a continuous variable indicating SIC	[65]

summarizes the main deep learning techniques and methods to support sea ice concentration estimation.

2.4. Sea Ice Concentration/Extent Forecasting

Sea ice concentration (SIC) forecasting involves using advanced methods and models to predict the proportion of sea ice coverage in a given ocean area over time. This process is closely related to sea ice extent (SIE) estimation, as SIE is defined as the area of the ocean with more than 15% SIC. These forecasts are important for several reasons. First, they are vital for ship navigation in polar regions, where unexpected changes in ice conditions can pose significant hazards. Second, they play a critical role in climate science as sea ice influences the heat balance of the Earth. Third, these forecasts contribute to the understanding of the impacts of climate change, as declining sea ice is a key indicator of global warming.

Several sources of data contribute to SIC forecasting. The NSIDC, NASA, and the Japan Aerospace Exploration Agency (JAXA) all provide SIC data through algorithm-based estimation from multiple satellite products [66,67,68,69]. These datasets are very helpful due to their large spatial and temporal coverage. However, their accuracy can be affected by atmospheric conditions and surface melt. Furthermore, discrepancies may arise due to the use of varying sensors, processing algorithms, or calibration methodologies. Reanalysis data incorporates ERA5 and ERA-Interim outputs from the European Centre for Medium-Range Weather Forecasts (ECMWF), creating a coherent long-term record of numerous atmospheric, terrestrial, and oceanic phenomena. Additionally, the Ocean Reanalysis and derived ocean heat content (ORAS4) dataset, another ECMWF product, offers insights into oceanic temperature, salinity, currents, and sea levels. Reanalysis datasets complement SIC forecasts by providing data on atmospheric and oceanic factors influencing sea ice dynamics, including air temperature, wind speed, ocean currents, and sea surface temperature [70,71,72,73]. Ice charts from the Canadian Ice Service (CIS) provide expert-verified information about ice conditions in Canadian waters. They contain information about ice type, concentration, thickness, and stage of development. While they do contain SIC data, it is often in a categorical rather than a continuous format (e.g., “9/10 to 10/10” rather than “95%”). These ice charts can be used as complementary data for SIC forecasting, providing high-accuracy, ground-truth data for model validation or data assimilation [70]. However, their spatial and temporal coverage is more limited than the satellite or reanalysis data.

Deep learning models used for SIC forecasting exhibit common characteristics. The input typically comprises a sequence of historical data points, including past SIC data and relevant atmospheric and oceanic conditions. Preprocessing and normalization of the gridded data are necessary, along with handling missing values through techniques such as interpolation. The output prediction for a future time frame mirrors the input data format, facilitating comparison and evaluation. U-Net, LSTM, and Convolutional Long Short-Term Memory (ConvLSTM) models are commonly used deep learning architectures for SIC forecasting.

U-Net’s structure allows for both high and low-level feature capture, and its symmetric expansion path helps regain spatial information lost during the contracting phase, aiding in precise localization. However, modifications to the architecture or data representation are often required, as U-Net, which was primarily designed for semantic segmentation, does not inherently manage temporal dependencies. A common solution is to use channels in spatial data as a proxy for the time dimension in forecasting problems [66,74]. In such cases, the multitemporal input data are stacked into multiple input channels, and the output, instead of being a single channel representing pixel-wise SIC quantification, will consist of multiple output channels, each containing the SIC estimation for a future timestamp. Other optimizations on the U-Net include the development of SICNet [68], a U-Net architecture with a temporal-spatial attention module to capture the spatiotemporal dependencies within the input data. Note that the use of U-Net is a surrogate for a forecasting task. Even though the model captures temporal relationships within the data, the problem remains solved as a spatial problem. It is not a sequence-to-sequence learning method commonly adopted in time series prediction.

Recurrent neural network (RNN) models are often employed in forecasting tasks due to their sequence-to-sequence learning ability. Choi et al. [75] applied a gated RNN to provide a 15-day prediction based on historical SIC data. Chi and Kim [76] developed an LSTM model that also uses SIC sequences for future one-month prediction of SIC. Wei et al. [77] developed an attention-based LSTM neural network model that considers the contribution from all hidden states in the forecasting. The results show that the proposed model outperforms traditional LSTM, which only considers information provided at the last hidden state. Similar works also include Ali et al. [78]. However, LSTM-based models capture only the temporal correlation, largely ignoring the spatial relationships. To address this issue, ConvLSTM, an LSTM variant that efficiently handles temporal dependencies by incorporating convolutions within the LSTM cell, allows for the simultaneous processing of both spatial and temporal data. Despite their resource-intensive nature, ConvLSTM models are particularly effective for spatiotemporal tasks like SIC forecasting [72,79]. Optimizations on these models include a two-stream ConvLSTM to learn the sparse and detailed characteristics of sea ice dynamics [71], a multi-task ConvLSTM for simultaneous SIC and sea ice extent (SIE) prediction [80], and the addition of a Conditional Random Field (CRF) module for boundary and edge detection [69].

Table 4. Deep learning solution techniques for sea ice concentration forecasting.

Sea Ice Application	Deep Learning Problem Formulated	Deep Learning Techniques (Models)	Output	References
Sea ice concentration forecasting	Time-series forecasting	Semantic segmentation (U-Net, SICNet)	Classify each pixel into a SIC type, with multi-channel output and each channel represents the SIC classification for a future timestamp (e.g., month)	[66,68,74]
		RNN (LSTM, attention-based LSTM, gated RNN)	Predict sea ice concentration in future timestamps through sequence-to-sequence learning. The forecast could be for one or multiple timestamps	[75,76,77,78]
		CNN+RNN (ConvLSTM, multi-task ConvLSTM)		[69,71,72,79,80]

summarizes the main deep learning techniques and methods to support sea ice extent forecasting.

2.5. Sea Ice Motion Estimation

Sea ice motion estimation refers to the assessment of sea ice displacement and velocity over time. Such movements, influenced by wind, ocean currents, and temperature variations, can have profound effects on the polar marine ecosystem, climate change, and human activities [81]. Sea ice motion contributes to the redistribution of ice thickness and coverage, therefore impacting the Earth’s albedo effect [82]. The movement and subsequent melting of ice in warmer regions contributes to the global thermohaline circulation, which significantly influences climate patterns [83]. For human activities, understanding sea ice motion is critical for safer navigation routes for shipping and other maritime operations in the Arctic regions [84]. Meanwhile, many related ice properties, such as ice divergence or opening, can be derived from ice drift, leading to the development of sea ice leads and cracks. Utilizing the momentum of sea ice motion, short-term predictions become possible based on estimated ice drift information [85]. Therefore, accurate sea ice motion estimation is important in both scientific and socioeconomic contexts.

Data used in sea ice motion estimation comes from a variety of remote sensing platforms. For instance, passive microwave sensors such as AMSR-E and AMSR2 are useful in sea ice motion estimation as they can detect ice concentration and type, and are unaffected by darkness or cloud cover, enabling continuous monitoring [57,86,87,88]. However, these sensors’ relatively coarser spatial resolution poses a challenge for detailed local studies. In contrast, SAR interferometry (InSAR) imagery from sources such as L-band PALSAR-1 provides high-resolution data, enabling the detection of smaller-scale ice movements [89]. The technology also operates in all weather conditions and during both day and night. Additionally, complementary data from sources like ASCAT (Advanced Scatterometer), which measures radar signal backscatter, and reanalysis data from Japanese 55-year Reanalysis (JRA55), can enhance the sea ice motion estimation process by providing surface wind data. Wind plays a significant role in driving sea ice motion, and thus data from ASCAT or JRA55 enhances the estimation process [88,90]. However, ASCAT measurements can be affected by heavy rain, and JRA55 data, based on model simulations and assimilated observational data, may contain biases and inaccuracies. Sea ice velocity data from NSIDC offers direct measurements of sea ice motion and can act as ground truth for model validation [90], although its coarser spatial and temporal resolution and potential errors due to the mix of satellite and buoy data may pose challenges.

The output of sea ice motion estimation often takes the form of two-dimensional maps or vectors showing the displacement over time. The vectors typically indicate the direction and speed of the ice movement, often provided in grid format or as displacement fields. These displacement maps offer a visual and quantifiable representation of sea ice movement, facilitating the understanding of ice dynamics and interactions with atmospheric and oceanic forces.

There are generally two methods for sea ice motion detection. The first is to predict a future image, and then, by identifying the same sea ice patches on the pair of images in consecutive timestamps, one can create the motion vectors for each pixel within the image scene. Srivastava et al. [91] developed an encoder-decoder-based LSTM model in which the encoder LSTM transfers the input into a 1D representation. Then, multiple decoder LSTMs are introduced to reconstruct images from the representation, learning to predict images in the future timestamps. Such tasks can also be achieved through video prediction techniques in an unsupervised learning framework [92]. The second approach is to first calculate optical flow, which indicates sea ice motion from the pair of images, creating a two-dimensional displacement field that symbolizes sea ice motion. Such optical flow can be incorporated into LSTM or ConvLSTM models for future motion prediction, leveraging these models’ ability to capture sequential data’s temporal dependencies [87,88].

The difference between LSTM and ConvLSTM is that LSTM takes in 1D sequence data, so the sea ice motion image will need to be serialized before applying it. ConvLSTM was able to add a convolutional module to an LSTM and has the ability to process 2D images in a time sequence. In another method, inputs including surface wind data, the previous day’s sea ice velocity, and sea ice concentration feed into a CNN, subsequently engaging in a patch-wise regression task to predict sea ice velocity [90]. This technique effectively utilizes environmental factors’ interaction and their impact on sea ice motion by allowing the CNN to extract complex spatial features from the input data. The results also demonstrated the superior performance of the CNN-based regression model as compared to other baseline models, such as linear regression and random forest. This is owing to the CNN’s ability to capture spatial relationships among the pixels within an image scene, whereas in other baseline models, each pixel is considered an independent sample. Recently, some studies have adopted super-resolution techniques to enhance satellite imagery resolution prior to optical flow computation [57,86], enabling a more detailed representation of sea ice, hence improving the optical flow and the overall motion prediction accuracy.

Table 5. Deep learning solution techniques for sea ice motion estimation.

Sea Ice Application	Deep Learning Problem Formulated	Deep Learning Techniques (Models)	Output	References
Sea ice motion estimation	Video/future image prediction	Unsupervised encoder-decoder LSTM	Predict the future sea ice images and generate the motion vector by comparing a pair of images in consecutive timestamps	[91]
	Point-based regression	CNN + fully connected layers	Predict the motion vector at both x and y directions for each pixel within the study area	[90]
	Time-series analysis and forecasting	LSTM (1D), Convolutional LSTM (2D)		[87,88]

summarizes the main deep learning techniques and methods to support sea ice motion estimation.

2.6. Sea Ice Type Classification

Sea ice type classification is research aimed at distinguishing between various types of sea ice, such as multi-year ice, first-year ice, new ice, and ice-free areas [22]. These classifications aid in understanding climate change and the functioning of polar ecosystems, as different types of ice have varying thermal properties, saline contents, and associated microorganism populations. Moreover, the presence of certain ice types can pose risks to shipping routes and offshore structures. With the increasing accessibility of the Arctic region due to ice melting, accurate sea ice type classification is becoming more important for safe navigation and efficient resource extraction.

Sea ice type classification utilizes a variety of data sources, each with distinct characteristics and applications. The data sources can be broadly categorized into satellite imagery and aerial photographs. Satellite data, particularly SAR systems including Sentinel-1, RADARSAT-2, and Gaofen-3, hold substantial value due to their all-weather and diurnal observation capabilities [93,94]. Sentinel-1 provides 10 m-resolution images in dual polarization, crucial for ice type differentiation through backscattering coefficients [95,96,97]. RADARSAT-2 provides flexible beam modes and polarizations, making it a useful tool for capturing different scales of ice features [98,99,100]. Gaofen-3 combines high-resolution imaging (at submeter level) and a large swath width, enabling detailed large-scale ice surveys [101]. In contrast, the multispectral imaging satellite Landsat-8 provides 30 m-resolution visible and infrared images. It is useful for ice type differentiation due to its ability to capture distinct spectral characteristics of different ice types [102,103]. However, unlike SAR systems, their usage is limited by cloud cover and polar darkness. Aerial photographs, offering extremely high-resolution (5 to 25 cm per pixel) data, capture detailed ice structures and features, but are resource and time-intensive with relatively small coverage [104].

The verification and training of machine learning models in sea ice type classification often rely on expert-annotated ice charts and researcher-led manual annotations. Ice charts, such as those from the Canadian Ice Service (CIS) and the Ice Service department of the German Federal Maritime and Hydrographic Agency (BSH), constitute human interpretation of ice conditions based on multiple sources, including aerial surveys, ship reports, and satellite data. Despite their use as reliable sources for large-scale, temporal studies [95,102], the spatial resolution of these charts is generally coarse and may not cover all regions or times of interest. Additionally, inconsistencies might arise from differential interpretations of ice conditions across analysts and institutions. Manual annotation offers another method for establishing ground truth, involving detailed examination of high-resolution imagery and annotating each pixel or region with the appropriate ice type. Although this yields high-accuracy and high-resolution labels suitable for model training and validation [103,104], it is a laborious process requiring substantial expertise in ice analysis, and feasibility depends on the availability of high-quality images and accessibility to ice areas.

The output from sea ice type classification is typically presented as thematic maps, indicating the geographical distribution of various ice types. This format facilitates a visual and spatial understanding of the ice conditions. The color-coded maps usually represent different types of sea ice and open water, along with metadata providing information about data source, time, and possibly uncertainties. Outputs may also be represented numerically, useful for climate models and further computational analysis.

The field of sea ice type classification utilizes both traditional machine learning techniques and more recent deep learning models. Traditional models, such as SVMs and MLPs, are commonly employed for patch-wise classification [102,103]. Despite their simplicity and computational efficiency, they may struggle with complex and high-dimensional data. In contrast, deep learning models, such as CNNs, can learn useful features directly from raw data. ResNet has been widely used for patch-wise classification due to its strong capability in handling image data [95,96,105,106]. Advanced architectures such as Multiscale MobileNet [101] and Normalizer-free ResNet [107,108] further improve performance by introducing efficient computation and improved training dynamics.

Some researchers approach the problem as a semantic segmentation task, employing models like U-Net or advanced architectures like the Swin Transformer-based U-Net. The Transformer is a new architecture with the capability of capturing data dependencies over a long range, thereby possessing stronger feature extraction capability than other feature extraction backbones. These models are also capable of generating high-resolution classification maps and are often better at preserving spatial structures [104]. Semi-supervised methods, such as an initial over-segmentation using Iterative Region Growing using Semantics (IRGS) segmentation, followed by ResNet-based patch classification, offer an interesting alternative [99,100]. This approach can improve the handling of boundaries and heterogeneous areas but might require careful parameter tuning and a thorough understanding of the data. Some researchers also incorporate temporal information, leveraging ConvLSTM networks [97]. This method can account for temporal variations in ice conditions but adds an extra layer of complexity and computational demand. In summary, the selection of the appropriate method largely depends on the specific requirements of the study, available resources, and the nature of the data at hand.

Table 6. Deep learning solution techniques for sea ice type classification.

Sea Ice Application	Deep Learning Problem Formulated	Deep Learning Techniques (Models)	Output	References
Sea ice type classification	Patch-based classification	CNN (ResNet, Multi-scale MobileNet, Normalizer-free ResNet)	Thematic map indicating sea ice types, e.g., “multi-year ice”, “first-year ice”, and “new ice” for each image patch	[94,95,96,101,105,106,107,108]
	Semi-supervised learning	Unsupervised segmentation + patch-level classification (IRGS+CNN/GCN)		[99,100]
	Pixel-based segmentation	Semantic segmentation (U-Net, Swin Transformer-based U-Net)	Thematic map indicating sea ice types, e.g., “multi-year ice”, “first-year ice”, and “new ice” for each image pixel	[104]
	Time-sequence analysis	ConvLSTM	Classify sea ice type based on past sea ice conditions	[97]

summarizes the main deep learning techniques and methods to support sea ice type classification.

3. Deep Learning Strategies Tailored for Sea Ice Research

Besides adopting different types of deep learning models, customized learning strategies, such as super-resolution and refined loss functions, have also been developed to support Arctic sea ice research. These advances bring new depth and accuracy to data analysis and forecasting tasks. The following section further explores and discusses these strategies.

3.1. Super-Resolution

In the domain of image processing, super-resolution refers to a set of methodologies that augment the spatial resolution of an imaging system. In the deep learning realm, super-resolution techniques employ trained neural networks to recreate high-resolution depictions from low-resolution equivalents, thereby filling in the details that the initial imaging process was unable to document [109].

For example, in the application of sea ice lead detection, high-resolution imagery is critical to ensure accurate lead delineation because of the typically small and narrow shape of these features. Satellite imagery, such as MODIS, carries inherent constraints in spatial resolution, making it less suitable for detecting smaller leads. As a solution to this, Yin et al. [26] proposed a model named DeepSTHF (deep sub-pixel turbulent heat flux), which employs MODIS Ice Surface Temperature (IST) images as inputs to generate enhanced output images and lead maps. The proposed architecture achieves this using two separate CNN branches. The first branch, composed of residual blocks, is committed to refining the MODIS IST image resolution, enabling complex image detail learning while mitigating information loss during the enhancement process. In parallel, the second branch, established on an encoder-decoder framework, is tasked with the generation of a high-resolution lead map.

In the context of SIC classification, continuous SIC monitoring largely depends on passive microwave images. However, due to the coarse spatial resolution of these images, SIC identification tends to be blurred at the ice-water boundaries. Addressing this issue, works by Liu et al. [57] and Feng et al. [56] applied the PMDRnet (progressive multiscale deformable residual network) to amplify the spatial resolution of sea ice passive microwave images based on unique characteristics of the images and sea ice movements. The network employs a strategy of multi-image super-resolution, with image alignment facilitated by a progressive alignment approach that dynamically switches reference images during the alignment of two neighboring images. This alignment is anchored on the principles of deformable convolution (DConv) and dilated convolution. DConv offers an augmentation of spatial sampling locations through learned offsets, thereby enhancing the geometric transformation modeling capacity of original CNNs that are constrained by a fixed kernel size in a convolutional layer. Furthermore, DConv alignment uses multiple offsets at each feature location, capturing information from local neighborhoods and mutually complementing them, which results in improved alignment accuracy.

Super-resolution has also been applied in sea ice motion research. Petrou et al. [86] employed super-resolved satellite images to derive optical flow and, therefore, a dense motion vector field with continuous values and subpixel precision, consequently achieving improved performance as compared using images at original resolution. Super-resolution can also be supported by incorporating temporal information. Liu et al. [57] leveraged temporal data to boost the resolution of coarse passive microwave data in their study. They employed a deep learning-based multi-image super-resolution approach to classify SIC. Here, the temporal attention mechanism helps extract and fuse temporal information across the data sequence, effectively compensating for potential image alignment issues, thereby ensuring a more accurate and fine-grained SIC classification.

3.2. Customizing Loss Function in AI Models

A loss function, in the context of deep learning, is a measure of how far the prediction of a model is from the true output (a.k.a., ground truth). It quantifies the error or discrepancy between the predicted and actual values, offering a way to evaluate the performance of a model. In a typical deep learning framework, the model is trained by adjusting its parameters iteratively to minimize the loss function. Different loss functions have different mathematical properties and are suitable for different types of prediction tasks. For instance, Mean Squared Error (MSE) is often used for regression tasks, while Cross-Entropy loss is commonly used for classification tasks. Sometimes, however, standard loss functions may not adequately capture the specific objectives of a certain task. In such situations, researchers may develop custom loss functions tailored to the problem at hand, as evident in the following examples.

In the realm of sea ice type classification, certain ice types are naturally less commonly seen, and consequently, a model trained on such data may tend not to predict these rare types. Pedersen and Kim [106], acknowledging this class imbalance, proposed a true negative weighted loss. This method reduces the loss value when the model correctly identifies the absence (a true negative) of a specific type of ice. Consequently, for ice types that are rare throughout the dataset, the model is not disincentivized to predict their presence. This approach ensures that the model’s predictions are largely influenced by the image content, as opposed to the overall statistical distribution of different ice types.

A different problem arises in SIC classification, where passive microwave radar satellite data is susceptible to interference from atmospheric variations, and calibration for one area may not translate well to another. To address this issue, Dominicus et al. [62] proposed a loss function that accounts for the uncertainty in passive measurements, incorporating uncertainty from a SIC product (e.g., OSI-401-b) into the loss computation. This uncertainty-weighed loss function scales the loss values by the uncertainty level, thereby preventing over-penalization of the model in instances where the concentration label exhibits high uncertainty. Similarly, Liu et al. [57] provided another example of tailoring loss functions to consider data characteristics. In addition to conventional reconstruction error for super-resolution, they proposed a SIC-specific loss function to enrich the model with more sea ice details for finer SIC predictions. The loss function leverages the polarization difference of brightness temperatures at the 89 GHz channel, a parameter frequently utilized for SIC computations. By assessing the difference in this value between the reconstructed and ground truth images, the loss function encourages the reconstructed image to retain equivalent SIC information as the ground truth.

In SIC forecasting, this task entails a sequence of SIC inputs coupled with a series of SIC predictions. Traditional loss functions, such as Mean Squared Error (MSE) or Mean Absolute Error (MAE), are commonly employed to measure the divergence between each forecast–ground truth pair [77]. However, this approach fails to acknowledge the temporal dependencies that characterize the underlying SIC trend, as SIC variations are not independent occurrences but are part of a sequential continuum. Addressing this gap, Liu et al. [79] proposed a gradient loss function (Grad-loss), designed to capture the SIC trend, thereby empowering the model to capture the temporal evolution of ice concentration. Similarly, Chi et al. [71] utilized a perceptual loss function that takes spatial patterns into account, thereby capturing pixel interrelations more effectively than pixel-level loss. This method involves passing both predicted and ground truth images through a CNN and computing loss based on the resultant feature maps. Moreover, SIC prediction regions may include land areas where sea ice cannot potentially form. Integrating predictions of these areas into loss computations may result in image content imbalance if the land cover area is extensive, potentially causing the model to skew towards predicting lower SIC values. Addressing this challenge, Kim et al. [80] and Andersson et al. [74] applied a land mask on SIC predictions to eliminate land regions from the loss computation, allowing for more efficient model optimization on grid cells that are feasible for ice formation.

3.3. New Learning Strategies to Address for the Lack of Ground Truth Data

As a data-driven technique, deep learning relies heavily on labeled ground-truth data to train the model and help it acquire sufficient domain knowledge. Conventional machine learning approaches, such as support vector machines or decision trees, usually require expert-crafted features as input. In contrast, deep learning models autonomously extract these features from raw data, thereby reducing the need for manual feature engineering. Nevertheless, this capability relies on a significant volume of labeled data to guide the learning process. The more complex the task, the more labeled examples the model typically needs to effectively learn useful patterns.

In sea ice research, acquiring a large amount of labeled data is often challenging due to several factors, such as logistical issues, the dynamic nature of sea ice, and the high cost of data collection. Some researchers have attempted to tackle these challenges by utilizing data from numerical simulation systems [45]. These systems leverage mathematical modeling to predict sea ice behavior and attributes, offering an abundance of data without the need for physical data collection. However, these systems are not without their own issues. Primarily, the simulations carry a degree of uncertainty. This uncertainty stems from the inherent limitations of modeling complex natural processes, inaccuracies in the input data, and the assumptions made in the models. In addition, the simulation systems also require human effort, such as data pre-processing, calibration of the simulation models, and validation of the simulation results against actual observations. This process can be labor-intensive and demands a high level of expertise.

In response to this challenge, iterative learning, also known as incremental learning, is a strategy that has been employed in recent literature [24,104]. This approach starts with the labeling of a small data subset, providing the groundwork for the initial training of the model. Subsequent to the training phase, the model is tasked with generating label predictions for the remaining unlabeled dataset. Human experts then review these predictions, rectifying evident errors and annotating notable omissions. The refined predictions are subsequently utilized to train another model iteration. This collaborative interplay between the model and human experts helps ensure that the labeling process retains a high level of accuracy while also benefiting from the efficiency of the AI system. Sometimes, some datasets serve as ground truth for the others. Yin et al. [26], for instance, derived higher-resolution MODIS data using Landsat-8 as ground truth. MODIS, while offering high temporal resolution data, is at a coarser spatial resolution (250 m to 1 km) in comparison to the finer spatial resolution (30 m) offered by Landsat-8, which has a lower temporal resolution. This is achievable due to the overlapping wavelengths of the two satellite systems, making inter-comparison possible.

Another popular approach is semi-supervised learning, leveraging both labeled and unlabeled data during the training process. This method not only utilizes the known labels to guide the learning but also exploits the underlying structure in the unlabeled data to improve generalization. For instance, similar data points are expected to have similar labels. Jiang et al. [99] used semi-supervised learning for classifying sea ice types. The method began with the iterative region growing using a semantics process to generate super-pixels. A GCN was then constructed to learn the features of each node representing these super-pixels, forming the basis for label assignment. Finally, a softmax assigned labels to the nodes in the graph.

3.4. Novel Models and Architectures

Deep learning has increasingly become an indispensable tool for researchers in sea ice studies over the past few years, harnessing the ability of these models to recognize intricate patterns in large, complex datasets. While conventional deep learning models have achieved significant success in tasks such as sea ice classification and tracking, they do have their limitations. Conventional CNN models, for example, often fail to fully utilize contextual information and can struggle with scale variations in image data. This can lead to less-than-optimal performance when classifying ice types that show similar spectral characteristics but different contextual information. LSTMs, while excellent at capturing long-term dependencies, in practice, can still struggle with very long sequences due to vanishing gradients. Moreover, their sequential nature makes it difficult to parallelize operations during training. In view of these limitations, sea ice researchers are progressively turning towards more advanced models and architectures.

Embracing the potential of more advanced models, sea ice researchers are now exploring the realm of attention mechanisms and Transformer architectures. Attention mechanisms allow models to focus on specific parts of the input while processing data, thereby enhancing the model’s ability to use contextual information. This can be particularly useful in sea ice studies, where distinguishing between similar-looking ice types often hinges on understanding the broader context [44,68,77]. On the other hand, Transformer models, initially designed for natural language processing tasks, have proven to be remarkably effective in image processing tasks as well, including image segmentation. The self-attention mechanism within Transformers enables the model to account for all pixels in the input image, thereby addressing scale variations and enabling a more global understanding of the image. This capability significantly enhances the accuracy of tasks like sea ice type classification [104], where classifying each pixel correctly is crucial. Unlike LSTMs, Transformers can process inputs in parallel, improving computational efficiency and facilitating large-scale applications.

Continuing the trajectory of innovation, newer deep learning models such as NFNets [110], PredRNN++ [111], and MobileNetV3 [112] are being incorporated into the toolkit of sea ice researchers, each offering distinct advantages over conventional models like VGGNets and LSTMs. NFNets, a family of models designed to provide high performance without the need for batch normalization, have shown promise in handling large-scale datasets. In the context of sea ice research, the improved performance and scalability of NFNets lead to better accuracy in tasks like sea ice classification [107], particularly when dealing with high-resolution satellite imagery. The elimination of batch normalization also reduces the computational resources needed during the training process, making the model more efficient to use. PredRNN++, an advanced recurrent model, overcomes the vanishing gradient problem in traditional LSTMs and can handle longer sequences, making it suitable for studying long-term sea ice dynamics. The enhanced ability of PredRNN++ to handle long-term spatio-temporal dependencies and improved performance in prediction tasks lead to more accurate forecasts of sea ice extent and thickness [45]. MobileNetV3, a highly efficient model architecture optimized for mobile devices, offers the benefit of lightweight structure and lower computational requirements. When combined with a multi-scale network such as FPN (Feature Pyramid Network), the benefits of MobileNetV3 can be further enhanced. FPN can help the model capture and integrate features at various scales, which is often critical for tasks such as sea ice classification, where the ice can display variations in scale [101].

4. Potential Future Research Directions

AI is a rapidly evolving field that is seeing continuous advances in new methods and techniques. While we are witnessing an increasing number of transdisciplinary research projects applying AI to Arctic sea ice remote sensing, we also see opportunities to further integrate and adapt cutting-edge AI into Arctic sea ice research. Below, we present a few research directions from which the sea ice research community could further benefit, leveraging the most recent and exciting developments in AI.

4.1. Enhanced Multimodal Deep Learning Capabilities

Multimodal deep learning is an advanced machine learning approach that harnesses information from multiple data types (or “modalities”) such as text, images, audio, and video to make more accurate and comprehensive predictions. It involves training deep neural networks on data that includes multiple types of information and creating a unified representation that captures the joint information present in these diverse data. This approach is particularly useful in scenarios where individual modalities might not provide a complete picture, and the integration of different types of data can lead to more robust and insightful models. For example, an image of a scene can be combined with textual descriptions to enhance content understanding.

Recent advancements in the field of multimodal deep learning have further broadened its applicability and effectiveness. For instance, researchers have been developing innovative algorithms that can not only process and integrate different data types but also account for the temporal dynamics and dependencies between these modalities, such as the synchronization of audio and video data in multimedia content [113,114,115]. Moreover, the advent of transformer-based models, such as ViLBERT [116] and LXMERT [117], has significantly improved the performance of multimodal tasks by better understanding and representing the complex interactions between different data modalities. Despite these advances, there remains substantial challenges, including the need for larger, more diverse datasets and methods to better interpret and explain the decisions of these complex models.

In parallel with these developments, the sea ice research field has begun to incorporate multimodal deep learning methodologies into its predictive models. An array of multi-source data, including sea ice data, meteorological data, and satellite-derived datasets, is already being used. For example, Karvonen et al. [118] fused CryoSat-2 radar altimeter and SAR imagery from Sentinel-1 to estimate sea ice thickness. In the deep learning context, researchers have approached data fusion by aligning them through input channels or processing the data streams separately and then combining them through concatenation [74]. This approach, while effective to an extent, still needs further refinement and optimization. The current progress of multimodal deep learning has opened a plethora of research opportunities for sea ice studies to understand and interpret the underlying interconnections among multi-source data.

Looking towards the future of sea ice research, the potential of multimodal deep learning to enhance our comprehension and prediction of sea ice transformations is unquestionably promising. A crucial aspect to consider is the necessity for ample, diverse, and high-quality datasets to fuel the evolution of multimodal deep learning within this research area. When it comes to handling data, the opportunity to align various data modalities is immensely valuable. Given the ever-changing nature of sea ice time-sequence data, temporal alignment has become the key. By combining datasets such as sea ice concentration, satellite imagery, and climate patterns that coincide with spontaneous temporal events, we may uncover intricate relationships that do not appear in individual datasets. Moving forward, fusion strategies for multimodal input that extend beyond basic concatenation or stacking need to be explored [119]. For instance, hierarchical fusion that consolidates information across different levels of a deep learning model could enhance our grasp on sea ice dynamics [120]. Likewise, implementing cross-modal attention mechanisms [121] could enable the model to adjust its focus across various modalities selectively, emphasizing salient features based on the context. In the realm of multimodal representation learning [122], there exists the possibility to devise innovative techniques to conjointly represent different modalities. Approaches like canonical correlation analysis (CCA) or its deep learning equivalent, DCCA, could optimize the interplay between various data modalities, thereby augmenting these models’ predictive power. In addition, the emergence of transformer-based models in the multimodal learning landscape opens a path for deeper investigation. With the ability to harness the self-attention mechanism of transformers, these models could better discern inter-modal and intra-modal dependencies, resulting in more accurate forecasts of sea ice conditions. While these advancements pose significant challenges, they hold the potential to fundamentally transform our understanding of sea ice dynamics.

4.2. Better Ability to Quantify Uncertainty

Uncertainty quantification (UQ) in the context of deep learning refers to the process of estimating the confidence of a model’s predictions. This uncertainty can arise from several sources, including the inherent noise in the data, incomplete or biased data, and model limitations such as overfitting or underfitting [123]. UQ is important in deep learning and, more broadly, in any modeling effort because, first, it allows users to understand the level of risk associated with each prediction, enabling them to make informed decisions. Second, it can lead to improved model performance by highlighting areas of the input space where the model is uncertain and may need additional training data. Third, it contributes to the development of more robust and trustworthy AI systems, as models that can quantify their uncertainty provide more transparent and interpretable outputs [124].

Significant efforts have been made in uncertainty quantification within the field of deep learning. A key development has been the incorporation of Bayesian methods into neural networks, leading to what is commonly known as Bayesian neural networks [125,126,127]. These networks offer a principled way of encoding uncertainty by treating model parameters as random variables, allowing for the computation of a posterior distribution over these parameters. Ensemble methods have also proven effective in quantifying uncertainty. By training multiple models and comparing their outputs, these methods can provide an estimate of the variation in predictions, offering a measure of uncertainty. This has been seen in Deep Ensembles [128], a technique that involves training multiple deep networks independently and then combining their predictions. Furthermore, advancements have been made in using out-of-distribution detection methods to quantify uncertainty [129,130]. These techniques aim to address challenging situations, such as making predictions on data that significantly differs from the training data, where the model’s predictions are inherently uncertain. This is also a well-known issue related to model generalizability [131,132]. Finally, research on uncertainty quantification has influenced the development of new loss functions, like the Variational Autoencoder’s (VAE) evidence lower bound [133,134] or the Wasserstein loss for GANs [135,136], which balance the reconstruction accuracy with the uncertainty in the model’s predictions. While these methods have marked considerable progress, uncertainty quantification in deep learning remains a challenging and active area of research.

Uncertainty quantification in sea ice research utilizing deep learning is a rapidly growing field with vast research potential, as evidenced by a limited but innovative body of work. One pioneering approach involves integrating uncertainty estimation directly from the dataset into the model’s training process. This is achieved by scaling the loss value based on the level of uncertainty, thereby allowing the model to better account for variability in the data and potentially enhancing its overall performance [62]. Another methodology is the implementation of an aleatoric uncertainty-embedded transfer learning approach for SAR-based sea ice classification. This technique involves placing a multivariate Gaussian distribution onto the embedding of input data in the latent space to model aleatoric uncertainty, a form of uncertainty that arises from inherent noise in the data [105]. Despite these advances, challenges and opportunities remain. Uncertainty in sea ice research can arise from multiple sources, including but not limited to the equations used for calculating SIC, the variability in numerical models, and the precision of the instruments employed. Furthermore, inherent limitations in deep learning models, such as overfitting or underfitting, can also contribute to uncertainty. The need to account for these diverse sources of uncertainty presents important research opportunities.

Moving forward, future research directions in sea ice research should emphasize the development and application of ensemble deep learning methods for uncertainty quantification. Ensemble methods, such as bootstrapped ensembles or Bayesian neural networks, offer a robust avenue for quantifying predictive uncertainty. They can leverage the power of multiple independently trained models to capture a broad range of uncertainties, including epistemic uncertainty, which arises from lack of knowledge or data. This could provide a more comprehensive uncertainty estimation, enhancing the credibility and reliability of sea ice predictions. In addition, integrating deep learning with traditional statistical methods for uncertainty quantification, such as Monte Carlo methods or Bayesian inference, can provide a more comprehensive approach to addressing uncertainty in both models and data [123]. This hybrid approach, leveraging the strengths of both deep learning and traditional statistical methods, could serve as a powerful tool not only for quantifying uncertainty but also for providing interpretable and reliable predictions in Arctic sea ice research.

4.3. Enhancing Explainable AI for Arctic Sea Ice Research

Another promising direction is the use of explainable artificial intelligence [137] techniques in deep learning-based sea ice research. The interpretability of deep learning models has been a long-standing challenge, and the lack of transparency can affect the trustworthiness of the modeling results [138]. Applying XAI techniques can help uncover how different AI models make predictions, providing insight into the relationship between input features (e.g., weather patterns and temperature) and model outputs (e.g., seasonal changes in sea ice concentration). This understanding not only improves trust but also enables researchers to identify potential biases or hidden patterns in the data that may not have been obvious through traditional methods.

Beyond uncertainty quantification, XAI could play a critical role in enhancing scientific understanding of complex Arctic systems. For instance, XAI can reveal how specific environmental factors—such as ocean currents, wind patterns, or atmospheric pressure—impact sea ice dynamics in different regions and seasons. By analyzing model explanations, researchers can gain insights into the relative importance of these factors, helping them prioritize variables for further study or refine existing sea ice models. Moreover, XAI can facilitate model validation and comparison by providing transparent justifications for why one model performs better than another in certain scenarios. This can be especially useful in ensemble modeling, where multiple AI models are used to predict sea ice extent or thickness. By applying XAI, researchers can determine which models are most reliable for specific tasks and which are prone to failure under particular conditions, thereby enhancing model selection and robustness.

A potential research direction with significant scientific value is leveraging explainable AI (XAI) to discover new physical laws or relationships hidden in the vast datasets generated by satellite observations and climate models. XAI can help identify previously unknown correlations or causative factors influencing sea ice behavior, offering the potential to inform the development of next-generation numerical models [139]. For example, deep learning models augmented with XAI could lead to the discovery of subtle interactions between sea ice melt patterns and atmospheric changes that are not well-represented in traditional climate models [140].

4.4. Deeper Integration with Physics-Based Models

Physics-based deep learning models are a fusion of traditional physics-based modeling and modern deep learning techniques [141]. In this approach, deep learning models are designed to adhere to known laws of physics or to integrate these principles directly into their architecture or learning process. The importance of physics-based deep learning models lies in their ability to leverage existing knowledge about the world to improve predictions. For instance, in climate science, physics-based models can use the laws of thermodynamics to better simulate climate patterns. By incorporating known physical laws, these models can make more accurate predictions, generalize better to unknown situations, and sometimes even learn more efficiently, as they are not starting from scratch, but building on a foundation of established understanding. They can also provide more interpretable predictions because the outputs are tied to physically meaningful mechanisms or principles. These models can make better predictions in situations with sparse or noisy data [142], as they can leverage prior physical knowledge to “fill in the gaps”.

The field of physics-based deep learning models is a rapidly evolving area with substantial contributions. One of the first works in this space was the use of CNNs in fluid dynamics [143,144], where the networks were taught to predict the time evolution of fluid flows with high accuracy, respecting the physical laws of fluid dynamics. In another seminal work, researchers integrated the Hamiltonian function, a concept from classical physics, directly into the architecture of a neural network, giving rise to Hamiltonian neural networks [145]. This enabled the model to learn and predict complex dynamic systems accurately while inherently conserving energy, a fundamental principle in physics. The concept of physics-informed neural networks (PINNs) is another significant contribution to this field [146]. PINNs incorporate differential equations, representing physical laws, directly into the loss function of a deep learning model. This allows the model to learn solutions that inherently follow the given physical laws. PINNs have been applied to a wide range of problems, from solving partial differential equations to modeling wave propagation and heat conduction. More recently, the notion of “deep equilibrium models” (DEQs) has gained traction [147]. These models are designed to find the equilibrium state of a physical system directly, rather than simulating its temporal evolution. DEQs leverage the power of deep learning to solve complex equilibrium equations that would be difficult or impossible to solve analytically.

Despite the promising advancements in the field of physics-based deep learning models, there has been a noticeable lack of such research specifically in the area of sea ice studies. In one of the few instances of such work, deep learning has been employed to improve sea ice classification performance through the creation of physics-aware training data [148]. The majority of deep learning-based sea ice research, however, relies heavily on data derived from physics-based simulation models such as ERA5. These datasets provide essential environmental parameters that are integral to understanding and predicting sea ice dynamics. A significant opportunity here is to integrate the strengths of physics-based models and deep learning. Physics-based models excel in their ability to simulate the complex interactions and processes in the sea ice environment, albeit often with some uncertainty due to approximations and assumptions in the model. On the other hand, deep learning models, with their ability to learn complex patterns from large volumes of data, can potentially enhance the predictive capabilities of these physics-based models. Furthermore, the incorporation of physical laws, as seen in the concepts of PINNs and HNNs, could provide a robust framework for dealing with the challenges inherent in sea ice prediction, such as data scarcity, noise, and the need to maintain conservation laws. Thus, the underutilized potential of physics-based deep learning models in sea ice research presents a promising avenue for future exploration and innovation.

Given the groundwork already laid by the application of physics-based deep learning models in earth science, sea ice researchers stand at the threshold of exciting new directions for future research. First, the application of PINNs in sea ice studies could be explored. PINNs have shown promise in various areas of Earth science, and their capacity to integrate physical laws directly into the learning process could provide a powerful tool for modelling and predicting sea ice dynamics. Similarly, Hamiltonian neural networks [145], with their focus on energy conservation, could offer unique insights into the energy balance processes that govern sea ice formation and melting. Furthermore, researchers could look to deep equilibrium models [147] as a potential framework for investigating the equilibrium states of sea ice under various atmospheric and oceanic conditions. Given that many Earth science applications involve finding equilibrium states, the transfer of this concept to sea ice dynamics could be highly beneficial. The high degree of similarity and overlap between Earth science and sea ice research, coupled with the potential of physics-based deep learning, presents an exciting frontier for future research and innovation in sea ice studies.

4.5. Better Leveraging of AI Foundation Models

AI foundation models are large-scale machine learning models that are pre-trained on a vast amount of data and can be fine-tuned for various downstream tasks [149]. The term, often associated with language models like GPT-4 [150] and image segmentation models like Segment Anything Model (SAM) [151], describes models that serve as a general basis for many different AI applications, offering a kind of “foundational” intelligence that can be built upon. The importance of AI foundation models lies in their generalizability and efficiency. By pre-training a model on a large dataset, it learns a broad base of knowledge that can be applied to many different tasks with very modest additional training efforts, a.k.a., fine-tuning. This saves time and resources compared to training a model from scratch for each new task. Additionally, these models often exhibit strong performance and domain adaptability across a wide range of tasks [152], in some cases even reaching or exceeding human-level performance.

Building upon this, AI foundation models can be categorized into several types, each with its unique functionality. Language models like GPT-4 [150] and LLaMA [153], trained on extensive text data, excel at generating human-like text and can be used for a wide array of tasks such as translation, summarization, and question-answering. Image models like SAM [151] and Florence [154], trained on image data, are adept at recognizing and segmenting objects in images, classifying images, or generating images. There are also multimodal models that can handle tasks necessitating the understanding of multiple modes of data [155,156]. Foundation models’ functionality is rooted in a two-step process: pre-training on extensive data in an unsupervised or self-supervised manner to learn useful data representations, followed by fine-tuning on task-specific data to optimize performance [157].

In the realm of sea ice research, the potential for deploying AI foundation models remains largely untapped, presenting a promising area for exploration and development. While no definitive foundation model for sea ice area has been established yet, the groundwork has been laid with the advent of several models focused on remote sensing imagery. These include the work by Cha et al. [158], which demonstrated the scalability of foundation models and was developed for detecting and segmenting various man-made features in satellite or aerial images, and RingMo [159], which proposed a large dataset of two million remote sensing images and trained a model to navigate dense, complex remote sensing scenes with small objects. Another notable model was proposed by Wang et al. [160], who introduced a new technique called rotated varied-size window attention (RVSA) to manage the large image sizes and objects of various orientations commonly found in remote sensing imagery. Recently, NASA and IBM jointly released an open-source geospatial foundation model called Prithvi, which includes several versions, such as Prithvi-weather-climate and Prithvi-100M. The former focuses on weather forecasting and climate applications, such as storm tracking, while the latter supports remote sensing image analysis and time-series image generation [161]. The existence of these models indicates a favorable momentum towards the creation of a sea ice foundation model. Furthermore, the interconnected nature of sea ice tasks, ranging from ice type classification to SIC classification/forecasting, suggests that a foundation model could serve as a highly effective tool, able to generalize across different tasks and offer significant acceleration in sea ice research and new discoveries.

Considering the significant progress already made, there are several plausible directions to consider for establishing a foundation model in sea ice research. First, leveraging the wealth of available data, encompassing satellite imagery, sea ice charts, and meteorological data, is key. These data could be used to pre-train an adaptive foundation model that can handle multiple sea ice-related tasks. Furthermore, the previous foundation model works [158,160,162] provide valuable insights for this endeavor. As the very first step, sea ice researchers could try to adapt existing AI foundation models, such as Meta’s SAM and NASA-IBM’s Prithvi model [161], and fine tune them for problem solving related to sea ice. Thinking more broadly, there is the prospect of building a foundation model that supports general Arctic research, trained with massive amounts of Arctic data with multimodality. This effort will benefit the Arctic and polar communities as a whole.

4.6. Open Benchmark Datasets and Open Models

To further advance sea ice remote sensing research, it is important to encourage the development of novel deep learning models to support the above sea ice applications and form a community-driven effort to openly share existing training datasets and models. AI-ready benchmark datasets are critically important resources to enable fair comparison and performance evaluation of various deep learning models. Notably, the Greenland Ice Service at the Danish Meteorological Institute and the Canadian Ice Service have jointly released the AI4Arctic dataset [163] to encourage new developments of deep learning models for the automated production of sea ice chart data, such as sea ice concentration, floe size, and sea ice development stage. Panchi and Kim [164] recently released the SeaIceWeather dataset with training and testing data collected on the GoNorth2023 cruise. This dataset supports ship-based sea ice analysis to understand weather-related degradation of Arctic remote sensing imagery. Andersson et al. [74] shared all the AI training data compiled from numerical models and remote sensing for seasonal Arctic sea ice forecasting. The code sharing and detailed documentation facilitate the reproduction and further extension of this exciting research area. Despite these efforts, the number and diversity of benchmark datasets in the sea ice domain are still limited compared to other Earth science domains, such as flood mapping. More efforts along this line will foster continuous advances in AI-sea ice research.

5. Conclusions

This paper provides a comprehensive overview of the transformative role that artificial intelligence (AI) has played in the field of Arctic sea ice remote sensing. The application of AI techniques has opened up new horizons for researchers, enabling them to extract invaluable insights from the vast and dynamic Arctic region. Throughout this paper, we have highlighted the key contributions of AI, particularly deep learning, across various aspects of sea ice remote sensing, including sea ice lead detection, thickness estimation, concentration and extent forecasting, motion detection, and sea ice type classification. Compared to traditional machine learning methods, such as random forest and support vector machines, deep learning models possess outstanding capability in enabling large-scale automated study and better capturing spatial and temporal dependencies within the datasets. These novel AI applications have demonstrated enhanced performance in both the accuracy and efficiency of sea ice classification, detection, and forecasting.

In addition to using off-the-shelf AI and deep learning models, we have also witnessed an increase in the development of customized learning strategies, such as self-attention and new loss functions, to tailor AI models to advance sea ice research. There has also been a rise in the adoption of cutting-edge AI models, such as transformers, to solve sea ice-related problems at a pan-Arctic scale. Built upon the momentum of adopting AI in Arctic sea ice research, we suggest future research directions where the Arctic sea ice research community could further benefit from the latest advancements of AI. This includes key research questions such as uncertainty quantification and a deeper integration of physics-informed sea ice modeling and deep learning, which are currently under-investigated.

Looking ahead, the future of AI for Arctic sea ice remote sensing holds great promise. As technology advances and more data become available, AI models will continue to evolve, enabling more accurate predictions, real-time monitoring, and an enhanced understanding of the Arctic ecosystem. This, in turn, will contribute to our global efforts to address climate change, ensure safer navigation in polar regions, and support sustainable resource extraction. In the future, in addition to the analysis of imagery at a single timestamp, research integrating time-series sea ice remote sensing products to enable both stronger spatial and spatiotemporal analysis capabilities is a desired goal. Advances in explainable AI (XAI) techniques will also help advance our knowledge of the physics of sea ice dynamics and help understand how long-term climate trends and short-term weather patterns in the Arctic and mid-latitude impact changes in sea ice concentrations and extent. Furthermore, besides developing new deep learning models, it is also important to leverage such models and multimodal sea ice remote sensing products to conduct pan-Arctic analyses to generate high-resolution and high-quality sea ice products. These areas, along with the development of benchmark AI-ready datasets, clearly require a community-driven effort.