GeoAI for Large-Scale Image Analysis and Machine Vision: Recent Progress of Artificial Intelligence in Geography

Abstract

GeoAI, or geospatial artificial intelligence, has become a trending topic and the frontier for spatial analytics in Geography. Although much progress has been made in exploring the integration of AI and Geography, there is yet no clear definition of GeoAI, its scope of research, or a broad discussion of how it enables new ways of problem solving across social and environmental sciences. This paper provides a comprehensive overview of GeoAI research used in large-scale image analysis, and its methodological foundation, most recent progress in geospatial applications, and comparative advantages over traditional methods. We organize this review of GeoAI research according to different kinds of image or structured data, including satellite and drone images, street views, and geo-scientific data, as well as their applications in a variety of image analysis and machine vision tasks. While different applications tend to use diverse types of data and models, we summarized six major strengths of GeoAI research, including (1) enablement of large-scale analytics; (2) automation; (3) high accuracy; (4) sensitivity in detecting subtle changes; (5) tolerance of noise in data; and (6) rapid technological advancement. As GeoAI remains a rapidly evolving field, we also describe current knowledge gaps and discuss future research directions.

1. Introduction

GeoAI, or geospatial artificial intelligence, is an exciting research area which applies and extends AI to support geospatial problem solving in innovative ways. The research in AI, which stems from computer science, focuses on developing computer systems to gain machine intelligence to mimic the way the human perceives, reasons, and interacts with the world and with each other [1]. Although the field of AI has experienced highs and lows in the past decades, it has recently gained tremendous momentum because of breakthrough developments in deep (machine) learning, immense available computing power, and the pressing needs for mining and understanding big data. With little doubt, AI has become the new space race in the 21st century because of its great importance in boosting the national economy, ensuring homeland security, providing rapid emergency response, and empowering a competitive workforce. AI technologies are widely applied in industry and science [2], notably in chemistry [3], mathematics [4], medical science [5], psychology [6], neuroscience [7], astronomy [8], and beyond.

The upward trend of research adopting AI does not stop with geography. In fact, geography is one of the fields which has made serious use of AI, having adopted it in the early days. Because of the interdisciplinary nature of its research agenda, Geography has the natural advantage of embracing new theories, methods, and tools from other disciplines. Back in the 1990s, Openshaw and Openshaw [9] published a book on “Artificial Intelligence in Geography”, which introduced AI techniques and methods, such as expert systems, neural networks, fuzzy system, and evolutional computation, which were state-of-the-art at the time, as well as their applications in Geography. Besides becoming the landmark reference for AI in Geography, it also drove discussion and criticism regarding the combination of the two fields and the scientific properties of AI [10]. Although some of the concerns, such as AI interpretability and the lack of “theory”, remain valid today, AI research has advanced so dramatically in recent years that it has evolved from modeling formal logic to exploration of the more data-driven, deep learning-based research landscape, which is in high demand as a powerful way to analyze ever-increasing big data.

Geography is becoming a field of big data science. In the domain of physical geography, global observation systems, such as operational satellites, which provide continued monitoring of the environment, atmosphere, ocean, and other earth system components, are producing vast amount of remote sensing imagery at high or very high spatial, temporal, and spectral resolutions. The distributed sensor network systems deployed in cities are also collecting real-time data about the status of physical infrastructures and movement of people, vehicles, and other dynamic components of a (smart) city [11]. For social applications, the prevalent use of location-based social media, GPS-enabled handheld devices, various Volunteer Geographic Information (VGI) platforms, and other “social sensors” have fostered the creation of massive information about human mobility, public opinion, and people’s digital footprints at scale. Besides being voluminous, these data sets contain a variety of formats, from structured geo-scientific data to semi-unstructured metadata to unstructured social media posts. These ever-increasing geospatial resources provide added value to existing research by allowing us to answer questions at a scale which was not previously possible. However, it also poses significant challenges for traditional analytical methods which were designed to handle small data sets of good quality [12]. To fully utilize the scientific value of geospatial big data, geographers started to switch gears toward data-driven geography, which relies on AI and machine learning to enable the discovery of new geospatial knowledge.

The term “GeoAI” was first coined at the 2017 ACM SIGSPATIAL conference [13]. It was then quickly adopted by high-tech companies, such as Microsoft and Esri, to refer to their enterprise solutions that combined location intelligence and artificial intelligence. Researchers frequently use this term when their research involves data mining, machine learning, and deep learning, a recent advance in AI. Here we define GeoAI as a new transdisciplinary research area that exploits and develops AI for location-based analytics using geospatial (big) data. Figure 1 depicts a big picture view of GeoAI. It integrates AI research with Geography, which is the science of place and space. If we agree that AI is about the development of machine intelligence that can reason like humans, GeoAI, which is the nexus of AI and Geography, aims at developing the next-generation machines that possess the ability to conduct spatial reasoning and location-based analytics, as do humans, with the aid of geospatial big data. Under the umbrella of AI, machine learning and other data-driven algorithms, which can mine and learn from massive amount of data without being explicitly programmed, have become cornerstone technology. And deep learning, as a subset of machine learning, represents the breakthrough development that advances machine learning from a shallow to a deep architecture allowing the modeling and extraction of complex patterns via the utilization of artificial neural networks. To better fuse AI and Geography and establish GeoAI as a research discipline that will last, there needs to be a strong interlocking of the two fields. Geography offers a unique standpoint for understanding the world and society through the guidance of well-established theories, such as Tobler’s first law of Geography [14] and the second law of Geography [15]. These theories and principles will expand current AI capabilities toward spatially-explicit GeoAI methods and solutions [16,17] so that AI can be more properly adapted to the geospatial domain. Its research territory can also be enlarged by integrating with geospatial knowledge and spatial thinking.

Just like any emerging topic that sits across multiple disciplines, the development of GeoAI has been undergoing three phases: (1) A simple importing of AI into Geography. In this phase, research is more exploratory and involves the direct use of existing AI methods by geospatial applications. The goal is really to test the feasibility in combining the two fields. (2) AI’s adaptation through methodological improvement. This phase identifies the challenges of applying and tailoring AI to help better solve various kinds of geospatial problems. (3) The exporting of geography-inspired AI back to computer science and other fields. In this phase, we will gain an in-depth knowledge of how AI works and how it can be applied, and we will focus on building new AI models by injecting spatial principles, such as spatial autocorrelation and spatial heterogeneity, for more powerful, general-purpose AI that can be adopted by many disciplines. Phase 2 and Phase 3 will build the theoretical and methodological foundation of GeoAI.

It is also important to discern the methodological scope of GeoAI. Researchers today frequently use GeoAI when their geospatial studies apply data mining, machine learning, and other traditional AI methods. Regression analysis and other shallow machine learning methods have existed for many decades, but it is deep machine learning techniques, such as the convolutional neural network (CNN), that have gained the interest of AI researchers and fostered the growth of the GeoAI community. Therefore, while a broad definition of GeoAI techniques shall include more traditional AI and machine learning methods, its core elements shall be deep learning and other more recent advances in AI in which important learning steps, such as feature selections, are done automatically rather than manually. In addition, methods should be scalable in processing geospatial big data.

This paper aims to provide a review of important methods and applications in GeoAI. We first reviewed key AI techniques including feed-forward neural networks, CNNs, Recurrent Neural Networks (RNNs), long- and short-term memory (LSTM) neural networks, and transformer models. These models represent some of the most popular neural network models that dominate modern AI research. We organize the review around the use of geospatial data. As the literature of GeoAI is growing so rapidly, every topic cannot be covered in a single paper. To ensure both depth and breadth of this review, we give preference to groundbreaking work in AI and deep learning, and seminal works that represent the most important milestones in expanding and applying AI to the geospatial domain. We also centered our review on research that leverages novel machine learning techniques, in particular deep learning, while touching on shallow machine learning methods for a comparative analysis. We hope this paper will serve as a fundamentally orienting paper for GeoAI that summarizes the progress of GeoAI research, particularly in tasks geospatial image analysis and machine vision.

The reminder of this paper is organized as follows: Section 2 briefly describes different types of geospatial big data, particularly structured and image data. Section 3 introduces popular methodology in GeoAI research. Section 4 reviews different applications that GeoAI enables. Section 5 summarizes the paper and discusses ways forward for this exciting research area.

2. Geospatial Big Data for Image Analysis and Mapping

• Remote sensing images

Remote sensing imagery is one of the most important and widely used data sources in Geography. It involves information extracted from the Earth’s surface and contains not only human-made features but also natural features. Recent advances in large-scale earth observation and unmanned aviation vehicles (UAVs) result in huge advantages to using remote sensing imagery, such as satellite and drone images, to support applications across multiple geographical scales [18].

• Google Street View

The recent availability of street-level imagery from high-tech companies, such as Google and Tencent, has become a useful way to derive information about the world without stepping into it [19]. In contrast to remote sensing imagery, street view data provide more human-centric observations which contain not only the physical environment but also the social environment [20], as well as other fine-grained information related to cities, such as human mobility and socioeconomic trends [21]. As more and more street view images are generated and machine learning techniques continue to be developed, street image data are being increasingly leveraged.

• Geo-scientific data

The study of Earth’s physical phenomena is important for the human condition. From understanding to predicting, for example, the weather and flooding, to environmental monitoring, geospatial research not only protects people from exposure to extreme events, but also ensures sustainable development of society. There are generally two types of data used in the research of Earth’s systems: sensor data and simulation data. Sensor data, such as temperature and humidity, became widely available because of advancements in hardware technology [16,22]. On the other hand, simulation data are the outputs of models which assimilate information about the Earth’s atmosphere, oceans, and other systems. Both types of data are structured, but they differ from natural images and therefore lead to unique challenges. For example, they are usually high-dimensional and in massive quantities. Their size can be in tera- to peta-byte levels with dozens of geophysical or environmental variables, while an ordinary image dataset is normally at gigabyte scale and has only three channels (RGB). In addition, different sensors may have different spatial and temporal resolutions, increasing the challenges for data integration. To address these challenges, various studies with different applications have been developed.

• Topographic map

Topographic maps contain fine-granule details and quantitative representation of the Earth’s surface and its features, both natural and artificial. On such a map, the features are labeled, and elevation changes are annotated. Topographic maps integrate multiple elements (e.g., features differentiated by color and symbols, labels for feature name, and contour lines showing the terrain changes) to provide a comprehensive view about the terrain. The U.S. Geological Survey is well known for creating the topographic map named U.S. Topo that covers the entire U.S. [23].

Compared to the use of other datasets, topographic mapping is often a primary focus of the government, such as by the United States Geological Survey (USGS). Usery et al. [24] has provided a thorough review of relevant GeoAI applications in topographic mapping, so we will focus on reviewing application using remote sensing images, street view images, and geoscientific data.

3. Methodology

In this review, we categorized articles into three types based on their use of data: remote sensing imagery, street view imagery, and geoscientific data. Each has its own characteristics and processing routines, so the corresponding techniques and methodologies vary. Based on data characteristics, we adopted different strategies for selecting and reviewing the literature. Remote sensing imagery has been used since 1960s or earlier, hence, various techniques have been developed and applied to such data before machine learning and GeoAI have become mainstream techniques, resulting in a large body of works in the area of remote sensing image analysis. To conduct this review, we categorized relevant publications by their tasks, e.g., image classification and object segmentation. Besides introducing applications (e.g., land use classification) of each task, we also describe the use of conventional methods and the more cutting-edge GeoAI/deep learning methods, as well as summarize their differences in a table. For conventional methods, we selected publications with a high number of citations from Google Scholar (~top 40 articles returned using search keywords, such as “remote sensing image classification”) in each task area. For deep learning methods, we selected breakthrough publications in terms of new model development in computer science based on our best knowledge and citation count from Google Scholar. Applications of deep learning methods in remote sensing image analysis are reviewed in more recent literature (2019–2022) to keep the audience informed on the recent progress in this area.

The second focused area of the review is street view imagery, the use of which has a relatively short history compared to remote sensing imagery. Techniques for collecting street view imagery started in 2001 and the data became available for research at around 2010. Because it is a new form of data, there are fewer studies in this area than for remote sensing imagery. Research that can benefit from street view imagery normally involves human activities and urban environmental analysis, which traditionally require in-person interviews or on-site examinations. Street view imagery offers a new way for obtaining information at a large-scale and GeoAI and deep learning enable automated information extraction from such data to reduce human effort and enable large-scale analysis. Here, we categorize our review by applications (e.g., quantification of neighborhood properties) and discuss how GeoAI and deep learning can support such applications. As most recent research in this area has been published after 2017, we did not specify the time range when doing the survey.

The third focus area includes the GeoAI applications of geo-scientific data. Compared to data in the other two categories, geo-scientific data are much more complex in structure and are heterogeneous when data come from different geoscience domains. Because of this, methods used to analyze such data also show large variances even though they are performing the same tasks in different applications. Therefore, we categorized publications by domain applications. Traditionally, scientists rely heavily on physics-based models to understand geophysical phenomena using geo-scientific data. As such, data are highly structured and can be represented as image-type data. In the recent years, GeoAI and deep learning have been increasing applied to derive new insights from these data and they be used as a complementary approach to the physics-based models. The review of traditional approaches or tools is based on their popularity and widespread adoption in large-scale study and forecasting, and the review of more recent deep learning applications is provided for comparison purposes.

4. Survey of Popular Neural Network Methods: From Shallow Machine Learning to Deep Learning

In this section, we review popular and widely used AI methods, particularly the deep learning models. Five major neural network architectures are introduced, including Fully Connected Neural Network (FCN) [25], which is a foundational component in many deep learning based neural network architectures; Convolutional Neural Network (CNN) [17] for “spatial” problems; Recurrent Neural Network (RNN) [26] and LSTM (Long, Short-Term Memory) Neural Network model [26,27] for time sequence; as well as transformer models [28], which have been increasingly used for vision and image analysis tasks. These methods also serve as the foundation for developing the research agenda for methodological development in GeoAI.

4.1. Fully Connected Neural Network (FCN)

Traditional artificial neural network models are the foundation of cutting-edge neural network architectures. For instance, the feed-forward neural network (Figure 2a) involves the placement of artificial neurons, each representing an attribute or a hidden node, in multiple layers. Each neuron in the previous layer has a connection with every neuron in the next layer. This type of neural network is also called a fully connected neural network and is capable of identifying non-linear relationships between the input and the output. However, they suffer from two major limitations: (1) the need to manually define the number of the input nodes, or independent variables, which are also important attributes that help to make final classification and (2) to gain a good predictive capability, the network needs to stack multiple neural network layers in order to learn a complex, non-linear relationship between the independent (the input) and dependent variable (the output). The learning process for such a complicated network is often very computationally intensive, and with its use, it is also difficult to converge on an optimal solution. To address these challenges, newer parallelly processing neural network models have been developed, one of which is CNN. Note that traditional models, particularly the fully connected neural networks, remain an essential component in many deep learning architectures for classification. The manual feature extraction is replaced by automated processing achieved by newer models. And CNN is one of them.

4.2. Convolutional Neural Network (CNN)

CNN is a breakthrough in AI that enables machine learning with big data and parallel computing. The emergence of CNN (Figure 2b) resolves the high interdependency among artificial neurons in an FCN by applying a convolution operation, which uses a sliding window to calculate the dot product between different parts (within the sliding window) of the input data and the convolution filter of the same size. The result is called a feature map and its dimensions depend on the design of the convolution filter. A convolution layer is often connected with a max-pooling layer, which conducts down-sampling to select the maximum value in the non-overlapping 2 by 2 subareas in the feature map. This operation ensures the prominent feature is preserved. At the same time, it reduces the size of the feature map, thus lowering computation cost. After stacking multiple CNN layers, the low-level features which are extracted at the first few layers can then be composed semantically to create high-level features which can better discern an object from others. CNN can be viewed as a general-purpose feature extractor.

Depending on the different types of data that a CNN can take, it can be categorized as 1D CNN, 2D CNN, or 3D CNN. The 1D CNN applies a one-dimensional filter which slides along the 1D vector space; it is therefore suitable for processing sequential data, such as natural language text or audio segment. The 2D CNN, in comparison, applies a filter with size at x × y × n, in which x and y are the dimensions for the 2D convolution filter and n is the number of filters applied to extract different features, e.g., horizontal edges and vertical edges. The 2D filter slides only in the spatial domain. When expanding 2D (image) data into 3D volume data, such as video clips in which the third z dimension is the temporal dimension, the filter is correspondingly in 3D and slides in all x, y, and z directions.

After feature extraction, the model can be further expanded for various applications. For image processing and computer vision, the model can be connected to a fully connected layer for image-level classification, or to a region proposal network for object detection or segmentation. For natural language processing (NLP), the text documents can be represented and converted as matrices of word frequency and then CNN can be leveraged for topic modeling and other text analysis tasks, such as semantic similarity measurement. For processing 3D data with properties of both space and time, or 3D LiDAR data depicting 3D objects, 3D CNN can be leveraged for motion detection or detection of 3D objects. Because of its outstanding ability in extracting discriminative features and its novel strategy in breaking the global operation into multiple local operations, a CNN gains much improved performance in both accuracy and efficiency compared to traditional neural networks. It therefore becomes a building block for many deep learning models.

4.3. Recurrent Neural Network (RNN)

While CNNs have found widespread application, particularly in computer vision and image processing, they are limited in the types of problems they can solve. Because a regular CNN takes a fixed-size input and created a fixed-size output, it cannot process a series of data with interdependency among the datasets at different time slices. To this end, RNN has been developed to add a hidden state to capture the context between the previous input and the output. In other words, the output is not only a function with an input at ‘time’ i (if we consider the dependency of the series of input is time), it also depends on the contextual information provided by the hidden state at time i − 1. Figure 2c illustrates a typical architecture of an RNN. Each hidden state node leverages a feed-forward NN, as shown in Figure 2a with two output nodes. In the example in Figure 2c, the RNN contains a chain of three hidden states (h (0) is a predefined hidden state). That means during the training, the RNN will learn to make a decision based on its current input and the hidden state memorizing contextual information from two previous states.

Similar to 2D CNN, which uses the same filter to perform convolution at different sub-regions of 2D data, the RNN also uses the same weights in <$W_x$, $W_y$, $W_h$> for the calculation at all recurrent states. The architecture of an RNN can be altered according to different application needs. For instance, a one-to-many model (one input, many outputs) can be used for caption learning from an image; a many-to-one model can be used for action classification from a video clip; and a many-to-many model can be used for language translation. Finally, a one-to-one RNN model will be simplified into a feed-forward neural network. By adding bidirectional connections between input, hidden, and output nodes, a bi-directional RNN can be created to capture the context from not only previous states but also future states.

An RNN model can also evolve to a deep RNN by increasing the length of the hidden states chain by adding depth to the transition between input to hidden, hidden to hidden, and hidden to output layers [29]. It is generally recognized that a deep RNN performs better than a shallow RNN because of the ability of a deep RNN to capture long-term interdependencies within the input series.

4.4. Long- Short-Term Memory (LSTM)

Deep RNN can capture long-term memory, and shallow RNN captures short-term memory in the input series. However, the memory they can capture is only at a fixed length. This limits an RNN’s ability to dynamically capture events with different temporal rhythms, negatively affecting its prediction accuracy. LSTM models are developed to address this limitation. As its name suggests, LSTM has the ability to flexibly determine whether and when short-term memory or long-term memory is more important in making decisions. It achieves this by introducing a cell state in addition to the hidden states in regular RNNs. The cell states preserve the long-term memory about event patterns, and the hidden states contain the short-term memory (Figure 2d). To determine which part of the memory should be considered to enable more accurate temporal pattern recognition, LSTM also introduces three gates: an input gate, a forget gate, and an output gate. The input gate determines the amount of input information from the previous state that should flow into the current state in an iterative training process. In other words, it will determine how much new information will be used. A forget gate decides which part of the memory is less important and therefore should be forgotten. And the output gate decides how to combine newly derived information with that filtered from memory to make an accurate prediction about a future state.

Because of its ability to capture long-term dependencies, LSTM has been widely used for time sequence predictions. For instance, a time series of satellite images can serve as the input of LSTM and the model predicts how land use and land cover will change in the future [30]. Depending on the application, LSTM input could be original time sequence data, or a feature sequence extracted using CNN models mentioned above. One interesting application of LSTM in image analysis is its adoption for object detection [31]. Although a single image does not contain time variance, the 2D image can be serialized into 1D sequence data by a scan order, such as row priming. In an object detection application, although the 2D objects will be partitioned into parts after the serialization, LSTM will be able to “link” the 1D subsequences belonging to the same object and make proper predictions because of its ability to capture the long-term dependency. When LSTM is used in combination with new objective functions, such as CTC (Connectionist Temporal Classification), it would be able to predict on a weak label instead of a per-frame label [27]. This significantly reduces labeling cost and increases the usability of such models in data-driven analysis.

LSTM can also be used to process text documents to predict the forthcoming text sequence or perform speech segmentation. These applications, however, are not the focus of this paper.

4.5. Transformer

Another very exciting neural network architecture is transformer, which was developed by the Google AI team in 2017 [28]. It is based on an encoder and decoder architecture and has the ability to transform an input sequence to an output sequence. This is also known as sequence-to-sequence learning. Transformers have been increasingly used in natural language processing, machine translation, question answering, and tasks related to processing sequential data. Different from other sequential data processing models, such as an RNN, a transformer model does not contain recurrent modules, meaning that the input data do not need to process sequentially, instead they can be processed in batch. A core concept that enables this batch or parallel processing is an attention mechanism. Once an input sequence is given, e.g., a sequence of words, the self-attention module will first derive the correlations between all word pairs. For a given word, this means calculating a weight to know how this word is influenced by all the other words in the sequence. These weights will be incorporated into the following computation to create a high-dimensional vector to represent each word (element) in the input sequence. This is also known as the encoding progress. Instead of directly using the raw data as input, the encoder will first conduct input embedding to represent the elements of the input sequence numerically. In addition, a positional encoding is introduced to notify the self-attention module the position of each element in the input sequence. A feed-forward layer is connected with the self-attention module for dimension translation of the encoded vector so it fits better with the next encoder or decoder layer. The encoder runs iteratively to derive the high-dimensional vector that can best represent the semantics of each element in the input sequence.

The decoder (Figure 2e) has an architecture similar to that of the encoder. It takes the output sequence as input (during the training process) and performs both position encoding and embedding on top of the sequence. The embedded vectors are then sent to the attention module. Here, the attention module is called masked attention because the calculation of attention values is not based on all the other elements in the sequence. Instead, because the decoder is used for predicting the next element in the sequence, the attention calculation for each element takes only those coming before it into the sequence rather than all elements in the sequence. This module is therefore called masked self-attention. Besides this module, the decoder also introduces a cross attention module that takes the embedded input sequence and already predicted output sequence to jointly make predictions about the upcoming element. The predictions could be single or multiple labels for a classification problem (i.e., to predict who the speaker is, given a piece of speech sequence), it can also be a non-fixed length vector for machine translation (i.e., from one language to another, or from speech to text).

Besides applications in natural language processing, transformer models have been increasingly used for image analysis and other machine vision tasks. The CNN focuses its attention on a smaller local window (a.k.a., a smaller receptive field) through its convolutional neural network. Comparatively, transformers can dynamically determine the size of a receptive field and can achieve similar or even better performance than CNN [32]. In recent years, Vision Transformers (ViTs) have been considered as the “Roaring 20s” and they surpass CNN models as the state-of-the-art image classification models. However, for more challenging image analysis tasks, such as object detection and semantic segmentation, CNNs still show more favorable performance than transformers [33].

In summary, the revolutionary development of new neural network models, particularly CNN, and LSTM and transformer models, have unique advantages in processing sequence and/or image data. They each therefore play an indispensable role in domain applications. Other machine learning models, such as deep reinforcement learning, generative adversarial network (GAN), and self-supervised learning, are high-level algorithms built upon these foundational neural network structures, and their applications for image and vision tasks will be reviewed in the next section.

5. Applications

5.1. Remote Sensing Image Analysis

To extract information from imagery, traditional approaches often employ image processing techniques, such as edge detection [34,35], and hand-crafted feature extraction, such as SIFT (Scale-Invariant Feature Transform) [36], HOG (Histogram of Oriented Gradients) [37], and BoW (Bag of Words) [38]. These methods require some or more prior knowledge and might not be adaptable to different application scenarios. Recently, CNN has proved to be a strong feature descriptor because of its superior ability to learn representations directly from the original imagery with little or no prior knowledge [39]. Much of current state-of-the-art work has adopted CNN as feature extractors, for example, for object detection [40] and sematic segmentation [41]. However, most of this work uses natural scene images taken from an optical camera and more challenges exist when the models are applied to remote sensing imagery. For instance, such data provide only a roof view of target objects, and the area coverage is large, but the objects are usually small. Therefore, the available information of objects is limited, not to mention issues of rotation, scale, complex background, and object-background occlusions. Therefore, expansion and customization are often needed when utilizing deep learning models with remote sensing imagery.

Next, we introduce a series of applications applying GeoAI and deep learning to remote sensing imagery.

Table 1. Summary of GeoAI and deep learning applications in remote sensing image analysis.

Task	Applications	Conventional Approaches	Limitations of Conventional Approaches	Deep Learning (DL) Approaches
Image-level classification
	Land use/land cover analysis Natural feature classify cation Manmade feature classification	Maximum likelihood Minimum distance Support vector machine (SVM) Principal component analysis (PCA)	Subjective feature extraction Not suitable for large datasets	Convolutional neural network (CNN) Graph neural network (GNN) Combination of CNN and GNN
Object detection
	Environmental management Urban planning Search and rescue operations Inspection of living conditions of underserved communities	Template matching Knowledge-based Object-based Machine learning-based	Sensitive to shape and density change Subjective prior knowledge and detection rules Lack of full automation process	Region-based CNN Regression-based CNN
Semantic segmentation
	Precision agriculture Land use/land cover analysis Infrastructure (road) extraction	Region-based Edge-based Clustering-based	Sensitive to contrast between objects and the background Subjective parameter selection	Encoder/decoder-based CNN
Height/depth estimation
	3D modeling Smart cities Ocean engineering	LiDAR and digital service model (DSM) Monocular estimation Stereo matching	Hand-crafted features Need for careful camera alignment	CNN-based monocular estimation CNN-based stereo matching
Image super resolution
	Image quality improvement in applications like medical imaging and remote sensing	Interpolation Statistical models Probability models	Subjective parameter selection Ill-posed problem, requirement of prior information	CNN-based methods GAN-based methods
Object tracking
	Vehicle tracking Automated surveillance Video indexing Human-computer interaction	Object detection, object feature selection and motion modeling	Hand-crafted features Subjective prior knowledge and detection rules	CNN-based single object tracking CNN-based multiple object tracking
Change detection
	Land use/land cover analysis Deformation assessment Damage estimation	Image differencing Image rationing PCA Change vector analysis	Subjective parameter selection (change threshold) Unable to extract detailed change information	Object detection-based approaches Segmentation-based approaches
Forecasting
	Weather forecasting Drought forecasting Land use/land cover forecasting Sales forecasting	Moving averages Exponential smoothing Linear regressions Probability models	Suitable for short-term/univariate forecasting Lag forecasts behind the actual trend	Deep Belief Network Long Short-Term Memory (LSTM) Transformer

summarizes these applications, methods used, and limitation of traditional approaches.

• Image-level classification

Image-level classification involves the prediction of content in a remotely sensed image with one or more labels. This is also known as multi-label classification (MLC). MLC can be used for predicting land use or land cover types within a remotely sensed image, it can also be used to predict the features, either natural or manmade, to classify different types of images. In the computer vision domain, this has been a very popular topic and has been a primary application area for CNN. Large-scale image datasets, such as ImageNet, were developed to provide a benchmark for evaluating the performance of various deep learning models [42]. The past few years have witnessed continuous refinement of CNN models to be utilized for MLC, particularly with remote sensing imagery. Examples include (1) the groundbreaking work on AlexNet [43], which was designed with five convolutional layers for automated extraction of important image features to support image classification, and (2) VGG [44], which stacks tens of convolutional layers to create a deep CNN. Besides the convolutional module, another milestone development in CNN is the inception module, which applies convolutional filters at multiple sizes to extract features at multiple scales [45]. In addition, the enablement of residual learning in ResNet [46] allows useful information to pass from shallow layers to not only their immediate next layer but also to much deeper layers. This advance avoids problems of model saturation and overfitting that traditional CNN encounters. Although different optimization techniques, such as dense connection and fine-tuning, are applied to further improve the model performance [47,48,49,50], they rest upon these building block and milestone developments of these CNN models.

In remote sensing image analysis, CNNs and their combination with other machine learning models are leveraged to support MLC. Kumar et al. [51] compared 15 CNN models and found that Inception-based architectures achieve the overall best performance in MLC of remotely sensed images. The UC-Merced land use dataset is used in this study [52]. Several CNN models also beat solutions using graph neural network (GNN) models for image classification on the same dataset [53]. These models benefit from transfer learning, which involves the training of the models on the popular ImageNet dataset to learn how to extract prominent image features and fine tune them based on the remote sensing images in the given tasks. Recent work by Li et al. [54] also shows that the combined use of CNN with GNN could in addition capture spatio-topological relationships, and therefore contributes to a more powerful image classification model.

• Object detection

Object detection aims to identify the presence of objects in terms of their classes and bounding box (BBOX) locations within an image. There are in general two types of object detectors: region-based and regression-based. Region-based models treat object detection as a classification problem and separate it into three stages: region proposal, feature extraction, and classification. The corresponding deep learning studies include OverFeat [55], Faster R-CNN [56], R-FCN [57], FPN [58], and RetinaNet [59]. Regression-based models directly map image pixels to bounding box coordinates and object class probabilities. Compared to region-based frameworks, they save time in handling and coordinating data processing among multiple components and are desirable in real-time applications. Some popular models of this kind include YOLO [60,61,62,63], SSD [64], RefineDet [65], and M2Det [66].

Object detection can find a wide range of applications across social and environmental science domains. It can be leveraged to detect natural and humanmade features from remote sensing imagery to support environmental management [67], urban planning [68], search and rescue operations [69], and the inspection of living conditions of underserved communities [70]. It has also found application in the aviation domain where satellite images are used to detect aircraft which can help track aerial activities, as well as other environmental factors, such as air and noise pollution owing to said traffic [71]. CapsNet [72] is a framework that enables the automatic detection of targets in remote sensing images for military applications. Li and Hsu [73] extends Faster R-CNN [56] to enable natural feature identification from remote sensing imagery. The authors evaluated performance of multiple deep CNN models and found that very complex and deep CNN models will not always yield the best detection accuracy. Instead, CNN models should be carefully designed according to characteristics of the training data and complexity of objects and background scenes. Other issues and strategies that may improve object detection performance, such as rotation-sensitive detection [74,75,76,77,78,79], proposal quality enhancement [80,81,82,83], weakly-supervised learning [27,84,85,86,87], multi-source object detection [88,89], and real-time object detection [90,91,92], also have been increasingly studied in recent years [93].

• Semantic segmentation

Semantic segmentation involves classifying individual image pixels into a certain class, resulting in the division of the entire image into semantically varied regions representing different objects or classes. It is also a kind of pixel-level classification. Several methods have been developed to support semantic segmentation. For instance, region-based segmentation [94,95] separates the pixels into different classes based on some threshold values. Edge-based segmentation [96,97] defines the boundary of objects by detecting edges where the discontinuity of local context appears. Clustering-based segmentation [98,99] divides the pixels into clusters based on certain criteria, such as similarity in color or texture. Recent neural network-based segmentation methods bring new excitement to this research. These models learn an image-to-image mapping from pixels to classes, which is different from image-to-label mapping, such as for image-level classification. Because it requires a finer granularity analysis of the image, semantic segmentation is also a more time consuming and challenging task in image analysis. To achieve this, most of the neural network-based models utilize an encoder/decoder-like architecture, such as U-Net [100], FCN [101], SegNet [102], DeepLab [103,104,105], AdaptSegNet [106], Fast-SCNN [107], HANet [108], Panoptic-deeplab [109], SegFormer [110], or Lawin+ [111]. The encoder conducts feature extraction through CNNs and derives an abstract representation (also called a feature map) of the original image. The decoder takes these feature maps as input and performs deconvolution to create a semantic segmentation mask.

Semantic segmentation is frequently employed in geospatial research to identify significant areas in an image. For example, Zarco-Tejada et al. [112] developed an image segmentation model to separate crops from background to conduct precision agriculture. Land use and land cover analysis detect land cover types and their distributions in an image scene. The automation of such an analysis is extremely helpful for understanding the evolution of urbanization, deforestation, and other urban and environmental changes. Many studies have been conducted to address the challenges of segmentation with remote sensing imagery. For example, Kampffmeyer et al. [113] proposed strategies, such as patch-based training and data augmentation, to solve the issue of small objects tending to be ignored in segmentation tasks to achieve better overall prediction accuracy. Fitoka et al. [114] developed a segmentation model to use remote sensing imagery to map global wetland ecosystems for water resource management and their interactions with other earth system components. Mohajerani and Saeed [115] used image segmentation to detect and remove clouds and cloud shadows from images to reduce error in biophysical and atmospheric analyses.

Road extraction and road width estimation is another interesting challenge that can be solved using segmentation. The idea is to combine remotely sensed images with monocular images taken at street level and other geospatial data to build a foundational infrastructure dataset for transportation research [116]. There are also techniques developed to enhance image segmentation to achieve real-time processing [117,118], the successful use of multi-spectral data [119,120,121], and the detection of small object instances [122,123,124,125].

• Height/depth estimation of 3D object from 2D images

Understanding 3D geometry of objects within remotely sensed imagery is an important technique to support varied research, such as 3D modeling [126], smart cities [127], and ocean engineering [128]. In general, two types of information can be extracted from remote sensing imagery about an 3D object: height and depth. LiDAR data and its derived digital surface model (DSM) data could support the generation of a height or depth map to provide such information. However, derivation of LiDAR data is often expensive, so it is difficult to achieve global coverage. In comparison, the development of satellites has allowed remote sensing imagery to become a globally achievable and low-cost alternative. There are generally two methods in the computer vision field to extract height/depth from 2D images: monocular estimation and stereo matching. The aim of monocular estimation is to map the image context to a real-world height/depth value from a single image. Because the depth/height of a particular location relates not only to the local features but also its surroundings, a probabilistic graphical model is often used to model such relations. For example, Saxena et al. [129] used a Markov Random Field (MRF) model to map the appearance features relating to a given point to a height value. Features can also be of various forms, such as hand-crafted features [129], semantic labels [130,131,132,133,134], and CNN-extracted features [135,136,137,138]. Eigen et al. [139] used two CNNs to extract information from global and local views, respectively, and later combine them by estimating a global depth structure and refining it with local features. This work was later improved by Eigen and Fergus [140] to predict depth information using multi-scale image features extracted from a CNN. D-Net [141] is a new generalized network that gathers local and global features at different resolutions and helps obtain depth maps from monocular RGB images.

In stereo matching, a model calculates height/depth using triangulation from two consecutive images and the key task is to find corresponding points of the two images. Scharstein and Szeliski [142] reviewed a series of two-frame stereo correspondence algorithms. They also provided a testbed for the evaluation of stereo algorithms. Machine learning techniques have also been applied in the stereo case and this often leads to better results by relaxing the need for careful camera alignment [143,144,145]. For estimating height/depth, images remotely sensed and from the field computer vision have different characteristics and offer different challenges. For example, remotely sensed images are often orthographic, containing limited contextual information. Also, they usually have limited spatial resolution and large area coverage but the targets for height/depth prediction are tiny. To address these issues, Srivastava et al. [146] developed a joint loss function in a CNN which combines semantic labeling loss and regression loss to better leverage pixel-wise information for fine-grained prediction. Mou and Zhu [135] proposed a deconvolutional neural network and used DSM data to supervise the training process to reduce massive manual effort for generating semantic masks. Recently, newer approaches, such as semi-global block matching, have been developed to tackle more challenging tasks, such as matching regions containing water bodies, for which accurate disparity estimation is difficult to identify because of the lack of texture in the images [147].

• Image super resolution

The quality of images is an important concern in many applications, such as medical imaging [148,149], remote sensing [150], and other vision tasks from optical images [151,152]. However, high-resolution images are not always available, especially those for public use and that cover a large geographical region, due partially to the high cost of data collection. Therefore, super resolution, which refers to the reconstruction of high-resolution (HR) images from a single or a series of low-resolution (LR) images, has been a key technique to address this issue. Traditional super resolution approaches can be categorized into different types, for example, the most intuitive method is based on interpolation. Ur and Gross [153] utilized the generalized multichannel sampling theorem [154] to propose a solution to obtain HR images from the ensemble of K spatially shifted LR images. Other interpolation methods include iteration back-projection (IBP) [155,156] and projection onto convex sets (POCS) [157,158]. Another type relies on statistical models for learning statistically a mapping function from LR images to HR images based on LR-HR patch pairs [159,160]. Others are built upon probability models, such as Bayesian theory or Markov random field [161,162,163,164]. Some super resolution methods operate in a different way than the image domain. For instance, images are transformed into a frequency domain, reconstructed, and transformed back to images [165,166,167]. The transformation is done by certain techniques, such as Fourier transformation (FT) or wavelet transformation (WT).

Recently, the development of deep learning has contributed much to image super-resolution research. Related work has employed CNN-based methods [168,169] or Generative Adversarial Network (GAN)-based methods [170]. Dong et al. [168] utilized a CNN to map between LR/HR image pairs. First, LR image are up-sampled to the target resolution using bicubic interpolation. Then, the nonlinear mapping between LR/HR image pairs are simulated by three convolutional layers, which represent feature extraction, non-linear mapping, and reconstruction, respectively. Many similar CNN-based solutions have also been proposed [169,171,172,173,174,175] and they differ in network structures, loss functions, and other model configurations. Ledig et al. [170] proposed a GAN-based image super resolution method to address the issue of generating less realistic images by commonly used loss functions. In a regular CNN, mean squared error (MSE) is often used as the loss function to measure the differences between the output and the ground truth. Minimizing this loss will also maximize the evaluation metric for a super-resolution task—the peak signal-to-noise ratio (PSNR). However, the reconstructed images might be overly smooth since the loss is the average of pixel-wise differences. To address this issue, the authors proposed a perceptual loss that encourages the GAN to create a photo-realistic image which is hardly distinguishable by the discriminator. Besides panchromatic images (PANs), dealing with hyperspectral images (HSIs) is more challenging due to difficulties collecting HR HSIs. Therefore, studies focusing on reconstruction of HR HSIs from HR PANs and LR HSIs [176,177,178,179,180] have also been reported. In more recent years, approaches, such as EfficientNet [181], have been proposed to enhance Digital Elevation Model (DEM) images from LR to HR by increasing the resolution up to 16 times without requiring additional information. Qin et al. [182] proposed an Unsupervised Deep Gradient Network (UDGN) to model the recurring information within an image and used it to generate images with higher resolution.

• Object tracking

Object tracking is a challenging and complex task. It involves estimating the position and extent of an object as it moves around a scene. Applications in many fields employ object tracking, such as vehicle tracking [183,184], automated surveillance [185,186], video indexing [187,188], and human-computer interaction [189,190]. There are many challenges to object tracking [191], for example, abrupt object motion, camera motion, and appearance change. Therefore, constraints, such as constant velocity, are usually added to simplify the task when developing new algorithms. In general, three stages compose object tracking: object detection, object feature selection, and movement tracking [192]. Object detection identifies targets in every video frame or when they appear in the video [56,193]. After detecting the target, a unique feature of the target is selected for tracking [194,195]. Finally, a tracking algorithm estimates the path of the target as it moves [196,197,198]. Existing methods differ in their ways of object feature selection and motion modeling [191].

In the remote sensing context, object tracking is even more challenging due to low-resolution objects in the target region, object rotation, and object-background occlusions. Work related to these challenges includes [183,184,192,199,200,201]. To solve the issue of low target resolution, Du et al. [199] proposed an optical flow-based tracker. An optical flow shows the variations in image brightness in the spatio-temporal domain; therefore, it provides information about the motion of an object. To achieve this, an optical flow field between two frames was first calculated by the Lucas-Kanade method [202]. The result was then fused with the HSV (Hue, Saturation, Value) color system to convert the optical flow field into a color image. Finally, the derived image was used to obtain the predicted target position. The method has been extended to multiple frames to locate the target position more accurately. Bi et al. [183] used a deep learning technique to address the same issue. First, during the training, a CNN model was trained with augmented negative samples to make the network more discriminative. The negative samples were generated by least squares generative adversarial networks (LSGANs) [203]. Next, a saliency module was integrated into the CNN model to improve its representation power, which is useful for a target with rapid and dynamic changes. Finally, a local weight allocation model was adopted to filter out high-weight negative samples to increase model efficiency. Other methods, such as Rotation-Adaptive Correlation Filter (RACF) [204], have also been developed to estimate object rotation in a remotely sensed image and subsequently detect the change in the bounding box sizes caused by the rotation.

• Change detection

Change detection is the process of identifying areas that have experienced modifications by jointly analyzing two or more registered images [205], whether the change is caused by natural disasters or urban expansions. Change detection has very important applications in land use and land cover analysis, assessment of deforestation, and damage estimation. Normally, before detecting changes, there are some important image preprocessing steps, such as geometric registration [206,207], radiometric correction [208], and denoising [209,210], that need to be undertaken to reduce unwanted artifacts. For change detection, earlier studies [211,212] employed image processing, statistical analysis, or feature extraction techniques to detect differences among images. For example, image differencing [213,214,215] is the most widely used method. It generates a difference distribution by the subtraction of two registered images and finds a proper threshold between change and no-change pixels. Other approaches, such as image rationing [216], image regression [217], PCA (Principal Component Analysis) [218,219], and change vector analysis [220,221], are also well developed.

Recent work has started to leverage techniques from AI [222,223,224] and deep learning [225,226,227,228,229,230,231] to conduct change detection. For example, Sun et al. [224] proposed a method to spatially optimize a variable k in k-nearest neighbor (kNN) algorithm for prediction of the percentage of vegetation cover (PVC). Instead of finding a globally optimal k value, a local optimum k was identified from place to place. A variance change rate of the estimated PVC was calculated at a given pixel by changing k in the kNN algorithm based on the pixel location. Next, a locally optimal k was selected such that the variance change rate curve becomes stable at this value. Wang et al. [231] employed an object detection network, Faster R-CNN [56], for change detection. The authors proposed two different networks, one aiming for detection from a single image merged from two registered images and the other doing the detection on the differences of two such images. Since the detection results of Faster R-CNN are bounding box regions, a snake model [232] is further applied to segment the exact change area. The Structural Similarity index (SSIM) [233] is a metric used to predict the perceived quality of television broadcasts and cinematic pictures by comparing the broadcasted and received images against each other for similarity. The higher the similarity the better the quality of the broadcast. Images (at two different timestamps) can be put through this index to determine how similar (or dissimilar) they are and hence the extent of change can be determined.

• Forecasting

Forecasting is the process employing statistical models on past and present observations to predict the future. Classic forecasting models include moving averages, exponential smoothing, linear regressions, probability models, neural networks, and their modifications. Observations could come from various sources and in either a spatial domain or temporal domain or both. Examples are found in numerous fields, such as the forecasting of weather, drought, land use, and sales, where the prediction could be lifesaving or of socioeconomic benefit. While forecasting brings advantages, it relies on historical data, which might be lacking due to resource or environmental constraints. Fortunately, remote sensing provides an opportunity for long-term and large-scale observation which can be integrated into numerical models for prediction and forecasting. However, in general, remotely sensed images are usually not directly utilized in a forecasting model. Instead, derived parameters or indices [234,235,236] computed from these images are often used. An index is an algebraic and statistical result from multispectral data and can be applied in different scenarios. For example, the Normalized Difference Vegetation Index (NDVI), Land Surface Temperature (LST), and Vegetation Temperature Condition Index (VTCI) [236,237] are indices of vegetation or moisture conditions and can be used for drought monitoring [238,239,240,241,242].

Recently, researchers have started to investigate the application of deep learning techniques for time-sequence data forecasting [243,244,245,246,247,248,249,250]. Since such forecasting involves temporal prediction, a model that can handle sequence data is often adopted, such as Deep Belief Network (DBN) [251] or Long Short-Term Memory (LSTM) models [252]. For example, Chen et al. [245] applied DBNs for short-term drought prediction using precipitation data. Poornima and Pushpalatha [248] used the same data with LSTM for long-term rainfall forecasts. Another application that makes direct use of remotely sensed imagery for forecasting is the prediction of the fluctuation of lake surface areas [253]. Applications such as these track pivotal hydrological phenomena, e.g., drought, that have severe socioeconomic implications. Forecasting the canopy water content in rice [254] using an Artificial Neural Network that integrates thermal and visible imagery is also one of the interesting forecasting applications. Recently, transformer models have been increasingly used as a tool for time series forecasting using remotely sensed or other geospatial data [250].

5.2. Applications Leveraging Street View Images

As a new form of data, street view images provide a virtual representation of our surroundings. Due to increased availability of this fine-grained image data, street view images have been adopted to quantify neighborhood properties, calculate sky view factors, detect neighborhood changes, identify human perception of places, discover uniqueness of places, and predict human activities.

Table 2. Summary of GeoAI and deep learning applications in street view image analysis.

Task	Applications	Conventional Approaches	Limitations of Conventional Approaches	Machine Learning (ML)/DL Approaches
Quantification of neighborhood properties
	Infer social and environmental properties of an urban region	In-person interviews	Labor-intensive Lack of data	Classification Object detection
Calculation of sky view factors
	Urban management Geomorphology Climate modeling	Direct measurement Simulation	On-site work required Hard to get precise parameters in complex scenes	Semantic segmentation
Neighborhood change monitoring
	Urban management Policies evaluation	In-person interviews Mailed questionnaires Visual perception surveys	Human bias Small region coverage	Semantic segmentation Classification
Identification of human perception of places
	Geospatial intelligence Cognitive and behavioral science	Indirect and direct human communication	Human bias Small region coverage	Object detection (face) Classification (emotion)
Personality and place uniqueness mining
	Human activity Socioeconomic factors	In-person interviews Social surveys	Human bias Small region coverage	Classification Object detection Semantic segmentation
Human activity prediction
	Urban planning Policies evaluation Assessment of health and environmental impacts	Household surveys In-person interviews Questionnaires	Labor-intensive Resource-intensive Lag of data	Object detection Classification

summarizes these applications, methods used to analyze street view images, and limitations of traditional approaches.

• Quantification of neighborhood properties

As street level data provide image views from human perspectives, they can be leveraged to infer different social and environmental properties of an urban region [255,256,257,258,259,260,261]. Gebru et al. [255] estimated demographic statistics based on the distribution of all motor vehicles encountered in particular neighborhoods in the U.S. They sampled up to 50 million street view images from 200 cities and applied a Deformable Part Model (DPM) [262] to detect automobiles. A convolutional neural network (CNN) [43,263] was also used to classify 22 million vehicles from street view images by their make, model, year, and a total of 88 car-related attributes which were further used to train models for the prediction of socioeconomic status. Another example is the prediction of car accident risk using features visible from residential buildings. Kita and Kidziński [257] examined 20,000 records from an insurance dataset and collected Google Street View (GSV) for addresses listed in these records. They annotated the residence buildings by their age, type, and condition and applied these variables to a Generalized Linear Model (GLM) [264,265] to investigate if they contribute to better prediction of accident risk for residents. The results showed significant improvement to the models used by the insurance companies for accident risk modeling. Street-view images can also be utilized to study the association between the greenspace in a neighborhood and its socioeconomic effects [266].

• Calculation of sky view factors

The sky view factor (SVF) [267] represents the ratio between the visible sky and the overlaying hemisphere of an analyzed location. It is widely used in diverse fields, such as urban management, geomorphology, and climate modeling [268,269,270,271]. In general, there are three types of SVF calculation methods [272]. The first is a direct measurement from fisheye photos [273,274]. It is accurate but requires on-site work. The second method is based on simulation, where a 3D surface model is built and SVFs are calculated based on this model [275,276]. This method relies on accurate simulation, but it is hard to get precise parameters in complex scenes. The last method is based on street-view images. Researchers use public street-view image sources, such as GSV, and project images to synthesize fisheye photos at given locations [277,278,279,280]. Due to the rapid development of street-view services, this method is applied at relatively low cost, because images of most places are becoming readily available. Hence, it has seen increasing application and has become a major data source for extracting sky view features.

Middel et al. [270] developed a methodology to calculate SVFs from GSV images. The authors retrieved images from a given area and synthesized them into hemispherical view (fisheye photos) by equiangular projection. A combination of a modified Sobel filter [281] and flood-fill edge-based detection algorithm [282] was applied on the processed images to detect the area of visible sky. The SVFs were then calculated at each location using tools implemented by [280]. The derived SVFs can be further used on various applications, such as local climate zone evaluation and sun duration estimation [270]. Besides SVF, view features of different natural scenes, such as trees and buildings, are also important in urban-environmental studies. Gong et al. [272] utilized a deep learning algorithm to extract three street features simultaneously (sky, trees, and buildings) in a high-density urban environment to calculate their view factors. The authors sampled 33,544 panoramic images in Hong Kong from GSV and segment images with Pyramid Scene Parsing Network (PSPNet) [283]. This network assigns each pixel in the image into categories, such as sky, trees, and buildings. Then, the segmented panoramic images are projected into fisheye images [278]. Since each image provides segmented areas of corresponding categories, a simple classical photographic method [284] was applied to calculate different view factors. Recently, Shata et al. [285] determined the correlation between the sky view factor and the thermal profile of an arid university campus plaza to study the effects on the university’s inhabitants. Sky view factor estimation is also a key technique for understanding urban heat island effects and how different landscape factors contribute to increased land surface temperatures in (especially desert) cities for developing mitigation strategies for extreme heat [286].

• Neighborhood change monitoring

In GSV, Google updates its street view image database regularly. Therefore, besides being able to access a single GSV image of a property, there are studies leveraging multiple GSV images to detect visible changes to the exteriors of properties (i.e., housing) over time. To monitor neighborhood changes, some scholars have collected data from in-person interviews, mailed questionnaires, or visual perception surveys [287,288,289,290,291]. Surveys can be customized thus the details are in-depth and fine-grained. However, this method may result in human bias or coverage of only a small geographical area. Other researchers have employed a GSV database but examined the images manually [292,293,294,295]. This reduces on-site efforts, but it is difficult to scale up in these studies. Recently, thanks to the advances of machine learning and computer vision, researchers are able to automatically audit the environment in a large urban center with huge quantities of socio-environmental data. For example, Naik et al. [296] used a computer vision method to quantify physical improvements of neighborhoods with time-series, street-level imagery. They sampled images from five U.S. cities and calculated the perception of safety with Streetscore, introduced in Naik et al. [297]. Streetscore includes (1) segmenting images into several categories, such as buildings and trees [298], (2) extracting features from each segmented area [299,300], and (3) predicting a score of a street in terms of its environmental pleasance [301]. The difference in the scores at a given location but with different timestamps can be used to measure physical improvement of the environment. The scores are found to have a strong correlation with human generated rankings. Another example is the detection of gentrification in an urban area [302]. The authors proposed a Siamese-CNN (SCNN) to detect if an individual property has been upgraded between two time points. The inputs are two GSV images of the same property at different timestamps and the output is the resulting classification indicating if the property has been upgraded.

• Identification of human perceptions of places

Quantifying the relationship between human perceptions and corresponding environments has been of great interest in many fields, such as geospatial intelligence, and cognitive and behavioral sciences [303]. Early studies usually used direct or indirect communications to investigate human perceptions [304,305,306]. This may result in human bias and is hard to apply to study large geographical (urban) regions. The emergence of new technologies, such as deep learning, and geo-related cloud services, such as Flickr and GSV, provide advanced methods and data sources for large-scale analysis of human sensing about the environment. For example, Kang et al. [307] extracted human emotions from over 2 million faces detected from over 6 million photos and then connected emotions with environmental factors. They first focused on famous tourist sites and their corresponding geographical attributes from Google Maps API and Flickr photos using geo-tagged information by Flickers API. Next, they utilized DBSCAN [308] to construct spatial clusters to represent hot zones of human activities and further used Face++ Emotion Recognition (https://www.faceplusplus.com/emotion-recognition/, accessed on 1 March 2022) to extract human emotions based on their facial expressions. Based on the results, the authors were able to identify the relationship between environmental conditions and variations in human’s emotions. This work extends the study to the global scale based on crowdsourcing data and deep learning techniques. Similar methodologies also appear in various works [297,309,310]. This research is extended to places beyond tourist sites with GSV services. Zhang et al. [303] proposed a Deep Convolutional Neural Network (DCNN) to predict human perceptions in new urban areas from GSV images. A DCNN model was trained with the MIT Places Pulse dataset [311] to extract image features and predict human perceptions with Radial Basis Function (RBF) kernel SVM [312]. To identify the relationship between sensitive visual elements of a place and a given perception, a series of statistical analyses, including segmenting images into object instances and multivariate regression analysis, were conducted to identify the correlation between segmented object categories and human perceptions. With the number of mobile devices crossing 4 billion in 2020 and a projected rise to 18 billion in the next 5 years, the best method for detecting and monitoring human emotions would be to make use of edge devices, e.g., IoT sensors. Also, with the increasing volume of data, edge computing for emotion recognition [313] using a CNN “on the edge” has also become a very efficient approach.

• Personality and place uniqueness mining

Understanding the visual discrepancy and heterogeneity of different places is important in terms of human activity and socioeconomic factors. Earlier studies for place understanding were mainly based on social surveys and interviews [314,315]. Recently, the availability of large-scale street imagery, such as GSV, and the development of computer vision techniques yield the ability for automated semantic understanding of an image scene and the physical, environmental, and social status of the corresponding location. Zhang et al. [316] proposed a framework which formalizes the concept of place in terms of locale. The framework contains two components, street scene ontology and the street view descriptor. In the street view ontology, a deep learning network, PSPNet [283], was utilized to semantically segment a street-view image into 150 categories from 64 attributes representing street scenes basics. For quantitatively describing the street view, a street visual matrix and street visual descriptor were generated from the results of scene ontology. These two values were then used to examine the diversity of street elements for a single street or to compare two different streets. Another example is the estimation of geographic information from an image at a global scale. Weyand et al. [317] proposed a CNN-based model with 91 million photos for image location prediction. To increase model feasibility, they partitioned the Earth’s surface based on a photo distribution such that densely populated areas were covered by finer-granule cells and sparsely populated areas were covered by coarser-granule cells. This work is extended by integrating long-short term memory (LSTM) into the analysis because photos naturally occur in sequences. This way, the model can share geographical correlations between photos and improve the prediction accuracy for the locations where an image is taken. Zhao et al. [318] leveraged the building bounding boxes detected from images and embeds this context back into the CNN model for prediction of a more accurate label describing a building’s functions (e.g., residential, commercial, or recreational). Another aspect of the personality of a place is the amount of criminal activity it witnesses. An interesting research article by Amiruzzaman et al. [319] proposed a model that makes use of street view images supplemented by police narratives of the region to classify neighborhoods as high/low crime areas.

• Human activity prediction

Understanding human activity and mobility in greater spatial and temporal detail is crucial for urban planning, policies evaluation, and the analysis of health and environmental impacts to residents of different design and policy decisions [320,321,322]. Earlier studies have often relied on data collected from household surveys, personal interviews, or questionnaires. These data provide great insight on personal patterns; however, it takes significant resources to collect them at regional to national levels and they are difficult to update. In recent years, emerging big data resources, such mobile phone data [323,324,325] and geo-tagged photo [326,327], have provided new opportunities to develop cost-effective approaches for gaining a deep understanding of human activity patterns. For example, Calabrese et al. [323] proposed a methodology to utilize mobile phone data for transportation research. The authors applied statistical methods on the data to estimate properties, such as personal trips, home locations, and other stops in one’s daily routine. In addition to phone and photo data, GSV images are another data source that are even more consistent, cost-effective, and scalable. Recent studies [320,328,329,330] that have employed GSV images have shown the data’s great potential for large-scale comparative analysis. For example, Goel et al. [328] collected 2000 GSV images from 34 cities to predict travel patterns at the city level. The images were first classified into seven categories of functions, e.g., walk, cycle, and bus. A multivariable regression model was applied to predict official measures from road functions detected from the GSV images. Human activity can also be reliably mapped [331] by making use of remote-sensing images to overcome the unavailability of mobile positioning data due to security and privacy concerns.

5.3. GeoAI for Scientific Data Analytics

In recent years, AI and deep learning have also been increasingly applied to understand the changing conditions of Earth’s systems. Reichstein et al. [332] identified five major challenges for the successful adoption of deep learning approaches to Earth and other geoscience domains. They are interpretability, physical consistency, complex data, limited labels, and computational demand. To address these challenges, various studies with different applications have been developed. In

Table 3. Summary of GeoAI and deep learning applications in geoscientific data analysis, as well as limitations of conventional techniques.

Task	Applications	Conventional Approaches	Limitation of Conventional Approaches	ML/DL Approaches
Precipitation nowcasting
	Safety guidance for traffic Emergency alerts for hazardous events	NWP-based method Optical flow techniques on radar echo map	Resource-intensive Subjective pre-defined parameters Lack of end-to-end optimization	2D/3D CNN RNN CNN (spatial correlation) + RNN (temporal dynamics)
Extreme climate events detection
	Disaster preparation and response	Simulation tools	Subjective pre-defined parameters Inconsistent assessments on the same events	CNN classification on multiple stacked spatial variables 3D CNN
Earthquake detection and phase picking
	Disaster alerts and response	Picking-based Waveform-based	Weak signal filtering capability Prior information requirement Less sensitive to smaller events	CNN classification on waveform signals RNN for waveform matching CNN (noise filter) + RNN (phase picking) CNN + RNN + transformer
Wildfire spread modeling
	Safety guidance for firefighters Public risk reducing Urban planning	Physics-based models Cell-based methods Vector-based methods	Subjective assumptions Limited model complexity	2D/3D CNN
Mesoscale ocean eddy identification and tracking
	Near-surface winds, clouds, rainfall, and marine ecosystems Ocean energy and nutrient transfer	Physical parameter-based Geometric based	Subjective pre-defined parameters Limited generalization Unclear geometrical features	Classification Object detection Semantic segmentation

, we summarize the applications of various kinds of geoscientific data, as well as traditional and novel methods (GeoAI and deep learning) in their analysis.

• Precipitation nowcasting

Precipitation nowcasting refers to the goal of giving very short-term forecasting (for periods up to 6 h) of the rainfall intensity in a local area [333]. It has attracted substantial attention because it addresses important socioeconomic needs, for example, giving safety guidance for traffic (drivers, pilots) and generating emergency alerts for hazardous events (flooding, landsides). However, timely, precise, and high-resolution precipitation nowcasting is challenging because of the complexities of the atmosphere and its dynamic circulation processes [334]. Generally, there are two types of precipitation nowcasting approaches: the numerical weather prediction (NWP)-based method and radar echo map-based method. The NWP-based method [334,335] builds a complex simulation based on physical equations of the atmosphere, for example, how air moves and how heat exchanges. The simulation performance strongly relies on computing resources and pre-defined parameters, such as initial and boundary conditions, approximations, and numerical methods [335]. In contrast, the radar echo map-based method is becoming more and more popular due to its relatively low computing demand, fast speed, and high accuracy at the nowcasting timescale. For the radar echo map-based method, each map is transformed into an image and fed into the prediction algorithm/model. The algorithm/model learns to extrapolate future radar echo images from the input image sequence. Two factors are involved in the learning process: spatial and temporal correlations of the radar echoes. Spatial correlation represents the shape deformation while temporal correlation represents the motion patterns. Thus, precipitation nowcasting is similar to the motion prediction problem from videos where input and output are both spatiotemporal sequences and the model captures the spatiotemporal structure of the data to generate the future sequence. The only difference is that precipitation nowcasting has a fixed view perspective which is the radar itself.

Early studies [336,337,338] used optical flow techniques to estimate the wind direction from two or more radar echo maps for predicting movement of the precipitation field. However, there are several flaws in optical flow-based methods. The wind estimation and radar echo extrapolation steps are separated so the estimation cannot be optimized from the radar echo result. Further, the algorithm requires pre-defined parameters and cannot be optimized automatically by massive amounts of radar echo data. Recently, deep learning-based models [339,340,341] have been developed to fix the flaws by end-to-end supervised training, where the errors are propagated through all components and the parameters are learned from the data. There are typically three deep learning-based architectures for precipitation nowcasting or video prediction: CNN, RNN, and CNN+RNN+based models. For CNN-based models, frames are either treated as different channels in a 2D CNN network [339,340] or as the depth in a 3D CNN network [342]. For RNN-based models, Ranzato et al. [341] built the first language model to predict the next video frame. The authors split each frame into patches, convoluted them with 1 × 1 kernel and encoded each patch by the k-means clustering algorithm. The model then predicts the patch at the next time step.

Srivastava et al. [343] further proposed a LSTM encoder-decoder network to predict multiple frames ahead. Although both CNN-based and RNN-based models can solve the spatiotemporal sequence prediction problem, they do not fully consider the temporal dynamics or the spatial correlations. By using RNN for temporal dynamics and CNN for spatial correlations, Shi et al. [344] integrated two networks together and proposed ConvLSTM. The authors replaced the fully connected layers in LSTM with convolutional operations to exploit spatial correlations in each frame. This work became the milestone for spatiotemporal prediction and the basis for various subsequent approaches; for example, ConvLSTM used with dynamic filters [345], with a new memory mechanism [346,347] optimized to be location-variant [348], with 3D convolution [349], and with an attention mechanism [350]. All these studies model data from the spatiotemporal domain, however, there are also studies that focus on the spatial layout of an image and the corresponding temporal dynamics separately [351,352].

• Extreme climate events detection

Detecting extreme climate events, such as tropical cyclones and weather fronts, is important for disaster preparation and response, as they may cause significant economic impact [353] and risk to human health [354]. Early studies [355,356] often defined occurrence of events when values of relevant variables were compared to a subjective threshold defined by domain experts. However, this brings the first challenge, different methods may adopt different threshold values for the same variables which leads to inconsistent assessment of the same data. Fortunately, deep learning enables the machine to automatically learn to extract distinctive characteristics and capture complex data distributions that define an event without the need for hand-crafted feature engineering. Li et al. [357] developed the first climate event classification CNN model. The authors stacked relevant spatial variables, e.g., pressure, temperature, and precipitation, together into image-like patches and processed them by CNN as a visual pattern classification problem to detect extreme climate events. Instead of single time-frame image classification, Racah et al. [358] developed a CNN model for multi-class localization of geophysical phenomena and events. The authors adopted and trained a 3D encoder-decoder convolutional network with 16-variate 3D data (height, width, and time). The result showed that 3D models perform better than their 2D counterparts. The research also finds that the temporal evolution of climate events is an important factor for accurate model detection and event localization.

Zhang et al. [359] also leveraged temporal information and constructed a similar 3D dataset for nowcasting the initiation and growth of climate events. One challenge is how to effectively utilize massive volumes of diverse data. Modern instruments could collect 10s to 100s of TBs of multivariate data, e.g., temperature and pressure, from a single area. This puts human experts and machines into a challenging position for processing and quantitatively assessing the big dataset. To address this challenge, parallel computing is the most common way to speed up model training and deploying. However, performance depends not only on the total number of nodes but how data are distributed and merged across the nodes. Kurth et al. [360,361] implemented a series of improvements to the computing cluster and the data pipeline, such as I/O (Input/Output), data staging, and network. The authors successfully scaled up the training from a single computing node to 9600 nodes [360] and the data pipeline to 27,360 GPUs [361]. As the data volume increases, the quality of training data becomes another important factor influencing model performance, especially for deep learning models where the performance is strongly correlated with the amount and quality of available training data. The ground-truth of climate event detection often comes from traditional simulation tools, for example, TECA (A Parallel Toolkit for Extreme Climate Analysis) [355]. These tools generate predictions following a certain combination of criteria provided by human experts. However, it is possible that errors occur in the results and the models learn from those errors as a result. To address this issue, various methods were developed including semi-supervised learning [358], labeling refinement [362], and atmospheric data reanalysis [363,364].

• Earthquake detection and phase picking

An earthquake detection system includes several local and global seismic stations. At each station, ground motion is recorded continuously, and this includes earthquake and non-earthquake signals, as well as noises. There are generally two methods to detect and locate an earthquake: picking-based and waveform-based. For the picking-based method, workflow involves several stages, including phase detection/picking, phase association, and event location. In the phase detection/picking stage, the presence of seismic waves is identified from recorded signals. Arrival times of different seismic waves (P-waves and S-waves) within an earthquake signal are measured. In the association stage, the waves at different stations are aggregated together to determine if their observed times are consistent with travel times from a hypothetical earthquake source. Finally, in the event location stage, the associated result is used to determine earthquake properties, such as location and magnitude.

Early studies used hand-crafted features, e.g., changes of amplitude, energy, and other statistical properties, for phase detection and picking [365,366,367]. For phase association, methods include travel time back-projection [368,369,370], grouping strategies [371], Bayesian probability theory [372], and clustering algorithms [373]. For event locating, Thurber [374] and Lomax et al. [375] developed corresponding methods, such as a linearized algorithm and a global inversion algorithm. In contrast to the multi-stage picking method, the waveform-based method detects, picks, associates, and locates earthquakes in a single step. Some methods, such as template matching [376,377] and back-projection [378,379], exploit waveform energy or coherence from multiple stations. Generally, the picking-based method is less accurate because some weak signals might be filtered out in the detection/picking phase. As a result, it is unable to exploit potential informative features across different stations. On the other hand, the waveform-based method requires some prior information and is computationally expensive because of an exhaustive search of potential locations.

Recently, deep learning-based methods have been exploited for earthquake detection. Perol et al. [380] developed the first CNN for earthquake detection and location. The authors input waveform signals into 2D CNN-like images to perform a classification task. The output indicates the corresponding predefined geographic area where the earthquake originates. A similar strategy was applied in the detection/picking phase of CNN to classify input waves [381,382,383,384]. Zhou et al. [385] further combined CNN with RNN as a two-stage detector. The first CNN stage was used to filter out noise and the second RNN stage for phase picking. Mousavi et al. [386] proposed a multi-stage network with CNN, RNN, and a transformer model to classify the existence of an earthquake, P-waves, and S-waves separately. As for the association phase, McBrearty [387] trained a CNN to perform a binary classification of whether two waveforms between two stations are from a common source. Differently, Ross et al. [388] used a RNN to match two waveforms to achieve cutting-edge precision in associating earthquake phases to events which may occur nearly back-to-back to each other. In addition to all the above work, Zhu et al. [389] proposed a multi-task network to perform phase detection/picking and event location in the same network. The network first extracts unique features from input waveforms recorded at each station. The feature is then processed by two sub-networks for wave picking and for aggregating features from different stations to detect earthquake events. Such a new deep learning-based model is capable of processing and fusing massive information from multiple sensors, and it outperforms traditional phase picking methods and achieves analyst-level performance.

• Wildfire spread modeling

Wildfires have resulted in loss of life and billions of dollars of infrastructure damage [390]. The U.S. National Climate Assessment points out a trend of increasing wildfire frequency, duration, and impact on the economy and human health [391]. At the same time, fighting wildfires is extremely complex because it involves the consideration of many location-variant physical characteristics. Wildfire modeling can simulate the spread of a wildfire to help understand and predict its behavior. This increases firefighters’ safety, reduces public risk, and helps with long-term urban planning. The current wildfire spread models are mostly physics-based and provide mathematical growth predictions [392,393,394,395]. Simulations incorporate the impact of related physical variables. These can be categorized into several levels based on their complexity, assumptions, and components involved [396]. For example, some simulations use only fixed winds while others allow ongoing wind observations. Besides the complexity, there are two main categories of implementation methods: cell-based [397,398,399] and vector-based [400,401,402]. The cell-based method simulates fire evolution by the interaction among contiguous cells while the vector-based method defines the fire front explicitly by a given number of points. Some researchers have proposed AI-based approaches to predict the area to be burned or the fire size [403,404,405]. For example, Castelli et al. [404] predicted the burned area using forest characteristic data and meteorological data by genetic programming.

Recently, machine learning/deep learning has been used in wildfire spread modeling because the data for wildfire simulations are similar for images and all gridded data, such as fuel parameters and elevation maps [406]. Unlike previous AI-based approaches, a deep learning-based method not only estimates the total burned area but also the spatial evolution of the fire front through time. Ganapathi Subramanian and Crowley [407] proposed a deep reinforcement learning-based method in which the AI agent is the fire, and the task is to simulate the spread across the surrounding area. As for CNN, the difference between various studies is how they integrate non-image data, such as weather and wind speed, into the model; how they transform these data into image-like gridded data [408]; how they take scalar input and perform feature concatenation [409]; or how they use graph models to simulate wildfire spread [410]. Radke et al. [408] combined CNN with data collection strategies from geographic information systems (GIS). The model predicts which areas surrounding a wildfire are expected to burn during the following 24 h given an initial fire perimeter, location characteristics (remote sensing images, DEM), and atmospheric data (e.g., pressure and temperature) as input. The atmospheric data are transformed into image-like data and processed by a 2D CNN network. Allaire et al. [409] instead processed the same data, only as scalar inputs. The input scalar was processed by a fully connected neural network into 1024-dimension features and later concatenated with another 1024-dimension features from the input image processed by convolutional operations.

• Mesoscale ocean eddy identification and tracking

Ocean eddies are ocean flows that create circular currents. Ocean eddies with a diameter ranging from 10 to 500 km and a lifetime of days to months are known as mesoscale eddies [411]. They have coherent vortices and three-dimensional spiral structures. Due to their vertical structure and strong kinetic energy, mesoscale eddies play a major role in ocean energy and nutrient transfer, e.g., heat, salt, and carbon [412]. Mesoscale eddies have also been shown to influence near-surface winds, clouds, rainfall [413], and marine ecosystems in nearby areas [414,415]. Thus, the identification and tracking of mesoscale eddies is of great scientific interest. Based on their rotation direction, eddies can be associated with two types of atmospheric conditions: cyclones (in the Northern Hemisphere) and anticyclones. These types of mesoscale eddies result in different satellite-derived data, such as those related to sea level anomalies (SLAs). Cyclonic eddies cause a decrease in SLA and elevations in subsurface density while anti-cyclonic eddies cause an increase in SLA and depressions in subsurface density. Therefore, these characteristics enable the identification of mesoscale eddies from the satellite data. Similar satellite data include sea surface temperature (SST), chlorophyll concentration (CHL), sea surface height (SSH), and that from synthetic aperture radar (SAR).

Early studies of mesoscale eddy detection can be divided into two categories: by physical parameter-based or geometric-based methods. The physical parameter-based method requires a pre-defined threshold for the target region, for example, as determined by the Okubo-Weiss (W) parameter [416,417] method. W-parameter measures the deformation and rotation at a given fluid point. A mesoscale eddy is defined based on the calculated W-parameter and a pre-defined threshold [418,419,420]. Another application of the physical parameter-based method is wavelet analysis/filtering [421,422]. On the other hand, the geometric-based method detects eddies based on clear geometrical features, e.g., streamline winding-angle [423,424]. Some studies [425,426] proposed a combination of two methods. As for the issues, the physical parameter-based method is limited in generalization because the threshold is often region-specific, while the geometric-based method cannot easily detect eddies without clear geometrical features.

Recent deep learning-based ocean eddy detection alleviates both issues by training with data across different regions and extracting high-level features. These studies can be categorized into different types based on the task performed. The first type is classification. George et al. [427] classified eddy heat fluxes from SSH data. The authors compared different approaches, including linear regression, SVM [428], VGG [44], and ResNet [46], and found CNNs significantly outperformed other data-driven techniques. The next type is object detection. Duo et al. [429] proposed OEDNet (Ocean Eddy Detection Net), which is based on RetinaNet [59], to detect eddy centers from SLA data and they applied a closed contour algorithm [430] to generate the eddy regions. The last type is semantic segmentation. This is the most commonly used method because it directly generates the desired output without extra steps. Studies related to its use include [431,432,433,434,435]. Lguensat et al. [432] adopted U-Net [100] to classify each pixel into non-eddy, anticyclonic-eddy, or cyclonic-eddy from SSH maps. Both Xu et al. [434] and Liu et al. [433] leveraged PSPNet [283] to identify eddies from satellite-derived data. Although these studies adopt various networks, most of them fuse multi-scale features from the input, e.g., spatial pyramid operation [436] in PSPNet and FPN [58] in RetinaNet. These studies rely mainly on data-level fusion; future research can exploit feature-level fusion and the use of multi-source data for improved ocean eddy detection [89].

6. Discussions and Future Research Directions

In this paper, we reviewed recent advances of AI in Geography and the rise of the exciting transdisciplinary area of GeoAI. We also reviewed mainstream neural network models, including traditional fully connected neural network models and the more recent deep structures of CNN, RNN, LSTM, and transformer models. The breakthrough development of these deep learning models makes AI more suitable for geospatial studies, which often involve processing large amount of data and pattern mining across local, regional, and global scales. As it is very difficult to cover every single topic of this rapidly growing field with its massive amount of available literature, we focused our review on the GeoAI applications using image and other structured data, particularly remote sensing imagery, street view images, and geoscientific data. We touched on traditional and classic methods in each application area but put more weight on the recent deep learning-based research. While challenges of traditional methods differ in different applications while analyzing diverse types of (structured) geospatial data, we have found through our review that GeoAI and deep learning approaches present major strengths for the following objectives: (1) Large-scale analytics. The ability of GeoAI and deep learning approaches to process spatial and temporal big data make it stand out from traditional approaches, which are tailored to analyze small data. This advantage makes GeoAI models more robust and the results derived from such models more generalizable than that from traditional models, which are “trained” on data covering only a small geographical area. (2) Automation. Automated feature extraction is a key aspect of GeoAI, which is capable of self-learning the driving factors (i.e., independent variables) for an event or phenomenon (i.e., dependent variables) from raw data directly. Traditional approaches, including shallow machine learning, require the analyst to define the independent variables manually. An incomplete list of variables adopted in the analysis may impede us from gaining a comprehensive understanding of the research problem. GeoAI and deep learning models overcome this issue by making discoveries in a new data-driven research paradigm. The feature extraction process facilitated with automation also allows a GeoAI model to discover exciting and previously unknown knowledge. (3) Higher accuracy. Because of GeoAI models can capture complex relationships between dependent and independent variables, the analytical results, especially when the model is used for prediction, are usually more accurate than traditional methods, which may only be able to capture a linear or a simple non-linear relationship between the variables. (4) Sensitivity in detecting subtle changes. The higher accuracy of GeoAI models and their ability to detect hidden patterns offer advantages over traditional approaches in capturing minor changes in (dynamic) events and discerning the subtle differences among similar events. (5) Tolerance of noise in data. Many traditional statistical approaches, such as regression, are tailored for use on small and good quality data. GeoAI models, which can simultaneously consider huge amounts of data in its decision process, are better at distilling important associations with the presence of noise. (6) Rapid technological advancement. Rapid development in this area, as witnessed by an exponential increase of research related publications, reflects a strong community recognition of the scientific value of GeoAI.

As we are passing the stage of “import” in the proposed “import-adaptation-export” frame of GeoAI, it is critical to develop more research regarding domain adaptations. In this paper we have reviewed a wide variety of research that fits into this category, including the adaptation of image-based analysis and deep learning for land-use and land-cover analysis, weather and drought forecasting, quantification of neighborhood properties and monitoring neighborhood changes, as well as identifying human perception of places and predicting human activity. Domain applications also include detecting extreme climate events, modeling wildfire spread, identifying and tracking mesoscale ocean eddy toward an exciting data-driven earth system science research paradigm.

These advances will lead us into the “export” phase, when geospatial domain knowledge and spatial principles will be intensively used to guide the design of better AI models that could be used in other science domains, further increasing GeoAI’s reach and impact [437]. For instance, recently we have seen more innovative GeoAI research which integrates geographically weighted learning into neural network models such that instead of deriving a uniform value, the learned parameter could differ from place to place [438]. Work such as this addresses an important need for “thinking locally” [439] in GeoAI research. Also, research, such as that of Li et al. [27], tackles the challenge for obtaining high quality training data in image and terrain analysis by developing a strategy for learning from counting. The authors use Tobler’s First Law as the principle to convert 2D images into 1D sequence data so that the spatial continuity in the original data can be preserved to the maximum extent. They then developed an enhanced LSTM model which can take the 1D sequence and perform object localization without the need for the bounding box labels used in general object detection models to achieve high accuracy prediction with weak supervision. Research of this type addresses a critical need for thinking spatially [440]. Future research that represents a deep integration between Geography and AI, which can help better solve both geospatial problems and general AI problems, will contribute significantly to the establishment of the theoretical and methodological foundation of GeoAI and broaden its impact beyond Geography.

Other commonly acknowledged concerns include building GeoAI benchmark datasets, robust and explainable models, fusing and processing multi-source geospatial data sets, and enabling knowledge driven GeoAI research [17,437,441,442]. In recent years, increasing attention has been paid to ethical issues in GeoAI research. As geospatial data potentially contain personal information that could be used to intrude on one’s privacy by, for instance, predicting one’s travel behavior and home and work locations. Hence, protecting geospatial data from being misused and developing robust GeoAI models with increased transparency and unbiased decision making is critical not only for GeoAI researchers and its users, but also for the public [443]. This way, we can work together as a community to contribute to the development of a healthy and sustainable GeoAI ecosystem to benefit the entire society [12]. In this vein, we hope that researchers aiming to achieve the above objectives in their work could refer to this review paper to identify important and relevant GeoAI literature to jump start their research. We also hope that this paper will become an important checkpoint for the field of GeoAI, encouraging more in-depth applications of GeoAI across environmental and social science domains.