GeoBoost: An Incremental Deep Learning Approach toward Global Mapping of Buildings from VHR Remote Sensing Images

Abstract

Modern convolutional neural networks (CNNs) are often trained on pre-set data sets with a fixed size. As for the large-scale applications of satellite images, for example, global or regional mappings, these images are collected incrementally by multiple stages in general. In other words, the sizes of training datasets might be increased for the tasks of mapping rather than be fixed beforehand. In this paper, we present a novel algorithm, called GeoBoost, for the incremental-learning tasks of semantic segmentation via convolutional neural networks. Specifically, the GeoBoost algorithm is trained in an end-to-end manner on the newly available data, and it does not decrease the performance of previously trained models. The effectiveness of the GeoBoost algorithm is verified on the large-scale data set of DREAM-B. This method avoids the need for training on the enlarged data set from scratch and would become more effective along with more available data.

1. Introduction

In recent years, the size of satellite image data sets has been considerably enlarged compared with a decade ago. In real-world applications, such as global or regional mappings, large-scale data sets are built at multiple stages. For instance, the widely used data set, WHU-RS [1], is built through three versions. This data set was expanded from 12 classes of aerial scenes [1] to 19 [2] and 20 [3] classes, and the number of samples in each class was also increased. Actually, it is a tough task in which we decide whether a certain size of the data set is big enough, or whether the configuration of the semantic classes is reasonable before diving into the solid verification of the data set. Inevitably, large-scale data sets are built in a multi-stage manner. Correspondingly, most current models, which are adopted for the increasing data sets, have the weak continual-learning ability over time. This arouses the need for the models which are capable of continually fitting the growing data sets.

The incremental learning approaches focus on tackling the issue of studying sequentially acquired data without forgetting. This type of method is also referred to as continual learning [4,5,6,7] and lifelong learning [8,9,10]. The incremental learning approaches resorting to neural networks can be divided into the following categories [6]: regularization approaches, dynamic architectures, and memory replay. Regularization approaches add certain constraints to the update of weights to alleviate forgetting [11,12,13]. Dynamic architectures expand the networks dynamically by creating new weights or layers [14,15,16,17]. Memory replay stores a subset of samples or long-term information from the previous stage [5,18,19]. Without constraining the capacity of the model, the method of dynamic architectures is a more appropriate choice to achieve better performance [7]. Therefore, we focus on this type of method for the incremental learning of satellite images in this paper.

The task of incremental learning in this paper refers to the data-incremental learning which keeps the same classes for the data at different stages. Progressive neural networks [16] reuse each part of intermediate layers in previous models. It strengthens the representations of the final model but discards the supervision information of output layers. Network architecture evolving [17] employs neural architecture search (NAS) to optimize the entire models based on the new coming data by choosing from three options: creating new neurons, reusing the existing parameters, and tuning the existing parameters. It keeps the model size reasonable, whereas the procedure of NAS is time-consuming. Deep adaptation networks [20] attach new convolutional filters to the existing base network. These new convolutional filters are linear combinations of the trained convolutional filters of the base network. This allows the augmented network to adapt representations from previous tasks. The weights of linear combinations are determined by the controller modules. Expert gate [21] is a network of experts. It employs an auto-encoder gate to decide which expert to use at test time. The relatedness between the test sample and the relevant experts are measured by the reconstruction error of the auto-encoder gate. Error-driven incremental learning [22] grows the deep convolutional networks in a hierarchical manner. The increased capacities of networks are only beneficial for the top layers, and the feature extractor part is not expanded. The self-organizing incremental neural network [23] combines a variant of the Growing When Required (GWR) network [24] with pre-trained convolutional neural networks (CNNs) to hierarchically learn human actions. The pre-trained CNNs limit the representational power of the model. Incremental feature learning [25] adds the neurons of hidden layers of a denoising autoencoder to enhance the capacity of the network. These newly added features are trained on the collected hard examples. Then, it merges similar features to reduce the redundancy.

Incremental boosting [14] trains a base learner for each sequence of new coming data to construct an ensemble model. Essentially, it is an additive model, so the performance of trained base learners are not affected by the subsequent models. Thus, we develop the new algorithms based on the incremental boosting method in this paper. The previous studies of utilizing boosting for incremental learning [14,15,26] employ the AdaBoost algorithm [27] to adjust the weights of samples. In recent years, the gradient boosting algorithm produces state-of-the-art results on many application benchmarks [28,29]. It is quite flexible to adopt diverse loss functions. Therefore, we employ gradient boosting [30] for incremental learning in this paper.

For the large-scale applications of satellite images, geospatial distribution is one of the most distinguishing characteristics. As shown in Figure 1, satellite images from different regions have their own regularities and are diverse in terms of color, texture, and morphological structure. In addition, for the large-scale satellite images, the available training data are not evenly distributed. Figure 2 illustrates the geospatial distribution of the training labels used in this paper, which are derived from Open Street Map (OSM) [31]. It can be seen clearly that the labels of Europe, North America, and East Asia are more available than that of other areas. The geospatial information of satellite images can be utilized to guide the continual-learning process of models for improving prediction results for semantic segmentation. Additionally, for the consideration of data privacy, the researchers of satellite images can not always obtain training data from previous research, whereas the trained models of them are publicly available in general. If we have a trained global model without access to the original data set, training a new model from scratch is not the best choice when the new regional data are collected. Fine-tuning the existing model is a common approach to use the information from the trained model, but it will decrease the performance of the model on the original data set. Therefore, we propose a novel algorithm for geographically incremental learning according to the corresponding geospatial distribution for improving the prediction results, which is called GeoBoost in this paper. The GeoBoost method simplifies the optimization of gradient boosting in the task of semantic segmentation via convolutional neural networks (CNN) and attaches the geospatial distribution information of the large-scale satellite images to the base learner of gradient boosting.

The remainder of this paper is organized as follows. Section 2 describes the essential idea of GeoBoost. In Section 3, the experiment design, and its results are presented in detail. We discuss the results for the experiments in Section 4. Finally, we draw some conclusions in Section 5.

2. Methods

Boosting of neural networks has been adopted for incremental learning [14,15,26], where each base learner is trained in a manner that is similar to AdaBoost [27]. Different from previous studies, the GeoBoost algorithm employs gradient boosting [30] for incremental learning. In this section, we illustrate how gradient boosting, composed of neural networks, can be trained in an end-to-end way for geographically incremental learning. Furthermore, when gradient boosting is applied to large-scale satellite images, we show how the proposed algorithm, GeoBoost, assembles base learners of the ensemble model according to the corresponding geospatial distribution of data to improve the prediction results.

2.1. Gradient Boosting

The gradient boosting method [30] is a special type of ensemble learning approaches, which constructs additive models in a stage-wise strategy. More formally, with the training data $\mathbf{x}=\{x_1,\dots ,x_N\}$ and the corresponding labels $\mathbf{y}=\{y_1,\dots ,y_N\}$, gradient boosting optimizes the ensemble model

$$F_m\left(\mathbf{x}\right)=\sum _{m=0}^M{f}_m\left(\mathbf{x}\right)$$

(1)

to minimize the loss function $L(\mathbf{y},F(\mathbf{x}\left)\right)$, where $f_m\left(\mathbf{x}\right)$ is a single base learner. The whole optimization procedure is divided into multiple stages. In the m-th stage, based on the ensemble model from the previous stage

$$F_{m-1}\left(\mathbf{x}\right)=\sum _{i=0}^{m-1}{f}_i\left(\mathbf{x}\right),$$

(2)

the base learner $f_m\left(\mathbf{x}\right)$ of the current stage can be derived from $F_{m-1}\left(\mathbf{x}\right)$ and the loss function $L(\mathbf{y},F_{m-1}\left(\mathbf{x}\right))$ as

$$f_m\left(\mathbf{x}\right)=-{\rho }_m{g}_m\left(\mathbf{x}\right).$$

(3)

Here, $g_m\left(\mathbf{x}\right)$ is the gradient direction

$$g_m\left(\mathbf{x}\right)=\frac{\partial L\left(\mathbf{y},F_{m-1}\left(\mathbf{x}\right)\right)}{\partial F_{m-1}\left(\mathbf{x}\right)},$$

(4)

and ${\rho }_m$ is the step of the linear search along the negative gradient direction

$${\rho }_m=\underset{{\rho }_m}{argmin}L\left(\mathbf{y},F_{m-1}\left(\mathbf{x}\right)-{\rho }_m{g}_m\left(\mathbf{x}\right)\right).$$

(5)

In summary, the essential idea of gradient boosting is to optimize the base learner $f_m\left(\mathbf{x}\right)$ along the negative gradient direction of the loss function. Since $g_m\left(\mathbf{x}\right)$ is the gradient direction with respect to $F_{m-1}\left(\mathbf{x}\right)$, parameters ${\theta }_m$ of the base learner $f_m(\mathbf{x};{\theta }_m)$ can not directly be trained by Equation (3). In practice, ${\theta }_m$ is obtained in an equivalent form

$${\theta }_m=\underset{{\theta }_m}{argmin}{[g_m\left(\mathbf{x}\right)-h_m(\mathbf{x};{\theta }_m)]}^2$$

(6)

and

$$f_m\left(\mathbf{x}\right)=-{\rho }_m{h}_m\left(\mathbf{x}\right),$$

(7)

where $h_m(\mathbf{x};{\theta }_m)$ is a parameterized function. Taking into account all the mentioned factors, the generic procedure of the gradient boosting method is shown in Algorithm 1.

Algorithm 1. The algorithm of gradient boosting [30].

Input: The training data, $\mathbf{x}$, and the corresponding labels, $\mathbf{y}$; the parameterized function, $h(\mathbf{x},\theta )$.

1: $F_0\left(\mathbf{x}\right)={argmin}_{{\rho }_0,h_0}L(\mathbf{y},{\rho }_0{h}_0\left(\mathbf{x}\right))$ 2: for$m=1$ to M do 3:    $g_m\left(\mathbf{x}\right)=\frac{\partial L\left(\mathbf{y},F_{m-1}\left(\mathbf{x}\right)\right)}{\partial F_{m-1}\left(\mathbf{x}\right)}$ 4:    ${\theta }_m={argmin}_{{\theta }_m}{[g_m\left(\mathbf{x}\right)-h_m(\mathbf{x};{\theta }_m)]}^2$ 5:    ${\rho }_m={argmin}_{{\rho }_m}L\left(\mathbf{y},F_{m-1}\left(\mathbf{x}\right)-{\rho }_m{h}_m\left(\mathbf{x}\right)\right)$ 6:    $f_m\left(\mathbf{x}\right)=-{\rho }_m{h}_m\left(\mathbf{x}\right)$ 7:    $F_m\left(\mathbf{x}\right)=F_{m-1}\left(\mathbf{x}\right)+f_m\left(\mathbf{x}\right)$ 8: end for

Output:$F_m\left(\mathbf{x}\right)={\sum }_{m=0}^M{f}_m\left(\mathbf{x}\right)$

2.2. End-to-End Gradient Boosting

While neural networks are adopted as base learners of the gradient boosting algorithm, the existing studies [32,33] just replace $g_m\left(\mathbf{x}\right)$ and ${\rho }_m$ with the corresponding versions of neural networks. Actually, the optimization of gradient boosting and the training of a single neural network can be combined together to further simplify the training process.

In practice, the previous studies treat the score of the softmax function as the output of neural networks for classification. Unlike the aforementioned mode, we can assume that the activation values $\mathbf{z}$ of the layer before the softmax function $\sigma$ is the output of neural networks. Certainly, $\mathbf{z}$ is a K-dimensional vector of real values with respect to the K classes. Then, $\mathbf{z}$ is normalized by the softmax function $\sigma$ to obtain the final normalized score:

$$\sigma {\left(\mathbf{z}\right)}_j=\frac{e^{z_j}}{{\sum }_{k=1}^K{e}^{z_k}};\mathrm{for}j=1,\dots ,K.$$

(8)

$$\sigma :{\mathbb{R}}^K\to {[0,1]}^K.$$

(9)

With the above definition of model, the base learner of gradient boosting can be expressed as $f_m\left(\mathbf{x}\right)={\mathbf{z}}_m$, and the ensemble model as

$$\begin{array}{ccc}\hfill F_m\left(\mathbf{x}\right)& =& \sigma \left(f_m(\mathbf{x};{\theta }_m)+\sum _{i=0}^{m-1}{f}_i\left(\mathbf{x}\right)\right)\hfill \end{array}$$

(10)

$$\begin{array}{cc}=& \sigma \left(\sum _{i=0}^m{f}_i\left(\mathbf{x}\right)\right)\hfill \end{array}$$

(11)

where ${\theta }_m$ are the parameters of the neural network $f_m$. In this form, the output of $F_m\left(\mathbf{x}\right)$ is normalized scores.

Since Algorithm 1 is proposed for the generic purpose, any corresponding optimization method of $h_m(\mathbf{x};{\theta }_m)$ can be utilized to find the solution of Equation (6). For instance, if $h_m(\mathbf{x};{\theta }_m)$ is a decision tree, it can be optimized by the CART algorithm [30,34]. Thus, Equations (6) and (4) are two separated steps. Generally, neural networks are trained by the gradient descent method. Likewise, gradient boosting is optimized along the negative gradient direction of the loss function. Instead of calculating $g_m(\mathbf{x}$) directly, we can incorporate this step into the process of gradient descent. In other words, the loss function can be optimized directly during the back propagation of neural networks without the extra step of obtaining $g_m\left(\mathbf{x}\right)$ explicitly. Therefore, this type of gradient boosting can be called end-to-end gradient boosting.

Supposing that we have the regression task with the squared loss $L(\mathbf{y},F_m\left(\mathbf{x}\right))={(y-F_m\left(\mathbf{x}\right))}^2$, the base model $f_0$ and $f_1$ have been learned, so we have the trained model $F_1\left(\mathbf{x}\right)=f_0\left(\mathbf{x}\right)+f_1\left(\mathbf{x}\right)$ and want to train the base model $f_2\left(\mathbf{x}\right)$ in the new iteration.

If $f_2$ is a decision tree, the gradient direction is

$$g_2\left(\mathbf{x}\right)=\frac{\partial L\left(\mathbf{y},F_1\left(\mathbf{x}\right)\right)}{\partial F_1\left(\mathbf{x}\right)}.$$

(12)

The decision tree $f_2$ has not been constructed yet, thus we can only get the gradients with respect to $F_1\left(\mathbf{x}\right)$ rather than $F_2\left(\mathbf{x}\right)=f_0\left(\mathbf{x}\right)+f_1\left(\mathbf{x}\right)+f_2\left(\mathbf{x}\right)$. After calculating $g_2\left(\mathbf{x}\right)$, the decision tree $f_2$ can be built with ${\theta }_2={argmin}_{{\theta }_2}{[g_2\left(\mathbf{x}\right)-f_2(\mathbf{x};{\theta }_2)]}^2$ using the CART algorithm [34]. In short, the gradient boosting method utilizes $g_2\left(\mathbf{x}\right)=\partial L/\partial F_1$ to guide the learning procedure of the parameter ${\theta }_2$.

If $f_2$ is a simple neural network $f_2\left(\mathbf{x}\right)={\theta }_2\mathbf{x}+b$, the gradient direction is

$$g_2\left(\mathbf{x}\right)=\frac{\partial L\left(\mathbf{y},F_2\left(\mathbf{x}\right)\right)}{\partial F_2\left(\mathbf{x}\right)}.$$

(13)

Different from decision trees, neural networks can be randomly initialized before the training procedure, so the gradient with respect to $F_2\left(\mathbf{x}\right)$ can be obtained. The backpropagation algorithm computes the gradients of parameters ${\theta }_2$ as

$$\frac{\partial L\left(\mathbf{y},F_2\left(\mathbf{x}\right)\right)}{\partial {\theta }_2}=\frac{\partial L\left(\mathbf{y},F_2\left(\mathbf{x}\right)\right)}{\partial F_2\left(\mathbf{x}\right)}\frac{\partial F_2\left(\mathbf{x}\right)}{\partial {\theta }_2}=g_2\left(\mathbf{x}\right)\frac{\partial F_2\left(\mathbf{x}\right)}{\partial {\theta }_2}$$

(14)

to guide the learning procedure of the parameters

$${\theta }_2^{\left(t\right)}={\theta }_2^{(t-1)}-\eta \frac{\partial L\left(\mathbf{y},F_2\left(\mathbf{x}\right)\right)}{\partial {\theta }_2},$$

(15)

where $\eta$ is the learning rate.

Using decision trees as base learners leads to two separate steps: calculate $\partial L/\partial F_1$, then use it to guide the learning procedure of the parameters. The advantage of gradient boosting is that the type of the base learner is not limited. Using neural networks as base learners does not need to calculate $\partial L/\partial F_2$ manually. It can be calculated by the backpropagation algorithm. Therefore, the end-to-end gradient boosting algorithm can only employ neural networks as base learners.

Different from line 3 in Algorithm 1 that $g_m\left(\mathbf{x}\right)$ is the gradient direction with respect to $F_{m-1}\left(\mathbf{x}\right)$, the base learner $f_m(\mathbf{x};{\theta }_m)$ can be optimized with respect to $F_m\left(\mathbf{x}\right)$ (Equation (10)) by gradient descent:

$$f_m(\mathbf{x};{\theta }_m)=\underset{{\theta }_m}{argmin}L\left(\mathbf{y},F_m\left(\mathbf{x}\right)\right).$$

(16)

During the training of $f_m\left(\mathbf{x}\right)$, the base learners from previous stages are frozen and non-trainable, and ${\theta }_m$ is the solely trainable parameters. Additionally, gradient descent of neural networks already searches the optimal point on the loss surface during training. Therefore, the linear search ${\rho }_m$ is redundant and can be omitted. Finally, we obtain a concise process of training gradient boosting in an end-to-end manner shown in Algorithm 2.

Algorithm 2. The algorithm of end-to-end gradient boosting.

Input: The training data, $\mathbf{x}$, and the corresponding labels, $\mathbf{y}$; the neural network, $f(\mathbf{x};\theta )$; and the softmax function, $\sigma$.

1: $F_0\left(\mathbf{x}\right)=\sigma \left(f_0(\mathbf{x};{\theta }_0)\right)$ 2: $f_0\left(\mathbf{x}\right)={argmin}_{{\theta }_0}L\left(\mathbf{y},F_0\left(\mathbf{x}\right)\right)$ 3: for$m=1$ to M do 4:   $F_m\left(\mathbf{x}\right)=\sigma \left(f_m(\mathbf{x};{\theta }_m)+{\sum }_{i=0}^{m-1}{f}_i\left(\mathbf{x}\right)\right)$ 5:   $f_m(\mathbf{x};{\theta }_m)={argmin}_{{\theta }_m}L\left(\mathbf{y},F_m\left(\mathbf{x}\right)\right)$ 6: end for

Output:$F_m\left(\mathbf{x}\right)=\sigma \left({\sum }_{i=0}^m{f}_i\left(\mathbf{x}\right)\right)$

2.3. GeoBoost

For the purpose of incremental learning, each base learner of boosting is trained on a set of new coming data rather than on the same one. For instance, the training collection $(\mathbf{X},\mathbf{Y})$ is composed by a group of image sets $\mathbf{X}=\{{\mathbf{x}}_0,\dots ,{\mathbf{x}}_m,\dots ,{\mathbf{x}}_M\}$ and the corresponding group of label sets $\mathbf{Y}=\{{\mathbf{y}}_0,\dots ,{\mathbf{y}}_{\mathbf{m}},\dots ,{\mathbf{y}}_M\}$, where each image set ${\mathbf{x}}_m=\{x_1,\dots ,x_N\}$ is a bunch of images and the corresponding label set is ${\mathbf{y}}_{\mathbf{m}}=\{y_1,\dots ,y_N\}$, respectively. If $x_1$ is an image with a size of $512\times 512$, then $y_1$ is the label matrix of the semantic segmentation task with a size of $512\times 512$. Thus, the image set ${\mathbf{x}}_1=\{x_1,x_2,x_3\}$ contains three images, and the corresponding label set ${\mathbf{y}}_1=\{y_1,y_2,y_3\}$ contains three label matrices. With two image sets ${\mathbf{x}}_1=\{x_1,x_2,x_3\}$, ${\mathbf{x}}_2=\{x_4,x_5\}$, their label sets are ${\mathbf{y}}_1=\{y_1,y_2,y_3\}$ and ${\mathbf{y}}_2=\{y_4,y_5\}$. Hence, the group of image sets $\mathbf{X}=\{{\mathbf{x}}_1,{\mathbf{x}}_2\}$ contains five images in total, and the group of label sets $\mathbf{Y}=\{{\mathbf{y}}_1,{\mathbf{y}}_2\}$ contains five label matrices in total.

As each base learner $f_m({\mathbf{x}}_m;{\theta }_m)$ trained on the data set $({\mathbf{x}}_m,{\mathbf{y}}_m)$, then Equation (10) is transformed into

$$F_m\left(\mathbf{X}\right)=\sigma \left(f_m({\mathbf{x}}_m;{\theta }_m)+\sum _{i=0}^{m-1}{f}_i\left({\mathbf{x}}_i\right)\right).$$

(17)

By utilizing gradient boosting for incremental learning [14,15,26], the performance of existing base learners is not affected by the newly added component, and the base learner in the current stage is trained without touching the data involved in previous stages. Additionally, the ensemble model can also reuse the classification capability of existing base learners.

For the large-scale satellite images, geospatial distribution is their most notable characteristic. Generally, the collected satellite images are clustered in certain areas (as shown in Figure 3), and the objects in these images from different areas are diverse in terms of color, size, and density. When gradient boosting is applied to the large-scale satellite images, the geospatial distribution information of data can be considered to improve the prediction results.

For instance, suppose that we separately collect satellite images from Europe and America, then we train a model $F_a$ with the data from Europe and train a model $F_b$ with the data from America. The model $F_a$ will perform well on the data from Europe, but deteriorate on the data from America. Images from these two areas are much diverse and $F_a$ has never touched the data from America, so it fails to predict other areas. One solution is to make the model dedicate to a certain area. In this manner, $F_a$ will be trained on the data from Europe and predict results for the data from Europe, and $F_b$ will be trained on the data from America and predict results for the data from America.

Specifically, in the m-th stage of gradient boosting, the coverage area of the training data set ${\mathbf{x}}_m$ can be expressed as geographic coordinates in the form of bounding box $B_m=(x_{min},y_{min},x_{max},y_{max})$. Since the base learner $f_m$ is trained on ${\mathbf{x}}_m$, we can set $B_m$ as the coverage area of the base learner $f_m({\mathbf{x}}_m;{\theta }_m,B_m)$. For a certain image $x_j$, it is solely classified by the base learners that cover the geographic location $p_j$ of the image. Consequently, the ensemble model becomes

$$F_m\left(x_j\right)=\sigma \left(\sum _{i=0}^{m}r(p_j,B_i)·f_i(x_j,p_j;B_i)\right),$$

(18)

where $r(p_j,B_i)$ is an indicator function

$$r(p_j,B_i)=\left\{\begin{array}{ccc}\hfill 1& ,\hfill & \hfill \mathrm{if}{p}_j\mathrm{is}\mathrm{in}{B}_i,\\ \hfill 0& ,\hfill & \hfill \mathrm{else}.\end{array}\right.$$

(19)

Due to the property that base learners adhere to certain areas, we name this algorithm GeoBoost. In the rest of this paper, the indicator function $r(p_j,B_i)$ is simply denoted as $r_i$, and $f_i(x_j,p_j;B_i)$ as $f_i\left(x_j\right)$. The entire process of GeoBoost is shown in Algorithm 3. The base learners of the original gradient boosting method do not take any geospatial information into consideration, which means that they can be applied to any area. In other words, they can be treated as a special case of GeoBoost that covers the entire range of the geographic coordinates system.

Boosting methods assemble a bunch of base learners to form the final ensemble model. Due to plenty of parameters, the ensemble model can easily cause the over-fitting problem. The gradient boosting method exerts a learning rate to each base learner as regularization for preventing over-fitting [30]. The learning rate $\nu$ and the number of base learners M jointly determine the performance of the boosting model. Small learning rates can reduce the dominance of a single base learner. With the presence of the learning rate $\nu$, we can reach the final algorithm as shown in Algorithm 4.

Algorithm 3. The algorithm of GeoBoost.

Input: The training collection of data, $\mathbf{X}=\{{\mathbf{x}}_0,\dots ,{\mathbf{x}}_M\}$, and the corresponding labels, $\mathbf{Y}=\{{\mathbf{y}}_0,\dots ,{\mathbf{y}}_M\}$; the neural network, $f(\mathbf{x};\theta ,B)$; the softmax function, $\sigma$; and the indicator function,$r_i$.

1: $F_0\left({\mathbf{x}}_0\right)=\sigma \left(f_0({\mathbf{x}}_0;{\theta }_0,B_0)\right)$ 2: $f_0\left({\mathbf{x}}_0\right)={argmin}_{{\theta }_0}L\left({\mathbf{y}}_0,F_0\left({\mathbf{x}}_0\right)\right)$ 3: for$m=1$ to M do 4:   $F_m\left(x_j\right)=\sigma \left({\sum }_{i=0}^m{r}_i·f_i\left(x_j\right)\right)$ 5:    $$f_m({\mathbf{x}}_m;{\theta }_m,B_m)=\underset{{\theta }_m}{argmin}{E}_{\underset{y_j\in {\mathbf{y}}_m}{x_j\in {\mathbf{x}}_m}}\left[L\left(y_j,F_m\left(x_j\right)\right)\right]$$ 6: end for

Output:$F_m\left(x\right)=\sigma \left({\sum }_{i=0}^m{r}_i·f_i\left(x\right)\right)$

Algorithm 4. The algorithm of GeoBoost with learning rates.

Input: The training collection of data, $\mathbf{X}=\{{\mathbf{x}}_0,\dots ,{\mathbf{x}}_M\}$, and the corresponding labels, $\mathbf{Y}=\{{\mathbf{y}}_0,\dots ,{\mathbf{y}}_M\}$; the neural network, $f(\mathbf{x};\theta ,B)$; the softmax function, $\sigma$; the learning rate of GeoBoost, $\nu$; and the indicator function, $r_i$.

1: $F_0\left({\mathbf{x}}_0\right)=\sigma \left(f_0({\mathbf{x}}_0;{\theta }_0,B_0)\right)$ 2: $f_0\left({\mathbf{x}}_0\right)={argmin}_{{\theta }_0}L\left({\mathbf{y}}_0,F_0\left({\mathbf{x}}_0\right)\right)$ 3: for$m=1$ to M do 4:   $F_m\left(x_j\right)=\sigma \left(r_m·f_m\left(x_j\right)+{\sum }_{i=0}^{m-1}\nu ·r_i·f_i\left(x_j\right)\right)$ 5:    $$f_m({\mathbf{x}}_m;{\theta }_m,B_m)=\underset{{\theta }_m}{argmin}{E}_{\underset{y_j\in {\mathbf{y}}_m}{x_j\in {\mathbf{x}}_m}}\left[L\left(y_j,F_m\left(x_j\right)\right)\right]$$ 6: end for

Output:$F_m\left(x\right)=\sigma \left({\sum }_{i=0}^m\nu ·r_i·f_i\left(x\right)\right)$

3. Experiment

3.1. Data Set

For the semantic segmentation tasks of satellite images, the commonly used data sets, such as the INRIA building data set [35] and ISPRS 2D Semantic Labeling Benchmark [36], are not distributed widely enough, which are built from just a few cities. Therefore, we create a new worldwide building data set to simulate the real application from 100 different cities as shown in Figure 2. This new data set is named as Building data set for Disaster Reduction and Emergency Management (DREAM-B). The training images are collected from Google Earth Engine [37], and the corresponding labels are obtained from the open-source map, Open Street Map (OSM) [31]. DREAM-B contains 626 image tiles with a size of $4096\times 4096$. We split out 250 tiles for training, 63 for validation, and 313 for testing. This data set solely contains two classes: the buildings and the non-building class. Each image tile of the DREAM-B data set is composed of red (R), green (G), and blue (B) bands, and its spatial resolution is 30 cm.

For the task of incremental learning, the 250 training image tiles are divided into four groups based on their geospatial location rather than sampled uniformly. As shown in Figure 3, we build a tiny global data set for the first group and add the local data sets for the other groups. The amount of image tiles of four groups is roughly equal. This manner of division is to simulate the real situation that a global dataset might be built in an incremental manner, e.g., from coarse to fine and from one region to another one.

3.2. Implementation Details

We combine the U-Net model [38] with the NASNet-Mobile model [39] for the architecture of the base learners of the GeoBoost algorithm. Specifically, the convolutional modules in U-Net are replaced by the neural cell obtained via neural architecture searching [39], which is more efficient for the consideration of computation efficiency. This model is called U-NASNetMobile in this paper. The input size of U-NASNetMobile is $512\times 512$, and the original image tiles of the DREAM-B data set is split into small patches with a size of $512\times 512$ to match the model. In the recent studies [40,41], researchers found that a larger training sample size produces better performance for neural networks. However, if the GPU memory is fixed, a larger training sample size will lead to the smaller batch size of gradient descending. According to the rule of linear scaling learning rate [40], a small batch size increases the noise in the gradient, so the learning rate of gradient descending may be decreased. For the consideration of the balance between the batch size and the training sample size, the input size of U-NASNetMobile is $512\times 512$. The architecture of U-NASNetMobile is shown in Figure 4. Data augmentations are employed to avoid over-fitting, including random flipping horizontally and vertically, random rotation, and random brightness jittering.

The Adam optimizer [42] is used for optimization. We train models with the cosine learning rate annealing schedule [43] with a maximum learning rate of $3\times {10}^{-4}$ and a minimum learning rate of $1\times {10}^{-6}$. Unless otherwise specified, all the experiments are trained for 200 epochs with a mini-batch size of 16. In addition, the Intersection over Union (IoU) is employed as the evaluation accuracy [44]. More specifically, the IoU accuracy is defined as

$$\mathrm{IoU}=\frac{\mathrm{Prediction}\bigcap \mathrm{GroundTruth}}{\mathrm{Prediction}\bigcup \mathrm{GroundTruth}}.$$

(20)

The metrics of Precision, Recall, Overall Accuracy, and Kappa Coefficient are also provided for reference.

4. Results and Discussion

In this section, we first analyze the impacts of technical factors in Section 4.1. Then, the quantitative and qualitative evaluation of different models are presented in the Section 4.2 and Section 4.3. Finally, we dive into the regional impacts of prediction in Section 4.4.

4.1. Technical Analysis

4.1.1. Pre-Training

Base learners of the boosting algorithm are trained sequentially. After the base model $f_0$ is trained, the weights of other base learners can be initialized by the trained weights of $f_0$ instead of random initialization. This is an application of the pre-training approach [45]. As illustrated in Figure 5, the base model $f_1$ with pre-training on $f_0$ converges faster than the model trained from scratch and also leads to better performance. Therefore, we adopt the strategy of pre-training in the rest of the experiments.

4.1.2. Learning Rates

For the reason that training ensemble models are time-consuming, all the base learners in

Table 1. Results for GeoBoost with different learning rates.

Learning Rate	Validation IoU
$1.0$	$0.6041$
$0.5$	$0.6183$
$0.2$	$0.6272$
$\mathbf{0.1}$	$\mathbf{0.6298}$
$0.05$	$0.6259$

are trained only 100 epochs to search for the optimum learning rate. From Table 1, it can be clearly seen that learning rates of base learners significantly affect the performance of the GeoBoost algorithm. With four base learners, the learning rate $\nu$ achieves the best result around $0.1$. This result is consistent with that in the gradient boosting algorithm [30] with much more base learners. Thus, the learning rate of $0.1$ is a stable value in practice and is the default configuration in the rest of the boosting experiments.

4.1.3. Complexity Analysis

The GeoBoost algorithm trains M base learners to form an ensemble model, thus its model complexity is $O\left(M\right)$. Therefore, it contains more parameters comparing with a single model. However, when more data are available, GeoBoost can be adopted without tuning hyper-parameters of the base models, and the architectures of these base models can also be different. Since each base learner is only assigned a portion of data, the training time of GeoBoost is not increased and keeps up with that of a single model. When testing, the ensemble model slows down the inference speed with time complexity of $O\left(M\right)$. Nevertheless, all the base learners are independent from each other, so the model is highly parallelizable to reduce the inference time.

4.2. Quantitative Evaluation

As shown in

Table 2. Results for the comparison of different methods. The result of a single model is obtained by training a net with all the available training data ${\mathbf{x}}_0$, ${\mathbf{x}}_1$, ${\mathbf{x}}_2$, and ${\mathbf{x}}_3$ from four areas. EGB is short for the end-to-end gradient boosting method.

Method	IoU	Precision	Recall	Overall Accuracy	Kappa
A single model (U-Net)	$0.6295$	$0.8049$	$0.7843$	$0.8877$	$0.7101$
A single model (U-NASNetMobile)	$0.6458$	$0.8283$	$0.8001$	$0.8985$	$0.7375$
EGB (U-Net)	$0.5465$	$0.7915$	$0.7443$	$0.8753$	$0.6736$
GeoBoost (U-Net)	0.6164	$0.8019$	$0.7646$	$0.8831$	$0.6953$
EGB (U-NASNetMobile)	$0.5887$	$0.8199$	$0.7276$	$0.8815$	$0.6784$
GeoBoost (U-NASNetMobile)	0.6372	$0.8112$	$0.7969$	$0.8905$	$0.7213$

, the result of a single net is obtained by training a U-NASNetMobile model with all the available training data ${\mathbf{x}}_0$, ${\mathbf{x}}_1$, ${\mathbf{x}}_2$, and ${\mathbf{x}}_3$ from four areas. It is the ideal case to train a model when all of the datasets are available. It might not be a scenario for the practical application of global mapping using VHR satellite images. The single net model has the highest accuracy of $0.6458$. This result is the baseline result for comparison.

The end-to-end gradient boosting algorithm for incremental learning obtains the worst performance of $0.5465$ (IoU) for U-net and $0.5887$ (IoU) for U-NASNetMobile in all the experiments. It can be seen from Figure 6a that the first base learner of gradient boosting $f_0$ already has an IoU accuracy of $0.5998$, and the successive results are only $0.5564$, $0.5747$, and $0.5887$. It means that the performance of the model is decreased by gradient boosting without considering the geospatial distribution of data.

Compared with a single net, GeoBoost obtains a comparable accuracy of $0.6372$, and it also outperforms the end-to-end gradient boosting algorithm by a large margin. Randomly sampling of training data for base learners can improve the performance of gradient boosting [46], whereas collecting satellite images from certain areas is actually equivalent to biased subsampling without repetition rather than uniformly sampling. That may be the reason that the end-to-end gradient boosting algorithm is failed on the DREAM-B data sets and GeoBoost achieves a satisfactory performance.

The results of experiments demonstrate the effectiveness of the GeoBoost algorithm by progressively utilizing the geospatial information of satellite images. Additionally, the result of a single net shows the importance of the large scale of data once more.

4.3. Qualitative Analysis

Figure 7 shows some semantic segmentation results of GeoBoost in different training stages from the cities of Chicago, Vienna, and Shanghai. Compared with the model $F_0\left(x\right)=\sigma \left(r_0{f}_0\left(x\right)\right)$, the model $F_3\left(x\right)=\sigma \left(r_0{f}_0\left(x\right)+r_1{f}_1\left(x\right)+r_2{f}_2\left(x\right)+r_3{f}_3\left(x\right)\right)$ produces better prediction results in terms of both accuracy and visual sense. According to the location of the image from Chicago, $r_2$ and $r_3$ is 0, and $r_0$ and $r_1$ is 1 for this image based on Equations (19) and (18). Therefore, the prediction of the image from Chicago can be simplified as $F_3\left(x\right)=\sigma \left(f_0\left(x\right)+f_1\left(x\right)\right)$. Similarly, predictions of images from Vienna and Shanghai can also be simplified in this way. Obviously, as pointed out by the yellow circles in Figure 7, some misclassified pixels of $F_0\left(x\right)$ are corrected by the model $F_3\left(x\right)$, and the others are not affected. Taken together, Figure 6 and Figure 7 suggest that the GeoBoost method is capable of continually learning, both quantitively and qualitatively.

Figure 8 presents some semantic segmentation results of different models from the cities of Chicago, Berlin, and Shanghai. Yellow circles indicate some notable discrepancies among different prediction results. Three models produce quite similar results. The prediction of GeoBoost is more in accordance with that of the single model. The end-to-end gradient boosting model misclassifies some tiny buildings. It should be noticed that these visualization results are merely for reference. The tiny differences among these models can also be triggered by the randomness of the training process and weights initialization. Quantitative evaluations in Section 4.2 are more reliable.

4.4. Discussion on the Regional Impacts

Some typical samples from the areas of $B_1$, $B_2$, and $B_3$ in Figure 3b are shown in Figure 9, Figure 10 and Figure 11, respectively. As shown in Figure 9, buildings in the area $B_1$ have unified distributions. The size of most buildings in this area is quite small, and the orientation of them is consistent within each image tile. Gaps among the individual buildings are obvious. Thus, images in this area are easy to be predicted correctly.

It can be seen from Figure 10 that buildings in the area $B_2$ are much different. Shapes of these buildings are not squares, and adjacent ones are connected to each other to form the big chunks.

Figure 11 shows that buildings in the area $B_3$ are more cluttered, and this area contains more high-rise buildings. Some pixels on the left-bottom corner of Figure 11i are mislabeled. This suggests that there may be some label noise in this area.

The accuracy of GeoBoost in Table 2 is measured by the overall performance of the model. For comparison, we can dive into more details of the trained model in different areas around the world. As shown in

Table 3. Results for the comparison of different areas on the test set. The coverage of different areas are the same as the bounding boxes of $B_1$, $B_2$, and $B_3$ in Figure 3.

Area	Testing IoU
America ($B_1$)	$0.6575$
Europe ($B_2$)	$0.7100$
East Asia ($B_3$)	$0.4758$

, the performance of GeoBoost fluctuates from one area to another. The model achieves the worst performance of $0.4758$ in East Asia, whereas the results of America and Europe are much better.

The divergence of the regional performance may be caused by two reasons. First, this may be caused by the accuracy measurement of IoU. It prefers big objects to the small ones. For simplicity, we assume that the buildings are squared with a side length of s pixels. Thus, the total number of pixels of a building is $s\times s=s^2$. Most misclassifications of the model occur near the border of buildings. Therefore, the total number of misclassified pixels is roughly $4\times s$. The proportion of the misclassified pixels can be inferred as $\frac{4s}{s^2}=\frac{4}{s}$. As the building size becomes bigger, the proportion of the misclassified pixels becomes much smaller, and the IoU becomes higher. The above inference is a rough estimation rather than a strict analysis. With all the available labels of the training set, we can take a glance at the distribution of building size in the DREAM-B data set. From Figure 12, it can be clearly seen that the building size of Europe is quite bigger than that of the other areas. Figure 10 presents the visualization of samples from three areas. Buildings in the city of Vienna are closely connected to each other and form the big building groups. In addition, big buildings have more context to infer the predicting results. This is why the prediction of Europe achieves the best result.

Second, the data of East Asia contain more high-rise buildings. The image in Figure 11f is a sample from the city of Shanghai. Obviously, the upper left corner of this sample has a considerable amount of high-rise buildings. This is very common in East Asia and does not appear very often in the other areas. Figure 13 better illustrates the affection of high-rise buildings. The labels of high-rise buildings are located on their roots. Though the model finds the positions of these high-rise buildings, there is no visual clue, such as edges and contrast, for precise segmentation. This is a tough task for CNNs to recognize 3D structures from a single image, whereas the segmentation of lower buildings is much easier and produces sharp edges of buildings as shown in Figure 7. For the reason that the image is not acquired with orthographic projection, the low accuracy of high-rise buildings is inevitable.

The above-mentioned two reasons may explain the divergence of the regional performance. Surprisingly, CNNs can capture 3D spatial information to some extent. These results qualitatively verify the necessity of utilizing the geospatial information of satellite images for semantic segmentation.

5. Conclusions

In this paper, we propose a novel approach, GeoBoost, for geographically incremental learning, which is trained in an end-to-end way. It enables the models of satellite images to learn continually based on geographical information of data without forgetting the previous knowledge. The effectiveness of the GeoBoost algorithm is verified on the large-scale data set of DREAM-B. The proposed method utilizing U-NASNetMobile as the base learner outperforms end-to-end gradient boosting by a large margin of 4.85% (IoU). Experiments with different base learners confirm that GeoBoost surpasses end-to-end gradient boosting consistently. At present, this algorithm is validated on the semantic segmentation task with the high-resolution satellite images. This method is quite flexible to be adapted to the lower resolution images. By adopting the appropriate base learner of GeoBoost, our proposed method may also be beneficial for the lower-resolution satellite images. The current GeoBoost algorithm focuses on the task of data-incremental learning. Based on its flexible framework, we will adapt it to the task of class-incremental learning in the future.