Isolation Forests to Evaluate Class Separability and the Representativeness of Training and Validation Areas in Land Cover Classification

Abstract

Supervised land cover classification from remote sensing imagery is based on gathering a set of training areas to characterise each of the classes and to train a predictive model that is then used to predict land cover in the rest of the image. This procedure relies mainly on the assumptions of statistical separability of the classes and the representativeness of the training areas. This paper uses isolation forests, a type of random tree ensembles, to analyse both assumptions and to easily correct lack of representativeness by digitising new training areas where needed to improve the classification of a Landsat-8 set of images with Random Forest. The results show that the improved set of training areas after the isolation forest analysis is more representative of the whole image and increases classification accuracy. Besides, the distribution of isolation values can be useful to estimate class separability. A class separability parameter that summarises such distributions is proposed. This parameter is more correlated to omission and commission errors than other separability measures such as the Jeffries–Matusita distance.

1. Introduction

Land cover classification is one of the most common remote sensing applications [1]. Classification can be both supervised, using ground truth training areas of each of the classes, or unsupervised, letting an algorithm automatically split the data into numerically meaningful clusters [2]. Maximum likelihood [3], support vector machines [4], random forest (RF) [5,6] and neural networks [7] are among the most widely used supervised classification algorithms [2,8], whereas ISODATA, k-means and fuzzy k-means are among the most used unsupervised classification algorithms [2,9].

In supervised classification, a set of land cover classes have to be decided beforehand. This set has to include all the classes of interest, but it also must be realistic, as classes must have reasonably different spectral signatures to be distinguishable from each other. Once this set of land cover classes are decided, training areas are needed to link class labels to spectral characteristics and to calibrate the classification algorithm to be used. The best option is to obtain the training areas through fieldwork; however, high resolution imagery (firstly airborne and more recently satellite-borne) is increasingly being used [10,11,12], especially since Landsat images are available free of charge and are used to obtain historical series of land use maps [13].

Validation is based on a different set of validation areas obtained in the same way as the training set. The previously calibrated model is used to predict land cover in the validation areas, and the comparison between reference and predicted classes is carried out with a confusion matrix. Several statistics can be obtained from such a matrix: overall accuracy, kappa index, omission errors and commission errors; as well as their confidence intervals [14,15]. Cross-validation [16] is a more accurate way of estimating classification errors, although it is more computationally demanding. This well known framework has two main problems [10,17].

Firstly, validation is only checking the coherence between training and validation areas (TVA). The validity of accuracy estimators as estimations of the real accuracy relies on three assumptions in the TVA: validation areas are randomly selected, training and validation areas are spatially independent and training areas are representative of the whole study area. The first assumption is easy to take care of and the second will hold as long as TVA are not too close to each other and their pixels are not split into training and validation sets [18]. However, the third assumption may be difficult to fulfil as classes might have different spectral properties in different places of the study area, especially in semiarid areas where the spectral signatures might be altered by the influence of the spatial fragmentation of agrarian landscapes due to patches of irrigated and rainfed plots and the effect of spectral properties of soil and lithology on reflectivity due to the low vegetation cover [19]. This variability may lead to some sectors being misrepresented by the TVA. As a consequence, a large number of training areas may be needed to cover the full variance of each class [10]. Some research was devoted to analysing the impact of different sampling schemes on accuracy [12,20] but to the best of our knowledge, no attempt was made to analyse how representative the TVA are. We suggest that trying to analyse this representativeness would be a more sensible approach than just trying to maximise the sample size without any knowledge on how representative the areas are.

A second issue is class separability. An attempt to perform a thematically very detailed land use classification will usually be impeded by the low separability of similar classes. Moreover, solving the aforementioned within-class variability issue may decrease the statistical separability of the classes as their internal variability may increase. A priori assessment of class separability is possible using Jeffries–Matusita (JM) distance [21] as long as the data for each class follows a multivariate normal distribution. Other approaches based on empirical distributions [22] can be computationally very expensive when used in remote sensing due to the high dimensionality of multispectral imagery, especially when using hyperspectral or multispectral-multitemporal data-sets.

Liu et al. (2008) proposed the use of isolation forests (IF), a particular case of random tree ensembles (RTE), as a way to estimate how isolated in the feature space different individuals in a sample are [23]. IF can be used to estimate how well represented each cell is by the TVA sample or by the subsample belonging to any particular class. In addition, class separability can be analysed by comparing the isolation scores of the TVA cells of classes A and B according to the IF model calibrated with one of them.

Thus, the aim of this paper is to analyse the use of an isolation score from IF to provide better understanding of TVA representativeness and class separability. Mapping isolation scores for the whole study area predicted by an IF model calibrated with the available TVA show how well different regions of the study area are represented by the TVA. Hence, new training areas can then be obtained in the most poorly represented regions, increasing the representativeness of the TVA sample. The effect of the new TVA on the accuracy statistics and on the final land use map are tested.

A new statistic to summarise isolation scores’ densities and to estimate class separability is proposed: the proportion of common isolation values (CIV). The capability of this statistic to predict omission and commission errors before the classification is also tested and compared with JM distance and with the average distance between randomly chosen pairs of pixels of both classes in the feature space.

2. Methodology

2.1. Study Area

The Segura Hydrographic Demarcation (DHS) is located in the southeast of the Iberian Peninsula (Figure 1) with an extension of 19,025 km² (considering only the continental surface) and 20,234 km² (including coastal waters). Height ranges from 0 in the coast to 2000 m in the northwestern area. The climate is typically Mediterranean and characterised by warm temperatures throughout the year and scarce and irregular rainfall. Average rainfall is 382 mm per year, with high spatial variability between sub-humid (precipitation greater than 500 mm/year), semi-arid and arid areas and temporally distributed between a maximum in autumn with a secondary maximum in spring and a minimum in summer. Episodes of torrential rain along with prolonged droughts are characteristic features.

The orographic and climatic characteristics coupled with the variety of soil types causes a high diversity in the vegetation of the region, dominated in natural areas by coniferous trees and Mediterranean scrubs like olive tree (Olea europaea), heather (Calluna vulgaris), carob tree (Ceratonia siliqua), mastic (Pistacia lentiscus), myrtle (Mirtus communis), rosemary (Rosmarinus officinalis) and rockrose (Cistus heterophyllus) developed as a consequence of the degradation of the Mediterranean forest. Crops are both irrigated and rainfed, although the latter is declining and the former increasing in area; groundwater and water from the Tagus-Segura transfer system are the main water sources.

2.2. Data Set

A total of 12 Landsat 8 scenes were used (

Table 1. Landsat 8 scenes used in the study.

Season	Path/Row	Date
spring	199/33	2/5/2017
spring	199/34	2/5/2017
spring	200/33	12/6/2017
spring	200/34	12/6/2017
summer	199/33	21/7/2017
summer	199/34	21/7/2017
summer	200/33	30/7/2017
summer	200/34	30/7/2017
autumn	199/33	10/11/2017
autumn	199/34	10/11/2017
autumn	200/33	19/11/2017
autumn	200/34	19/11/2017

). Winter images were excluded as cloud free data for those months were unavailable. However, Gomariz Castillo et al. (2017) concluded, in a multi-temporal study, that winter images are the least informative for this region [24]. The images were downloaded through the Semi-automatic Classification Plugin (SCP) of the GIS software QGIS [25], which also allows the preprocessing of images. The reflectivity of the Earth’s surface was obtained using the algorithm described in [26] and the effect of the atmosphere was corrected using the Chavez (1996) [27] method. Bands 2 to 7 and 10 were used as predictors (

Table 2. Landsat 8 bands used in the study.

Name	Wavelength	Resolution
Band 2 Blue	0.45–0.51	30
Band 3 Green	0.53–0.59	30
Band 4 Red	0.64–0.67	30
Band 5 Near Infrared (NIR)	0.85–0.88	30
Band 6 SWIR 1	1.57–1.65	30
Band 7 SWIR 2	2.11–2.29	30
Band 10 Thermal Infrared	10.6–11.19	100

) along with the following four normalised indices:

The normalized vegetation index (NDVI) is calculated using Equation (1) where ${\rho }_5$ is the reflectivity of band 5 and ${\rho }_4$ is the reflectivity of band 4. This index was correlated with the leaf chlorophyll content, net flow of CO₂, photo-synthetically active radiation, vegetation net productivity, leaf area index and the plant phenological dynamics, among other variables [28].

$$NDVI=\frac{{\rho }_5-{\rho }_4}{{\rho }_5+{\rho }_4}$$

(1)

The normalized index of buildings (NDBI) allows built-on surfaces to be detected [29]; it measures the proportion of buildings in each pixel. In semi-arid zones where soils have little vegetation, it is difficult for this index to discern bare soils and built areas, since the reflectivity of both classes can be similar [30]. It is generated with Equation (2), where ${\rho }_5$ is the reflectivity of band 5 and ${\rho }_6$ is the reflectivity of band 6.

$$NDBI=\frac{{\rho }_6-{\rho }_5}{{\rho }_6+{\rho }_5}$$

(2)

The normalized soil bareness index (NDBaI) [31] identifies areas without vegetation. It is generated with Equation (3), where $N{D}_5$ is the digital level of band 5 and $N{D}_{10}$ is the digital level of band 10. Since band 10 is the thermal band, digital level (ND) values are used instead of reflectivities.

$$NDBaI=\frac{N{D}_5-N{D}_{10}}{N{D}_5+N{D}_{10}}$$

(3)

The modified normalized water index (MNDWI) measures the amount of water present in the pixel. This method was proposed by Xu (2006) [32] and uses Equation (4), where ${\rho }_3$ is the reflectivity of band 3 and ${\rho }_6$ is the reflectivity of band 6.

$$MNDWI=\frac{{\rho }_3-{\rho }_6}{{\rho }_3+{\rho }_6}$$

(4)

2.3. Training and Validation Areas

TVA were digitised from the 2016 high resolution images from the National Aerial OrthoPhotography Plan (PNOA). Ten land covers present in the study area were differentiated: 1. Forest: includes natural arboreal vegetation; 2. Shrubs: corresponds to typical Mediterranean shrub areas and is easily localised in sunny hillslopes; 3. Low density tree crops: including mainly almond trees and olive trees, 4. Dense tree crops: including citrus and stone fruit trees; 5. Rainfed grass crops: soil dedicated mainly to the cultivation of cereals and some legumes; 6. Irrigated crops: includes areas dedicated to horticultural crops and rice fields; 7. Impermeable surfaces: soil with strong anthropic transformation causing maximum runoff and preventing water infiltration. 8. Water surfaces: areas of the soil that are covered with water. 9. Bare soil: Surfaces with very sparse vegetation cover, both natural and artificial (e.g., quarries). 10. Greenhouses: built-up greenhouses used for growing crops or land patches covered with plastic. Figure 2 shows the distribution of TV areas in the study area. The difference in density in different locations reflects differences in variability and in interest in the areas.

2.4. Random Forest, Random Tree Ensembles and Isolation Forests

Ensemble learning [33] consists of training several classifiers and averaging the results. Decision trees are well known for being a low bias, high variance algorithm when used for classification [33,34]. An ensemble of decision trees will integrate the results of those low bias, high variance trees and produce a low bias, low variance estimation. The decision trees in the ensemble should not be highly correlated in order to prevent all trees from giving the same prediction. Different strategies were proposed to decrease correlation in order to increase their variance and thus reduce the bias of the averaged result, as in bagging, one of the simplest tree ensembles, where each tree is trained with a bootstrapped subset of the training cases [33,34].

In random forest [5], besides randomising the cases in each of the trained trees, the features among which the one producing the most homogeneous split is chosen are also randomised in each node. The number of features randomly selected in each split is a parameter set by the user whose default value for classification is $\sqrt{p}$ rounded to the lowest integer, where p is the number of features. This counterintuitive approach reduces among-tree correlation, and increases the generalisation capacity of the model, because the higher the variability of the trees the lower the bias of the ensemble. The default number of trees in RF is 500 [35]. It was shown [35] that the default values for both parameters perform well in most of the cases so they are not usually optimised.

Random tree ensembles (RTE) [36,37,38] is a different kind of tree ensemble, which takes the decorrelation approach to the extreme. All trees are trained with all cases but both the feature to split each node and the corresponding threshold are randomly selected. The growth of each tree is limited. Fan et al. (2003) advise using a maximum depth equal to half the number of predictors [36]. This author also recommends a very low number of trees (around 30). Accuracy results similar to other classification methods were reported [36,37,38] due to the success of the increase in tree variability when generalising the results; however, RTE were scarcely used beyond the initial proposals.

Liu et al. (2008) [23] proposed a modification of RTE as a way to estimate case isolation in the feature space. Each random tree is fully grown until every final node contains an isolated case, or several cases with exactly the same values for all the features. The number of splits that a particular individual needs to reach a final node is a measure of its isolation in the feature space. The larger the number of splits, the lower the isolation. Individuals within large clusters will need a larger amount of splits to reach a leaf node, whereas isolated individuals will reach a leaf node with just a few splits. An IF is an ensemble of such random trees, in which the average number of splits is the final isolation score. Once an IF model was calibrated using training cases, any new individual can be let through the model to obtain its isolation with respect to the sample of TVA used to calibrate the model. According to Liu et al. (2008) [23], individuals with a low isolation score are anomalies in the sample. IF can then be considered an anomaly or novelty detection procedure [39]; in particular, it can be considered a semisupervised anomaly detection method, in which only normal cases are required to train the model, whose output is an isolation/anomaly score.

2.5. Isolation Forests and TVA Representativeness

In remote sensing imagery classification, IF can be used to determine if any new pixel is well represented by the available TVA. We propose to train an IF with all the pixels in the TVA and to apply it to evaluate the isolation score of all the pixels in the image. In this way we discover which pixels in the image are not properly represented by the training areas and, consequently, the model may not have enough information to classify them. The results can be mapped in the geographical space to show areas not properly represented by the TVA. It is then straightforward to digitise new TVA in those areas and to repeat the procedure to check whether the new sample is representative enough. This procedure can be repeated until no clearly missrepresented areas appear in the map.

Figure 3 shows a simple example with 3 classes and just 2 predictors. The plot in (a) shows the feature space in which classes B and C partially overlap and class A is more isolated (distinguishable) from the others. The plot in (b) maps the isolation score from an IF trained with samples from the 3 classes. The sectors of the feature space less represented by the sample have lower values, and the sectors corresponding with the sample values have higher values.

As a consequence, variability of the different classes will increase, producing multimodality in some classes; however, this is not a problem when using a tree based model to classify, because such models can create several separate areas in the feature space without affecting other classes and without the need for any modification in the configuration of the model. Conversely, a neural network would need more nodes in the layers or even more layers to deal with increasing heterogeneity in class definitions; maximum likelihood would need the creation of new classes to deal with multimodality; support vector machine would need an increasingly complex boundary between classes in the feature space, requiring complex kernels, larger penalty coefficients and a more difficult optimisation process.

2.6. Isolation Forests and Class Separability

IF can also be used to analyse class separability. If an IF model is trained with data from a class A, the isolation of each cell in the data set with respect to that IF can be used to estimate how well different cells of class A and of the other classes are represented by class A cells. Logically, the isolation value of the TVA cells of the same class A will be high (meaning high representativeness) and the isolation values of cells from other classes will vary depending of the separability of the classes. Interestingly, the relation is not symmetrical, if all cells in a class A, for example, are very well represented by cells of a class B, it is not necessarily the case that all cells in the class B will be well represented by cells of a class A. This could happen, for instance, if classes A and B partially intersect in the feature space but class B has a larger variance than class A. Hence, IF can be used not only as an estimation of class separability but also as an estimation of the amount of omission and commission errors of any algorithm; however, this estimation will be more accurate with any tree based classification algorithm because it will deal with multimodality in the same way.

Figure 3 (plot c) shows the distribution of classes A, B and C isolation scores calculated from the IF trained only with class A. The distributions of classes B and C do not overlap with the distribution of A, indicating that class A is perfectly separable from the others. Plot (d) and (e) show the same information but now with the IF calibrated with class B and C data respectively. In both cases there is a clear overlap of the value distributions of classes B and C, whereas class A distribution does not overlap. That means that classes B and C are not as separable from each other as they are from class A.

A parameter summarising the difference between the isolation scores of a class A and those of the reference class B with which an IF was trained would be useful, although not easy to define as isolation variability is quite high even when a class is tested with its own IF. In order to quantify this difference, we propose the proportion of common isolation values (CIV) as such parameter. It represents the area under both density curves over the sum of the areas under the density curves (Figure 4). CIV can be calculated numerically, splitting the isolation values into several bins, with Equation (5).

$$CI{V}_{A,B}=\frac{{\sum }_i^{N}min(density\left(A_i\right),density\left(B_i\right))}{{\sum }_i^{N}max(density\left(A_i\right),density\left(B_i\right))}$$

(5)

where N is the number of bins in which the range of isolation scores is discretised (in this case we used bins of width = 0.25), $density\left(A_i\right)$ and $density\left(B_i\right)$ are the frequency of isolation values in bin i for classes A and B, respectively. The numerator is the area under both density curves (c in Figure 4), and the denominator gives the sum of areas A, B and C (Figure 4).

A noteworthy property of this measure is that it is not symmetrical; that is $CI{V}_{A,B}\ne CI{V}_{B,A}$, as can be seen both from Figures 7 and 8 and in Table 4. When computing the isolation of classes A and B in the class B IF, CIV is related with the risk of omission error in class A and commission error in class B. However, it is related with the risk of omission error in class B and commission error in class A, when isolation of classes A and B are computed in the class A IF.

Jeffries–Matusita distance [3] was proposed as a measure of class separability; however, it assumes that features in the compared classes follow a multinormal distribution. The equation is:

$$J_{ij}=2(1-e^{-d_{ij}}),$$

(6)

where $d_{ij}$ is the Bhattacharyya distance between classes i and j. If multinormality is assumed, it can be written as:

$$d_{ij}=\frac{1}{8}{(m_j-m_i)}^T{\left[\frac{{\Sigma }_i+{\Sigma }_j}{2}\right]}^{-1}(m_j-m_i)+\frac{1}{2}ln\frac{\frac{|{\Sigma }_i+{\Sigma }_j|}{2}}{\sqrt{|{\Sigma }_i\left|\right|{\Sigma }_j|}}$$

(7)

A different approach to estimate the separability between two classes would be to compute the average distance between pairs of cases extracted from each class. As computing all the possible pairs of cells would be quite time consuming, a sample was extracted using the algorithm described in Figure 5:

A plot of these three separability measures against omission and commission errors will show how good these measures are to predict errors. A generalised additive model (GAM) [40] using the gamma distribution allowed us to analyse these relations and to select the most effective measure to predict such errors.

2.7. Procedure

We used the steps given below in order to check how IF can help to assess the representativeness of the TVA. They are also represented in the flow chart in Figure 6.

• An RF model was used to classify the study area using the R package randomForest [35]; 70% of the TVA were used for training and 30% for validation. Complete TVA were selected for each set, thus preventing the pixels from being split into training and validation sets.
• A global IF was obtained for the whole sample of TVA. R package h2o [41] allows IFs to be calibrated and used.
• The global IF was then fed with data from each cell in the study area to obtain a map that reflects how each cell is represented by the TVA (Figure 11).
• Sectors of the study area specially misrepresented by the TVA were identified and then additional TVA were digitised in such sites and added to the TVA set.
• With this enlarged TVA set, a new global IF was obtained and a new representativeness map produced (Figure 12).
• To test the benefits in accuracy, we validated the original RF model with the new set of TVA and cross-validated another model including all TVA. leave-one-area-out cross-validation [18] was the approach used for validation. The corresponding confusion matrices (Table 5) and the statistics extracted from them helped quantify the improvement in accuracy.

To check how IF can help to assess separability, the following steps were carried out:

• Class-specific IFs were calculated for each one of the 10 classes. (Step 7 in Figure 6).
• The cells of every single class were fed into each of the class-specific IFs to obtain the isolation of each class in the IF model from every other class. Results appear as density plots (Figure 7 and Figure 8). (Step 8 in Figure 6).
• CIV was calculated for each pair of classes. Table 4 shows the non-symmetric matrix of CIV values. (Step 9 in Figure 6).
• Jeffries–Matusita distance was also calculated using the R package selVar [42]. (Step 10 in Figure 6).
• The average distance between cells of two classes was calculated using the algorithm presented in Section 2.6. (Step 10 in Figure 6).
• The percentage of wrongly classified cells outside each class (omission errors) and inside each class (commission errors) were calculated from the confusion matrix of the initial RF model. (Step 11 in Figure 6).
• A plot of omission and commission errors vs. JM distance, vs. the corresponding CIV values and vs. the mean distance among classes shows which magnitude is more related to classification errors and, consequently, is more useful as a class separability estimation. To analyse the relation among these variables a generalized additive model (GAM) was fitted using the GMCV R package [40]. (Step 11 in Figure 6).

3. Results

3.1. Initial RF Model

The initial TVA set contained 1060 areas including 75,063 cells.

Table 3. Confusion matrix of the initial random forest (RF) model.

Prediction	Reference										Com. Error
	Forest	Scrub	Low Dens. Tree Crops	Dense Tree Crops	Rainfed Grass Crops	Irrigated Crops	Impermeable	Water	Bare Soil	Greenhouses
Forest	2765	173	2	0	56	0	38	0	0	0	0.089
Scrub	69	3582	39	115	85	12	23	0	7	0	0.089
Low density tree crops	2	88	1013	19	41	20	28	0	38	0	0.189
Dense tree crops	0	0	42	776	0	64	34	2	1	0	0.156
Rainfed grass crops	0	58	37	2	1185	41	14	0	124	0	0.189
Irrigated crops	20	26	12	88	2	993	62	1	0	0	0.175
Impermeable	14	21	328	13	70	177	3319	7	17	73	0.178
Water	0	1	0	8	0	11	0	1670	0	0	0.012
Bare soil	0	55	21	3	43	14	94	0	164	0	0.584
Greenhouses	0	0	0	0	0	0	2	0	0	181	0.011
Om. error	0.036	0.105	0.322	0.242	0.2	0.254	0.081	0.006	0.533	0.287

shows the results of the first RF model with the initial set of TVA. The overall accuracy is 0.864, and the kappa index is 0.841. Table 3 also shows commission and omission errors of every class. The highest errors appear when classifying bare soil and low density tree crops.

3.2. Class Separability

Figure 7 and Figure 8 show the density of isolation scores of each class in the IF model built with every other class. They also show the isolation scores of a class in its own IF model. The top left plot of Figure 7 shows the density of all classes in the IF built using only forest cells. It is clear that forest cell has a high score, as expected, and the score of the other classes is lower, while only the distribution of scrubs’ isolation scores shows a slight overlapping. It can be assumed that forest class is separable from other classes and that commission errors are expected to be low. The same can be said of the classes scrubs, water and greenhouses.

Greater problems appear with other classes. Irrigated grass crops and low density tree crops are not properly isolated from dense tree crops. Low density tree crops are also not separable from rainfed grass crops. Rainfed grass crops and impermeable have the same problem in relation with bare soil. Finally, most of the classes seem difficult to distinguish from irrigated grass crops.

The most serious problem appears when most of the training cells in a class have isolation scores in another class IF model that are almost as large as those of the reference class. This happens with low density tree crops and rainfed grass crops, dense tree crops and irrigated grass crops, rainfed grass crops and bare soil and impermeable and bare soil. In this case the classification model has a larger risk of producing commission errors with the reference class and omission errors with the compared class.

An interesting case arises when density plots of classes A and B in the class B IF model are very similar but the density plot of class B in the class A IF model is different from that of class A. This means that class A occupies a subset of class B spot in the variable space. It is then very likely that the cells of class A are confused with those of class B, but it is not so likely that the opposite occurs. A good example is that of the classes impermeable and bare soils. The isolation scores of the class impermeable when compared with the IF of bare soil fits almost perfectly the isolation scores of bare soil, creating the potential for substantial confusion. However, isolation scores of the class bare soil are smaller than those of the class impermeable.

The matrix of CIV parameters is shown in

Table 4. Common isolation values (CIV). Rows represent the different IF models and columns represent the classes whose isolation is tested with the IF models.

Prediction   Test Class	Forest	Scrub	Low Dens. Tree Crops	Dense Tree Crops	Rainfed Grass Crops	Irrigated Crops	Impermeable	Water	Bare Soil	Greenhouses
(1) Forest	1.0000	0.1725	0.0770	0.0963	0.0667	0.0871	0.0703	0.0093	0.0713	0.0110
(2) Scrub	0.1066	1.0000	0.1792	0.1124	0.1599	0.1809	0.1398	0.0076	0.1423	0.0060
(3) Low density tree crops	0.0587	0.2130	1.0000	0.2185	0.3004	0.2785	0.2578	0.0069	0.2151	0.0407
(4) Dense tree crops	0.0863	0.1200	0.2593	1.0000	0.1751	0.3505	0.1416	0.0081	0.1129	0.0237
(5) Rainfed grass crops	0.0625	0.2429	0.5885	0.1896	1.0000	0.2198	0.1619	0.0069	0.2129	0.0231
(6) Irrigated crops	0.1255	0.4129	0.3465	0.7013	0.2483	1.0000	0.3365	0.0123	0.2120	0.0398
(7) Impermeable	0.0547	0.1947	0.2748	0.1423	0.1332	0.2622	1.0000	0.0085	0.1905	0.0620
(8) Water	0.0107	0.0080	0.0083	0.0093	0.0056	0.0163	0.0115	1.0000	0.0138	0.0063
(9) Bare soil	0.0511	0.2367	0.6938	0.1465	0.4807	0.3198	0.3148	0.0063	1.0000	0.0417
(10) Greenhouses	0.0013	0.0011	0.0101	0.0066	0.0048	0.0036	0.0380	0.0018	0.0136	1.0000

. CIV values of the class water are smaller than 0.02 in all cases. Conversely, several classes show very high isolation scores in the IF model of bare soil (Figure 8). This is reflected in the high CIV values in the row corresponding to bare soil in Table 4. When isolation is calculated with respect to greenhouses, all classes show very low scores, indicating a low risk of confusion; thus, all CIV values in the greenhouse row in Table 4 have values smaller than 0.04; however, the column shows a CIV value of 0.062 when impermeable class is compared with the IF model of greenhouses. This value is reflected in the 73 greenhouse cells classified as impermeable (Table 3).

Figure 9 shows the plots of the three class separability measures that were tested against omission and commission errors. A GAM model is also fitted and the RMSE values computed with cross-validation are added to the plots. Commission errors seem, in general, easier to predict than omission errors (lower RMSE values in all cases). JM distance seems the worst option to predict errors: it not only has the highest RMSE value but also the models do not predict a clear relation among JM distance and omission and commission errors. Probably, the reason is that the multinormality assumption is not met. On the other hand, CIV and mean distance among cases have a similar accuracy when predicting errors, the RMSEs of the later being slightly higher. However, it is probably a better idea to use a threshold both for CIV and mean distance. When CIV is smaller than 0.1 and when mean distance values are larger than 0.9, the confusion proportions are smaller than 0.05. Beyond those thresholds the proportion of misclassified cells is more variable.

3.3. Representativeness of the Training Sample

Figure 10 shows the distribution of isolation score for the whole study area calculated with the initial TVA set (red) and with the final TVA set (black). There is a clear reduction in the frequency of values between 5.5 and 6.75 and a clear increase in values between 6.75 and 7.

Figure 11 shows an isolation score map according to the IF built using all TVA. Some areas are characterised by quite low isolation scores, meaning a high isolation in feature space. They are mostly inland water bodies, bare soil areas with very high reflectivity, some small urban areas and the northwestern sector that corresponds to an image of a different satellite path. Although a visual inspection and a rapid statistical analysis did not find this difference among the two images, the differences appeared in this analysis because the area was clearly undersampled. This problematic northwestern sector corresponds mainly to forest areas.

A new set of TVA was digitised from these low isolation score sectors and they were added to the previous one so the new TVA set contained 1077 areas including 97,835 cells. This meant only 17 new areas but a large number of new cells because a very large area was digitised in a lagoon in the eastern part of the study area (Figure 12). We repeated the analysis and obtained the map in Figure 12. The increase in the isolation score in several areas is clear; the map also shows the location of the new TVA, demonstrating that with just 17 new TVA a high increase in representativeness was achieved. This is especially evident in the northwestern forest area, in the west and NE sectors where there are several bare soil areas and in several water bodies that have a very low isolation score in Figure 11 and somewhat larger values in Figure 12.

Figure 13 shows detailed views of three areas with a predominance of the three mostly underrepresented classes in the initial TVA set. The isolation scores calculated with the initial TVA set, the isolation scores calculated with the final TVA set and the high resolution ortophotograph used to digitise the TVA are shown in the Figure. The borders of these areas are included in Figure 11. The colour scale is the same as in Figure 11 and Figure 12. A forested area corresponding to the image of a different satellite path appears in the top images. In this case it is clear how the entire area is best represented by the final TVA. In the middle, there is a mix of crops and bare soil areas. The whole image, especially the bare soil areas, is also best represented by the final TVA. At the bottom, a reservoir and a small pond show the increase in representativeness in the water surfaces. Among the guidelines for digitising the first TVA set is using only clear waters and the consequence was to increase the isolation in the feature space of several inner water surfaces.

Table 5. Confusion matrix calculated with the new set of TVA of the RF model calibrated with the first set of TVA (left) and confusion matrix calculated with cross validation using the new set of TVA of the RF model calibrated with the first set of TVA (right).

Prediction	RF Calibrated with the Initial TVA				RF Calibrated with the New TVA
	Forest	Impermeable	Water	Bare Soil	Forest	Impermeable	Water	Bare Soil
1. Forest	2263	0	0	0	2266	0	0	0
2. Scrub	3	0	0	1	2	0	0	0
3. Low density tree crops	0	0	0	1450	0	6	0	284
4. Dense tree crops	0	0	0	18	0	0	0	11
5. Rainfed grass crops	0	1	0	3604	0	0	0	577
6. Irrigated crops	0	0	0	1	0	0	0	0
7. Impermeable	0	24	1	125	0	19	0	1195
8. Water	6	0	12,207	2	4	0	12,209	2
9. Bare soil	0	204	0	3455	0	212	0	6585
10. Greenhouses	0	0	5	98	0	0	4	100

shows the confusion matrices obtained with the new TVA. The confusion matrix on the left corresponds to the first RF model validated with the new TVA, and the confusion matrix on the right corresponds to the RF model calibrated with the augmented set of TVA and validated, using cross-validation, with the new TVA. In these confusion matrices, only the columns corresponding to classes in the new TVA appear. The augmented TVA set allows an increase in accuracy from an overall accuracy of 0.765 to 0.9, with kappa values of 0.644 and 0.833, respectively. In this case, the classes that were worst represented by the initial sample were only four: forest, impermeable, water and bare soil. Although the four classes increase in accuracy, only bare soil shows a significant increase. Omission error in bare soils decreases from 0.605 to 0.248. There is also a considerable decrease in bare soil cells classified as low density tree crops (16.56% to 3.21%) and rainfed grass crops (41.17% to 6.6%). However, there is an increase in bare soil classified as impermeable (1.43% to 13.65%).

4. Discussion

Plots of isolation score density reflect how likely it is that a cell belonging to the test class is misclassified as belonging to the class with whose cells the corresponding IF model was trained. The higher the parameter the more likely this is. So it could partly increase omission and commission errors of the test classes. When computing the isolation of classes A and B in the class B IF, CIV is related with the risk of omission error in class A and commission error in class B, whereas the values obtained when computing the isolation of classes A and B in the class A IF, are related with the risk of omission error in class B and commission error in class A. For instance, the IF model of the class water has a large difference with the rest of the classes, so commission errors of class water are very low. In addition, water class isolation scores with the IF models of the rest of the classes are also very low, so the occurrence of omission errors is not likely either.

CIV is a good class separability indicator. It is more related with errors of omission and commission than JM distance, probably because CIV does not depend on class multivariable normality as JF distance does. On the other hand, CIV performance predicting omission and commission errors is slightly worse than mean distance among cells. However, it demands slightly less computing time. It would be interesting to compare CIV with other methods to assess class separability, and we intend to follow this line of research in the future.

We used IF to increase representativeness of the whole TVA set, but it could also be used just for the training areas or just for the validation areas. It could be used to complete the data set or just to have a clear idea of what sectors of the study area, if any, are poorly represented by the training set, the validation set or both. When used to improve the representativeness of the set, attention may be necessary to keep the same percentage of classes as in the study area by including other new areas.

The flow chart in Figure 6 shows the methodology followed in this paper but the logical flow when applying IFs to increase representativeness would be to go from four (new set of TVA) back in order to complete the initial TVA set.

In the case of prediction, random forest can provide, for every pixel, the amount of votes cast for each class. The proportion of votes for each class is not exactly a probability but can serve as an uncertainty measure using the random forest margin, that is the difference in votes percentage between the first and the second most voted classes [5,34]. It would be tempting to use uncertainty evaluated by a model, e.g., probability in maximum likelihood or margin with random forest, as a proxy of how well a cell is represented by the TVA sample; however, such uncertainty measures reflect both lack of representativeness in the sample and lack of separability among classes. A cell between two classes that are very close to each other would have high uncertainty but also a high isolation score. Furthermore, a cell at the border of the feature space might not be well represented, but, at the same time, may have low uncertainty because it is closer to a class than to any others. This is what seems to happen with forest and water cells, which are not properly represented by the initial TVA set but mostly well classified by the first RF model. We calculated the correlation coefficient between isolation and random forest margin metric to be just –0.034.

5. Conclusions

Isolation forest may be considered a good tool for estimating how well the TVA sample represents all the cells of a remote sensing image. Using the prediction of an IF model trained with the set of available TVA indicates sectors of the study area not properly represented by the TVA. Including some of these highlighted sectors as new training areas produces a more complete training areas set. The model calibrated with this augmented set is more generalise than the original. When using IF to estimate class separability, it is possible to obtain interesting graphic results and also a measure, common isolation values (CIV), which is more related with the omission and commission errors than the well known Jeffries–Matusita distance, although slightly worse than the average of distances between cases. However, CIV calculation is less computer demanding. CIV values lower than 0.1 and mean distance values higher than 0.9 results in confusion proportions below 0.05. Beyond those thresholds the proportion of misclassified cells is more variable.