Analyzing the Losses and Gains of a Land Category: Insights from the Total Operating Characteristic

Abstract

This manuscript provides guidance concerning how to use the Total Operating Characteristic (TOC) when (1) analyzing change through time, (2) ranking a categorical independent variable, and (3) constraining the extent for a gaining category. The illustrative variable is the marsh land-cover category in the Plum Island Ecosystems of northeastern Massachusetts, USA. The data are an elevation map and maps showing the land categories of water, marsh, and upland in 1938, 1971, and 2013. There were losses and gains near the edge of the marsh between 1938 and 1972 and between 1972 and 2013. The TOC curves show that marsh gained most intensively at intermediate elevations during the first time interval and then had a weaker association with elevation during the second time interval. Marsh gains more intensively from water than from upland during both time intervals. The TOC curves also demonstrate that the marsh gains occurred where marsh was previously lost, a phenomenon called Alternation. Furthermore, eliminating far distances and extreme elevations from the spatial extent decreased the area under the curve (AUC) for distance and increased the AUC for elevation. We invite scientists to use the TOC because the TOC is easier to interpret and shows more information than the Relative Operative Characteristic.

1. Introduction

The Total Operating Characteristic (TOC) can help researchers understand land change because the TOC analyzes the relationships between a ranked independent variable, such as distance, and a binary variable, such as the presence or absence of a land category. The TOC has gained increasing attention in the field of spatiotemporal analysis. This is because the TOC shows the total information in an error or change matrix, which other popular methods like the Receiver Operating Characteristic (ROC) fail to show. Pontius Jr and Si [1] first proposed the TOC as a modification of the ROC, which has been popular in diverse fields such as genetics [2,3], radiology [4], psychology [5,6], machine learning [7,8], and remote sensing [9,10].

Pontius and Si [1] described the use of the TOC to analyze the change in a land-cover category during a time interval. Subsequently, Bilintoh et al. [11] described how to use the TOC to analyze the losses and gains of land-cover categories during two intervals. Analyzing land change during more than one time interval provides an opportunity to compare the patterns of losses and gains during consecutive time intervals, which can reveal a change pattern called Alternation [12]. Alternation derives from pairing losses and gains through time at the same location.

The TOC compares a binary variable to a ranked index variable. A threshold of the ranked variable determines the diagnosis of an observation’s presence or absence based on whether an observation’s index value exceeds a threshold [1]. For our application, we diagnosed the presence or absence of gains and losses of marsh based on multiple indexes. The TOC can analyze ranked continuous variables such as elevation and distance. Furthermore, the TOC can analyze categorical variables [13]. For instance, scientists might want to analyze how a category such as urban gains from other categories such as forest or agriculture. Using the TOC to analyze the relationship between land-cover categories can reveal information about the categories that a gaining category targets or avoids.

Figure 1 shows the matrix for threshold t. The reference and the diagnosis can agree in two ways: Hits (H_t) and Correct Rejections (C_t) at threshold t. Similarly, the reference and the diagnosis can disagree in two ways: Misses (M_t) and False Alarms (F_t) at threshold t. Figure 1 shows the two types of agreements and disagreements and the total number of observations, which is P + Q (Pontius Jr and Si 2014 [1]). Scientists require four bits of information to complete the matrix for threshold t. These pieces of information could be H_t; M_t; F_t; and C_t. Alternatively, the bits could be P + Q; P; H_t + F_t; and H_t. Other combinations of the four bits of information are possible.

Table 1. Mathematical notation (modified after Pontius Jr and Si 2014 [1]).

Symbol	Meaning
T	Index for a threshold
Ht	Hits, which is the number of observations that are reference presence and diagnosed presence at threshold t
Mt	Misses, which is the number of observations that are reference presence and diagnosed absence at threshold t
Ft	False Alarms, which is the number of observations that are reference absence and diagnosed presence at threshold t
Ct	Correct Rejections, which is the number of observations that are reference absence and diagnosed absence at threshold t
P	Number of observations that are reference presence, also known as Abundance
Q	Number of observations that are reference absence

defines the mathematical notation in Figure 1.

The area under the curve (AUC) is a popular metric among scientists using the ROC and the TOC. Some scientists consider particular AUC values to designate the results as good, which is problematic because an arbitrary spatial extent of absence influences the AUC. For example, Naghibi et al. [14] used the TOC to evaluate the accuracy of an urban gain model in a region of Iran. However, we suspect that they failed to mask urban areas at the initial time point, given the shape of the curve. Failing to mask pixels that are not candidates for change can result in inflated AUC values, which may lead to a flawed interpretation of the TOC curves. Similarly, Chakraborti et al. [15] used the TOC to analyze LULCC in the Siliguri region of India, but we suspect they also failed to mask urban areas at the first time point. Another situation exists where scientists include vast regions that have zero probability of change, which inflates the AUC. This causes some scientists to question the usefulness of the AUC, because the AUC increases when the scientists arbitrarily include places where change is not plausible. The arbitrariness of the spatial extent causes confusion when comparing the AUC values across case studies [16].

The TOC’s ability to provide detailed information about the relationship between a ranked variable and a reference binary variable makes the TOC a valuable method for evaluating land changes and the accuracy of models [13,17]. However, authors must apply the TOC appropriately. This manuscript illustrates universal mathematical concepts by using the TOC to analyze how the losses and gains of land-cover categories during two time intervals relate to three types of variables: distance; elevation; and two land-cover categories. In addition, this manuscript addresses two crucial concepts in TOC analysis: (1) ranking a categorical independent variable and (2) constraining the extent of a gaining category.

2. Materials and Methods

2.1. Study Region

Figure 2 shows the Plum Island Ecosystem (PIE) Long-Term Ecological Research site in northeastern Massachusetts. The PIE’s marshes provide several ecosystem services, including storm protection, biodiversity habitats, nutrient cycling, and carbon storage. Rising sea levels threaten these ecosystem services. For example, sea-level rise could cause the cordgrass Spartina alterniflora to become flooded, thus causing Spartina alterniflora to shift to higher elevations. Measuring and visualizing these changes is crucial for understanding the relationship between changes and the ecosystem function.

2.2. Data

Figure 3 and Figure 4 show the data, which are land-cover maps in 1938, 1972, and 2013 and an elevation map. Each land-cover map has a spatial resolution of 10 m by 10 m, where each pixel shows one of three land-cover categories: water; marsh; and upland. The land-cover maps were derived from the Georgia Coastal Ecosystems website [18]. The distance to the edge of the marsh in the 1938 map in Figure 3 ranges between 0 and 865 m. A distance of 0 m indicates marsh, while a distance of 865 m indicates the furthest distance away from the edge of the marsh.

2.3. Methods

We used the TOC Curve Generator version 1.2.7 [19] to generate the TOC curves. The software requires the following inputs to generate a TOC curve: a binary variable; an index variable; and an optional mask. The binary variable contains information about a category’s presence or absence, where 1 shows the presence and 0 shows the absence of the category. A mask restricts the spatial extent of the analysis. Figure 5 shows the binary variables in the form of maps of marsh change, including gains, losses, and persistence during each time interval. We then segmented each time interval’s maps into four binary maps: gain and loss during the first time interval and gain and loss during the second time interval. Thus, each pixel uses 1 to denote gain or loss while using 0 to denote other. The second step is to create index variables. We used maps of the distance from the edges of the marsh at the start of each time interval, a map of elevation, and a map showing the two non-marsh land categories at the beginning of each time interval.

Pontius [13] described the procedure for a TOC analysis involving a categorical index variable. We adopted this procedure to create the TOC curves that show the relationship between marsh gain and the other two categories during each time interval. We excluded marsh from the categorical TOC curve because an observation must be non-marsh at the initial time point to experience marsh gain during the time interval. We had two land categories to rank after eliminating marsh. The land categories were water and upland. We computed each category’s intensity of marsh gain to rank the water and upland. The greater intensities received earlier ranks. An intensity is a ratio where the numerator is the size of the marsh’s gain from the losing category while the denominator is the size of the losing category [13].

The next step was to create masks to eliminate locations that were not plausible candidates for the gain of marsh. Our approach examines the thresholds for the index variable where the TOC curve touches the upper or lower bounds of the TOC parallelogram. Thresholds to the right of the point that touches the upper bound must be on the upper bound, so we considered those locations to be not plausible. Thresholds to the left of the point that leaves the lower bound must be on the lower bound, so we also considered those locations to be not plausible. This step is relevant for the TOC curves of a gaining land-cover class. We therefore conducted this step for gains of marsh during the first and second time intervals where the index variables were elevation and distance to the edge of the marsh.

Figure 6a shows the TOC curve for distance to the edge of the marsh and marsh gain during the second time interval, without applying a mask. We use the red rectangles to zoom in on segments of the distances that did not experience marsh change. The zoomed upper-right corner shows that the curve touches the upper bound at a distance of 310 m and continues to hug the upper bound until the farthest distance. Any distance beyond 310 m did not experience change, and so we excluded those large distances from the analysis. Figure 6c,d show the analysis for the masking of elevation. Figure 6c shows the entire TOC before the masking of elevation, while Figure 6d shows the zoomed lower-left and upper-right corners of Figure 6c.

The zoomed version shows that the curve starts from the origin and hugs the horizontal axis until after an elevation of −1.9. Any elevation below −1.9 m did not experience change; thus, we excluded elevations below −1.9 m from the analysis. Similarly, the zoomed upper-right corner shows that the curve touches the upper bound at an elevation of 7 m and continues to hug the upper bound until the highest elevation. Thus, any elevation above 7 m did not experience change. We therefore excluded elevations above 7 m from the analysis.

3. Results

Figure 7 provides the results for the marsh’s loss, while Figure 8 provides the TOC curves for the marsh’s gain. The TOC curves start at the lower-left corner of the parallelogram with coordinates (0,0), where the number of Hits and False Alarms is zero. Each TOC curve consists of segments, where two thresholds bound each segment. Labels on the segments give numerical thresholds when the index variable is numerical. Labels are words when the index variable is categorical. The gray regions of Figure 7 and Figure 8 show regions where it is impossible for a TOC curve to reside. The left gray triangle is an impossible region because Hits cannot be greater than Hits plus False Alarms. The right gray triangle is an impossible region because Hits cannot be less than Hits plus False Alarms. The slope of each TOC curve’s segment is the intensity with which change occurs. Scientists must compare the steepness of a TOC curve’s segment to the uniform line. If a curve’s segment is steeper than the uniform line, then the change occurs more intensively than across the spatial extent in that segment. Conversely, if the segment is flatter than the uniform line, then the change occurs less intensively than across the spatial extent in that segment.

Figure 7a shows the TOC curves for marsh loss between 1938 and 1972, while Figure 8b shows the TOC curves for marsh loss between 1972 and 2013. The steeper parts of the blue curves show that marsh losses were most intensive closer to the marsh edge during both time intervals. The steep parts of the orange curves show that marsh losses were most intensive at the lowest and highest elevations during both time intervals. The steepest part of the red curve shows that marsh losses were most intensive during the second time interval, where marsh gained during the first time interval, which is at the threshold labeled Alternation.

Figure 8a shows the TOC curves for marsh gain between 1938 and 1972, while Figure 8b shows the TOC curves for marsh gain between 1972 and 2013. The steepest parts of the blue curves show that marsh gains were most intensive between 10 and 20 m from the marsh’s edge. The steep part of the orange curves shows intensive marsh gains at intermediate elevations. The green curves show that marsh gains slightly more intensively from water than from upland during both time intervals. Finally, the first segment from the origin for the red curve in Figure 8b shows that marsh gains were most intensive during the second time interval, where marsh was lost during the first time interval. The TOC curve for elevation shows the largest absolute deviation from 0.5.

4. Discussion

Examining changes during at least two time intervals presents an opportunity to observe patterns that would be impossible within a single time interval [20,21]. Figure 7b and Figure 8b display the relationship between the distance to change during the first time interval and marsh change during the second time interval. Figure 7b demonstrates that marsh losses were most intensive where marsh was previously gained, which demonstrates Alternation. Alternation is a pattern where a category alternates between presence and absence through time. Figure 8b shows that marsh gains were most intensive where marsh was previously lost, which is also Alternation. Figure 7b and Figure 8b show the thresholds at which Alternation occurs. Alternation has several implications for interpreting change patterns. For instance, Alternation may indicate a map error. For example, if marsh persists on the ground during both time intervals, while the map erroneously shows an absence of marsh at the middle time point, then the time series of maps will erroneously show loss followed by gain. Alternatively, Alternation on the ground occurs for some categories. Erosion followed by accretion along the edges of the marsh will cause Alternation of the marsh. Cropland alternates when farming practices include sequential cultivation and fallow years [22,23]. Increasing the number of time points increases the possibility of observing Alternation, regardless of whether Alternation is a true change or a result of map errors.

The AUC is a metric that measures the strength of a monotonic relationship between the binary variable and the ranked variable [13]. The maximum AUC is 1, while the uniform line has an AUC of 0.5. A common practice in the profession is to compare 0.5 to the AUC of a ranked variable. However, the AUC does not provide detailed information about the shape of the TOC curve; thus, scientists have cautioned against using the AUC in assessing the performance of models [24,25]. Imagine a situation where a TOC curve crosses the uniform line. This scenario indicates a non-monotonic relationship between the dependent and ranked index variables, potentially resulting in an AUC of 0.5. This scenario highlights the risk of exclusively using the AUC to assess the overall relationship. Scientists may overlook a substantial non-monotonic relationship between the independent and ranked index variables, evident from the shape of the TOC curve but not necessarily conveyed by the AUC alone.

Some authors may be inclined to establish universal rules and anoint the AUC values as poor, acceptable, good, excellent, etc. However, such universal rules do not address particular research questions or applications precisely because they are universally applied. Instead, these universal rules tend to cater more to the psychological desires of scientists rather than serving scientific purposes [13]. Scientists should, therefore, focus on aligning the interpretation of the AUC to the intended purpose of their research. Researchers must decide whether the AUC is a relevant metric. For our example, the AUC is not appropriate to compare the strength of the relationship with distance to the strength of the relationship with elevation because elevation has a non-monotonic relationship with the change intensity of the marsh. Therefore, we take a different approach to interpreting the relationship between the uniform line and the other TOC curves. Specifically, we compare the slope of each curve’s segments to the slope of the uniform line to determine the change intensities along the TOC curve.

Figure 7a,b show a monotonic decreasing trend in the relationship between marsh loss intensity and distance to the edge during both time intervals. Figure 8b shows a non-monotonic decreasing trend for the relationship between marsh gain intensity and distance to the edge. Several factors could account for the observed patterns, ranging from geomorphic processes to the misregistration of images at different time points. For instance, this relationship could be because marsh changes along its edges, which a process of erosion and accretion would cause. Elevation plays a critical role in a plethora of ecological studies [26,27]. Figure 7a,b show that marsh loss occurs most intensively at lower and higher elevations, avoiding intermediate elevations of marsh. The study region has three distinct elevation patterns: (1) lower elevation dominated by water; (2) intermediate elevation occupied by marsh; and (3) high elevation occupied by upland. Figure 8a,b show that marsh gains occur most intensively at intermediate elevations during the first and second time intervals. The green lines in Figure 8a,b show that marsh gains slightly more intensively from water than from upland. The horizontal distance of a segment for a categorical variable shows the size of the category. The TOC curves show that the size of water is greater than that of upland during both time intervals.

Masking the spatial extent to show data only for locations where the phenomenon is plausible helps interpret the AUC. The masked parts of the maps in Figure 5 show where the landscape has not experienced marsh change during the two time intervals. These locations are primarily at the lowest and the highest elevations. Figure 5 shows that marsh does not gain at the lowest elevations or the highest elevations. Failure to mask these areas from the TOC analysis may result in flat segments in the TOC curve near the origin or the upper-right corner of the parallelogram, thus impacting the AUC values. Several studies exhibit this pattern [14,28]. Interpreting the AUC values resulting from the TOC analysis that failed to mask the implausible region of interest leads to misleading conclusions. The next logical question is how to define implausible regions for masking. A straightforward approach is to examine the thresholds for the index variable at locations where the curve leaves the lower bound and touches the upper bound of the parallelogram. Hits equal zero at all thresholds to the left point where the TOC curve leaves the lower bound of the parallelogram. Hits equal Prevalence at all thresholds to the right of where the curve first arrives at the upper bound of the parallelogram. We gain no insight by the inclusion of thresholds to the left of the threshold where the TOC curve leaves the lower bound or to the right of where the curve arrives at the upper bound.

Section 2.3 described the procedure to eliminate the distances to the edge and elevations that do not experience the gain of marsh. Comparatively, Figure 6a has an AUC of 0.9, while the TOC curve for distance to the edge of the marsh in Figure 8b has an AUC value of 0.6.

The green curves in Figure 8 analyze the relationship between marsh gains and land cover in 1938 and 1972. The land-cover maps in 1938 and 1972 have the same categories: water; marsh; and upland. The TOC requires ranking to establish a hierarchy among the categories that lose. The hierarchy derives from the intensities with which the category experiences gain of marsh; thus, the category with the greatest intensity is ranked first, while the category with the least intensity is ranked last. The intensity of each category is calculated as a ratio, with the numerator representing the size of the marsh’s gain from the category and the denominator representing the size of the category. Water and upland have, respectively, 9% and 8% intensities during the first time interval. Similarly, water and upland have 8% and 6% intensities during the second time interval. Water has the greater intensity during both time intervals; thus, the green TOC curves show the segment for water first and upland second.

5. Conclusions

Our manuscript provides scientists with a blueprint for using the TOC to analyze the spatial and temporal patterns of losses and gains of a land category. We show the association between change and four variables to illustrate the TOC’s applicability to varying factors that may influence change in a landscape. Our manuscript shows a change pattern called Alternation, which pairs the losses and gains of a category during sequential time intervals at the same location. We show how to rank a categorical variable for use in the TOC. In addition, we provide a methodology to constrain the TOC analysis to relevant parts of the spatial extent when analyzing a category’s gain. The constraint influences the AUC values and the interpretation of the relationship between the binary and index variables.