Extension of an Open GEOBIA Framework for Spatially Explicit Forest Stratification with Sentinel-2

Abstract

Spatially explicit information about forest cover is fundamental for operational forest management and forest monitoring. Although open-satellite-based earth observation data in a spatially high resolution (i.e., Sentinel-2, ≤10 m) can cover some information needs, spatially very high-resolution imagery (i.e., aerial imagery, ≤2 m) is needed to generate maps at a scale suitable for regional and local applications. In this study, we present the development, implementation, and evaluation of a Geographic Object-Based Image Analysis (GEOBIA) framework to stratify forests (needleleaved, broadleaved, non-forest) in Luxembourg. The framework is exclusively based on open data and free and open-source geospatial software. Although aerial imagery is used to derive image objects with a 0.05 ha minimum size, Sentinel-2 scenes of 2020 are the basis for random forest classifications in different single-date and multi-temporal feature setups. These setups are compared with each other and used to evaluate the framework against classifications based on features derived from aerial imagery. The highest overall accuracies (89.3%) have been achieved with classification on a Sentinel-2-based vegetation index time series (n = 8). Similar accuracies have been achieved with classification based on two (88.9%) or three (89.1%) Sentinel-2 scenes in the greening phase of broadleaved forests. A classification based on color infrared aerial imagery and derived texture measures only achieved an accuracy of 74.5%. The integration of the texture measures into the Sentinel-2-based classification did not improve its accuracy. Our results indicate that high resolution image objects can successfully be stratified based on lower spatial resolution Sentinel-2 single-date and multi-temporal features, and that those setups outperform classifications based on aerial imagery only. The conceptual framework of spatially high-resolution image objects enriched with features from lower resolution imagery facilitates the delivery of frequent and reliable updates due to higher spectral and temporal resolution. The framework additionally holds the potential to derive additional information layers (i.e., forest disturbance) as derivatives of the features attached to the image objects, thus providing up-to-date information on the state of observed forests.

1. Introduction

The availability of information on the state and condition of forest ecosystems is a key element in sustainable forest management and forest monitoring. With data needs covering scales from global monitoring to regional forest management, some can be addressed by operational state forest inventories and aerial surveys, whereas others call for more timely acquisitions and multi-temporal assessments of the studied forest systems [1,2]. Mapping forest resources through Earth Observation (EO) satellites has been successful on global, continental, national, and regional scales [3,4,5,6,7]. In this regard, the twin satellites of Sentinel-2 support a broad range of environmental studies, through the open access strategy of the Copernicus Program. The satellites provide multispectral imagery with a high resolution (HR) of up to 10 m in a high revisit frequency, enabling the continuous monitoring of dynamic ecosystems. The implementation of the temporal domain into environmental studies has become more feasible and successfully shows improvements in tree species classification [2,7,8], forest phenology characterization [9,10], and disturbance mapping [10].

Along with these advances in open access EO data, the quality of derived forest information is improving due to the integration of different data sources, data fusion, and the development of effective machine learning processing chains. However, the production of up-to-date information in sufficient spatial detail for operative forest management and monitoring at forest district level still does not entirely satisfy data needs [1,5,11]. As very high-resolution (VHR) satellite imagery is currently only provided by commercial companies, or can only be accessed through open data programs for scientific use (i.e., the European Space Agency Third Party Missions [12]), aerial imagery remains a crucial dataset in multiple operational forestry applications on a local scale [13,14,15]. Indeed, VHR imagery provided in at least yearly cycles would be necessary to provide the capability to monitor changes and disturbances (i.e., wind fall), whereas intra-annual data in sufficient detail would be necessary for capturing dynamic changes caused by biotic and abiotic drivers (i.e., insect calamities, drought effects) [16,17]. EO systems such as Sentinel-2, which provide image data in short time intervals with increased spectral resolution, thus offer the potential to capture these dynamics, but might not deliver the spatial detail needed. The advantages of VHR aerial imagery as well as the spectral and temporal resolution of HR satellite imagery can thus deliver the crucial combination for the development of local forest monitoring. Although it does not deliver the spatial resolution needed, the use of multi-temporal satellite imagery in environmental studies has shown a reliable increase in the accuracy of forest stratification compared with a single-date aerial acquisition [18].

In this context, we investigate the integration of VHR aerial imagery with HR satellite imagery of Sentinel-2 into an existing geographic object-based image analysis (GEOBIA) framework with a focus on forest stratification of leaf types (needleleaved, broadleaved, non-forest) [19]. Accurate information on leaf type at stand level is the basis for more specific applications. The delineation of leaf type (needleleaved and broadleaved) and up-to-date information on non-forest sites are the basis to investigate the development of forests on an object-specific basis. The integration of multi-temporal data into the GEOBIA framework enables the investigation of recent forest cover changes on the derived objects, while keeping detailed object geometry intact over a longer time period.

VHR imagery comes with its own challenges in processing, and GEOBIA tries to resolve this complexity through the creation of image objects through segmentation of the VHR imagery source. These image objects are used as the fundamental unit for subsequent processing. Due to this shift from pixel to vector objects as the processing unit, GEOBIA workflows can simplify the integration of data from additional sensors and auxiliary vector data through standard tools available in geoinformation systems (GIS) and mitigates problems arising from pixel-based analysis (i.e., salt-and-pepper effects) [20].

The overall aim of the presented GEOBIA framework is thus to develop a workflow to regularly update of forest type maps in the spatial detail that supports their use in operational forest management. In addition, this workflow is based exclusively on open data and free and open-source software (FOSS) with the intent to enable the creation of seamless maps across administrative boundaries and to deliver a replicable framework. Although the initial framework’s segmentation and classification was solely based on aerial imagery [19], the capability to extend the framework is tested through the integration of Sentinel-2 imagery. With the capability to handle data from multiple sources (VHR aerial imagery, multi-temporal, and multispectral HR satellite imagery), the continued development of the framework might thus improve its ability to create more frequent and reliable updates rather than rely on subsequent aerial imagery acquisitions, with their limited temporal and spectral resolution. The open access policy of Sentinel-2 facilitates these updates to the framework. To monitor the dynamics of change processes is an aspect of utmost importance regarding climate change and its effect on ecosystems.

The presented GEOBIA framework consists of an automatic procedure to delineate forest boundaries from color infrared (CIR) aerial imagery through segmentation that satisfies the geometric requirements of operational forest management within central European forests, which consist of a meaningful proportion of small-structured and fragmented units. An MMU of 0.05 ha has been set based on the lower boundary of the UNFCCC forest definition, [21] and is a common MMU for monitoring on a local forest scale [22,23]. While preserving this VHR geometry to reflect forest characteristics, Sentinel-2 provides the spectral and temporal information to provide timely updates on forest status. Sentinel-2 spectral information from eight different acquisition dates over the vegetation period are used to subsequently enrich the image objects. In order to gain some understanding about the potential of Sentinel-2 integration, different temporal (single scenes, multi-temporal) scene combinations and vegetation index time series are tested as features for leaf type stratification in supervised classification using a random forest classifier.

With the intent to support operational forest management with frequent updates, the following questions shall be answered in the context of this contribution:

• Can multi-temporal MR satellite data compensate for the inadequacies of VHR aerial imagery (acquisition time and frequency, shadow cast in the imagery, spectral resolution), while still maintaining the spatial detail aerial imagery delivers towards image objects?
• Is Sentinel-2 able to improve leaf type classification compared with aerial imagery, which is temporally non-uniformly acquired and does not cover phenologically optimal dates?
• What is the optimal temporal setup of Sentinel-2 acquisition dates for receiving reliable updates for leaf type stratification?

We believe the approach of object-based classification of leaf type could prove to provide advantages through segmentation on VHR aerial imagery, while at the same time harvesting spectral and multi-temporal information from Sentinel-2 satellite data to deliver continuous updates for forest stratification. The investigation of the framework extension is finalized by a critical review of the initial framework through a comparative assessment with the integration of the most recent 2020 CIR aerial imagery.

2. Materials and Methods

2.1. Study Area

The Grand Duchy of Luxembourg is located in central Europe and comprises an area of 2586 km² (Figure 1). The landscape is divided into two geologically and geomorphologically different regions: The northern part, which belongs to the Ardennes low mountain range, was formed during the quarterly genesis of the Rhenish Massif, and the southern part, which is dominated by Mesozoic sediments over the Paleozoic basement as part of the Paris basin [24]. The elevation ranges between 129 m and 560 m above sea level [25]. The temperate climate is mostly dominated by west winds, the average temperature is 9 °C, and the mean annual precipitation is around 897 mm. In the northern part, temperatures are lower, and precipitation higher [26].

The geological background becomes especially apparent regarding the landscape’s structure. Forest coverage is higher in the hilly northern part, where forests are mostly concentrated on slopes (Figure 1). In Luxembourg, the total forest covered area amounts to 940 km², which is 36% of the land surface. Private forests account for 35% of the area; 65% is publicly managed forest [27]. Two-thirds of the forest area is covered by broadleaved species, with European beech (Fagus sylvatica L.), sessile and pedunculate oak (Quercus petraea (Matt.) Liebl.; Quercus robur L.), and European hornbeam (Carpinus betulus L.). Representative needleleaved species include Norway spruce (Picea abies (L.) H. Karst.) and Douglas fir (Pseudotsuga menziesii (Mirbel) Franco) [27,28]. The European larch (Larix decidua Mill.) needs to additionally be mentioned as a deciduous needleleaved tree species, which should be of special concern in the presented methodology with its focus to stratify needleleaved and broadleaved forest areas.

2.2. Data

The GEOBIA framework heavily relies on access to several data sources. Luxembourg operates an extensive open data portal (https://data.public.lu, accessed on 8 November 2021) [25], which includes a collection of aerial imagery for several years, making it possible to implement the entire GEOBIA framework exclusively using publicly available data sources. Although the aerial imagery is thus restricted to the national borders of Luxembourg, the additional data sources used in the framework are derived from OpenStreetMap (OSM) as well as Sentinel-2 data accessed through the Copernicus API hub in the FORCE processing chain [29].

2.2.1. Aerial Imagery

Luxembourg’s open data portal offers geographic and non-geographic datasets useful for the investigation of forest ecosystems. Part of the collection of available datasets is aerial imagery in different spatial resolutions up to 20 cm, dating back to 2001. Flyovers of the 2018 aerial imagery took place with flights on the 2 July, 8 July, 27 July, and 5 August [30] at a flight altitude of 3000 m above ground level. The data are supplied in JPEG2000 format as 8-bit true color (RGB) or color-infrared (CIR) image [30,31]. The acquisition of the 2020 aerial imagery took place over a longer period, with nine flights ranging between 30 July and 19 of September. It is available with the same specifications as the 2018 orthophotos, but with an increased ground resolution of 10 cm [32].

2.2.2. OpenStreetMap

Access to official vector data is often limited or restricted to state-owned forests. Volunteered geographic information (VGI), such as OSM, can thus play a vital role in the data access towards different topics. Based on its geographic and thematic information content, the OSM database is utilized in manifold ways in this study. It constitutes a data source with high cross-border consistency, which is a major constraint of administrative land cover datasets [5], and is able to surpass the quality of administrative datasets in areas of active and continuous contribution patterns [33].

Contribution to OSM ranges from hobby volunteers, to organized editing events and bulk imports. OSM is an open database where geographic information is saved in the form of nodes, ways, or relations and additional tags are used to describe thematic information. Several tags, consisting of key = value pairs, are used to depict forest areas in the OSM database (i.e., landuse = forest or natural = wood). Informative tags in addition to these main tags are quite rare in OSM. Although information on leaf type (key leaf_type with values needleleaved, broadleaved or mixed) ranks highest on the list of the tags combined most often with the mentioned main tags, it is only combined with 11.15% of all landuse = forest and 10.65% of all natural = wood (10.65%) objects in the OSM database, as reported on https://taginfo.openstreetmap.org, accessed on 8 November 2021. Nevertheless, those areas carry the potential to be used as training data with a high thematic and positional accuracy [19,34,35].

A substantial amount of the forest areas in Luxembourg is represented in the OSM database. Jaccard’s coefficient [36] can be used to compare the forest areas present in the OSM database with those reported in Luxembourg’s Land Use Map 2018 (LIS-L LU 2018) [37]. The coefficient is defined as the intersect over union of two maps given by:

n11/(n11 + n10 + n01),

(1)

where n11 are the intersecting areas covered by forest according to both maps, and n10 and n01 are the areas covered by forest in one or the other map [38]. The result is a coefficient of 0.93 for the forest areas in Luxembourg, which attests to the large amount of spatial overlap between the two forest maps.

As OSM guidelines do not specify a minimum mapping unit (MMU), a first step to integrate forest areas into the database is to digitize forest boundaries and tag them with the main tag landuse = forest or natural = wood. In the case of small forest plots, the leaf_type = * (mixed, broadleaved, needleleaved) tag is also commonly added. In the case of larger forest plots, a common strategy is to tag the whole area with the tag leaf_type = mixed. Given the lack of MMU in OSM, this strategy is technically correct, if there is a mixture of broadleaved and needleleaved forest. Trees with different leaf types might occur right next to each other or distinct patches of trees of the same leaf type might be visible and could thus be captured separately (Figure 2). Regarding enrichment of the database through additional tags, most of the larger mixed forests contain this kind of recognizable stand borders, which would allow for a separation of needleleaved and broadleaved classes. In 2020, around 4600 landuse = forest polygons amounted to an area of 90 km². Around 67% of all polygons had been tagged with an additional leaf_type key. Around half the amount of these areas were tagged with the mixed value but contributed to 85% of the whole forest area in Luxembourg.

2.2.2.1. sBoundaries for Segmentation

Study sites to be investigated were selected based on the OSM database in the beginning of 2020 and comprise forest areas tagged with leaf_type = mixed in addition to landuse = forest. The data have been acquired through the interactive OSM interface using the Overpass API (http://overpass-turbo.eu/, accessed on 8 November 2021) [39]. Investigation of these forest sites has shown the value mixed for leaf_type is not appropriate for implementation of those forest areas into analysis related to local forest management, as can be seen in Figure 2. The make-up of these forest sites is defined by small-structured and fragmented needleleaved, broadleaved, and mixed stands, offering the potential for further delineation through their subdivision. The selected forest sites have a sufficient size and contain a complex spatial composition of leaf types to investigate the presented approach.

In OSM, most land cover objects are delineated through so-called “armchair mappers”, which rely on imagery provided through OSM editors. According to OSMstats (https://osmstats.neis-one.org/, accessed on 8 November 2021) for Luxembourg, there are two contributors who actively work on land use features [40]. In the case of Luxembourg, the available data include the country’s most recent aerial imagery acquisition, which allows OSM contributors to make use of up-to-date and very high-resolution imagery for mapping [41]. In addition to the high value of Jaccard’s index, visual inspection of the delineated forest sites in Luxembourg attests to the spatial accuracy of the mapped objects which were thus deemed appropriate to delineate our study sites.

These study sites function as the boundaries used in the segmentation process of the GEOBIA workflow (see Figure 1 for an overview of all study sites). The inclusion of vector boundaries is a common procedure in GEOBIA workflows as known boundaries can be implemented to restrict the segmentation and lead to more accurate results [42], which can be implemented into an existing data structure.

2.2.2.2. Training Data

Deriving training data from OSM is a promising approach for supervised classification as the information includes detailed geometries with additional thematic information. If contribution of volunteers is high in a specific region and the target classes are covered by the thematic information of attached tags, OSM data can be readily used as areas for the training of supervised classifiers [43,44]. Depending on the target classes, OSM training data must be semantically preprocessed to match the intended classification scheme [45].

The training data for the presented framework have been derived from OSM in the same way as the boundaries for segmentation. Instead of the mixed value, data were filtered by needleleaved and broadleaved values for the key leaf_type where the tag landuse = forest was present. Through this approach, it is assured the training and target datasets are spatially independent as there is no overlap between those entities. The target objects, which will be created through a segmentation process on the mixed forest sites, thus do not contain any overlap with the training objects.

Pre-processing of the derived OSM data was needed to minimize the influence of mixed pixels and the spatial discrepancy between the training data and Sentinel-2 pixels. Accordingly, a negative buffer of 10 m was applied and training data above the MMU of 0.05 ha were retained (

Table 1. Training data derived from OpenStreetMap with number of samples, their minimum, maximum and mean size per class in ha and visual examples of the minimum and maximum sized training plots per class with their negative buffer and an example of the smallest training plot superimposed on the 10 m resolution of Sentinel-2.

		Class
		Broadleaved	Needleleaved	Non-Forest
Sample Size		166	203	22
Minimum (ha)		0.09	0.05	0.05
Maximum (ha)		27.20	8.41	0.54
Mean (ha)		2.52	0.71	0.24
Examples	Minimum
	Superimposed Sentinel-2 pixels
	Maximum

).

This procedure ensures the calculation of zonal statistics is based on at least four Sentinel-2 pixels for each training area. As can be seen from Table 1, the training areas can be situated adjacent to fields or settlements, and especially with smaller areas, the influence of mixed pixels could potentially be high if not corrected. An example of a minimum size training area has been superimposed with Sentinel-2 pixels to show the positive effect of the negative buffer selection process (Table 1).

In addition, since it cannot be ruled out that errors or outdated information are present in the OSM database, the retrieved training data have been visually checked to make sure the tags match the leaf type present in the aerial imagery of 2020 [32]. In this process, there was indeed a mismatch between OSM and the aerial imagery in some areas (i.e., harvested or disturbed areas). The dynamic of the 2018–2020 summer heat waves and the mass spread of bark beetles could be due to two explanations. Those mismatch occurrences contain useful information and have thus been retained in an additional non-forest class. The class describes areas that will probably return to a forested state but do not contain any tree cover at present. Non-forest, therefore, describes the current state of land cover with soil, grass, small bushes, and remainders from the disturbance (i.e., logs, leaf material) rather than the land use.

After these checks, 391 training areas were retained to train broadleaved, needleleaved, and non-forest classes. Table 1 gives an overview of the training areas and their characteristics. It is noticeable that broadleaved training areas tend to be larger in size in comparison with needleleaved areas, which should be kept in mind as an influential characteristic for training and classification. Additionally, non-forest is a minority class, with only 22 training areas.

2.2.3. Sentinel-2

For the extension of the GEOBIA framework, Sentinel-2 provides optimal premises. Operating since June 2015 (S2A), and since March 2017 (S2B) as a twin satellite system, multispectral image data are provided in short term intervals (5 days revisit time at nadir) at a ground resolution of up to 10 m [46]. The large swath of 290 km allows a spatially and temporally consistent mapping for large areas. The data are provided free of charge [47].

Sentinel-2 imagery was acquired through the Framework for Operational Radiometric Correction for Environmental Monitoring (FORCE) [29]. The FORCE pre-processing chain comprises full radiative transfer modelling, based on Tanré et al. [48]. From this imagery collection, suitable images for the year 2020 were selected. The selected tiles were confirmed by the quality flags, delivered with the FORCE pre-processing chain. All scenes have been assured to be cloud-free over the areas of interest (namely OSM training and OSM boundaries for segmentation).

Since its start, Sentinel-2 has been able to provide a basis for operational applications related to forest monitoring [46] and for scientific applications regarding land monitoring and management with forests among other domains. One of the first studies in the forest domain by Immitzer et al. [49] has shown 93% accuracy in leaf type classification but was not able to achieve satisfactory results in tree species classification based on a single-date Sentinel-2 scene. One drawback that was specifically highlighted was the general heterogeneity of the forest in the selected study area, which could not be fully captured by the spatial resolution of Sentinel-2 [49].

Observation and monitoring of dynamic vegetation characteristics are enabled through the temporal coverage of Sentinel-2 and other medium resolution satellites [50,51,52]. The high repetition rate of the Sentinel-2 twin satellites naturally favors the application of multi-temporal imagery in classification of forest ecosystems. It enables the selection of several acquisitions during appropriate periods, e.g., during leaf-off conditions in March to discriminate between leaf types to be able to cover larger study areas [5].

Moreover, there are several examples of multi-temporal imagery being used as features in machine learning classifications, without aggregation or specific selection [8,53]. For the satellite image-based stratification of forest types in a temperate zone, seasonal variability of the target object and the time of data acquisition are crucial. Significant spectral differences in the forest’s reflectance properties, which are necessary for the distinction of needleleaved and broadleaved forest, occur during the leaf-off period, during the onset and throughout the early foliage development of broadleaved forest.

Consequently, data were acquired starting in March, with a focus on covering the onset of the greening of broadleaved forest and the phenological development until the end of summer (

Table 2. List of Sentinel-2 scenes used in the study.

Scene	Sensor	Acquisition Date
S1	Sentinel-2 A	2020 March 28
S2	Sentinel-2 A	2020 April 7
S3	Sentinel-2 B	2020 April 22
S4	Sentinel-2 A	2020 May 7
S5	Sentinel-2 A	2020 May 17
S6	Sentinel-2 A	2020 May 27
S7	Sentinel-2 B	2020 July 31
S8	Sentinel-2 A	2020 August 5

). In this period, the sun position is already high enough to provide acceptable image data quality. The scenes from July and August have been acquired to investigate the capabilities of single-date scenes to distinguish between broadleaved and needleleaved forest in less optimal phenological conditions, i.e., when spectral differences become less pronounced. These scenes allow the comparability with previous studies based on VHR aerial imagery acquired in July and August [19].

2.3. Methodology

The initial methodological GEOBIA framework was developed as a straightforward example to derive spatially explicit broadleaved and needleleaved target objects as subdivisions of large mixed forests in the OSM database [19]. An extension to the initial framework presented in this study can be found in the selection of training areas, which have been updated to include a non-forest class described in Section 2.2.2.2, the use of more recent validation data described in Section 2.3.3, and updates to the random forest classification with extended input of multi-temporal Senitel-2 imagery (Section 2.2.3). A comparison with the initial framework will be drawn by a comparative assessment with recent CIR aerial imagery of 2020 (Section 3.2.4). The extended framework (Figure 3) delivers resulting forest leaf type maps for the study sites, but more importantly it aims at presenting a reproducible combination of methods for GEOBIA classification workflows in the context of forest management units that do not include such detailed spatially explicit information. The initial GEOBIA framework this work is based on has been developed with the use of aerial imagery only [19]. Classification was solely based on spectral bands and texture parameters derived from a single-date aerial image of 2018. A random forest classification based on visible and near infrared bands and texture parameters achieved the highest overall accuracy of 85%, when being trained on manually selected OSM training data in comparison with 84% overall accuracy for training on automatically derived data from OSM based on leaf_type = * tags and filtered by their size. The results revealed that training on OSM-derived areas is feasible and most misclassifications could be attributed to the influence of shadow proportion in sparse broadleaved stands and to young dense stands of needleleaved forest being misclassified due to their small spectral difference to broadleaved forest, as well as texture information that does not match mature stands.

2.3.1. Region Growing Segmentation

The segmentation of remote sensing imagery is a fundamental step in GEOBIA procedures, guiding all subsequent quality of the derived data. Although GEOBIA does not always lead to static classifications, the influence of the segmentation towards classification accuracy is well established [54,55,56]. During the segmentation, spectrally homogenous image objects are delineated, which should represent the intended target objects of the analysis in their spatial boundaries. These boundaries are then used to derive the statistical measures from the selected image data, and thus can strongly affect the subsequent classification.

This strong influence of the segmentation also requires that the spectral and spatial resolution need to be able to capture these target objects for an accurate delineation. The derivation of image objects through VHR satellite imagery and aerial imagery show that spectral information in the infrared is of high importance to delineate leaf types. This has previously been shown in the context of the presented GEOBIA framework in the comparison of segmentation derived by aerial imagery with and without the inclusion of an infrared band, where RGB imagery has not been sufficient to derive a meaningful segmentation [19].

One of many segmentation methods is region growing, a type of algorithm where segments are iteratively merged with their most similar neighbor, based on a similarity measure, to create image objects. Region growing algorithms either start the iterative merging process with all pixels in an image set as segments, or at predefined pixel locations. Whether the most similar neighbors are merged is decided by the threshold or scale parameter, which is the most important tuning parameter of the segmentation, and inherently determines the size of resulting image objects. Its value specifies the maximum level of dissimilarity under which two segments are merged. Thus, a large threshold value leads to a larger amount of merging iterations, and subsequently larger and more heterogeneous image objects.

Tuning the parameters of the region growing procedure often relies on testing different parameters through a trial-and-error approach. Results of different threshold combinations are then screened for their ability to delineate the target objects by a visual assessment [23,57]. Although most object-based approaches still rely on this visual assessment of the segmentation, and it was the only available method at the beginning of GEOBIA approaches [58], several automated methods to derive an indication of the best segmentation parameters do now exist.

One example of such an automated parameter tuning procedure is available in GRASS GIS through the i.segment.uspo module [59]. Unsupervised testing of all parameters of the region growing procedure (threshold, radioweight, smoothweight, minsize) is possible when combining the module with the i.segment [60] or i.segment.gsoc [61] module. This unsupervised segmentation parameter optimization (USPO) is based on Espindola et al. [62] and Johnson et al. [63] and results in a measure based on within-segment homogeneity and between-segment heterogeneity for each parameter combination that is described in more detail in regard to a GEOBIA processing chain in an urban context by Grippa et al. [64]. For the final measure of the USPO, an additional weight (α parameter) can be set either to emphasize homogeneity inside an image object or heterogeneity between the image objects. This allows for additional adaptation of the measure towards the specific needs of the intended segmentation and target objects. Based on this measure, the optimal parameters of the segmentation can be retrieved. However, even though the optimization results in a good approximation of segmentation parameters based on the inherent spectral image statistics, a visual assessment of the resulting segmentation should not completely be forgone to make sure the image objects reflect the target objects. Grippa et al. have used the USPO procedure to derive segmentation parameters for a heterogenous urban region and found that locally optimized thresholds for subsets of the region outperform a globally derived threshold on the whole dataset [65].

2.3.2. Random Forest Classification

Image objects derived through the region growing segmentation are the basis for the prediction of the target objects through a random forest classification [66]. Random forests are ensemble classifiers of decision trees, where the final classification is derived through a majority vote. In many remote sensing studies, random forest classification is employed as an easy-to-use classifier, which removes the need to tune many hyperparameters and has no assumptions concerning data distribution [52,66]. Random forest classifiers have also been successfully implemented in GEOBIA workflows and show low sensitivity concerning feature selection and the use of multi-source features [67,68]. For an extensive overview and review about random forest classifications in the field of remote sensing, see Belgui and Drăguţ [68].

Classifications in GEOBIA workflows are based on the features calculated for each delineated object, and are thus not conducted on a per-pixel basis. Additional information is attached to each image object through the calculation of zonal statistics based on the bands of optical or multispectral datasets (e.g., aerial imagery, Sentinel-2), features derived from these datasets (e.g., vegetation indices) and optional thematic or topographic datasets. In most cases, derived features are the central tendency of each image band and measures of their statistical dispersion [67].

Multi-temporal Sentinel-2 imagery has successfully been used in random forest classification setups. The use of all available bands from each involved acquisition as features for the random forest model makes it possible to draw conclusions based on the importance measures reported by the random forest classifier [2,8]. Classification based on single-date Sentinel-2 scenes show the timing of the acquisition is of utmost importance if targets of interest have dynamic spectral characteristics. Low accuracies based on a single date Sentinel-2 acquisition do not necessarily mean Sentinel-2 is incapable of differentiation, but might give an indication of an untimed acquisition [1].

Studies in the forest domain commonly use additional datasets such as airborne laser scans (ALS), digital elevation models, or texture as features for classification [69,70,71]. Texture measures based on the Gray-Level Co-Occurrence Matrix (GLCM) tend to increase the classification accuracy in natural settings, which could also be confirmed in previous experiments and other studies [13,19,72]. The calculation of texture measures is processing intensive, which is why most studies focus on a small selection of measures. Hall-Beyer (2017) [73] provides practical guidelines to which measures are preferential regarding classification on different spatial scales. The selection is still very subject-specific, and additional parameters regarding window size and step width of the moving window need to be chosen.

Other features employed in GEOBIA analysis are features derived through the object geometry itself (i.e., compactness, elongation). These features can tremendously improve the classification accuracy in GEOBIA workflows, but their contribution is highest in settings where the target objects have an inherently distinct geometry. Features based on an object’s geometry are more effective in the classification of objects with clear boundaries than in natural settings, where boundaries tend to be fuzzy. As such, studies have found them to deliver no additional value to the classification accuracy [13].

2.3.3. Validation via Reference Map

Data on forest areas in Luxembourg have been derived from Luxembourg’s Land Use Map 2018 (LIS-L LU 2018). This map includes four forest classes in the forest block category (needleleaved, broadleaved, mixed, and young forests) and three forest classes in the clearing category (burnt area, storm damage, clear-cuts, and other) with an MMU of 0.05 ha [74]. The land use map has been created in an object-based approach integrating several data sources, i.e., 0.2 m aerial imagery of 2018, land use map of 2015, additional vector datasets for water, roads, and buildings as well as elevation models [37]. This land use product is a land use product used for the validation of our results, but not a ground truth dataset and thus in itself subject to accuracy assessment. The reported overall accuracy for the land use 2018 map is 96.5% with user accuracies of over 90% in all classes of the forest block category and over 93.3% for the classes in the clearing category. Although the land use map and the image objects of the presented framework are both based on aerial imagery from 2018, we can assume higher accuracy for the land use 2018 map due to the use of the auxiliary data sources and digitization and manual improvements in its production. As such, the validation dataset adheres to the recommendations by Olofsson et al. for sources of reference data [75].

Mixed and young forests in the forest block category present special cases and are not part of our target classes due to missing training data and potential overlap to the defined classes. The reference polygons from these classes have thus been retained from the accuracy assessment, although these classes are certainly present in the larger mixed forest sites from the OSM database. Ideally, young forests are classified as either needleleaved or broadleaved based on their phenological development over the investigation period, whereas mixed forests should be separated if their composition is that of mixed stands with apparent stand borders as introduced in Figure 2. The young and mixed forest classes are thus evaluated in a visual assessment.

The sampling strategy for the accuracy assessment is based on a regular point grid at 50 m. This strategy ensures that complex spatial conditions are represented in the accuracy metrics (e.g., stand border situations and image objects with high shadow proportion). The reported overall accuracies are based on confusion matrices. The best performing classification is discussed in depth with statistical measures of producer’s and user’s accuracy, allocation and quantity agreement, as well as a detailed visual assessment with a critical discussion about the results and their limitations.

3. Results

The segmentation results as well as the results for the best performing classification are available to be interactively explored in the web map viewer via https://t1p.de/hgi9, accessed on 8 November 2021. The map viewer additionally includes bookmarks to allow quick access to the areas covered in the presented figures and an interactive legend to filter out certain areas of interest.

3.1. Region Growing Segmentation with USPO

Very high-resolution 2018 CIR aerial imagery from Luxembourg has been used as input for the segmentation procedure. To optimize processing time and regarding the intended scale of analysis, the imagery has been resampled from the initial spatial resolution of 0.2 m to a lower resolution of 2 m. The segmentation on this resolution assures the target objects of broadleaved and needleleaved forest areas can still be detected (see Figure 4), whereas smaller variations in spectral values have less negative influence on the segmentation procedure. Especially in the last step of the segmentation, these small variations would lead to unpredictable behaviour, as the minsize parameter would force them to merge with the most similar neighbour to arrive at the MMU of 0.05 ha. This step is processing intensive and if resulting segments are far below the MMU, several iterations are needed to arrive at the image object. The parameters radioweight and smoothweight were kept constant at the defaults of 0.9 and 0.5, whereas the α parameter was set to the value 2 to evaluate the segmentations rather based on within-segment homogeneity than on between-segment heterogeneity. Therefore, the parameter optimization in the USPO procedure was focused on the threshold parameter.

The final segmentation into image objects is derived ‘polygon-wise’ for each mixed forest polygon derived from the OSM database. Each polygon is used as a region of interest for the USPO procedure and the best segmentation parameters are reported separately for each polygon. This approach to retrieve a separate threshold parameter for each mixed forest polygon is preferred over a global threshold parameter, which was not fit to cover the different spatial contexts of the mixed forest polygons in the study area (e.g., surrounding topography, forest management schemes or different ecological stand conditions). Visual assessment of segmentation is still important, as was proven in previous experiments on segmentation of aerial imagery in RGB and CIR [19], where a more detailed assessment of the segmentation can be found. The derived image objects of the two mixed forest sites introduced in Figure 2 can be investigated in Figure 4. The two example sites show image objects of different size and complexity have been created in the mixed forest boundary, but the different leaf types are delineated into mostly homogenous image objects.

3.2. Classification in Different Setups

Based on the available input data, various measures can be calculated on a per-object basis to provide aggregate information for the classification of each image object. Features for classification are the median of each spectral band and each date for the eight dates of acquired Sentinel-2 imagery (80 features). Additionally, the Simple Ratio (SR) has been calculated for each date to investigate the power of a simple Sentinel-2 derived vegetation index (eight features). Furthermore, CIR aerial imagery of 2020 was used to have a fair comparison, if data from a recent aerial acquisition mosaic could be sufficient to classify and update leaf types on the objects derived from 2018 aerial imagery. We run these classifications on the median values of the spectral bands (four features) and all available texture measures derived from GRASS module r.texture (13 features) [76]. Texture measures have been calculated on the NDVI of the CIR aerial imagery on a window size of seven at a step width of one pixel. Due to processing time in the texture calculation, the imagery has been resampled to 1 m resolution.

Overall, this results in a set of 113 potential features to be used for the classification based on Sentinel-2 imagery, as well as aerial imagery with derived texture measures. Although it is common to introduce all features into classifications in random forests, the tests have been restricted to a few different classification setups (

Table 3. Selection of classification setups with features derived from different input combinations with associated number of features and their achieved overall accuracy.

Combinations						OAA
Sentinel-2 Scene(s)			Index Time Series	Aerial Imagery	Texture
S3 (10)						88.3%
S8 (10)						84.9%
S3 (10)	S4 (10)					89.1%
S3 (10)	S4 (10)	S5 (10)				89.3%
All (80)						88.9%
			SR (8)			89.4%
				CIR (4)		73.0%
				CIR (4)	T (13)	74.5%
			SR (8)		T (13)	89.4%

). We decided on this procedure as more realistic suggestions can be derived based on these setups, as the result of classification based on all features often leads to in an impractical mix of important features not fit for our objective (i.e., single bands from multiple different dates and different sensors). Classification on all derived Sentinel-2 scenes is still performed and compared with the classifications based on specifically selected feature combinations. Additionally, we investigate if the texture measures based on aerial imagery of 1 m are able to further improve the best performing model derived from the setups based on Sentinel-2 derived features.

3.2.1. Single-Date Sentinel-2 Classifications

Classification models based on single-date Sentinel-2 acquisitions have been able to achieve overall accuracies in a range from 84.9% to 88.3%. The classifications based on the acquisitions from 22 April (S3) and 7 May (S4) achieved the highest accuracies, whereas the lowest overall accuracy was derived with the scene acquired on 5 August (S8) (see

Table A1. Classification setups with features derived from different Sentinel-2 Scenes with associated number of features and their achieved overall accuracy.

Sentinel-2 Scene	Sensor	Acquisition Date	OAA (%)
S1 (10)	Sentinel-2 A	28 March 2020	86.5
S2 (10)	Sentinel-2 A	7 April 2020	85.7
S3 (10)	Sentinel-2 B	22 April 2020	88.3
S4 (10)	Sentinel-2 A	7 May 2020	88.2
S5 (10)	Sentinel-2 A	17 May 2020	88.0
S6 (10)	Sentinel-2 A	27 May 2020	86.2
S7 (10)	Sentinel-2 B	31 July 2020	87.0
S8 (10)	Sentinel-2 A	5 August 2020	84.9

in Appendix A for all single-date accuracies). In general, the classification accuracies do not differ tremendously, and confirm scenes in spring are preferable to summer scenes in leaf type classifications. Visual assessment of the target objects (Figure 5) shows significant differences between the classifications based on S3 (best) and S8 (worst). Through the aerial imagery in the background, it is apparent that the classification based on S8 is not able to capture all occurrences of broadleaved forest. In both cases, the non-forest plot was identified, but it is noticeable that the associated image object does not completely cover the plot. The classification based on S3 performs much better and results in the wanted target objects.

The single-date classifications differ in their reported importance scores, with the best performing classification based on S3 reporting B3 (green), B5 (red edge) and B6/B11 (red edge/SWIR) as the most important bands in that order, rather than B5 and then B3/B6/B11 as equally important for S4 (Figure 6). Interestingly, the importance scores of the single-date classifications show a certain shift. Although the most important bands were B11 and B12 (SWIR) in March (S1), B5 (red edge) has higher importance than B11 in April (S2). This shift carries through April and May (S3 to S6), where especially B12 carries low importance in the classification. Although B3 (green), B5 and B6 (red edge) show high importance in S3 and S4, importance further shifts to the NIR and red-edge bands (B7, B8, B8a) in higher spectral wavelengths in May. Finally, July (S7) and August (S8) do not show an enormous difference in importance scores, but most importance is reported on B11, followed by B6, B8, and B12. This progression clearly shows the importance is related to the phenological development of the target vegetation. A comparison with the spectral profiles of the training areas (Figure 6) additionally confirms this. In the scenes S1 and S2, the spectral profile of broadleaved areas follows the spectral profile of needleleaved areas in the red-edge and NIR bands, whereas differences are visible in the visible spectrum and in the SWIR region. In S3, the spectral profile of broadleaved forest shows differences in the green, red-edge and NIR bands. Concerning all three target classes, the spectrums show similarities and dissimilarities in different regions of the spectrum for each scene, and S4 seems to be the best scene to separate the classes visually, whereas S3 reports a slightly higher accuracy.

3.2.2. Multi-Temporal Sentinel-2 Classifications

The collection of eight Sentinel-2 scenes allows for several temporal combinations to be covered through multi-temporal classification setups. Although a total of 247 multi-temporal combinations would be possible when using between two and up to eight scenes, we have picked a subset of scenes to be tested based on the phenological development of the target classes and based on results indicated by the single-date classifications.

In the case of multi-temporal classifications based on two scenes (with a total of 28 possible combinations), we have selected the best performing single-date classification (S3) as a baseline for comparison. All other scenes (S2, S4–S8) are combined with this baseline scene for the multi-temporal classification. In general, the change in accuracy is small compared with the baseline, with a maximum decrease of 0.38 and maximum increase of 0.76 percentage points (pp). An increase in accuracy in comparison with the baseline (single-date S3) indicates the inclusion of the added scene delivered useful information (Figure 7). It is apparent that most of the information is already contained in the baseline scene S3 and only a small contribution to accuracy (<1 pp) can be achieved when including scene S4 (7th May) or S5 (17th May) in the model (S3 + S4; S3 + S5).

Multi-temporal models based on three scenes have been tested in the combination S3+S4+S5. This combination further improves the accuracy by 0.2 pp and thus by a total of 1 pp in comparison with the baseline of single-date classification on S3. The importance scores of this multi-date model are concentrated on S4 features (B5, B3, B12, B7), on B3 and B5 from S3 and B6/B7/B8(a) from S5 (see Figure A1 in Appendix A). Although this can partly be traced from the single-date models in Figure 6, it shows the temporal development of some bands seems to be a decisive factor in the classification and again shows the features of scene S4 are generally of high importance.

The final multi-temporal model based on all Sentinel-2 scenes (S1–S8) achieves an accuracy of 88.9% and thus performs better than the single-date baseline model S3, but lacks behind the models based on multi-date S3 + S4 + S5 and based on multi-date S3 + S4. The importance scores for the model based on all Sentinel-2 scenes show features of high importance in all scenes except S7 (July) and S8 (August). Although the importance scores of the features also available in the S3 + S4 + S5 model have not changed their order among themselves, the importance of features based on S2 and S6 is emphasized in the importance scores of the model (see Figure A2 in Appendix A).

3.2.3. Classifications Based on Index Time Series

Classification based on multi-temporal scenes has shown a slight increase in accuracy in comparison with the single-date baseline model S3. In order to assess the power of vegetation indices over the investigated time period, classification was conducted on the Simple Ratio (SR) of B4 (red) and B8 (NIR). The SR achieved the highest accuracy reported for all tests at 89.4%, which is an increase of 1.1 pp to the baseline model. The progression of the SR index for each leaf type (including non-forest) shows the early foliage development phase of broadleaved forest (Figure 8). As SR values rise in the broadleaved class due to greening, a large overlap exists in the values of broadleaved to needleleaved class at the scene S3 (22nd April) used as baseline for the single-date and multi-temporal classifications. The index of this scene does thus not have the same differentiation power. This is additionally confirmed through the importance measures with highest importance reported on indices based on scenes S1 and S2 and lowest importance on the indices derived from scenes S3 and S8.

As the classification on SR time series is thus the classification that has best performed among the selected models, a closer look into its results is needed. The confusion matrix in

Table 4. Confusion matrix and derived accuracies and disagreements in % for the SR time series classification.

	Reference
		Needleleaved	Broadleaved	Non-Forest
Prediction	Needleleaved	5208	864	114
	Broadleaved	1697	25423	389
	Non-forest	347	275	345
OAA = 89.4	PAc	71.8	95.7	40.7
CAA = 69.4	UAc	84.2	92.4	35.7
Q = 3.1	qc	3.1	2.7	0.3
A = 7.5	ac	5.6	6.6	2.9

OAA: Overall Accuracy; CAA: Class-averaged Accuracy; PA_c: Class-specific Producer’s Accuracy; UA_c: Class-specific User’s accuracy; Q: Quantity Disagreement; q_c: Class-specific Quantity Disagreement; A: Allocation Disagreement; a_c: Class-specific Allocation Disagreement.

shows the reference and prediction regarding the validation grid, which results in the overall accuracy of 89.4% for all test sites combined. Due to the imbalance in classes the class-averaged accuracy is given with 69.4%. Class-specific accuracy measures of producer’s and user’s accuracy as well as quantity and allocation disagreement [77] are reported additionally.

Visual comparisons between the SR time series classification and the S3 baseline model can be investigated in Figure 9. The comparisons cover forest sites with different spatial configurations with site-specific accuracy of the SR time series classification additionally reported. It is visible from the examples the SR time series classification correctly identifies broadleaved and needleleaved leaf types in the cases where the S3 baseline model overestimates needleleaved and non-forest plots. Interestingly, although visible in the uppermost example, areas of needleleaved forest plots still stocked, but without signs of vitality due to bark beetle infestation, are classified as non-forest in the SR time series model.

3.2.4. Reevaluating Classifications Based on Aerial Imagery of 2020

As final setups, classification has been performed on the CIR aerial imagery of 2020 and associated texture measures. Previous experiments on the GEOBIA framework have resulted in a leaf-type classification (broadleaved/needleleaved) with an overall accuracy of 85% based on 2018 CIR aerial imagery with derived texture measures [19]. Although the new setup is not entirely comparable due to an update to the training data, the introduction of an additional class and an update to the validation strategy, the setup still calls for an investigation. A classification procedure that is based on the same data source used for segmentation, and for which the data source, in the case of Luxembourg, is published as an open dataset for each year, could be nicely integrated into monitoring frameworks. Apart from the processing intensive segmentation and calculation of texture measures, the classification framework could be toned down to just a several steps and would not need to include specific selection of appropriate scenes.

Unfortunately, the results of the classification based on aerial imagery and texture of 2020 are not in its favor. Although the classification based on aerial imagery only reaches an accuracy of 73.0%, the classification with the inclusion of 13 texture measures based on the GLCM achieved 74.5% overall accuracy. This is a small improvement, but the achieved accuracy does not come close to the accuracy achieved with Sentinel-2 based features. The results and associated importance scores indicate the classifications of Sentinel-2 scenes in summer use B11 and B12 as most important features (Figure 6), spectral bands that are not present in the CIR aerial imagery.

Although the summer scenes (S7, S8) performed worse in the single-date Sentinel-2 setups, their accuracy is 10 pp higher than the accuracy based on CIR aerial imagery. Visual comparison of the classification on S8 versus the classification based on CIR aerial imagery confirm the same problems identified in our initial GEOBIA framework (Figure 10). Open broadleaved stands, where larger gaps occur between neighboring trees, are prone to be misclassified as needleleaved due to the shadows cast, leading to lower spectral values when averaging image bands over the image objects.

The 13 texture measures slightly improved the accuracy of CIR classification and could thus deliver some additional information to the Sentinel-2-based classifications; the model of the SR index time series was tested with the extension of adding the 13 texture measures (Table 3). In this case, the texture was not able to deliver any important added information for the classification and lead to the same accuracy as classification on the SR time series (89.4%). This is also confirmed by the importance measures of the random forest, where all SR features rank higher in importance than the texture measures.

4. Discussion

Several aspects of the framework are discussed in the following sections, including related issues and particular problems identified in the procedure.

4.1. Image Objects and Their Influence on Classification

Segmentation is considered the most crucial step in a GEOBIA framework [54,55,78,79]. Not only is it a daunting task to derive spectrally homogenous image objects through the segmentation procedure in a natural context using VHR imagery, it additionally needs to be ensured that derived objects represent thematic objects or subsets of thematic objects for further analysis. Ideally, a derived image object and its thematic content should coincide. Alternatively, the possibility to merge the derived objects into one thematic object should be given (for example, to merge spectrally different objects, i.e., covering forest floor and tree crowns into a thematic object covering the whole forest plot). It is the spectrally and structurally difficult setting that leads to difficulties in the segmentation procedure of forest areas [23,78]. The need to further develop segmentation and the whole GEOBIA concept in the context of natural environments has been identified by Johnson et. al, where natural forest areas rank high among the environments survey participants identified as insufficiently researched [54].

The segmentation procedure in our framework is based on recommendations by Grippa et al. to derive optimal local thresholds for each study area [65]. This procedure worked well in most parts of the study area characterized by mixed forests but evoked over-segmentation in study areas covered by only one target class. In these cases, the threshold is still optimized to separate objects based on their spectral difference, leading to over-segmentation with an optimized local threshold much lower than a spectrally rich study area with needleleaved and broadleaved forests (Figure 11). The area in Figure 11 (left) is structurally and spectrally heterogenous due to many gaps in the tree crown occurring as a shadow in the aerial image, but thematically homogenous. The result is an over-segmentation of the target objects which strongly follow the gaps in the forest canopy at their boundaries. Judging from our results, these over-segmented, smaller image objects can still successfully be classified into target objects, even though the derived object statistics are based on fewer Sentinel-2 pixels. Although we made sure the training areas are covered by at least four Sentinel-2 pixels, object statistics of derived image objects (n = 36,010) are computed on less than three Sentinel-2 pixels in a few cases (n = 116).

In contrast, under-segmentation poses a more challenging problem. If an image object covers several target classes, the classification naturally leads to one definite target class. An example of where this becomes problematic is visible in Figure 12 (left), where an image object classified into the needleleaved target class covers both needleleaved and broadleaved areas. Although these occurrences should be minimized during the segmentation procedure, under-segmentation can also become a problem due to temporal changes as presented in Figure 12 (right). Figure 12 depicts another problem still present, which is complex image object shapes. The image object in the center extends along the forest boundary. These occurrences should be mitigated by incorporating the shape parameters into the segmentation. Although the shape parameters improve the segmentation compared with segmentation based on spectral values alone, image objects such as this are still sometimes created.

Although as previously noted, the shadow problem of VHR imagery is mitigated by using spectral features derived from Sentinel-2 for classification, difficulties remain where image objects have been created based on gaps in sparse forest areas or at forest area borders. These borders can occur at the boundary of the mixed forest study areas or can constitute borders within the larger area. In the final result, these areas are commonly classified as either needleleaved or non-forest, depending on the environmental context with examples also visible in Figure 12.

Another aspect to discuss is the temporal stability of the image objects considering the GEOBIA framework as a whole.

In the example of Figure 12, changes to the forest areas have developed between 2018 and 2020. The image objects employed in this framework have been derived from the aerial imagery of 2018. Keeping the image objects stable to enable progressive updates, thus comes with the downside that changes inside the image objects might not be easily detected.

Considering segmentation is the most computing intensive procedure in the GEOBIA framework, the image objects should ideally be unchangeable. Although forests are a rather stable land cover and land use class, there are enough examples of when a change to the geometry of an object might be necessary, i.e., storm, fire, bark beetle damage, or clear-cuts. Updating the image objects additionally carries the need to recalculate all features previously attached to the former image objects.

Calculated as exemplary for one forest site, Figure 13 depicts image objects based on 2018 aerial imagery (left) and 2020 aerial imagery (right), whereas both image objects have been classified with the Sentinel-2 SR time series of 2020. It is noticeable that the segmentation based on 2020 aerial imagery leads to very different image objects and less under-segmentation of the target classes. Although some of the previously under-segmented areas are now correctly classified, the image objects in the center are still falsely classified as needleleaved. This example demonstrates potential issues due to temporal inconsistencies, but also points out the potential of regular input of spectral information from Sentinel-2 imagery. This issue is further discussed in chapter 4.3 (Temporal Inconsistencies).

4.2. Sentinel-2 as (Multi-Temporal) Input Data

The goal of extending an established GEOBIA framework with Sentinel-2 data has shown the utility and suitability of Sentinel-2 for mapping leaf types. The framework is easily extendable, not only with the spectral information of the Sentinel-2 bands, but with vegetation index derivatives, such as SR.

It was possible to use a good collection of acquisitions in spring, covering the complete foliage development, with a maximum range of 15 days between acquisitions. Introducing additional scenes after the foliage development would presumably not have supplied meaningful additional information. Due to cloud cover, no Sentinel-2 scenes could be derived in June and no more than one Sentinel-2 scene could be derived for July and August. It was thus not possible to estimate the power to discriminate the target classes in this temporal frame. Classification of the aerial imagery of 2020 confirms the imagery does not have the differentiation capability of Sentinel-2 for the target classes. Apart from the fact the data for the 2020 aerial imagery have been acquired over nine different dates, reaching into middle September, the acquisition of aerial imagery usually takes place in July and August and thus falls into the time period where it is harder to differentiate the target classes. Texture features were able to slightly improve the classification of CIR aerial imagery (+1.5%) but the importance scores confirm the infrared information of the CIR aerial imagery is the most important feature to separate the target classes. Using the texture parameters derived from the aerial imagery to the Sentinel-2-based SR time series, models also did not improve the classification accuracy. These results are in line with the findings of Hemmerling et al., stating that the texture measures did not improve the classification accuracy significantly [2].

Given that all Sentinel-2-based classifications (single-date and multi-date) achieved moderately high classification accuracies, the sensor’s capabilities are well fitted to extend a GEOBIA framework with continuous updates to the image object’s characteristics and is able to deliver updates to the object’s classification. This can be attributed to the sensor and sensor platform characteristics, with a spatial, temporal, and spectral resolution specifically developed as continuous support for forest ecosystem research and applications [46]. Although the classification on the single-date S8 model was not as reliable as the classification based on single-date S3, an investigation of class changes between these models can also deliver additional information about the development of the target objects. For example, comparing S3 to S8 classifications, areas of very recent damage or clear cuts in 2020 can be highlighted. In Figure 14, we see some examples, where a rule-based procedure based on the single-date classes is able to detect recent changes of the objects not detected by the SR time series model. This simple procedure to compare classifications of different dates has the potential to analyze changes regarding the target objects based on temporally consistent object geometries. Interestingly, the best-performing classification (SR time series) classified these recently cut or damaged areas as needleleaved. This calls for an important discussion about the temporal specifications inherent to the GEOBIA framework.

4.3. Temporal Inconsistencies

The CIR aerial imagery used in this study was acquired over a longer time period in 2020, starting from the end of July, and barely falling into the same time period of Sentinel-2 acquisitions. This temporal inconsistency leads to classification into one of the leaf type classes based on the SR-based model, whereas in the aerial imagery of 2020 these areas already show a cleared forest plot. Of the training areas derived from the OSM database, 22 have been reclassified to non-forest areas, where changes to the original OSM tag were detected in the aerial imagery of 2020. Training areas relabeled as non-forest might thus still have been covered with forest in the selected Sentinel-2 acquisitions of 2020, which cover the vegetation period until the beginning of August (Table 2). Even though the temporal gap between both data sources is only a few weeks (with a maximum of six weeks for some parts of the study area), significant disturbances between the acquisitions might not be part of the resulting classification, especially if the changes occurred at the end of the covered time series.

Some clearcuts visible in the VHR image are not covered by the Sentinel-2 time series as depicted in Figure 13 and Figure 14. The clearcut would certainly result in a sharp decrease in SR index values at the end of the time series, especially in the case of harvesting healthy forest or harvesting healthy needleleaved forest to prevent further spread of the bark-beetle. On one hand, it is possible this disturbance is not part of the time series at all. On the other hand, as was visible from the importance analysis of the SR time series model, it is possible the disturbance is indeed covered by the time series, but the low importance of the vegetation index for the later scenes carries into the classification result. For example, the SR index of S8 is not considered important by the SR time series model. Regarding the size of the image objects and the spectral features joined to those, it also has to be taken into account that the dynamic needs to cover a substantial proportion of the image object to change the spectral measures.

Furthermore, a larger temporal distance exists between the creation of the image objects, which are based on CIR aerial imagery of 2018 and the features used for classification, which are based on CIR aerial imagery and Sentinel-2 acquisitions of 2020. The decision to keep the image objects intact has been decided deliberately, since it ensures regular updates to the GEOBIA framework are easily implemented without the need to change the image object geometry in each update. There is room for discussion to draw up a suitable timeframe to initiate an update to the geometry of the image objects as depicted in Figure 13. The normal repetition cycle for forest inventories is around five years and a yearly or less frequent update might not be necessary. However, a frequent update would be mandatory if the framework is employed for stand-wise disturbance detection. Additionally, procedures on how to manage this change of geometry in an ongoing management and monitoring system are needed.

The same temporal inconsistency exists between the features used for classification (2020) and the reference map (2018). As this is a common and extensively debated topic in remote sensing, an investigation of where this leads to differences between map and reference classes is needed. It also shows that our approach is able to deliver timely updates. The open data access and the high temporal resolution of image data from ESA’s Copernicus program provide good opportunities for locally restricted updates of image objects.

4.4. Validation via Reference Map

The land use 2018 reference map was the most recent map in the open data portal of Luxembourg, and especially interesting due to its specifications. Making use of similar data sources, the reference map is well fitted for validation purposes of the derived classification. The accuracy assessment via the confusion matrix showed considerable overlap between the classes of broadleaved and needleleaved, whereas the non-forest class was subject to the lowest class-specific accuracies. According to the validation grid, 64% of the objects classified as non-forest belong to the class of broadleaved or needleleaved in the reference map. Figure 15 illustrates this setup. Although the needleleaved plot at the top centre is classified as non-forest in the SR time series classification, the area is classified as needleleaved in the reference map. The aerial imagery of 2020 in the background reveals our classification is accurate but overestimated by the geometry of the image object. Additionally, an area accurately classified as needleleaved is regarded as broadleaved in the reference map. This might be due to the plot geometry, too—the area belongs to the delineation of the broadleaved forest in the lower left. Ideally, this would be classified as mixed forest in the reference map.

Mixed forests were deliberately retained from the accuracy assessment since the class cannot be effectively modelled through the present GEOBIA framework. It was still possible to separate these areas into needleleaved and broadleaved at least in the cases where the image objects sufficiently delineated the separate classes. In addition to the previous example, the mixed area in the centre of the reference map was separated into needleleaved and broadleaved through the SR time series classification. Although the upper part of the mixed area shows good separation, we are doubtful about the lower part. There seem to be a few broadleaved trees; thus, it can be assumed the neighbouring agricultural field had some influence on the classification.

5. Conclusions

The extended GEOBIA framework for the stratification of leaf types uses single-date and multi-temporal Sentinel-2 imagery as input data for several classification setups. The implementation of a data source that can cover phenologically relevant time periods led to improvements of the stratification of leaf types and non-forest areas, which were an update to the specified target classes. The extended GEOBIA framework can be replicated and built-upon, since free and open-source software was used throughout the development. The whole processing chain can be followed if open data of the necessary spatial detail to delineate the image objects is available. As could be shown in the discussion, a whole array of analysis techniques can be derived from this setup, which is based on temporally consistent image objects.

Our contribution shows that plots of 0.05 ha MMU can be successfully classified based on 10 m Sentinel-2 features. Although the previous initial framework with classifications based on spectral bands of CIR aerial imagery and texture have shown overestimation of the needleleaved class due to shadow cast in open broadleaved areas [19], this problem has been largely mitigated by the classifications based on Sentinel-2 single-date, multi-temporal, and index time series setups. The classification results confirm a particularly important aspect. If it is not possible to include multi-temporal data in the classification of natural features, it is of tremendous importance to cover the time periods where the highest spectral difference between the target classes is present. We could prove that a single-date Sentinel-2 scene can reach classification accuracies comparable with the accuracies of multi-temporal classifications (n = 2, n = 3, n = 8), if the scene represents characteristic phenological differences between leaf types.

A remaining challenge, in consideration of the target classes and their spatial patterns, is the ability to handle mixed forests without apparent stand borders (Figure 2). Additional procedures, i.e., the integration of different scales in the segmentation, would be necessary in order to discern them. A procedure that has shown potential in a recent study by Oreti et al. uses decision rules based on segmentations at different scales [23].

The presented extension of the GEOBIA framework can produce regular updates to forest leaf type maps through the incorporation of multi-temporal Sentinel-2 imagery from the growing period of broadleaved forests. Although this is another step forward, the framework holds potential for more extensive classification problems and for intra-annual information retrieval (i.e., disturbances) on the target objects as was already touched in the discussion as well as the need to develop robust measures that can prompt a recalculation of the image objects. The presented framework should thus be evaluated on a more extensive classification problem (e.g., tree species classification, age classes, health status) or by fully incorporating the possibilities of the object-based framework in a decision-based system. Applicability and transferability of the framework into an operative forest management system is given, as the OSM mixed forest boundaries could be substituted by local forest management units in the segmentation procedure. This is an advantage of the presented approach, as the image objects do not extend over the predefined management units. This possibility of direct implementation into a well-known expert system can deliver the spatially explicit information that is often needed.