BAMFORESTS: Bamberg Benchmark Forest Dataset of Individual Tree Crowns in Very-High-Resolution UAV Images

Abstract

The anthropogenic climate crisis results in the gradual loss of tree species in locations where they were previously able to grow. This leads to increasing workloads and requirements for foresters and arborists as they are forced to restructure their forests and city parks. The advancements in computer vision (CV)—especially in supervised deep learning (DL)—can help cope with these new tasks. However, they rely on large, carefully annotated datasets to produce good and generalizable models. This paper presents BAMFORESTS: a dataset with 27,160 individually delineated tree crowns in 105 ha of very-high-resolution UAV imagery gathered with two different sensors from two drones. BAMFORESTS covers four areas of coniferous, mixed, and deciduous forests and city parks. The labels contain instance segmentations of individual trees, and the proposed splits are balanced by tree species and vitality. Furthermore, the dataset contains the corrected digital surface model (DSM), representing tree heights. BAMFORESTS is annotated in the COCO format and is especially suited for training deep neural networks (DNNs) to solve instance segmentation tasks. BAMFORESTS was created in the BaKIM project and is freely available under the CC BY 4.0 license.

1. Introduction

The anthropogenic climate crisis (ACC) is probably humanity’s most urgent and most significant problem to solve and adapt to in the coming years. Just as the years 2017–2022, 2023 was, according to the German Umweltbundesamt, the warmest year in Germany and worldwide [1]. This rapid increase in temperature puts much pressure on ecosystems worldwide, including forests and city trees [2,3,4,5]. As forests are natural carbon sinks, this is a self-reinforcing problem [6]. The effects are less resilient trees and increased pests and secondary pests such as bark beetles and mistletoes [7,8]. Especially in forests that are heavily used for forestry and, therefore, often structured as tree monocultures, this can lead to the loss of all trees in large areas [9].

To mitigate the effects of ACC on forests and city trees, arborists and foresters need to restructure their forests, city parks, and solitary city trees. Experience in the BaKIM (BaKIM is a cooperation project of the Cognitive Systems Chair of the University of Bamberg and the City of Bamberg. It is funded with EUR 450,000 by the Bavarian Ministry of Digital Affairs) project shows that, to accomplish this, German foresters and arborists rely on incomplete and often old data, which do not allow for a timely and regular evaluation. Furthermore, research shows that this is not a problem of Germany alone, but a European and probably worldwide one [10]. BaKIM shows that, to support foresters and arborists in their work, current data on the position of individual trees, tree species, tree vitality, and infestation with pests are needed.

Reliably detecting single trees is the underlying task, which helps all other previously mentioned tasks, making the information traceable on the single-tree level. Furthermore, foresters and arborists in the BaKIM project state that this information is also important and helpful for debating with local decision makers and politicians when it comes to the funding of city trees and forest areas. The rise of cheap UAV technology and deep learning in the past few years has built the base for solving this task of single tree detection. Overview papers such as Kattenborn et al. [11] showed that very-high-resolution UAV data with a ground sampling distance (GSD) of below 2 cm and deep neural networks (DNNs) from the computer vision domain yield good results. For example, Gan et al. [12] compared the widely used Python deep learning packages Detectree2 [13] and Deepforest [14] on different very-high-resolution GSDs. Xi et al. [15] tested different dimensionality reductions to overcome the drawbacks of multispectral images for ITCD approaches, which can yield better results than just RGB-imagery.

One remaining problem with supervised DNNs is the lack of high-quality training data. Annotating very-high-resolution UAV data on a single-tree level is time-consuming and expensive. Therefore, only a few small, openly accessible datasets exist. The Million Trees project [16] initiated by Weinstein et al. [17] tries to solve this by collecting and combining as many available datasets as possible. Nevertheless, this is an ongoing effort which, according to Ben Weinstein, will take place until fall 2025. To the best of our knowledge, none of the existing datasets come with a proposed dataset split to help compare the results of different methods and DNN architectures created by different researchers.

With this work, we publish BAMFORESTS: relative to other datasets with ITCD labels, it is a large benchmark dataset of very-high-resolution UAV imagery and individual delineated tree crowns (ITCD). To make it a benchmark dataset, we specify a fixed split of training, validation, and testing data. Furthermore, for ease of use, we publish the data in the COCO format in two different tile sizes so that researchers can easily test their methods and compare them with other results. BAMFORESTS will become part of the Million Trees benchmark dataset.

2. Theoretical Background

Deep learning in the form of deep convolutional neural networks (DCNNs) brought great advancement to different tasks in computer vision. Image classification, object detection, semantic segmentation, and instance segmentation are all tasks that extract information from image data [18,19]. Kattenborn et al. [11] provided a detailed overview on the following aspects of computer vision in the forest domain. While image classification assigns a label to a whole image, the latter tasks generate spatial information in differing detail. Object detection produces bounding boxes for multiple objects in an image. These bounding boxes are represented by pixel coordinates and describe the outer bounds of detected objects. As multiple objects can be separately detected, these objects can be counted. For example, with a working object detection, it is possible to count trees in an image but not possible to obtain the exact area covered by the tree crowns. Semantic segmentation, on the other hand, is not capable of differentiating between instances as it predicts a class for each pixel. If overlapping instances of the same class exist, semantic segmentation does not separate them, therefore making it impossible to count the instances. The improvement over object detection is that additional spatial information on the class occurrence is given, as the class of each pixel is predicted. Assuming correct classification, semantic segmentation for example enables the calculation of the area-related share of different tree species. Instance segmentation basically combines the two aforementioned tasks. It classifies each pixel in an image, but at the same time keeps information of single instances. If instances of the same class overlap, they can still be counted. Therefore, instance segmentation enables us to count trees, obtain their area-related share, and obtain their exact tree crown dimensions.

Image classification, due to its simple input–output structure of an image as the input and a single label as the output, was the first problem that greatly benefited from a large benchmark dataset, more powerful computer hardware, and DCNNs in 2012 [20,21]. Huge performance leaps in the more complex tasks of object detection, semantic segmentation, and instance segmentation followed in the years after with the release of the COCO benchmark dataset [22].

The following subsections will summarize the application of such DCNNs in the domain of forests and trees in aerial images and deal with the state of benchmark datasets in this domain.

2.1. Supervised Deep Learning on Forest Datasets

DCNNs are the state-of-the-art method for different computer vision tasks and are also applied in the forest domain. Recent publications show that, for tree detection and individual tree crown delineation (ITCD), technically an instance segmentation, DCNNs are the best-performing models. The widespread, successful use of these models in this domain is summarized by different overview papers and systematical literature reviews, such as Kattenborn et al. [11] and Zhao et al. [23]. Furthermore, Fan et al. [24] showed that deep learning approaches outperform statistical segmentation approaches such as marker-controlled watershed transformation (MCWST).

For example, Weinstein et al. [17] implemented DeepForest, a RetinaNet pre-trained on several datasets, capable of performing the object detection of trees, even in closed canopies. Schiefer et al. [25] used the U-Net architecture to map tree species via semantic segmentation. Ball et al. [13] developed Detectree2, an instance segmentation DCNN based on Facebook’s Detectron2 architecture. While Schiefer et al. [25] published their dataset and code, Deepforest and Detectree2 are publicly available Python packages, including pre-trained weights and pre- and post-processing steps. Therefore, they enable researchers without profound knowledge in programming and machine learning to utilize DCNNs for their aerial image datasets.

While all of these approaches and applied methods show that DCNNs are a very good solution to the task of tree detection and segmentation, the performance in the forest domain is still far behind when compared with the performance of the same methods on datasets like ImageNet and COCO, reaching over 90% accuracy in image classification and AP.50 scores of over 70% for object detection [26,27].

2.2. Forest Datasets

Benchmark datasets in computer vision, such as the COCO dataset or ImageNet, consist of hundreds of thousands or even millions of images and millions of annotations [22,28]. Compared with these, datasets in the forest domain are quite small, covering only tens of thousands of instances in very-high-resolution datasets with ITCD annotations, and there are at most a few hundred thousand instances in datasets with lower resolution and lower-quality labels like point annotations. Datasets in the forest domain can be divided into the following categories:

• Label completeness:

  –

  All tree instances are labeled;

  –

  Only some tree instances are labeled;

• Ground sampling distance (GSD):

  –

  Low resolution (satellite imagery): >1.2 m per pixel;

  –

  High resolution (airplane): 40 cm–6 cm per pixel;

  –

  Very-high resolution (UAVs): <5 cm per pixel;

• Label type:

  –

  Points: the center of each tree crown is labeled;

  –

  Bounding boxes: the outer extents of each tree crown are labeled;

  –

  Polygons:

  ∗

  Semantic segmentation: tree species are labeled on a pixel level;

  ∗

  Instance segmentation: each tree crown is delineated on a pixel level.

Furthermore, less clear categories such as forest density, geo-location, forest type, species distribution, and vegetation phase as well as weather conditions in the weeks before data acquisition should influence the transferability of models trained on a dataset, as research in land cover detection from satellite data has suggested [29].

The Million Trees project [16] aims to collect as many existing datasets covering all above-mentioned categories and creating a combined benchmark dataset of at least one million labeled trees. As BAMFORESTS falls into the category of a very-high-resolution datasets with complete polygon labels of single trees, we summarize similar datasets in

Table 1. Datasets in the forest domain similar to BAMFORESTS.

	BAMFORESTS	FORTRESS [30]	Quebec Trees Dataset [31,32]	SiDroForest [33]	Name NA [34,35]
GSD	1.61 cm–1.81 cm	<1.35 cm	1.81 cm–2.02 cm	3 cm	1.70 cm–2.00 cm
Labeled Area	105 ha	47 ha	44 ha	13.25 ha	7ha
N of Labels	27,160	–	22,933	872 (19,342 *)	2547
Label Type	Polygons	Polygons	Polygons	Polygons	Polygons
Segmentation Type	Instance segmentation	Semantic segmentation	Instance segmentation	Instance segmentation	Instance segmentation
Label Completeness	Yes	Yes	Yes	No	Yes
Acquisition Period	Jul 2022 – Aug 2022	Mar 2017 – Sep 2019	May 2021 – Oct 2021	Jul 2018 – Aug 2018	Apr 2021 – Jun 2021
Region	Bavaria, Germany	Baden- Württemberg, Germany	Quebec, Canada	Yakutia and Chukotka, Siberia	Northern Territory, Australia

* Shapes are automatically extracted and of differing quality.

.

Cloutier et al. [32] recently published their dataset with 22,933 ITCs and image data from seven consecutive monthly flights. Their dataset is taken from two AOIs. One main AOI is used for training and validation, and a smaller second AOI, separated by a small body of water, serves as the test set. After BAMFORESTS, it is the second largest dataset freely available to the knowledge of the authors. Schiefer et al. [30] published a dataset for semantic segmentation, where not single trees, but patches of the same tree species, were labeled. Therefore, the number of ITCs is not known. Considering that FORTRESS consists of 47 ha, it is approximately half as big as BAMFORESTS. Due to the labels for semantic segmentation, it cannot be used to train and validate ITCD methods. Kruse et al. [33] made their dataset SiDroForest also publicly available. While it consists of a total of 19,342 shapes, only 872 of these are labeled by hand. The rest is generated automatically and is, therefore, of lower quality and probably incomplete. Jansen et al. [34] published a small unnamed dataset consisting of 2547 ITCs. Due to the small size of the dataset, it is, on its own, not suitable for the training of DCNNs.

3. Dataset

BAMFORESTS data come from four different AOIs, which are shown in Figure 1. All AOIs are located within a radius of 20 km in and around Bamberg, and the aerial surveys were carried out as part of the BaKIM project [36]. Figure 2 shows a detailed example view of each AOI. Figure 1 and Figure 2 are scaled equally so that the relative size of the AOIs and trees can be seen.

3.1. AOIs

The Hain-AOI is the cities forest-like park with a size of about 61 ha, and it is located south-east of the city center. As a park, it is heavily frequented, and arborists take a lot of measures to keep trees and reduce hazards of falling branches near paths. Furthermore, the Hain has the highest diversity of tree species because of its main functions as a recreational area and special area of conservation (SAC). It mainly consists of deciduous trees and, the resulting orthomosaic has a GSD of 1.82 cm.

The Stadtwald-AOI covers an area of 152 ha and is located about 5 km south-east of Bamberg’s city center. On the one hand, it is a managed forest, and on the other hand, it is a drinking water reservoir for parts of Bamberg and an SAC. In comparison with the city park, no measurements to keep single trees are taken, but if necessary, trees are removed and the timber is sold. It is a mainly coniferous forest, but in recent decades, second and third tree layers of deciduous trees were planted beneath the older coniferous trees. The resulting orthomosaic has a GSD of 1.70 cm.

The Tretzendorf-1-AOI and Tretzendorf-2-AOI share similar characteristics. The first one covers an area of 65 ha and the latter one an area of 47 ha. Both are located about 20 km west to north-west of Bamberg and are SACs. Because coniferous as well as deciduous trees grow in the Tretzendorf-AOIs, they can be described as managed mixed forests, and the resulting orthomosaics have a GSD of 1.61 cm and 1.79 cm.

3.2. UAVs and Sensors

Two different UAVs were used for data acquisition. A small DJI Phantom 4 (DJI-Shenzhen, China) with a 1” CMOS 20MP sensor and 84° angle of view (AOV), acquired images over the Hain region near the city center. For the forest areas, a larger Quantum Systems Trinity F90+ (Quantum Systems-Gilching, Germany) fixed-wing UAV with a Sony RX1 RII (Sony-Tokyo, Japan 42.4 MP sensor and 63° AOV was used. Both sensors are RGB sensors, and the resulting orthomosaics contain a red, green, and blue channel.

3.3. Data Acquisition

The data acquisition of all images in BAMFORESTS took place in the summer of 2022. Both Tretzendorf AOIs were captured on the fifth of July, the Stadtwald AOI was captured on the sixth of July, and the Hain AOI was captured on the third of August. While the flights of the Tretzendorf and Stadtwald AOIs were carried out at midday, due to legal reasons the flights of the Hain AOI took place after sunrise in the morning until early midday.

In all flights, we aimed for a forward overlap and sidelap of 80%. As the Trinity F90+ flies at fixed speeds, we compensated the lower forward overlap of about 65% with a higher sidelap of 85–90%. With the Trinity F90+, we always flew 120 m above ground level (AGL) and used a smoothed terrain follow mode in the hilly Tretzendorf-AOIs. With the DJI Phantom 4, we flew 85 m AGL to reach a similar ground sampling distance (GSD). The Hain-AOI was covered with a total of 3087 images, the Stadtwald-AOI with 3250 images, the Tretzendorf-1-AOI with 1577 images, and the Tretzendorf-2-AOI with 1229 images.

3.4. Orthomosaic Generation

For the generation of the orthomosaics, we chose the commercial software Agisoft Metashape (Version 1.8.4) after comparing the process and results to those of WebODM (Version 1.9.15). While WebODM is free and open-source, the setup is more difficult, and the flexibility in setting ground control points (GCPs) and influencing the resulting orthomosaic is lower. Nevertheless, the resulting orthomosaics can be described as of equal quality, as seen in Figure 3.

Agisoft Metashape uses a structure from motion algorithms to generate a georeferenced point cloud from the single overlapping images acquired with the UAVs. This tie point cloud is then cleaned with three statistical methods and a manual step where the remaining outliers below ground level and above the tree crowns are deleted by hand. For the statistical filtering, we found the best results were obtained by deleting about 20% of the tie points by reconstruction uncertainty, about 10% by reprojection error, and 10% by projection accuracy. After each filter step, we optimized the camera alignment. Based on the cleaned tie point cloud, we first generated the dense point cloud, then the digital surface model, and lastly the orthomosaic.

3.5. Labeling Process

The individual tree crowns were delineated and labeled by an experienced forester in 2023. To help the labeling process, they had additional access to orthomosaics taken in March and April for some of the AOIs. During and after labeling, they checked annotations on the sites. After this labeling, apprentices of the local forestry office randomly checked the labels onsite and corrected them if necessary. In addition to the tree species, the tree vitality was assessed from the imagery and categorized into vital, degrading, and dead. The tree species information and tree vitality information are used in this work to balance the dataset splits.

3.6. Dataset Metrics

BAMFORESTS consists of a total of 27,160 individually delineated tree crowns (ITC). Of those, 1978 are located in the 15 labeled hectares of the Hain-AOI, 15,473 are located in the 46 labeled hectares of the Stadtwald-AOI, 5900 are located in the 29 labeled hectares of the Tretzendorf-1-AOI, and 3809 are located in the 15 labeled hectares of the Tretzendorf-2-AOI. For an overview of the species and vitality distribution of the whole dataset, see Figure 4. With this publication, as we focus on the ITCD task, we only publish the ITC shapes and not all existing labels. Nevertheless, the labels are important for comparability and the splitting of the dataset.

3.7. Benchmark Dataset Split

Based on the AOI location, the tree species distribution, and vitality distribution, we define a fixed testing set for BAMFORESTS and propose a training set and validation set split. This ensures the comparability of different applied methods for ITCD and therefore helps the community in the forest domain to compare their models.

To ensure that spatial autocorrelation, as described by Kattenborn et al. [37], does not lead to an overestimation of the performance of methods, Test-Set-1 (15 ha, 14%) solely consists of the complete Hain-AOI. It is 5–20 km away from the other AOIs and is of severely different structure. Furthermore, it was captured with another UAV and sensor. All of these factors are good for estimating the real-world performance of methods. On the other hand, we want researchers to gain better insight into their methods performance on more alike data. This is why we define Test-Set-2 (14 ha, 13%), which is taken from the same AOIs as the training and validation set. To make sure that no redundant image information is in Test-Set-2, the split is based on complete hectare plots. This allows for overlapping tiles inside the hectare plots, thereby making the best use of the existing image data.

The validation set (Val-Set) consists of 16 ha (15%). Like Test-Set-2, it is taken from the AOIs Stadtwald and Tretzendorf. It follows the same split method based on the hectare plots. The training set (Train-Set) consists of the remaining 60 ha (57%) from the Stadtwald and Tretzendorf AOIs. Except for Test-Set-1, which is the Hain-AOI, all sets are as similar as possible, considering the distribution of classes. See

Table 2. Distribution of classes in the proposed split of sets in BAMFORESTS.

	Train-Set	Val-Set	Test-Set-2	Test-Set-1
N of shapes	17,212	4390	3580	1978
Pinus	36.23%	31.94%	27.54%	1.11%
Fagus	23.11%	20.71%	23.10%	23.96%
Quercus	23.30%	20.27%	22.43%	19.01%
Picea	5.53%	7.22%	9.11%	1.06%
Larix	2.70%	1.75%	2.21%	1.26%
Pseudotsuga	1.12%	1.80%	1.20%	0.15%
Abies	1.19%	1.12%	1.01%	0.00%
Other	6.83%	15.19%	13.41%	52.88%
Vital	86.34%	84.99%	83.97%	91.20%
Degrading	11.88%	12.35%	13.18%	8.49%
Dead	1.78%	2.67%	2.85%	0.30%

for the relative class share in each set.

3.8. COCO Label Generation

The COCO dataset is a commonly used dataset for deep learning-based object detection. BAMFORESTS is released in the same data format as the COCO dataset [38] so that it can be easily used with state-of-the-art deep learning models. The images in the dataset are divided into the mentioned 57.14% Train-Set, 15.24% Val-Set, 14.29% Test-Set-1, and 13.33% Test-Set-2. Corresponding annotations are available in their respective .GeoJSON files.

Based on the division of the hectare plots, the orthomosaics and associated annotations are cropped and then divided and collectively converted into the COCO format. There is a 50% overlap between the images both horizontally and vertically. To meet the needs of the various scenes and available GPU-VRAM, the BAMFORESTS dataset provides two different image sizes (1024 × 1024 pixels and 2048 × 2048 pixels). At the edges of the images, individual crowns are cropped. In the center of the image, the individual crowns are completely recorded, as shown in Figure 5.

3.9. COCO Label Stats

The data description of the BAMFORESTS dataset in the COCO format can be seen in

Table 3. The stats of the BAMFORESTS dataset in the COCO format.

Size	Number of Images				Number of Annotations
	Train	Val	Test-Set-1	Test-Set-2	Train	Val	Test-Set-1	Test-Set-2
1024	7521	2008	1675	1668	96,908	25,316	12,843	20,332
2048	1439	382	313	322	58,235	15,180	6720	12,321

. Due to image cropping and overlapping, a single tree crown may be captured several times in different images. Therefore, the number of annotated tree crowns is much higher than the actual number of individual trees in BAMFORESTS.

Test-Set-1 is based on the Hain plot, which is completely independent of the Train-Set and Val-Set. However, Test-Set-2 comes from the same forests as the Train-Set and Val-Set. As shown in Figure 6 and Table 2, the images and annotations from the dataset show noticeable differences, such as tree species, crown sizes, and planting densities between Test-Set-1 and Test-Set-2. The design of the test sets is based on the intention that the trained model will be tested in both similar and different situations. The different results of these two test sets in training models are valuable for researchers to better explore the robustness and transferability of training models.

4. Discussion

With BAMFORESTS, we do not present the first openly accessible dataset of individual delineated tree crowns but rather the first dataset in the forest domain that proposes a fixed dataset split; therefore, it serves as a benchmark dataset. Furthermore, with a total of 105 annotated hectares, a total of 27,160 trees, and a GSD of 1.6–1.8 cm, to the best of our knowledge, BAMFORESTS will be the single largest very-high-resolution dataset.

Nevertheless, BAMFORESTS is still a relatively small dataset for application in deep learning, but projects like Million Trees from Weinstein [16] will hopefully solve this in the future. Furthermore, the labels in BAMFORESTS are not of perfect quality. Especially in overlapping deciduous tree crowns, it is impossible to correctly label all trees. We solved this by controlling the labels with random samples, but it must be assumed that some trees are over- or under-segmented.

With two different test sets, we enable two different and important insights on method or model performance: Test-Set-2 gives good insights on the performance on similar data, covering applications where a model is trained or fine-tuned on labeled data of the same AOI and then applied to the remaining unlabeled area; meanwhile, Test-Set-1 gives good insights of the models real-world performance when it is applied to data from severely different AOIs and other sensors. With a ready-to-train dataset, split into tiles and made available with COCO annotation jsons, we make it as easy as possible for other researchers to train and test their methods on BAMFORESTS. Therefore, BAMFORESTS is an important and very much-needed contribution to deep learning in the forest domain and ITCD task.

5. Outlook and Future Work

To further improve BAMFORESTS, we are working on the following improvements: first, we will apply and update the labels from 2022 to orthomosaics from other sensors but the same AOIs in 2023; and second, as BaKIM and BAMFORESTS is open source, we plan to gather datasets and labels from other municipalities and cities in Germany. With these measures, we plan to reach 50,000 labeled ITCs in 2024. Additionally, we plan to publish the tree species and vitality information with BAMFORESTS-2.