Data Quality of National Monitoring Schemes: Filling the Gap between Specialists and the General Public

Abstract

Worldwide, large-scale biodiversity monitoring schemes are developing and involve many non-specialist volunteers. If the opening of schemes to non-specialists allows for the gathering of huge amounts of data, their quality represents a controversial issue. In the framework of the French Garden Butterfly Observatory (FGBO), we studied non-specialist volunteer identification errors based on identifications provided during a one-shot experiment. With 3492 butterfly pictures sent by 554 non-specialist volunteers, we directly measured identification errors and misidentification rates for each butterfly species or species group targeted by the FGBO. The results showed that when non-specialist volunteers identified butterflies at the species level, identification errors (i.e., the misidentification rate) reached 20.9%. It was only 5.0% when FGBO species groups were used. This study provides novel insights into the trade-off between data quantity and quality provided by non-specialist volunteers and shows that if protocols, research questions and identification levels are adapted, participatory monitoring schemes relying on non-specialists represent a powerful and reliable tool to study common species at a large scale and on a long-term basis.

1. Introduction

Biodiversity monitoring schemes based on the involvement of volunteers in research programs are now being established in many countries (e.g., [1,2,3]). Volunteer involvement is widely used to reduce the cost of biodiversity monitoring at a large scale [4,5,6,7] and has the added benefit of increasing citizen awareness of scientific processes and environmental issues [8,9]. Biodiversity monitoring schemes are also useful to build scientifically sound decision-making tools (e.g., biodiversity indicators) for conservation, particularly at a large scale [10,11,12]. They allow for the assessment of spatial and temporal trends of biological diversity with the collection of a large amount of data [5,13].

However, volunteer-based monitoring may be perceived as simplistic by the scientific community and prone to higher biases compared to professional monitoring schemes [14,15,16,17]. In some cases, data gathered by volunteers have been excluded [18]. Thus, the quality of data provided by volunteers is subject to debate [4,19]. It has been argued that volunteers are not sufficiently trained to prevent both false positive and false negative data, especially for species identifications [20,21]. This feeling is reinforced by the fact that papers based on data collected by volunteers usually do not deal with training or compensation for volunteer error (e.g., [20]). Different solutions have been proposed, such as the selection of volunteers and training (e.g., [21]), the use of skilled volunteers only (e.g., [22]) or data validation and calibration (e.g., [20,23,24,25]). However, with the increase in large monitoring schemes based on volunteers who may come from the general public [26], it is of prime importance that the quality of data provided by such schemes is reliable for scientific analysis [27,28,29,30,31,32].

Among insect taxa, butterflies are generally advocated as good indicators of many aspects of environmental changes at the landscape level [33,34,35]. They are also well documented, easy to recognize and popular with the general public [35]. Thus, specific volunteer monitoring schemes based on butterflies have been launched all over Europe [22]. The French Garden Butterfly Observatory (FGBO) began in 2006. This national citizen science program based on a standardized protocol involves thousands of volunteers from the general public each year. No minimum identification skills are requested of volunteers for them to be allowed to take part in the scheme.

In this study, we tested volunteer identification errors using data produced via a side participatory program involving FGBO volunteers [36], which allowed us to directly measure volunteer error rates in butterfly identifications. Moreover, since the FGBO scheme relies on non-specialist volunteers, in addition to species that are easy to identify, it uses species groups. These groups represent closely related and look-alike species that cannot be a priori discriminated from each other by non-specialists (e.g., the Pieris species). We thus also estimated the relevance of these species’ groups in FGBO to investigate the trade-off between precision and data quality when non-expert volunteers are involved in participatory schemes.

2. Materials and Methods

We used data from a specific participative survey realized in the framework of the FGBO, called “flowers for butterflies” [36]. The initial purpose of this experiment was to study butterfly diet preferences. A collection of pictures of butterflies on flowers made by FGBO volunteers all over France was produced between March and October 2008 (Figure 1). Volunteers were asked to identify butterfly species or species groups and provide their pictures with identifications. The identifications provided by participants were used to estimate misidentification rates by comparison with identifications made by an expert (Benjamin Bergerot, a French university academic associate professor). Based on the expert identifications, only pictures of species monitored in the framework of the FGBO were selected (i.e., species belonging to the 28 common species or species groups,

Table 1. Butterfly species list at the species level and the French Garden Butterfly Observatory (FGBO) level. Butterfly species were ordered by family and grouped by FGBO level (Ns: butterfly number identified by the specialist in the data collection, Nv: butterfly number identified by volunteers in the data collection, Nm: number of matching identifications between the specialist and volunteers, MR: misidentification rate in %, UR: undetection rate in %).

		French Garden Butterfly Observatory Level						Species Level
Family	Scientific Name	FGBO Groups	Ns	Nm	Nv	MR	UR	Ns	Nm	Nv	MR	UR
Heliconiidae	Argynnis paphia (Linnaeus, 1758)	Silver-washed fritillary	87	74	74	0.0	14.9	87	74	74	0.0	14.9
Hesperiidae	Ochlodes sylvanus (Esper, 1777)	Orange skippers	23	9	9	0.0	60.9	4	0	0	NA	100.0
	Thymelicus acteon (Rottemburg, 1775)							13	2	2	0.0	84.6
	Thymelicus sylvestris (Poda, 1761)							6	3	7	57.1	50.0
	Carcharodus alceae (Esper, 1780)	Speckled skippers	12	4	4	0.0	66.7	7	4	4	0.0	42.9
	Pyrgus sp.							5	0	0	NA	100.0
Limenitidae	Limenitis camilla (Linnaeus, 1764)	Admirals	27	18	19	5.3	33.3	1	0	3	100.0	100.0
	Limenitis reducta (Staudinger, 1901)							26	15	16	6.3	42.3
Lycaenidae	Cacyreus marshalli (Butler, 1898)	Geranium bronze	72	69	70	1.4	4.2	72	69	70	1.4	4.2
	Callophrys rubi (Linnaeus, 1758)	Green hairstreak	2	2	3	33.3	0	2	2	3	33.3	0
	Celastrina argiolus (Linnaeus, 1758)	Blues	264	234	244	4.1	11.4	27	10	11	9.1	63.0
	Cupido alcetas (Hoffmannsegg, 1804)							3	1	1	0.0	66.7
	Cupido argiades (Pallas, 1771)							5	1	1	0.0	80.0
	Cupido minimus (Fuessly, 1775)							1	1	1	0.0	0
	Lampides boeticus (Linnaeus, 1767)							4	3	4	25.0	25.0
	Leptotes pirithous (Linnaeus, 1767)							12	9	9	0.0	25.0
	Maculinea arion (Linnaeus, 1758)							2	1	1	0.0	50.0
	Plebejus argyrognomon (Bergsträsser, 1779)							1	1	1	0.0	0
	Polyommatinae (subfamily)							79	69	186	62.9	12.7
	Polyommatus icarus (Rottemburg, 1775)							125	24	24	0.0	80.8
	Polyommatus bellargus (Rottemburg, 1775)							5	1	5	80.0	80.0
	Lycaena phlaeas (Linnaeus, 1761)	Coppers	137	115	117	1.7	16.1	118	106	109	2.8	10.2
	Lycaena tityrus (Poda, 1761)							17	7	7	0.0	58.8
	Lycaena virgaureae (Linnaeus, 1758)							2	0	1	100.0	100.0
Nymphalidae	Aglais urticae (Linnaeus, 1758)	Small tortoiseshell	37	36	38	5.3	2.7	37	36	38	5.3	2.7
	Inachis io (Linnaeus, 1758)	European peacock	302	292	293	0.3	3.3	302	292	293	0.3	3.3
	Polygonia c-album (Linnaeus, 1758)	Comma	99	98	98	0.0	1.0	99	98	98	0.0	1.0
	Vanessa atalanta (Linnaeus, 1758)	Red admiral	185	182	183	0.5	1.6	185	182	183	0.5	1.6
	Vanessa cardui (Linnaeus, 1758)	Painted lady	83	76	79	3.8	8.4	83	76	79	3.8	8.4
Papilionidae	Iphiclides podalirius (Linnaeus, 1758)	Scarce swallowtail	106	95	97	2.1	10.4	106	95	97	2.1	10.4
	Papilio machaon Linnaeus, 1758	Common yellow swallowtail	92	84	89	5.6	8.7	92	84	89	5.6	8.7
Pieridae	Anthocharis cardamines (Linnaeus, 1758)	Orange tips	10	8	9	11.1	20.0	9	7	8	12.5	22.2
	Anthocharis euphenoides (Staudinger, 1869)							1	1	1	0.0	0
	Colias crocea (Fourcroy, 1785)	Clouded yellows	57	50	55	9.1	12.3	34	28	49	42.9	17.7
	Colias hyale (Linnaeus, 1758)							23	2	6	66.7	91.3
	Gonepteryx cleopatra (Linnaeus, 1767)	Brimstones	232	217	221	1.8	6.5	9	7	7	0.0	22.2
	Gonepteryx rhamni (Linnaeus, 1758)							223	209	214	2.3	6.3
	Leptidea sp.	Whites	514	497	504	1.4	3.3	8	7	8	12.5	12.5
	Pieris brassicae (Linnaeus, 1758)							137	23	37	37.8	83.2
	Pieris napi (Linnaeus, 1758)							126	12	14	14.3	90.5
	Pieris rapae (Linnaeus, 1758)							186	15	29	48.3	91.9
	Pieris sp.							57	56	416	86.5	1.7
	Aporia crataegi (Linnaeus, 1758)	Black-veined white	11	11	21	47.6	0.0	11	11	21	47.6	0
Satyridae	Brintesia circe (Fabricius, 1775)	Great banded grayling	37	34	39	12.8	8.1	37	34	39	12.8	8.1
	Coenonympha pamphilus (Linnaeus, 1758)	Small heath	41	34	71	52.1	17.1	41	34	71	52.1	17.1
	Lasiommata megera (Linnaeus, 1767)	Wall browns	89	76	89	14.6	14.6	63	49	70	30.0	22.2
	Lasiommata maera (Linnaeus, 1758)							26	13	19	31.6	50.0
	Maniola jurtina (Linnaeus, 1758)	Meadow brown	384	320	347	7.8	16.7	384	320	347	7.8	16.7
	Melanargia galathea (Linnaeus, 1758)	Marbled whites	112	106	107	0.9	5.4	111	106	107	0.9	4.5
	Melanargia lachesis (Hübner, 1790)							1	0	0	NA	100.0
	Pararge aegeria (Linnaeus, 1758)	Speckled wood	86	74	81	8.6	14.0	86	74	81	8.6	14.0
	Pyronia tithonus (Linnaeus, 1771)	Gatekeepers	277	227	253	10.3	18.1	270	220	246	10.6	18.5
	Pyronia bathseba (Fabricius, 1793)							7	6	7	14.3	14.3
Sphingidae	Macroglossum stellatarum (Linnaeus, 1758)	Hummingbird hawk-moth	208	199	199	0.0	4.3	208	199	199	0.0	4.3
	Total counts		3586	3241	3413	5.0	9.6	3586	2699	3413	20.9	24.7

).

We defined two identification levels: the “species level” and the “FGBO level”. At the species level, we investigated whether specific identifications made by volunteers matched those made by the expert. Based on the pictures, some butterflies could not be identified at the species level by the specialist, and four groups were created (which are different from the FGBO groups): Pieris sp., Leptidea sp., Pyrgus sp. and Polyommatinae (Table 1). When volunteers identified a butterfly at the species level while it was in theory not possible to identify that butterfly in a picture, we considered the identification as a misidentification.

At the FGBO level, butterflies were identified following the FGBO guidelines, i.e., some species such as Vanessa atalanta are identified at the species level (low misidentification risk), whereas others are attributed to species groups (e.g., blue lycaenids, white pierids). In the latter case, misidentifications at the species level inside FGBO groups (e.g., whites between Pieris rapae and P. napi, coppers between Lycaena tityrus and L. phlaeas) were considered good identifications.

At both the species and the FGBO levels, we calculated the volunteer misidentification rate (MR) for species x (in %) with the following Equation (1):

$$\mathrm{Misidentification}\mathrm{rate}\left(\mathrm{MRx}\right)=100-\left({\displaystyle \frac{\mathrm{Nmx}}{\mathrm{Nvx}}}\times 100\right)$$

(1)

with Nmx being the number of correct identifications by volunteers of species x as indicated by the specialist and Nvx being the number of pictures identified by volunteers as species x, correctly or not.

We also calculated the volunteer undetection rate (UR) for species x (in %) with the following Equation (2):

$$\mathrm{Undetection}\mathrm{Rate}\left(\mathrm{URx}\right)={\displaystyle \frac{\mathrm{Nsx}-\mathrm{Nmx}}{\mathrm{Nsx}}}\times 100$$

(2)

with Nsx being the total number of photos of species x identified by the specialist and Nmx being the number of correct identifications by volunteers of species x, as indicated by the specialist.

These two indices are complementary and allow for distinction between species that are well identified but difficult to detect (low MR and high UR) and species that are well detected but regularly misidentified.

All correlation tests were conducted in R 4.2.3 (www.R-project.org, accessed on 16 January 2024).

3. Results

In 2008, 3619 volunteers sent data to the FGBO and 554 (i.e., 15.3% of the 2008 active participants) took part in the “flowers for butterflies” experiment (Figure 1), a one-shot participatory science experiment to analyze butterfly feeding preferences, sending 3492 pictures. A total of 3586 butterflies belonging to the 28 common species or FGBO species groups were identified by the specialist in the data collection. It included 3437 butterflies identified at the species level (95.8% of the data). A total of 149 butterflies were unidentifiable at the species level (4.2%) and were identified at the group level (Table 1).

Volunteers identified 3413 butterflies which, according to their identifications, belonged to the FGBO. For the whole dataset, at the species level, 20.9% of the pictures were ill identified (2699 pictures were well identified by volunteers), while this error rate reached 5.0% (3241 butterflies were well identified by volunteers) when the identification was conducted with FGBO species or species groups. The undetection rates reached 24.7% and 9.6% for species and FGBO levels, respectively (Table 1).

At the species level, misidentification rates ranged from 0% (Polygonia c-album, for instance) to 100% (for two species with fewer than three pictures in the whole dataset). Considering that species were represented in more than 10 pictures, the highest misidentification rate was 66.7% for Colias hyale. For species represented in more than 100 pictures, the three Pieris spp. (P. napi, P. brassicae and P. rapae) had the highest misidentification rates (14.3%, 37.8% and 48.3%, respectively), followed by Pyronia tithonus (10.6%) and Maniola jurtina (7.8%). The three Pieris sp. had undetection rates above 80%, while it was 18.5% for Pyronia tithonus and 16.7% for Maniola jurtina. The other eight species with more than 100 pictures had a misidentification rate below 3%. Unexpectedly, Polyommatus icarus had a misidentification rate of 0%, but this was counterbalanced by the fact that it had an undetection rate of 80.8%; this species is poorly detected but always well identified when it is detected.

At the FGBO level, MRs ranged from 0% (both skipper groups, Argynnis paphia, Polygonia c-album and Macroglossum stellatarum) to 52.1% for Coenonympha pamphilus. A total of 15 out of 28 species/species groups had misidentification rates below 5%. Volunteer undetection rates (URs) were the highest for speckled and orange skippers (66.7% and 60.9%, respectively) and admirals (33.3%). URs and MRs were significantly correlated at the species level (Spearman correlation test, r = 0.3, p < 0.05, Figure 2A), but not at the FGBO level (Spearman correlation test, r = 0.07, p = 0.72, Figure 2B).

4. Discussion

Unlike many European butterfly monitoring schemes [22], identification skills at the species level are not required in the FGBO. However, based on a large sample of FGBO volunteers (15.3% of all active observers), we showed that identification errors (MRs) made during the “flowers for butterflies” project reached 5.0% at the FGBO level (species easy to identify and groups of look-alike species). By comparing misidentification rates at the species and FGBO species group levels, we also illustrated the trade-off between data quality and the desired level of precision: at the species level, the misidentification rate reached 20.9%. The use of FGBO species groups limits the result analysis (e.g., the group “blues” includes more than 50 species in France with a wide range of ecological needs, which hampers result interpretation at the species level). The use of species groups increases data reliability and allows for analyses when species identification is not required (e.g., when species-specific analyses or species richness issues may not be possible). However, opening the schemes to non-specialists allows for the gathering of huge amounts of data that can be used for large temporal and spatial-scale studies (e.g., [37]). This is particularly true for common whites, where misidentification rates were high at the species level but only 1.4% at the FGBO level, with an undetection rate of 3.3%, making the data collected on this group reliable.

Moreover, even if there is no clear-cut line between UR and MR as expected [38,39,40], the use of species groups in such schemes is necessary to avoid volunteer frustration, since some of the most commonly encountered butterflies in gardens (i.e., Pieridae, Lycaenidae) are difficult to identify (high MR value, low UR value). This social aspect is not to be underestimated because it favors volunteer involvement [41]. We consider that the bias related to the fear of misidentification was low because, in the initial purpose of this experiment [36], volunteers were not obliged to provide identifications.

Some methodological caveats should be pointed out in the conclusions based on the “flowers for butterflies” experiment to study FGBO error misidentifications. First, as the “flowers for butterflies” experiment was a one-shot additional survey in the framework of the FGBO, it may have attracted mostly highly motivated volunteers (with a mean number of 6.3 pictures sent by each volunteer) who may have been more skilled than average volunteers [42]. Perhaps more importantly, butterflies were identified in photos. It implies that volunteers had time to identify butterflies, allowing them to use documents to determine species names. However, compared to FGBO protocol, based on field observations, the misidentification rate could be either underestimated since it may be more difficult to identify a butterfly in the field than in a picture, or overestimated since volunteers could hesitate between a larger range of species when they are confronted with guidebook plates.

Interestingly, identification in pictures provided useful information on the species that were mostly affected by misidentifications, such as Aporia crataegi, Maniola jurtina and Coenonympha pamphilus. It also pointed out which species are often undetected. For example, orange and speckled skippers were widely undetected but well identified, whereas the black-veined white (Aporia crataegi) was always detected but often misidentified. These results may help researchers target specific communication campaigns in order to improve volunteer skills and monitoring scheme reliability.

URs and MRs are not specific to butterflies and could easily be implemented in many other monitoring programs. Many monitoring programs are subject to the same bias. Data quality could be improved by incorporating metrics that highlight the specific characteristics of species and group identifications, enabling fine characterization of fluctuations in UR and MR at the species or species group level. One strategy could be to identify, on the basis of MR and UR, the level of identification that optimizes data quality (e.g., for plants [43] or insects [44]). Our results provide novel insights into the trade-off between data quantity and quality provided by volunteers in monitoring schemes, and future research should better examine misidentification rates directly in the field.