Quality Assurance Framework Development Based on Six New ECV Data Products to Enhance User Confidence for Climate Applications

Abstract

Data from Earth observation (EO) satellites are increasingly used to monitor the environment, understand variability and change, inform evaluations of climate model forecasts, and manage natural resources. Policymakers are progressively relying on the information derived from these datasets to make decisions on mitigating and adapting to climate change. These decisions should be evidence based, which requires confidence in derived products, as well as the reference measurements used to calibrate, validate, or inform product development. In support of the European Union’s Earth Observation Programmes Copernicus Climate Change Service (C3S), the Quality Assurance for Essential Climate Variables (QA4ECV) project fulfilled a gap in the delivery of climate quality satellite-derived datasets, by prototyping a generic system for the implementation and evaluation of quality assurance (QA) measures for satellite-derived ECV climate data record products. The project demonstrated the QA system on six new long-term, climate quality ECV data records for surface albedo, leaf area index (LAI), fraction of absorbed photosynthetically active radiation (FAPAR), nitrogen dioxide (NO₂), formaldehyde (HCHO), and carbon monoxide (CO). The provision of standardised QA information provides data users with evidence-based confidence in the products and enables judgement on the fitness-for-purpose of various ECV data products and their specific applications.

1. Introduction

Climate change mitigation and adaptation have risen to the top of the agenda for many governments and international organisations [1]. In particular, the Paris Agreement from 2015 [2], aiming at strengthening the global response to the threat of climate change, is requesting systematic observation of the climate system. This has led to the establishment of space and research agency programs dedicated to increasing scientific understanding of the Earth system and its response to natural and/or human-induced changes. Key to this is the derivation of quantitative variables describing the chemical and physical properties of the biosphere using Earth observation (EO) satellites. More recently, emphasis has been placed on the development of climate data records (CDRs) of essential climate variables (ECVs) to characterise and monitor long-term (20+ years) trends and fluctuations. There are currently 54 ECVs defined by the Global Climate Observing System (GCOS) spanning the atmospheric, oceanic, and terrestrial domains [3,4]. These variables can be derived directly from in situ observations or indirectly from remote-sensing instruments flown on airborne or satellite platforms.

Fundamental to the scientific understanding of the Earth system, and its response to change and progress in policymaking, is a rigorous quantification of the accuracy and validity of these CDRs produced from EO satellites [3,4,5]. Although EO data and products are widely available, it is rare for them to have reliable and fully traceable information concerning their generation process and their quality. Setting achievable accuracy requirements that can be quantified with confidence is a challenging task that is dependent on the intended use of the data. Both the GCOS and the World Meteorological Organisation’s (WMO) observing systems capability analysis and review (OSCAR) tool publish and regularly review the quality requirements that satellite-derived variables should satisfy to support the work of the United Nations Framework Convention on Climate Change (UNFCCC), the Intergovernmental Panel on Climate Change (IPCC), and the WMO programmes. But these requirements do not specifically address specific applications, and their usage to quantify confidence in existing data products remains difficult. The situation is exacerbated, because different versions of the same ECV parameter are offered by various data providers. For example, an internet search (2018) revealed that upward of 30 satellite-derived leaf area index (LAI) and fraction of absorbed photosynthetically active radiation (FAPAR) data products are available for download, with even more for other ECVs such as sea surface temperature (~55), soil moisture (~62), and ozone and aerosols (~180). Further, most operationally derived ECV products adhere to different definitions and assumptions, which are not standardised among the international EO and ecological communities. These data products are created with independent or multiple sources of EO data using an array of retrieval algorithms and assumptions. They are also provided at different spatial and temporal resolutions over varying time periods.

Regulatory frameworks requiring EO data and product producers to be held accountable for ensuring the quality, accuracy, and validity of the information provided do not currently exist nor do the standards against which data quality should be monitored [6]. However, given the increasingly prominent role that quantitative EO products assume in climate monitoring applications, it is inevitable that the quality of these data will come under increasing scrutiny in the future [6]. There is a clear requirement for continued investment by data product providers for the following: (1) detailed assessment of EO data product quality including characterisation of their associated uncertainties; (2) provision of this product quality information in a standardised and comprehensible format to help data users navigate the wealth of ECV data products available to them and ensure they are applying the best data for their application; and (3) progression of internationally endorsed methods and good practices for this purpose.

2. Quality Assurance for Essential Climate Variables (QA4ECV)

Quality Assurance for Essential Climate Variables (QA4ECV) was a European Union (EU) Seventh Framework Programme (FP7) funded project (2014–2018) comprising of a partnership of key European scientists, data providers, and developers of future climate services, as well as a national metrology institute. The partners have significant roles in the following: international metrology; EO coordinating bodies such as the Committee on Earth Observation Satellites (CEOS) and the Coordination Group for Meteorological Satellites (CGMS); space programs such as the European Space Agency’s Climate Change Initiative (ESA CCI), the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT) operational provision of CDRs, and international satellite data validation activities; ground-based reference measurement networks such as the Network for the Detection of Atmospheric Composition Change (NDACC); and various related FP7 and H2020 projects such as CHARMe (CHARacterization of Metadata), GAIA-CLIM (Gap Analysis for Integrated Atmospheric ECV Climate Monitoring and FIDUCEO (Fidelity and uncertainty in climate data records from Earth Observations).

The project had three main objectives: (1) the development of a generic system for quality assurance of satellite data products that can be applied to many ECVs as a prototype of a sustainable service; (2) the generation of multi-decadal CDRs for atmospheric and terrestrial ECVs that are based on inter-satellite calibrated data and state-of-the-art retrievals and are traceable with uncertainty metrics; and (3) engagement with stakeholders, governance bodies, and end-users to demonstrate how trusted satellite data and a reliable means of interoperability can facilitate users in judging the fitness-for-purpose of the ECV CDRs. All project information is available online at: http://www.qa4ecv.eu/.

3. QA4ECV QA System

The purpose of developing and implementing a quality assurance (QA) system is twofold: (1) to provide ECV data product producers/science teams with the necessary resources (internationally endorsed tools, standards, methodologies) to develop products with embedded QA information that is presented in a clear and common format throughout the EO community; and (2) to provide data users with robust QA information as a means to quantitatively assess uncertainty and fitness-for-purpose of the data and derived products. The provision of such QA information demonstrates the traceability of products and simplifies comparisons between the same ECV produced by independent science teams. It also provides data users with evidence-based confidence in the products and enables judgement on the fitness-for-purpose of various ECV CDRs for their specific applications. Figure 1 outlines the QA system framework. Essentially, all the data and methodologies used to derive these data products (i.e., satellite, ancillary (climate/elevation models, etc.), and reference (in situ or model data used to calibrate and validate the algorithms)) should go through a quality checking process before being made available for climate data usage. The QA service components are highlighted in the grey box and will be described in more detail throughout the following sections. The utility of this QA system is demonstrated on the six QA4ECV data products, which are described in Section 4: albedo, leaf area index (LAI), fraction of absorbed photosynthetically active radiation (FAPAR), nitrogen dioxide (NO₂), formaldehyde (HCHO), and carbon monoxide (CO).

3.1. QA System Development

The QA4ECV QA system has been developed over a four-year period and is described in detail within the service specification document [7]. The initial requirements were scoped within a user requirements activity that employed a survey to gauge the current state of and need for quality assurance in satellite-derived data products [8]. The QA4ECV QA system framework aligns with IPCC guidelines [9] and builds upon other relevant and successful EU projects that consider EO data quality and provenance issues, for example CHARMe [10] and CORE-CLIMAX (COordinating Earth observation data validation for RE-analysis for CLIMAte ServiceS) (http://www.coreclimax.eu/), as well as international coordination bodies including the CEOS Working Group on Calibration and Validation (WGCV), the joint CEOS-CGMS Working Group on Climate, and GCOS, among others. In order to ensure content is current and captures relevant findings and user-endorsed methods from other initiatives, a review of 12 projects and initiatives dedicated to improving the quality of satellite-derived data streams was conducted [11]. The QA4ECV system collates the following types of quality indicators (QIs) associated with EO-derived ECV products: product details; algorithm traceability; quality flags; validation; uncertainties; and assessment against standards.

3.1.1. Product Details

Product details summarise the basic meta-information about the product, including for example: product DOI (Digital Object Identifier), spatial resolution, temporal resolution, spatial coverage, temporal coverage, in situ/reference datasets, and satellite and/or airborne datasets used. Documentation, which describes the data product, including the algorithm theoretical basis document (ATBD) and product user manual (PUM)/product user guide (PUG)/product specification document (PSD), is also captured here.

3.1.2. Algorithm Traceability Chain

A QA4ECV traceability chain is a diagrammatic and partly interactive representation of the processing steps taken to produce the final data product. It shows sub-processing chains and intermediate products/parameters, as well as provides a short description of each step and where to find more detail on the process implemented. The traceability chain aids a user in understanding the data production and the assumptions that are made during implementation. It also helps producers identify and understand potential sources of discrepancies between two similar data products produced by different methods or algorithms. The traceability chains for the six QA4ECV demonstrator products are shown in Section 4, and interactive versions can be found at: http://www.qa4ecv.eu/ecvs/.

3.1.3. Quality Flags

Quality flags (QFs) are provided as a product data layer and indicate quality information about the product at the ground pixel level. There are currently no standards specifying if and what QFs should be provided with a product, and therefore, the content can vary in complexity and detail of information for different data products. Commonly implemented flags have been recommended in the QA system to provide information such as the following: number of observations used in the calculation; snow/cloud cover; back-up algorithm implementation; fill-values utilised; pixel-based uncertainty estimates, and so on.

3.1.4. Validation and Inter-Comparison

Validation is the process of assessing by independent means the quality of the data products derived from the system outputs [12]. The content of the validation section is dependent on the ECV domain chosen; however, all data producers are asked to provide the validation report(s), comparisons with independent reference data, and comparisons with other satellite-derived ECV datasets (i.e., inter-comparisons). Given the ambiguity associated with determining if a global data product is “validated”, a hierarchical approach to classify validation stages was adopted by CEOS through a consensus of the land product validation (LPV) community [13]. While for the atmospheric domain, the data producer is asked to provide information about the validation protocol steps, to which the product has been subjected (i.e., those developed in the context of CEOS initiatives and ESA projects and tailored in the prototype QA/Validation Service for Atmospheric ECV Precursors—Detailed Processing Model [14]. Building on this prototype, the QA4ECV Atmosphere ECV Validation Server, available online at https://qa4ecv-dev.stcorp.nl, was implemented to validate all QA4ECV atmospheric ECV datasets and transferred recently to the Sentinel-5p Mission Performance Centre for the operational validation service for TROPOMI (the TROPOspheric Monitoring Instrument) atmospheric data products (see http://www.tropomi.eu).

3.1.5. Uncertainties

The QA system captures details about the uncertainty information and derivation of the uncertainty estimates associated with the ECV data products. This includes information on how the uncertainties from each dataset have been included into the final uncertainty estimate and how uncertainties introduced at each stage of the processing have been accounted for. The concept of metrological traceability with EO data and products is a multi-faceted problem that—though it is being tackled with greater detail in dedicated projects such as FIDUCEO (www.fiduceo.eu) and others—requires considerable further research.

3.1.6. Assessment against Standards

This QI of the QA System seeks to determine the fitness-for-purpose of the data product for various applications, including climate-based applications, and the degree to which the data producers follow good practises (e.g., for uncertainty estimation and validation in generating the data products). Two key internationally recognised standardised references are addressed, including the GCOS ECV product requirements for climate applications [3], as well as the core climax system maturity matrix scores [15].

GCOS—The ECV product target numerical requirements for climate applications can be found within GCOS-200 [3]. The requirements specify the physical quantities (products), as well as the frequency, resolution, uncertainty, and stability, necessary to meet climate monitoring needs.

System maturity matrix (SMM)—The maturity model for assessing the completeness of CDR production systems developed within the CORE-CLIMAX project has been applied [15]. The CORE-CLIMAX maturity model is an adaptation of the model of Bates et al. (2012) [16], which was revised to be more generic so that it can be applied not only for satellite data sets but for all CDRs (in situ, combined satellite and in situ, and reanalyses). The aim of adopting the SMM in the QA4ECV (described in [15]) is to evaluate the production process of the ECV CDRs to ensure that they follow best practices for science, engineering, and utilisation; it is not to assess the quality of the data itself. The results of this exercise help the data user to build confidence in data producers that systematically apply best practises. In addition, the maturity scores help to determine the strengths and weaknesses in the data record generation process. This information is useful for data record producers and agencies responsible for such data products to steer activities to mitigate weaknesses in the process (e.g., insufficient validation activities). This should indeed lead to improved data quality.

3.2. QA System Process

In order for the QA4ECV QA system to be successful and widely adopted, it must be simple and intuitive, offer a wide range of tools and resources relevant to multiple EO disciplines, be documented appropriately, and ensure that the evaluation process is streamlined and follows a user-friendly ‘checklist’-type strategy. Figure 2 outlines the process of the QA4ECV QA system, the specific detail of each component is discussed below. To effectively develop, implement, and operate the system, the organisational structure and the skill sets of specific persons involved must be defined for a suite of tasks. This section outlines the requirements for the QA evaluation organisation that should implement the QA4ECV framework, as well as the responsibilities of the ECV CDR developing organisation.

3.2.1. A Dedicated QA Administration

A dedicated QA administration provides the backbone for the operation of the QA system. The office is responsible for a range of tasks including the following: preparing QA evaluation criteria; ensuring the tools, information, templates, and training modules provided within the QA system are current, state of the art, and comply with higher level requirements or recommendations; offering guidance and support for use of the QA system; conducting the independent evaluation (“audit”) by at least two domain expert evaluation officers; providing evaluation feedback to the product producer in an iterative process; and awarding endorsement for QA compliance. Noting that this type of QA system and evaluation activity is ‘new territory’ for the EO community, the framework set out within this document has been designed to allow ‘levels of QA compliance’, which will be key in encouraging ECV developers to continually improve their product QA evidence through time as the algorithm and evaluation work matures and to aid in data users’ understanding of the products. For the purpose of the prototype QA system, land and atmosphere measurement experts from the National Physical Laboratory (NPL) in the United Kingdom (UK) and the Royal Belgian Institute for Space Aeronomy (BIRA-IASB) performed the role of administrators and product evaluators.

3.2.2. Product Producers

The data product producers are responsible for completing all of the required information within each of the quality indicators. This includes uploading references and documentation, providing justification for processes and statements made concerning the development and validation of the data product, and being responsive to constructive analysis and recommendations for improvement provided by the QA office assessor. This is because the product producer knows their product best. The ECV product producer is consulted through the evaluation process, and producer consent must be gained before a QA evaluation report is made publicly available. ECV product producers need to be well motivated to complete the QA report (i.e., they need to understand that systematic assessment of product quality and the processes to generate them is advantageous for further evolution of the data products). It needs to be clear to them that the whole process, including multiple interaction and feedback, takes approximately 2–4 h to complete.

3.3. Training and Guidance

The training developed to accompany the QA system consists of guidance to support product developers in navigating and completing the required fields of quality information. A ~3-min video provides a visual overview of the QA system categories. A short 5-page quick start guide is provided to accompany the video. While a detailed QA system user manual has been produced [17]. This documentation, as well as the video, can be found on the QA4ECV website: http://www.qa4ecv.eu/qa-system/training. Further, face-to-face and web conferencing training sessions have been run by NPL and BIRA-IASB to demonstrate functionality and use of the QA system.

3.4. QA Evaluation (“Auditing”) Process

This section describes a formalised process undertaken by the QA office to ensure effective and traceable implementation of the QA4ECV framework. This includes consideration of the level of compliance to be achieved, the process for undertaking each stage of compliance demonstration, and how the framework is driven by continual improvement. An overview of the process is given in Figure 2. The QA4ECV service uses the term “audit” (a systematic, independent, and documented process for obtaining objective evidence and evaluating it objectively to determine the extent to which the evaluation criteria are fulfilled: ISO9001:2015 [18]).

The aim of checking quality records is to ensure the consistency of information between ECV datasets. Once the ECV data product producer has completed the QA system QI categories to the best of their ability, they are prompted to submit their report for evaluation by the QA office. The iterative process for checking records is demonstrated in Figure 2. A quality evaluation checklist has been derived, which contains three levels of increasing compliance (amount of detail/justification provided), for each quality indicator. The three levels include the following:

• Basic—Some information is provided on the quality of the product to allow the users to make a simple distinction between the product and others. (Light grey).
• Intermediate—Detailed information is provided on the product, allowing the user to understand how it was made and the quality and uncertainty information available to them. (Blue).
• Advanced—Significant detailed information is provided on the product, providing the user with enough information to make an informed decision about how the product should be used. (Green).

Further, the QA system generates a QA label, which will be applied to a dataset. This label provides a quick visual overview of the quality ranking a product has achieved for each QI (basic—light grey; intermediate—blue; advanced—green). The label is based on the GEO label utilised by GEOSS (Global Earth Observation System of Systems) in their datasets (http://www.geolabel.info/).

The QA evaluation/audit should be undertaken by at least two independent product experts and consolidated by an impartial QA officer to ensure a fair and robust assessment. Once the QA assessment has been completed, the product producer is invited to review the audit. This provides them with the chance to improve, update, or provide further justification for their answers. When both parties are in agreement with the evaluation, the final product quality summary report will be made publicly available. The QA summary reports are generated for two main purposes: (1) to allow the data producers to gauge how well they are achieving standardised quality assessment criteria, and where they may need to focus their efforts; and (2) for data product users to use the reports and identify suitable datasets for their requirements and/or discover information about the existing datasets they use to improve knowledge and value of applications. Detailed information regarding the QA evaluation process can be found in [7]. The development of QA summary reports for multiple data products will also have the added benefit of signalling key research gaps for future funding efforts.

4. Six New ECV Climate Data Records

The second key objective of the QA4ECV project was to generate multi-decadal CDRs for atmospheric and terrestrial ECVs that are based on inter-satellite calibrated data, state-of-the-art retrievals and are fully traceable with uncertainty metrics. Each CDR is described below, and the static top-level traceability chain is shown. The traceability chain key is shown in Figure 3. Dynamic versions of these traceability chains along with access to the eight data products produced within the QA4ECV project are available at http://www.qa4ecv.eu/.

4.1. Broadband Albedo

The QA4ECV broadband albedo product is based on processing a 35-year (1982–2016) daily time series of 0.05° polar orbiting National Oceanic and Atmospheric Administration advanced Very High-Resolution Radiometer (NOAA-AVHRR) and re-projected geostationary (Meteosat, GOES (Geostationary Operational Environmental Satellite system), GMS (Geostationary Meteorological Satellite)) level-2 surface visible and near-infrared (NIR) bidirectional reflectance factors (BRFs) into top-of-canopy bidirectional reflectance distribution functions (BRDFs). These are then integrated into surface albedo. The daily polar orbiting (BRFs) were generated within the AVHRR long-term data record (LTDR) V5 developed by [19]. The geostationary BRFs were generated using the standard SCOPE-CM (Sustained, Coordinated Processing of Environmental Satellite Data for Climate Monitoring) processing scheme [20]. The processing used to retrieve the BRDFs uses the ESA GlobAlbedo processing chain, which employs optimal estimation [21] to generate daily 0.05° (≈5 × 5 km) products over the 35-year time period. In addition, as part of the QA4ECV processing chain automated machine learning-based methods were developed and rigorously tested to screen out clouds, flag snow, and sea-ice over shallow water and water bodies in order to focus on land only pixels. The AVHRR BRFs from channels 1 and 2 were converted to 3 broadbands: visible (VIS, 0.4–0.7 µm), NIR (0.7–3 µm), and shortwave (SW, 0.4–3 µm). The GEO (geostationary satellite) BRFs were converted to SW (0.4–3 µm) as only the panchromatic bands were used, which straddle the red edge at 0.7 µm. For each annual set of BRDF/albedo retrievals, 18 months of input low Earth orbit satellites (LEO) BRFs (derived from AVHRR from seven different NOAA spacecrafts) were employed for each daily product within the central 12 months. For the area within ±60° latitude, GEO top-of-atmosphere data were processed by EUMETSAT to SW-BRF, and these were included in the joint retrieval [22]. A background dataset consisting of broadband daily climatology in the 3 wavelength regions, derived from 16 years of MODIS (MODerate resolution Imaging Spectro-radiometer) BRDFs, was employed to ensure that there were no gaps when there was persistent cloud cover or during polar night. From these daily products, using energy conservation for upscaling, 0.5° daily and monthly products were produced. For each and every pixel, an estimate of the uncertainty produced using the processes shown within the traceability chain was generated. These pixel-level uncertainties consisted of standard errors called “sigma” and a cross-product covariance term called “alpha” between the VIS and NIR channels, which were subsequently employed for the two-stream inversion package (TIP) processing (see Section 4.4). Output products also include quality measurements, such as a weighted number of samples and a relative entropy related to the influence of the MODIS prior, in addition to snow and water body flags. This QA4ECV broadband albedo product is the first ever fused product from GEO + LEO and the longest time series ever produced of the Earth’s land surface albedo. It has been extensively tested by our collaborators at the Ludwig Maximilian University of Munich for fitness-for-purpose in climate models (papers in preparation). The QA4ECV broadband albedo product top-level traceability chain is shown in Figure 4.

4.2. Spectral Albedo

In the ESA GlobAlbedo project (www.GlobAlbedo.org), surface spectral BRFs were calculated from medium-resolution imaging spectrometer (MERIS) and VEGETATION sensors. These spectral BRFs were then converted into visible, NIR, and shortwave broadbands using narrow-to-broadband coefficients calculated at Freie Universität Berlin [23]. In QA4ECV, the time series of MODIS spectral albedos (specifically daily MCD43C (MODIS BRDF/Albedo Product) at 0.05°) was extended back in time using VEGETATION as the primary sensor. This process started by generating a set of spectral coefficients, which allowed VEGETATION spectral BRFs to be converted into their equivalent for MODIS. This was performed by matching up millions of MODIS spectral BRFs from MOD09 (MODIS Surface Reflectance Product) with the closest possible matchups in view and solar angles from MERIS and VEGETATION spectral band BRFs to determine the sensor-to-MODIS spectral band mapping. For VEGETATION, with a 1.6 µm band, this mapping works very well for the first 6 spectral bands of MODIS, but it does not function correctly for MODIS band 7 (≈2.13 µm). This matchup also works surprisingly well for all the MERIS bands, even though there are no bands above 1 µm. The GlobAlbedo processing chain was modified to process all the input VEGETATION only for 1998–2000 daily data and to test this for future use with Sentinel-3. This was tested for 2005 with MERIS + VEGETATION, as employed in GlobAlbedo. We then compared these synthesised MODIS spectral albedos with the actual MODIS albedos and found extremely high correlations. A spectral version of the aforementioned MODIS BRDF climatology was employed in the optimal estimation retrieval scheme. The same technique could also be applied to generate a long-time series of MERIS-like or ocean and land colour instrument (OLCI)-like spectral channels going back to 1998. Uncertainties were calculated per band and not between bands, as this was too computationally challenging. Only data for 16 tiles over Europe were processed for the same reason. The QA4ECV spectral albedo product top-level traceability chain is shown in Figure 5.

4.3. Sea-Ice Albedo

Sea-ice albedo is a key climate change indicator as there is strong feedback between sea-ice albedo and direct radiative forcing. Up until now, models have been employed for sea-ice albedo retrieval using instruments such as AVHRR [24] for shortwave only and at low resolution (25 km) on weekly time-steps. Sea-ice packs many kilometres in size move at up to 15 km/day, so any method such as time-composting smears out each individual albedo record, rendering it unfit for retrieval of sea-ice albedo. What was needed was an instantaneous measurement of spectral BRF to allow an instantaneous retrieval of spectral albedo. The NASA (National Aeronautics and Space Administration) multi-angle imaging spectro-radiometer (MISR) instrument is the only such instrument in orbit, which records information at sufficiently high spatial resolution (1.1 km in all 4 spectral bands of blue, green, red, and NIR), albeit with a narrow 380-km swath. Surface spectral BRFs were specially processed at NASA Langley over the Arctic and Antarctic regions from ±60° of latitude to the northernmost point at 83° due to the inclination of the NASA Terra orbit. The cloud mask derived from MISR is not yet sufficiently robust to be able to differentiate cloud from sea-ice, so a separate orbital 1-km sea-ice product derived from MODIS, called MOD29 [25], was employed to mask out the clouds and sea-only areas from the MISR-derived BRF and albedos. This land surface BRF and MISR albedo product is described in [26], and the special product is described in Kharbouche and Muller (in review). The 1-km, 5-km and 25-km products are processed every orbit and then integrated over ±24 h, ±3 days, ±7 days, and monthly on a daily time-step. Although each retrieved spectral BRF and albedo has an uncertainty, this is not employed to generate the global product. Instead, for the time-composited products, a standard deviation of albedo is employed. This product has been used to generate a 16-year time series (2000–2016) for April–September of each year when the solar zenith angle ≤70°. This time series has been validated using data from a tower-mounted albedometer when converted to shortwave bi-hemispherical diffuse reflectance (BHR, sometimes called “white sky” albedo), as well as data from the NASA cloud absorption radiometer (CAR) instrument using the methods described in [27]. There is huge interest in this product across the climate–cryosphere community. The QA4ECV sea-ice albedo product top-level traceability chain is shown in Figure 6.

4.4. TIP LAI/FAPAR

Leaf area index (LAI) and fraction of absorbed photosynthetically active radiation (FAPAR) ECVs along with their per pixel uncertainty were consistently retrieved using the two-stream inversion package (TIP) [28,29] applied to visible (VIS) and near-infrared (NIR) broadband albedos from the QA4ECV project. The TIP is the inversion of the two-stream model developed by [30], which implements the two-stream approximation of radiative transfer for a homogeneous one dimensional canopy (“1D-canopy”). The 1D radiative transfer model is potentially consistent with large-scale climate and Earth system models and does not require assumptions about other factors (e.g., biome type) to be made. Owing to the 1D approach, TIP-LAI is an effective quantity, describing the optical effects of the leaves. The implementation used in QA4ECV (TIP5D) uses the full variance–covariance matrix of the BHRs, which is an enhancement beyond previous applications of the TIP [31,32], while maintaining reproducibility of the results.

In QA4ECV, TIP LAI, and FAPAR were produced globally for 0.5-degree and 0.05-degree regular grids, for each day of 1982–2016. Full per pixel processing information and extra quality information is available through the provided retrieval flags. This product is best suited for use in soft constraint data assimilation into dynamic models, which use a similar radiative transfer scheme, yielding maximum gain from the consistency of LAI, FAPAR, and the albedos and their uncertainties (e.g., [33]). As the uncertainties can be quite large, they should always be taken into account. Depending on the application it may be advisable to mask out some data according to the retrieval flag (see the product user guide for details [34]); for instance, trend analysis should not use data that were filled with the albedo prior (most notably in late 1994, where no AVHRR data is available). However, version 1.0.1 suffers from artefacts introduced by problems further up the processing chain, as detailed in the uncertainty and validation document [35]. The QA4ECV TIP LAI/FAPAR product top-level traceability chain is shown in Figure 7.

4.5. AVHRR FAPAR

Joint Research Centre (JRC) methodology was used to compute daily fraction of absorbed photosynthetically active radiation (FAPAR) from daily spectral measurements acquired by advanced very high-resolution radiometer (AVHRR) onboard a series of National Oceanic and Atmospheric Administration (NOAA) platforms, namely 07, 09, 11, 14, and 16. The methodology itself is based on previous JRC-FAPAR algorithms, such as the ones developed for the medium-resolution instrument sensor (MERIS) and the ocean land colour instrument (OLCI) [36,37], except surface reflectances in Band 1 and Band 2 were used as inputs data instead of top-of-atmosphere ones [19]. The retrieval method assumes that the leaves are alive and photosynthesising, hence the ‘green’ FAPAR is assumed. Also, contrary to the TIP FAPAR Section 4.4), the values correspond to instantaneous definition (i.e., under direct illumination). The QA4ECV products span from 1982 to 2006 at 0.05° × 0.05°. In addition to daily products, 10-day and monthly products were provided as well over a coarser resolution for biosphere changes studies (e.g., 0.5° × 0.5°). The products contain several uncertainty metrics, such as error propagation derived from inputs uncertainties and both temporal and spatial standard deviation for re-gridded products. These products are unique, as they are the only ones containing three types of uncertainties and can be used together with SeaWiFS (Sea-Viewing Wide Field-of-View Sensor), MERIS, and Sentinel-3 FAPAR products using the same retrieval algorithm and definition. However, despite the recent calibration and atmospheric correction performances made by [19], the products still contain, at the end of a few NOAA satellites, some artefacts that must be corrected for global changes studies [38]. This will be conducted in the new version. The QA4ECV AVHRR FAPAR product top-level traceability chain is shown in Figure 8.

4.6. NO₂

The QA4ECV NO₂ ECV precursor product contains harmonised vertical NO₂ columns from the ERS-2 GOME (Global Ozone Monitoring Experiment), Envisat SCIAMACHY (SCanning Imaging Absorption SpectroMeter for Atmospheric CHartographY), Aura OMI (Ozone Monitoring Instrument), and MetOp-A GOME-2(A) sensors. The main product is the tropospheric vertical column density. The datasets cover the period of July 1995–December 2017, a 22+ year record. The spatial resolution varies from 320 × 40 km² (GOME) to 13 × 24 km² (OMI in nadir), with mid-morning (10:00, local time) overpasses for GOME, SCIAMACHY, and GOME-2 and early afternoon (13:40, local time) for OMI. Global coverage is achieved every 1–6 days, depending on the instrument field-of-view and measurement conditions (e.g., presence of clouds, snow, or ice). The QA4ECV NO₂ ECV precursor product contains detailed information on retrieval uncertainty (from uncertainty propagation calculations embedded in the retrieval algorithm) and quality flags. The main uncertainty metric is the uncertainty in the tropospheric NO₂ column, but the dataset also includes a breakdown of the individual contributions to the overall uncertainty budget (i.e., from detector noise, fitting techniques, radiative transfer calculations, and assumptions made on ancillary data). A full description of the retrieval approach, uncertainty analysis and auxiliary data, and some preliminary validation is provided in [39]. The data product has been registered using unique DOIs for the four sensor subsets (e.g., as in [40]). What is unique about the QA4ECV NO₂ data record is that it is the first cross-calibrated, multi-sensor dataset, with very detailed quality information embedded and that it spans a period of more than 20 years. Tropospheric NO₂ columns are being used widely, especially for estimating NO_x emissions (e.g., [41]), for improving the estimates and attribution of ozone and aerosols (e.g., [42,43]), for trend analyses (e.g., [44]), and for reanalysis studies (e.g., [45]). Users are advised to use the tropospheric NO₂ columns by taking into account detailed information on measurement flags, spatio-temporal representativeness, and vertical sensitivity. For more detail on this and practical recommendations on how to use the data, we refer the readers to [46]. The QA4ECV NO₂ product top-level traceability chain is shown in Figure 9.

4.7. HCHO

The QA4ECV HCHO ECV precursor product contains harmonised HCHO tropospheric vertical column densities for the period of 1996–2016. The HCHO ECV data provides geophysical information for every ground pixel observed by each satellite sensor (GOME, SCIAMACHY, OMI, and GOME-2A). Global Earth coverage is achieved within 1–6 days, depending on the sensor and on the observation conditions. In addition to the vertical HCHO column densities, the product contains intermediate results for every ground pixel, such as the result of the spectral fit, fitting diagnostics, the averaging kernel, cloud information, uncertainty estimates detailed for each retrieval step, and quality flags. A full description of the retrieval algorithm, uncertainty analysis, and auxiliary data is provided in [47]. Satellite HCHO observations are widely used to gain knowledge on non-methane volatile organic compounds (NMVOC) emissions, tropospheric ozone formation, and biogenic aerosols [48]. Uncertainties in satellite HCHO observations are dominated by their random components. Users are therefore advised to average the data in space and/or in time, in order to reduce this contribution. The QA4ECV algorithm is now being transferred to the TROPOMI sensor, offering a significantly improved signal-to-noise ratio and extending the 20-years QA4ECV HCHO dataset. The QA4ECV HCHO product top-level traceability chain is shown in Figure 10.

4.8. CO

The QA4ECV CO ECV precursor product consists of a 10-year archive of CO total columns from the IASI (Infrared Atmospheric Sounding Interferometer) sensor (2008–2017). The columns are calculated from the CO profiles, retrieved from IASI day and night level 1C radiances with the FORLI software (Fast Optimal Retrievals on Layers for IASI v20100815+v20140922), on 19 vertical layers in the troposphere. The columns are provided with error estimates and quality flags at the native IASI resolution (i.e., for individual elliptical pixels with sizes ranging from IASI 12 km by 12 km at nadir to 20 km by 39 km at the largest angles). IASI provides bi-daily coverage of the Earth, with overpass times at around 9:30 and 21:30. Generally, the sensitivity to the boundary layer is better for the daytime observations, as documented in [49], and the varying sensitivity should therefore be carefully accounted for when analysing the time series in the columns. A general description of the retrieval software is provided in [50], and an algorithm technical basis document has been generated in the context of the EUMETSAT Satellite Application Facility (https://acsaf.org/products/iasi_co.html). A product description, with first analyses and consistency check with the MOPITT (Measurement of Pollution in the Troposphere) data record is provided in [51]. The QA4ECV data record is available from the French Atmosphere Infrastructure (AERIS). There are remaining non-homogeneities in the time series, which have been traced back to changes in the meteorological input parameters. The data record from IASI-A is being extended from 2012 with IASI-B, and there is no bias between the two missions; IASI-C will continue the record after 2018. The IASI CO product is supporting near real-time applications (operational dissemination via EUMETCast and assimilation in the Copernicus Atmospheric Monitoring Service (CAMS)), emission inventories, and tropospheric chemistry models [52]. The QA4ECV CO Product top level traceability chain is shown in Figure 11.

5. QA4ECV Product Quality Reports

The summary QA4ECV product QA reports are hosted on a public-facing area of the QA system. Login information is not required for general users to view the completed product QA reports. At the time of product evaluation, the QA4ECV products were still under development and not scheduled for completion until the end of the project (early 2018). Therefore, key quality indicators, such as validation and product inter-comparison studies, have not been conducted and evaluation of this QI could not be done. The maturity matrix assessment has been repeated at the end of the project and resulted in some distinct improvements for individual data records, in particular for increased completeness of validation activities and documentation. The QA evaluations for each of the eight QA4ECV products are shown below. Each product achieved varying levels of quality based on the defined criteria [7] (note grey = basic; blue = intermediate, and green = advanced quality information). This process highlights the need for an iterative and flexible QA evaluation approach, which gets vital product QA information to the user community but allows the product producer to improve the QA as further research is conducted.

Broadband Albedo   The QA4ECV AVHRR + GEO broadband albedo product produced by UCL (University College London, MSSL (Mullard Space Science Laboratory) and Geography) and Brockmann Consult has achieved an advanced status for the product details, traceability chain, and assessment against standards; intermediate status for quality flags; and basic status for uncertainty assessment and validation.

Spectral Albedo   The QA4ECV spectral albedo product produced by UCL (MSSL and Geography) and Brockmann Consult has achieved an advanced status for the product details and traceability chain; intermediate status for assessment against standards; and basic status for quality flags, uncertainty assessment, and validation.

Sea-Ice Albedo   The QA4ECV sea-ice albedo product produced by UCL (MSSL and Geography) and Brockmann Consult has achieved an advanced status for the traceability chain and assessment against standards; intermediate status for product details; and basic status for quality flags, uncertainty assessment, and validation.

TIP LAI/FAPAR   The QA4ECV FAPAR/LAI product produced by FastOpt has achieved an advanced status for the information provided for product details, traceability, quality flags, and assessment against standards; intermediate status for uncertainty assessment; and basic status for validation.

AVHRR FAPAR   The QA4ECV AVHRR FAPAR product produced by JRC has achieved an advanced status for the traceability chain; intermediate status for information provided for product details, quality flags, and assessment against standards; and basic status for uncertainty assessment and validation.

NO2   The QA4ECV NO2 ECV precursor product has achieved advanced status for the information provided for product details, traceability, quality flags, and assessment against standards; intermediate status for uncertainty assessment; and basic status for validation.

HCHO   The QA4ECV HCHO ECV precursor product has achieved advanced status for the information provided for product details, traceability, quality flags, and assessment against standards; intermediate status for uncertainty assessment; and basic status for validation.

CO   The IASI FORLI CO product (version 20140922 and version 20100815) has achieved advanced status for the information provided for traceability and quality flags; intermediate status for product details; and basic status for uncertainty assessment, validation, and assessment against standards.

6. Conclusions

Here, we present the service specification for a prototype, pre-operational quality assurance (QA) system for ECV data records and services, based on the experiences within the FP7-QA4ECV project. The QA system is designed to translate the complex information about data products that is contained in algorithm theoretical basis documents (ATBDs), product user guides (PUGs), product producers heads, and other documents into a standard format based on a simple set of questions for each quality indicator. Standardisation of this information between data products enables fair comparison of data products and facilitates guidance on best use of the data products for climate applications. It also helps identify gaps in current knowledge to drive forward scientific advancement and good practice.

Feedback on the operational utility of the prototype QA4ECV QA system was sourced from the following: the QA4ECV product producers [53]; the independent QA office auditors (NPL and IASB-BIRA) [53]; and a selected number of product “champion” users that were identified during the project [54]. The QA4ECV product producers agreed that the QA system is streamlined and relatively easy to use, and likewise, champion users were positive concerning the product QA report content, traceability chains, and accessibility of this standardised information. However, reviewing the quality information provided by the product producers within the QA system revealed several improvements that can be made to the system architecture, content, and governance. While the feedback is detailed in [7,54], some key elements for improvement are outlined below.

The product users requested a different report layout streamlining if information was not available, provision of more detail in sections related to the quality flags, validation, and data uncertainties, as well as more product usability case studies to be presented. Further, concern was raised about the usability of the GCOS requirements and the system maturity matrix information in the QA context. The SMM should only be used as an overarching management tool by funding organisations to track the maturity of their data products, and consideration as to how relevant this information is to data users, needs to be investigated further. The QA system architecture could be improved through a more robust software framework, as well as enhancing the evaluation “audit” functionality and communication exchanges between reviewer and product producer. The “audit” functionality could become part of a review process that leads to the release authorisation of a data record to the public. Such reviews are common practise with operational data providers, such as EUMETSAT, and would lead to mandatory provision of information to the QA system by the data record producers. Such a review process is important for operational activities, such as the Copernicus Climate Change Service, to ensure that published data products are good and mature enough to support the authoritative character of the C3S services that make official statements about climate change on behalf of the EU. The system content may be improved by tailoring content more specifically to each ECV domain (land, ocean, and atmosphere) and re-evaluating the audit categories and evaluation scheme, as well as integrating the traceability chains with the algorithm uncertainty information.

The QA4ECV QA system provides a solid architecture for the concepts of QA for ECV data products derived from EO satellites. The system addresses the science and product quality attributes; however, additional quality dimensions, such as that of stewardship and services, to the overall quality of these ECV datasets should be investigated further and incorporated into future iterations of CDR evaluation and quality control processes [55,56]. In particular, this relates to the standardisation of QI information and subsequent requirement for data producers to incorporate all relevant field within their product’s metadata. This issue is highly relevant to international coordination bodies, such as CEOS, and this forum should be used to encourage product development and funding organisations to ensure quality information standardisation and provision within all data products.

The QA system was applied successfully to the six QA4ECV data products. A summary quality report has been generated for each data product and made available to data users via the QA system to aid in fitness-for-purpose assessments for different application requirements. Throughout the project, product producers and external QA4ECV “champion users” have provided positive and constructive feedback on the architecture, content, and governance of the prototype QA4ECV QA system presented. All product producer and champion user feedback has been consolidated and will be taken into account for future iterations of the QA system that will be applied in a revised form as part of the evaluation and quality control functionality of the European Copernicus Climate Change Service (C3S).