Time Series Homogenization with ACMANT: Comparative Testing of Two Recent Versions in Large-Size Synthetic Temperature Datasets

Abstract

Homogenization of climatic time series aims to remove non-climatic biases which come from the technical changes in climate observations. The method comparison tests of the Spanish MULTITEST project (2015–2017) showed that ACMANT was likely the most accurate homogenization method available at that time, although the tested ACMANTv4 version gave suboptimal results when the test data included synchronous breaks for several time series. The technique of combined time series comparison was introduced to ACMANTv5 to better treat this specific problem. Recently performed tests confirm that ACMANTv5 adequately treats synchronous inhomogeneities, but the accuracy has slightly worsened in some other cases. The results for a known daily temperature test dataset for four U.S. regions show that the residual errors after homogenization may be larger with ACMANTv5 than with ACMANTv4. Further tests were performed to learn more about the efficiencies of ACMANTv4 and ACMANTv5 and to find solutions for the problems occurring with the new version. Planned changes in ACMANTv5 are presented in the paper along with related test results. The overall results indicate that the combined time series comparison can be kept in ACMANT, but smaller networks should be generated in the automatic networking process of the method. To improve further the homogenization methods and to obtain more reliable and more solid knowledge about their accuracies, more synthetic test datasets mimicking the true spatio-temporal structures of real climatic data are needed.

1. Introduction

The purpose of the homogenization of observed climatic time series is to remove possible non-climatic biases which often occur for changes in station location, instrumentation, instrument installation, station surrounding, or observing practices. The effects of such technical changes are referred to as inhomogeneities, and they usually manifest as sudden, non-climatic shifts (breaks) in the section mean values of the time series and in other statistical properties of the probability distribution of the observed data, although sometimes gradually growing biases also occur. The homogenization of climatic time series (hereafter: homogenization) intends to separate non-climatic biases from true climate variation. In homogenization, a candidate series is usually compared to several neighbor series, and the differences are evaluated using appropriately designed statistical methods [1,2,3,4]. This procedure is named relative homogenization, and its advantage is that regionally common climatic variations are not present in the spatial difference series. Therefore, the use of spatial difference series (they can be arithmetic differences or ratios according to climatic variables, and all of them can be referred to as relative time series) helps to separate non-climatic changes from true climate variation. Homogenization can be performed with the help of documents (so-called metadata) about the history of technical changes in the observations, but the inclusion of statistical homogenization is preferred even when metadata support homogenization. When insufficient station density limits or impedes the use of neighbor series, reanalysis data or other kinds of auxiliary climate series may help homogenization [5,6], and in some special cases absolute homogenization, i.e., homogenization of a candidate series without the use of any other time series, can be performed.

In this study, the accuracy of some relative homogenization methods without metadata use will be analyzed in synthetically developed test datasets. For the tests, the use of synthetically developed datasets is needed, since true inhomogeneity properties are known only in synthetic data.

A large number of statistical methods have been developed to provide accurate relative homogenization. Some important milestones of this development were the creation of the MASH [7], PRODIGE [8], PHA [9], HOMER [10], Climatol [11], and ACMANT [12] homogenization methods. In the international method comparison tests, ACMANT was found to be the best performing method more frequently than any other tested method [13,14,15,16,17], although the rank order of efficiency depends on test dataset properties, efficiency measures and the tested method versions. Regarding recent developments, some promising novel techniques are included in ACMANTv5 [18] and in a new, HOMER-based method, Bart [19]. The new MASH version (MASHv4 [20]) also includes valuable elements. Notwithstanding, in this study only some ACMANT versions are tested due to limits of technical and working capacity.

Domonkos [18] reported that the method of “combined time series comparison”, including a break detection step with pairwise comparisons and another break detection step with composite reference series use, increases the efficiency of homogenization when time series are affected by synchronous or semi-synchronous breaks (they are also referred to as clustered breaks). That research used a test dataset [16] having been part of the Spanish MULTITEST project [17] and is referred to as the MULTITEST dataset in this study, and the findings showed that the new method tends to give more accurate homogenization results even for datasets free of clustered breaks. However, newer tests with the synthetic daily temperature test dataset developed by Killick [13] (hereafter the K2016 dataset) gave somewhat less favorable results, which inspired the author to perform even more tests and search for possible refinements of the methodology. The objective of this paper is to share the results of the new tests with the research community, and discuss the knowledge provided by these tests.

The K2016 dataset is used throughout the paper, but in two different forms: it is used in its original form, but also in a modified form where subsets of 50 time series are selected from the original dataset. ACMANTv4 and two experimental versions of ACMANTv5 are tested.

The structure of the study is as follows: In Section 2, a brief review is presented about the main concepts and tools of climate data homogenization. Section 3 contains the detailed descriptions of the test datasets. In Section 4, the ACMANT versions used in this study are presented. The measures of homogenization accuracy used are described in Section 5. Section 6 and Section 7 are dedicated to the presentation of the results and the discussion of the results, respectively. Finally, the conclusions are presented in Section 8.

2. Main Concepts and Methods of Climate Data Homogenization

The purpose of homogenization is the removal of non-climatic effects, so-called inhomogeneities, from the temporal evolution of the recorded observed values. While the climate signal is approximately the same within a given climatic region, most inhomogeneities are station-specific, since they are linked to some local change in the technical conditions of the observation or data recording. For the latter reason, inhomogeneities can be referred to as the temporal changes in the station effect. Then, the time series of observed data X consists of the climate signal (U), station effect (V) and noise (ε) where the noise is composed of the spatially uneven weather effects and non-systematic observation errors (Equation (1)).

$$\mathbf{X}=\mathbf{U}+\mathbf{V}+\mathsf{\epsilon } (\mathbf{X}=x_1, x_2,\dots x_i\dots x_{n }\left(i=\mathrm{1,2}\dots n\right))$$

(1)

When the relative time series (T) of X and a neighbor series (F) is examined, U will be 0 (approximately), and only the station effects of the two series and a noise series remain (Equation (2)).

$$\mathbf{T}=\mathbf{X}-\mathbf{F}={\mathbf{V}}_{\mathbf{X}}-{\mathbf{V}}_{\mathbf{F}}+\mathsf{\epsilon }\mathbf{\prime }$$

(2)

The neighbor series used for spatial comparisons are often referred to as reference series. When the reference series is homogeneous, V_F is constant, and thus only the noise might mask the inhomogeneities of X. Time series models used in homogenization usually contain break type inhomogeneities only.

The efficient homogenization of long time series is generally a challenging task for several reasons.

When long time series are examined, all or most of the time series include inhomogeneities of different sizes, so we cannot find homogeneous reference series a priori. The average frequency of break occurrences is usually around 1 per 15–20 years [2,21].

The signal-to-noise ratio (SNR) is often lower than expected. A low SNR may occur due to insufficient spatial density of observations, low correlations between the observed time series, or the small magnitude or short duration of inhomogeneities. Inhomogeneities of small magnitude or short duration cause relatively small biases from the true climate signal, but they may affect the correct detection of larger inhomogeneities [22,23]

Long time series are usually affected by multiple inhomogeneities, making the correct detection of individual breaks and the removal of the complex bias that they cause in the time series of observed data difficult.

Sometimes several time series are affected by the same type of inhomogeneity bias, which may notably reduce the efficiency of time series comparisons in relative homogenization.

A large number of statistical homogenization methods have been developed by mathematicians and climate researchers, but some common features exist between them. A relative homogenization method includes at least three steps: (a) time series comparison; (b) detection of inhomogeneities; and (c) corrections for removing inhomogeneity biases. Relative homogenization methods usually also contain outlier filtering and gap filling steps, and the overall number of steps is usually much higher than three for the attenuation of the possible effects of large inhomogeneities (can be referred to as pre-homogenization) before the final homogenization cycle of time series comparison, inhomogeneity detection, and bias removal. Most homogenization methods are multiphase procedures including two or more homogenization cycles. Relative homogenization methods often also calculate the seasonal differences and some higher statistical moments of inhomogeneity biases. Here, only the three main parts of relative homogenization, the effects of the iterative approach to inhomogeneity bias removal, and the role of metadata are discussed.

(a)

Time series comparison: This is usually solved either using (i) composite reference series or using (ii) pairwise comparisons. Further possibilities are (iii) the use of multiple reference series or (iv) combined time series comparison.

(i)

A composite reference series includes the simple or weighted average of some selected neighbor series [1,24]. With the averaging, the impacts of possible inhomogeneities in the neighbor series are attenuated. Weightings can be applied according to the squared spatial correlations [24] or using ordinary kriging [25].

(ii)

Pairwise comparisons: The candidate series is compared one-by-one to each of the selected neighbor series [8,26]. The detected breaks in individual time series comparisons are subjected to a final evaluation. An automated version of this procedure has been developed [9]. Pairwise comparison is often considered a modern and powerful homogenization tool [4,10,27,28], although this method has both advantages and drawbacks in comparison to the use of composite reference series [29].

(iii)

Multiple reference series: The selected neighbor series are sorted into some groups, and composite reference series are edited from the series of a group in each group, according to (i). The pieces of the break detection results are jointly evaluated as in (ii) [7,30].

(iv)

Combined time series comparison: First, the break detection is performed using pairwise comparisons, then the break detection is repeated using composite reference series. In the second step of the combined time series comparison, the dates of the breakpoints detected in the first step are set as fix break positions, and only possible additional breaks are searched in the second step [18].

(b)

Detection of inhomogeneities: A large number of inhomogeneity detection methods have been used in the history of climate data homogenization. Most inhomogeneity detection methods detect breaks only, and in the modern homogenization practice inhomogeneity detection always means break detection. This is because the modeling of more complex inhomogeneity structures than breaks acts as an additional error source in the inhomogeneity detection [31]. In the use of break detection methods, gradually changing inhomogeneity biases are approached with the detection of one or more breaks, and the methods must be prepared to detect multiple break occurrences. From this point of view, break detection methods can be sorted to three main groups: (i) iterative, hierarchical approach to the final solution with the detection of one break in an iteration step, and cutting the time series into two parts at the detected breakpoint (binary segmentation) [32] to examine them separately for the detection of possible further breaks; (ii) direct detection of multiple break structures; and (iii) the use of moving windows (i.e., sections of time series) for detecting one or zero breaks in each window.

(i)

There are a large variety of single break detection methods. The maximum likelihood methods like the Bayesian test [33] or Penalized Maximal t Test (PMT) [34] are the best methods, but their advantage over many other methods is minor [31]. Another type of method, the Standard Normal Homogeneity Test (SNHT) [35], is the most commonly used single break detection method. SNHT is a likelihood ratio test, and its simplicity compensates for the minor and practically imperceptible decrease in accuracy relative to the maximum likelihood methods.

(ii)

The direct detection of multiple break structures is performed mostly using step function fitting, following the model of [8]. Its bivariate version was developed by Domonkos [12]. A different kind of multiple break detection method is applied by Szentimrey [7,20]. Beyond these, there exists a method in which all breaks of a network of time series are jointly detected (joint segmentation [36]). It is included in HOMER [10,37,38] and Bart [19,39] and is known as Joint detection.

(iii)

Break detection using moving windows: such methods are generally not recommended since their results are generally poor due to the limited coherence between the pieces of the break detection results. However, in Climatol [11,40,41,42] a combination of binary segmentation and moving windows is applied with very good results [13,16,17,43].

(c)

Corrections for removing inhomogeneity biases: Various methods are in use, but in relative homogenization one method clearly outperforms the others, and it is the joint calculation of correction terms for the network of time series whose data are homogenized together. The so-called ANOVA correction model was introduced to homogenization by Henry Caussinus and Olivier Mestre [8]. The model does not presume conditions other than those that are generally used in relative homogenization; hence, it does not have any specific error source. The joint calculation of correction terms is based on an equation system, in which the slightly modified forms of Equation (1) are written for each time point of the time series and also for each homogeneous section (i.e., sections between two adjacent detected breakpoints, or between an endpoint of the time series and the closest detected break to the endpoint) of each time series [8,12,37,44,45]. The ANOVA correction model has a more developed version, referred to as the weighted ANOVA model [12] in which the spatial variation in the climate signal is taken into account. The ANOVA correction model is applied in PRODIGE, HOMER, Bart, AHOPS [30], and ACMANT.

When using different inhomogeneity removal methods than ANOVA corrections, individual adjustments are applied for each detected break, or, alternatively, spatial interpolation can be applied. In both cases, the final solution is approached with a large number of iterations. Such iterations often cause undesired error propagation between the jointly homogenized time series [29], resulting in elevated biases in the regional mean trends. It is a serious problem since the importance of the accuracy of the regional mean trends is enhanced [29,46,47] The Pairwise Homogenization Algorithm (PHA) [9,28,48] excludes any iteration, and it uses only the presumably homogeneous sections of the neighbor series for the calculation of correction terms. This strategy has specific advantages and drawbacks [16,29]. It is noted again that none of the methods discussed in this paragraph have competitive performance with the ANOVA corrections.

The accuracy of homogenization depends both on the homogenization methods and the statistical properties of the homogenized time series. Metadata are generally considered to be important [1,3,49,50], but dense and spatially highly correlated time series can be homogenized without metadata. However, the consideration of metadata is particularly important when they indicate synchronous or semi-synchronous technical changes in many time series [4,16,51,52], since such incidents may make the separation of inhomogeneities from true climatic variation difficult or impossible. The potential benefit of metadata increases with the magnitude of inhomogeneity biases and with decreasing station density [53]. Thus, metadata are generally even more important in the homogenization of the old data of early instrumental observations [2,54,55] than in that of the modern climate observations.

Metadata showing station-specific changes can be used within automatic homogenization procedures, and the most efficient method is the introduction of metadata dates into the final application of the ANOVA correction model. ACMANTv5 includes this automatic metadata use [53]. ACMANT is an effective and user-friendly homogenization method, and its use is recommended before climatological analyses [17,43,45,56,57,58,59].

Interested readers can find many more details about the homogenization of climatic time series in a recently published book [29].

3. Data

The study uses the synthetic daily temperature dataset of K2016 [13]. In the first part of this section (Section 3.1), some characteristics of the original dataset are presented, then the two ways that it is used in this study are described in Section 3.2 and Section 3.3. However, the description starts here with the definition of some terms.

“Synthetic”, “surrogate” and “simplified” test data: During the European project HOME [21] the concept of surrogate data was introduced for artificially generated data mimicking most of the spatial and temporal variations in observed data in a given geographical area well, while test data developed using simpler methods are called synthetic data. A drawback of this use of terms is that any artificially developed dataset can be called a synthetic dataset. Using the term “simplified dataset” would likely be better for datasets excluding the reproduction of truly occurring low-frequency changes in spatial climatic gradients.

In this study, the term “network” is reserved for groups of time series whose data are homogenized together, and it is not used in other contexts, e.g., for sections of a dataset specific for a geographical region or for the method of the dataset generation.

3.1. Properties of the Source Dataset

The source dataset K2016 is a surrogate daily temperature dataset representing four U.S. regions of 2 × 10⁵ to 3 × 10⁵ km² size for each. They are referred to as Wyoming (WY), Northeastern (NE), Southeastern (SE) and Southwestern (SW) regions. The base of the dataset development was the 20th Century Reanalysis dataset [60], but observed climatic data and large-scale circulation indices were also used in the formation of the temporal and spatial structures of the data. The test dataset has a “clean” section which does not contain inhomogeneities or missing data, while its “corrupted” section includes both inhomogeneities and missing data. Three or four inhomogeneous sections were developed for each region; such sections sometimes differ also in station density. The overall number of inhomogeneous dataset sections is 13, and each of them includes 75 to 222 time series. All of the time series cover the 1970–2011 period with less than 25% missing data. The median spatial correlation is around 0.75, although it is only ~0.6 for the SW region (see Figure 3.6 of [13]). These correlations allow the use of neighbor series for any candidate series from almost all parts of a region. However, the simulated climate and its temporal evolution are not constant spatially. Temporal changes in spatial climatic gradients might cause the detection of false breaks, since they cause the presence of some climate effects in relative time series. Killick [13] tested the frequency of false break detection using the PHA method in the homogeneous sections of the dataset. Domonkos et al. [29] performed the same tests using ACMANTv5. The results of these tests (

Table 1. Number of detected breaks in the homogeneous section of the K2016 dataset. Mean magnitudes (°C) of ACMANT detected breaks are shown in brackets. Adopted from [29].

Data Section	Number of Series	Number of Detected Breaks in Homogeneous Data
		PHA	ACMANTv5
WY1	75	4	7 (0.15)
WY2	158	5	11 (0.12)
WY3	158	16	9 (0.18)
WY4	75	3	12 (0.61)
SE1	153	13	53 (0.09)
SE2	210	9	58 (0.10)
SE3	210	15	69 (0.10)
NE1	146	11	4 (0.09)
NE2	207	9	11 (0.09)
NE3	207	11	5 (0.06)
SW1	151	50	77 (0.16)
SW2	222	28	131 (0.15)
SW3	222	31	100 (0.18)

) are important for the correct interpretation of the test results for the inhomogeneous sections of the dataset.

The frequency of time series with detected false breaks may be expected not to exceed 5–10% of the number of tested time series. However, many false breaks were detected for the SW region using both tested methods due to the complex geographical composition of that region [13]. Overall, ACMANT detected many more false breaks than PHA, but not in every region. Table 1 shows that the ratio of false breaks depends both on the spatio-temporal changes in climate and on the homogenization methods. For instance, in the SE region only ACMANT detected more false breaks than expected. Fortunately, the magnitudes of these breaks are generally very small, hence their effect on the accuracy of inhomogeneity bias removal is minor [13,29].

3.2. Dense Test Dataset

The source K2016 dataset was used in its original form, and it is referred to as the dense dataset. The mean distance between the adjacent stations is about 40–50 km.

3.3. Moderately Dense Test Dataset

The core operation is the random selection of 50 stations from a section of the source K2016 dataset. This operation was performed 10 times for each of the WY1, NE1, SE1 and SW1 sections of the K2016 dataset. In this way, the moderately dense test dataset was generated and included 40 dataset sections. The mean distance between adjacent stations is ~80 km. The main goal of this dataset creation was to use a dataset with a higher number of dataset sections than the source dataset in order to reduce the random component of the estimated homogenization accuracies.

4. ACMANT Homogenization Method

The development of ACMANT started around 2010 based on the PRODIGE method (ACMANT = Adapted Caussinus–Mestre Algorithm for the homogenization of Networks of climatic Time series). It approaches the final solution with three homogenization cycles, and ensemble homogenization helps to attenuate random effects. A brief description of ACMANT is provided in a recent daily temperature and precipitation dataset development for Catalonia [45], while the full description of ACMANTv4 was published by Domonkos [12]. ACMANTv4 was used in the method comparison tests of the MULTITEST project. In those tests, ACMANTv4 often produced more accurate homogenization results than any other tested method [16,17], but a problem of ACMANTv4 was also revealed: the method cannot effectively treat inhomogeneities of clustered breaks. To address this issue, the combined time series comparison was introduced to ACMANTv5 [18]. The development of ACMANTv5 is ongoing, and some of the recent developments have not been published in other documents. Here, the differences from ACMANTv4 in three subversions of ACMANTv5 are presented. Note that from the three subversions only ACMANTv5.1 is currently available and the other subversions, referred to as A52 and A53, are under development.

4.1. ACMANTv5.1

The method of combined time series comparison (Section 2) has been introduced into the first homogenization cycle of the method, and it has exchanged the ensemble homogenization of that cycle in the earlier versions. A set of detected breaks produced with the combined time series comparison contains the detected breaks from the first step together with the additionally detected breaks from the second step. Of course, the number of detected breaks can be zero in any step. Inhomogeneity bias removal is performed only after both steps of the combined time series comparison have been finished, and it is completed using the ANOVA correction model. The full description of the combined time series comparison was presented by Domonkos [18].

Two more important novelties of ACMANTv5 are as follows: (i) this method has both automatic and interactive versions [61] and (ii) metadata can be treated using both of the automatic and interactive versions [53]. The details of these aspects are not provided here, since the present study examines only automatic homogenization without metadata.

In ACMANTv5.1, the parameterization of the final ensemble homogenization (at step 17.3.2 of ACMANTv4) is modified. The nine linear combinations of the minimum of the ensemble results (z′) of the second homogenization cycle and the arithmetical average of the same ensemble results (z⁺) are used both in ACMANTv4 and ACMANTv5. Using ACMANTv4 the weights of z′ (denotation of weight: c′) change from −3 to 4, while those of z⁺ (c⁺) decrease from 4 to −3. In ACMANTv5, weights c′ increase by 0.5, while c⁺ decrease by 0.5 (

Table 2. Coefficients of adjustment terms z’ and z⁺ in the nine ensemble members of the final homogenization cycle in ACMANTv5.

	c′	c+		c′	c+		c′	c+
1	−2.50	3.50	4	0.31	0.69	7	2.44	−1.44
2	−1.30	2.30	5	1.00	0.00	8	3.30	−2.30
3	−0.44	1.44	6	1.69	−0.69	9	4.50	−3.50

).

4.2. Version A52

The changes introduced by ACMANTv5.1 are kept, but three further modifications are applied.

(i)

The length of the overlapping periods in the use of relative time series for break detection is changed. The concept and practice of using overlapping relative time series are presented at section B6 and step 10.1 of the ACMANTv4 description [12]. As a first approach, only one relative time series is used, always the one with the highest β score. This score is determined primarily using the number of neighbor series included in the composite reference series, but some other factors are also considered. However, close to any endpoint of a relative time series (which can be different to the endpoints of the candidate series), the reliability of break detection is reduced. Therefore, overlapping of relative time series is applied when it helps to cease or reduce such edge effects. In ACMANTv5.1, the maximum length of the overlap is 9 years, while in A52 it is increased to 15 years. However, when a detected break point is close to the endpoint of the previously used relative time series, the overlap by the lately used relative time series extends only to the timing of that detected break. This parameter change is applied in all break detection steps of A52 when multiple relative time series are used.

(ii)

The creation of relative time series for break detection in the first homogenization cycle is modified. The applied modifications partly change the content of steps 9.1–9.3 of ACMANTv4. Note that in ACMANTv5, these steps are part of the combined time series comparison.

In A52 networks are classified to be small networks or large networks. In the classification, the mean number of time series with comparable observed data (N*) is considered. For the calculation of N*, the period from the earliest starting year (Y_A ≥ 1) of all homogenized periods to the latest ending year (Y_B ≤ n) of all homogenized periods is used. N denotes the number of years in the study period defined by the user, while the homogenized period defines the period of a time series in which the ratio and compactness of the observed data, as well as the availability of spatially comparable data of neighbor series, make it possible to perform homogenization with ACMANT [61]. When the total number of time series in the network is N, the number of truly comparable data N′ (N′ ≤ N) may vary over time (i), Equation (3).

$$N^{*}=\frac{1}{Y_B-Y_A+1}\sum _{i=Y_A}^{Y_B}N\mathrm{\prime }\left(i\right)$$

(3)

In method A52 a network is considered to be a small network if N* ≤ 15, while it is considered to be a large network in the opposite scenario.

In small networks, only one relative time series is edited for each candidate series. It covers the whole homogenized period of the candidate series. The composite reference series includes all neighbor series which have a homogenized period overlapping with the homogenized period of the candidate series. When the considered neighbor series have missing data, they are completed over the homogenized period of the candidate series, and the completed series are used for the creation of the relative time series. Neighbor series are equally weighted in small networks.

In large networks, the neighbor series are weighted using their squared spatial correlations with the candidate series. There is no other change in the edition of multiple relative time series for large networks.

Note again that this methodology is used only at the second step of the combined time series comparison and is not used in other relative time series edition steps of A52.

(iii)

In the gap filling steps of A52, the use of monthly data is preferred in several details of the procedure, even when daily data homogenization is performed. The earlier concept of always using daily data for gap filling in daily data homogenization was based on the fact that monthly values may have elevated uncertainty when some of their daily data are missing. However, tests proved (not shown) that the use of daily data in gap filling does not yield perceptible accuracy improvement in the final results, except for a few details of the procedure, which are presented here and still considered in A52. The motivation of these changes is that the reduction in using daily data in gap filling steps often significantly reduces the computational time.

The gap filling for monthly temperatures is performed using Equations (4)–(7) adopted from Section B12 of the ACMANTv4 description:

$${gc}_{h0}=\frac{1}{W^{\prime }}{\sum }_{s=1}^{N^{″}}{w}_s{g}_{s,h0}^{\prime }$$

(4)

$$g_{s,h0}^{\prime }=g_{s,h0}+\frac{1}{H^{″}}\sum _{h=h_1}^{h_2}({gc}_h-g_{s,h})$$

(5)

$$W^{\prime }=\mathrm{m}\mathrm{a}\mathrm{x}(p_4,W)$$

(6)

$$W=\sum _{s=1}^{N^{″}}{w}_s$$

(7)

Denotations: gc—candidate series; g_s—neighbor series s; h—serial number of month; h₀—timing (month) of missing data in the candidate series; N″—number of neighbor series used; w_s—weight (depending most on the squared spatial correlation between gc and g_s); h₁ and h₂ are the applied time window around h₀; H″—number of months with observed data in both gc and g_s within the time window; and p₄—parameter (usually 0.4).

When the time resolution is changed from monthly to daily, d (day) can be written instead of h in Equations (4) and (5), with which the formulas are converted to Equations (8) and (9).

$${gc}_{d0}=\frac{1}{W^{\prime }}{\sum }_{s=1}^{N^{″}}{w}_s{g}_{s,d0}^{\prime }$$

(8)

$$g_{s,d0}^{\prime }=g_{s,d0}+\frac{1}{D^{″}}\sum _{d=d_1}^{d_2}({gc}_d-g_{s,d})$$

(9)

In ACMANTv4 and ACMANTv5.1, Equations (8) and (9) are applied in all gap filling steps of daily data homogenization. (Gap filling for precipitation data is somewhat different, but its presentation is excluded in this study.) However, Equation (9) is not used in A52, except for at the initial generation of monthly data (next paragraph). In the preliminary operations and within the first two homogenization cycles, only monthly data are used in gap filling, and Equations (4) and (5) are also used there for daily data homogenization. It is possible, because in the first two homogenization cycles the other steps of the homogenization are also completed in monthly or annual resolution. In the last homogenization cycle and in the final gap filling step, the daily values are determined using Equation (8), but still monthly data are used for calculating the differences between the averages of station series, as shown with Equation (10)

$$g_{s,d0}^{\prime }=g_{s,d0}+\frac{1}{H^{″}}{\sum }_{h=h_1}^{h_2}({gc}_h-g_{s,h})$$

(10)

In the daily data homogenization with ACMANT, a monthly data is considered to be observed when at least 75% of the daily data in the month are observed. Differences between the mean climate anomaly of the observed data of a month and that of the other days of the month may cause biased estimations of monthly values. To reduce such biases, gap filling with daily data is performed in the initial generation of monthly data. In this step, only the data of the month including the target missing data (d₀) are used. Here, Equation (9) is used with d₁ = 1, and d₂ equals the number of days in the month.

4.3. Version A53

The changes introduced with A52 are kept, but further modifications are included. All of the newly introduced changes are related to the automatic network construction. In A53 two networks are used when the input dataset contains more than 22 time series. One of the networks is constructed in the same way as in the earlier ACMANT versions, while the other network is constructed with the modification of a few parameters. These new networks are generally smaller than the networks of the earlier method versions, and to distinguish the two network types easily they will be referred to as large networks and small networks.

(a)

Generation of large networks: identical to the network construction of the earlier method versions (see step 3.6 of the ACMANTv4 description).

(b)

Generation of small networks:

(i)

First, the best correlating 20 neighbor series are selected.

(ii)

When the first 20 neighbor series do not cover parts of the homogenized section of the candidate series sufficiently, further neighbor series are selected when neighbor series s with index S > 0 can be found (Equation (11)).

$$S\left(s\right)=S_1+S_2+S_3$$

(11)

S₁ is an empirically constructed index characterizing the frequency of those observed monthly values of the candidate series, which are paired with fewer than 10 synchronous observed data of the neighbor series. S₂ is also an empirically constructed index, with which the frequency of fewer than 20 synchronous observed data of neighbor series are considered in overlapping 10-year-long sections of the homogenized period of the candidate series. There is no change in the calculation of S₁ and S₂ relative to their use in large networks. Index S₃ is a penalty term for the excess in the network size (N′, see Equation (12)).

$$S_3={-(N^{\prime }-q)}^2$$

(12)

In the construction of large networks q = 31, while in that of the small networks q = 21. When S is positive for more than one neighbor series, the one with the highest S is selected. The network construction is finished when no neighbor series with S > 0 can be found.

(c)

Use of small networks and large networks in A53: In most parts of A53 the small network is used. The exceptions are the second step of the combined time series comparison, i.e., the break detection with composite reference series in the first homogenization cycle, and the preparatory steps for that break detection step.

4.4. Selection of Method Versions

Here, the reasoning for the actual method version selection is provided. As preliminary examinations indicated that the network size impacts the test results more than the other modifications applied, A52 and A53 are selected to compare two method versions differing only in the construction and use of networks. The results of these ACMANTv5 versions are contrasted with the results of ACMANTv4 to analyze the favorable and unfavorable features of the recent modifications in ACMANT.

5. Efficiency Measures

The accuracy of the homogenization results is evaluated through examining the differences between the homogenized series (A) and the perfectly homogeneous series (B). In this study, the centered root mean square error (RMSE) of daily values, the RMSE of annual means, the absolute value of linear trend bias for the whole period of the individual time series and the absolute value of network mean linear trend bias are examined. Note that a particularity of the centered RMSE is that possible deviations in the overall mean values are not considered, but only the deviations in the temporal variation. The use of centered RMSE was introduced to the evaluation of homogenization accuracy during HOME [21].

(i)

RMSE of daily values:

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}(\mathrm{d})=\sqrt{\frac{1}{M}\sum _{i=1}^M{\left(a_i-b_i-\frac{1}{n}\sum _{j=1}^n{(a}_j-b_j)\right)}^2}$$

(13)

In Equation (13) M and n stand for the length of time series in days and years, respectively.

(ii)

RMSE of annual values:

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}(\mathrm{y})=\sqrt{\frac{1}{n}\sum _{i=1}^n{\left(a_i-b_i-\frac{1}{n}\sum _{j=1}^n{(a}_j-b_j)\right)}^2}$$

(14)

(iii)

Absolute value of linear trend bias (Trb) when trend slopes are denoted with α:

$$\mathrm{T}\mathrm{r}\mathrm{b}=\left|{\alpha }_a-{\alpha }_b\right|$$

(15)

Equation (15) can be applied for the calculation of trend bias both in individual time series and in area mean series. In the calculation of regional mean trend bias, the regional mean annual values are calculated before the application of Equation (15). In this study, the default unit of Trb is °C per 100 years, but some exceptions will be indicated when another unit is applied for visualization purposes.

(iv)

The improvement in ACMANTv5 in comparison with the ACMANTv4 results is characterized with the Z-index [18].

$$Z=\frac{E\left(\mathrm{A}\mathrm{C}\mathrm{M}\mathrm{A}\mathrm{N}\mathrm{T}\mathrm{v}5\right)}{E\left(\mathrm{A}\mathrm{C}\mathrm{M}\mathrm{A}\mathrm{N}\mathrm{T}\mathrm{v}4\right)}-1$$

(16)

In Equation (16) E stands for the mean of a given type of error (RMSE or Trb) calculated using Equations (13)–(15). Negative (positive) values of Z indicate an improvement (worsening) relative to ACMANTv4. When the absolute value of the denominator is small, the Z-index might show unrealistic values. Therefore, Equation (16) is applied only to sufficiently large samples of homogenization results.

6. Results

6.1. Results for the Dense Test Dataset

Figure 1 shows the inhomogeneity bias reduction using the A53 method in the 13 sections of the original, dense K2016 dataset.

Figure 1a shows that high ratios of the raw data errors are removed in all error types of daily RMSE, annual RMSE and trend bias for the individual time series. The highest error reduction is achieved for the Trb of the individual series. By contrast, the error reduction in the regional mean trend bias (Figure 1b) is much smaller, but note that the regional mean trend biases of the raw data are generally small, particularly if the shortness of the time series (42 years) is also considered. When the found efficiencies according to error types are ranked from the highest to the lowest, the order is Trb for individual time series, annual RMSE, daily RMSE and Trb for the regional mean series. This rank order is typical for homogenization results, and it was found in several other studies [16,21] with the only difference that the monthly RMSE is tested instead of the daily RMSE when the test data have a monthly resolution. In the present test results, the error reductions in RMSE and Trb for individual time series are higher than in many other tests, mainly for the high station density, high spatial correlations and generally high SNR in the K2016 dataset.

In Figure 2 the residual homogenization errors are compared between different ACMANT versions. Figure 2a compares the results of A52 and ACMANTv4, while Figure 2b shows the same comparison for A53 and ACMANTv4.

Increasing (decreasing) errors relative to ACMANTv4 results are shown with red (blue) color. Figure 2a shows that although the absolute values of differences are small, the dominance of positive sign changes indicates that the errors with A52 are mostly larger than with ACMANTv4. The typical magnitude of the difference is lower than 0.02 °C RMSE or 0.02 °C/100 yr Trb. In the trend bias results, and particularly in the regional mean trend bias results, the differences between A52 and ACMANTv4 results are sometimes larger. Note, however, that this difference in the magnitude of differences is likely for the relatively large sampling errors of regional trend bias values (there are only 13 regions). The presence of sampling errors in the results explains that a large error decrease from ACMANTv4 to A52 also occurred (in region SW1), which, however, does not change much in the generally unfavorably picture regarding the A52 results.

Figure 2b shows that the A53 results tend to be more accurate than the ACMANTv4 results, although the magnitudes of the differences are very small. The regional trend bias results are exceptions. No decreasing tendency can be found for them, i.e., in six regions the A53 results are the most accurate, in further six regions the ACMANTv4 results are the most accurate, while in one region the regional trend bias is exactly the same with any of the two examined methods.

6.2. Results for the Moderately Dense Dataset

Figure 3 shows the residual errors of homogenization with A53 in comparison with the raw data errors.

The results of Figure 3 show that the reduction in station density relative to that of the original dataset has little effect on the homogenization accuracy, except for in the SW region where the error increase relative to the errors of the dense dataset is notable.

In Figure 4 the residual homogenization errors of A52 and A53 are compared to those of the ACMANTv4 results in the same way as they were compared in Figure 2 for the dense test dataset.

In comparing the A52-ACMANTv4 differences between the results of Figure 4a and those of Figure 2a, one can find both similarities and notable differences. An important similarity between Figure 4a and Figure 2a is the dominance of red color indicating generally poorer results for A52 than for ACMANTv4. However, an important difference is that the regional differences are generally larger in Figure 4a. Here, the difference between A52 and ACMANTv4 results is practically zero for the WY and NE regions, the differences are higher for the SE region and the highest for the SW region. These results indicate that false breaks for the complexity of spatial-temporal climatic structures (see Table 1 in Section 3.1) may have a greater effect on the homogenization results when the station density is moderated, and they affect the accuracy of A52 more than the accuracy of ACMANTv4.

In Figure 4b the regional differences are smaller than in Figure 4a, and most pieces of the A53 results indicate small improvement in comparison with the ACMANTv4 results. However, the regional trends are exceptions, for them the ACMANTv4 results are better. When the frequency of false breaks is low (Table 1), the reduction in network size between A52 and A53 caused an improvement in the RMSE errors but worsening in the regional trend estimations. These results are concordant with the results of an earlier study where the network size effect on the homogenization accuracy was examined [62] with ACMANTv3 and with large surrogate datasets based on a WY section of the K2016 dataset.

6.3. Z-Index of Accuracy Improvement

Z-index values are examined for the averages of homogenization errors in the 13 (40) sections of the dense (moderately dense) datasets.

Table 3. Z-index (percentage) of the change in homogenization errors when comparing versions A52 and A53 of ACMANTv5 to ACMANTv4. Mean results for entire datasets. RMSE-d—RMSE of daily values; RMSE-y—RMSE of annual values; Trb—mean absolute trend bias for station series; and Reg-Trb—regional mean absolute trend bias for dataset sections.

		Z (%)
		RMSE-d	RMSE-y	Trb	Reg-Trb
Dense dataset	A52	2.1	3.7	2.2	7.4
	A53	−0.4	−0.6	−1.5	1.0
Moderately dense dataset	A52	4.3	5.7	7.9	18.2
	A53	−1.5	−2.3	−3.0	18.4

presents the results.

The reduction in network size between the A52 and A53 methods results in a small, but systematic accuracy improvement in the RMSE and station-specific trend estimations. However, the results are less clear for the regional trend bias: in dense datasets the network size reduction improved the results, while in the moderately dense dataset the regional trend estimations are almost the same with A52 and A53, and they are both poorer than with ACMANTv4. The advantage of ACMANTv4 in the regional trend estimations seems to be notable in the Z-index results, but note that the absolute differences are very small, as shown in Section 6.1 and Section 6.2.

7. Discussion

The results show generally small differences between the accuracies of the tested method versions, therefore one question is whether such small differences merit attention. I think the correct answer is yes, for two main reasons:

(i)

Homogenization methods are applied for several climate variables and for data observed under varied geographical conditions and using varied observation practices. Therefore, the differences found between homogenization efficiencies might be related to larger and more important absolute differences in climate characteristics than those for the K2016 dataset.

(ii)

Small differences in the mean errors often indicate more significant differences in the risk of committing large homogenization errors. This was demonstrated with the MULTITEST results [16,29].

The results from the K2016 dataset do not show any advantage of the pre-homogenization with combined time series comparisons, and the regional mean trend errors were found to be larger with the ACMANTv5 versions than with ACMANTv4. The likely reason for the disappointing weak results with methods including the combined time series comparison is that regional mean trend biases are generally very small in the K2016 dataset, which, however, is not rare in real observational datasets. As the accuracy of regional mean trends is a particularly important issue in climate science [29,46,47], the found problems need further research. The differences in the two experimental versions of ACMANTv5 relative to ACMANTv5.1 have the greatest influence on the homogenization of large datasets and the automatic networking procedure, while their effect is minor on previously defined small networks like most segments of the MULTITEST dataset. Therefore, the results of a comparison between ACMANTv4 accuracy and ACMANTv5 accuracy in an earlier study [18] in which the MULTITEST dataset and ACMANTv5.1 were used are considered to be usable for the joint analysis with the results of this study. Given that ACMANTv5 performed better than ACMANTv4 in most sections of the MULTITEST dataset, some slight advantage of the combined time series comparison can be concluded by the joint evaluation. The use of reduced network size in A53 is found to be more favorable than the former parameterization of automatic networking included in A52 and in previous ACMANT versions. Note, however, that these evaluations do not free from the subjectively judged importance dedicated to individual pieces of the results, and later tests might lead to other conclusions. One key issue of the further progress in relative homogenization is to learn more about the frequency and magnitude of low frequency changes in climatic gradients between time series with seemingly sufficient spatial correlations. To find reliable answers to these questions, the development of homogeneous surrogate datasets should be continued using the method started with the creation of the K2016 dataset, but further climatic zones and further climatic variables should be included.

A peculiarity of the automatic networking with A53 is that the old parameterization is used in only one break detection step of the procedure, i.e., in the second step of the combined time series comparison. There, the use of larger networks is advantageous since only relatively large breaks are searched by using a stricter significance threshold than in the other break detection steps [12]. Stricter significance thresholds could also be used in the pairwise detection step of the combined comparisons but experiments with them (not shown) did not yield better results than when the original parameterization of ACMANTv5 [18] is used.

The A53 version of ACMANT was selected as the base for further methodological developments, but later research results might modify this preference in three different ways: (i) it is possible that a more effective method of combining pairwise comparisons and the use of composite reference series will be found; (ii) future tests might indicate the superiority of the ensemble pre-homogenization method of ACMANTv4; or (iii) the method of pre-homogenization will depend on some parameters and/or previously calculated statistics for the input data in a possible future version of ACMANT. Nevertheless, regarding the available homogenization methods, both of the ACMANTv5 and ACMANTv4 methods are recommended since the inclusion of the ANOVA correction model, bivariate homogenization (when applicable) and ensemble homogenization provide the superior performance of the analyzed ACMANT versions in comparison with many other homogenization methods.

8. Conclusions

Some versions of the ACMANT homogenization method were tested with the original dense version and with a derived moderately dense version of the synthetic daily temperature dataset developed for four U.S. regions (K2016 dataset) [13]. Two experimental versions of ACMANTv5, referred to as A52 and A53, were used, and the changes in homogenization accuracy relative to the ACMANTv4 accuracy were analyzed. The main conclusions are summarized as follows:

• The found differences between ACMANTv5 accuracy and ACMANTv4 accuracy are generally small, and ACMANTv5 often gave slightly worse results than ACMANTv4.
• A reduction in network sizes reduced the RMSE and station specific trend errors of ACMANTv5, while it did not change the regional mean trend biases significantly.
• Regional mean trend biases are particularly sensitive both to the simulated climate properties of the test dataset and to the fine details of the applied homogenization method. Therefore, further improvements in the creation and use of more high-quality homogenous test datasets are needed.
• The joint analysis of the results of this study and an earlier study indicates that the inclusion of combined time series comparison in ACMANT is likely favorable.

The results of this study confirm that both of the ACMANTv4 and ACMANTv5 methods are efficient and recommended tools for the preparation of observed climatic data for climate change and climate variability studies.