Operating Cost Coverage vs. Water Utility Complaints

Abstract

In addressing the topic of decision making in the water sector, what has been suggested to be useful up to this point is benchmarking. This paper aims at exploring the relationship between the Operating Cost Coverage Index and customer complaints about water and wastewater services for 1898 water utilities in 11 countries, i.e., Australia, Bangladesh, Brazil, Lithuania, Peru, Poland, Russia, Serbia, Tanzania, Zambia, and Zimbabwe. The data used are the most recent available per country through the International Benchmarking Network for Water and Sanitation Utilities (2013–2016). In order to examine the aforementioned relationship and further reveal the key policy messages that are derived by this analysis, parametric and non-parametric group comparisons are employed. The results of those comparisons concerning the two examined indicators of this study consist of two main findings: The first sheds light on the rational argument that well performing utilities open their data to others—at least more so than those not performing as well. The second is that there could be an association between financial performance and the complaints received about water and wastewater services, but this finding requires further investigation.

1. Introduction

Performance measurement of water utilities is a complex task due to the nature of the service [1]. The reason behind this is that, in principle, the quality is related to investments and effective management, with debate in measuring economic costs [2], impacts [3,4], and service quality [5,6]. Several approaches have been employed so far for evaluating water utilities, and a number of variables have been found to affect their performance.

A crucial variable in the evaluation of water utilities is their size. Economies of scale and scope have been found applicable to water utilities in Australia [7], Israel [8], Italy [9], Portugal [10,11], England and Wales [12], and Japan [13]. Tsagarakis [14] showed that significant gains can be achieved by larger utilities by comparing the financial performance of differently sized groups for several countries (for a literature review on this topic, see the work published by Carvalho et al. [15]). However, Guerrini et al. have reported improved efficiency for wastewater services by smaller utilities in Denmark [16]. Whatever the index and evaluation methodology employed, water utilities should operate in a sustainable way involving, apart from economic and quality indicators, social, environmental, governance, and asset management aspects [17].

When data for water utility comparisons concern different countries, there is a (data) compatibility issue due to the fact that those utilities operate under different socio-cultural and political conditions [18]. However, when utilities publicize their data in the form of annual reports or report to a central agency, cross-national comparisons are feasible [19]. One major factor for cross-national [19] and national comparisons is the ownership of the water utilities [20], with private or partially private water utilities showing higher profitability compared to those fully publicly owned [21,22]. The time span is normally within one or a few years due to data limitations [23]. However, private involvement in the water industry should be implemented after institutional arrangements, especially at introductory applications [24]. Depending on effective management practices such as successful customer involvement, public and private water suppliers can perform equally well [25,26]. A good review on the impact of private participation for water projects in developing countries was developed by Estache and Philippe [27]. There are no, however, general rules that are universally applicable, since there are cases where small or public water utilities achieved effective water supply models [28].

A useful database to bridge the gap of data scarcity is the International Benchmarking Network (IBNET), which provides an online resource network of national databases covering the performance of water utilities all over the world [29]. The benefit of data openness consists in the possibility of exploring how other water utilities operate [30] within a country by measuring and comparing their performance [31,32]. The International Water Association (IWA) also has special groups that report on different performance indicators that can assist in the benchmarking process, such as the IWA Task Forces on Water Losses [33], the Specialist Group on Economics and Statistics [34], and several focus publications [35,36].

Utilizing available IBNET data, this study aims at investigating two hypotheses by exploring the statistical relationship between selected indicators: (a) Do water utilities that perform financially well monitor and provide their data publicly better than those operating not so well? (b) Do customers of financially well performing utilities complain more than customers of not-so-well performing utilities? The rest of the paper is structured as follows: Section 2 consists of the research methodology. In Section 3, the results of the data analysis are presented, and are discussed in Section 4. Section 5 consists of overall conclusions and further research suggestions.

2. Research Methodology

The data used in this paper were downloaded from the IBNET database in June 2017. For investigating the relationship between financial performance and quality of service, the two respective indicators were used. The data availability of the database poses a limitation, while the selected indicators should present variability to allow valid comparisons.

Although quality of service should be of focus in performance measurement, historically it has received limited attention [37]. A basic quality indicator is the number of complaints about the water and wastewater services recorded per year, divided by the number of water and wastewater connections expressed as a percentage (WWC), as shown in Equation (1). This indicator was selected among others reported in the database based on data availability. It typically includes the total number of customer complaints during each year. Depending on the utility’s methods of recording complaints, they may have been made by telephone, in person, or in other forms [38]. Note that connection may differ within utility or country, since it could apply to either the building or the user.

$$\mathrm{WWC}=\frac{\mathrm{Number}\mathrm{of}\mathrm{Water}\&\mathrm{Wastewater}\mathrm{Complaints}\mathrm{per}\mathrm{Year}}{\mathrm{Number}\mathrm{of}\mathrm{Water}\&\mathrm{Wastewater}\mathrm{Conections}}\times 100.$$

(1)

While it is complex to fully evaluate the financial performance of a water utility, a simple and widely used approach is comparing revenues with costs on a yearly basis. IBNET reports this indicator as the Operating Cost Coverage (OCC), and merely divides revenues per costs over a calendar year [39], as shown in Equation (2). Although quite simple, this indicator can be the basis of comparisons for a water utility over time, for water utilities within the same country, or utilities among different countries.

$$\mathrm{OCC}=\frac{\mathrm{Total}\mathrm{annual}\mathrm{operational}\mathrm{revenues}}{\mathrm{Total}\mathrm{annual}\mathrm{operating}\mathrm{costs}}$$

(2)

Operational revenues typically include total billing of water and wastewater services, connection fees, well abstraction fees, reconnection fees, and other operational revenues including subsidies, but excluding all taxes. Operating costs typically include labor (salaries, wages, pensions, other benefits, etc.), energy, and contracted out service costs, excluding depreciation and financing charges (interest and capital repayments) [40].

A first approach is to decide how good the financial performance is based on OCC categories defined by threshold values. In the IBNET blue book [41], water utilities are classified based on their OCC in three class groups, defined by the “physiological” threshold of unity (below which utilities do not cover their annual costs), and a value of 1.4 is reported to be the minimum acceptable value for ensuring good financial performance. Other values reported in the literature as acceptable range between 1.3 and 1.6 [42].

Performance indicators can assist with a comparison of utilities within the same country or can locate key messages by investigating statistically significant correlations of financial performance with other indicators [43]. Based on the available online data, two hypotheses will be investigated in this study:

• How open are water utilities with their data?/Who reports? These are the first two questions to be investigated through statistical comparisons. Can we support the rational reasoning that financially well performing utilities will be more open to their data compared to those performing not so well? To address this question, groups of financially well performing utilities need to be compared, ceteris paribus, to those performing not so well.
• What is the relationship between the financial performance of a utility and the complaints it receives?/Who complains? The complaints received in water utilities are expected to be related with technical or quality performance. However, is there any relation with the financial performance of the utility? Do utilities that perform financially well, better than others, report more complaints, compared to those utilities not performing so well?

The limitations of IBNET are well described in the Water Supply and Sanitation Performance Blue Book (pp. 14–15) [44]. The span of the time frame restricts findings to only the static view of performance. There are quality limitations, since the data are submitted by individual utilities or the utilities’ associations. Nevertheless, the quality screening assists in minimizing the obviously wrong or inconsistent entries. The voluntary nature of membership also limits the range of utilities or countries that report to IBNET. Although there is a significant representation of developing countries, there is a gap from utilities from Europe and North American countries.

3. Results

The search of the IBNET database resulted in a number of countries giving a considerable amount of utilities for the WWC indicator; the present analysis aims at making the most out of these data. The availability of the indicators per country as well as the reference year, which is the most recent year with available data, are presented in

Table A1. Data inventory.

Country	Reference Year	Number of Utilities Reporting
		OCC	WWC
Australia	2013	65	58
Bangladesh	2016	56	1
Brazil	2015	1430	596
Lithuania	2014	22	17
Peru	2014	45	44
Poland	2015	22	17
Russia	2015	80	2
Serbia	2015	111	78
Tanzania	2016	25	3
Zambia	2016	11	11
Zimbabwe	2014	31	27

. The utilities used in this analysis are not necessarily a representative sample of the respective countries’ water sector. In this study, when referring to a country, the utilities that report to the IBNET database are implied. The data employed for this study are the most readily available, while findings are restricted to the number of observations, i.e., the number of water utilities reporting this indicator.

3.1. Who Reports?

As far as the complaints for the water and sanitation services are concerned, they are related to changes in water quality or quantity [45,46] or network damages. Therefore, time series monitoring and analysis would be the most effective approach to address this issue. In the absence of time series data, statistical evidence from many utilities in one year can provide key messages on customer complaints.

Some countries (Bangladesh, Russia, and Tanzania) provided a very small amount of utilities that report the examined indicator in the IBNET database. Other countries report a small amount of utilities without WWC values (e.g., only one for Peru). These countries are, however, examined, although statistical test comparisons may not allow for high significance.

Table 1. Descriptive statistics of the complaints about water and wastewater services indicator.

Statistics	Australia	Brazil	Lithuania	Peru	Poland	Serbia	Zambia	Zimbabwe
N	58	596	17	44	17	78	11	27
Missing	7	834	5	1	5	33	0	4
Mean	0	0.2304	0.0138	0.1227	0.0085	0.0240	0.2033	0.1191
Std. Deviation	0	0.2715	0.0290	0.0853	0.0137	0.0372	0.1356	0.0701
Minimum	0	0.0000	0.0000	0.0027	0.0000	0.0000	0.0734	0.0188
25% Percentile	0	0.0336	0.0000	0.0668	0.0004	0.0038	0.0835	0.0615
Median	0	0.1288	0.0001	0.1170	0.0018	0.0099	0.1736	0.1090
75% Percentile	0	0.3132	0.0101	0.1749	0.0110	0.0267	0.3333	0.1652
Maximum	0	1.0000	0.0928	0.3996	0.0501	0.2298	0.4753	0.2544

presents the descriptive statistics of the WWC indicator per country. A country is included here if it reported for four or more utilities. The remaining countries are only included in

Table A2. Complaints about water and wastewater services.

Statistics	Bangladesh	Russia	Tanzania
N	1	2	3
Missing	55	78	22
Mean	0.0164	0.2581	0.1926
Std. Deviation	n.a.	0.1407	0.1970
Minimum	0.0164	0.1587	0.0764
25% Percentile	n.a.	0.1587	0.0764
Median	n.a.	0.2581	0.0814
75% Percentile	n.a.	n.a.	n.a.
Maximum	0.0164	0.3576	0.4201

n.a.: not applicable

.

As can be observed, mean values vary considerably by country, with the lowest and highest average among the countries examined being achieved by Australia, with zero reported complaints, and Brazil with 0.2304, respectively. On the other hand, if Australia is excluded, the median values show a lower variation, i.e., 0.0001 and 0.1288, for utilities from Lithuania and Brazil, respectively, which, considering the upper percentile and maximum values, indicate the existence of extreme values. The descriptive statistics of the utilities’ OCC, for countries included in Table 1, are analyzed in

Table 2. Operating Cost Coverage (OCC) descriptive statistics for water utilities in the examined countries.

Statistics	Australia	Brazil	Lithuania	Peru	Poland	Serbia	Zambia	Zimbabwe
N	65	1430	22	45	22	111	11	31
Mean	1.956	0.849	1.348	0.926	1.529	1.427	1.066	1.757
Std. Deviation	0.911	0.696	0.453	0.163	0.292	0.968	0.284	1.129
Minimum	0.167	0.000	0.949	0.602	1.142	0.108	0.500	0.654
25% Percentile	1.551	0.165	1.105	0.846	1.304	0.897	0.892	1.075
Median	1.830	0.945	1.264	0.908	1.469	1.077	1.052	1.490
75% Percentile	2.196	1.219	1.411	1.010	1.665	1.618	1.365	1.912
Maximum	7.254	5.528	3.164	1.654	2.392	5.490	1.437	6.046

, while the remaining countries are only included in

Table A3. Operating cost coverage descriptive statistics for water utilities.

Statistics	Bangladesh	Russia	Tanzania
N	56	80	25
Mean	1.158	0.959	1.054
Std. Deviation	0.427	0.115	0.273
Minimum	0.208	0.624	0.242
25% Percentile	0.912	0.910	0.994
Median	1.135	0.972	1.082
75% Percentile	1.392	1.036	1.196
Maximum	2.337	1.214	1.532

. Out of the 77 countries reporting this indicator in IBNET, the median country average is 0.03, and the upper quartile average is 0.11 [47].

Note that, although considerable variations can be observed for the OCC with mean values ranging from, e.g., 0.849 for Brazil to 1.956 for Australia, these data are provided by the countries’ utilities’ and, as mentioned above, are not necessarily a representative sample. Furthermore, there are utilities in the database that do not report their OCC or utilities not registered at IBNET. Out of the 128 countries reporting this indicator in IBNET, the median country average is 1.15, and the upper quartile average is 1.42 [48].

To provide statistical evidence for the discussed hypotheses, a large number of utilities and variability is necessary. Furthermore, it is essential that comparisons remain within the same legal political and economic system; otherwise, data compatibility issues may arise [18]. In this analysis, cross-national comparisons could only be safe at the exploratory descriptive statistics phase. For this reason, all hypotheses are examined at a national level individually.

Table 3. Independent sample t-test and Mann–Whitney U test comparisons for the OCC of water utilities based on whether or not they report their customers’ complaints in the IBNET database.

Country	Report	Utilities	Mean OCC	St Dev	t-Test	p	Mean Ranks	Z	p
Australia	Yes	58	1.9944	0.9205	0.981	0.330	33.72	0.889	0.374
	No	7	1.6365	0.8214			27.00
Brazil	Yes	569	1.1337	0.5440	4.332	<0.001	632,56	5.650	<0.001
	No	583	0.9769	0.6786			521,79
Lithuania	Yes	17	1.3985	0.4963	0.972	0.343	12.29	1.058	0.319
	No	5	1.1744	0.2059			8.80
Poland	Yes	17	1.5358	0.3142	0.203	0.841	11.76	0.353	0.762
	No	5	1.5049	0.2272			10.60
Serbia	Yes	78	1.4524	0.9936	0.423	0.673	56.97	0.490	0.624
	No	33	1.3671	0.9169			53.70
Tanzania	Yes	3	1.0765	0.0719	0.152	0.881	12,00	0.251	0.844
	No	22	1.0504	0.2912			13,14
Zimbabwe	Yes	27	1.8410	1.1825	1.086	0.286	16.85	1.355	0.193
	No	4	1.1860	0.3664			10.25

reports group comparisons for the parametric and equivalent non-parametric tests for the OCC of water utilities separated by whether or not they report their customers’ complaints in the IBNET database. It is evident for the utilities in Brazil that those reporting the selected quality indicator for the complaints by their customers perform financially better compared to those that do not report it by about 15.68% (t = 4.332, p < 0.001).

Valid comparisons require an adequate number of water utilities per group compared. This is not the case here for most of the countries, since one of the groups has a low number of observations (reporting the WWC for Tanzania and not reporting WWC for Australia, Lithuania, Poland, and Zimbabwe). In order for the differences among the financial performance of utilities based on whether or not they report this quality indicator to be more evident the mean values of each group considered per country are calculated, as depicted in Figure 1. It is evident that in all examined countries, apart from Peru where only one utility does not report complaints, the mean OCC of the utilities reporting the complaints of their customers is higher than those not reporting them.

It should be made clear that financial performance is not the only parameter related to the data openness of the utility. This finding is implicit information elicited by statistical comparisons of the OCC. Indeed, it is expected that there are other parameters that will affect the degree of openness in data for the water utilities apart from their financial performance. It has been demonstrated that the financial performance is also related to the size of the utility [14]. Larger utilities are often obliged, due to their legal status, to be more open in their data compared to the smaller ones, which may have fewer staff members and less access to resources that would allow for the public posting of data. In Brazil, for example, comparing the size of utilities that provide their data for the WWC with those that do not, the former have an average value of 83,014 connections, while the latter have an average of 6714 connections (t = 3.877, p < 0.001). Note, however, that only 1082 out of the 1430 of the water utilities in Brazil have entered their number of connections in IBNET.

3.2. Who Complains?

The next step is to further examine the financial performance of the utilities based on the complaints reported. Is the number of complaints higher in the well performing utilities or those that are not performing so well? To address this, the correlations between the two examined indicators are examined, where this is possible, subject to data availability. Positive significant correlation is only found for Brazil (r = 0.282, p < 0.001) and negative for Zambia (r = −0.751, p = 0.008); all correlations can be found in

Table A4. Correlations among financial performance and complaints received.

Country	N	r	p
Brazil	594	0.282	<0.001
Lithuania	17	−0,214	0.409
Peru	44	0.099	0.524
Poland	17	0.087	0.739
Serbia	78	0.016	0.886
Zambia	11	−0.751	0.008
Zimbabwe	27	−0.072	0.722

. Though the opposite signs of the Pearson coefficient could be suggested to lead to contradictory findings, it is important to note that for Brazil, the correlation was calculated based on 569 cases, while for Zambia from just 11. The results for the other countries give insignificant correlations (p ≥ 0.409), which could be attributed to the low number of reporting utilities. However, for water utilities in Peru (N = 44) and Serbia (N = 78), it is evident that there is no significant correlation (p = 0.524 and p = 0.886, respectively).

Further investigation is performed per group in Brazil enabled from the large number of utilities, as presented in

Table 4. Comparisons of the financial performance for water utilities in Brazil according to the complaints reported.

Comparisons	OCC Groups	Number of Utilities	Mean OCC	St Dev	t-Test	p	Mean Ranks	Z	p
A	<1.7	541	0.2137	0.2493	3.290	0.002	291.61	3.064	0.002
	≥1.7	55	0.3947	0.4004			366.27
B	<1.4	478	0.2030	0.2379	3.978	<0.001	285.91	3.592	<0.001
	≥1.4	118	0.3413	0.3585			349.48
C	<1	238	0.1639	0.2334	5.178	<0.001	243.50	6.359	<0.001
	≥1	358	0.2746	0.2860			335.07

. It is observed that every possible group comparison (A, B, and C for groups defined by OCC values of lower and higher than: 1.7, 1.4, and 1.0, respectively) results to the same finding, i.e., the better the financial performance, the more the reported complaints. Note that the OCC is not claimed to be the only variable affecting the number of complaints. Although reporting complaints may be affected by some cultural, geographical, or organizational variables, in Brazil the size of the utilities, i.e., the number of connections, is positively related to both the number of complaints (r = 0.096, p = 0.019) and the OCC (r = 0.092, p = 0.003).

The major statistical finding supported only by the utilities in Brazil is that those utilities that perform financially well also report more complaints compared to those performing not so well. Considering that there are several socio-cultural factors that may result in higher rates of complaints for goods and services [49,50,51], it is risky to assume any generalization, considering that there are also countries with much fewer data but result in no or opposite sign correlation. It should be emphasized that complaints are negatively related to service quality and efficiency, and whenever possible it is important to study relations among both people’s perceptions of quality and actual chemical/technical quality [7,52].

4. Discussion

Public and private water utilities have several methods of self evaluation that they may or may not practice or publicly share. There is much to gain by the sharing the data of utilities and agencies with a country’s citizens [53], while this sharing policy offers great opportunities for benchmarking toward recent environmental and climate change challenges [54]. Although the advances in information and communication technologies have facilitated the open data policies in water utilities [55], there is still much to gain by effective open data policies [56,57].

Simple rationale will lead to an attitude not only applicable to water utilities but also valid for any entrepreneurial activity. The managers of utilities that perform well, know and want others to know it as well, while they are willing to be evaluated and provide information (Figure 2). Based on the findings of this paper, it is suggested that well performing utilities in Brazil (the country with the most utilities in the database) do report their customers’ complaints—a quality of service indicator—compared to those performing not so well. Though this hypothesis is only statistically supported for Brazil, it is also visually evident by the mean comparisons of the OCC for almost all other countries included in the analysis. Further statistical evidence cannot be argued at this point, due to the low number of observations and/or different variations among the two compared groups. Note that only data from the IBNET database are used.

The number of complaints is positively related to the financial performance of the water utilities in Brazil. All other examined countries showed no evidence of statistical significance, apart from very few utilities in Zambia (N = 11), where the correlations were negative. Therefore, the second hypothesis cannot be addressed. Several more questions arise about the positive relation of the financial performance to the received complaints:

• Do financially well performing utilities receive/report more complaints or do they provide the conditions for customers to do so (a more effective complaint reporting system)?
• Do customers complain due to higher expectations from paying more?
• Is this finding only related to sociocultural and political factors, or to other performance indicators?
• Is the high financial performance related to not addressing all damages, thus resulting in more complaints?

These questions raise a fundamental question on how complaints are defined. Is it, for example, just a phone call, or a complaint of an insignificant issue? This could result in a very high variation among countries that may follow different reporting protocols. On the other hand, customers complain when they have faith on the reporting system efficiency. If the water utility is not responsive to complaints, then consumers know that it is a waste of time and thus refrain from complaining. Thus, a lower number of complaints is recorded. These questions lead to the finding that reporting the number of complaints only cannot result in uniform findings. The most reliable indicator would be the recording of the response time for efficient evolvement, but such an indicator, although found elsewhere, is not reported in IBNET. Finally, as the water sector changes in most countries from a single decision-making authority to a multi-level, polycentric governance [58], the level of customer-involvement in the decision-making process in water utilities will play a critical role in service satisfaction and complaints received.

5. Conclusions

Statistical comparisons can reveal some key messages for water utility management. This paper opens the debate on a data support basis on whether or not well performing utilities receive more complaints compared to those performing not so well. In addition, a known argument that the well performing utilities open their data is statistically evident for water utilities in Brazil, a country with a high number of utilities in the available database. This is also inferred by the available data from the other countries examined. One policy conclusion that can be advised from the findings of this paper is that, once utilities are forced to publish their data, they may start improving performance.

The proper recording of customer complaints by offering different means of communication and protocols for reporting back is essential in evaluating the quality of service. Although utilities from 128 countries report the OCC, only 77 (60%) of these countries report customer complaints. This shows limited attention in quality issues.

However, this study has limitations. The hypothesis could not be further statistically supported due to data availability, fragmented by the country data availability for the water utilities. Data availability does not allow for time series analysis or allow for flexibility on variables for regression analysis. More data and in-depth research is needed to generalize that financial performance and complaints received are related. The value of benchmarking and data variability is evident, since such aspects will work to support key policy messages for water utility managers and other decision-makers in the water sector.