Performance Indicators Framework for Assessment of a Sanitary Sewer System Using the Analytic Hierarchy Process (AHP)
Abstract
:1. Introduction
2. Research Methodology
2.1. Selection Criteria of Performance Indicators
- Management indicators (A1): evaluate the efficiency and effectiveness of personnel undertaking management activities for long-term sewer services; these indicators include functions, qualifications, and safety training.
- Operation and maintenance indicators (A2): are used for assessment the performance of the undertaking related to the operation activities and maintenance programs for SSSs.
- Service indicators (A3): determine the level of services provided to customers as well as customer satisfaction. Information on the sewer service and responses to complaints are measured to ensure quality of service.
- Environmental indicators (A4): evaluate the performance of the sewer undertaking regarding environmental impacts, including sanitary sewer overflows and infiltration/inflow. The compliance with wastewater discharge standards and regulations is also assessed.
- Financial indicators (A5): assess the financial status of the undertaking company, as well as the financial effort (such as reinvestment rate for a sewer system) to maintain the service life of the infrastructures.
2.2. Design of Survey Questionnaire for Experts’ Opinion
2.3. AHP Procedures
2.3.1. Step 1. Defining the Problem and Determining Its Goal
2.3.2. Step 2. Structuring the Decision Problem into the Hierarchy
2.3.3. Step 3. Making Pairwise Comparisons and Obtaining the Judgmental Matrix
2.3.4. Step 4. Computation of Local Weights and Consistency of Comparisons
2.3.5. Step 5. Aggregation of Weights across Various Levels to Obtain the Final Weights of Alternatives
3. Analysis of the AHP Results
4. Extraction of Key Performance Indicators for Sanitary Sewer Systems
4.1. Proposed Set of KPIs
- Provides measurements to compare the overall performance among wastewater utilities
- Provides key information needed to determine the efficiency and effectiveness of delivered services, and focuses management attention on what matters most
- Monitors asset management processes and provides a common language for communication
- Identifies potential problems and areas for improving the performance of systems
- Provides snapshots of systems’ performances focusing on goals and strategies
- Evaluates the performance of systems in accordance with the classification categories of consistent, transparent, clear, specific, auditable, more effective, and easier to implement
- Assigns responsibility and encourages accountability
- Supports decision-making and provides a way to see if the strategic plan is working
- Functions as a tool to drive desired behaviors.
- The AHP result: the selected indicators must reflect the important indicators that were evaluated by experts (as expressed in Section 3)
- The set of KPIs must include all important aspects when judging an SSS
- KPIs should adhere to SMART criteria and other criteria to be directly relevant to objectives, should be as few as necessary, should be applicable to the options under consideration, should be comprehensive, meaningful, and relevant to all stakeholders, and applicable over time.
4.2. Performance Scores for Sanitary Sewer Systems
- : Poor
- : Average
- : Good
- : Excellent.
5. Conclusions
Supplementary Materials
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- Matos, R.; Cardoso, A.; Duarte, P.; Ashley, R.; Molinari, A.; Schulz, A. Performance Indicators for Wastewater Services; IWA Publishing: London, UK, 2003. [Google Scholar]
- Danilenko, A.; Berg, C.; Macheve, B.; Mofitt, L. The IBNET Water Supply and Sanitation Blue Book 2014; World Bank: Washington, DC, USA, 2014. [Google Scholar]
- OFWAT. Key Performance Indicators—Guidance; Water Services Regulation Authority: Birmingham, UK, 2013. Available online: https://www.ofwat.gov.uk/wp-content/uploads/2015/12/prs_web_kpiprintable.pdf (accessed on 8 May 2019).
- U.S. EPA. Guide for Evaluation Capacity, Management, Operation, and Maintenance (CMOM) Programs at Sanitary Sewer Collection Systems; United States Environmental Protection Agency: Cincinnati, OH, USA, 2005.
- AWWA. Benchmarking Performance Indicators for Water and Wastewater Utilities–2013 Survey Data and Analyses Report; American Water Works Association: Denver, CO, USA, 2015. [Google Scholar]
- Alegre, H.; Baptista, J.M.; Cabrera, E., Jr.; Cubillo, F.; Duarte, P.; Hirner, W.; Merkel, W.; Parena, R. Performance Indicators for Water Supply Services (Manual of Best Practice), 2nd ed.; IWA Publishing: London, UK, 2006. [Google Scholar]
- Alegre, H.; Hirner, W.; Baptista, J.M.; Parena, R.; Cubillo, F.; Cabrera, E.; Matos, R. The IWA Systems of Performance Indicators for Urban Water Services. In Proceedings of the Workshop Views and Experience Gained Through Implementing IWA Performance Indicators Project, Melbourne, Australia, 7–12 April 2002. [Google Scholar]
- Guérine-Schneider, L.; Brunet, E. Performance Indicators for the Regulation of the Water and Sewerage Services: The French Experience. In Proceedings of the Enviro 2002 IWA World Water Congress, Melbourne, Australia, 7–12 April 2002. [Google Scholar]
- Milutinovic, B. Benchmarking and Performance Indicators in Water Supply and Wastewater Services. Master’s Thesis, Postgraduate Program in Water Resources and Environmental Management, Faculty of Civil Engineering, University of Belgrade, Beograd, Serbia, 2013. Available online: https://www.slideshare.net/BorisavMilutinovic/benchmarking-and-performance-indicators-borisav-milutinovic (accessed on 25 March 2019).
- Ministry of Environment. Sewage Statistics 2014; South Korea, 2013. Available online: http://eng.me.go.kr/eng/web/index.do?menuId=310 (accessed on 8 May 2019).
- Mels, A.R.; Nieuwenhuijzen, A.F.; van der Graaf, J.H.J.M.; Klapwijk, B.; de Koning, J.; Rulkens, W.H. Sustainability indicators as a tool in the development of new sewage treatment methods. Water Sci. Technol. 1999, 39, 243–250. [Google Scholar] [CrossRef]
- Ashley, R.; Hopkinson, P. Sewer systems and performance indicators—Into the 21st century. Urban Water 2002, 4, 123–135. [Google Scholar] [CrossRef]
- Balkema, A.J.; Preisig, H.A.; Otterpohl, R.; Lambert, F.J. Indicators for the sustainability assessment of wastewater treatment systems. Urban Water 2002, 4, 153–161. [Google Scholar] [CrossRef]
- Milman, A.; Short, A. Incorporating resilience into sustainability indicators: An example for the urban water sector. Glob. Environ. Chang. 2008, 18, 758–767. [Google Scholar] [CrossRef]
- Ioris, A.R.; Hunter, C.; Walker, S. The development and application of water management sustainability indicators in Brazil and Scotland. J. Environ. Manag. 2008, 88, 1190–1201. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhou, Q. A Review of Sustainable Urban Drainage Systems Considering the Climate Change and Urbanization Impacts. Water 2014, 6, 976–992. [Google Scholar] [CrossRef] [Green Version]
- Shinde, V.R.; Hirayama, N.; Mugita, A.; Itoh, S. Revising the existing Performance Indicator system for small supply utilities in Japan. Urban Water J. 2013, 10, 377–393. [Google Scholar] [CrossRef]
- Statistics Korea. Population Projections for Korea (2015~2065); Statistics Korea: Daejeon, Korea, 2016.
- Fenner, R.A. Approaches to sewer maintenance: A review. Urban Water 2000, 2, 343–356. [Google Scholar] [CrossRef]
- Palme, U.; Tillman, A.-M. Sustainable development indicators: How are they used in Swedish water utilities? J. Clean. Prod. 2008, 16, 1346–1357. [Google Scholar] [CrossRef]
- Saaty, T.L. The Analytic Hierarchy Process. In Encyclopedia of Biostatistics, 2nd ed.; Armitage, P., Colton, T., Eds.; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2005. [Google Scholar]
- Subramanian, N.; Ramanathan, R. A review of application of Analytic Hierarchy Process in operations management. Int. J. Prod. Econ. 2012, 138, 215–241. [Google Scholar] [CrossRef]
- Saaty, T.L. Fundamentals of Decision Making and Priority Theory with Analytic Hierarchy Process, 1st ed.; RWS Publications: Pittsburgh, PA, USA, 2000; Volume 6. [Google Scholar]
- Saaty, T.L. Decision making with the analytic hierarchy process. Int. J. Services Sci. 2008, 1, 83–98. [Google Scholar] [CrossRef]
- Kurka, T. Application of the analytic hierarchy process to evaluate the regional sustainability of bioenergy developments. Energy 2013, 62, 393–402. [Google Scholar] [CrossRef]
- Saaty, T.L. Time dependent decision-making: Dynamic priorities in the AHP/ANP: Generalizing from points to functions and from real to complex variables. Math. Comput. Model. 2007, 46, 860–891. [Google Scholar] [CrossRef]
- Parmenter, D. Key Performance Indicators: Developing, Implementing, and Using Winning KPIs, 3rd ed.; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2015. [Google Scholar]
- Vilanova, M.R.N.; Filho, P.M.; Balestieri, J.A.P. Performance measurement and indicators for water supply management: Review and international cases. Renew. Sust. Energ. Rev. 2015, 43, 1–12. [Google Scholar] [CrossRef]
- Salvetti, M. The network efficiency rate: A key performance indicator for water services asset management? In Proceedings of the 7th IWA International Conference on Efficient Use and Management of Water, Paris, France, 22–25 October 2013. [Google Scholar]
- Water Supply Policy and Economics. Key Performance Indicators for Annual Performance Reporting for Queensland Urban Water Service Providers—Definitions Guide; Water Supply Regulation of Natural Resources, Department of Natural Resources, Mines and Energy: Queensland, Australia, 2016.
- Pietrucha-Urbanik, K.; Tchórzewska-Cieślak, B. Water supply system operation regarding consumer safety using Kohonen neural network. In Safety, Reliability and Risk Analysis: Beyond The Horizon, 1st ed.; Steenbergen, R.D.J.M., van Gelder, P.H.A.J.M., Miraglia, S., Vrouwenvelder, A.C.W.M., Eds.; CRC Press: Boca Raton, FL, USA, 2014; pp. 1115–1120. [Google Scholar]
- Pietrucha-Urbanik, K.; Tchórzewska-Cieślak, B. Approaches to failure risk analysis of the water distribution network with regard to the safety of consumers. Water 2018, 10, 1679. [Google Scholar] [CrossRef]
- Mashford, J.; Marlow, D.; Tran, D.; May, R. Prediction of sewer condition grade using support vector machines. J. Comput. Civ. Eng. 2011, 25, 283–290. [Google Scholar] [CrossRef]
Main Category | Code | Major Group | Code | Sub-Group | Code | Calculation Level | Code | Unit |
---|---|---|---|---|---|---|---|---|
Management | A1 | Laws, regulations, and required manuals | M1 | Act, provisions, ordinances | C1 | Regulatory compliance | D1 | (%) |
Professional standards and manuals | C2 | Professional standards and manuals | D2 | (%) | ||||
Contract document | C3 | Prosecutions/notices issued to contractors | D3 | (%) | ||||
Sewage management recordkeeping list | C4 | Sewage management recordkeeping | D4 | (%) | ||||
Organizational structure | M2 | Work role and responsibility | C5 | Employment levels | D5 | (No./100 km sewer) | ||
Management personnel | D6 | (%) | ||||||
Technical personnel | D7 | (%) | ||||||
Employment qualification | C6 | Employment qualification | D8 | (%) | ||||
Work continuity of employee | C7 | Absenteeism | D9 | (days/100 employees/month) | ||||
Working accidents | D10 | (No./100 employee/year) | ||||||
Employee education and training | M3 | On-the-job training | C8 | On-the-job training | D11 | (hours/employee/year) | ||
Environment, health, safety (EHS) training | C9 | Environment, Health, Safety (EHS) training | D12 | (hours/employee/year) | ||||
Emergency response training and exercises | C10 | Emergency response training | D13 | (hours/employee/year) | ||||
Operation and maintenance | A2 | Operational monitoring and inspection | OM1 | Flow rate/water quality/odor monitoring | C11 | Wastewater quality monitoring | D14 | (%) |
Sewer cleaning | D15 | (%) | ||||||
Flow monitoring | D16 | (%) | ||||||
Sewer condition inspection | C12 | Sewer inspection | D17 | (%) | ||||
OM2 | Drainage system maintenance | C13 | Drainage system maintenance | D18 | (%) | |||
Manhole maintenance | C14 | Manhole maintenance | D19 | (%) | ||||
Sewer maintenance | C15 | Sewer maintenance | D20 | (%) | ||||
Pump maintenance | C16 | Pump maintenance | D21 | (%) | ||||
Information and communication facility maintenance | OM3 | Database server and security inspection | C17 | Database server and security inspection | D22 | (No./year) | ||
Electronic and communication equipment maintenance | C18 | Electronic and communication equipment maintenance | D23 | (%) | ||||
Supplies and equipment maintenance | OM4 | Supplies and equipment inventory | C19 | Supplies and equipment inventory | D24 | (%) | ||
Supplies and equipment preparedness | C20 | Supplies and equipment preparedness | D25 | (%) | ||||
Safety and protective equipment | C21 | Safety and protective equipment | D26 | (%) | ||||
Service | A3 | Sewer service information | S1 | Level of Service, and Index value | C22 | Population covered | D27 | (%) |
Adequacy of sewer system | D28 | (%) | ||||||
Notice to customer of service interruptions | C23 | Notice to customer of service interruptions | D29 | (%) | ||||
Complaint and response | S2 | Sewer service complaint | C24 | Total complaints | D30 | (No./100/year) | ||
Complaint resolution | C25 | Response to complaints | D31 | (%) | ||||
Environment | A4 | Sanitary sewer overflows | E1 | SSOs frequency | C26 | SSOs frequency | D32 | (No. SSO events/100km) |
SSOs volume | C27 | SSOs inspection volume | D33 | (%) | ||||
Infiltration/Inflow | E2 | Infiltration volume | C28 | Infiltration volume | D34 | (m3/km/year) | ||
Inflow volume | C29 | Inflow volume | D35 | (m3/km/year) | ||||
Finance | A5 | Financial soundness | F1 | Debt to total asset value | C30 | Debt to total asset | D36 | (%) |
Operation and service cost | F2 | Operation cost to annual expenditures | C31 | Operating ratio | D37 | (%) | ||
Customer service cost to annual expenditures | C32 | Customer service cost per account | D38 | ($/account) | ||||
Capital investment plan | F3 | Budget planning | C33 | Current ratio | D39 | (%) | ||
Asset turnover ratio | D40 | (%) | ||||||
Medium and long-term budget and business planning | C34 | Return on asset | D41 | (%) |
A1 | A2 | A3 | A4 | A5 | Eigenvalues (Weights) | |
---|---|---|---|---|---|---|
A1 | 1 | 0.317 | 0.895 | 0.915 | 0.899 | 0.1310 |
A2 | 3.153 | 1 | 3.065 | 3.348 | 2.730 | 0.4298 |
A3 | 1.117 | 0.326 | 1 | 1.741 | 1.205 | 0.1681 |
A4 | 1.093 | 0.299 | 0.574 | 1 | 1.347 | 0.1365 |
A5 | 1.113 | 0.366 | 0.830 | 0.742 | 1 | 0.1346 |
CR | 0.0131 |
C13 | C14 | C15 | C16 | Eigenvalues (Weights) | |
---|---|---|---|---|---|
C13 | 1 | 1.213 | 0.509 | 0.491 | 0.1770 |
C14 | 0.825 | 1 | 0.624 | 0.517 | 0.1718 |
C15 | 1.966 | 1.601 | 1 | 1.104 | 0.3250 |
C16 | 2.035 | 1.936 | 0.906 | 1 | 0.3263 |
CR | 0.0028 |
Level 1 | Level 2 | Level 3 | Ranking | |||||
---|---|---|---|---|---|---|---|---|
Criteria | Weight | Major Group | Gross Weight | Sub-Group | Gross Weight | Level 1 | Level 2 | Level 3 |
Management (A1) | 13.1% | Laws, regulations, and required manuals | 0.032 | Act, provisions, ordinances | 0.006 | 5 | 13 | 33 |
Professional standards and manuals | 0.005 | 34 | ||||||
Contract document | 0.011 | 30 | ||||||
Sewage management recordkeeping list | 0.010 | 31 | ||||||
Organizational structure | 0.044 | Work role and responsibility | 0.023 | 10 | 17 | |||
Employment qualification | 0.010 | 32 | ||||||
Work continuity of employee | 0.011 | 29 | ||||||
Employee education and training | 0.055 | On-the-job training | 0.026 | 6 | 16 | |||
Environment, health, safety (EHS) training | 0.012 | 27 | ||||||
Emergency response training and exercises | 0.017 | 22 | ||||||
Operation and maintenance (A2) | 43.0% | Operational monitoring and inspection | 0.146 | Flow rate/water quality/odor monitoring | 0.056 | 1 | 2 | 5 |
Sewer condition inspection | 0.090 | 1 | ||||||
Sewer system maintenance | 0.186 | Drainage system maintenance | 0.033 | 1 | 13 | |||
Manhole maintenance | 0.032 | 14 | ||||||
Sewer maintenance | 0.060 | 4 | ||||||
Pump maintenance | 0.061 | 3 | ||||||
Information and communication facility maintenance | 0.055 | Database server and security inspection | 0.035 | 7 | 12 | |||
Electronic and communication equipment maintenance | 0.021 | 18 | ||||||
Supplies and equipment maintenance | 0.043 | Supplies and equipment inventory | 0.017 | 11 | 21 | |||
Supplies and equipment preparedness | 0.012 | 28 | ||||||
Safety and protective equipment | 0.014 | 26 | ||||||
Service (A3) | 16.8% | Sewer service information | 0.049 | Level of service, and index value | 0.030 | 2 | 9 | 15 |
Notice to customer of service interruptions | 0.019 | 20 | ||||||
Complaint and response | 0.119 | Sewer service complaint | 0.045 | 3 | 6 | |||
Complaint resolution | 0.074 | 2 | ||||||
Environment (A4) | 13.7% | Sanitary sewer overflows | 0.053 | SSOs frequency | 0.036 | 3 | 8 | 11 |
SSOs volume | 0.017 | 23 | ||||||
Infiltration/Inflow | 0.084 | Infiltration volume | 0.040 | 4 | 10 | |||
Inflow volume | 0.044 | 7 | ||||||
Finance (A5) | 13.5% | Financial soundness | 0.042 | Debt to total asset value | 0.042 | 4 | 12 | 9 |
Operation and service cost | 0.062 | Operation cost to annual expenditures | 0.042 | 5 | 8 | |||
Customer service cost to annual expenditures | 0.020 | 19 | ||||||
Capital investment plan | 0.031 | Budget planning | 0.015 | 14 | 25 | |||
Long-term budget and business planning | 0.016 | 24 |
Key Performance Indicator | Code | Unit | Concept | |
---|---|---|---|---|
KPI1 | On-the-job training | D11 | (hours/employee/year) | On-the-job training = [(Number of training hours during a year)/(total number of employees)] |
KPI2 | Wastewater quality monitoring | D14 | (%) | Wastewater quality monitoring = [Total number of tests related to wastewater quality (i.e., BOD, COD, TOC…) that are carried out/total number of tests related to wastewater quality (i.e., BOD, COD, TOC…) required by applicable standards or legislation] × 100 |
KPI3 | Sewer cleaning | D15 | (%) | Sewer cleaning = (Length of sewers cleaned/total sewer length) × 100 |
KPI4 | Flow monitoring | D16 | (%) | Flow monitoring = (Number of flow metering performed at locations within sewer system/total number of flow metering locations within sewer system required by operation strategy of the system) × 100 |
KPI5 | Sewer inspection | D17 | (%) | Sewer inspection = (Length of sewers inspected/total length of sewers network) × 100 |
KPI6 | Sewer maintenance | D20 | (%) | Sewer maintenance = (Length of defective sewers rehabilitated, renovated, replaced/total sewer length) × 100 |
KPI7 | Pump maintenance | D21 | (%) | Pump maintenance = (Number of pumps replaced, renewed, renovated, or repaired/total of pumps) × 100 |
KPI8 | Response to complaints | D31 | (%) | Response to complaints = [(Total number of responses to complaints)/(total number of complaints related to sanitary sewer system)] × 100 |
KPI9 | SSOs frequency | D32 | (No. SSO events/100km) | SSOs frequency = (Total number of SSOs events that occurred during a year of sanitary sewer systems × 100/total km of sewer collection system) |
KPI10 | Infiltration volume | D34 | (m3/km/year) | Infiltration volume = Volume of water entering sewers from groundwater during a year/total sewer length of sanitary sewer system |
KPI11 | Inflow volume | D35 | (m3/km/year) | Inflow volume = Volume of water entering sewers from erroneous connection during a year/total sewer length of sanitary sewer system |
KPI12 | Debt ratio | D36 | (%) | Debt ratio = (Total liabilities/total assets) × 100 |
KPI13 | Operating ratio | D37 | (%) | Operating ratio = (Total O&M costs/total operating revenue) × 100 |
Code | Key Performance Indictor | Gross Weight (%) | Weight (%) |
---|---|---|---|
Management (A1) | |||
KPI1 | On-the-job training | 2.60 | 4.55 |
Operation & Maintenance (A2) | |||
KPI2 | Wastewater quality monitoring | 1.87 | 3.27 |
KPI3 | Sewer cleaning | 1.87 | 3.27 |
KPI4 | Flow monitoring | 1.87 | 3.27 |
KPI5 | Sewer inspection | 9.00 | 15.76 |
KPI6 | Sewer maintenance | 6.00 | 10.51 |
KPI7 | Pump maintenance | 6.10 | 10.68 |
Service (A3) | |||
KPI8 | Response to complaints | 7.40 | 12.96 |
Environment (A4) | |||
KPI9 | SSOs frequency | 3.60 | 6.30 |
KPI10 | Infiltration volume | 4.00 | 7.01 |
KPI11 | Inflow volume | 4.40 | 7.71 |
Finance (A5) | |||
KPI12 | Debt to total asset | 4.20 | 7.36 |
KPI13 | Operation ratio | 4.20 | 7.36 |
Sum | 57.10 | 100.00 |
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Nam, S.-N.; Nguyen, T.T.; Oh, J. Performance Indicators Framework for Assessment of a Sanitary Sewer System Using the Analytic Hierarchy Process (AHP). Sustainability 2019, 11, 2746. https://doi.org/10.3390/su11102746
Nam S-N, Nguyen TT, Oh J. Performance Indicators Framework for Assessment of a Sanitary Sewer System Using the Analytic Hierarchy Process (AHP). Sustainability. 2019; 11(10):2746. https://doi.org/10.3390/su11102746
Chicago/Turabian StyleNam, Seong-Nam, Thao Thi Nguyen, and Jeill Oh. 2019. "Performance Indicators Framework for Assessment of a Sanitary Sewer System Using the Analytic Hierarchy Process (AHP)" Sustainability 11, no. 10: 2746. https://doi.org/10.3390/su11102746
APA StyleNam, S.-N., Nguyen, T. T., & Oh, J. (2019). Performance Indicators Framework for Assessment of a Sanitary Sewer System Using the Analytic Hierarchy Process (AHP). Sustainability, 11(10), 2746. https://doi.org/10.3390/su11102746