Next Article in Journal
Modeling/Simulation of the Dividing Wall Column by Using the Rigorous Model
Next Article in Special Issue
Scheduling of Energy-Integrated Batch Process Systems Using a Pattern-Based Framework
Previous Article in Journal
Combining Microwave Pretreatment with Iron Oxide Nanoparticles Enhanced Biogas and Hydrogen Yield from Green Algae
Previous Article in Special Issue
Dynamic Performance Assessment of Primary Frequency Modulation for a Power Control System Based on MATLAB
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Integration Multi-Model to Evaluate the Impact of Surface Water Quality on City Sustainability: A Case from Maanshan City in China

School of Information and Statistics, Guangxi University of Finance and Economics, Nanning 530003, China
*
Author to whom correspondence should be addressed.
Processes 2019, 7(1), 25; https://doi.org/10.3390/pr7010025
Submission received: 23 November 2018 / Revised: 27 December 2018 / Accepted: 2 January 2019 / Published: 8 January 2019
(This article belongs to the Special Issue Design and Control of Sustainable Systems)

Abstract

:
Water pollution is a worldwide problem that needs to be solved urgently and has a significant impact on the efficiency of sustainable cities. The evaluation of water pollution is a Multiple Criteria Decision-Making (MCDM) problem and using a MCDM model can help control water pollution and protect human health. However, different evaluation methods may obtain different results. How to effectively coordinate them to obtain a consensus result is the main aim of this work. The purpose of this article is to develop an ensemble learning evaluation method based on the concept of water quality to help policy-makers better evaluate surface water quality. A valid application is conducted to illustrate the use of the model for the surface water quality evaluation problem, thus demonstrating the effectiveness and feasibility of the proposed model.

1. Introduction

It has long been recognized that water pollution has negative impacts on human health and ecosystems [1]. Regarding human health, the most immediate and most severe impact is the lack of improved sanitation, associated with the lack of safe drinking water, which currently affects more than one third of the world’s population [2]. Water pollution is caused by industrial and anthropogenic activities, such as industrial accidents [3,4], urbanization [5,6] and natural phenomena like soil erosion [7,8]. Additional threats include, for example, exposure to pathogens or chemical toxicants via the food chain (e.g., as a result of irrigating plants with contaminated water and of the bioaccumulation of toxic chemicals by aquatic organisms, including seafood and fish) or during recreation (e.g., swimming in contaminated surface water) [9,10]. Serious water pollutants will cause different degrees of loss to industry. Therefore, the prevention and control of water pollution is a top priority in all of life and industry [11].
To effectively solve the problem of water pollution at several spatial scales, water quality modeling is of utmost relevance for evaluation. A Multiple Criteria Decision-Making (MCDM) model is developed in this case for evaluating water quality. The measurement of water quality is one way of comparing degrees of water pollution [12]. There have been several works devoted to evaluating water pollution. For example, Reference [13] used MCDM techniques to present different aspects of water pollution control and reported the results of a case study for developing a master plan for water resources pollution control in Isfahan Province in Iran. Reference [14] reported the application of fuzzy MCDM for ranking different types of industries based on water pollution potential in the State of Gujarat, India. Fuzzy analytic hierarchy process, as one of the most versatile MCDM methods, was used for managerial decision-making in complex surface water pollution situations with multiple and varied measures [15]. An alternative evaluation index system was developed to establish the priorities of a range of water pollution alternatives using MCDM techniques [16].
However, existing methods may obtain different evaluation results in the same case, so it is difficult for policy-makers to decide which method to employ. In this circumstance, ensemble vote, as a kernel of ensemble learning, can represent a good assistant to solve the multi-MCDM model problem [17]. Ensemble learning can not only solve the problem of evaluation result non-conformity in water pollution, but also solve water pollution environmental management issues.
MCDM methods have been applied to a wide range of application areas. For instance, the Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) is used primarily in the manufacturing, industry, and government sectors. Analytic Hierarchy Process (AHP) is used for evaluating weight derivation and it is still the most popular method despite some criticism in recent years [18]. Meanwhile, the hybrid learning style was applied in research suggesting that an integrated TOPSIS model could be used to solve the truck selection problem of a land transportation company [19]. A novel conjunctive MCDM approach that combines AHP and TOPSIS was used to develop an innovation evaluation system for location selection [20]. Similarly, a hybrid ensemble approach was used in a suppliers’ selection problem, applying ensemble methods to obtain a better predictive performance than that achieved by an individual algorithm [21]. A novel hybrid ensemble learning paradigm integrating ensemble empirical mode decomposition was proposed for nuclear energy consumption forecasting, and empirical results demonstrated that the ensemble learning paradigm could outperform some forecasting models in both prediction and directional forecasting [22]. Furthermore, water quality is of great importance to our daily lives and is forecasted by using a multi-task multi-view learning method to blend multiple datasets from different domains [23]. The results of this evaluation are inconsistent between different models even under the same cases. Therefore, the proposed ensemble learning paradigm is used for the evaluation of water quality in this paper.
To address the aforementioned concerns, this paper proposes hybrid ensemble learning for a data-mining evaluation framework, which combines the TOPSIS [24], Grey Relational Analysis (GRA) [25], AHP [26], and Takagi-Sugeno (TS) fuzzy neural network [27,28].
The main objective of this work is to select and combine some algorithms of optimum configuration for an integration of multiple MCDM models in a framework based on multi-criteria evaluation. The main motivation is to evaluate water quality in six sites. Related work is discussed in Section 2. Section 3 proposes the frame and the methodology. The results and discussion are given in Section 4. The conclusions of this work are summarized in Section 5.

2. Related Work

Generally, most of the criteria required in MCDM cannot be evaluated accurately, since it is impossible to obtain precise data concerning the assessment of the decision makers. Moreover, some of the criteria are only evaluated subjectively, leading to incorrect results [29]. MCDM is widely used in many evaluation cases, and it is used in the evaluation of water quality in our research. Therefore, we focus on comparing water quality in different sites to compare the water pollution degree of different sites according to the required water criteria [30]. Water quality evaluation refers to the understanding of water quality conditions, which is a complex problem involving multiple disciplines. Many countries established relevant standards for different types of water systems. Water quality standards for surface waters are established by foundation of the water quality-based pollution control schemes under the Clean Water Act (CWA) in the USA [31]. Meanwhile, China promulgated the surface water quality standards (GB3838-2002) for the surface water quality evaluation of rivers and lakes in 2002 [32]. Evaluation theory and evaluation methods have enjoyed great developments in the recent years. For instance, the index evaluation method [33], fuzzy evaluation method [34], Grey theory evaluation method [35], and neural network evaluation method have been put forth by different researchers [36].
As an active research area, several evaluation methods have been proposed. For example, TOPSIS is used to evaluate the nature of green supplier selection. This is a complex multi-criteria problem, including both quantitative and qualitative factors, and there may be conflicts and uncertainness. The identified components are integrated into a novel hybrid fuzzy MCDM model that combines the TOPSIS with the Analytical Network Process [37]. Meanwhile, the hybrid learning style is applied, and some research has suggested integrating the TOPSIS model to help the industrial practitioners with performance evaluation in a fuzzy environment [38]. Similarly, a novel conjunctive MCDM approach that combines AHP and TOPSIS order was employed to develop an innovation support system that considers the interdependence of higher education institutes to comprehensively evaluate their innovation performance [39]. Another study presented the application of multi-criteria evaluation in the selection of an optimal allocation for an air quality model [40], which was developed to support managers in exploring the strengths and weaknesses of each alternative. This approach identifies a preferred result based on the integration of Bayesian networks and fuzzy logic to rank and evaluate suppliers [41].
MCDM and integrating machine learning algorithms have been developed in recent years, including some hybrid methodology techniques that have effectively been used to conduct multi-attribute inventory analyses. For instance, naive Bayes, Bayesian network, artificial neural network (ANN), and support vector machine (SVM) algorithms [42], they have successfully dealt with inventory classification problems. Financial risk prediction is used by MCDM methods (e.g., TOPSIS) to develop a two-step approach to evaluate classification algorithms, and results show that linear logistics, Bayesian network, and ensemble methods are the top-three classifiers [43]. In the assessment of machine learning algorithms, selecting the appropriate classifier from the list of available candidates is a time-consuming and challenging task. Therefore, an accurate MCDM methodology evaluates and ranks classifiers based on experience, allowing end-users or experts to select the top-ranked classifier for their applications to learn and build classification models for their specific purposes [44].

3. Methods

To demonstrate the above concerns, we propose a hybrid ensemble multi-model for a data-mining evaluation framework combining GRA, TS Fuzzy Neural Network, TOPSIS, and AHP, as shown in Figure 1. We know from Figure 1 that our framework consists of other key phases, including indicator preprocessing, a weights model, and an evaluation training model. In this work, the main contribution is shown in node 3, which have solved different results from multiple model evaluation, that is to say, a consistent result is obtained by ensemble vote. We describe the phases in detail in the following sections.

3.1. Ensemble Learning Approach

Ensemble learning is a learning task by building and combining multiple learners [45]. There are two main advantages in this method. One is multiple training machines which can achieve the same performance if the learning task is large from the perspective of statistics. Also, this multiple model would reduce risk, because it may not lead to the wrong selection compared to a single model. The other one is learning algorithms, which tend to fall into the local minimum, and lead to poor generalization performance from a computational perspective. Meanwhile, a combination of multiple models and learning algorithms can reduce the risk of falling into a bad local minimum. In addition, some learning tasks may not satisfy the current hypothesis learning algorithm, and a single model would definitely be invalid in this case. By combining with multiple models, the corresponding hypothesis space has a better expanded learning better. Therefore, the ensemble learning is often adopted to improve the overall accuracy of regression and classification methods [46]. We used the ensemble vote integration results of a multi-MCDM model evaluation to obtain a more accurate result. The given ensemble includes T model h 1 , h 2 , , h T , where the h t output is h t ( x ) in training set x. The voting is involved in the testing stage, when the independent prediction is combined to predict an invisible instance. Some popular methods of voting include equal voting, weighted voting, and naive Bayesian voting. This paper employs the weighted voting method, in which the weights of each models are the different, as shown in Equation (1).
H ( x ) = 1 T t = 1 T ω t h t ( x )
where h t ( x ) is a single training model result, ω is the model result weights, H ( x ) is the ensemble value of each model’s result, and T is the count.

3.2. Evaluation Based on TS Fuzzy Neural Network System

ZadehLA, an American cybernetics expert at the University of California, pioneered the concept of fuzzy sets in 1965 [47]. Fuzzy theory has captured the characteristics of the ambiguity of human thinking and can thus be used to solve conventional problems.
TS fuzzy systems have a very strong self-adaptive ability, which can automatically update and modify the membership function of fuzzy subsets. Therefore, the error of a model is modified slowly in the running process, and the result is relatively better. A TS fuzzy system has a set of “if-then” rules to be defined. Concerning the rules for the R , the fuzzy reasoning is as follows:
R i : If x 1 is A 1 i , x 2 is A 2 i , , x n is A n i , then y i = p 0 i + p 1 i x 1 + + p n i x n ,
where i = 1 , 2 , 3 , , m , j = 1 , 2 , 3 , , n , and so on, hereinafter; A j i is the fuzzy sets for the fuzzy systems; p j i is the parameters of the fuzzy system; and y i is the output according to the fuzzy rules. If the input part (i.e., “if”) is fuzzy, then the output part (i.e., “then”) is determined.
For input value x = [ x 1 , x 2 , , x n ] , the membership of each input variable is x j according to fuzzy rules is:
μ A j i = e ( x j c j i ) 2 b j i , i = 1 , 2 , , m , j = 1 , 2 , , n ,
where c j i , b j i represent the membership function center and width, respectively; n is the input parameters; m is the number of fuzzy subsets.
The calculation of the fuzzy degree of membership, fuzzy operator for the multiplication operator is as follows:
w i = μ A j i ( x 1 ) × μ A j i ( x 2 ) × × μ A j i ( x n ) .
According to the fuzzy results, to compute the model output value y i :
y i = i = 1 m w i ( p 0 i + p 1 i x 1 + + p n i x n ) i = 1 m w i .
However, model is lack of ability to learn in the application of the fuzzy comprehensive evaluation, due to complex optimization of the model parameters. The ANNs have self-learning, ability of self-organization and self-adaptation. If the combining them, they can effectively play their respective advantages.
To obtain the evaluation result, the back-propagation (BP) neural network combined with the TS fuzzy system for training was employed. Figure 2 displays a neural network training procedure.
There is also a θ k (or threshold b k = θ k ). The above can be expressed mathematically:
u k = j = 1 n w k j x j , v k = u k θ k , y k = ϕ ( v k ) ,
where x 1 , x 2 , x 3 , , x n , are input signals; w k 1 , w k 2 , w k 3 , , w k p , are the weights of neurons k; u k is a linear combination; θ k is a threshold; ϕ ( ) is the activation function; and y k is the output of neurons k.
The TS Fuzzy Neural Network is divided into the input layer, the fuzzy classification layer, fuzzy programming to calculate the layer, and output layer. The input layer relates to the input vector x i , which is the same as the number of nodes and the dimensions of the input vector. The fuzzy classification layer uses the fuzzy degree of the membership function of input values to obtain the fuzzy membership value. The layers of fuzzy rules are based on the fuzzy multiplication formula. The output layer uses this formula to calculate the output of the fuzzy neural network. The learning algorithm of the fuzzy neural network is as follows.
(1)
The error computing
e = 1 2 ( y d y c ) 2 ,
where y d is the network expectation output; y c is the actual output of the network; e is the actual output of the error.
(2)
The modified coefficient
p j i ( k ) = p j i ( k 1 ) α e c i j ,
e p j i = ( y d y e ) 2 w i i = 1 m w i x j ,
where p j i ( k ) is the coefficient of the neural network; α is the network learning rate; x y is the network input parameters; and w i is the product of the degree of membership of input parameters.
(3)
The modification parameter
c j i ( k ) = c j i ( k 1 ) β e c i j ,
b j i ( k ) = b j i ( k 1 ) β e b i j ,
where c j i , b j i represent the membership function center and width, respectively.

3.3. Evaluation Based on TOPSIS

TOPSIS was first proposed by Hwang and Yoon in 1981 and developed by Chen and Hwang in 1992 [39]. The TOPSIS model was used to evaluate the characteristics of the comprehensive evaluation index system of water quality and the purpose of evaluation in this paper. The basic idea of the TOPSIS method is to calculate the distance between the best scheme and the worst scheme in the continuous time series of samples and use the relative degree of the ideal solution as the standard of comprehensive evaluation. The TOPSIS method is detects the evaluation object and the optimal solution, as well as the worst solution of the distance to sort. It is best when the evaluation object is closest to the optimal solution and as far away as possible from the worst solution, otherwise it is will not be optimal. Firstly, we standardized the original data sets. Then R = r i j m × n was employed as the for normalized dimensionless data matrix, where r i j = x i j i = 1 n x i j 2 , i = 1 , 2 , , m ; j = 1 , 2 , , n .
Secondly, we determined the positive ideal and negative ideal solutions. The best and worst values are respectively represented by Y + and Y . Then:
Y + = { ( m a x i r i j | j n ) } = { y 1 + , y 2 + , , y n + } , Y = { ( m i n i r i j | j n ) } = { ( y 1 , y 2 , , y n } .
Thirdly, we determined the relative closeness to the ideal solution. The relative closeness C i to the ideal solution can be expressed as follows:
C i = D i D i + + D i ,
where D i + and D i are the separation of each alternative from the positive ideal solution and the negative ideal solution, respectively. Each is D i + = j = 1 n ( r i j y j + ) 2 , D i = j = 1 n ( r i j y j ) 2 , i = 1 , 2 , , m . The larger the C i , the closer the sample is to the optimal sample point.
Fourth, in this paper, the fuse neural network and entropy method was used to determine the weights of the indicators. The main idea can be summarized by the Reference [24].

3.4. Evaluation Based on Grey Relational Analysis

The Grey system theory has been widely applied in various fields [25], and has been proven to be useful in dealing with information on poverty, incompleteness, and uncertainty. We used GRA to evaluate water quality. Firstly, we determined the evaluation matrix, then determined each indicator’s weights, as shown in Section 3.3. We then calculated the Grey relational coefficient:
ξ ( x 0 j , x i j ) = min Δ i j + ρ max Δ i j Δ i j + ρ min Δ i j ,
where Δ i j = x 0 j x i j and ξ i ( x 0 j , x i j ) are the Grey relational coefficients between x i j and x 0 j ; ρ is the distinguishing coefficient, ρ [ 0 , 1 ] . Finally, the Grey weighting relational was calculated,
r X 0 , X i = j n w j ξ ( x 0 j , x i j ) ,
where w is the weight of attribute j; while r X 0 , X i is the Grey relational grade between X i and X 0 , which represents the level of correlation between the reference sequence and the comparability sequence.

3.5. Evaluation Based on AHP

AHP was put forth for operations research by Professor Saaty at the University of Pittsburgh in the 1970s [48]. The main indicators of this method reflect the nature of complicated decision-making problems. This process conducts an in-depth analysis of the influencing factors and their relationship with the base station. It also involves less quantitative information to make the decision-making process of mathematical thinking, for the multi-objective and multi-criteria or non-structure decision-making method of simple and complex decision-making problem. It is difficult to completely apply a quantitative decision-making model or method to a complex system.
AHP can be succinctly summarized for water quality modeling as follows: structuring the decision hierarchy of interrelated decision elements, collecting input data by pairwise comparisons of decision elements, evaluating the consistency of managerial judgements, and applying the eigen-vector method to compute relative weights.
Concerning the details of the process reported in the literature [49,50], the main difference that arises is establishment of the matrix. We determined the matrix, in accordance with the theory of scale relation and the relationships between indicators, as shown in Table 1. Similarly, we aimed to overcome the difficulty and discrimination of only expert scoring using this scale.

4. Case Study

The numerical experiment was performed with surface water quality data. The MCDM method was particularly pragmatic for visualizing the structure of complex causality. MCDM is a comprehensive approach to establishing and analyzing structural models that contain causal relationships among complex factors. It can clearly reveal the standard causality when measuring a problem. MCDM is a good technique for evaluating problems, and the relationship of systems is generally given by crisp values in building a structural model, as shown in node 3 of Figure 1.

4.1. Indicator Description and Data Preprocess

An environmental system is a complex system; therefore, water quality evaluation must also be a result of a multi-factor interaction. The impact of different factors on the degree of pollution varies, and that which has the greatest impact on the role of water pollution is known as the “main factor”. We thus needed to collect data of different action factors, analyze the data, and introduce the quantification index to evaluate the influence degree. According to this quantitative index, we could determine the water pollution degree shadow size. Therefore, based on the relevant water science data, we identified several factors that have a significant impact on water quality.
The relevant indicators surface water quality standard (i.e., GB3095-2002 standard [51]) from Ministry of Environmental Protection of China, combined with indicator selection for water quality. The final selection indicators were: D i s s o l v e d O x y g e n ( F 1 ) , B O D 5 ( F 2 ) , M a n g a n a t e ( F 3 ) , A m m o n i u m ( F 4 ) , N i t r i t e N i t r o g e n ( F 5 ) , V o l a t i l e P h e n o l ( F 6 ) , C y a n i d e ( F 7 ) , T o t a l M e r c u r y ( F 8 ) , T o t a l A r s e n i c ( F 9 ) , and H e x a v a l e n t C h r o m i u m ( F 10 ) . The standards are shown in Table 2, in which is Level 1 is best, Level 5 is worst.
Ten indicators were chosen to be measured for the determination of the best water quality. However, positive and negative values represent different meanings. For instance, D i s s o l v e d O x y g e n is a positive indicator, while B O D 5 is a negative indicator. Therefore, we choose Equations (16) and (17) to normalize the positive and negative values. Data preprocessing was necessary and completed using two methods. The first was the max-minimum method that is used to train the weights, and the second was the standard method that is used to calculate the comprehensive score [52].
Positive indicators:
x i j = u i j min ( u i j ) max ( u i j ) min ( u i j )
Negative indicators:
x i j = max ( u i j ) u i j max ( u i j ) min ( u i j )
where u i j represents the original data, x i j represents the data after normalization.

4.2. Study Area

Ma’anshan, also colloquially written as Maanshan, is a prefecture-level city in the eastern part of Anhui province in Eastern China. It also is an industrial city stretching across the Yangtze River. Maanshan’s rivers mainly include the Ma’anshan Reach of the Yangtze River, Quarry River, Yushan River, Cihu River, Yushan Lake, and Suoxi River, corresponding to sites 1–6 respectively, all of which belong to the Yangtze River system, and samples were correct from surface water. Location maps are shown in Figure 3.

4.3. Training Weights

Firstly, we used the entropy calculation of the initial weights of each indicator to calculate the values. Next, we calculated the feature value of each indicator, as shown in line A of Table 3. We obtained initial weights by the entropy method, as shown in line B of Table 3. Because none were close to the threshold value, the results did not improve entropy method, and the weights values shown in line C of Table 3 are those obtained by training using the neural networks.
We found that the Std (standard deviation) was 0.0416, shown in line B of Table 3. A Std of 0.0208 was found in line C of Table 3; this advantage from variance result improved after training by neural networks.
Similarly, we computed four method weights, as shown in Table 4.

4.4. TOPSIS Evaluation

Similarly, based on the TOPSIS evaluation scoring system, we calculated the surface water quality scores in each study site, and the comprehensive evaluation scores represented the pre-selection programs. The weighted decision matrix V = ( v i j ) m × n was constructed after determining the weights, and the results are shown in Table 5.
Finally, the positive and negative ideal values, D i + and D i , as well as the optimal sample point C are shown in Table 6. From Table 6, the analysis results show that A 5 represents the most serious level of water pollution, and A 1 represents best water quality.

4.5. GRA Evaluation

GRA is an impact evaluation model used to measures the degree of similarity or difference between two sequences based on the relationship level. The proposed methodology applied GRA to rank water quality with respect to urban water quality [53].
Similarly, we obtained an evaluation result, shown in Table 7, using the GRA method. We know from Table 7 that A 5 represents the most serious water pollution, and A 1 is the best water quality.

4.6. TS Fuzzy Neural Networks Evaluation

In this real-world situations, crisp values are insufficient. Many evaluation criteria are really imperfect and probably influenced by uncertain factors. Fuzzy method is used by many researchers in the literature, as considering human judgement of preferences is often unclear and hard to estimate by exact numerical values. We constructed the fuzzy neural network according to the dimensions of the training samples, and then determined the number of inputs/outputs of the fuzzy neural network and the fuzzy membership function number. Finally, we built membership function using Gauss the function fuzzy reasoning and sum-product. The function center value b and function witch value c are shown in Figure 4 and Figure 5, respectively. The ambiguity resolution used the weighted average method.
The normalized data was used for the training neural network, and we then used the trained fuzzy neural network to comprehensively evaluate the surface water quality. Results and rankings are shown in Table 8.

4.7. AHP Evaluation

One of the advantages of AHP is the possibility to evaluate quantitative and qualitative criteria and alternatives on the same preference scale [54]. Similarly, we used AHP to evaluate all the indicators of surface water quality. In particular, a decision matrix V = ( v i j ) n × n was first constructed, as shown in Table 9, by the weights size from Table 3, which does not show a diagonal matrix. The final C I = 0.0126 , obtained through a consistency test, and C R = 0.0084 < 0.1 . Therefore, its mean value met the requirements.
Next, we constructed the matrix for each indicator. The indicator D i s s o l v e d is shown as an example in Table 10.
C I = 0 , as obtained through a consistency test, and C R = 0 < 0.1 , so its mean value also met requirements. Then, the eigenvalues of the criteria layer were found to be 0.2381, 0.2857, 0.1905, 0.0952, 0.0476, and 0.1429. Next, we computed the matrix of all evaluation indicators at criteria layer, as shown in Table 11.
As show in Table 12, final values from high quality to low quality were ordered as follows: A 1 > A 2 > A 3 > A 6 > A 4 > A 5 .
Consistent with the abovementioned final ranking, we found the order of A 1 > A 2 > A 3 > A 6 > A 4 > A 5 by weighted voting from Table 6, Table 7 and Table 8 and Table 12 according to the ensemble vote, as shown in Table 13.

4.8. Discussion

The comprehensive use of the ensemble multi-MCDM method was applied to the trend analysis of surface water quality to provide effective support for governmental implementation of management strategies for reservoir water resources. After detecting the similarity in surface water quality between different sampling sites, a representative sampling site is selected to optimize from each sampling group. For example, the surface water quality of the Ma’anshan Reach of the Yangtze River was found to be best from the results of Table 13, so it may be used as a life water source. The spatial pattern of water quality can also help local governments understand the pollution status of the managed areas and protect their respective aquatic ecosystems. The ensemble multi-MCDM method can avoid inconsistencies among different MCDM methods and help to determine and manage priorities by emphasizing the regional distinctions.
Weights showed that the values of indicators represent the importance of pollution sources. Line C of Table 3 presents the rankings of all the indicators. We can see from line C of Table 3 that M a n g a n a t e ( F 3 ) , N i t r i t e N i t r o g e n ( F 5 ) and T o t a l A r s e n i c ( F 9 ) rank higher than other features, implying that they play a dominant role in water pollution. In particular, the weights of these three factors account for 40.73% of the total, while other seven factors account for the other 59.27%.

5. Conclusions and Future Work

Surface water quality has a major impact on the sustainable development of cities, especially in developing countries. How to efficaciously evaluate the impacts of surface water quality on social and economic development is an important issue for sustainable urban development. Studies have shown that integrating ensemble vote with MCDM is an effective means to improve traditional MCDM technique. This study has illustrates the usefulness of the proposed multi-MCDM approach in combination with ensemble vote as a tool for the evaluation of surface water quality. By comparing the results of four comprehensive evaluation methods, we found that different methods evaluating the same sample can results in different evaluation scores. However, we need same ranking results of various evaluation methods to employ a combined approach. Ensemble vote can solve this problem. According to the characteristics of the comprehensive evaluation index system of water quality and the purpose of the evaluation, this paper has used the TOPSIS, GRA, AHP, and TS Fuzzy Neural Network to evaluate the surface water quality of the study samples.
Although all results of this study illustrate the effectiveness of multi-MCDM methods in extracting characteristics from surface water quality datasets, there are some limitations to be noted. The disadvantage of this study is that the pollution sources are not identified to enable a fully understanding of the temporal and spatial variations of surface water quality. Further studies on surface water quality parameters, including TEMP, pH, and NH4–N, should be carried out [55]. These parameters can be further monitored for more accuracy and controlled. In addition, to obtain the contribution of different pollutants in each area, it is necessary to quantitatively evaluate pollution sources.
The purpose of this work is to explore water pollution evaluation, with the additional aim of improving water quality in the future. According to the influence of surface water quality evaluation, we hope that the government can put forward the countermeasures forth the management of water pollution in the future. Furthermore, based on the found compatibility between the compliance assessments and the practical surface water quality evaluation, a compatible grading evaluation and management scheme has been developed for better private and public decision-making.

Author Contributions

Z.C. contributed to collecting datasets and analyzing data. Z.C. and H.Z. designed and implemented the algorithm. H.Z. and M.L. contributed to the interpretation of the results. Z.C. took the lead in writing the manuscript. H.Z. and M.L. revised the manuscript.

Funding

This research was funded by the China Key projects of the National Bureau of Statistics grant number 2016564 and the APC was funded by open project of Applied Economics First-class Discipline (Cultivation) at the Guangxi University of Finance and Economics in 2018: Evaluation of the Digital Eco-economic Construction in Guangxi.

Acknowledgments

This work is partially supported by the China Key projects of the National Bureau of Statistics under Contracts 2016564, and open project of Applied Economics First-class Discipline (Cultivation) at the Guangxi University of Finance and Economics in 2018: Evaluation of the Digital Eco-economic Construction in Guangxi. The authors would also like to thank the anonymous reviewers for their useful comments.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:
MCDMMultiple Criteria Decision-Making
TOPSISTechnique for Order Preference by Similarity to an Ideal Solution
GRAGrey Relational Analysis
AHPAnalytic Hierarchy Process
TS-FNNTakagi-Sugeno Fuzzy Neural Network
ANNArtificial Neural Network

References

  1. Programme, W.W.A. The United Nations World Water Development Report: Water in a Changing World; UNESCO Publishing; Berghahn Books: New York, NY, USA, 2009. [Google Scholar]
  2. Oki, T.; Kanae, S. Global hydrological cycles and world water resources. Science 2006, 313, 1068–1072. [Google Scholar] [CrossRef] [PubMed]
  3. Duan, W.; He, B. Emergency Response System for Pollution Accidents in Chemical Industrial Parks, China. Int. J. Environ. Res. Public Health 2015, 12, 7868–7885. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Wang, Q.; Yang, Z. Industrial water pollution, water environment treatment, and health risks in China. Environ. Pollut. 2016, 218, 358–365. [Google Scholar] [CrossRef] [PubMed]
  5. Tu, J. Spatial Variations in the Relationships between Land Use and Water Quality across an Urbanization Gradient in the Watersheds of Northern Georgia, USA. Environ. Manag. 2013, 51, 1–17. [Google Scholar] [CrossRef] [PubMed]
  6. Gorgoglione, A.; Bombardelli, F.A.; Pitton, B.J.; Oki, L.R.; Haver, D.L.; Young, T.M. Uncertainty in the parameterization of sediment build-up and wash-off processes in the simulation of sediment transport in urban areas. Environ. Model. Softw. 2019, 111, 170–181. [Google Scholar] [CrossRef]
  7. Shi, H.; Shao, M. Soil and water loss from the Loess Plateau in China. J. Arid Environ. 2000, 45, 9–20. [Google Scholar] [CrossRef]
  8. Issaka, S.; Ashraf, M.A. Impact of soil erosion and degradation on water quality: A review. Geol. Ecol. Landsc. 2017, 1, 1–11. [Google Scholar] [CrossRef]
  9. Al-Isawi, R.H.; Almuktar, S.A.; Scholz, M. Monitoring and assessment of treated river, rain, gully pot and grey waters for irrigation of Capsicum annuum. Environ. Monit. Assess. 2016, 188, 287. [Google Scholar] [CrossRef] [PubMed]
  10. Alisawi, R.H.K.; Scholz, M.; Alfaraj, F.A.M. Assessment of diesel-contaminated domestic wastewater treated by constructed wetlands for irrigation of chillies grown in a greenhouse. Environ. Sci. Pollut. Res. 2016, 23, 25003–25023. [Google Scholar] [CrossRef] [Green Version]
  11. Le, C.; Zha, Y.; Li, Y.; Sun, D.; Lu, H.; Yin, B. Eutrophication of Lake Waters in China: Cost, Causes, and Control. Environ. Manag. 2010, 45, 662–668. [Google Scholar] [CrossRef]
  12. Nasiri, F.; Maqsood, I.; Huang, G.; Fuller, N. Water quality index: A fuzzy river-pollution decision support expert system. J. Water Resour. Plan. Manag. 2007, 133, 95–105. [Google Scholar] [CrossRef]
  13. Karamouz, M.; Zahraie, B.; Kerachian, R. Development of a Master Plan for Water Pollution Control Using MCDM Techniques: A Case Study. Water Int. 2003, 28, 478–490. [Google Scholar] [CrossRef]
  14. Lad, R.K.; Desai, N.G.; Christian, R.A.; Deshpande, A.W. Fuzzy modeling for environmental pollution potential ranking of industries. Environ. Prog. 2010, 27, 84–90. [Google Scholar] [CrossRef]
  15. Li, L.; Shi, Z.H.; Yin, W.; Zhu, D.; Ng, S.L.; Cai, C.F.; Lei, A.L. A fuzzy analytic hierarchy process (FAHP) approach to eco-environmental vulnerability assessment for the danjiangkou reservoir area, China. Ecol. Model. 2009, 220, 3439–3447. [Google Scholar] [CrossRef]
  16. Chung, E.S.; Lee, K.S. Prioritization of water management for sustainability using hydrologic simulation model and multicriteria decision making techniques. J. Environ. Manag. 2009, 90, 1502–1511. [Google Scholar] [CrossRef] [PubMed]
  17. Tebaldi, C.; Knutti, R. The Use of the Multi-Model Ensemble in Probabilistic Climate Projections. Philos. Trans. A Math. Phys. Eng. Sci. 2007, 365, 2053–2075. [Google Scholar] [CrossRef]
  18. Kubler, S.; Robert, J.; Derigent, W.; Voisin, A.; Le Traon, Y. A state-of the-art survey & testbed of fuzzy AHP (FAHP) applications. Expert Syst. Appl. 2016, 65, 398–422. [Google Scholar] [Green Version]
  19. Baykasoglu, A.; Kaplanoglu, V.; Durmusoglu, Z.D.; Sahin, C. Integrating fuzzy DEMATEL and fuzzy hierarchical TOPSIS methods for truck selection. Expert Syst. Appl. 2013, 40, 899–907. [Google Scholar] [CrossRef]
  20. Choudhary, D.; Shankar, R. An STEEP-fuzzy AHP-TOPSIS framework for evaluation and selection of thermal power plant location: A case study from India. Energy 2012, 42, 510–521. [Google Scholar] [CrossRef]
  21. Hosseini, S.; Al Khaled, A. A hybrid ensemble and AHP approach for resilient supplier selection. J. Intell. Manuf. 2016, 1–22. [Google Scholar] [CrossRef]
  22. Tang, L.; Yu, L.; Wang, S.; Li, J.; Wang, S. A novel hybrid ensemble learning paradigm for nuclear energy consumption forecasting. Appl. Energy 2012, 93, 432–443. [Google Scholar] [CrossRef]
  23. Liu, Y.; Zheng, Y.; Liang, Y.; Liu, S.; Rosenblum, D.S. Urban Water Quality Prediction based on Multi-task Multi-view Learning. In Proceedings of the International Joint Conference on Artificial Intelligence, New York, NY, USA, 9–15 July 2016; pp. 2576–2582. [Google Scholar]
  24. Wang, Q.; Dai, H.N.; Wang, H. A Smart MCDM Framework to Evaluate the Impact of Air Pollution on City Sustainability: A Case Study from China. Sustainability 2017, 9, 911. [Google Scholar] [CrossRef]
  25. Lin, C.T.; Chang, C.W.; Chen, C.B. The worst ill-conditioned silicon wafer slicing machine detected by using grey relational analysis. Int. J. Adv. Manuf. Technol. 2006, 31, 388–395. [Google Scholar] [CrossRef]
  26. Jaramilloa, P.; Mullerb, N.Z. Air pollution emissions and damages from energy production in the U.S.: 2002–2011. Energy Policy 2016, 90, 202–211. [Google Scholar] [CrossRef]
  27. Angelov, P.P.; Filev, D.P. An approach to online identification of Takagi-Sugeno fuzzy models. IEEE Trans. Syst. Man Cybern. Part B 2004, 34, 484–498. [Google Scholar] [CrossRef]
  28. Haykin, S. Neural Networks and Learning Machines; Pearson: London, UK, 2009. [Google Scholar]
  29. Lee, A.H.I.; Chen, W.; Chang, C. A fuzzy AHP and BSC approach for evaluating performance of IT department in the manufacturing industry in Taiwan. Expert Syst. Appl. 2008, 34, 96–107. [Google Scholar] [CrossRef]
  30. Altenburger, R.; Aitaissa, S.; Antczak, P.; Backhaus, T.; Barcelo, D.; Seiler, T.; Brion, F.; Busch, W.; Chipman, K.; De Alda, M.L.; et al. Future water quality monitoring—Adapting tools to deal with mixtures of pollutants in water resource management. Sci. Total Environ. 2015, 512, 540–551. [Google Scholar] [CrossRef]
  31. Boyd, J. Compensating for Wetland Losses under the Clean Water Act. Environment 2002, 44, 43–44. [Google Scholar] [CrossRef]
  32. Zou, Z.; Yun, Y.; Sun, J. Entropy method for determination of weight of evaluating indicators in fuzzy synthetic evaluation for water quality assessment. J. Environ. Sci. 2006, 18, 1020–1023. [Google Scholar] [CrossRef]
  33. Liou, S.; Lo, S.; Wang, S. A Generalized Water Quality Index for Taiwan. Environ. Monit. Assess. 2004, 96, 35–52. [Google Scholar] [CrossRef]
  34. Icaga, Y. Fuzzy evaluation of water quality classification. Ecol. Indic. 2007, 7, 710–718. [Google Scholar] [CrossRef]
  35. Liu, L.; Zhou, J.; An, X.; Zhang, Y.; Yang, L. Using fuzzy theory and information entropy for water quality assessment in Three Gorges region, China. Expert Syst. Appl. 2010, 37, 2517–2521. [Google Scholar] [CrossRef]
  36. Shi, B.; Wang, P.; Jiang, J.; Liu, R. Applying high-frequency surrogate measurements and a wavelet-ANN model to provide early warnings of rapid surface water quality anomalies. Sci. Total Environ. 2018, 610–611, 1390–1399. [Google Scholar] [CrossRef] [PubMed]
  37. Büyüközkan, G.; Çifçi, G. A novel hybrid MCDM approach based on fuzzy DEMATEL, fuzzy ANP and fuzzy TOPSIS to evaluate green suppliers. Expert Syst. Appl. 2012, 39, 3000–3011. [Google Scholar] [CrossRef]
  38. Hanine, M.; Boutkhoum, O.; Tikniouine, A.; Agouti, T. Comparison of fuzzy AHP and fuzzy TODIM methods for landfill location selection. SpringerPlus 2016, 5, 501. [Google Scholar] [CrossRef] [PubMed]
  39. Chen, J.K.; Chen, I.S. Using a novel conjunctive MCDM approach based on DEMATEL, fuzzy ANP, and TOPSIS as an innovation support system for Taiwanese higher education. Expert Syst. Appl. 2010, 37, 1981–1990. [Google Scholar] [CrossRef]
  40. Almanza, V.; Batyrshin, I.; Sosa, G. Multi-criteria selection of an Air Quality Model configuration based on quantitative and linguistic evaluations. Expert Syst. Appl. 2014, 41, 869–876. [Google Scholar] [CrossRef]
  41. Ferreira, L.; Borenstein, D. A fuzzy-Bayesian model for supplier selection. Expert Syst. Appl. 2012, 39, 7834–7844. [Google Scholar] [CrossRef]
  42. Kartal, H.; Oztekin, A.; Gunasekaran, A.; Cebi, F. An integrated decision analytic framework of machine learning with multi-criteria decision making for multi-attribute inventory classification. Comput. Ind. Eng. 2016, 101, 599–613. [Google Scholar] [CrossRef]
  43. Peng, Y.; Wang, G.; Kou, G.; Shi, Y. An empirical study of classification algorithm evaluation for financial risk prediction. Appl. Soft Comput. 2011, 11, 2906–2915. [Google Scholar] [CrossRef]
  44. Ali, R.; Lee, S.; Chung, T.C. Accurate multi-criteria decision making methodology for recommending machine learning algorithm. Expert Syst. Appl. 2017, 71, 257–278. [Google Scholar] [CrossRef]
  45. Gomes, H.M.; Barddal, J.P.; Enembreck, F.; Bifet, A. A Survey on Ensemble Learning for Data Stream Classification. ACM Comput. Surv. 2017, 50, 23. [Google Scholar] [CrossRef]
  46. Wang, Q.; Xia, L.Y.; Chai, H.; Zhou, Y. Semi-Supervised Learning with Ensemble Self-Training for Cancer Classification. In Proceedings of the 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), Guangzhou, China, 8–12 October 2018; pp. 796–803. [Google Scholar]
  47. Aliahmadipour, L.; Torra, V.; Eslami, E. On Hesitant Fuzzy Clustering and Clustering of Hesitant Fuzzy Data. In Fuzzy Sets, Rough Sets, Multisets and Clustering; Number 671; Springer: New York, NY, USA, 2017; pp. 157–168. [Google Scholar]
  48. Saaty, T.L. How to make a decision: The analytic hierarchy process. Eur. J. Oper. Res. 1990, 48, 9–26. [Google Scholar] [CrossRef]
  49. Chang, K.H. Generalized multi-attribute failure mode analysis. Neurocomputing 2016, 175, 90–100. [Google Scholar] [CrossRef]
  50. Mon, D.L.; Cheng, C.H.; Lin, J.C. Evaluating weapon system using fuzzy analytic hierarchy process based on entropy weight. Fuzzy Sets Syst. 1994, 62, 127–134. [Google Scholar] [CrossRef]
  51. Environmental Quality Standards for Suface Water; Ministry of Environmental Protection: Beijing, China, 2002.
  52. Garciacascales, M.S.; Lamata, M.T. On rank reversal and TOPSIS method. Math. Comput. Model. 2012, 56, 123–132. [Google Scholar] [CrossRef]
  53. Tseng, M. Using linguistic preferences and grey relational analysis to evaluate the environmental knowledge management capacity. Expert Syst. Appl. 2010, 37, 70–81. [Google Scholar] [CrossRef]
  54. Ishizaka, A.; Labib, A. Review of the main developments in the analytic hierarchy process. Expert Syst. Appl. 2011, 38, 14336–14345. [Google Scholar] [CrossRef]
  55. Wu, Z.; Zhang, D.; Cai, Y.; Wang, X.; Zhang, L.; Chen, Y. Water quality assessment based on the water quality index method in Lake Poyang: The largest freshwater lake in China. Sci. Rep. 2017, 7, 17999. [Google Scholar] [CrossRef] [Green Version]
Figure 1. Our proposed hybrid ensemble multi-model framework.
Figure 1. Our proposed hybrid ensemble multi-model framework.
Processes 07 00025 g001
Figure 2. Neural network training procedure.
Figure 2. Neural network training procedure.
Processes 07 00025 g002
Figure 3. Numbers 1–6 represent the Ma’anshan Reach of the Yangtze River, Quarry River, Yushan River, Cihu River, Yushan Lake and Suoxi River, respectively.
Figure 3. Numbers 1–6 represent the Ma’anshan Reach of the Yangtze River, Quarry River, Yushan River, Cihu River, Yushan Lake and Suoxi River, respectively.
Processes 07 00025 g003
Figure 4. Neural network training value b.
Figure 4. Neural network training value b.
Processes 07 00025 g004
Figure 5. Neural network training value c.
Figure 5. Neural network training value c.
Processes 07 00025 g005
Table 1. Saaty’ s fundamental scale.
Table 1. Saaty’ s fundamental scale.
Intensity of ImportanceDefinition
1Equal importance/preference
3Moderate importance/preference
5Strong importance/preference
7Very strong or demonstrated importance/preference
2, 4, 6, 8In between these two adjacent levels
9Extreme importance/preference
Table 2. National environmental quality standards for surface water (partial) (/mg·L 1 ).
Table 2. National environmental quality standards for surface water (partial) (/mg·L 1 ).
Level F 1 F 2 F 3 F 4 F 5 F 6 F 7 F 8 F 9 F 10
172.820.0150.060.0020.00550.050.01
26340.020.10.0020.0550.050.05
35460.020.150.0050.2100.050.05
43680.210.010.21000.10.05
5210100.210.10.21000.10.1
Table 3. List of every indicator training value.
Table 3. List of every indicator training value.
LineExplain F 1 F 2 F 3 F 4 F 5 F 6 F 7 F 8 F 9 F 10
AEntropy e0.78140.82690.73170.84040.61840.78620.78330.85980.73670.7945
BWeights of entropy0.09800.07760.11580.07150.17110.09580.09720.06280.11800.0921
CWeights after training0.09670.08030.13670.07250.14790.10450.08600.06340.12270.0893
Table 4. Weights of every model.
Table 4. Weights of every model.
IndicatorTS-FNNTOPSISGRAAHP
Weights0.22780.21570.30930.2472
Table 5. Normalized process table for Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS).
Table 5. Normalized process table for Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS).
Site F 1 F 2 F 3 F 4 F 5 F 6 F 7 F 8 F 9 F 10
A 1 0.090160.096510.096510.088640.096510.096510.096510.056690.096510.07857
A 2 0.080140.067920.072270.080140.052700.080140.076290.080140.020080.08014
A 3 0.079440.129220.042960.055060.019220.136430.116200.103700.034180.13643
A 4 0.011760.022720.010360.000150.002270.012090.010880.047500.018130.01359
A 5 0.000300.000300.000300.098600.000300.000300.000300.000300.000300.00030
A 6 0.057190.068630.059710.047890.053880.060950.060090.064860.065310.05225
Table 6. D i + , D i , C Value and Ranking.
Table 6. D i + , D i , C Value and Ranking.
Site D i + D i CRanking
A 1 0.04480.11020.71081
A 2 0.06770.16520.70922
A 3 0.21050.24860.54153
A 4 0.11030.06130.35725
A 5 0.29490.09830.25006
A 6 0.03580.04030.52934
Table 7. Alternatives and ranking at Grey Relational Analysis (GRA).
Table 7. Alternatives and ranking at Grey Relational Analysis (GRA).
Site A 1 A 2 A 3 A 4 A 5 A 6
Value0.92560.81730.62210.38620.35270.5363
Ranking123564
Table 8. Alternatives and ranking according to Takagi-Sugeno Fuzzy Neural Network (TS-FNN).
Table 8. Alternatives and ranking according to Takagi-Sugeno Fuzzy Neural Network (TS-FNN).
SiteGlobal PerformanceRanking
A 1 −0.99591
A 2 −0.80152
A 3 −0.27474
A 4 0.01385
A 5 0.50086
A 6 −0.38723
Table 9. Surface water quality evaluation indicator matrix.
Table 9. Surface water quality evaluation indicator matrix.
F 1 F 2 F 3 F 4 F 5 F 6 F 7 F 8 F 9 F 10
F 1 1/23/21/35/37/62/31/64/35/6
F 2 8/32/3324/31/37/35/3
F 3 2/78/75/73/71/76/74/7
F 4 7/25/23/21/232
F 5 2/31/31/65/61/2
F 6 1/21/45/43/4
F 7 1/223/2
F 8 32
F 9 1/2
F 10
Table 10. D i s s o l v e d indicator matrix.
Table 10. D i s s o l v e d indicator matrix.
A 1 A 2 A 3 A 4 A 5 A 6
A 1 5/65/45/255/3
A 2 3/2362
A 3 244/3
A 4 22/3
A 5 1/3
A 6
Table 11. Total evaluation indicator matrix at the criteria layer.
Table 11. Total evaluation indicator matrix at the criteria layer.
Site F 1 F 2 F 3 F 4 F 5 F 6 F 7 F 8 F 9 F 10
A 1 0.23810.28570.28570.23810.28570.25000.28570.30000.25000.1818
A 2 0.28570.19050.23810.28570.23810.25000.23810.20000.16670.2727
A 3 0.19050.23810.14290.09520.14290.25000.19050.20000.16670.2727
A 4 0.09520.09520.09520.04760.09520.08330.09520.15000.16670.0909
A 5 0.04760.04760.04760.14290.04760.04170.04760.05000.04170.0455
A 6 0.14290.14290.19050.19050.19050.12500.14290.10000.20830.1364
Table 12. Alternatives and ranking according to Analytic Hierarchy Process (AHP).
Table 12. Alternatives and ranking according to Analytic Hierarchy Process (AHP).
SiteGlobal PerformanceRanking
A 1 0.26121
A 2 0.23552
A 3 0.18513
A 4 0.10245
A 5 0.05316
A 6 0.16274
Table 13. Global performance ranking with ensemble learning.
Table 13. Global performance ranking with ensemble learning.
SiteTOPSISGRATS-FNNAHPH ValueRanking
A 1 11111.0001
A 2 22220.8812
A 3 33430.5573
A 4 55550.2015
A 5 66660.0006
A 6 44340.4954

Share and Cite

MDPI and ACS Style

Chen, Z.; Zhang, H.; Liao, M. Integration Multi-Model to Evaluate the Impact of Surface Water Quality on City Sustainability: A Case from Maanshan City in China. Processes 2019, 7, 25. https://doi.org/10.3390/pr7010025

AMA Style

Chen Z, Zhang H, Liao M. Integration Multi-Model to Evaluate the Impact of Surface Water Quality on City Sustainability: A Case from Maanshan City in China. Processes. 2019; 7(1):25. https://doi.org/10.3390/pr7010025

Chicago/Turabian Style

Chen, Zhanbo, Hui Zhang, and Mingxia Liao. 2019. "Integration Multi-Model to Evaluate the Impact of Surface Water Quality on City Sustainability: A Case from Maanshan City in China" Processes 7, no. 1: 25. https://doi.org/10.3390/pr7010025

APA Style

Chen, Z., Zhang, H., & Liao, M. (2019). Integration Multi-Model to Evaluate the Impact of Surface Water Quality on City Sustainability: A Case from Maanshan City in China. Processes, 7(1), 25. https://doi.org/10.3390/pr7010025

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop