Research on Sustainable Supplier Evaluation Index System in Architectural Design Industry

Abstract

Sustainable development has been popularized and emphasized in many industries. Great attention has been paid to the sustainability performance of the whole supply chain, with evaluating suppliers’ sustainability being particularly critical. However, research on sustainable development in the consulting service industry remains to be discovered, and few academic studies have analyzed the supplier selection indices derived from this industry. This paper conducted a case study in a design institute using the Delphi method to investigate the concept and characteristics of sustainable suppliers. Keywords derived from surveys and literature were collected, sorted, and established into a comprehensive hierarchy index system for sustainability evaluation. Finally, based on the improved failure mode and effects analysis (FMEA) and analytic hierarchy process (AHP), the failure risk of the indices was analyzed with weighted risk factors to obtain the high-risk key evaluation indices, which provided a simplified scheme for enterprises to streamline the evaluation process on sustainable suppliers. The results show the reliability of the improved FMEA model by differentiating the risk value and ranking. Suggestions for sustainable improvement are provided.

1. Introduction

The WCED [1] first proposed the concept of sustainable development in the Brundtland Report. By improving the economic, environmental, and social performance of the entire supply chain, enterprises can have a significant positive impact on society [2] with higher sustainable creativity, while continuing to extend and balance the relationship among triple bottom line (TBL) dimensions [3]. Based on sustainable development theory, scholars have paid attention to supply chain management in multiple industries [4,5,6]. Nevertheless, research on sustainable development in the consulting service industry remains to be discovered, and few academic studies have analyzed the supplier selection indices derived from this industry.

Nowadays, greater attention has been paid to the sustainability performance of the whole supply chain, with evaluating suppliers’ sustainability being particularly critical [7]. Compared with dynamic suppliers, seeking strategic cooperation and sustainability will help enterprises maintain long-term competitiveness [8], thus significantly enhancing the stability of the supply chain. Developing a scientific way of evaluating suppliers’ sustainability is crucial. However, various discussions have been conducted on traditional supplier evaluation indices (Appendix A 

Table A1. Common evaluation indices and frequency of occurrence.

Major Index	Secondary Index	Occurrence
quality	quality certificate or standard	19
	pass rate	19
	repair and return rate	9
	quality management system	18
	abnormal quality handling capability	2
	quality improvement system	7
	material science	5
delivery capacity	on-time delivery	42
	delivery quantity	14
	delivery flexibility	17
	return goods	4
	geographical location and transportation distance	11
	inventory turnover	4
	packaging quality	5
technology	R&D capability and investment	21
	production equipment and technical level	21
	design capability	4
	productivity	13
price and cost	price	43
	cost	10
	freight	19
	after-sale service cost	2
	cost reduction capability/plan	7
	product life cycle cost	1
service	after-sale service	15
	customer complaint handling	8
business management	organization management	10
	financial stability	34
	market competitiveness	7
	management information systems	11
	adaptability/flexibility	21
	environmental management and pollution control	9
	employee management	6
	corporate reputation	16
	strategic partnership	15
	policies and regulations	6
	corporate culture and welfare	8

) instead of TBL, while most studies provide limited indices or criteria without a comprehensive overview.

Taking an architectural design institute as the research object, this study analyzes the understanding of sustainable development in the design industry. It helps the design institute identify, evaluate, and select the indices of sustainable suppliers from the perspective of risk evaluation and constructs a sustainable supplier index system to provide methods and decision support for enterprises to judge the sustainability of suppliers. The remainder of this paper includes the following sections: Section 2 provides a literature review on the Delphi method, analytic hierarchy process (AHP), and failure mode and effects analysis (FMEA), and constructs the model of the study; Section 3 establishes the index system based on Delphi method; Section 4 filters the key indices through the risk analysis based on improved FMEA and AHP; Section 5 provides a discussion; and Section 6 is the conclusion.

2. Literature Review and Methodology

A comprehensive analysis combined with literature research and data collection is necessary to understand the meaning of sustainable suppliers in the design consulting industry. Considering the feasibility of enterprise applications, the basic model for supplier selection should meet several conditions: ① the model is mature and has been widely practiced in the existing research and application, and has sufficient reliability; ② the algorithm and formula are simple and convenient, which is easy for software programming and staff learning; ③ the data processing is clear, and the majority can rely on the existing statistical data as the basis for data collection, with low processing difficulty; and ④ the model can be improved under optimization demand to reduce the loss of learning new tools.

To ensure that the analysis results have objective and practical significance and to avoid the differences and difficulties caused by open-ended problems, this study intends to use the Delphi method to continuously improve concept analysis and find a general understanding through multiple rounds of investigation. Based on the indices filtered from Delphi results, a hierarchy index system is developed and their importance is analyzed based on the integration of AHP and FMEA. The improved FMEA model supports risk analysis in that indices with a higher risk of failure will be regarded to be of greater importance, as these high-risk indices might cause serious effects compared with others. Meanwhile, to differentiate the role of risk factors, AHP is utilized to define their weights.

2.1. Literature Review

The Delphi method [9] is a systematic method of gathering opinions from a group of experts through a series of questionnaires, where there is a mechanism of feedback through rounds of questions held while maintaining the anonymity of respondents’ responses. It is suitable for the study of the following characteristics: ① insufficient data and information; ② lack of theoretical basis but subjective judgment; ③ wide range and complex content; ④ analysis of large differences in views; and ⑤ difficult in quantification. This method has been utilized for supplier selection and evaluation in multiple ways, including criteria and sub-criteria selection [10,11] and decision model improvement [12,13]. An integration between the Delphi method and AHP can also be found in previous studies on green supplier selection [14,15,16]. However, studies that applied the Delphi method normally selected criteria from an existing criteria pool with limited criteria numbers, lacking the background of criteria establishment and its developing process. In this study, the Delphi method is used to analyze the concept of sustainable suppliers, which can refine abstract and fuzzy definitions through multiple rounds of research and obtain a clearer conclusion. Meanwhile, the initial index system for sustainable supplier evaluation is formed by keywords provided by experts.

Supplier selection is typically a multi-index decision-making process. As a commonly used decision-making approach, the advantage of AHP is that it can decompose complex decision-making problems and deeply study the internal relationship of its influencing factors by simplifying the decision-making process through quantification. It is very effective for decision-making problems with multi-objective, multi-index, or unstructured characteristics [17], and provides convenience for determining the index weight. Furthermore, AHP helps prioritize the evaluation criteria of supplier selection for sustainability [18]. It is the most commonly used technique for evaluating sustainable supplier decisions, but integrating AHP with other approaches may be required to address the problem more efficiently and flexibly [19]. Because of its great flexibility and wide applicability, integrated AHP approaches have been studied extensively for the past decades, e.g., the technique for order preference by similarity to an ideal solution (TOPSIS) method [20], spherical fuzzy theory [21,22], and Delphi method [23]. A review of its integrations and applications can be found in [24].

Suppliers’ sustainability can be evaluated positively from the perspective of performance, or negatively from the perspective of risk. However, as a major challenge that almost all supply chains need to face nowadays, risk management is rarely considered in research on sustainable supplier selection [25]. The selection of evaluation indices can be determined based on the FMEA model by regarding the evaluation index as failure modes, which means that failure modes with higher risks would be the evaluation indexes with higher importance. It is an inductive analysis method based on the discussion of the FMEA expert team to systematically analyze the design, development, and production process of products or services, identify potential failure modes and their effects, and evaluate and classify them according to risk factors [26]. This method can not only identify the key evaluation indices according to the risk ranking but also efficiently solve the errors or problems by analyzing the failure causes, consequences, and improvement measures in the failure mode table. To improve the safety and reliability of the system, FMEA has also been studied and widely used in supplier selection [27], group decision-making [28], condition monitoring and fault diagnosis [29], and optimal maintenance policies [30] in various industries [31]. Meanwhile, since traditional FMEA cannot show the importance of risk factors and may achieve parallel results with different value combinations of risk factors, it has been improved by integrating AHP [32], data mining [33], K mean clustering [34], quality function deployment [35], etc.

Current discussions on sustainable supplier evaluation focus on the weight of evaluation indices and the optimization of the decision-making model, which does not illustrate the process of developing an index system. Meanwhile, features of the consulting service industries have not been distinguished from others in most studies, while the evaluation of service quality relies more on customer subjective judgment. It is also evident that integration among the Delphi method, AHP, and FMEA is feasible in supplier evaluation research, while enterprises prefer an easy-to-learn method in practice. As a result, this paper aims to conduct an in-depth analysis of the understanding of sustainable suppliers in the consulting service industry and develop a comprehensive hierarchy index system for sustainability evaluation with an easily adopted integrated method. The process of developing an index system combines both rounds of questionnaires and a large literature collection, aiming to provide a whole picture of the detailed index structure.

2.2. Methodology

Firstly, the Delphi method with three rounds of questionnaires is utilized to gather information related to sustainable supplier evaluation from selected experts. The concept of a sustainable supplier is defined and the initial evaluation index system is developed. Secondly, we improve the traditional FMEA model and analyze the importance of risk factors by using AHP. The indices with higher failure risk will be considered important indices that may significantly affect supplier sustainability.

When filtering the key evaluation indices of sustainable suppliers, AHP is used to analyze the weight of FMEA risk factors: severity (S), occurrence (O), and detection (D). Meanwhile, an improved FMEA model is used to analyze the failure risk of evaluation indices, and those with higher risk will be regarded as key indices with higher importance. The key indices receive priority when evaluating the sustainable performance of suppliers.

Since the weight of risk factors needs to be incorporated into the risk priority number (RPN) calculation formula, the direct superposition of multiplication cannot reflect the effect of method improvement. Therefore, we first convert $RPN=S\times O\times D$ into a natural logarithm, that is,

$$\mathit{ln}(RPN)=\mathit{ln}(S\times O\times D)=\mathit{ln}S+\mathit{ln}O+\mathit{ln}D$$

Then, assuming that the weight of risk factors is $w_0=(w_S,w_O,w_D)$, after one-to-one correspondence to risk factors, we can obtain the improved formula:

$$RP{N}^{’}=w_S\ast \mathit{ln}S+w_O\ast \mathit{ln}O+w_D\ast \mathit{ln}D$$

The calculation results of the improved formula not only include the weight of risk factors but can also clearly differentiate the evaluation results through the conversion of the natural logarithm and further clarifying the ranking. The improved method helps provide decision-making references directly for the design institute.

3. Development of a Sustainable Supplier Evaluation Index System

Evaluation indices of sustainable suppliers in the consulting service industry have their features and must be determined by enterprises instead of relying purely on literature. An in-depth case study in a design institute can provide sufficient and reliable information to understand the concept of a sustainable supplier in the consulting service industry and develop the evaluation index system. Institute A is one of the most widely qualified design consulting companies and has always been committed to providing high-quality design services to society. Its supply chain includes many manufacturing and service industries. This research will effectively evaluate the sustainability of upstream suppliers in combination with TBL to understand the sustainability of its development and industry. The research process can be divided into three steps: ① investigate the institute to understand the enterprise demand and its supply chain environment, and determine the enterprise’s understanding of sustainable suppliers by using the Delphi method; ② collect and sort out relevant literature on sustainable supplier evaluation indexes, and form an index database in combination with research; and ③ use AHP and FMEA to screen and filter the key indices to build a more operable evaluation index system.

3.1. Definition of Sustainable Suppliers

The Delphi method was used to analyze the understanding of sustainable suppliers. Experts from Institute A were invited to complete the questionnaire online, and three rounds of surveys were formulated to determine the characteristics of sustainable suppliers.

3.1.1. Identify Research Experts

When selecting research experts at Institute A, the following conditions must be met: ① know the concept of sustainable development; ② professional field or major work involves suppliers; ③ be willing to participate in multiple rounds of research and have plenty of time. In addition, the preferred optional conditions for selecting experts are: ① have direct contact with suppliers, such as procurement, audit, etc.; ② the department or enterprise has a clear sustainable vision such as strategic planning; ③ have rich experience and knowledge, with the latest industry development information. After preliminary screening, nine experts were finally identified, of which 66% had worked in the design industry for more than 11 years (included), and two thirds had direct contact with suppliers. Questionnaires answered by each expert were given equal attention without differences in weight.

3.1.2. The First Round

The first round of the survey aimed to collect the types of suppliers involved in the business of a design institute and the important indices when evaluating suppliers’ comprehensive performance. The questionnaire adopted an open Q&A so that experts could fully express their views without being constrained by the options. The results show that the suppliers of Institute A include electrical appliance manufacturers, intelligent companies, precast concrete deepening design (subcontracting), steel structure deepening design (subcontracting), etc. The key indices for assessing suppliers’ performance are shown in

Table 1. Keywords of supplier performance evaluation indices.

Keyword	Occurrence	Keyword	Occurrence
price/expense	9	adaptability	3
brand qualification	6	problem-solving ability	2
professional technology	6	customer satisfaction	2
overall competence	5	responsiveness	2
quality	5	team ability	1
service	4	—	—

, with price/expense on the top, followed by brand qualification and professional technology. These indices are economic benefits oriented and lack the awareness of environmental protection and social responsibility.

3.1.3. The Second Round

The second round of the survey targeted the evaluation of sustainable suppliers. Experts were required to supplement their understanding of sustainable development and provide sustainability evaluation indices with corresponding rankings to show the importance of keywords. The survey made full use of the flexibility of the questionnaire system to cross-process the ranking of keywords so that experts could obtain keywords from each other. They were asked to evaluate their importance through a five-point scale, and the importance of indices was calculated by weighted average (

Table 2. Keywords and their importance for sustainable suppliers.

Keyword	Occurrence	Importance	Keyword	Occurrence	Importance
professional technology	5	4.00	green innovation	6	3.00
quality	7	4.00	corporate social responsibility	4	2.75
waste disposal	4	4.00	overall competence	2	2.50
recycling	8	4.00	problem-solving ability	4	2.50
resource efficiency	7	3.57	price/expense	5	2.40
green design	3	3.33	staff training	3	2.33
adaptability	7	3.14	customer satisfaction	3	2.00
brand qualification	5	3.00	team ability	2	2.00
service	4	3.00	carbon footprint	3	2.00
responsiveness	5	3.00	corporate culture	3	1.67

).

The most important keywords in Table 2 are professional technology, quality, waste disposal, and recycling (4.00 points). However, the price/expense highlighted in the first round is of lower importance (2.40 points), indicating that when facing a reasonable demand for sustainable development, the cost and economic indices can make appropriate concessions in exchange for the maintenance or optimization of environmental protection. In addition, experts believe that the concept of sustainable suppliers should be extended to other enterprises with business connections. As a result, we determined that sustainable suppliers in this study refer to suppliers (or contractors) who are willing to improve their sustainability along with upstream and downstream companies while maintaining economic benefits as well as environmental and social sustainability.

3.1.4. The Third Round

The final round of consultation aimed to filter and improve the description and key indices of sustainable suppliers and test the consistency of expert feedback. It required experts to repeatedly evaluate the importance of the keywords listed in Table 2. The order of the keywords was random every time to avoid the influence of previous evaluations. The results described the characteristics as a reference for developing the evaluation index system of sustainable suppliers.

Based on the understanding of sustainable development in the design industry, a sustainable supplier is defined as an enterprise or individual that supplies various required resources to enterprises or competitors for economic, environmental, and social sustainability, including providing raw materials, equipment, energy, services, etc., and they are willing to improve their sustainability along with upstream and downstream enterprises. Traditional suppliers with sustainability can generally be regarded as sustainable suppliers. The concept of traditional suppliers pays more attention to business and functionality, whereas sustainable suppliers attach development ability on this basis. They should improve themselves as well as supervise and urge adjacent enterprises, which is an evolution with continuity characteristics. This particularity encourages suppliers to cultivate sustainability, which is a complementary process.

The results of the importance evaluation are shown in

Table 3. Keywords and importance ranking of sustainable supplier evaluation indices.

Keyword	Importance	Ranking	Keyword	Importance	Ranking
recycling	4.00	1	green innovation	3.11	11
quality	3.89	2	carbon footprint	3.11	11
professional technology	3.89	2	adaptability	2.89	13
resource efficiency	3.78	4	customer satisfaction	2.89	13
waste disposal	3.67	5	overall competence	2.78	15
problem-solving ability	3.67	5	team ability	2.78	15
brand qualification	3.33	7	corporate social responsibility	2.67	17
service	3.33	7	corporate culture	2.44	18
green design	3.22	9	price/expense	2.33	19
responsiveness	3.22	9	staff training	2.11	20

. The keywords related to environmental sustainability are generally at the top, whereas those related to social sustainability are relatively at the bottom. Balance among the three dimensions still needs to be observed and improved.

3.2. Establishment of Sustainable Supplier Evaluation Index Database and System

The dimensions of evaluation indices include economic, environmental, and social sustainability, while economic indices have been well developed. Based on the literature, the evaluation indices mentioned in existing studies were sorted out and classified according to TBL dimensions, and various indices corresponding to their occurrence frequencies were obtained (Appendix A, Appendix B and Appendix C). Although the secondary and primary indices can cover most industries such as manufacturing and service, they are still not sufficiently focused on the design industry to show the measurement requirements and matching degree of design enterprises on suppliers’ sustainability.

The majority of indices in Table 1, Table 2 and Table 3 are consistent with the conclusion of literature retrieval, indicating that the enterprise demand matches theoretical research. Therefore, the keywords in Table 3 were used as the classification basis of secondary indices and properly adjusted in combination with theoretical research. Specific combing operations included deduplication, merging, separation, and arrangement. The major indices in Appendix A, Appendix B and Appendix C were mapped to the secondary indices, while an evaluation index system was constructed with the help of experts (

Table 4. Sustainable supplier evaluation index database and system.

Primary Index	Secondary Index	Tertiary Indicators
economics (A1)	quality (B1)	quality certificate or standard (C1)
		qualified rate (C2)
		repair and return rate (C3)
		product performance (C4)
		quality management and improvement (C5)
	service (B2)	schedule management (C6)
		process management (C7)
		communication and docking (C8)
	price expense (B3)	cost (C9)
		price (C10)
		financial stability (C11)
	professional technology (B4)	R&D and investment (C12)
		core technology (C13)
		equipment conditions (C14)
	flexibility (B5)	on-time delivery (C15)
		delivery quantity (C16)
		responsiveness (C17)
		fit degree (C18)
		problem-solving ability (C19)
	comprehensive strength (B6)	management information systems (C20)
		brand qualification (C21)
		market share and competitiveness (C22)
		customer satisfaction (C23)
		team ability (C24)
		partnership (C25)
		corporate culture (C26)
environment (A2)	green design (B7)	material design (C27)
		process design (C28)
		environmental design (C29)
		building energy saving design (C30)
	green innovation (B8)	green packaging (C31)
		green storage (C32)
		green logistics (C33)
		clean technology (C34)
		green R&D and innovation (C35)
	recycling (B9)	internal material circulation (C36)
		external circulation (C37)
	resource use (B10)	environmentally friendly materials (C38)
		use of toxic and harmful substances (C39)
		resource utilization (C40)
	waste disposal (B11)	discharge of toxic and hazardous substances (C41)
		general solid waste treatment (C42)
		ordinary wastewater treatment (C43)
		ordinary waste gas treatment (C44)
		waste disposal optimization (C45)
	carbon footprint (B12)	carbon emissions (C46)
		environmental responsibility commitment (C47)
		green supply chain integration (C48)
sociology (A3)	employee interests and rights (B13)	working hours (C49)
		salary (C50)
		training and career development (C51)
		employee welfare (C52)
		culture and religion (C53)
	health and safety (B14)	health and insurance (C54)
		training and records (C55)
		safety and health management system (C56)
	stakeholder (B15)	information protection and disclosure (C57)
		participation (C58)
		customer feedback (C59)
	corporate social responsibility (B16)	moral (C60)
		basic services (C61)
		community welfare (C62)
		cultural property (C63)

).

Table 4 shows a comprehensive evaluation index database with its hierarchy structure. It can be divided into three layers: the primary indices based on the TBL dimensions; 16 secondary indices derived from the Delphi method; and 63 tertiary indices, most of which can be quantified directly through the enterprise’s data in the evaluation process. The sorting process fully combines numerous literatures and the results of the Delphi method, which has high reliability and comprehensiveness. The evaluation index system is not limited to suppliers’ products and technical capabilities, nor is it focused only on the scope of environmental protection, but also radiates to a wider range of social impact and supply chain coordination. It is effective in evaluating the performance and progress of overall interests and sustainability. Thus, the three-level index system in Table 4 is highly comprehensive with practicability.

4. Key Indices Based on Risk Analysis

The 63 primary indices in Table 4 can comprehensively and carefully evaluate the supplier sustainability of a design institute, providing strong support for judging whether they meet the requirements of being a sustainable supplier. However, a large number of indices might bring difficulties in practice, such as the heavy workload of expert evaluation, countless supporting documents submitted, less feasibility to suppliers of different businesses, and the complexity of quantification or calculation.

During the Delphi research, we discussed with experts whether there was a more reliable scheme when the number of evaluation indices is large, retaining comprehensiveness with a streamlined evaluation process with higher operability. Two compromise schemes were obtained for choice: ① the complete index system should be applied to new suppliers and suppliers with large performance fluctuations, while a group of key indices could be selected for stable cooperative suppliers; ② regular sustainability evaluation should be conducted on cooperative suppliers accordingly, using the whole evaluation index system every three years and key indices for inspection and maintenance during the period. Both schemes provided some convenience and emphasized the importance of refining key evaluation indices, while the filtering process could be designed from a risk perspective. Accordingly, by transferring the indices into failure modes, their importance can be analyzed by risk evaluation.

4.1. Weight Formulation of Risk Factors Based on AHP

Experts participating in the risk evaluation of the indices should be identified again. An additional requirement was put forward: their work had to have direct contact with suppliers. Five experts met all the conditions and had time for the evaluation. Four of them had worked in the design industry for at least 11 years. Based on their job responsibilities and knowledge of sustainability, the weight of experts was set as λ = (0.15, 0.2, 0.2, 0.2, 0.25).

4.1.1. Comparison Judgment Matrix and Weight Vector

Each expert was asked to compare the importance of the risk factors (S/O/D) and construct a comparison judgment matrix. The 1–9 scale recommended by Saaty [36] was adopted, and

Table 5. Comparison judgment matrix and weight vector.

Expert	Risk Factors	S	O	D	wi
D1	S	1	5	1/4	0.304
	O	1/4	1	1/6	0.092
	D	3	4	1	0.604
D2	S	1	4	5	0.685
	O	1/5	1	1/2	0.121
	D	1/5	2	1	0.194
D3	S	1	4	1/4	0.310
	O	1/5	1	1/4	0.116
	D	4	2	1	0.574
D4	S	1	3	5	0.631
	O	1/3	1	3	0.260
	D	1/4	1/4	1	0.109
D5	S	1	7	5	0.725
	O	1/6	1	1/5	0.082
	D	1/5	3	1	0.193

shows the results of the weight vector wi.

4.1.2. Consistency Test

Calculate the maximum eigenvalue λ_max of the five experts and determine the consistency test results (

Table 6. Maximum eigenvalue and consistency test results.

Expert	λmax	CI	RI	CR
D1	3.017	0.008	0.52	0.016
D2	3.003	0.002	0.52	0.003
D3	3.099	0.049	0.52	0.095
D4	3.070	0.035	0.52	0.067
D5	3.031	0.015	0.52	0.029

).

The consistency test results showed that all C_R < 0.1, indicating that the comparison judgment matrix had satisfactory consistency.

4.1.3. Weight of Risk Factors

The composite weight of risk factors was obtained: w₀ = (0.552, 0.134, 0.314), which showed that severity was the most important risk factor among all, followed by detection difficulty and occurrence. The emphasis on severity is reasonable, as the suppliers of the design institute include construction units, in which construction safety is an extremely important factor. Therefore, when evaluating the failure risk of supplier indices, the severity factor bears the brunt.

4.2. Risk Evaluation on Indices

Based on the weight of risk factors calculated by AHP, the FMEA was then used to sort the failure risk of the evaluation indices to reflect their importance. The failure modes in the model corresponded to 63 evaluation indices in the system.

4.2.1. Scoring Standard

The scoring standard referred to the 1–10 scale of the traditional FMEA. The higher the score, the higher the risk of S, O, and D. Experts evaluated the failure modes one by one according to the scoring standard table to form a series of risk evaluation tables. Meanwhile, the failure of the indices was transferred according to their definition. For example, failure of a quality certificate or standard (C₁) meant a lack of data and documents. If the evidence provided by the suppliers met the requirement, it would be converted to a lower score (less failure).

4.2.2. Calculation and Results Analysis

The evaluation experts who participated in the weight formulation were invited to conduct a unified evaluation of risk factors and failure modes C₁-C₆₃. Part of the evaluation is listed in

Table 7. Failure evaluation results of sustainable supplier evaluation indices (partially).

Evaluating Indicator	Severity S	Occurrence O	Detection D
quality certificate or standard (C1)	9	3	1
qualified rate (C2)	9	5	4
repair and return rate (C3)	7	6	4
product performance (C4)	8	4	5
quality management and improvement (C5)	5	6	4
schedule management (C6)	7	8	4
process management (C7)	4	8	6
communication and docking (C8)	9	4	6
cost (C9)	7	6	4
price (C10)	8	5	4
financial stability (C11)	5	3	8
R&D and investment (C12)	6	5	4
core technology (C13)	6	6	5
equipment conditions (C14)	7	6	4
on-time delivery (C15)	9	6	4
delivery quantity (C16)	8	5	5
responsiveness (C17)	6	4	5
fit degree (C18)	7	4	4
problem-solving ability (C19)	8	4	5
management information systems (C20)	6	5	4

(for full table, see Appendix D).

Based on Table 7, the traditional RPN and the improved RPN’ were calculated with the weight of risk factors w₀ = (0.552, 0.134, 0.314). Part of the results according to the descending order of RPN’ is shown in

Table 8. Ranking of RPN and RPN’ (partially).

Evaluating Index	RPN	RPN Ranking	RPN’	RPN’ Ranking
carbon emissions (C46)	567	1	2.118	1
moral (C60)	315	3	1.979	2
communication (C8)	216	18	1.961	3
health and insurance (C54)	270	7	1.958	4
wastewater treatment (C43)	343	2	1.946	5
external circulation (C37)	294	4	1.925	6
waste disposal optimization (C45)	294	4	1.925	6
information protection and disclosure (C57)	294	4	1.925	6
customer satisfaction (C23)	192	24	1.896	9
resource utilization (C40)	270	7	1.895	10
material design (C27)	240	14	1.893	11
safety and health management system (C56)	240	14	1.893	11
on-time delivery (C15)	216	18	1.888	13
environmental design (C29)	252	10	1.877	14
clean technology (C34)	252	10	1.877	14
emissions of toxic and hazardous substances (C41)	100	58	1.870	16
delivery quantity (C16)	200	23	1.869	17
pass rate (C2)	180	26	1.864	18
product performance (C4)	160	38	1.839	19
problem-solving ability (C19)	160	38	1.839	19

(for full table, see Appendix E).

(1)

Ranking differentiation

Based on the results of Appendix E, the distribution of both RPN and RPN’ values can be drawn in Figure 1, including the number of repetitiveness. It can be found that various failure modes share the same values in traditional RPN. For example, nine failure modes received 120 (ranked 47), including quality management and improvement C₅; financial stability C₁₁; R&D and investment C₁₂; responsiveness C₁₇; management information systems C₂₀; green storage C₃₂; environmentally friendly materials C₃₈; solid waste treatment C₄₂; and culture and religion C₅₃. Six failure modes received 160 (ranked 38). The traditional RPN calculation has obvious disadvantages in terms of the number of repetitions.

In contrast, the maximum number of the repeated RPN’ value was reduced to less than 3, such as the sixth place included external circulation C₃₇, waste disposal optimization C₄₅, and information protection and disclosure C₅₇; the 33rd place involved repair and return rate C₃, cost C₉ and equipment conditions C₁₄; and the 51st place were R&D and investment C₁₂, management information systems C₂₀, and solid waste treatment C₄₂.

Through the ranking comparison, the optimization results based on RPN’ value can more effectively differentiate the score and ranking, which is conducive to enterprises drawing lines when selecting key evaluation indices with less confusion.

(2)

Reasons for some significant difference between the two rankings

The distribution difference within the top ten rankings was not significant, except communication C₈ (RPN ranking 18 vs. RPN’ ranking 3) and customer satisfaction C₂₃ (24 vs. 9). In addition, large gaps emerged in emissions of toxic and hazardous substances C₄₁ (58 vs. 16); use of toxic and hazardous substances C₃₉ (60 vs. 25); cultural property C₆₃ (9 vs. 39); community welfare C₆₂ (16 vs. 47); and process management C₇ (24 vs. 55).

The reason is traceable as the weight of the risk factors has been added and brings an obvious impact. The failure modes with higher severity increased significantly in the new ranking. This phenomenon fully reflects the importance and necessity of weight formulation in the evaluation process. The entire failure risk evaluation provides higher accuracy by measuring the relative importance of risk factors. Thus, the RPN’ ranking can be considered with higher reliability.

(3)

Key evaluation index screening

High-risk failure modes can be regarded as critical evaluation indices. The screening method can be formulated according to the enterprise. For example, selecting the top 20% failure modes for further detailed analysis based on the Pareto principle; dividing the failure modes into different risk levels according to the ABC classification method; giving priority to solving failure modes in class A; and formulating a standard line for further analysis and improvement. After discussion with experts of Institute A, the third choice was selected and the top 20 indices according to RPN’ results were determined as the key evaluation indices of sustainable suppliers that would conduct further failure analysis. The screened indices can be found in Table 8.

These key evaluation indices are mainly used to evaluate suppliers with long-term and stable cooperation, or routine evaluation. The complete evaluation index system in Table 4 can still be used for new suppliers and periodic evaluations. It should be noted that the screened indices will significantly impact the design institute in the case of failure (i.e., failure and error), which requires daily and continuous attention.

5. Discussion

Index ranking based on risk assessment can effectively reflect the impact of suppliers on the business of the design institute when they fail to complete the corresponding tasks. The extracted key indices have the priority of conducting in-depth failure analysis because they obtain higher risk values and may cause serious effects compared with other indices. Both the sustainable supplier evaluation index system and its filtered results were sent to experts as well as managers in Institute A. Discussion on their reliability and feasibility was conducted, and the research received positive feedback. Enterprises should follow the suggestions on different supplier relationships and evaluation periods while implementing the index system and key indices. The key indices provide the possibility of easy evaluation, and inspire Institute A to refresh high-risk indices periodically for further improvement. The research results provide important reference guidance for design institutes to evaluate the sustainability of suppliers in the future. Specific suggestions are as follows:

(1)

Pay attention to environmental sustainability from a total life cycle perspective

Among the 20 key evaluation indices, the number of environmental sustainability indices (nine) is the greatest. The content not only involves the environmental protection problems of suppliers in the design stage (e.g., material design C₂₇ and environmental design C₂₉) but also includes the production and operation stage (e.g., carbon emissions C₄₆; wastewater treatment C₄₃; external circulation C₃₇; waste disposal optimization C₄₅; resource utilization C₄₀; clean technology C₃₄; and emissions of toxic and hazardous substance C₄₁). These indices are logically related and build up a picture of a total life cycle for products or services. For example, whether a supplier can optimize waste treatment will indicate whether it can effectively and continuously reduce the emissions of wastewater and toxic and hazardous substances in the future; environmental design refers to both the project construction and the project operation environment, which have direct requirements on green materials and technology; external circulation can reflect the integration ability of the supply chain that improves the efficiency and circulation of products or services, the utilization of resources, and the reduction of carbon emissions of the entire process.

In practice, considering a green total life cycle has a positive impact on the supply chain’s sustainability. Involving suppliers in the design stage will bring convenience to the construction project as they can provide more detailed information at an earlier stage. Design institutes need to not only evaluate the environmental sustainability of suppliers through certificates and documents but also carefully observe their management and operation levels as a whole picture, so that they can speculate whether suppliers have a high awareness of environmental protection and high-level purification technology at present and in the future.

(2)

Strengthen the monitoring of social sustainability

Only four social sustainability indices were filtered according to the RPN’ result, which is significantly less than the other two dimensions. Three obvious problems of the social dimension include: ① Difficult to quantify. Most indices in social dimensions are abstract and refer to humanization and social impact. It is difficult to evaluate through data instead of expert subjective opinion. Therefore, the formulation of measurement standards is a tough issue, and the selection of experts is crucial. ② Time-consuming. The contribution and impact of enterprises on employees, stakeholders, and surrounding communities are slow to be observed and have long-term effects, which means that the evaluation of indices may not be real-time. Especially for temporary or short-term cooperation suppliers, it is hard to check their social performance. ③ Low attention. Many researchers have discussed environment and green development when studying sustainability, ignoring the fact that the social dimension weights the same. Research on social sustainability is relatively new and has not received sufficient attention [37].

For design institutes, their products and services (such as buildings and their facility management) have direct and long-term impacts on the surrounding environment and communities. Construction projects may last for years and need positive support and enrollment from employees and all stakeholders. Materials and equipment provided by sustainable suppliers contribute to a cleaner site, lower noise, and an efficient work cycle. Thus, enhancing community satisfaction as well as suppliers’ awareness of social responsibility is crucial. Starting with the key indices is a suggestion for formulating the measurement method for the social dimension. These indices can reflect the supplier’s attitude towards internal and external stakeholders, as well as the degree of implementation of normative management and measures. In particular, the health and safety B₁₄ contains two key indices, which are the top priorities to be observed by a design institute. Since the contracting business inevitably involves construction projects, the management of employee’s health and safety should be the most basic responsibility of the suppliers.

(3)

Combine procurement with the index system

At present, the common procurement methods of the design institute include tendering, bid negotiation, and direct commission. The assessment of suppliers is mainly based on the investigation of projects, schemes, works, and cooperative relations. The content relies mostly on economic benefits, which are relatively direct and general. However, these perspectives cannot provide sufficient details related to sustainability, including whether the specific construction process and technology meet the requirements of sustainable development, and whether the supplier is capable of managing employee welfare, health, and safety. Therefore, it is necessary to expand the sustainable evaluation indices to the procurement process properly and gradually shift the focus from economic benefits to the overall importance of sustainable development.

The design institute should increase the scope of investigation of suppliers during tendering and bid negotiation based on the evaluation index system and its key indices. Conditions such as green materials, clean technologies, and tidy sites could be added as part of the procurement evaluation. In particular, more attention needs to be paid to hidden indices that are difficult to observe and obtain data directly from.

(4)

Establish a sustainable supplier database

This paper provides a sustainable supplier evaluation index database and index system for design institutes. When selecting specific suppliers, a similar sustainable supplier database can be established, while suppliers can be classified according to past cooperation and historical evaluation. The suppliers at each level correspond to different ranges of evaluation indices, while the process of selecting evaluation indices can be flexible according to needs and conditions. Although a complete evaluation index system can observe the supplier’s overall ability, it can save much time and simplify operations by using key indices in the case of repeated evaluation and middle-stage evaluation. Establishing and managing a sustainable supplier database corresponds to the evaluation index system. For example, key indices are used in the middle-stage evaluation of excellent cooperative suppliers, whereas a complete index system must be used in the initial evaluation of new cooperative suppliers. This not only improves the efficiency of supplier evaluation but also ensures the evaluation results and further measures.

(5)

Build an information management platform for sustainable suppliers

Based on the sustainable supplier database, we can build an information management platform by referring to the sustainable management methods of other manufacturing or chemical industries, and carry out information and dynamic management to evaluate, select, monitor, and promote suppliers’ sustainability. At present, several industries have established sustainable management platforms with the help of third-party institutions to share the sustainability information of most suppliers in the industry and to supervise each other. Before cooperation, the suppliers should upload the documents, certificates, or data corresponding to each index on the platform, while their performance will be analyzed and published in the system afterward. Through score comparison and specific information sharing, enterprises can judge if suppliers have the ability for sustainable development and then decide whether to cooperate. Simultaneously, suppliers can obtain the sustainability evaluation results of potential competitors on the platform and observe their position in the industry, then urge themselves to improve their competitive advantage.

Currently, the implementation of building information modeling (BIM) has been widely recognized and adopted. The system provides a technical platform for information sharing from the very start of a construction project. From the perspective of supplier selection, a similar platform can also be established to enhance the effectiveness of a sustainable supply chain. Building an information management platform for sustainable suppliers is critical in the long run. It can also support the dynamic management of suppliers based on the information update cycle. Due to the difference in the business content and contract validity of suppliers, their business category and evaluation cycle can be formulated when suppliers join the platform. Those of the same type can be classified into similar evaluation index systems that will warn automatically when periodic evaluations are needed. This information-based method can help the design institute monitor the sustainable development ability of multiple suppliers in real time. The platform involves fresh information about suppliers who have not yet cooperated in the industry, thus providing a wider range of alternatives for procurement and promoting fuller competition.

6. Conclusions

Taking sustainable development as the background, this study investigated the concept of sustainable suppliers in the design industry and constructed an evaluation index system with further suggestions. Firstly, the importance of TBL in sustainable development was clarified, and the Delphi method was used to conduct three rounds of surveys to define the concept of sustainable suppliers with their keywords in the design industry. Secondly, sustainable supplier evaluation indices were collected through the literature, and a database and hierarchy system were established. Finally, based on the improved FMEA and AHP, the failure risk of the indices was analyzed with weighted risk factors to obtain the high-risk key evaluation indices, which provided a simplified scheme for enterprises to streamline the evaluation process of sustainable suppliers.

The research results provide an effective evaluation reference for the design institutes and the industry and point out improvement and development suggestions: ① pay attention to environmental sustainability from a total life cycle perspective; ② strengthen the monitoring of social sustainability; ③ combine procurement with the index system; ④ establish a sustainable supplier database; and ⑤ build an information management platform for sustainable suppliers. In addition, integrating the improved FMEA model and AHP shows its feasibility and effectiveness in differentiating the RPN results, reducing the possibility of repetitiveness value and ranking. Thus, it is suitable for supplier sustainability ranking based on evaluation indices and provides enterprises with accurate and easy-to-operate tools.

In the future, more studies on implementing a sustainable supplier evaluation index system and its key indices should be validated through case studies to support the research results of this paper. The conclusion could be further analyzed and expanded to other service industries. Moreover, the dynamic management of the index database and supplier pool is a potential research direction supporting sustainable suppliers’ long-term management and monitoring.