Framework for Evaluating the BIM Application Performance: A Case Study of a Grid Information Modeling System

Abstract

BIM has played an important role in promoting sustainable development in the construction industry. The lack of an effective system for evaluating BIM application performance has become a major obstacle to BIM application. Therefore, this study develops an indicator evaluation framework that includes benefit factors and cost factors to systematically evaluate the BIM application performance. The evaluation indicators are determined through a scientometric literature review and expert evaluations, and the AHP method is employed to assess the weights of each indicator. A performance index is established and measured through a cost–benefit measurement. The developed evaluation framework and index are applied in a case study of a grid information modeling (GIM) system implemented in a specific UHV substation project. The sensitivity of the evaluation index is further examined. Finally, the recommendations for developing BIM applications like GIM are discussed. Accordingly, this research mainly contributes to developing the BIM application performance evaluation framework and index, which can be used to assess the application performance of digital technologies in the construction industry worldwide. The case experience and recommendations could promote BIM application in the power generation construction industry.

1. Introduction

Building information modeling (BIM) is one of the core applications of information technology (IT) in the construction industry. BIM technology builds virtual buildings with geometric information through digital means. The geometric information in virtual buildings includes not only the visual information of building shape but also nongeometric information such as fire resistance grade and procurement information. Moreover, BIM technology can summarize the data throughout the whole life cycle of projects and process the document data from the design stage to the operation stage in an integrated way, which will continue to play a role in the construction and operation stage of the project. BIM can integrate all parametric modeling information, functional requirements, and component performance throughout the life cycle of a building project into a single architectural model. It includes process information such as during construction processes and service processes [1,2]. Accordingly, the application of BIM technology can improve the productivity of the construction industry and assist its transformation toward digitization [3]. However, the lack of effective BIM application evaluation measures has hindered BIM application in the construction industry [4].

With the rapid development of the national economy and the requirements of sustainable development, investment in power generation projects in China has increased rapidly. However, due to the complexity of the various on-site conditions, project stakeholders, and expectations, the question of how to enhance the construction performance and sustainability of power generation projects is a critical challenge for management practices and decision-makers. To solve these challenges, developing and applying digital technologies in substation construction projects has recently become a priority [5]. BIM technology should be an important and effective tool for power engineering projects due to its ability to achieve flexible information sharing and adaptive interaction [6]. However, BIM only reflects information on a single building project and cannot meet the needs of the grid network and crossover and information exchange between different power grids in power transmission and transformation projects. Accordingly, the application of BIM technology in power transmission and transformation projects is expected to be studied.

Based on BIM, the China State Grid Corporation has developed a computer-aided design technology that enables the digital representation of information to meet the 3D design needs of transmission and substation projects, which is named grid information modeling (GIM). GIM technology has improved the shortcomings of the previous BIM to a certain extent. However, GIM cannot meet the relevant national requirements in terms of application scope, depth of application, and user incentives [7]. Moreover, analyzing and evaluating the GIM application in substation construction projects can improve work efficiency and enable rapid development of these projects. Therefore, it is very necessary to establish an evaluation system for assessing the GIM application in the power generation construction industry.

In order to promote BIM application, this study develops an evaluation system for assessing BIM application performance. The GIM technology in China will be employed as a case study to investigate and examine the reliability of the evaluation system. The rest of this paper is organized as follows: Section 2 outlines the relevant studies on BIM and the indicators for evaluating the BIM application; Section 3 describes the methodology of the paper; and Section 4 demonstrates the application of the methodology in the case study. Section 5 discusses the obtained results, and, finally, Section 6 summarizes the conclusions within the context of the field.

2. Literature Review

2.1. Performance Evaluation of BIM Application

With the development of the digital economy and industrial digitization, BIM has gradually been applied to various fields of construction projects. For example, more and more engineering projects have been adopting BIM technology, such as power engineering [8], large airport projects [9], and hydropower engineering [10]. BIM can not only accurately and quickly measure project costs [11] and minimize project labor and time costs [12], but also conduct real-time quality information collection and processing on construction sites, identify construction problems, and support real-time quality control [13]. Moreover, based on the integration of BIM with artificial intelligence, construction robots, big data, and other new-generation information technologies, it also benefits to establish an information-based management platform, improve the collaborative work of stakeholders, and achieve multi-stage information transmission and full-life-cycle project management [14]. Therefore, BIM has been an important platform for promoting sustainable development in the construction field.

Recently, in order to efficiently promote the BIM application, its application performance has been evaluated. Li et al. established an evaluation model for the effectiveness of BIM application by focusing on owners and analyzed the correlation between the five secondary and tertiary benefit indicators of product, finance, organization, management, and strategy [15]. Chan et al. discussed the critical success factors of BIM implementation in the architecture, engineering, and construction industry of Hong Kong, aiming to improve the successful delivery of projects as well as coordination and cooperation in project design, construction, and management stages [16]. Liu et al. focused on the application performance of BIM during the construction phase [17]. Barlish and Sullivan developed a computational model for assessing the benefits of BIM, highlighting the significant impact of information transfer, change, and improved project duration [18]. Al-Ashmori et al. pointed out that the hindrance in BIM implementation is largely due to stakeholders’ lack of understanding of the technology’s benefits, and that cost, production advantage, and coordinated communication are the most important application benefits for stakeholders [3]. Deng et al. employed the fuzzy comprehensive evaluation (FCE) method to study the application benefits of BIM in project cost control [19].

Evaluating the BIM application performance could promote the BIM application in the construction field and then improve construction productivity and project management performance. Wang et al. demonstrated that evaluating and improving the application ability of BIM has a positive effect on the application efficiency of BIM, and they evaluated the application efficiency of BIM from three dimensions of application ability: technology, organizational management, and personnel using an interval gray clustering analysis model [20]. Badrinath and Hsieh found that the key factors of project delivery success using BIM technology can drive the critical success factors of the project [21]. Alaloul et al. emphasized the important role of BIM in risk identification, prediction, and mitigation during the construction process through the analysis of semistructured questionnaire data using SPSS 26 software, and they recognized that BIM can significantly improve construction project safety and reduce accident rates [22]. Therefore, it is essential to assess the performance of BIM applications in the construction industry.

Previous studies generally developed different performance indicators to evaluate the BIM application performance. Luo et al. established a BIM application benefit evaluation model based on the four dimensions of the Balanced Scorecard—learning and growth, internal operation, finance, and customers—and also analyzed the model using the network data envelopment analysis (DEA) performance evaluation method [4]. Sinoh et al. analyzed 184 questionnaires from Malaysian construction, engineering, and construction companies and found that nontechnical factors such as management, leadership, and coordination had a higher correlation with BIM performance compared to technical factors such as software and hardware [23]. Phang et al. evaluated the benefits of applying BIM to precast concrete companies and identified eight important factors that influence the successful application of BIM technology, including policy support, high-level attention to BIM, competitive advantage, design intent, accurate information on BIM, increased productivity, reduced time and support teams [24]. Mahamadu et al. used the fuzzy entropy weight method to analyze the application performance of BIM and divided the influencing factors in the pre-qualification and bidding stages into different dimensions such as ability, resources, culture and attitude [25]. Nonirit et al. evaluated the performance of BIM implementation from the aspects of efficiency, the relevance of BIM to specific environments, and its usefulness [26].

However, there is a lack of a systematic evaluation system that focuses on application benefits and costs. Yang and Chou proposed an overall benefit evaluation framework for BIM implementation, including formative evaluation (requirement evaluation, structured conceptualization, implementation evaluation, process evaluation, etc.) and summary evaluation (structural evaluation, economic evaluation, impact evaluation, secondary analysis, and meta-analysis) [27]. However, this evaluation system only assessed the BIM application benefits without considering the application cost. The cost related to BIM infrastructure, training, and operation is an obstacle to BIM application [28]. Therefore, this article comprehensively investigates the application benefits and costs of BIM. More importantly, previous studies have often overlooked the roles of application organizations, which are crucial for the successful implementation of BIM. In summary, this study develops a systematic framework for evaluating BIM applications.

2.2. Evaluation Methods of BIM Application

Previous research has adopted different research methods for the performance evaluation of BIM applications. The main evaluation methods include Delphi, principal component analysis (PCA), gray relational analysis (GRA), DEA, analytic hierarchy process (AHP), and fuzzy comprehensive evaluation (FCE). For example, Denecke et al. used the Delphi method to obtain evaluation indicators through a literature review, and obtained the health robot application evaluation model after three rounds of expert evaluation [29]. Wyke et al. identified 26 factors affecting the cost and time overrun of construction projects through interviews with project managers and a literature review, and they grouped the 26 factors into four dimensions of quality control, project preparation, user management, and project management by using principal component analysis [30]. Bai and Liu calculated the weights of factors affecting the quality of the construction process using the GRA method [31]. Luo et al. built a BIM application evaluation index system and evaluated the application performance of BIM based on DEA [4]. Gudiene et al. used AHP to analyze the weight of the critical success factors of the construction project and ranked them [32]. The comparison among them is shown in

Table 1. Common methods of performance evaluation in BIM application.

Method	Introduction	Advantages	Disadvantages	Suitable Object
Delphi	Organize experts to conduct independent evaluations, summarize and re-evaluate, and carry out round by round.	Simple operation, give full play to the role of experts, take the essence and discard the dregs.	Strong subjectivity, long operation time, and poor usability for complex problems.	Simple, unquantifiable objects.
PCA	By reducing dimensions, multiple indicators are simplified into several important indicators to avoid information duplication.	Comprehensive, objective, and comparable.	It requires a large amount of basic data, which cannot reflect the comprehensive level of a certain element, and is not for evaluating qualitative indicators.	Classify the evaluation objects.
GRA	Quantify the dynamic development state of the system and judge the development trend of the gray process.	It requires less sample size and is easy to operate.	Defining curve similarity is slow and requires comparability of the selected variables.	Technical performance, service level evaluation.
DEA	Determine the weight coefficient of each unit and evaluate the relative effectiveness and relative ineffectiveness.	The index dimension is not limited; objectivity is strong, and it is suitable for a multi-input multi-output system	Data are too sensitive; there are no absolute indicators, only to measure the relative level, not for actual conditions.	Production efficiency, scale effectiveness, etc.
AHP	According to the relationship of each influencing factor, a hierarchical structure model is formed, the weights of the indicators are determined by comparing with each other, and the importance of the overall goal is finally determined.	The principle is simple, the error is small, the result is reliable, and the combination of qualitative and quantitative.	The subjectivity is strong, the correction cannot be optimized, and the pairwise comparison of the evaluation factors is too difficult.	Establishment of evaluation index and determination of weight.
FCE	The membership function is used to determine the evaluation matrix and quantify the evaluation index.	For fuzzy systems with more qualitative indexes, comprehensive decision making can be realized and the result is clear.	If information duplication cannot be avoided, there will be subjective membership.	Qualitative index analysis, systematic evaluation.

. AHP can effectively combine qualitative and quantitative analysis, thereby increasing evaluation reliability. Therefore, this study employs AHP to effectively evaluate the performance of BIM applications.

3. Methodology

3.1. The Research Method Framework

The research method framework of this study is displayed in Figure 1. The methods consist of a scientometric literature review and content analysis, expert evaluation, AHP method, cost–benefit measurement, and case study.

The first step in Figure 1 is to use the scientometric literature review and content analysis to search relevant literature and then to identify the performance indicators for assessing BIM applications. The databases used for the review include Scopus, CNKI, and Web of Science. The search keywords were determined as “BIM application/implementation” “application/implementation performance” and “evaluate/assess”. The CNKI search is in Chinese. A total of 511 pieces of literature were obtained, and then 127 publications were initially obtained by eliminating duplicate literature and neglecting articles that were evidently not in this field. As this study focuses on the indicators of evaluating the BIM application performance, a total of 27 pieces of literature were finally selected to formulate the evaluation framework. Through a systematic review and summary of the identified literature, the indicators for evaluating the performance of BIM applications were identified.

In order to ensure the reliability and authenticity of the performance evaluation indicators, 15 experts from relevant fields were invited to further validate these indicators. The experts include three professors engaged in BIM-related research in universities, four BIM technology developers from companies, and eight BIM technology managers. All the experts have a minimum educational qualification of a master’s degree and a minimum work experience of 5 years in the field of BIM technology. The detailed information of the experts is presented in Appendix A. A three-round assessment was conducted by the 15 experts. First, the experts were invited to evaluate the rationality and comprehensiveness of the evaluation indicators. Second, the experts and professionals were asked to score the indicators to obtain the weights of each performance evaluation indicator for digital technology applications. During the scoring process, experts were not allowed to communicate with each other to reduce the influence of subjective judgments. Finally, the performance evaluation indicators were applied to actual cases, and the experts were invited to score the indicator scores in real projects based on field visits and literature reviews.

In this study, the AHP method was used to assign weights to each indicator in the evaluation system. A hierarchical structure framework was established based on the relationships among different indicators. Pairwise comparisons were then conducted to determine the relative importance of each indicator at each level of the framework, and the indicators were ranked according to the expert ratings. Based on the scores provided by the experts for the lower-level indicators of the performance evaluation system for digital technology applications, the importance weights of the higher-level indicators were derived. Different weights were assigned to different indicators to ensure the reliability and accuracy of the evaluation system.

Based on the derived weights for each indicator, the cost–benefit measurement was applied to calculate the performance index of BIM application. This measurement covers all indicators from both the benefit and cost sides in the framework. To validate the confidentiality of the index, sensitivity analysis was conducted to test the evaluation system’s sensitivity.

3.2. AHP Method for Weighting the Evaluation Indicators of BIM Application Performance

When using AHP for analysis, two tasks are crucial: establishing an indicator evaluation system, and determining a weight matrix. The indicator system has been determined by the scientometric literature review, content analysis, and expert evaluation. The weight matrix will be decided through AHP. For example, there are K evaluation factors at the first level such as $A_1 ,A_2,\cdots , A_{\mathrm{m}}$, $A_k$. The second level influencing the first-level indicator $A_k$ consists of n factors $B_1,B_2,\cdots ,B_n$. For the indicator of $A_k$, $b_{ij}$ represents the relative importance of the $B_i$ indicator relative to the element $B_j$. $b_{ji}$ is the expert evaluation score. Generally, experts are requested to provide a judgment on the mutual importance of the indicators at each level of the system evaluation index system.

AHP uses Sati’s 1–9 scale method to construct a judgment matrix [33]. The scale values $b_{ij}$ are shown in

Table 2. Sati’s 1–9 scale.

Scale	Representation
1	Indicates equal importance factors of $B_i$ and $B_j$
3	Indicates $B_i$ slightly more important than $B_j$
5	Indicates $B_i$ clearly more important than $B_j$
7	Indicates $B_i$ much more important than $B_j$
9	Indicates $B_i$ very much more important than $B_j$
2, 4, 6, 8	Represents the intermediate value of the two adjacent judgments mentioned above

.

The hierarchical weights can be attributed by calculating the eigenvectors of the judgment matrix.

$$BW={\lambda }_{MAX}W$$

(1)

Among these, ${\lambda }_{MAX}$ is the maximum eigenvalue of $B$ and $W$ is the corresponding ${\lambda }_{MAX}$ normalized eigenvector. Using the square root method, the calculation is as follows:

$$W_i=\frac{\sqrt[n]{{\prod }_{j=1}^n{b}_{ij}}}{{\sum }_{i=1}^n\sqrt[n]{{\prod }_{j=1}^n{b}_{ij}}}$$

(2)

Among these, $i,j=1,2,\cdots ,n$. Then $W=(W_1,W_2,\dots W_n)$ is the desired feature vector.

$${\lambda }_{MAX}=\frac{1}{n}{\displaystyle \sum _{i=1}^n\frac{{(BW)}_i}{W_i}}$$

(3)

Among these, ${(BW)}_i$ represents the ith component of $BW$.

In order to ensure that the conclusions of the AHP are reasonable, it is also necessary to conduct consistency checks on each judgment matrix. The consistency check adopts the following formula:

$$CR=CI/RI$$

(4)

Among these, $CR$ is the random consistency ratio of the judgment matrix.

$RI$ is the average random consistency index of a matrix. $CI$ is the general consistency indicator of the judgment matrix. The calculation formula for $CI$ is as follows:

$$CI=\frac{{\lambda }_{max}-n}{n-1}$$

(5)

$n$ is the order of the judgment matrix. When $CR<0.10$, it is believed that the judgment matrix has satisfactory consistency. Otherwise, it is necessary to adjust the judgment matrix to ensure satisfactory consistency.

3.3. Cost–Benefit Measurement for the BIM Application Performance Index

Application performance is an effectiveness indicator that measures application benefits under certain application costs. The BIM application performance measures and compares the benefits and costs that are resulted from BIM application. Therefore, this study employs a cost–benefit measure to evaluate the BIM application performance. For example, there are N ($i=\mathrm{1,2}, \dots , N$) benefit indicators and M ($j=\mathrm{1,2},\dots , M$) cost indicators. The application performance index P can be expressed as:

$$P=\frac{{\sum }_{i=1}^N{w}_i{Y}_i}{{\sum }_{J=1}^M{w}_j{X}_j}$$

(6)

$w_i$ and $w_j$ are, respectively, the weight coefficients of the evaluation benefit and cost indicators. $Y_i$ and $X_j$ are the result of experts scoring the benefit and cost indicator, respectively. The higher the value of the index, the higher the BIM application performance in the project.

4. Developing the Three-Level Indicator Framework for Evaluating the Application Performance of BIM

When a BIM application evaluation system is to be constructed, it is necessary to ensure that the primary indicators truly reflect the application performance on multiple dimensions. Based on the Scientometric literature review and content analysis, the developed framework and the determined weights of the three-level framework indicators for evaluating BIM application are shown in

Table 3. Weights of indicators in the evaluation system.

Groups	Primary Indicators	Secondary Indicators
Application capability performance [34] (0.44)	BIM management Capability [20] (0.398)	Attention paid by executives [20] (0.447)
		Attention paid by middle managers [20] (0.287)
		Degree of cooperation of technicians [20] (0.265)
	Ability to formulate BIM implementation plan [35] (0.331)	Completeness of workflow [36,37] (0.716)
		Completeness of software maintenance scheme [35] (0.126)
		Completeness of database maintenance scheme [35] (0.158)
	BIM technology utilization level [36] (0.271)	Number of disciplines covered [36] (0.543)
		Accuracy [36] (0.340)
		Application degree [35] (0.117)
Application organization performance [4,20] (0.30)	Technical basis [35] (0.625)	Number of BIM hardware facilities [20,35] (0.617)
		Number of BIM team members [20] (0.261)
		Database capacity [35] (0.122)
	Technical environment [36] (0.214)	Technical standards [38] (0.642)
		Internal regulations of the enterprise [36] (0.180)
		Relevant policies outside the enterprise [36] (0.178)
	Team collaboration [36] (0.161)	Employee training [20] (0.457)
		Communication between employees [37] (0.405)
		Interdepartmental cooperation [36] (0.138)
Application economic benefit [35,39] (0.26)	Cost benefits [24] (0.546)	Cost savings of scheme design [4,38] (0.515)
		Cost savings of construction scheme [4,38] (0.305)
		Cost savings in personnel, materials [4,38], and machinery (0.180)
	Construction period benefits [14] (0.257)	Construction period savings of project design [37] (0.517)
		Construction period savings of construction process [37] (0.246)
		Construction period savings of personnel, materials, and machinery [37] (0.237)
	Enterprise competitiveness [40] (0.197)	Customer satisfaction [40] (0.443)
		Number of project awards [40] (0.364)
		Enterprise reputation [40] (0.193)
Application cost [41,42]	Design phase costs [4,38] (0.545)	Project design costs [4,38] (0.421)
		Project promotion costs [4] (0.405)
		Technology R&D costs [4,36,38] (0.174)
	Construction phase costs [4,38] (0.240)	Operating costs [4,38] (0.655)
		Management costs [38] (0.215)
		Maintenance costs [38] (0.130)
	Other costs [38] (0.215)	Sunk costs (0.611)
		Risk costs [38] (0.231)
		Efficiency costs [20] (0.158)

. The weight coefficient of each indicator is a value determined in line with the indicator’s importance through the AHP method. The framework comprises four subsystems, the benefit side including application capability, application organization, and application benefit, and the cost side of application cost.

4.1. The Capability Performance of the BIM Application

BIM application capability refers to the comprehensive ability to introduce and apply. The application process of BIM includes three aspects: management, technology, and talents. BIM application capability is not only the ability to apply BIM to the production process, but also it is a combination of the enterprise’s ability to manage BIM, the enterprise’s ability to formulate a BIM implementation plan, and the designated personnel’s ability to effectively use BIM [34]. Therefore, based on the expert evaluation and AHP calculation, the weight of the capability performance of the BIM application is with the highest value of 0.44.

• BIM management ability: The BIM management ability of enterprises can be analyzed from the three levels of senior management, middle management, and technical workers. The application of digital technology in enterprises generally lies in whether the senior managers can accept the application of BIM in actual projects. Middle managers play an important role in uploading and issuing, not only accepting the orders of senior managers but also supervising the practical application of digital technology. Although it is not possible to decide to apply BIM technology by technical workers, they can implement the relevant planning and scheduling. Finally, the weight value is measured as 0.398 at the secondary level.
• Ability to develop a BIM implementation plan: The plan needs to cover workflow, BIM software maintenance plan, and database maintenance plan. Among them, workflow accounts for the most weight, as it can provide a good basis for information exchange between different links and departments, which is crucial for the control of construction project schedules and task scheduling. Building construction is an extremely large and complex system, involving many professionals, participants, and users; the information generated in the production process is huge, and the information changes dynamically with the construction and use process and needs to be coordinated by many parties. In the construction process, it is necessary not only to maintain and update the BIM software, but also to maintain the database as a whole to ensure that the BIM data meet the needs of the construction project. By establishing a work plan network diagram, managers can organize the whole planning task strictly according to the objective law of production, so that the formulation and execution of the production plan can be based on scientific calculation. Managers can understand the schedule of production and production requirements for their work, and technicians can clearly understand their position and role in the overall situation. Finally, the weight value is assigned as 0.331 in the group of primary indicators.
• The level of BIM technology utilization. The accuracy, scope, and the number of disciplines covered by BIM can be used as observation indicators to assess the level of BIM utilization. BIM technology can not only carry out information processing of engineering projects, but also realize the effective storage, rapid and accurate calculation, and analysis of massive data in the process of project management. Based on the efficient calculation, accurate data, and scientific analysis ability of BIM, the traditional management status quo relying on experience can be greatly improved, and the project refinement and enterprise intensive management and control can be gradually realized. BIM can integrate multi-stage resources and promote multi-agent collaboration [36]. Therefore, the accuracy and scope of BIM application are used to evaluate whether the BIM model established by the enterprise is accurate and whether it is completely applied to the production process. The number of disciplines represents whether different disciplines can use BIM in practical work. Covering a comprehensive range of disciplines with BIM will promote information exchange between departments during the whole life cycle of construction projects. In summary, the weight value is evaluated as 0.271.

4.2. The Organization Performance of the BIM Application

Organizational ability is an organic combination of unique abilities gradually formed in the process of enterprise development. Application organization generally includes three elements: technical foundation, technical environment, and teamwork [20]. Among them, the technical foundation decides the success and level of BIM application, the technical environment provides an effective guarantee for the practical application of BIM technology, and the team cooperation provides a driving force for the application of BIM technology. The weight value of the organization performance of the BIM application is evaluated as 0.30.

• Technical foundation of BIM application: The technical foundation of BIM application includes not only the hardware facilities of BIM, but also the organization of BIM related personnel. When evaluating the technical basis of BIM application, the number of BIM hardware facilities and the number of BIM technicians are selected as evaluation indicators. The key to the application of BIM technology in actual projects lies in the purchase of corresponding BIM hardware facilities and the training of professional talents using BIM technology. Therefore, the number of facilities and personnel needs to be mainly increased in the process of promoting the application of BIM technology. The increase in the number of facilities and personnel will lead to excessive information capacity, so it is necessary to ensure that necessary information management is adopted in the application process. Accordingly, this indicator has the largest value of 0.625 in this cluster.
• Legal and technical environment: In the application organization of BIM, the technical environment plays an important role from both inside and outside the enterprise [35]. The technology environment is assessed against the dimensions of technical standards and regulations/policies within and outside the enterprise. Technical standards indicators are measured mainly by the number of technical standards currently adopted by enterprises, while regulatory/policy indicators are the policies and programs currently driving BIM adoption. The application of new technologies cannot be separated from the support of policies. Providing a good technical environment to ensure the implementation of technologies is an important factor that the government and enterprises need to comprehensively consider when formulating relevant policies at this stage [36]. Finally, the weight result is 0.214.
• Team collaboration in BIM application: Team collaboration in BIM application refers to the degree of collaboration of the entire BIM team in the process of striving to achieve the enterprise objectives [34]. The level of team collaboration can effectively reflect the level of the application. Team cooperation includes training of team members, communication between team members, and cooperation between departments. When applying BIM technology, enterprises will take into account the training of relevant professional and technical personnel and often ignore the level of cooperation between members and departments, but the research results show that the level of team cooperation is as important as the importance of training professional and technical personnel. Therefore, in the evaluation process of team cooperation in the BIM application, it is necessary to fully consider three key factors: personnel, team, and department. Compared to the other two indicators, this indicator has the least weight value of 0.161.

4.3. The Economic Performance of the BIM Application

The economic performance of the BIM application generally divides into visible and invisible benefits. When application benefits are manifested in saving costs, they are visible benefits. The application benefits associated with quality, safety, and enterprise competitiveness are considered invisible benefits. This study evaluates the application benefit in relation to the dimensions of cost benefits, construction period benefits, and enterprise competitiveness in different project stages [39]. The weight value of this primary indicator is assessed as 0.26.

• Cost-effectiveness. Cost-effectiveness is divided into three elements along with the application stage: design optimization; structure optimization; and resource management. The cost-effectiveness of design optimization is based on the cost savings in collision detection and optimization scheme design, the cost-effectiveness of construction optimization demonstrates the cost savings in visual construction guidance and construction scheme optimization, and the cost-effectiveness of resource management presents the cost savings in personnel, materials, machinery, capital, etc. The application of BIM technology in the design stage can not only directly reduce the design cost, but also reduce the indirect cost caused by structural collision in the construction stage [4]. The weight result is measured as 0.546.
• Construction period benefit. Construction projects generally have a long construction period, and the application of BIM technology to actual projects can provide a reliable guarantee for construction period management [37]. In the traditional construction design stage, the building, structure, and equipment are designed in stages and in order, and BIM technology can help the three links to carry out synchronous design, greatly reducing the time required in the design stage. In the technical disclosure of the construction stage, the visual model of the building structure is provided to reduce the rework events caused by the imperfect construction drawing design. Finally, 0.257 is its weight value.
• Enterprise competitiveness. Enterprise competitiveness is evaluated based on three dimensions: customer satisfaction; number of project awards; and enterprise reputation. Customer satisfaction refers to whether customers are satisfied after the application of BIM, the number of project awards measures whether the use of BIM has improved project quality, and the reputation of the enterprise gauges the improvement in the enterprise’s influence within the industry after the application of BIM. Completely, in this group, this weight indicator is assessed with the minimum value of 0.197.

4.4. The BIM Application Cost Performance

The application of BIM can be divided into four phases: the decision-making phase; the design phase; the construction phase; and the post-completion settlement phase. Most BIM application costs occur in the design and construction phases. Therefore, this study focuses on the design and construction phases in analyzing BIM application costs. In addition, a host of invisible costs should be considered. Therefore, the evaluation of the application cost covers the costs in the design phase, the construction phase, and other costs.

• Design phase costs. The design phase is critical to the project. The cost in the design stage mainly includes three elements: the direct cost incurred, the BIM technology R&D cost, and the engineering design costs. Digital technology will display different structural models depending on the input parameters of the segment, and through a specific algorithm, the final output is the best solution for a specific project [26]. However, relevant BIM developers pointed out that the process of developing BIM technology needs investing the corresponding funds, which leads to a low level of BIM application in actual projects. When applying BIM technology, enterprises need to comprehensively consider research and promotion costs and cost savings in the design stage. The research costs are the expenses of searching and studying how to design, develop, and apply BIM in the enterprise. The promotion costs are incurred to make BIM better known to the stakeholders and training costs. Therefore, the weight value of this indicator is 0.545, which has the largest influence in this group.
• The cost of the construction phase. Typically, the majority of a project’s expenditure occurs during the construction phase [42]. After the design phase, both machines and labor incur costs. For BIM application projects, the construction phase is the key stage to the control of operating costs, management costs, and maintenance costs. However, in the actual construction process, due to the input of BIM implementation will produce the corresponding equipment operation cost and database maintenance cost, so the weight of operation cost and maintenance cost is larger in the evaluation process of construction stage cost. Finally, the weight value of this indicator is 0.240.
• Other costs. Other costs include sunk costs, risk costs, and efficiency costs. Some experts suggested adding two additional indicators, namely “sunk costs” and “efficiency costs”. When implementing new digital technologies in actual construction projects, it can be achieved by reducing sunk costs through selling obsolete software or equipment and reducing efficiency costs through improved production efficiency with digital technologies. Therefore, it would result in an incomplete evaluation if without considering the two costs. Sunk cost can be identified in the assessment process as the cost incurred by the enterprise when investing in the application of BIM technology and the total cost incurred by selling abandoned equipment. Risk cost is accompanied by the whole life cycle of a construction project, which is faced with risk cost both in the bidding stage and the construction stage. In the bidding stage, the bidder must make the quotation of the entire construction project scientific and reasonable and put forward high requirements on the cost budget. BIM technology can help the bidder plan the total cost, so as to reduce the increase in risk cost caused by bidding failure. The value of 0.215 is its weight value.

5. Case Study

5.1. Description of the Case Study

This study selected an important hub substation in the eastern region of China as the case study. This project won the award in the first BIM competition in the Chinese construction industry. The case was one of the first batches of ultra-high-voltage (UHV) trial projects after GIM was introduced in China. Detailed designs were worked out for every phase of the project before the commencement of construction, and GIM was fully implemented in the project. The total land area of the project is 13.3785 hectares. Basic information about the scale of the project is given in

Table 4. Introduction information of the case project.

Scale	Current Period	Prospect
Main transformer (MVA)	2 × 3000	4 × 3000
1000 kV outgoing line (circuit)	4	8
1000 kV high-voltage reactor (MVAR)	2 × 840	6 groups
500 kV outgoing line (circuit)	4	8
110 kV low-voltage reactor (MVAR)	2 × 2 × 240	4 × 2 × 240
110 kV low-voltage capacitor (MVAR)	2 × 2 × 210	4 × 4 × 210

.

5.2. GIM Development and Application

Based on BIM, GIM is a technical standard developed by the State Grid Corporation of China to meet the 3D design needs of transmission and substation projects, and it is applicable throughout the life cycle of such projects. GIM is a computer-aided design technology that enables the digital representation of information. GIM not only enables the detailed design of the project, but also allows for real-time monitoring of equipment and facility operations, effectively controlling project costs and building energy consumption. As a result, GIM has gradually gained attention from industry professionals and has been applied in substation construction projects.

As the first GIM pilot UHV project of the State Grid, the degree of 3D design and application is the top priority of the project, which lays the foundation for the promotion of UHV engineering design in the later stage. 3D models are used in the design stage for rapid layout and repeated optimization to determine the optimal scheme. After the design is completed, the design data will be uploaded to the Enterprise Resource Planning (ERP) platform for accurate procurement, reducing material waste, and applying technical means such as construction progress simulation and complex process simulation in the construction stage. The data are associated and integrated through coding to form a data information database, and the operation and maintenance platform can extract the information, perform intelligent management, and control simulation maintenance, equipment disassembly training, and other functions, and finally realize the digital application of the whole life cycle.

5.3. The Evaluation of GIM Application Performance

First, the evaluation results for the secondary indicators of the project were calculated using the weighted average method and the scoring of the decision experts. Then, the results of the primary indicators could be estimated. Finally, these indicators were weighted and summed into the group level, and the performance results of the GIM application were obtained. The application capability score of the project was 86.75; the application organization score was 85.11; the application benefit score was 86.53; and the application cost score was 81.51. Based on Equation (6) in Section 3.3, the GIM application performance index was calculated as 1.058. The index greater than 1 indicates that the GIM application benefits are higher than the application costs. GIM has been effectively and successfully developed and applied in the case study.

In order to demonstrate the effectiveness of the evaluation, the sensitivity analysis with different weight values was implemented to observe the impacts of different weight values on the final performance results. The specific weight change is shown in

Table 5. The application performance values under different weight settings.

Number	Change the Weight of Benefit Indicators			Application Performance Value
Group 1	a = 0	b = 0.5	c = 0.5	1.0528
Group 2	a = 0.5	b = 0	c = 0.5	1.0630
Group 3	a = 0.5	b = 0.5	c = 0	1.0542
Group 4	a = 0.5	b = 0.25	c = 0.25	1.0585
Group 5	a = 0.25	b = 0.5	c = 0.25	1.0535
Group 6	a = 0.25	b = 0.25	c = 0.5	1.0578
Group 7	a = 1	b = 0	c = 0	1.0642
Group 8	a = 0	b = 1	c = 0	1.0442
Group 9	a = 0	b = 0	c = 1	1.0615

, where a, b, and c represent the weights of application capability, application organization, and economic benefit, respectively. The value changes of a, b, and c represent the weight changes of these indicators. It can be seen that whatever the weight changes, the application performance values are all better than 1. Therefore, GIM has been successfully implemented in the power generation project. The rich experience in 3D design gained in this project can inform the GIM design of subsequent UHV stations.

5.4. Recommendations for GIM Application

(1)

Improve GIM application performance from the perspective of developing application capability

The application standards of GIM for power enterprises should include not only data exchange standards but also the application scope and depth, submission formats, and development standards according to the requirements or project objectives of power enterprises. Improving the IT level in the power industry requires promoting the integration of GIM technology within the industry and, ultimately, improving the information management of power projects and power lines as a whole. Therefore, in a future where application standards are better generalized, comprehensive GIM applications will be realized to support future power engineering management activities in a networked and service-oriented manner. At the same time, many verification functions that cannot be achieved by 2D technology will be realized, such as gap verification with points and wind deflection verification.

The overall GIM application performance can be enhanced by improving the capability to apply GIM. Improving an enterprise’s GIM management capability and implementation plan formulation capability can effectively enhance the GIM application performance of the enterprise. It is also feasible to enhance the GIM application performance of an enterprise by improving the utilization level of GIM technology. To improve the GIM management capability of an enterprise, the executives and middle managers must attach great importance to GIM. However, this is not enough. It is also necessary to enhance the cooperation and motivation of the technicians through various measures. In the process of adopting GIM technology, the existing workflow will be affected. A careful analysis of the changes in the workflow can help planners formulate an effective GIM implementation plan. When considering how to improve the GIM application performance of an enterprise, it is necessary to pay attention to the number of disciplines covered, the accuracy, and the extent of the application.

(2)

Improve the GIM application performance from the perspective of the application organization

Leadership is a key driver in the development of GIM technology. Whether at the national or project level, leadership determines the depth of GIM technology application and the binding relationship that exists between leadership and the depth of GIM technology application, affecting the standardization of GIM technology. After implementing a series of measures, it is necessary to check whether the measures have successfully improved the GIM application performance. In this regard, application organization is an indispensable observation variable, and the technical basis, technical environment, and team collaboration are important components of application organization. To improve the GIM application performance, it is necessary to increase the number of GIM hardware facilities and team members, as well as develop regulations and technical standards to create an excellent technical environment. In application organization, team collaboration also plays an important promoting role, and training employees and improving interdepartmental communication and cooperation are effective measures to improve application organization.

It is also important to strengthen the training of complex talents. The development of GIM technology ultimately depends on the ability to build networks, but, currently, many technicians only know how to use the software and lack expertise in power project management, while some electrical engineering management professionals are lacking in both software and network knowledge. Both situations seriously restrict the development of GIM technology to the continuous improvement stage in China. Training of interdisciplinary personnel who understand GIM technology and electrical engineering management expertise are urgent requirements for the development of GIM technology.

(3)

Improve GIM application performance from the perspective of economic benefits

Power engineering projects should develop and implement project information strategies and plans according to their requirements and the functions of GIM technology. In order to achieve data sharing and coordination, project stakeholders should prepare data software and hardware, build a data environment, and establish a management operation mechanism that includes permission control for various users, version control of software and files, and consistency control of the model. The GIM model is not just a simple geometric model but also comes with complete data information that is logically linked to the database and can be easily transferred digitally.

For enterprises, introducing GIM technology will obtain benefits in all these aspects. Therefore, accurately evaluating the application benefit of GIM technology is of great importance. The application benefit of GIM is reflected mainly in the saving cost, construction period benefits, and enterprise competitiveness. The cost benefits and construction period benefits are visible benefits that can be measured. In contrast, enterprise competitiveness is a potential benefit that is crucial to the enterprise despite its invisibility. Therefore, when evaluating the benefit of applying GIM technology, we should not only pay attention to the cost savings resulting from better design, reduced consumption for personnel, materials, and machinery, and shortened GIM development period, but also consider the improvement of enterprise competitiveness.

6. Conclusions

BIM application has played an important role in promoting sustainable development in the construction area. Although some studies have attempted to evaluate the application performance, different studies generally selected various performance evaluation indicators. There is a lack of a systematical framework for evaluating BIM application performance. Therefore, this study developed an assessment framework that includes capability, organization, economy, and cost to systematically evaluate the application performance of BIM. First, the scientometric literature review, content analysis, and expert evaluation were used to identify and assess the benefit factors of capability performance, organizational performance, and economic performance and the cost factor of application cost. Subsequently, the AHP method was used to calculate the weights of the benefit and cost indicators, where the weights of application ability, organization, and economy were 0.44, 0.30, and 0.26, respectively. Additionally, an empirical analysis of the evaluation framework was conducted using a case study of a UHV project. The application performance was further examined using the cost–benefit measurement method. Finally, recommendations were offered to promote the sustainable development of GIM in power grid construction projects.

The developed framework is a three-level indicator evaluation system for the application benefits and cost. In the benefit group, the application capability performance is with the highest weight value, consisting of BIM management capability, ability to formulate a BIM implementation plan, and BIM technology utilization level. The following evaluation performance is the application organization performance, which consists of technical foundation, technical environment, and team collaboration. The application economic benefits including cost benefits, construction period benefits, and enterprise competitiveness have the least influence in the benefit group. The application cost consists of design phase costs, construction phase costs, and other costs. Based on the benefit and cost evaluation indicators, the application performance index is established. Finally, the developed evaluation framework and index have been implemented in a case study of a GIM system. The performance index in this case is bigger than one, indicating the GIM system has played a positive role in promoting construction enterprise and project management performance. Promoting the application capability, organizational ability, and economic benefits can improve the application performance of BIM.

The main contributions of this study are summarized as follows: (1) the three-level indicator framework for evaluating the BIM application has been developed; (2) the application performance index was established; and (3) the recommendations for how to promote the application of digital technologies were provided. The developed performance evaluation indicators and index can be applied to assess the application performance of digital technologies in the construction field of the world. The application experience and recommendations developed from the GIM case can be used for developing the BIM application in power generation projects around the whole world. However, this study also has limitations, such as: (1) the evaluation weight and values are mainly developed through expert evaluation in China, which may lead to a slight lack of objectivity and may be not suitable for other countries; (2) the evaluation indicators for the application benefits are established from the company and project levels, without considering other benefits such as for social and government benefits. Future research could further investigate the application evaluation framework and index for promoting the sustainable development of digital technologies in the construction field. The GIM system has great potential to improve the performance of enterprise and project management performance, which is expected to be studied in the power generation construction industry worldwide.