Innovation Performance Indicators for Architecture, Engineering and Construction Organization

Abstract

It is known that organizations can gain a competitive advantage only by managing effectively for today, while simultaneously creating innovation for tomorrow, and sustainability is one of the innovative strategies in major architecture, engineering, and construction (AEC) organizations. Innovation is vital to AEC organizations’ growth, yet most do not have a comprehensive measurement of innovation performance. Similar to the balanced scorecard approach, key indicators should be identified for the measuring of innovation performance to facilitate management. This article presents a study by using a triangulation approach that integrates systematic literature reviews and two-step consultations with experienced senior professionals to compile a set of key indicators for innovation performance measures for the AEC Industry.

1. Introduction

Much research has argued that the technological era of the 21st century is based on knowledge, information, and innovation [1,2,3]. Tushman and Nadler, in their study [4], emphasize that organizations can gain a competitive advantage only by managing effectively for today, while simultaneously creating innovation for tomorrow. However, a report presented by the Boston Consulting Group for the World Economic Forum in 2017 [5] indicated that the Architecture, Engineering, and Construction (AEC) industry had largely not benefited from the process, product, and service innovations. The AEC industry has been hesitant to embrace the latest innovation opportunities fully, and its labor productivity has stagnated or even decreased over the past 50 years [6]. The report suggested firms in the AEC industry did not emphasize enough innovation. For example, Virtual Design and Construction (VDC), as defined by Kunz and Fischer [7] as the use of integrated multi-disciplinary performance models of design–construction projects to support explicit and public business objectives, offers a platform for effective management and the innovation development of AEC. Though VDC has been introduced for years, many AEC firms still do not fully recognize this platform [8]. This indeed is not a new finding. Many researchers have shared a similar view [9,10]. A report published by the National Endowment for Science, Technology and the Arts of the United States in 2007 [11] has indicated that the AEC industry is regarded as a low innovative industry as it cannot match the conventional policy definition of innovation. The major factors causing the lag behind situation can be summarized as follows:

(1)

The industry largely favors a traditional way of business procedures as the acts of the parties in the industry are rigidly restricted by procurements and data are presented in paper documents [6].

(2)

The end-users of the product (a building or structure) may favor conventional building form and/or building style rather than contemporary design [12,13].

(3)

As the designers and contractors work under difficult time and cost restrictions, in order to minimize the risk of the rejection of building plans, the client, designer and contractors always prefer the design and construction to be in compliance with the prescriptive building codes rather than using a performance-based approach [14].

Though AEC firms have been plagued by low innovation levels, characterized by lagging productivity growth, limited digitalization, limited use of new materials and inefficient project management, they have recently paid more attention to innovation in their organizational strategies; in particular, sustainability is a widely used innovative strategy. A review on the annual reports [15] of 10 major AEC firms in the US has revealed that they all pay attention to innovation. A summary of the relevant descriptions in the annual reports is given in Appendix A. The descriptions presented in the reports have indicated that the firms have their strategies on innovation. However, it is not clear if the AEC firms have managed the innovation related strategies effectively to ensure that they remain competitive for the future.

It is well known that performance measurement and analysis are crucial for an organization’s management [16,17,18]. Managing innovation, as part of the general management of the organization, should also be supported by innovation performance measurement. Innovation performance measurement is considered a broad concept which includes the evaluation of various aspects such as firms’ strategies on innovation, ideas and knowledge; customers’ needs and the market situation; learning, training and knowledge sharing; and employees’ engagement and creativity, etc. [19,20] In other words, it is logical for the AEC firms’ management to understand innovation performance [21,22,23]. In the general management field, there are studies on performance measures for the innovation of organizations [24,25,26,27,28] and countries [29,30], but can these findings be directly applied to AEC firms? To respond to this question, we have to examine the characteristics of the AEC industry.

It is known that the AEC industry is extensively regulated [31]. Modern buildings and civil works are complex. Planning permission and the approval of building plans are normally required at the planning and design stage. License and consent are always required for implementing building works. Occupation permits and operation licenses should be obtained prior to the use of the product (the building). The low-price rule and tight schedules are always applicable to the construction process. Moreover, no two sites are the same, which implies that no two products are identical. Under this circumstance, the industry pays more attention not only to the product, but also to the process. Manufacturing, information technology and other industries may usually emphasize the products more than the process when measuring innovation performance, as product measurement can be easily quantified and objectively represented. This may cause the AEC firms difficulties in managing innovation in the same way as other industries. For example, a contractor (construction firm) may invest in developing innovative construction techniques or technologies, such as developing robots for laying bricks, in order to improve its productivity and innovation. Though the robot itself may be a product, the goal of creating this robot is not for selling, but for the enhanced productivity, hence, process innovation.

The manufacturing industry, pharmaceutical industry and software industry, for instance, may consider a new patent or product as the key to success. They may depend on patents to provide unique products for customers, generating necessary revenue, and reinvesting a portion of this revenue in innovation for sustained competitiveness. They may evaluate financially new product sales or revenue growth as the indicator for innovation. Other common evaluation indicators include measuring the number of patents, Research & Development (R&D) expenditures, trademarks or number of new products produced. These indicators are mainly designed for industries where an innovation is typically a new product for customers and can be quantified through sales and investment on generating the innovation. However, it is not a common practice for AEC firms to apply for patents to measure new product sales or trademarks, etc. Zawawi et al. [32] in their study have indicated that each industry or firm has their own specific method of innovation to ensure good performance; therefore, a specific innovation model for each industry should be developed. In view of the specific characteristics of the AEC industry, it is suggested that the innovation performance measures adopted in other industries should not be directly applied. Instead, modification is needed in order to effectively apply innovation to the AEC industry. The key objective of this study is to explore the current pool of innovation indicators used in other industries as described in the body of literature and review these indicators. Through professional judgement from practitioners in senior management roles, a set f indicators that are most appropriate for the AEC industry can be established.

2. Previous Studies

In order to explore the previous studies on innovation performance for the AEC industry, a literature review by keyword search on Web of Science (WoS) [33] was performed for the period from 1 July 2011 to 30 June 2021 and is summarized in

Table 1. Keyword Search on Web of Science (1 July 2011 to 30 June 2021).

Keyword Search by:	No. of Articles	Remarks
Innovation + Performance + Measure	4	None relating to AEC industry
Innovation + Performance + Evaluation	6	None relating to AEC industry
Innovation + Performance + Assessment	2	None relating to AEC industry
Innovation + Performance and Construction	19	9 out of 19 articles relating to AEC industry

. Nine articles (listed in

Table 2. Keyword Search on Web of Science (1 July 2011 to 30 June 2021).

Articles	Key Objectives of the Research (Information Extracted from the Abstract of the Article)
1. Li, YF; Song, Y; Wang, JX; Li, CW [34]	Intellectual capital not only has a direct positive influence on the innovation performance of construction enterprises, but also positively affects their innovation performance through knowledge sharing.
2. Zheng, JW; Wu, GD; Xie, HT [35]	Project managers should promote a higher stimulation of a leadership behavior, encouraging knowledge management, and establishing social capital, thus improving the innovation performance of project-based organizations in construction projects.
3. Wang, Q; Zhao, LW; Chang-Richards, A; Zhang, YY; Li, HJ [36]	The research results can not only improve an understanding of effects of social capital on the innovation performances of construction enterprises, but also validates the importance of knowledge transfer in stimulating innovation performance.
4. Kim, DG; Choi, SO [37]	It is found that any improvement of IT convergence innovation competence such as business efficiency IT index, collaboration IT index, and strategy management IT index has a positive impact on the production process, financial performance, and customer satisfaction with the services the companies provided.
5. Xue, XL; Zhang, RX; Wang, L; Fan, HQ; Yang, RJ; Dai, J [38]	The decomposition of a collaborative relationship with network analysis contributes to a better understanding of the innovation process in construction projects. In particular, key nodes which influence construction innovation through collaborative relationships are revealed and analyzed.
6. Chen, T; Huang, GQ; Olanipekun, AO [39]	This study reveals the mutual effects of the factors in the inner innovation system in Large Scale Construction Enterprises and provides an effective model for internal systems analyses in the construction industry and in other sectors.
7. Ozorhon, B [40]	Collaborative working among team members and strong commitment prove to be the primary enablers of innovation; reluctance, inexperience, and cost are regarded as barriers to innovation. This study helps to develop a better understanding of the inter-organizational nature of construction innovations, thereby improving innovation performance.
8. Lai, XD; Liu, JX; Georgiev, G [41]	An integration innovation management evaluation model is needed for the sustainability evaluation in the construction practice. Its development is the main scientific objective of this study, by taking into consideration the entire life cycle assessment and various other factors.
9. Le, Y; Wan, JY; Wang, G; Bai, J; Zhang, B [42]	The purpose of this paper is to analyze the relationship between the demographic characteristics of top management teams (i.e., age, gender, administrative level, senior management experience and educational background) and megaproject performance, with respect to schedule, cost, quality, safety and technological innovation. The findings revealed that age has a significant influence on schedule performance; gender has a significant influence on safety performance; senior management experience has a significant influence on cost performance; and educational background has a significant influence on both schedule and technological innovation performance.

) were found to present studies on innovation performance related to the AEC industry. Only Science Citation Index (SCI) or Social Science Citation Index (SSCI) listed journals are considered in this study, thus the search uses WoS to ensure quality inputs. Details of the selection process are elaborated on in the information collection section.

The aforesaid articles present research on various issues of innovation performance. However, they do not specifically relate to the establishment of an innovation performance evaluation approach.

Today, the rapid advancement of information technology and new materials have changed our working patterns and living behavior. Even though the AEC industry lags behind others in adopting innovation, new technologies/innovative approaches can help the industry work efficiently and improve our architectural environment. Several previous pieces of research have acknowledged the management of new technology, such as technology implementation dynamics in project teams [43]; dynamics of project-based collaborative networks for BIM implementation [44]; understanding and managing systemic innovation in project-based industries [45,46]; modeling the circumstances under which a management-defined communication structure can add value to an organization [47]; organizational culture and knowledge transfer in project-based organizations [48]; and exploring key factors that influence the design of the project-based organization [49]. Nevertheless, the works which access innovation performance for the AEC industry are limited.

There are research works in the general management field, which may provide insights for studying the innovation performance for AEC organizations. The purpose of measuring performance through tracking the Key Performance Indicators (KPIs) is to enable reliable information, both quantitative and qualitative, to be used for benchmarking purposes, critical for any organization aiming to operate effectively [50]. Innovation KPIs allow managers to better understand the organizations’ strengths and weaknesses, identify room for improvement and monitor success.

In general, innovation performance should not be evaluated by a single indicator [51]. It should be measured by a complex evaluation process which comprises a set of indicators relating to the organization strategies, process, and outputs:

Innovation Performance = f(set of indicators)

(1)

To formulate innovation performance evaluation, two main challenges need to be addressed:

(1)

The identification and formulation of evaluating indicators;

(2)

The establishing of methods to quantify and measure in conformance with the indicators.

In general, there are two terms that should be clarified: indicators and metrics. In this study, we define the terms as:

(1)

an indicator is used to measure and indicates performance and success;

(2)

a metric is solely a judgement, whether it is a number or a qualitative description within an indicator that helps track performance and progress.

The establishment of Key Performance Indicators (KPI) for measuring innovation performance has been reported [52,53]; however, there appears to be little agreement about what should be measured and how it should be measured [54].

Direct measurement in terms of financial return may seem to be a simple indicator for measuring innovation performance. However, Henttonen et al. in their study [55], indicated that the importnace of financial measures for innovation is not significant. Measurability should not be considered as a primary criterion for selecting indicators, especially when measuring innovation performance. For example, innovativeness, a broad conceptual term to represent a perception for an idea, is widely chosen by managers to judge innovation performance [56]. As mentioned above, some firms measure new product sales or revenue growth, the number of patents, R&D expenditures, trademarks, or the quantity of products produced as the indicators for innovation in practice. However, Chobotová and Rylková, in their study [57], commented that these measurements do not necessarily correlate to innovation activities. Moreover, such measurements may not be applicable to the AEC industry. For instance, the adoption of building information modelling (BIM)/virtual design and construction (VDC), considered one of the innovation initiatives for design, construction, and construction project management, cannot be directly measured through the measurements of number of patents, trademarks or number of new products.

It is evident that quantitative indicators are reasonably simple to measure and track. Nevertheless, they require great attention to detail to interpret effectively. It may be misleading to concentrate on readily available figures in some situations without considering what they truly represent. Neely and Hii [58] have indicated that innovation has a multi-dimensional character and cannot be easily measured directly by quantity. While qualitative indicators focus more on human performance as a measure of effectiveness, their accuracy will depend on the “goodness” of experts’ opinions and may be trade-specific. They may be biased due to over- or under-measuring [59]. Rogers [60] indicated that innovation might be a risky activity and could disrupt organizational operations, and its impact on firms’ performances is difficult to measure and predict. Even if it provides “benefits” to the organization, the return may only be accounted for after several years. Measuring the annual return owing to innovation operations may not truly reflect its effectiveness. Taques et al. [61] have also mentioned that a quantitative single-dimension analysis may cause measurement bias in organizational innovation.

Innovation performance should be viewed in the context of the corporate model of innovation and the process of creative accumulation in the organization [62], i.e., the measurement criteria should be contextual and considered at the management level. Good innovation performance evaluation is subject to the judgments and interpretations of experts. Schwartz et al. [56] further related the stressed criteria of practitioners should be simple and easily judged. Experienced practitioners are experts with practical knowledge and insights, most suitable to provide the required judgments.

From a broad view of innovation performance at the management level of an organization, there should not be too many indicators. Each indicator should include a broader spectrum that can be judged by senior managers who have adequate knowledge in the field. From the decision-maker’s point of view, they should not be overloaded by too many time-consuming indicators that may not even provide valuable information [54]. Too many indicators may also result in the sending of confusing or contradictory information. Nonetheless, in the general business management field, an investigation initiated by Boston Consultant Group 2007 [63] has reported that most firms use less than five indicators to track the performance of innovation, which is falls very short of the number necessary to conduct a thorough investigation.

With respect to the views described above, we summarized the following issues that are considered for establishing a set of indicators for AEC industry:

(a)

innovation performance evaluation is a complex process comprising of a set of indicators;

(b)

the number of indicators selected for the assessment should be limited to between five and ten;

(c)

the evaluation should be proceeded initially by considering two components, namely efficiency (examine how things can be carried out, referring to the input and process) and effectiveness (examine the capability of producing a desired output and outcome, referring to the output and outcome);

(d)

the measurement for an indicator should not be a quantitative figure only; clear qualitative indicators should also be included and be easily understandable to experts.

Innovation performance can be conducted at the project or organizational level. Product innovation, for example, should be considered at the project level [64,65]. When evaluating the overall innovation performance of an AEC firm and not for a particular project, it should be carried out at the organizational level [66,67].

3. Approach and Methodology

The approach of this study first started by adopting the Systematic Literature Review (SLR) [68] approach on the basis of a seven-step approach proposed by Xiao and Watson [69]. As described above, there are studies concerning innovation performance in general management and other fields. It is therefore possible to establish an initial pool of innovation performance indicators by reviewing studies in various fields. Innovation performance indicators were identified by a systematic review of previous studies.

It is also indicated above that there are not many closely related studies for identifying innovation performance indicators for the AEC industry. The pool established in the first stage then formed the basis for the second stage processing. In this stage, we adopted a two-step consultation processing approach. The triangulation approach is used for establishing the sets of innovation performance indicators through SLR and the two-stage consultation [70,71]. Carter et al. [70] mentioned that the four types of triangulation methods are (i) method triangulation, (ii) investigator triangulation, (iii) theory triangulation, and (iv) data source triangulation. We adopted the method triangulation approach [72] in that the three different methods of data collection were SLR and two separate conversation interviews. Figure 1 below outlines the method of triangulation approach:

The pool of indicators was established by SLR, and was then reviewed by two groups of senior professionals through conversation interviews/discussions/written communications. The first group comprised of senior professionals, who were experienced professionals holding senior management positions in the AEC industry in the US. Then, the new pool of indicators compiled by the first group of experts, including the set of indicators drawn from SLR, was reviewed/commented through conversation interviews/discussions/written communications by a second extended group of experts, who were also experienced professionals holding senior management positions in the AEC industry in the US, Hong Kong and China. At this second step, much broader views from senior professionals in several regions were gathered. A final pool of indicators was reached through the conversation interviews, discussions and written communications, consolidated through conversation and discussion until a steady set of indicators was reached.

Conversation interviews, discussions and written communications were adopted for identifying and defining the indicators as some of the indicators were qualitative and had a broader meaning. Discussions with the experts were necessary to assure they fully understood the meaning of the indicators and provided valuable comments, such as compiling the definition of the indicators. Moreover, the consolidated views and definition of the metrics for measuring each indicator were also compiled. A detailed description of the process is given in Section 4 below.

4. Information Collection and Triangulation Approach

4.1. Systematic Literature Review

At this stage, we started initial checks by using the keywords “innovation + performance” and “performance + indicators” in the search engine, Web of Science, for 2020–2021, which returned 537 and 2,183 journal articles across all disciplines, respectively. This implied that there were numerous studies concerning innovation performance or performance indicators. However, most of these articles were not relevant to our study; emphasis should be placed on studying the evaluation approach and establishing corresponding indicators. We then structured the search by using the established Systematic Literature Review (SLR) approach. There are some guidelines on the adoption of SLR [73,74,75,76]. On the basis of the method proposed by the study of Xiao and Watson [69], a modified seven-step SLR exercise is adopted and illustrated in Figure 2.

Step 1: Formulating the aim/problem of the review 

Literature review is a form of research enquiry. It is necessary to formulate the aim of the review and the problem for the investigation. This review exercise aims to identify key innovation performance indicators adopted in previous research works. The indicators are those used for performance evaluation in management [76,77]. However, there are different performance evaluations applicable in other aspects of management. We shall concentrate on performance evaluation for innovation [78].

Step 2: Developing the review protocol 

Review protocol is then developed to improve the review process’s creditability [79,80]. The protocol is a plan that allows other researchers to follow and repeat the process to yield similar results [73,74]. With reference to the studies by Gates [81] and Gomersall et al. [82], we have developed the following protocol for the study and shown it in

Table 3. Protocol for the Review.

Criteria	Action/Remarks
Purpose of the Study	To investigate organizations’ performance evaluation in respect of innovation issues
Research Issues	To establish the innovation performance indicators/metrics for evaluation
Inclusion Criteria	To include performance indicators for AEC organizations
Search Strategies	To search the information from high-quality literature, i.e., from renowned electronic indexing systems
Screening Procedures	To compile a list of keywords according to the research issues To identify a high-quality electronic indexing system—SCI/SSCI listed journals in the Web of Science system To perform searches in the electronic system To identify articles that have relevant titles To extract the abstracts of relevant articles for review To identify articles that have relevant abstracts To download full relevant articles for scrutinization
Data Extraction Approach	To extract the information, in particular the indicators and metrics presented in the relevant articles To remark the background information of the studies To categorize the information according to the nature of the articles, the field, and the time
Synthesis and Reporting	To summarize the information collected To group the information according to the categories given in the previous step To consolidate the information and provide remarks to each indicator To report the list of indicators

below:

Step 3: Determining the pool for searching: literature of high quality 

The quality of the literature undoubtedly affects the quality of the review. Developing accurate and objective strategies to identify well-recognized journals that publish high-quality research can help researchers select the most valuable literature to review. To assure the quality of the literature, we searched the literature published in renowned journals, particularly those journals listed in the SCI/SSCI in Web of Science. Nazim [83] mentioned that the indexing system for SCI/SSCI-listed journals had a generally stipulated policy that no journal could be indexed until a period had elapsed, allowing the publisher to determine if the publication had merit and was recommended by well-known scholars in the respective discipline. The publications in SCI/SSCI journals are thus, recognized as good quality and reliable articles. Accordingly, an electronic search in the Web of Science for publications is used in this study. Moreover, Scopus Indexing and other similar sources are also useful databases for searching for journal articles, conference proceedings, theses, and reports. These sources provide extensive information to support the research.

Step 4: Brief screening and reviewing for inclusion and exclusion 

The keywords included in the study served as the primary key for the electronic search. Having compiled a list of articles for reference, the abstracts of listed articles are reviewed for relevancy with respect to the aim of the study. The objective is to investigate innovation performance measures. The shortlisted articles are then scrutinized (see later in this paper) to extract the information concerning innovation performance, assessment/evaluation framework, corresponding indicators, and metrics. There is no consensus on which articles should be considered relevant [84]. In order to avoid selection bias and to maintain the generalizability of review findings, articles describing innovation performance and mentioning evaluation indicators are included regardless of the basis of the works, i.e., research works carried out in different countries/cities, for different organizations/parties and different industries are included. Figure 3 illustrates the flow of the process in this step.

Step 5: Restrictive reviewing for category establishment 

This step identifies indicators and establishes the corresponding categories, in order to select the indicator categories and compile an initial list of indicators. The procedures are as follows:

(1)

To adopt a restrictive searching approach, by using the keywords “innovation + performance + indicators”/“innovation + performance + measure” for the list of articles compiled in Step 4;

(2)

To obtain the articles for a full-text review;

(3)

To review the full text of the new set of articles and extract the “indicators” presented/mentioned in the selected articles.

On the basis of the steps mentioned earlier, a list of key relevant articles, the corresponding aim and objectives of the articles, the indicators presented, and the designated categories are summarized in

Table 4. Summary of articles extracted by first stage literature review.

Article	Aim/Objective of the Study	Indicators/Metrics (Presented or Mentioned in the Article)	Implied Ideas/Meaning of the Indicators	Indicator Categories Designated
Chen F, Zhao T, Liao Z (2020) [85]	To study the influence of technological–environmental innovation indicator systems on CO2 emissions of China’s transportation sector.	Innovation performance Innovation resources Knowledge innovation Innovation environment	Innovation activities Financial and human resources Knowledge creation Organization support (training and knowledge sharing, etc.)	innovation process finance resources people & organization; creativity of employees training knowledge sharing
Dziallas, M., & Blind, K. (2019) [86]	Literature review on innovation indicators based on the different stages of innovation. The review identifies 82 unique indicators to evaluate innovations under different dimensions.	Note: Only company-specific categories will be listed below: Strategy Innovation culture Competence and knowledge Organizational structure R&D activities and input Financial performance Innovative products Innovation process Innovation project management	Strategy Innovation culture Competence and knowledge Organizational structure R&D activities and input Financial performance Innovation outcome (innovation product, etc.)	people & organization knowledge sharing people & organization creativity of employees financial resources innovation process resultant of new ideas implementation of new ideas
Piro FN (2019) [87]	To study the R&D composition of European countries and whether the profiles are associated with Research and Innovation performance indicators.	R&D systems Intellectual assets Culture Creativity	Ideas creation and implementation Organization strategies People and organization structure/culture Creativity of people	implementation of ideas quality of ideas people & organization creativity of employees resultant of new ideas
Brogi S, Menichini T (2019) [88]	To study to what extent the ISO certification of Environmental Management Systems influences eco-innovation performance.	Financial or human resources R&D personnel and researchers Early stage investments Level of advancement and implementation of eco-innovation Resource efficiency Outcomes	Financial and human resources Organization support R&D investment Innovation process and implementation Output/product efficiency	finance resources people & organization finance resources quality of ideas innovation process implementation of new ideas resultant of new ideas customers’ satisfaction
Popadic et al. (2018) [89]	This article presents the results of innovation activities analyses on a given sample of Small-Medium Enterprises.	Income growth rate Level of customer’s satisfaction Deadlines of responses to customer requirements Employee productivity No of new products/services Available technological knowledge of employees Growth rate of new customers The number of key customer orders increase Customer Loyalty (Number of Repeated Orders)	Output/product returns Output/product efficiency Processing efficiency New/innovative idea Knowledge Customers’ satisfaction Competence of employees	people & organization financial resources resultant of new ideas customer’s satisfaction implementation of new ideas innovation process quality of new ideas creativity of employees
Garcia-Granero EM, Piedra-Munoz L Galdeano-Gomez E (2018) [90]	To perform a critical review of literature on eco-innovation performance indicators.	Product Process Organization Marketing	Customer’s expectation and satisfaction Process (efficiency) Organization (support/strategy) Marketing (returns/financial situation)	customers’ satisfaction resultant of new ideas quality of new ideas innovation process financial resources people & organization
Mehta S (2018) [91]	To study empirically the national innovation system of India.	Input: Expenditure on R&D Human capital: knowledge & education Output: Productivity Patents Proportion of high-tech exports	Financial support Knowledge/Training Process efficiency Quality of new ideas (patents)	financial resources knowledge sharing training quality of new ideas innovation process implementation of new ideas
Nwachukwu C, Chladkova H, Fadeyi O (2018) [92]	To study the link between strategy formulation process and innovation performance indicators in microfinance banks.	Product Process Market Strategic formulation	Product: customer’s expectation and satisfaction Process: implementation and efficiency Returns/financial situation Strategy: organizational strategy	customers’ satisfaction financial resources implementation of new ideas quality of new ideas innovation process resultant of new ideas
Mazur K, Inkow M (2017) [93]	To examine what innovation performance indicators are appropriate for research on innovation process in the IT sector.	Workforce creativity Knowledge sharing R&D input and outputs New products or patents	Employee creativity Knowledge/training Financial support New ideas	creativity of employees training knowledge sharing quality of new ideas resultant of new ideas people & organization
Ocnarescu I, Bouchard, C (2017) [94]	To study the mechanism of aesthetic experiences of work in a research and innovation context—an R&D laboratory of a multinational communications and information technology company.	R&D researchers’ aesthetic experiences People Organization structure	Employee creativity Researchers’ Knowledge	employees’ creativity quality of ideas people and organization implementation of new ideas
Moagar-Poladian S, Folea V, Paunica M (2017) [95]	The article presents a study on the competitiveness of European Union Member States in terms of research and innovation from the perspective of attracting EU funding for research, as well as from the viewpoint of key science and innovation performance indicators.	Technological innovation Employment in knowledge-intensive activities Competitiveness of knowledge-intensive goods and services Employment R&D intensity	Knowledge/Training Customer’s satisfaction/competitiveness Financial and human resources	knowledge sharing; training customers’ satisfaction; quality of idea resultant of new ideas finance resources; people and organization
Plewa M (2017) [96]	To study the long-run dynamics between product life cycles and key innovation performance indicators.	Time-To-Market Development Speed Impact Profitability Industry Consequences Capabilities Quality Success	Process (efficiency) Customer’s satisfaction (impact) Profit/Returns Quality of products Success	innovation process implementation of new ideas customers’ satisfaction resultant of new ideas quality of new ideas finance resources
Noktehdan M, Shahbazpour M, Wilkinson S (2015) [97]	To study the relationship between innovation and productivity improvement in the construction industry.	Innovation Type Innovation Novelty Innovation Benefit	New ideas Creativity Number of new ideas Returns Customers satisfaction	employee’s creativity quality of new ideas implementation of new ideas resultant of new ideas customers’ satisfaction
Cheng CCJ, Huizingh EKRE (2014) [98]	To study the relation between open innovation and innovation performance.	new product/service innovativeness new product/service success customer performance financial performance	New idea Quality of new ideas Number of new ideas Financial support Customer’s satisfaction	quality of new ideas implementation of new ideas finance resources; resultant of new ideas customers’ satisfaction

below.

Step 6: Grouping of identified indicators 

The information extracted from the identified articles is recorded/coded. The performance indicators, whether the studies are for general organizational performance or the innovation performance of an organization, are recorded and remarked. The procedures for this grouping of indicators and naming of categories are as follows:

(1)

To assign name/title to each indicator category;

(2)

To classify/group the indicator to the relevant indicator category and provide a detailed description/explanation for each indicator.

Data extraction in the above process is summarized. Indicators that included similar meanings are then grouped and organized to establish a set of 4–10 indicators [99].

Step 7: Analyzing and synthesizing for initial set of indicators 

Manoochehri [100] has recommended organizations use 8–12 indicators for the innovation dashboard. Birchall et al. [54] put forward the criteria for selecting innovation indicators (metrics), suggesting that they should:

(1)

cover critical issues;

(2)

be simple and clear to all parties;

(3)

not rely on complex and challenging data collection;

(4)

be understood and reliable;

(5)

be easy for interpretation and evaluation;

(6)

be presentable in scale for use in executive decision-making.

This list is then further consolidated into 10 indicator categories, and descriptions/explanations are provided for each indicator with reference to the criteria mentioned above.

Table 5. Consolidated list of indicators—initial set.

Consolidated Metric Category (Indicators)	Descriptions/Explanations
Financial Resources	Describes the financial resources an organization is willing to invest in innovation-related activities, including, but not limited to, establishing grants for supporting pilot projects, holding idea competitions with financial incentives and financial support on investing in new technologies, etc.
Innovation Process	Describes the systematic process in the firm that an innovative idea will go through, including how the process helps incentivize idea generation, implementation, the flow of the new ideas from generation to implementation (or even commercialization), etc.
Creativity of Employees	Describes the creativity of employees when solving problems in their work process and how willing they are to share their new ideas with the organization.
People and Organizations	Includes organization culture, management structure, technological environment, infrastructure, and leadership. Describes employee engagement in the innovation process and how well the current organization is to support innovation.
Training	Includes training sessions provided by the organization for the employee to learn new technologies or approaches. It also includes how the organization supports the self-development of the employee, such as subsidiaries for employees to take related online courses.
Knowledge Sharing	Includes the means and methods by which the organization supports knowledge sharing among its employees; how advanced implementation can be transferred from one team to another.
Implementation of New Ideas	Includes the efficiency of the implementation of new ideas, the diffusion rate of new ideas, etc. How the new idea/concept/method is being implemented in the organization; the strategies used.
Quality of New Ideas	Includes the measurement of the sustainability, scalability, and effectiveness of the idea, reflecting how well the idea is for bringing value to the organization.
Resultant of New Ideas	Includes enhancement of project satisfaction, cost, time, quality, organization’s image, competitiveness, etc. In other words, the performance chang led by the innovation, e.g., return of investment.
Customers’ Satisfaction	Clients’ satisfaction; end-users’ satisfaction; and revenue.

shows the consolidated list of indicators and the corresponding descriptions/explanations.

This consolidated set of innovation performance indicators is regarded as the “initial set of indicators” for further processing using the two-step consultation process described below.

4.2. Second Stage Two-Step Consultation Process

Patton [101] mentions that multiple data collection methods and analyses provide more support for the research process. The triangulation approach refers to the use of multiple methods and data collection to compile a comprehensive study to achieve agreement or convergence of information from different sets of data [102]. As described above, we adopted the method triangulation approach [103,104,105] for processing the data. By integrating the SLR and two-step consultation processes [106], a set of indicators was compiled for the innovation performance evaluation in the AEC industry.

Two groups of senior professionals were invited for a consultation in a two-step approach. The first step was proceeded by inviting 12 AEC professionals, who had at least 10 years of experience at senior management positions in the AEC industry in the United States, to have in-depth discussions/online chatting/written communications to ensure that all the interviewees fully understood and referred to the similar meanings of the indicators. After extensive communication, the opinions of the professionals were confirmed and compiled to a selective set of indicators. Figure 4 shows the flow of the process.

Table 6. Description of First-Step Conversation Interviews.

Major Items	Descriptions/Information	Decision/Actions
Establish initial set of indicators by SLR	----	----
Inviting senior professionals in the US to join the conversation/discussions/chatting/written communications for vetting an initial set of indicators	Discuss with experts’ and solicit their comments on the definition/usefulness/measurement, etc., of each indicator by using open-ended questions	Record comments and discuss with other experts
Assess the usefulness of the indicator: “training and knowledge sharing”	Collect experts’ comments on the “training and knowledge sharing” indicator None of the experts specifically considered training as an independent indicator Knowledge sharing and training are regarded as similar and can be grouped together It is suggested that the “idea pool” can be set as an indicator to replace knowledge sharing	Delete “training and knowledge sharing”; group them together and form a new indicator “ideas and knowledge” which includes idea pool (how many more ideas are in the pipeline, waiting for investment, piloting, or implementation, etc.)
Add durability to explain the resiliency of the innovation	Experts commented that they would like to know how the “value” of the innovation is sustained over time, i.e., the resiliency of the innovation It is suggested to use “durability” to describe the resiliency	Add a new indicator: durability
Add accessibility to reflect the ease of implementation of the innovation	Experts commented that the technological maturity of the environment supports the idea The ease to apply and repeat The scale the idea	Add a new indicator: accessibility
Add strategic alignment to describe how the ideas from the innovation team align with operational and business strategies of the company	Experts expect that the organizational strategy concerning innovation is important; It is necessary to know how business plans, objectives, operational goals support the innovative activities	Add a new indicator: strategic alignment
Establish second set of indicators	Comments and suggestions collected Further review of literatures to supplement the comments from experts Confirmation of the set of indicators	Finalized the second set of indito cater for for the next stage process

describes the major remarks and decisions in the process.

The second-step was proceeded by inviting 20 AEC professionals, who also had extensive experience (at least 10 years’ experience) holding senior management positions in the AEC industry in the United States, Hong Kong and Mainland China, for conversation interviews/discussions/online chatting/written communications. Similar to the first step, we proceeded with an in-depth consultation. Discussions with the senior professionals from different regions were carried out in order to gather an extended view of experts with wider international and regional experience. The set of indicators compiled in the first step was shown to the interviewees in this step. After an in-depth consultation, the opinions of the professionals were confirmed and a final set of indicators was established. Figure 5 shows the flow of the process,

Table 7. Description of Second Step Conversation Interviews.

Task	Information	Decision
Secondset of indicators established in the first step consultation process	---	----
Invite senior professionals in US, Hong Kong and Mainland China to join the conversation/discussion for vetting the second set of indicators	Discuss with experts and solicit their comments on the definition/usefulness/measurement, etc., of each indicator by using open-ended questions	Record comments and discuss with other experts
Assess the usefulness of the indicator: “resultant of new ideas”	Discuss with experts and collect their views/comments on the “resultant of new ideas” indicator Most experts commented that using the term “return on investment” (ROI) should be better since ROI is more specific and commonly used in the field already Experts also indicated that customer satisfaction should be reflected by the “resultant of new ideas” indicator	Customers’ satisfaction to be included in “resultant of new idea”; change the wording of “resultant of new idea” to “return of investment (ROI)”
Comment on the indicator: “implementation of new ideas“	Experts commented that this term should be considered and incorporated into the “innovation process” indicator The innovation process should include the implementation idea	“implementation of new idea” should be included in the indicator: “innovation process”
Add Impact as a new indicator to represent the soft side of “resultant”	The majority of experts have the view that the soft side of resultant, such as the image of the company should be included A term “impact ”was suggested to the indicator list to indicate the softer impact of innovation such as branding and the organization’s image They also commented that it is necessary to distinguish impacts; it refers to the issues that can hardly be quantified, such as typical design, cost, and schedule ROI	Add a new indicator: Impact
Merge financial resources into strategic alignment	Experts commented that it is better to merge financial resources into strategic alignment in order to describe a broader concept of the resources allocated to innovation through the organization’s strategy or policies, which includes the financial/human resources assigned to innovation activities	Delete the “financial resources” indicator and add explanation to “strategic alignment” to include the concept of financial and human resources
Amend “people and organization” into engagement	Experts, especially those from the US, mentioned that engagement level of employees should be one of the indicators for measuring innovation performance This indicator also includes the culture of organization	Change “People & Organization” to “Engagement”; add further explanation to the term “engagement”
Delete the indicator “accessibility”	Accessibility is deleted and incorporated into the “innovation process” indicator where it includes the process of vetting innovation ideas, which should be a measure for the “accessibility” of the ideas	Delete “accessibility” and add further explanation to “innovation process”
Retain the indicator “durability”	Experts commented this term is not very clear; however, they all agreed that it describes a very specific portion of innovation performance where one may want to assess the performance in a longer time horizon; When we evaluate innovation, it is difficult to tell the value of the innovation in a short time frame; we should also consider its performance in the long run	Keep the term “durability” and add further explanation to clarify this indicator
Establish Final Set of Indicators	Consultation with experts to reach a final set of indicators with corresponding explanations/definition to each indicators	Finalize the Final Set of Indicators (for subsequent works)

describes the major remarks/decisions in the process and

Table 8. Finalized Set of Innovation Performance Indicators.

Categories	Indicators	Explanations
Leading (Input/Process)	Creativity of Employees	Creativity of Employees describes the creativity of employees when solving problems faced in their work process and how willing they are to share their new ideas with the organization.
	Innovation Process	Innovation Process describes the system within the organization that supports innovation, including how the system helps incentivize idea generation, implementation, and the flow of the new ideas from generation to implementation (or even commercialization), etc.
	Durability	Durability describes how “durable” the value of the innovation will be in the market and how resilient this innovation will be to change.
	Ideas and Knowledge	Ideas and Knowledge includes the organizations’ idea pool and how knowledge transfers between communities and groups.
	Engagement	Engagement describes the engagement level of employees in carrying out innovation. This includes how engaged employees are in generating new ideas, and implementing these new ideas in projects, etc. This engagement is also hindered by other factors, which are included in this indicator, such as the organization’s culture.
	Strategic Alignment	Strategic Alignment includes aligning organizational policies/strategies that promote and manage innovation, management style, organization structure, etc.
Lagging (Output/Outcome)	Return on Investment (ROI)	ROI describes the measurement of the return on the investment of the innovations. In general, measuring the enhancement in performance results from the idea, such as project satisfaction, cost, time, quality, etc. In other words, the performance change led by the innovations. Customers’ satisfaction is included.
	Impact	Impact measures the “soft” impact of innovations such as their effect on the organization‘s image, competitiveness in the industry, etc.
	Quality of New Ideas	Quality of Ideas includes the measurement of the applicability, scalability, etc., of the idea, representing the value of the ideas.

shows the Final Set of Indicators. In addition, the metrics for the indicators’ measurements were also suggested (Appendix B) and confirmed in the consultation processes.

5. Discussions

This study searched through the current research body on innovation performance measurements. It integrated knowledge with industry-specific inputs for compiling a customized set of innovation performance measures for the AEC industry. This set of metrics provides the means for industry practitioners to evaluate their innovation performances for the organization and to identify the factors that industry experts deem necessary for innovation. These factors may change through time and require an adaptive learning process to identify such factors; hence, this method should be repeated regularly for the best results. This research illustrates an approach for collecting and integrating industry experts’ opinions and transforming them into a valuable set of metrics. This approach is repeatable for other industries and essential for developing a tailored set of innovation measures to reflect unique industry characteristics.

Limitations and Future Works

During the search for innovation indicators within the current research body, this research only considered SCI and SSCI journals and was limited to the last 10 years of published papers to ensure the quality of published references. Elaborated discussion on the choice of this scope of reviewed works can be found in step three of Section 4. Nonetheless, aside from SCI and SSCI journals, other reviewed or non-reviewed works such as proceedings, industry publications, etc., might also provide quality insights for this research. Therefore, future research can be carried out to expand the pool of search for innovation indicators.

As the concept of innovation and what pertains to each industry varies over time, results from this study should be updated by incorporating more recent inputs from industry experts. Though the core idea of innovation might not vary over time, innovation studies are bound to be time-dependent, and results should be used with care. Aside from industry characteristics, an organization’s vision, mission, or culture may affect the choice of indicators. Thus, this set of innovation performance indicators are provided as a reference for the AEC industry, which can be readily adopted or tailored with company-specific inputs. This research can be further expanded to incorporate different characteristics of AEC stakeholders, especially the difference between architecture and engineering firms (AE firms), construction firms, and owners.

In addition to the difference introduced by industry and company characteristics, attributes such as the organization’s size, regional differences, including both cultural and government policy differences, are possible factors that might affect the applicability and effectiveness of the resulting set of innovation performance indicators. Future studies may focus on each aspect and collect data inputs accordingly to better understand the probable effects of these factors.

6. Conclusions

Although AEC industry is regarded as lagging behind in innovation, firms have recently paid more attention to innovation. To effectively manage innovation, investigations on performance evaluation for various fields such as manufacturing, information technology, etc., were widely initiated recently. However, there are not many works on innovation performance evaluation for the AEC industry and this study attempts to fill this research gap. In view of the specific characteristics of the AEC industry, the performance indicators in other fields may not directly apply to AEC firms. On the basis of the works in other fields, a set of innovation performance indicators was compiled by the method triangulation approach, which integrates SLR and two-step consultation process. Six leading (input/process) indicators and three lagging (output/outcome) indicators are identified. Metrics for the indicators’ measurements are also compiled in the consultation processes and shown in Appendix B. These metrics as compiled in this study provide a set of readily applicable customized innovation performance indicators for organizations within the AEC industry. This set of metrics supports the measurement of innovation performance in AEC organizations and provides a common basis for comparisons and benchmarking, adding a set of AEC-specific innovation performance indicators to the current literature body. Furthermore, this helps organization managers to monitor the innovation activities within their organization, and identify their strengths and weaknesses for improvement. This set of metrics can also provide valuable insights for policy makers where they assess the performance of firms and identify areas with regard to supporting innovation activities in the industry. For example, if the performance is generally lower than other regions, one may want to consider incentivizing the industry through supportive government policies, such as providing funding or recognizing the usage of innovative approaches in public projects.

Previous studies have argued that the number of indicators for performance evaluation should be limited. A total of nine indicators can serve for practical innovation performance evaluation by using a simple auditing approach. Equation (1) may be re-written as:

$$InnovationPerformance={{\displaystyle \sum }}_i^6{w}_i{I}_i+{{\displaystyle \sum }}_j^3{w}_j{O}_j$$

(2)

where I = input indicators and O = output indicators, and w is the corresponding weighting for input and output indicators.

This paper presents a study to compile the set of innovation performance indicators. Future works should be carried out to establish the evaluation framework. The SMART (Simple Multi-Attribute Rating Technique) method [107], with example applications [108,109,110,111], can be used to evaluate innovation performance, and Equation (2) shows a simple form of this evaluation framework. However, more works should be initiated, particularly via the elicitation of weightings for each of the indicators. These weightings may be affected by factors such as the region, size, and nature of the organizations. Nevertheless, the set of indicators and the corresponding measurement metrics can serve as the basis for researchers and practitioners to formulate the evaluation and support the decision-making of the management.