Performance Measurement in R&D Projects: Relevance of Indicators Based on US and German Experts

Abstract

To turn technologies into successful products, it is necessary to understand the development process from ideas to the market and to know how to measure performance. Performance measurement is critical for technology developers and investors in monitoring whether performance meets expectations to make decisions about actions for improving R&D characteristics. This article emphasizes indicators for R&D project performance measurement, especially relevant for measuring project performance in company, start-up and spin-off companies, where the project is perceived as an independent business unit. A clear set of indicators for measuring and controlling the performance of R&D projects for policy representatives would allow them to identify problematic areas in the implementation of R&D projects and to make well-aimed decisions for the promotion and financing of technology development. What indicators should be used to measure the performance of R&D projects? Attempts to find the answer to the question in science were unsuccessful. This article aims to select indicators for measuring the performance of R&D projects and identify and compare their relevance among US and German experts. Research is carried out in different countries, and their results create opportunities for mutual learning and more intensive international cooperation in technological development. In order to achieve a goal, essential decision-making points in R&D projects were identified, and a general set of R&D performance evaluation indicators were prepared based on a literature analysis. Later, two groups of experts from the US and Germany selected from the general list indicators suitable only for evaluating R&D projects and evaluated their relevance. The obtained evaluation results of the US and German experts were processed using the MCDM method and compared.

1. Introduction

In recent years, changing business environments have challenged companies to improve their R&D processes in terms of effectiveness and efficiency, and the attention of technology developers is now focused on R&D’s contribution to competitive advantage. R&D was considered a unique, creative and unstructured process that was difficult, if not impossible, to control. The control techniques used in other fields of business were considered inappropriate for the R&D function. R&D processes have several characteristics that differentiate them from other processes. Performance measurement in R&D can be described as chaos or problems involving dynamic and behavioural complexity. Dynamic complexity is characterized by a long period from input to output, making it challenging to visualize cause-and-effect relationships. In contrast, behavioural complexity requires high interpersonal and support skills [1]. However, it is difficult to accept that R&D processes are unmanageable; therefore, there is a growing acceptance of the need to control these processes. However, more and more often, technologies are developed into a product in small start-up or spin-off companies developing only one or a few technology-based projects. Technology developers and investors are interested in taking over experience from R&D experts and have a clear set of indicators for measuring R&D projects. This article emphasizes the R&D project performance measurement, where the project is perceived as an independent, individual business unit, and it aims to answer the question: What indicators are suitable to measure the performance of R&D projects? After an unsuccessful attempt to find studies dedicated to R&D project performance measurement, a decision was made to prepare a general list of indicators for measurement of R&D performance and later, with the help of experts, to select indicators from this list suitable for measuring R&D projects. The most concrete and at the same time the most useful results were presented by Kerssens van Drongelen, Bilderbeek (1999) [2], Bremser, Barsky (2004) [3], Cedergren et al. (2010) [1], Cedergren and Larsson [4], Markham and Lee (2013) [5] and Barbosa et al. (2020) [6].

For investigation, experts from the US and Germany were chosen as representatives of the R&D sector of these countries. These two countries can always be found in the first ten positions of the Global Innovation Index rankings; they are successful examples of technology development policies representing different continents with different R&D business traditions and systems. This investigation was inspired by the work of the National Academy of Engineering (1997) [7], which explored and compared the national technology transfer systems of Germany and the United States. Motives that led to the work mentioned above coincide with the motives of this study: conducting research in different states and comparing them and identifying areas and opportunities for further mutual learning between the two national systems; it also offers a starting point from which each nation can pursue new paths toward strengthening economic and technological performance, as well as cultivating more intensive, mutually advantageous international collaboration in technology transfer. The mentioned motive led to this research, considering the scientific literature and the opinion of practitioners/experts.

This article aims to select indicators used to measure R&D projects and determine and compare their relevance among US and German experts. In order to achieve this goal, we defined the process during which performance measurement could be studied, and a general set of R&D performance evaluation indicators were prepared with the help of literature analysis. Later, two groups of experts from the US and Germany were asked to select from the general list of indicators suitable only for evaluating R&D projects and later to evaluate their relevance. The second stage of expert research was conducted to establish the relative weight of the indicators in the set, in other words, to determine the relevance of the indicators to compare them. The obtained results of the expert evaluation were processed using the MCDM method.

2. Performance Measurement in the R&D Process

2.1. Content and Measurement Points in the R&D Process

In general, R&D signifies systematic activities to create knowledge and use this knowledge for developing new products, processes or services. Innovation processes are strongly involved in the concept of R&D. In the broadest meaning, R&D consists of every activity, from basic research to the successful marketing of a product or (effective) launching of a new process (R&D&I) [8]. In the scientific literature that deals with R&D management issues, there are many different approaches to the development process, from invention to product introduction to the market and concepts of this process. Different names are often used for the same procedure. It is necessary to examine the models used in the scientific literature and practice reducing chaos and defining the most critical decision-making stages in which it would be appropriate to use certain indicators.

The technology commercialization (TC) development schemes were examined in science and some technology transfer centres to connect the TC process to the concept and identify key decision points. The content of TC development schemes can be decided considering the schemes used in practice [9,10,11,12,13]. Some schemes were analysed in scientific articles [14,15,16,17,18]. The models of TT were analysed in scientific articles [19,20,21] and used by different organizations responsible for TT transfer [22,23,24,25,26] to connect an understanding of the technology transfer (TT) process to the concept. The TT procedures match all TC processes except the Goldsmith TC model [13].

The TC and TT procedure usually starts with research and follows the disclosure of the invention and its assessment and legal protection. It ends with the transfer of ownership to a company willing and able to develop the technology and bring it to market. The overall successful procedure of TC and TT is often divided into estimation and realization phases. The estimation results cause an impact on the implementation decision. The traditional TC and TT model is reflected in the stages of TC at Stanford University (2022) [9]. Many models of TC and TT have a commercialization stage as the last stage of the procedure, which includes three possible choices finishing with the technology transfer to industry, which will lead to the implementation of a product or service in business.

Some universities [13] use the Goldsmith TC scheme. If we compare it to other models, this model is the most extensive, covering the entire R&D process, or the TC, TT and PD processes together. This model includes stages from discovery to the enterprise’s optimization profit potential, which means that the process involves the stages from the initial stage to the moment return from end users can be assessed, called the post-lunch stage. Each step includes technical, market and business activities developed in parallel. This process results in a company that generates profits through multiple revenue streams.

Based on the analysis of TC and TT models, it became clear that the commercialization process in the literature and practice can be understood in three ways: (1) in the narrow sense—it starts with research or the disclosure of the intervention and ends with the transfer of intellectual property rights to a company; (2) in a broad sense—covering the entire R&D process, or the TC, TT and PD processes together (Goldsmith TC model) [13]; (3) as the last stage of the TT/TC process.

When it comes to the earliest stages of R&D, an assessment phase aims to answer the question: Does the invention/idea merit expending any effort? [27]. In order to implement this, we need to consider financial possibilities—identifying resource needs and gaps, preliminary technical characteristics—in the beginning, laboratory feasibility of the idea under ideal conditions, later related to the maturity of the technology, also matters of patentability, market issues—competitors’ alternatives and consumer needs.

The new product development (PD) models were analysed, presented in scientific articles, and used by practitioners to identify critical points of assessment and decision making. The most popular Stage-Gate system created by R. G. Cooper (2009) [27] is a PD tool that is used by many successful enterprises worldwide and is cited in virtually every scientific article that deals with the topic of PD [3,28,29]. Works by Markham, Lee (2013) [5], Cedergren et al. (2010) [1], Bremser and Barsky (2004) [3] and Bimal et al. (2011) [29] were also examined and investigated.

New product creators use the Stage-Gate schemes or its modifications. This scheme is a road map for moving a new product project from an idea to a product launch. Stage-Gate divides the effort into different stages separated by management control decision gates. The cross-functional crew must complete the outlined set of related cross-functional functions in each stage before receiving management/the gatekeeper’s approval to proceed to the next step of PD. This scheme consists of: 0 Stage—discovery: the activities designed to discover opportunities and generate new product ideas; 1 Stage—scoping; 2 Stage—development of the business case; 3 Stage—development, when plans are translated into concrete results—focus on new product design and preliminary tests regarding potential customers; 4 Stage—testing and validation; 5 Stage—in launch starts of production, marketing and sales; Post-launch reviews—quality-monitoring operations and any post-launch reviews follow after this stage. Gates are points where a decision is made to pass the project into the next stage, end the project or send it back to the previous stage: Gate 1—Idea Screen; Gate 2—Second Screen; Gate 3—Go to Development; Gate 4—Go to Testing; Gate 5—Go to Launch. The gatekeepers are the senior people in the business who own the resources required by the project leader and team to move forward. For major new product projects, the gatekeepers should be a cross-functional senior group—the heads of Technical, Marketing, Sales, Operations and Finance. The gatekeeper group must involve executives from the resource-providing areas to achieve alignment and the necessary resources. In addition, a multi-faceted view of the project leads to better decisions than a single-functional view [27]. In general, Gates is rated one of the weakest areas in PD. The 3rd Gate has earned the most attention and is considered one of the most responsible because, after 3 Gates, a decision is made regarding implementing plans and significant investments.

Cedergren et al. (2010) [1] break down the PD process into planning, implementation, and sales and delivery. The authors emphasize that when the performance of the PD function is under investigation, the focus quickly turns to the implementation activities at the expense of the planning activities. Scientists and developers must understand the importance of planning performance evaluation. In the early stages of development, decisions are made about what customer needs are to be met and what the value of the PD investment can be. Activities that produce consequences are related to the implementation of what was decided in the planning process.

It is vital to acknowledge the importance of the evaluation of planning activities. Considering Cooper’s observations regarding the critical stages in the PD process, Gates is rated one of the weakest areas in PD. The 3rd Gate earned the most attention and is considered one of the most responsible because, after 3 Gates, a decision is made regarding implementing plans and significant investments. Additionally, from a value creation point of view, during the early activities of the development, it is decided what customer needs will be satisfied and what the possible value of the PD investments can be. The focus tends to be on the realization phase of the PD process too. The research states that the PD function cannot be considered successful from a company perspective until all three activity categories have been completed and a product has been delivered to the customer. It is also emphasized that the outcome in the post-launch assessment is carried out regularly by only one of the five case companies.

2.2. Compilation of a Set of Indicators for Measuring the Performance of R&D Projects

To determine the relevance of indicators for measurement performance in R&D projects, it is necessary create a set of indicators. The set of indicators is the foundation for determining their relevance based on evaluating multi-criteria decision making (MCDM). Due to the lack of scientific research dedicated to evaluating R&D projects, it was decided first to prepare a general set of R&D performance indicators based on scientific literature analysis and later to select indicators for R&D projects and determine their relevance with the help of expert research.

The current interest in R&D control, specifically in R&D performance measurement, is reflected in articles with titles and abstracts featuring words such as effectiveness, performance, success, control, monitoring, assessment, measurement, benchmarking, auditing and evaluation. These words are often used as synonyms [2]. To clarify how the article interprets R&D performance measurement, it will first describe performance measurement as part of performance control. Firms decide to measure the performance of their R&D activities for multiple purposes: motivate researchers and engineers; monitor the progress of R&D activities; evaluate the profitability of R&D activities; favour coordination and communication among the different people taking part in R&D activities; reduce the level of uncertainty; stimulate and support individual and organizational learning [30].

At the early stage of development, the performance of R&D activities is measured as the TT Office outcomes. Regarding Tseng and Raudensky (2014) [19], many investigators studied university TTOs’ performance, and many metrics have been selected to assess their performance. The following metrics can quantify the performance measure for a TTO: TTO revenue; the number of invention disclosures; the number of patent applications; the number of patents granted; the number of licenses signed; the number of start-ups formed; research expenditure of university scientists; expenditure of patenting activities; operation expenditure; the number of new commercial products; employment and productivity growth of start-up partners; changes in stock prices of industrial partners, etc. The first six metrics are more frequently applied for measuring the performance or accomplishment of a TTO. The Milken Institute released a report ranking more than 200 universities across the US for their prowess in developing basic research into new technologies, products and companies—a process known as TT. The University Technology Transfer and Commercialization Index is based on data collected by the Association of University Technology Managers (AUTM) via the AUTM’s Annual Licensing Activity Survey, except for the University of California System. The Index is measured using four-year averages (2012–2015) for four key indicators of TT success: patents issued, licenses issued, licensing income, and start-ups formed [31].

If we come back to the primary purpose of this article (i.e., project evaluation indicators in the R&D process from invention to product in the market), many TTO activity indicators such as the number of invention disclosures, the number of patent applications, the number of patents granted, the number of licenses signed and the number of start-ups formed basically states the beginning of the project. Such indicators as the research expenditure of university scientists and patenting activities identify efforts/contributions to achieving project start. In summary, these reasons justify the inappropriateness of the above indicators for measuring R&D projects.

When it comes to PD, the inventor can create a business unit to produce and market an invention, or sign contracts to transfer the invention to business; in this case situation regarding the used performance measurement indicators changes. Concerning Chiesa et al. (2009) [30], the companies measure R&D performance by taking into account: the economic and financial aspects associated with R&D (financial perspective); the extent to which R&D identifies and satisfies the needs of its internal and external customers (customer perspective); the efficiency with which specific tasks and processes are carried out (business process perspective); and the extent to which R&D contributes to generating new knowledge and innovation opportunities (innovation and learning perspective). Kerssens van Drongelen and Bilderbeek (1999) [2] presented the results of an empirical study focusing on the effectiveness of R&D performance measurement practices in the Netherlands. The authors intentionally did not include the question regarding performance measures in a predetermined list of metrics. They were interested to find out how the respondents themselves defined the measures, and they did not want to tempt them to list metrics they did not really use. Later, they clustered the measures mentioned in groups: the customer perspective; internal business perspective; innovation and learning perspective; and financial perspective. In many companies, the feasibility of projects is measured periodically during the process at milestones or gates, for example, in terms of market, strategic, economic and technical feasibility. Bremser and Barsky (2004) [3] respond to calls in the R&D literature to explore integrated performance measurement systems that capture financial and non-financial performance. They submit the survey results of the most frequently used R&D metrics: R&D spending as a percentage of sales; new products approved/released; the number of approved projects ongoing; total active projects supported; total patents filed/pending/awarded; current percentage of sales of new products; percentage of budget resources dedicated to R&D; change in R&D headcount; percentage of resources dedicated to sustaining existing products; average development cost per product. Cooper & Edgett (2008) [32] use the concept of productivity, which is output over input, or “the most bang for the buck”. More specifically, in the field of new PD, productivity is defined as output measured as new product sales or profits divided by input measured as R&D or new PD costs and time. Cedergren et al. (2010) [1] claim that the following five measurements were identified as being those most commonly used: R&D spending as a percentage of sales; total patents filed/pending/awarded/rejected; total R&D headcount; current-year percentage of sales due to new products released in the preceding X-number of years; the number of new products released. According to the author, these measurements are, without a doubt, vital. However, they do not support evaluating the current PD performance since all measurements are either resource (cost) oriented or output (outcome) oriented. There is excellent potential for adding measurements for the planning activities. Performance measurement systems focusing on the later stages of the implementation activities do not support the PD function, only reporting the end result [1]. Cedergren et al. (2010) [1] criticize the commonly used indicators but later, in another publication, present only a set of factors for performance measurement without indicators. In a survey, Markham and Lee (2013) [5] explored how new PD activities and performance are measured. They compared the time spent on each development activity in each new PD stage. The researchers assessed which indicators are most important for the business units to measure results. The top two indicators are profit from new product sales and new product sales as a per cent of total sales. Net margin ROI, market share trends, project cost versus budget, and total cost of new product effort as a per cent of revenue are also marked as important issues to measure results. The authors cite Cooper (2011) and claim that for some time, product developers believed companies that focus on financial measures of product performance did not perform as well as others. Data from the research conducted by the authors suggest just the opposite. It is important to note that financial measures are not only essential to all companies; they are also more likely to be related to the higher performing companies [5]. Barbosa et al. (2020) [6] opted to consider both R&D output indicators (patent and publication indicators) and traditional project performance indicators (cost, schedule and quality indicators) used in open innovation R&D projects.

The current level of knowledge in measuring R&D activities is not abundant, and the indicators for evaluating R&D projects could not be detected in general. For this reason, a decision was made to prepare a general list of indicators for measuring R&D results and later select suitable ones from that list for evaluating R&D projects. In the last twenty years, a small contribution has been made in this field. Much attention in the scientific literature has been paid to measuring TTO performance. Indicators for that purpose, which state the beginning of the project or identify efforts to achieve project start, are irrelevant in this article. As for the general indicators for the evaluation of R&D activities, no significant breakthrough has been made here: basically, there is a dispute in the literature about the expediency of using financial indicators; it is proposed to focus on the integration of R&D measurement systems; in different scientific works, R&D indicators are repeated and do not change but were enough to make a general list of R&D indicators.

Table 1. Indicators for measurement of R&D performance.

	Authors and Years	Kerssens van Drongelen and Bilderbeek (1999) [2]	Bremser and Barsky (2004) [3]	Cedergren et al. (2010) [1]	Markham and Lee (2013) [5]	Barbosa et al. (2020) [6]
Indicators for Measurement R&D Performance
Financial perspective:
Profit due to R&D/Profits from new product sales		•		•	•
Project cost versus budget/average development cost per product/total cost of new product effort as % of revenue/R&D spending as a percentage of sales			•		•	•
% of sales by new product(s)/current % of sales of new products/new product sales as a % of total sales/current-year percentage of sales due to new products released in the preceding X-number of years		•	•	•	•
Realized IRR/ROI/net margin ROI		•	•		•
New products sales					•
% of budget resources dedicated to R&D			•
% of resources to sustaining existing products			•
% of R&D budget for radical innovation products					•
Efficiency/keeping within budget		•
Customer satisfaction with new products/market response/% of products succeeding in the market/new product acceptance rate		•	•
Market share gained due to R&D		•
Market share trends					•
Business process perspective:
No. new products approved/released/no. of approved projects ongoing/the sheer numbers of projects/number of innovative projects			•	•	•
Products completed speed/the length of time spent on each development activity in each NPD stage/actual project schedule versus planned.		•	•		•	•
Product life cycle in the market			•
Total no. active projects supported.			•
Quality of output/work		•
Technical quality/maturity of the project execution						•
Change in R&D headcount/total R&D headcount			•	•	•
Pricing and profit planning accuracy		•	•
Innovation and learning perspective:
Number patents filed/pending/awarded		•	•	•	•
Comparison of the number of patent applications of the project assessed and other R&D projects of the company						•
The measure of the importance of patens					•
Number of ideas/findings		•
Creativity/innovation level		•
Strategic skill coverage ratio by competency category			•
R&D competency vs. competitors (innovation level)			•
Employee survey measures			•
Employee training (hours)			•
Network building		•
Other:
Professional esteem		•
Behaviour (in the group)		•

presents a prepared general list of R&D indicators, excluding those intended to assess the performance of TTOs.

Each row presents R&D indicators of a similar nature but treated slightly differently. For example, the first row shows indicators based on profit, the second on costs, and the third on the sale of new products.

Due to the lack of scientific research to evaluate R&D projects, a set of standard indicators was developed based on R&D indicators mentioned in the scientific literature (Table 1). Later, this list was used in expert research from the general list indicators, selecting only suitable indicators for R&D projects. In this case, the R&D project is an independent, individual business unit not related to the company’s finances, which has its budget. For the reliability of the research results, it was decided to select only indicators suitable for evaluating R&D projects with the help of R&D experts.

3. Materials and Methods: Selection of a Set of Indicators and Determining the Relevance of Indicators for Measuring the Performance in R&D Projects

Making a decision is a process where alternatives are assessed to select a choice or a course of action to fulfil desired objectives and goals. A suitable decision-making process can be essential for success in an organization [33]. The MCDM method defined the research structure and was used to summarize the results of the expert study. This method’s choice is based on a motive related to the evaluation purpose [34]—to determine indicators’ relevance among US and German experts. The set of indicators is the foundation for determining their relevance based on the multi-criteria decision-making (MCDM) methods of determination of weights. Due to the lack of scientific research dedicated to indicators for R&D projects, a general set of indicators was developed in Table 1. In this case, the R&D project is understood as an independent, individual business unit not related to the company’s finances in which it may or may not be carried out. For the reliability of the research results, it was decided to select indicators for R&D projects with the help of R&D experts and later, in the same way, determine their relevance. After the experts’ selection, the following remained on the list of R&D project indicators:

• Profit due to R&D/Profits from new product sales;
• Project cost versus budget/Average development cost per product/Total cost of new product effort as % of revenue/R&D spending as a percentage of sales;
• Realized IRR/ROI/Net margin ROI;
• New product sales;
• Customer satisfaction with new products/market response/% of products succeeding in the market/New product acceptance rate;
• Market share gained due to R&D;
• Products completed speed/the length of time spent on each development activity in each NPD stage/actual project schedule versus planned;
• Product life cycle in the market;
• Technical quality of the project execution/maturity of the project.

The second stage of expert research was conducted to establish the relative weight of the indicators in the set, in other words, to determine the relevance of the indicators with the intention compare them later. Experts were asked to express their opinion on the relevance of indicators for evaluating R&D projects. The experts had to divide 100% for each indicator: the more relevant indicator received a higher percentage, the less relevant indicator a lower percentage.

Calculations summarized by calculating in three ways: taking the most probable arithmetic mean, median and geometric mean. The weights were defuzzified of the computation [35]. Formulas 1–5 below were used for the calculation:

$w_{jl}-\underset{k}{\min }{y}_{jk},j=\overline{1,n},k=\overline{1,p}$ is minimum possible value, $w_{j\alpha }={\left({\displaystyle \prod }_{k=1}^p{y}_{jk}\right)}^{\frac{1}{p}},j=\overline{1,n},k=\overline{1,p}$ is the most possible value and $w_{j\beta }=\underset{k}{\max }{y}_{jk},j=\overline{1,n},k=\overline{1,p}$ is the maximal possible value of j-th indicator $w_{j\gamma }=\underset{k}{\min }{y}_{jk},j=\overline{1,n},k=\overline{1,p}.$

The experts assign the raw rating to each indicator on a scale of 0 to 100.

$$w_{cj}=p_{cj}/{\displaystyle \sum }_{c=1}^m{p}_{cj},$$

(1)

where $w_{cj}$-weight computed for indicator c from the rating given by judge j $p_{cj}$–rating is given by judge j to criterion c, and w_c is calculated as follows:

$$w_c={\displaystyle \sum }_{j=1}^n{w}_{cj}/{\displaystyle \sum }_{j=1}^n{\displaystyle \sum }_{c=1}^m{w}_{cj}.$$

(2)

Equations (3) and (4) using rules of fuzzy arithmetic are modified as follows:

$$\begin{array}{c}{\tilde{w}}_{cj}={\tilde{p}}_{cj}/{\displaystyle \sum }_{c=1}^m{\tilde{p}}_{cj}=\left(p_{c\alpha j}/{\displaystyle \sum }_{c=1}^m{p}_{c\gamma j};p_{c\beta j}/{\displaystyle \sum }_{c=1}^m{p}_{c\beta j};p_{c\gamma j}/{\displaystyle \sum }_{c=1}^m{p}_{c\alpha j}\right);\\ {\tilde{w}}_c=\left(w_{c\alpha };w_{c\beta };w_{c\gamma }\right)={\displaystyle \sum }_{j=1}^n{\tilde{w}}_{cj}/{\displaystyle \sum }_{j=1}^n{\displaystyle \sum }_{c=1}^m{\tilde{w}}_{cj}=\\ \left({\displaystyle \sum }_{j=1}^n{w}_{c\alpha j}/{\displaystyle \sum }_{j=1}^n{\displaystyle \sum }_{c=1}^m{w}_{c\gamma j};{\displaystyle \sum }_{j=1}^n{w}_{c\beta j}/{\displaystyle \sum }_{j=1}^n{\displaystyle \sum }_{c=1}^m{w}_{c\beta j};{\displaystyle \sum }_{j=1}^n{w}_{c\gamma j}/{\displaystyle \sum }_{j=1}^n{\displaystyle \sum }_{c=1}^m{w}_c\right);\end{array}$$

(3)

there, $w_{jl}-\underset{k}{\min }{y}_{jk},j=\overline{1,n},k=\overline{1,p}$ is the minimum possible value, $w_{j\alpha }={\left({\displaystyle \prod }_{k=1}^p{y}_{jk}\right)}^{\frac{1}{p}},j=\overline{1,n},k=\overline{1,p}$ is the potential value, when decision-makers use the geometric mean values, $w_{j\alpha }=\frac{{{\displaystyle \sum }}_{k=1}^p{y}_{jk}}{p},$ when decision-makers use the arithmetic mean values, $w_{j\alpha }=med\left(y_{jk}\right)j=\overline{1,n},k=\overline{1,p},$ when decision-makers use the median of a set of data, and $w_{j\beta }=\underset{k}{\max }{y}_{jk},j=\overline{1,n},k=\overline{1,p}$. The main difference between the median and the average is that unlike the average the median is not affected by outliers. Therefore, when data sets have outliers, the median is the preferred measure of central tendency.

A defuzzification is necessary before making final decisions. A defuzzification technique, a centre of gravity, is used in the case study:

$$w_c=\frac{1}{3}\left(w_{c\alpha }+w_{c\beta }+w_{c\gamma }\right).$$

(4)

Decision-makers calculated the average values of obtained criteria using the following equation:

$$w_c=\frac{1}{3}\left(w_{cA}+w_{cG}+w_{cM}\right).$$

(5)

The significance of the indicators is determined (

Table A1. Relevance of performance measurement indicators for R&D projects based on US R&D expert’s research results.

R&D Expert’s Research Results	Performance Measurement Indicators for R&D Projects
	Profit due to R&D/Profits from New Product Sales	Project Cost Versus Budget/Average Development cost per Product	Realized IRR/ROI/Net Margin ROI	New Product Sales	Customer Satisfaction with New Products/Market Response	Market Share Gained due to R&D	Products Completed Speed	Product Life Cycle in the Market	Technical Quality of the Project Execution/Maturity of the Project
1	10	13	14	20	6	6	6	20	5
2	30	20	5	5	20	5	5	5	5
3	10	11	15	12	17	16	3	1	15
4	20	10	9	6	20	4	3	8	20
5	8	5	5	20	20	4	10	8	20
6	15	20	4	3	8	20	20	5	5
7	20	5	5	10	10	10	30	5	5
8	30	20	5	5	20	5	5	5	5
9	10	20	18	23	4	3	1	20	1
10	20	4	3	5	5	11	15	12	25
11	4	20	4	20	4	4	4	20	20
12	10	20	5	11	15	12	10	7	10
13	10	10	10	3	12	10	20	5	20
14	3	8	18	5	11	11	15	12	17
15	8	11	15	12	12	4	10	8	20
16	11	12	12	19	12	4	10	16	4
17	10	15	11	15	12	12	4	10	11
18	10	15	30	20	5	5	5	5	5
19	17	15	4	15	12	16	1	4	16
20	9	8	20	12	4	10	17	16	4
21	10	3	12	10	8	17	11	15	14
22	12	17	16	12	6	4	10	3	20
23	8	4	10	17	7	12	10	17	15
24	9	10	19	4	15	15	11	10	7
25	12	20	4	10	10	8	12	10	14
Av.	0.1366	0.0953	0.1350	0.1069	0.0942	0.0921	0.1333	0.0920	0.1146
Med.	0.1276	0.1039	0.1276	0.1128	0.1039	0.0979	0.1217	0.0861	0.1187
Geom.	0.1363	0.1050	0.1296	0.1110	0.1040	0.0952	0.1189	0.0899	0.1100

and

Table A2. Relevance of performance measurement indicators for R&D projects based on German R&D expert’s research results.

R&D Expert’s Research Results	Performance Measurement Indicators for R&D Projects
	Profit due to R&D/Profits from New Product Sales	Project Cost Versus Budget/Average Development Cost per Product	Realized IRR/ROI/Net margin ROI	New Product Sales	Customer Satisfaction with New Products/Market Response	Market Share Gained due to R&D	Products Completed Speed	Product Life Cycle in the Market	Technical Quality of the Project execution/Maturity of the Project
1	10	10	15	14	10	8	3	10	20
2	12	17	16	4	9	8	6	8	20
3	11	26	12	2	12	15	4	10	8
4	17	4	5	10	8	20	17	16	20
5	10	15	15	12	17	1	2	12	16
6	20	11	2	12	4	3	8	20	20
7	11	20	4	10	17	16	4	10	8
8	13	15	4	10	8	12	10	8	20
9	17	11	3	8	11	6	15	12	17
10	12	17	16	4	10	8	13	16	4
11	10	15	30	20	5	5	5	5	5
12	11	15	12	10	8	20	16	4	4
13	10	8	11	15	12	19	3	8	14
14	11	10	12	20	4	10	10	3	20
15	11	20	4	10	17	16	4	10	8
16	11	15	12	8	10	8	16	8	12
17	12	12	16	11	15	6	10	8	10
18	20	4	10	8	4	10	17	16	11
19	3	8	20	10	12	20	4	10	13
20	10	8	17	11	15	12	3	8	16
21	10	8	13	3	12	10	12	16	16
22	10	9	12	10	10	8	13	14	14
23	16	8	12	3	12	10	8	13	18
24	10	8	12	17	16	10	5	12	10
25	10	3	12	10	12	8	12	17	16
Av.	0.1068	0.1348	0.1530	0.1045	0.0922	0.1047	0.0892	0.1059	0.1090
Med.	0.1093	0.1286	0.1415	0.1029	0.1029	0.0997	0.0868	0.1061	0.1222
Geom.	0.1131	0.1303	0.1386	0.1014	0.1020	0.0995	0.0866	0.1089	0.1196

in Appendix A;

Table 2. Relevance of performance measurement indicators for R&D projects based on the research of US and German R&D experts.

Performance Measurement Indicators for R&D Projects	Evaluation Results Based on US Experts					Evaluation Results Based on German Experts					Differences (4–9)
	Av.	Med.	Geom.	(1 + 2 + 3)/3	Rank	Av.	Med.	Geom.	(6 + 7 + 8)/3	Rank
	1	2	3	4	5	6	7	8	9	10	11
(1) Profit due to R&D/Profits from new product sales	0.1366	0.1276	0.1363	0.1335	1	0.1068	0.1093	0.1131	0.1098	4	0.0238
(2) Project cost versus budget/Average development cost per product	0.0953	0.1039	0.1050	0.1014	6	0.1348	0.1286	0.1303	0.1312	2	−0.0298
(3) Realized IRR/ROI/Net margin ROI	0.1350	0.1276	0.1296	0.1308	2	0.1530	0.1415	0.1386	0.1443	1	−0.0136
(4) New product sales	0.1069	0.1128	0.1110	0.1102	5	0.1045	0.1029	0.1014	0.1029	6	0.0073
(5) Customer satisfaction with new products/market response, etc.	0.0942	0.1039	0.1040	0.1007	7	0.0922	0.1029	0.1020	0.0990	8	0.0017
(6) Market share gained due to R&D	0.0921	0.0979	0.0952	0.0951	8	0.1047	0.0997	0.0995	0.1013	7	−0.0062
(7) Products completed speed/the length of time spent on each development activity in each NPD stage, etc.	0.1333	0.1217	0.1189	0.1246	3	0.0892	0.0868	0.0866	0.0875	9	0.0371
(8) Product life cycle in the market	0.0920	0.0861	0.0899	0.0893	9	0.1059	0.1061	0.1089	0.1070	5	−0.0177
(9) Technical quality of the project execution/maturity of the project	0.1146	0.1187	0.1100	0.1145	4	0.1090	0.1222	0.1196	0.1169	3	−0.0025

) reflecting the elements’ influence on the assessed object. There, $w_{cA}$ is indicator significance when the arithmetic means are the possible values, $w_{cG}$ is indicator significance when the geometric means are the possible values, and $w_{cM}$ is criterion significance when the medians are the possible values [36,37].

Experts were selected from the US and German R&D organizations according to the following criteria: (1) experience in the process of TC or process the development of products/services, or research, the subject of which is the process of TC, or the development of products/services; (2) positions of the person in organizations and institutions responsible for the TC, development of products/services, or scientific research in the field of R&D. All specialists had at least ten years of experience in the field of technology commercialization. The research in the USA involved eleven experts who were representatives of technology transfer centres, six start-up employees and founders, five representatives of large corporations and three researchers studying the process of technology commercialization. In the study conducted in Germany, two representatives of technology transfer centres, fifteen start-up employees and founders, six representatives of big technology developing companies, and two investors in technology projects participated.

4. Results: Relevance of Indicators for Measuring the Performance of R&D Projects

In the second stage, the experts expressed an opinion on the weight/relevance of indicators. In this case, the most relevant indicators receive the highest point, whereas the least relevant is the lowest. After the research, fifty correctly completed R&D indicator relevance questionnaires/tables were selected: twenty-five questionnaires from R&D organizations in US and twenty-five from R&D organizations in German. In Table A1 and Table A2 in Appendix A are provided all research results, results with ranks and differences in Table 2.

After the calculations, the final relevance of the indicators was determined. Regarding the results of the study by US experts, the first, third and seventh indicators are considered the most relevant here. In the middle position are the ninth, fourth and second indicators. The fifth, sixth and eighth indicators appeared in the lowest position. Regarding the results of the study by German experts, the third, second and ninth indicators are in the highest position. In the middle position, first, eighth and fourth indicators, and in the lowest position sixth, fifth and seventh indicators.

Comparing the difference between the evaluations of the US and German experts, we can see the most significant difference between the relevance of the seventh, second and first indicators. A much smaller difference can be seen regarding the third and eight indicators. The differences in assessing the remaining indicators, fourth, fifth, sixth and ninth, are minimal.

Many indicators are intended to measure the performance of the R&D project after the product is introduced to the market. However, experts were also asked about the possibility of using these indicators in the planning stage or later stages. Indicator (5) ‘Customer satisfaction with new products/market response/% of products succeeding in the market/new product acceptance rate’ was mentioned as the most important in the initial stage of PD. Indicator (7) ‘Products completed speed/the length of time spent on each development activity in each NPD stage/actual project schedule versus planned’ should be monitored throughout the project. The seventh indicator is closely related to indicator (2) ‘Project cost versus budget/Average development cost per product’.

5. Discussion and Conclusions

This article emphasizes the R&D project performance measurement, where the project is perceived as an independent, individual business unit. As a rule, it was not possible to answer the question about the most suitable indicators for measuring R&D projects. After an unsuccessful attempt to find studies dedicated for R&D project performance measurement, a decision was made to prepare a general list of indicators for measurement R&D performance and later, with the help of experts, to select indicators from this list, suitable for measuring R&D projects. The most specific and at the same time the most useful results were presented by Kerssens van Drongelen, Bilderbeek (1999) [2], Bremser, Barsky (2004) [3], Cedergren et al. (2010) [1], Cedergren and Larsson (2014) [4], Markham and Lee (2013) [5] and Barbosa et al. (2020) [6]. These authors examine the problems of measurement R&D performance, and all unanimously emphasize the gap between scientific research and practice. As for the general indicators for the evaluation of R&D activities, no big breakthrough has been made here too: basically, there is a dispute in the literature about the expediency of using financial indicators; it is proposed to focus on the integration of R&D measurement systems; in different scientific works, R&D indicators are repeated and do not change but were enough to make a general list of R&D indicators. Much attention in the scientific literature has been paid to measuring TTO performance and indicators for that purpose, which state the beginning of the project or identify efforts to achieve project start; therefore, they are irrelevant in this article.

For selection and evaluation relevance of indicators, experts were chosen from the US and Germany as representatives of the R&D sector. By conducting research in different countries and comparing their results, areas and opportunities for further mutual learning are identified, as well as more intensive, mutually beneficial international cooperation in the field of technology development is developed.

All research results are provided in Table A1 and Table A2 in Appendix A, final results with ranks and differences are provided in Table 2. Regarding the results of the study by US experts, the first, third and seventh indicators are considered the most relevant here. The fifth, sixth and eighth indicators appeared in the lowest position. In terms of the results study by German experts, in the highest position are the third, second and ninth indicators, and in the lowest position are the sixth, fifth and seventh indicators.

A list of indicators is presented in Table 2, fill the research gap and meet the needs of practitioners. In addition, here can be seen quite pronounced differences in the evaluations of German experts and the US, which reflect differences in experiences and in business environment conditions. Comparing the difference between the evaluations of the US and German experts, we can see the most significant difference between the relevance of the 7th indicator: ‘Products completed speed/the length of time spent on each development activity in each NPD stage, etc.’, 2nd indicator: ‘Project cost versus budget/Average development cost per product’ and 1st indicator: ‘Profit due to R&D/Profits from new product sales’. Taking into account the discussion in the literature about the appropriateness of financial indicators, we can state that the results of this study support the opinion of proponents of financial indicators; most of the indicators selected with the help of experts was financial, others reflecting problems in the market and reflecting dimensions of product development.

It is vital to acknowledge the importance of evaluation of the planning activities. Considering Cooper’s observations regarding the critical stages in the PD process, Gates is rated one of the weakest areas in PD. The 3rd Gate earned the most attention and is considered one of the most responsible because, after 3 Gates, a decision is made regarding implementing plans and significant investments. Additionally, from a value creation point of view, during the early activities of the development, when it is decided what customer needs will be satisfied and what the possible value of the PD investments can be. The focus tends on the realization phase of the PD process too. The research states that the PD function cannot be considered successful from a company perspective until all three activity categories have been completed and a product has been delivered to the customer.

Several limitations of the research need to be mentioned. It was established lack of previous studies in the research area, identified literature gap and the need for further development in the area of study. Additionally, the literature in this field lacks concreteness in naming the place of use of indicators in technology development projects. Attempt to find studies dedicated for measurement the performance of R&D projects in science were unsuccessful. A lack of scientific research for the measurement, control and setting indicators for measurement performance of R&D projects does not allow policy representatives to identify problematic areas of R&D project implementation and to make targeted decisions regarding the promotion and financing of technology development. In addition, after applying specific technology development promotion instruments, it is difficult to determine the effectiveness of these measures.

The research was carried out by interviewing the US and German R&D experts; basically, the goal was achieved, and suitable indicators for R&D projects were identified and evaluated, but we cannot claim that the research reflects the opinion of US and German experts due to the geographical limitations of the research. The US study was conducted in R&D organizations in San Francisco and Los Angeles, while in Germany, the study was conducted at the Hannover Messe 2022 technology fair. If the limited ability to access a more comprehensive geographic range of participants could have been avoided, results mirroring those of the US and Germany may have been obtained.

Future research directions should include a more accurate determination of the place of indicators in the technology development process, broader and more specific studies in determining indicators for R&D projects and efforts to develop an evaluation scale for each indicator. It would significantly contribute to creating a functional model, a tool for performance measurement in R&D projects.