Assessment and Selection of Technologies for the Sustainable Development of an R&D Center

Abstract

The central role of R&D centers in the advancement of technology within industrial enterprises is undeniable and clearly affects their strategies, their competitiveness and their business sustainability. R&D centers assume responsibility for technology recognition, collection, acquisition, development and transition. Among their activities, the efficient choice of emerging technologies in the Technology Management Process is becoming a real challenge. In such heterogeneous scenarios, Multiple Criteria Decision Making (MCDM) models are commonly proposed as an appropriate decision-making approach. Multiple research works address the selection of particular technologies in industrial applications, but very few references can be found related to research institutions, and R&D centers in particular. Therefore, a decision-making model is provided in this study following the MIVES multi criteria method for the assessment of one or more technologies. The model is then applied to two case studies related to the selection process of new technologies at a Spanish R&D Center specialized in manufacturing.

1. Introduction

In advanced societies, R+D+i (Research+Development+innovation) is widely recognized to be necessary in order to ensure sustainable socio-economic development [1]. Its relevance is understood both for the well-being of the most advanced countries, and for the development of countries at different stages of industrial and technological development [2,3,4]. In recent years, special emphasis is being placed on the importance of increasing the effectiveness of the R&D development chain, seeking two main objectives:

• Shorten the technology’s maturation period to the limit, so that the period between the initial testing activities and the market penetration of its findings is as short as possible.
• Increase the efficiency of the duration of testing. Efficiency is defined in two ways: in terms of the correct selection of the technology to be developed (i.e., minimizing the effort devoted to low industrial and socioeconomic return technologies) [5]; and in terms of the efficiency of the process, adjusting resources in the production chain to optimize the ratio between return and investment [6,7].

R&D Centers have a significant role to play to ensure the quality and the efficiency of the R+D+i activities, since they are technology developers and providers, and their activity is linked to science and academy on one side of the R+D+i chain, and to industry on the other side. One of the activities that is crucial to achieve the above highlighted objectives is the correct selection of the technologies to acquire and develop. This is key for any organization involved with technology, and especially for R&D centers. Nevertheless, very few references can be found in the literature dealing with the topic of strategic technology selection in R&D centers. A significative number of research works can be found applied in the industrial scenario dedicated to the selection of particular technologies, and a lower number of works related to public administration and universities [8,9], mainly dealing with general research program orientation. For R&D centers, the references found in literature are very limited and basically oriented to evaluate the capabilities of R&D centers, and their contribution to the industrial innovation process [10,11], not focusing on the technology identification, assessment and selection process, which is the focus of this research work.

In this context, this research work is oriented to the design and development of a model to overcome that methodological gap, addressing the phase of technology assessment and selection within the technological development activity of an R&D center.

One of the main contributions of the developed model is its structure, covering the complete activity frame of an R&D center, addressing the characteristics of the technology, the characteristics of the R&D center in relation to the evaluated technology and the characteristics of its industrial customers. This approach is supported by the fact that the R&D process cannot be understood as an isolated element, but as a process that is part of a global strategy, in which an R&D center has to observe technological and scientific impacts, but at the same level, the impact generated to the industry by means of its technology transfer activities.

1.1. The Role of R&D Centers

R&D Centers play an important role in the R+D+i scenario, due to their place in the technology development chain. Their activities cover from basic and applied science up to technology transfer (i.e., from Technology Readiness Level—TRL 2-3 to TRL 7-8, with their core activity being focused at TRLs 3-6). Their relevance is also supported in the fact that they are, to a large extent, the connection link between the generation of knowledge that is carried out in most academic research activities in the university, industry, business and market sectors [12,13]. EARTO (European Association of Research and Innovation Organizations) itself defines R&D centers, within the broader category of Research and Technology Organizations (RTOs), as providers of technology and innovation for businesses and governments, maintaining an important part of their activity integrated into basic science in cooperation with the university [14]. This position in the knowledge and technology development chain obliges technology centers to adopt development strategies that must necessarily take into account both the aspect of knowledge generation (R) and the commercial transfer and exploitation of results (i), in addition to their nuclear activity of technological development (D) [14,15,16]. This is the situation also in the Basque Country, where recently the Basque Government has reinforced the role of R&D centers, creating a new technological Alliance, the Basque Research and Technology Alliance (BRTA), which integrates in a collaborative organization the whole of the Basque R&D centers, under the statement “The bet of a country on research and technology” [17].

Ideko, one of the members of this Alliance, with more than 30 years of experience specialized in Machine-Tool and Manufacturing technologies is represented in this research work, providing the case studies, framed in the decision making process of it’s strategic reflection, and the selection of technologies carried out within that reflection.

1.2. Management of Technology and Technology Assessment and Selection Process

In the most commonly accepted model [18], the Management of Technology Process is structured into five stages: recognition, selection, acquisition, exploitation and security. In the case of R&D centers, the identification and selection activities are especially critical from the point of view of decision-making, as they are usually framed in the processes of strategic and technological planning, thereby defining future lines of development, the fundamentals for their scientific and economic competitivity and sustainability [11,19].

In this phase of identification and selection of technologies, the identification activities are quite structured and systematized at R&D centers, based on tools such as Technological Surveillance methodologies [20] that enable proper and effective identification of technologies.

The scientific literature presents numerous experiences of selecting specific technologies at a high (TRL) level of maturity, with a view to their implementation in industrial companies [4,21,22,23,24,25,26]. There are also research works that deal with selection and decision-making in technological fields of interest analyzed by public administrations for the definition of their R&D support programs [8]. However, there are very few examples relating to universities [9,27], which are, at a more incipient stage of the technologies, applied in the process of selecting technologies or technological lines to be developed. The examples related to R&D centers are even fewer in the literature [10,11].

Technology selection is one of the most important decision-making processes that a company, and even more so a technology center, must address within its strategies and activities [28,29]. This challenge is made more complex by the increasing availability of technology alternatives to select and the need to select the appropriate technology to achieve the organization’s objectives [30]. Furthermore, the criteria for determining the most appropriate technology may differ depending on the specific requirements of a company and the context in which the process is carried out [31]. Furthermore, technology selection is a process that is closely related to other processes in the organization and is associated with a broader technological, organizational and business environment. This decision therefore encompasses more than a pure technical evaluation of the available alternatives [32]. Taking into account this global vision of the business, where factors related to knowledge, strategy, capacity, organization, culture, finance and market are necessary to correctly evaluate the suitability of a technology, it can be understood that the problem of evaluating and selecting a technology is, definitively, a complex problem in which heterogeneous factors intervene [16,33].

1.3. Multi Criteria Decision Making Methods for the Selection of Technology

Based in this complexity, there is an outstanding use of Multicriteria Decision Making Methods (MCDM) as a central tool for objectifying the process of analysis, evaluation and decision-making in the technology evaluation and selection process [34,35,36,37]. MCDM configure a family of analysis tools that have been gaining relevance in the analysis of complex problems due to their inherent capacity to assess different alternatives based on the establishment, evaluation and weighting of homogeneous or heterogeneous criteria for the selection of the most suitable alternative. The differential characteristics of the methods lie in their capacity to incorporate multiple non-proportional and heterogeneous criteria, qualitative and quantitative, with different units of measurement between them, and the presence of very different alternatives [38,39].

This versatility has allowed them to be applied in areas as diverse as energy, environment and sustainability; strategic management; production management; supply chain management; quality management; projects management; security and risk management; information management, technology and manufacturing systems; software and hardware development; tourism management; education, public administration programs [35,37].

Related to the central topic of this research work, the assessment and selection of technologies, the application of MCDM tools is highly widespread, as mentioned above and detailed in the literature review below. The applications of these methods for the evaluation and selection of technologies, in particular in industrial scenarios, vary from the selection of very particular technologies [40,41,42] and technological equipment [40,43,44] to wider strategic decisions about technology and business, in business fields such as suppliers and partnerships [32,45,46], strategy and technology portfolio configuration [30,47,48,49,50]. Within the family of MCDM methods, MIVES (Integrated Value Model for Sustainable Assessment) [51] is one of the youngest ones, developed in a collaborative research between universities and R&D centers in Spain. Based on other relevant methods, such as Delphi and AHP, MIVES introduces the use of value functions to characterize the factors of the evaluation and process, minimizing the subjectivity in the assessment [52,53]. So far, MIVES has already been widely used in several fields of building sector: selection of constructive materials [54,55,56], urban planification [57,58,59], environmental and sustainability evaluation in construction and urban management [53,60,61], hydraulic structures [61,62], post-disaster sites and housing selection [63,64]. The relevance of the method in the construction sector in Spain is such that it has been adopted as a reference for assessing the sustainability of concrete structures in the current Spanish Structural Concrete Code [65].

1.4. Problem Statement and the Contribution of This Research

Although, as has been described, the problem of selecting technologies in the industrial environment, and the use of MCDM methods to address it, is widely extended, it is no less true than in the case of R&D centers, the references found in literature are very limited [10,11]. Moreover, the references that can be found are oriented to evaluate the capabilities of R&D centers, and their contribution to the industrial innovation process, not focusing on the technology identification, assessment and selection process, which is key for their own sustainability and the competitivity and sustainability of their industrial customers.

In that scenario, the objective of this research work has been focused to the design and development of a model to overcome that methodological gap, addressing the phase of technology assessment and selection within the technological development activity of an R&D center.

The developed model has been designed to represent the complete activity frame of an R&D center, addressing the characteristics of the technology, the characteristics of the R&D center in relation to the evaluated technology, and the characteristics of its industrial customers. This approach is supported in the fact that the R&D process cannot be understood as an isolated element, but as a process that is part of a global strategy, in which an R&D center has to observe technological, and scientific impacts, but on the same level, the impact generated to the industry by means of its technology transfer activities [12,13].

This integrated scenario to be modelized encompasses scientific, technological, organizational, human, strategic, market and operational factors, configuring a heterogeneous assessment environment that seems appropriate to apply the characteristics and virtues of MCDM methods. Among these methods, the research team has considered that the application of MIVES provides a novelty contribution in two aspects, one related to the organization subject of the research, an R&D center, in the process of evaluating and selecting technologies. Additionally, the extension of MIVES, widely applied in the construction sector, to the manufacturing sector, in which Ideko R&D center develops its activities, introduces an originality aspect to the proposed approach.

In the following sections, we will first present the literature review on the application of MCDM methods, followed by the research methodology and the detailed description of the development process of the technology selection model. Then, the results of applying this model to two real case studies, along with the overall results of the sensitivity analysis will be described. In the Discussion section, the relation of these results to previous contributions will be explained and, in the Conclusions section, the main contributions of the research, its limitations, its implications and directions for future work will be described.

2. Literature Review

The literature review has been structured carrying out a chronological description of the most representative works related to the application of MCDM methods for the selection of technologies. The contributions of the different authors are presented, and the chapter is completed with a table that summarizes the most relevant contributions. As mentioned above, it must be remarked that most of the papers found refer to actions related to industry.

Kleindorfer and Partovi [50] provided an analysis of competitive strategy to construct their methodology for the selection of technology in manufacturing companies, gathering the core characteristics for the formulation of company strategy. Those characteristics helped to define the relevance of cost, quality, flexibility and dependence for profitability and long-term viability. The evaluation of performance potential was done by applying the Analytic Hierarchy Process (AHP) method, developed by Saaty [66]. In a similar approach, Mohanty and Deshmukh [23], developed a model for the selection of manufacturing technologies based on a progressive process that began with the definition of strategic objectives, the identification of advanced manufacturing technologies, the identification of key attributes of those technologies and, finally, evaluation and selection.

The model proposed by Yap and Souder [67] contributed a remarkable combination of the analytical aspects of technology selection decisions and the impacts that internal aspects, both behavioral and organizational, generated within the company. The model presented by Punniyamoorthy and Ragavan [68] introduced the consideration of tangible and intangible factors into the technology selection process, which were evaluated by AHP.

Bayazit [69] presented a case of AHP applied to the selection of a Flexible Manufacturing System. Its originality lies on the approach to evaluate a single technology, instead of a selection from several alternatives.

Mohanty et al. [70] presented one of the few works focused on the selection of R&D projects. Methodologically, they defended the advantages of the Analytical Network Process (ANP) method, developed by Saaty [71] as an evolution of AHP. Eilat, Golany and Shtub [5], proposed a model for the evaluation of R&D projects throughout their life cycle. Huang, Chu and Chiang [59] presented a conventional AHP-based multi-criteria selection model, with the particularity that it is one of the very few works aimed at the selection of R&D projects, in this case by the Public Administration.

Shehabuddeen, Probert and Phaal [31] introduced the concept of a “framework” for technology selection. They structured this framework into four key areas and proposed the use of two filtering sequentially applied phases, so that the number of candidate technologies was progressively reduced.

Büyüközkan and Çifçi [72] developed a model that combined three MCDM methods: fuzzy Decision-Making Trial and Evaluation Laboratory (DEMATEL) [73], to generate the relationship matrix, fuzzy ANP, for pairwise comparisons, and fuzzy TOPSIS, for the evaluation of the alternatives. The authors concluded that the combined approach provided a more precise analysis.

Cho and Lee [74], presented the application of two conventional methods, Delphi and fuzzy AHP, for decision-making relating to the commercialization of products based on new technology. Based on a conventional approach, they proposed an interesting hierarchical structure taking into account technological, organizational, marketing and business aspects.

Ma, Chang and Hung [33] developed a work based on applying the fuzzy AHP and Delphi methods to the selection of industrial solar energy technologies, applied to the case of Taiwan. The differentiating aspect of the work was that it focused on an in-depth analysis of the characteristics of the technology, under the criteria of innovation, significance, IPR, extension potential, sustainability, benefits and risks.

Ordoobadi [75], in a study on the selection of advanced manufacturing technologies, introduced the importance of correctly characterizing both the positive and the negative impacts of critical factors. The methodology in use was AHP, and provided a systematic approach to quantify intangible risks and benefits.

Meesapawong et al. [10] proposed a method for the orientation of innovation in Public R&D Organizations, especially focused on developing countries for a correct choice of research strategy. Applying a conventional methodology, combining Delphi and AHP, the factors used were very interesting, associated with strategic decisions, such as mission, research strategy, technological collaboration and technology management. Farooq and O’Brien [32] updated their 2010 and 2012 work by introducing two differentiating factors into the decision model: risk analysis for the selection of manufacturing technology and the most global vision of the company, taking into account the supply chain.

Viñolas et al. [51] presented the Modelo Integrado de Valor para Evaluaciones de Sostenibilidad (MIVES) method. It integrated other significant methods such as Delphi and AHP and contributed with the inclusion of more adjustable and precise value functions to obtain the assessments (from 0 to 1) of the factors at the last level of the hierarchical tree. Shen et al. [30] developed a classic approach with the combination of Delphi and AHP that included the Patent Co-citation Approach (PCA) [76], in the final phase to build the ranking of technologies.

Cuadrado et al. [60] develop a methodology for the evaluation of sustainable solutions for industrial buildings. They use the MIVES method with the main objective that the resulting methodology can be applied both to the evaluation of future constructions and existing buildings. Pons et al. [53] make a comparative analysis of MIVES with other multi-criteria methods in civil engineering and architecture applications and support the applicability of MIVES with two case studies. Pujadas et al. [59] apply MIVES for evaluating the impact on the sustainability (economic, environmental and social) of public investment projects. It is completed with a case study in the city of Barcelona, concluding that the MIVES method provides accurate, consistent and repeatable evaluations.

Table 1. Main contributions in the application of MCDM methods for technology assessment.

Author	Contribution
Saaty [77,78]	AHP method, the most widely applied, combined and adopted. ANP method, as a variant of AHP, introducing the interrelations between different levels of the hierarchy tree
Kleindorfer and Partovi [50]	Introduce organization strategy concepts on the evaluation by AHP
Bayazit [69]	Structures AHP hierarchy tree in risk and benefict factors One of the few works that evaluate a single technology
Mohanty et al. [70]	One of the few works related to R&D activities selection, starting from a strategic view
Huang et al. [79]	Oriented to the selection of R&D projects in Public Administration decisions
Eilat et al. [6]	In a model for the assessment of R&D projects introduce the evaluation throughout their life cycle
Shehabuddeen et al. [31]	Introduced the concept of a “framework” for technology selection. They proposed a sequential filtering to progressively reduce the number of candidate technologies
Büyüközkan and Ifi [72]	Combine three methods: fuzzy DEMATEL, fuzzy ANP and fuzzy TOPSIS
Cho y Lee [74]	Proposed an interesting hierarchical structure taking into account technological, organizational, marketing, and business aspects
Ordoobadi [75]	AHP combined with Taguchi. Characterized both the positive and the negative impacts of critical factors with a systematic approach to quantify intangible risks and benefits
Meesapawong et al. [10]	The use case was the orientation of innovation in Public R&D Organizations, especially focused on developing countries for a correct choice of research strategy. The factors used were associated with strategic decisions, such as mission, research strategy, technological collaboration and technology management.
Pons et al. [53]	Comparative analysis of MIVES with other multi-criteria methods in civil engineering and architecture applications and support the applicability of MIVES with two case studies.

summarizes the main contributions found in the above described literature review.

3. Research Methodology

The objective of this study was to develop a model capable of objectifying the process of evaluation and decision about the adoption of new technologies in an R&D center. To ensure the observance of all the factors that characterize the success of the decision, and the impact on the sustainability of the center, the model must cover the scientific-technological factors of the technology itself and the particularities of the center, but it cannot avoid the factors related to the potential customers, which are the key elements to ensure the success and the sustainability of the commitment to technology.

With this perspective, the model will include factors associated with the characteristics and the position of the technology, the strategies and capabilities of the R&D center, the capabilities and the proper adaptation of the technology to the customers of the R&D center and the accessibility to the market or potential markets of the technology. MIVES method, introduced in the previous section, was selected as the methodology for the development of this model. This selection is based on certain favorable and differentiating particularities of MIVES: the integration of fully recognized methods such as Delphi and AHP and the customizable value functions tuned to the essence of the variables that constitute the last level of the decision tree. With this aim, the research was structured into three phases:

• Configuration of the decision tree, by means of a semi-systematic literature review and the application of Delphi with a panel of experts.
• Applications of MIVES method, with two main activities included: Quantification of the decision tree, calculating the weights of the parameters by means of AHP pairwise comparison and characterization of the indicators by the design of the value functions.
• Application of the model to two case study.

These three phases are described in more detail as follows.

3.1. Configuration of the Decision Tree

In this first phase, the identification of the parameters that characterize the technology and its adaptation to the development scenario (capabilities and characteristics of the R&D center and its customers) was addressed. To do so, an analysis of the scientific literature has been carried out following a “semi-systematic literature review” approach [80,81]. The semi-systematic or narrative review approach is usually applied in the analysis of topics that have been conceptualized in different contexts and studied by different groups of researchers within different disciplines, so that a complete systematic review process is difficult. The semi-systematic analysis methodology is usually completed with a qualitative analysis, although there are exceptions. This type of analysis is considered useful to identify topics, theoretical perspectives or common themes within a specific discipline or research methodology or to identify components of a theoretical concept [82].

In this research work, the adoption of this approach is justified by the fact that the parameters identified in the literature review will be then transferred to the panel of experts for their evaluation, selection and filtering, applying a Delphi consensus process. Therefore, the result of the semi-systematic literature review is a preliminary identification and selection of parameters. The most frequently cited parameters from the literature review were selected, to configure the first version of the hierarchy tree.

This preliminary wide set of parameters was then analyzed by the panel of experts following a Delphi based procedure. The panel of experts reviewed and refined this preliminary version of the hierarchy tree, and after the achievement of the consensus, the outcome of the process is the definitive hierarchy tree. The panel of experts was composed of relevant people from the research work scenario: managers and researchers of R&D centers, professors from universities and top-level managers and technicians from customer companies that operate in the technological and geographical universe of the R&D center. Experts from international R&D centers in the field of analysis were also included. The expert selection process followed the recommendations made by Boje and Murnighan [83] and Linstone and Turoff, [84] criteria and, finally, 14 experts formed the final panel.

The outcome of this configuration phase was the first version of the hierarchy decision tree.

3.2. Aplication of MIVES Method

In this phase, the definitive hierarchy decision tree was completed, following the MIVES methodology. The main activities in this phase are:

• The definition of the value functions that characterize the indicators, lower level of the hierarchy tree. This activity is carried out by the research team.
• Assignment of weights to the parameters in the different levels of the hierarchy tree. Following the AHP procedure that MIVES integrates, the weighting is performed by a pairwise comparison. This activity is completed by the research team members, using as reference the weights assigned and comments provided by the panel of experts during the Delphi consensus rounds.

3.3. Application to Two Case Studies

Having developed the model, it was applied to the evaluation of two case studies, two manufacturing technologies, in a real assessment process at the selected R&D center. The application scenario is Ideko, an R&D center located in the Basque Country and specialized in machine tools and manufacturing processes. Its main customers are manufacturers of machine tools, in particular the companies that lead the sector in Spain, and are considered amongst the references in Europe, in the fields of grinding and milling. In the process of Strategic Planning 2009–2012, The R&D center adopted the decision to include two new technologies in its research agenda: Additive Manufacturing, more precisely Laser Cladding or Laser Metal Deposition (LMD) technology [85], and Micromachining technology, focusing of Micro-milling and Micro-turning [86]. These case studies were selected with a time perspective, which provided the necessary data to complete the model and offered a temporal visualization of the real evolution of the analyzed technologies and, therefore, the appropriateness of the results provided by the model.

The application to the case studies is completed by a sensitivity analysis. The sensitivity analysis is an activity included as optional in the MIVES method, recommended to test the stability and reliability of the model, proving that small modifications in the weights assigned do not generate significant modifications in the final value provided by the model [36,87].

4. Results

In this chapter the research work conducted throughout the three phases introduced in the previous section will be described in detail.

4.1. Configuration of the Decision Tree

The objective of the research work was to generate a tool to assist decision-making processes at an R&D center in the evaluation and selection of potential technologies or technological lines to be included in a research agenda. This decision-making process can occur during a strategic reflection process, usually involving the evaluation of several candidate technologies, but can also occur in daily activity, with the emergence and identification of new technologies or a single technology to be evaluated. Therefore, the model should allow a comparison between alternatives and an evaluation of a single one. It should, moreover, be reasonably agile in its application, particularly for obtaining the objective, real data that represent and characterize the indicators.

Most importantly, the model must represent the scenario of actuation of an R&D Center, attending to all the relevant areas and variables that define the adequacy of a technology and the success of its acquisition and development. For this purpose, the success scenario of developing a technology at an R&D Center was structured into three areas:

• Area 1: The characteristics or internal indicators of the technology itself.
• Area 2: The adequacy of the technology to the R&D Center, or vice versa, how the R&D Center is qualified and organized to develop that technology and to transfer it.
• Area 3: The industrial scenario, a key aspect for the sustainability of any technology development, the extent to which the industrial customers of the R&D center are qualified and committed to the technology or technologies that are under evaluation and the access of those industrial customers to the technology center and their relations with it.

The semi-systematic literature review performed analyzed the structure of the hierarchies proposed by the different authors, and the key parameters that they proposed, to characterize the research cases. Although the papers referring to R&D activities and institutions were very limited, the works related to industrial needs provided very interesting and helpful criteria. These criteria, which concern the characteristics of the technology, the characteristics of the entities that are involved, and the boundary conditions, were collected, analyzed and adapted to the proposed structure of our model. The works reviewed, the parameters collected and the way they fit to the three key areas of the model are presented in

Table 2. Key parameters identified in the literature review.

Key Area	Parameter	Authors
Technology-maturity	Position of the technology in its own life-cycle; Fundamental research; Advanced research; Engineering research; Advancement of technology; Innovation of technology; Technology maturity	Mohanty et al. [70]; Davoudpou et al. [55]; Shen et al. [30]; Ordoobadi [88]; Anand and Kodali [89]; Ma et al. [33]
Technology-relevance	Competitiveness of technology; Proprietary technology; Relevance of technology; Technological expandability; Technology supportability; Originality; Imitability; Patentability; Threat of substitution technologies; Potential technical interaction with existing products and technologies	Huang et al. [79]; Davoudpou et al. [55]; Ma et al. [33]; Cho and Jaeho [74]; Shen et al. [30]; Davoudpou et al. [55]; Kim et al. [90]; Choi et al. [91]; Kim et al. [90]
Technology-market	Potential market; Potential market size; Expected market share; Effect on existing market share; Commercialization potential; Entry barriers; Market attractiveness; Market competition	Mohanty et al. [70]; Cho and Jaeho [74]; Cooper [92]; Shen et al. [30]; Huang et al. [79]; Shen et al. [30]; Ma et al. [33]; Zirger and Maidique [93]; Slater and Narver [94]; Mishra et al. [95]; Balbontin et al. [96]; Jain et al. [97]
Technology-risks	Commercial risk; Economic risk; Technical risk; Evidence of scientific feasibility	Mohanty et al. [70]; Shen et al. [30]; Huang et al. [79]; Dedasht et al. [98]
R&D Center/customers—qualification	Human dimension-training cost; Competence and experience on similar project; Research staff availability; Technical Ability; Experience of the R&D personnel; Capability of research team; Alignment with organization objective and capability	Nouri et al. [99]; Bayazit [69]; Farooq and O’Brien [32]; Mohanty et al. [70]; Jain et al. [97]; Davoudpour et al. [100]; García-Valderrama et al. [101]; Huang et al. [79]
R&D Center/customers—equipment	Facilities available; Usefulness of infrastructures; Value of laboratories; Value of equipment	Mohanty et al. [70]; Davoudpour et al. [64]; García-Valderrama et al. [32]
R&D Center/customers—strategy	Strategic need; Strategy alignment; Innovation strategies and initiatives; Synergy with base business; Alignment with organization objective and capability; Top management commitment	Mohanty et al. [70]; Ordoobadi [88]; Shehabuddeen et al. [31]; Huang et al. [79]; Jain et al. [97]; Farooq and O’Brien [32]; Cho and Jaeho [74]; Cooper [102]; Im and Workman [103]; Zirger and Maidique [93]; Montoya-Weiss and Calantone [104]; Davoudpour et al. [100]; Bayazit [69]; Anand and Kodali [89]; Balachandra and Brockhoff [105]; Souder and Song [106]; Bastic [107]
R&D Center/customers—Costs	Financial dimension-capital; Cost of operation; Financial factors; Reasonableness for research cost; Cost of operation; Risk for development cost; Commercialization cost	Nouri et al. [99]; Bayazit [69]; Ordoobadi [88]; Shehabuddeen et al. [31]; Farooq and O’Brien [32]; Anand and Kodali [89]; Yurdakul [108]; Braglia et al. [109]; Huang et al. [79]
R&D Center/customers—Timing	Long implementation lead-time; Reasonableness for research period; Long implementation lead-time; Expected time to commercialization	Bayazit [69]; Ordoobadi [88]; Yurdakul [108]; Braglia et al. [109]; Huang et al. [79]; Bernoider and Koch [110]; Shen et al. [30]

.

Based on the parameters identified in Table 1, the Delphi process was performed with the panel of experts. The process reached to a consensus in two iterations, and provided the final version of the hierarchy tree, presented in Figure 1.

To complete the work of the panel of experts, they assigned weights to the factors in the three levels: requirements, criteria, indicators. This weighting was used to guide the pairwise comparison in the MIVES process, described in the next section.

4.2. Quantification of the Decision Tree

The MIVES model construction is completed by the weighting of the factors (requirements, criteria and indicators in MIVES terminology) and the characterisation of the indicators, by means of the value functions.

4.2.1. Weighting by AHP Methodology

As described in previous chapters, the MIVES methodology incorporates AHP to assign weights to the indicators, criteria and requirements that configure the hierarchy tree. The pairwise comparison has been conducted by the analysis team, using as reference the weights assigned by the panel of experts in the final stage of Delphi. The pairwise comparison matrix has been constructed for each block of factors (indicators related to the same criteria, criteria related to the same requirement), the weights vector obtained and the consistency of each matrix has been calculated. The MIVES method provides a very useful tool to evaluate the consistency index, which is very helpful to guide the analyst, in particular when the matrix is bigger than 3 × 3 [53].

4.2.2. Characterization of the Indicators

The procedure to obtain the value for each indicator was defined to complete the model. Defining value functions is also crucial to add homogeneity to different indicators, which have different measurement units. Defining each value function requires measuring preference or the degree of satisfaction produced by a certain alternative. Each measurement variable may be given in different units; therefore, it is necessary to standardize these into units of value or satisfaction, which is basically what the value function does. The method proposes a scale for which value 0 reflects minimum satisfaction and value 1 reflects maximum satisfaction. MIVES outlines a procedure consisting in the definition of the tendency (increase or decrease) of the value function; the points corresponding to values 0 and 1 (maximum and minimum satisfaction); the shape of the value functions (linear, concave, convex, S-shaped); and finally the mathematical expression of the value function.

MIVES also provides the possibility to use a valuation table to obtain the indicator valuation (0 to 1) when the indicator characterizes a discrete attribute.

Table 3. Characterization of indicators related to criteria Maturity.

Requirement	Technology
Criteria	Maturity
Indicator	Characterization
Development cycle	The TRL value of the technologies is used as reference. The value function takes values of 1 for TRL 1 and 0 for TRL9, with a decreasing convex shape.
Technological risk	Characterized by a risk matrix that collects two factors: the risk of cost overrun and the risk of non-completion of the development. These two factors are evaluated by the user in two aspects, probability and impact. The values obtained by the combination of both aspects are normalized to a scale of 0–1.

,

Table 4. Characterization of indicators related to criteria Relevance.

Requirement	Technology
Criteria	Relevance
Indicator	Characterization
Originality	Originality is characterized in terms of scientific papers relating to the technologies under evaluation. As the research is oriented towards the manufacturing sector, mature technologies such as milling, turning, and grinding were considered to set the limit values for the scaling. Thus, a value of 4000 references or more, i.e., a highly referenced topic, is considered the minimum satisfaction value, scored 0, and up to 100 references received a value of 1. Intermediate values are calculated by a linear value function
Potential of extension	Defined as the potential of the technology to enter other sectors. That is, it is intended to determine whether it is a vertical technology or a more transversal technology, applicable in a wider variety of sectors. This indicator is characterized by a convex value function, representing the number of machine tool types to which the technology is applicable. This value is obtained from the machine tool types classified by the Spanish Association of the Machine Tool Industry and related Manufacturing Technologies, AFM.
Patentability	The number of patents related to the technology is considered. Represented by a convex value function with the limit values established using mature, widely implemented manufacturing technologies as a reference.

,

Table 5. Characterization of indicators related to criteria Market.

Requirement	Technology
Criteria	Market
Indicator	Characterization
Dimension	The size of the market is measured in terms of business volume, and for each technology the structure of the market must be identified. The value function was defined as an S curve, representing the percentage of market covered by the evaluated technology.
Fragmentation	The AFM machine classification reference is used. From this document the classification of machines is accompanied by with the application markets, and market volume is presented. This value is introduced in a linear value function.
Competitors	The evaluation of this indicator took into account the number of competing industrial companies that were in the process of developing the technologies under evaluation, the size of those competing companies, their temporary tendency and intensity in the use of the technologies. Based on the evaluation of these (low, medium, high) aspects, the score 0 to 1 is obtained.

,

Table 6. Characterization of indicators related to criteria CT Internal factors.

Requirement	R&D Center
Criteria	Internal Factors
Indicator	Characterization
Professional capabilities	Two factors are evaluated: the percentage of researchers already trained in the technologies and the percentage of researchers with previous experience on the technologies. Applying the corresponding weights, the score between 0 and 1 is obtained.
Equipment	Following an analogous procedure, in this case the factors under evaluation are equipment currently available to work with the technologies and adaptable equipment.
Alignment with strategy	The value function is defined as S-shaped, boosting the high alignment cases and penalizing the less aligned technologies. Alignment with the strategy is evaluated in percentile terms at the center and the value is introduced in the function.
Access to the market	Evaluated by means of three factors: previous technology implementation experiences in the objective market, previous knowledge of market particularities, and relationships with industrial companies, and other potential customers. Each one of the factors was scored at four levels, and the values weighted to obtain the score between 0 and 1.
Costs	Cost of personnel (recruitment and training), equipment (acquisition and maintenance), and development (materials, testing and implementation) were evaluated at four levels, from low to high, and weights were assigned to each of the three ones, to obtain the score from 0 to 1.
Timing	Evaluated by performing a comparison between the overall TRL of the technology and the TRL of the technology at the R&R center. The value moves from low in case the center is a leader in this technology and highest when the TRL of the technology is higher than the position of the center. By means of a matrix Technology TRL/TRL in the center, the position on that matrix provides the value between 0 and 1.

and

Table 7. Characterization of indicators related to criteria customers internal factors.

Requirement	Customers
Criteria	Internal Factors
Indicator	Characterization
Professional capabilities	This indicator was evaluated by means of two factors: level of training and level of experience with the technologies under evaluation. The factors were weighed by the panel of experts at 40% and 60%, respectively, and were evaluated from low to high at four levels.
Equipment	In a similar procedure, two factors completed the evaluation of the indicator equipment: availability of equipment dedicated to the technologies under evaluation and availability of adaptable equipment, weighed by the panel of experts at 60% and 40%, respectively.
Alignment with strategy	The evaluation follows exactly the same procedure as for the R&D center.
Costs	Following the same procedure as for the R&D center, personnel costs, equipment cost, and, in the case of the industrial customers, implementation costs, are the factors for evaluation. The evaluation procedure is the same, with four levels from low to high and by weighting each factor.
Timing	Timing evaluation followed exactly the same argument as the evaluation of the R&D center, through a comparison between the global TRL of the technologies under evaluation and the TRL of the potential customers of the R&D center.

describe the characterization function defined for each one of the indicators.

The valuation scheme described above summarizes the procedure that will be followed by the users of the model. The model as it is developed and presented in this section can be applied by decision-making teams at an R&D Center. The aim of the research was to develop a useful tool, moderately comfortable for the user, but what is more important, providing a scenario of thinking and discussion for the team that was trying to take into account all the variables, criteria and indicators that may affect the decision on the adequacy of a technology for each R&D center.

The use of AHP and Delphi as the basis for building the model and conducting the analysis and discussion provided a solid foundation to the model, while the deep and detailed analysis of the literature gave a set of variables and criteria that, adapted to the scenario of manufacturing and machine tools, assisted the construction of the decision-making tree.

Upon that solid basis, the research team worked on the detailed definition of the criteria, and the valuation method, configuring the value functions of the MIVES methodology, through objective, quantitative sources of information when possible, and objectifiable decisions in other cases.

4.2.3. Application to Two Case Studies

The application scenario is an R&D center located in the Basque Country and specialized in machine tools and manufacturing processes. Its main customers are manufacturers of machine tools, in particular the companies that lead the sector in Spain, and are considered amongst the references in Europe, in the fields of grinding and milling. In its Strategic Plan 2009–2012, the R&D center adopted the decision to include two new technologies in its research agenda: Additive Manufacturing, more precisely Laser Cladding or Laser Metal Deposition (LMD) technology [85], and Micromachining technology, focusing on Micro-milling and Micro-turning [86].

The selection of these technologies was carried out in a process of analysis by the members of the center Management Board according to subjective criteria of opportunity for the center itself and of potential applicability and adaptation to the capabilities of its clients. Some of the favorable aspects valued for the decision were: the development and growth opportunity for the center; the complementarity of the technology for the customer companies and, therefore, the ease of integration with their capabilities; accessibility to the sectors of application; the opportunity for growth of the client companies, from the creation of new lines of products, based on machines for laser cladding and combined machines of additive and subtractive manufacturing on the one hand, and ultra-precision machines for micro-milling and micro-turning.

The case studies evaluation carried out simulates what the process of applying the model would have been and the result of both technologies’ evaluation, had this model been available at that time. To that end, in the assessment of the different criteria, the objective information available over the period in which the decisions were taken (2009–2010) were all compiled. Likewise, the case studies were completed by the Technology Management team of the center, so that subjective aspects were also subject to the criteria of the same people taking the decision. In that way, the exercise is perfectly transferable to that time and decision scenario.

It must be remarked that this evaluation carried out is not comparative for the selection of one of the two cases, the evaluation is individual and independent. The results will be presented together to facilitate the reading and further explanation comprehension, in

Table 8. Results for Technology—Maturity indicators.

Requirement	Technology
Criteria	Maturity
Indicator	Laser Cladding	Micromachining
Development cycle	0.937	0.035
Technological risk	0.562	0.375

,

Table 9. Results for Technology—Relevance indicators.

Requirement	Technology
Criteria	Relevance
Indicator	Laser Cladding	Micromachining
Originality	0.989	1.000
Potential of extension	0.452	0.356
Patentability	1.000	1.000

,

Table 10. Results for Technology—Market indicators.

Requirement	Technology
Criteria	Market
Indicator	Laser Cladding	Micromachining
Dimension	0.911	0.511
Fragmentation	0.420	0.300
Competitors	0.300	0.367

,

Table 11. Results for R&D Center—Internal Factors.

Requirement	R&D Center
Criteria	Internal Factors
Indicator	Laser Cladding	Micromachining
Professional capabilities	0.510	0.786
Equipment	0.6	0.6
Alignment with strategy	1	0.919
Access to the market	0.6	0.2
Costs	0.7	0.5
Timing	0.7	0.3

,

Table 12. Results for Customers—Internal Factors.

Requirement	Customers
Criteria	Internal Factors
Indicator	Laser Cladding	Micromachining
Professional capabilities	0.400	0.640
Equipment	0.79	0.51
Alignment with strategy	1	0.375
Costs	0.886	0.528
Timing	0.604	0.228

and

Table 13. Global Index. Technology Adoption Index.

	Laser Cladding	Micromachining
Technology Adoption Index	0.698	0.428

Laser Cladding technology has received a significantly higher score, although its value is moderately good. In the case of Micromachining, the value obtained seems too low to recommend a favorable decision for its adoption.

The main differences found between both case studies are related to market and development related indicators: market access for the center, industrialization and commercialization costs and timing are penalizing the Micromachining score, while both technologies obtain rather low values in terms of market fragmentation, presence of competitors (due to relevant big size competitors already involved in the technologies) and also the potential of extension is low scored. On the opposite side, relevance factors receive good evaluation, observing good publishing and patenting options.

Analyzing the real time perspective, it must be said that the decision adopted by the center, in 2009, on the adoption and development of Laser Cladding technology was favorable. Ten years after that decision, the center had consolidated the technology, with an established research team and reasonable results for scientific contributions, access to funding sources and industrial implementation at one of its strategic industrial customers. However, it should be said that the business volume for both the center and its customer was moderate, at least within the company technology and product portfolio, even though the decision is still yielding positive commercial and scientific results.

In the case of Micromachining, after a period of around six years of investment in R&D activities with low materialization in the market and a future perspective in which a change in trend was not visualized, the center decided to abandon the activity in Micromachining, transferring the gained experience to conventional machines, such as precision grinding machines and lathes. It must be noted that, while the valuation carried out with the MIVES model points to some of the negative aspects indicated in this analysis, such as market access, production capacity or timing by the company, the evolution over time has been more negative than that foreseeable in 2009.

4.3. Sensitivity Analysis

After the completion of the model and the application to the case study, a sensitivity analysis was performed to evaluate the consistency of the model.

The Technology Adoption Index calculated by the model is directly related to the weights assigned to the requirements, criteria and indicators. The sensitivity analysis was performed to verify whether slight changes in their corresponding relative weights could cause large changes in the final value of the evaluation [36,87]. To do so, simulations were carried out in which the weights of the criteria or requirements were varied, and the final result was calculated. It can also be done by varying the weight of the indicators, but that approach is not common [51,111], since its influence on the final result is not usually as high as a change in the weight of the criteria or requirements.

The sensitivity analysis was performed using the One at a Time Sensitivity Analysis (OAT), one of the most popular approaches to sensitivity analysis in AHP-based methodologies [112,113]. In addition, it is simple to perform, it requires low computing resources and the results are easy to interpret [114]. In the two exercises that were conducted, the weights of the requirements were varied, with individual modifications of ±30%, ±50% and ±80%. Once the weight of the corresponding requirement was altered, the rest of the weights were proportionally recalculated, so that the total sum was still 100%. The results for both cases studies are presented in

Table 14. Sensitivity analysis performed for both case studies.

Case Study 1: Laser Cladding Original Value: 0.698	REQ1 +30%	REQ1 +50%	REQ1 +80%	REQ1 −30%	REQ1 −50%	REQ1 −80%
	1.9%	3.2%	5.1%	−1.9%	−3.2%	−5.1%
	REQ2 +30%	REQ2 +50%	REQ2 +80%	REQ2 −30%	REQ2 −50%	REQ2 −80%
	−0.2%	−0.3%	−0.4%	0.2%	0.3%	0.4%
	REQ3 +30%	REQ3 +50%	REQ3 +80%	REQ3 −30%	REQ3 −50%	REQ3 −80%
	−3.5%	−5.8%	−9.3%	3.5%	5.8%	9.3%
Case Study 2: Micromachining Original Value: 0.428	REQ1 +30%	REQ1 +50%	REQ1 +80%	REQ1 −30%	REQ1 −50%	REQ1 −80%
	1.0%	1.7%	2.7%	−1.0%	−1.7%	−2.7%
	REQ2 +30%	REQ2 +50%	REQ2 +80%	REQ2 −30%	REQ2 −50%	REQ2 −80%
	−1.4%	−2.3%	−3.6%	1.4%	2.3%	3.6%
	REQ3 +30%	REQ3 +50%	REQ3 +80%	REQ3 −30%	REQ3 −50%	REQ3 −80%
	1.3%	2.2%	3.5%	−1.3%	−2.2%	−3.5%

.

When the results of the evaluation are very similar even with significantly modified weights, it can be concluded that the evaluation model is correct [113,115]. In the present study, the maximum variation of the evaluation result was 9.30%, which can be considered a small variation, taking into account that it was the consequence of an extreme variation in the weight of the dimensions, of up to 80%.

5. Discussion

The model for the evaluation and selection of technologies that is presented in this research work has been designed to assist an R&D center when taking a decision on the acquisition and the development of a technology or a technological line. This process of technology selection, one of the five that configure the Technology development process [18], is key for any entity, but especially critical in the case of R&D centers when it comes to selecting a technology area, which can condition a high volume of investment, in both material and human resources. Competitiveness and sustainability of the center itself and the industrial partners that are the final beneficiaries of the research outcomes, are highly dependent on the success in the center’s technological commitment, and this commitment is based on the technologies adopted by the center.

The selection of a technology is conditioned by many heterogeneous factors. First of all, those factors that are characteristic of the intrinsic nature of the technology [70,88], which are evident from a preliminary assessment, or a comparison between several technologies, in relation to their attractiveness, both scientific and technological, as well as their potential exploitation on the market [74].

At the second stage, it is important to analyze the fit of the technology with the nature, characteristics and position of the entity that will develop and implement it. Alignment with the strategies and support of the management [32,88], capabilities, both in material and human environments, are factors that can facilitate or condition the success of the decision in technology development or acquisition.

Focusing on the scenario of an R&D center, this research work has emphasized the relevance of including in the analysis the factors associated with the potential customers, in order to evaluate the feasibility of transferring the technology to the market.

This work represents a unique approach in this research area, with novelty contributions at different levels. On the conceptualization of the problem, the references found in the literature about technology assessment in R&D organizations are very few [13,68,77], and they are not directly focused on R&D centers, nor oriented to an strategic process for the assessment and selection of technologies. On the configuration of the model, the global vision of the R&D center scenario is also original. This research work integrates factors related to science and technology that can be found in research works related to technology management [49,116,117] and disruptive innovation [118,119]. The model also integrates factors related to the access and relation between the R&D center and its industrial customers that are also used in research works related to R&D and industrial collaboration [15,120,121,122,123]. And finally, the factors related to the capabilities of the potential industrial customers and the fitting of the technology with them are also included. These factors encompass aspects related to strategy, collaboration and capacitation that are collected under the absorptive capacity concept [124,125,126,127,128]. The literature review, from which the first wide set of parameters candidate to be included in the hierarchy tree has been built, has observed the factors identified by the authors in the above-mentioned fields, to ensure the observance of the global scenario vision.

MIVES methodology has been selected for the development of the model. This selection has been based on two main factors: The first positive value of MIVES is the balance between reliability, provided by the integration of relevant methods, Delphi and AHP, and novelty, by including the value functions for the characterization of the indicators, providing customization and objectivization. The second favorable aspect observed is related to the extension of the MIVES method to a new sector, starting from the relevance acquired by the method in Spain, in the context of the civil construction sector. The application to an R&D center and to the manufacturing sector can be understood as a novelty of this research work. It must be highlighted that the virtues of the MIVES method, already addressed in other research works [53,129] have been proven in this research work, especially the agility of application the user friendliness and the flexibility and customization provided by the value functions.

The model presented in this research has a universal character, applicable to the decision-making of an R&D center of any technological sector. The customization to the manufacturing sector was intended to give the method more precise configurations of the value functions for characterizing the indicators, their evaluation and the sources of the objective data, which have been restricted to the manufacturing sector and, more specifically, the Machine-Tool.

The main contribution of the research work on the MIVES methodology is focused on the formulation of the value functions and the collection of the reference data for the evaluation of the criteria. This section has also introduced one of the most complex areas in the search for the double objective of objectifying the analysis and facilitating the data for the evaluation of the indicators. Thus, the assessment of indicators such as originality or patentability have been resolved through easily accessible data sources such as the TRL of the technology or the number of existing patents, while other indicators such as technological risk or those related to the target market and the fragmentation have required more analysis and discussions within the research team. This is a problem frequently discussed in the MCDM methods application, solved in several research works by a combination of methods [72,130,131].

Two case studies have been applied to evaluate the behavior of the model on a real decision-making process that took place some ten years ago at a Manufacturing R&D Center in Spain. This approach was supported because the research team had full access to all the information required for the evaluation and had a timeline perspective on the evolution of the decision-making process, yielding a qualitative analysis of the adequacy of the valuation granted by the model. Finally, a sensitivity analysis was performed to verify the stability and reliability of the model behavior. The qualitative assessment of the case studies, carried out in detail, and complemented with the sensitivity analysis, has given a positive assessment of the behavior of the model, both in terms of its usability and the result of the evaluation of the technology that was finally selected.

6. Conclusions and Further Implications

The model has been configured for use in an R&D center, but it has the same applicability for an academic agent in its technology transfer strategy and for industrial companies and corporations with their own R&D department or unit, which can characterize both their R&D capabilities and their industrial and market capabilities. Evaluation in a company is even easier as the evaluators have direct access to company data, required to characterize and evaluate factors in the industrial customers branch of the model.

The applicability of the method is increased when several technologies are compared in order to select only one, as the model generates a quantitative ranking, but it also provides information on the detailed valuations of the indicators and the weights assigned to the different criteria and variables. A decision-making team that may apply the model therefore has at its disposal an analysis and reflection tool based on a quantitative base and on real and objective data.

As future lines of development on the model, the possibility of drawing more complex relationships between indicators and criteria could be evaluated, as variants in the combination of the same criteria can generate different evaluation results, in line with the ANP method that Saaty [79,132] proposed as an evolution of the AHP, developed for cases where interdependencies are observed between levels of the hierarchical tree.

The refinement of the value functions defined for the indicators and the sources of obtaining the objective data are likewise a future line of work, seeking the minimization of the subjective factors applied to some of them and facilitating the application of the method.