Applicability of Technology Maturity Level Evaluation Methodologies within Small- and Medium-Sized Organizations: Prospects and Proposals

Abstract

Most industrial companies (e.g., SMEs or start-ups) have very limited assets (time, human, and financial efforts, etc.) for the development of their R&D projects. To be competitive, these organizations must carefully evaluate the feasibility of their R&D projects and select the one that best suits their business priorities and their needs. Introducing new technologies can provide a significant competitive advantage, but it also involves risk. Moreover, in accessing public funding, the maturity level of a technology is one of the factors to be considered in terms of access to specific calls for proposals. In recent times, one of the most common measures to assess the maturity of a technology is based on the concept of “technology readiness level” (TRL), which allows the classification of the technology according to its maturity (research, development, or market readiness of the solution). But it has some drawbacks related to the SMEs’ lack of specialized resources and the subjectivity of the current assessing methods. Determining the TRL of a given technology in an objective way, based on indicators, would be of great help for the formulation of R&D projects. This paper considers whether it is possible to use existing methodologies for the determination of the TRL from indicators in organizations with limited access to bibliographic resources. Based on the state-of-the-art, modifications of the methodologies will be proposed to make them applicable to organizations with fewer resources. The outcome of this study will form the basis of a novel approach to achieve a more appropriate methodology for these organizations.

1. Introduction

At the present time, the number of technology-based SMEs and start-ups is increasing, especially in this all-around digital era. This is related to the application of digitalization as one of the key technologies to improve the manufacturing sector [1] and how Industry 4.0 technologies can positively impact several value creation factors within the organizations such as equipment, human, organization, product or process [2]. The digitalization of the manufacturing sector is a European challenge which is demonstrated by national or supra-national technology roadmaps across the European Union [2]. In addition, other countries, such as the USA or China, are also aware of Industry 4.0 technologies as a tool to increase industrial competitiveness. All stakeholders (venture capital, politicians, entrepreneurs, police enforcers, etc.) are aware of their importance for economic growth [3]. In Europe, SMEs are significant drivers for the economy as they represent a high percentage of private organizations [4]. In particular, SMEs in manufacturing industries represent 9% of the total number, 19% based on the number of people employed, and 19% considering value added for SMEs [5]

Despite the relevance of small- and medium-sized organizations for economic growth, there is a lack of SME-target tools to help them to plan and to organize their R&D activities. Furthermore, they have limited resources (both human and financial) to be assigned on their own Research and Development (R&D) or innovation projects. As a result, they often need to be supported by Research and Technology Centers and to seek additional investment (public or private). Specifically, to the best of authors’ knowledge, there is not any suitable tool which enables companies to rank R&D and innovation projects according to their available means in an easy and cost-effective manner. In this context, SMEs and entrepreneurs do not always have an easy pathway to advance from research to development in cooperation with several bureaucratic agencies, as authors highlight [6].

Moreover, the features of technology-based projects (risks, funding accessibility, etc.) are highly dependent on the level of technology maturity. Hence, low mature technology-based projects imply research projects where risks are higher, but the public funding rate is also high, while high mature technology based projects mean lower risks and lower public funding rates [7]. Therefore, the capability to establish the level of technology maturity will ease the decision-making process to rank projects in any type of organization, especially for small- and medium-sized ones, where the prioritization is crucial for their business success and optimization of resources assignment. Moreover, the complexity in management is even more relevant when complex systems, such as R&D ones, are involved. It requires a holistic approach to find a suitable solution where aspects such as creativity, flexibility or non-linear thinking are crucial [8].

Recently, some concepts related to technology research development are becoming more relevant than ever and they are being used as R&D funding program related indicators. Accordingly, Technology Readiness Level (TRL) is applied by European Commission to classify the technology under Research and Development Funding Program Horizon 2020 [9]. Subsequently, Europe is widely applying TRL to classify technology-based projects according to their development status. In the EIC Accelerator (formerly SME Instrument) and Fast Track to Innovation (FTI), at least TRL5 must be completed at the time of proposal submission. This means that the research must be (largely) completed, that a technology must function in a relevant environment, and that a functioning demonstrator must exist shortly thereafter. The European Commission wants to ensure that the bridge between research/development and market launch is shortened.

The novelty of the paper relies on the potential of concepts such as Technology Readiness Level (TRL) to determine the technology maturity in the present, past, or even to predict future technology trends. Nevertheless, these TRLs are mostly determined on a qualitative basis, considering experts’ opinions—implying potential bias—while non-subjective and quantitative methods for these calculations are not widely exploited or investigated. Moreover, SMEs can directly benefit from quantitative and objective alternatives to the ones presented in this paper, for the faster development of their business and R&D strategy while minimizing the number of allocated resources.

The main findings of this paper are related to the research of existing methodologies (both qualitative and quantitative) to determine the TRL of a technology in an industrial application. The identified quantitative methodologies are analyzed to check their applicability for target organizations, such as SMEs. For this purpose, the methodologies need to comply with several criteria such as objectivity, suitability for free data sources, etc. As a result, a novel approach is proposed to overcome current limitations and to provide added value to SMEs, including more data sources beside bibliographic ones (e.g., projects, standards, or social media trends).

2. State of the Art

2.1. The Origin of Technology Readiness Level (TRL) and Its Potential

The need for a measurement scale of technology development was clear in the case of National Aeronautics and Space Administration (NASA) since the early 1970s. In 1989, Stanley [10] presented the technology readiness concept. The paper proposed a new strategy focused on the development of new technologies not directly driven by space programs. The availability of a growing set of technologies in different development levels requires efforts in categorization. In this paper, a 7-level categorization schema is proposed. In 1991, NASA used a 9-level TRL categorization, adding the 8–9 level proposed by Mankins [11]. The authors apply the following definition: “Technology Readiness Levels (TRLs) are a systematic metric/measurement system that supports assessments of the maturity of a particular technology and the consistent comparison of maturity between different types of technology. This figure of merit is discipline-independent and can be used for more effective assessment of, and communication regarding the maturity of new technologies” [12]. In 1999, the US General Accounting Office reviewed various defense programs using TRL schema and recommend that the DoD (Department of Defense) programs start it TRL7 or above. This forced subcontractors of the DoD to use the TRL scale [13]. The ISO 16290:2013 “Space systems—definition of the Technology readiness Levels (TRLs) and their criteria of assessment” [14] defines the TRL in a similar way to Mankins, and it became standard use by European Space Administration (ESA). TRL is not only applied for the aerospace or defense sectors, but it has been extended to other macro developments related to chemical [15] or nuclear [16] industries.

In 2009, the European Union identifies Key Enabling Technologies (KETs) for their potential impact in industrial development. In the document COM 512 [17], it was recommended that project proposals should be designed to assure the link between research output and industrial impact because, depending on the maturity level of the Key Enabling Technology, integration between experimental research, innovation, and industrial exploitation is essential. A High-Level Expert Group was created to elaborate a European strategy to develop six KETs (advanced manufacturing was one of them). This expert group integrated representatives from European Member States and from relevant European industries, including small and medium enterprises and research technology organisations [18]. The gap between basic knowledge generation and the subsequent commercialization of this knowledge in marketable products is the main weakness identified and is known in broad terms as the “valley of death” issue. To cross this “valley of death”, a three-pillar bridge, related to three stages of development, is proposed: the first stage is Technological Research, the second one is the Product Demonstration, and the third one is Competitive Manufacturing. The first recommendation made is applying Technology Readiness Level (TRL) to outline the deployment steps. The TRLs can be defined by the Horizon 2020 program [19] and are shown in Figure 1. These levels have very little differences with the NASA proposal.

The three-pillar bridge approach and TRL match perfectly. The TRL1 concerns basic research, TRL2–4 describe the steps of technological research (pillar 1), and TRL5–8, product development (pillar 2). The manufacturing activities (pillar 3) start at TRL9.

One of the features of the European Horizon 2020 R&D program was the use of TRL in many of the proposals. For example, in the proposal for the Research and Innovation Actions (RIA) [20] the definition is included in the Concept and Methodology section. As an example, the topic LC-BAT-1-2019: Strongly improved, highly performant and safe all solid state batteries for electric vehicle demands the proposed project should start at TRL3 and reach TRL6 at the end of the project [21].

TRL scales allow for the defining of a common technology status terminology, which is very useful in the communication between researchers and funding, but it suffers from a lack of specific guidelines to easily establish their value.

Another relevant aspect is that the same technology evolves differently depending on the application field. This is highlighted in the literature for the case of Additive Manufacturing [22]. Likewise, Robotics have several maturity levels depending on the sector where this technology is applied [23]. Furthermore, other authors [24] have also identified the existing gap to measure innovation or innovativeness. They have proposed some new data-mining-based metrics in order to solve some current limitations, such as surrounding factors affecting the final value (e.g., industrial application).

2.2. Techonology Maturity and Its Impact on Business Strategy

Business planning needs to successfully consider technology in business strategy. It means that technology-related decisions will enable us to provide a competitive and sustainable position for the company. Technology road mapping (TRM) is a common technique the suitability of which has been widely demonstrated since its origin in 1970 [25,26] and even at the end of the 1990s. In this context, technology road mapping is a supple and recurrent tool used for strategic and future decision planning. Within the process of technology roadmap development, technology assessment is an important step where a TRL-based concept methodology can positively help. TRM is widely implemented by large companies but they find important difficulties in keeping it updated [27]. Although they have higher resources, large companies cannot easily use these types of tools so for small- and medium-sized companies it is inconceivable unless new methodologies arise to support this tedious work. Furthermore, factors that facilitate and hinder the achieving of a successful TRM are also identified in the literature [28]. One of the most important barriers to success is the lack of availability of required data or information. This is mainly related to the dependence of experts in the field so there is a need to find alternatives that are not based on specialist profiles.

The definition of a research strategy using TRM or other techniques and, specifically, the formulation of R&D projects in SMEs, requires knowledge of the state of development of the technologies involved (TRL). Due to SMEs’ resource scarcity, there are additional requirements for the TRL evaluation methodology as: use of free sources, expert non-dependent methodology, suitability for any industrial technology application, etc. This paper focuses on assessing whether the tools available to evaluate the TRL of a technology in an objective way are applicable to SMEs, and to propose the necessary adaptations for this purpose. This paper validates the applicability of quantitative and non-expert approach methodologies of TRL determination. The authors aim to identify the strengths and weaknesses of the available methodologies as a baseline for their future works and made some proposals to adapt them to SMEs’ limitations. Accordingly, once the TRL is defined, additional steps will be included regarding a methodology to rank projects more efficiently within small and medium organizations. Although the development of a new methodology is out of the scope of this paper, the sought results will enable the achieving of a tool for project management of R&D and innovation activities within small and medium organizations.

Briefly, this work begins with the state of the art of quantitative and non-expert-based methodologies to determine TRLs. The identified methodologies will be analyzed to validate their capability to rank TRL for a given technology. For each method, its ability to consistently evaluate the readiness value was checked, as was whether it can be performed using only free sources among other requirements. Taking their limitations as a basis, a proposal on how to adapt them to SMEs particularities is made.

2.3. Literature Review of Existing Tools for TRL Estimation

The existence of a cost-effective, semi-automated, non-expert, and neutral based methodology to determine the readiness level of a technology is a desirable objective for project management in small and medium organizations. The definition of the technology readiness level (TRL) applied for a technological project will enable organizations to easily quote their financial needs, resources, and related risks [7].

There are some approaches to the determination of TRL, based, in whole or in part, on the expert knowledge extracting method, such as:

• Guidelines for Technology Readiness procedure. These are based on nuclear energy [29] and include a set of templates where questions/indicators can be found to establish a technology maturity level based on TRLs. They only include rules to determine from TRL1 to TRL6 while TRL7, TRL8, and TRL9 are out of the evaluation. In addition, this method is only based on experts’ answers related to the template questions and on qualitative information.
• Strategic analysis for research and technology. Related to aerospace applications, an integrated methodology is developed named ITAM (Integrated Technology Analysis Methodology) [7]. It is based on a set of elements and parameters to quantify the state of a technology. However, this methodology only includes a new parameter as there are no data about factually establishing the current TRL.
• Calculator of technology level. This calculator gives the value of TRLs in the aerospace sector, based on NASA definitions [30]. This calculator is based on experts’ answers and now it is available for the whole TRL scale (TRL1–TRL9). Before, it was only able to calculate up to TRL6. Based on this calculator, the one found in reference [31] is the one applied in the defense sector in Turkey. A similar calculator based on an Excel tool was developed by Air Force Research Laboratory (AFRL) and is also available, but it is also dependent on experts’ answers [32].

If the search is focused on non-expert-based and complete methodologies, there are only two that are well-developed and described in scientific papers. The first one is the “Scientific method for determining Emerging Technologies”, a patented methodology proposed by Abercrombie [33], that will be referred in this paper as SmET. The second one is the “bibliometric method for assessing technological maturity” [34], that will be referred to as BINATEM. Both methodologies are analyzed in detail to determine their potential application in small or medium organizations.

Table 1. Comparison of TRL definition methodologies.

Methodologies	Quantitative	Qualitative	Main Features	Range
SmET	✓	🗴	Based on numerical indicators and procedures over science metric data	TRL1–TRL9
BINATEM	✓	🗴	Based on numerical indicators and procedures	TRL1–TRL9
Guidelines for Technology Readiness procedure	🗴	✓	Questionnaire based on experts’ answers	TRL1–TRL6
Strategic analysis for research and technology	🗴	✓	There are no data about establishing current TRL, only parameters based on differences of TRL	-
Calculator of technology level	🗴	✓	TRL1-TRL9Questionnaire based on experts’ answers	TRL1–TRL9

resumes the characteristics of the aforementioned TRL evaluation methodologies:

2.3.1. Scientific Method for Determining Emerging Technologies (SmET)

In this context, ”Scientific method for determining Emerging Technologies”, SmET, ref. [33] is a patented methodology which details a more complete quantitative and “non-expert” procedure to determine the year when a TRL is achieved. This procedure is organized into four steps (Figure 2):

Due to there being a very different dataset, to sum up, the datasets with all be normalized. The normalization of each dataset is performed by calculating a percentage (from 0 to 100); each yearly data point is divided by the maximum yearly data point count for such a database. Afterwards, the datasets are fitted by applying polynomial trend lines. The methodology indicates that an R² (values from 0 to 1) may be calculated for each of the datasets to determine the goodness of fit for the linear regression. Then, the system starts the search and determines the TRL for a given technology or application.

Four datasets are used: Scholarly data (academic articles and academic citations), Patent data (patents), Web News data (web news sources), and Business Products starts data (application specific product emergence). The data sources used to define each dataset are (as referred to in the patent):

• ISI Web of Knowledge—Science Innovation Index for Scholarly dataset.
• Thomson Innovation—Derwent World Patents Index (DWPI) for Patent dataset.
• Google News for Web News dataset.
• Google Maps for Business Products starts dataset.

Although the ISI Web of Knowledge database and Thomson Innovation database are the references applied on the referenced patent, the current references are Web of Science (service provided by Clarivate Analytics integrated within ISI Web of Knowledge) and Derwent Innovation is the new denomination for former Thomson Innovation, also provided by Clarivate Analytics.

Each TRL is calculated based on the applicable dataset(s) according to the TRL–dataset, as detailed in

Table 2. US. US9177429 procedure—TRL and dataset relationship (own elaboration from [33]).

	Scholarly Data	Patent Data	Web News	Business/ Product Starts
TRL1	✓
TRL2	✓
TRL3	✓	✓	✓
TRL4		✓
TRL5		✓
TRL6		✓
TRL7		✓	✓	✓
TRL8				✓
TRL9				✓

.

Hence, the procedure establishes the year when a TRL is achieved. Next, the criteria to define such a year along the TRL scale according to this patented methodology are:

• TRL1: If at least one paper is related to the technology (Scholarly dataset).
• TRL2: If two criteria are achieved: the first is if the number of publications is relevant and the second one is if the found papers are cited by a new paper. (Scholarly dataset).
• TRL3: If patent grants peak for the technology, irrespective of the assignee, and if first news data sources are prevalent (a relevant number of mentions). This is determined by the resulting values of three datasets (1. Scholarly dataset, 2. Patent Data, and 3. Web News data). The arithmetic average of three resulting values is calculated and this value (date) is when TRL3 is achieved. Dataset 1 is used to determine TRL2. Dataset 2 determines when patent grants achieve a crest regardless of assignee name. Dataset 3 resolves when the first technology-related news is widespread.
• TRL4: If a first patent for the technology is granted in any country (Patent dataset).
• TRL5 and/or TRL6: If the highest number of patent grants is made (Patent dataset).
• TRL7: This is determined by the results of three datasets (1. Patent data, 2. Web News data, and 3. Business/Product Starts data). The arithmetic average of three resulting values is calculated and this value (date) is when TRL7 is achieved. Dataset 1 determines when the patent grants’ crest by technology and assignee name is reached. Dataset 2 calculates when the first announcement of a product occurs, and Dataset 3 determinates when first announcement is produced in Business/Products Data.
• TRL8: If there is an announcement of a product offering or a product launch (Business/Products Starts dataset).
• TRL9: If there are multiple product offerings (Business/Products starts dataset).

These criteria are supposed to be applied to the normalized databases. However, it is not indicated how the normalized databases are applied and the fitted linear regression is applied to the assignment of the TRL. The calculation of the R² related to the fitting seems to be optional and its value does not affect the application of the linear regression, whether it is high or not.

2.3.2. Bibliometric Method for Assessing Technological Maturity (BIMATEM)

Lezama-Nicolás, R. et al. proposed a Bibliometric Method for Assessing Technological Maturity [34] (BIMATEM) to quantitatively determine a range of TRL for a given technology, now or even in the past. This methodology is completely based on bibliography sources and is structured in the following four stages:

• Technology selection: This begins with a terminology selection that needs to be as clear and concise as possible. It implies particular attention to select terms and extensive literature revision (even expert validation). Secondly, database selection is important depending on Technology Lyfe Cycle (TLC) or state of maturity (emerging, mature). A list of suitable databases is included in the paper (especially related to medical and biological sciences).
• Search query: This refers to the design of a search query, by defining the conditions that results must fulfill regarding title fields. If required, additional searches with “Keywords” or “Abstract” fields can be performed. Lastly, a revision of database guidelines and syntax is performed, by amending the search query to the selected database.
• Retrieval of results: This step means the download of search query results where special attention is paid to results such as time periods and results’ relevance. The given time periods need to be reliable, and, in general terms, they will be represented by years. Furthermore, it is essential to guarantee coherent results and to avoid duplications. Authors detail their strategy considering each applied database: Patseer^TM, commercial database for patent search; Factiva^TM, commercial database for news, etc.
• Technology maturity assessment: This step is focused on the interpretation of results by applying a mathematical model to assign a TRL. It includes the construction of a maturity assessment table, the verification that initial conditions are met, and the application of non-linear regression to each database. Data is grouped by period. The BIMATEM authors use statistical software, Minitab 18^TM. If the technology development is in TRL from 1 to 7, scientific papers and patents are the data source and logistic growth is used for curve adjustment. Non-zero data from four periods is needed to use logistic growth approximation. If the TRL are from 7 to 9, a news source is added. In this case, hype-type evolution is used and eight non-zero data periods are required. If the number of periods is not fulfilled, the respective stage should be rejected within the BIMATEM workflow. Otherwise, the application of non-linear regression to each database is applied.

Hence, logistic growth regression (1) is applied on the data from research, applied research, and development databases (scientific articles and patents), while hype-type regression (2) is applied on the data from news databases. According to BIMATEM, the applicable equations are the following.

The logistic growth behavior is represented mathematically as:

$$\mathrm{r}\left(\mathrm{t}\right)=\frac{\mathrm{k}}{1+\mathrm{a}{\mathrm{e}}^{-\mathrm{b}\left(\mathrm{t}-{\mathrm{t}}_0\right)}}$$

(1)

where k is the upper limit of the growth of r(t). The initial stage of diffusion is presented by a and b is the velocity of diffusion.

The hype-type behavior was modeled by the authors of reference [35] as the superposition of functions Q(t) and S(t’) where:

$$\mathrm{H}\left(\mathrm{t}\right)=\mathrm{Q}\left(\mathrm{t}\right)+\mathrm{S}\left({\mathrm{t}}^{\mathrm{\prime }}\right)$$

(2)

and Q(t) is the derivative of the logistic function (1).

$$\mathrm{Q}\left(\mathrm{t}\right)=\frac{\mathrm{d}\mathrm{R}\left(\mathrm{t}\right)}{\mathrm{d}\mathrm{t}}=\frac{\mathrm{a}\mathrm{b}\mathrm{k}{\mathrm{e}}^{\mathrm{b}\left(\mathrm{t}-{\mathrm{t}}_0\right)}}{\left[\mathrm{a}+{\mathrm{e}}^{\mathrm{b}\left(\mathrm{t}-{\mathrm{t}}_0\right)}\right]}$$

(3)

In addition, S(t’) is a modified logistic growth function and shows the slope of enlightenment and plateau of productivity:

$$\mathrm{S}\left({\mathrm{t}}^{\mathrm{\prime }}\right)=\mathrm{j}\mathrm{R}\left({\mathrm{t}}^{\mathrm{\prime }}\right)=\frac{\mathrm{j}\mathrm{k}}{1+\mathrm{a}{\mathrm{e}}^{-\mathrm{b}\left(\mathrm{t}-{\mathrm{t}}_0\right)}}$$

(4)

where j is the proportionality constant and t’ = t − t*. t* is the transformer of the delay to attain the so-called plateau of productivity.

A continuous function is generated that fits the points obtained, using the Levenberg–Marquardt somewhat-rhythm of nonlinear regression. This type of regression fits well for the logistic growths typical of innovation. Standardized data on scale 1 will be used, with the initial value of k set to 1 while the rest of the parameters are initialized to the value 0.5 (standard error of the regression) against the Acceptance Threshold for S value (ATS), as shown in Figure 3.

As authors have explained, the standard error of the regression (S) is used as the estimator of goodness of fit and it was selected instead of the coefficient of determination (R² more commonly used) because research shows that R² is invalid for non-linear regression models [36]. The TRL will be assigned according to the S value obtained, as detailed in Figure 4. Therefore, the different databases will be used depending on the S value obtained after data fitting.

Hence, SmET and BIMATEM propose quantitative methodologies to calculate the TRL of a given technology. Both approaches will be further analyzed with the aim of verifying their suitability for small and medium organizations.

3. Materials and Methods

This work aims to validate the applicability of existing methodologies (SmET and BIMATEM) for determining the status of a technology based on the TRL concept, in non-expert environments and with resource restrictions, such as those found in SMEs. Both methodologies are based on quantitative and “non-expert” procedures to determine the year when a TRL is attained for a given technology or technological application (SmET) or to assign the TRL of a given technology at a present or past moment (BIMATEM).

3.1. Hypotheses

This paper aims to test these methodologies to validate the following hypotheses (Hi):

Hypothesis 1.

Ranking of TRL for a given technology: the methodology can rank the technology according to the concept of TRL.

Hypothesis 2.

Unequivocal TRL allocation: the methodology provides a clear TRL for a given technology in a specific field at a specific moment.

Hypothesis 3.

Use of free data sources for the search: the methodology is based on the use of free data sources to create its databases.

Hypothesis 4.

Applicability to a wide range of applications related to industrial technologies: the methodology is valid for applications related to industrial technologies (e.g., KETs—Key Enabling Technologies).

Hypothesis 5.

Quantitative procedure not dependent of experts’ experience: there is no need to be an expert to use the methodology.

Hypothesis 6.

Suitability of the methodology by applying open data sources: the methodology enables the calculation of a TRL using open data sources, even if the original databases are not open access.

3.2. Input Data

To validate the methods, it is necessary to have an element of comparison. Focusing the search on the industrial sector, additive manufacturing as a technology that is in various states of reference was found to be a very suitable option.

The AMR Sub-Platform is an initiative of the MANUFACTURE Technology Platform groups, which is the key stakeholder of Additive Manufacturing in Europe [37]. Additive Manufacturing uses computer-aided design to build objects layer by layer [38]. In 2014, the Strategic Research Agenda of the AM Platform [39] published a report detailing the range of TRL for several Additive Manufacturing (AM) technologies: the AM of plastics is, in general, at a higher TRL scale with “no good” properties (TRL7–9), while the AM of metals (TRL3–7) or the AM of other materials (e.g., ceramics) is at a lower level (TRL1–3). Using this reference, the next validation tests are proposed:

• Test 1.1. To estimate the year when TRL1, TRL2, and TRL3 are attained for AM in ceramics.
• Test 1.2. To determine the year when TRL3, TRL4, TRL5, TRL6, and TRL7 are attained for AM in metals.

Test 1.3. To determine the year when TRL7, TRL8, and TRL9 are attained for AM in plastics with “no good” properties. Test 1.3 will not be applied for the validation (Section 4) of both methodologies, as neither of them able to calculate high TRLs while complying with defined hypotheses. In particular, the SmET methodology applies a Business/Product Starts module for this calculation and it is only suitable for determining TRLs in the present, not for in the past, such as in 2014. The BIMATEM methodology uses Factiva to evaluate TRL7 and over. In this case, there is no access to Factiva^TM or similar databases (under free access) so the final TRL cannot be evaluated.

The European-Commission-funded project AM-Motion CSA tries to reinforce the European AM ecosystem. One of their objectives was to map the AM landscape and it includes a TRL evaluation of some new AM applications in 2016 when the project began [40] as Organ Bioprinting (TRL1–3), producing larger airframe structures (TRL3–4) and ensuring the quality and consistency of powder production (TRL6). Based on the information of the TRL evolution, a new group of tests are proposed, including the following:

• Test 2.1. To determine the year when TRL1, TRL2, and TRL3 are attained for Organ bioprinting
• Test 2.2. To determine the year when TRL3 and TRL4 are attained for the production of larger airframe structures through AM technology.
• Test 2.3. To determine the year when TRL6 is attained for the quality and consistency of (AM) powder production.

Both methodologies (SmET and BIMATEM) will be applied to calculate the proposed test and compared with the expected result, as shown in

Table 3. TRL results of the tests (expected result)—input data.

Set	Test	TRL
Set 1	Test 1.1	TRL1–3 in 2014
	Test 1.2	TRL3–7 in 2014
	Test 1.3 *	TRL7–9 in 2014
Set 2	Test 2.1	TRL1–3 in 2016.
	Test 2.2	TRL3–4 in 2016.
	Test 2.2	TRL6 in 2016.

* Check previous explanation.

. Additive manufacturing is one of the Key Enabling Technologies and it is a good reference to validate the capability of both methodologies to obtain the TRL and to determine the main limitations of such a procedure and ways to improve it.

4. Validation Tests

In this section, SmET and BIMATEM procedures are applied in Additive Manufacturing technology to validate the fulfilment of defined hypotheses. SmET is used to determine the year that a given TRL is achieved in selected use cases, whereas BIMATEM provides the TRL (range of TRLs) for a given technology at a present or past moment.

4.1. Datasets

Both methodologies use data from up to four datasets. Since there are many similarities in the collection of the records, it is considered preferable, for the sake of clarity, to indicate them in a common way prior to the application of the mathematical procedures and criteria indicated in both methodologies.

SmET uses four datasets obtained from the correspondent source: the ISI Web of Knowledge database for the Scholarly dataset, Espacenet for the Patent dataset, Google News for the Web News data, and Google Maps for the Business Products starts. This methodology applies some paying databases such as the ISI Web of Knowledge database for the Scholarly dataset or the Thomson Innovation—Derwent World Patents Index (DWPI) for the Patent dataset. To check hypothesis H6, the authors employ the ISI Web of Knowledge database for the Scholarly dataset whilst the rest of the sources are free data sources: Espacenet for patents and Google News and Google Maps as the original methodology. As, in Spain, Google News is not available anymore due to copyright issues, it is applied by using Google Search and, afterwards, filtered through News.

BIMATEM applies up to four paid databases (Science Citation Index, INSPEC^TM, Patseer^TM or Factiva^TM) to collect the data and a paid statistical software (Minitab18^TM) to perform the non-linear regression fitting. In order to check the fulfillment of the defined hypotheses, the methodology will apply only a paying source (Science Citation Index) while Patseer^TM will be replaced by Espacenet [41]. Figure 4 summarizes the TRL assignment given by BIMATEM and the adjustment by using free data sources. In particular, Figure 4 represents the approach to be followed when the BIMATEM method (summarized in Figure 3) is applied using free databases. Firstly, basic research is calculated using the same database in both cases (Science Citation Index) so the procedure is similar. Secondly, the “traditional” BIMATEM methodology (Figure 3) applies an additional paying database named INSPEC (papers) and Patseer (patents) to differentiate between applied research and development, respectively. However, the proposed improvement in BIMATEM (Figure 4) cannot distinguish between applied science and development due to the limitations of free available sources (Espacenet). As it is not possible to classify between applied science and development, BIMATEM is modified to include only three split stages. Hence, Figure 4 shows that the proposed approach can exclusively indicate if the TRL is within applied research and development, or beyond.

4.2. Set #1 of Tests—Additive Manufacturing (AM Platform)

4.2.1. Test 1.1 AM on Ceramics (TRL1–3 in 2014)

The keywords “3D printing”, “Additive manufacturing”, and “ceramics” were selected for the case related to “AM on ceramics”. The searches for all data sources are presented in

Table 4. Test 1.1 Searches used in data sources.

Test	Boolean Search	Source
1.1 SmET	“3D printing OR Additive manufacturing AND ceramics” REFINEMENT TI = (3D printing AND ceramics) OR TI = (Additive manufacturing AND CERAMICS)	Science Citation Index
	(ta all “3D print * AND ceramics” OR ta all “additive manuf * AND ceramics”) AND pd ≤ “2014”	Espacenet
	3d printing OR additive manufacturing AND ceramics	Google search. Filter by News
1.1 BIMATEM	TI = (3D print * AND ceramics) OR TI = (Additive manufact * AND CERAMICS)	Science Citation Index
	(ta all “3D print * AND ceramics” OR ta all “additive manuf * AND ceramics”) AND pd ≤ “2014”	Espacenet

* is part of the query sintax; It implies to search words includiing word modfications. E.g., “manuf *” will search “manufacture”, “manufacturing”, etc.

. (They are restricted to 2014 due to the fact that this is when TRL references were established.)

The retrieved results from Science Citation Index were collected and normalized according to every methodology (

Table 5. Retrieved records of additive manufacturing on ceramics from Science Citation Index^TM (WoS)—normalized results according to BIMATEM and Abercrombie methodologies.

Year	Period	Records	BIMATEM	SmET
			Normalized Records	Normalized Records (%)
1997	0	1	0.02631579	10.00%
2005	8	3	0.07894737	30.00%
2006	9	1	0.02631579	10.00%
2008	10	3	0.07894737	30.00%
2009	11	3	0.07894737	30.00%
2010	12	1	0.02631579	10.00%
2011	13	2	0.05263158	20.00%
2012	14	7	0.18421053	70.00%
2013	15	7	0.18421053	70.00%
2014	16	10	0.26315789	100.00%

).

SmET Application

In SmET, data are fitted to a polynomial regression. The equation and its regression coefficient R², as well as the retrieved data, are shown in Figure 5 (y = −5 × 10⁻⁵x⁶ + 0.643x⁵ − 3226.9x⁴ + 9 × 10⁶x³ – 1 × 10¹⁰x² + 1 × 10¹³x – 3 × 10¹⁵ and R² = 0.8976, respectively). The authors noted that such regression is not mandatory to determine the TRL in SmET. Furthermore, the value of R² is not defined as critical to decide if the regression can be applied or not. Due to the lack of details regarding the procedure, an additional application of the normalized data cannot be carried out.

The assignment of TRL based on SmET methodology is as follows:

TRL1 search: The first result is reference [42], so, in 1997, TRL1 was achieved.

TRL2 search: The year when the number of articles became prominent, according to Abercrombie et al.’s procedure description, can be understood as being 2012 and beyond. After checking the content of the articles in 2012, this is the year when a prominent (up to 5, 10 …) number of articles on the technology were published. Then, it was checked whether the second criteria was fulfilled, meaning as at least one article [43] is already cited by other articles and authors in that year. TRL2 was achieved in 2012. The results of WoS are shown in Table 5.

TRL3 search: the TRL2 is already defined as being 2012, according to the previous explanation. Afterwards, based on Espacenet, a search of patents related to the topic is carried out based on the search query detailed in Table 4. as Test ID A1. After a cleaning phase, the retrieved results are not related to the topic. As this TRL is calculated as an average of three years and the second one (related to patents) cannot be fixed, TRL3 cannot be determined.

BIMATEM Application

The retrieved records of “Additive manufacturing on ceramics” are shown in Table 5, where results are also normalized. After this calculation (results detailed in

Table 6. Logistic growth fit of normalized result of additive manufacturing on ceramics from Science Citation Index^TM (WoS)—according to BIMATEM methodology.

BIMATEM–R Summary for Logistic Growth Fit of AM Ceramics Records from Science Citation IndexTM Method: Algorithm: Levenberg–Marquardt Nonlinear Least-Squares Algorithm Max Iteration: 50
Equation: AMceramics records from SCI = 1/(1+6.70539 × EXP(−0.11148 × ‘Year-Initial_Year’)
Summary: Iterations: 13 DFE (degrees of freedom for error): 8 S: 0.0858

), S is 0.09. Due to the S value, the search was performed in a patent database (Espacenet). In this case, the previous search query syntax was applied. According to BINATEM, the use of IPC (International Patent Code) in emerging technologies, as is the case in AM ceramics, is not recommended because the IPC code for AM (additive manufacturing) was created in 2015 and its consideration may exclude technology-related patents before this year. In this case, this is particularly important as the target year is 2014. The database (Espacenet) returned 199 patents but, after a cleaning phase, they were not related to the topic.

BIMATEM Criteria

In order to check the accuracy of the search, the search query was performed applying the same syntax, choosing the IPC BY33 related to additive manufacturing and enabling the searching of results up to the present day. It returned 12 patents related to AM ceramics that were published between 2015 and 2020—out of the target period which ends in 2014. Due to the lack of information about patents within the time frame under study, there were no data to be fitted to a logarithmic growth and S could not be calculated. Based on the previous results of S, AM Ceramics was on a TRL comprising TRL1 to TRL3 during the year 2014, according to BINATEM.

4.2.2. Test 1.2. AM on Metals (TRL3–7 in 2014)

This search was carried out based on the data sources detailed in the section above. In this case, the keywords to be applied during this search were “Additive manufacturing”, “3D printing”, and “metal”. The searches used can be seen in

Table 7. Test 1.2 Searches used in data sources.

Methodology	Boolean Search	Database
1.2 SmET	TI = (3D printing AND metal) OR TI = (Additive manufacturing AND metal)	Science Citation Index
	(ta all “3D print * AND metal” OR ta = (“Additive manufact *” prox/distance < 3 “metal”)) AND pd ≤ “2014”	Espacenet
	“3d printing OR additive manufacturing AND metal”	Google search. Filter by news
	“3d printing OR additive manufacturing AND metal AND product business”	Google maps
1.2 BIMATEM	TI = (3D printing AND metal) OR TI = (Additive manufactur * AND metal)	Science Citation Index
	(ta all “3D print * AND metal” OR ta all “additive manuf * AND metal”) AND pd ≤ “2014”“	Espacenet

.

The retrieved results from the Science Citation Index were collected and normalized according to every methodology (Table 8).

SmET Fit

As with the previous case, the data were fitted to a polynomial regression, the formula of which is y = 3 × 10⁻⁶x⁶ – 0.0388x⁵ + 194.26x⁴ − 519281x³ + 8 × 10⁸x² − 6 × 10¹¹x + 2 × 10¹⁴, and R² = 0.9863.

SmET Criteria

TRL3 search: Based on the normalized results (Figure 6), the year when TRL3 was achieved seems to be 2014. Secondly, the year when the related patent grants peak arose was also estimated. The retrieved results were analyzed to find the patent grants within the patent forms and, then, the year was achieved in 2014. The search of the Web News dataset (Figure 6) displayed 1999, 2000, 2001, and 2002 with 1 new per year; 2003, 2004, 2005, and 2006 did not really show any related new papers. In 2007, there was one related new paper [44]. In 2008, there were four [45,46]. In 2009, there were more than seven [47,48,49,50,51], so 2009 can be considered for TRL3 calculation.

As all values are equally weighted, TRL3 is (2014 + 2014 + 2009)/3 = 2012. In this case, the description “prevalent” does not help to normalize TRL definition.

TRL4 search: Based on the patent’s dataset, this was calculated as the year when a first patent grant was made (Patent dataset). In this case, Espacenet only provided the total number of patents forms so there was not an automatic method for collecting this information. After research, the year would be 2005, based on the information provided by Espacenet and the limited existing information on several patents. This is an incoherence in the application of the SmET methodology due to the fact that it does not assure TRL3 is achieved before TRL4.

TRL5 and TRL6 search: for this case, it is the year when the highest number of patent grants is made. Based on previous searches, it would be 2014, so TRL5–6 was achieved in 2014.

TRL7 search: this is calculated based on three datasets: patents, news, and business/products. The first calculation is related to the patents peak by technology and company name. Considering the previous search, this year should be 2014. Then, the module for news data, we calculated the year when a first offer of a product was released. In this case, the Boolean search is based on the equation “3d printing OR additive manufacturing AND metal AND product” and the first information that appeared was in 2009 [52]. Then, the module for Business/Product Starts was calculated using the following Boolean search: “3d printing OR additive manufacturing AND metal AND product business” in Google Maps. This did not work; thus, other keywords had to be applied: “metal additive manufacturing”. In this case, four solutions appeared for the geographical area under study (the area in which the search was performed), but it is not possible to filter by year or by other places. So, for this purpose, SmET cannot be applied directly. An option is to analyze the year when such businesses were created but a problem emerges as this cannot be applied broadly. In brief, when a Business/Product Starts module is applied, the TRL cannot be determined.

BIMATEM Fit

Retrieved records of “Additive manufacturing on metals” are shown in Table 8.

Table 8. Retrieved records of additive manufacturing on metals from Science Citation Index^TM (WoS)—normalized results according to SmET and BIMATEN methodologies.

Year	Year-Initial Year	Records	SmET	BIMATEM
			Normalized Records	Normalized Records
1997	0	1	3.57%	0.00127551
1998	1	1	3.57%	0.00127551
1999	2	0	0.00%	0
2000	3	0	0.00%	0
2001	4	2	7.14%	0.00255102
2002	5	0	0.00%	0
2003	6	0	0.00%	0
2004	7	2	7.14%	0.00255102
2005	8	0	0.00%	0
2006	9	0	0.00%	0
2007	10	0	0.00%	0
2008	11	2	7.14%	0.00255102
2009	12	3	10.71%	0.00382653
2010	13	4	14.29%	0.00510204
2011	14	6	21.43%	0.00765306
2012	15	8	28.57%	0.01020408
2013	16	13	46.43%	0.01658163
2014	17	28	100.00%

Table 8. Retrieved records of additive manufacturing on metals from Science Citation Index^TM (WoS)—normalized results according to SmET and BIMATEN methodologies.

Year	Year-Initial Year	Records	SmET	BIMATEM
			Normalized Records	Normalized Records
1997	0	1	3.57%	0.00127551
1998	1	1	3.57%	0.00127551
1999	2	0	0.00%	0
2000	3	0	0.00%	0
2001	4	2	7.14%	0.00255102
2002	5	0	0.00%	0
2003	6	0	0.00%	0
2004	7	2	7.14%	0.00255102
2005	8	0	0.00%	0
2006	9	0	0.00%	0
2007	10	0	0.00%	0
2008	11	2	7.14%	0.00255102
2009	12	3	10.71%	0.00382653
2010	13	4	14.29%	0.00510204
2011	14	6	21.43%	0.00765306
2012	15	8	28.57%	0.01020408
2013	16	13	46.43%	0.01658163
2014	17	28	100.00%

This fitting returned an S value <0.15 so the next step was performed (the results are shown in

Table 9. Logistic growth fit of normalized result of AM metals from Science Citation Index^TM (WoS)—according to BIMATEM methodology.

BIMATEM—R Summary for Logistic Growth Fit of AM Metals Records from Science Citation IndexTM Method: Algorithm: Levenberg–Marquardt Nonlinear Least-Squares Algorithm Max Iteration: 50
Equation: AMMetal records from SCI = 1/(1 + 8928000 × EXP(−0.8998 × ‘Year-Initial_Year’)
Summary: Iterations: 21 DFE (degrees of freedom for error): 50 S: 0.05196

).

In this case, the next step relies on performing a search query related to patents in the topic. It was performed using Espacenet and by applying the search query syntax detailed in Table 9.

As the returned results (2013: 3 patents and 2014: 11 patents) did not fulfill YWR >4, the S could not be calculated.

BIMATEM Criteria

As a consequence, this methodology just enables to determine that AM metal is on a TRL between TRL3 and TRL5.

4.3. Set #2 of Tests—Additive Manufacturing (AM Motion Project Roadmap)

This second set of tests was performed considering the TRL classification for Additive Manufacturing technological applications [40]. This document collects a detailed roadmap for AM technologies applied in several fields. It details the initial TRL, when the document was created in 2016, and the target TRL, once such actions are finished.

4.3.1. Test 2.1. Organ Bioprinting (TRL1–3 in 2016)

The data search process and the datasets involved were the same as for test set #1. The searches used for each case are as indicated in

Table 10. Test 2.1 Searches used in data sources.

Test ID	Methodology	Boolean Search	Database
2.1	SmET	TI = (Bioprinting AND organ)	Science Citation Index
		ta all “organ AND bioprint *” OR ta = (“print *” prox/distance < 3 “organ”)	Espacenet
2.1	BIMATEM	TI = (Bioprinting AND organ)	Science Citation Index

below.

The retrieved results from the Science Citation Index were collected and normalized according to every methodology (

Table 11. Retrieved records of Organ bioprinting from Science Citation Index^TM (WoS)—normalized results according to SmET and BIMATEM methodologies.

Year	Year-Initial Year	Records	BIMATEM	SmET
			Normalized Records	Normalized Records
2006	0	1	0.04347826	16.67%
2007	1	0	0	0.00%
2008	2	0	0	0.00%
2009	3	0	0	0.00%
2010	4	3	0.13043478	50.00%
2011	5	0	0	0.00%
2012	6	2	0.08695652	33.33%
2013	7	2	0.08695652	33.33%
2014	8	3	0.13043478	50.00%
2015	9	6	0.26086957	100.00%
2016	10	6	0.26086957	100.00%

).

The patent search in Espacenet showed one related patent in 2014 and one in 2016, as shown in

Table 12. Organic bioprinting—patent grants.

Year	Grants
2013	0
2014	1
2015	0
2016	1

.

SmET Fit

The normalized publication data were fitted to a polynomial regression (y = −0.0001x⁶ + 1.3796x⁵ – 6933.1x⁴ + 2 × 10⁷x³ − 3 × 10¹⁰x² + 2 × 10¹³x − 8 × 10¹⁵) but it was not a good fitting as R² was 0.881.

SmET Criteria

TRL1: As previously mentioned, the definition of TRL does not really need to be used to identify the year when the first article to be published on the topic was reference [50]. So, TRL1 was achieved in 2006.

TRL2: considering the results, the year when the number of publications became prominent was 2010 (three publications). In this year, at least one article [53] was cited by another author. Thus, TRL2 was attained in 2010.

TRL3: the year when this TRL was attained was calculated as the average of three years. The first year was 2010 (when TRL2 was achieved). The second year was the one when there was a patent grant peak for the technology, in this case this was 2014 (within the period until 2016, as shown in

Table 13. Logistic growth fit of normalized result of organ printing production from Science Citation Index^TM (WoS)—according to BIMATEM methodology.

BIMATEM—R Summary for Logistic Growth Fit of AM Metals Records from Science Citation IndexTM Method: Algorithm: Levenberg–Marquardt Nonlinear Least-Squares Algorithm Max Iteration: 50
Equation: Organ printing production records from SCI = 1/(1 + 12.25121 × EXP(−0.39362 × ‘Year-Initial_Year’)
Summary: Iterations: 8 DFE (degrees of freedom for error): 3 S: 0.0369

). Then, the third year when the first news data sources were prevalent was 2013 (Figure 7).

As all values are equally weighted, TRL3 is (2010 + 2014 + 2013)/3 = 2012 (2012.33). In this case, it is clear that the description “prevalent” does not help to normalize the TRL definition.

BIMATEM Criteria

This fitting returned (using the data from 2012 to 2016) an S value < 0.15 so this means that the TRL is higher than TRL1–2 and at least TRL3 is achieved (the results are shown in Table 13). However, further calculations could not be performed as available data for patents only covers 2 years so YWR > 4 was not fulfilled.

4.3.2. Test 2.2. Production of Larger Airframe Structures through AM Technology (TRL3–4 in 2016)

The searches used for each case are indicated in

Table 14. Test 2.2 Searches used in data sources.

Test ID	Methodology	Boolean Search	Database
2.2	SmET	TI = (additive manufacturing AND aerospace) OR TI = (additive manufacturing AND aircraft)	Science Citation Index
		(ti all “additive manufact * AND aerospace” OR ta all “additive manufact * AND aircraft”) AND pd ≤ “2016”	Espacenet
		(additive manufacturing AND aerospace) OR (additive manufacturing AND aircraft)	Google search. Filter by News
2.2	BIMATEM	TI = (additive manufacturing AND aerospace) OR TI = (additive manufacturing AND aircraft)	Science Citation Index
		(ti all “additive manufact * AND aerospace” OR ta all “additive manufact * AND aircraft”) AND pd ≤ “2016”	Espacenet

below.

The retrieved records were normalized as shown in

Table 15. Retrieved records of larger airframe structure AM from Science Citation Index^TM (WoS)—normalized results according to SmET and BIMATEM methodologies.

Year	Year–Initial Year	Records	BIMATEM	SmET
			Normalized Records	Normalized Records
2010	2	0	0.14285714	50.00%
2011	1	1	0.07142857	25.00%
2012	0	2	0	0.00%
2013	1	3	0.07142857	25.00%
2014	3	4	0.21428571	75.00%
2015	3	5	0.21428571	75.00%
2016	4	6	0.28571429	100.00%

.

The patent search in Espacenet retrieved the results shown in

Table 16. Retrieved records of larger airframe structure AM from Espacenet—normalized results according to BIMATEM methodology.

Year	Year–Initial Year	Number Records	BIMATEM Normalized Results
2010	0	1	0.06666667
2011	1	0	0
2012	2	0	0
2013	3	1	0.06666667
2014	4	2	0.13333333
2015	5	5	0.33333333
2016	6	6	0.4

.

SmET Fit

The normalized data are fitted to polynomial regression and the equation and coefficient are y = 0.0024x⁶ – 29.347x⁵ + 147641x⁴ – 4 × 10⁸³ + 6 × 10¹¹x² − 5 × 10¹⁴x + 2 × 10¹⁷, and R² = 1, respectively

SmET Criteria

TRL1: This was attained in the year 2010 due to references [54,55].

TRL2: 2016 is the year when related articles became prominent. In addition, during this year, at least one article was cited by another author. Then, TRL2 was attained in 2016.

TRL3: This was calculated as the average of three items. The first item was 2016 (when TRL2 was achieved). The second item was the year when there was a patent grant peak for the technology; in this case it was 2004 (this is the first year when a peak arose). Then, the third item, when first news data sources were prevalent, was 2016. As all values are equally weighted, TRL3 is (2016 + 2002 + 2016)/3 = 2014. As for previous cases, this result is inconsistent, as TRL2 cannot be achieved before TRL3.

BIMATEM Criteria

This fitting for publication records returned an S value <0.15, so the next step was performed (the results are shown in

Table 17. Logistic growth fit of normalized result of larger airframe structure AM production from Science Citation Index^TM (WoS)—according to BIMATEM methodology.

BIMATEM	R Summary for Logistic Growth Fit of Larger Airframe Structure AM Records from Science Citation Index Method: Algorithm: Levenberg–Marquardt Nonlinear Least-Squares Algorithm Max Iteration: 50
Equation	Larger airframe structure AM records from SCI = 1/(1 + 22.0366 × EXP(−0.3619 × ‘Year-Initial_Year’)
Summary	Iterations: 14 DFE (degrees of freedom for error): 5 S: 0.06678

). This means that TRL was higher than TRL1–2.

Hence, a fitting for patents records was also carried out and an S value <0.21 was found (results shown in

Table 18. Logistic growth fit of normalized result of larger airframe structure AM production from Espacenet—according to BIMATEM methodology.

BIMATEM	R Summary for Logistic Growth Fit of Larger Airframe Structure AM Records from Science Citation Index Method: Algorithm: Levenberg–Marquardt Nonlinear Least-Squares Algorithm Max Iteration: 50
Equation	larger airframe structure AM records from SCI = 1/(1 + 14.000 × EXP(−8.216 × ‘Year-Initial_Year’)
Summary	Iterations: 12 DFE (degrees of freedom for error): 5 S: 0.2

). This means that TRL was higher than TRL4–5. However, further calculations could not be performed due to the limitations of this methodology.

4.3.3. Test 2.3. Quality and Consistency of AM Powder Production (TRL5–6 in 2016)

The searches used for each case are indicated in Table 19.

SmET Fit

The retrieved results were collected and normalized, as shown in

Table 20. Retrieved records of AM powder production from Science Citation Index^TM (WoS)—normalized results according to SmET and BIMATEM methodologies.

Year	Year–Initial Year	Records	BIMATEM	SmET
			Normalized Records	Normalized Records
1996	0	1	0.0078125	2.00%
1997	1	0	0	0.00%
1998	2	0	0	0.00%
1999	3	0	0	0.00%
2000	4	0	0	0.00%
2001	5	0	0	0.00%
2002	6	1	0.0078125	2.00%
2003	7	2	0.015625	4.00%
2004	8	0	0	0.00%
2005	9	1	0.0078125	2.00%
2006	10	1	0.0078125	2.00%
2007	11	2	0.015625	4.00%
2008	12	1	0.0078125	2.00%
2009	13	4	0.03125	8.00%
2010	14	7	0.0546875	14.00%
2011	15	3	0.0234375	6.00%
2012	16	3	0.0234375	6.00%
2013	17	5	0.0390625	10.00%
2014	18	19	0.1484375	38.00%
2015	19	28	0.21875	56.00%
2016	20	50	0.390625	100.00%

and

Table 21. Retrieved records of AM powder production from Espacenet—normalized results according to BIMATEM methodology.

Year	Year-Initial Year	Number Records	BIMATEM Normalized Results
2013	0	3	0.214286
2014	1	11	0.785714

. They were fitted to the following polynomial regression y = 5 × 10⁷x⁶ − 0.0066x⁵ + 33.06x⁴ − 88331x³ + 1 × 10⁸x² − 1 × 10¹¹x + 4 × 10¹³ of which the coefficient was R² = 0.9824.

Table 19. Test 2.3 Searches used in data sources.

Test ID	Methodology	Boolean Search	Database
2.3	SmET	((ta all “AM powder” AND ta = (“AM” prox/ordered “powder”)) OR (ta all “additive manuf * powder” AND ta = (“additive manuf *” prox/ordered “powder”))) AND pd ≤ “2016”	Espacenet
		TI = (AM powder OR additive manufacturing powder)	Science Citation Index
2.3	BIMATEM	((ta all “AM powder” AND ta = (“AM” prox/ordered “powder”)) OR (ta all “additive manuf * powder” AND ta = (“additive manuf *” prox/ordered “powder”))) AND pd ≤ “2016”	Espacenet
		TI = (additive manufacturing AND aerospace) OR TI = (additive manufacturing AND aircraft)	Science Citation Index

Table 19. Test 2.3 Searches used in data sources.

Test ID	Methodology	Boolean Search	Database
2.3	SmET	((ta all “AM powder” AND ta = (“AM” prox/ordered “powder”)) OR (ta all “additive manuf * powder” AND ta = (“additive manuf *” prox/ordered “powder”))) AND pd ≤ “2016”	Espacenet
		TI = (AM powder OR additive manufacturing powder)	Science Citation Index
2.3	BIMATEM	((ta all “AM powder” AND ta = (“AM” prox/ordered “powder”)) OR (ta all “additive manuf * powder” AND ta = (“additive manuf *” prox/ordered “powder”))) AND pd ≤ “2016”	Espacenet
		TI = (additive manufacturing AND aerospace) OR TI = (additive manufacturing AND aircraft)	Science Citation Index

SmET Criteria

TRL1 was attained in 1996, related to reference [54].

TRL2: Five articles or more in 2010, and at least one in reference [55], were cited by another author.

TRL3: As with the previous case, the year when this TRL was attained was calculated as the average of three items. The first item is 2010 (when TRL2 was achieved). The second item is the one when there was a patent grant peak for the technology, and this is 2014 (after a manual examination of the retrieved patents for the period). Then, the third item is when first news data sources were prevalent, and this is 2013. Then, the year when TRL3 was attained is (2010 + 2014 + 2013)/3 = 2012·(2012.33).

TRL4 was achieved in the year when a first patent grant was made, so TRL4 was achieved in 2007 (which is incoherent as TRL4 cannot be reached before TRL3).

TRL5 and/or TRL6 was achieved in the year when the highest number of patent grants was achieved, and this corresponds to 2012. So, TRL5–6 was achieved in 2012.

BIMATEM Criteria

This fitting returned an S value <0.15, so the next step was performed (the results are shown in

Table 22. Logistic growth fit of normalized result of AM powder production from Science Citation Index^TM (WoS)—according to BIMATEM methodology.

BIMATEM	R Summary for Logistic Growth Fit of AM Metals Records from Science Citation IndexTM Method: Algorithm: Levenberg–Marquardt Nonlinear Least-Squares Algorithm Max Iteration: 50
Equation	AMPowder production records from SCI = 1/(1 + 79.9713 × EXP(−0.2267 × ‘Year-Initial_Year’)
Summary	Iterations: 21 DFE (degrees of freedom for error): 10 S: 0.05287

).

In this case, the next step relies on performing a search query related to patents in the topic. It was performed using Espacenet and applying the search query syntax detailed in Table 18. As the returned results (2013: 3 patents and 2014: 11 patents) did not fulfill YWR >4, the S could not be calculated. Consequently, this methodology just enabled us to determine that AM metal is on a TRL between TRL3 and TRL5.

5. Results

5.1. Comparison of Results

Table 23. Comparison of results: SmET vs. BINATEM.

Set	Test	Target TRL	SmET (TRL) Results	BIMATEM (TRL) Results
Set 1	Test 1.1	TRL1–3 in 2014	TRL1 in 1967; TRL2 in 2012; TRL3 could not be calculated.	TRL1–3 in 2014.
	Test 1.2	TRL3–7 in 2014	TRL3 in 2012; TRL4 in 2005; * TRL5–6 in 2014; TRL7 could not be calculated.	TRL3–5 in 2014.
	Test 1.3	TRL7–9 in 2014	No data.	No data.
Set 2	Test 2.1	TRL1–3 in 2016.	TRL1 in 2006. TRL2 in 2010. TRL3 in 2012.	TRL1–3 in 2016
	Test 2.2	TRL3–4 in 2016.	TRL1 in 2010. TRL2 in 2016. * TRL3 in 2014.	Higher than TRL4–5 in 2016.
	Test 2.3	TRL6 in 2016.	TRL1 in 1996. TRL2 in 2010. TRL3 in 2012. * TRL4 in 2007. TRL5–6 in 2012.	TRL3–TRL5 in 2016.

* Incoherent results.

summarizes the results obtained after applying both methodologies (SmET and BIMATEM) on the identified datasets and tests, considering the established hypotheses. For each test, SmET was able to calculate the TRL (or range, for the case of TRL5–6) and the year when it was achieved for five out of six tests (except test 1.3). However, the accuracy in terms of the TRL, and the year when it was attained, is limited. Although SmET is capable of determining the TRL based on fixed criteria and it corresponds to the target value (according to the bibliography references), the associated year when it is reached does not always match with such a reference. In addition, a particular weakness of this method is its inconsistency since it does not provide any means to assure that high TRLs are not attained later than low TRLs. Hence, SmET calculates each TRL based on a criterion or several criteria (e.g., the calculation of TRL3). When several criteria are considered, the year when the TRL is attained is based on the average of the years obtained considering all the criteria. This was the case in Test 1.2 when TRL4 was attained in 2005 before TRL3 (2012).

On the other hand, BIMATEM can provide an accurate range of TRL that are coherent with the target ones provided by the literature of reference. Nevertheless, it is not capable of providing a unique value of TRL instead of a range. Moreover, it is not able to differentiate high TRLs beside the range TRL6–TRL9. In general terms, BIMATEM is more accurate than SmET, as the retrieved results are like the target values provided in the bibliography.

5.2. Analysis of Applicability

The achieved results after validating both methodologies (SmET and BIMATEM) demonstrate that none of them totally comply with all the hypotheses (Hi) detailed in Section 3. In terms of methodology, a review of the achievement of the hypotheses by both methodologies is detailed above. Each methodology was reviewed for its limitations, as a way of identifying opportunities for improvement.

5.2.1. SmET

Based on the characteristics and the practical uses, the next results are extracted in relation to the hypothesis (Hi) defined in Section 3. Hypothesis H1 was based on the capability of the methodology to rank the TRL of a given technology, and SmET fulfills H1 as it enables the ranking of the year when a given technology attained a specific TRL. Additionally, its potential to allocate an unequivocal TRL was analyzed, as stated in hypothesis H2. However, SmET presented limitations in assigning a unique TRL for a given technology at a specific instant. This was mainly due to the definition of some TRLs underlying this methodology, as it provides a range of results instead of a unique value. Considering the origin of the data sources, the third hypothesis (H3) was related to the use of free data sources for searches as a priority. In this case, SmET partially accomplished this as it is based on a set of data sources where a limited number of them present free access (the ones related to Google services) while others are based on paying services (Science Citation Index and Derwent Innovation Index). The applicability of the methodology for a wide range of industrial-technologies-related applications is also studied as part of the fourth hypothesis (H4). Although SmET was developed for ICT applications, it was validated for one of the KETs defined by the European Commission (AM) in a complete set of applications. Hence, the use of this methodology can be extended to applications related to industrial technologies by accomplishing this hypothesis. Another aspect of the research was focused on checking that the target methodology was based on a quantitative approach and independent of expert procedure (H5). The study revealed that SmET is based on data sources of which the results and interpretation can be carried out by non-experts. However, the selection of keywords to optimize the search of a given technology needs to be improved as this patented procedure was performed for ICT applications and limited information exists about the way datasets are normalized. In this case, the participation of experts is required so this hypothesis is not completely fulfilled. SmET was also analyzed to validate its suitability when applying open data sources (H6) and demonstrated validity when the majority of the applied sources are free.

The application of this methodology has been conducted to identify the following limitations:

• Regarding its potential to comply with H1. Ranking of TRL for a given technology and with H2. Unequivocal TRL allocation:

Limitation 1.

TRL5 and TRL6 have the same definition to characterize the year when they are attained (non-fulfilment of H2).

Limitation 2.

The year when some TRLs are attained (e.g., TRL3 and TRL7), is established as an average of the years obtained by several datasets. The calculation of such an average is carried out considering all data as equally weighted, which may not be valid for all technologies as it depends on their maturation speed.

Limitation 3.

Some years where a specific TRL is achieved (e.g., TRL2, TRL3, and TRL9) are established as the years when an ambiguous value is achieved for several parameters. For example, TRL2 indicates that, when academic paper publications become prominent, papers are published. For the authors, the term “prominent” means the first year up to 5, 10, 15… Such values require a review regarding the type of technology, its level of maturity, its application, etc. as not all technologies achieve the same status at the same moment or with the same speed.

• For H3. Use of free data sources for the search:

Limitation 4.

Two of four used datasets were set on paid data sources (Science Citation Index and Derwent Innovation Index—non-fulfilment of H3).

• Regarding H4. Applicability to a wide range of applications related to industrial technologies:

Limitation 5.

This patented procedure was validated for ICT-based use cases where datasets were normalized, but not for other applications such as the case of industrial technologies, referring to the original patent document. However, this work has demonstrated that it can be applied for industrial technology applications.

• Considering H5. Quantitative procedure not dependent on experts’ experience and H6. Suitability of the methodology in applying open data sources:

Limitation 6.

The procedure includes content which was presented in an article by the authors [56] where additional issues to improve the procedure were foreseen. However, such improvements were not included on the patented procedure referred to as SmET.

Limitation 7.

There is no information about the procedure followed to create keywords for each specific use case, thus the search results are limited to the suitability of selected keywords and these mainly depends on the user expertise or seniority (non-fulfilment of H5).

Limitation 8.

Business/Product starts module cannot be applied at present to calculate the information for TRL7, TRL8 or TRL9. The required information to determine such TRLs must be consulted for past events (e.g., businesses created in the past, before the present moment).

• Other aspects, such as the strength of the method, are:

Limitation 9.

This methodology does not have any way to contrast the TRLs evolution (e.g., technology maturity velocity). Hence, due to the TRLs ambiguity in description, some incoherent situations may occur such as the achievement of TRL4 before TRL3, as shown in the test results.

Limitation 10.

For the cases where TRL is calculated as the average of several values (articles, patents, and news), if one of them cannot be obtained, the year when the TRL is attained cannot be determined.

Limitation 11.

Although the methodology details how the datasets are normalized and indicates that these datasets are fitted to a polynomial regression, there is not a clear definition about how it is used. Likewise, the R² associated to the polynomial regression is calculated but it is not employed as a criterion to dismiss the dataset or to determine the goodness of fit.

5.2.2. BIMATEM

Based on the characteristics and the practical use, the next results are extracted in relation to the hypothesis defined in Section 3. First, this methodology can be adapted to determine the TRL at the current moment or in the past, so H1 is accomplished. In this case, the methodology provides a range of TRLs instead of a unique TRL and does not satisfy H2. BIMATEM considers several elements from paying databases: for low TRLs (TRL1 to TRL5), it is only based on scientific papers (science); for medium TRLs (TRL6 and TRL7), the calculation relies on patents (technology); for higher TRLs (TRL8 and TRL9), news are considered (market). Hence, it does not fulfill H3. BIMATEM was originally applied for a use case related to additive manufacturing and this paper has validated the methodology for additional cases related to additive manufacturing. Hence, BIMATEM is valid for applications related to industrial technologies and H4 is fulfilled. The results and methodology can be performed by non-experts. However, the selection of keywords requires a deep analysis of the state of the art or expert profiles. In contrast, the methodology explains how to optimize the databases and expertise is not required for this. Nevertheless, H5 is not completely fulfilled. BIMATEM exclusively uses paid data sources and proprietary statistical software, but it can be adjusted to the use of free data sources and free statistical software by fulfilling H6.

A further study of BIMATEM has been conducted and the following limitations were identified:

• Considering compliance with H1. Ranking of TRL for a given technology and H2. Unequivocal TRL allocation:

Limitation 1.

The calculation of TRL is performed for ranges: science (TRL1–2, TRL3, TRL4–5), technology (TRL6–7), and market (TRL8–9) (non-fulfilment of H2).

• For H3. Use of free data sources for the search:

Limitation 2.

All applied datasets are four payment-based ones (non-fulfillment of H3).

• Considering H6. Suitability of the methodology by applying open data sources:

Limitation 3.

The determination of S (critical value to determine the TRLs) implies the definition the behavior that best fits the databases: logistic growth or hype-type evolution. This step implies the use of paid proprietary software (Minitab 18™). The approach of this article is to apply free software [57] but this requires programming knowledge so additional high-skilled users are required.

Limitation 4.

For novel technologies, in the first years, publications may appear in an irregular manner. This is the case for organic bio-printing, for which results cannot converge into a good fit due to the intermediate years where publications are 0. In addition, data cannot be disregarded as YWR > 4 must be fulfilled and this is not always the case for immature technologies.

Limitation 5.

The fitting of results to a logarithmic growth is based on Levenberg–Marquardt (L–M) algorithm using free software R. L–M algorithm is also applied by BIMATEM, using the paid source Minitab 18™. During the validation, many tests (using software R) cannot be performed because the algorithm cannot converge due to several reasons (e.g., Limitation 4) and this limits many of the tests such that they are not able to achieve a result. Hence, this methodology depends on a high number of causes that hinder the determination of a final result in an important range of situations (see also Limitation 6).

Limitation 6.

The calculation has limits for the higher TRLs as it depends on the retrieved results of a paying source (Factiva^TM) and there are not free alternatives to collect them (non-fulfillment of H6). In this sense, TRLs within TRL7–9 cannot be applied using free-data-sources-based BIMATEM.

5.2.3. Suitability for Small and Medium Organizations

None of the studied methods fully complies with all the initial hypotheses but both have positive aspects for potential application in small and medium organizations. On the one hand, SmET provides a unique TRL (a year when a unique TRL is attained) for most of the cases and half of the databases rely on free sources; BIMATEM offers a range of TRL and none of the applied databases are open access. On the other hand, BIMATEM clearly details its methodological steps (e.g., the normalization of results/databases, search query, etc.) as well as being originally validated for industrial applications. This is not the case for SmET, although it has been demonstrated suitability for industrial applications.

Still, both methodologies present limitations, hindering their direct application to target organizations (small- and medium-sized ones). Considering the TRL assignment, none can provide an unequivocal value of TRL. While SmET has problems differentiating TRL5 and TRL6, BIMATEM only provides a range of TRLs, excluding TRL3. The relevance of assigning a unique and unequivocal value of TRL is related to the capability of the methodology to enable further steps for project ranking related to other issues directly linked to the technology maturity: level of risk, access to finance, need for technology providers, etc. Moreover, the use of paid databases (both), the lack of unique TRL or use of complex (and paid) statistical software and methods to determine the range of TRLs (BIMATEM) or the inconsistencies during TRL calculation (SmET) complicate their application for this type of organization where time and financial resources are narrower than for others.

As the target organizations have limited resources, the importance of using exclusively free sources for the databases is essential and this is not complied with by SmET or BIMATEM. In addition, BIMATEM applies a paid statistical tool (MiniTab18^TM) to calculate the parameter S and this is the critical step to determine the TRL of the technology according to this methodology.

During TRL calculation, inconsistencies have been identified, in particular when applying SmET (e.g., TRL4 is attained before TRL3 due to the TRL definition and calculation indications for the method). All these issues are due to the ambiguity of the definition (average of several years, the moment when a given value of a parameter is attained (5, 10, 15…), etc.) without considering the different speeds for technology maturity depending on the final application. On the other hand, BIMATEM applies a statistical concept (S) based on the accuracy of the fitting of the retrieved data for non-linear regression. Therefore, BIMATEM has no ambiguity regarding the defined S and the thresholds which determine the TRL are based on a calculation built on emerging technologies. It is remarkable that Abercrombie refers to a linear regression fitting for the datasets and the calculation of coefficient of the regression R² to determine the goodness of fit. Despite both methodologies being oriented to assigning the TRL for a given technology, one of them (BINATEM) is totally based on statistical concepts, applying the standard error of the non-linear regression (S) as the criteria to determine the TRL. Nevertheless, SmET only applies a linear regression and the related R² of the regression is only considered as a reference to know how good the fitting is, instead of being a critical parameter to determine the TRL.

Albeit that these methodologies apply different sources depending on the type of TRL (top or bottom of the TRL scale, close to the market or research, respectively), these sources are mainly bibliometric ones. Some references, even if they are free, are not valid to calculate the TRL for past events (e.g., Google maps in SmET). Additionally, the selection of keywords to perform the search is very dependent on experts’ experience and, in the case of SmET, how this choice is made is not detailed.

In this context, the authors have drafted the pathway to define a novel methodology that overcomes such identified limitations as have been detailed in Section 6.

6. Improvement Proposals

The state of art has been studied to identify suitable methodologies for technology-based project prioritization considering the concept of TRL. The sought solutions pivoted around it but not all of them enabled us to calculate the TRL of a given technology in a quantitative manner and non-dependent on experts. SmET and BIMATEM were the two methodologies that accomplished the majority of such features. Both were further analyzed as suitable tools for project management in small and medium organizations.

Furthermore, the proposed procedure will include several modules based on free sources to create the databases. The data sources will encompass a wide range of references considering the variety of TRLs (low TRLs related to research or high TRLs more oriented to market solutions). Therefore, a tentative proposal of modules would be:

• Scholarly module: this will be focused on quantifying the R&D potential of the use case technology, based on articles and citations. In this case, the data source may be Google Scholar [58] instead of Science Citation Index™ [59], as applied in the state of the art. In brief, authors will prioritize the use of several open science repositories such as Google Scholar (worldwide reference) [60], European Open Access Repositories or Regional Open Access repositories such as RIA [61].
• Patents module: this will be based on identifying the potential patents related to the use case technology, based on the number of patents, assignee names, patent classifications, etc. The data source may be Espacenet or Google Patents instead of Derwent Innovation Index. Likewise, authors will study a range of potential open data sources such as Espacenet, WIPO [62] or USPTO [63], among others.
• Projects module: this will be oriented to identify R&D projects for the use case technology, as many funding programs already classify funded projects using the TRL range. Suitable free sources will be the data sources: EC Quick tools [64] and Cordis [65]. Many papers are also related to projects or funding agencies, and this will enable the creation of a link.
• Business module: this will be oriented to determine technology-related businesses. In this case, the idea is to combine several sources to solve the limitations of current solutions (e.g., the use of Google maps to analyze the existence of businesses on the past). Applicable data sources will rely on a combination of Google maps, sectorial, and standardization sources in the field.
• News/Social Media module: this will be oriented to quantify the related news on the topic. In this case, the applicable data sources will be: Twitter [66] and Google news (in Spain, this means a Google search using the filter of “News” as Google News does not exist anymore due to country particularities). In addition, further studies may tackle other social media sources such as TikTok [67].

The TRL will be calculated by applying a sole module or a combination. The results will enable us to determine the TRL in a quantitative manner based on impartial criteria. It will be suitable for non-experts on the topic as it is not based on a survey or answers from experts. After the calculation, the outcome will be a TRL and this information will help to rank projects accordingly. Once the technology-based project is classified according to a TRL, other aspects will be analyzed, such as the need of financial and human resources, the level of risks, and the degree of difficulty to access public or private funding. Therefore, the outcome of the proposed methodology (TRL for a given technology and application) will be the input for further project assessment to determine project risks and funding needs to prioritize such projects within the target organizations.

The proposed methodology aims to go beyond the state of the art as summarized in

Table 24. Comparison between SmET and BIMATEM procedures against proposed methodology.

	SmET	BIMATEM	Novel Approach
Approach	To determine the year when a TRL of a technology “use case” is attained.	To determine the TRL of a technology at the present. It may be adapted to be determined for a past date.	To determine the TRL of a technology “use case” in the present (main approach) or past. To determine the year when a TRL of a technology use case is attained (optional).
Quantitative method	Yes (quantitative indicators are given)	Yes (quantitative indicators are given)	Yes (quantitative indicators are given)
Need for experts	No. General procedure without need of experts. However, some steps (normalization of datasets) require experts for their definition.	No If an extensive literature revision is done, experts are not required. However, their expertise may notably reduce the required time.	No
Uniqueness on TRL characterization	No (TRL5 and TRL6 have the same definition).	No (Range of TRLs)	Yes (Addition of modules and criteria)
Missed information of several procedure steps	Yes. Need of experts for dataset normalization and selection of sources (keywords).	No. Experts may be required for terminology definition (keywords) but it is not mandatory.	No. Methodology will be defined to make it semi-automated, minimizing experts’ inputs during its application.
Lack of free data sources	2 out of 4 datasets are based on paid data sources (WoS for Scholarly dataset and DSII for Patents dataset).	All data sources and statistical tool are available under license.	All modules are based on free data sources.
Application	ICT application	Additive manufacturing technologies	Wider application with special attention to industrial technologies.

. SmET, BIMATEM, and the proposed methodology have common points, such as being quantitative methods oriented towards calculating the TRL for a given technology. Nevertheless, the proposed methodology will provide an unequivocal TRL for a given use case, while this is not the case for SmET or BIMATEM. It will rely on a detailed systematic search, where additional modules related to social media, projects, and standardization initiatives will be included in order to improve the definition of intermediate and high TRLs. Social media inputs have been demonstrated to be a relevant solution as a result of a review of existing methodologies [68,69,70]. Likewise, the patent data search can be improved as detailed in existing technologies [71]. In addition, the novel method will exclusively use free data sources, going beyond the state of the art. On the other hand, all the methodologies can be almost totally performed without experts, but for some crucial steps (e.g., the selection of keywords), expert involvement would be required (SmET) or recommended (BIMATEM). However, the proposed methodology will include a refinement of keywords selection by using “text mining” techniques, in particular for technology as terms evolve with time, even for the same concept (e.g., 3D printing, free forming or additive manufacturing). Likewise, the proposed methodology will provide normalized databases and criteria to calculate the TRL to avoid the participation of experts. In addition, the proposed methodology will be focused on the needs of target users (e.g., small and medium organizations), meaning the use of free databases and tools, ease of use, non-dependence on experts, etc.

Lastly, this proposal also has the potential to be extended to other types of organizations (beside small and medium organizations) that do not have their own R&D intensive infrastructure or any capability to have it. This is the case in Public Administration, Universities, hospitals, or non-profit organizations, among others.

7. Conclusions

This work includes the study, validation, and comparison of two technology assessment methodologies based on quantitative criteria: SmET and BIMATEM. Both can determine the TRL of a technology “use case”—meaning a technology applied for a specific application—for a given moment (present or past) based on quantitative indicators (number of articles, patents, news…) and, in general terms, independent of expert input. Neither of them can provide a unique TRL (SmET has the same definition for TRL5 and TRL6, while BIMATEM provides a range of TRLs) or use exclusively free data sources (SmET uses two free ones while BIMATEM applies none). In relation to their application, SmET has been defined for ICT issues but, after testing, it has been demonstrated that it is valid for industrial applications related to additive manufacturing. Moreover, BIMATEM has been developed for specific additive manufacturing technologies and it has been tested for broader applications related to additive manufacturing, demonstrating its validity.

Based on the defined hypotheses, neither of the methodologies fulfills all of them. Hence, SmET can rank a technology according to the concept of TRL, relies on free data sources (50% of applied sources), and is suitable for industrial technology applications and for applying free databases. However, it is not able to allocate an unequivocal TRL and some steps (e.g., definition of keywords) are dependent on experts. Regarding data analysis, there is not a clear link between the definition of each TRL and the performed data analysis (normalization of data, linear regression fitting, etc.). BIMATEM only satisfies the hypotheses related to ranking a given technology according to TRL, and in its validity for industrial technologies. Despite its limited independence from expert experience, the normalization of data, non-linear regression fitting, criteria of TRL based on calculated S, etc., is fully objective and based on quantitative parameters.

Regardless of their limitations, both methodologies have advantages that can be used as a baseline for further developments:

• SmET is based on a high number of free data sources and is suitable for this type of source. In addition, it is valid for industrial technology applications, even though it was developed for ICT-related issues.
• BIMATEM has been demonstrated to be a strong approach in determining TRL based on quantitative indicators and including a deep description of data processing and analysis. In addition, it was originally developed for manufacturing technologies, so it is a good basis to be extended for other industrial applications.

Hence, this work presents a framework where the concept of TRL is the pillar around which the proposed methodology pivots. In this sense, TRL will determine the maturity of a technology applied in a given case and it will be based on factual and quantitative criteria, overcoming the constraints found in the state of art solutions as result of the previous analysis.

Therefore, this novel approach will enable us to simplify the project management of small- and medium-sized organizations. This work proposes a tool to help them during their technological project selection while planning their internal R&D and innovation activities, as well as during the definition of their medium- and long-term R&D and innovation strategies and programs. This instrument will benefit such entities by reducing the time related to decision-making or by minimizing the associated risks. Once the novel approach classifies projects according to their TRL, the organization can make strategic decisions based on this information. Hence, the tool balances the uncertainty associated with high-risk projects (R&D) against the high level of consumed efforts or the access to public funding for research and development activities—public funding determines low TRLs as R&D activities and higher rates of funding (grants) can be attained. In contrast, innovation activities feature low risk—according to public funding mechanisms—and complexity in reaching public funding on the topic. In this context, organizations can rank projects according to their availability of resources, priorities, and project needs resulting from methodology application.

In addition, the proposed solution will be developed to answer the needs of small and medium organizations considering their limited resources (funding, personnel, time, etc.) and the impossibility of having high-level skilled staff experienced in key competences (target technologies, programming, or statistical analysis, among others).

To sum up, the authors suggest a novel approach to assess the maturity of a given technology for small and medium organizations considering their particularities:

• Lack of access to experts: the proposed solution will not depend on experts’ feedback, so it will be defined based on quantitative and objective indicators. This will also be made possible by including a wider range of databases and types of data sources (e.g., social media).
• Limited number of resources: the target organizations (small and medium size) cannot dedicate many resources to calculate the technology maturity for selected use cases. In this sense, the proposed methodology will be based on free data sources and a semi-automated approach to answer their needs.
• Resilience to a changing environment: this TRL-based approach will enable organizations to rapidly adapt and redirect their strategy and decision plan (low TRLs for opening new R&D lines and high TRLs for boosting technology-based new business lines). Hence, it will enable them to face sudden crises or events that may arise (e.g., pandemics) based on the valuable knowledge provided by this tool.