Technology Readiness Levels (TRLs) in the Era of Co-Creation

Abstract

Technology readiness levels (TRLs) is a well-established and widely used approach for defining the readiness of new technology. It assesses technology maturity against specific benchmarks, ranging from level 1 (concept) to level 9 (market solution). Although this is a useful classification service that allows us to establish a common language, there are cases where we find that this conceptual approach cannot adequately highlight the maturity of certain innovative endeavors and effectively steer their development to higher TRLs. We will present an empirical case where the TRL approach presented a critical shortcoming in highlighting the true and effective readiness of a specific technological development and could not suggest the next natural step in ascending the maturity ladder. We will seek to generalize for the case of co-creation at large, analyze why co-creation may be poorly serviced by the current TRL model, and suggest an amendment that would allow the observed shortcomings of the traditional TRL approach to be overcome and its use extended into such co-creative settings, thus allowing stakeholders to enhance the effectiveness and impact of their collaborative innovation efforts.

1. Introduction

Technology readiness levels (TRLs) is a methodological tool that emerged to assess the maturity of new technology. The approach was first developed and standardized by NASA [1]; the initial intention was to provide a disciplined way to differentiate between technology readiness levels. The original definition of TRL was a scale from level 1, corresponding to the formulation of a new concept, up to level 7, which ascertained that the system had been successfully demonstrated in a real environment [1]. Although initially used within NASA and aimed at securing a common language between research and operational personnel, the TRL approach has now extended well beyond NASA and the aerospace industry and has been widely adopted across research and industry. In the process, it has also expanded to nine levels [2]; indeed, this nine-level scale corresponds to the formulation currently in use worldwide. Numerous entities beyond NASA are using the TRL system, and Mankins [2] provides evidence of a global adoption, even though some users may opt for some slight modifications or more extensive documentation on the meaning of every single readiness level. One example is the European Space Agency’s Handbook [3], which defines each level by numerous standards that a technology must meet to be categorized at the particular level.

Interestingly, the TRL approach is widely used as a mechanism to classify innovation. Innovation is strongly related to the development and implementation of new technologies, and TRL can be a useful tool for assessing the progress and potential of these technologies. For example, investors may use TRL to evaluate the potential risks and rewards of investing in a particular technology, while policymakers may use TRL to assess the feasibility of implementing a new technology in a particular sector.

Thus, in Horizon Europe, the EU’s program for research and innovation, proposers and contractors are typically required to position their research and development plan against this ladder and explain what both the starting TRL and target TRL are. Figure 1, below, illustrates the nine levels of the EC’s TRL ladder and provides a short description of each level.

The nine-level model was also implemented by the US Department of Energy (DoE) with small variations in some levels [4]. In other cases, the TRL was modified to assess the readiness not only of technology but of the process of incremental innovation [5], and the innovation readiness level was defined.

Two more levels, level 10 and level 11, have also been recommended in the literature [6,7], and this is something which demonstrates the constantly evolving nature of technology maturity classification towards an always greater resolution. However, in the following discussion, we will restrict ourselves to the nine levels shown in Figure 1, as the authors consider levels 10 and 11, in which technology is continuously improved or applied to new contexts, to be beyond TRL 9. Levels 10 and 11 have not been standardized or widely recognized in the TRL framework. Therefore, limiting a conversation on TRL to the first nine levels is practical because it covers the entire spectrum of technological development and represents a widely accepted framework in the scientific and engineering communities.

While TRLs offer valuable guidance, they have certain limitations that may affect their ability to fully reflect the readiness of innovations and suggest the next natural steps for ascending the maturity ladder. In the discussion below, we will present an empirical case which provides evidence that the TRL approach cannot always provide guidance as regards the next step up the innovation ladder. We will discuss why this has been the case and what the explicit limitations and eventual workarounds have been found to be. We will also attempt to generalize and delineate the particular features that render the TRL approach insufficient in specific settings. Broad guidelines for extending the TRL framework to address the observed limitations will also be discussed.

In conclusion, this work will offer a different approach to the current TRL model, which, even though it provides a valuable foundation for assessing technology readiness, may need to be adapted and supplemented to fully capture the complexity of innovation developments and guide effective decision making. By addressing the limitations of the TRL model and adopting complementary approaches, the authors aim to present to future stakeholders an alternative way to better evaluate the readiness of innovations and navigate the path to maturity.

We structure the remainder of our paper as follows: initially, we review the relevant literature on innovation, with a special emphasis on the collaborative innovation models that, we will argue, are poorly served by the current TRL assessment scheme. Then, we describe an empirical case of a specific scientific development that was of a low TRL and for which we could not elaborate any strategy to push it up the TRL ladder. A discussion follows on why this was the case and what the explicit limitations encountered were, and we introduce the hidden opportunity that was unleashed when we brought this development into a co-creative environment as a distinct value-adding component. We generalize and delineate the conditions that need to be in place in order for such a strategy to be promising. Most importantly, we suggest broad guidelines that render the current TRL approach insufficient in specific settings and illustrate some adaptations to the TRL scheme in order to allow for the modeling of co-creative pathways. We end with a discussion and highlights of planned future work in innovation assessment.

2. Related Work

The current operational and technological context of many manufacturing companies often obliges them to develop innovative solutions, leading to a complex asset lifecycle composed of different phases, starting from conception and planning before progressing through design to engineering, construction, validation, verification, and commercialization. Roman F. Bastidas Santacruz et al. [4] developed and proposed a TRL template that aimed to support the lifecycle management of manufacturers’ assets, fostering the exploitation of related data and knowledge from the development phase of the technologies employed onwards. The proposed TRL model was structured at nine levels, classified into three main phases, and was considered an improvement on the EC and NASA TRL standards. Nonetheless, some weaknesses were identified during the verification phase. Improvements were recommended, intended to make the assessment and verification of the assets’ TRL easier.

In 2022, during the Sustainable Places 2022 conference, eleven European projects joined forces on a clustering workshop entitled ‘Low-TRL Renewable Energy Technologies’ [8]. These EU Horizon 2020 research and innovation programs addressed high-risk technology developments for breakthrough renewable energy and fuel technologies, and their activities focused on providing knowledge and scientific proof of the feasibility of the proposed concepts. They were presented and brought together, trying to co-address similar challenges to bring their technologies to TRL 9. In order to do so, a validation of their TRL was a prerequisite and their rank was TRL 3–4. The recommendations that followed proposed a roadmap to move the technologies further up the TRL ladder for market uptake and full integration into the energy system.

The accelerating use of artificial intelligence (AI) and machine learning (ML) technologies led Alexander Lavin et al. [9], to research the definition of a principled TRL process to ensure robust, reliable, and responsible systems. Their proposed Machine Learning TRL framework highlights that the existence of data-related requirements along every step would ensure that the development process considers data readiness and availability, which will not only help define ML-specific testing considerations (levels 5 and 7) but also help highlight ML-specific failure modes early. Its synergy with the existing industry standard software engineering practices would allow it to handle unique challenges. Their research concludes with the assumption that the proposed framework’s distinct advantage is the nomenclature: an agreed-upon grading scheme for the maturity of an AI technology and a framework for how/when that technology fits within a product or system, enabling everyone to communicate effectively and transparently [9]. Thus, for AI technology, TR models differ from other cases [10,11], as AI is a multidimensional construct, hitherto not widely explored in this context.

It should be highlighted that, no matter the sector, successful innovation involves more than a great idea [12], as collaboration with others makes things happen [13]; even though new ideas may often be born into a reactionary environment, collaboration can overcome inertia [14] and unlock the chains of convergent thinking, resulting in something innovative. On the other hand, co-creation means including ‘trans-disciplinary actors’ [15] and other key stakeholders, especially people who would be affected by a specific decision in a decision-making process. In order to do so, co-creation is a technique that is based on iterations during a creation process [16], which takes collaboration to the next level. When co-creating, stakeholders dive in deep [17,18] they bring each person’s unique perspective, skill sets, and experiences to produce the best possible solution with the highest impact [19]. However, a collaborative partnership alone is not enough, especially when creating new products or strategies, as new technologies encounter many implementation challenges that can hinder or halt their introduction [20]. As co-creation produces something that did not exist before [21], this implies that the result of a co-creation process is to have created something or to have brought something into existence [22].

An interesting background to this treatise is the classification work on modern innovation presented in the literature by Carliss Baldwin et al. [23]. These researchers start with a description of the three key types of innovation: single user innovation, where an individual or a firm contemplates investment in an innovative design; producer innovation, where producers undertake larger designs aiming at many end users; and open collaborative innovation, involving users and various third parties that collectively generate and share a design. The authors rigorously investigate the particular conditions affecting the costs incurred and the value generated to conclude on what effectively makes a particular setup helpful and competitive. They also discuss hybrid types of innovation, where more than one of the above three fundamental types coexist. We will argue below that co-creation is often a setup that cannot be well described by the current TRL scheme. According to the classification presented in this work, co-creation, in fact, is a hybrid innovation model, including aspects of the open collaborative innovation model and the producer innovation model. Apparently, the current TRL approach mostly reflects the producer innovation model that has been the dominant approach in the 20th century. Indeed, Schumpeter [24] positions producers at the center of the innovation stage, stating that ‘it is the producer who initiates economic change’. Teece [25,26] and Romer [27] echoed Schumpeter’s views by suggesting that the vast majority of innovative design results in private profit-maximizing firms. The dramatic reduction of communication costs in recent times essentially enabled the open-source innovation model to emerge and occupy a measurable and increasingly important role in the innovation arena. To this extent, perhaps the current TRL model, by being based on the key hypotheses and practices of the 20th century, may fail to fully and adequately model the maturity evolution of new-era innovations that are of the open-source model or are hybrids, sharing an important part thereof.

The details of open collaborative innovation have also received increasing attention in the literature. Ghosh [28], Raymond [29], and Baldwin [30,31] showed that the key condition for collaborative innovation to be a rational and potentially preferable option when compared to the produced innovation paradigm is that the communication costs incurred by each participant joining the collaboration are well offset by the value of the design contributed by the other parties. This condition essentially allows design modularity and propels open collaboration; it is a condition that is met more and more in current times, when the cost of communication is rapidly reducing. This comes in stark contrast to the centralized innovation model of the 20th century, where the firm itself was by far the most suitable entity to innovate in mass-produced products and processes, something that is also well established in the literature [32,33]. The high cost of communication in those days did not allow for the potential of open collaboration to be unleashed. Currently, the facility to locate and use several communication channels allows even the public sector to approach co-creation.

Co-creation has also captured the attention of the public sector, as the new public management theory relies on the concept of collaborative interaction, networks, and partnerships with the private sector and industry. In the literature review, Nuno Baptista et al. [34] classified co-creation benefits in the public sector as innovation-related and improving decision making. According to the authors, the present context of austerity and reduced public resources has led policymakers and politicians to consider the benefits of co-creation, co-production, and innovation. Within this context, for innovations to produce the outcomes that matter, it is important that the key stakeholders are involved in the design [34] and, hence, the TRL is adequate. According to Henry William Chesbrough, the open innovation model enables the advancement of valuable technologies both within and outside the current business models, promoting growth through investments [35]. Thus, once again, the question emerges of whether the current TRL model can highlight the true and effective readiness of these innovation developments and suggest the next natural step in ascending the maturity ladder, as there are studies that suggest that TRLs have shortcomings in manufacturing environments [36] because of their imprecise interpretation and complex success factors [37] and that they fall short in assessing integration readiness and compatibility with human capabilities and limitations [38].

It is also important to distinguish between open collaboration and co-creation. Although a co-creative setup may also be an open, collaborative one, this is not always the case. Co-creation may also apply in completely restricted, multistakeholder setups, as it may also exist in a closed, collaborative environment, where the typical producer innovation coexists with open collaborative aspects. This scheme is the one that underpinned our empirical case and to which we will turn in the next section.

A radically different approach to innovation and its assessment was introduced by Harvard professor Clayton Christensen and his co-authors in their seminal book The Prosperity Paradox [39]. In this work, Christensen et al. provide a compelling historical description of numerous incidents of so-called market-generating innovations and how they have benefited not only their initiators but, far more importantly, the public domain, having pulled in many subsequent investments for social infrastructure.

The approach suggested by Christensen et al. is radically different because it pays little attention to the ‘newness’ of a development, a trait we typically consider as defining innovation. Instead, they take a value-driven view, whereby they assess innovation not in terms of newness but in terms of its market-generation potential. In such an approach, the traditional TRL model is seriously challenged, as its very foundations, rooted in the newness concept, are questioned. In his research and in this book, in particular, Christensen recounts numerous cases stripped of any ‘newness’ that, however, excelled in terms of market generation, especially in emerging economies. To be fair, one has to credit Schumpeter as the first to clearly trace the line between ‘invention’ fostering newness and ‘innovation’ fostering value. Here is how he puts it in his landmark Fundamentals of Economic Development (1949): ‘Although entrepreneurs of course may be inventors just as they may be capitalists, they are inventors not by nature of their function but by coincidence and vice versa’. Further, the innovation which it is the function of entrepreneurs to carry out need not necessarily be inventions at all. It is, therefore, not advisable—and it may be downright misleading—to stress the element of invention as much as many writers do. Thus, Schumpeter clearly draws the line between the two terms. Innovation is, for the first time, clearly not a synonym for invention.

3. The Case of an AI-Based Energy Forecasting Technology

TRUST-AI (www.trustai.eu, accessed on 14 April 2024) is a Horizon-funded research project running in the period 2020–2025 and aiming to develop a framework of so-called explainable AI approaches and their validation in diverse use cases. One such case was in the energy domain [40,41], where our specific responsibility was to customize an explainable energy forecasting approach. Though a variety of AI forecasting models have been developed over the last twenty years, our forecasting approach would differ in that it could provide user explanations of the forecast. Explainability refers both to model level explainability achieved via symbolic expressions and instance explainability, which, in our case, was centered on the concepts of feature importance and counterfactuals. This explainability would create confidence on the side of users, and this confidence is something often considered essential in the literature; an example is the case of demand response applications, which rely on forecasts and where user perception may impede uptake of flexible pricing and participation in such demand response schemes. We will not provide any further technical detail here, as our purpose in this paper is not the AI energy forecasting itself but the classification of its particular development along the TRL scale.

At the project’s outset, we started from a TRL 2, having a clear formulation of the technology concept that we would seek to develop and validate in real settings. Our validation was done in a real building—one that could be considered to be between ‘relevant’ (TRL 6) and ‘operational’ (TRL 7). Depending on the application context, it may sometimes be difficult to draw a clear line between these two levels. Our interpretation was that a truly operational environment would present more data collection uncertainty than the one in which we tried our algorithm, in which data issues were already streamlined, and which presented no real difficulties. Thus, an operational environment might present some further data challenges, which we were, in this case, not exposed to.

However, the question that soon arose was not so much whether the achieved TRL was 6 or 7. Instead, the questions pertinent to our discussion here were: how one can climb further up the ladder, reaching TRL 8 and 9. What can TRL 8 mean in our case? How can you qualify a forecasting algorithm? Even more, how can you possibly generate a market model around a fully-developed system when the system is a forecasting algorithm?

The answer is, simply, that you cannot. It is highly unlikely that one will find in the market such a technology, per se. Such a development may indeed generate some new value but will never amount to a self-standing market value.

We would also argue that this is hardly a rare case. On the contrary, it is quite typical that a new technology presents some innovative features but falls short of enabling, by itself, any viable business case. At that point, one may, of course, consider traditional licensing instruments and arrangements. However, if one wishes to remain on the maturity upgrading path, this is not a viable option. Licensing may provide some financial benefit, but it will take you off the TRL ladder for good. If you wish to push your way upwards on the TRL ladder, licensing is obviously not a solution. Indeed, TRLs have been designed for endeavors strategically looking forward to reaching the highest level. Licensing may be a way out of this aspiration, but what happens if it is not possible or desirable?

We would argue that co-creation is, then, the only option left for our technology endeavor. Through co-creation, we effectively acknowledge that our algorithm is not a carrier of standalone market value and that a multistakeholder environment is due, one where our specific development would be an integral part of a broader solution, of which it might increase the market value.

Figure 2, below, illustrates the exact positioning of the demand forecasting (dark gray rectangles) in such a multistakeholder environment. The term multistakeholder denotes that many third parties can integrate applications (third-party app library) and extract data (third-party data). The demand forecasting in question is just one case of such a third-party app.

Regarding the innovation classification discussed in the review, this environment is hybrid; it includes open-source functionality (open collaborative innovation paradigm), but there are also parts of the functionality that are priced (producer innovation paradigm).

In the following, we will not be interested in the technical details of the above architecture, and we will restrict ourselves solely to how this co-creative setup evolved and especially how this impacts the TRL classification.

3.1. TRL in Co-Creation

New technologies that reach TRL 6 or 7, as in our particular case, but which have no possibility to move further up the ladder, will need to branch off to an alternative, co-creative path. The following Figure 3 shows what we have practically found this path to include. Indeed, four more steps have been found necessary to accommodate the new value setup. Incidentally, the 4S method (State, Structure, Solve, and Sell) used in project management comes close to effectively describing these missing steps. Essentially, the fourth step is just a reiteration of TRL 7 (technology demonstrated in operational environment) which is now rephrased in TRL 7C4 (co-creative solution demonstrated in operational environment). This new structure is illustrated in Figure 3 below.

We will now describe how we were guided empirically to define these additional steps and the specific responses we provide in our endeavor.

3.1.1. 7C1—Concept Development for a Co-Creative Setup

After realizing that there was no way for the particular forecasting development to independently move up from TRL 6/7, we embarked on the formulation of a co-creative concept. This is shown as the first step of the branch off, and called 7C1: concept development for a co-creative setup. The challenge for this 7C1 step would be threefold: first, to define a concept that could reach beyond TRL 7; second, to make sure that our particular development (explainable forecasting algorithm) was a value-adding part of this concept, as this would be imperative to achieve our main goal to push this item up the TRL ladder; and, third, to make sure that the new development could add value to the other parts of the co-creative solution. Indeed, as highlighted by Baldwin [23], in order to be viable, any collaborative solution needs to generate tangible value to all parties, exceeding the effort consumed, one by one.

To address these requirements, we initially considered bringing together the forecasting technology with a building management system (BMS) technology that we deliver in house and that has been market-available for some time. Figure 2 illustrates this co-creative venture. The light gray is the pre-existing functionality, and the dark gray is the newly embedded one.

However, linking a BMS with novel forecasting did not seem to result in tangible added market value. In particular, the third condition set above was not fulfilled, as the BMS did not appear to acquire any tangible market value from the demand forecasting. Although some value would result for the BMS, it was rather unlikely that this value could be monetized.

Thus, this approach alone was not sufficient; although linking with the BMS was potentially interesting, it did not secure a co-creative solution that would be beneficial for all parties. At that point, we tried to sketch what this new value could be. Following several internal brainstorming sessions, we came to the conclusion that a promising direction could be that of a demand response solution. Demand response technology, in a narrow sense, assists users in planning their energy uses so that they may benefit from low prices. The more dynamic the pricing tariff, the bigger the potential benefit. Demand response is also used in a broader sense to denote any change in demand patterns that may be informed and driven not necessarily by price but by other aspects (e.g., behavior-related). An example could be a thermostat change to reduce energy costs without compromising thermal comfort. Even more, we were now confident that, as demand response relies on forecasts to issue advice and recommendations to the building users, our forecasting technology was a vital element of the overall solution and would add new value that could be monetized.

Most importantly, this approach surpassed the limitation identified in the early phases; the demand response was potentially a tangible value–adding layer to the BMS, and, therefore, the BMS gained in value by feeding its data to the demand response controller. All three parties appeared to benefit from this arrangement.

Thus, the 7C1 phase concluded by having defined a market-oriented concept, one that would bring together three value-adding components, as illustrated in Figure 4 below:

It is interesting to note at this point that the multistakeholder environment was not there in the first place. It only emerged empirically in 7C1 as a potential direction for pushing the forecasting application up the TRL ladder.

Integrating demand forecasting in the co-creative framework allowed us to consider more broadly the integration of third-party apps in the same framework, as illustrated by the notation ‘1a’ in Figure 2. Indeed, it is only half of the truth that co-creation leveraged our demand-forecasting development and allowed it to proceed up the TRL ladder. More than this, co-creation was essentially conceived as a powerful concept that could, in addition to leveraging the forecasting app, also create additional benefits by increasing the potential of any third party to link to the platform, allowing it to emerge in a collectively managed environment, something we are currently investigating how best to manage.

Challenges for 7C1: for this setup, two challenges are highlighted: first, to provide for a market opportunity for the weaker (in the TRL perspective) component (item(1)). And, second, to make sure that the scheme is also beneficial to all other partners (two more in this case). If the first challenge was not met, the scheme would be meaningless, while, if the second was not met, it would not be sustainable.

3.1.2. 7C2—The Definition of the Co-Creative Actors—STRUCTURE How the Value Is Delivered

After the co-creative concept definition, we entered 7C2, i.e., the definition of the co-creative actors that could provide their part while also subscribing to the co-creative value. Although this is presented as a second step, in practice, it was performed rather iteratively together with 7C1; various actor setups were considered, and these helped formulate the value itself. Depending on the case, step 7C2 can run iteratively or sequentially with step 7C1.

In principle, there may be several approaches to investigate, depending on the links, alliances, and partners that may be available. In our case, we did not have to reach far outside the business group, as item (1) was already available within the company and item (3) was in development by a close business partner, who had no difficulty in understanding the merit of the specific co-creative value setup proposed. We considered value adding the engagement of a consumer association for validation, although it has until now proven difficult to conceptualize a clear cooperation framework.

Challenges for 7C2: overall, 7C2 is a risk management exercise; the more you have to reach out of your human network, the greater the risk and the communication costs. The only means of mitigation is a clearly formulated concept in 7C1, one that makes clear what the contribution and the expectation from the co-creative setup can be.

3.1.3. 7C3—The Prototyping of the Co-Creative Solution—SOLVE the Details of the Solution

This is a currently ongoing phase, aimed at the prototyping of the co-creative environment, as shown in Figure 2. Methodologically, there is nothing new in this step, as it is more or less already provided for in the classical TRL formulation. It includes a mix of activities typically characterized as TRL 4–6, with a main view on the integration and interoperability of the three items, each of which had already independently climbed up to 7 or beyond.

Challenges for 7C3: interoperability and seamless integration of the three items.

3.2. The Case of an Early Inception of Co-Creation

Of course, co-creation may be deliberately established as such from the very beginning. Should this be the case, moving up from TRL 6/7 should not be impossible; having a market goal from the very beginning makes moving up the TRLs realistic and potentially workable. In this case, the same concepts discussed above can be moved lower down the ladder, while 7C3 is not needed as its activities are dispersed over TRL 4–6. This is illustrated in Figure 5 below.

Indeed, this was not the case for us! With our demand forecasting algorithm, our key limitation occurred because the specific development came about without even a remote market concept in mind. It started as a formal piece of research, which, day after day, appeared more and more interesting as it successfully climbed up to TRL 5/6, at which moment market models became an issue of concern.

We would also argue that this late scenario is not a rare event; it is quite typical, especially for academia-originating research and development activities which are, more often than not, pursued with no clear market horizon in mind. But these may, on the way, generate some tangible value, albeit not of the type required to reach, on its own, the market.

4. Results

In our empirical case, that of an innovative development in explainable energy demand forecasting, at a certain point in time, we realized that the specific development was inherently limited in terms of its market perspective and could therefore not reach beyond a TRL 6/7. We understood that co-creative setups were a possible and promising way towards providing a market potential for this development, a direction we are currently working on.

We also found that the classical formulation of the TRL model of innovation has important shortcomings in describing the details of our approach. The classical TRL has emerged based on notions of producer innovation and not on collaborative and co-creative setups that present challenges and opportunities not adequately traced on the ladder. However, such setups are more and more frequently encountered in the innovation arena and foster the need for an adaptation of the TRL approach.

It is important at this point to draw the line between classical teamwork and co-creation in the way we experienced it and presented it above. Teamwork typically has a common agenda, known from the very beginning. Here, by co-creation, we imply something well beyond teamwork. Essentially, we imply a type of synergy that may be created by bringing together different developments and the opportunities created by this for some of the individual components to assume a critical role in the co-creative setup—one that they could never enjoy outside of it.

We presented a conceptual framework to extend the TRL ladder to account for co-creation. Three more steps were found to be necessary: to define the co-creative value, to define the co-creators, and to make sure that the economics of the co-creation are clear and rational to all, as well as to secure the interoperability of their solution items. The challenges of each one of these were briefly discussed.

Our case was one of late co-creation, meaning that the co-creative idea occurred later, while the innovative result was already at TRL 6/7. However, most of the considerations are also pertinent in early co-creation when the co-creative setup is in place from the very onset.

As confirmed by extensive research of the literature, we also found that the communication costs are the main concern for the viability of a co-creative pathway. These have to be counterbalanced by the additional value resulting independently and for all co-creators, and not just in aggregate terms. This is a most critical condition for the viability of the scheme and for securing an effective move up the TRL ladder.

5. Conclusions and Future Work

The producer innovation model that completely dominated the innovation scene in the past is no longer the only one in use. Both the user and the open collaborative model are increasingly entering the innovation scheme and shaping the course of innovation. Baldwin [23] provides a comprehensive review of the economics of these three models to understand the conditions under which each presents benefits, as well as understanding the mechanisms at play that may erode this advantage.

In our work, we were guided by a practical requirement to push a specific scientific development, a novel energy forecasting algorithm, up the TRL ladder. We soon realized that, if considered as a standalone development, this could never be achieved. We found that a way to bypass this inherent limitation was to resort to collaborative schemes which bore the potential to drastically reposition the specific component and leverage its value in the collaborative development. We also believe that this empirical result has a high generalization potential and can provide guidance and a new mindset for similar endeavors.

Such cases, as well as the many hybrids that may result therefrom, need to be meticulously considered in the light of the current TRL formulation. The TRL approach is a product of the 20th century and reflects the dominant producer innovation model of that period. We have empirically found that this model does not allow us to see the true potential value as it emerges in our time; a co-creative (and not just a producer) mindset is required for such a refined and adapted model to clearly emerge. To this end, we proposed an amendment to the TRL ladder scheme to address this type of opportunity.

Indeed, similar empirical evidence needs to be collected and similar investigations will need to be performed over all possible hybrids, bringing together producer, collaborative, and user innovation, to see if and how these can be described by the current TRL formulation, or if some amendments are due, such as those suggested in our particular hybrid case study, merging producer and open collaborative innovation. Even more dynamically, one needs to consider how such setups can unleash new value in innovative approaches which, if left trapped in the producer model and mindset, cannot reach high on the TRL ladder.

As we approach the end of this treatise, we find it pertinent to refer again to Christensen and his robust, empirical evidence–based market generating approach to innovation. His conceptual framework clearly challenges any ‘newness’-centered TRL formulation to describe the evolution of the value generating innovation. Indeed, we believe this would be a fascinating area of future research, similar to the case of co-creation discussed in more detail in this paper. In the light of related empirical evidence, we would once again need to consider an even more pervasive overhaul of the current TRL formulation which, by reflecting almost exclusively the ‘producer innovation’ and the ‘newness’ concepts, appears to be inadequate for providing an effective structuring of the maturity of a multifold and multipurpose, modern-era innovation.

Of course, the traditional approach to innovation and its accompanying assessment schemes are well present and bound to remain dominant in the innovation arena. To this extent, TRLs will still serve as a common language and continue to play a valuable role in guiding technology development, assessing readiness, and managing risks. In this work, we showed that there is important evidence that, more and more, this innovation model is challenged by novel approaches, such as collaborative and co-creative schemes. Further evidence in the same direction arises from a robust consideration of innovation value. All this will necessitate some reconsideration and adjustment of the typical, current TRL approach, so that it can better fulfill its purpose in the current moment.