Next Article in Journal
Efficient Feature Selection Using Weighted Superposition Attraction Optimization Algorithm
Previous Article in Journal
The Gastrointestinal Microbiota as a Potential Cause and Target in Chronic Kidney Disease Accentuating Treatment and Intervention Strategies
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 13
Issue 5
10.3390/app13053221
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Development of an Approach for the Holistic Assessment of Innovation Projects in Manufacturing Including Potential, Effort, and Risk Using a Systematic Literature Review and Expert Interviews

by
Quirin Gärtner
1,*,
Ermanno Ronco
2,
Anna Corinna Cagliano
2 and
Gunther Reinhart
1
1
Institute for Machine Tools and Industrial Management, Technical University of Munich, Boltzmannstr. 15, 85748 Garching near Munich, Germany
2
Department of Management and Production Engineering, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Torino, Italy
*
Author to whom correspondence should be addressed.
Appl. Sci. 2023, 13(5), 3221; https://doi.org/10.3390/app13053221
Submission received: 31 January 2023 / Revised: 22 February 2023 / Accepted: 24 February 2023 / Published: 2 March 2023
(This article belongs to the Section Mechanical Engineering)
Browse Figures
Review Reports Versions Notes

Abstract

:
Manufacturing companies face severe challenges from rapid technological developments. Industry 5.0 indicates the need for a sustainable, human-centered, and resilient industry. In striving for transformation, innovation becomes critical. However, a careful allocation of resources implies the evaluation of innovation projects. Moreover, diverse types of innovation and limited amounts of information represent a significant challenge. Therefore, this contribution presents an approach for holistically assessing innovation in manufacturing. First, a systematic literature review (SLR) was conducted to frame the current research state and identify assessment criteria. Second, a multiple-attribute decision-making method (MADM) was developed using the findings of the SLR and expert interviews. Finally, the criteria and the assessment approach were verified and validated by expert interviews, a workshop, and an industrial use case application. As the main findings, three criteria groups were derived and detailed: potentials, efforts, and risks. These criteria groups were used in a MADM approach incorporating Fuzzy set theory within a hybrid technique, combining the Analytical Hierarchical Process with the Technique for Order Preference by Similarity to Ideal Solutions. In conclusion, an enhancement of innovation assessment in manufacturing was achieved through the integration of different criteria and the balance between complexity and industrial applicability.

1. Introduction and Motivation

Companies operating in the manufacturing industry nowadays face significant challenges due to globalization, rapidly changing markets [1,2,3], digitalization [4], new regulations [5], and increased demand for product customization [6]. Industry 5.0 represents the manufacturing industry’s most recent technological and social transformation. Starting with the first industrial revolution in the second half of the 18th century, which led to a transition from an agricultural to an industrial society, the second and third industrial revolutions of automation and digitalization (Industry 4.0) followed. While Industry 4.0 represents a digital, data-driven, and interconnected industry, Industry 5.0 complements and extends these features with a need for a sustainable, human-centered, and resilient industry [7]. Innovation in manufacturing, particularly the adoption of new technologies and processes [8,9,10], was identified as a crucial driver for the success of this transformation processes, granting competitiveness [5].
Nevertheless, the innovation process requires significant resources [10], encounters barriers to its effective completion [2], and involves changes that inherently present uncertain outcomes. The potential economic utility [11] alone does not provide the basis for a thorough analysis of the different alternative projects to be assessed. For this reason, a holistic perspective in innovation assessment is necessary [12,13].
Therefore, the scope of the research presented in this contribution is the solution to a design problem. It is derived from the above-mentioned need for an approach for the holistic assessment of innovation projects in manufacturing. A systematic and extensive literature review was conducted to identify existing approaches and the associated constraints. A holistic set of evaluation criteria was derived and synthesized. Those preparatory works allowed the structuring of a Multiple Attribute Decision-Making (MADM) problem to be solved by combining the Analytical Hierarchical Process (AHP) in a Fuzzy variant and the Technique for Order Preference by Similarity to Ideal Solutions (TOPSIS). First, the proposed assessment methodology was verified by both academic experts in the fields of innovation and manufacturing and industrial professionals. Then, the method was implemented, leading to its initial validation in a use case based on data regarding three innovation projects in an aerospace company.

2. Fundamentals of Innovation Assessment in Manufacturing

The concepts of innovation and invention were distinguished by Schumpeter [14] as original combinations of new or existing knowledge, resources, equipment, and other factors. Innovation is conducted commercially, while an invention can be generated without a commercial scope. True innovation is not only an invention or an idea, but the definition also implies the development of something successfully used in the manufacturing process. Additionally, the innovation processes present an outstanding degree of uncertainty. Five possible types of innovation in a manufacturing context are identified: product, process, organization, manufacturing system, and management innovation [15].
The innovation process in manufacturing encompasses all the initiatives from ideation and initiation to its implementation. To be more precise, five main phases can be distinguished in an innovation process [2]: the Impulse and the Idea Generation, corresponding to problem-identification, research, and problem-solving activities; Idea Conceptualization, focusing on the analysis, evaluation, and subsequent selection of a concept; the Idea Realization (Change) phase, corresponding to the implementation of the selected idea; and finally, the Serial Production phase, where the innovation is exploited and eventually transferred to another context or sector. As a result, a cyclic framework was proposed in the context of Manufacturing Innovation Management (MIM) [2] and was later expanded in academia to bring the innovation process to a continuous improvement perspective [16].
Innovation management and its core processes aim to enable the proper addressing of such complex changes in the manufacturing system. In particular, the present research aims to contribute to innovation controlling, specifically within its “strategic innovation planning” component, by expanding the existing approaches, referring to the evaluation of innovation from the perspective of different criteria to solve decision-making problems [15]. The process of ranking and selecting alternatives within a finite set according to their performances measured with multiple criteria falls under the definition of MADM [17]. Several methods have been developed over the years to structure and solve this type of problem. The AHP [18], ELECTRE [19], PROMETHEE [20], and TOPSIS [21] represent the most used approaches. Fuzzy set theory [22] was also exploited to express the imprecision and uncertainty inherent to human judgments in mathematical terms.
In particular, the AHP, as presented by Saaty [18], aims to structure the decision-making problem and solve it within a hierarchical approach driven by the uppermost goal. In detail, pairwise comparisons of the attributes associated with the possible alternatives are used to form reciprocal matrices, synthesizing the individual subjective judgments to estimate the relative weight of each attribute. Then, the aggregation of the resulting weights with the performance rating of each alternative according to each identified decision-making criterion is conducted to determine the best alternative, and therefore the best strategy [17].
TOPSIS is based on the concept of compromise solutions, found by comparing alternatives with ideal positive and negative solutions. An information table is deployed by including the normalized performance scores of each alternative under the evaluation criteria and the weights of the attributes. Two ideal solutions are identified, one with the best scores for each criterion among the alternatives (positive solution) and one with the worst (negative solution). The best compromise is represented by the alternative closest in terms of Euclidean distance to the positive ideal solution [17,21].
In the literature, there are countless implementations of Fuzzy set theory in MADM. Primarily, the linguistic Fuzzy approach is frequently applied to represent the uncertainty associated with using qualitative linguistic evaluation scales in mathematical terms [23,24]. It has been proven that the implementation of triangular and trapezoidal Fuzzy membership functions, with dedicated aggregation and de-fuzzification methods, is suitable for addressing the problem of group decisions [25].

3. State of the Art

The present section systematically delimits the scope of this research by providing an analysis of the state of the art. Therefore, drawing on the basic principles explained in the previous section, the delimitation is made object-, subject-, and process-related. Firstly, the considered objects of the developed approach are described. The main object-related categories are manufacturing and the innovation project. Therefore, only approaches aiming at improvements within the functional area of manufacturing are to be considered. Second, innovation projects can be differentiated regarding their content dimension. In manufacturing, innovations related to processes, organizations, manufacturing systems, and management are relevant [12]. Consequently, Ulich [26] proposes the Human-Technology-Organization Model (HTO), differentiating the content dimensions of innovation projects in human, technological, or organizational innovation.
Regarding the process-related delimitation, the proposed model serves as a support for the early phases of the innovation process, whereby the initial phases of Problem Identification and Idea Generation are not considered. The approach aims to accompany the phase of Idea Conceptualization, including evaluation and decision-making. The subject-related delimitation describes the users of the approach. It is primarily directed to industrial and scientific committees and general experts in innovation management.
The following section presents the state of research based on a comprehensive literature analysis to identify the relevant criteria for holistic innovation assessment and existing methodical approaches. Figure 1 provides an overview of the conducted Systematic Literature Review according to Rowley and Keegan [27]. A specific method structured in consequential steps was deployed to derive the most critical and relevant contributions. To reduce the number of results, sources that lay outside the scope were excluded by analyzing their titles, abstracts, and tables of contents. The remaining sources were listed and passed on to the final full-text analysis. In addition, a Backward and Forward Snowballing search, according to Webster and Watson [28], was applied to find further relevant contributions.
More than 1600 journal and conference papers, dissertations, and books were identified using the databases OPAC, Scopus, Web of Science, Google Scholar, and SciVal. The search terms (keywords) displayed in the following table (Table 1) and their German translations were used, as numerous relevant authors in the research field have published their works and doctoral theses in German. The keywords were linked with logical operators of Boolean algebra for conjunction (and) and disjunction (or) to refine the results. To identify the most recent publications but also to understand the evolution of models for assessing innovation in manufacturing over time, publications from 1970–2022 were analyzed. Further, the results were filtered using the criteria shown in Table 1.
As a result, 121 relevant contributions, including 27 methodical innovation assessment approaches, were identified and analyzed systematically. The review results are presented in Table 2. Within the analysis, various criteria were used to classify the approaches. First, it was examined whether the approach was focused on the manufacturing area. Second, the approaches were temporally assigned to the innovation process to examine their applicability within the early phase of the process. Third, the approach was focused on the innovation objects considered. Here, a differentiation following the HTO-Model proposed by Ulich [26] was applied to identify if the approach focused on human, technological, or organizational innovation. Further, the criteria used to assess the innovation projects were analyzed. There, potentials, efforts, and risks were differentiated according to the identified approaches. Fifth, the MADM assessment methods used by the state-of-the-art contributions were examined. Finally, the industrial applicability was evaluated based on the implementation effort and the practical relevance of the results.
As displayed in Table 2, it was found that there is already a comprehensive state of the art on methods of technological assessment [3,6,10,13,29,30,31,32,33,34,35,36,37,38] and change management in manufacturing [39,40,41,42,43]. One- or multi-sided evaluation perspectives are adopted, focusing exclusively on the potentials, risks, or efforts, or pairs of these aspects. Further, some more holistic evaluation approaches have already been used to select production strategies [44,45] or assess innovation [46,47,48,49].
Based on the analysis, different needs for further research were derived. More than one third (37%) of the identified models were not fully focused on the early phase of the innovation process. However, to identify inadequate or promising projects, it is necessary to evaluate them at an early stage of the innovation process. Further, a method for innovation assessment must be specifically adapted to manufacturing to enable a target-oriented evaluation incorporating imprecision.
Additionally, Table 2 shows that only three approaches considered all relevant types of innovation in manufacturing [13,41,46], while the majority (66%) of approaches were solely focused on technological innovation. In the context of Industry 5.0, as presented by the European Union [7], it is essential also to consider innovation increasing prosperity for humans and organizations within the assessment. Consequently, a broader catalog of criteria for evaluating manufacturing innovation is necessary to enable a holistic evaluation. However, almost all (85%) of the approaches only considered two or fewer of the relevant criteria groups (potential, effort, risk) within their assessment.
Looking at the decision-making methods applied, the majority (81%) of the analyzed approaches deliver a structured decision-making approach. Utility analysis and AHP form the most common methods to structure decision-making problems, followed by TOPSIS. The analysis also shows that the Fuzzy set theory is widely used to address uncertainty and guarantee a translation from linguistic variables to numerical values. However, the implementation of a structured and proven MADM method, enabling a practical innovation assessment tool, is less common than expected.
Despite the identified research gaps, the analyzed literature provides an excellent scientific basis for developing an approach for the holistic assessment of the early phases of the innovation process in manufacturing. In particular, the approaches of Farooq and O’Brien [33], Eßer et al. [47], and Liang and Li [48] already consider a wide range of potential-, effort-, and risk-related criteria. Further, human-, technology-, and organization-focused innovation in manufacturing is already included in the approaches presented by Essakly et al. [13], Plehn et al. [41], and Arabshahi and Fazlollahtabar [46]. Additionally, a structured and practical applicable MADM method enabling a transparent decision-making process within the early phases of the innovation process was proposed by Ordoobadi [34], Schuh et al. [51], and Hofer et al. [31]. The existing approaches can be combined and extended using scientific methods to derive a holistic approach for innovation assessment in manufacturing, with reference to process innovation.

4. Methodical Approach

The following section presents the systematic, application-oriented procedure that was used to elaborate the proposed approach for assessing innovation projects in manufacturing. Thereby, the identified starting points for further research from the previous section were addressed. The procedure can be divided into three major phases as displayed in Figure 2.
In the first phase, the relevant criteria for the holistic assessment of innovation projects in manufacturing need to be derived. Therefore, the results of the structured literature review presented in Section 3 were used as a basis for the criteria definition. Following, the selected criteria were structured using propositions identified from the scientific literature. In addition, the derived criteria were challenged and complemented methodically conducting eight semi-structured interviews with experts from the manufacturing industry. Lastly, the selected criteria were verified through an expert workshop with eight participants with a scientific background. On this basis, a criteria catalog for the holistic assessment of innovation projects in manufacturing was derived. This catalog provides the basis for the second phase.
As a starting point for the second step, existing assessment methods were identified based on the structured literature analysis presented in Section 3. Next, the requirements for an innovation assessment method were derived from the state of the art and expert interviews. On this basis, an approach for innovation assessment in manufacturing was developed to fulfill the defined requirements and to include the identified evaluation criteria from the first phase. Subsequently, the elaborated approach was verified by conducting two semi-structured expert interviews with participants from academia. In conclusion, a prototype of the approach was implemented using a MATLAB script to enable a practical industrial application and to provide the basis for the third phase.
In the third phase, an initial evaluation of the developed assessment approach was conducted within an industrial use case application. Thereby, three production innovation projects were compared regarding potential, effort, and risk they imply. Subsequently, a management overview and a recommended action were derived. Finally, the approach’s results, efforts, and practical benefits were critically reviewed by the authors and two industrial experts that were involved in the application. These results state the last step of the approach, building the basis for the discussion and the perspective.

5. Developing the Approach for the Assessment of Innovation Projects in Manufacturing

5.1. Phase I: Criteria Derivation for Innovation Project Assessment in Manufacturing

As a first step, the results of the systematic literature analysis presented in Section 3 were used as a scientific basis for deriving the relevant criteria for innovation assessment in manufacturing. According to the guidelines for a literature review [27], research categories and keywords were derived from various scientific contributions (see Table 1 Section 3). A detailed explanation of the systematic literature analysis approach, the used keywords, and the applied limitations can be obtained from Section 3. After performing the two-step analysis presented in Section 3, the remaining 121 pertinent publications were used to identify relevant innovation assessment criteria. Table 3 shows an excerpt of the identified 121 publications and the included evaluation criteria. The reader should note that the list of criteria presented in this table is the first one that was released while developing the holistic innovation assessment approach, before the expert review.
With the help of the identified sources, relevant descriptive criteria were derived, collected, and combined into groups. Based on the investigated scientific literature, three main criteria groups for innovation assessment were derived: potentials, efforts, and risks.
Potentials determine the prospective evaluation of the change impacts caused by the innovation project [31]. The second criteria group describes efforts related to an innovation project, as innovation projects are associated with a significant expenditure of resources and time [68]. Due to uncertainties within the innovation process, risk states, the third criteria group, are necessary for a holistic assessment of manufacturing [69]. The three main criteria groups were further specified using relevant scientific contributions. In this way, further groups and sub-groups of criteria could be identified, containing detailed assessment criteria. In Figure 3, a schematic structure of the catalog after the expert review is displayed. The criteria catalog was not detailed further to provide a catalog which, on the one hand, enables a company- and use-case-specific application and, on the other hand, provides a sufficient level of detail to cover all relevant criteria for project assessment in manufacturing.
In addition, eight expert interviews were conducted with representatives from the manufacturing industry. The industrial experts were involved in the production management of an internationally active construction materials and tools manufacturer and two SMEs manufacturing machines, to include both small and large companies in the panel. Figure 3 provides a more detailed insight into the profiles of the interviewed experts. The interviews were conducted using a guideline to carry out semi-structured systematizing expert interviews [70]. By applying this systematic and theory-based procedure, the model was supplemented with exclusively available knowledge and validated already in the first stage.
For the data collection within every single semi-structured systematizing expert interview, a three-step procedure was chosen. First, the research methods and the evaluation criteria already identified were presented to the expert. Secondly, the already identified criteria were discussed individually with the expert and, if necessary, supplemented or questioned until a new criteria basis was created that additionally represented the expert’s views. In the last step, the results were processed and made available to the expert. These results then constituted the input for the next expert interview. As an outcome, it could be determined that the detailing of the detailed criteria mentioned above could be improved compared to the initial results of the literature analysis. Lastly, the final verification of the criteria catalog was carried out in a workshop with eight researchers from a German research institute for production technology. During the half-day workshop, a three-step approach was conducted. First, the criteria catalog was presented and discussed within the group regarding completeness. Second, the group was challenged to use the catalog to describe their recent research projects to test the practical applicability of the catalog. Third, the workshop results were summarized by creating the final criteria catalog and made available to the participants. After completing this structured approach, 11 criteria groups emerged, containing 34 sub-criteria groups with 123 detailed criteria. The final catalog is detailed in the following figures (Figure 4, Figure 5 and Figure 6).
The first main criteria group (Potential) describes what relevant changes can be expected after the implementation of an innovation project. The relevant outcomes are derived from the generic target dimensions of manufacturing described by Hofer et al. [31] and include costs, quality, time, changeability, social, ecological, and strategic factors.
Cost represents the first main criterion within the potentials, as cost reduction is the main contribution of manufacturing to secure a company’s profitability. The cost-specific evaluation of an innovation project focuses exclusively on the long-term effects on the cost structure of the company since one-off costs are considered in the effort evaluation. The further structuring of costs is based on the process-related cost analysis in manufacturing according to Pfeffer and Gottmann [71]. With reference to the scope, organization, production, logistics, and quality (customers), processes are further detailed in the sub-criteria. In addition, the perspective on manufacturing factors according to Kern [72] was used in the catalog. Following this, all costs of manufacturing factors, such as personnel or material, can be attributed to the respective activities within the detailed criteria. Subsequently, the cost-relevant criteria were detailed as displayed in Figure 4 with the help of the relevant preliminary work [52,71,72,73,74,75].
Quality is the second main criterion of the potential assessment. Generally, quality in manufacturing is divided into an internal and an external perspective. Internally, process quality is defined by error-free and efficient processes. From an external customer perspective, product quality is the decisive factor. Therefore, it is necessary to identify characteristics for both perspectives to assess the project-related effects. To describe process quality in manufacturing, four general sub-objectives and characteristics were identified based on the scientific literature [53,54,55,56]. Effects on product quality are indirect, e.g., a better surface quality due to a new manufacturing process. As products are diverse, a generic definition of quality characteristics is not possible. Consequently, the quality characteristics are to be detailed by the user. The complete detailing of the criterion can be found in Figure 4.
The third main criterion is time. There is a common understanding in the literature that a short lead time is one of the main objectives in manufacturing. It describes how long a product needs to pass through the entire manufacturing process. Accordingly, the effects of project-related changes on the lead time are to be evaluated. Therefore, the internal manufacturing lead time was broken down into waiting, transport, and processing, as well as planned and unplanned downtime [57,76]. Further detail is displayed in Figure 4 above.
Changeability forms the fourth main criteria dimension for innovation projects in manufacturing. According to Zäh et al. [58], changeability consists of flexibility and adaptability, where flexibility is defined as the ability to meet necessary functional, dimensional, and structural requirements at different manufacturing system levels within a specific range. Additionally, adaptability is a potential going beyond this range to adjust the manufacturing system to changing conditions. The classification of Sethi and Sethi [59] is used to systematize changeability. Here, the individual types of manufacturing changeability are differentiated regarding processes, routing, product-mix, volume, and expansion. Figure 4 gives a detailed overview of the identified changeability assessment criteria.
Social impacts form the fifth relevant main criteria dimension within innovation project assessment. From a manufacturing point of view, social influences are decisive since people form an essential production factor. Further, a correlation between performance and employee satisfaction can be observed [60,77]. Manufacturing innovation can influence employee needs regarding a safe and healthy workplace. The workplace and its human-related activities can be further subdivided into environmental and activity-related stress [61]. Further, activity-related stress can be expressed as physical or psychological stress. A detailing of the relevant social assessment criteria is displayed in Figure 4.
Since a purely economic evaluation of strategic decisions is no longer appropriate, ecological factors form the sixth main criteria dimension. Consequently, the goals of a manufacturing unit regarding the corporate ecological environment are to be considered using input–output relationships, which are already well described in the literature [62]. As displayed (Figure 4), operating materials and energy can be used as input factors, while emissions and waste are to be considered on the output side.
Strategic factors form the final main criteria of Potential. In the context of the innovation project evaluation, strategic target values were considered that lie in functional strategies [1]. To fulfill this strategy, it is necessary to consider conformity and strategic impact [63]. For further strategic improvement, strategic knowledge must be generated, and non-value-adding activities must be identified, based on the Lean philosophy. Following, competency and transparency along the value stream is decisive [64]. The further detailing of these strategic criteria according to Hofer et al. [31], Cooper [63], and Knoll et al. [65] is displayed in Figure 4.
In addition to the evaluation of potentials, a differentiated consideration of necessary efforts is required to decide between alternative innovation projects. However, so-called “sunk costs” [78] are no longer relevant to the decision as they have already occurred. Consequently, only future efforts were considered. Within manufacturing innovation projects, efforts can arise in many ways. Nevertheless, the project management theory considers two effort-critical resources: money and time [68]. The temporal assessment considers the required execution time, while the monetary view focusses on the expected expenses.
Within the economic and temporal criteria, five dimensions were identified in the literature: development, organization, structures, qualification and consulting, and opportunity [39,66]. Technology-specific efforts in innovation initiatives are summarized through research and development activities, designing, prototyping, testing, and activities of the specific implementation. Organizational efforts can be summarized by necessary resources for project planning and control. Structural efforts subsume all expenditures that arise from physical changes to the existing production system and its production factors. Qualification and consulting summarize all efforts that occur, either through one-off personnel expenses such as training or through expenses for external consulting services. Finally, opportunity efforts describe all anticipated costs arising from unused opportunities, such as downtimes or scrapped materials. A further detailing of the defined effort criteria can be found in Figure 5.
The evaluation of innovation projects in their early phase is characterized by a high degree of uncertainty, leading to the fact that the risk dimension cannot be neglected [69]. Risk criteria to be considered in the assessment should be systematized into the different risks relevant to manufacturing. Risk factors of the innovation initiatives are classified as internal or external to manufacturing according to whether they can be controlled by the company managing the production process or not [79]. The internal risks are further divided into risks affecting the innovation project and the manufacturing system. Within the project-specific risks, a distinction can be made between cost, time, and organization- and resource-related risks. Manufacturing-specific risks are subdivided into technological, social, and organizational exploitation risks following the HTO approach [26]. Risks external to the company represent the company’s environment (political, market-dependent, environmental (ecological), social, and legal factors) that has a direct influence on the innovation project. Since this environment is subject to constant change, further detail is neither desirable nor meaningful. A detailing of the relevant internal risk criteria, derived from Schuh et al. [11], Li [53], and Spur et al. [67], is displayed in Figure 6.

5.2. Phase II: Approach Development for Holistic Innovation Assessment in Manufacturing

A three-step approach was chosen to develop an appropriate MADM method addressing the gaps in the state of the art. Figure 7 provides an overview of the derivation of the assessment method. In the first step, a structured literature review (see Section 3) was used to identify the most relevant methodological approaches and the used assessment methods. The table shown in Section 3 (Table 2) gives an overview of the identified approaches. Subsequently, the requirements were derived based on the research gap and an evaluation method was developed by combining the identified methods in a meaningful way based on their advantages and disadvantages. In the second step, the approach was presented to two academic experts from the innovation and manufacturing fields, using semi-structured expert interviews to discuss the overall concept and the criteria selected for the assessment. Lastly, a prototype of the approach was implemented using a MATLAB script to initialize the initial validation of the approach.
Among the plethora of MADM approaches presented in the literature, AHP and TOPSIS stand out as the most suitable methods because of their comprehensible, methodical, and individually adaptable procedure. Moreover, Liu et al. [25] proposed a methodology for implementing Fuzzy sets in MADM methods according to the characteristics of the data to be parsed by the model and included the possibility of managing group decision scenarios by discussing several aggregation methods for judgments. Figure 8 provides a schematic overview of the elaborated approach for holistic innovation assessment in manufacturing.
Within an industrial scenario, a finite set of alternatives will be compared by decision makers aiming to select the best initiative to undertake according to the attributes and criteria identified in Section 5.1. This kind of problem is well represented by the definition of a MADM method. Based on the state of the art, the approach requires applicability to human-, technological-, and organizational innovation, and demands a holistic assessment, taking potentials, efforts, and risks into account. Further, the approach needs to be industrial-applicable within the early phases of the innovation process.
The development of a decision support system of this kind involves the implementation of the criteria identified in Section 5.1 with evaluation scales and a suitable mathematical method allowing to synthesize the assessment activities’ resulting in a ranking of the alternatives [50].
The model presented exploits the strengths of AHP to transform the decision-making problem into a hierarchic structure of criteria to be selected among the attributes discussed in Section 5.1. The relative weights of the attributes were obtained using pairwise comparisons performed with linguistic judgments translated in Fuzzy triangular numbers utilizing a variation of the fundamental scale proposed by Saaty [18], defined according to the principles discussed by Liu et al. [25]. The comparisons were performed at each level of the hierarchical structure, as proposed by Saaty and Shang [80]. Afterwards, the weights of the attributes were extracted by each pairwise comparison matrix through the eigenvalue method, linking the eigenvector associated with the maximum eigenvalue of each pairwise comparison matrix to an expression of the relative dominance of the criteria. The weights of the lowest-level attributes of the structure were multiplied by the relative weights of their parent clusters. Aggregation of the matrices corresponding to different decision-makers was achieved through the geometric mean. The de-fuzzification of the matrices using the centroid method, consistent with Yager [81], is the preliminary step for the consistency check and the extraction of the weights [82]. The judgments relative to the performance assessment of the alternatives were assigned by the user(s) of the proposed approach according to a qualitative evaluation scale, including a modified version of the risk matrix proposed by Goddard [83]. The Fuzzy results were eventually aggregated with a simple average [25]. The weights set resulting from the Fuzzy application of the AHP and the performance ratings constitute the information table representing the starting point for applying TOPSIS for the final ranking and selecting the best alternative [84]. The application of TOPSIS, as mentioned in Section 2, allows us to rank the alternatives according to an index, comparing them to two ideal solutions. These solutions include the best and worst individual performance ratings for each criterion in terms of Euclidean distance. Again, as described in Section 2, the ratings were normalized and multiplied by the weights obtained through the AHP.
The output of the model is a set of three separate rankings of the alternatives according to the global dimensions of potential, effort, and risk. The final decision is left to the user and the decision-maker, as the interdependence between potentials, efforts, and risks cannot be neglected. To address this, the approach offers a synthesis of the assessment results that can be represented in a portfolio overview.
Second, semi-structured expert interviews were conducted. Within the semi-structured expert interviews, a two-step approach was chosen. Firstly, the method was presented to the experts, explaining why specific MADM methods were combined. The experts’ feedback regarding the logic and applicability was collected in the second step within a discussion. The verification obtained the expert’s appraisal regarding the approach because it combines the potential from a hierarchical representation of the decision-making problem (AHP) with a more refined ranking process (TOPSIS) and includes uncertainty factors (Fuzzy) present in human thinking and linguistic variables.
Thirdly and finally, a prototype of the final approach was implemented using a MATLAB script to facilitate the necessary mathematical operations.

5.3. Phase III: Initial Application

The prototypical script built the starting point for the following validation. Thereby, the approach was applied to a use case, aiming to evaluate different approaches to improve internal logistics for an aerospace company. The application was carried out in collaboration with two experts from the respective company. The criteria selection, weighting, and the qualitative performance assessment of the three alternatives was performed by the experts and is presented in Table 4 together with the local and absolute weights. The alternative projects were assessed and weighted according to the selected criteria. Eighteen detail criteria were used to assess potentials and efforts, while the exploitation risk was summarized in a single overall rating. The application of the method resulted in the final closest-to-ideal values, summarized in Figure 9. Based on the obtained values a portfolio was created, displaying efforts on the abscissa, potentials on the ordinate, and risks in terms of the bubble size. The portfolio can help decision makers to clearly identify the best alternative innovation project in the top right corner of the portfolio. Within the present use case, the second alternative results as the best-in-class, as it has the most potential and the lowest costs compared to the other two alternatives. It is also not affected by any risks, as displayed in Table 4. The advantages of the second alternative are also shown in the portfolio illustration in Figure 9. The second alternative is closest to the ideal for potentials and efforts and has a small bubble size, indicating a low risk.
In conclusion, the fulfillment of the requirements identified for the developed innovation project assessment approach was discussed with the industrial experts. The derived portfolio helped the industrial decision-makers to identify the most valuable project within the three alternatives based on a structured approach at an early stage. On this basis, the requirement of industrial applicability in the early phase of the innovation process can be considered fulfilled. Further, the industrial users confirmed that the inclusion of all three assessment dimensions enabled a holistic assessment. Lastly, coherently with the idea of Industry 5.0, the selected criteria enable an inclusion of human, technological, and organizational points of view within the assessment approach, even if the projects considered were of a mainly technical and organizational background. Finally, the use case application helped to identify various starting points for the future development of the method, building the basis for the following discussion and perspective.

6. Implications of Findings

The approach was verified within the respective phases of the research methodology. The identified assessment criteria were verified within a workshop including scientific experts. The resulting assessment approach was verified based on two expert interviews with members of the academic community and initially validated through one industrial use case. On this basis, a discussion of the results can be carried out, and perspectives for further research can be derived.
Based on the expert interviews and the initial use case application, it can be concluded that the identified gaps in the state of research were sufficiently addressed. In contrast to existing MADM methods (e.g., [8,39]), the method developed enables an evaluation of innovation projects within manufacturing in the early phases of the innovation process. The structuring of existing assessment criteria into potentials, efforts, and risks, as well as the inclusion of non-monetary criteria such as ecological or social impacts, state the first main contribution to the state of research within the innovation project assessment. In contrast to existing evaluation methods, which are mainly focused on technology assessments (e.g., [29,31]), a consideration of the various relevant types of innovation in manufacturing is ensured by a comprehensive catalog of criteria, oriented towards the HTO concept according to Ulich [26]. Furthermore, a holistic evaluation is provided by considering potentials, efforts, and risks within the assessment criteria. Finally, the combination of AHP, TOPSIS, and Fuzzy approaches states the second contribution to the state of research, enabling an evaluation that operates in a structured manner, takes different dimensions and input variables into account, but is still practically applicable. The developed MADM method combines the strengths of the different approaches, transforming the problem into a hierarchic criteria structure and ranking the alternatives according to ideal solutions. This enables the user to evaluate different innovation projects through the company-specific selection and structuring of evaluation criteria. On the other hand, comparing the evaluation results with the ideal solution enables meaningful comparison of different projects within a company.
Nevertheless, based on the application and the discussions with a plethora of different experts, three possible starting points for the further development and continuation of research activities could be identified:
  • Point 1—Extension and further specification of the assessment criteria: With the constant change and enhancement of target dimensions within manufacturing as well as the release of innovative and complex technologies the assessment criteria selection must be continuously adapted. Further, to enable the ongoing development of the Industry 5.0 concept, emerging potentials, efforts, and risks may need to be integrated into the assessment approach to enable an evaluation that not primarily aims at increasing cost-efficiency but also includes workers, consumers, society, and the environment. Additionally, Key Performance Indicators (KPI) could specify the assessment criteria. By setting up a KPI model for each criterion, the changes within the respective dimension could be indicated more precisely.
  • Point 2—Further inclusion of cause-effect relationships: Cause-effect relationships between the potential or effort targets and the company’s strategic goals could be integrated to elaborate the model perspective towards a broader, stakeholder-focused analysis.
  • Point 3—Further industrial validation: The approach should be validated with the help of other use cases from different industries and the representation of group decision scenarios. On this basis, the theoretical strengths of the approach could be validated and potential starting points for further development work could be identified.

7. Conclusions

A holistic innovation assessment approach has the potential to enable manufacturing companies to deal with fundamental and complex challenges, such as globalization or Industry 5.0, in a systematic and structured manner. This allows managers to select innovation projects in a structured way and to deploy resources in a targeted manner. However, the holistic assessment of innovation projects in manufacturing is complex due to a lack of approaches that provide a structured, but still easily applicable MADM method, enabling a transparent and logical decision-making process considering all relevant criteria in the early phase.
This contribution presents a holistic criteria catalog for innovation project assessment in manufacturing and an approach for practically applicable decision-making, including Fuzzy set theory based on a literature review and expert interviews. First, the catalog proposes the relevant assessment criteria in detail and structures them into groups according to three dimensions; namely, potentials, efforts, and risks. To this end, a comprehensive literature review and expert interviews have identified relevant assessment criteria. The findings of the interviews and the previous scientific publications were analyzed, and relevant criteria were selected and structured. In conclusion, an expert workshop was held to verify the elaborated structure and the detailed criteria catalog. Second, an approach for holistic manufacturing innovation assessment was elaborated. Therefore, the state of research was analyzed, and suitable methods were combined to derive an approach to incorporate the benefits of the various methodologies already existing in the literature. It was found that a combination of AHP, TOPSIS, and Fuzzy set theory is suitable to fill the identified needs regarding the structure and industrial applicability of the developed assessment approach. Third, the elaborated approach underwent an initial validation within an industrial use case application and the fulfillment of the identified requirement was challenged with the help of two industrial experts. In addition, starting points for further research and adaption of the approach were identified. To summarize, an enhancement of innovation project evaluation in manufacturing was achieved through the integration of different criteria and the balance between methodological complexity and industrial applicability of the presented assessment approach. The approach constitutes a relevant support for manufacturing companies in the transformation process towards digitalization and Industry 5.0, enabling a holistic assessment including social, technological, and ecological potentials, risks, and efforts.

Author Contributions

Conceptualization, Q.G. and E.R.; methodology, Q.G. and E.R.; software, E.R.; validation, Q.G. and E.R.; formal analysis, E.R.; investigation, Q.G. and E.R.; resources, Q.G. and G.R.; data curation, E.R.; writing—original draft preparation, Q.G. and E.R.; writing—review and editing, Q.G., A.C.C. and G.R.; visualization, Q.G.; supervision, A.C.C. and G.R.; project administration, Q.G. and A.C.C.; funding acquisition, G.R. and A.C.C. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the German Research Foundation (DFG) (No. 431696521).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

The authors would like to thank the DFG for its generous support. We would also like to sincerely thank the reviewers for their helpful and constructive comments that clearly contributed to improving this manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Mack, O.; Khare, A. Perspectives on a VUCA World. In Managing in a VUCA World; Mack, O., Khare, A., Krämer, A., Burgartz, T., Eds.; Springer International Publishing: Cham, Switzerland, 2016; pp. 3–19. [Google Scholar]
  2. Hofer, A.; Brandl, F.J.; Bauer, H.; Haghi, S.; Reinhart, G. A Framework for Managing Innovation Cycles in Manufacturing Systems. Procedia CIRP 2020, 93, 771–776. [Google Scholar] [CrossRef]
  3. Becattini, N.; Cascini, G.; Pierluigi, P.; Pucciarini, A. Production Processes Modeling for Identifying Technology Substitution Opportunities. Procedia Eng. 2015, 131, 14–29. [Google Scholar] [CrossRef] [Green Version]
  4. Cagliano, A.C.; Mangano, G.; Rafele, C. Determinants of digital technology adoption in supply chain. An exploratory analysis. Supply Chain Forum Int. J. 2021, 22, 100–114. [Google Scholar] [CrossRef]
  5. Hofer, A.; Schnell, J.; Beck, B.; Reinhart, G. Potential-based technology planning for production companies. Procedia CIRP 2019, 81, 1400–1405. [Google Scholar] [CrossRef]
  6. Rebentisch, E.; Schuh, G.; Rudolf, S.; Breunig, S.; Brakemeier, C. Technology Assessment for Modular Product Platforms with Fuzzy Numbers. Procedia CIRP 2016, 50, 601–606. [Google Scholar] [CrossRef] [Green Version]
  7. Breque, M.; de Nul, L.; Petridis, A. Industry 5.0: Towards a Sustainable, Human-Centric and Resilient European Industry; European Commission: Brussels, Belgium, 2022.
  8. Millen, R.; Sohal, A.S. Planning processes for advanced manufacturing technology by large American manufacturers. Technovation 1998, 18, 741–750. [Google Scholar] [CrossRef]
  9. Greitemann, J.; Hehl, M.; Wagner, D.; Reinhart, G. Scenario and roadmap-based approach for the analysis of prospective production technology needs. Prod. Eng. 2016, 10, 337–343. [Google Scholar] [CrossRef]
  10. Greitemann, J.; Zaggl, M.A.; Hehl, M.; Raasch, C.; Reinhart, G. Technology lifecycle-oriented search for production technologies. CIRP J. Manuf. Sci. Technol. 2017, 16, 21–33. [Google Scholar] [CrossRef]
  11. Schuh, G.; Scholz, P.; Seichter, S.; Schuh, G. Identification of Indicators for the Assessment of Technological Risks within Technology Selection. In Proceedings of the 2020 61st International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS), Riga, Latvia, 15–16 October 2020; pp. 1–8. [Google Scholar] [CrossRef]
  12. Arnal, Á.; Dıáz-Ramırez, M.; Acevedo, L.; Ferreira, V.J.; Garcıá-Armingol, T.; LópezSabiron, A.M.; Ferreira, G. Multicriteria analysis for retrofitting of natural gas melting and heating furnaces for sustainable manufacturing and Industry 4.0. ASME J. Energy Resour. Technol. 2020, 142, 022203. [Google Scholar] [CrossRef] [Green Version]
  13. Essakly, A.; Wichmann, M.; Spengler, T.S. A reference framework for the holistic evaluation of Industry 4.0 solutions for small- And medium-sized enterprises. IFAC-PapersOnLine 2019, 52, 427–432. [Google Scholar] [CrossRef]
  14. Schumpeter, J.A. Konjunkturzyklen: Eine Theoretische, Historische und Statistische Analyse des Kapitalistischen Prozesses; Vandenhoek & Ruprecht: Göttingen, Germany, 1961. [Google Scholar]
  15. Brandl, F.J.; Kagerer, M.; Reinhart, G. A Hybrid Innovation Management Framework for Manufacturing—Enablers for more Agility in Plants. Procedia CIRP 2018, 72, 1154–1159. [Google Scholar] [CrossRef]
  16. Gärtner, Q.; Hofer, A.; Reinhart, G. Identification and systematization of strategic technology demands in manufacturing. Procedia CIRP 2021, 104, 32–37. [Google Scholar] [CrossRef]
  17. Shen, K.-Y.; Tzeng, G.-H. Advances in Multiple Criteria Decision Making for Sustainability: Modeling and Applications. Sustainability 2018, 10, 1600. [Google Scholar] [CrossRef] [Green Version]
  18. Saaty, T.L. The Analytic Hierarchy and Analytic Network Processes for the Measurement of Intangible Criteria and for Decision-Making. Int. Ser. Oper. Res. Manag. Sci. 2016, 233, 363–419. [Google Scholar] [CrossRef]
  19. Figueira, J.R.; Mousseau, V.; Roy, B. ELECTRE Methods. Mult. Criteria Decis. Anal. 2016, 233, 155–185. [Google Scholar] [CrossRef]
  20. Brans, J.-P.; de Smet, Y. PROMETHEE Methods. Mult. Criteria Decis. Anal. 2016, 233, 187–219. [Google Scholar] [CrossRef]
  21. Hwang, C.-L.; Yoon, K. Multiple Attribute Decision Making: Methods and Applications; Springer: Berlin/Heidelberg, Germany, 1981; ISBN 978-3-642-48318-9. [Google Scholar]
  22. Zadeh, L.A. Fuzzy logic—A personal perspective. Fuzzy Sets Syst. 2015, 281, 4–20. [Google Scholar] [CrossRef]
  23. de Barros, L.C.; Bassanezi, R.C.; Lodwick, W.A. A First Course in Fuzzy Logic, Fuzzy Dynamical Systems, and Biomathematics: Theory and Applications; Springer: Berlin/Heidelberg, Germany, 2017; ISBN 9783662533246. [Google Scholar]
  24. Rangone, A. A reference framework for the application of the fuzzy set theory to the assessment of investments in advanced manufacturing technologies. In Proceedings of the 1995 IEEE International Conference on Systems, Man and Cybernetics, Vancouver, BC, Canada, 22–25 October 1995; Volume 3, pp. 2880–2885. [Google Scholar] [CrossRef]
  25. Liu, Y.; Eckert, C.M.; Earl, C. A review of fuzzy AHP methods for decision-making with subjective judgements. Expert Syst. Appl. 2020, 161, 113738. [Google Scholar] [CrossRef]
  26. Ulich, E. Man—Technology—Organization: An European production concept. Concr. Precast. Plant Technol. 1999, 65, 26–31. [Google Scholar]
  27. Rowley, J.; Keegan, B.J. An overview of systematic literature reviews in social media marketing. J. Inf. Sci. 2020, 46, 725–738. [Google Scholar] [CrossRef] [Green Version]
  28. Watson, R.T.; Webster, J. Analysing the past to prepare for the future: Writing a literature review a roadmap for release 2.0. J. Decis. Syst. 2020, 29, 129–147. [Google Scholar] [CrossRef]
  29. Schuh, G.; Scholz, P.; Latz, T. Determining the strategic potential of technologies for manufacturing companies. Vorausschau Technol. 2021, 17, 471–494. [Google Scholar]
  30. Schuh, G.; Scholz, P. Risk and potential evaluation of technologies in the early stages of the innovation process. Vorausschau Technol. 2019, 15, 381–400. [Google Scholar]
  31. Hofer, A.; Pemsel, B.; Putz, M. Potential-based Evaluation of Innovative Technologies in Production. Z. Wirtsch. Fabr. 2021, 116, 294–298. [Google Scholar] [CrossRef]
  32. Larsson, L.; Stahre, J.; Warrol, C.; Rönnbäck, A.Ö. An Assessment Model for Production Innovation: The Program Production 2030. Procedia Manuf. 2018, 25, 134–141. [Google Scholar] [CrossRef]
  33. Farooq, S.; O’Brien, C. Risk calculations in the manufacturing technology selection process. J. Manuf. Technol. Manag. 2010, 21, 28–49. [Google Scholar] [CrossRef]
  34. Ordoobadi, S.M. Fuzzy logic and evaluation of advanced technologies. Ind. Manage. Data Syst. 2008, 108, 928–946. [Google Scholar] [CrossRef]
  35. Schöning, S. Potenzialbasierte Bewertung Neuer Technologien. Ph.D. Dissertation, RWTH Aachen, Aachen, Germany, 2006. [Google Scholar]
  36. Hall, K. Ganzheitliche Technologiebewertung: Ein Modell zur Bewertung Unterschiedlicher Produktionstechnologien. Ph.D. Dissertation, Universität Leoben, Leoben, Austria, 2002. [Google Scholar]
  37. Yap, C.M.; Souder, W. A filter system for technology evaluation and selection. Technovation 1993, 13, 449–469. [Google Scholar] [CrossRef]
  38. Servatius, H.-G.; Peiffer, S. Ganzheitliche und Evolutionäre Technologiebewertung. In Technologiefrühaufklärung; VDI-Technologiezentrum, Ed.; Schäffer-Poeschel Verlag: Stuttgart, Germany, 1992; pp. 73–98. [Google Scholar]
  39. Kröger, S.; Bauer, H.; Reinhart, G. Cost Structure for Change Impact Evaluation in Manufacturing Systems. Z. Wirtsch. Fabr. 2019, 114, 530–534. [Google Scholar] [CrossRef]
  40. Bauer, H.; Schönmann, A.; Reinhart, G. Approach for model-based change impact analysis in factory systems. In Proceedings of the 2017 IEEE International Systems Engineering Symposium (ISSE), Vienna, Austria, 11–13 October 2017; Volume 1, pp. 1–7. [Google Scholar] [CrossRef]
  41. Plehn, C.; Stein, F.; de Neufville, R.; Reinhart, G. Assessing the Impact of Changes and their Knock-on Effects in Manufacturing Systems. Procedia CIRP 2016, 57, 479–486. [Google Scholar] [CrossRef]
  42. Cichos, D.; Aurich, J.C. Planning and Controlling of Multiple, Parallel Engineering Changes in Manufacturing Systems. Procedia CIRP 2015, 33, 81–86. [Google Scholar] [CrossRef] [Green Version]
  43. Malak, R.C.; Aurich, J.C. Software Tool for Planning and Analyzing Engineering Changes in Manufacturing Systems. Procedia CIRP 2013, 12, 348–353. [Google Scholar] [CrossRef]
  44. Brönner, M. Decision-Support for Production Strategies for Developing Economies. Doctoral dissertation, Technical University of Munich, Munich, Germany, 2020. [Google Scholar]
  45. Krebs, P.; Reinhart, G. Evaluation of interconnected production sites taking into account multidimensional uncertainties. Prod. Eng. 2012, 6, 587–601. [Google Scholar] [CrossRef]
  46. Arabshahi, H.; Fazlollahtabar, H. Risk analysis for innovative activities in production systems using product opportunity gap concept. TQM J. 2019, 31, 1028–1048. [Google Scholar] [CrossRef]
  47. Eßer, G.; Brykczynski, G.; Hipp, C. Management von Produktionsinnovationen mit TREX. Z. Wirtsch. Fabr. 2016, 111, 671–675. [Google Scholar] [CrossRef]
  48. Liang, C.; Li, Q. Enterprise information system project selection with regard to BOCR. Int. J. Project Manage. 2008, 26, 810–820. [Google Scholar] [CrossRef]
  49. Granig, P. Innovationsbewertung: Potentialprognose und -steuerung durch Ertrags- und Risikosimulation. Ph.D. Dissertation, Universität Klagenfurt, Klagenfurt, Austria, 2005. [Google Scholar]
  50. Sambasivarao, K.V.; Deshmukh, S.G. A decision support system for selection and justification of advanced manufacturing technologies. Prod. Plann. Control 1997, 8, 270–284. [Google Scholar] [CrossRef]
  51. Schuh, G.; Scholz, P.; Ando, Y.-S. Assessing the application based utility potential of new technologies. In Proceedings of the 2020 IEEE Technology & Engineering Management Conference (TEMSCON), Novi, MI, USA, 3–6 June 2020; pp. 1–8. [Google Scholar] [CrossRef]
  52. Deutsches Institut für Normung. DIN 55350 Begriffe zum Qualitätsmanagement; Beuth: Berlin, Germany, 2021; pp. 1–48. [Google Scholar]
  53. Li, G. Research on the Investment Decision-Making on the Application of Advanced Manufacturing Technologies in Enterprises. Adv. Mater. Res. 2011, 323, 60–64. [Google Scholar] [CrossRef]
  54. Westkämper, E. Einführung in die Organisation der Produktion; Springer: Berlin/Heidelberg, Germany, 2006; ISBN 9783540260394. [Google Scholar]
  55. Erlach, K. Wertstromdesign: Der Weg zur schlanken Fabrik; Springer: Berlin/Heidelberg, Germany, 2020; ISBN 978-3-662-58906-9. [Google Scholar]
  56. Geisler, E. Creating Value with Science and Technology; QUORUM Books: Westport Connecticut, CT, USA, 2001; ISBN 978-1567204056. [Google Scholar]
  57. Deutsches Institut für Normung. DIN 8743 Verpackungsmaschinen und Verpackungsanlagen; Beuth: Berlin, Germany, 2014; pp. 1–26. [Google Scholar]
  58. Zäh, M.F.; Müller, N.; Prasch, M.; Sudhoff, W. Methodik zur Erhöhung der Wandlungsfähigkeit von Produktionssystemen. Z. Wirtsch. Fabr. 2004, 99, 173–177. [Google Scholar] [CrossRef]
  59. Sethi, A.; Sethi, S. Flexibility in manufacturing: A survey. Int. J. Flexible Manuf. Syst. 1990, 2, 289–328. [Google Scholar] [CrossRef]
  60. Korder, S.; Krauel, M.; Vernim, S.; Reinhart, G. Identification of workplace-related turnover predictors in production. Procedia CIRP 2021, 104, 1476–1481. [Google Scholar] [CrossRef]
  61. Deutsches Institut für Normung. DIN EN ISO 10075-1 Teil 1: Ergonomische Grundlagen Bezüglich Psychischer Arbeitsbelastung: Allgemeine Aspekte und Konzepte und Begriffe; Beuth: Berlin, Germany, 2018; pp. 1–19. [Google Scholar]
  62. Schneider, D.; Paul, M.; Vernim, S.; Zäh, M.F. Sustainability Assessment of Manufacturing Systems—A Review-Based Systematisation. Proceedings CARV 2022, 1, 1023–1030. [Google Scholar] [CrossRef]
  63. Cooper, R.G. Managing technology development projects. IEEE Eng. Manag. Rev. 2007, 35, 67. [Google Scholar] [CrossRef]
  64. Forno, A.J.D.; Pereira, F.A.; Forcellini, F.A.; Kipper, L.M. Value Stream Mapping: A study about the problems and challenges found in the literature from the past 15 years about application of Lean tools. Int. J. Adv. Manuf. Technol. 2014, 72, 779–790. [Google Scholar] [CrossRef]
  65. Knoll, D.; Reinhart, G.; Prüglmeier, M. Enabling value stream mapping for internal logistics using multidimensional process mining. Expert Syst. Appl. 2019, 124, 130–142. [Google Scholar] [CrossRef]
  66. Pfeffer, M. Bewertung von Wertströmen; Springer Fachmedien Wiesbaden: Wiesbaden, Germany, 2014. [Google Scholar]
  67. Spur, G.; Eßer, G.; Brykczynski, G. Risikomanagement produktionstechnischer Innovationen. Z. Wirtsch. Fabr. 2012, 107, 485–488. [Google Scholar] [CrossRef]
  68. Cagliano, A.C.; Grimaldi, S.; Rafele, C. Choosing project risk management techniques. A theoretical framework. J. Risk Res. 2015, 18, 232–248. [Google Scholar] [CrossRef] [Green Version]
  69. Spur, G.; Eßer, G. Produktionsinnovationen als unternehmerische Herausforderung. Z. Wirtsch. Fabr. 2010, 105, 763–766. [Google Scholar] [CrossRef]
  70. Adams, W.C. Conducting Semi-Structured Interviews. In Handbook of Practical Program Evaluation; Newcomer, K.E., Hatry, H.P., Wholey, J.S., Eds.; John Wiley & Sons, Inc: Hoboken, NJ, USA, 2015; pp. 492–505. [Google Scholar]
  71. Pfeffer, M.; Gottmann, J. Chancen nutzen in Krisenzeiten durch Produktionsoptimierung. Z. Wirtsch. Fabr. 2009, 104, 358–361. [Google Scholar] [CrossRef]
  72. Kern, S. Produktions-Controlling für Fertigungsinseln. Ph.D. Dissertation, Universität Saarbrücken, Saarbrücken, Germany, 1993. [Google Scholar]
  73. Gottmann, J.; Pfeffer, M.; Sihn, W. Process Oriented Production Evaluation. Procedia CIRP 2013, 12, 336–341. [Google Scholar] [CrossRef]
  74. Arnold, D.; Isermann, H.; Kuhn, A.; Tempelmeier, H.; Furmans, K. (Eds.) Handbuch Logistik; Springer: Berlin/Heidelberg, Germany, 2008; ISBN 9781281276100. [Google Scholar]
  75. Pérez-Fernández, L.; Sebastián, M.A.; González-Gaya, C. Methodology to Optimize Quality Costs in Manufacturing Based on Multi-Criteria Analysis and Lean Strategies. Appl. Sci. 2022, 12, 3295. [Google Scholar] [CrossRef]
  76. Dyckhoff, H.; Spengler, T.S. Produktionswirtschaft; Springer Berlin Heidelberg: Berlin/Heidelberg, Germany, 2010; ISBN 9783642136849. [Google Scholar]
  77. Palupski, R. Management von Beschaffung, Produktion und Absatz; Gabler Verlag: Wiesbaden, Germany, 2002. [Google Scholar]
  78. Liebermann, M. Sunk Costs. In The Palgrave Encyclopedia of Strategic Management; Augier, M., Teece, D.J., Eds.; Palgrave Macmillan: London, UK, 2016; pp. 1–3. [Google Scholar]
  79. Cagliano, A.C.; de Marco, A.; Grimaldi, S.; Rafele, C. An integrated approach to supply chain risk analysis. J. Risk Res. 2012, 15, 817–840. [Google Scholar] [CrossRef] [Green Version]
  80. Saaty, T.L.; Shang, J.S. An innovative orders-of-magnitude approach to AHP-based mutli-criteria decision making: Prioritizing divergent intangible humane acts. Eur. J. Oper. Res. 2011, 214, 703–715. [Google Scholar] [CrossRef]
  81. Yager, R.R. A procedure for ordering fuzzy subsets of the unit interval. Inf. Sci. 1981, 24, 143–161. [Google Scholar] [CrossRef]
  82. Wind, Y.; Saaty, T.L. Marketing Applications of the Analytic Hierarchy Process. Manag. Sci. 1980, 26, 641–658. [Google Scholar] [CrossRef]
  83. Goddard Space Flight Center. Risk Management Reporting GSFC-STD-0002; Goddard Space Flight Center: Greenbelt, MD, USA, 2009.
  84. Kusumawardani, R.P.; Agintiara, M. Application of Fuzzy AHP-TOPSIS Method for Decision Making in Human Resource Manager Selection Process. Procedia Comput. Sci. 2015, 72, 638–646. [Google Scholar] [CrossRef] [Green Version]
Applsci 13 03221 g001 550
Figure 1. Overview of the systematic literature review analysis approach and the results.
Figure 1. Overview of the systematic literature review analysis approach and the results.
Applsci 13 03221 g001
Applsci 13 03221 g002 550
Figure 2. Methodical procedure to derive a holistic assessment approach.
Figure 2. Methodical procedure to derive a holistic assessment approach.
Applsci 13 03221 g002
Applsci 13 03221 g003 550
Figure 3. Overview of the methodological derivation of the criteria and schematic structure of the criteria catalog for innovation project assessment.
Figure 3. Overview of the methodological derivation of the criteria and schematic structure of the criteria catalog for innovation project assessment.
Applsci 13 03221 g003
Applsci 13 03221 g004 550
Figure 4. Relevant criteria to assess the potential of an innovation project in manufacturing.
Figure 4. Relevant criteria to assess the potential of an innovation project in manufacturing.
Applsci 13 03221 g004
Applsci 13 03221 g005 550
Figure 5. Relevant criteria to assess the efforts of innovation projects in manufacturing.
Figure 5. Relevant criteria to assess the efforts of innovation projects in manufacturing.
Applsci 13 03221 g005
Applsci 13 03221 g006 550
Figure 6. Relevant criteria to assess the risks of innovation projects in manufacturing.
Figure 6. Relevant criteria to assess the risks of innovation projects in manufacturing.
Applsci 13 03221 g006
Applsci 13 03221 g007 550
Figure 7. Overview of the methodological derivation of the approach for the holistic assessment of innovation in manufacturing.
Figure 7. Overview of the methodological derivation of the approach for the holistic assessment of innovation in manufacturing.
Applsci 13 03221 g007
Applsci 13 03221 g008 550
Figure 8. Overview of the approach for the holistic assessment of innovation in manufacturing.
Figure 8. Overview of the approach for the holistic assessment of innovation in manufacturing.
Applsci 13 03221 g008
Applsci 13 03221 g009 550
Figure 9. Results of the use case application of the approach in the form of closest-to-ideal values and a portfolio analysis.
Figure 9. Results of the use case application of the approach in the form of closest-to-ideal values and a portfolio analysis.
Applsci 13 03221 g009
Table
Table 1. Inclusion criteria for the systematic literature review.
Table 1. Inclusion criteria for the systematic literature review.
Inclusion CriteriaDescription
Search termsManufacturing, Production, Process, Technology, Innovation, Assessment, Evaluation, Effort, Risk, Potential, Management
Subject areaDecision-making, Risk-assessment, Technology assessment, Decision support systems, Investments, Strategic planning, Performance assessment, Economics, Industrial management, Innovation, Cost–benefit analysis, Technological forecasting, Economic and social effects
Year1970–2022
LanguageEnglish, German
Table
Table 2. Result analysis for the investigated methodical approaches.
Table 2. Result analysis for the investigated methodical approaches.
Characteristics
FocusProcessInnovationCriteriaMethodApplication
TitleCitationManufacturingEarly PhaseHumanTechnologyOrganizationPotentialEffortRiskDecision SupportUtility AnalysisFuzzyAHPTOPSISIndustrial
Applicability
Production Processes Modeling for Identifying Technology Substitution Opportunities[3]Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i002Applsci 13 03221 i002Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001
Technology Assessment for Modular Product Platforms with Fuzzy Numbers[6] Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i002Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i003
Planning processes for advanced manufacturing technology by large American manufacturers[8]Applsci 13 03221 i001Applsci 13 03221 i002 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i002
A reference framework for the holistic evaluation of Industry 4.0 solutions for small- And medium-sized enterprises[13]Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i002 Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001
Identification and systematization of strategic technology demands in manufacturing[16]Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i002 Applsci 13 03221 i002Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001
Determining the strategic potential of technologies for manufacturing companies[29]Applsci 13 03221 i001Applsci 13 03221 i003 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001
Risk and potential evaluation of technologies in the early stages of the innovation process[30]Applsci 13 03221 i002Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i002
Potential-based Evaluation of Innovative Technologies in Production[31]Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i004
An Assessment Model for Production Innovation[32]Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i004
Risk calculations in the manufacturing technology selection process[33]Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i003
Fuzzy logic and evaluation of advanced technologies[34]Applsci 13 03221 i002Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001
Potential-based assessment of new technologies [ger.][35]Applsci 13 03221 i002Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i003
Holistic technology assessment. A model for the evaluation of different production technologies [ger.][36]Applsci 13 03221 i001Applsci 13 03221 i003 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i002 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i003
A filter system for technology evaluation and selection[37]Applsci 13 03221 i002Applsci 13 03221 i003 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i002Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001
Holistic and Evolutionary Technology Assessment [ger.][38]Applsci 13 03221 i002Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i004
Cost Structure for Change Impact Evaluation in Manufacturing Systems[39]Applsci 13 03221 i001Applsci 13 03221 i002Applsci 13 03221 i002Applsci 13 03221 i001Applsci 13 03221 i002 Applsci 13 03221 i001 Applsci 13 03221 i002 Applsci 13 03221 i001
Approach for model-based change impact analysis in factory systems[40]Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i002 Applsci 13 03221 i002
Assessing the Impact of Changes and their Knock-on Effects in Manufacturing Systems[41]Applsci 13 03221 i001Applsci 13 03221 i003Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i003
Planning and Controlling of Multiple, Parallel Engineering Changes in Manufacturing Systems[42]Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001
Software Tool for Planning and Analyzing Engineering Changes in Manufacturing Systems[43]Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i002 Applsci 13 03221 i002 Applsci 13 03221 i003
Decision-Support for Production Strategies for Developing Economies[44]Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001
Evaluation of interconnected production sites taking into account multidimensional uncertainties[45]Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001
Risk analysis for innovative activities in production systems using product opportunity gap concept[46]Applsci 13 03221 i002Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001
Management of production innovations with TREX [ger.][47]Applsci 13 03221 i001Applsci 13 03221 i003 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i002 Applsci 13 03221 i004
Enterprise information system project selection with regard to BOCR[48]Applsci 13 03221 i002Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001
Innovation assessment: Potential forecasting and control through yield and risk simulation [ger.][49]Applsci 13 03221 i002Applsci 13 03221 i002 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001
A decision support system for selection and justification of advanced manufacturing technologies[50]Applsci 13 03221 i001Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001 Applsci 13 03221 i001Applsci 13 03221 i001Applsci 13 03221 i001
Legend: Applsci 13 03221 i001
: fully fulfilled		Applsci 13 03221 i003
: well fulfilled		Applsci 13 03221 i002
: partly fulfilled		Applsci 13 03221 i004
: hardly fulfilled
Table
Table 3. Excerpt of the used sources for the derivation of assessment criteria.
Table 3. Excerpt of the used sources for the derivation of assessment criteria.
PotentialEffortRisk
TitleCitationCostQualityTimeFlexibilitySocialEcologyStrategyDevelopmentOrganizationStructureQualification
& Consulting		OpportunityProjectProductionExternal
Cost Structure for Change Impact Evaluation in Manufacturing Systems[39] XXXXX
Quality management [ger.][52]XX
Research on the Investment Decision-Making on the Application of Advanced Manufacturing Technologies in Enterprises[53]XXXX XX
Production Management. An introduction [ger.][54] X
Value stream design. The way to lean production [ger.][55]XXXX
Creating Value with Science and Technology[56] X
Packaging machines and packaging lines [ger.][57] X
Methodology for increasing the adaptability of production systems [ger.][58] X
Flexibility in manufacturing: A survey[59] X
Identification of workplace-related turnover predictors in production[60] X
Ergonomic principles regarding mental workload-General aspects and concepts and terms [ger.][61] X
Sustainability Assessment of Manufacturing Systems—A Review-Based Systematisation[62] X
Managing technology development projects[63] X
Value Stream Mapping: a study about the problems and challenges found in the literature from the past 15 years about application of Lean tools[64] X
Enabling value stream mapping for internal logistics using multidimensional process mining[65] X
Evaluation of value streams [ger.][66] XXXXX
Risk management of innovation in production [ger.][67] XXX
Legend:X: adressed[ger.]: german title translated
Table
Table 4. Use case application: criteria and weights.
Table 4. Use case application: criteria and weights.
Main
Criteria		CriteriaWeight (Local)Sub-
Criteria		Weight
(Local)		Detail
Criteria		Weight
(Local)		Weight
(Global)		Alternative
1		Alternative
2		Alternative
3
PotentialCost0.545Organizational costs0.193Warehouse planning and controlling1.0000.1050.0000.000−0.500
Production costs0.193Supply1.0000.1050.0000.0002.000
Logistic costs0.193Transport0.5000.053−3.000−1.500−3.000
Warehousing0.5000.053−0.5000.000−0.500
Quality costs0.422Internal defects0.4570.1050.0000.0000.000
Testing and inspection0.5430.1251.5001.5001.500
Time0.125Waiting and layover1.000Waiting and layover1.0000.125−0.7501.500−1.500
Changeability0.125Flexibility1.000Process0.3330.042−0.7500.000−1.500
Product mix0.3330.042−1.5000.000−1.500
Expansion0.3330.0420.0000.0000.000
Social0.125Physical stress0.500Harmful effects (long-term potential)1.0000.0630.0000.000−3.000
Psychological stress0.500Harmful effects (long-term potential)1.0000.063−0.7500.000−3.000
Ecology0.080Input1.000Operating supplies1.0000.080−0.5000.000−0.500
EffortFinancial
effort		1.000Structures0.750Buildings and spaces0.2500.1880.0000.0000.000
Utilities and tools0.2500.1881.0001.0002.000
IT systems (software, hardware)0.2500.1883.0000.0005.000
Conversion and installation0.2500.1881.0001.0002.000
Qualification and consulting0.250Supplier qualification1.0000.2503.0000.0003.000
RiskInternal risks1.000Production (utilization)1.000Exploitation1.0001.0002.0000.0000.000
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Gärtner, Q.; Ronco, E.; Cagliano, A.C.; Reinhart, G. Development of an Approach for the Holistic Assessment of Innovation Projects in Manufacturing Including Potential, Effort, and Risk Using a Systematic Literature Review and Expert Interviews. Appl. Sci. 2023, 13, 3221. https://doi.org/10.3390/app13053221

AMA Style

Gärtner Q, Ronco E, Cagliano AC, Reinhart G. Development of an Approach for the Holistic Assessment of Innovation Projects in Manufacturing Including Potential, Effort, and Risk Using a Systematic Literature Review and Expert Interviews. Applied Sciences. 2023; 13(5):3221. https://doi.org/10.3390/app13053221

Chicago/Turabian Style

Gärtner, Quirin, Ermanno Ronco, Anna Corinna Cagliano, and Gunther Reinhart. 2023. "Development of an Approach for the Holistic Assessment of Innovation Projects in Manufacturing Including Potential, Effort, and Risk Using a Systematic Literature Review and Expert Interviews" Applied Sciences 13, no. 5: 3221. https://doi.org/10.3390/app13053221

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop