Assessing the Content Quality of Industry Technology Roadmaps

Abstract

An industry technology roadmap (hereafter ITRM) is a strategic planning tool to predict the technologies and innovations demanded by the future market, allowing the industry to leverage capital and other investments in a range of alternative technologies and achieve sustainable development. To date, ITRM has been adopted by various global organizations in different industries; however, the majority of research has focused on roadmapping techniques only. Although success factors have been mentioned in some of the literature, little work has been conducted to assess the success of any ITRM. This research, therefore, developed an assessment model, including a theoretical and knowledge framework, assessment methods, and quantitative indices, to systematically assess the contents of an ITRM. We then used it to assess four global textile ITRMs. The assessment results led us to recommend five success factors of an effective ITRM: (1) methodological industry technology roadmapping; (2) a multi-organizational background; (3) systematic presentation of ITRM contents; (4) balanced contents for market and technology forces; and (5) appropriate databases. Compared with the success factors proposed in the previous literature, these five success factors are more practical for roadmap content construction.

1. Introduction

Industry technology roadmaps (hereafter ITRMs) have been widely used in the textile, semiconductor, energy and advanced materials industries, amongst others, for sustainable development [1,2,3,4]. There have been various techniques and approaches to roadmapping [5,6,7,8], but roadmaps yet well evaluated [9,10]. A small number of scholars have assessed the success of corporate-level technology roadmaps, in most of which success or failure was decided by the users’ perceptions and attitudes [11,12,13]. Recently, a few studies have initiated new assessment perspectives, such as the roadmapping process [14,15], roadmap achievements [16] and structuring contents [12,17]. Little research for roadmap assessment with a theoretical foundation and systematic methods can be found in the literature.

By focusing on content quality, an assessment model was developed to fill the research gap. First, a theoretical framework with a set of attributes for ITRM content assessment was established. To explore the inner relations among these attributes, a knowledge framework of ITRM contents was illustrated in the context of industrial developments. Second, content analysis was proposed as the assessment approach, with assessing procedures and measuring rubrics tailored for ITRMs. With the results of the content analysis, a measurement to quantify the overall content quality and any internal deficiency (between contents of market and technology forces) of an ITRM was developed. Last but not least, four global ITRMs for the textile industry were assessed using the proposed model. Relevant success factors of an effective ITRM in terms of content construction were recommended.

This work contributes significantly to the body of literature on the assessment of ITRMs. This is the first time that an original model with a theoretical foundation and assessment approach to assess the quality of ITRM content has been proposed. Within the development of the model, the intellectual understanding of relations among different attributes for an ITRM has been reflected upon. Moreover, a quantitative measurement has been developed for the internal assessment of an ITRM, which might initiate assessment innovation for the field.

2. Literature Review

Over the past three decades, technology roadmapping has evolved as a scientific methodology for strategic planning and resource management after Motorola developed its first technology roadmap in 1987 [18]. The publication was comprised of roadmapping approaches [19,20,21,22,23] and practical applications [24,25,26,27]. Recently, assessment methods for technology roadmaps have been marked in some literature as one of the issues left unaddressed [9,10].

A small but growing body of literature has assessed the success of roadmaps for companies and industries. Phaal et al. [28] proposed the fast-start process of technology roadmapping and applied it to ten selected companies. In order to assess the effectiveness of each case and make improvements to the processes, they conducted an assessment of the usefulness, functionality, and usability of the roadmap through the ratings of the company users. Ten success factors and corresponding barriers were then identified [7,11]. The study has commonly been taken as the first formal assessment of the success of roadmaps. The main limitation, however, has been that the theoretical foundation or application scopes have not yet been developed. Gerdsri et al. [12] proposed several measures for the success of a corporate-level roadmap. However, specific assessment methods of those measures have not been discussed further. Lee et al. [13] identified the factors that could improve the credibility of technology roadmaps and may have been the first time that a fundamental theory was developed for roadmap assessment. This research mainly focused on the communication attitudes and styles of the roadmapping team and the roadmap users. Jeffery et al. [16] developed nine metrics to assess the success of ITRMs in the renewable energy sector. Both the content architectures and implementation results were assessed by the authors. It can be taken as one of the first practices of ITRM assessment; however, there was a lack of theoretical support for the metrics and a scientific method for assessment.

The existing technology roadmaps are generally assessed in terms of the process or approach of roadmapping [13,15], the stakeholder’s comments [11,12], the actual impacts on the target areas [16], and the contents [12,17].

Major research gaps in the studies discussed so far include theoretical foundation and systematic assessment method. Without a solid theoretical foundation, it has been hard to judge to which level or scope of the assessment can be applied. Only systematic methods with feasible procedures can be repeated in different cases and are guaranteed to be reliable. Moreover, little work has been conducted other than a user satisfaction survey to assess the content quality of any roadmap.

3. Assessment Model

There are three types of ITRM assessments: (i) methodological (process and approach), (ii) efficiency (stakeholder’s comments and actual impacts on external target areas), and (iii) quality (contents). In this study, the assessment of ITRM is focused on the quality of contents within a roadmap only. For this research, a theoretical framework and a model to assess the quality of the contents of ITRM have been developed. The Cambridge Dictionary defines content as the ideas that are contained in a piece of writing [29]. Our model was designed to assess the contents contained within a roadmap, saying the words or information of texts, graphs, and other visual forms that analyze and predict the direction of market demands and technology development in the medium to long term [21,25].

3.1. Development Process

Technology roadmapping is a relatively flexible tool facilitating strategic planning for corporates, industries and even governmental sectors [8]. The roadmapping architectures, processes, and approaches have been widely studied [7,8,22,30,31,32]. However, only a few researchers focused on the construction of contents [7,12]. This limitation is a problem leading to the lack of assessments of the quality of contents per se [17].

There is little literature proposing systematic models for ITRM content quality assessment. In order to fill this research gap, the existing literature surrounding technology roadmaps at both corporate and industrial levels has been reviewed, and the key attributes involved have been summarized, as shown in Table 1. Using keywords including “technology roadmap”, “technology roadmapping”, “assessment”, “evaluation”, “success factor”, and “metric”, research searched from the databases (Web of Science, Scopus, and Google Scholars) has been reviewed. Some of the attributes have been renamed from the corresponding items in the literature. Definitions of these attributes will be further discussed in this section.

Table 1. Brief summary of attributes mentioned in previous research.

	Key Attribute Mentioned in Existing Literature	References	Terms in Figure 1 and Figure 2
1	Roadmapping Methods	[12,17,31]	Roadmapping Methods
2	Expert Panels	[12,31,33]	Expert Panels
3	Operation/Research Team	[12,16,33]	Research Team
4	Value Creation and Activity	[34,35]	Value Chain
5	Demand and Supply	[10,34,36]	Supply Chain
6	Science and Application	[10,17,34]	Innovation Chain
7	Targets or Goals	[7,32,34,35,36,37]	Industrial Goals
8	Key Technology or R&D Projects to develop	[17,35,36,38,39]	Key Technologies
9	Social, Economic, Political, Ecological, Resources, Culture, Lifestyle, Demographical	[17,34,36,39,40,41]	Macro Environment
10	Infrastructure Investment, Regulation, Policy, Education, Training	[17,34,36,39,41]	Micro Environment
11	Market Pull	[23,42,43,44,45]	Market Forces
12	Technological Push	[8,19,21,45,46]	Technology Forces

Table 1. Brief summary of attributes mentioned in previous research.

	Key Attribute Mentioned in Existing Literature	References	Terms in Figure 1 and Figure 2
1	Roadmapping Methods	[12,17,31]	Roadmapping Methods
2	Expert Panels	[12,31,33]	Expert Panels
3	Operation/Research Team	[12,16,33]	Research Team
4	Value Creation and Activity	[34,35]	Value Chain
5	Demand and Supply	[10,34,36]	Supply Chain
6	Science and Application	[10,17,34]	Innovation Chain
7	Targets or Goals	[7,32,34,35,36,37]	Industrial Goals
8	Key Technology or R&D Projects to develop	[17,35,36,38,39]	Key Technologies
9	Social, Economic, Political, Ecological, Resources, Culture, Lifestyle, Demographical	[17,34,36,39,40,41]	Macro Environment
10	Infrastructure Investment, Regulation, Policy, Education, Training	[17,34,36,39,41]	Micro Environment
11	Market Pull	[23,42,43,44,45]	Market Forces
12	Technological Push	[8,19,21,45,46]	Technology Forces

As only a few publications have indicated the key attributes of the content construction of ITRM, a theoretical framework is necessary so that the intellectual understanding of relations among different attributes of an ITRM can be reflected. Based on the renamed key attributes listed in Table 1, a theoretical framework for assessing the ITRM internal contents has been established (Figure 1). The assessment covers internal legitimacy and a knowledge framework of the roadmap contents.

Figure 1. Theoretical framework for internal assessment of ITRM.

Figure 1. Theoretical framework for internal assessment of ITRM.

Figure 2. Knowledge framework of ITRM internal assessment model.

Figure 2. Knowledge framework of ITRM internal assessment model.

3.2. Theoretical Framework

The assessment model for the ITRM is based on a theoretical framework comprising 12 key attributes that cover internal legitimacy and a knowledge framework of the roadmap contents.

3.2.1. Internal Legitimacy

Internal legitimacy analysis refers to an examination of whether the actions involved are appropriate and whether they meet the demands of the social system of norms, values, beliefs, and definitions [47]. The first three attributes in Table 1 (roadmapping methods, expert panels, and research team) were categorized as the internal legitimacy in this assessment model. The roadmapping methods can evaluate the credibility of the roadmapping design and process. The expert panels can affect the precision and timeliness of market and innovation forecasting, and the research team may affect the credibility of data collection and roadmap writing.

3.2.2. Knowledge Framework of Roadmap Contents

The key objective of the technology roadmap is to help an industry’s sustainable development by seeking synergy between profitability and sustainability. A successful ITRM can provide sufficient and valuable contents that help the targeted groups of stakeholders to capture the general industrial landscapes, opportunities, and threats and to make good use of the market and technology forces to achieve their objectives of future development.

The 4th to the 12th attributes in Table 1 were integrated into a knowledge framework of the ITRM internal assessment model, as presented in Figure 2. It simulates the mechanism for linking the contents of an industry technology roadmap within the context of industrial development. The development of an industry involves a complicated interaction between the value chain (Attribute 4) and the supply chain (Attribute 5) driven by the market forces; and between the supply chain (Attribute 5) and the innovation chain (Attribute 6) driven by the technology force. The industrial goals (Attribute 7) inform the value chain to affect the innovation chain. The key technologies (Attribute 8) inform the innovation chain to affect the value chain. The industrial goals are influenced by the macro environment (Attribute 9), while the key technology is influenced by the micro environment (Attribute 10).

• Market forces and technology forces

For industry roadmaps, market and technology forces are complicated and involved in larger contexts. Market forces mean the “aggregate influence of the buyers and sellers on prices and quantities of goods and services offered in a market” [48]. Technology forces are the influences of technology developments on customers, businesses, and society [49]. As stated by Phaal et al. [8], the technology force links the market and technology.

From the industrial perspective, the interaction among the value chain, supply chain, innovation chain, industrial goals, key technology, macro environment, and micro environment is a process in which to strike a balance between the market forces and technology forces. An imbalance exists between market forces and technology forces. The more well-matched market forces and technology forces are, the smaller the deficiency is and the better the industry develops.

A successful ITRM should provide effective information, predictions and plans from the perspectives of both market and technology to minimize the deficiency between the market forces and technology forces within the target industry. Therefore, the eight attributes (Figure 1) have been categorized into two groups, as follows.

Market forces: value chain (price and benefits influence), supply chain (influence of goods and service offering), macro environment (influence of buying power and consuming behavior), and micro environment (influence of productivity and sales behavior).

Technology forces: internal legitimacy (industrial influence of ITRM methodology and participants), supply chain (influence of technology for production), innovation chain (influence of technology status), industrial goals (influence of the strategies of technology development), and key technologies to develop (influence of the objectives of technology to develop).

The internal deficiency between the above two groups’ content quality is a key index to assess an ITRM’s quality. To date, the concept of internal deficiency has originally been established for roadmap assessment. The detailed measurement is presented in Section 3.2.2.

• Value chain, supply chain, and innovation chain

ITRM covers the general landscapes of the target industry with a complete set of the value chain, supply chain, and innovation chain at an industrial level not limited to one firm, not only upstream supply or downstream demand, and not limited to main technologies adopted or internal activities [34,35]. However, the examples have not been fully illustrated in the literature yet.

According to Porter [50], the value chain has primary activities and support functions. Primary activities involve inbound logistics for adding value by processing the product, outbound distribution to the points of sale, marketing and sales to brand it and promote it, as well as post-sales service. The support functions include the infrastructure, management systems, human resource, procurement in the required speed, accuracy, and quality. All these multi-linked functions are in the value system [51,52]. The value chain involves value creation and competition amongst participants within the same segment [53,54]. Maximum value creation is a common objective for all business units in an industry. Therefore, it is an essential attribute of the ITRM.

Supply chain refers to the supply and demand of goods and services across various segments in the longitudinal relationship between the upstream suppliers and downstream buyers. Previous studies [17,36] have defined the generic framework of the contents of the supply chain for ITRM, as in Figure 3.

The innovation chain involves a process (Figure 4) from the new knowledge of science to technological invention, industrial application, and commercialization which may have an influence on customer behavior and the market [55]. Following previous studies on various technology roadmaps [10,17,56], the innovation chain is also included in this study as a technology-driven attribute in the ITRM assessment model.

The value, supply, and innovation chains are interrelated. For example, when there is a disruptive innovation, new products are developed and supplied on a small scale. Due to the high demand and low supply, the price increases. Then the higher-value product will attract competing producers with a greater supply. The demand will then become stable, which will lower the price, and finally, the market will become mature.

• Industrial goals and key technology

As mentioned at the beginning of 3.2.2, the industrial goals inform the value chain, and the key technologies inform the innovation chain. Specific industrial goals are set to provide the products that the market treasures and develop technologies for future needs [7,21]. The industrial goals determine the focus and direction of the industry development. In order to realize industrial goals, key technologies are developed to improve production effectiveness or create potential new products. If the key technologies match with the industrial goals, the roadmap can provide effective routes to balance the market forces and technology forces.

• Macro and micro environments

The industrial goals are influenced by the macro environment, while the key technology is influenced by the micro environment. Roadmaps are contextual. Contents in the roadmaps cannot be isolated from the multi-interacted social contexts [37,39]. Based on the concepts of contextual analysis [57], the macro and micro environment are important in describing the contextual contents of an ITRM [39]. By combining the attributes of Porter’s Five Forces Model [50], the PESTLE Analysis [58] and related studies on roadmaps [34,39], six attributes: (i) economic factors, (ii) societal challenges, (iii) environmental protection, (iv) resources provision, (v) political, cultural and lifestyle, and (vi) population features are considered as the macro environment, and four attributes: (vii) infrastructures, (viii) governmental policies and international agreements, (ix) capital investment, and (x) education and training are regarded as comprising the micro environment in the knowledge framework of ITRM internal assessment model (Figure 2).

3.3. Assessment Procedures

3.3.1. Content Analysis

Content analysis is a methodological measurement applied to texts (or other meaningful matters) with respect to the contexts of their use [59,60]. Content analysis has been considered a reliable methodology [61,62] and has been widely adopted in management research [63,64]. It is applicable to both qualitative and quantitative data, as well as inductive and deductive analysis [65]. Although the processes are purpose-oriented and relatively flexible [59,63], the common feature of content analysis is that the words of the text (or information of other visual forms) are divided into categories [66,67]. Content analysis generally includes the steps as follows: (1) propose the research question, (2) select the unit of analysis, (3) select coding/categorization scheme, (4) conduct measurement, (5) reliability check of results, and (6) report results [59,65,68,69].

Following the steps of content analysis, four experts who hold doctoral degrees and have published in roadmapping were invited to assess the contents of the ITRM. They were from academic, industrial, or governmental departments.

The contents were coded based on the 24 subattributes, and the experts rated the quality of the coded contents using a five-point Likert scale (1 means a lack of relevant information, 3 means relevant information presented, and 5 means relevant information fully presented with critical analysis and solid references).

3.3.2. Quality Measurement

The quality of the contents of different roadmaps can be presented in terms of the overall quality score (Q_i) and the internal deficiency index (D_I).

First of all, the experts’ scores are transferred into ranks. The scores for ITRMs are collected from invited experts by content analysis which is usually in a small sample size and distribution-free. Therefore, the nonparametric test should be chosen [70]. The ranks, which are used to perform a nonparametric test, are adopted in this research, because the difference between adjacent scores for each expert may not necessarily be the same [71].

The method of assigning ranks is to order the data of the same investigation group from smallest to largest. The lowest score is assigned a rank of 1, the next lowest one a rank of 2, and the largest score is assigned a rank of the total number of scores. For each sub-attribute in content analysis, the largest scores are 16 (four experts rated each of the four ITRMs, and the total number of scores is sixteen). When there are ties for the same sub-attribute, the average rank of the ties is assigned to each [70,71]. In using this method, scores of each sub-attribute for all the four ITRMs rated by different experts are assigned by ranking, which makes their comparisons meaningful.

• Quality Scores

After assigning ranks, the average of each key attribute’s quality score (Q_i) is calculated for the ranks of different sub-attributes (Equation (1)):

$$Q_i=\sum x_n/m$$

(1)

where Q_i is the quality score of the ith key attribute (listed in Figure 1); x_n means the rank of each sub-attribute for the ith key attribute; and m means the total number of corresponding sub-attributes for the ith key attribute.

With the results of Q_i, the quality scores of market-force-related attributes, technology-force-related attributes, and the overall quality score are calculated respectively by Equations (2)–(4):

Q_MF = (Q₂ + Q₃ + Q₇ + Q₈)/4

(2)

where Q_MF is the quality score of market-force-related attributes; Q₂ is the quality score of the value chain; Q₃ is the quality score of the supply chain; Q₇ is the quality score of macro environment analysis; Q₈ is the quality score of micro environment analysis.

Q_TF = (Q₁ + Q₃ + Q₄ + Q₅ + Q₆)/5

(3)

where Q_TF is the quality score of technology-force-related attributes; Q₁ is the quality score of internal legitimacy; Q₃ is the quality score of the supply chain; Q₄ is the quality score of innovation chain; Q₅ is the quality score of industrial goals; Q₈ is the quality score of key technology.

$$Q_O=\sum Q_i/8$$

(4)

where Q_O is the overall quality score of an ITRM.

2.

Internal deficiency index

To compare the quality of the contents of the market forces and technology forces, internal deficiency (D_I) is calculated by the absolute value of the difference between their quality scores in Equation (5):

D_I = |Q_MF − Q_TF|

(5)

In order to assess the deficiency in the quality scores relative to the average, a relative internal deficiency (R_DI) is also calculated (Equation (6)):

$$R_{DI}=D_I/\left[\left(Q_{MF}+Q_{TF}\right)/2\right]\times 100\%$$

(6)

4. Assessment of Four ITRMs

The developed theoretical framework (Figure 1) and knowledge framework (Figure 2) was used to assess the internal quality of the contents of four textile ITRMs at an industrial level, as shown in

Table 2. The four selected roadmaps for case study analysis.

Year	Roadmaps	Origin	Author Type
2007	Mapping of evidence on sustainable development impacts that occur in the life cycles of clothing [41]	UK	Government
2008	Technology roadmap for Canadian textile industry [72]	Canada	Industry
2009	Industry technology roadmap for the flushable pre-moistened nonwoven wipes industry [38]	US	Academia
2010	Development of Technology Roadmap for Guangdong Textile and Clothing Industry [36]	China	Academia/Industry/Government

. The developers of these four roadmaps include academia, industrialists, and government officers. Their contents were analyzed, and the attributes that contributed to good content quality were identified.

The following six steps of content analysis (Section 3.2.1) were used in the assessment of the four roadmaps.

• Step 1: Propose the research question

The research question is whether the newly developed ITRM internal assessment model is feasible to assess the quality of contents in four selected roadmaps.

• Step 2: Select the unit of analysis

Four global ITRMs for textiles were content analyzed to investigate their success in content and seek certain attributes bearing a relationship to content quality. Four selected ITRMs are shown in Table 2. All of the four roadmaps are industrial level and written by authors from academia, industry, and government.

• Step 3: Develop the coding/categorization scheme

Inductive content analysis is recommended when the knowledge is fragmented, while deductive content analysis is recommended for theory or model testing [65,73,74]. For the internal assessment of ITRMs, deductive content analysis was adopted for the theoretical framework. The categorization scheme for deductive content analysis includes a categorization matrix and data coding [65].

Based on the theoretical framework (Figure 1) and the knowledge framework of the ITRM contents assessment model (Figure 2), the categorization matrix of assessment criteria is shown in

Table 3. Categorization matrix of assessment criteria.

Main Criteria	Key Attribute	Generic Attribute	Assessment Items
Internal Legitimacy	1. Internal Legitimacy	Roadmapping methods	definitions for ITRM, roadmapping techniques, and process
		Expert panel	authority level of expert panel
		Research team	authority level of research team
Knowledge framework of ITRM contents	2. Value Chain	Value chain	price index and value addition for each industrial segment
	3. Supply Chain	Supply chain	supply and demand status among industrial segments
	4. Innovation Chain	Innovation chain	technology innovation capacity and innovation clusters among industrial segments
	5. Industrial Goals	Industrial gaps	main gaps for industry development
		Research areas	target research areas
		Development strategy	development strategy to achieve industrial goals
		Key projects	key projects to develop
	6. Key Technology	Technology barrier	main technological barriers to the industry
		Objectives of technology	objectives of technologies to develop
		Current status of technology	current status of technologies, such as in mature, advanced or frontier levels
		Key technologies to develop	key technologies to develop in different terms, i.e., short-term, medium-term or long-term
	7. Macro Environment	Economics factor	external economic threats/shocks for the industry from structure, conduct and performance
		Societal challenges	social effects that can influence the industry directly, such as labor force and residential living level etc.
		Environmental protection	impacts on environment and responding solutions for ecological protection and balance
		Resources provision	supply, demands and distributions of resources of raw materials, energy and others
		Politics, cultures and lifestyles	key political, cultural and lifestyle elements and their changing trends
		Population feature	key structural features and changing trends of human population
	8. Micro Environment	Infrastructures	current status and future demands of infrastructural facilities and platforms
		Government policies and international agreements	existing international agreements and policies in the nations or regions
		Capital investment	structure and trends of capital investments from governments and industries/regions
		Education and training	status and demands of higher education and professional training

. There is no time dimension in these categories, and they can be applied at any time point to meet varied purposes.

By using the newly developed categorization matrix and the rules of data coding, each of the four ITRMs was reviewed for content and coded for correspondence with or exemplification of the identified categories [65,75]. Before the formal data coding by the experts, a pilot study was conducted to ensure that the corresponding or exemplified contents in each ITRM could be coded into the categories. Then the full contents of the four selected ITRMs and the categorization matrix were sent to the experts, with written instructions and follow-up explanations via video calls, for them to code the relevant contents into the corresponding category for each ITRM.

• Step 4: Measurement

Delphi is applied for this step. Scoring with a rubric is more reliable than scoring without it [76] because different experts may use different criteria in rating the ITRM contents. Therefore, the rubrics (

Table A1. Rubrics for content assessment.

No.	Internal Assessment Criterion		Rubrics
			1	2	3	4	5
1	Internal Legitimacy	Definitions of TRM, methodology, roadmap development process	not mentioned nor stated clearly	unclear concepts or methodologies	stated clearly	stated clearly with detailed plan	stated clearly with detailed plan and meet scientifically rigorous requirements with references and critical analysis
		Level of authority of expert panels	No reputable professionals from academia or industry	Some reputable professionals from academia or industry from local and regional community	Balanced reputable professionals from academia and industry from local and regional community	Balanced reputable professionals from academia and industry from local and regional and national community	Balanced reputable professionals from academia and industry from local, regional, national and international community
		Level of authority of research team	No reputable professionals from academia or industry	Some reputable professionals from academia or industry from local and regional community	Balanced reputable professionals from academia and industry from local and regional community	Balanced reputable professionals from academia and industry from local, regional, national community	Balanced reputable professionals from academia and industry from local, regional, national and international community
2	Value Chain	The price index and value addition for each segment, including raw materials, fabric processing, design, clothes processing, marketing and recycle	not identified nor analyzed	partially identified and analyzed	systematically identified and analyzed	systematically identified and analyzed with convincing evidence	systematically identified and analyzed with convincing evidence and sound references
3	Supply Chain	The supply and demand status among segments, such as leading actors, clusters, industrial scale, production capacity, logistics etc.	not identified nor analyzed	partially identified and analyzed	systematically identified and analyzed	systematically identified and analyzed with convincing evidence	systematically identified and analyzed with convincing evidence and sound references
4	Innovation Chain	The technology innovation capacity and innovation clusters among segments, such as innovation input, R&D power of industry and academia, innovative projects, industrial clusters, cooperation ways etc.	not identified nor analyzed	partially identified and analyzed	systematically identified and analyzed	systematically identified and analyzed with convincing evidence	systematically identified and analyzed with convincing evidence and sound references
5	Industrial Goals	Identification of industrial gap	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		Identification of research areas	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		Identification of development strategy	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		Identification of key projects	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
6	Key Technology	Identification of technology barrier/gap	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		Identification of objectives of technology	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		Identification of current status of technology	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		Identification of key technologies to be developed in short, medium and long terms	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
7	Macro environment	The external economic threats/shocks for business firms from structure, conduct and performance aspects	not identified and analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		The societal challenges that can influence the industry directly, including industrial basis, labour force, geographical distributions of sectors, residential living level and requirements etc.	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		The impacts of industrial chains on environment and requirements and solutions for ecological protection and balance	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		The resources of raw materials, energy, water and others, their supply, demands, distributions and regions of origin	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		The key political, cultural and lifestyle elements and their changing trends	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		The key structural features and changing trends of human population	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
8	Micro environment	The current status and future demands of infrastructural facilities and platforms, including platforms for product, information and finance, quality control and management and transport facilities, etc.	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		Existing policies and relevant international agreements in the nations/regions	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		The structure and trends of capital investments from governments and industries/regions	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references
		The demands and trends of training and educational resources, and key effects of higher education and professional training that can influence the industry directly	not identified nor analyzed	partially identified and analyzed	thoroughly identified and analyzed	thoroughly identified and analyzed with convincing evidence	thoroughly identified and analyzed with convincing evidence and sound references

) were developed and provided for them to assess specific contents of the ITRMs. The score ranges from 1 (bad performance) to 5 (good performance) for each category. The four experts coded the ITRM contents into the categorization matrix and assessed each category alongside the provided rubrics.

Table 4. Information of the four invited experts.

No	Expertise	Background	Origin
1	Material science and textile innovation management	Academia	UK
2	Textile testing technology and textile technology industrialization	Academia	Hong Kong
3	Textile engineering and textile project management	Industry Association	Hong Kong
4	Computing science, simulation and visualization of textile materials	Industry	Mainland China

shows the information of the four invited experts.

• Step 5: Reliability Check

Interrater reliability refers to the extent to which the independent raters agree on the coding and rating of the contents in the same categorization/coding scheme [77]. Interrater reliability is widely accepted as the standard measure for research quality [78,79], and it is a critical component of content analysis. In this research, the average pairwise percentage agreement [80] and Cohen’s Kappa index [79,81,82] were adopted to determine interrater reliability. The average percentage agreement was 73.96% and is considered reliable [83]. Cohen’s Kappa was 0.625 and regarded as substantially reliable [84].

• Step 6: Result Report

The experts’ scores, as nonparametric data, were transferred into ranks (Section 3.2.2). The results are reported in the following section in detail.

5. Results

5.1. Content Analysis

5.1.1. Internal Legitimacy

The median and range of the ranks for the three sub-attributes of internal legitimacy for the four ITRMs are presented below in

Table 5. Results of ranks (median and range) for sub-attributes of internal legitimacy.

Sub-Attribute	UK	Canada	US	China
Roadmapping methods	7.50 (6.50)	7.50 (0.00)	15.00 (7.50)	7.50 (0.00)
Expert panel	11.00 (10.00)	3.50 (7.50)	11.00 (7.50)	11.00 (0.00)
Research team	6.25 (10.50)	9.00 (0.00)	3.50 (0.00)	14.00 (0.00)
Overall score of internal legitimacy	8.25 (9.00)	6.67 (2.50)	8.58 (2.50)	10.83 (0.00)

.

The Canadian roadmap, as shown in Figure 5, among the four ITRMs rated, received the lowest quality score, 6.67 out of 16, for internal legitimacy, since it failed to address the definition and methodology of roadmapping and only invited nine stakeholders from Canadian textile companies as expert panels. The ITRM for the US received the best rating for roadmapping methods but the worst rating for the authority level of the research team. The Chinese ITRM was ranked the highest for the internal legitimacy average due to the diversified and reputable members in both the expert panel and research team. The UK ITRM received a score of 8.25 out of 16, and its main weakness was a lack of reputable participants from academia and governmental department.

5.1.2. Value Chain, Supply Chain, and Innovation Chain

From the original expert scores, the rating of the value chain for all four ITRMs was not satisfactory. In the US and Chinese ITRMs, information on the value chain was fragmented and coded from the sections of supply and innovation chains.

The ranking order for the quality of the supply chain was Canadian, Chinese, the UK, and the US ITRM. Over ten years, it is possible to collect data and information about the supply and demand of industry from various databases and websites. The main challenge is how to filter and analyze a huge amount of information.

In the ratings for the innovation chain, the US ITRM ranked the lowest because it only conducted a patent search for technology and innovation analysis. The contents of the innovation chain had been taken as an important attribute for roadmaps.

5.1.3. Industrial Goals and Key Technology

The median and range for sub-attributes of industrial goals and the key technology for the four ITRMs are presented in

Table 6. Results of ranks (median and range) for the sub-attributes of industrial goals and key technology.

Key Attribute	Sub-Attribute	UK	Canada	US	China
Industrial goals	Industrial gaps	6.00 (7.50)	6.00 (0.00)	6.00 (0.00)	13.50 (2.50)
	Research areas	6.00 (7.50)	6.00 (0.00)	6.00 (0.00)	13.50 (2.50)
	Development strategy	8.50 (0.00)	8.50 (0.00)	8.50 (0.00)	8.50 (0.00)
	Key projects	9.00 (8.00)	5.00 (0.00)	5.00 (8.00)	13.00 (0.00)
Key technology	Technology barrier	10.50 (13.50)	10.50 (0.00)	2.50 (2.50)	10.50 (0.00)
	Objectives of technology	9.00 (0.00)	9.00 (0.00)	5.25 (7.50)	9.00 (7.00)
	Current status of technology	10.50 (6.00)	7.50 (5.00)	2.50 (5.00)	13.50 (0.00)
	Key technologies to develop	7.50 (8.00)	7.50 (0.00)	7.50 (0.00)	7.50 (8.00)

. The Chinese and UK ITRMs ranked the top two for the key projects and current status of technology, showing the benefits of using diversified databases.

5.1.4. Macro and Micro Environment

The median and range for the sub-attributes of macro and micro environments for the four ITRMs are shown in

Table 7. Results of ranks (median and range) for the sub-attributes of macro and micro environments.

Key Attribute	Sub-Attribute	UK	Canada	US	China
Macro environment	Economics Factor	9.75 (7.50)	6.00 (0.00)	6.00 (5.00)	13.50 (0.00)
	Societal Challenges	7.25 (7.50)	11.00 (7.50)	3.50 (7.50)	11.00 (5.00)
	Environmental Protection	10.00 (14.00)	4.50 (5.00)	7.00 (0.00)	13.00 (0.00)
	Resources Provision	12.00 (11.50)	8.25 (7.50)	4.50 (7.50)	8.25 (7.50)
	Politics, Cultures and Lifestyles	9.50 (0.00)	9.50 (8.00)	9.50 (8.00)	9.50 (0.00)
	Human Population Feature	7.25 (10.50)	13.50 (0.00)	5.00 (0.00)	8.50 (5.00)
Micro environment	Infrastructures	7.50 (12.00)	14.50 (3.00)	2.50 (5.00)	7.50 (4.00)
	Policies and Agreements	11.00 (14.50)	11.00 (6.50)	4.50 (3.00)	11.00 (0.00)
	Capital Investment	8.50 (11.00)	14.00 (0.00)	8.50 (5.50)	3.00 (5.50)
	Education and Training	11.00 (7.50)	11.00 (0.00)	3.50 (2.50)	11.00 (0.00)

.

The ITRMs for the UK, Canada and China received both very good and unsatisfactory scores for different sub-attributes, and the US ITRM was rated with quite low scores for most of these attributes. This result proved that the content construction of an ITRM had not been well developed yet.

5.2. Quality Measurement

5.2.1. Quality Scores

After analyzing the quality of each sub-attribute (Section 5.1.1, Section 5.1.2 , Section 5.1.3 and Section 5.1.4), the newly proposed methods and equations (Section 3.2.2) will also be used to analyze the overall quality of the four ITRMs. The quality scores (Q_i) of eight key attributes for each ITRM are illustrated in a radar diagram for comparison in Figure 5.

The area of Canadian ITRM was the largest because it ranked the highest in the value chain, supply chain, innovation chain, and micro environment analysis. However, for the other two attributes—internal legitimacy and industrial goals—it ranked the lowest. Amongst the four ITRMs, the US one ranked the worst and achieved the smallest area in Figure 5. It ranked the lowest in seven key attributes out of eight, except internal legitimacy. The Chinese ITRM had the best quality scores of internal legitimacy, industrial goals, key technology, and macro environment analysis. The UK ITRM ranked in the medium for all the key attributes.

Figure 6 presents the quality scores (median and range) of the market force and technology forces in the four ITRMs.

The Canadian ITRM received the best quality score for market forces, and China’s ITRM received the best score for technology forces. Moreover, the UK and Canadian ITRMs had better quality scores for market forces than for technology forces, and the US and China ITRMs scored better for technology forces than for market forces.

5.2.2. Internal Deficiency Index

The calculated results (median and range) of internal deficiency and relative internal deficiency are plotted in Figure 7.

The values of internal deficiency as well as the relative internal deficiency, were bigger in the Canadian and Chinese ITRMs, although they ranked top two for overall quality. The bars show that the contents of market forces and technology forces were not well developed in balance in the four ITRMs examined.

The UK ITRM, initiated by the government department, collected sufficient information on the market and technology status for the industry from different databases of industry, academia, and government, but it only presented second-hand sources without incorporating new knowledge from the roadmapping processes.

The Canadian ITRM, initiated by a national industrial association, performed very well on the information and analysis of industrial statuses, such as value chain, supply chain, innovation chain, and micro environmental analysis; however, its strategic decisions in close relation to technology forces, such as industrial goals and key technology to develop, needed further improvement.

The US roadmap, as an academic thesis, had strength in methodology, but the analysis of the industrial status and strategic decisions was not sufficient. Although the relative internal deficiency of the US ITRM was the smallest, it did not seem to be the result of an awareness of the balanced development of market and technology forces due to the poor ranks for the content quality of both market and technology forces.

The Chinese roadmap, as a cooperative work of industry, academia, and government, generally had good ratings across various attributes except for the value chain. With opposite results to the Canadian ITRM, the Chinese ITRM received good rankings on key attributes in close relation to technology forces rather than market forces.

6. Discussion

According to the assessments of the four ITRMs investigated, the contents in all four ITRMs were not systematically organized, and some relevant attributes were not mentioned or only partially covered. It revealed that there was a lack of systematic framework for ITRM content presentation, and the authors might only present the contents with easy access.

Moreover, different types of organizations focused on different attributes of an ITRM’s contents. For example, the government department (UK roadmap) paid more attention to macro environmental analysis, such as environmental protection, resources, and policies, while the industrial association (Canadian roadmap) emphasized attributes in relation to the conditions for realizing industrial goals, such as infrastructure, capital investment, and education and training. The organizational backgrounds also had an influence on the content quality. For industry-status-related contents, such as value, supply and innovation chains, and micro environment analysis, the research team and expert panels from the industry (Canadian roadmap) did the best job because they had more direct and sensitive judgments on actual industrial statuses. For technology-planning-related contents and macro environment analysis, research teams and expert panels from multiple organizations in academia, industry and the government department had better performance because multi-organizational backgrounds could focus on the overall situations and generate collaborative opinions across various organizations rather than the interests of particular entities.

Different data sources also affected the quality of information collection and analysis. The Canadian roadmap used the database of the Canadian Textile Industry Association as a source, and the ITRM attributes relating to industrial development, like infrastructures, capital investment, and educational human resources, had better scores. The Chinese roadmap used the database “Web of Knowledge” and “SciFinder” to explore the updated status of technology and innovation, and it received high scores on the relevant attributes. The UK roadmap was created through desk research on second-hand information but still received medium scores on all the attributes.

7. Recommendations

7.1. Five Success Factors

Based on the assessment framework and results, the following five success factors are recommended to aid in the organizing and writing of ITRM, particularly for the textile industry. With this experience, more case studies in other industries can be performed to widen the applications of the proposed internal assessment model.

(1)

Methodology of industry technology roadmapping

As a future development planning tool, industry technology roadmapping has been developed, amended, and applied in various areas with different objectives. It is necessary to choose suitable methods for different levels (industry or corporate), roadmapping techniques and processes to adapt different objectives for roadmaps.

(2)

Multi-organizational background

A research team and expert panels containing renowned experts in balanced technology, business and governing areas of the target industry worldwide are recommended. Usually, ideas about the overall industry trends as well as specific technology development, can be generated from expert forums, workshops, interviews, etc., depending on the decisions made by the research team.

(3)

Systematic presentation of ITRM contents

It is important for the ITRM to have a clear and comprehensive analysis of the current status of the targeted industry to ensure that the audience gets updated knowledge of the development of the industry. Presenting data and analysis from three aspects—value, supply, and innovation chains—can provide a dynamic vision of the entire industry rather than only fragmented information.

Setting industrial goals and prioritizing key technologies/barriers for various developing periods are crucial tasks for ITRMs, and should be carefully generated to cover a range of industrial participants (academia, industry, and government) to ensure implementation effectiveness.

Since industrial development cannot be isolated from the macro environment, to avoid predictable risks and utilize potential advantages, it is necessary to analyze the economic factors, societal challenges, environmental protection, resource provision, politics and culture, and population features. In order to connect the macro environment with the target industrial status, the micro environment, including infrastructure, policies, agreements, capital investment, and education and training, is also recommended for investigation.

(4)

Balanced content for market and technology forces

The core task for an ITRM is to minimize the gaps between the market demands and the technology and innovation development. Therefore, the contents for both current and potential market forces and technology forces should be developed in balance. The more well-matched market and technology forces are, the smaller the internal deficiency is and the better the target industry develops.

(5)

Appropriate databases

Using corresponding databases for collecting different kinds of information is helpful in improving the quality of presented data and analysis. For example, industry association databases, customer databases, national statistical yearbooks, governmental statistics, and international organizations’ databases can be valuable for industrial status and macro and micro environmental analysis. Scientific tools, such as “Web of Science”, “Web of Knowledge”, “Scopus”, “SciFinder”, and “Google Scholar”, can be adapted for technology and innovation analysis.

7.2. Advantages over the Previous Suggestions of Success Factors

Phaal et al. [7] identified ten success factors, including a “clear and effective process for developing ITRM”, “effective tools/techniques/methods”, and “right people/functions were involved” that are similar to our first two recommendations. Their success factors were generated from the surveys on the roadmapping process, while our success factors were revealed from the results of a systematic assessment of four actual ITRMs.

Jeffrey et al. [16] suggested eight success factors for ITRM based on the assessment results of four ITRMs in the renewable energy sector. There were also two success factors, “having the right people/author in place” and “robust method for developing the roadmap”, similar to our first two recommendations. People with multi-organizational backgrounds were mentioned in both studies, but Jeffery et al. did not mention the expert panels.

The recommended success factors (1) and (2) also agree with that proposed by the previous research [7,12,16]. The success factors (3), (4) and (5) are newly emerged from the analysis of internal quality assessment results of four ITRMs in textiles. The advantage is that the recommended five success factors in this research are developed based on the internal quality of the ITRM contents, so they can provide more practical guidelines for ITRM content construction.

8. Conclusions

The assessment of the success of an industry technology roadmap is a complex process. In order to maximize the effectiveness of an ITRM, the content of an ITRM should be elucidated. This can lead to more effective processes and roadmapping techniques.

The essence of the proposed internal assessment model is a theoretical framework connecting different content attributes. Besides assessment, this framework can be used for the systematic presentation of ITRM contents. The concept of internal deficiency was originally developed to emphasize the balanced integration between market forces and technology forces in an ITRM, and relevant indices have also been developed to open a new chapter in the quantitative assessment of roadmaps. The findings of this research can help the practitioner to develop an effective ITRM with clear guidelines and a knowledge framework to achieve the industry’s sustainable development.

Focusing on content quality, this proposed model is limited to an internal assessment of roadmaps. Future studies of external assessment on the actual performance of roadmaps are highly recommended.