A Maturity Model for Diagnosing the Capabilities of Smart Factory Solution Suppliers and Its Pilot Application

Abstract

For the successful and sustainable deployment and diffusion of smart factories, both the capabilities of the adopters who operate the factories and the capabilities of the suppliers who supply information technology and equipment play very important roles. However, since the existing models for diagnosing the capabilities of smart factories are mainly focused on evaluating the capabilities of the manufacturing companies themselves, such as technological capabilities and digital transformation, there are not many models that diagnose the supply capabilities of suppliers from the perspective of demand companies. Unlike models that diagnose the level of smart factories, when diagnosing the capabilities of suppliers, various factors such as supply experience and management capabilities must be comprehensively evaluated in addition to the capabilities of the company itself. Therefore, this study proposes a new model to diagnose the capabilities of suppliers from the perspective of adopters who want to build smart factories and verifies the validity of the model by applying the model for a pilot diagnosis for 32 suppliers. In addition, based on the survey results obtained from both adopters and suppliers participating in the pilot diagnoses, this study proposes an institutionalization plan for capability diagnosis.

1. Introduction

With the advancement of information technology in the era of the fourth industrial revolution, represented by artificial intelligence, the Internet of Things, and big data, the business environment of manufacturing companies is changing faster than ever before. Companies are finding it difficult to survive without digital transformation, and factories that produce goods for companies are also finding it difficult to survive in an infinitely competitive market unless they utilize information technology such as predictive analysis and cyber-physical systems to increase productivity and innovation [1].

For this reason, since the mid-2010s, the manufacturing industry, especially manufacturing plants, has been working to leverage digital technologies to integrate, automate, and intelligently manage processes in product planning, design, production, transportation, inventory control, and sales management to increase manufacturing productivity and agility to quickly respond to changing markets, and to create a safe, human-centered manufacturing environment, under the name of ‘smart factory’. Compared to traditional factories, the key difference of smart factories is the use of innovative techniques such as the Internet of Things and machine learning with the power of technological advances.

In this respect, these smart factory innovations require many specialized skills. In an ideal world, each company that builds and operates a smart factory would internalize relevant knowledge and skills to design, build, and continuously operate and improve the appropriate factory for the company. But for many companies, except for some large enterprises, it is not feasible to have all the relevant skills in addition to industry expertise and technology.

Therefore, smart factory solution suppliers with specialized capabilities have emerged to serve these companies, analyzing the environment of the factory and providing services from planning to the design, development, construction, operation, and maintenance of smart factories suitable for each factory. Since 2014, the Korean government has implemented a program to support smart factory innovation by connecting competent suppliers with small and medium-sized enterprises (SMEs) that can improve their competitiveness by adopting smart factory solutions. Through this program, many SMEs have succeeded in converting their existing factories into smart factories, and smart factory solution suppliers have been able to grow as they continue to build their capabilities.

However, there are several problems with this supplier-centric approach to smart factory transformation, one of which is that there is a lack of information on the core technology areas and capabilities of suppliers. In addition, the information available to demanding companies, i.e., adopters, may not be standardized, making it difficult for adopters to find the most suitable and capable suppliers for implementing smart factories. According to the survey conducted in this study, 70.5% of the companies that have built smart factories through specialized suppliers said that they would like to have information on the objective capabilities of suppliers, and 36.5% of the companies that provide smart factory solutions said that they would prefer to have their capabilities objectively evaluated.

In both academia and practice, assessment models exist to evaluate a firm’s capabilities or competencies in smart factory-related operations, technologies, and practices [2,3,4,5,6,7,8,9]. Some evaluation models measure a firm’s digital transformational competency, which is fundamentally related to its smart factory capabilities [10,11,12,13,14,15,16]. However, these models are designed for firms operating smart factories, not for smart factory solution suppliers. To the best of our knowledge, there is no assessment model to evaluate the capability or competency of a smart factory solution supplier. Although some competency areas may overlap, there are unique and more demanding capabilities required for suppliers. These include, but are not limited to, maintenance and support, project management, technology integration, communications, etc. Existing models may be used to evaluate some aspects of a supplier’s smart factory capabilities but cannot provide a full spectrum of capabilities needed by the supplier.

Therefore, this study aims to develop a model that can objectively evaluate the capabilities of smart factory suppliers and propose an evaluation process to systematically operate them in order to make corporate smart factory innovation more efficient and increase the transparency of the smart factory construction market. The proposed model is based on the organizational maturity concept [17], widely utilized as an underlying framework for numerous capability and competency models. These models are designed to evaluate the processes or aspects of an organization and provide structured guidelines for continuous improvement and innovation [18,19]. Following the most accepted convention, we use the five maturity levels to represent the overall capability level of a smart factory solution supplier. We provide detailed evaluation criteria and processes, including the development of appraisal auditors akin to CMMI [20] and other appraisal models.

Furthermore, this study conducts a pilot application of the proposed model with 32 suppliers to check if the criteria and evaluation are appropriate. Lastly, based on the survey results obtained from the participants in the pilot, we suggest ways to improve and institutionalize the proposed maturity model and evaluation process.

This paper contributes to the literature on organizational maturity models by providing a maturity model to holistically evaluate the capabilities of smart factory solution suppliers. The study also offers managerial implications for practitioners in the smart factory industry.

2. Theoretical Foundation and Related Models

2.1. Theoretical Foundation: Organizational Maturity and Maturity Models

The concept of organizational maturity encapsulates the evolutionary journey of an organization as it enhances its people, processes, and technological readiness and capability through the adoption of quality practices. It serves as a metric for evaluating the quality of a company’s operations, with higher maturity levels indicative of an organization’s ability to confront challenges and capitalize on opportunities [21].

An organizational maturity model provides a structured framework for gauging a company’s maturity, typically dividing it into distinct levels or stages. Once organizational leaders comprehend their current maturity level, they can strategically work towards achieving higher levels. Typically comprising five stages, these models outline defining characteristics for each stage and are applicable across diverse domains such as change management, digital maturity, finance maturity, and HR maturity.

Ferradaz et al. [22] highlight an increasing interest in the adoption of organizational maturity models (OMMs) within the academic and entrepreneurial communities. Through a bibliometric analysis, the authors discern trends and evidence of convergence, aiming to identify opportunities for further study. The study emphasizes the growing interest in OMMs from academic and entrepreneurial spheres, particularly in fields such as occupational health and safety.

In another study, Kucińska-Landwójtowicz et al. [23] focus on determining research areas and recognizing the current direction in the development of maturity models. They aim to identify key areas of OMM development and classify them, shedding light on research gaps and potential areas for development in the context of scientific research and management needs. Their empirical classification identifies 12 categories, ranging from information technology to health and safety management, serving as criteria for classifying OMM models.

The capability maturity model integration (CMMI), a widely recognized model for evaluating organizational maturity, was developed by the Software Engineering Institute (SEI) at Carnegie Mellon University [20]. This model offers a structured framework for organizations to assess and enhance their processes, featuring maturity levels ranging from Initial to Optimizing. As organizations progress through these levels, they cultivate capabilities that signify a more mature state. Several studies contribute to the understanding of organizational maturity, including the impact of organizational maturity on information system skill needs [17], systematic literature reviews on organizational maturity model architectures [18], and state-of-the-art analyses of maturity models for information systems [19]. Additionally, research by Becker et al. [24] provides insights into the development of maturity models for IT management, while Wendler [25] conducts a systematic mapping study on the maturity of maturity model research, contributing to the broader understanding of this concept.

More recently, organizational maturity models have been studied to evaluate an organization’s capabilities in digital or fourth industrial revolution maturity models [26,27,28,29]. Although many studies have been conducted in both the literature and in practice, there are few appropriate maturity models that can objectively evaluate the capabilities of smart factory solution suppliers, to the best of our knowledge. Thus, the main contribution of this paper is to provide the literature with knowledge of how to evaluate smart factory suppliers.

2.2. Related Models

This study investigates models that evaluate the capabilities of smart factories themselves, rather than the supplier, and models that evaluate corporate management capabilities, briefly describing the representative models.

Table 1. Summary of smart factory capability/competency diagnosis model.

Organization	Model Name	Description
Acatech	Industrie 4.0 Maturity Index	Resources, Information Systems, Organization, and Culture are defined as the main core competencies for a company to become an agile organization capable of implementing a smart factory, and the level of each competency is divided into six stages: Computerization, Connectivity, Visibility, Transparency, Predictive Capacity, and Adaptability [2,3].
EDB Singapore/TÜV SÜD	Smart Industry Readiness Index (SIRI)	This model consists of eight key areas (Operations, Supply Chains, Automation, Connectivity, etc.) that support three core components (Process, Technology, Organization). The final layer consists of 16 assessment dimensions, such as Vertical Integration, Shop Floor, Facility, and Leadership Competency, that should be referenced when assessing the maturity level of a manufacturing facility [4,5].
VDMA	Guideline Industrie 4.0	The Products section evaluates innovation in terms of product development, and the Production section assesses the overall production process and cost efficiency. In addition, the guidelines are categorized into five stages: Preparation, Analysis, Creativity, Evaluation, and Implementation, and detailed evaluation indicators are utilized for each stage [6].
NIST	Smart Manufacturing System Readiness Level (SMSRL)	To diagnose the readiness to select and improve manufacturing technologies and implement smart factories, this model classifies maturity into four categories: Organization, Information Technology, Performance Management, and Information Connectivity [7].
Fraunhofer	Digital Maturity Assessment (DMA)	This model measures the level of digital maturity of a company in the areas of market analysis (threat of new entrants, bargaining power of buyers, bargaining power of suppliers, competition with existing competitors, threat of substitutes), business (customer, culture/workforce, digital strategy, governance, digital market building, transition management, digital operations), and IT (interoperability, IT security, big data, connectivity) [8].
Korea Smart Manufacturing Office (KOSMO)	Smart Factory Level Verification System	Based on KS X 9001, the level of a company’s smart factory is evaluated through 4 areas, 10 categories, and 44 detailed evaluation items, including promotion strategy, process, information system and automation, and performance. It diagnoses the level of each area from Level 0 (no ICT) to Level 5 (advanced) in five stages, providing a level confirmation certificate and presenting customized guidelines that can be used to build and advance smart factories [9].

summarizes the major capability or competency diagnosis models related to smart factories. In addition to the six diagnostic models, other major diagnostic models related to smart manufacturing include Rockwell Automation’s The Connected Enterprise Maturity Model [30], VDMA’s IMPULS-Industry 4.0 Readiness [31], Fraunhofer Oberösterreich’s Reifegradmodell Industrie 4.0 [32], PwC’s Maturity Model–Industry 4.0 capabilities [33], and the Smart Manufacturing System Readiness Level [34].

The smart factory stands as a crucial element in the ongoing digital transformation of today’s businesses. Organizational capabilities or competencies associated with digital transformation or the fourth industrial revolution are inherently tied to smart factory capabilities. Consequently, we explored models designed to diagnose digital transformation-related capabilities for general companies, summarizing our findings in

Table 2. Competency diagnostic model for enterprise digital transformation.

Organization	Model Name	Description
University of Aarhus	Digital Maturity Assessment Tool (DMAT)	This diagnostic tool provides a digital maturity assessment, a maturity overview, and a personalized mini-report. It consists of six dimensions (strategy, culture, organization, processes, technology, customers, and partners) that make up 30 sub-dimensions. An online tool to assess digital maturity in questionnaire format, measured on a five-level scale ranging from 1 to 5 [10].
Boston Consulting Group (BCG)	Digital Acceleration Index	It allows you to assess your organization’s digital capabilities and compare them to peer industry averages, digital leaders, and other groups. The assessment dimensions are organized into eight areas: digitally enabled business strategy, customer delivery and go-to-market, operations, support functions, new digital growth, changing the way we work, leveraging data and technology capabilities, and integrated ecosystem [11].
KPMG	Digital Business Aptitude	By assessing your digital business aptitude (DBA), you can answer the question of “how successful are you at digital transformation” and “identify gaps that need to be addressed”. It consists of five domains and four to six attributes for each domain that describe key capabilities related to an organization’s ability to successfully undertake digital business transformation [12].
Ministry of SMEs and Startups	InnoBiz	It was developed and used as an index to evaluate the technological innovation system of SMEs based on the OECD’s “Oslo Manual”, a manual for evaluating technological innovation activities. It is divided into four categories: technology innovation ability, technology commercialization ability, technology innovation management ability, and technology innovation performance, and each category is composed of large items such as R&D activity indicators, technology innovation system, technology commercialization ability, marketing ability, management innovation ability, manager’s values, and technology management performance [13].
Ministry of SMEs and Startups	MainBiz	The evaluation indicators are divided into three strategic directions: management innovation infrastructure, management innovation activities, and management innovation performance, and each strategic direction is subdivided into categories, evaluation items, and evaluation indicators. According to the evaluation score, it is divided into creative, growth, basic, and basic, and each focuses on preventing problems from occurring in advance, building a flexible system that can respond to changes, and establishing plans to improve capabilities for key processes [14].
Korea Productivity Center	Productivity Management System (PMS)	After evaluating the current level of the management innovation system, improvement tasks are proposed to drive productivity innovation to achieve performance goals. Core values are required through seven categories: leadership, innovation, customers, measurement, analysis and knowledge management, human resources, processes, and management performance. Therefore, the PMS model consists of seven audit categories, 19 basic items, and 80 detailed items [15].
National IT Industry Promotion Agency	Software Process Quality Certification	In order to improve quality and secure reliability while developing and managing software and information systems, we have developed a system to assess and grade the level of software process quality capabilities of software companies and development organizations. This model diagnoses the level of quality competence through a total of 70 questions based on five areas: project management, development, support, organization management, and process improvement [16].

.

However, as mentioned earlier, these models focus on evaluating the capabilities of firms operating smart factories, referred to as smart factory adopters in this paper. They are not designed for smart factory solution suppliers, who are responsible for designing, implementing, operating, or maintaining smart factory systems for adopters. Suppliers require unique capabilities that may not be necessary for adopters, and some capabilities are more critical for suppliers than for adopters. To the best of our knowledge, there is no assessment model available to evaluate the capabilities or competencies of a smart factory solution supplier in both the literature and in practice. While existing models can be used to evaluate certain aspects of a supplier’s smart factory capabilities, they fall short in providing a comprehensive spectrum of capabilities needed by suppliers.

However, it is worth noting that some capability or competency areas in the related models may overlap with those for smart factory solution suppliers. We utilized these related models, including those not listed here due to limited space, as references to construct our model.

3. Methodology

3.1. Framework to Build Our Maturity Model

Based on the methodology proposed by Becker et al. [24], we established the following procedure for developing the maturity model for diagnosing the capabilities of smart factory solution suppliers.

(1)

Define the Scope and Objectives:

Clearly define the scope of the organizational maturity model. Identify the key aspects and objectives specific to smart factory solution suppliers that the model aims to address. Establish a clear understanding of the desired outcomes and the purpose of the model within the context of the smart factory domain.

(2)

Literature Review:

Conduct an extensive literature review to understand existing maturity models, particularly those relevant to IT management and smart factory solutions. Identify key concepts, best practices, and critical success factors that should be considered in the development of the maturity model. This step ensures that the model is built upon a solid theoretical foundation.

(3)

Stakeholder Engagement:

Engage with stakeholders, including smart factory solution suppliers, industry experts, and relevant professionals. Gather insights into the specific challenges, opportunities, and requirements within the smart factory domain. This collaborative approach ensures that the maturity model is tailored to the unique needs and expectations of the target audience.

(4)

Identify Key Dimensions and Levels:

Define the key dimensions that contribute to the maturity of smart factory solution suppliers. Based on the literature review and stakeholder engagement, identify the critical areas that influence organizational maturity. Establish distinct levels or stages that represent the evolutionary progression of maturity within each dimension.

(5)

Develop Assessment Criteria:

Create detailed assessment criteria for each level within the identified dimensions. These criteria should be measurable and indicative of the organization’s capabilities and practices. Align the criteria with the specific characteristics and requirements of smart factory solutions, ensuring relevance and accuracy in the assessment process.

(6)

Validation and Iteration:

Validate the preliminary maturity model with stakeholders and subject matter experts. Gather feedback on the clarity, applicability, and completeness of the model. Iterate on the model based on the received feedback, making necessary adjustments to enhance its validity and practicality.

(7)

Documentation and Communication:

Document the finalized organizational maturity model, including detailed descriptions of dimensions, levels, and assessment criteria. Clearly communicate the intended use of the model, its benefits, and guidelines for implementation. Provide supporting documentation to facilitate understanding and adoption by smart factory solution suppliers.

(8)

Pilot Testing:

Conduct a pilot test of the maturity model with a select group of smart factory solution suppliers and adopters. Evaluate the model’s effectiveness in assessing organizational maturity and gather insights from the pilot test to make any final refinements.

(9)

Implementation Support:

Develop support materials, training programs, and resources to assist smart factory solution suppliers in implementing and using the maturity model. Provide guidance on how to interpret assessment results and leverage them for continuous improvement.

(10)

Continuous Improvement and Institutionalization:

Establish a mechanism for continuous improvement of the organizational maturity model. Incorporate feedback from users, monitor industry trends, and update the model periodically to ensure its relevance and effectiveness over time. Set up the mid-term and long-term promotion and institutionalization plans.

3.2. Normal Group Technique (NGT)

As specified in Step 3 in the established procedure, we formed an external stakeholder group, consisting of researchers in smart factory technologies, related government agencies, industry experts in both smart factory adopters and suppliers, and industry experts in general IT services management, and conducted group discussion sessions to evaluate the in-progress artifacts developed during the development of the proposed maturity model as well as the evaluation of the pilot testing results. In particular, in Steps 4 (Identify Key Dimensions and Levels) and 5 (Develop Assessment Criteria), this group actively participated in decision-making processes. We used a well-known group decision tool, referred to as the normal group technique (NGT). The NGT is a structured process for group brainstorming that involves generating ideas, listing them, and clustering them into coherent groups. NGT is an effective method for group decision-making because it encourages participation and collaboration among group members [35].

3.3. Validation of the Proposed Model

Validating the usefulness of the proposed maturity model is challenging, particularly given its novelty and the absence of a benchmark method. This is reflected in the fact that, according to Wendler [25], about 53% of the maturity model literature in related studies does not incorporate any empirical validation.

However, in alignment with established practices reviewed by Wendler [25], we undertake a pilot test to validate our proposed maturity model. This step is included in the procedure used to create the proposed model. Further detailed information about the pilot testing is in Section 6.

4. Capability Diagnostic Model

Based on the existing diagnostic models discussed above, we define and propose a model to evaluate the capability of smart factory technology and solution suppliers. The proposed capability diagnosis model is called K-SSC (Korea Smart Factory Suppliers Competency).

4.1. Diagnosis Areas

Referring to the main evaluation areas of existing models and using a survey of smart factory suppliers and adopters that have built or are planning to build smart factories, this study defined three areas as representative diagnostic areas: business management, technology, and project management. The capability part of the ‘process’ aspect, which is used as one of the pillars of general evaluation, was limited to ‘project management’ capability, given that the suppliers provide smart factory solutions.

The business management capabilities were developed by benchmarking domestic and international management system models such as InnoBiz, MainBiz, productivity management system (PMS), and the Baldrige excellence framework [13,14,15,36]. We added items to diagnose ESG management, which is a recent issue, IT infrastructure, and performance creation capabilities such as productivity. Business management capabilities consist of three sub-areas: leadership and strategy, human resources and infrastructure, and business performance.

The technology capabilities were developed by benchmarking domestic and international technology/innovation system models such as InnoBiz, ISO 56000 (Innovation management), and Hyundai Motor Group’s Technology 5 Star [13,37,38], and added items to diagnose the level of smartization of the building systems utilized by major products, export capabilities, and smart factory R&D capabilities. Technology capabilities are organized into three sub-areas: products and services, technology innovation, and technology performance.

The project management capabilities were developed by benchmarking domestic and international project management models such as ISO 12207 (software life cycle processes), the project management body of knowledge (PMBOK), capability maturity model integration (CMMI), and software process quality certification [16,20,39,40]. We also added items to diagnose the level of smartization of the construction systems used for the main products, export capabilities, and R&D capabilities. Project management capabilities consist of three areas: process management, support and improvement, and project management performance.

Table 3. Smart factory supplier capability diagnosis model diagnosis areas and scores.

Area	Sub-Area	Evaluation Category	Scoring
1. Business management	1.1. Leadership and strategy	1.1.1. Leadership	75	250
		1.1.2. Strategies
	1.2. Human resources and infrastructure	1.2.1. Human resources	75
		1.2.2. Business infrastructure
	1.3. Business performance	1.3.1. Financial performance	100
		1.3.2. Non-financial performance
2. Technology	2.1. Products and services	2.1.1. Product and service infrastructure	150	500
		2.1.2. Design and development
	2.2. Technology innovation	2.2.1. Creating new markets	150
		2.2.2. Increase technology competitiveness
	2.3. Technical performance	2.3.1. R&D activity metrics	200
		2.3.2. R&D performance metrics
3. Project management	3.1. Process management	3.1.1. Plan and control	75	250
		3.1.2. Manage deliverables
	3.2. Support and improvements	3.2.1. Support and organization management	75
		3.2.2. Process improvement
	3.3. Project management performance	3.3.1. Adopter performance	100
		3.3.2. Follow-up performance

shows the three diagnostic areas of the K-SSC model and the sub-areas within each area. The K-SSC model is divided into three major sub-areas in each of the three areas, for a total of nine major sub-areas, and each major sub-area is further divided into two evaluation categories, for a total of 18 evaluation categories and 36 evaluation items. For each area, a third sub-area contains a quantitative evaluation category and diagnostic items. The diagnostic scores for each sub-area are also shown, with different scores depending on the importance.

4.2. Capability Maturity

Similar to related enterprise maturity assessment models, the K-SSC model is designed to establish a step-by-step innovation roadmap by subdividing the capability maturity of smart factory suppliers into five stages to respond to the requirements of adopters and stakeholders. The five stages consist of the Informal, Basic (Managed), Standard-Based (Defined), Agile (Predictable), and Optimizing stages. The five stages are defined by the extent to which an organization leverages the Activities, Results, Planning, and Evaluation processes and how they are used to define and improve policies or standards.

Figure 1 depicts the required maturity capabilities for each of these stages, along with the required inputs and outputs and associated information, while

Table 4. Scoring criteria for qualitative diagnostic items.

Maturity Level	Evaluation Criteria
5. Optimizing	Learn from entrenched continuous improvement activities, revise new policies and standards based on feedback, and move toward optimization Proactively anticipate and respond to environmental changes and exceptional circumstances
4. Predictable	Quantitatively measure and manage workforce capabilities and activities and predict their performance Quickly and flexibly respond to environmental changes and exceptional circumstances
3. Defined	Plans, outcomes, and performance evaluations are based on enterprise-wide policies or standards Standardized systems, processes, people, resources, etc. exist and are supported Empowerment aligned with job responsibilities and training aligned with core competencies
2. Managed	Planned activities and capabilities drive results Basic systems, processes, people, resources, etc. exist and are supported Clarify job responsibilities, share limited authority, and provide training for skills and knowledge
1. Informal	Results are driven by the activities and capabilities of individuals without a plan Reactive and ad hoc, with few institutions, processes, people, resources, etc. Unclear job responsibilities, performing routine tasks
0. None	No activity or results No preparation or interest in systems, processes, people, resources, etc.

and

Table 5. Evaluation criteria for quantitative diagnostic items.

Maturity Level	Evaluation Criteria
5. Clear market leader	Excellent upward trend or excellent levels of performance across all metrics Recognized as a clear market leader
4. Benchmarks from other companies	Continued upward trend in most metrics You’re outperforming your competitors and are being benchmarked within your industry
3. Positive trends in most metrics	Overall upward trend in most metrics Outperforming peers and competitors across the board
2. Positive trends in key metrics	Key metrics are trending upward Performing at a similar level overall to your peers and competitors
1. Negative trends in key metrics	Trending downward on key metrics or flat on most metrics Overall lower performance than peers and competitors
0. Negative trends in most metrics	Overall downward trend in most metrics Recognized as the lowest ranked company in your industry

describe the qualitative and quantitative evaluation criteria for each maturity stage as assessed by the diagnostic items. In Figure 1, activities produce results (Stage 1) according to planning (Stage 2), which is based on policies and standards (Stage 4). When planning, the results should be estimated (Stage 5). Improvements in each stage lead to an increase in the level of smartness.

The scoring system for all evaluation categories in Table 3 is determined by detailed diagnostic items (evaluation indicator) in

Table 6. Detailed diagnostic items.

Evaluation Category	Diagnostic Item (Evaluation Indicator)
1.1.1. Leadership	Smart factory vision and goals ESG management
1.1.2. Strategy	Strategic planning and deployment Performance management
1.2.1. Human resources	Human resources management Human resources development
1.2.2. Business infrastructure	Information systems Office & workplace environment
1.3.1. Financial performance	Corporate credit rating Value added per capita
1.3.2. Non-financial performance	Level of welfare Turnover and employee satisfaction
2.1.1. Product/service infrastructure	Level of product/service International export infrastructure
2.1.2. Design and development	Product design and development Information security
2.2.1. Creating new markets	New market analysis New product launches
2.2.2. Increase technology competitiveness	Driving technology innovation Technology cooperation
2.3.1. R&D activity metrics	Ratio of technical developers to R&D assignments
2.3.2. R&D performance metrics	Number of IPRs registered Number of product/test certifications
3.1.1. Planning and control	Project planning and control Supplier management
3.1.2. Manage deliverables	Analysis and design Implementation and testing
3.2.1. Support and organization management	Quality assurance and configuration management Measurement and analysis
3.2.2. Process improvement	Troubleshooting and risk management Process improvement management
3.3.1. Construction performance	Process improvement performance System utilization and satisfaction level
3.3.2. Follow-up performance	Maintenance contract performance Track record of selected good/bad practices

according to quantitative and qualitative diagnostic items (in Table 4 and Table 5). We applied the evaluation criteria of quantitative and qualitative diagnostic items such as the examples in

Table 7. Example of a management area: Smart factory vision and goals.

Area	Sub-Area	Evaluation Category	Diagnostic Item
1. management	1.1. Leadership and strategy	1.1.1. Leadership	1.1.1.1. Smart Factory Vision and Goals
Item Description	It refers to a comprehensive process that brings together strategic planning and organizational capabilities to connect future goals and realities to strengthen the competitiveness of smart factory suppliers in the long term. The three components of a vision: ① a meaningful purpose (mission), ② core values, and ③ a blueprint for the future (consultative vision).
Key Things to Look For	Supplier’s vision and management objectives map Company homepage, company letterhead and brochure, company promotional video, CEO message, etc.
Evaluation Criteria	5	The organization’s smart factory vision system and operation activities are so excellent that they are becoming an example for other companies.
	4	The organization’s smart factory vision and goals are closely aligned with its strategy and business plan and are periodically evaluated and improved.
	3	The organization’s smart factory vision and goals are systematically established, and various systems and activities are implemented according to the strategies and plans associated with them.
	2	The organization’s smart factory vision and goals are systematically established, but they are underutilized for strategic and business planning.
	1	The organization has a smart factory vision and goals, but they are not specific or structured.
	0	No evidence of an organizational smart factory vision and goals.
Diagnostic Notes	Ensure your company vision includes the three Ps. - Product: Describe a challenge for a key product/solution, such as a goal or position in the market. - Process: State your goals for competitiveness on quality, delivery, cost, etc. - People: Describe your company’s mission to customers or internal workforce. Make sure your vision has specific goals and evaluate them using SMART principles: - Specific, measurable, action-oriented, realistic, time-bound Ensure that the visioning process is aligned with systematic strategic planning and key business initiatives for the year.

and

Table 8. Example of a management area: Smart factory vision and goals.

Area	Sub-Area	Evaluation Category	Diagnostic Item
3. Project management	3.1. Process management	3.1.1. Planning and control	3.1.1.1. Project planning and control
Item Description	A standardized set of processes by which a supplier creates a project plan based on a customer’s requirements statement and controls resources, budget, schedule, etc. Managing phased completion schedules to meet project deadlines and developing a response system in case of unforeseen events is one of the core competencies of a supplier.
Key Things to Look For	Project planning and control management performance and driving plans Company standard processes, guidelines, handbooks, regulations, etc. Project planning and control management monitor and improve performance Information systems and infrastructure for project planning and control management, etc.
Evaluation Criteria	5	Predict environmental changes and exceptional situations in advance for optimized response
	4	Ability to plan and control projects and quantitatively measure and manage activities and predict outcomes
	3	Project planning and controls are based on enterprise-wide policies or standards to guide evaluation.
	2	Project planning and control results from the activities and capabilities through planning
	1	Project planning and control results from the activities and capabilities of individuals without a plan
	0	No activity or results for project planning and control
Diagnostic Notes	Refer to the Korea Smart Manufacturing Office (KOSMO)’s “Deliverables List of Smart Factory Construction and Advancement Program” - Project management detail tasks: schedule management, reporting management, change management, performance management, etc. - Deliverables: work breakdown schedule (WBS), weekly/monthly reports, change logs, performance measures, etc. Refer to NIPA’s “SW Process Quality Certification (SP Certification)” project management area (1.1~1.2) - Specific, measurable, action-oriented, realistic, time-bound Ensure that the visioning process is aligned with systematic strategic planning and key business initiatives for the year.

.

4.3. Detailed Diagnostic Items and Criteria

The K-SSC model is divided into 18 subcategories, and each subcategory has two detailed diagnostic items (evaluation indicators), as shown in Table 6. Therefore, capability diagnosis is performed through qualitative and quantitative evaluation of a total of 36 detailed evaluation indicators.

For the evaluation of each diagnostic item, the main points to be checked in the item description and evaluation are specified, and detailed evaluation criteria for each of the 36 diagnostic items are defined based on the basic evaluation criteria described in Table 4 and Table 5. In addition, situations that should be referred to during the evaluation are defined and presented. In this paper, the evaluation criteria for ‘Smart Factory Vision and Goals’ and ‘Project Planning and Control’ are given as examples as shown in Table 7 and Table 8.

5. Evaluation Process: Auditor Training Operational Process

To ensure objective and fair diagnoses and audits, we use the same method as other similar audit models at home and abroad, such as InnoBiz and CMMI, to train professional auditors to diagnose directly. Therefore, we defined the training and operation process of auditor curriculum development, auditor training, and auditor management, as shown in Figure 2.

The purpose is to select and train auditors to understand the indicators and level system of the capability diagnosis model so that they can train auditors by specialty and put them into diagnosis, to select and train auditors to diagnose according to the situation of each company, and to establish auditor qualification standards and management measures to strengthen the reliability and professionalism of the capability diagnosis system.

The auditor training consisted of four main parts: basic training to provide basic knowledge on the overview of smart factories, major policies, key technologies and application cases, and the status of suppliers; practical training to provide detailed inspection criteria and diagnostic exercises of the Air Enterprise Capability Diagnostic Framework; supplementary training on inspection criteria, consisting of case studies and discussion of practical inspection results among auditors; and supplementary training on supplementary diagnostic models and criteria.

In addition to the training operation process, it is also necessary to organize constraints such as the formation and operation of the qualification review committee for objective and responsible auditor operation, auditor qualifications and obligations to complete training, limitations on the period of training recognition, and participation in continuing education and evaluation to renew auditor qualifications.

6. Pilot Diagnostics and the Results

6.1. Auditor Training and Supplier Selection

To validate the K-SSC capability diagnosis model and the auditor training operation process, we conducted a pilot diagnosis of 32 suppliers in South Korea. In accordance with the auditor training operation process and auditor management plan, we developed a curriculum and trained 28 auditors to conduct the pilot diagnosis. By technical specialty, the group consisted of 10 operational technology (OT), 11 information technology (IT), and seven automation technology (AT) auditors, and the conflict of interest between the auditors and the diagnosed companies was checked in advance.

We first selected suppliers that were judged to have a relatively high level of interest and understanding of capability diagnosis, and at least one company was included according to the basic classification of each solution. The selected companies were given a detailed explanation of the purpose of the diagnosis so that they would not feel reluctant to participate, as the institutionalization plan and incentives for supplier capability diagnosis had not yet been finalized. Based on the solutions that suppliers mainly deal with, ten were MES (Manufacturing Execution System), six were ERP (Enterprise Resource Planning), and four were Big Data/AI. Additionally, 18 were distributed in the Seoul metropolitan area and 14 in the provinces.

6.2. Pilot Diagnostic Results

The average overall capability level of the 32 companies was 2.10 on a five-point scale, which is approximately Level 2. Relatively high levels of capability were found in business performance, technical performance, process management, and project management performance, while relatively low levels of capability were found in human resources and infrastructure, products and services, technology innovation, and support and improvement as shown in Figure 3. The range (maximum–minimum), which is one of the indicators of differences between suppliers, is the largest for business performance (2.75) and the smallest for project management performance (1.50).

Based on the 36 detailed diagnostic items, relatively high capabilities were found in the areas of building system utilization and satisfaction, smart factory vision and goals, and welfare ratios, while relatively low capabilities were found in the areas of overseas export infrastructure and ESG management. The items with the largest ranges (max–min) were value added per capita, turnover rate and employee satisfaction, and number of patents, all of which are related to quantitative performance.

The diagnosis results, candidate tasks, and roadmap for implementation were presented in the form of a diagnosis report, as shown in Figure 4. The diagnostic report was presented to the participating companies so that they could analyze their differences from other companies.

Given that this was our initial attempt to assess the capabilities of smart factory solution suppliers, we lacked objective benchmark data. We engaged a panel of experts comprising researchers, government agencies, industry experts, and pilot appraisal auditors. This group utilized the NGT to assess the effectiveness and accuracy of the pilot appraisal results across all suppliers. The evaluation involved comparing results among suppliers and taking into account qualitative feedback from the market, along with other performance measures related to the suppliers. The presented pilot results are the outcomes of this effort. We also used the experience and results of this pilot testing to calibrate the maturity model.

7. Promotional Strategy and Institutionalization

7.1. Mid- to Long-Term Roadmap

In order to activate the supplier capability diagnosis system, we designed an operation management system, as shown in

Table 9. K-SSC mid- to long-term implementation roadmap.

Phase 1. Foundation	Phase 2. Diffusion	Phase 3. Stabilization
Diagnosis model development Auditor training and piloting Establishment of an institutionalization plan KS standardization	Supplementing industry-specific certification criteria Development of audit process and manual Development of auditor qualification and training system Certificate development	Complementing industry-specific certification standards and audit manuals and auditor training systems Online authentication system development and operation

, and established a mid- to long-term promotion roadmap with a promotion strategy and stages: Phase 1—Foundation, Phase 2—Diffusion, and Phase 3—Stabilization.

The first phase aims to develop a model for diagnosing the capabilities of smart factory suppliers, create an operational process for training auditors, train auditors for actual pilot diagnosis as a pilot concept, and conduct audits of selected suppliers. The second phase, the diffusion phase, aims to improve the operation plan of the capability diagnosis model and auditors developed in the first phase by reflecting the results of the pilot diagnosis and to spread and apply it to various supplier diagnosis. The main tasks include supplementing diagnostic criteria for each industry, expanding the pool of operating organizations and auditors, and developing audit processes and manuals. The final phase, Phase 3, aims to advance the capability diagnosis model and auditor operation plan developed in this study. It will develop industry-specific diagnostic criteria, audit manuals, and auditor training systems.

7.2. Institutionalization

It is necessary to institutionalize the capability diagnosis model presented in this study to effectively utilize it and expand its positive impact on the smart factory-related consumer and supplier markets.

The purpose of institutionalization is as follows. First, the capability diagnosis model will be utilized as an innovation tool to understand the supply technology and market trends that are being advanced through self-diagnosis of suppliers and to leap forward as an excellent adopter company. Second, by utilizing external professional auditors to check the technology status and management status of smart factory suppliers, it is possible to objectively diagnose the current level of suppliers and derive improvement measures. Third, it serves as a criterion for granting incentives to excellent suppliers and support to deficient companies. Finally, it aims to enhance the competitiveness of the smart factory industry by establishing standards for diagnosing the capabilities of suppliers and fostering experts for diagnosing the capabilities of suppliers. In practice, an additional pilot project is being conducted to reflect the results of this study, and the government is establishing specific measures, including capacity diagnosis, based on the strategy to foster smart factory solution suppliers.

There are differences in requirements between suppliers and adopters, such as the form of operation of the capability diagnosis system, the incentive system based on the results, and the method of disclosing the results. A survey was conducted to identify the differences in stakeholders’ perceptions and understanding of institutionalization, and

Table 10. The most appropriate form of capability diagnosis system operation.

	Supplier Eligibility Prerequisites	Supplier Qualification Preferences	Check Optional Capability Levels	Capability Build-Up Consulting Prerequisites	Etc.
Supplier	29.0	45.4	21.3	4.3	0.0
Adopter	36.2	15.3	34.6	13.7	0.2

,

Table 11. Incentives for capability diagnosis system.

	Offer a Small Incentive to All Participating Companies	Incentivize a Select Group of Highly Capable Companies	Penalize Some Companies with Low Capability Levels	Provide Support for Some Less Capable Organizations	Etc.
Supplier	61.2	24.4	4.0	10.1	0.3
Adopter	33.2	29.1	24.9	12.6	0.2

,

Table 12. Appropriate outcome management practices for capability diagnosis system.

	Disclosures in Governmental Programs	Public Only for High-Level Companies	Disclose Only for Low-Level Companies	All Unpublished	Etc.
Supplier	59.2	21.3	2.6	16.7	0.3
Adopter	77.6	13.3	6.2	3.0	0.2

and

Table 13. Concerns about introducing a capability diagnosis system.

	New Regulations in Action	The Burden of Additional Work	Additional Cost and Time	The Burden of External Evaluation	Doubts about Effectiveness	Etc.
Supplier	25.6	30.2	33.0	5.2	6.0	0.0
	Formal Institutional Operations		Rising Supply Costs	Supply-Demand Imbalance		Etc.
Adopter	53.1		36.6	10.1		0.2

show the results of the survey on the capability diagnosis system by supplier versus adopter companies. Responses were received from 348 suppliers and 437 adopters.

It is necessary to reduce the difference in perception between the supplier and the adopter and design an incentive-oriented system for the common benefit of the stakeholders to minimize the gap and make it a system that can be mutually recognized and satisfied. For this purpose, it is necessary to prepare an institutionalization plan considering the following points.

• Full or partial coverage of diagnostic costs, depending on the roadmap phase, until institutionalization
• Needs for careful design to avoid socialization
• Spreading the system by providing preferential conditions for participation in government dissemination projects
• Information provision in the KOSMO system based on the strong areas of the supplier’s diagnostic results.
• Use of basic information and diagnostic results for future research and statistical analysis after obtaining consent

Lastly, we acknowledge that an organization’s capabilities are dynamic rather than static. One of the key purposes of a maturity model is to furnish the evaluated organization with structured guidelines for enhancing or developing necessary capabilities to progress to higher maturity levels. Aligned with the practices of other maturity models such as CMMI [20], we intend to establish a routine reappraisal process to continuously monitor the capabilities of the evaluated organization.

8. Contributions and Limitations

Many prior studies have suggested assessment models to evaluate smart factory-related capabilities of organizations that operate smart factories for their businesses [2,3,4,5,6,7,8,9]. To the best of our knowledge, in both the literature and in practice, there is no appropriate maturity model that can objectively evaluate the capabilities of smart factory solution suppliers. Therefore, this study contributes to the literature on smart factory research and corporate capability assessment by presenting a model to objectively assess the capability of smart factory suppliers to provide smart factory-related solutions and services for the first time in academia. Additionally, there is organizational maturity model literature proposing a new maturity model to evaluate an organization’s capabilities in the digital sphere or the fourth industrial revolution [26,27,28,29]. This study also contributes to the maturity model literature by adding a new maturity model for smart factory solution providers. In addition, the results of this study have great practical significance in that they lay the foundation for providing accurate and objective information about smart factory suppliers, which has been a challenge for many adopters. In the future, we plan to change the diagnostic items such as the number of test certification acquisitions and process improvement performance to other items such as the satisfaction of the adopting company, considering the variability depending on the applied industry.

While this study is significant in that it is the first to present a competency model for smart supply companies, it does not include a detailed assessment of the competencies of the technologies possessed by suppliers. This is because the unique characteristics of each technology make it difficult to create common evaluation criteria, and the constantly changing and evolving nature of technology makes it almost impossible to include detailed evaluation methods for all technologies in an evaluation model. For this reason, other types of skill assessment models such as CMMI do not include detailed software development skills. However, considering that adopters who want to build smart factories also need such detailed technology evaluation information, it seems necessary to develop a detailed technology evaluation model for representative technologies of smart factories such as MES. Another limitation lies in the validation of the proposed model. Although we present pilot diagnosis results and survey findings demonstrating that the proposed model can assess the capabilities of smart factory solution suppliers, a more detailed and rigorous validation is needed to determine whether our evaluation criteria are closely related to other crucial performance or quality measures. These measures include financial outcomes, market shares, and customer satisfaction for both suppliers and adopters. Exploring these aspects could be a potential extension of this study. Lastly, it is important to note that this study was conducted in South Korea. While our model does not account for geographical or cultural aspects, a replicated study is necessary to verify whether our model functions effectively in different geographical or cultural settings.

9. Conclusions

The existing software capability models, such as CMMI [20], measure maturity by focusing on general software engineering backgrounds, and there is a lack of understanding of the capabilities of suppliers in terms of smart factories.

In this study, we developed a model to diagnose the capabilities of suppliers that provide services such as planning, design, construction, operation, and maintenance of smart factory solutions for adopters that want to transform and innovate smart factories. The capability diagnosis model was developed by referring to existing research results and existing models that are used as standards in the industry, such as the existing smart factory-related maturity model, the corporate management-related maturity model, and the software/information service supplier competency model. It is composed of three areas, nine sub-areas, 18 evaluation categories, and 36 diagnostic items, and is presented to evaluate capability or maturity on five levels. In addition, this study presents an operational process that can objectively evaluate and manage suppliers by utilizing the developed capability diagnosis model. The operational process basically follows the method of training professional auditors to conduct independent audits similarly to existing corporate competency assessment models such as InnoBiz and CMMI. It includes criteria for auditor selection, training and management, and company selection and evaluation methods for auditor-centered assessment. Finally, in order to verify the effectiveness of the proposed capability diagnosis model and operational process, this study selected, trained, and developed a total of 28 auditors according to the proposed operational process and conducted a pilot diagnosis for 32 supplier companies. The pilot diagnosis identified improvements to the capability diagnosis model and operational process, and a proposal for the institutionalization of capability diagnosis was made. This allows assessed organizations to know where they stand in each area, identify capabilities that need further improvement, and support rapid decision-making to strengthen those capabilities. This study contributes to the related literature by providing the first maturity model to objectively and holistically evaluate the capabilities of smart factory solution suppliers. It also offers practical implications for the practitioners in the smart factory industry. This study is a first step to build a maturity model for smart factory suppliers rather than adopters. Further follow-up research is necessary to validate the proposed model, improve the proposed model, and investigate the effects of the model on other performance measures of suppliers and adopters, including financial and operational performances etc.