Developing an Evaluation Framework for Selecting Optimal Medical Devices

Abstract

The objective of this study is to summarize the current general evaluation process that stakeholders use to evaluate single or multiple medical devices to select the optimal medical device. This study constructed a novel evaluation framework to select the optimal medical device by overcoming the limitations of assessments that were previously confined to the medical device cycle and the purpose of the specific assessment. It also presents a general framework for incorporating usability evaluations into the frame to reflect the objectives and characteristics of key stakeholders in the medical device industry. In addition, the results of medical device evaluation are further analyzed using the analytical hierarchy process (AHP).

1. Introduction

The medical device industry is closely related to human life and public health. It is a field of the health industry whose importance is increasing alongside the technological developments in the 21st century. In addition, as the demand for high-quality medical devices rises and the development of advanced medical devices is accelerating due to technological developments, medical devices are being recognized as a high-value-added industry. With the advancement of the age of population of Korea, developing countries, and developed countries, the importance of medical device development and the medical device industry are also becoming more apparent [1].

However, the multitude of perspectives throughout the medical device industry and among stakeholders is becoming a risk factor for the development of the medical device industry. Stakeholders are not evaluating the value of medical devices uniformly because they use different evaluation methods with different purposes and outcomes. In addition, different evaluation criteria may not only increase the period of entry of medical devices into the market but may also cause the objective evaluation and efficiency of medical devices to deteriorate, which may hinder industrial development.

Therefore, in order for the industry to continue to develop, it should accept and consider the variety of stakeholders’ perspectives through an evaluation framework that evaluates medical devices objectively and comprehensively. Stakeholders in the medical device industry need to unify their views of each other’s medical devices. To do so, it is necessary to develop an evaluation framework for selecting optimal medical devices that can be used objectively and universally. An evaluation framework that can be used universally can select medical devices that meet the objectives and characteristics of stakeholders according to the results of objective evaluation results and can improve the quality of medical care by improving the reliability and efficiency of the selection process. In addition, when selecting multiple medical devices, it would be possible to maximize the benefits to the stakeholders by providing the results of numerical comparisons.

The objectives of this study are as follows. First, in order to select the optimal medical device, stakeholders have outlined standardized methods of evaluating existing single or multiple medical devices. Second, the existing frameworks that have limited evaluation objectives were considered in the new universal framework to counter the difficulties of evaluating similar medical devices across the market. Third, usability evaluations that were previously limited to the users themselves were included in the new universal evaluation framework to add an element of objectivity.

First, we review the limitations of previous studies on medical device evaluations and the different stakeholders. We then review the literature on the theoretical basis of the evaluation system of medical device evaluation methods. Subsequently, a novel general framework for evaluating medical devices is presented with a specific indicator system. Next, the results of the survey are analyzed quantitatively according to the frames, analytical hierarchy process(AHP) is performed, and a comprehensive evaluation is presented. Finally, suggestions for medical devices based on the analysis results are presented.

2. Literature Review

2.1. Previous Evaluation Frameworks for Medical Devices

There are two main methods for evaluating medical devices. One is the evaluation of a single medical technology or device, and the other is to evaluate several medical technologies and devices simultaneously.

The methods of evaluating a medical device or technology can be classified according to the development period of the device. Which follows the model of the total product life cycle and is also used by the United State(US )food and drug administration to reflect the characteristics of medical devices [2]. However, as the experience of product testing and use accumulates, the stage of development of medical devices overlaps or iterates. After the product enters the market, product evaluations in the market and resulting improvements continue.

In the early stages of invention and prototyping and in the preclinical stage, technology valuation is conducted, and evaluation is carried out at the clinical stage and the regulatory decision stage. Although the evaluation of medical devices is done the Research and development(R&D) throughout this period, there is an opportunity to determine the development of a device and conduct clinical trials through the evaluation of technology value at the early stage of technology development. The evaluation of clinical trials at the regulatory decision stage involves the evaluation of new medicines and medical technology by the government as part of the evaluation carried out in the licensing process of medical devices.

Clinical trial evaluation is a system for reviewing efficacy and adverse reaction data and managing its availability to humans, while the food and drug administration. It is also responsible for the evaluation of medical technology through a system by which the Korea health and medical research institute evaluates medical technology [3]. Further economic evaluation has been performed by the health insurance review and assessment service for the determination of benefits. The results of the clinical trials and the evaluation at the regulatory decision stage are summarized in

Table 1. Evaluations at the clinical trials and regulatory decision stages.

	Clinical Trial Evaluation	New Medical Technology Evaluation	Economic Evaluation
Related Laws	Medical device technique	Medical law	Health Insurance Act
Purpose	Determining whether medical devices are allowed	Determining the purpose, subject, and procedure of the new medical practice	Deciding whether to pay for health insurance
Subject	Medical Device	New medical practice (Including medical devices)	Medical Devices and Medical Practices
Evaluation factor	Safety and efficacy in clinical trials	Stability and efficacy in medical settings—Assessing medical practices, including medical devices	Appropriateness of health insurance benefits, economic efficiency—Evaluation of medical activities including medical devices
Evaluation viewpoint	Safety: Physical chemistry, biological safety performance Validity: The observed outcome in clinical trials	Safety: Validity of outcome indicators Validity: Accuracy and impact on medical results	Appropriateness of salary: principle of insurance pay, insurance financial status Economics: alternative possibilities, cost effectiveness
Evaluation index	Analytical sensitivity and specificity, accuracy, precision, clinical sensitivity and specificity, correlation, reference material	Side effects, complications, mortality, accuracy of diagnosis and effects on outcome	Cost-Effectiveness Evaluation
Evaluation materials	Clinical trial result data	Systematic review of peer-reviewed published literature, expert judgment	Results of new medical technology evaluation, cost-effective data, domestic and overseas papers, etc.

, as safety and efficacy, clinical value, and economic value are evaluated individually for each evaluation method [4].

Although product evaluation is conducted after the product is released in the product lifecycle model, methods for comparing and evaluating multiple medical devices or medical technologies are used, rather than evaluating a single product. Evaluations are carried out to compare medical devices with their alternatives by companies and to compare similar medical devices for introduction in hospitals. The evaluation items and criteria are applied differently according to the hospitals where they are to be introduced, and the evaluation methods mainly used are summarized in

Table 2. Methods for comparing and evaluating multiple medical devices.

	Profitability Evaluation	Aging Evaluation	User Preference	Utilization Evaluation
Purpose	Analysis of profitability due to introduction of medical device	Evaluate the value of medical device and reflect it in introduction	Selection of necessary medical device in department of use	Measures the utilization according to the necessity, basis, and selection condition of medical device required for the operation of medical institution
Evaluation factor	Profit from medical device introduction	Evaluation of the value of medical device	Priority by user needs	Evaluation of variables for introducing medical device, including whether device is essential
Evaluation viewpoint	Evaluation of profit / loss ratio within 5 years against device amount or profitability, evaluation of whether basic / essential device or quality of care is improved	Year of introduction, current production status, frequency of failure, cumulative repair cost of purchase amount, ability to support company service, necessity to department of use, evaluation of person in charge	Evaluation of the purchase priority given by the use department	Reviewing and evaluating the average number of months or hours per use
Evaluation index	Annual revenue is based on device estimates; Revenues from depreciation	Current production status (model discontinuance) / discontinuance / purchase date / frequency of failure / cumulative repair ratio	Claiming department with 1 ranking or 1 to 5 ranking	Operational need / Investment profitability / Appropriateness of utilization

[5].

There are several methods for evaluating medical devices that meet the purposes and values of stakeholders, and the stakeholders that are the subject here are the government, hospitals, and businesses. From the standpoint of stability and effectiveness, the government evaluates devices in terms of efficiency and technology, and hospitals in terms of user aspect and economics. At first glance, it would appear that the evaluation of the medical device is performed to suit the characteristics of each stakeholder, but because each method considers only one viewpoint, it has the limitation that the whole value of the medical device cannot be considered.

Unlike the international trends of evidence-based medical technology evaluation, Korea’s medical technology evaluation system supports economic evaluation [6].

The evaluation of a medical device has different stages, each of which requires different information. Medical devices for hospitals need high quality information on clinical efficacy, especially for innovative medical devices, but there is no information on clinical data. In Europe, the Conformite Europeenne(CE) marking of the safety and performance of government devices is only available on the market based on a small number of clinical studies that are not methodologically accurate. This shows how difficult it is to share experiences in the field and to evaluate the medical device appropriately during the practical period for decisions [7]. Also, Kazanjian and Green [8] state that the manufacturers have had the usual limited point of view of the development process, because they are mainly focused on proof of concept technologies, which may result in a lack of information suited to the user’s needs and on the economic aspects of the device, thereby affecting the evaluation of the device.

Comparative evaluation methods for the introduction of medical devices involve a variety of evaluations with different types of results reflecting different preferences, such as the type of institution, medical device, and area of ​​introduction, which determine the details of the introduction. In this process, the evaluation of the medical device faces considerable uncertainty [9]. Rather than considering the priorities of different values, these methods require the ability to appropriately allocate the value of medical devices to various stakeholders, but each participant lacks the means to convey its values to the other stakeholders.

Kang et al. [5] report that, according to the improvement reports of medical device purchasing management systems of affiliated hospitals, the highest-priority selection criterion is user preferences in surveys on medical device selection but given that the priority evaluation items by users account for only 8%. It is unlikely that users will be able to introduce a satisfactory medical device under current standards. Usability evaluation is an engineering research methodology that identifies the problems and needed improvements of products by observing and analyzing actual users using actual products [10]. Therefore, it is a scientific method of evaluating a medical device based on the users and reflecting the users’ preferences, which are an important factor in the introduction decision. However, it is difficult to perform usability testing to evaluate an existing medical device because the users find it difficult to apply the method to assess the direct use of the device due to expense or bulkiness. However, currently, usability testing is being introduced into various fields due to the need to better test the medical devices being applied in medical technology, or due to the creation of conditions favorable to usability testing by companies with laboratories with a view to demonstrating their medical devices. Therefore, the importance of usability evaluation will increase in future medical device evaluation.

These limitations can be summarized by saying that the evaluation of existing medical devices makes it difficult to introduce medical devices because clinical data and objective evaluation data about medical devices are not shared. In addition, comparative evaluations of medical devices with different purposes and perspectives make it difficult to compare superior medical devices because of their lack of objectivity, as well as introducing uncertainties about predictability and comparison effects when developing and investing in medical devices. The introduction of medical devices also lowers user preferences, making the selection of medical devices more difficult, and does not reflect the need for usability evaluation in the medical device industry.

2.2. Necessity of a New Evaluation Framework

Thus, because of the nature of the medical device industry, the means and purposes of evaluating medical devices differ by the time, purpose, systems, and variety of stakeholders, and so a generalized evaluation frame is needed to overcome the limits of stakeholders in introducing medical devices. Objective standards are needed to meet the objectives of various stakeholders and to maximize their mutual benefits while increasing efficiency in the distribution flow of medical devices. The medical device industry is growing quickly and is one of the most valuable industries in an aging society [11].

Although the position of high-end medical devices is limited in domestic companies, low-end medical devices are steadily gaining ground in small and medium-sized enterprises (SMEs). In response to the domestication of these industries, the government actively supports SMEs and venture companies and also invests in the development of medical devices. If we can verify the excellence of domestic medical devices that can replace high-priced medical devices, the efficiency-oriented medical device industry market will be able to strengthen the position of domestic medical device companies. When considering evaluation frames from the viewpoint of various stakeholders, it is necessary to summarize the necessary aspects of evaluation from various perspectives in order to construct a universal evaluation framework.

1).

Necessary aspects of an evaluation framework from the viewpoint of the hospital

①

Economic aspect: It is possible to create a medical devices by selecting multiple pieces from other cost-effective medical devices.

②

Competitiveness aspect: It is possible to improve the quality of medical services by selecting excellent medical devices and improving the reliability of the hospital, thereby enhancing the brand image of the hospital.

③

Business aspect: Effective medical device selection can reduce business costs.

④

Improvement of work efficiency: The respective departments can select the most necessary and effective medical device, thereby improving work efficiency and user satisfaction.

2).

Necessary aspects of an evaluation framework from the viewpoint of the government

①

Economic aspect: The evaluation of medical devices through the evaluation framework can lower the investment risk of the government.

②

Competitiveness aspect: It can shorten the time of relevant processes by integrating various evaluations conducted by the ministries and reduce the indiscriminate introduction of medical device, thereby raising the competitiveness of the medical device industry.

③

Business aspect: By reducing rebates and unfair contracts of medical device business, it is possible to create a transparent medical device introduction market and induce technological competition to help the medical device industry develop. It can be used to support R&D commercialization, such as the development of new materials and design technologies, and the development of key parts and technology.

3).

Necessary aspects of an evaluation framework from the viewpoint of the medical device companies

①

Economics: Reducing rebates and unfair transactions related to the introduction of medical devices can increase investment in medical device technology and development.

②

Business aspect: It is possible to raise the market competitiveness of domestic medical device makers and to lower the barriers to entry, and it is possible to utilize the evaluation results in the promotion of the relevant products.

③

Improvement of work efficiency: Contributes to efficient sales strategy by referring to objective evaluation criteria at the development stage or planning stage of medical device. In addition, based on the evaluation data, it is possible to improve the product improvement and quality control standards, and to provide feedback on product upgrades and new developments.

Stakeholders seek a variety of benefits based on their own needs and interests, but in the end a single objective evaluation frame is needed. Although there have been systematic studies of individual evaluations in previous studies, there have been no studies of the evaluation of medical devices taking into consideration the variety of stakeholders.

However, there are various efforts to this end being made in the medical technology sector. The evaluation frame for the medical technology of recent US high-tech medical device industry association (advanced medical technology association) presents us the example of a health technology evaluation framework taking into account the various stakeholders that was constructed by Deloitte Consulting. They have launched a strategic value creation program in partnership with limited liability partnership (LLP) to develop principles and approaches to assess the value of medical technology that can be adopted by medical technology companies, healthcare systems, payers, and other stakeholders. There are eight principles of inclusiveness, evidence, cost, specificity, flexibility, participation, transparency, and relevance. These principles are used to assess the value of medical technology effectively and equitably. This can be a means to that end. This approach integrated into the evaluation process represents four categories of value creation factors:

• Clinical impact: The range of clinical efficacy and health outcomes associated with providing medical technology.
• Non-clinical patient impacts: Impacts of the patient (or caregiver) on the non-medical benefits: patient experience and patient economy Out-of-pocket cost (OOP cost).
• Care revenues and cost impacts: The impact of technology on revenues or costs on providers, payers, provider sponsored plans, etc., on bonuses or fines associated with care quality metrics and clinical trial studies and other operational efficiency resources.
• Public and population impacts: The impact of technology on large-scale and employer or public health systems [12].

These four categories are intended to reflect the views of the various stakeholders, whose priorities may vary from organization to organization. The framework for comprehensive evaluation of medical technology, which is AdvaMed’s approach, ensures that the appropriate analysis supports the valuation. Stakeholders are interested in evaluating the value of a particular medical technology and benefiting patients, providers, and others, taking into account the economic impact of the introduction and all related risks. AdvaMed believes that an effective appraisal process can result in such an analysis of the expected “value proposition” [12].

This approach of AdvaMed resembles the nature of this project. It provides a wide-ranging framework for evaluating the stakeholders’ perspectives and satisfying stakeholders’ value factors based on patient values.

Therefore, in this study, to overcome the limitations of these previous studies, AdvaMed’s evaluation framework is supplemented and revised to create a general evaluation model. In addition, we aim to develop objective indicators based on existing methodologies to overcome the barriers to the creation and activation of an objective and transparent medical device industry ecosystem through an evaluation frame that considers various stakeholders.

2.3. Development of A New Evaluation Framework

In this study, we will approach the evaluation framework from an ecological point of view, reflecting the characteristics of the medical device industry. The medical device industry has a complex structure of relationships among the industries. Here, the stakeholders that form the main axis in the medical device industry are designated government, hospital, and company. Ideally, in such an ecosystem, the government will establish the foundation of the industry through investment and support and the companies will generate technological development and management performance, while the hospitals will increase usability as well as management performance on this basis, and the performance is comparable in other industries and communities across cultures [13].

The general model of AdvaMed is derived from the difference between medical technology and medical devices and from the issue of a suitable model for universal use. First, consideration should be given to stakeholders in the evaluation of medical devices. Given the definition of stakeholders above, the main stakeholders in the medical device industry can be thought of as governments, hospitals, and businesses.

The value factor for AdvaMed’s comprehensive evaluation frame is included in the device evaluation frame as follows. The value factors of clinical effect include, through the impact of its effectiveness and reliability, the revenue impact of medical technology. The nonclinical effect is a value factor reflecting the patient’s experience and benefit, while the user’s introduction of the medical device is affected by the characteristics of the medical device; this can be classified as the user’s position. Care revenues and cost impacts can be categorized as economic factors related to medical devices and treated as a value factor of the costs incurred during the entire medical technology cycle. Finally, the value factors related to public and population impacts are similar to the value factor considering the effects of medical devices obtained from the evaluation of stability and effectiveness that is part of the evaluations carried out by the government. By taking into consideration the characteristics of medical technology and medical devices, it is possible to construct an evaluation frame for medical devices having the same purpose and characteristics by comparing the values of the stakeholders with other aspects of the industry.

AdvaMed’s evaluation frame takes into account the evidence for each evaluation method, which depends on the valuation of the technology. An evaluation method that conforms to the aforementioned principles is applied to various medical technologies. To evaluate medical devices, the clinical performance, technology, stability, and effectiveness of a medical device can be defined in a broader technological sense, and the usability, taking into account the influence on the user and the cost and cost effectiveness of the medical device, can be defined in terms of economic efficiency. Therefore, these major items must be taken into account on the basis of this concept when constructing an evaluation frame for medical device technology, affordability, and usability, and this must take into account the stakeholders and value factors in the medical device industry when performing an evaluation within the AdvaMed frame. The evaluation frame comprises a large medical device evaluation frame and an index selected based on certain attributes: the characteristics of safety and efficacy, functionality, economics, and usability according to the characteristics of each stakeholder. Classifying these individual evaluations according to their purposes, it is necessary to evaluate the performance and technology of medical device, as well as its usability.

The performance and technology of the medical device are obtained through evaluations of the technical documents and an examination of the medical device. The broad view obtained through these evaluations can be defined as the evaluation of the technology. In addition, a number of methods for evaluating the economic value of medical devices, such as cost, benefit, and utility, which essentially involve values rather than technological criteria, can be defined as economic evaluations. Finally, the usability evaluation includes the evaluations of users, which are not currently taken into account in the general evaluation of a medical device.

For the items that make up the larger framework of the evaluation, it is necessary to take into account the characteristics and value drivers of the category, while bearing in mind that evaluation must satisfy both parties to the medical device in a broad sense and include the evaluation methods common to current evaluations, namely the technological evaluation and economic evaluation of medical devices, and that the usability evaluation, which considers the user’s perspective, was added to constitute a major item of the general evaluation framework.

In order to derive the sub-items and indicators of the evaluation framework outlined in the previous section, additional measures are needed. It is necessary to define an evaluation from the user’s perspective that has not been used in general medical device evaluation, and evaluation within the evaluation frame is not limited to pass or conformity. It is important to be able to compare medical devices, and it is important to be able to easily evaluate medical devices without applying difficult methods of evaluation requiring complex analyses. Therefore, in order to achieve the goal of an evaluation framework for the final introduction of the optimal medical device, it is necessary to evaluate the specific results of detailed indicators. Therefore, we decided to construct indicators that can evaluate the activities unique to this project. In particular, the index for usability evaluation, which is not included in the general frame, and the index for technology evaluation, which is difficult to include in the general frame, have been added to constitute the characterization index. Such a configuration, by allowing the creation of a general model, requires complementary indicators when analyzing a particular case, while taking advantage of the fact that the current evaluation can be applied to improve the analysis of other cases in the future. The indicators that can be integrated into the evaluations for each item are classified as follows.

3. Methodology

3.1. Analytical Hierarchy Process Analysis Method and Survey Sample

Our research uses the analytical hierarchy process (AHP) method to derive the relative weights of importance of assessments by the stakeholders, cycle of medical devices, and factor. These studies are well documented in the studies of Kim et al. [13,14,15] and Lee et al. [16].

The AHP, also referred to as stratified analysis, is the main method of the expert questionnaires developed by Saaty in the early 1970s. AHP is used to address the issue of selecting from competing priorities under conflicting criteria and incomplete information [17]. AHP classifies system components into different levels, which is an analytical approach that reflects the natural human thinking process of humans by grouping similar elements, which has the following advantages. First, it provides a way to set scales and priorities that can measure intangibles and enables the verification of the logical consistency of judgments used to determine priorities. In addition, it can analyze the sensitivity to changes of information, and it makes it possible to revise judgments and understanding through repetition. Finally, it does not seek to impose a unitary consensus, but allows the integration of representative results from various judgments and to select a method for weighting the evaluation indexes of the evaluation framework for the introduction of the optimal medical device [16].

This questionnaire used the method of pairwise comparison in setting the weights. This is a method of comparing the relative importance differences between two selected characteristics, A and B, among all the various characteristics and collecting these pairwise comparisons to identify the differences between all the characteristics. The relative importance of each question-specific component is determined by comparing each component in terms of the relative impact of the features it contains (one-by-one comparison of each evaluation factor). The way to compare the two elements is to ask which of A and B is more important (serious) and compare their severity. Currently, the scale of the pairwise comparison used generally has a score between 1 and 9.

For the AHP analysis of the evaluation framework, 23 questionnaires were administered to experts in government, hospitals, academia, and companies. In order to prevent the concentration of experts by type of institution, five experts from government, five medical device experts from hospitals, five medical-device-related experts, and eight experts from medical device companies were selected as a wide range of experts representing all stakeholders. The questionnaire concerned the priorities of the evaluation indices of each evaluation framework on a 9-point scale. Therefore, we conducted research into four groups, (1) Group 1: government officials (5 people), (2) Group 2: hospital officials (5 people), (3) Group 3: Academia officials (5 people), (4) Group 4: company officials (5 people).

Table 3. Statistics of survey distribution.

Actor Group	Total Survey Distributed	Survey Gathered	Inconsistent Survey	Final Valid Sample
Group 1	5	5	0	5
Group 2	5	5	0	5
Group 3	5	5	0	5
Group 4	8	8	3	5
Total	23	23	3	20

shows the statistics of the survey distribution, response rates for each group, and the results of consistency.

The results of the questionnaire were obtained by using the expert choice program, and consistency ratios (CR) were determined for the questionnaires of the 23 experts to investigate the consistency of their responses. As a result of the survey, three questionnaires were identified that failed to maintain a consistency of response greater than 0.15. Therefore, the final result was derived from the opinion of 20 experts after excluding these three people.

3.2. AHP Framework

In order to derive detailed indicators of the evaluation frame, the following goals were established. The first goal is to identify and classify the evaluation indicators that reflect the characteristics of stakeholders from three perspectives. The second goal is to classify the indicators that can be used universally as major items, middle items, and minor items to form the evaluation framework. The third goal is to complete the general evaluation framework by weighting the evaluation indicators through the questionnaire.

3.2.1. Deriving Detailed Indicators of Technology Evaluation

Technological evaluation comprises technology performance analysis or clinical testing and the evaluation of the medical device, safety evaluation, and other tests. The analysis involves comparing the test results for competing technologies and products. It is an evaluation that gives opinions or ratings based on contents [18]. The technological evaluation is conducted to assess the optimal medical devices in order to obtain an indication for the Korea Food & Drug Administration (KFDA) medical device performance evaluation and the National Entertainment Collectibles Association(NECA) new medical device evaluation was obtained as a reference, and the medical device used to define the standard of performance and the electrical, mechanical, and selected chemical evaluations of the performance tests for similar devices or those generally available via international standard methods were included as indices of new devices. In the evaluation of medical technology, a usability evaluation for medical technology or medical device technology was selected as an index through a systematic literature review. The most common medical device in the country was subjected to two technological evaluation methods from a single list of technological evaluations to refine the range.

The evaluation of the performance of the medical device can be performed using the test items of the performance evaluation of medical devices that meet international and domestic standard specifications. This can be divided into technological performance and clinical performance, as the evaluation criteria are divided into indicators that show the accuracy and precision of the results of the performance test and clinical indicators that show the clinical test results of the medical device [19]. The detailed indicators to be evaluated include accuracy, precision, correlation, analytical sensitivity, analytic specificity, clinical sensitivity, and clinical specificity. The evaluation of usability is an evaluation of whether the technology of the medical device is safe and valid for use, which is consistent with the purpose of the government evaluation, and it can be divided into safety and efficacy [20]. The results of these analyses are used to evaluate the accuracy and effectiveness of the device.

3.2.2. Deriving Detailed Indicators of Economic Evaluation

The evaluation of economic efficiency can be divided into two methods. One is to compare the direct cost of the medical device with the efficiency of the cost, and the other is to evaluate the competitiveness of the medical device. An economic evaluation is an evaluation method that simultaneously analyzes cost (input) and result (output). In other words, if the same money spent on one alternative is more effective and entails lower costs to obtain a given result or effect, this allows us to determine its economic value, and thereby determine the priority given to introducing the device into use. Economic evaluations may entail applying a variety of evaluations. According to Ahn [6], many medical technology evaluation agencies recommend cost effectiveness analysis as a comprehensive economic evaluation analysis method for a standard rather than including a cost effectiveness analysis in the economic evaluation of medical technology. However, rather than comparing the final value in the cost–benefit analysis, the evaluation for the decision to introduce a medical device involves a more detailed evaluation of cost-effectiveness than just comparing the final value in accordance with the stakeholder’s priorities. In addition, in order to increase the objectivity of the presented cash value based on a cost–benefit analysis, the price of the actual medical device is evaluated through reverse price analysis and the cost value is converted into the monetary value by the cost–benefit analysis to increase the objectivity.

According to the medical device industry analysis report [21], the characteristics of the medical device market are as follows. The medical device market is characterized by limited demand. Medical devices are a major demand for hospitals with expertise in medical diagnosis and treatment. Also, since it is related to health, the safety and reliability of the product are given priority. Therefore, market buyers tend to be conservative and continue to use famous products, so the market has relatively high entry barriers and low price elasticity. Product awareness and brand power are very important indicators, and marketing sensitivity and barriers are relatively low due to high marketing barriers and loyalty. The competitiveness in the market is a matter that should be treated as important from the standpoint of a company or a hospital. Therefore, the competitiveness of medical device makers, vendors, and medical devices themselves can be divided into domestic and overseas levels, so that economic evaluation can be defined and utilized while taking a broad, objective view.

3.2.3. Deriving Detailed Indicators of Usability Evaluation

Usability evaluation evaluates whether the device has a function that is suitable for the purposes of the user through a user interface that allows the user to easily and conveniently manipulate the complex functions of the product and dictionary definition [22]. Usability evaluation is an evaluation carried out using a mobile application or a product that is in close contact with a user. It is an evaluation method that observes and analyzes actual users’ patterns and reasons for use according to the process of use, and collects the opinions of the users, making it possible to reflect the comprehensive opinions of the hospitals that use medical devices. Usability evaluation has several characteristics. According to ISO, the international organization for standardization, usability evaluation is defined as “validity within a specific use situation (ability to complete a task), efficiency (effort to perform a task), satisfaction (have experience satisfaction)” [23], the degree to which a product may be used to attain a certain goal by a certain user to have meaning. Usability expert Steve Krug’s evaluation of usability evaluation is to judge it based one seven features. “Useful: Can people do what they need? Learnable: Can you tell yourself how to use it? Memorable: Should you learn how to use it again?, Effective: Have you achieved what you want with your product?, Efficient: Time and effort, Desirable: Do people want products?, Delightful: Are they good to use?” [23]. In the process of developing or introducing medical devices, the answer to this question can only be made by the actual user in line with the need for a given medical device.

There are limitations in applying various attributes of usability evaluation and its principles to medical device evaluation as-is. The usability evaluation methods differ depending on the evaluation target or the user. It may be categorized as an analytical method that involves an empirical examination that is often highly sensitive, and usability testing depends on the pros and cons of the device in the user’s situation, so the evaluation depends on how the usability evaluation method is to be used to select a device [22]. Thus, in a usability evaluation, the indicators should be selected after considering the properties to be evaluated, only after which the evaluation should proceed. Therefore, the evaluation should be structured, taking into account the characteristics of the medical device and the future directions of the medical device industry. The purpose of this study is to evaluate medical devices of medical device classes 2–4.

In usability testing, an actual user interacts with one or more medical device models, prototypes, and production units to assess their ease of use, efficiency, and user appeal. Usability tests can be performed in laboratory settings, simulated environments, or actual intended use environments. Usability testing can, among other things, detect usage errors when performed locally [24].

According to the association for the advancement of medical instrumentation [25], the general principles of fitness for use require the development of a compatible design for evaluating the suitability of a medical device for use, and optimization of the user interface for stability and efficiency is required. If the compatibility of the medical device is poor, it will make it difficult to learn how to use the device, inducing usage errors. Therefore, instead of a partially reflective interface, a design for compatibility must be undertaken, reflecting possible developments of effective medical devices, and for medical device usability, problems that occur during use require the application of the international standards of International Electrotechnical Commission (IEC) 62366 to understand and resolve them, and risk management is essential to this task [22].

In summary, measures of ease of learning, ease of use, efficiency, compatibility, and stability should be included in the evaluation of the usability of medical devices. The usability evaluation of medical devices can be divided into efficiency and compatibility depending on the nature of each indicator, in light of the items presented above. Again, efficiency can be divided into efficiency and ease of use to construct an index of efficiency. Compatibility can be achieved by evaluating the compatibility and stability of use environment as an evaluation index.

The evaluation framework of the medical device that is the basis of this study is constructed to include the attributes and characteristics belonging to the medical devices in the general evaluation framework widely used for medical devices as well as medical technologies. Thus, they can be used in each step and for any purpose. The above is summarized and shown in

Table 4. Evaluation framework for optimal selection of medical devices.

Level 1	Level 2	Level 3	Index Reference
Technology evaluation	Performance evaluation	Technology performance evaluation	Medical Device Performance Test Guideline (KFDA) [19]
		Clinical performance evaluation	Clinical Trial Guidelines (KFDA) [19]
	Usability evaluation	Validity Evaluation	New Medical Technology Evaluation (NECA) [20]
		Stability evaluation	New Medical Technology Evaluation (NECA) [20]
Economic Evaluation	Cost evaluation	Cost evaluation	Estimated cost analysis = reverse price analysis [13]
		Cost Efficiency Evaluation	Cost-Benefit Analysis [13]
	Competitiveness Evaluation	Domestic Competitiveness Evaluation	Medical device market share survey [26] (Korea Health Industry Development Institute)
		Evaluation of overseas competitiveness	Medical device market share survey [26] (Korea Health Industry Development Institute)
Usability evaluation	Efficiency Evaluation	Usage efficiency evaluation	Usability evaluation [27] (Korea Health Industry Development Institute-Senior Friendly Industry Support Center)
		Ease of Use Evaluation	Usability evaluation [27] (Korea Health Industry Development Institute-Senior Friendly Industry Support Center)
	Compatibility Evaluation	Use Compatibility Evaluation	Guidelines for Evaluation Criteria for Software Technology [28] (National IT Industry Promotion Agency)
		Usability Stability Evaluation	Guidelines for Evaluation Criteria for Software Technology [28] (National IT Industry Promotion Agency)

.

4. Results

As seen in

Table 5. Prioritization of the evaluation indices for selecting optimal medical devices.

Level 1	Weight	Level 2	Weight	Weight Total	Level 3	Weight	Weight Total	Ranking
Technological evaluation	0.321	Performance evaluation	0.471	0.151	Technology performance evaluation	0.259	0.039	10
					Clinical performance evaluation	0.741	0.112	4
		Usefulness evaluation	0.529	0.170	Validity evaluation	0.307	0.052	9
					Stability evaluation	0.693	0.118	3
Economic evaluation	0.297	Cost evaluation	0.409	0.121	Cost evaluation	0.252	0.031	12
					Cost-effectiveness evaluation	0.748	0.091	6
		Competitiveness evaluation	0.591	0.176	Domestic competitiveness evaluation	0.367	0.064	8
					Evaluation of overseas competitiveness	0.633	0.111	5
Usability evaluation	0.382	Efficiency evaluation	0.697	0.266	Usage efficiency evaluation	0.518	0.138	1
					Ease of use evaluation	0.482	0.128	2
		Compatibility evaluation	0.303	0.116	Usability compatibility evaluation	0.317	0.037	11
					Usability stability evaluation	0.683	0.079	7
Sum	1.000		Sum	1.000		Sum	1.000

, the priorities of the evaluation indicators are different according to each stakeholder. Stakeholders are highly aware of the importance of each evaluation based on their perspectives. The fact that the evaluation framework of this study, which is based on the framework for medical technologies, has different priorities for different indicators shows that the indicators are configured in a balanced manner according to the viewpoint of the respective stakeholder. In addition, the timing of the evaluation of the stakeholders in accordance with the distribution flow of medical devices, whether in the initial stages or after the introduction of other medical devices in the evaluation, causes the method of evaluation of medical devices to differ from the method of evaluation used in prior cycles. AHP analysis of the weights gives highly variable results depending on the perspective and position of each of the stakeholders, whereas the general evaluation framework presented in this study showed each indicator to be balanced with uniform deviations.

In

Table 6. Evaluation indicators and rankings by stakeholders in evaluation framework for optimal medical device selection.

Level 1	Level 2	Level 3	Government				Hospital				Academia				Company
			Lv1 Weight	Lv2 Weight	Lv3 Weight	Total Weight (Rank)	Lv1 Weight	Lv2 Weight	Lv3 Weight	Total Weight (Rank)	Lv1 Weight	Lv2 Weight	Lv3 Weight	Total Weight (Rank)	Lv1 Weight	Lv2 Weight	Lv3 Weight	Total Weight (Rank)
Technological evaluation	Performance evaluation	Technology performance evaluation	0.412	0.318	0.243	0.032 (10)	0.206	0.571	0.263	0.031 (9)	0.234	0.632	0.222	0.033 (11)	0.278	0.420	0.263	0.031 (12)
		Clinical performance evaluation			0.757	0.099 (4)			0.737	0.087 (5)			0.778	0.115 (4)			0.737	0.086 (4)
	Usefulness evaluation	Validity evaluation		0.682	0.318	0.089 (6)		0.429	0.263	0.023 (12)		0.368	0.155	0.013 (12)		0.580	0.5	0.081 (6)
		Stability evaluation			0.682	0.192 (1)			0.737	0.065 (7)			0.845	0.073 (6)			0.5	0.081 (7)
Economic evaluation	Cost evaluation	Cost evaluation	0.309	0.376	0.165	0.019 (12)	0.265	0.636	0.162	0.027 (11)	0.530	0.483	0.153	0.039 (8)	0.273	0.329	0.396	0.036 (11)
		Cost-effectiveness evaluation			0.835	0.097 (5)			0.838	0.141 (3)			0.847	0.217 (1)			0.604	0.054 (10)
	Competitiveness evaluation	Domestic compe-titiveness evaluation		0.624	0.247	0.048 (9)		0.364	0.307	0.030 (10)		0.517	0.356	0.098 (5)		0.671	0.674	0.123 (2)
		Evaluation of overseas competitiveness			0.753	0.145 (2)			0.693	0.067 (6)			0.644	0.176 (2)			0.326	0.060 (9)
Usability evaluation	Efficiency evaluation	Usage efficiency evaluation	0.279	0.707	0.392	0.077 (7)	0.529	0.75	0.462	0.183 (2)	0.236	0.648	0.769	0.118 (3)	0.449	0.671	0.64	0.193 (1)
		Ease of use evaluation			0.608	0.120 (3)			0.538	0.213 (1)			0.231	0.035 (9)			0.36	0.108 (3)
	Compatibility evaluation	Usability compatibility evaluation		0.293	0.273	0.022 (11)		0.25	0.297	0.039 (8)		0.352	0.404	0.034 (10)		0.329	0.433	0.064 (8)
		Usability stability evaluation			0.727	0.059 (8)			0.703	0.093 (4)			0.596	0.050 (7)			0.567	0.084 (5)

, we see that the importance of each indicator was ordered differently, and the priorities of the indicators differed. First, opinions from the Tier 2 evaluation showed that governments, hospitals, academia, and companies all have different priorities. This was known from the early studies, which identified that this mainly occurs from the use of different evaluations, and because each stakeholder differs due to their respective needs and the details of the distribution stages of different medical devices. Because of the importance of the effectiveness, safety, and performance of medical devices, the priority of government officials designated by the questionnaire for these indices is high, and the major property of economic efficiency is cost-effectiveness, which has a high priority in academia, government, and hospitals. Usability was found to be highly significant for hospitals and companies using actual medical devices.

Government officials ranked the indices in order of usability, competitiveness, and efficiency; hospitals ranked in order of efficiency, cost, and compatibility; the academic sector ranked in order of competitiveness, cost, and efficiency; and company experts ranked in order of efficiency, competitiveness, and usability. The evaluations on Tier 2 can be thought of as reflecting the difference in governments and companies in the importance they give to the evaluation indices, while hospitals and academia differed in their respective priorities of technology and economic feasibility and usability, while academics tended to prioritize economic performance indices over other indicators.

The indicators with the highest priority are stability for the government, ease of use for hospitals, cost efficiency for academia, and use efficiency for companies. Although the hospitals and companies were similar in terms of the number of items, there was a difference in prioritization among the sub-indices, and the government had a higher weight on the stability index, but the other index-weighted values were relatively high. There does not appear to be a tendency to look at many indicators in a balanced way among hospitals; four availability indicators had higher-weighted equivalent values for ease of use, which showed even higher deviations. As this concerns medical devices, the ease of use and efficiency are important. In academia, a higher priority for cost-effectiveness indicators was shown. As with hospitals, their weighted conversion values also showed large deviations from the other indicators and demonstrated a high priority for economic indicators, unlike those of other stakeholders. When we compare the weighted conversion value of the priority indicators, it found that the industry prizes a variety of indicators, such as efficiency index, domestic competitiveness, and clinical performance.

Therefore, it can be said that the narrow framework for each stakeholder has been taken into account in the evaluation framework from the perspective of all the stakeholders, as well as throughout the entire life cycle of the medical device as shown in Figure 1. Through this evaluation frame, the government can include the usability characteristics of the medical device; hospitals can obtain information about the technological characteristics of the medical device and review the economic indicators; and companies can concentrate on selecting medical devices based on the development of the product and analyzing their technological characteristics.

The AHP analysis of each stakeholder’s evaluation index was able to determine the priorities of opposing criteria and multiple alternatives. It does not assert a conclusion but suggests a general framework by analyzing representative evaluations of various indicators. When using evaluation frames for each stakeholder, it weighs more highly the indicators, reflecting the characteristics of each stakeholder.

5. Conclusions and Discussion

The future development of the medical device industry is promising, but it is imperfect due to the competing natures of various industries. The presence of different perspectives and values pursued by various stakeholders when selecting medical devices can pose a major difficulty to the medical device industry because it can cause deviations to a single outcome.

Therefore, it is necessary to factor in the viewpoints of stakeholders and evaluate their values properly through an evaluation framework that assesses medical devices objectively and comprehensively.

The results of the AHP analysis of the representative evaluation methods presented in the evaluation framework showed that the importance of many indicators was rated differently by each stakeholder. The higher weights that were given to certain indicators by different stakeholders show that the evaluation framework takes various stakeholders’ perspectives into account. In general, the deviation of the weight value of the proposed evaluation indicators is constant across indicator categories. Additionally, a particular frame can be constructed to suit the specific viewpoint of each stakeholder in the AHP analysis of the characteristics of each specialized indicator.

The AHP analysis of each stakeholder’s evaluation index was able to analyze the conflicting criteria and priorities. It does not assert a conclusion, but instead suggests a general framework through an analysis of stakeholder representatives’ evaluations of various evaluation indicators. When using the evaluation frame for each stakeholder, the weight value reflecting the characteristics of that stakeholder may be used for the indicators

The values of the weights from the AHP analysis can be given numerically as a representative set of evaluation priorities that are applicable to the evaluation of medical devices. In this way, not only can a comparison of different medical devices be conducted, but the superiority of a single medical device can be confirmed (by a higher-weighted number). By including usability evaluations that were not included in the general evaluation framework for medical devices, we were able to broaden the evaluation base of medical devices and balance the evaluations conducted by different stakeholders. Considering that the usability evaluation showed a high priority in the survey results, we think that usability evaluation will become an important factor in future general medical device evaluation frames.

The AHP analysis showed that the weights vary widely depending on the viewpoint and position of each stakeholder. However, a uniform deviation was found in the indicators from the general evaluation framework presented in this study. Therefore, it can be said that the narrow framework of each stakeholder has been factored into this evaluation framework, which may be suited to the particular perspective of each of the stakeholders, as well as being applicable throughout the entire life cycle of the medical device. Through this evaluation frame, the government can include the usability characteristics of the medical device, while hospitals can obtain information on the technological characteristics of the device and review its economic aspects, and companies can select and concentrate on the development of the product and analyze the technology.

By objectively evaluating medical devices from various points of view, the government can contribute to investments, as well as industrial development, by providing objective data on numerous factors, including economic efficiency, usability, safety, and effectiveness of medical devices. Hospitals can obtain information about medical devices and evaluate their usability, thereby enhancing the reliability of medical device selection. This can raise the efficiency of medical device introduction, improve the quality of medical care, and maintain the industrial ecosystem of the users. Since the company has access to a framework that allows stakeholders’ interests to be considered from the initial stage of development to the introduction of medical devices, it can unify the consideration of a diversity of aspects currently not treated together and thereby maximize profits. By understanding the stakeholders who are most focused on the evaluation framework for selecting optimal medical devices and studying methods to evaluate medical devices that take these perspectives into consideration, we hope to spread a better understanding of the different ways of assessing the value of medical devices that will result in a more objective and transparent evaluation of medical devices, which will provide a good basis for the development of the medical device industry ecosystem.

6. Scope and Limitation of Research

The AHP analysis method used in this study has a limitation, as it required the sincere response of the respondent, and it is difficult to consistently respond to more than a certain number of items. The number of subjects that could be analyzed by expert choice software, which is the AHP decision-making program used in this study, was limited to 25 respondents, which implies that the maximum number of subjects that can be analyzed is 25 people. If more answers are received from the survey respondents in future studies, it is assumed that better results will be obtained.

7. Topics for Further Research

Previous studies conducted by Kim et al., [13,14,15] and Lee. J. et al., [29,30,31] and Lee. M. et al., [32,33,34] and Yoon et al., [35] presented frameworks which were developed into quantitative analyses, and the evaluation frame of this paper can be developed into a digitized evaluation study of medical devices through quantitative analysis in the future.