Interoperability Reference Models for Applications of Artificial Intelligence in Medical Imaging

Abstract

Medical imaging is currently being applied in artificial intelligence and big data technologies in data formats. In order for medical imaging collected from different institutions and systems to be used for artificial intelligence data, interoperability is becoming a key element. Whilst interoperability is currently guaranteed through medical data standards, compliance to personal information protection laws, and other methods, a standard solution for measurement values is deemed to be necessary in order for further applications as artificial intelligence data. As a result, this study proposes a model for interoperability in medical data standards, personal information protection methods, and medical imaging measurements. This model applies Health Level Seven (HL7) and Digital Imaging and Communications in Medicine (DICOM) standards to medical imaging data standards and enables increased accessibility towards medical imaging data in the compliance of personal information protection laws through the use of de-identifying methods. This study focuses on offering a standard for the measurement values of standard materials that addresses uncertainty in measurements that pre-existing medical imaging measurement standards did not provide. The study finds that medical imaging data standards conform to pre-existing standards and also provide protection to personal information within any medical images through de-identifying methods. Moreover, it proposes a reference model that increases interoperability by composing a process that minimizes uncertainty using standard materials. The interoperability reference model is expected to assist artificial intelligence systems using medical imaging and further enhance the resilience of future health technologies and system development.

1. Introduction

A key component of the Fourth Industrial Revolution is the convergence of information and communication technology (ICT). Artificial intelligence and big data comprise the ICT field and is highly expected to be future growth industries. As such, the scope in which data are used is rapidly changing. In the past, research using data was applied in controlled environments in which resources could be collected. The range of artificial intelligence and big data has expanded the data application to the medical imaging domain [1,2]. Therefore, artificial intelligence and big data are applied to medical imaging.

As the range in which data could be applied expanded from a single institution to multiple institutions, a uniform data standard was important, and data collectors faced challenges in controlling the entire process by which data were collected and applied [3]. Medical imaging standards such as Health Level Seven (HL7), Digital Imaging and Communications in Medicine (DICOM), and other standards were implemented with multiple institutions using the same standards [4]. There is also research being conducted in developing de-identifying methods in order to protect any personal information present in medical imaging collected from environments in which collectors are unable to monitor the entire data collection process. De-identifying methods process any personal information that could potentially be used to specify an individual and retain any data necessary for research. In the case of de-identifying methods, as any personal identifiers are removed, it does not conflict with personal information protection laws. These methods minimize the risk of personal information leaks and maximize accessibility towards data [5].

Multiple medical institutions have their medical specialists review medical imaging that have different measurement values due to differing medical equipment. As interest in artificial intelligence capable of deciphering medical imaging increases, there is a need for quantifying medical imaging for more precise diagnoses [6]. Currently, a large amount of artificial research in the medical imaging field is being conducted with magnetic resonance imaging (MRI), which is used to identify and predict diseases and to provide forecasts after prognosis, among others [7]. In the case of MRI, different environments at medical institutions and the conditions of the patient may affect the magnetic field, resulting in vastly different measurement values even with the same subject. Such results are defined as measurement uncertainty, and other medical imaging measurement equipment have similar occurrences of measurement uncertainty. If measurement uncertainty can be minimized, medical imaging data can have a meaningful value regardless of which institution collected it. To minimize uncertainty, the measurement value of phantoms, which are standard materials, can be used as standardized values in a process that supplements actual measured values.

In order to establish the interoperability of medical imaging and allow for medical imaging to be used at different institutions, this study aimed to propose an interoperability reference model that uses methods which minimize measurement uncertainty, enable de-identifiers, and standardize pre-existing data used for research.

The contributions of this study can be summarized as follows:

• It proposes an approach to increase interoperability of medical imaging measurements from different medical institutions by using methods reference phantoms.
• It addresses the systematic compatibility issue by applying pre-existing medical data specification standards in HL7 and personal information protection methods in the Health Insurance Portability and Accountability Act (HIPAA) and DICOM.
• It follows the characteristics of interoperability as defined by the Healthcare Information and Management Systems Society, Inc. (HIMSS), and applied in medical imaging.

2. Materials and Methods

2.1. Materials

2.1.1. Interoperability

The general definition of interoperability means the characteristic of different systems being compatible with one another without any limitations. In the healthcare information field, interoperability is defined as a function by which different medical information systems, medical equipment, and applications can optimize the health of an individual or humanity through compatibility [8].

HIMSS includes three stages for interoperability in healthcare and medical technologies [9] as “foundational”, “structural”, and “semantic”. Each stage can be summarized as follows:

• Foundational: A basic characteristic of interoperability is establishing an interconnectivity protocol between two different medical systems and enabling them to share medical data.
• Structural: An exchange format and structure must be defined in order to maintain the integrity of medical data.
• Semantic: No issues should arise in the process of safely exchanging and interpreting medical data between two or more medical systems.

According to HIMSS, interoperability is the ability to safely exchange or interpret medical data created by multiple systems or institutions without any excessive effort by the user. The establishment of interoperability is an element that is essential to the use of medical imaging data in the future. This study composed a reference model based on interoperability as defined by HIMSS.

2.1.2. HL7 FHIR

For the sharing of medical imaging and medical data, an HL7 Fast Healthcare Interoperability Resources (FHIR) standard server was used. HL7 is a standards committee established to promote the exchange of information between different medical information systems and recognized worldwide with countries such as the United States, United Kingdom, Australia, Japan, Taiwan, New Zealand, and others designating and using HL7 as national standards [10]. The HL7 Application Programming Interface (HAPI) FHIR server is the official FHIR test server of HL7 [11]. There are multiple test servers such as “Graham’s Test Server”, “Healthcare Services Platform consortium (HSPC) Sandbox”, and “Vonk”, among others, but the HAPI FHIR server is open-source and can be operated on local systems. Operation guidelines are also available and were used in this study.

The HAPI FHIR server environment was executed on Windows 10 with “jdk1.8.0_211” and “jre1.8.0_211” used for Java based application development software. Library builds and management needed for server operations used Apache Maven with Apache Tomcat 8 used to execute the server.

HL7 FHIR designates XML and JavaScript Object Notation (JSON) as the data format standard and uses the term “resource” as the smallest unit of exchanged information [12]. HL7 officially lists resources and the respective resource used for different layouts is listed in

Table 1. Resources for Health Level Seven (HL7) Fast Healthcare Interoperability Resources (FHIR) Layouts.

Layout	Resource
Clinical	Observation, ImagingStudy, Procedure
Identification	Patient, Practitioner, Organization
Workflow	Schedule, Task, Encounter
Financial	Coverage, Invoice, Claim
Conformance	OperationDefinition, ImplementationGuide
Infrastructure	Questionnaire, MessageHeader, Parameters

.

Resources are constantly being added and edited following further research by HL7, and there are currently 146 resources listed as of 2021 [13].

2.1.3. DICOM and CTP

Along with HL7 FHIR, DICOM is another medical imaging data standard that contributes to the establishment of increased interoperability. In the case of general medical imaging, only information regarding the image itself is included such as Bitmap, Tagged Image File Format (TIFF), JPEG, and the bit count of each pixel. However, DICOM medical imaging also includes the name of the patient, data imaging date, medical equipment used for imaging, and the imaging specialist, among other information [14].

The clinical trial processor (CTP) of the Medical Imaging Resource Center (MIRC) used de-identifying modules for medical imaging. According to a study conducted by K.Y.E. Aryanto, M. Oudkerk, P.M.A. van Ooijen, the modules showed the highest performance rate in different environments among all freely accessible DICOM de-identifying modules [15]. CTP’s de-identifier tool follows DICOM PS 3.15 Annex E’s Attribute Confidentiality Profile. The section includes a total of seven profiles comprising a “Basic Profile” and six other profiles that can be applied to different situations [16].

As is the case with HAPI FHIR, CTP was operated on Windows 10 and used “jdk1.8.0_211” and “jre1.8.0_211.”

2.2. Methods for Adapting Medical Imaging

2.2.1. Designing a Medical Imaging Data Exchange System

As this study focuses on proposing a reference model regarding medical imaging, a data exchange system was created using a HAPI FHIR server according to HL7 FHIR standards. HAPI FHIR supports the XML and JSON data formats with system compatibility confirmed by sending randomly generated medical data in XML format to the server. The medical data used were made by resources according to HL7 parameters, and a resource called Phantom was added to the pre-existing resources (ImagingStudy, Patient, Endpoint) used. As the Phantom resource is not a resource in pre-existing HL7 FHIR, this study will discuss this matter in detail in the section regarding measurement uncertainty designs. The Patient resource handles any data relating to the patient while the ImagingStudy resource focuses on data relating to medical imaging. The actual medical imaging does not exist within this resource and instead includes Unique identifier (UID) values such as Series UID and Instance UID. The Endpoint resource handles the address in which the actual medical image is saved. In Figure 1, the diagram shows the relationship among other resources based on the ImagingStudy resource.

2.2.2. Personal Information Protection Methods within Medical Imaging

In order to protect personal information within medical imaging, this study referred to the privacy regulations related to the Protected Health Information (PHI) section of the Health Insurance Portability and Accountability Act (HIPAA). In the relevant section, there are the expert determination and the safe harbor methods [17].

The expert determination method requires experts to take part in an 11-step process in which de-identification is processed with personal information existing within medical data. The first step requires the determination of identifiers that relate to personal information and, in the next steps, to prepare a model for potential threats, select the necessary information, and conduct an analysis of minimized personal information damage. Afterwards, a review of the actual threats is conducted with sample data and provided to external sources in the final step. This is a high level de-identifier method but it requires a significant amount of time for data to be collected and processed and has the downside of requiring an expert.

On the other hand, the safe harbor method defines 18 different elements that can be classified as personal information and makes sure that such data are not included with medical data. Personal information such as one’s name, date of birth, phone number, email address, date of hospitalization, and leave are accepted as protected personal information without the involvement of an expert as long as they are removed. This method allows for a quick and easier application compared to the expert determination method.

DICOM also offers a de-identifying method. Information regarding the security and management of medical imaging can be found in DICOM PS 3.15. Moreover, the Attribute Confidentiality Profile in Annex E offers a method for a de-identifier process for DICOM medical imaging. There are a total of seven profiles with the most basic profile being the “Basic Profile.” The other six profiles can be applied according to different situations. The de-identifier subjects classified by DICOM’s basic profile satisfy the requirements of HIPAA’s safe harbor [18]. As such, the study used DICOM’s basic profile as it enabled the use of medical imaging for medical data and as it desired to use the study data of the quickly developing artificial intelligence technology.

De-identifier methods through CTP were applied according to DICOM PS 3.15 Annex E. Attribute Confidentiality Profile’s basic profile and all authentic medical imaging within the CTP module were processed with the de-identifier method according to the basic profile. Figure 2 below displays the de-identifier items according to the DICOM basic profile within the CTP module.

2.2.3. Minimization of Measurement Uncertainty in Medical Imaging

With medical imaging, there are a variety of measurement equipment and methods such as MRI, CT, X-ray, and others, but as aforementioned in this study, the main focus will be on MRIs as they are the subject of research regarding artificial intelligence. In designing a measurement uncertainty minimization method, an analysis of the elements influencing uncertainty and a survey of the minimization methods used is needed. Based on research by Peter Kellman and Michael S Hansen, an analysis of data elements needed to design a medical imaging measurement uncertainty method was conducted. According to research by them, measurement uncertainty elements influencing T1 mapping values are as displayed in

Table 2. MRI Measurement Uncertainty Elements in Medical Imaging [19].

Category	Detail
Protocol parameters	Matrix size
	Parallel imaging
	Partial Fourier
	Flip angle
	Echo-spacing (BW and TR)
	Raw filter
Sequence design	Slice profile
	Inversion pulse efficiency and BW
	SSFP steady state run-up
Scanner adjustments	Shim
	Center frequency adjustment
	B1 transmit ampl (flip angle)
	z-FOV
Fit model	2 vs. 3 parameters
	Multi-fit MagIR vs SPIR
Tissue characteristics	T2
	MT
	Fatty infiltration
	Flow
Patient	Heart rate
	Respiratory motion

BW: Bandwidh; TR: Repetition Time; SSFP: Steady State Free Precession; SPIR: Spectral Presaturation with Inversion Recovery; MT: Magnetization Transfer.

[19]. T1 mapping is a cardiac magnetic resonance (CMR) imaging method that allows for the early detection of cardiovascular fibrosis [20].

Uncertainty elements and factors that affect T1 mapping values are protocol parameters, sequence design, scanner adjustments, fit model, tissue characteristics, patient, and others. Factors such as the flip angle and heart rate affect T1 values and may cause uncertainty.

This study proposes a method of using standardized phantoms in order to minimize the uncertainty elements that exist within such medical imaging. Environmental measurement factors affect measurement uncertainty, and if these factors are assigned a standardized value, all medical imaging measurement values can be calibrated and quantified according to those standards. As such, a standardized phantom that can provide standardized values is needed.

Gabriella Captur and Peter Gatehouse conducted research on T1 value measurement phantoms that can be used in cardiovascular MRI diagnoses, and the standardized phantoms used in this study were composed in the structure shown in Figure 3 [21].

Phantoms are composed of a gel matrix that surrounds multiple tubes. Each tube is a mixture of agarose and NiCl₂, and the T1 value can be created according to the situation or target needed to be measured by T1 values in the blood before and after the insertion of a contrast medium by changing the concentration of NiCl₂.

As T1 values from standardized phantoms are used as standardized values, it is important to maintain stable and consistent levels. The preservation periods are affected by the storage and usage temperature.

Based on this research, this study considered the storage and usage temperature of phantoms, the targeted area, and the period of stability and, in the case of the inner tubes and gel matrix within the phantom, considered the components, concentration, and T1 value to design a data model. Data elements for phantoms, tubes, and the gel matrix are listed in

Table 3. Phantom Data Model Elements and Explanation.

Data Elements	Explanation of Element
ID	Phantom ID
Volume	Phantom volume
Length	Phantom length
Inner body cross section	Phantom inner cross-section
Correct orientation	Correct phantom orientation to be used
Date of manufacture	Date of manufacture
Number of tubes	Number of tubes within phantom
Period of stability	Period of stability
Magnetic field	Strength of the MRI magnetic field applied on the phantom
Target Area	Target area to be measured

,

Table 4. Tube data model elements and explanation.

Data Elements	Explanation of Element
ID	Tube ID
T1 value	Measured T1 value
T2 value	Measured T2 value
Component	Components within tube
Date of manufacture	Date of manufacture
Period of stability	Period of stability
Description target	Value displayed by tube
Storage temperature	Adequate temperature for storage
Usage temperature	Adequate temperature range in use

and

Table 5. Gel matrix data model elements and explanation.

Data Elements	Explanation of Element
ID	Gel matrix ID
T1 value	Measured T1 value
T2 value	Measured T2 value
Component	Gel matrix components
Date of manufacture	Date of manufacture
Period of stability	Period of stability
Description target	Value displayed by tube
Storage temperature	Adequate temperature for storage
Usage temperature	Adequate temperature range in use

.

Pre-existing medical imaging data did not handle phantom-related data models, and there were no resources for the relevant data model on the HL7 FHIR server. As such, extension functionalities were used in compliance to the HL7 FHIR operations rules [22].

When using the extensions, data elements not defined by HL7 FHIR can occur. Through the use of extensions, data elements related to phantom, tubes, and the gel matrix were included in Figure 4.

As using extensions on HL7 FHIR requires data elements that did not exist previously, a process defining each data element is needed. This process is called profiling and each profile on specific resources and extensions are saved in StructureDefinition within the HL7 FHIR server. The profiling module used Forge [23] and the data elements of the phantom are defined in

Table 6. Example of Phantom Data Element Profiling.

Data Element	Path	Definition	Type
ID	Phantom.Phantom_id	Phantom intrinsic ID	id
Volume	Phantom.Volume	Phantom volume	decimal
Length	Phantom.Length	Phantom length	decimal
Inner body cross section	Phantom.Cross_Section	Inner cross-section within phantom	decimal
Correct orientation	Phantom.Orientation	Correct orientation of Phantom in use	string
Date of manufacture	Phantom.Date	Phantom date of manufacture	date
Number of tubes	Phantom.tubesNumber	Number of tubes within phantom	positiveInt
Period of stability	Phantom.StablityPeriod	Phantom period of stability	positiveInt
Magnetic field	Phantom.MagneticField	MRI strength applied to phantom	decimal
Target area	Phantom.TargeArea	Measured target area of phantom	string

and

Table 7. Example of Tube, Gel Matrix Element Profiling within Phantom Data.

Data Element	Path	Definition	Type
Tube/Gel Matrix	Phantom.Material	Tube or gel matrix specified	id
T1 value	Phantom.Material.T1	Measured T1 value from material	decimal
Component	Phantom.Material.Component	Material components	decimal
Description target	Phantom.Material.DescriptionTarget	Target described by the material	decimal
Storage temperature	Phantom.Material.StorageTemperature	Adequate storage temperature	string
Usage temperature	Phantom.Material.UsageTemperature	Adequate usage temperature	date

.

The Phantom resource must be referenced as it must be used with the pre-existing ImagingStudy resource. Using the functionality allowing for references to the Specimen resource in the lower series section of the ImagingStudy resource, the design including information on phantom can be seen in Figure 5. This study used the Specimen resource as the Phantom resource.

3. Results

3.1. Results of the Medical Imaging Data Exchange System

Using the HAPI FHIR server composed based on the HL7 medical data standard, the results of transmitted XML information from ImagingStudy, Patient, and Endpoint were confirmed. Data were successfully saved only in cases in which the actual resource structure classified by the FHIR was used, and its results can be seen in Figure 6.

In the case of the ImagingStudy resource, the ID references the Patient, Endpoint, and Specimen resource. The referenced IDs are 70104 for the Patient resource, 70103 for the Endpoint resource, and 70107 for the Specimen resource. The Specimen resource was used as the resource to store information regarding the phantom. Below the respective section, UIDs related to the series and instance are included as information on the medical imaging.

In the Patient resource sample XML transmission results in Figure 7, the id 70104 can be seen as referenced in the ImagingStudy in Figure 6. Moreover, basic patient-related information is included.

In the Endpoint resource sample XML transmission results in Figure 8, the data related to Endpoint id 70103 referenced by the ImagingStudy resource in Figure 6 can be seen. In this resource, the server address where the medical image is saved and information regarding the server’s connection method can be confirmed.

In Figure 9, the specimen resource example of the XML transmission results using the Phantom resource includes information on the phantom using the Extension. The Specimen id 70107 referenced by the ImagingStudy resource in Figure 6 can be deduced as the relevant data.

Each data element complies to the StructureDefinition rules set within the server when designing the Phantom resource. As such, it includes the StructureDefinition URL information that defines each element per extension that relates to the phantom data element.

3.2. Results of Personal Information Protection within Medical Imaging

When comparing MRI data using the CTP from the original to the data after the de-identifier process, data that are at risk of personal information leaks are either removed or de-identified.

In Figure 10, data that needs to be protected for personal information related issues is redacted in the cardiovascular MRI original image. Redacted information are items related to DICOM de-identifier methods and basic profile and include the patient’s name, MR imaging location, name of the specialist, and date of imaging, among others. Information processes by de-identifier processes are shown in Figure 11.

In Figure 11, data elements that were shown in the original are nonexistent or changed to different value. The risk of personal information leaks are minimized and only information needed for research is retained.

3.3. Minimization Process of Measurement Uncertainty in Medical Imaging

Figure 12 displays the measurement uncertainty minimization process using standardized phantom data and results from an analysis of measurement uncertainty from medical imaging.

Environmental variables that are classified as uncertainty elements when measuring medical imaging can have an effect on MRI T1 values. In order to calibrate the measurement values, the phantom T1 value can be calibrated based on the phantom’s standardized T1 value affected by environmental variables. Through this process, the measurement value calibration function can be deduced and applied to the patient’s measured MRI T1 value. As a result, the calibrated MRI T1 value is attained with the respective value being the same regardless of where the MRI T1 medical imaging was collected as long as the same standardized phantom T1 value is applied.

For example, in order to detect diseases in specific areas such as cardiovascular symptoms, the patient MR imaging and phantom are captured together. In this case, environmental variables such as sequence design or protocol parameters can be applied as factors of uncertainty and affect the MRI T1 value. We can use a standardized phantom that displays the MRI T1 based value on cardiovascular symptoms. By using the phantom standardized MRI T1 value of cardiovascular symptoms saved on a HL7 FHIR server, we can calibrate the phantom MRI T1 value. By applying the same level of calibration as the phantom MRI T1 value to the patient’s MRI T1 measurement value, we can derive the patient’s calibrated MRI T1 value.

Figure 13 shows that when such a process is applied, different medical imaging from different institutions can be decoded on the same quantifiable levels.

If there is information about the phantom that reflects the characteristics of the other measurement site, it can save the phantom data and use the same method.

Through the quantified phantom, which minimizes measurement uncertainty elements, MRI results can be translated into quantitative MRI (qMRI) and provide high quality data to artificial intelligence medical imaging diagnosis equipment. Moreover, this process would contribute greatly to clinical practices by allowing the earlier detection of diseases, the replacement and supplementation of biopsies, and precise differentiation of diseases due to numerical differences among others [24].

3.4. Interoperability Reference Model

The final interoperability reference model structure is displayed in Figure 14. Centered on the medical data exchange system, the medical imaging measurement calibration system, which is an uncertainty minimization system, and personal information de-identifier system are connected with one another with the three systems comprising the interoperability reference model.

The collecting institution applies to the system collecting medical imaging including hospital Picture Archiving Communication System (PACS). Medical imaging received through the medical data exchange system is processed through the de-identifier and measurement uncertainty minimization system, quantified and saved as de-identifier-based medical imaging data. Medical imaging data that minimizes the accessibility limitations of data due to personal information protection regulations and uses a quantified measurement value with standardized data structures will enable use in environments that require large amounts of data, such as in artificial intelligence research.

4. Discussion

The application range of data including medical imaging is quickly expanding, and the need for data collected and used in limited locations is now being demanded in multiple different locations. As such, the need to establish an environment that enables the exchange and use of medical imaging among different institutions is becoming more important and is the main goal of establishing interoperability.

In approaching medical imaging from the concept of interoperability, this study allowed for the exchange of medical data between two systems through an HL7 FHIR server and designated the exchange format and structure through resources and XML according to HL7 FHIR standards. Moreover, for the safe exchange of data, it used a CTP module using de-identifier methods by HIPAA and DICOM in order to minimize personal information related issues that could occur systematically. It also designed a process that minimized measurement uncertainty through the use of phantoms in order to display meaningful values regardless of the institution in which the medical imaging data derived from and simultaneously confirmed the de-identifier and medical data standards currently being used.

In the case of medical imaging, as different measurement standards are used for each hospital and medical equipment, it is normal for a medical professional to perform a qualitative review. Uncertainty minimization methods using standardized phantoms allow medical imaging to be displayed in a qualitative manner and also allows for the use of quantitative medical imaging between different institutions when the same standards are used.

Pre-existing medical data standards such as HL7 FHIR can be used for multiple medical institutions to use the same medical data standards and guarantee compatibility between systems. Moreover, by using non-identifier methods such as HIPAA and DICOM for the protection of personal information in medical data, legal issues can be minimized as well.

When applying the interoperability features as defined by HIMSS, the first basic feature is defined as such that medical data should be able to be shared between two systems. We confirmed that medical imaging related data such as imaging study, patient, endpoint, and phantom data can be transferred through a medical data exchange server based on HL7 standards.

The second feature as defined by HIMSS states that formats and structures must be defined so that data cannot be tampered with or modified. The data formats within the HL7 FHIR server are defined as XML and JSON with the data structure following Resource.

The third feature states that no issues should arise in the safe exchange, use and interpretation of medical information between two or more medical systems. By using phantom data, medical images collected from different locations can be used in a qualitative manner by applying the same standards.

The interoperability reference model created can enable the use of medical imaging on a large scale. As a large amount of data is required for deep learning, the process of increasing the quality of data before image training is important. It is necessary to check whether quantified images increase deep learning performance; we will conduct a further study that shows the results of applying deep learning techniques to MRI images with a reference model.

The use of medical devices that apply artificial intelligence is growing within the medical devices industry. For example, integrating the reference model to other medical visualization modules that can transfer informative medical volume such as clustering data [25] and are suitable for handing large amounts of medical imaging data [26] is able to contribute to improving medical image artificial intelligence performance.

In addition, the U.S. Food and Drug Administration (FDA) has defined the concept of Software as a Medical Device (SaMD) and allowed the foundation for artificial intelligence to be certified as a medical tool [27]. The interoperability reference model proposed by this study is believed to be able to provide further developments to the artificial intelligence medical devices industry.