Breakthrough and Challenging Application: Mixed Reality-Assisted Intracardiac Surgery

Abstract

Background: While several studies investigate the utility and clinical value of 3D printing in aiding diagnosis, medical education, preoperative planning, and intraoperative guidance of surgical interventions, there is a scarcity of literature regarding concrete applications of mixed reality in the cardiovascular domain due to its nascent stage of study and expansion. This study goes beyond a mere three-dimensional visualization of the cardiac district, aiming to visualize the intracardiac structures within the scope of preoperative planning for cardiac surgery. Methods: The segmentation of the heart was performed through an open-source and a professional software and by applying different procedures. Each anatomical component of the heart, including the aortic valve, was accurately segmented and a 3D model was built to represent the entire heart. Results: Beyond the three-dimensional visualization of the cardiac region, the intracardiac structures were also segmented. A mixed-reality app was implemented with the possibility of exploding the model, interacting with it, and freely sectioning it with a plane. Conclusions: The proposed segmentation methodology allows a segmentation of the valve and the intracardiac structures. Furthermore, the mixed-reality app has confirmed the potential of this technology in diagnostic and preoperative planning, although some limitations should still be overcome.

1. Introduction

In recent years, augmented reality (AR), virtual reality (VR), and mixed reality (MR) have emerged as prominent technologies across various fields, including medicine. While AR overlays virtual elements onto the real environment in real-time, VR provides users with an entirely immersive experience [1]. In contrast, MR blends the digital and physical worlds, enabling interactions between the two dimensions. This technology allows virtual objects to be placed in the real environment or facilitates digital presence in the physical world for collaboration across different locations and times [2]. The introduction of such technologies in surgery brings several advantages, including streamlined procedures, enhanced medical training, improved patient data visualization, and surgical support.

Several studies [3,4,5,6,7,8,9,10,11,12] examine the utility and clinical value of three-dimensional (3D) reconstruction and printing in aiding diagnosis, medical education, preoperative planning, and intraoperative guidance of surgical interventions. However, there is a scarcity of literature regarding concrete applications of MR in the cardiovascular domain due to its nascent stage of study and expansion.

In the field of cardiovascular medicine, conventional 2D medical images and their corresponding 3D reconstructions are integral components of an efficient surgical planning system. The segmentation process is at the base of the 3D reconstruction and it can be conducted from different medical imaging techniques such as computed tomography (CT), Magnetic Resonance Imaging (MRI), doppler ultrasound [13], and advance imaging [14]. Moreover, to understand and simulate the physiological behavior of the heart, computational models have been studied to descript electro-physiology [15] and personalized morphology [16]. However, these processes still request deep analysis and development.

However, current practices often rely on flat screens for viewing 3D images, resulting in the loss of crucial spatial information necessary for understanding anatomical regions of interest and their associated pathologies. In the last three years, some research studies [10,17,18,19] have already presented the idea of introducing MR or VR in the cardiovascular domain to overcome the limitations of standard surgical procedures. These limitations include the need for continuous switching between CT image visualization and the patient’s body, as well as the restricted two-dimensional visualization of complex anatomical regions. The usefulness of the 3D visualization in pre-interventional planning and the assistive intraoperative tool has been already demonstrated in different studies [20,21,22]. Surgeons consider favorably this type of application where they can discuss a 3D model at the same time and see the 3D model in a holographic mode during the surgery. This functionality also reduces problems related to miscommunication, limited access to the mapping system, and direct interaction with technical instrumentation [23].

Generally, in VR and MR experiences specifically developed for surgical planning or training, the 3D model is visualized as a hologram and no interactions are considered, although the device allows far more elaborate implementations. The 3D model should be ideally observed in 360 degrees and virtually dissected according to the type of pathology. However, the complexity of the anatomical region and surgical procedures still limits the application of MR in cardiological surgery. The selection of the software for segmentation and the manual intervention is fundamental. Ye and co-workers [24] proposed one of the first MR applications with a 3D model of the heart for the first version of HoloLens. In this study, the segmentation was conducted with an open-source software 3DSlicer (version number 5.6.2.), and the inner membranes of the anatomical component are not clearly visible. Similarly, the authors of [25] presented an MR visualization of the heart with a 3D model built with a 3DSlicer, which allows users to section the model as desired. In [26], the software Mimics Innovation Suite Medical (Materialise, Leuven, Belgium) was used for the segmentation of the heart to subsequently 3D print it. For the same purpose, Materialise was used in [27] for the processing of CT data to finally present a procedure to develop and produce a physical 3D stamp of a Left Ventricular Aneurysm. Also, in [28], Materialise was used to segment the heart from multidetector computed tomography or 3D balanced steady-state free precession images with the aim of implementing 3D printing of the organ. Although 3D printing provides a tangible and realistic perception of the model [29], the holographic model can be sectioned and it can show outer and inner structures of the model, as proposed in this study.

Probably, the most complete MR application for heart segmentation, including intracardiac membranes and surfaces, is ARTICOR (developed by Artiness srl, Milan, Italy), which was used in [30]. In this case, the segmentation is semi-automatic, and the MR experience is exclusively for holographic visualization. Although the MR experience increased the attention and enthusiasm of the medical staff, ref. [30] demonstrates that no significant improvement of knowledge of the medical users was found. In fact, although an accurate 3D model is fundamental for the surgical planning process, also developing a Heart Navigation MR application full of different interactions is important to assist physicians.

The aim of this study is to propose an MR experience realization. The potential and limits of this methodology would be discussed, as well as the challenges to be overcome to meet the wishes of surgical teams. This project, conducted in collaboration with cardiothoracic surgeons at the Umberto I Hospital, involved reconstructing the 3D volume of a patient’s heart affected by aortic stenosis, starting from CT scans. Each anatomical component of the heart, including the aortic valve, was segmented and a 3D model was built to represent the entire heart. The critical aspect lies in achieving a reproduction of the anatomical region of interest, including internal structures. This alignment was achieved through meticulous manual segmentation work and by using a professional software to segment in a distinguishable way the outer and inner surfaces of the membranes. The MR app was implemented to propose different interactions and activities with the 3D model other than the prefixed visualization. Finally, the cardiac surgery team has reviewed the model and tried the MR experience to evaluate the accuracy of the segmentation and the efficiency in supporting orientation and decision making.

2. Materials and Methods

2.1. Data Acquisition

This project included data from a patient affected by aortic stenosis awaiting aortic valve replacement surgery.

The dataset used for the acquired heart valve surgery use case included CT images with a slice thickness of 1 mm, a spacing between slices of 0.5, a spiral acquisition, and an in-plane acquisition matrix of 512/512 pixels. The Iodine contrast agent enhanced the phases acquisition, and the resulting CTDI value was 41.5.

2.2. Heart Segmentation

The starting point for crafting the 3D heart model is derived from Digital Imaging and Communications in Medicine (DICOM) images obtained from CT scans.

A subject with aortic stenosis undergoing cardiac CT was considered. The MicroDicom Viewer program was downloaded and utilized to visualize the CT images (in DICOM format). The selection of the slices relative to the heart facilitates their transfer to segmentation software. The professional software called Materialise Mimics was used to process the CT data. This platform is employed by researchers and users seeking to customize production processes to meet specific requirements. Generally, open-source software only identifies the external surface of an anatomic part. Instead, Materialise Mimics overcomes these technological limitations, allowing to segment and visualize internal cardiac structures. So, Materialise Mimics was chosen for this project because it allows to exhibit a realistic mesh of infrastructures and both the external and internal surface of the anatomic portion. To ascertain the suitability of this platform, the first step was to segment the heart and then dissect it with a plane to examine its interior presentation and verify the realistic mesh, distinct from the surface mesh.

Two segmentation approaches were employed: (i) region-based segmentation and (ii) edge-based segmentation. The edge-based approach facilitates segmentation by studying and identifying edges. It partitions an image based on abrupt changes in local grayscale levels, which must be sufficiently distinct from each other. Conversely, the region-based segmentation method relies on similarities between regions, excluding edges and considering regions within the anatomical region of interest.

The first procedure is the region-based segmentation. The objective is to identify the region, which could be an anatomical district within the organ or the organ itself. However, region-based segmentation alone is not sufficient to obtain a comprehensive 3D model, because of its tendency to be approximate in edge detection. Therefore, both segmentation methods were applied to ensure an accurate delineation. Herein are delineated the primary algorithms employed in the construction of the 3D model:

Threshold—This tool is employed at the onset of the segmentation process. It selects pixels within a specified intensity range contained within a chosen bounding box. The outcome is a preliminary mask that typically necessitates further refinement using additional tools.

Multiple Slice Edit—Operating on one of the principal CT views, this tool interpolates masks drawn on a limited number of slices. Users segment the most significant slices, resulting in a mask for each section within the selected interval. It can be paired with threshold definition to confine the interpolation to pixels within a specified intensity range. It is particularly effective for structures exhibiting a distinct and consistent section across principal planes, enabling the creation of a comprehensive segmentation mask with minimal errors from manually built slices.

Region Grow—Beginning with the selection of a seed pixel (or series of seeds), this tool isolates individual portions of a pre-existing mask through connectivity analysis. It is useful for refining a threshold-derived mask by eliminating isolated parts. It can be utilized to isolate major volumes of the left ventricle/left atrium, right ventricle/right atrium, and connected vascular system.

Level Tracing—defines a contour in which all pixels share the same background value as the selected pixel.

Hollow—replaces the selected segment with a uniformly thick shell.

Smoothing—Smooths the segments by filling holes and/or removing extrusions. Various smoothing methods (Joint, Median, Closing–Filling Holes) with a kernel size of 3 mm were applied, followed by an additional smoothing factor of 0.5.

Dynamic Region Grow—Similar to Region Grow, but it accepts multiple seed points across multiple layers, allowing for varied gray levels to control connectivity analysis. Threshold values on gray level similarity and mask expansion can also be applied. Particularly beneficial when standard Region Grow yields unsatisfactory results or when contrast agent effects vary significantly across the blood pool. Useful for generating valid starting points for chamber segmentation.

3D Interpolate—Performs interpolation akin to the Multiple Slice Edit, but utilizes seed masks from every principal anatomical plane. It is suitable for structures with complex shapes that cannot be described from a single perspective, such as left atrium/right atrium and left ventricle/right ventricle shapes. It is particularly advantageous for ventricle volumes due to their intricate shape.

Smart Fill—Analyzes an input mask and fills enclosed groups of inactive pixels, known as holes. The maximum hole size filled can be controlled by selecting the number of voxels.

Each step of the segmentation process was discussed with the radiologist and approved by the cardiac surgery team. The statistical data analysis has showed a standard deviation level for the methodology of segmentation applied about 7% lower than the gold standard, i.e., the manual segmentation performed by the radiologist.

2.3. Aortic Valve Segmentation

Materialise Mimics provides the capability to visualize intracardiac areas and consequently segment the aortic valve affected by stenosis. Also for the segmentation of the valve, the procedure is primarily manual, although in the literature, deep learning algorithms have been proposed [31,32].

In relation to the segmentation of the aortic valve, the initial step involves identifying the stenotic valve across the three planes. Subsequently, the multiplanar reconstruction (MPR) technique was employed to optimize the visualization and segmentation of the valve. The MPR technique is the most straightforward image processing technique and has been readily available for several years. This technique allows to rotate each plane differently from the standard axial, coronal, and sagittal sections. So, MPR enables the most effective visualization of anatomical structures that extend across planes different from the standard ones, aiding in accurate diagnostic analysis.

The final step involved utilizing the automatic threshold algorithm to identify the calcifications to be added to the 3D model. The density range of 389 ÷ 868 [hnsf’U] was set to extract the calcifications, considering that the range of adjacent tissues was 148 ÷ 195 [hnsf’U].

2.4. Implementation of the Application

Materialise Mimic allows to export the 3D model in an STL format. However, the conversion of the file from an STL to FBX format was necessary in order to integrate the model into the 3D environment engine and to enhance the object manageability. For this purpose, Blender 4.3, an open-source 3D creation software, was selected. Blender was also used to make other adjustments to the model. In particular, ensuring the optimal sizing, orientation, and alignment of the components with the desired model configuration is important. In fact, the 3D model of the heart obtained from the segmentation phase consists of nine 3D objects corresponding to the different functional parts of the heart. All the 3D objects have their reference systems in the same position, the center of the global heart mesh. Blender was indispensable for centering the local reference system of each 3D object in the geometric center of its mesh. This step is fundamental for the realization of the interactions (including translation, rotation, and scaling) in the 3D environment engine. In fact, once imported into Unity, the interactions with each object would take place with respect to the local system of the model. In order to be able to structurally reconstruct the global heart according to anatomy, the single 3D object was repositioned with the original coordinates in relation to the global heart system obtained from Blender. In this way, the conformation of the heart was preserved.

Subsequently, the finalized object in an FBX format, enriched with distinct color assignments for individual anatomical segments, was generated. The 3D model is then imported into the Unity platform (Unity version 2020.3.30f1, Visual Studio 2019), in which the MR experience is implemented for the HoloLens 2 headset (HL2). To implement interactions specific of MR, the Mixed Reality toolkit (MRTK) from Microsoft was integrated into the Unity project by configuring it to use the HL2 visor. In addition, the UnityVolumeRendering asset was integrated into the project to enable DICOM image display and 3D rendering. Meanwhile, the object with the “clipping plane” material provided by Unity was used to make a plane that could dissect the 3D object to allow the internal observation of the mesh.

When developing an app for HL2, the size limit of 500 MB for a 3D object that HL2 can process should be taken into consideration. Materialise can segment and then realize the mesh according to the level of details set by the operator. With a hundredth of a millimeter of detail of the order of magnitude, the limit set by the HL2 had been reached by the heart firstly segmented. In this context, the application cannot be built and the device should be used in the Holographic Remoting mode (i.e., connected to the computer via cable so as to utilize the computer’s calculation capacity). To match the limitation in a model file dimension supported by the HL2 and to allow the MR experience to be more fluent and stand-alone, the 3D model was smoothed. The details for the segmentation were set to a lower order of magnitude. Once the Unity project configuration is completed, all necessary parameters must be set to compile and deploy the application to the HL2.

A comprehensive workflow of the methodology is reported in Figure 1, which summarizes the main steps from the DICOM images to the MR experience realization.

3. Results

In Figure 2, the 3D model of the heart obtained through the segmentation is depicted, showing all the heart membranes.

Appling the MPR technique, the aortic valve was precisely identified and manually segmented (Figure 3a), while the calcifications were added to the 3D model of the aorta (Figure 3b).

The primary function of the MR application is to enable the exploration of a holographic 3D model of the heart reconstructed from CT images. This capability meets the surgeon’s requirement for a 360° observation of each anatomical site and provides a sense of depth that is otherwise unattainable. The user can navigate within the scene and interact with the elements in the scene effortlessly. A simple and intuitive user interface contains the following application features:

• Explosion of the model into its main components and interaction (sizing, rotating, and moving) with each one of them;
• Display of corresponding original CT images for the selected section from the planes;
• Selection of the anatomical components of interest through a user-friendly menu that includes user tracking (Figure 4a);
• Sectioning the heart components and observation of intracardiac structures by freely rotating and moving a semitransparent plane (Figure 4b);
• Reset option available to revert to the full view of the heart, restoring the scene to its original state.

4. Discussion

The final implementation of Heart Navigation represents a prototype of how cardiac surgery planning could be performed using MR. Feedback from physicians demonstrates how an MR system integrated into the HL2 can drastically reduce procedure execution times, thereby improving accuracy in selecting the appropriate surgical therapy. In fact, as stated in the review [33] about studies concerning congenital heart surgery, MR allowed a significant reduction in time for surgical planning and intraoperative preparation. The spatial representation and visualization of relevant anatomy offered by MR improved depth perception and speeded up intraoperative recognition of structures, resulting in the better processing of pathological structures and surgical phases. However, the segmentation and holographic visualization of intracardiac structures are still a challenge. In fact, the segmentation processes are still in development in order to be more accurate and faster. In this context, the attention has been moved to the deep learning algorithms [34,35,36] and to implement a specific dataset of DICOM images useful for the training phase [37].

In this particular case, the patient was 76 years old and was affected by aortic stenosis, which is a chronic disease that occurs over years, and due to its risk factors, it is strongly associated to the atherosclerotic disease.

The treatment option, at the severe stage of aortic valve disease, is mainly represented by the aortic valve replacement. In such patients, two options are available to perform the cardiac surgery: the standard surgical or the percutaneous route (TAVR or TAVI). While the standard valve replacement usually requires an open-heart procedure with a surgical access, extracorporeal circulation and, often, heart and great vessels’ manipulation. The TAVR or TAVI procedure can be completed through very small openings, or even percutaneously.

This study aims at going beyond the three-dimensional visualization of the cardiac region by accurately illustrating the intracardiac structures. Such a specific visualization is really crucial for preoperative planning in cardiac surgery. Inner organs are as significant as external ones, as they may provide additional or distinct information to experts. Given the patient’s condition, focusing on the aortic valve and on the ascending aorta was essential for diagnosis and determining the intervention type. Therefore, the aortic valve was manually segmented and integrated into the 3D model of the heart.

In fact, the preoperative planning process and deciding which treatment to adopt are based on an overall assessment of the patient’s risks (clinical, procedural, and anatomical risks). The key points are represented by, principally, age and therefore the fragility index of the patient, associated pathologies, degree of atherosclerotic pathology, and the possible involvement of the ascending aorta by the atherosclerotic process. The overall evaluation is based on the clinical visit and, above all, the integration of the information obtained from instrumental tests (chest X-ray, echocardiography, CT angiography). Computed axial tomography images are often analyzed and examined in both axial sections and multiplanar reconstructions.

In this specific case, the MR visualization has added some very important elements:

-

Better visualization of the aortic valve and the specific location of the annular calcifications and the valve leaflets;

-

An optimal spatial resolution, allowing the surgeon to better visualize and localize aortic calcifications, making surgical intervention and the use of extracorporeal circulation possible, minimizing the risk of peripheral embolism or acute aortic syndromes.

Since surgery is based on programming, in order to minimize possible unexpected events, the integration of the information obtained from our model made possible to translate and transform radiological images into a three-dimensional reality that can be cross-examined in every detail, corresponding to the real intraoperative features.

The accuracy in the heart visualization and 3D model realization is fundamental. Furthermore, the MR visualization requires high resolution to enable surgeons to examine even the slightest imperfections in anatomical features. In this context, HL2 presents some technical limitations regarding the resolution of the hologram it can generate and on the stand-alone functionality for a more complex MR experience. Indeed, using Holographic Remoting settings with a wired connection may be necessary for projects demanding high device performance.

The MR experience enables an interactive exploration of individual parts of the heart and sectioning in every direction. Additionally, the original DICOM images can be viewed simultaneously across all three-dimensional planes. Other functionalities can be added and integrated, such as measurements, marker positioning, and visualization from different perspectives. In particular, to further optimize the application, several future developments are planned, as follows:

-

Adding the capability to measure the aortic valve annulus point by point;

-

Incorporating blood flow dynamics into the hologram;

-

Creating a 3D model of the mitral valve using echocardiography;

-

Utilizing the ‘sharing experience’ feature, which allows multiple users, each with their own device, to view and interact collectively with the same hologram.

Although only one patient’s result is reported in this study, the potential and limitations of the proposed methodology were clear to the surgical team. The MR experience, completed with different activities and interactions, speeds up the recognition of structures and decision making. It provides additional information to traditional 2D images and it is also more manageable and interactive than the 3D printed model. At the same time, the segmentation process is currently flawed. Indeed, the procedure to achieve accurate segmentation of even intracardiac structures still requires manual intervention, although deep learning procedures are increasingly demonstrating high results.

5. Conclusions

While several studies investigate the utility and clinical value of 3D printing in aiding diagnosis, medical education, preoperative planning, and intraoperative guidance of surgical interventions, there is a scarcity of literature regarding concrete applications of mixed reality in the cardiovascular domain due to its nascent stage of study and expansion.

Cardiovascular medicine has a long history of using predictive modeling to assess patient risk. Recent studies have discovered methods to predict heart failure and other severe cardiac events in asymptomatic individuals. When combined with personalized prevention strategies facilitated by MR in precision medicine, these 3D models can positively impact disease incidence, consequence, and treatments.

This study goes beyond a mere three-dimensional visualization of the cardiac district, aiming to accurately visualize the intracardiac structures within the scope of preoperative planning for cardiac surgery.

MR has been expanding significantly, especially in the medical field, garnering considerable attention in experimental research [38,39,40]. The medical sector particularly requires lightweight, small, user-friendly, and cost-effective mobile devices. Key advancements in technology, including device miniaturization, the availability of open-source software, and the widespread use of smartphones among medical personnel, have played a crucial role in facilitating the adoption of MR. Certainly, the future inclusion of technologies like MR and deep learning in the clinical practice is subjected to the evaluation of ethical [41] and regulatory [42,43] aspects.

MR overlays real images with holographic images, often providing users with the perception of seeing through the patient’s body in medical settings. Understanding anatomical characteristics and generating realistic three-dimensional visualizations of congenital heart diseases remain challenging due to the complexity and wide spectrum of coronary diseases. Emerging technologies, such as 3D printing and MR, have the potential to overcome these limitations by utilizing 2D and 3D reconstructions of standard DICOM images. However, there has been limited research on the clinical value of these new technologies in coronary disease.