Medical Augmented-Reality Visualizer for Surgical Training and Education in Medicine

Abstract

This paper presents a projection-based augmented-reality system (MARVIS) that supports the visualization of internal structures on the surface of a liver phantom. MARVIS is endowed with three key features: tracking of spatial relationship between the phantom and the operator’s head in real time, monoscopic projection of internal liver structures onto the phantom surface for 3D perception without additional head-mounted devices, and phantom internal electronic circuit to assess the accuracy of a syringe guidance system. An initial validation was carried out by 25 medical students (12 males and 13 females; mean age, 23.12 years; SD, 1.27 years) and 3 male surgeons (mean age, 43.66 years; SD, 7.57 years). The validation results show that the ratio of failed syringe insertions was reduced from 50% to 30% by adopting the MARVIS projection. The proposed system suitably enhances a surgeon’s spatial perception of a phantom internal structure.

1. Introduction

Medical applications of virtual and augmented reality (AR) have been extensively developed during the last decade [1,2]. In the current era of data fusion, AR provides an ideal approach for educational purposes [3], especially those related to medicine [4,5,6], as it supports the training of surgeons without risk to patients. Virtual reality and AR techniques may also be employed to shorten the learning curves [7] related to surgical procedures and increase the overall educational efficiency, thereby providing increased patient safety.

Shortening the learning curves for surgical personnel is a societally important and widely researched issue. It aims to improve the accuracy and performance of surgical procedures in the most effective manner. Key factors related to learning of surgical procedures include proper real-time feedback and trial evaluation [8]. Studies have shown that the preferred feedback type is haptic (e.g., tool vibration) [9], suggesting that the success of surgical training is notably influenced by the degree of realism of the implemented haptic feedback [10]. In addition, surgeons may prefer additional haptic feedback, as it can provide valuable tactile information during surgery [9]. More generally, haptic feedback is believed to enhance perception by providing another mode of sensorial information. A review of simulation platforms available within endourology [11] presents a set of outcome measures for the validation of simulation and assessment tools. They include such parameters as:

• face validity (i.e., extent to which the examination resembles the situation in the real world),
• content validity (i.e., extent to which the intended content domain is being measured by the assessment exercise),
• construct validity (i.e., extent to which a test is able to differentiate between a good and bad performer),
• concurrent validity (i.e., extent to which the results of the test correlate with gold-standard tests known to measure the same domain),
• predictive validity (i.e., extent to which this assessment will predict future performance),
• acceptability (i.e., extent to which assessment procedure is accepted by subjects in assessment),
• educational impact (i.e., extent to which test results and feedback contribute to improve the learning strategy on behalf of the trainer and the trainee),
• cost effectiveness (i.e., technical and nontechnical requirements for implementation of test into clinical practice)

An initial validation of the system described in this paper with regard to the aforementioned parameters will be presented in further sections.

Currently available systems that support surgical procedures by means of augmented visualization utilize virtual reality headsets, goggles with translucent displays [12], or external computer displays [13]. For instance, head-mounted displays like Google Glass or Microsoft Hololens are being used in research on medical procedure training [14,15]. Although the headsets are often reported to be useful, they create a physical barrier between the surgeons and their environment, imposing an undesirable situation during medical procedures. Moreover, most translucent devices require additional markers for external tracking to achieve precise positioning of the virtual structures [16], especially in changing environments. Most of these devices are uncomfortable to wear and use for prolonged periods and often cause eye strain, headaches, and substantial discomfort [17,18,19]. Likewise, external screens can cause discomfort by the frequent switching of the surgeon’s focus between the patient and screen, thus prolonging operation time.

Another known disadvantage is that stereoscopic AR projection systems can exhibit the vergence–accommodation conflict. Conventional 3D displays present images on a surface, requiring both eyes to focus at a certain distance regardless of the intended distance to the virtual object. Hence, it is essential to properly set the depth of the display considering the depth of the depicted scene. Otherwise, eye strain, discomfort, and fatigue can result by excessive use of the lens muscles in the eyes to focus on the correct plane [20].

In this paper, we introduce an AR system called MARVIS (Medical AR VISualizer) that provides support for surgical procedures training by overlaying images of internal structures onto the surface of a phantom organ. The human liver was chosen as the test organ, for which biopsy and thermoablation were considered as exemplary procedures. This study aimed to develop a system that closely resembles the real conditions of a surgeon’s workspace. The key aspects of the MARVIS system are the following:

• Optical tracking of the user’s head while practicing on the organ phantom. Information related to the spatial relationship between the phantom and operator’s head is calculated in real time.
• Projection of internal structures onto the surface of the liver phantom in conjunction with naked-eye observation. This type of observation along with monoscopic motion parallax provides 3D perception through simple motions of either the operator’s head or the phantom. No additional goggles or headsets are required.
• A dedicated phantom composed of segments with different electric conductivities and an electronic module. This subsystem provides objective information about the procedure results in terms of the correct targeting rate of the internal structures.

Various AR systems for medical applications have been proposed. In Ref. [21], a system that displays anatomical objects in three dimensions combined with a 2D slicing plane on an external stereoscopic display is introduced. This system allows trainees to feel a virtual patient using their hands through palpation. In addition, projection-based AR systems for medicine have also been proposed. In Ref. [22], an interactive computer-generated intelligent virtual agent is introduced, where visual information is projected onto the physical shell of a head. Another system trains ultrasound-guided needle placement augmented with computer-generated ultrasound imagery, which emulates the physical process of ultrasound scanning on patients [23]. A system presented in Ref. [24] simulates the fermoal palpation procedure in a virtual environment and allows to train needle insertion into the vessel.

However, the systems in Refs. [21,24] show the generated auxiliary data on external monitors, and the operator must constantly switch the viewpoint, possibly leading to problems in long-term perception. The technique described in Ref. [21] needs a wired stylus to manipulate the virtual object. Furthermore, the system in Ref. [24] requires dedicated hardware to deliver tactile feedback. The systems in Refs. [22,23] do not consider the operator’s viewpoint, possibly causing manipulation problems during the virtual medical procedure due to different coordinate systems. Moreover, the approach presented in Ref. [23] renders all information on screen as monochromatic visualization only.

The approach we propose in this paper has the abovementioned key aspects and comprises a novel projection-based AR system. The approach is implemented on the MARVIS dedicated system, which allows to visualize previously segmented computer tomography (CT) medical data of internal structures on the liver phantom.

2. Materials and Methods

2.1. Overview

The MARVIS system consists of four modules, as illustrated in Figure 1. The phantom-tracking module (PhTM) detects the phantom in the real-time video stream and determines its 3D position and orientation using data stored in the model database (MDB). The position of the surgeon’s head is estimated using a marker-based optical head-tracking module (HdTM). Images of internal structures projected by the projection module (PrM) are transformed (in terms of perspective and deformation) considering the relative positions of the phantom and operator’s head. The rendered images are visible on the surface of the phantom, which is connected using an electronic structure interaction module (SIM) to detect the physical (electrical) contact of the surgeon’s tool with the internal structures of the phantom. In the next subsection, we detail each module and its role in data processing.

Tracking the phantom relies on its contours acquired through a tracking camera. Contour tracking has lower computational complexity than comparison based on 3D models, which usually requires components such as smart-pixel cameras [25], time-of-flight cameras [26], or at least two standard cameras to get the spatial coordinates of each point [27]. A set of contours is created for the digital liver model (using edge detection on a binary silhouette of a digital model, as shown in the left column of Figure 2) for a range of rotations (±60° with respect to the X axis, ±45° with respect to the Y axis). A resolution of 2° per axis is related to the degrees of freedom of a real liver. For each angle (a combination of rotations around two axes on the horizontal plane, perpendicular to the tracking-camera axis), a contour-based shape descriptor (CSD) is calculated and stored in the MDB. A CSD analysis for the physical phantom is performed by the user during training and allows estimation of its orientation in real time.

2.2. Contour-Based Shape Descriptor

The position and orientation of the liver phantom are calculated from its contour. The initial contour is a vector comprising 2D points represented in a Cartesian coordinate system (XY plane). The contour may be obtained by either using the computer-generated image of a projection of the 3D model onto a 2D plane of the virtual camera or analyzing video footage. The proposed CSD is invariant to object scale and rotation about the camera axis and is represented using polar coordinates with the pole located at the geometric center of mass of the 2D object silhouette (Figure 2). The polar axis is defined by the contour start point, which is selected using two points (P_CP1, P_CP2) at the intersection between the contour and the best-fitting line (L_BFL) of all the contour points. Then, the contour start point is selected as the farthest point from the pole (Figure 3). The scale-invariant feature is achieved by normalizing the distances in terms of the distance from the pole to the contour start point. The rotation-invariant feature is achieved by using line L_BFL and the contour start point. The CSD is obtained by uniform angular sampling (every 6°) of the contour and storing the parameter pair (α, d) per point, where α is the polar angle and d is the normalized radius of the contour point.

2.3. Model Database

The MDB in this study stored a 3D surface model of the liver along with a set of possible rotations and corresponding CSDs. Each surface model is built once in the form of a triangle mesh per physical phantom, using real imaging data. For each orientation (θ,Φ) defined by rotations about the (X,Y) axes of the model (Figure 4), the contour is determined and a corresponding CSD is calculated. Input silhouette images used for CSD calculation are created by projecting the 3D triangle mesh onto a 2D plane, followed by contour extraction. The sampling of angles (θ,Φ) affects the final angular resolution of phantom tracking, and a balance should be maintained between the image analysis requirements regarding accuracy and statistical errors.

2.4. Phantom-Tracking Module

The PhTM locates the liver phantom in the field of view of the tracking camera (Figure 5), which is a pinhole-calibrated [28] color camera, and calculates its position and orientation. During system operation, the current phantom position is calculated by determining the geometric center of mass of the contour and its transformation into a global coordinate system. Phantom rotation (θ,Φ) is calculated by correlating the CSD for the current frame with precomputed CSDs from the MDB (Figure 6). Similarity is determined using the minimum of contour difference S_AB, calculated for contours A and B as follows:

$$S_{AB}={{\displaystyle \sum }}^{}\frac{{\left(d_{Ai}-d_{Bi}\right)}^2}{\sqrt{N}}$$

(1)

where d_Ai and d_Bi are the distances to point i of contours A and B from the pole, respectively, and N is the number of points per contour, which is sampled according to the required angular accuracy. Hence, similarity S_AB measures the difference between two CSDs and is calculated between the current CSD of the phantom image and CSDs for every orientation, as stored in the MDB, providing an estimate for the current angles (θ,Φ) of the phantom.

Note that the third rotation angle (about the axis perpendicular to the work table) is calculated directly from the image captured by the phantom-tracking camera, because the current CSD is invariant to Z-axis rotation. This reduces the MDB size and improves the speed of the database search. The PhTM implements the abovementioned key aspect 1.

2.5. Head-Tracking Module

The viewing perspective of the user is determined from the calculated position of a single marker located on the forehead between the user’s eyebrows, referred to as the mid-eyes point. We use the marker for head position tracking to avoid problems when the system is simultaneously used by some individuals. The marker clearly identifies the viewpoint to be rendered by MARVIS. The proposed solution utilizes monoscopic motion parallax, allowing the viewer to determine spatial relationships between the internal structures of the phantom. This approach is used assuming that the head pitch with respect to the phantom remains stable and that the mid-eyes point is located 45 mm below the marker. These assumptions increase the accuracy of the viewing angle. The HdTM locates the user’s mid-eyes point, determining the marker position within the 3D coordinate system. If the HdTM loses the marker, the system uses the last known position. This approach is supported by a pinhole stereo-calibrated setup [28] consisting of two color cameras (HdTC₁ and HdTC₂ in Figure 5). The HdTM is also related to key aspect 1.

2.6. Projection Module

Key aspect 2 is implemented using the PrM. All the visual information rendered by a graphics card using the segmented medical data and recalculated using MARVIS is projected onto the destination phantom surface by the PrM projector, which is calibrated as the inverse pinhole camera [29]. More details on the projection system are provided in Section 2.9.

2.7. Structure Interaction Module

The phantom in the proposed system consists of molding silicone, which imitates the tactile texture of an organ’s surface. In fact, similar types of materials have been used as sensor covering to imitate human fingers [30]. The internal structures within the phantom are made of an electrically conductive material. During phantom design, we used CT scan data from a real liver. Then, we obtained a custom mold of the liver external shape and hepatic portal vein structure by image segmentation. In addition, a mold of tumorous tissue with an arbitrary shape was fabricated. The custom molds of internal structures were made of a material that absorbs X-rays and is electrically conductive. The internal structures were wired with cables and then immersed in silicone, which resembled the liver tissue. Finally, the manufactured liver phantom was CT scanned to obtain its detailed 3D model including the fabricated internal structures.

As the internal structures of the phantom are detectable by CT imaging, its digital 3D model accurately corresponds to the physical phantom. The electrical connectors of the internal structures located at the bottom of the phantom connect each structure to the SIM (Figure 7). The SIM algorithms are implemented in a hardware module based on a Microchip PIC12F675 microcontroller. The hardware module is connected to both the internal structures and a syringe probe needle. The SIM network interface allows to start/reset a stopwatch, whereas the needle that closes the circuit with either of the internal structures stops the stopwatch and adds a new entry to the database (the structure is impinged and the time is measured). The SIM implements part of key aspect 3.

2.8. Global Calibration

Initially, each device in the modules uses its local coordinate system (Figure 8), and hence the output data is expressed into a common global coordinate system with the origin on the working table (X_w, Y_w, and Z_w axes in Figure 8). These modules must be set up and calibrated before operation. First, the PhTM must be calibrated to estimate the internal and external parameters of the projector in the PrM and the camera in the PhTM, using the pinhole method proposed in Ref. [29]. The same algorithm and calibration pattern are used to calibrate the pair of stereo cameras (HdTC₁ and HdTC₂) in the HdTM. All calculations are then realized in the same global coordinate system.

2.9. Prototype Fabrication

We implemented a prototype of the MARVIS system to assess and validate its concept. The system frame was made from aluminum profiles of 45 × 45 mm. The PhTM consists of a 2 MP color camera (Flea3 FL3-U3-20E4C; Point Grey Research Inc, Wilsonville, OR, USA) equipped with a narrow-angle lens (f = 25 mm; Pentax Corp., Tokyo, Japan; green area in Figure 9). The camera optical axis is nearly parallel to the projector optical axis, with a working distance of 1450 mm. The HdTM consists of two 2 MP color cameras (Flea3 FL3-U3-20E4C; Point Grey Research Inc, Wilsonville, OR, USA; red areas in Figure 9) with wide-angle lenses (f = 8 mm; Pentax Corp., Tokyo, Japan;), pointing at the user’s head. The distance between the HdTM cameras is approximately 1200 mm. The liver phantom illumination and AR displaying are obtained from a full HD projector (EH415; Optoma Corp., New Taipei, Taiwan; PrM, blue area in Figure 9). The projection distance is approximately 1300 mm, and the optical axis is perpendicular to the working table. The SIM consists of a PIC microcontroller with an additional hardware module based on the Banana PI board (Sinovoip Co., Ltd, Shenzhen, China) and IBM Node-RED software for capturing data from the user’s syringe probe (Figure 10). The phantom was cast from silicone molding rubber [31], and the internal structures from agar [32]. The molding casts were designed using 3ds Max 2017 and printed from ABS using a µPrint 3D printer (Stratasys Ltd., Valencia, CA, USA). The print quality was set to the highest value, layer height 0.254 mm. The liver phantom view is augmented with the internal structures as shown in Figure 11. The area in red represents a tumor, and that in blue represents blood vessels.

The developed software was implemented in C++ using Visual Studio 2015 (Microsoft Corp., Redmond, WA, USA) and the OpenGL API. It runs on a 64-bit Windows 10 Pro computer with an Core i7 6700K (4.0 GHz; Intel Corp., Santa Clara, CA, USA) processor and GeForce 980 graphics card (NVIDIA Corp., Santa Clara, CA, USA). The CSDs of the liver model were sampled with resolution of 5° about the Z-axis (60 points for the contour). Prerendered views were generated with resolution of 2° about both the X and Y axes. These resolutions consider the tradeoff between computational complexity and phantom simulation accuracy. The ranges of rotation per axis were chosen after consultation with medical experts, within ranges −60° < Φ < 60° and 45°< θ < 45° relative to a default anatomic phantom position on the working table. These values consider the surgical experience from two of the coauthors and the limitations of liver anatomical movements in the human body. These ranges, which were implemented in MARVIS, are broader than the usual angles found in surgeon’s practice.

MARVIS generates an OpenGL scene of a liver model, including its internal structures, at each timeframe and projects its transformed image onto the physical phantom. First, a scene is rendered with respect to the user’s viewpoint (Figure 12, step 1), beginning with the phantom shell without internal structures. Next, the system renders the internal structures in a separate framebuffer for superimposition onto the framebuffer with the shell image, disregarding depth information from the Z-axis buffer (Figure 12, projection of A onto A_p). The system then virtually re-projects the pixels (Figure 12, step 2) from the user’s viewpoint onto the liver phantom surface based on the global coordinate system and projects these pixels onto the projector image plane (Figure 12, step 3). Therefore, the PrM accesses the information required to display an image with suitable perception characteristics for the operator (Figure 12, step 4).

The elements of the required transformation matrices are partially calculated during calibration, such as matrices containing the intrinsic and extrinsic parameters for all the cameras and projector. Other elements, such as matrices containing information of liver phantom orientation and user’s head position, are updated frame-by-frame in real time.

3. Results

We performed various evaluations to validate the performance of the MARVIS system. The analyzed functionalities include angular accuracy of phantom tracking, quality of mutual calibrations at particular modules, and accuracy of head tracking. Finally, a test of the whole system was conducted.

3.1. Angular Accuracy of Phantom Tracking

To assess the accuracy of angular position estimation for the phantom, we employed a dedicated hardware configuration. Specifically, the phantom was placed on a stack comprising a tilting table and a rotary table. The phantom was positioned flat and attached to the table surface. The starting orientation of the phantom was estimated based on the flat arrangement of its bottom part, such that it was tangent at two points to a vertical line projected by the PrM.

Each axis of rotation was independently validated in

Table 1. Detection accuracy of phantom rotation angles. (a) Difference ∆|φ| between reference rotation φ_ref and angle φ_meas about the X axis detected by the system for the phantom placed on the table (Unit: degrees, °). (b) Difference ∆|θ| between reference rotation θ_ref and angle θ_meas about the Y axis detected by the system for the phantom placed on the table (Unit: degrees, °). (c) Difference ∆|ψ| between reference rotation ψ_ref and angle ψ_meas about the Z-axis detected by the system for the phantom placed on the table. (Unit: degrees, °).

(a)			(b)			(c)
φref	φmeas	∆|φ|	θref	θmeas	∆|θ|	ψref	ψmeas	∆|ψ|
30.0	28.0	2.0	30.0	31.0	10	0.0	0.01	0.01
25.0	24.0	1.0	25.0	23.0	2.0	22.5	22.89	0.39
20.0	18.0	2.0	20.0	17.0	3.0	45.0	45.70	0.70
15.0	16.0	1.0	15.0	11.0	4.0	67.5	68.60	1.10
10.0	10.0	0.0	10.0	9.0	1.0	90.0	89.47	0.53
5.0	6.0	1.0	5.0	5.0	0.0	112.5	111.35	0.15
0.0	0.0	0.0	0.0	–1.0	1.0	135.0	135.42	0.42
−5.0	–4.0	1.0	–5.0	–7.0	2.0	157.5	156.87	0.62
–10.0	–10.0	0.0	–10.0	–13.0	3.0	180.0	179.27	0.73
–15.0	–12.0	3.0	–15.0	–17.0	2.0
–20.0	–18.0	2.0	–20.0	–21.0	1.0
–25.0	–22.0	3.0	–25.0	–25.0	0.0
–30.0	–28.0	2.0	–30.0	–33.0	3.0

a–c. Angular tracking about the X and Y axes was conducted with a 2° resolution, corresponding to the angular spatial resolution of the prerendered positions in the MDB. The average error was 1.38° (maximum 3.0°) about the X axis, and 1.92° (maximum 4.0°) about the Y axis. We believe that the accuracy of the silicon mold is the most important factor that influences the matching procedure, because the crucial contour detection operation is applied to local changes in shape (Figure 13). The detection of rotation resolution about the Z-axis is limited by the resolution of the PhTM camera.

3.2. Mutual Calibration of Phantom-Tracking and Projection Modules

The mutual calibration of the PhTM and PrM was assessed by estimating the marker reprojection error which is a common method for systems consisting of image detectors and projectors. Four markers from a calibration board were selected and measured at seven distances from the working table at a range of 55.5–112.5 mm (Figure 14,

Table 2. Object-tracking reprojection error at specific height (along Z-axis) above the table.

	Marker 1				Marker 2				Marker 3				Marker 4
	On Camera		On Projector		On Camera		On Projector		On Camera		On Projector		On Camera		On Projector
Z	X	Y	X	Y	X	Y	X	Y	X	Y	X	Y	X	Y	X	Y
[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]
55.5	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
65.5	−0.02	0.01	−0.04	0.02	0.011	0.00	0.03	0.02	−0.02	0.01	0.07	0.00	0.014	−0.00	0.02	0.00
75.5	−0.01	0.02	0.00	0.03	−0.01	−0.02	−0.02	0.02	0.01	0.02	0.17	0.00	0.00	−0.01	0.00	0.00
85.5	−0.01	0.04	0.00	−0.08	0.01	0.01	0.11	0.03	−0.01	−0.01	0.05	−0.03	−0.02	−0.02	−0.02	0.00
95.5	−0.02	0.05	−0.01	0.03	−0.02	0.01	0.03	0.04	0.01	0.01	0.12	−0.01	−0.01	−0.02	−0.01	−0.05
105.5	−0.02	0.06	−0.01	−0.01	−0.04	0.01	−0.04	0.05	0.02	−0.00	0.17	−0.02	0.02	−0.04	0.02	−0.18
112.8	0.01	0.05	0.00	−0.05	−0.01	0.00	−0.02	0.03	0.00	0.00	0.11	−0.01	0.00	−0.02	−0.02	−0.16

), which corresponds to the phantom surface position. The average reprojection error within the calibrated measurement volume was 0.01 mm (maximum 0.17 mm). The average size of a projected pixel was 0.20 mm, and the average size of the pixels captured by the PhTM camera was 0.25 mm. The results are satisfactory when compared to commercially-available tracking systems which achieve accuracy of 0.61 ± 0.55 cm (Oculus Rift) and 0.58 ± 0.45 cm (HTC Vive) [33]. However, it must be noted that MARVIS uses a much smaller measurement volume.

3.3. Head-Tracking Accuracy

To validate the head-tracking accuracy of MARVIS, several points (Figure 15) near the edges of the calibration volume were selected for calculating the reprojection error. For the image plane in the middle of the calibrated volume, one pixel of the HdTM camera image corresponds to 2.5 mm. The obtained results listed in

Table 3. Head-tracking reprojection error in calibration volume.

X0	Y0	Z0	X1	Y1	Z1	∆X	∆Y	∆Z
[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]	[mm]
−504.35	338.53	1367.02	−505.87	338.90	1370.98	1.52	0.37	3.96
−517.29	−348.87	1350.01	−517.96	−348.57	1351.60	0.67	0.31	1.59
329.76	112.35	964.45	330.37	111.57	966.91	0.61	0.78	2.46
339.01	−135.37	968.48	339.60	−135.44	970.28	0.59	0.08	1.80
46.13	174.12	588.10	46.24	174.62	592.71	0.11	0.50	4.61
28.39	−128.10	578.11	28.41	−128.06	582.33	0.01	0.04	4.22
204.44	137.02	632.78	205.57	137.01	637.43	1.14	0.01	4.65
206.63	−148.44	629.07	207.64	−148.42	633.28	1.01	0.02	4.21
236.37	19.10	682.92	237.92	18.94	687.45	1.55	0.15	4.53
45.74	21.10	877.70	45.91	20.99	881.10	0.17	0.11	3.39
−118.27	63.50	1069.90	−118.63	63.38	1072.95	0.35	0.12	3.05
−270.32	69.22	1234.31	−270.96	69.11	1237.19	0.64	0.10	2.88
−186.02	−257.16	1080.75	−186.49	−257.02	1082.93	0.46	0.14	2.18
−195.27	228.51	1040.83	−196.05	228.67	1044.49	0.78	0.17	3.66
Average error						0.69	0.21	3.37
Median error						0.63	0.13	3.53

Each row corresponds to a 3D point in the global coordinate system. (X₀,Y₀,Z₀) describes the position of the first calculation of the 3D reprojection of 2D pixels onto the global coordinate system. (X₁,Y₁,Z₁) describes the result of projecting (X₀,Y₀,Z₀) onto the camera pixels and their reprojection to the 3D coordinate system.

confirm an angular accuracy below 0.1 degree [34], which is sufficient to track the user’s head viewing direction. This is used for determining the orientation of the user’s viewpoint in the global coordinate system and then to enable parallax motion. The accuracy of this measurement is crucial to prevent the vergence–accommodation conflict.

3.4. System Validation

An initial validation of MARVIS was conducted as a trial involving pricking internal structures of the liver phantom (Figure 16) using either phantom image data displayed in a traditional way on an external LCD monitor (control experiment) or structures being projected (Figure 12) directly onto the phantom’s surface (test experiment).

The total of 25 students (12 males and 13 females, mean age, 23.12 years; SD, 1.27 years) from the faculties of medicine at the Medical University of Warsaw took part in the validation. Additionally, three experienced male surgeons performed the same test (mean age, 43.66 years; SD, 7.57 years). Given the small number of participating surgeons, we analyzed the data from all the participants together. For each participant, the experiment was carried out as follows:

• Control experiment: Author R.G. selected an internal structure from the 3D model displayed on the external monitor. The 3D model was generated using CT data of the first phantom (two phantoms with different internal structure shapes were used for each participant of the experiment in order to avoid the influence of the user remembering the particular case). The participant was asked to prick the structure in the first liver phantom with a syringe probe by looking at the 3D model. The 3D model could be rotated accordingly using a computer keyboard. The time to complete the task was measured manually by the test supervisor until the participant informed task accomplishment.
• Test experiment: Author R.G. selected an internal structure projected by the MARVIS system onto the second phantom. The participant was asked to prick the structure with the syringe probe. Holding the phantom in his hand in a way that would not interfere with phantom contour detection, the participant inspected the internal structures as they were projected onto the phantom’s surface. Phantom position and participant’s head were tracked with PhTM and HdTM, respectively.
• When the participant decided to prick the structure, he had to put the phantom on the desk (for stability) and pause PhTM tracking by depressing a foot pedal. Otherwise, his hands or the syringe occluding the phantom contour would cause PhTM to provide false data about phantom position. Only then pricking could be performed. HdTM and PrM were working continuously so the projection parameters of the internal structures depended on the current position of the participant’s head. The duration time of the pricking procedure was measured as in the control experiment.
• The test experiment was performed with the first phantom.
• The control experiment was performed with the second phantom.

The tests showed that the failure rate (i.e., participants not being able to hit the desired structure) dropped from 50% without projection to 30% with projection (

Table 4. MARVIS validation results.

	Without Projection						With Projection						Comparison
Participant	Number of Correct Attempts	Number of Incorrect Attempts	Average Time to Correctly Prick Needle	SD	Median Time to Correctly Prick Needle	Failure Rate	Number of Correct Attempts	Number of Incorrect Attempts	Average Time to Correctly Prick Needle	SD	Median time to Correctly Prick Needle	Failure Rate	Median Time Difference	Failure Rate Difference
-	-	-	[s]	[s]	[s]	[%]	-	-	[s]	[s]	[s]	[%]	[s]	[%]
1	9	1	26.25	49.05	4.26	10.0	6	4	11.97	1.59	11.97	40.0	7.7	30.0
2	8	2	14.19	8.15	14.47	20.0	8	2	16.58	13.38	10.88	20.0	−3.6	0.0
3	9	1	14.04	9.18	15.42	10.0	8	2	11.29	10.99	10.35	20.0	−5.1	10.0
4	8	2	8.11	2.63	8.67	20.0	7	3	10.42	3.71	11.67	30.0	3.0	10.0
5	8	2	27.75	16.24	23.51	20.0	5	5	4.05	1.99	3.74	50.0	−19.8	30.0
6	5	5	18.21	11.71	12.26	50.0	7	3	15.97	13.8	10.73	30.0	−1.5	−20.0
7	4	6	26.95	10.92	33.21	60.0	7	3	10.89	10.4	6.55	30.0	−26.7	−30.0
8	5	5	12.48	3.1	11.45	50.0	8	2	12.04	2.61	12.55	20.0	1.1	−30.0
9	3	7	18.77	3.39	20.84	70.0	2	8	13.16	2.67	11.98	80.0	−8.9	10.0
10	5	5	19.67	16.56	13.89	50.0	4	6	9.39	3.97	10.44	60.0	−3.5	10.0
11	6	4	17.56	6.37	17.75	40.0	7	3	31.88	15.85	31.6	30.0	13.9	−10.0
12	4	6	22.14	9.41	21.8	60.0	5	5	11.58	10.33	6.27	50.0	−15.5	−10.0
13	4	6	20.66	14.71	11.46	60.0	6	4	12.92	5.37	10.83	40.0	−0.6	−20.0
14	2	8	21.74	5.32	21.74	80.0	7	3	11.17	4.19	9.75	30.0	−12.0	−50.0
15	6	4	35.19	24.93	28.34	40.0	6	4	14.9	9.26	13.91	40.0	−14.4	0.0
16	3	7	38.22	41.95	18.85	70.0	5	5	11.01	8.38	11.95	50.0	−6.9	−20.0
17	5	5	28.22	16.59	23.02	50.0	6	4	21.33	8.17	19.32	40.0	−3.7	−10.0
18	4	6	16.31	8.75	17.45	60.0	2	8	21.6	11.45	18.35	80.0	0.9	20.0
19	4	6	28.1	8.02	27.12	60.0	7	3	8.59	6.72	7.12	30.0	−20.0	−30.0
20	5	5	27.42	10.12	25.8	50.0	7	3	12.7	8.58	12.68	30.0	−13.1	−20.0
21	3	7	15.06	8.99	10.96	70.0	6	4	14.37	4.01	13.93	40.0	3.0	−30.0
22	1	9	5.27	0.0	5.43	90.0	8	2	6.21	5.3	3.04	20.0	−2.4	−70.0
23	3	7	23.72	9.66	16.96	70.0	7	3	13.62	13.34	11.24	30.0	−5.7	−40.0
24	6	4	21.25	7.94	20.27	40.0	5	5	8.32	6.57	7.01	50.0	−13.3	10.0
25	8	2	25.92	25.77	14.61	20.0	10	0	15.18	6.21	15	0.0	0.4	−20.0
26	5	5	62.35	35.96	41.28	50.0	7	3	21.9	13.5	14.16	30.0	−27.1	−20.0
27	6	4	24.13	7.14	23.1	40.0	8	2	12.46	6.26	13.49	20.0	−9.6	−20.0
28	8	2	11.93	8.05	8.26	20.0	10	0	24.77	34.45	11.32	0.0	3.1	−20.0
Avarage			22.56	13.59	18.29	47.50			13.94	8.68	11.32	35.36
Median			21.50	9.30	17.60	50.0			12.58	7.45	11.32	30.0

, failure rate columns, and Figure 17 and Figure 18). Moreover, the median time to perform one correct prick dropped from 17.6 s to 11.5 s (Table 4, median time columns). Most of the participants consistently claimed that projection simplified and reduced the time to complete the test procedure.

3.5. System Performance

The framerates of the PhTM camera and HdTM cameras were 60 fps. The average image projection framerate of the PrM was 25 fps regardless of the position/orientation of the phantom. The lag between moving the phantom and the change in the projected image was approximately 0.5 s (as measured using an external 120 fps camera). Each prick took 21.5 ± 9.3 s in the worst case, and a needle probe remained sufficiently sharp for efficient pricking over approximately 150 trials. Every custom-made liver phantom was pricked approximately 50 times by students and it still keeps its mechanical material properties. However, conductivity drops after approximately 1 month, when agar dries.

4. Discussion

The proposed system fulfills our design requirements and provides 3D perception of internal structures when either the operator’s head or the phantom moves. The simulator provides content validity by clearly determining whether the training participant was able to prick the proper internal structure during the test. The duration of the pricking procedure is measured, and faulty hits are recorded to differentiate between a good and bad performer (construct validity). Although no dedicated surveys were conducted, the user acceptability during the tests was very high. The educational impact was positive as shown by the failure rate distribution and median procedure time histograms. The cost of a single pricking procedure is mainly connected with the wear of the phantom and is approximately USD 0.50 (the cost of the system hardware is not considered here).

Although the design requirements have been met, certain aspects of the presented solution can be further improved. In the current setup, the user must pay attention not to occlude the phantom edge from the PhTM camera. Moreover, the silicone mold may be deformed if mishandled and provide a faulty CSD. Furthermore, head tracking with a single marker makes the system sensitive to yaw/pitch/roll rotations of the head, and considerable rotations can lead to inaccuracy and distortion in the projected image.

For a focused image to be formed on the retina, the eyes accommodate at a proper distance and adjust the required vergence. The allowable tolerance of the focusing distance for the observer to perceive of a clear image is ±0.3D (diopter) [35], and the acceptable vergence error is at the level of 15–30′ (arc minutes) [36]. Larger values impede the generation of stereoscopic vision [37]. Although small discrepancies do not cause the complete disappearance of binocular fusion, they notably undermine depth perception [38]. Therefore, the long-term use of conventional 3D displays causes fatigue and discomfort to the viewer [39,40,41]. Those symptoms may be caused by the lack of synchronization between vergence and accommodation [20]. In contrast, monoscopic visualization supported by motion parallax avoids these problems.

After a few days of tests, we noticed declining performance of the participants, and one of them complained about a blunt needle. Then, we verified that this was the reason for the observed performance decay. In fact, after replacing the needle, the effectiveness returned to the expected levels. Furthermore, some students had difficulties interpreting the CT data displayed on the external LCD during the control experiment, possibly because of their inexperience. In addition, frozen pose detection was problematic, as some of students found it difficult to coordinate the pedal usage. On the other hand, some participants pointed out the low resolution of the projected data as the main drawback of the MARVIS system. Nevertheless, most participants claimed that using MARVIS led to easier and faster task completion by the projection-supported 3D perception and understanding of the spatial relationships between internal structures. They also agreed on their interest to train with the MARVIS system on more complex cases.

The initial validation demonstrates the applicability of the MARVIS system for surgical training and education in medicine. The corresponding results show that methods employing natural phantom manipulation (i.e., using the operator’s hands) along with visualization unobstructed by external devices and enhanced by AR are promising for the development of surgical research and applications.

5. Conclusions

We propose a low-cost glasses-free medical AR visualizer called MARVIS for use in surgical training and education in medicine. The proposed approach allows the projection of internal structures on the surface of a silicone liver phantom considering the user’s viewpoint. The system setup prevents problems such as eye strain, discomfort, and fatigue, which are generally present when using AR headsets or glasses. Moreover, marker-less shape matching simplifies the phantom preparation. The overall accuracy of the MARVIS system from the viewpoint of natural human perception is highly satisfactory. Regarding the employed hardware, MARVIS accuracy is in the range of a single pixel of the projected image.

MARVIS may be used to support the education in medicine and improve the training of surgeons without risk to patients. Using MARVIS, we demonstrated that an AR system can influence the performance to accomplish surgical procedures. Using a real phantom provides a natural tactile experience for future surgeons and can simplify the process of surgical operation planning.

In future work, we will focus on improving visual data representation and decreasing the system response time to improve the user experience. We also intend to render more complex texture information related to internal structures using computer-generated 3D lighting. Further development of 3D mapping algorithms is also planned.