Validation of a White Light and Fluorescence Augmented Panoramic Endoscopic Imaging System on a Bimodal Bladder Wall Experimental Model

Abstract

Background: Fluorescence visualization of pathologies, primarily neoplasms in human internal cavities, is one of the most popular forms of diagnostics during endoscopic examination in medical practice. Currently, visualization can be performed in the augmented reality mode, which allows to observe areas of increased fluorescence directly on top of a usual color image. Another no less informative form of endoscopic visualization in the future can be mapping (creating a mosaic) of the acquired image sequence into a single map covering the area under study. The originality of the present contribution lies in the development of a new 3D bimodal experimental bladder model and its validation as an appropriate phantom for testing the combination of bimodal cystoscopy and image mosaicking. Methods: An original 3D real bladder-based phantom (physical model) including cancer-like fluorescent foci was developed and used to validate the combination of (i) a simultaneous white light and fluorescence cystoscopy imager with augmented reality mode and (ii) an image mosaicking algorithm superimposing both information. Results: Simultaneous registration and real-time visualization of a color image as a reference and a black-and-white fluorescence image with an overlay of the two images was made possible. The panoramic image build allowed to precisely visualize the relative location of the five fluorescent foci along the trajectory of the endoscope tip. Conclusions: The method has broad prospects and opportunities for further developments in bimodal endoscopy instrumentation and automatic image mosaicking.

1. Introduction

Currently, cystoscopy is the mainstay of bladder cancer detection and usually works together with cytologic analysis, biomarkers, and imaging [1,2]. Allowing visual inspection of the bladder wall and main cancer lesions, white light cystoscopy (WLC) is the most commonly used in vivo diagnostic imaging modality [3,4] and is still considered to be the “gold standard” for the detection and localization of bladder cancers [2,5]. As a clinical examination, WLC provides direct visual access to the inner wall of the bladder tissue and allows to obtain highly informative images for real-time diagnosis, surgical guidance, and follow-up.

However, WLC has inadequate sensitivity (lower than 70%) for detecting small papillary tumors, carcinoma in situ (CIS), or dysplasia. This can result in early recurrence rates up to nearly 40% one year after WLC-guided transurethral resection of the bladder [6]. Moreover, WLC can be inaccurate and is unlikely to distinguish inflammation from malignancy [3].

Fluorescence cystoscopy (FC) can improve the detection efficiency of bladder cancer and, in the case of non-muscle invasive bladder cancer, decrease its recurrence rate and its progression towards muscle invasive bladder cancer.

The basic principle of FC is based on the detection of light emitted by fluorescing molecules of special agents introduced into the body. After being excited by the appropriate wavelength light, these molecules relax to the ground state and emit photons with less energy than the excitation ones. This approach allows to separate signals at different wavelengths and to distinguish between normal and cancerous tissues, because the latter preferentially accumulate the photosensitive agents.

The advancement in the development of photosensitizers (PSs) [7] allows to move forward the progress in fluorescence-guided endoscopy. Commonly used substances for the selective emission of fluorescence are Chlorin e6 (Ce6) and Protoporphyrin-IX (PpIX)-based (5-aminolevulinic acid (5-ALA)) substances, methylene blue (MB), and hexaminolevulinate (HAL). Light absorption occurs mainly due to endogenous chromophores in tissues, which reduce the efficiency of the photodynamic process by competing with PS in the absorption process. For Ce6 and PpIX photosensitizers, two excitation regions are typical: about 400 nm and 620–660 nm [8,9,10,11,12,13,14,15,16]. Methylene blue does not have absorption peaks in the region of 400 nm; therefore, it is classically excited in the red wavelength range [17,18,19,20,21]. For PSs that have significant absorption peaks in the blue region, excitation at about 400 nm allows to excite fluorescence more effectively, while excitation in the red region allows light to penetrate deeper into the tissue and travel in the optical phototherapeutic window of biological tissues [22,23], in particular, to bypass significant absorption by blood components [14].

Fluorescence endoscopy allows intraoperative enhanced images of the tumor borders, i.e., surgical margins, to be visualized and analyzed. FC is known to be more sensitive than WLC in the detection of cancer lesions [1,3,24]. This is most important in the detection and complete resection of flat incipient and residual lesions of the bladder wall that might otherwise be missed [25].

As endoscopic imaging techniques, both WLC and FC provide images with a limited field of view of ~1 cm² that requires the urologist to scan the bladder with the risk for non-exhaustive scanning of the entire bladder wall. Furthermore, this procedure generates long sequences of images, i.e., large size files with highly redundant data, to be archived, which implies the tedious and time-consuming retrieving of information and generates problems for the systematic archiving of videos.

There are different medical devices on the market for fluorescence-based diagnosis of the inner surface of hollow organs and other cavities, including those operating in the augmented reality mode.

However, they have known disadvantages, among which are (i) a limited depth of tissue probing for all commercially available devices using blue light excitation, (ii) a small field of view (FoV), (iii) inappropriate data output format in the sense of the visualization of individual areas of interest that are not formed into a single map, which complexify the orientation between fluorescent areas for surgeons and significantly raise the number of errors when interpreting the fluorescence intensity, and (iv) a change in the distance between the detector and the surface of the biological tissue during image registration, which affects the value of the fluorescence intensity and leads to signal distortion, since the intensity of the recorded fluorescence signal depends on the distance between the detector and the surface.

Among the strategies for overcoming some of these limitations and, in particular, depth probing and FoV, red-shifted bimodal panoramic imaging techniques have been investigated, consisting of the automatic building of mosaics of high-resolution images superimposing WL and fluorescence (FL) information from PS excited in the red.

One of the directions of such methods is the development of image stitching algorithms aiming at obtaining a single map of the surface of a hollow organ using video data from an endoscopic examination [26,27,28]. The ongoing scientific research in image stitching is aimed at finding the optimal algorithm, the main aspects of which are high speed, minimization of the number of artifacts obtained during the construction of a single image-map, and compatibility with the existing technical equipment.

The present work describes the development of a new 3D bimodal experimental bladder model and its validation as an appropriate phantom for testing the combination of bimodal cystoscopy and image mosaicking. Unique results of image registration in augmented reality mode (including the registration of WL and fluorescent streams separately and synchronously) and mapping of realistic cancer-like fluorescing foci within the panoramic map build are presented.

2. Materials and Methods

2.1. Three-Dimensional Phantom of the Bladder Wall

A 3D “bimodal” phantom of a bladder wall embedding fluorescing inclusions was designed and prepared in order to validate the combined capabilities of the bimodal cystoscopic device (see Section 2.2) and the mosaicking algorithm developed (see Section 2.3). First, a 3D solid plastic base mold was printed in the form of a hemisphere (radius = 7 cm) mimicking the standard shape and dimension of a human half-bladder. Then, a 1-mm-thick silicone layer was prepared by pouring and distributing liquid silicone onto the plastic base until solidification. Next, to recreate a bladder wall surface as realistic as possible, a real porcine bladder tissue (obtained from a free market in dried form) was cut and fixed along the hemispheric silicone, hence allowing to acquire WL images with realistic bladder colors and textures.

Our so-called “bimodal” model refers to the fact that fluorescing foci mimicking bladder CIS fluorescence can also be imaged (using the fluorescence imaging modality of the bimodal cystoscope). To do so, PSs were directly injected at three different locations under the upper layer of the mucous membrane of the porcine bladder (Figure 1 and Figure 2). In order to perform our experimental validation for different PSs approved for clinical use, each of the three locations was injected with Ce6, MB, and PpIX, respectively. Moreover, a separate flap of a piglet bladder containing MB PS was also applied to mimic the papillary type of tumors (Figure 2).

The imaging system has already been tested in clinical settings and has shown that it detects fluorescence in real conditions with sufficient sensitivity [29,30]. In the present work, the experiment was designed to be as close as possible to the picture obtained in real conditions. The injected concentrations of PSs in the phantoms were chosen according to our practical experience in measurements of the average PS concentrations on real bladders (1 and 2 mg/kg for Ce6, 0.5 and 1 mg/kg for MB, and 2 mg/kg for PpIX).

In addition, in real conditions, we normalize to “normal” tissue; that is, we first remove the signal from obviously healthy tissue (including its autofluorescence) and assign the value “1” to the fluorescence intensity in this place. Due to the higher concentration, areas with PSs glow brighter, so their fluorescence index becomes greater than one (as a rule, in practice, it is 2–14). In this study, neither digitization of the fluorescence index nor exact PS concentrations were carried out, since this was not the purpose of the study and was not induced in the data processing.

The vascular network was reproduced with the use of a red marker. The phantom allowed to obtain intraoperative-like visualization through the bimodal cystoscopic system.

2.2. Bimodal Endoscopic System and Image Acquisition Protocol

A bimodal video system with a rigid endoscope featuring a 45-degree bevel was used for image acquisition. The main technical features of this device, developed by our team, are provided in [29,30]. Briefly, it consists of two cameras (a highly sensitive black-and-white digital CCD with an optical filter and a color camera), beam splitter, cystoscope, and illuminating devices (WL source, laser). The black-and-white camera formed a fluorescence image, while the color camera served as a navigating unit for the usual image acquisition.

The peak wavelength of the laser (635 nm) was chosen as the best trade-off between (i) using only one fluorescence excitation light source; (ii) extending tissue depth probing (compared to blue light excitation); and (iii) enabling the detection of three different PS (Ce6, PpIX, and MB). Regarding Ce6, despite its second main absorption peak in the region of 660 nm, its molar extinction at 635 nm is still about 5000 cm⁻¹/M (which cannot be seen clearly in Figure 3, where the graphs are peak-normalized to 1). However, it appears that this low level of absorption is sufficient to be detected in clinics by specific sensitive devices or for antibacterial use [31,32,33,34,35,36,37]. PpIX has a pronounced maximum at 635 nm (Figure 3), and examples of its detection in different organs being excited at 630–635 nm can be found in [13,14,29,30].

The phantom was irradiated with the laser light at a 635-nm peak wavelength to excite PS fluorescence (Figure 3) and, at the same time, illuminated with white light through the illuminating fibers of the endoscope to obtain the standard color image of the bladder wall. The images at a frame rate of 15 fps were stored as photos in BMP format along the endoscope tip manual displacement path (about 15 cm long, represented by the black dotted arrow in Figure 2). The distance between the endoscope and the phantom surface was maintained in the range of 3–4 cm. In the fluorescence excitation mode, the fluorescence emitted by all three PSs was clearly recorded (see scanning route at the surface of the bladder phantom at the Supplemented Materials). A total of 174 images in each modality (color images in WL mode and gray-level images in FL mode) was obtained over the endoscope displacement trajectory and selected for the analysis and formation of two spatially stackable WL and FL panoramic images.

2.3. Image Mosaicking in the White Light Modality

The principle of the bimodal mosaic construction is to find the geometrical transformations between white light image pairs and to apply them to their corresponding fluorescent images, with the assumption that the pixel correspondence is known between the two modalities by calibrating the two-camera acquisition set-up (this calibration is described in Section 2.4). The image registration and stitching algorithm developed for constructing the panoramic images in the white light modality runs on a computer with an Intel Core i5 12400 CPU (Intel, Santa Clara, CA, USA) and Nvidia GTX 1660 GPU (Nvida, Santa Clara, CA, USA) and was implemented in Python using the OpenCV library and materials from [40,41]. The image processing algorithm consists of the following steps:

• The image pre-processing [42] step includes image resampling (down to 0.6 mega-pixels), area masking, noise reduction, dead pixel removal, radial distortion compensation (using the OpenCV library and a chess board pattern), and shading correction.
• Finding the key points (relating to vessel textures/structures in the images; see Figure 2 and Figure 4) using the Scale-Invariant Feature Transform (SIFT) [43] algorithm (3 octave layers, 0.03 contrast threshold, 12-edge threshold, and a sigma value of 1.6) and the determination of homologous SIFT key points between the image pairs by the nearest neighbors search method with subsequent filtering by the threshold ratio test. These homologous key points are later used to determine the parameters of the homography that superimposes the common scene parts seen in two images. A homography is an appropriate geometrical transformation, since quasi-planar bladder parts are seen in the small field of view of the endoscopes.

• The stitching process exploits a graph in which the vertices correspond to images and where the edge weights are given by the number of correct matches of homologous key points between the two images linked by the edge. The images used to determine the mosaic are those with the largest weights.
• Simultaneous evaluation and optimization of the parameters of the homographies linking all image pairs with enough SIFT matches (i.e., with large weight values in the previous step) by minimizing a functional depending on the color differences of the superimposed pixels using the least squares method.
• Construction of a panorama from aligned images either by the seam search method [44], which gives the smallest path in the relative brightness graph of images (placement of seams in areas where the colors of the image are the most similar), or the Laplace pyramid blending method.

2.4. Augmented Visualization

An augmented image was obtained by overlaying the standard color panoramic image (WL mode) and the gray-level fluorescence panoramic image in green color through the alpha channel of the four-channel 32-bit RGBA model. Their alignment was carried out by a pre-computed projective transformation between the two cameras using a checkerboard pattern. Indeed, the intrinsic camera parameters defining the perspective 3D/2D projection matrix of the two cameras and the relative pose (extrinsic parameters) of the fluorescence camera with respect to the white light camera were calibrated using a calibration board (i.e., the chess board pattern). The knowledge of these geometrical parameters of the acquisition set-up was exploited to associate the pixels of the fluorescence images with their corresponding pixels in the white light images, making possible the augmented visualization of the mosaic.

3. Results

3.1. Image Acquisition

An example of WL and FL images acquired on two fluorescent regions of interest of the bladder phantom are shown in Figure 4. Fluorescence signals emitted by MB and Ce6 foci are clearly visible.

3.2. Image Mosaicking and Augmented Visualization

As a final result, color, fluorescence and augmented panoramic images (2292 × 1937 pixels with a spatial density of 176 pixels/cm) of the bladder wall model were obtained (Figure 5). Additionally, a simple desktop application was created that allows to control the visibility of the fluorescent layer and numerically estimate the fluorescence intensity in a number of rectangular regions, indicating and comparing the accumulation of the different PSs. The gray levels in the panoramic image were 20% and 43% for the full scale for MB, 10% and 13% for Ce6, and 15% for PpIX, which correlated with the injected concentration.

4. Discussion

Different works describe various bladder models created for such purposes as (i) training new endoscopic tools for white light cystoscopy and biopsy (phantoms with tumor simulations, including expanding ones) [45,46] or (ii) obtaining panoramic images of the inner wall surface of the bladder (phantoms made of different materials with different geometry, surface texture, and stiffness) [47]. Our model was reproduced using a unique technique and represents a three-dimensional hollow bladder with mucosa, tumor areas, and a simulated vascular pattern. The developed model takes into account the geometric, anatomical, vascularization, and fluorescence features of the bladder wall in the presence of a CIS tumor and fluorescent foci. This made it possible to perform imaging of the bladder wall under conditions close to the real one.

Different algorithmic approaches for endoscopic image registration, stitching, and surface reconstruction have been developed in recent years [48]. T. Soper et al. [49] proposed an algorithm for stitching images of the surface of the bladder in an offline mode. The method makes it possible to recreate a complete three-dimensional (3D) mosaic from endoscopic video using the structure from motion (SfM) technique. The software aligns the images by the mutual reconstruction of endoscopic motion and bladder shape. Next, a set of consistent characteristic points of the endoscopic video and matching pairs of overlapping video frames is determined. Then, a nonlinear least squares optimization method is performed to obtain the coordinates of the 3D points, the position and direction of the camera, needed for the reconstruction. The software successfully generated a 3D spherical bladder mosaic consisting of hundreds of images and thousands of 3D feature points. The study was carried out using a specifically designed flexible thin endoscope with a scanning fiber.

Stitching of the color endoscopic images can be performed to increase the field of view. In their minimally invasive surgery panoramic endoscopic system, Peng et al. [50] achieved enhancement of the image size up to 155% and construction of larger images with an additional 55.2% pixel increase.

Another example of the 3D modeling of non-fluorescent images of phantoms and different organs is a 3D reconstruction on phantoms using a dense Optical Flow (OF) for lightly textured surfaces such as the stomach [51]. The latter work presents a reconstruction of each 3D scene point using a large set of 2D homologous points extracted from a reference image and its overlay image obtained from different viewpoints. The results on the phantoms show that the accuracy of the proposed dense OF-based SfM method can closely approach the accuracy of a feature detection-based method.

K.L. Lurie et al. [52,53] developed a computational method for the reconstruction and visualization of a 3D model of organs from endoscopic video, which captures the shape and appearance of the organ. The post-processing algorithm consisted of sequential operations: pre-processing and the selection of key images, creation of a cloud of bladder surface points, and determination of camera poses to describe the position and orientation of the cystoscope in each key frame. Next, a coordinate grid was generated, and the texture was reconstructed.

Several studies [54,55] have investigated algorithms for constructing image maps of hollow organs from video data recorded during fluorescence endoscopy [56]. In the study [55], fluorescence maps of a 5-ALA-induced PpIX photosensitizer, excited with blue light, were built. However, this method was limited in stitching only fluorescence images and by the difficulty of placing the endoscope close to the bladder wall in order to obtain distinct images (due to the limited field of view).

Among the strong points of the method proposed in the present contribution are the ability to pan fluorescence images due to the presence of a color camera, as well as the simultaneous registration and visualization of a color image as a reference and a black-and-white fluorescence image with an overlay of the two images. In addition, image stitching is still rarely used for medical purposes; therefore, it has broad prospects and opportunities for development in this direction. Moreover, the correlation of the photosensitizer concentration with the gray level in the fluorescence panoramic image suggests the possibility of its use for the quantitative assessment of fluorescence.

However, when working with the presented method, it should be taken into account that the algorithm is sensitive to a loss of focus, glare, or foreign objects in suspension and sharp bends of surfaces when it is impossible to apply homography. For example, the obtained panoramic image contains errors in vessel alignment along the edges of the phantom due to sharp deviations from the plane, which can potentially be avoided by using the method of rotational advancement acquisition. Another interesting perspective of the experimental solution proposed is to provide an effective experimental framework to evaluate the performance of new “bimodal” similarity criteria used in the image registration and mosaicking algorithms, further exploiting the wealth of color and fluorescence information from both modalities.

5. Conclusions

The present work describes the development of a new 3D bimodal experimental bladder model, including realistic cancer-like fluorescing foci, and its validation as an appropriate phantom for testing the combination of a white light and fluorescence cystoscopy imager with a real-time augmented reality mode and an image mosaicking algorithm co-locating the cancer-like sites onto the color image map of the bladder wall. Valuable results of image registration in the augmented reality mode (including the registration of WL and fluorescent streams separately and synchronously) and mapping of realistic cancer-like fluorescing foci within the panoramic map build were obtained, which validated the proposed experimental model as an appropriate phantom for testing the combination of bimodal cystoscopy and image mosaicking.