Quantitative Analysis of Retinal Vascular Leakage in Retinal Vasculitis Using Machine Learning

Abstract

Retinal vascular leakage is known to be an important biomarker to monitor the disease activity of uveitis. Although fluorescein angiography (FA) is a gold standard for the diagnosis and assessment of the disease activity of uveitis, the evaluation of FA findings, especially retinal vascular leakage, remains subjective and descriptive. In the current study, we developed an automatic segmentation model using a deep learning system, U-Net, and subtraction of the retinal vessel area between early-phase and late-phase FA images for the detection of the retinal vascular leakage area in ultrawide field (UWF) FA images in three patients with Behçet’s Disease and three patients with idiopathic uveitis with retinal vasculitis. This study demonstrated that the automated model for segmentation of the retinal vascular leakage area through the UWF FA images reached 0.434 (precision), 0.529 (recall), and 0.467 (Dice coefficient) without using UWF FA images for training. There was a significant positive correlation between the automated segmented area (pixels) of retinal vascular leakage and the FA vascular leakage score. The mean pixels of automatic segmented vascular leakage in UWF FA images with treatment was significantly reduced compared with before treatment. The automated segmentation of retinal vascular leakage in UWF FA images may be useful for objective and quantitative assessment of disease activity in posterior segment uveitis. Further studies at a larger scale are warranted to improve the performance of this automatic segmentation model to detect retinal vascular leakage.

1. Introduction

The diagnosis and management of posterior uveitis with retinal vasculitis can be challenging. Posterior segment inflammation in uveitis is managed through ocular and systemic symptoms, clinical examinations, and chorioretinal imaging [1,2]. Multimodal imaging using fundus photographs, optical coherence tomography (OCT), OCT-angiography (OCTA), fluorescein angiography (FA), and indocyanine green angiography (IA) is critical for the clinical management of posterior segment uveitis [3,4].

Recent studies have demonstrated that ultrawide field (UWF) imaging is useful for peripheral findings in various posterior segment inflammations [5,6]. UWF provides a field of view of over two hundred degrees (82% of the retina region), while conventional fundus cameras capture only 30–55° of the posterior segment. By making a montage of several conventional fundus images, 60–90 degrees of wide field fundus images can be captured. However, the montage technique requires patient cooperation and fixation stability of the patient, and the capture time is too long. Moreover, a montage image might have artifacts which could affect uveitis analysis, considering the narrower view than the UWF camera. The current UWF imaging system is based on a confocal scanning laser ophthalmoscope and uses blue laser for FA. For detection of uveitis findings in the center to peripheral region, the current study used UWF FA images.

FA is widely used for detecting inflammatory lesions in the posterior segment, such as vascular leakage and ocular complications, including the retinal non-perfusion area and neovascularization in uveitis. FA is useful for assessing disease activity and severity, as well as in monitoring the response to treatment [1,2]. However, since the assessment of FA findings is subjective and descriptive, objective and quantitative measurements need to be developed for standardization of the clinical practice of uveitis [7].

Recently, machine learning has been applied for diagnosis and quantitative analysis of ocular findings in various retinal diseases, including diabetic retinopathy [8,9]. Studies regarding the automated measurement of vascular leakage in the assessment of retinal vasculitis are limited [10,11]. In a related recent study, leakages were automatically detected by evaluating the change over time in early- and late-phase UWF FA images [12]. Vascular segmentation was based on Gaussian operation, and registration was provided by using Fourier correlation of retinal vascular patterns. The intensity of leakage regions was equalized by spectral information, and leakage areas were binarized using a fixed threshold value. Parameters were set based on initial iterative feedback from expert readers on an initial preliminary training set. One problem to be solved, however, is the need for feedback from experts. The number of vessels in a UWF image is too great, and the vascular diameter is small; thus, vascular segmentation is challenging work for human experts. Conventional studies had to set many parameters for vascular segmentation, registration, and detection of the leakage area by expert feedback. Therefore, machine learning based on easy parameter setting should be discussed for leakage detection, although machine learning generally needs many images, and it is difficult to collect many uveitis UWF images. In the current study, we present an automated and quantitative analysis of retinal vascular leakage, including capillary leakage, using machine learning with a small dataset. Retinal vascular leakages were extracted on a subtraction image between early- and late-phase FA images based on weak supervised learning and unsupervised learning where early- and late-phase images were registered by vascular key points.

2. Materials and Methods

2.1. Patients and FA Images

The present study was conducted in accordance with the tenets of the Declaration of Helsinki and was approved by the Kyorin University Hospital Research Ethics Committee (protocol code 780-07 and 31 July 2020). In addition, a retrospective chart review and a waiver of informed consent were approved by the Ethics Committee. Patients were given the opportunity to “opt out”. The patients were referred to the Ocular Inflammation Service of the Kyorin Eye Center, Kyorin University Hospital, between 2016 and 2019, and diagnosed with either idiopathic uveitis with retinal vasculitis (three cases) or Behçet’s disease (three cases), using the Behçet’s Disease Research Committee of Japan criteria for all patients. Both “complete type” and “incomplete type” were included [13]. The anatomical location of inflammation was based on the Standardization of Uveitis Nomenclature (SUN) guidelines [14]. A single eye from each patient was included in this study. FA was performed at baseline and after the initiation of treatment.

2.2. Acquisition of FA Images

All UWF FA images were captured on Optos 200Tx or Optos California devices (Optos, PLC, Dunfermline, Scotland, UK). All patients received a standard infusion of 5 ml of 20% sodium fluorescein through the antecubital vein. Early-phase (45 s–1 min) images and late-phase (8–12 min) images were used for analysis. Exclusion criteria for this study included poor image quality, media opacification, and severe artefacts obscuring the fundus view.

2.3. Automated Retinal Vascular Leakage Segmentation

For automatic segmentation of the retinal vessel, we performed (1) deep-learning-based segmentation of retinal vessels in early-phase and late-phase UWF FA images using the U-Net network [15]; (2) registration of segmented retinal vessels between early-phase and late-phase UWF FA images; (3) generation of subtraction images for extraction of retinal vascular leakage in late-phase images; and (4) segmentation of retinal vascular leakage using cluster analysis.

2.3.1. Automated Segmentation of Retinal Vessels

The U-Net network structure was used as a learning machine for segmentation of retinal vessels [15]. The structure of the U-Net is a symmetrical U-shape, with the encoder on the left and the decoder on the right. The down-sampling path consisted of five stages, and the number of convolution filters was 64, 128, 256, 512, and 1024. The convolution layer was repeated twice in each stage, with a kernel size of 3 × 3 filters and a stride of 1, and a max pooling operation layer with a kernel size of 2 × 2 and stride of 2. The up-sampling path was performed using up-sampling 2D of Keras. The final convolution was performed with a kernel size of 1 × 1 and stride of 1. The accuracy was used for assessment of this U-Net network.

2.3.2. Pre-Processing for U-Net Training

Transfer learning, which has been called weakly supervised learning, was applied to U-Net, and it was found to improve retinal vessel segmentation [16]. Note that transfer learning needs a small number of supervised FA image data sets. Therefore, this paper proposes a learning method for U-Net that does not need supervised FA data sets by transforming RGB color images with supervised data sets in this section. In the current study, 60 color retinal images and 12 UWF images were used for training and testing of U-Net, respectively.

For U-Net training, 40 RGB color retinal images from color photographs of the Digital Retinal Images for Vessel Extraction (DRIVE) database [17] and 20 images from the Structured Analysis of the Retina (STARE) database [18] were used as training data for learning how to use the U-Net network. Each database includes manual vessel segmentation images by experts. The resolution of DRIVE images is 565 × 584 pixels and STARE images are 700 × 605 pixels. The resolution of FA images is 2500–4000 × 2500–4000 pixels; thus, the input size of U-Net was unified at 1024 × 1024 pixels. The training images were converted to 1024 × 1024 pixels by zero-padding of peripheral pixels. The fundus region of the training images was not enlarged to prevent the vascular width increasing. On the other hand, test images were reduced to 1024 × 1024 pixels, and their vascular width was also narrow. In order for the FA images to resemble RGB color images, as shown in Figure 1a, RGB images were pre-processed as follows. Grayscale images were generated after reading the RGB color fundus images. To improve limb darkening of the fundus image, contrast limited adaptive histogram equalization (CLAHE), as a histogram flattening method, was used for enhancement of the contrast in local areas of the fundus images. After that, gamma correction was used to adjust the intensity values, and negative-positive inversion was performed. U-Net was trained using these 60 pre-processed RGB color retinal images without FA images.

For pre-processing of the test data images (retinal vasculitis consisted of three cases: early-phase images, three images; late-phase images, three images, for a total of six images. Behçet’s disease consisted of a total of three cases: early-phase images, three images; late-phase images, three images, for a total of six images; 12 UWF FA images underwent CLAHE and gamma correction. U-Net was tested using these 12 pre-processed UWF images.

2.3.3. Assessment of Automated Segmentation of Retinal Vessels

Retinal vessels were manually segmented in UWF FA early and late-phase images of six patients using Photoshop v23 by an expert grader. For assessment of the performance level of automated segmentation of retinal vessels using U-Net, the Dice coefficient was analyzed between automatic segmented retinal vessels and manually segmented retinal vessels.

2.3.4. Automated Segmentation of Retinal Vascular Leakage

Following retinal vascular segmentation, early and late FA images were normalized using a mean intensity of 32 and mean deviation of intensity of 16 in early images, and a mean intensity of 128 and mean deviation of intensity of 64 in late images, since the signal intensity was variable between images. After normalization of each image, early and late image registration was performed using k-nearest neighbors from key points detected by the “KAZE” algorithm [19]. Then, projective transformation was conducted for the early FA images by parameter estimation, using the random sample consensus (RANSAC) algorithm. If global matching is performed, misregistration occurs in the retinal peripheral area. The retinal region was divided into four regions with horizontally and vertically centered macula, and registration and projective transformation were performed in each region.

Following registration and projective transformation, subtraction images were generated to extract retinal vascular leakage in the late-phase images. For the segmentation of the vascular leakage area, cluster analysis using the k-means method was used [20]. The vascular leakage area was classified into 10 clusters, and the cluster showing the highest intensity was defined as the vascular leakage area, and a binarized image was generated in white (=255) for the vascular leakage area and in black (=0) for the others. In late-phase FA images accompanied by a marked vascular leakage, leakage areas were classified into the top two clusters, because the leakage area was wide. In such FA images, many vascular regions were invisible due to high leakage. Therefore, the top two clusters were determined as leakage regions if the number of segmented vascular region pixels in the late-phase FA image was 50% lower than that of the early image.

2.3.5. Assessment of Automated Segmentation of Retinal Vascular Leakage

Retinal vascular leakage was manually segmented in UWF FA late-phase images using Photoshop v23 by an expert grader. For assessment of the performance level of automated segmentation of the retinal vascular leakage area, precision, recall, and the Dice coefficient were analyzed between the automatic segmented vascular leakage area and the manually segmented vascular leakage area in late-phase images.

2.4. FA Scoring System

The UWF FA score was determined by grading FA images based on an FA scoring system previously reported [7]. This FA scoring system consists of a total of 40 points summarized from nine parameters of FA signs: optic disc hyperfluorescence (maximum score 3 points); macular edema (4 points); retinal vascular staining/leakage (7 points); capillary leakage (10 points); retinal capillary nonperfusion (6 points); neovascularization of the optic disc (2 points); neovascularization elsewhere (2 points); pinpoint leaks (2 points); and retinal staining/pooling (4 points). We used four of the nine parameters (optic disc hyper fluorescence, macular edema, retinal vascular staining/leakage, and capillary leakage, for a total maximal score of 24 in grading the FA vascular leakage, according to the previous report [21]. One uveitis specialist determined the FA score using late-phase FA images before and after treatment in six patients through this system.

2.5. Statistics

Correlation analysis was performed using Spearman’s correlation coefficient test. Comparison between the two groups was performed using the paired t test. A p value less than 0.05 was considered statistically significant. Statistical analyses were performed using SPSS software version 28.0 (SPSS Inc., Chicago, IL, USA).

3. Results

3.1. Automated Segmentation of Retinal Vessels and Assessment of Registration of Retinal Vessel

Retinal vessels in six patients (early-phase and late-phase UWF FA images) before treatment were manually or automatically segmented using U-Net (Figure 2). The mean accuracy of automatic segmentation of retinal vessels was 0.962 ± 0.013 (0.939–0.980). The mean accuracies of early-phase FA images and late FA images were 0.955 ± 0.012 (0.939–0.970) and 0.970 ± 0.007 (0.964–0.980), respectively.

3.2. Assessment of Performance Level of Automated Segmentation of the Retinal Vascular Leakage Area

The retinal vascular leakage areas in six patients before treatment were manually or automatically segmented. The mean precision was 0.434 ± 0.164 (0.270–0.652), the mean recall was 0.529 ± 0.147 (0.340–0.767), and the mean Dice coefficient was 0.467 ± 0.138 (0.301–0.664). Figure 3 demonstrates a representative comparison between the manual and automatic segmented vascular leakage area.

3.3. The Correlation between the Automated Retinal Vascular Leakage Area and FA Score

To analyze the correlation between the automated retinal vascular leakage area and FA score in six patients before and after treatment, we evaluated the FA score according to the FA scoring system demonstrated in a previous study [7]. Figure 4a shows a statistically significant positive correlation between the automated segmented area (pixels) of the retinal vascular leakage area and the total FA score (Spearman’s correlation coefficient = 0.890, p < 0.001). Furthermore, Figure 4b shows a significant positive correlation between the automated segmented area (pixels) of the retinal vascular leakage area and FA vascular leakage score (Spearman’s correlation coefficient = 0.880, p < 0.001). Fluorescein angiography images and automatic segmentation of retinal vascular leakage of a representative case before and after initiation of IFX is shown in Figure 5.

The FA images revealed leakage from temporal, inferior, and nasal periphery retinal vessels before treatment. At one year after initiation of treatment, fluorescein leakage was improved. Automatic segmentation revealed white lesions (30,003 pixels) corresponding to the retinal vascular leakage area observed in FA images before treatment, and white lesions were reduced after treatment (528 pixels).

3.4. The Comparison of the Pixels of Automatic Segmented Vascular Leakage in FA Images before and after treatment

We compared the pixels of automatic segmented vascular leakage in FA images before and after treatment. As shown in Figure 6a, we compared the FA score from six patients before treatment and after treatment, and the mean FA score after treatment was significantly reduced compared with that before treatment. Similarly, as shown in Figure 6b, the mean pixels of automatic segmented vascular leakage with treatment was significantly reduced compared with that before treatment. These findings suggested that measurement of pixels for automatic segmented vascular leakage may be useful for objective assessment of vascular leakage before and after treatment in eyes with posterior uveitis.

4. Discussion

FA is the gold standard in detection of inflammatory lesions in the posterior segment, such as in retinal vascular leakage. FA is useful for the evaluation of disease activity and severity, and for monitoring during treatment [7]. However, since the evaluation of FA findings is subjective and descriptive, quantitative analysis of vascular leakage is required. In the current study, we explored the feasibility of automatic segmentation of retinal vascular leakage in UWF FA images in patients with uveitis. Our study implemented a deep learning method to extract retinal vessels using the U-Net system, and automated segmentation of retinal vascular leakage was performed. Subsequently, vascular leakage areas were quantified automatically, and the FA score was significantly associated with the automated segmented area (pixels) of retinal vascular leakage.

Retinal vascular leakage areas were detected by subtraction of early and late-phase images in the current study. Registration was applied by using vascular feature points as key points; thus, vascular segmentation was especially important. Recent reports have demonstrated an automated tool to extract retinal vasculature using a deep learning algorithm [22,23]. In the current study, we used a deep learning system, the U-Net system, for segmentation of retinal vessels in early-phase and late-phase UWF FA images. For learning using the U-Net network, the DRIVE and STARE databases were used as training data, rather than using UWF fundus images and UWF FA images as the training data set, suggesting that a supervised FA dataset is not required and can be replaced by transforming RGB color images with the supervised dataset. The whole region tends to be white in FA image with strong leakage, and only a part of the vascular area is described. The mean accuracy of vascular segmentation in early-phase FA images was 0.970. On the other hand, reported accuracies of color retinal images in recent years have been 0.97–1.0. U-Net training without target images achieved a good enough performance for the current study. The automated segmentation of retinal vessels using the U-net system in the present study may be feasible for the assessment and quantification of retinal vascular features in not only uveitis, but also other retinal diseases, such as diabetic retinopathy.

Three well-known registration methods, Oriented FAST and Rotated BRIEF (ORB) [24], KAZE, and Accelerated-KAZE (AKAZE) [25], were discussed for the current study. ORB performs as well as SIFT for feature detection and works rapidly. AKAZE is based on nonlinear diffusion filtering like KAZE, and it works faster than KAZE. For assessment of performance level of registration methods, the Dice coefficient was compared between the automatic segmented vascular areas in early and late-phase FA images. Mean Dice coefficients of ORB, KAZE, and AKAZE in 12 FA images were 0.469 ± 0.149 (0.105–0.658), 0.577 ± 0.111 (0.367–0.717), and 0.567 ± 0.111 (0.337–0.713), respectively. The current study applied KAZE, which had the highest Dice coefficient, although mean values of KAZE and AKAZE were similar. We conducted a paired t-test for Dice coefficients of KAZE and AKAZE and found no significant difference between the conditions (p = 0.15 > 0.05). Note that given the lower numbers of FA image pairs, this may be attributable to a lack of power. The reason for the low Dice coefficient was that the visible vascular area in the late-phase FA images was lower than that in the early images due to high vascular leakage. In our vision assessment, all 12 subtraction images showed good enough results for leakage segmentation.

The present study demonstrated that the automated model for segmentation of the retinal vascular leakage area through a subset of 12 UWF FA images reached 0.434 (precision), 0.529 (recall), and 0.467 (Dice coefficient), suggesting that the segmentation of retinal vascular leakage using UWF FA images is a challenging task for the current automatic segmentation model. Since brightness and contrast in FA images between the early-phase and late-phase images varied, normalization via pre-processing was performed in this study. In addition, since the signal intensity of the hyperfluorescent area in FA images was different between patients, we applied cluster analysis using the k-means method to extract the leakage area. Automated segmentation tended to over-extract the area between the leaking areas, as shown in Figure 3. Manual segmentation and automated segmentation were globally matched. However, this automated segmentation was based on unsupervised learning, so the Dice coefficient was not as high. As a result, the automated segmented area (pixel) and the FA score had a strong correlation, as shown in Figure 4. These results show the potential of automatic detection of the retinal vascular leakage area by registration of retinal vessels in early-phase and late-phase UWF FA images for the objective assessment of retinal vascular leakage in patients with uveitis.

Retinal vascular leakage is known to be an important biomarker to monitor the disease activity of uveitis [26,27]. For grading FA images, the FA scoring system proposed by the angiographic scoring for uveitis working group is widely used [7]. We investigated the correlation between the FA score and the retinal vascular leakage area (pixels) detected by our automatic segmentation system. We observed that there was a significant positive correlation between total FA score and pixels of the automated detected leakage area. Furthermore, the vascular leakage FA score was also significantly correlated with pixels of the automated detected leakage area. In addition, the mean pixels of automatic segmented vascular leakage, as well as the FA score, were significantly decreased compared with before treatment. Recently, Young et al. developed a U-Net based algorithm to detect retinal vascular leakage in the FA of patients with uveitis, and the trained algorithm obtained a Dice similarity coefficient of 0.563, suggesting that the algorithm can help clinicians more objectively compare vascular leakage between FA images [11]. U-Net is a supervised technique; thus, many annotated FA images are needed for U-Net training. Moreover, the results of leakage detection depend on annotation images made by human observers. It is difficult for a human observer to mark vascular leakage regions with high repeatability, because any abnormality is represented by a faint outline. Since the evaluation of FA images is subjective and descriptive, an objective and quantitative method to detect vascular leakage is needed to assess the disease activity in uveitis [7]. Our results suggest that automated quantification of the retinal vascular leakage area may contribute to the more precise and objective evaluation of inflammatory status in patients with uveitis in the posterior segment.

In the present study, since the signal intensity was variable between FA images, normalization was performed on both early-phase images and late-phase images, and we determined the mean intensity and mean deviation for segmentation of vascular leakage as much as possible in each early/late UWF FA images. Since the mean intensity and mean deviation for normalization depends on the signal level of each UWF FA image, it was difficult to determine the fixed signal strength in the UWF FA images. Further investigations are needed for determination of appropriate normalization values for machine learning using UWF FA images.

There were several limitations in this investigation. First, this study was a retrospective analysis and the sample size was small. Furthermore, the FA images were collected from a single institution. Second, the assessment of performance level of automated segmentation was not compared between Behçet’s disease and idiopathic uveitis with posterior segment inflammation due to the small sample size. Third, it was difficult for the automatic segmentation system in this study to distinguish retinal vascular leakage due to retinal vasculitis from fluorescein leakage due to neovascularization of the optic disc and neovascularization elsewhere, pinpoint leaks, and retinal staining/pooling, because the retinal vascular leakage area was segmented by subtraction of retinal vessels in the early-phase UWF FA image from the late-phase UWF FA image. Fourth, since the intensity of leakage in the peripheral area showed a trend toward being weaker than that in the posterior pole, several vascular leakage areas may have been missed. Fifth, several factors, such as image quality, media opacification, eye movement, uneven illumination, and eye lashes may have influenced the detection of retinal vessels and segmentation of vascular leakage. Further investigations are needed to improve the performance of this automated segmentation system to detect the retinal vascular leakage area in a large-scale study.

In summary, we propose the potential of automated segmentation of retinal vascular leakage in patients with uveitis in the posterior segment. Objective and quantitative analysis of FA images is required to evaluate clinical trials of new drugs for patients with intermediate, posterior, and panuveitis. Although the proposed model requires further improvement and validation before clinical use, this automatic segmentation system may provide helpful information for the objective assessment of retinal vascular leakage areas and precise monitoring of the clinical response to treatment. Assessment of FA images using automatic detection of retinal vascular leakage may be able to be developed into a clinical outcome measure in patients with uveitis in the posterior segment.

5. Conclusions

The present study demonstrated an automated model for segmentation of the retinal vascular leakage area using UWF FA images from patients with uveitis. Our study showed high accuracy of segmentation of retinal vessels and modest concordance regarding identifying vascular leakage compared to manually segmented vascular leakage from retinal vessels. A significant positive correlation was observed between the automated segmented area (pixels) of retinal vascular leakage and the FA vascular leakage score, and the mean pixels of automatic segmented vascular leakage was significantly reduced compared with that before treatment. The automated segmentation of retinal vascular leakage in UWF FA images may be useful for objective and quantitative assessment of disease activity in uveitis in the posterior segment. Further study at a large scale is required to improve the performance of this automatic segmentation model to identify retinal vascular leakage of eyes with uveitis.