Recurrent and Concurrent Prediction of Longitudinal Progression of Stargardt Atrophy and Geographic Atrophy towards Comparative Performance on Optical Coherence Tomography as on Fundus Autofluorescence

Abstract

Stargardt atrophy and geographic atrophy (GA) represent pivotal endpoints in FDA-approved clinical trials. Predicting atrophy progression is crucial for evaluating drug efficacy. Fundus autofluorescence (FAF), the standard 2D imaging modality in these trials, has limitations in patient comfort. In contrast, spectral-domain optical coherence tomography (SD-OCT), a 3D imaging modality, is more patient friendly but suffers from lower image quality. This study has two primary objectives: (1) develop an efficient predictive modeling for the generation of future FAF images and prediction of future Stargardt atrophic (as well as GA) regions and (2) develop an efficient predictive modeling with advanced 3D OCT features at ellipsoid zone (EZ) for the comparative performance in the generation of future enface EZ maps and prediction of future Stargardt atrophic regions on OCT as on FAF. To achieve these goals, we propose two deep neural networks (termed ReConNet and ReConNet-Ensemble) with recurrent learning units (long short-term memory, LSTM) integrating with a convolutional neural network (CNN) encoder–decoder architecture and concurrent learning units integrated by ensemble/multiple recurrent learning channels. The ReConNet, which incorporates LSTM connections with CNN, is developed for the first goal on longitudinal FAF. The ReConNet-Ensemble, which incorporates multiple recurrent learning channels based on enhanced EZ enface maps to capture higher-order inherent OCT EZ features, is developed for the second goal on longitudinal OCT. Using FAF images at months 0, 6, and 12 to predict atrophy at month 18, the ReConNet achieved mean (±standard deviation, SD) and median Dice coefficients of 0.895 (±0.086) and 0.922 for Stargardt atrophy and 0.864 (±0.113) and 0.893 for GA. Using SD-OCT images at months 0 and 6 to predict atrophy at month 12, the ReConNet-Ensemble achieved mean and median Dice coefficients of 0.882 (±0.101) and 0.906 for Stargardt atrophy. The prediction performance on OCT images is comparably good to that on FAF. These results underscore the potential of SD-OCT for efficient and practical assessment of atrophy progression in clinical trials and retina clinics, complementing or surpassing the widely used FAF imaging technique.

1. Introduction

Stargardt disease (also called juvenile macular dystrophy) is the most common cause of macular degeneration in children and young adults. It results in retinal damage and progressive vision loss and can result in eventual blindness [1,2,3,4,5,6,7,8,9,10,11,12,13,14]. AMD is the leading cause of blindness in people aged 65 and older in the Western world, with many of these patients appearing to eventually lose vision due to the development of macular neovascularization and non-neovascularization or GA. Stargardt atrophy and GA represent the endpoints of Stargardt disease and non-neovascularization AMD, respectively, in Food and Drug Administration (FDA)-approved clinical trials. The atrophic formation in juvenile and age-related macular dystrophy has been the most severe cause of vision loss and blindness. Until 2023, there had been no proven effective treatments for macular atrophy. Currently, two complement inhibitors, pegcetocoplan (Apellis Pharmaceuticals, Waltham, MA, USA) and avacincaptad (Iveric Bio, Parsippany-Troy Hills, NJ, USA), demonstrated positive Phase 3 clinical trial results in treating AMD atrophy and were cleared for clinical use by the FDA. As such, detecting macular atrophy and predicting the expected progression, and selecting the optimal therapy have been topics of pressing need and critical importance.

Fundus autofluorescence (FAF) is one commonly used imaging method in evaluating Stargardt disease and AMD [15]. With FAF, lipofuscin build-up can be assessed on the produced image of the retina. While FAF imaging has been widely used for diagnosing and predicting macular atrophy progression, it has notable limitations. FAF provides 2D imaging with restricted depth penetration and specificity. Additionally, the bright blue excitation light used in FAF can cause patient discomfort. In contrast, spectral-domain optical coherence tomography (SD-OCT) is a 3D imaging modality that excels in delivering high-resolution, cross-sectional images of the macula, making it ideal for a detailed tissue analysis in retinal layers [16,17]. OCT typically operates at wavelengths around 850 nm or 1050 nm, which generally enhances patient comfort. Hence, beyond FAF, it is important to develop effective approaches for diagnosing and predicting macular atrophy progression on OCT.

The convolutional neural network (CNN) architecture has been widely used in the task of automated semantic segmentation of Stargardt disease and GA [18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37]. However, current approaches to an automated analysis for macular atrophy generally do not incorporate longitudinal data, focusing on one instant in time. One way to incorporate longitudinal data is through the use of recurrent neural networks, such as the long short-term memory (LSTM) architecture, a modification of the recurrent neural network [38,39,40,41,42,43,44,45]. The LSTM architecture has been used in medical image analyses for the detection of cardiomegaly, consolidation, pleural effusion, and hiatus hernia on X-ray and for Alzheimer’s disease diagnosis [46,47]. In the field of ophthalmology, LSTMs have been successfully used for assessing glaucoma progression, predicting diabetic retinopathy, and predicting late AMD progression. However, in these cases, the LSTM architecture was used for prediction in the classification, not for prediction for the segmentation, of images. Additionally, it is important to being able to predict (or generate) future images (FAF or OCT enface) as well to assist clinicians for decision-making on treatment by visualizing both the predicted future atrophy regions and the associated generated or predicted images. However, there are no reported studies that can generate or predict future images for both FAF and OCT enface. Moreover, the retinal layers in volumetric SD-OCT images include rich information beyond the generally used image features (original retinal layer mean intensity and thickness features). To take advantage of the rich 3D retinal layer information on SD-OCT images beyond 2D FAF images, ensemble CNNs, an extension of CNNs, could be used, where several neural networks each handle separate inputs of the novel features based on the retinal layer and work together to concurrently predict the output of atrophy progression with the CNN features from the ensemble networks together [18,29,30].

Per our knowledge, there is so far only one Stargardt disease progression prediction reported that used a self-attended U-Net to predict atrophy progression at twelve months. It is indeed from our own group [25]. The limitation of our previous algorithm is that it only used baseline FAF images as input without additional longitudinal information. Other work in this area is largely focused on GA prediction. In Lee et al., CNNs were used to extract features from color fundus photographs, and these features were subsequently fed into an LSTM. However, they predict the 2- and 5-year risk of progression to late AMD rather than generating prediction masks of regions of atrophy [45]. In Banerjee et al.’s study, an RNN model is used with OCT and demographic data as input to predict progression to exudative AMD with certain timeframes but again does not attempt to generate prediction masks of atrophy [48]. In Zhang et al.’s study, a bi-directional LSTM prediction module is used with a 3D U-Net refinement module to predict locations of future GA growth on OCT imaging [49]. However, as with Lee et al., each timepoint is sent to a separate model which does not concurrently take advantage of ensemble features from rich 3D OCT retinal layer information.

Thus, in this study, we first develop an efficient predictive model for the generation of future FAF images and the prediction of future Stargardt atrophic (as well as GA) regions and then develop an efficient ensemble predictive model that concurrently utilizes advanced 3D OCT features at the retinal layer of ellipsoid zones (EZ) (where the atrophy occurs) for the comparative performance in the generation of future enface EZ maps and prediction of future Stargardt atrophic regions on OCT as on FAF. To achieve these goals, we propose a set of novel deep convolutional neural units that are enhanced with recurrent network (i.e., LSTM in this project) units and the concurrent learning of ensemble network unit networks (termed ReConNet) for the prediction of the future progression of atrophy in Stargardt and GA based on longitudinal patients’ data.

Our major contributions include the following: (1) The ReConNet (recurrent and concurrent neural network) architecture incorporates LSTM connections with encoder and decoder pathways for use on longitudinal data on FAF. (2) On OCT, the ReConNet further takes advantage of the enhanced OCT enface EZ feature maps in ensemble networks (termed ReConNet-Ensemble) to capture higher-order inherent EZ retinal layer features associated with atrophy for an enhanced algorithm performance. (3) The ReConNet can be used for the generation or prediction for both future FAF images and future Stargardt atrophy and geographic atrophy. (4) While the OCT suffers from lower image quality, our prediction performance on OCT images with ensemble EZ features is comparably good to that on FAF with the available data of Stargardt disease.

2. Methods

2.1. Imaging Dataset and Ground Truth

The Stargardt dataset consisted of 141 eyes from 100 patients from ProgStar study. Each eye had FAF imaging (Spectralis HRA+OCT 1.11.2.0; Heidelberg Engineering, Heidelberg, Germany) [1,2,3,4] performed at baseline (zero-month) and at six-, twelve-, and eighteen-month follow-up. Of these eyes, 71 were identified as additionally having SD-OCT imaging (Spectralis HRA+OCT; Heidelberg Engineering) performed at initial baseline, six-month, and twelve-month visits. The dataset consisted of 60 eyes from 60 patients obtained from Doheny Eye Institute clinics that had FAF imaging performed at initial baseline visit and six-, twelve-, and eighteen-month follow-ups.

The ground truth masks of both GA and Stargardt atrophic regions for all FAF images were manually delineated by certified graders at Doheny Image Reading Center. We used 85% of the images along with the ground truth for the training of the deep learning algorithms and used 15% of them for testing. For the GA dataset, the field of view was 30° with pixel dimensions of 768 × 868. For the Stargardt disease dataset, the field of view was 20° with pixel dimensions of 512 × 512. The FAF image scan dimensions were resized to a standard size of 256 × 256. The SD-OCT volumes had scan protocols with 496 × 1024 × 49 (depth × A-scans × B-scans) pixels or 496 × 512 × 49 pixels. All B-scans were resized to be 1024 × 496 pixels. After segmentation, enface feature maps were generated from each volume with dimensions of 1024 × 49 pixels. We resized the enface feature maps to 512 × 512 pixels when registering them to the initial-visit FAF images. They were further resized to 256 × 256 pixels when assembled for input to the neural network because of limited memory space. All the right (OD) eye scans and images were horizontally flipped so as to match them to left (OS) eye scans and images, providing consistency through the course of the analysis. As described in our previous work, all Stargardt dataset FAF images and SD-OCT-derived feature maps were registered to the initial baseline visit through feature-based image registration [50]. All GA dataset FAF images were similarly registered to their initial visit images. For Stargardt FAF images, contrast-limited adaptive histogram equalization was performed as part of pre-processing.

The ground truths utilized in this study for both GA and Stargardt atrophy were based on FAF images, as thus far, FAF images have been utilized as standard imaging modality for macular atrophy assessment in clinical trial studies. GA and Stargardt atrophy lesions on FAF images were graded using the semiautomated software tool RegionFinder 2.6.6.0 (Heidelberg Engineering, Heidelberg, Germany). Images were initially graded by a certified reading center grader, and the grading was subsequently reviewed by a senior grader. A senior investigator (SS) resolved discrepancies between the two graders.

2.2. Neural Network Structure

To achieve the two primary goals, we designed two network structures (ReConNet, ReConNet-Ensemble) for the macular atrophy prediction applications based on multiple patient visits: (1) prediction of future GA and Stargardt atrophy regions using longitudinal FAF images (ReConNet); (2) prediction of future Stargardt atrophy regions using longitudinal SD-OCT images (ReConNet-Ensemble). Additionally, we also investigated the feasibility of the prediction of interval growth of GA and Stargardt atrophy regions using longitudinal FAF images (ReConNet-Interval). All three algorithms were implemented using the open-source deep learning framework Keras.

The first neural network, ReConNet, used in this study was for longitudinal prediction of progression of GA and Stargardt atrophy on FAF images based on the network architecture, which has an encoding pathway and a decoding pathway, with a concatenation linking the output of an encoding pathway convolution to the input of a decoding pathway convolution. The term ReConNet was used for both FAF and OCT to be flexible in the future to include additional ensemble network inputs for concurrent learning. Such a deep learning architecture can be used for semantic segmentation using a relatively small training set of images [18]. To incorporate longitudinal data, the neural network in this study had multiple encoding pathways, one for each timestep. At each layer of the encoding pathway, the outputs of the encoding pathway convolutions were combined using a 2D convolutional-LSTM layer. The output of this layer was concatenated with the input of the corresponding decoding pathway convolution. Note that the overall network architectures for ReConNet and ReConNet-Interval were similar, and the difference was that the input for ReConNet was original FAF image (and related ground truth) and for ReConNet-Interval was the FAF image with the region of atrophy set to zero (and related interval ground truth), reflecting the growth regions between any two adjacent patient visits. A schematic of the neural network used in this study is depicted in Figure 1.

The second neural network structure, ReConNet-Ensemble, used in this study was for longitudinal prediction of progression of Stargardt atrophy on OCT images. The ensemble inputs included multiple enhanced enface feature maps derived from the OCT data surrounding the EZ in the depth (z-direction) of the OCT, the region of the retina most affected by Stargardt atrophy. These feature maps also extended beyond the traditional mean intensity maps, additionally including minimum intensity, maximum intensity, median intensity, standard deviation, skewness, kurtosis, gray-level entropy, and thickness of the EZ similar to that used in our previous work [18]. The network structure of ReConNet-Ensemble was similar to that previously described by our team [18]. The difference was that the previous one was for prediction using baseline alone, and the ReConNet-Ensemble in this project included longitudinal data with three patient visits (month 0, month 6, and month 12), reflecting dynamic changes of atrophy over time. In ReConNet-Ensemble neural network, multiple of the above-described neural networks were used to take in the multiple OCT-derived enface feature maps as input, allowing for the incorporation of the three-dimensional OCT information at EZ. In our previous work, we found significant improvement for the prediction of Stargardt atrophy when incorporating these advanced feature maps compared to using mean intensity alone. The logit layers of the individual neural networks were combined through averaging, with the combined result subsequently being sent through a softmax function to predict probability maps of atrophy. The ReConNet-Ensemble neural network is depicted in Figure 2.

Note that the ReConNet neural networks can predict both future images and atrophy. They are optimized by different loss functions depending on if they are used for image generation or for segmentation. For the future image generation (or prediction), the ground truth is the original image at that timestep. The cosine similarity loss is designed to optimize the performance of the CNN model by maximizing the similarity of the generated future image and the original image at that timestep. The cosine similarity loss is expressed as follows:

$$L_{cos}=-{\displaystyle \frac{{\sum }_{x\in \Omega }{p}_l\left(x\right)g_l\left(x\right)}{\sqrt{{\sum }_{x\in \Omega }{p_l}^2\left(x\right)}\sqrt{{\sum }_{x\in \Omega }{g_l}^2\left(x\right)}}}$$

The Dice loss is designed to optimize the performance of the CNN model by maximizing the overlap between predicted interesting regions and manually annotated interesting regions (i.e., ground truth). Thus, for the segmentation of atrophic regions, Dice loss is used as follows:

$$L_{Dice}=1-{\displaystyle \frac{2{\sum }_{x\in \Omega }{p}_l\left(x\right)g_l\left(x\right)}{{\sum }_{x\in \Omega }{g_l}^2\left(x\right)+{\sum }_{x\in \Omega }{p_l}^2\left(x\right)}}$$

where $g_l(x)$ is 1 if the pixel $x\in \Omega$ belongs to class $l$, otherwise 0, and $p_l(x)$ is the predicted probability that pixel $x$ belongs to class $l$.

2.3. Atrophy Prediction

2.3.1. Prediction of Future GA and Stargardt Atrophy Regions Using Longitudinal FAF Images (ReConNet)

The prediction of progression of geographic and Stargardt atrophy using longitudinal FAF images was trained with two steps (initial step of ReConNet1 and final step of ReConNet2; see Figure 3 for details). The initial step consisted of the prediction/generation of FAF image and atrophy regions (size and location) on the FAF image. The input for the neural network was the zero-month, six-month, and twelve-month FAF images, each paired with the manually graded atrophy label for that visit. The outputs were the predicted/generated FAF image and atrophy at eighteen months. In the second step, the generated/predicted FAF image and label were then used in conjunction with the zero-month, six-month, and twelve-month images in the neural network for enhanced atrophy prediction. An example of this process with the input and output at each step is shown in Figure 3.

2.3.2. Prediction of Progression of Stargardt Atrophy Using Longitudinal SD-OCT Images (ReConNet-Ensemble)

With the ensemble neural network for Stargardt atrophy progression using longitudinal SD-OCT images, a similar process of two steps (ReConNet1-Ensemble and ReConNet2-Ensemble) was performed. Note that for OCT data, we only had data from three patient visits (month 0, month 6, month 12). Instead of using single input of FAF image for each patient visit, in step 1, various enhanced feature maps at each patient visit derived from the zero-month and six-month SD-OCT scans (each again paired with the manually graded atrophy label for that visit) were sent through the ensemble network, producing both a predicted region of atrophy and a predicted enface image. As mentioned, the enhanced features demonstrated rich 3D atrophy information beyond the mean intensity associated features shown in our previous paper [18]. In step 2, rather than attempt to predict the individual twelve-month OCT-derived feature maps, the predicted enface image was used as a stand-in for the twelve-month feature maps and sent through the neural network again in conjunction with the zero-month and six-month feature maps.

2.3.3. Prediction of Interval Growth of GA and Stargardt Atrophy Regions Using Longitudinal FAF Images (ReConNet-Interval)

In this task, the interval growth of GA and Stargardt atrophy between visits was examined. In the zero-month, six-month, and twelve-month FAF images, the areas labeled as atrophy by graders was set to zero, and interval growth labels were paired with each visit. A placeholder label created by inverting the twelve-month atrophy region ground truth was paired with the twelve-month FAF image. The neural network, ReConNet-Interval, was trained to identify only the interval growth of the region of atrophy. Note that different from ReConNet and ReConNet-Ensemble, which were performed by two steps where the generated/predicted image and label were used in conjunction with the zero-month, six-month, and twelve-month images in the neural network for enhanced atrophy prediction, the ReConNet-Interval can only be performed for the first step, as the second step will need an additional visit after 18 months to obtain the interval. Hence, the performance of the one step ReConNet1-Interval may not be comparable with the enhanced atrophy prediction results after two steps from ReConNet2 and ReConNet2-Ensemble.

2.4. Performance Evaluation

Four metrics were used to evaluate the performance of the neural networks: pixel-wise accuracy, Dice coefficient, sensitivity, and specificity. Pixel-wise accuracy measures all correctly identified pixels in the image:

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(1)

where TP is true positives, TN is true negatives, FP is false positives, and FN is false negative. Sensitivity describes how much atrophy is correctly labeled compared to the total amount of atrophy:

$$Sensitivity={\displaystyle \frac{TP}{TP+FN}}$$

Specificity describes how much non-atrophied tissue is correctly labeled compared to the total amount of non-atrophied tissues:

$$Specificity={\displaystyle \frac{TN}{TN+FP}}$$

The Dice coefficient of two sets A and B describes the spatial overlap between the two sets:

$$Dice={\displaystyle \frac{2(A\cap B)}{A+B}}$$

3. Results

Neural network predictions were evaluated using the pixel-wise accuracy, Dice coefficient, sensitivity, and specificity. The initial and final prediction results for ReConNet are shown in

Table 1. Results for ReConNet (Median/Mean (SD)).

		Accuracy	Dice Coefficient	Sensitivity	Specificity
Stargardt Atrophy	ReConNet1-Initial	0.919|0.904 (0.072)	0.568|0.577 (0.163)	0.406|0.432 (0.163)	0.998|0.996 (0.008)
	ReConNet2-Final	0.980|0.973 (0.021)	0.922|0.895 (0.086)	0.876|0.84 (0.120)	0.998|0.996 (0.007)
	p-value	<0.001	<0.001	<0.001	0.020
GA	ReConNet1-Initial	0.901|0.896 (0.060)	0.867|0.827 (0.129)	0.84|0.839 (0.107)	0.971|0.938 (0.089)
	ReConNet2-Final	0.928|0.919 (0.042)	0.893|0.864 (0.113)	0.964|0.945 (0.062)	0.920|0.901 (0.077)
	p-value	<0.001	<0.001	<0.001	<0.001

. The initial and final results for ReConNet-Ensemble are shown in

Table 2. Results for ReConNet-Ensemble (Median/Mean (SD)).

		Accuracy	Dice Coefficient	Sensitivity	Specificity
Stargardt Atrophy	ReConNet1-Ensemble-Initial	0.955|0.92 (0.093)	0.742|0.662 (0.238)	0.691|0.63 (0.292)	0.992|0.977 (0.032)
	ReConNet2-Ensemble-Final	0.98|0.968 (0.033)	0.906|0.882 (0.101)	0.912|0.894 (0.086)	0.991|0.983 (0.029)
	p-value	<0.001	<0.001	<0.001	0.140

. The results for ReConNet-Interval are shown in

Table 3. Results for ReConNet-Interval (Median/Mean (SD)).

		Accuracy	Dice Coefficient	Sensitivity	Specificity
Stargardt Atrophy	ReConNet1-Interval	0.988|0.985 (0.009)	0.559|0.557 (0.094)	0.676|0.673 (0.145)	0.993|0.991 (0.006)
GA	ReConNet1-Interval	0.968|0.959 (0.025)	0.601|0.612 (0.089)	0.718|0.711 (0.155)	0.981|0.975 (0.020)

.

When comparing the performance of the neural network with and without the inclusion of a predicted eighteen-month FAF image, the Wilcoxon ranked-sign test was used due to the non-normal distribution of the data, resulting in p-values less than 0.05 for all metrics for all neural network configurations for both Stargardt atrophy and GA, except for with specificity in one case. Examples of inputs, output predicted regions of atrophy, and ground truths after ReConNet, ReConNet-Ensemble, and ReConNet-Interval are shown in Figure 4, Figure 5 and Figure 6, respectively. In Figure 4, the ReConNet model is applied to longitudinal FAF images to generate future images and predict the full extent of atrophic regions for both Stargardt atrophy and GA. In Figure 5, the ReConNet-Ensemble model combines features from multiple longitudinal OCT enface maps to generate future images and predict the full extent of Stargardt atrophic regions. Figure 6 demonstrates the use of the ReConNet-Interval model to predict the growth interval, rather than the entire atrophic regions, for both Stargardt atrophy and GA. Each of these algorithms has distinct advantages depending on the imaging modality used (2D FAF with ReConNet or 3D OCT with ReConNet-Ensemble) and the specific treatment goals. For instance, ReConNet-Interval is suited for scenarios where the focus is on controlling the rate of atrophic growth, while ReConNet is useful for assessing whether early atrophic changes have been addressed beyond merely suppressing growth.

4. Discussions

This paper represents the first attempt to use longitudinal imaging data to predict future regions of macular atrophy in the setting of Stargardt disease using recurrent (LSTM) and concurrent neural network architecture (ReConNet and ReConNet-Ensemble). The Stargardt atrophy progression prediction performance on OCT images is comparably good to that on FAF. These results underscore the potential of SD-OCT for the efficient and practical assessment of atrophy progression in clinical trials and retina clinics, complementing or surpassing the widely used FAF imaging technique. Additionally, the ReConNet is also applied for the GA progression prediction on FAF, demonstrating its feasibility potentially for general macular atrophy progression prediction.

For the prediction of atrophy growth in FAF using ReConNet, as shown in Table 1, the initial ReConNet1 predictions achieved a median Dice coefficient of 0.568 for eighteen-month predictions of atrophy in the setting of Stargardt atrophy using baseline zero-month, six-month, and twelve-month data. When this initial generated/predicted eighteen-month images and labels were used in conjunction with prior visits of longitudinal data, a median Dice coefficient of 0.922 for ReConNet2 was achieved, showing significant improvement (62% improvement, p < 0.05). For GA, median Dice coefficients of 0.867 for ReConNet1 and 0.893 for ReConNet2 were achieved with modest enhancement, probably due to the good performance of ReConNet1 and additionally the smaller size of the GA dataset (60 eyes) compared with the Stargardt atrophy dataset (141 eyes); hence, the improvement space is limited.

For the prediction of Stargardt atrophy growth in OCT, the ensemble neural network with various OCT feature maps resulted in a similar pattern of performance. Note that the ensemble neural network was only applied on Stargardt atrophy prediction (with month 0 and 6 to predict month 12 since further longitudinal data were not available). Additionally, the longitudinal OCT data for GA patients were not accessible in this study. Using initial zero-month and six-month data, the ensemble neural network achieved a median Dice coefficient of 0.742 for twelve-month predictions in the setting of Stargardt disease (71 eyes) for the initial ReConNet1-Ensemble, and using the initial prediction in conjunction with the prior visits of longitudinal data resulted in a median Dice coefficient of 0.906 for ReConNet2-Ensemble, again showing significant improvement (22% improvement, p < 0.05). Similar patterns of results were seen with pixel accuracy, sensitivity, and specificity, with the only exception of specificity for the prediction of Stargardt atrophy when using FAF images. In comparison to our previous work, we have achieved higher Dice coefficients for atrophy prediction through the inclusion of longitudinal data.

It is worth noting that the prediction results of the Dice coefficient for Stargardt atrophy using OCT based on ReConNet-Ensemble (Table 2 and Figure 5) is comparable to that using FAF based on ReConNet (Table 1 and Figure 4). This finding is critically important. Usually, FAF has been considered as the best imaging modality regarding imaging quality and has been used as a main imaging modality for clinical trials of new atrophy drugs. However, the FAF imaging acquisition process is very uncomfortable for patients, and it is in a two-dimensional imaging modality. OCT imaging is much more tolerable for patients, is a three-dimensional imaging modality, and is becoming the most popular imaging modality in major retina clinics. Hence, our novel recurrent and concurrent neural networks based on multiple advanced enface OCT maps, which reflect higher-order inherent retinal layer features associated with atrophy, provide a new, more efficient, and practical way for the prediction of atrophy progression in clinical trials and clinics.

When predicting only the interval growth of the atrophic lesions in FAF using ReConNet-Interval, in the setting of Stargardt atrophy, a median Dice coefficient of 0.557 was achieved. For GA, a median Dice coefficient of 0.601 was achieved. As mentioned, the ReConNet-Interval can only be performed for the first step, as the second step will need an additional visit after 18 months to obtain the interval that is not available for this project. Based on our investigation of the two steps’ schema for ReConNet and ReConNet-Ensemble, we can reasonably expect that the performance of the median Dice coefficient for the interval growth of Stargardt and geographic atrophic lesions for ReConNet2-Interval should be both greater than 0.80.

In Figure 4, it is possible to compare the initial and final predictions of the ReConNet neural network to the input FAF images and ground truths. Of interest in these images is the improved detail of the final prediction compared to the initial prediction. This indicates that in predicting the FAF image, the neural network may capture features important to the growth of atrophy that are not captured by segmentation alone. In Figure 5, we demonstrate a set of comparisons with the initial and final predictions based on the ensemble neural network alongside the input OCT feature maps and their corresponding ground truths. Once again, there is an improved final prediction after the incorporation of an initially predicted FAF image and ground truth. This indicates that the benefit of incorporating a predicted FAF image for the final prediction is consistently present across imaging modalities.

In Figure 6, it is possible to compare the predicted interval growth to the input-edited FAF images. It can be seen that the predicted interval growth tends to underestimate the ground truth. This is also apparent in the high specificity and relatively low sensitivity.

While these results are promising, this study is not without limitations. First, while our ReConNet architecture performs well with a limited sample size, the dataset sizes for both Stargardt disease and GA are relatively small, particularly for the GA dataset. Second, the longitudinal patient visits are limited, especially for the goal for the evaluation of the growth interval. Either more longitudinal patient visits or denser visits for ReConNet-Interval should be included to achieve a reasonably good atrophy progression prediction by applying two steps of ReConNet-Interval (i.e., ReConNet1-Interval and ReConNet2-Interval) similar as the ReConNet and ReConNet-Ensemble. Third, the registration process may introduce some errors due to the relatively low image resolution in one dimension. It results in slightly sheared, rotated, or translated registered OCT enface maps in comparison to the corresponding baseline FAF images. Fourth, while the Dice coefficient has been widely used as a standard metric for the evaluation of macular atrophy [51,52], the investigation of additional new metrics, for instance, as used in [53], may be useful, which can be a future direction of investigation.

5. Conclusions

In this study, we reported multiple methods of incorporating longitudinal data in the generation of future images and prediction of future regions of atrophy in Stargardt disease and AMD. There is high agreement with the manual gradings when predicting the whole region of atrophy, which is further improved when cycling the initial predictions, and incorporating them into the originally prior input longitudinal data. The prediction performance in OCT images is comparable to that using FAF, which opens a new, more efficient, and practical door in the assessment of atrophy progression for clinical trials and retina clinics, beyond widely used FAF. These results are highly encouraging for a high-performance interval growth prediction when more frequent or longer-term longitudinal data are available in our clinics. This is a pressing task for our next step in the ongoing research.