Study of Diagnostic Accuracy: Fundus Photography vs. Optical Coherence Tomography

Featured Application

This study assesses the diagnostic accuracy of non-mydriatic FP vs. OCT in posterior pole screenings; it determines the level of agreement between optometrist and ophthalmologist in the assessment of retinal abnormalities. It aims to prevent avoidable cases of blindness or low vision in the relevant population by achieving the better and wider detection of pathologies of the posterior pole.

Abstract

(1) Background: This study aimed to determine the diagnostic accuracy that optical coherence tomography (OCT) can add to fundus photography (FP) in assessing the condition of the retinal posterior pole. (2) Methods: We conducted two blocks of analysis: First, the posterior pole of each eye was examined using an FP non-mydriatic imaging device. Second, OCT was used in addition to FP. After consolidating the specific diagnostic criteria, the assessments were evaluated using two blinded independent investigation groups (by optometrists, and by ophthalmologists who were considered the gold standard). (3) Results: We calculated the diagnostic accuracy of FP compared to OCT and found that they had similar sensitivity. FP had a slightly higher specificity (p-value: 0.01), and OCT had a higher kappa coefficient with 0.50 (95% CI: 0.46–0.55) vs. 0.39 (95% CI: 0.34–0.45) for FP. (4) Conclusions: On the basis that the role of the optometrist in Spain is not to diagnose but to detect lesions and refer patients to an ophthalmologist, the results of this study support the use of OCT, which provided gradable images in almost all examined eyes (97.5%), compared to FP (73.5%). However, optometrists need a detailed and standardized guide in order to conduct evaluations according to the ophthalmologist’s criteria.

1. Introduction

According to the World Health Organization, globally, at least 2.2 billion people have a vision impairment or blindness, of whom at least 1 billion have a vision impairment that could have been prevented or has yet to be addressed [1]. “Eye conditions and vision impairment are widespread, and far too often they still go untreated”, stated Dr Tedros Adhanom Ghebreyesus, the WHO’s Director General.

Furthermore, the fact that 80% of cases of visual impairment are preventable shows that there is a great deal of room for improvement. The number of newly registered people with maculopathies in the Spanish National Organization of the Blind increased by 14% in 2022 and 5.9% for those with glaucoma [2,3,4,5].

Optical coherence tomography (OCT) has revolutionized the way in which vitreoretinal diseases are assessed and studied. The usefulness of OCT in clinical practice is becoming increasingly relevant [6,7,8,9]. Frequently, the general population undergoes eye examinations to procure eyeglasses. Eye exams with fundus photography that are conducted by optometrists are limited, because FP does not show all of the possible posterior pole alterations. In Spain, the optometrist cannot use mydriatic drops, meaning that using FP as part of an optometrist’s practice encounters some problems, such as miosis. Furthermore, non-mydriatic FP is associated with some limitations, such as corneal opacity or the opacity of the ocular media [10].

So far, various articles have compared OCT to FP in relation to different characteristics, such as their ability to improve glaucoma screening [11], or in the screening of age-related macular degeneration [12]. The findings demonstrated substantial differences in the diagnostic accuracy between screenings using FP with OTC and FP alone. In a glaucoma screening, FP’s sensitivity was 55.4% while the sensitivity of FP + OCT was 80%; however, their levels of specificity were similar (92%). In the screening of age-related macular degeneration, 97.7% of the images were gradable using OCT while only 52.4% were gradable using FP.

This study would allow us to determine the clinical relevance of OCT in identifying posterior pole abnormalities [13,14,15,16]. It focused on two main areas: determining the value that OCT adds to FP in optometric practice and assessing what proportion of alterations are visible in OCT and not in FP. We further sought to identify differences in the judgment accuracy between ophthalmologists (gold standard) and optometrists.

To achieve the early detection of pathologies or retinal alterations such as diabetic retinopathy, glaucoma, or macular holes, is important that the optometrist, who is visited by patients more frequently, has the optimal criteria and instruments to carry out high-quality population screenings. Regarding the assessment of FP, in an article by Carmen Novo-García et al. published in the Journal of Clinical Nursing [17], one shortcoming is that FP of various patients could not be assessed by the nursing professionals in charge in a study of the prevalence of diabetic retinopathy; therefore, patients were not referred to the ophthalmologist. The National Health System faces specific problems such as the lack of development of primary care. The following are two problematic situations in Spain: over referral, due to distrust of FP that sometimes collapses the emergency service; and sick cases not being diagnosed in time which generates a higher cost in the long term.

We followed the Standards for Reporting of Diagnostic Accuracy (STARD) guidelines to conduct this comparative study. Designing a study to evaluate the performance of these tests presents several challenges, not least the need to have a reference standard, as the tests are subject to human interpretation [18,19,20].

For these reasons, we consider this article of interest, since the intention is to quantify the diagnostic precision involved in evaluations based on FP alone, as are currently performed in Spain, for those patients with posterior pole abnormalities whose alterations are not visible in FP. The problem is that their pathology is not prevented or has not yet been addressed, it worsens and its good treatment or solution decreases. It is important to know the percentage of patients who would have been referred to the ophthalmologist in time and could be dealt with appropriately using OCT. In Spain today, this situation is a problem of the National Health System. It is obvious that FP does not provide the same information as OCT, but there are no studies so far that quantify the undiagnosed patients and to what extent the inclusion of the OCT would solve this situation.

2. Materials and Methods

This is a prospective, observational, and comparative study of diagnostic capability. The study adhered to the principles of the Declaration of Helsinki and was approved by the Ethics Committee of the European University of Madrid. These principles were followed during all stages of the study. Approval Code: CIPI/22.220. Approval Date: 29 July 2022. Written informed consent was obtained from all the participants.

In this study, two main areas are analyzed: how much OCT contributes to FP in the diagnosis of the posterior pole, and what difference there is between the assessments carried out by the ophthalmologist and the optometrist based on FP and FP + OCT. To this end, the following methodology was used.

2.1. Description of the Patients

The study population consisted of people who came to the optometry clinic for various reasons over a period of four months, between July 2022 and October 2022. The only inclusion criterion was age; patients had to be between 40 and 90 years of age, regardless of their gender or other characteristics. Anyone entering the optometry clinic was offered the opportunity to enter the study. The recruitment center was a Spanish optometry clinic. Because of the inclusion criterion and the participant recruitment plan, the sample was defined as a consecutive series of participants between 40 and 90 years of age. The sample included people who gave their consent before the tests were carried out; information related to the eyes was then obtained, and the researchers analyzed the results (Figure 1).

2.2. Test Procedure

Participating subjects were given a verbal and written explanation of the purpose of the study and the methods to be used. Written informed consent was obtained from all the participants.

The steps commonly used to carry out tests in an optometry clinic in its usual practice were followed, in order to achieve the greatest possible affinity with reality and, in this way, to calculate the precision of eye examinations in adults. Therefore, complete and detailed eye examinations were performed (corrected distance visual acuity (CDVA) and refraction, PF, macular and papilar OCT). The diagnostic accuracy assessment included components such as background information and images, and it was conducted in a conventional optometry clinic. The examinations were performed by an optometrist external to this study in order to avoid bias.

The proposed methodology can be divided into three main steps: an assessment step, an analysis step, and a diagnostic precision calculation step. During the assessment step, the two groups of evaluators (ophthalmologists and optometrists) evaluated and classified the posterior poles of the patients according to the information given in each stage (FP and FP + OCT). During the analysis step, the classifications made by each group were compared. In the last step, all the information obtained was compiled and calculations were conducted.

2.2.1. Assessment Step

An optometrist external to the analysis performed the tests on each patient and, afterwards, encrypted the information derived from the examinations performed. The data were sent to the groups of examiners (optometrists and ophthalmologists) in two different files: one with the patients’ information, visual acuity, and FP (stage I) and another with this information plus the reports obtained using the macula 3D OCT and optic nerve 3D OCT (stage II). Figure 2 and Figure 3 show an example of the information sent to examiners at Stages I and II to assess each eye.

The information gathered during these examinations, the best corrected distance visual acuity, and the stored digital images (FP non-mydriatic imaging and report of macula 3D OCT and optic disc 3D OCT) were independently graded by the two research groups. Both independently graded the posterior pole of each eye based on the masked information provided in each assessment stage.

Assessment stages:

-

Stage I: based on a conventional eye examination and FP.

-

Stage II: based on a conventional eye examination, FP, and OCT.

The eye fundus classifications were made following the criteria previously established by the group of ophthalmologists and optometrists to ensure the consistency of the analysis methods. The eye fundus was classified as follows:

(a)

Not evaluable: images are of insufficient quality to categorize the state of the fundus of the eye.

(b)

Evaluable: images are of sufficient quality to categorize the state of the eye fundus. The evaluable cases could be healthy or have abnormalities. The eyes with abnormalities were classified in two groups depending on the severity of the alteration, referable or preferential referable (Figure 4).

• Healthy includes cases that have a normal appearance, which rules out any pathological alterations in its characteristics aside from those associated with age.
• Referable includes cases with physiognomic characteristics that lead to an alteration that must be assessed as minor or major, as measured by an ophthalmologist at any point in time. These cases include those that present an abnormal appearance that modifies the retinal morphology and a structure that shows deterioration in the ocular health of the posterior pole.
• Preferential referable includes cases with characteristics that suggest any alteration that must be evaluated by an ophthalmologist within a short period of time.

2.2.2. Statistical Analysis

After completing the evaluations, the group of ophthalmologists and optometrists analyzed and compared the results; the group of ophthalmologists represents the gold standard.

All statistical analyses were performed using SPSS version 27. Differences between groups were analyzed using Pearson-Chi-square tests for categorical variables. Logistic regression models were used to analyze the odds ratios of the outcomes with 95% confidence intervals.

To carry out this stage, archives were used with the evaluations of the two groups of evaluators collected in tables. In these tables, the information of the patients collected at the beginning was the number assigned to the patient, right or left eye, age, sex, and CDVA. Secondly, the assessment in stage I (FP) and stage II (FP + OCT) by ophthalmologists and optometrists took place, with numerical assessments. The meanings of the numerical evaluations are as follows: 0: not evaluable, 1: healthy, 2: referable, and 3: preferential referable.

This information was used to perform statistical calculations.

During this step, the diagnostic precision, sensitivity, specificity, likelihood ratios, and kappa index calculations were performed, in order to quantify the improvement resulting from the joint use of the FP and the OCT. At the same time, we aimed to determine whether or not the degree of agreement between the ophthalmologist and the optometrist improved when using both techniques.

2.3. Instrumentation

A Nidek ARK-510A automated refractometer/keratometer, (NIDEK, Gamagori, Japan) and a Snellen chart were used for the conventional eye examination. Subsequently, a Topcon Maestro2 (TOPCON, Tokyo, Japan) was used for the FP, macula OCT, and optic nerve OCT. The 3D macula option was used for the OCT with a scan area of 6 mm × 6 mm. A report was generated for each eye, which included a retinal thickness map and a reference database. The 3D optic disc OCT combines disc topography, FP, and measurements of the retinal nerve fiber layer (RNFL)’s thickness.

3. Results

3.1. Sample

After a period of 14 months of analyses and evaluations of the eye fundus, conducted independently by the two research teams, a total of 1334 eyes from assessments of 667 Spanish patients were obtained in this study, 68.5% of which were for female patients. The mean age of the participants was 60 ± 18 years (interquartile range of 40–90 years). The CDVA was 0.89 and the median was 1, meaning that 50% of the records had a CDVA of 1. Two people were excluded because they did not want to participate in the study for personal reasons.

The sample was made up of 1334 eyes from 667 people. Although the sample is representative of the Spanish population, as seen in Figure 5, the participant group aged between 40 and 51 was smaller than this group is in the overall Spanish population; there were slightly more people aged between 58 and 84 years old.

To simulate the daily practice of the optometrist who has the FP and the one who has the OCT, the following assessments were conducted. In the first stage, each research group reviewed and classified the posterior pole of each eye, based on the information given and on the conventional eye examination together with the FP images. During this stage, of the 1334 eyes, 974 could be evaluated. In the second stage, the research groups classified the status of the posterior pole based on the information used in the previous stage together with the reports provided by the 3D macula OCT and 3D optic nerve OCT. During this stage, of the 1334 eyes, 1278 could be evaluated (Figure 6). The results of first stage (FP), according to the ophthalmologist, were as follows: there were 354 non-valuable eyes and 980 valuable eyes, of which 817 were healthy eyes and 163 were altered eyes, and 133 were referable eyes and 30 were preferential referable eyes. According to the optometrist, there were 141 non-valuable eyes and 1193 valuable eyes, of which 694 were healthy and 499 were altered, and 484 were referable and 15 were preferential referable. According to the ophthalmologist, the results of the second stage (FP + OCT) were as follows: there were 33 non-valuable eyes and 1301 valuable eyes, of which 963 were healthy eyes and 338 were altered eyes, and 268 were referable eyes and 70 were preferential referable eyes. According to the optometrist, there were 24 non-valuable and 1310 valuable eyes, of which 719 were healthy eyes and 594 were altered eyes; these were divided into 548 referable eyes and 43 preferential referable eyes.

The results obtained by the ophthalmologists are summarized in

Table 1. Distribution of disease severity according to the opthalmologist’s assessment.

Assessment		FP	FP + OCT
Evaluable	Healthy	817 (61.2%)	963 (72.2%)
	Referable	133 (10.0%)	268 (20.1%)
	Preferential Referable	30 (2.3%)	70 (5.2%)
Not Evaluable		354 (26.5%)	33 (2.5%)

. They show that 24% of the sample could be evaluated by adding OCT to the FP examination. The OCT provided evaluable images in almost all of the examined eyes (97.5%), compared to the FP value of 73.5%. Figure 7 shows an example of a posterior pole alteration that was not visible when using FP.

When analyzing the results using a gold standard assessment, 13% of the sample, which was not assessable when using FP, exhibited alterations, and an additional 2.9% of cases exhibited abnormalities requiring an urgent referral. The results shown in Table 1 display 40 eyes with severe retinal alterations that were not referred as quickly as they needed to be due to OCT not being used for their assessment.

3.2. Diagnostic Accuracy Values and Interobserver Agreement between Ophthalmologists and Optometrists

To calculate the diagnostic precision of both instruments, FP and OCT, a 2 × 2 confusion matrix was created (Appendix A). The reference values, true positives, and true negatives were defined as cases diagnosed by the group of ophthalmologists, which was the gold standard in this study. To calculate the predictive values (sensibility, specificity…) and the Kappa coefficient, we used the 2 × 2 confusion matrix with contingency tables (

Table 2. Confusion matrix used to define the optometrist’s performance. Comparison based on FP. The total sample (n) is reduced from n = 1334 to n = 974.

	Positive	Negative
Positive	561	250
Negative	11	152

and

Table 3. Confusion matrix used to define the optometrist’s performance. Comparison based on FP and OCT. The total number of observations is reduced from n = 1334 to n = 1287.

	Positive	Negative
Positive	681	272
Negative	31	303

).

For the statistical analysis, the sample was divided into posterior poles without abnormalities (healthy) and posterior poles with abnormalities (including those that are referable and urgently referable). Not evaluable cases were excluded [21,22,23]. The sample was reduced from 1334 to 974 in the FP-based assessment and from 1334 to 1287 in the assessment based on OCT and FP due to the poor image quality caused by the opacity of the cornea or of the ocular media, miotic pupils, or a lack of patient cooperation.

A series of statistical measures were used to provide safe and reliable results, with the aim of ensuring that optometrists can provide high-quality and valid assessments in their daily practice. As a result, these methods can be used to confirm the presence or absence of posterior pole abnormalities [24,25,26,27,28,29,30,31,32,33,34,35,36].

Table 4. Summary of diagnostic accuracy results. Diagnostic analysis of non-mydriatic fundus images vs. OCT reports from patients in a retinal screening; comparison between ophthalmologists (gold standard) and optometrists (95% confidence interval: CI 95%).

Variable (CI 95%)	FP	OCT + FP
Accuracy	0.73 (0.70–0.76)	0.76 (0.74–0.79)
Sensitivity %	98 (0.97–0.99)	96 (0.94–0.97)
Specificity %	38 (0.33–0.43)	53 (0.49–0.57)
PPV	0.69 (0.66–0.72)	0.71 (0.69–0.74)
NPV	0.93 (0.88–0.97)	0.91 (0.87–0.94)
Likelihood (+)	1.58 (1.46–1.70)	2.02 (1.85–2.20)
Likelihood (−)	0.05 (0.03–0.09)	0.08 (0.06–0.12)
Kappa coefficient	0.39 (0.34–0.45)	0.50 (0.46–0.55)

shows the calculated values of sensitivity, specificity, positive and negative predictive values (PPV, NPV), the likelihood ratios, and the kappa coefficient [27,28,29,30,31,32,33,34].

The results show that, when comparing the assessment of the optometrist with that of the ophthalmologist (gold standard), based on retinography, the sensitivity is markedly higher than the specificity, with 0.981, CI 0.97–0.99, versus 0.378, CI 0.33–0.43. The same is true of sensitivity and specificity for OCT, but with a less marked difference: 0.956, CI 0.94–0.97, versus 0.527, CI 0.49–0.57. This is due to an increase in false positives in the assessments conducted by the optometrist, that is, the optometrist tends to judge more cases to exhibit alternations when the ophthalmologist does not consider this to be the case. The presence of high sensitivity values compared to the relatively low specificity values is affirmed by the likelihood ratio values. After performing the contrast test to see whether there were differences between the results obtained using FP and those obtained using FP together with OCT, we confirmed that there are statistically significant differences, since the p-value obtained was 0.01.

According to the classification of Landis and Koch when comparing the optometrist and ophthalmologist evaluations in each block, we found that there was fair (0.39, CI: 0.34–0.45) agreement between the evaluations based on FP, which increased to moderate agreement (0.5, CI: 0.46–0.55) when OCT was added. Therefore, it can be deduced that the degree of agreement, beyond chance, between the optometrist and the ophthalmologist is greater when adding OCT to the FP for the evaluation of the patient’s posterior pole.

4. Discussion

In Spain, the waiting list for outpatient consultations in the National Health System as of June 2023 is, on average, almost three months [35]. Based on a review of the literature on screening studies carried out in Spain, the coverage of diabetic retinopathy screening is 32.4% (95% CI: 30.8–34.0%) [36]. This level of coverage falls far short of the recommendation of the portfolio of standardized services, according to which all diabetic patients should have a biennial examination of the eye fundus. It is therefore vital for optometrists to be in a position to help in the screening of patients during the referral process, not only from the anterior pole but also from the posterior pole, with sufficient criteria to determine whether the patient can wait three months or must be referred via a faster route.

In this study, for the first time, a direct comparison is made between FP and OCT in patients who were unaware of the health status of their posterior pole, thus avoiding bias on the part of the evaluators, or a timelier fixation on specific pathologies. It also provides a comparison of the assessments carried out by the optometrist using both instruments and those carried out by the ophthalmologist. In a 2021 study by Ouyang Y [37], 3D OCT exhibited a good capacity to detect most of the characteristics of different alterations. It noted that OCT will be added to FP screening for chorioretinal disorders in the future. The studies published to date focus on specific pathologies, such as Jain, N.’s quantitative comparison of drusen segmented on SD-OCT versus drusen delineated on color fundus photographs [38], or Hao Y’s paper [39], which indicated that the combination of OCT with fundus photography can effectively identify disease. In a study of high levels of myopia, the sensitivity, specificity, PPV, and NPV of OCT combined with fundus photography were more than 94.0%, which were significantly higher than OCT or fundus photography alone (p < 0.05).

Our research verifies the validity of using OCT as an addition to FP by calculating diagnostic accuracy values when comparing the use of these tests (p-value: 0.01). The kappa coefficient also indicates that the proportion of better-than-chance agreement compared to the maximum possible agreement is higher when OCT is used in addition to FP.

In an optometry clinic, retinal assessments based on FP and OCT are similar in terms of sensitivity; however, FP combined with OCT is more specific than FP alone. These results are similar to those reported by Tomoyuki Watanabe et al. [11], who evaluated the accuracy of glaucoma screening using FP combined with OCT and determined the agreement between ophthalmologists and ophthalmology residents. The diagnostic accuracy of the glaucoma screening significantly increased with the use of OCT. Ophthalmologists exhibited the same sensitivity (80%) as ophthalmology residents in glaucoma screenings.

These improvements would enable optometrists to provide quality assessments with sufficient criteria to refer posterior pole abnormalities to an ophthalmologist as appropriate. This is of particular relevance in Spain, where optometrists do not diagnose but are responsible for detecting any abnormalities that should be referred to an ophthalmologist.

Although the investment required to acquire OCT equipment is high, the prices are becoming more and more affordable. Optometrists are the first point of contact for patients suffering from visual impairments, so are well placed to assist in the early detection of pathologies of the posterior pole. Therefore, the price does not diminish the significance of the findings, as they highlight the importance of complementing FP with OCT in optometry clinics.

In recent years, artificial intelligence has shown great promise in detecting pathologies in the posterior pole (see [9]). Medical imaging is expanding globally at an unprecedented rate, resulting in an ever-increasing amount of data that must be interpreted by humans with experience with the criteria. There is currently a relative shortage of professionals who are trained to perform diagnoses of this magnitude [40]. A review of the bibliography shows that algorithms are being generated to identify retinal pathologies based on OCT images in data-limited situations, such as that described by Karri SP et al. [41], who discuss the algorithms used for the detection of diabetic macular edema and dry age macular degeneration (AMD). In the future, artificial intelligence may speed up patient screening. In the meantime, however, a good assessment of the posterior pole by optometrists could include improved patient screening and earlier referrals to the ophthalmologist. According to Sarraf, a clinical professor of ophthalmology in the Retinal Disorders and Ophthalmic Genetics Division at Jules Stein Eye Institute at University of California [42], advances in imaging and the experience of a skilled clinician are still essential in fundus examinations; while machines can conduct exams, no machine can give a patient a sense of comfort and gratification.

OCT imaging appears to improve the power of screening compared to FP alone. More complete studies should be carried out with a greater number of patients and more patient diversity in order to evaluate the option of performing screening using FP with OCT; this would verify the results obtained here.

After conducting a careful analysis of the results, we identified certain limitations in this study. We must bear in mind that the sensitivity values were high compared to the specificity values obtained for FP and OCT. This can be explained by the fact that optometrists tend to over refer, that is, to consider more posterior poles to be altered than the ophthalmologist does. Therefore, it is important to generate a detailed and standardized guide containing all possible alterations of the posterior pole, in order to determine according to the ophthalmologist’s criteria which cases should be referred and with what degree of preference. In addition, more studies with more researchers are needed for an analysis with more contrast.

5. Conclusions

OCT provides optometric clinics with the possibility of conducting more reliable analyses with a greater degree of interobserver agreement between the ophthalmologist and the optometrist. As the first point of contact for the patient, optometrists would therefore have a more highly informed assessment criterion for referral to an ophthalmologist, but they also need a detailed and standardized guide to obtain a higher degree of agreement with the ophthalmologist’s assessment criteria.

The addition of OCT to a screening modality with FP could resolve the issue that 1 in 10 eyes with visible abnormalities on OCT images are not identified by the ophthalmologists when the assessment is based on FP alone. This highlights the need for further research and data collection on this issue to confirm whether OCT should be included as part of an outpatient screening program, with the aim of detecting all patients with posterior pole abnormalities. This would eliminate 80% of the cases of visual impairment that are currently preventable.