Next Article in Journal
Impact of Liver Fibrosis on Survival of Patients with Intrahepatic Cholangiocarcinoma Receiving Gemcitabine-Based Chemotherapy
Next Article in Special Issue
Comparison between Suspected and Confirmed COVID-19 Respiratory Patients: What Is beyond the PCR Test
Previous Article in Journal
Is the JAK-STAT Signaling Pathway Involved in the Pathogenesis of Depression?
Previous Article in Special Issue
Does the Serum Concentration of Angiotensin II Type 1 Receptor Have an Effect on the Severity of COVID-19? A Prospective Preliminary Observational Study among Healthcare Professionals
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
JCM
Volume 11
Issue 7
10.3390/jcm11072054
jcm-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Systematic Review

Artificial Intelligence for COVID-19 Detection in Medical Imaging—Diagnostic Measures and Wasting—A Systematic Umbrella Review

by
Paweł Jemioło
1,*,
Dawid Storman
2 and
Patryk Orzechowski
1,3
1
AGH University of Science and Technology, Faculty of Electrical Engineering, Automatics, Computer Science and Biomedical Engineering, al. A. Mickiewicza 30, 30-059 Krakow, Poland
2
Chair of Epidemiology and Preventive Medicine, Department of Hygiene and Dietetics, Jagiellonian University Medical College, ul. M. Kopernika 7, 31-034 Krakow, Poland
3
Institute for Biomedical Informatics, University of Pennsylvania, 3700 Hamilton Walk, Philadelphia, PA 19104, USA
*
Author to whom correspondence should be addressed.
J. Clin. Med. 2022, 11(7), 2054; https://doi.org/10.3390/jcm11072054
Submission received: 23 February 2022 / Revised: 24 March 2022 / Accepted: 26 March 2022 / Published: 6 April 2022
(This article belongs to the Special Issue Updates in Management of SARS-CoV-2 Infection)
Browse Figures
Versions Notes

Abstract

:
The COVID-19 pandemic has sparked a barrage of primary research and reviews. We investigated the publishing process, time and resource wasting, and assessed the methodological quality of the reviews on artificial intelligence techniques to diagnose COVID-19 in medical images. We searched nine databases from inception until 1 September 2020. Two independent reviewers did all steps of identification, extraction, and methodological credibility assessment of records. Out of 725 records, 22 reviews analysing 165 primary studies met the inclusion criteria. This review covers 174,277 participants in total, including 19,170 diagnosed with COVID-19. The methodological credibility of all eligible studies was rated as critically low: 95% of papers had significant flaws in reporting quality. On average, 7.24 (range: 0–45) new papers were included in each subsequent review, and 14% of studies did not include any new paper into consideration. Almost three-quarters of the studies included less than 10% of available studies. More than half of the reviews did not comment on the previously published reviews at all. Much wasting time and resources could be avoided if referring to previous reviews and following methodological guidelines. Such information chaos is alarming. It is high time to draw conclusions from what we experienced and prepare for future pandemics.

1. Introduction

In early December, 2019, a new coronavirus epidemic was identified in Wuhan [1]. Coronavirus disease 2019 (COVID-19) is a viral infection spread by direct contact with people experiencing the illness (from droplets generated by sneezing and coughing) or indirectly [2]. It is caused by Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2). As of 23 February 2022, over 420 million people have been diagnosed with COVID-19, with nearly 5.89 million associated deaths [3]. The consecutive waves of COVID-19 affected many societies, as well as scientific foundations and organisations [4,5,6,7]. On 30 January 2020, the World Health Organisation (WHO) issued a public health emergency of international concern (PHEIC) associated with COVID-19 and declared the state of a pandemic on 11 March 2020 [8].
Disease manifestation is variable, with some infected people remaining asymptomatic (even up to 57% [9]) and others suffering from mild (including fever, cough, and aches) to severe (involving lethargy with dyspnoea and increased respiratory rate) and critical manifestations (requiring mechanical ventilation). It may lead to serious neurological, musculoskeletal, or cerebrovascular disorders or may even progress to a life-threatening respiratory syndrome in some patients [10,11].
Moreover, in 80% of patients, COVID-19 may leave one or more long-lasting symptoms, with fatigue, headaches, attentional difficulties, anosmia, and memory loss manifesting the most frequently [12]. Wide-ranging longer-term morbidity has also been described in the absence of severe initial illnesses [13].
The essence of stopping the significant increase in morbidity is, in addition to treatment, quick diagnostics. The identification of those infected allows for better management of the pandemic (e.g., isolation, quarantine, hospital admission or admission to the intensive care unit) [14]. Understanding the accuracy of tests and diagnostic features seems essential to develop effective screening and management methods [15].
As the pandemic unfolded, many ways have been found to diagnose COVID-19. The primary method for diagnosing COVID-19 is Nucleic Acid Amplification Tests (NAATs). It utilises respiratory tract samples (mainly from the nasopharynx or oropharynx). However, some guidelines recommend nasal swabs [16], and some evidence suggests lower respiratory samples, such as sputum, may have higher sensitivity [17].
From the pandemic onset, chest radiography (X-ray) has been a helpful tool for COVID-19 diagnosis [18]. Nevertheless, even routine chest radiography does not confirm that the patient has COVID-19, especially early on [19], so diagnosing based on an X-ray is challenging. On the contrary, a computed tomography (CT) has been able to discover COVID-19 abnormalities with sensitivity exceeding 97% [20]. However, it was reported to have only 25% to 83% specificity for symptomatic patients [21]. Some evidence suggests it helps to detect COVID-19 earlier than manifested by the positive reverse transcription-polymerase chain reaction (RT-PCR) test [22,23]. Additionally, from the beginning of the pandemic, COVID-19 diagnosis based on ultrasound imaging proved to be of sensitivity and accuracy, which is similar to differentiating with CTs [24].
With the rising role of medical imaging as a diagnostic tool for COVID-19, a question arose if and to which extent automated tools could be included in clinical diagnosis. Up to this day, artificial intelligence (AI), or more specifically, deep learning (DL), have started to play an increasingly vital role in medicine [25]. AI can be employed in the first step of diagnosis, or the results it produces may be used to confirm hypotheses generated by clinicians. In some recent studies and clinical trials, AI has been demonstrated to match or even exceed the performance of expert radiologists, which could potentially offer expedited and less expensive diagnostics [26,27,28,29,30,31]. A study and meta-analysis by [32] with 31,587 identified and 82 included studies shows DL is even capable of slightly outperforming health care professionals in detecting diseases from medical images with a pooled sensitivity of 87% (vs. 86%) and a pooled specificity of 93% (vs. 91%), respectively.
Since the emergence of the COVID-19 pandemic, around 237,000 related papers (and growing) have been published [33,34]. The urgency of reporting novel findings and high pressure to publish COVID-19-related research quickly has been reported to lead to exceptions to high standards of quality [35,36], an increase in overlap [37], lowering methodological credibility of some of the articles [38], or even accepting papers with numerous analytical errors [39].
Almost two years after the pandemic onset, it is the right time to start drawing conclusions [40]. We should also pay attention to the mistakes we have committed and avoid them in the face of the upcoming threats. The current situation is an opportunity to learn lessons on dealing with crises.
This systematic umbrella review aims to screen reviews on AI techniques to diagnose COVID-19 in patients of any age and sex (both hospitalised and ambulatory) using medical images and assess their methodological quality. Additionally, our goal was to evaluate the research publishing process and the degree of overlap to assess the legitimacy of creating new works in the unfolding pandemic.

2. Materials and Methods

2.1. Data Sources and Searches

In order to determine whether there are any eligible papers, we conducted a pre-search in the middle of August 2020 via Google Scholar by browsing. Next, we searched seven article databases (MEDLINE, EMBASE, Web of Science, Scopus, dblp, Cochrane Library, IEEE Xplore) and two preprint databases (arXiv, OSF Preprints) from inception to 1 September 2020 using predefined search strategies. In developing the search strategy for MEDLINE, we combined the Medical Subject Headings (MeSH) and full-text words. In Text S1, we present the used strategies. No date or language restrictions were adopted. Additionally, we searched the references of included studies for eligible records.

2.2. Study Selection

We focused on any review (systematic or not) that includes primary studies utilising AI methods with medical imaging results to diagnose COVID-19. We were particularly interested in the performance of such classification systems, e.g., accuracy, sensitivity, specificity. Based on available guidelines [16], we excluded these primary studies that used reference standards other than assay types (NAATs, antigen tests, and antibody tests) from nasopharyngeal or oropharyngeal swab samples, nasal aspirate, nasal wash or saliva, sputum or tracheal aspirate, or bronchoalveolar lavage (BAL) [41].
Additionally, due to overlapping and double referencing of the post-conference articles (particular chapters), we excluded entire proceedings and post-conference books as they contain little information about the topics (presented in chapters) per se. However, we did not exclude reviews (chapters) as they were still present in our search.
The protocol of this review was published [42] and registered [43] on the OSF platform.
Using Endnote X8 (Clarivate Analytics ®) and Rayyan [44], we checked identified references for duplicates. P.J., D.S., and P.O. independently screened the remaining references using the latter application, and subsequently, independently assessed the full texts for meeting the inclusion criteria.
To improve the understanding of the criteria among the reviewers, we carried out pilot exercises before the screening of titles and abstracts and full texts assessment. We achieved consensus via discussion if any conflicts occurred.

2.3. Definitions

We defined the terms used in our eligibility criteria below. Review refers to a paper identified by authors as a review or a survey. AI refers to computer programs that can perform tasks as intelligent beings [45]. COVID-19 refers to a disease caused by the SARS-CoV-2 virus [46]. Imaging refers to individuals’ medical imaging results (e.g., CT scans, X-rays, ultrasound images) [47,48].
Diagnosis refers to the identification of an illness (here: COVID-19) [49]. Performance metrics refers to evaluating machine learning algorithms. These measures are utilised to juxtapose observed data (actual labels) with the predictions of the model [50].

2.4. Data Extraction and Quality Assessment

Before the extraction phase, we checked included preprints for peer-reviewed versions and included them, if available. We predefined an extraction form, and P.J. and D.S. collected all necessary data independently. We gathered information about authors, funding, population, models, outcomes—AI diagnostic metrics, and additional analyses.
We also extracted bibliometric data about publishing dates (availability), sending to the editors (first and last version), and acceptance in a journal or a conference of included reviews. Moreover, we checked the availability dates for primary studies.
To provide a common understanding of the criteria, we performed calibration exercises before data extraction and credibility assessment. When the conflict occurred, we discussed the final version.
P.J. and D.S. conducted quality evaluations independently. We assessed the methodological credibility using AMSTAR 2 [51] with critical items (2, 4, 7, 9, 11, 13, and 15), indicated as such by AMSTAR 2 authors, and not yet validated extended version of QASR [52].
The general quality across the study was evaluated as critically low when more than one item in a critical domain was considered a flaw [51].
In this paper, we concentrate only on the results of applying AMSTAR 2 (as it is suggested for evaluating systematic reviews [53]), while a full assessment of both instruments will be included in the next methodologically focused article.
Additionally, we assessed the quality of reporting in included studies using the Preferred Reporting Items for Systematic Reviews and Meta-analyses for Diagnostic Test Accuracy (PRISMA-DTA) checklist [54]. We rated each module on the 3-item scale: 0 (no with no compliance), 0.5 (partial yes with fragmentary compliance), 1 (yes with total compliance). Next, the results were summed, and the overall score was then assigned.
Based on the method of Li et al. [55] and taking into account two more items in the DTA extension [54] (comparing to the original instrument [56]), we differentiated the quality of reporting as follows:
  • Major flaws when the final score was ≤ 17.0 ,
  • Minor flaws when the final score was ≥ 17.5 and ≤ 23.0 ,
  • Minimal flaws when the final score was ≥ 23.5 .
In the case of reviews without meta-analysis, we lowered the cut-offs by 1 point following PRISMA-DTA [54].

2.5. Data Synthesis and Analysis

In this umbrella review, we focus on the descriptive summary of included papers regarding the quality and reporting on the most significant characteristics, such as population, models, interpretability, and outcomes.
We did not synthesise the results quantitatively because of the quality of included reviews, the agreement between them, and the percentage of non-reported data (data we intended to extract, e.g., accuracy of diagnostic methods or AI model type, see Section 3). Therefore, we do not present a subgroup analysis and investigation of heterogeneity, sensitivity, and publication bias analyses.
As for the in-depth characteristics, all the primary studies were divided into two groups: included in one review only and included in at least one review. The studies included in more than one review were analysed in 2 ways (A and B) considering not reported data. In analysis A, whenever not reported data from one review occurred together with a specific value from the other paper, we considered it non-overlapping and excluded it. In analysis B, we ignored not reported data and included the specific value. After the exclusion of non-overlapping data for continuous variables, we calculated the statistics, namely means with ranges.
For diagnostics metrics, we prepared the scatter plots. We considered data regardless of non-overlapping. Whenever disagreements between reviews occurred (in specific primary studies), we averaged the values. In case of lack of data (not reported), we provided two charts with modal imputation and without it.
From the above analyses, we excluded those primary studies, which included more than one DL model.
We analysed how extensive was the search performed by the authors of the reviews, i.e., percentage of the identified primary studies available up to the selected date. In the first case, we considered the reference date, by which we mean the day that the review was either received, accepted by the editors, or published. In the second analysis, we relaxed this condition to the date the last cited paper included in the review was available.
Investigating the citations between the reviews, we considered two different scenarios: citing only published reviews and citing both published and preprint versions.
We assessed inter-review agreement only if at least two different reviews included the same paper (and only one DL model). We determined the inter-review agreement as a percentage of overlapping values within all extracted data (text and non-text) and subgroups of characteristics (text and non-text) and outcomes. The text variables considered the dataset used, architecture, and post-processing. The inter-review agreement was assessed in 2 ways (analyses A and modified B—exclude a pair, instead of ignoring it).
All analyses were conducted using Python 3.7.10 (including libraries: Matplotlib 3.2.2, Seaborn 0.11.1, Pandas 1.1.5 and NumPy 1.19.5).

3. Results

3.1. Included Studies

After removing duplicates, we screened 725 studies, of which 33 were read in the form of full texts. In total, we included 22 reviews [57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78] for qualitative synthesis. We followed PRISMA guidelines [56]. The full study flow is presented in Figure 1.
The included and excluded studies (with reasons) are presented in Tables S1 and S2, respectively. The detailed characteristics of included reviews are shown in Table 1, while its extended version is ensured in Table S3.
We present the in-depth characteristics of primary studies in Table S4. Tables S5 and S6 focus on the non-overlapping between reviews and non-reporting in terms of specific extracted variables. Figures S1–S6 introduces visualised diagnostic metrics.
None of the reviews provided information about ethnicity, smoking, and comorbidities. Only one review [68] reported on age and gender proportion as well as the study design of discussed primary papers. None of the studies conducted a meta-analysis, but one summarised the results with the use of averages [71].
The analysed reviews described 165 primary papers (on average: eight primary studies per review, range: 1–11). Of these, 138 of them were included in at least one review, of which 73 were included only in one. Only 27 of the primary studies considered more than one DL model for diagnosis.

3.2. Quality of Included Studies

The general quality of all included studies is critically low (see Figure 2). Six reviews [62,63,64,66,68,77] provided full information about sources of funding and conflict of interest. It was the most satisfied item. None of the studies provided a list of excluded papers, explanation of eligible study design, and sources of funding in included studies.
A heatmap with the authors’ judgements regarding AMSTAR 2 items can be found in Figure S7. In Figure S8, we also included results per specific review.
In terms of reporting, major flaws are present among 21 of the included papers. Only one review [68] contains minor flaws (see Figure 3 and Figures S9 and S10). The most affected domains were those concerning additional analyses both in terms of methods and results (all reviews). Similarly, a summary of evidence was not reported in any review.
On the contrary, 12 of the included papers [57,59,62,63,64,66,68,69,70,71,77,78] reported fully on funding. Additionally, 11 reviews [57,58,60,65,66,69,70,72,73,76,77] and eight reviews [57,59,60,65,66,67,71,77] described the rationale and objectives in the introduction adequately. These were the most satisfied domains.
The mean overall score of reporting quality across the included reviews equals 6.23 (1.5–17.5). Across all items and studies, the most frequent score was 0 (no) with 67%; 19% of the time, we assessed the items as 0.5 (partially yes). A heatmap with all authors’ judgements regarding PRISMA-DTA items can be found in Figure S9. In Figure S10, we also included summarised results per specific review.

3.3. Resources and Time Wasting Analyses

The included studies were published or available online without peer reviewing from 11 April 2020 to 12 October 2020 (see Figure S11). In Figure 4, we presented a cumulative chart of all 165 primary studies included in the discussed reviews.
The number of included interesting studies (related to our research question) in selected reviews ranged from 1 [62,63,64] to 106 [73]. Moreover, we present the percentage of articles introduced by (first appear in) a particular review. Figure S11 additionally depicts the appearance of included reviews and interesting primary studies over time.
Half (50%) of all (165) primary studies (the half-saturation constant) were included at least once before the end of July 2020. However, the same number of papers was available for inclusion three months earlier.
Next, we investigated the extent to which review authors performed the search. Regarding the reference date (see Figure S11), the mean percentage of the primary studies covered was 14% (1–64). When relaxed, the mean percentage of covered studies increased to 24% (1–65). More details about the search are presented in Table S7.
Out of all the studies, 14% did not include any new paper into consideration. The mean primary studies that were introduced by a particular review was 7.24 (0–45).
Analysing published versions of reviews only, the cross-citing equals 0.81 (0–4). Including also preprints, 1.1 (0–7) published papers or preprints were quoted by the authors of the subsequent reviews. Notably, 12 (55%) of reviews did not refer to any previously available ones at all.
Figures S12–S21 present results regarding the agreement between reviews (pairs of reviews) in the reporting of characteristics and outcomes.

4. Discussion

Generally, we report that the quality of the included reviews was critically low. Similar findings were found by Jung et al. [79], who observed lowered methodological credibility in 686 of 14,787 screened COVID-19 papers. Analyses of reviews on COVID-19 by Yu et al. [80] and Al-Ryalat et al. [81] also showed their unsatisfactory credibility. It adds on top of the generally low quality of reporting of DL performance from medical images, with a high risk of bias present in 58 out of 81 of the existing studies (72%) [82].
Poor quality is not related only to COVID-19 and AI. Still, it occurs in many fields such as bariatric surgery with up to 99% critically low articles [83], psychology with 95% of papers [84], or methodology where 53 out of 63 publications were of critically low quality [85].
What is more, we also noticed major flaws in reporting. Nagendran et al. [82] observed the same, but they used the original PRISMA instrument. In our research, three PRISMA-DTA domains were fully violated by all reviews. Although the authors focused on diagnostics, they poorly reported on accuracy measures and explicit description of the extraction process. None of the included studies performed a meta-analysis, similar to what Adadi et al. [86] found in their study.
The low credibility of evidence and flawed reporting (e.g., population characteristics) can be associated with a lack of knowledge of reporting standards and clinical practice or misunderstandings regarding AI methods and additional analyses.
We also observed multiple disagreements between the included reviews. However, excluding them from synthesis is associated with a vast information loss, e.g., the number of participants. Inconsistencies were also noticed in the reporting of DL architecture. For instance, the following names were used across multiple studies: ResNet-18, ResNet18, resnet-18, 18-layer ResNet. In some of the papers, the architecture was not reported at all. It made it challenging to group models into similar subsets. Some discrepancy was also observed in extracting the measurements of AI models performance, e.g., diagnostic effectiveness metrics. Such negligence may lead to further replicating the errors by subsequent studies and should be corrected before releasing the paper or soon after in an updated version or in an associated erratum.
Many of the reviews included in this paper did not strengthen the evidence on using AI in diagnosing COVID-19 from medical imaging. Those works have not identified and correctly cited pre-existing primary papers, which is deemed the essence of any research. Some of the potential explanations are that multiple similar studies might have been initiated around the same time, and prolonged review times impacted their content. Alternatively, the research objectives of some articles were broad enough to preclude a deeper analysis of the use of artificial intelligence in medical imaging.
Wasting may also (or in particular) be observed on the primary studies level. Failure to consider previous results may lead to publishing new papers describing models similar to those presented by other researchers. Shockingly, these newer DL architectures rely sometimes on fewer participants or COVID-19 cases, so they probably reflect reality less adequately. As researchers suggest, the amount of waste and poor biomedical research quality is staggering [35,87]. Papers that do not bring any additional evidence to the field can be considered redundant [88,89,90,91].
Proper reporting of deep learning performance from primary studies is challenging. Naudé [92] has pointed out some of the significant concerns regarding the adoption of AI in COVID-19 research, including data availability and its quality. Still, many studies do not ensure that the utilised code is open-source, which highly limits the reproducibility of their findings [93,94]. Therefore, we suggest sharing it so that you can react faster and more effectively from the perspective of the upcoming, similar breakdowns.
On average, when not considering credibility issues, the diagnostic metrics of described models exceed the human’s ability to diagnose COVID-19 from medical imaging [32]. Sadly, no evidence of such an advantage could be transferred to any implications for practice because of not following reporting and quality instruments. It resembles a situation when a long jumper did not break the world record just because they stepped on the foul line.

Study Strengths and Limitations

Our umbrella review has the following strengths. First, the search strategy was comprehensive. It is based on adequate inclusion criteria related to the research question and spanned across a wide selection of existing data sources: papers and preprints. This selection was further expanded by searching the references of included papers to identify additional works. It is noteworthy that the searches were not limited in terms of format or language (we imposed no restrictions). The process of our review was rigorous as the study was preceded by the publication of protocol. We used the most up-to-date and applicable instruments to assess the credibility and quality of reporting—AMSTAR 2 and PRISMA with extension for DTA, respectively.
Nevertheless, these two have been designed for reviews in medicine and health sciences, where the formulation of the research question is structured, the methodology is validated, and the quantitative synthesis of results is popular.
It must be noted, though, that the vast majority of the included studies focused on a broader context than purely diagnosing COVID-19 from medical images.
In this study, we have also investigated wasting among the reviews. We based on the date of publishing of the last included primary study in a specific review. By doing so, we aimed at assessing the depth of the search performed by the authors. We assumed that if the authors included a given study, they should have had the required knowledge about all the papers available before it was released. This approach relaxes the strict requirement to include all the studies that appeared before the review was published and seems to measure the quality of the search more objectively.
The level of agreement between reviews differs remarkably depending on the extracted variable, so without comparing with the primary studies, we cannot be fully convinced of the data correctness reported in the reviews. In the assessment of the inter-review agreement, we considered only these reviews that included the same paper (and described one DL model).

5. Conclusions

The COVID-19 research is quickly moving forward. Each day hundreds of new papers are published [95,96,97]. As AI starts to play an increasingly important role in clinical practice [98,99], it is crucial to evaluate its performance correctly.
In this paper, we synthesised and assessed the quality of the 22 reviews that mention using AI on COVID-19 medical images. We reviewed them and critically assessed their reporting and credibility using well-established instruments such as PRISMA-DTA [54] and AMSTAR 2 [51].
We explored the beginning of the pandemic when much uncertainty and confusion existed in the world of science. It seems that the number of articles and the pace of their publishing during the future global outbreaks might be even faster. Thus, it is essential to draw the appropriate conclusions now and treat this briefing as an opportunity to optimise work and avoid wasting in publishing.
In order to accomplish this, we urge the authors of the reviews to use PRISMA [56] and AMSTAR 2 [51] and the authors of primary studies to follow appropriate tools [100,101].
It is high time to adopt best practices, improve the research quality, and apply higher scrutiny in filtering out non-constructive contributions.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/jcm11072054/s1, Table S1: Included studies with dates: first received, last received, accepted and published; Table S2: Exluded studies with reasons; Table S3: Full characteristics of included reviews [102,103,104,105,106,107,108,109,110,111,112]; Table S4: In-depth characteristics of primary studies; Table S5: Non-overlapping of in-depth characteristics variables; Table S6: Non-reporting of in-depth characteristics variables; Figure S1: CT-based COVID-19 diagnosis, without imputation; Figure S2: CT-based COVID-19 diagnosis, with modal imputation; Figure S3: X-Ray-based COVID-19 diagnosis, without imputation; Figure S4: X-Ray-based COVID-19 diagnosis, with modal imputation; Figure S5: COVID-19 diagnosis with full patients number data provided, without imputation; Figure S6: COVID-19 diagnosis with any patients number data provided, with modal imputation; Figure S7: Review authors’ judgements about each AMSTAR 2 item across all included studies; * denotes critical items; Figure S8: AMSTAR 2 score in each included review; Figure S9: Review authors’ judgements about each PRISMA-DTA item across all included studies; Figure S10: PRISMA-DTA score in each included review; Figure S11: Appearance of included reviews (vertical lines, reference date, see Table S1 and interesting primary studies (dots); Table S7: Time and resource wasting statistics; Figure S12: Level of agreement based on overlapping in extracted data (characteristics only, without text data) between included reviews; analysis A; Figure S13: Level of agreement based on overlapping in extracted data (characteristics only) between included reviews; analysis A; Figure S14: Level of agreement based on overlapping in extracted data (outcomes only) between included reviews; analysis A; Figure S15: Level of agreement based on overlapping in extracted data (all variables, without text data) between included reviews; analysis A; Figure S16: Level of agreement based on overlapping in extracted data (all variables) between included reviews; analysis A; Figure S17: Level of agreement based on overlapping in extracted data (characteristics only, without text data) between included reviews; analysis B; Figure S18: Level of agreement based on overlapping in extracted data (characteristics only) between included reviews; analysis B; Figure S19: Level of agreement based on overlapping in extracted data (outcomes only) between included reviews; analysis B; Figure S20: Level of agreement based on overlapping in extracted data (all variables, without text data) between included reviews; analysis B; Figure S21: Level of agreement based on overlapping in extracted data (all variables) between included reviews; analysis B; Text S1: Search Strategies; PRISMA 2020 Checklist.

Author Contributions

Conceptualization, P.J., D.S., and P.O.; methodology, D.S. and P.J.; software, P.J. and P.O.; validation, P.J. and D.S.; formal analysis, P.J. and D.S.; investigation, P.J., D.S. and P.O.; resources, P.J.; data curation, P.J.; writing—original draft preparation, P.J., D.S. and P.O.; writing—review and editing, P.J., D.S. and P.O.; visualization, P.J. and P.O.; supervision, P.J.; project administration, P.J.; funding acquisition, P.J. All authors have read and agreed to the published version of the manuscript.

Funding

Research project supported by program Excellence initiative— research university for the University of Science and Technology. P.O. was supported by National Institutes of Health (grant number AI116794).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Code used in this research is available on GitHub: https://github.com/pawljmlo/covid-ur-wasting (accessed on 25 March 2022). The data presented in this study are available on request from the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Abbreviations

The following abbreviations are used in this manuscript:
AIartificial intelligence
BALbronchoalveolar lavage
COVID-19Coronavirus disease 2019
CTcomputed tomography
DLdeep learning
NAATsNucleic Acid Amplification Tests
PHEICpublic health emergency of international concern
X-rayradiography
RT-PCRreverse transcription-polymerase chain reaction
SARS-CoV-2Severe Acute Respiratory Syndrome Coronavirus 2
WHOWorld Health Organisation

References

  1. Kahn, N. New virus Discovered by Chinese Scientists Investigating Pneumonia Outbreak 2020. Available online: https://www.wsj.com/articles/new-virus-discovered-by-chinese-scientists-investigating-pneumonia-outbreak-11578485668 (accessed on 23 February 2022).
  2. World Health Organization. Report of the WHO-China Joint Mission on Coronavirus Disease 2019 (COVID-19). 2020. Available online: https://reliefweb.int/report/china/report-who-china-joint-mission-coronavirus-disease-2019-covid-19?gclid=EAIaIQobChMI2vX_nJro9gIVVj5gCh2LDQKuEAAYASAAEgLn9PD_BwE (accessed on 23 February 2022).
  3. World Health Organization. WHO Coronavirus (COVID-19) Dashboard. 2020. Available online: https://covid19.who.int/ (accessed on 23 February 2022).
  4. Simon, S.; Frank, B.J.; Aichmair, A.; Manolopoulos, P.P.; Dominkus, M.; Schernhammer, E.S.; Hofstaetter, J.G. Impact of the 1st and 2nd Wave of the COVID-19 Pandemic on Primary or Revision Total Hip and Knee Arthroplasty—A Cross-Sectional Single Center Study. J. Clin. Med. 2021, 10, 1260. [Google Scholar] [CrossRef] [PubMed]
  5. Vahabi, N.; Salehi, M.; Duarte, J.D.; Mollalo, A.; Michailidis, G. County-level longitudinal clustering of COVID-19 mortality to incidence ratio in the United States. Sci. Rep. 2021, 11, 1–22. [Google Scholar] [CrossRef] [PubMed]
  6. Saito, S.; Asai, Y.; Matsunaga, N.; Hayakawa, K.; Terada, M.; Ohtsu, H.; Tsuzuki, S.; Ohmagari, N. First and second COVID-19 waves in Japan: A comparison of disease severity and characteristics: Comparison of the two COVID-19 waves in Japan. J. Infect. Dis. 2020, 82, 84–123. [Google Scholar]
  7. Coccia, M. The Effects of the First and Second Wave of COVID-19 Pandemic on Public Health. 2020. Available online: https://www.researchsquare.com/article/rs-110013/latest.pdf (accessed on 23 February 2022).
  8. World Health Organization. World Health Organization coronavirus disease 2019 (COVID-19) Situation Report. 2020. Available online: https://apps.who.int/iris/handle/10665/331686 (accessed on 23 February 2022).
  9. Kimball, A.; Hatfield, K.M.; Arons, M.; James, A.; Taylor, J.; Spicer, K.; Bardossy, A.C.; Oakley, L.P.; Tanwar, S.; Chisty, Z.; et al. Asymptomatic and presymptomatic SARS-CoV-2 infections in residents of a long-term care skilled nursing facility—King County, Washington, March 2020. MMWR 2020, 69, 377. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  10. Guan, W.j.; Ni, Z.y.; Hu, Y.; Liang, W.h.; Ou, C.q.; He, J.x.; Liu, L.; Shan, H.; Lei, C.l.; Hui, D.S.; et al. Clinical characteristics of coronavirus disease 2019 in China. N. Engl. J. Med. 2020, 382, 1708–1720. [Google Scholar] [CrossRef] [PubMed]
  11. Le, T.T.; Gutiérrez-Sacristán, A.; Son, J.; Hong, C.; South, A.M.; Beaulieu-Jones, B.K.; Loh, N.H.W.; Luo, Y.; Morris, M.; Ngiam, K.Y.; et al. Multinational Prevalence of Neurological Phenotypes in Patients Hospitalized with COVID-19. medRxiv 2021. [Google Scholar] [CrossRef]
  12. Lopez-Leon, S.; Wegman-Ostrosky, T.; Perelman, C.; Sepulveda, R.; Rebolledo, P.A.; Cuapio, A.; Villapol, S. More than 50 Long-term effects of COVID-19: Asystematic review and meta-analysis. SSRN 2021, 11, 3769978. [Google Scholar]
  13. Greenhalgh, T.; Knight, M.; Buxton, M.; Husain, L. Management of post-acute covid-19 in primary care. BMJ 2020, 370, m3026. [Google Scholar] [CrossRef]
  14. Sun, Q.; Qiu, H.; Huang, M.; Yang, Y. Lower mortality of COVID-19 by early recognition and intervention: Experience from Jiangsu Province. Ann. Intensive Care 2020, 10, 1–4. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  15. Islam, N.; Salameh, J.P.; Leeflang, M.M.; Hooft, L.; McGrath, T.A.; Pol, C.B.; Frank, R.A.; Kazi, S.; Prager, R.; Hare, S.S.; et al. Thoracic Imaging Tests for the Diagnosis of COVID-19. Cochrane Database Syst. Rev. 2021, CD013639. [Google Scholar] [CrossRef]
  16. Centers for Disease Control and Prevention. Interim Guidelines for Collecting and Handling of Clinical Specimens for COVID-19 Testing. 2020. Available online: https:/cdc.gov/coronavirus/2019-nCoV/lab/guidelines-clinical-specimens.html/ (accessed on 23 February 2022).
  17. Wang, W.; Xu, Y.; Gao, R.; Lu, R.; Han, K.; Wu, G.; Tan, W. Detection of SARS-CoV-2 in different types of clinical specimens. JAMA 2020, 323, 1843–1844. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  18. Smith, D.L.; Grenier, J.P.; Batte, C.; Spieler, B. A Characteristic Chest Radiographic Pattern in the Setting of COVID-19 Pandemic. J. Thorac Imaging 2020, 2, e200280. [Google Scholar] [CrossRef] [PubMed]
  19. Cleverley, J.; Piper, J.; Jones, M.M. The role of chest radiography in confirming covid-19 pneumonia. BMJ 2020, 370. [Google Scholar] [CrossRef] [PubMed]
  20. Kovács, A.; Palásti, P.; Veréb, D.; Bozsik, B.; Palkó, A.; Kincses, Z.T. The sensitivity and specificity of chest CT in the diagnosis of COVID-19. Eur. Radiol. 2020, 31, 1–6. [Google Scholar] [CrossRef] [PubMed]
  21. Park, J.Y.; Freer, R.; Stevens, R.; Neil, S.; Jones, N. The Accuracy of Chest CT in the Diagnosis of COVID-19: An Umbrella Review. 2021. Available online: https://www.cebm.net/covid-19/the-accuracy-of-chest-ct-in-the-diagnosis-of-covid-19-an-umbrella-review/ (accessed on 23 February 2022).
  22. Chua, F.; Armstrong-James, D.; Desai, S.R.; Barnett, J.; Kouranos, V.; Kon, O.M.; José, R.; Vancheeswaran, R.; Loebinger, M.R.; Wong, J.; et al. The role of CT in case ascertainment and management of COVID-19 pneumonia in the UK: Insights from high-incidence regions. Lancet Respir Med. 2020, 8, 438–440. [Google Scholar] [CrossRef] [Green Version]
  23. Ai, T.; Yang, Z.; Hou, H.; Zhan, C.; Chen, C.; Lv, W.; Tao, Q.; Sun, Z.; Xia, L. Correlation of chest CT and RT-PCR testing for coronavirus disease 2019 (COVID-19) in China: A report of 1014 cases. Radiology 2020, 296, E32–E40. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  24. Sultan, L.R.; Sehgal, C.M. A review of early experience in lung ultrasound in the diagnosis and management of COVID-19. Ultrasound Med. Biol. 2020, 46, 2530–2545. [Google Scholar] [CrossRef]
  25. Ching, T.; Himmelstein, D.S.; Beaulieu-Jones, B.K.; Kalinin, A.A.; Do, B.T.; Way, G.P.; Ferrero, E.; Agapow, P.M.; Zietz, M.; Hoffman, M.M.; et al. Opportunities and obstacles for deep learning in biology and medicine. J. R. Soc. Interface 2018, 15, 20170387. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  26. Hosny, A.; Parmar, C.; Quackenbush, J.; Schwartz, L.H.; Aerts, H.J. Artificial intelligence in radiology. Nat. Rev. Cancer 2018, 18, 500–510. [Google Scholar] [CrossRef] [PubMed]
  27. McBee, M.P.; Awan, O.A.; Colucci, A.T.; Ghobadi, C.W.; Kadom, N.; Kansagra, A.P.; Tridandapani, S.; Auffermann, W.F. Deep Learning in Radiology. Acad. Radiol. 2018, 25, 1472–1480. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  28. Wang, P.; Berzin, T.M.; Brown, J.R.G.; Bharadwaj, S.; Becq, A.; Xiao, X.; Liu, P.; Li, L.; Song, Y.; Zhang, D.; et al. Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: A prospective randomised controlled study. Gut 2019, 68, 1813–1819. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  29. Liu, X.; Faes, L.; Kale, A.U.; Wagner, S.K.; Fu, D.J.; Bruynseels, A.; Mahendiran, T.; Moraes, G.; Shamdas, M.; Kern, C.; et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: A systematic review and meta-analysis. Lancet Digit Health 2019, 1, e271–e297. [Google Scholar] [CrossRef]
  30. Loo, J.; Clemons, T.E.; Chew, E.Y.; Friedlander, M.; Jaffe, G.J.; Farsiu, S. Beyond performance metrics: Automatic deep learning retinal OCT analysis reproduces clinical trial outcome. Ophthalmology 2020, 127, 793–801. [Google Scholar] [CrossRef] [PubMed]
  31. Bullock, J.; Luccioni, A.; Pham, K.H.; Lam, C.S.N.; Luengo-Oroz, M. Mapping the landscape of artificial intelligence applications against COVID-19. J. Artif. Intell. Res. 2020, 69, 807–845. [Google Scholar] [CrossRef]
  32. Li, Y.; Cao, L.; Zhang, Z.; Hou, L.; Qin, Y.; Hui, X.; Li, J.; Zhao, H.; Cui, G.; Cui, X.; et al. Reporting and methodological quality of COVID-19 systematic reviews needs to be improved: An evidence mapping. J. Clin. Epidemiol. 2021, 135, 17–28. [Google Scholar] [CrossRef] [PubMed]
  33. Chen, Q.; Allot, A.; Lu, Z. Keep up with the latest coronavirus research. Nature 2020, 579, 193. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  34. National Institutes of Health. COVID-19 Portfolio. 2020. Available online: https://icite.od.nih.gov/covid19/search/ (accessed on 23 February 2022).
  35. Glasziou, P.P.; Sanders, S.; Hoffmann, T. Waste in COVID-19 Research. BMJ 2020, 369, m1847. [Google Scholar]
  36. London, A.J.; Kimmelman, J. Against pandemic research exceptionalism. Science 2020, 368, 476–477. [Google Scholar] [CrossRef] [Green Version]
  37. Quinn, T.J.; Burton, J.K.; Carter, B.; Cooper, N.; Dwan, K.; Field, R.; Freeman, S.C.; Geue, C.; Hsieh, P.H.; McGill, K.; et al. Following the science? Comparison of methodological and reporting quality of covid-19 and other research from the first wave of the pandemic. BMC Med. 2021, 19, 1–10. [Google Scholar] [CrossRef] [PubMed]
  38. Mahase, E. Covid-19: 146 researchers raise concerns over chloroquine study that halted WHO trial. BMJ 2020, 369, 2197. [Google Scholar] [CrossRef] [PubMed]
  39. Ioannidis, J.P. Coronavirus Disease 2019: The Harms of Exaggerated Information and Non-Evidence-Based Measures. Eur. J. Clin. Invest. 2020, 50, e13222. [Google Scholar] [CrossRef] [PubMed]
  40. Osterholm, M.T. Preparing for the Next Pandemic. N. Engl. J. Med. 2005, 352, 1839–1842. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  41. The Coronaviridae Study Group of the International Committee on Taxonomy of Viruses. The species Severe acute respiratory syndrome-related coronavirus: Classifying 2019-nCoV and naming it SARS-CoV-2. Nat. Microbiol. 2020, 5, 536. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  42. Jemioło, P.; Storman, D.; Moore, J.H.; Orzechowski, P. Diagnosing COVID-19 from Medical Images with Artificial Intelligence—An Umbrella Survey. 2020. Available online: https://osf.io/kxrmh/ (accessed on 23 February 2022).
  43. Jemioło, P.; Storman, D.; Moore, J.H.; Orzechowski, P. Diagnosing COVID-19 from Medical Images with Artificial Intelligence—An Umbrella Survey (Registration). 2020. Available online: https://osf.io/hkwfq/ (accessed on 23 February 2022).
  44. Ouzzani, M.; Hammady, H.; Fedorowicz, Z.; Elmagarmid, A. Rayyan—A web and mobile app for systematic reviews. Syst. Rev. 2016, 5, 210. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  45. Copeland, B. Artificial Intelligence: Definition, Examples, and Applications. 2020. Available online: https://www.britannica.com/technology/artificial-intelligence (accessed on 23 February 2022).
  46. Wang, L.; Wang, Y.; Ye, D.; Liu, Q. Review of the 2019 novel coronavirus (SARS-CoV-2) based on current evidence. Int. J. Antimicrob. Agents 2020, 55, 105948. [Google Scholar] [CrossRef] [PubMed]
  47. Leondes, C.T. Medical Imaging Systems Techniques and Applications: Computational Techniques; CRC Press: Boca Raton, FL, USA, 1998; Volume 6. [Google Scholar]
  48. Santosh, K.; Antani, S.; Guru, D.S.; Dey, N. Medical Imaging: Artificial Intelligence, Image Recognition, and Machine Learning Techniques; CRC Press: Boca Raton, FL, USA, 2019. [Google Scholar]
  49. Cambridge Dictionary English Dictionary, Translations & Thesaurus. 2021. Available online: https://dictionary.cambridge.org/ (accessed on 23 February 2022).
  50. Botchkarev, A. Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology. arXiv 2018, arXiv:1809.03006. [Google Scholar]
  51. Shea, B.J.; Reeves, B.C.; Wells, G.; Thuku, M.; Hamel, C.; Moran, J.; Moher, D.; Tugwell, P.; Welch, V.; Kristjansson, E.; et al. AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017, 358, j4008. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  52. Jemioło, P.; Storman, D. Quality Assessment of Systematic Reviews (QASR). 2020. Available online: https://osf.io/dhtw3/ (accessed on 23 February 2022).
  53. Lorenz, R.C.; Matthias, K.; Pieper, D.; Wegewitz, U.; Morche, J.; Nocon, M.; Rissling, O.; Schirm, J.; Jacobs, A. A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool. J. Clin. Epidemiol. 2019, 114, 133–140. [Google Scholar] [CrossRef] [PubMed]
  54. McInnes, M.D.; Moher, D.; Thombs, B.D.; McGrath, T.A.; Bossuyt, P.M.; Clifford, T.; Cohen, J.F.; Deeks, J.J.; Gatsonis, C.; Hooft, L.; et al. Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: The PRISMA-DTA statement. JAMA 2018, 319, 388–396. [Google Scholar] [CrossRef] [PubMed]
  55. Li, J.l.; Ge, L.; Ma, J.c.; Zeng, Q.l.; Yao, L.; An, N.; Ding, J.x.; Gan, Y.h.; Tian, J.h. Quality of reporting of systematic reviews published in “evidence-based” Chinese journals. Syst. Rev. 2014, 3, 1–6. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  56. Page, M.J.; Moher, D.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ 2021, 372, n160. [Google Scholar] [CrossRef] [PubMed]
  57. Shi, F.; Wang, J.; Shi, J.; Wu, Z.; Wang, Q.; Tang, Z.; He, K.; Shi, Y.; Shen, D. Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for COVID-19. IEEE Rev. Biomed. Eng. 2020, 14. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  58. Ilyas, M.; Rehman, H.; Naït-Ali, A. Detection of covid-19 from chest X-ray images using artificial intelligence: An early review. arXiv 2020, arXiv:2004.05436. [Google Scholar]
  59. Dong, D.; Tang, Z.; Wang, S.; Hui, H.; Gong, L.; Lu, Y.; Xue, Z.; Liao, H.; Chen, F.; Yang, F.; et al. The role of imaging in the detection and management of COVID-19: A review. IEEE Rev. Biomed. Eng. 2020, 14. [Google Scholar] [CrossRef] [PubMed]
  60. Ito, R.; Iwano, S.; Naganawa, S. A review on the use of artificial intelligence for medical imaging of the lungs of patients with coronavirus disease 2019. Diagn. Interv. Radiol. 2020, 26, 443. [Google Scholar] [CrossRef] [PubMed]
  61. Kumar, A.; Gupta, P.K.; Srivastava, A. A review of modern technologies for tackling COVID-19 pandemic. Diabetes Metab. Syndr. 2020, 14, 569–573. [Google Scholar] [CrossRef]
  62. Raj, V. Role of Chest Radiograph (CXR) in COVID-19 Diagnosis and Management. J. Indian Med. Assoc. 2020, 118, 14–19. [Google Scholar]
  63. Cui, F.; Zhou, H.S. Diagnostic methods and potential portable biosensors for coronavirus disease 2019. Biosens. Bioelectron. 2020, 165, 112349. [Google Scholar] [CrossRef]
  64. Jalaber, C.; Lapotre, T.; Morcet-Delattre, T.; Ribet, F.; Jouneau, S.; Lederlin, M. Chest CT in COVID-19 pneumonia: A review of current knowledge. Diagn. Interv. Imaging 2020, 101, 431–437. [Google Scholar] [CrossRef]
  65. Salehi, A.W.; Baglat, P.; Gupta, G. Review on machine and deep learning models for the detection and prediction of Coronavirus. Mater. Today Proc. 2020, 33, 3896–3901. [Google Scholar] [CrossRef]
  66. Farhat, H.; Sakr, G.E.; Kilany, R. Deep learning applications in pulmonary medical imaging: Recent updates and insights on COVID-19. Mach. Vis. Appl. 2020, 31, 1–42. [Google Scholar] [CrossRef] [PubMed]
  67. Shaikh, F.; Anderson, M.; Sohail, M.R.; Mulero, F.; Awan, O.; Dupont-Roettger, D.; Kubassova, O.; Dehmsehki, J.; Bisdas, S. Current landscape of imaging and the potential role for artificial intelligence in the management of COVID-19. Curr. Probl. Diagn. Radiol. 2020, 50. [Google Scholar] [CrossRef] [PubMed]
  68. Wynants, L.; Van Calster, B.; Collins, G.S.; Riley, R.D.; Heinze, G.; Schuit, E.; Bonten, M.M.; Dahly, D.L.; Damen, J.A.; Debray, T.P.; et al. Prediction models for diagnosis and prognosis of covid-19: Systematic review and critical appraisal. BMJ 2020, 369, 2204. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  69. Chen, J.; Li, K.; Zhang, Z.; Li, K.; Yu, P.S. A Survey on Applications of Artificial Intelligence in Fighting against COVID-19. arXiv 2020, arXiv:2007.02202. [Google Scholar] [CrossRef]
  70. Pham, Q.V.; Nguyen, D.C.; Huynh-The, T.; Hwang, W.J.; Pathirana, P.N. Artificial Intelligence (AI) and Big Data for Coronavirus (COVID-19) Pandemic: A Survey on the State-of-the-Arts. IEEE Access 2020, 8, 130820–130839. [Google Scholar] [CrossRef] [PubMed]
  71. Chen, D.; Ji, S.; Liu, F.; Li, Z.; Zhou, X. A review of Automated Diagnosis of COVID-19 Based on Scanning Images. arXiv 2020, arXiv:2006.05245. [Google Scholar]
  72. Mohamadou, Y.; Halidou, A.; Kapen, P.T. A review of mathematical modeling, artificial intelligence and datasets used in the study, prediction and management of COVID-19. Appl. Intell. 2020, 50, 3913–3925. [Google Scholar] [CrossRef]
  73. Shoeibi, A.; Khodatars, M.; Alizadehsani, R.; Ghassemi, N.; Jafari, M.; Moridian, P.; Khadem, A.; Sadeghi, D.; Hussain, S.; Zare, A.; et al. Automated Detection and Forecasting of COVID-19 Using Deep Learning Techniques: A Review. arXiv 2020, arXiv:2007.10785. [Google Scholar]
  74. Nguyen, T.T. Artificial Intelligence in the Battle Against Coronavirus (COVID-19): A Survey and Future Research Directions. arXiv 2020, arXiv:2008.07343. [Google Scholar]
  75. Islam, M.N.; Inan, T.T.; Rafi, S.; Akter, S.S.; Sarker, I.H.; Islam, A. A Survey on the Use of AI and ML for Fighting the COVID-19 Pandemic. arXiv 2020, arXiv:2008.07449. [Google Scholar]
  76. Islam, M.M.; Karray, F.; Alhajj, R.; Zeng, J. A Review on Deep Learning Techniques for the Diagnosis of Novel Coronavirus (COVID-19). IEEE Access 2021, 9, 30551–30572. [Google Scholar] [CrossRef] [PubMed]
  77. Roberts, M.; Driggs, D.; Thorpe, M.; Gilbey, J.; Yeung, M.; Ursprung, S.; Aviles-Rivero, A.I.; Etmann, C.; McCague, C.; Beer, L.; et al. Machine Learning for COVID-19 Detection and Prognostication Using Chest Radiographs and CT Scans: A Systematic Methodological Review. 2020. Available online: https://www.researchgate.net/publication/343689629_Machine_learning_for_COVID-19_detection_and_prognostication_using_chest_radiographs_and_CT_scans_a_systematic_methodological_review (accessed on 23 February 2022).
  78. Ulhaq, A.; Born, J.; Khan, A.; Gomes, D.P.S.; Chakraborty, S.; Paul, M. Covid-19 control by computer vision approaches: A survey. IEEE Access 2020, 8, 179437–179456. [Google Scholar] [CrossRef] [PubMed]
  79. Jung, R.G.; Di Santo, P.; Clifford, C.; Prosperi-Porta, G.; Skanes, S.; Hung, A.; Parlow, S.; Visintini, S.; Ramirez, F.D.; Simard, T.; et al. Methodological quality of COVID-19 clinical research. Nat. Commun. 2021, 12, 1–10. [Google Scholar] [CrossRef] [PubMed]
  80. Yu, Y.; Shi, Q.; Zheng, P.; Gao, L.; Li, H.; Tao, P.; Gu, B.; Wang, D.; Chen, H. Assessment of the quality of systematic reviews on COVID-19: A comparative study of previous coronavirus outbreaks. J. Med. Virol. 2020, 92, 883–890. [Google Scholar] [CrossRef]
  81. Al-Ryalat, N.; Al-Rashdan, O.; Alaaraj, B.; Toubasi, A.A.; Alsghaireen, H.; Yaseen, A.; Mesmar, A.; AlRyalat, S.A. Assessment of COVID-19-Related Meta-Analysis Reporting Quality. Ir. J. Med. Sci. 2021, 1–5. [Google Scholar] [CrossRef]
  82. Nagendran, M.; Chen, Y.; Lovejoy, C.A.; Gordon, A.C.; Komorowski, M.; Harvey, H.; Topol, E.J.; Ioannidis, J.P.; Collins, G.S.; Maruthappu, M. Artificial intelligence versus clinicians: Systematic review of design, reporting standards, and claims of deep learning studies. BMJ 2020, 368, m689. [Google Scholar] [CrossRef] [Green Version]
  83. Storman, M.; Storman, D.; Jasinska, K.W.; Swierz, M.J.; Bala, M.M. The quality of systematic reviews/meta-analyses published in the field of bariatrics: A cross-sectional systematic survey using AMSTAR 2 and ROBIS. Obes. Rev. 2020, 21, e12994. [Google Scholar] [CrossRef]
  84. Leclercq, V.; Beaudart, C.; Tirelli, E.; Bruyère, O. Psychometric measurements of AMSTAR 2 in a sample of meta-analyses indexed in PsycINFO. J. Clin. Epidemiol. 2020, 119, 144–145. [Google Scholar] [CrossRef] [Green Version]
  85. Pieper, D.; Lorenz, R.C.; Rombey, T.; Jacobs, A.; Rissling, O.; Freitag, S.; Matthias, K. Authors should clearly report how they derived the overall rating when applying AMSTAR 2—A cross-sectional study. J. Clin. Epidemiol. 2021, 129, 97–103. [Google Scholar] [CrossRef]
  86. Adadi, A.; Lahmer, M.; Nasiri, S. Artificial Intelligence and COVID-19: A Systematic Umbrella Review and Roads Ahead. J. King Saud Univ. Comput. Inf. Sci. 2021. [Google Scholar] [CrossRef]
  87. ESHRE Capri Workshop Group. Protect us from poor-quality medical research. Hum. Reprod. 2018, 33, 770–776. [Google Scholar] [CrossRef] [PubMed]
  88. International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical journals: Writing and editing for biomedical publication. Indian J. Pharmacol. 2006, 38, 149. [Google Scholar]
  89. Johnson, C. Repetitive, duplicate, and redundant publications: A review for authors and readers. J. Manipulative Physiol. Ther. 2006, 29, 505–509. [Google Scholar] [CrossRef] [PubMed]
  90. Yank, V.; Barnes, D. Consensus and contention regarding redundant publications in clinical research: Cross-sectional survey of editors and authors. J. Med. Ethics 2003, 29, 109–114. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  91. Huth, E.J. Repetitive and divided publication. In Ethical Issues in Biomedical Publication; JHU Press: Baltimore, MD, USA, 2000. [Google Scholar]
  92. Naudé, W. Artificial intelligence vs COVID-19: Limitations, constraints and pitfalls. AI Soc. 2020, 35, 761–765. [Google Scholar] [CrossRef] [PubMed]
  93. Corrado, E.M. The Importance of Open Access, Open Source, and Open Standards for Libraries. 2005. Available online: https://library.educause.edu/resources/2005/1/the-importance-of-open-access-open-source-and-open-standards-for-libraries (accessed on 23 February 2022).
  94. Beaulieu-Jones, B.K.; Greene, C.S. Reproducibility of computational workflows is automated using continuous analysis. Nat. Biotechnol. 2017, 35, 342–346. [Google Scholar] [CrossRef]
  95. Born, J.; Beymer, D.; Rajan, D.; Coy, A.; Mukherjee, V.V.; Manica, M.; Prasanna, P.; Ballah, D.; Guindy, M.; Shaham, D.; et al. On the role of artificial intelligence in medical imaging of covid-19. Patterns 2021, 2, 100269. [Google Scholar] [CrossRef]
  96. Chee, M.L.; Ong, M.E.H.; Siddiqui, F.J.; Zhang, Z.; Lim, S.L.; Ho, A.F.W.; Liu, N. Artificial Intelligence Applications for COVID-19 in Intensive Care and Emergency Settings: A Systematic Review. Int. J. Environ. Res. Public Health 2021, 18, 4749. [Google Scholar] [CrossRef]
  97. Syeda, H.B.; Syed, M.; Sexton, K.W.; Syed, S.; Begum, S.; Syed, F.; Prior, F.; Yu, F., Jr. Role of machine learning techniques to tackle the COVID-19 crisis: Systematic review. JMIR Med. Inform 2021, 9, e23811. [Google Scholar] [CrossRef]
  98. Soltan, A.A.; Kouchaki, S.; Zhu, T.; Kiyasseh, D.; Taylor, T.; Hussain, Z.B.; Peto, T.; Brent, A.J.; Eyre, D.W.; Clifton, D.A. Rapid triage for COVID-19 using routine clinical data for patients attending hospital: Development and prospective validation of an artificial intelligence screening test. Lancet Digit Health 2021, 3, e78–e87. [Google Scholar] [CrossRef]
  99. The Lancet Digital Health. Artificial intelligence for COVID-19: Saviour or saboteur? Lancet Digit Health 2021, 3, e1. [Google Scholar] [CrossRef]
  100. Mongan, J.; Moy, L.; Kahn, C.E. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. Radiol. Artif. Intell. 2020, 2, e200029. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  101. Balsiger, F.; Jungo, A.; Chen, J.; Ezhov, I.; Liu, S.; Ma, J.; Paetzold, J.C.; Sekuboyina, A.; Shit, S.; Suter, Y.; et al. MICCAI Hackathon on Reproducibility, Diversity, and Selection of Papers. In Proceedings of the MICCAI Conference, Strasburg, France, 27 September–1 October 2021. [Google Scholar]
  102. Gil, D.; Díaz-Chito, K.; Sánchez, C.; Hernández-Sabaté, A. A Early screening of sars-cov-2 by intelligent analysis of X-ray images. arXiv 2020, arXiv:2005.13928. [Google Scholar]
  103. Albahri, A.S.; Hamid, R.A.; Alwan, J.K.; Al-Qays, Z.; Zaidan, A.A.; Zaidan, B.B.; Alamoodi, A.H.; Khlaf, J.M.; Almahdi, E.M.; Thabet, E.; et al. Role of biological Data Mining and Machine Learning Techniques in Detecting and Diagnosing the Novel Coronavirus (COVID-19): A Systematic Review. J. Med. Syst. 2020, 44, 122. [Google Scholar] [CrossRef]
  104. Bressem, K.K.; Adams, L.C.; Erxleben, C.; Hamm, B.; Niehues, S.M.; Vahldiek, J.L. Comparing different deep learning architectures for classification of chest radiographs. Sci. Rep. 2020, 10, 13590. [Google Scholar] [CrossRef]
  105. Chamola, V.; Hassija, V.; Gupta, V.; Guizani, M. A Comprehensive Review of the COVID-19 Pandemic and the Role of IoT, Drones, AI, Blockchain, and 5G in Managing its Impact. IEEE Access 2020, 8, 90225–90265. [Google Scholar] [CrossRef]
  106. Bragazzi, N.L.; Dai, H.; Damiani, G.; Behzadifar, M.; Martini, M.; Wu, J. How Big Data and Artificial Intelligence Can Help Better Manage the COVID-19 Pandemic. Int. J. Environ. Res. Public Health 2020, 17, 3176. [Google Scholar] [CrossRef]
  107. Nagpal, P.; Narayanasamy, S.; Garg, C.; Vidholia, A.; Guo, J.; Shin, K.M.; Lee, C.H.; Hoffman, E.A. Imaging of COVID-19 pneumonia: Patterns, pathogenesis, and advances. Br. J. Radiol. 2020, 93, 20200538. [Google Scholar] [CrossRef]
  108. Bansal, A.; Padappayil, R.P.; Garg, C.; Singal, A.; Gupta, M.; Klein, A. Utility of Artificial Intelligence Amidst the COVID 19 Pandemic: A Review. J. Med. Syst. 2020, 44, 156. [Google Scholar] [CrossRef]
  109. Rezaei, M.; Shahidi, M. Zero-shot learning and its applications from autonomous vehicles to COVID-19 diagnosis: A review. Intell. Med. 2020, 3, 100005. [Google Scholar] [CrossRef]
  110. Kharat, A.; Duddalwar, V.; Saoji, K.; Gaikwad, A.; Kulkarni, V.; Naik, G.; Lokwani, R.; Kasliwal, S.; Kondal, S.; Gupte, T.; et al. Role of edge device and cloud machine learning in point-of-care solutions using imaging diagnostics for population screening. arXiv 2020, arXiv:2006.13808. [Google Scholar]
  111. Albahri, O.S.; Zaidan, A.A.; Albahri, A.S.; Zaidan, B.B.; Abdulkareem, K.H.; Al-Qaysi, Z.T.; Alamoodi, A.H.; Aleesa, A.M.; Chyad, M.A.; Alesa, R.M.; et al. Systematic review of artificial intelligence techniques in the detection and classification of COVID-19 medical images in terms of evaluation and benchmarking: Taxonomy analysis, challenges, future solutions and methodological aspects. J. Infect. Public Health 2020, 13, 1381–1396. [Google Scholar]
  112. Manigandan, S.; Wu, M.-T.; Ponnusamy, V.K.; Raghavendra, V.B.; Pugazhendhi, A.; Brindhadevi, K. A systematic review on recent trends in transmission, diagnosis, prevention and imaging features of COVID-19. Process Biochem. 2020, 98, 233–240. [Google Scholar] [CrossRef]
Jcm 11 02054 g001 550
Figure 1. PRISMA flow chart.
Figure 1. PRISMA flow chart.
Jcm 11 02054 g001
Jcm 11 02054 g002 550
Figure 2. Quality graph: our judgements on each AMSTAR 2 item presented as the percentage of all the included studies; * denotes critical domains.
Figure 2. Quality graph: our judgements on each AMSTAR 2 item presented as the percentage of all the included studies; * denotes critical domains.
Jcm 11 02054 g002
Jcm 11 02054 g003 550
Figure 3. Quality of reporting graph: our judgements about each PRISMA-DTA item presented as averages (with 95% confidence intervals—black lines) across all included studies. Different shades of blue are used just to improve the chart’s clarity.
Figure 3. Quality of reporting graph: our judgements about each PRISMA-DTA item presented as averages (with 95% confidence intervals—black lines) across all included studies. Different shades of blue are used just to improve the chart’s clarity.
Jcm 11 02054 g003
Jcm 11 02054 g004 550
Figure 4. The cumulative chart of included, available (by the date), and introduced primary papers among discussed reviews.
Figure 4. The cumulative chart of included, available (by the date), and introduced primary papers among discussed reviews.
Jcm 11 02054 g004
Table
Table 1. Detailed characteristics of included reviews.
Table 1. Detailed characteristics of included reviews.
VariableNumber (Percentage)Mean
(Range)2
Number of reviews with the authors from a specific country
 United States of America8 (18%)NA
 Australia4 (9%)NA
 China4 (9%)NA
 India4 (9%)NA
 United Kingdom3 (7%)NA
 Other22 (49%)NA
Total number of authors of the reviews1718 (1-43)
Type of publication
 Journal article (mean IF1: 4.14; range: 0–30.31)13 (59%)NA
 IEEE Access2 (9%)NA
  IEEE Reviews in Biomedical Engineering2 (9%)NA
  Diagnostic and Interventional Imaging2 (9%)NA
 Diabetes & Metabolic Syndrome:
 Clinical Research & Reviews		1 (5%)NA
  Applied Intelligence1 (5%)NA
 British Medical Journal1 (5%)NA
 Biosensors and Bioelectronics1 (5%)NA
 Machine Vision and Applications1 (5%)NA
 Current Problems in Diagnostic Radiology1 (5%)NA
 Journal of the Indian Medical Association1 (5%)NA
 Preprint article8 (36%)NA
 Conference article1 (5%)NA
Was the review specified as systematic by the authors?
 No20 (91%)NA
 Yes2 (9%)NA
Number of reviews that searched a given data source505 (3-7)
 arXiv8 (36%)NA
 medRxiv6 (27%)NA
 Pubmed/Medline6 (27%)NA
 Google Scholar6 (27%)NA
 bioRxiv5 (23%)NA
 IEEE Xplore3 (14%)NA
 Science Direct3 (14%)NA
 ACM digital library2 (9%)NA
 Springer2 (9%)NA
 MICCAI conference1 (5%)NA
 IPMI conference1 (5%)NA
 Embase1 (5%)NA
 Web of Science1 (5%)NA
 Elsevier1 (5%)NA
 Nature1 (5%)NA
Number of studies
 Reported by review authors as included35851 (20–107)
 Applicable for this review question (total)45121 (1–106)
 Applicable for this review question (unique only)1657.5 (0–11)
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Jemioło, P.; Storman, D.; Orzechowski, P. Artificial Intelligence for COVID-19 Detection in Medical Imaging—Diagnostic Measures and Wasting—A Systematic Umbrella Review. J. Clin. Med. 2022, 11, 2054. https://doi.org/10.3390/jcm11072054

AMA Style

Jemioło P, Storman D, Orzechowski P. Artificial Intelligence for COVID-19 Detection in Medical Imaging—Diagnostic Measures and Wasting—A Systematic Umbrella Review. Journal of Clinical Medicine. 2022; 11(7):2054. https://doi.org/10.3390/jcm11072054

Chicago/Turabian Style

Jemioło, Paweł, Dawid Storman, and Patryk Orzechowski. 2022. "Artificial Intelligence for COVID-19 Detection in Medical Imaging—Diagnostic Measures and Wasting—A Systematic Umbrella Review" Journal of Clinical Medicine 11, no. 7: 2054. https://doi.org/10.3390/jcm11072054

APA Style

Jemioło, P., Storman, D., & Orzechowski, P. (2022). Artificial Intelligence for COVID-19 Detection in Medical Imaging—Diagnostic Measures and Wasting—A Systematic Umbrella Review. Journal of Clinical Medicine, 11(7), 2054. https://doi.org/10.3390/jcm11072054

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop