Next Article in Journal
Prediction of Resting Energy Expenditure in Children: May Artificial Neural Networks Improve Our Accuracy?
Next Article in Special Issue
Machine Learning and Virtual Reality on Body Movements’ Behaviors to Classify Children with Autism Spectrum Disorder
Previous Article in Journal
Comparison of Clinical Outcomes between Hyaluronic and Polylactic Acid Filler Injections for Penile Augmentation in Men Reporting a Small Penis: A Multicenter, Patient-Blinded/Evaluator-Blinded, Non-Inferiority, Randomized Comparative Trial with 18 Months of Follow-up
Previous Article in Special Issue
The Benefits of emotion Regulation Interventions in Virtual Reality for the Improvement of Wellbeing in Adults and Older Adults: A Systematic Review
 
 
Retraction published on 28 July 2020, see J. Clin. Med. 2020, 9(8), 2404.
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

RETRACTED: Use of Virtual Reality for the Management of Anxiety and Pain in Dental Treatments: Systematic Review and Meta-Analysis

by
Nansi López-Valverde
1,
Jorge Muriel Fernández
1,
Antonio López-Valverde
1,*,
Luis F. Valero Juan
2,
Juan Manuel Ramírez
3,
Javier Flores Fraile
1,
Julio Herrero Payo
1,
Leticia A. Blanco Antona
1,
Bruno Macedo de Sousa
4 and
Manuel Bravo
5
1
Department of Surgery, University of Salamanca, Instituto de Investigación Biomédica de Salamanca (IBSAL), 37007 Salamanca, Spain
2
Department of Biomedical and Diagnostic Sciences, University of Salamanca, Avda. Alfonso X El Sabio S/N, 37007 Salamanca, Spain
3
Department of Morphological Sciences, University of Cordoba, Avenida Menéndez Pidal s/n, 14071 Cordoba, Spain
4
Institute for Occlusion and Orofacial Pain Faculty of Medicine, University of Coimbra, Polo I - Edifício Central Rua Larga, 3004-504 Coimbra, Portugal
5
Department of Preventive and Community Dentistry, Facultad de Odontología, Campus de Cartuja s/n, 18071 Granada, Spain
*
Author to whom correspondence should be addressed.
J. Clin. Med. 2020, 9(4), 1025; https://doi.org/10.3390/jcm9041025
Submission received: 26 February 2020 / Revised: 27 March 2020 / Accepted: 27 March 2020 / Published: 5 April 2020

Abstract

:
Background: Dental treatments often cause pain and anxiety in patients. Virtual reality (VR) is a novel procedure that can provide distraction during dental procedures or prepare patients to receive such type of treatments. This meta-analysis is the first to gather evidence on the effectiveness of VR on the reduction of pain (P) and dental anxiety (DA) in patients undergoing dental treatment, regardless of age. Methods: MEDLINE, CENTRAL, PubMed, EMBASE, Wiley Library and Web of Science were searched for scientific articles in November 2019. The keywords used were: “virtual reality”, “distraction systems”, “dental anxiety” and “pain”. Studies where VR was used for children and adults as a measure against anxiety and pain during dental treatments were included. VR was defined as a three-dimensional environment that provides patients with a sense of immersion, transporting them to appealing and interactive settings. Anxiety and pain results were assessed during dental treatments where VR was used, and in standard care situations. Results: 31 studies were identified, of which 14 met the inclusion criteria. Pain levels were evaluated in four studies (n = 4), anxiety levels in three (n = 3) and anxiety and pain together in seven (n = 7). Our meta-analysis was based on ten studies (n = 10). The effect of VR was studied mainly in the pediatric population (for pain SMD = −0.82). In the adult population, only two studies (not significant) were considered. Conclusions: The findings of the meta-analysis show that VR is an effective distraction method to reduce pain and anxiety in patients undergoing a variety of dental treatments; however, further research on VR as a tool to prepare patients for dental treatment is required because of the scarcity of studies in this area.

1. Introduction

Pain suppression during dental interventions has been a major accomplishment for humankind. In 1842, William E. Clarke gave ether to a patient for the removal of a tooth; later, in 1844, a dentist named Horace Wells used nitrous oxide as an anesthetic for dental extractions; and in 1846, another dentist, William T. G. Morton, became a pioneer in the use of inhaled ether as an anaesthetic at the Massachusetts General Hospital [1].
Patient anxiety when facing dental procedures is determined by two circumstances: on the one hand, the prior act of anesthetizing, which in itself frequently causes a state of phobia [2] and, on the other hand, the subsequent dental treatment. The number of studies on this pathology has exponentially increased over the last few years, growing from a very low number in the 1940s to the more than 6000 papers that are currently available, according to the U.S. National Library of Medicine (Figure 1).
Different therapies have been proposed for the prevention and treatment of P and DA, among them virtual reality (VR) distraction techniques [3].
Although it is a concept that is difficult to define, VR is generally accepted as a three-dimensional environment generated by means of computer technology that creates a sense of immersion in the user, transporting the individual to appealing and interactive settings [4].
The benefits of using VR for the reduction of dental anxiety (DA) and pain (P) levels during dental procedures has been extensively addressed in scientific literature [5,6,7,8,9], and its usefulness as a distraction tool is receiving increasing attention in medical contexts [9]. During aversive experiences, VR can improve pain management [10] and reduce the perceived duration of the procedure [11]. Moreover, a recent systematic review examined the effectiveness of virtual reality distraction in reducing pain [12]. This could be an advantage for many patients who reject DA control using anti-anxiety drugs because of their disadvantages or side effects, which can be, among others, impaired cognitive function and coordination, since they act as depressants on specific areas of the central nervous system [13,14,15].
The purpose of this study was to conduct a systematic review of literature comparing the effectiveness of the use of VR as a method for reducing anxiety and pain levels during dental treatment. This systematic review constitutes an essential tool to synthesize the scientific information available, increasing the validity of the conclusions and of individual studies, and identifying areas of uncertainty, where research is necessary. Meta-analysis (when possible) provides very useful information, to facilitate understanding of the effect of a treatment or intervention, both in general and in specific groups of patients. In addition, it allows us to increase the precision in the estimation of the effect, detecting effects of moderate magnitude, but of clinical importance, that could go unnoticed in primary studies.

2. Methods

The study selection process was carried out according to the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) guidelines for systematic reviews and meta-analysis [16].

2.1. Protocol

The search strategy was conducted using the population, intervention, comparison and outcome (PICO) framework, based on the following question:
“Are distraction techniques using VR effective against the anxiety and pain caused by dental procedures?”
To answer this question, a population of patients undergoing dental treatment, with no age limit, were selected. The intervention consisted in using audio–visual or VR distraction methods. Controls were patients who were not subjected to audio–visual or VR distraction methods. The results revised in the literature were the DA or P values obtained using different validated scales:
-
For pain: Visual Analogic Scale (VAS), Wong–Baker Faces Scale (W–BFS) and Faces Pain Scale-Revise (FPS-R).
-
For anxiety: Consolability Scale (FLACC), Verbal Rating Scale (VRS), Modified Dental Anxiety Scale (MDAS), (Norman Corah’s anxiety questionnaire (NCAQ) and Venham’s Clinical Anxiety Rating Scale (VCARS).

2.2. Search Method for the Identification of Studies

A search of the MEDLINE, CENTRAL, PubMed, EMBASE, Wiley Library and Web of Science electronic databases was conducted in November 2019 to identify relevant scientific articles. The search terms used were: “virtual reality”, “distraction systems”, “dental anxiety”, “pain”.

2.3. Inclusion and Exclusion Criteria

Inclusion criteria:
(a)
Articles published in English.
(b)
Randomized controlled clinical trials related to dental anxiety and pain associated with dental procedures in children and adults.
(c)
Studies assessing anxiety in said procedures.
Exclusion criteria:
(a)
Non-randomized studies or non-controlled clinical trials.
(b)
Comparative studies.
(c)
Narrative reviews and systematic reviews.
(d)
Case studies.
(e)
Irrelevant and duplicate studies and those that did not meet the established inclusion criteria.

2.4. Data Extraction and Analysis

Studies that made no reference to the research question were removed, and the titles and abstracts of the articles selected were obtained and entered in an Excel spreadsheet. Two reviewers (NL-V and JMF) selected the titles and abstracts independently. Discrepancies in terms of study inclusion were discussed between the two mentioned reviewers until consensus was reached. Subsequently, the full texts of the selected studies were obtained for their review and inclusion.

2.5. Risk of Bias (RoB) of Included Articles

The Cochrane Collaboration, London, UK, tool was used to assess the methodology of the scientific evidence in all the selected studies [17].

2.6. Quality of the Reports of the Included Studies

This was assessed according to the modified Strengthening the Reporting of Observational studies in Epidemiology (STROBE) statement [18], which includes a total of 22 items. Each item was assessed by reviewers NL-V and JMF, who attributed scores of 0 (not reported) or 1 (reported), carrying out a complete count of all the studies included (Table 1).

2.7. Statistical Analysis

In the meta-analysis, four studies were excluded on the grounds that two of them [7,19] did not present results and the other two [20,21] presented confusing results at the time of their assessment. Physiological data, such as pulse rate, degree of oxygen saturation, blood pressure and more were not included, incorporating data related to pain and anxiety only, during dental treatments. Pain and anxiety were analyzed separately in children and adults. The mean scores and SDs (standard deviations) for pain and anxiety, during the procedure with VR and control, were extracted from the selected articles, using mean scores and interquartile ranges, or reported, directly, by the authors of the studies. Other information not related to VR was not taken into consideration in our meta-analysis. The different measurement scales and VR devices used were not considered. The meta-analysis was performed using Stata v.14.2 (StataCorp LP, College Station, TX, USA) and closely followed the methods proposed by the Cochrane collaboration [17]. The methods can be observed in the different tables and figures. The standardized difference of means (SMD) was used as a measure of effect to account for different measurement scales both for anxiety and pain. Statistical heterogeneity among studies was assessed using the Q test according to Dersimonian and Laird and the I2 index (heterogeneity: I2 > 30% being moderate, >50% substantial and >75% considerable [17]). We decided to pool the study-specific estimates with the random effects model to protect our composite estimates (for anxiety and pain) from heterogeneity in the context of a relatively limited number of studies. We also decided a priori to present the results not only for all the studies together, but also as a subgroup analysis according to age group (children and adults) derived from the different clinical usefulness and interpretation. Finally, funnel graphs (not shown) and p-value calculation (Egger test) were used to assess the publication bias.

3. Results

3.1. Characteristics of the Studies

Until November 2019, a total of 31 studies were gathered and subsequently assessed by the reviewers. Three duplicate studies were removed after an initial detection. A second detection led to the removal of 14 studies, which left a total of 14 full-text studies for the final selection [2,6,7,9,19,20,21,22,23,24,25,26,27,28] (Figure 2). Pain levels were assessed in four studies, two in children [20,28] and two in adults [2,21]; anxiety levels in three, two in adults [9,19] and one in children [26]; and anxiety and pain together in seven studies, five in children [22,23,24,25,27] and two in adults [6,7].
Table 2 provides a general description of the details of each study. The risk of bias (RoB Cochrane Collaboration Tool) in the studies considered is shown in Figure 3. All the studies complied with random sequence. Two of the studies included [25,28] did not comply with allocation concealment and only 35% complied with blinding of participants and personnel. It should be noted that none of the studies included complied with blinding of outcome assessment.
The STROBE criteria reported an average score of 27.4 (± 0.85), the maximum scores corresponding to the studies by Al-Halabi et al. [20] and Mitrakul et al. [27]. It is also noteworthy that Item 9 (Bias) was only reported in five studies [21,22,24,26,27], which is also the case with Item 19 (Limitations) [20,21,22,26,27] (Table 1).

3.2. VR and Anxiety Management

Regarding anxiety, the composite measure is not significant, neither for all together (children + adults) (p = 0.302), nor for children (p = 0.243) (Figure 4) nor adults (p = 0.567) (Figure 5). The heterogeneity seems to be moderate (Table 3).

3.3. VR and Pain Management

For pain, there is a significant protection for children (SMD = −0.82, i.e., a substantial effect according to Cohen’s scale [29]) (Figure 6), but not for adults (Figure 7), since the 95%-confidence interval (CI) includes the null value 0 (Table 4). Nevertherless, it is based only on two studies.

3.4. Publication Bias and Heterogeneity

All estimates seem not to be affected by publication bias, according to the Egger test (Table 1 and Table 2), but heterogeneity seems to be substantial.

4. Discussion

While anxiety and pain are usually associated with dental treatments, the number of studies addressing their management, especially during anesthetic block, which is one of the procedures that usually causes great anxiety among patients, is very limited [2,30].
Although most studies are based on pediatric population, fear of dental treatment affects 15%–20% of the population, being recognized by the World Health Organization (WHO) as a real pathology [3,31,32] that leads those who are affected by it to reject even the most basic dental treatments, such as simple dental check-ups or cleanings [33]; thus, its management is essential to improve the patient’s quality of life [34].
Despite the existence of rigorous reviews of literature [5], we believe that this is the first systematic review and meta-analysis focused on the efficacy of VR in patients, regardless of age, who suffer from anxiety triggered by dental treatments.
Our meta-analysis is based on 10 studies: two on pain, two on DA and six on DA and P together, and has proved that VR is an effective tool for reducing pain (SMD = −0.82), as reported by child patients during a variety of dental procedures.
Some studies proved that, both for pain and anxiety, the use of VR was more effective in children than in adults. A possible reason for this could be that VR is especially appealing to children, since they become more engaged in whimsical thinking and are fascinated by imaginative play [35].
Nevertheless, regarding age, it should be noted that the differences in the efficacy of VR, in each study, could be due to the interpretive problems to which these analyses are susceptible; such phenomenon, known as ecological fallacies, could be associated either to the heterogeneity of the study’s characteristics (methodological diversity), or to the study populations (clinical diversity) [36].
VR was found to be more effective for treating P and DA than conventional treatments; however, it is difficult to assess its efficacy as compared to other types of distraction. Klassen and colleagues [37] conducted a meta-analysis on distraction using music therapy as an alternative method to reduce anxiety and pain in different medical and dental procedures, finding a significant reduction, with an effect size of 0.35. A Cochrane review of psychological interventions using different types of distraction to relieve pain in children and adolescents, published in 2006 and updated in 2013 and 2018 [38,39,40], reported different distraction techniques such as musical therapy, reading, watching films, hypnosis, breathing techniques and combined cognitive-behavioral strategies, as effective tools to reduce pain and anxiety during needle procedures. However, the reviewers considered the level of certainty of the review to be low, since in most of these studies there was no blinding of participants and assessors. This is consistent with our meta-analysis, according to which only 35% of the studies included met this requirement.
The heterogeneity of VR software and hardware is also relevant to the immersive approach, which is influenced by the interaction with the virtual environment, either through translation or change of position, rotation or change of direction, viewpoint or perspective and visual field. This aspect is difficult to analyze when referred to patients who are undergoing dental treatment (especially in children), since adequate patient immersion is hindered by the fact that they are expected to remain with their heads as still as possible to facilitate the professional’s work [41].
Large devices (hardware), also hinder the dentist’s work, limiting vision of and access to the dental operation area [42].
Another interesting aspect that has not been given due regard by researchers is gender difference and its significance in terms of fear of dental treatment. Patients who suffer from dental phobia are characterized by a gender-specific brain structure. Such differences have not been sufficiently addressed by researchers and clinicians, and could contribute to greater effectiveness in the management of dental phobia [43].
Likewise, patients’ different personality traits were not considered in the studies included, either. Patients with dental fear and a high predisposition to anxiety magnify their pain expectations when they are exposed to critical situations. When patients with dental anxiety undergo dental treatment, their beliefs about the negative consequences of bodily excitement can negatively influence their assessment of pain linked to such treatment [44]. DA as a predisposing factor is associated with a state of anxiety, which has a constant impact on pain during the patient’s entire dental treatment; hence, anxiety should be assessed as a critical step not only towards anxiety management in patients with high DA, but also towards P management in all dental patients [45].
On the other hand, the studies included used different pain and DA assessment scales: VAS, W-BFS, FPS-R FLACC, VRS, MDAS, NCAQ and VCARS. Likewise, none of the studies included single scales for joint assessment by dentist and patient. Vital signs as emotional state indicators were also assessed in some of the studies, thus it would be convenient and appropriate to find validated scales that might be used to adequately assess all of these aspects [46].
Eijlers and colleagues [5,47] presented two preparatory studies for pediatric surgery based on VR training programs; however, the use of this type of preparatory program before certain dental procedures is yet to be explored by researchers and, therefore, it is currently not possible to compare effects with and without preparation.
Moreover, none of the studies drew attention to factors that could moderate VR’s effectiveness, such as a subject’s sensitivity to anxiety or their temperament. Shy and emotional temperaments could be associated with dental anxiety [48,49,50,51].
For all these reasons, we believe that this systematic review has certain limitations in terms of number, quality and methodology of the studies included: only three studies in adults were included [2,6,9], which is too scarce to consider them significant in our analysis, the authors themselves even acknowledge the limited sample size in one of them [2], and the existence of major limitations with regard to participant and assessor blinding, in another [6].
Hence, to determine the effects of VR on anxiety and pain in dental treatments, it would be necessary to reduce the risk of bias, to remove confusion factors and to establish a clear definition of the adequate parameters, all with the purpose of obtaining results that can be translated into broad clinical applications, so that the evidence can effectively support the practice of clinical dentistry.

5. Conclusions

This systematic review and meta-analysis leads to the conclusion that VR is a useful tool to reduce P in children undergoing dental treatment. No significant effect was found for DA. Studies in adults are scarce. On the other hand, most of the studies chose to focus on immersion in the pediatric population, neglecting a series of aspects that should be considered, such as training programs, the different types of software and hardware of virtual reality devices, the temperament and patient personality, gender difference and more. Due to all this, the role of virtual reality in the control of anxiety and dental pain in children and adults should be considered as a topic for future research.

Author Contributions

Study concept and design: N.L.-V., A.L.-V., J.M.F.; Acquisition of data (Literature search and Study Selection): N.L.-V., J.F.F., J.M.R.; Analysis and interpretation of data (literature): M.B., L.F.V.J., J.H.P., L.A.B.A.; Drafting of the manuscript: N.L.-V.; A.L.-V., B.M.d.S.; Critical revision of the manuscript for important intellectual content: A.L.-V., M.B. All authors have read and agreed to the published version of the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

VRVirtual Reality
DADental Anxiety
VASVisual Analogic Scale
W–BFSWong–Baker Faces Scale
FPS-RFaces Pain Scale-Revise
FLACCConsolability Scale
VRSVerbal Rating Scale
MDASModified Dental Anxiety Scale
NCAQNorman Corah’s anxiety questionnaire
VCARSVenham’s Clinical Anxiety Rating Scale
RoBRisk of Bias
SDsStandard Deviation
SMDStandard Mean Deviation
CIConfidence Interval

References

  1. López-Valverde, A.; Montero, J.; Albaladejo, A.; Gómez de Diego, R. The discovery of surgical anesthesia: Discrepancies regarding its authorship. J. Dent. Res. 2011, 90, 31–34. [Google Scholar] [CrossRef] [PubMed]
  2. Sweta, V.R.; Abhinav, R.P.; Asha, R. Role of Virtual Reality in Pain Perception of Patients Following the Administration of Local Anesthesia. Ann. Maxillofac. Surg. 2019, 9, 110–113. [Google Scholar] [PubMed]
  3. Ougradar, A.; Ahmed, B. Patients’ perceptions of the benefits of virtual reality during dental extractions. Br. Dent. J. 2019, 227, 813–816. [Google Scholar] [CrossRef] [PubMed]
  4. Carl, E.; Stein, A.T.; Levihn-Coon, A.; Pogue, J.R.; Rothbaum, B.; Emmelkamp, P.; Asmundson, G.; Carlbring, P.; Powers, M.B. Virtual reality exposure therapy for anxiety and related disorders: A meta-analysis of randomized controlled trials. J. Anxiety Disord. 2019, 61, 27–36. [Google Scholar] [CrossRef]
  5. Eijlers, R.; Utens, E.; Staals, L.M.; de Nijs, P.F.A.; Berghmans, J.M.; Wijnen, R.M.H.; Hillegers, M.H.J.; Dierckx, B.; Legerstee, J.S. Systematic Review and Meta-analysis of Virtual Reality in Pediatrics: Effects on Pain and Anxiety. Anesth. Analg. 2019, 129, 1344–1353. [Google Scholar] [CrossRef]
  6. Gujjar, K.R.; van Wijk, A.; Kumar, R.; de Jongh, A. Efficacy of virtual reality exposure therapy for the treatment of dental phobia in adults: A randomized controlled trial. J. Anxiety Disord. 2019, 62, 100–108. [Google Scholar] [CrossRef]
  7. Raghav, K.; Van Wijk, A.J.; Abdullah, F.; Islam, M.N.; Bernatchez, M.; De Jongh, A. Efficacy of virtual reality exposure therapy for treatment of dental phobia: A randomized control trial. BMC Oral Health 2016, 27, 25. [Google Scholar] [CrossRef]
  8. Wiederhold, M.D.; Gao, K.; Wiederhold, B.K. Clinical use of virtual reality distraction system to reduce anxiety and pain in dental procedures. Cyberpsychol. Behav. Soc. Netw. 2014, 17, 359–365. [Google Scholar] [CrossRef]
  9. Tanja-Dijkstra, K.; Pahl, S.; White, M.P.; Andrade, J.; Qian, C.; Bruce, M.; May, J.; Moles, D.R. Improving dental experiences by using virtual reality distraction: A simulation study. PLoS ONE 2014, 2, e91276. [Google Scholar] [CrossRef]
  10. Dahlquist, L.M.; McKenna, K.D.; Jones, K.K.; Dillinger, L.; Weiss, K.E.; Ackerman, C.S. Active and passive distraction using a head-mounted display helmet: Effects on cold pressor pain in children. Health Psychol. 2014, 26, 794–801. [Google Scholar] [CrossRef]
  11. Schneider, S.M.; Hood, L.E. Virtual reality: A distraction intervention for chemotherapy. Oncol. Nurs. Forum 2007, 34, 39–46. [Google Scholar] [CrossRef] [PubMed]
  12. Malloy, K.M.; Milling, L.S. The effectiveness of virtual reality distraction for pain reduction: A systematic review. Clin. Psychol. Rev. 2010, 30, 1011–1018. [Google Scholar] [CrossRef] [PubMed]
  13. American Society of Anesthesiologists Task Force on Sedation and Analgesia by Non-Anesthesiologists. Practice guidelines for sedation and analgesia by non-anesthesiologists. Anesthesiology 2002, 96, 1004–1017. [Google Scholar] [CrossRef] [PubMed]
  14. Dixon, R.A.; Kenyon, C.; Marsh, D.R.; Thornton, J.A. Midazolam in conservative dentistry. A cross-over trial. Anaesthesia 1986, 41, 276–281. [Google Scholar] [CrossRef]
  15. Foley, J. The way forward for dental sedation and primary cares? Br. Dent. J. 2002, 193, 161–164. [Google Scholar] [CrossRef]
  16. Hutton, B.; Ferrán Catalá-López, F.; Moher, D. The PRISMA statement extension for systematic reviews incorporating network meta-analysis: PRISMA-NMA. Med. Clin. 2016, 16, 262–266. [Google Scholar] [CrossRef]
  17. Higgins, J.P.T.; Altman, D.G.; Sterne, J.A.C. Cochrane Handbook for Systematic Reviews of Interventions, version 5.1.0; Higgins, J.P.T., Green, S., Eds.; The Cochrane Collaboration: London, UK, 2006; Chapter 8; Available online: http://www.cochrane-handbook.org (accessed on 20 October 2019).
  18. Vandenbroucke, J.P.; von Elm, E.; Altman, D.G.; Gøtzsche, P.C.; Mulrow, C.D.; Pocock, S.J.; Charles, P.; James, J.S.; Matthias, E. STROBE Initiative. Strengthening the reporting of observational studies in epidemiology (STROBE): Explanation and elaboration. PLoS Med. 2007, 4, e297. [Google Scholar] [CrossRef]
  19. Tanja-Dijkstra, K.; Pahl, S.; White, M.P.; Andrade, J.; May, J.; Stone, R.J.; Bruce, M.; Mills, I.; Auvray, M.; Gabe, R.; et al. Can virtual nature improve patient experiences and memories of dental treatment? A study protocol for a randomized controlled trial. Trials 2014, 22, 90. [Google Scholar] [CrossRef]
  20. Al-Halabi, M.N.; Bshara, N.; AlNerabieah, Z. Effectiveness of audio visual distraction using virtual reality eyeglasses versus tablet device in child behavioral management during inferior alveolar nerve block. Anaesth. Pain Intensive Care 2018, 22, 55–61. [Google Scholar]
  21. Bentsen, B.; Svensson, P.; Wenzel, A. Evaluation of effect of 3D video glasses on perceived pain and unpleasantness induced by restorative dental treatment. Eur. J. Pain 2001, 5, 373–378. [Google Scholar] [CrossRef]
  22. Asl Aminabadi, N.; Erfanparast, L.; Sohrabi, A.; Ghertasi Oskouei, S.; Naghili, A. The Impact of Virtual Reality Distraction on Pain and Anxiety during Dental Treatment in 4-6 Year-Old Children: A Randomized Controlled Clinical Trial. J. Dent. Res. Dent. Clin. Dent. Prospect. 2012, 6, 117–124. [Google Scholar]
  23. Nunna, M.; Dasaraju, R.K.; Kamatham, R.; Mallineni, S.K.; Nuvvula, S. Comparative evaluation of virtual reality distraction and counter-stimulation on dental anxiety and pain perception in children. J. Dent. Anesth. Pain Med. 2019, 19, 277–288. [Google Scholar] [CrossRef] [PubMed]
  24. Niharika, P.; Reddy, N.V.; Srujana, P.; Srikanth, K.; Daneswari, V.; Geetha, K.S. Effects of distraction using virtual reality technology on pain perception and anxiety levels in children during pulp therapy of primary molars. J. Indian Soc. Pedod. Prev. Dent. 2018, 36, 364–369. [Google Scholar] [CrossRef] [PubMed]
  25. Shetty, V.; Suresh, L.R.; Hegde, A.M. Effect of Virtual Reality Distraction on Pain and Anxiety during Dental Treatment in 5 to 8 Year Old Children. J. Clin. Pediatr. Dent. 2019, 43, 97–102. [Google Scholar] [CrossRef]
  26. Al-Khotani, A.; Bello, L.A.; Christidis, N. Effects of audiovisual distraction on children’s behaviour during dental treatment: A randomized controlled clinical trial. Acta Odontol. Scand. 2016, 74, 494–501. [Google Scholar] [CrossRef] [PubMed]
  27. Mitrakul, K.; Asvanund, Y.; Arunakul, M.; Paka-Akekaphat, S. Effect of audiovisual eyeglasses during dental treatment in 5–8 year-old children. Eur. J. Paediatr. Dent. 2015, 16, 239–245. [Google Scholar] [PubMed]
  28. Asvanund, Y.; Mitrakul, K.; Juhong, R.O.; Arunakul, M. Effect of audiovisual eyeglasses during local anesthesia injections in 5- to 8-year-old children. Quintessence Int. 2015, 46, 513–521. [Google Scholar]
  29. Cohen, J. Statistical Power Analysis for the Behavioral Sciences, 2nd ed.; Lawrence Erlbaum Associates: New York, NY, USA, 1988. [Google Scholar]
  30. Yamashita, Y.; Shimohira, D.; Aijima, R.; Mori, K.; Danjo, A. Clinical Effect of Virtual Reality to Relieve Anxiety During Impacted Mandibular Third Molar Extraction Under Local Anesthesia. J. Oral Maxillofac. Surg. 2019, 26, 31343–31346. [Google Scholar] [CrossRef]
  31. Hill, K.B.; Chadwick, B.; Freeman, R.; O’Sullivan, I.; Murray, J.J. Adult Dental Health Survey 2009: Relationships between dental attendance patterns, oral health behaviour and the current barriers to dental care. Br. Dent. J. 2013, 214, 25–32. [Google Scholar] [CrossRef]
  32. Seligman, L.D.; Hovey, J.D.; Chacon, K.; Ollendick, T.H. Dental anxiety: An understudied problem in youth. Clin. Psychol. Rev. 2017, 55, 25–40. [Google Scholar] [CrossRef]
  33. De Stefano, R. Psychological Factors in Dental Patient Care: Odontophobia. Medicina 2019, 8, 55. [Google Scholar] [CrossRef] [PubMed]
  34. Fiorillo, L. Oral Health: The First Step to Well-Being. Medicina 2019, 7, 55. [Google Scholar] [CrossRef] [PubMed]
  35. Bolton, D.; Dearsley, P.; Madronal-Luque, R.; Baron-Cohen, S. Magical thinking in childhood and adolescence: Development and relation to obsessive compulsion. Br. J. Dev. Psychol. 2002, 20, 479–494. [Google Scholar] [CrossRef]
  36. Thompson, S.G.; Higgins, J.P. How should meta-regression analyses be undertaken and interpreted? Stat. Med. 2002, 21, 559–573. [Google Scholar] [CrossRef]
  37. Klassen, J.A.; Liang, Y.; Tjosvold, L.; Klassen, T.P.; Hartling, L. Music for pain and anxiety in children undergoing medical procedures: A systematic review of randomized controlled trials. Ambul. Pediatr. 2008, 8, 117–128. [Google Scholar] [CrossRef]
  38. Uman, L.S.; Chambers, C.T.; McGrath, P.J.; Kisely, S. Psychological interventions for needle-related procedural pain and distress in children and adolescents. Cochrane Database Syst. Rev. 2006, 18. [Google Scholar] [CrossRef]
  39. Uman, S.L.; Birnie, K.A.; Noel, M.; Parker, J.A.; Chambers, C.T.; McGrath, P.J.; Kisely, S.R. Psychological interventions for needle-related procedural pain and distress in children and adolescents. Cochrane Database Syst. Rev. 2013, 10. [Google Scholar] [CrossRef]
  40. Birnie, K.A.; Noel, M.; Chambers, C.T.; Uman, L.S.; Parker, J.A. Psychological interventions for needle-related procedural pain and distress in children and adolescents. Cochrane Database Syst. Rev. 2018, 10. [Google Scholar] [CrossRef]
  41. Hoffman, H.G.; Sharar, S.R.; Coda, B.; Everett, J.J.; Ciol, M.; Richards, T.; Patterson, D.R. Manipulating presence influences the magnitude of virtual reality analgesia. Pain 2004, 111, 162–168. [Google Scholar] [CrossRef]
  42. Yu, M.; Zhou, R.; Wang, H.; Zhao, W. An evaluation for VR glasses system user experience: The influence factors of interactive operation and motion sickness. Appl. Ergon. 2019, 74, 206–213. [Google Scholar] [CrossRef]
  43. Schienle, A.; Scharmüller, W.; Leutgeb, V.; Schäfer, A.; Stark, R. Sex differences in the functional and structural neuroanatomy of dental phobia. Brain Struct. Funct. 2013, 218, 779–787. [Google Scholar] [CrossRef]
  44. Klages, U.; Kianifard, S.; Ulusoy, O.; Wehrbein, H. Anxiety sensitivity as predictor of pain in patients undergoing restorative dentalprocedures. Community Dent. Oral Epidemiol. 2006, 34, 139–145. [Google Scholar] [CrossRef] [PubMed]
  45. Lin, C.S.; Wu, S.Y.; Yi, C.A. Association between Anxiety and Pain in Dental Treatment: A Systematic Review and Meta-analysis. J. Dent. Res. 2017, 96, 153–162. [Google Scholar] [CrossRef] [PubMed]
  46. Astramskaitė, I.; Juodžbalys, G. Scales used to rate adult patients’ psycho-emotional status in tooth extraction procedures: A systematic review. Int. J. Oral Maxillofac. Surg. 2017, 46, 886–898. [Google Scholar] [CrossRef] [PubMed]
  47. Eijlers, R.; Legerstee, J.S.; Dierckx, B.; Staals, L.M.; Berghmans, J.; van der Schroeff, M.P.; Wijnen, R.M.; Utens, E.M. Development of a Virtual Reality Exposure Tool as Psychological Preparation for Elective Pediatric Day Care Surgery: Methodological Approach for a Randomized Controlled Trial. JMIR Res. Protoc. 2017, 6, e174. [Google Scholar] [CrossRef] [PubMed]
  48. Jain, A.; Suprabha, B.S.; Shenoy, R.; Rao, A. Association of temperament with dental anxiety and behaviour of the preschool child during the initial dental visit. Eur. J. Oral Sci. 2019, 127, 147–155. [Google Scholar] [CrossRef] [PubMed]
  49. Stenebrand, A.; Wide Boman, U.; Hakeberg, M. Dental anxiety and temperament in 15-year olds. Acta Odontol. Scand. 2013, 71, 15–21. [Google Scholar] [CrossRef]
  50. Lundgren, J.; Elfström, M.L.; Berggren, U. The relationship between temperament and fearfulness in adult dental phobic patients. Int. J. Paediatr. Dent. 2007, 17, 460–468. [Google Scholar] [CrossRef]
  51. Bergdahl, M.; Bergdahl, J. Temperament and character personality dimensions in patients with dental anxiety. Eur. J. Oral. Sci. 2003, 111, 93–98. [Google Scholar] [CrossRef]
Figure 1. Increasing tendency of publications, according the U.S. National Library of Medicine database, using “dental phobia” as the keyword.
Figure 1. Increasing tendency of publications, according the U.S. National Library of Medicine database, using “dental phobia” as the keyword.
Jcm 09 01025 g001
Figure 2. Flow chart of the study selection process. PRISMA (Preferred Reporting Items for Systematic Review and meta Analyses) [16].
Figure 2. Flow chart of the study selection process. PRISMA (Preferred Reporting Items for Systematic Review and meta Analyses) [16].
Jcm 09 01025 g002
Figure 3. Risk of bias.
Figure 3. Risk of bias.
Jcm 09 01025 g003
Figure 4. Forest plot for anxiety in Children.
Figure 4. Forest plot for anxiety in Children.
Jcm 09 01025 g004
Figure 5. Forest plot for Anxiety in Adults.
Figure 5. Forest plot for Anxiety in Adults.
Jcm 09 01025 g005
Figure 6. Forest plot for pain in Children.
Figure 6. Forest plot for pain in Children.
Jcm 09 01025 g006
Figure 7. Forest plot for Pain in Adults.
Figure 7. Forest plot for Pain in Adults.
Jcm 09 01025 g007
Table 1. Checklist of Reporting of Observational studies in Epidemiology (STROBE) criteria reported by the included studies. Each item was judged as “0” (not reported) or “1” (reported). The total score of each of included studies was also recorded.
Table 1. Checklist of Reporting of Observational studies in Epidemiology (STROBE) criteria reported by the included studies. Each item was judged as “0” (not reported) or “1” (reported). The total score of each of included studies was also recorded.
AuthorsAsl Aminabadi et al. [22]Tanja-Dijkstra et al. [9]Nunna et al. [23]Gujjar et al. [6]Niharika et al. [24]Al-Halabi et al. [20]Raghav et al. [7]Tanja-Dijkstra et al. [19]Shetty et al. [25]Al-Khotani et al. [26]Mitrakul et al. [27]Asvanund et al. [28]Bentsen et al. [21]Sweta et al. [2]
Section and item
1. Title and Abstract11111111111111
Introduction
2. Background11111111111111
3. Objectives11111111111111
Methods
4. Study design11111111111111
5. Setting11111110111111
6. Participants11111111111111
7. Variables11111111111111
8. Data sources/ Measurement11111111111111
9. Bias10000100011010
10. Study Size11111111111111
11.Quantitative variables11111111111111
12. Statistical Methods 11111111111111
Results
13. Participants11111111111111
14. Descriptive data11111111111111
15. Outcome data11111111111111
16. Main results11111111111111
17. Other analyses00111110101101
Discussion
18. Key results11111111111111
19. Limitations10000100011010
20. Interpretation11111111111111
21. Generalisability 11111111111111
Other information
22. Funding11111111111111
Total score2119202020222018202122202120
Mode Value: 27.4 (± 0.85).
Table 2. Details of each study.
Table 2. Details of each study.
Study
(Year)
JournalChildren Values (Ma, n, Ar)Adult Values (Ma, n, Ar)Dental ProcedureVR Device EquipmentMeasuring ScalesOutcomes
DAP
Asl Aminabadi et al. 2012 [22]J Dent Res Dent Clin Dent ProspectMa = 5.4
n = 120
Ar = 4–6
Restorative treatment in primary molars.I-glasses 920HR Ilixco, Inc. Menlo Park, CA, USA.MDASW-BFSThere was a significant decrease in pain perception and anxiety scores with the use of VR eyeglasses during dental treatment.
Tanja-Dijkstra et al 2014 [9] Plos One J Ma = 33.1
n = 69
Ar = Nr
A simulated dental treatment.Eyeglasses EVuzix iWear VR920 headset. Dual-core, 1.3GHz Intel processor with Nvidia GT
540 M graphics card.
MDAS Participants with higher
dental anxiety showed more interested in using VR during real dental treatment than those with lower levels of dental anxiety.
Dental patients who have a positive dental treatment experience thanks to VR, might have fewer unpleasant memories and thus be less likely to postpone
a future dental visit.
Nunna et al 2019 [23]J Dent Anesth Pain MedMa = Nr
n = 70
Ar = 7–11
- Counter-stimulation.
- Local anaesthesia administration with virtual reality distraction.
Lenovo smartphone, Sennheiser earphones, and ANTVR glasses.VCARSW-BFS Assessment of mean anxiety scores showed a significant difference in girls belonging to the VR group.
Gujjar et al. 2019 [6]Journal of Anxiety Disorders Ma = group 1, 25.3
group 2, 23
n = 30
Ar = Nr
Routine dental
treatment.
Eyeglasses. Dell XPS-8700 desktop with 4th Generation
Intel Core i7-4790 processor (8M Cache, up to 4.0 GHz) and ASUS
NVIDIA GEFORCE GTX 750 TI OC 2GB GDDR5 graphic card.
MDASVASThe results of this study provide evidence to support the efficacy of
VR in the treatment of dental phobia.
Study limitations:
- No blinding of patients or therapist in the interventions.
- VR compared with pamphlet information.
Niharika et al. 2018 [24]J Indian Soc Pedod Prev DentMa = Group A (7.17 ± 0.316)
Group B
(7.28 ± 0.300)
n = 40
Ar = 4–8
Routine dental care
(pulp therapy in mandibular primary molars). Local anaesthetic.
Google VR Box and Anti-Tank Virtual Reality 3D GlassesMDASW-BFSTwo groups. Childhood Anxiety-Related Disorders scores did not differ significantly between the two groups.
In both groups, a statistically significant difference was detected between the two treatment sessions (with and without VR).
Al-Halabi et al. 2018 [20]Anaesth Pain & Intensive CareMa = 7.4
n = 102
Ar = 7–10
Local anaesthesia in mandibular arch (inferior alveolar nerve block).Eyeglasses
(BlackBug™ Virtual Reality Glasses 3D VR Box
Headsets, China)
W-BFSThree groups. There was no significant difference in the anxiety of groups. There was a statistically significant difference in the anxiety and pain level in pulse rate.
Limitations:
- No blinding of the external investigator.
- The size of the VR Box was a little too big.
Raghav et al. 2016 [7]BMC Oral Health J Ma = Nr
n = 30
Ar = 19–45
1. Restorative dental procedure which may or may not be requiring local anaesthesia.
2. Extraction procedure requiring local anaesthesia.
Oculus development kit 2HMD, with a resolution of
960 × 1080 per eye and with a 100 degree field of view.
MDASVASTwo groups, VR (Idle, Mirror, Syringe,
Soundless Drill, Drill with Sound, 35- second exposures)
and informational pamphlet control groups.
Efficacy of VR in the
treatment of dental phobia in the setting of the dental procedure.
A limitation of the
present study is the absence of in vivo exposure therapy as gold standard
control group.
Tanja-Dijkstra et al. 2014 [19] Trials J Ma = Nr
n = 90
Ar = Nr
Dental treatmentEyeglasses.
A Sony personal 3D viewer, connected to an Alienware gaming laptop.
Participants can walk around in
the virtual environment by using a Zeemote JS1 thumb
stick controller.
MDAS This study compared two types of VR, natural environment and urban environment.
It included both referred patients and inhouse patients from a dental clinic and two separate procedures are described, one for each type of patient.
Shetty et al. 2019 [25]The Journal of Clinical Paediatric DentistryMa = Nr
n = 120
Ar = 5–8
Dental treatment
(vital pulp therapy)
Eyeglasses.
VR device (i-glasses 920HR, Ilixco Inc.,
Menlo Park, CA, USA)
MDASW-BFSTwo groups. The group with VR distraction, reported a decrease in the severity of anxiety.
Lower pain scores were observed in the VR group.
Al-Khotani et al. 2016 [26]Acta Odontologica ScandinavicaMa = 8.2
n = 56
Ar = 7–9
Dental examination, oral hygiene information, prophylaxis, restorative treatment.Eyeglasses.
DVD Players, gaming systems like Sony Play Station
Pro, Microsoft X-BOX, Nintendo WII
MDAS Two groups. VR and control group. Significant reduction in anxiety throughout the restorative procedure (including injection with local anaesthesia) in VR group.
Mitrakul et al. 2015 [27]European Journal of Paediatric DentistryMa = 6.9 ± 0.9
n = 42
Ar = 5–8
Restorative dental treatment in maxilla or mandible under local anaesthetic
Injection.
Eyeglasses. (Shenzhen Longway Vision Technology Co. Ltd, Shenzhen, China).FPS-RFLACCTwo groups.
Group 1 received treatment without wearing VR
in the first visit and wearing VR in a second visit. In Group 2, VR was used vice versa.
Asvanund et al. 2015 [28]Quintessence InternationalMa= 7 ± 0.8
n = 49
Ar = 5–8
Restorative dental
treatment (local anaesthetic injection in the maxillary arch or mandibular block).
Eyeglasses (Shenzhen Longway Vision Technology
Co. Ltd, Shenzhen, China).
FPS-RFLACCTwo groups.
The study assesses pain and anxiety without making a distinction.
The limitation of this study is that the FLACC score was assessed by playing back the video recording of each
visit, which was done by two paediatric dentists who could not be blinded to the child’s use of VR.
Bentsen et al. 2001 [21]Eur J of Pain- London Ma = Nr
n = 23
Ar = 20–49
Dental treatmentVideo glasses
(NV-HD 660 PanasonicTM)
VASThe study was a split-mouth, randomized design (2 dental filling). Dental treatments were performed without anaesthesia.
74% of patients would prefer VR if they were to receive a second dental filling.
Sweta et al. 2019 [2]Ann Maxillofac Surgery Ma = 39.72 ± 15.93.
n = 50
Ar = Nr
Local anaesthesia in patients undergoing a dental procedure.NrNCAQ Local anaesthesia and extractions reported the highest anxiety levels among the patients.
Limitations of this study:
- Small sample size.
- Patients were not in control of their VR environment.
n (Participant number); Ma (Mean age years); Ar (Age range years); Nr (Not reported); VAS (Visual Analogic Scale); W–BFS (Wong–Baker Faces Scale); FPS-R (Faces Pain Scale-Revised); FLACC (Consolability Scale); VRS (Verbal Rating Scale); MDAS (Modified Dental Anxiety Scale); NCAQ (Norman Corah’s anxiety questionnaire); VCARS (Venham’s Clinical Anxiety Rating Scale). DA (Dental Anxiety); P (Pain).
Table 3. Meta-analysis of anxiety. Characteristics of individual studies and meta-analysis. a: Standardized difference of means.
Table 3. Meta-analysis of anxiety. Characteristics of individual studies and meta-analysis. a: Standardized difference of means.
Age group/StudyYearScaleTestControlSMDaHeterogeneity
I2 (p-Value)
Public.bias
p-Value (Egger Test)
nMean ± sdnMean ± sdWeightMean95%-CIp-Value
Children + Adults (n = 9) 282 284 −0.54−1.58 to 0.490.30231% (p = 0.169)p = 0.399
Children (n = 7) (see Figure 5)
Shetty V2 [25]2019MDAS6011.3 ± 3.56016.5 ± 3.514.6%−1.48−1.88 to −1.07
Nunna M1 [23]2019VCARS350.57 ± 0.61351.00 ± 0.8414.5%−0.58−1.06 to −0.10
Niharika P1 [24]2018MDAS2019.6 ± 0.92017.3 ± 0.813.8%2.741.86 to 3.63
Mitrakul K2 [27]2015FLACC2126.0 ± 9.12128.0 ± 12.014.3%−0.18−0.79 to 0.43
Mitrakul K1 [27]2015FLACC2129.5 ± 11.32127.3 ± 10.614.3%0.20−0.41 to 0.80
Asl Aminabadi N1 [22]2012MDAS6012.6 ± 1.06017.7 ± 1.214.2%−4.46−5.14 to −3.78
Al-Khotani A [26]2016MDAS280.14 ± 0.36280.75 ± 0.5214.3%−1.34−1.93 to −0.76
Total 245 245 100%−0.74−1.99 to 0.510.24338% (p = 0.139)p = 0.536
Adults (n = 2) (see Figure 6)
Tanja-Dijkstra K [9]2014MDAS223.73 ± 0.65243.33 ± 0.8754.6%0.51−0.08 to 1.10
Gujjar KR1 [6]2019MDAS1518.3 ± 2.61518.8 ± 2.845.4%−0.18−0.90 to 0.54
Total 37 39 0.20−0.48 to 0.870.5670% (p = 0.317)-
Table 4. Meta-analysis of pain according to random effect models. Characteristics of individual studies and meta-analysis. a: Standardized difference of means.
Table 4. Meta-analysis of pain according to random effect models. Characteristics of individual studies and meta-analysis. a: Standardized difference of means.
Age group/StudyYearScaleTestControlSMDaHeterogeneity
I2 (p-Value)
Public.bias
p-Value (Egger Test)
nMean ± sdnMean ± sdWeightMean95%-CIp-Value
Children + Adults (n = 10) 355 355 −0.77−1.28 to −0.260.00365.8% (0.002)0.173
Children (n = 8) (see Figure 7)
Shetty V1 [25]2019W-BFS602.00 ± 0.50602.42 ± 1.4713.6%−0.38−0.74 to −0.0
Nunna M2 [23]2019W-BFS353.03 ± 2.22352.97 ± 2.4913.2%0.03−0.44 to 0.49
Niharika P2 [24]2018W-BFS202.56 ± 0.39205.22 ± 0.517.8%−5.71−7.16 to −4.25
Mitrakul K4 [27]2015FPS-R211.90 ± 2.93211.90 ± 3.3212.5%0.00−0.60 to 0.60
Mitrakul K3 [27]2015FPS-R210.86 ± 1.49211.62 ± 2.9412.5%−0.32−0.93 to 0.29
Asvanund Y2 [28]2015FPS-R492.23 ± 2.29492.46 ± 3.4613.5%−0.08−0.47 to 0.32
Asvanund Y1 [28]2015FPS-R491.57 ± 2.29493.04 ± 3.0813.5%−0.54−0.94 to −0.13
Asl Aminabadi N2 [22]2012W-BFS601.89 ± 0.65603.00 ± 0.8113.4%−1.50−1.91 to −1.10
Total 315 315 100%−0.82−1.42 to −0.220.00871% (p = 0.001)p = 0.180
Adults (n = 2) (see Figure 7)
Sweta VR [2]2019VAS251.28 ± 0.89252.60 ± 1.3852.3%−1.12−1.72 to −0.52
Gujjar KR2 [6]2019VAS1568 ± 101570 ± 1447.7%−0.18−0.90 to 0.53
Total 40 40 −0.67−1.58 to 0.240.1490% (p = 0.317)-

Share and Cite

MDPI and ACS Style

López-Valverde, N.; Muriel Fernández, J.; López-Valverde, A.; Valero Juan, L.F.; Ramírez, J.M.; Flores Fraile, J.; Herrero Payo, J.; Blanco Antona, L.A.; Macedo de Sousa, B.; Bravo, M. RETRACTED: Use of Virtual Reality for the Management of Anxiety and Pain in Dental Treatments: Systematic Review and Meta-Analysis. J. Clin. Med. 2020, 9, 1025. https://doi.org/10.3390/jcm9041025

AMA Style

López-Valverde N, Muriel Fernández J, López-Valverde A, Valero Juan LF, Ramírez JM, Flores Fraile J, Herrero Payo J, Blanco Antona LA, Macedo de Sousa B, Bravo M. RETRACTED: Use of Virtual Reality for the Management of Anxiety and Pain in Dental Treatments: Systematic Review and Meta-Analysis. Journal of Clinical Medicine. 2020; 9(4):1025. https://doi.org/10.3390/jcm9041025

Chicago/Turabian Style

López-Valverde, Nansi, Jorge Muriel Fernández, Antonio López-Valverde, Luis F. Valero Juan, Juan Manuel Ramírez, Javier Flores Fraile, Julio Herrero Payo, Leticia A. Blanco Antona, Bruno Macedo de Sousa, and Manuel Bravo. 2020. "RETRACTED: Use of Virtual Reality for the Management of Anxiety and Pain in Dental Treatments: Systematic Review and Meta-Analysis" Journal of Clinical Medicine 9, no. 4: 1025. https://doi.org/10.3390/jcm9041025

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop