Next Article in Journal
Behavioral Theory-Based Framework for Prediabetes Self-Care System—Design Perspectives and Validation Results
Previous Article in Journal
Associations between Depression, Anxiety, Fatigue, and Learning Motivating Factors in e-Learning-Based Computer Programming Education
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Effect of Distraction Intervention for Needle-Related Pain and Distress in Children: A Systematic Review and Meta-Analysis

Department of Nursing Science, Chungbuk National University, Cheongju 28644, Korea
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2021, 18(17), 9159; https://doi.org/10.3390/ijerph18179159
Submission received: 5 August 2021 / Revised: 24 August 2021 / Accepted: 28 August 2021 / Published: 31 August 2021
(This article belongs to the Topic Oncogenomics and Pediatric Health)

Abstract

:
A systematic review and meta-analysis conducted to evaluate the combined effect of distraction intervention for needle-related pain in order to provide the basis for developing an effective nursing intervention for children. We searched three electronic databases, PubMed, Embase, and CINAHL, for original articles published in the period from 1 January 2011 to 31 July 2019. In addition, a manual search was performed on the basis of references in the literature and the references of the articles in pursuit of comprehensive data until 10 December 2019. Meta-analysis was performed by the synthesis of the effect size, homogeneity, heterogeneity, and trim-and-fill method using MIX 2.0 Pro. Well-planned RCTs, single-center studies, high-quality studies, participants older than 10 years of age, and visual and clown distraction interventions were effective for needle-related pain and distress management among children. The results showed evidence supporting the effect of distraction interventions for children’s needle-related pain and distress. Through this review, strategies were identified to design better interventions to improve the outcomes.

1. Introduction

Several international pain-related expert groups, including the World Health Organization (WHO), contend that optimally effective pain management is a fundamental human right and good and ethical practice [1,2]. Therefore, healthcare providers, including pediatric nurses, have an ethical obligation and responsibility to use the up-to-date scientific ground to relieve any pain and fear that children may experience during treatment [3].
Both healthy and medically ill children may have difficulty with needle-related procedures [4]. Vaccination, blood collection, and intravascular injection are treatment procedures causing pain and fear that children may most frequently experience [4,5,6]. They may have such painful experiences in diverse places, including public health clinics and hospitals [1]. As children grow and receive routine vaccinations, they are regularly exposed to injections and pain. The current recommendations are that healthy children should get 20 to 30 vaccinations before the age of 18 years [7] and the majority of them are given within the first six years after birth [8]. Children with acute or chronic diseases frequently undergo needle-related procedures to determine their condition [9]. Children can have greater difficulty in understanding, accurately expressing, or overcoming the distress and pain they experience during the treatment process. Even a minimal medical procedure can arouse meaningful pain and distress among some children [10]. Needle-related procedures induce anxiety and fear in children due to the related pain and are also associated with negative emotions and feelings [11,12]. In addition, they often involve significant pain and distress for both children and parents [9]. The response to pain is a learned behavior and needle phobia can have many different causes; previous research reported that it was associated with any negative experience of pain [10]. Children remember the past painful experiences, and the negative memory of pain or fear caused by poor management during treatment can affect the subsequent treatment, consequently having an adverse long-term impact on future responses to pain [13]. It is; therefore, necessary to make an intervention for children that would properly, immediately, and effectively control needle-related pain and distress in nursing practice.
Because it is very important and difficult to evaluate and manage children’s pain, many hospitals regard pain as the fifth vital sign [14,15,16,17]. It has been suggested that pain be managed to give children the optimal comfort before, during, and after pain-causing treatments [2]. Although pain management for children has continually received attention for 20 years and its importance has been recognized, and the understanding of children’s pain has grown and considerable knowledge about its management has been accumulated through scientific exploration, many children still receive improper pain management, along with poor specific practice [1,2].
With the report that children’s pain and fear associated with needle insertion can adversely affect the process of treatment, various interventions have recently been introduced to relieve them. Albeit it is necessary to apply interventions that can effectively reduce pain and distress in children, it is difficult to apply them in clinical practice when special preparation and training and complicated procedures are required. As distraction techniques are often applied by nurses and parents and can relieve pain in children during treatment, they have recently attracted attention as practical nursing interventions [7].
Distraction intervention is defined as a cognitive or behavioral strategy to relieve pain, stress, and anxiety by distracting children from a harmful stimulus to what they do or something attractive through a non-pharmacological method of pain management [18]. Literature review presents evidence that distraction intervention is a simple and effective non-pharmacological pain management strategy for both healthcare providers and parents to relieve procedural pain in children with ease and at a low cost [5,17].
Pain management needs to be provided on a scientific ground, not based on personal opinions or beliefs; every healthcare provider needs to provide optimal pain management [2]. Nurses who apply any non-pharmacological intervention to relieve procedural pain, including distraction intervention, need to determine if it is appropriate and effective [3]. Some researchers presented a direction for further research and clinical practice related to distraction techniques through a critical review of the research on distraction interventions applied in children to relieve procedural pain [5]. Birnie et al. [7] performed a systematic review and meta-analysis related to the efficacy of distraction and hypnosis for relieving needle-related pain and distress in children and adolescents. However, there has been no meta-analysis of the effectiveness of distraction interventions for relieving needle-related pain and distress in children. Therefore, in this study, we performed a comprehensive comparative analysis of the effectiveness of distraction interventions to relieve needle-related pain and distress in children of diverse backgrounds to identify the most effective intervention program and suggest a direction for effective distraction interventions required to control.

2. The Review

2.1. Aims

This study aimed to estimate the overall effect size of distraction interventions on needle-related pain and distress in children and provide a ground for developing a nursing intervention that would effectively relieve needle-related pain and distress in children.

2.2. Design

We performed a systematic and comprehensive review and a meta-analysis to determine the effects of distraction interventions on needle-related pain and distress in children. The study was performed according to the systematic consideration handbook presented by Cochrane Collaboration [19] and the systematic consideration reporting guidelines presented in Systematic Reviews and Meta-Analysis by Littell, Corcoran, and Pillai [20].

2.3. Search Methods

Articles were first searched based on the criteria of population, intervention, comparison, outcome, and study design (PICO-SD). The population was limited to children, intervention to distraction intervention for relieving needle-related pain and distress, comparison to patients given usual care, outcome to needle-related pain and distress, and study design to randomized controlled trials (RCTs) and studies with a quasi-experimental design. An electronic database search was performed for national and international studies according to the inclusion and exclusion criteria to choose articles published in English that showed specific statistics of the intervention effects, including the mean and standard deviation and the sample number.
Three electronic databases—PubMed, Embase, and CINAHL (Supplementary Table S1)—were used to search for articles published from 1 January 2011 to 31 July 2019. In addition, a manual search was performed based on references in the literature and the references of the articles in pursuit of comprehensive data until 10 December 2019. Medical Subject Heading (MesH) terms, synonyms, and related terms for expressing “child” or “children”, “distraction” or “distraction intervention”, “pain” or “needle-related pain” or “venipuncture pain” or “immunization pain” according to PICO were identified in PubMed and adapted to the properties of each database by using such search functions as MesH terms, text words, logical operators, and truncation search properly.

2.4. Search Outcomes

Of the total 686 screened articles (456 from PubMed, 133 from the Embase, and 97 from CINAHL), 27 articles were finally included based on the exclusion criteria for overlapping, population, study design, intervention, and variables. Of these, 13 articles with two experimental groups were included in the analysis, with the interventions in each group separated from each other. As a result, a total of 45 studies were included in this analysis (Figure 1).

2.5. Quality Appraisal

The Checklist for Randomized Controlled Trials and the Checklist for Quasi-Experimental Studies in the Joanna Briggs Institute of Critical Appraisal Tools [21] were used to evaluate the quality of the selected RCTs and quasi-experimental studies, respectively, and to access the risk of bias of individual studies. As a pilot test, two researchers independently evaluated a total of four articles—two RCTs and two quasi-experimental studies—to investigate whether the quality evaluation tools had the same score for each item. For the items with inconsistent scores, the agreement was reached through discussion to clarify the evaluation criteria, followed by independent quality evaluation and comparison of the scores for the quality evaluation items in the final articles. The quality evaluation items for RCTs were as follows: random assignment, allocation concealment, treatment groups similarity, blinding of participants, blinding of treatment delivering, blinding of outcome assessor, similar treatment, follow-up complete and if not, adequately described and analyzed, the intention-to-treat analysis, same way of outcome measures, reliable way of outcome measures, appropriate statistical analysis, and appropriateness of the trial design. The score for each quality evaluation item ranged from 0 (No, Unclear) to 1 (Yes), with a total score of 13. The quality evaluation items for quasi-experimental studies were as follows: clarity of the cause and outcome effects, treatment groups similarity, similar treatment, comparison of the treated group, multiple measures, follow-up complete and if not, adequately described and analyzed, same way of outcome measures, reliable way of outcome measures, and appropriate statistical analysis. The score for each quality evaluation item ranged from 0 (No, Unclear) to 1 (Yes), with a total score of 9. The absence of tables or contents in the text, even if described in the abstract, was scored 0.

2.6. Data Abstraction

Throughout the course of data screening and collection, two researchers independently reviewed each study included in the analysis. The EXCEL program was used to exclude overlapping articles. Next, titles and abstracts were reviewed to identify articles that met the inclusion criteria; if it was difficult to determine if they met the criteria on the basis of titles or abstracts alone, the original text was referenced to determine inclusion. The bibliographical information of every article was coded in the EXCEL program and managed in the identical way, with the excluded articles recorded by stages. For the final articles, authors, publication year, country, number of centers, population size, participants’ age, Institutional Review Board (IRB) approval status, funding status, research design, intervention type and characteristics, intervention in the control group, outcome variables, and quality evaluation scores were collected and recorded in the coding table.

2.7. Synthesis

The characteristics of the studies were presented as frequencies or the mean and standard deviation. MIX 2.0 Pro (Ver. 2.0.1.6, BiostatXL, 2017, BiostatXL, CA, USA) was used for statistical analysis of the method of combining the effect size. For the effect size, standardized mean difference and 95% confidence intervals (CIs) were used for the same outcome variable and the weight of each effect size was estimated using the inverse number of variances [22]. The random effects model for resetting weighted values, considering the variability among the participants in different studies and inter-study heterogeneity, was used to estimate the overall effect. A sub-analysis was performed on the basis of the participants’ age and the following study characteristics: number of centers, IRB approval and funding status, study design, intervention type, and scores in the quality evaluation. For the heterogeneity, Higgin’s I2 for the actual variance of the observed total variance or the interobserver variance rate was estimated [23], with I2 > 50% regarded as indicator of heterogeneity [19]. The funnel plot was used to modify the standardized mean difference to test the studies for publication bias [24].

3. Results

3.1. Study Characteristics

There were 14 studies published before 2015 and 31 after 2015; six in East Asian countries, such as India and Taiwan, 12 in American countries, such as the United States, Canada, and Brazil, and 27 in other countries. There were 40 single-center studies and 36 had a population of 50 to less than 300 persons. In 41studies, the mean age of the population was <10 years and four studies (study ID: 28, 29, 30, 33) had participants aged ≥10 years. Forty-one studies had IRB approval and 15 were funded. There were 41 RCTs, including two crossover RCTs, and four quasi-experimental studies. Regarding distraction intervention types, visual distraction was used in 19 studies, audial distraction in seven, clowns in two, cognitive behavioral therapy in two, multiple methods, including touch and acupressure, in 13, and local anesthesia in two studies. Thirty-eight studies involved no intervention or usual care given by parents in the control group. Characteristics of the included studies are presented in Table 1.

3.2. Methodological Quality

For the quality evaluation, two RCTs and two quasi-experimental studies among the final 45 studies were included in the pilot test for scores. There was 90.5% consistency, and it was agreed that inconsistent items would be scored 0 if the homogeneity was “not tabulated in the results of the text though described in the abstract”. In the methodological quality evaluation, 41 RCTs scored 9.05 (range = 7–12) and four quasi-experimental studies scored 7.00 (range = 5–9). As for the qualitative level of the studies, the conclusion had no possibility of being changed.

3.3. Effect of Distraction Intervention on Needle-Related Pain in Children

In the 41 studies, the mean of the differences between the pre-test and the post-test in both groups, the standard deviation of the differences, and the sample size were used to estimate standardized mean differences, which were presented in a synthesis forest plot (Figure 2). Since the characteristics of the included study were heterogeneous, two methods were used to calculate the combined effect size in this study. First, for the calculation of the combined effect size in this study, hedge’s g was weighted the standard deviation by its sample size for corrected effect size, because Cohen’s d tends to overestimate the population variance, especially for small samples (n < 20) [25]. Second, the random effect model was used rather than the fixed effects model to analyze the random error of the effect sizes of each study and the error according to the characteristics of the study [26].
For the children, the overall effects of needle-related pain were at the medium level of −0.50 (95% CI = −0.82 to −0.18) and the needle-related pain decreased significantly after distraction intervention (Z = −3.09, p = 0.002). The effect size was highly heterogeneous: I2 = 92.1% [24]. Given that exploratory explanation of the background for the heterogeneity of the effect size was required, a sub-analysis was performed on the basis of the participants’ age and the following study characteristics: number of centers, IRB approval and funding status, study design, intervention type, and scores in the quality evaluation (Table 2).
For the 26 studies with participants aged ≥10 years, the overall effects were at a high level of –1.07 (95% CI = −1.60 to −0.53) and the needle-related pain decreased significantly after distraction intervention (Z = −3.92, p < 0.001). For the 38 single-center studies, the overall effects were at a medium level of −0.51 (95% CI = −0.85 to −0.17) and were statistically significant (Z = −2.96, p = 0.003). For the 37 studies without IRB approval, the overall effects were at a medium level of –0.64 (95% CI = −0.89 to −0.39) and the needle-related pain decreased significantly after distraction intervention (Z = −5.04, p < 0.001). For the 15 studies with funding, the overall effects were at a high level of –0.83 (95% CI = −1.20 to −0.46) and the needle-related pain decreased significantly after distraction intervention (Z = −4.41, p < 0.001).
For the total 37 RCTs, including the two crossover RCTs, the overall effects were at a medium level of −0.61 (95% CI = −0.92 to −0.30) and the needle-related pain decreased significantly after distraction intervention (Z = −3.85, p < 0.001). While the overall effects of multiple distraction methods, including audial, cognitive, and touch and acupressure, were statistically insignificant, those of the visual distraction method was at a high level of −0.76 (95% CI = −1.15 to −0.37), which was statistically significant (Z = −3.86, p < 0.001). The overall effects of clowns were at a high level of −3.64 (95% CI = −4.76 to −2.52), which was statistically significant (Z = −6.36, p < 0.001).
For the 17 studies with a score ≥10 in the quality evaluation, the overall effects were at a medium level of −0.65 (95% CI = −0.98 to −0.32) and the needle-related pain decreased significantly after distraction intervention (Z = −3.84, p < 0.001).

3.4. Effect of Distraction Intervention on Needle-Related Distress in Children

In the 45 studies, besides needle-related pain in children as a principal variable, observer pain was measured in parents and nurses, anxiety in children, parents, and nurses, and blood cortisol in children (Table 3). Twenty-three studies involved observer pain (parents), 28 observer pain (nurses), three child-reported anxiety, and five cortisol measurement; the overall effects of these variables were statistically insignificant after distraction intervention. For the 15 studies related to observer anxiety (parents), the overall effects were at a medium level of −0.68 (95% CI = −1.13 to −0.23) and observer anxiety (parents) decreased significantly after distraction intervention (Z = −2.96, p = 0.003). For the 14 studies related to observer anxiety (nurses), the overall effects were at a high level of −1.03 (95% CI = −1.67 to −0.40) and observer anxiety (nurses) decreased significantly after distraction intervention (Z = −3.20, p = 0.001).

3.5. Publication Bias Analysis

In the publication bias analysis for validation, funnel plot analysis was first performed, as usually recommended, to identify publication biases [20]. As seen in Figure 3, the effect size was visually symmetrical in a funnel shape. Nfs (fail-safe number), one of the criteria for reliability of meta-analysis results, was estimated to secure the reliability of the results. Nfs is a bias that can occur when the mean effect size is only based on the results of the published article without unpublished results, and Orwin’s (1983) method was applied in this study. For the 41 studies concerning needle-related pain in children after distraction intervention, the combined effect size hedge’s g = −0.50 and Cohen’s (1977) small effect size d = 0.2 were used to estimate the Nfs, and 61.5 additional studies were required to reduce the effect size to d = 0.2. Based on this finding, it cannot be said that the analyzed studies had no publication error, but the errors were not so significant as to reverse the entire results (Figure 3).

4. Discussion

A total of 45 studies published from 2011 to 2019 were included in the meta-analysis for the effect size of distraction intervention on needle-related pain and distress in children. The results can be discussed as follows.
Of the 45 studies, 32 (69.6%) were published after 2015 and the majority of them were published in the United States, Canada, and Brazil (12; 26.1%). Twenty-six articles on distraction and 13 on hypnosis for needle-related pain and distress in children and adolescents [7] indicated relatively strong effects. We analyzed the effects of distraction intervention alone, excluding those of hypnosis, among psychological interventions. From the results, the research on distraction intervention had increased after 2015. Distraction intervention still failed to be used in many countries. This supports the finding that despite the far-reaching extension of knowledge about pain management and the development of effective ground-based strategies, pain management using distraction intervention is not positively applied in clinical practice and there are clear differences between the knowledge about pain management and practice [2,27].
Of the 45 studies, 41 (89.1%) were conducted in participants aged <10 years on average and 17 (37.0%) had participants aged ≥10 years. This implies that for the study design, consideration was given to the fact that young children might be vulnerable to fear of needle insertion. However, it is necessary to pay attention to the decrease of the overall effects in children aged ≥10 years in the subgroup analysis for needle-related pain. Since children aged ≥10 years may experience needle-related pain and distress in various settings, it is necessary to conduct an RCT on distraction intervention in this population.
The meta-analysis found that the overall effects of needle-related pain were at a medium level and that the needle-related pain and distress decreased significantly after distraction intervention. Regarding the type of distraction interventions, visual distraction and clown programs were effective in relieving needle-related pain and distress; therefore, it is necessary to actively develop methods of relieving needle-related pain and distress taking these intervention types into account. There are diverse settings and methods of using distraction intervention [9]. Recently, some researchers successfully used a humanoid robot as a cognitive-behavioral intervention to relieve needle-related pain and distress during annual flu vaccination [28]. This implies the development of applicable technology for new distraction interventions to relieve needle-related pain and distress. It is necessary to accumulate the effects of such interventions as evidence through strict experimental research and identify a way of applying it actively in practice.
The RCTs that were conducted at a single center and scored ≥10 in the quality evaluation showed medium to high levels of effects, which were statistically significant. Journal editors have recently tried to apply diverse methods for securing the quality and transparency of studies. One of the methods is to ensure that researchers comply with the guidelines for reporting results; the items in the guidelines are consistent with those in the quality evaluation [29]. Evidence-based nursing requires that the most valid, reliable, high-quality results obtained using scientific methods be applied in practice; to achieve this, researchers should report their results according to the relevant reporting guidelines [30]. Therefore, considering each item in the reporting guidelines according to the study design from the stage of study planning to the writing of an article is expected to further improve the quality of studies.
In the 45 studies besides needle-related pain directly measured in children, observer pain and anxiety were measured in parents and nurses. After distraction intervention, the overall effects on needle-related pain and observer pain were statistically insignificant. Similarly, Uman et al. [17] failed to get clear effects on distress. In contrast, observer anxiety decreased significantly at medium and high levels in parents and nurses. When a child experiences distress, the parent and the healthcare provider may often feel anxious, helpless, and guilty, accept the procedure with greater difficulty, and experience stress. In many cases, children experiencing needle-related pain and distress are given distraction intervention by their parents and the nurse as a healthcare provider and their emotions can be transmitted to the children [9]. Recently, Birnie et al. [9] found that distraction positively affected self-reported pain and distress, observer-reported pain and distress, the behavioral measures of distress, and the heart rate. In this respect, the finding of this study that distraction intervention was significantly effective in relieving anxiety in parents and nurses as observers is very meaningful. It is necessary to find a way of relieving distress in parents and nurses while developing distraction interventions to relieve needle-related pain and distress in children through repetitive research. Since children’s age, developmental level, temperament, and treatment type can affect their ability to cope with the procedural course, it is necessary to pay more attention to individual children’s preferences and disposition in pursuit of optimal results. Further research should be conducted to examine distress following distraction interventions appropriate for children’s characteristics [5,27]. Another research can be conducted to compare needle-related pain and distress by stages because it can differ from distraction interventions before, during, and after a needle-related procedure.
This study has the limitations of a meta-analysis; it is difficult to include unpublished articles and those not retrieved in the analysis. Because some of the included studies were narrow in scope, there were limitations in generalizing the results. It was a necessary process for planning new interventions in the future by analyzing various interventions, but the low statistical power of the meta-analysis results obtained for some variables of the interventions analyzed was. Furthermore, articles not written in English were excluded, which limits the interpretation. Other limitations are that few studies were conducted in older children or adolescents and that limited information about the distraction intervention procedure was presented for other researchers to conduct repeated research and was insufficient to solve any problems with the usability of distraction intervention in clinical practice. In addition, no studies have been conducted to determine whether applying distraction interventions was effective in terms of time and costs.

5. Conclusions

This study aimed to determine the effect size of distraction intervention on needle-related pain and distress in children, analyze the contents of the intervention, and present a direction for its practical application and further research through a meta-analysis of 45 studies published from 2011 to 2019. Despite several limitations of the previous studies, distraction intervention was significantly effective in relieving needle-related pain and distress in children. Well-planned RCTs conducted at a single-center, high-quality study, participants older than 10 years of age, and visual and clown distraction interventions were effective for needle-related pain and distress management among children. Based on these results, the following suggestions can be made:
First, it is necessary to examine the effects of distraction intervention on needle-related pain and distress more scientifically and increase intervention research to expand it into the area of nursing.
Second, it is necessary to apply it to various pediatric nursing settings with various types of longer intervention based on the theoretical framework; randomized experimental study, which is a study design effective in applying the results, is required. Methodological strictness is required in terms of study design, implementation, and reporting.
Third, it is necessary to provide education and training to parents and healthcare professionals, including nurses, to narrow the gap between theories and practice concerning distraction intervention.

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/ijerph18179159/s1, Table S1. Search Strategy for Databases.

Author Contributions

Conceptualization, M.-K.C. and M.-Y.C.; methodology, M.-K.C.; formal analysis, M.-K.C.; writing—original draft preparation, M.-K.C. and M.-Y.C.; writing—review and editing, M.-K.C. and M.-Y.C. Both authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Curtis, S.; Wingert, A.; Ali, S. The Cochrane Library and Procedural Pain in Children: An Overview of Reviews. Evid. Based Child Health A Cochrane Rev. J. 2012, 7, 1363–1399. [Google Scholar] [CrossRef]
  2. Petovello, K. Pediatric Procedural Pain Management: A Review of the Literature. Int. J. Child Youth Fam. Stud. 2012, 3, 569–589. [Google Scholar] [CrossRef]
  3. Czarnecki, M.L.; Turner, H.N.; Collins, P.M.; Doellman, D.; Wrona, S.; Reynolds, J. Procedural Pain Management: A Position Statement with Clinical Practice Recommendations. Pain Manag. Nurs. 2011, 12, 95–111. [Google Scholar] [CrossRef] [PubMed]
  4. Taddio, A.; Ipp, M.; Thivakaran, S.; Jamal, A.; Parikh, C.; Smart, S.; Sovran, J.; Stephens, D.; Katz, J. Survey of the Prevalence of Immunization Non-Compliance due to Needle Fears in Children and Adults. Vaccine 2012, 30, 4807–4812. [Google Scholar] [CrossRef] [Green Version]
  5. Koller, D.; Goldman, R.D. Distraction Techniques for Children Undergoing Procedures: A Critical Review of Pediatric Research. J. Pediatr. Nurs. 2012, 27, 652–681. [Google Scholar] [CrossRef]
  6. Blount, R.L.; Piira, T.; Cohen, L.L.; Cheng, P.S. Pediatric Procedural Pain. Behav. Modif. 2006, 30, 24–49. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  7. Birnie, K.A.; Noel, M.; Parker, J.A.; Chambers, C.T.; Uman, L.S.; Kisely, S.R.; McGrath, P.J. Systematic Review and Meta-Analysis of Distraction and Hypnosis for Needle-Related Pain and Distress in Children and Adolescents. J. Pediatr. Psychol. 2014, 39, 783–808. [Google Scholar] [CrossRef] [Green Version]
  8. Buscemi, N.; Vandermeer, B.; Curtis, S. The Cochrane Library and Procedural Pain in Children: An Overview of Reviews. Evid.-Based Child Health A Cochrane Rev. J. 2008, 3, 260–279. [Google Scholar] [CrossRef]
  9. Birnie, K.A.; Noel, M.; Chambers, C.T.; Uman, L.S.; Parker, J.A. Psychological Interventions for Needle-related Procedural Pain and Distress in Children and Adolescents. Cochrane Database Syst. Rev. 2018. [Google Scholar] [CrossRef]
  10. Noel, M.; McMurtry, C.M.; Chambers, C.T.; McGrath, P.J. Children’s Memory for Painful Procedures: The Relationship of Pain Intensity, Anxiety, and Adult Behaviors to Subsequent Recall. J. Pediatr. Psychol. 2009, 35, 626–636. [Google Scholar] [CrossRef] [Green Version]
  11. Cummings, E.A.; Reid, G.J.; Finley, G.A.; McGrath, P.J.; Ritchie, J.A. Prevalence and Source of Pain in Pediatric Inpatients. Pain 1996, 68, 25–31. [Google Scholar] [CrossRef]
  12. Kazak, A.E. Comprehensive Care for Children with Cancer and their Families: A Social Ecological Framework Guiding Research, Practice, and Policy. Child. Serv. Soc. Policy Res. Pract. 2001, 4, 217–233. [Google Scholar] [CrossRef]
  13. Cramton, R.E.; Gruchala, N.E. Managing Procedural Pain in Pediatric Patients. Curr. Opin. Pediatr. 2012, 24, 530–538. [Google Scholar] [CrossRef]
  14. Casey, G. Pain-the Fifth Vital Sign. Kai Tiaki Nurs. N. Z. 2011, 17, 24. [Google Scholar]
  15. Erdoğan, Ç.; Turan, T.; Pınar, B. The Effect of Maternal Voice for Procedural Pain in Paediatric Intensive Care Unit: A Randomised Controlled Trial. Intensive Crit. Care Nurs. 2020, 56, 102767. [Google Scholar] [CrossRef] [PubMed]
  16. Stinson, J.; Yamada, J.; Dickson, A.; Lamba, J.; Stevens, B. Review of Systematic Reviews on Acute Procedural Pain in Children in the Hospital Setting. Pain Res. Manag. 2008, 13, 465891. [Google Scholar] [CrossRef]
  17. Uman, L.S.; Chambers, C.T.; McGrath, P.J.; Kisely, S. A Systematic Review of Randomized Controlled Trials Examining Psychological Interventions for Needle-Related Procedural Pain and Distress in Children and Adolescents: An Abbreviated Cochrane Review. J. Pediatr. Psychol. 2008, 33, 842–854. [Google Scholar] [CrossRef] [PubMed]
  18. Kleiber, C.; McCarthy, A.M. Evaluating Instruments for a Study on Children’s Responses to a Painful Procedure when Parents are Distraction Coaches. J. Pediatr. Nurs. 2006, 21, 99–107. [Google Scholar] [CrossRef]
  19. Higgins, J.P.; Altman, D.G.; Gøtzsche, P.C.; Jüni, P.; Moher, D.; Oxman, A.D.; Savović, J.; Schulz, K.F.; Weeks, L.; Sterne, J.A. The Cochrane Collaboration’s Tool for Assessing Risk of Bias in Randomised Trials. BMJ 2011, 343, d5928. [Google Scholar] [CrossRef] [Green Version]
  20. Littell, J.H.; Corcoran, J.; Pillai, V. Systematic Reviews and Meta-Analysis; Oxford University Press: Oxford, UK, 2008. [Google Scholar]
  21. JBI. Checklist for Case Control Studies. Available online: https://jbi.global/critical-appraisal-tools (accessed on 21 November 2020).
  22. Borenstein, M.; Hedges, L.V.; Higgins, J.P.; Rothstein, H.R. Introduction to Meta-Analysis; John Wiley & Sons, Ltd.: Hoboken, NJ, USA, 2011. [Google Scholar]
  23. Mavridis, D.; Salanti, G. How to assess publication bias: Funnel plot, trim-and-fill method and selection models. Evid. Based Ment. Health 2014, 17, 30. [Google Scholar] [CrossRef]
  24. Lakens, D. Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for T-Tests and ANOVAs. Front. Psychol. 2013, 4, 863. [Google Scholar] [CrossRef] [Green Version]
  25. Bell, A.; Fairbrother, M.; Jones, K. Fixed and Random Effects Models: Making an Informed Choice. Qual. Quant. 2019, 53, 1051–1074. [Google Scholar] [CrossRef] [Green Version]
  26. Im, E.S.; Kim, J.S. Distraction Techniques for Children Undergoing Procedures: A Critical Review of Korean Intervention Research. Child Health Nurs. Res. 2014, 20, 340–349. [Google Scholar] [CrossRef]
  27. Beran, T.N.; Ramirez-Serrano, A.; Vanderkooi, O.G.; Kuhn, S. Reducing Children’s Pain and Distress Towards Flu Vaccinations: A Novel and Effective Application of Humanoid Robotics. Vaccine 2013, 31, 2772–2777. [Google Scholar] [CrossRef] [PubMed]
  28. PLOS Medicine Editors. Observational Studies: Getting Clear about Transparency. PLoS Med. 2014, 11, e1001711. [Google Scholar] [CrossRef] [Green Version]
  29. Oermann, M.H.; Turner, K.; Carman, M. Preparing Quality Improvement, Research, and Evidence-Based Practice Manuscripts. Nurs. Econ. 2014, 32, 57. [Google Scholar]
  30. Moadad, N.; Kozman, K.; Shahine, R.; Ohanian, S.; Badr, L.K. Distraction using the BUZZY for Children during an IV Insertion. J. Pediatr. Nurs. 2016, 31, 64–72. [Google Scholar] [CrossRef]
Figure 1. Search outcomes.
Figure 1. Search outcomes.
Ijerph 18 09159 g001
Figure 2. The effect of distract intervention on needle-related pain of the child. ES: Effect size, CI = Confidence interval.
Figure 2. The effect of distract intervention on needle-related pain of the child. ES: Effect size, CI = Confidence interval.
Ijerph 18 09159 g002aIjerph 18 09159 g002b
Figure 3. Publication bias analysis.
Figure 3. Publication bias analysis.
Ijerph 18 09159 g003
Table 1. Characteristics of the included studies.
Table 1. Characteristics of the included studies.
Study IDAuthor (Year)CountryCenterProcedure (Sample Size)Age (Year)IRBFundDesignType of Intervention
(Characteristics)
ControlOutcomeQuality Assessment
1Canbulat et al. (2014)Turkey1 hospitalPhlebotomy (n = 126), E: 63, C: 637–11 (8.8 ± 1.5)YesYesRCTVisual distraction (distraction cards)No interventionPain: WB-FACES, Anxiety: CFS9
2Canbulat et al. (2014)Turkey1 hospitalPhlebotomy (n= 125), E: 62, C: 637–11 (8.8 ± 1.5)YesYesRCTVisual distraction
(Kaleidoscope)
No interventionPain: WB-FACES (0–10), Anxiety: CFS (0–4)9
3Hsieh et al. (2017)Taiwan1 hospitalIV (n = 68), E: 35, C: 336–12 (E:8.3 ± 1.6, C:7.8 ± 1.5)YesNoQuasiCognitive-behavioral program
(pre: Educational photo book 10–15 min, during: Favorite music video 5–10 min (40–60 dB))
Routine carePain: NRS (0–10)9
4Rimon et al. (2016)Israel1 hospitalIV (n = 53), E: 29, C: 242–15 (E:5.6 ± 2.8, C:6.9 ± 3.4)NoYesRCTMedical clown (pre: Distraction via humor, during: Distraction via humor 15 min)Parent regular distraction and comfortPain: VAS, FPS-R, S-cortisol7
5Kuo et al. (2016)Taiwan1 medical centerIV (n = 184), E: 92, C: 923–7 (E:4.5, C: 4.51)YesNoRCTVisual distraction (story book reading)Oral instructionOSBD-R (1–4)8
6Kuo et al. (2016)Taiwan1 medical centerIV (n = 184), E: 92, C: 923–7 (E:4.5, C:4.51)YesNoRCTVisual distraction (cartoon viewing (iPad))Oral instructionOSBD-R (1–4)8
7Sahiner and Bal (2016)Turkey1 hospitalPhlebotomy (n = 60), E: 30, C: 306–12 (9.1 ± 1.6)YesNoRCTVisual distraction (distraction cards)Routine carePain: WB-FACES, Anxiety: CFS11
8Sahiner and Bal (2016)Turkey1 hospital60 children for Phlebotomy (n = 60), E: 30, C: 306–12 (9.1 ± 1.6)YesNoRCTAudial distraction (music playback)Routine carePain: WB-FACES, Anxiety: CFS11
9Sahiner and Bal (2016)Turkey1 hospitalPhlebotomy (n = 60), E: 30, C: 306–12 (9.1 ± 1.6)YesNoRCTTouch distraction (balloon inflation)Routine carePain: WB-FACES, Anxiety: CFS11
10Miller et al. (2016)Australia1 hospitalIV (n = 40), E: 20, C: 203–12 (E:8.1 ± 3.0, C:7.0 ± 2.3)YesNoRCTVisual distraction (PSP: Sony handheld PSP, age- appropriate games)Standard distractionPain: WB-FACES, VAS (0–10), FLACC (0–10)11
11Miller et al. (2016)Australia1 hospitalIV (n = 40), E: 20, C: 203–12 (E:8.1 ± 3.0, C:6.4 ± 2.9)YesNoRCTMultisensory stimuli (animated visual content, sound) and vibration; Ditto-D: PSP + interactive games)Standard distractionPain: WB-FACES, VAS (0–10), FLACC (0–10)11
12Miller et al. (2016)Australia1 hospitalIV (n = 39), E: 19, C: 203–12 (E: 8.1 ± 3.0, C: 6.2 ± 2.8)YesNoRCTMultisensory stimuli (animated visual content, sound) and vibration; Ditto-PP: interactive procedural preparation story)Standard distractionPain: WB-FACES, VAS (0–10), FLACC (0–10)11
13Miller et al. (2016)Australia1 hospitalIV (n = 39), E: 19, C: 203–12 (E:8.1 ± 3.0, C:6.0 ± 2.3)YesNoRCTMultisensory stimuli (animated visual content, sound) and vibration; Ditto-C: story + interactive stories and games)Standard distractionPain: WB-FACES, VAS (0–10), FLACC (0–10)11
14Sadeghi et al. (2018)Iran1 hospitalIV (n = 60), E: 30, C: 304-6NoYesQuasiTouch distraction (press a soft ball with the opposite handNo interventionPain: WB-FACES7
15Aydin et al. (2016)Turkey1 hospitalPhlebotomy (n = 60), E: 30, C: 307–12 (E:9.3 ± 1.8, C:9.9 ± 2.0)YesNoRCTVisual distraction (distraction cards)Routine carePain: WB-FACES, Anxiety: CFS9
16Aydin et al. (2016)Turkey1 hospitalPhlebotomy (n = 60), E: 30, C: 307–12 (E:9.7 ± 2.2, C:9.9 ± 2.0)YesNoRCTTouch distraction (soft ball)Routine carePain: WB-FACES, Anxiety: CFS9
17Aydin et al. (2016)Turkey1 hospitalPhlebotomy (n = 60), E: 30, C: 307–12 (E:9.7 ± 2.4, C:9.9 ± 2.0)YesNoRCTTouch distraction (balloon inflation)Routine carePain: WB-FACES, Anxiety: CFS9
18Pour et al. (2017)Iran2 hospitalPhlebotomy (n = 80), E: 40, C: 406–12 (8.0 ± 1.8)YesNoRCTLocal anesthesia (2g EMLA cream, 45 min)Routine carePain: FLACC (0–10)8
19Pour et al. (2017)Iran2 hospitalPhlebotomy (n = 80), E: 40, C: 406–12 (8.0 ± 1.8)YesNoRCTAcupressure (Yintang (extra 1) and Laogong (P-8))Routine carePain: FLACC (0–11)8
20McCarthy et al. (2014)USA3 hospitalsIV (n = 232), E: 116, C: 1164–10 (7.3 ± 1.9)YesYesRCTVisual distraction (enhanced: RA distraction coaching)Patent directed distractionPain: Oucher (0–10), PRCD (1–7), OSBD-R (1–4), Salivary cortisol8
21McCarthy et al. (2014)USA3 hospitalsIV (n = 240), E: 124, C: 1164–10 (7.3 ± 1.9)YesYesRCTVisual distraction (professional: RA distraction support)Patent directed distractionPain: 23 Oucher 24 (0–10), PRCD25 (1–7), OSBD-26R (1–4), Salivary cortisol8
22Hillgrove-Stuart et al. (2013)Canada1 clinicIV (n = 66), E: 32, C: 341–2 (1.3 ± 0.2)YesYesRCTTouch distraction (parent directed distraction)Routine carePain: M29BPS (0–10), 30MAISD11
23Hillgrove-Stuart et al. (2013)Canada1 clinicIV (n = 67), E: 33, C: 341–2 (1.3 ± 0.2)YesYesRCTTouch distraction (RA directed distraction)Routine carePain: MBPS (0–10), MAISD11
24Cohen et al. (2015)USA1 hospitalImmunization (n = 60), E: 30, C: 304–6.5 (4.8 ± 0.8)YesYesRCTVisual distraction (age- appropriate movie+ parent teaching program)Routine carePain: CAPS, parent knowledge, parent behavior, children behavior11
25Cohen et al. (2015)USA1 hospitalImmunization (n = 60), E: 30, C: 304–6.5 (4.8 ± 0.8)YesYesRCTVisual distraction+ education (age-appropriate movie)Routine carePain: CAPS, parent knowledge, parent behavior, children behavior11
26Inal and Kelleci (2012)Turkey1 hospitalIV (n = 123), E: 61, C: 626–12 (E:9.4 ± 2.1, C:9.3 ± 1.8)YesNoRCTVisual distraction (distraction cards)Routine carePain: CAPS (0–5), FPS (0–10), anxiety12
27Jibb et al. (2018)Canada1 hospitalSCP needle insertion for chemo (EMLA 60 min) (n = 39), E: 18, C: 214–9 (E:6.3 ± 1.4, C:6.1 ± 1.5)YesYesRCTWeb-based service cognitive-behavioral program (MEDiPORT humanoid robot)Active distraction robot with dancing and singingPain: FPS, NRS (0–10), fear: CFS (0–10), distress: BAADS (1–5)9
28Kristja and Kristja (2011)Canada1 schoolImmunization (n = 77), E: 38, C: 3913–15 (14 ± 0.18)YesNoRCTAudial distraction (musical distraction: With headphone)Standard carePain: VAS (0–10), anxiety, fear (0–10)7
29Kristja and Kristja (2011)Canada1 schoolImmunization (n = 80), E: 41, C: 3913–15 (14 ± 0.18)YesNoRCTAudial distraction (musical distraction: without headphone)Standard carePain: VAS (0–10), anxiety, fear (0–10)7
30Nilsson et al. (2015)Sweden3 schoolHPV vaccination (n = 37), E: 37, C: 3711–12YesYesCrossover RCTVisual distraction (relaxation and guided imagery)Standard carePain: CAS (0–10), FAS (0.04–0.97), salivary cortisol10
31Oliveira et al. (2016)Brazil1 hospitalIV (n = 40), E: 22, C: 186–12 (E:8.3 ± 2.1, C:8.7 ± 1.8)YesNoCrossover RCTAudiovisual distraction (Disney movie)Routine carePain: VAS (0–10), FPS (0–10), PCS-C (13 item, 1–5), CSS (35 item, 0–4)8
32Hartling et al. (2013)Canada1 hospitalIV (n = 42), E: 21, C: 213–11YesYesRCTAudial distraction (musical distraction: with headphone)Standard carePain: FPS-R (0–10), OSBD-R, anxiety10
33Redfern et al. (2018)USA1 hospitalVaccination (n = 50), E: 25, C: 253–18 (E:10.7 ± 4.7, C:10.5 ± 4.7)YesNoRCTVisual distraction (distraction cards)No interventionPain: WB-FACES, parent satisfaction (0–10)12
34Risaw et al. (2017)India1 hospitalPhlebotomy (n = 210), E: 105, C: 1054–6 (4.8 ± 0.8)YesNoRCTVisual distraction (distraction cards)Standard carePain: FLACC (0–10)10
35Bergomi et al. (2018)Italy1 hospitalIV (n = 75), E: 36, C: 395–12 (E:8.3 ± 2.2, C:9.4 ± 2.3)YesNoRCTTouch distraction (BUZZY device)Standard carePain: WB-FPS, CEMS, parent anxiety: NRS (0–10)7
36Bergomi et al. (2018)Italy1 hospitalIV (n = 76), E: 37, C: 395–12 (E:9.4 ± 2.0, C:9.4 ± 2.3)YesNoRCTVisual distraction (cartoon)Standard carePain: WB-FPS, CEMS, parent anxiety: NRS (0–10)7
37Bergomi et al. (2018)Italy1 hospitalIV (n = 77), E: 38, C: 395–12 (E:8.6 ± 2.1, C:9.4 ± 2.3)YesNoRCTTouch distraction (BUZZY device + cartoon)Standard carePain: WB-FPS, CEMS, parent anxiety: NRS (0–10)7
38Meiri et al. (2016)Israel1 hospitalIV (n = 66), E: 33, C: 332–10 (E:5.4 ± 2.6, C:5.5 ± 2.6)YesYesRCTMedical clown (medical clown)Standard carePain: VAS (0–10), anxiety (0–10), crying time 7
39Meiri et al. (2016)Israel1 hospitalIV (n = 67), E: 34, C: 332–10 (E:5.0 ± 2.4, C:5.5 ± 2.6)YesYesRCTLocal anesthesia (EMLA cream, 50min)Standard carePain: VAS (0–10), anxiety (0–10), crying time 7
40Karakaya and Duygu G€ozen (2016)Turkey1 hospitalIV (n = 144), E: 72, C: 727–12YesNoRCTVisual distraction (kaleidoscope)No interventionPain: FPS (0–10), pulse oximeter, thermometer7
41Cerne et al. (2014)Italy1hospitalImmunization (n = 35), E: 18, C: 176YesNoRCTAudiovisual distraction (cartoon movie)Standard carePain: WB-FACES Distress: OSBD-A8
42Vagnoli et al. (2015)Italy1hospitalPhlebotomy (n = 50), E: 25, C: 254–11 (E:7.1 ± 1.8, C:7.38 ± 2.5)YesNoRCTVisual distraction (dog presence)Standard carePain: WB-FACES Distress: OSBD-A10
43Singh (2012)India1hospitalImmunization (n = 60), E: 30, C: 301.5–2NoNoQuasiTouch distraction (toy)Standard carePain: FLACC (0–10)5
44Singh (2012)India1hospitalImmunization (n = 60), E: 30, C: 301.5–2NoNoQuasiAudial distraction (musical distraction)Standard carePain: FLACC (0–10)5
45Minute et al. (2012)Italy1hospitalImmunization (n = 97), E: 47, C: 504–10 (median: 7)YesNoRCTVisual distraction (Wii videogame+ EMLA cream)EMLA creamPain: FLACC (0–10), FPS-R9
Notes. E: experimental group, C: control group, IV: intravenous, SCP: subcutaneous port, EMLA: eutectic mixture of local anesthetics, HPV: Human papillomavirus, IRB: institutional review board, RCT: randomized controlled trial, Quasi: quasi-experimental design, PSP: playstation portable, Ditto-D: ditto distraction, Ditto-PP: ditto procedural preparation, Ditto-C: ditto combined procedural preparation and distraction, RA: research assistant, WB-FACES: Wong Baker faces pain rating scale., CFS: children’s fear scale, NRS: numerical rating scale, VAS: visual analogue scale, FPS-R: face pain scale—revised, OSBD-R: observational scale of behavioral distress—revised, FLACC: face, legs, activity, crying and consolability behavioral scale, PRCD: parent report of child distress, MBPS: modified behavior pain scale, MAISD: measure of adult and infant soothing and distress, CAPS: children’s anxiety and pain scales, FPS: faces pain scale, BAADS: behavioral approach-avoidance scale, CAS: colored analogue scale, PCS-C: pain catastrophizing scale for children, CSS: child stress scale, WB-FPS: Wong-Baker faces pain rating scale, CEMS: children’s emotional manifestation scale, OSBD-A: amended observation scale of behavioral distress.
Table 2. The subgroup analysis of the effect of distract intervention on needle-related pain of the child by the study characteristics (N = 41).
Table 2. The subgroup analysis of the effect of distract intervention on needle-related pain of the child by the study characteristics (N = 41).
CharacteristicsSubgroupKNOverall ES95% CIZ (p)I2 (%)
Lower LimitUpper Limit
Age (year)<101514190.27−0.18 0.73 1.17 (0.242)95.4
≥10261761−1.07−1.60 −0.53 −3.92 (<0.001)85.3
CenterOne382671−0.51−0.85 −0.17 −2.96 (0.003)92.7
Multi-center3509−0.43−0.96 0.09 −1.61 (0.108)0.0
IRBNo 42331.97−4.318.250.61 (0.539)98.7
Yes372947−0.64−0.89−0.39−5.04 (<0.001)84.9
FundYes151340−0.83−1.20−0.46−4.41 (<0.001)90.5
No261840−0.18−0.840.47−0.55 (0.581)93.0
Study designRCT372932−0.61−0.92 −0.30 −3.85 (<0.001)91.3
Quasi-E42480.94−3.60 5.48 0.40 (0.685)96.9
Type of interventionAudial7394−0.38−2.10 1.33 −0.44 (0.661)94.9
Visual171790−0.76−1.15 −0.37 −3.86 (<0.001)89.6
Clown2119−3.64−4.76 −2.52 −6.36 (<0.001)91.9
Cognitive2107−0.47−1.64 0.70 −0.79 (0.428)0.0
Multi methods127030.31−0.350.970.92 (0.360)93.4
Local anesthesia1670.67
Quality assessment score<10242077−0.35−1.16 0.46 −0.84 (0.400)93.6
≥10171103−0.65−0.98 −0.32 −3.84 (<0.001)89.3
Notes. K: Number of studies, N: Number of participants, ES: Effect size, CI: Confidence interval, IRB: Institutional Review Board, RCT: Randomized controlled trial, Quasi-E: Quasi-experimental design.
Table 3. The effect of distract intervention on distress of the Child.
Table 3. The effect of distract intervention on distress of the Child.
VariablesKNOverall ES95% CIZ (p)I2 (%)
Lower LimitUpper Limit
Observer pain (parent)231891−0.40−1.260.45−0.93 (0.355)94.1
Observer pain (nurse)2821731.43−1.964.820.83 (0.408)99.7
Child-reported anxiety3207−0.33−1.270.62−0.68 (0.496)0.0
Observer anxiety (parent)151163−0.68−1.13−0.23−2.96 (0.003)85.9
Observer anxiety (nurse)141330−1.03−1.67−0.40−3.20 (0.001)94.0
Cortisol5340−1.11−2.09−0.12−2.20 (0.274)66.8
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Cho, M.-K.; Choi, M.-Y. Effect of Distraction Intervention for Needle-Related Pain and Distress in Children: A Systematic Review and Meta-Analysis. Int. J. Environ. Res. Public Health 2021, 18, 9159. https://doi.org/10.3390/ijerph18179159

AMA Style

Cho M-K, Choi M-Y. Effect of Distraction Intervention for Needle-Related Pain and Distress in Children: A Systematic Review and Meta-Analysis. International Journal of Environmental Research and Public Health. 2021; 18(17):9159. https://doi.org/10.3390/ijerph18179159

Chicago/Turabian Style

Cho, Mi-Kyoung, and Mi-Young Choi. 2021. "Effect of Distraction Intervention for Needle-Related Pain and Distress in Children: A Systematic Review and Meta-Analysis" International Journal of Environmental Research and Public Health 18, no. 17: 9159. https://doi.org/10.3390/ijerph18179159

APA Style

Cho, M. -K., & Choi, M. -Y. (2021). Effect of Distraction Intervention for Needle-Related Pain and Distress in Children: A Systematic Review and Meta-Analysis. International Journal of Environmental Research and Public Health, 18(17), 9159. https://doi.org/10.3390/ijerph18179159

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop