Next Article in Journal
Study on Emotional Perception of Hangzhou West Lake Scenic Area in Spring under the Influence of Meteorological Environment
Next Article in Special Issue
Short- and Long-Term Stability of Aromatic Amines in Human Urine
Previous Article in Journal
Simplified Unified BARGE Method to Assess Migration of Phthalate Esters in Ingested PVC Consumer Products
Previous Article in Special Issue
The Serum Levels of the Heavy Metals Cu, Zn, Cd, and Pb and Progression of COPD—A Preliminary Study
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 20
Issue 3
10.3390/ijerph20031908
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Chemical Profiles and Toxicity of Electronic Cigarettes: An Umbrella Review and Methodological Considerations

by
Nargiz Travis
1,*,
Marie Knoll
1,
Steven Cook
2,
Hayoung Oh
1,
Christopher J. Cadham
3,
Luz María Sánchez-Romero
1 and
David T. Levy
1
1
Lombardi Comprehensive Cancer Center, Georgetown Medical University, Washington, DC 20007, USA
2
Department of Epidemiology, School of Public Health, University of Michigan, Ann Arbor, MI 48104, USA
3
Department of Health Management and Policy, School of Public Health, University of Michigan, Ann Arbor, MI 48104, USA
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2023, 20(3), 1908; https://doi.org/10.3390/ijerph20031908
Submission received: 1 November 2022 / Revised: 13 January 2023 / Accepted: 18 January 2023 / Published: 20 January 2023
(This article belongs to the Special Issue Tobacco Product-Related Health Issues: Toxicants, Biomonitoring, Health Risks, and Chronic Disease)
Browse Figure
Versions Notes

Abstract

:
Background: Electronic cigarettes (ECs) are often marketed as a safer alternative to combustible tobacco products. The global EC market has rapidly expanded since their introduction, creating an urgent need for research describing the toxicity and chemical composition of ECs. We conducted an umbrella review to summarize the evidence from existing systematic reviews (SRs). Methods: The search for SRs was conducted across four electronic databases through 25 January 2022. Methodological quality was assessed using the AMSTAR-2 quality appraisal tool. Results: Twenty-five SRs were included in our umbrella review. Chemical profiles widely varied across studies included in the reviews, which was mainly attributed to the lack of standardized protocols investigating the constituents, and differences in EC devices and e-liquids tested. Metals were more abundant in some EC aerosols than cigarettes, while carbonyls were typically found at lower levels. There was consistent evidence of in vitro toxicity from EC aerosol and e-liquid exposure. AMSTAR-2 revealed important limitations across reviews. Conclusions: While most reviews concluded that ECs were likely less harmful than cigarettes, there was hesitancy to draw clear conclusions due to variable analytical procedures and inconsistent findings among the included studies. Future SRs with improved methodology and reporting are needed to adequately inform tobacco regulatory actions.

1. Introduction

The electronic cigarette (EC) is a nicotine delivery system that is often marketed as a safer alternative to combustible tobacco products [1]. ECs have been constantly evolving since their invention in 2003, with mainly four types of devices available to the user: disposable cig-a-likes prefilled with a liquid (i.e., first-generation devices), prefilled or refillable cartridges (i.e., second-generation devices), refillable and modifiable tank cartridges, also known as “mods”, that allow for a change in voltage or power, (i.e., third-generation devices), and prefilled or refillable pod cartridges with a modifiable system, also known as “pod-mods” (i.e., fourth-generation devices) [2].
Unlike conventional cigarettes (CC), which involve tobacco combustion, exposing users to thousands of chemicals, ECs heat and aerosolize liquid generally containing propylene glycol, vegetable glycerin, nicotine (in nicotine-containing solutions), flavorings (in flavored solutions), and other chemicals [3,4]. The thermal degradation of the EC liquid (e-liquid) during the heating process emits potentially harmful compounds, such as heavy metals, low molecular weight carbonyl compounds (e.g., formaldehyde, acetaldehyde, and acetone), and tobacco-specific nitrosamines [5]. However, with aerosol constituents and their emission levels varying across different EC devices, mixed information exists regarding their likely chemistry [6].
With the rapid expansion of the global EC market, reliable information is needed regarding the chemical composition and the potentially toxic effects of ECs so that evidence-based regulations can limit the harm associated with EC use. Given limited short-term and lack of long-term data on human health outcomes [7], an important source of EC risk assessment relies on the results from toxicity studies [8,9]. Toxicity assessments are commonplace in tobacco research, since they can help identify levels of exposure that are safe for humans [8]. Such assessments are also essential for testing and understanding the mechanisms by which chemicals affect animals and humans at the cellular level, thereby bridging the gap between in vitro and in vivo studies.
An emergent body of research has examined the chemical profiles (i.e., chemical elements contained in EC aerosol and/or e-liquid) and toxicity (i.e., in vitro testing of cytotoxic and genotoxic effects on cultured cells exposed to aerosol or e-liquid) of ECs, and several systematic reviews have attempted to synthesize the findings from these studies. However, these reviews produced heterogeneous results, making it important to summarize the findings and areas of disagreement of existing systematic reviews and critically evaluate their methodological quality. In this umbrella review, we aim to summarize the main findings, conclusions, and recommendations of these reviews, critically assess their methodological quality, highlight limitations of current research, and identify needs for future research.

2. Methods

Our review protocol was developed a priori in adherence with the guidelines for umbrella reviews [10], and registered with PROSPERO (Registration Number: CRD42021237878). The original review protocol also planned to include evidence on health outcomes of ECs. However, the volume of the literature and the significance of differences related to the chemical constituents and EC toxicity literature warranted two separate publications. The present paper is an extension of our previously published umbrella review on the health effects of ECs and follows the same methodology [7]. It covers chemical profiles and toxicity of ECs and was reported in adherence with the Preferred Reporting Items for Systematic reviews and Meta-analyses (PRISMA) [11].

2.1. Search Strategy and Terms

A structured literature search was conducted on 27 May 2020 in PubMed, Web of Science, Embase, and Cochrane Database of Systematic Reviews by two reviewers (NT, CJC) and was updated on 25 January 2022. The search terms included “electronic cigarette”, “e-cigarette”, “electronic nicotine delivery system”, “personal vaporiser”, “personal vaporizer”, “e-liquid”, “nicotine content”, “systematic review”, “meta-analysis”, and “review”. The complete search strategy has been previously described in more detail in our earlier umbrella review [7], and the PubMed database search is available in Supplementary Table S1.

2.2. Eligibility

This umbrella review included systematic reviews and meta-analyses published in English that examined outcomes related to: (1) chemical profiles of e-liquids/EC aerosols, or (2) toxicity of e-liquids/EC aerosols. We considered a literature review to be systematic if it explicitly identified itself as such in the title, abstract, keyword, or methods. Editorials, commentaries, and letters to the editor were excluded. Evidence from experimental chemical and in vitro human and animal cell studies were included. Reviews reporting on chemicals found in human biosamples of EC users were also included, since they reflect user exposure to chemical constituents present in ECs. EC devices and e-liquids of all types were considered.

2.3. Data Extraction

Two reviewers (NT, MK) developed a standardized data extraction tool. Extracted information from reviews included: citation, authors’ financial disclosures, review objectives, details of the literature search, designs of included studies, description of exposure (e.g., duration of exposure, EC device, e-liquid or aerosol), outcome variables, cell/tissue types, quality appraisal tool used, key findings, authors’ concluding views along with recommendations, and reported study limitations. Data from each review was extracted and documented by one reviewer (MK), and checked for completeness and accuracy by the second reviewer (NT).

2.4. Assessment of Quality and Risk of Bias

Included reviews were independently appraised for quality and bias by two reviewers (NT and MK) using the AMSTAR-2 checklist [12]. The description and the exact application of this tool in the framework of our umbrella review have been previously published [7].

3. Results

We identified 1108 individual articles from the databases searched on 27 May 2020, and an additional 772 unique records were found through our re-run search conducted on 25 January 2022 (Figure 1).
Full-text appraisal was conducted for 354 of these studies. The most common reason for excluding full texts was the unfulfilled criterion of being a systematic review or meta-analysis (n = 238). Twenty-five systematic reviews including one meta-analysis were deemed eligible for inclusion in our study. Chemical profiles of ECs were investigated in twenty [6,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31] and the toxicity of ECs was covered in fourteen reviews [13,15,18,19,21,23,26,28,30,32,33,34,35,36]. Nine reviews investigated both outcomes [13,15,18,19,21,23,26,28,30]. Table 1 presents the characteristics of the included reviews.

3.1. AMSTAR 2 Assessment of Quality and Risk of Bias

Table 2 shows the results from the AMSTAR-2 Assessment tool. Twenty-three reviews failed to report whether their review methods were established in a written protocol prior to conducting the review, as recommended by the PRISMA Guidelines [11]. Only five reviews applied an appropriate technique for systematically assessing the potential risk of bias within their included studies, and only four accounted for the risk of bias in individual studies when interpreting their results. Twenty-three reviews disclosed their sources of funding, while only eight reviews reported information on funding and any conflicts of interest for the included studies. Fourteen reviews did not apply a comprehensive literature strategy, and only one review received full credit for this domain (e.g., searching at least two databases, searching reference lists of included studies, trial/study registries). In nine reviews, the study selection was performed in duplicate and seven reported to have performed data extraction in duplicate. None of the reviews provided a list of excluded studies with justifications for exclusion of each potentially relevant study.

3.2. Chemical Profiles of E-Liquids and EC Aerosols

Twenty reviews investigated the chemical composition of e-liquids and EC aerosols [6,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]. Results were generally reported without a clear stratification between constituents found in e-liquid vs. EC aerosols. Below, we focus on the constituents that were most commonly reported across the reviews.

3.2.1. Metals/Metalloids

Twelve reviews examined chemical evaluations of metals and metalloids (i.e., semimetals) in e-liquids and EC aerosols [6,14,15,16,19,21,22,23,25,26,30,31]. Reviews consistently found evidence of these elements being present in ECs, although considerable heterogeneity was reported in the detected levels across the included studies [6,14,15,16,19,21,22,23,25,26,30,31]. Higher levels of metals were seen in EC aerosols after e-liquids came in contact with the heating coil and metal components of the EC device, while metal levels varied by brand and device characteristics, such as power and coil material [6,14,16,21,22,23,31]. Two reviews reported that the levels of metals/metalloids in ECs were significantly lower than in CC smoke, while one suggested that the levels in ECs may be higher, likely due to the metal components of EC devices [16,19,21]. Reviews of studies examining human biosamples confirmed similar or higher levels of metals and metalloids in EC users’ urine and serum compared to CC or cigar users [14,26].

3.2.2. Carbonyls

Carbonyl levels were examined in 15 reviews [6,15,18,19,20,21,22,23,24,25,26,28,29,30,31]. Carbonyls such as formaldehyde and acetaldehyde were generally found at lower levels in EC aerosols and e-liquids compared to CCs, with levels substantially differing between and within EC samples depending on device type and the specific study [6,15,18,19,20,21,23,24,25,26]. In particular, aerosol emissions from fourth generation nicotine-salt-containing devices, such as Juul, were shown to contain lower levels of carbonyls when compared to other ECs and CCs [28]. Carbonyl concentrations were found to be positively correlated to higher power levels and temperatures of EC devices [6,20,23,24,25,30,31].

3.2.3. Other Common Constituents

Volatile organic compounds (VOCs), tobacco-specific nitrosamines (TSNA), N’-Nitrosonornicotine (NNN), 4-(methylnitrosamino)-1-(3-pyridyl)-1-butanone (NNK), and polycyclic aromatic hydrocarbons (PAHs) were among other common chemical constituents measured in chemical profile analyses within 15 reviews [6,13,15,17,18,21,23,25,26,27,28,29,30,31,36]. Reviews found that, while present in many e-liquid and EC aerosol samples, the levels of these chemicals were often below the level of detection or significantly lower than levels found in CCs [6,13,15,17,18,21,23,25,26,28,31]. However, while levels were lower, it was commonly found across reviews that levels widely varied depending on study techniques and were inconsistent with EC labeling [6,15,17,18,21,23,25,26,31]. Notably, reviews emphasized the difficulty in identifying true exposure to carcinogenic compounds due to user-adjustable operating parameters of EC devices and unregulated e-liquids [29,30]. Similarly, reviews that included human experimental studies found the presence of NNN, NNK, and other carcinogens in the saliva, urine, and blood of EC users, though most often at lower levels than that of CC users [13,26,27]. When looking at flavored e-liquids, 173 chemical compounds were identified with ester and alkene as the most frequent chemical class, while aryl, alcohol, ketone, aldehyde, and lactone were among other common chemical classes identified [29]. Predictions of chemical transformations of these flavoring components upon aerosolization suggest their possible correlation to aerosol toxicants [29].

3.3. Toxicity of ECs

Fourteen reviews covered studies investigating the toxicity of ECs, mostly relying on evidence from studies exposing human cell cultures to EC aerosol and/or e-liquid [13,15,18,19,21,23,26,28,30,32,33,34,35,36]. Reviews consistently found that, upon exposure to EC aerosol and e-liquid, there was evidence of reduced cell viability, increased apoptosis, DNA damage, oxidative stress, impaired immune function, and an increase in inflammatory cytokines [13,15,18,19,21,23,26,28,30,32,33,34,35]. Generally, reviews with studies evaluating e-liquids reported more cytotoxic effects compared to EC aerosols. One review found evidence that while exposure to e-liquid and/or EC aerosol was associated with cytotoxicity, oxidative stress, reduced viability, delayed fibroblast migration, aberrant morphology, and genotoxicity, cigarette smoke exposure was found to be significantly more toxic to human cell cultures [34].

Flavoring Additives

Reviews that included evidence from studies testing the toxicity of flavored e-liquids reported cytotoxic effects of flavoring additives [13,15,16,17,18,19,21,26,30,31,32,33,35,36]. Among flavors, cinnamon [15,17,18,19,21,30,31,33,35] and menthol/mint [13,30,32,33] were singled out for their toxic properties (i.e., increased cell necrosis, altered cell morphology). One review in particular found sweet, fruity, and citrus flavored e-liquids to be more damaging to human tissues (e.g., initiate pathological processes, oxidative stress) compared to vanilla-flavored or non-flavored e-liquids [35]. Diacetyl and acetyl propionyl, artificial butter flavoring agents used in e-liquids, have been associated with an increased risk of developing bronchiolitis obliterans, a pulmonary disease previously detected among microwave popcorn industry workers [19,30,31,36].

3.4. Main Conclusions of Systematic Reviews

Due to the variable nature of product use among EC users and varying design parameters, most reviews that examined EC toxicity concluded that evidence is still too limited to draw definite conclusions about the potential human health effects of ECs, and that while ECs are not innocuous at the cellular level, they are likely much safer than CCs [6,13,15,17,18,19,20,21,22,23,24,25,26,30,31,32,33,34,36]. However, there was an agreement among several reviews that the evidence of negative impact on health warrants stronger regulation of ECs [14,21,28,35]. One review offered a more cautious interpretation of current evidence, determining that there is insufficient quality research to conclude that ECs are less harmful than CCs [32]. Similarly, another review specifically reviewing new generation EC devices, such as Juul, found that while e-liquids and aerosols from such devices may contain less harmful constituents than other types of e-cigarettes and cigarettes, there is no evidence that these levels are safe for young adults [28]. Two reviews concluded that ECs are a major concern for exposure to metals/metalloids, questioning whether they should even be marketed as a safer alternative to CCs [14,16]. While the evidence on the toxic effects of flavoring additives is limited to e-liquid testing, one review concluded that there is a possible correlation of flavor compounds in e-liquids to aerosol toxicants, calling for a conceptual framework that can further enhance this knowledge [29].

3.5. Main Limitations of Current Research

Authors of the included reviews identified several methodological limitations within their included studies. A major limitation mentioned in nearly half of the reviews was the lack of validated and standardized methods for collecting evidence on chemical profiles and toxicity evaluations that could explain variations in the estimated levels of toxicants [6,14,15,16,19,20,23,26,31,32,35,36]. Further, reviews emphasized that methods currently used to evaluate chemical compositions and cellular toxicity do not necessarily reflect actual exposure from realistic EC use behaviors in humans [6,18,19,21,22,25,29,31,35,36]. Thus, authors cautioned directly applying findings to in vivo situations. Evidence from human biosample studies included in some reviews were deemed weak by review authors due to small sample sizes, lack of control groups, missing definitions of smoking status, and short exposure periods [13,14,15,19,26,27]. Several reviews expressed concern that most research lacked information on devices used, and, those that did, often included evaluations from first-generation devices which may not reflect current consumer use [6,26,27].

3.6. Recommendations of Systematic Reviews

Future recommendations from authors of reviews examining chemical profiles of ECs included standardizing laboratory protocols for methods of extracting compounds from EC aerosols and e-liquids, and uniform units of detection [6,14,18,19,20,21,25,31]. One review suggested that future experimental research should address typical EC use behavior so that accurate puff intervals can be established [22]. Additionally, several reviews highlighted the need to emphasize the importance of conducting long-term in vivo studies, which is essential for understanding true human exposure [13,21,24,26,28,32,36]. One review emphasized the importance of focusing on both exclusive EC users and dual EC and CC users in future research in order to better distinguish health consequences of ECs [33]. Methods of validation and identification of appropriate control groups and relevant biomarkers were among other recommendations [13,19]. Continued testing of the toxicity of flavored EC aerosols and e-liquids was also a common recommendation [13,20,21,29], with two reviews explicitly recommending that diacetyl should not be used as a flavoring component in e-liquids [26,36]. In order to improve the safety profiles of ECs, one review suggested that regulatory agencies play a central role in guiding the acceptable levels of exposure to the various constituents [31]. With new products constantly emerging, reviews generally recommended focusing future research on newer generations and models.

4. Discussion

This umbrella review summarized evidence of the chemical constituents and in vitro toxicity of ECs from existing systematic reviews while highlighting the limitations of current research. Wide variations were reported in the chemical compositions of ECs across studies included in reviews, likely due to a lack of standardized protocols for collecting evidence on chemical profiles and the heterogeneity of devices and e-liquids evaluated. Reviews generally found that metals/metalloids were more abundant in EC aerosols than in e-liquids due to contact with a heating coil and metal components of the device. Carbonyls and other constituents (e.g., tobacco-specific nitrosamines, polycyclic aromatic hydrocarbons) were consistently found at lower levels in ECs than in CCs. Reviews found evidence of in vitro toxicity of ECs, although to a lesser extent than CCs, with more cytotoxic effects reported for e-liquids compared to EC aerosols. Notably, evidence from human cell culture studies showing significantly higher cytotoxicity of cigarette smoke compared to EC aerosol has been well documented in the past [37,38]. Among flavoring additives in e-liquids, cinnamon and menthol/mint were found to be particularly cytotoxic. However, considerations of dose, mode of administration, and lack of human in vivo data leave unanswered the clinical importance of these findings.
The need for standardized and validated protocols for chemical profile and toxicity studies was a predominant theme in the limitations and discussions across nearly all included reviews. While approaches for testing combustible tobacco products are relatively well established, approaches for ECs are less developed [9]. Because of the large and constantly evolving varieties of ECs, developing standard protocols is challenging, especially for toxicity assessments [39,40]. Recently, tools and frameworks have been proposed to standardize puffing patterns and toxicology assessments [40,41]. In particular, rigorous analytical approaches for accurate assessment of the concentrations of toxic metals in EC aerosols have been proposed and validated as part of good inorganic analytical practice to help eliminate possible sources of leachable metal background in the sample collection materials [42,43]. The use of such validated methods would enable future research to better understand the relative chemical profiles and toxicity of ECs and improve the validity and comparability of research findings.
A lack of transparency in the methods of the included studies was another major concern across the reviews. Inadequate detail regarding the EC device types limited the ability to compare results across studies. Importantly, reviews found that the identified constituent levels often deviated from those reported on labels. More research is needed to identify if these deviations are due to inaccurate measurement methods or poor labeling standards. Future reviews should focus on the degree to which ECs are less toxic than CCs, with clear interpretation as to how these findings can be translated into regulatory and public health guidelines. Furthermore, reviews should compare the different classes of EC products in terms of their toxicity, the determinants of toxicity, and explore steps that could be taken to reduce such toxicity. For this purpose, future reviews may also consider unpublished, but otherwise publicly available, data on toxicity and chemical constituents as submitted by manufacturers to FDA in their pre-market and modified-risk applications.
The abundance of available flavored e-liquids on the market and the lack of mandatory labeling of flavoring agents further extend the question of safety of ECs. While some country- and market-specific EC flavor classifications have been proposed, a universal classification system of EC flavoring compounds is needed [44,45]. With most available studies on the toxicity of flavoring additives being restricted to e-liquids, future research should focus on how these additives contribute to toxicant emissions in EC aerosols. Additionally, more in vivo studies are needed to better understand the toxic health effects of EC use on humans.
While reviews identified common limitations of the current empirical research, our study found important methodological weaknesses of these systematic reviews. The assessment of methodological quality and risk of bias using the AMSTAR-2 checklist found many incomplete domains, indicating major methodological flaws and a potentially high risk of bias in the included systematic reviews. As such, over 90% of the included reviews failed to report whether their review methods were established in a written protocol prior to conducting the review, despite recommendations by Cochrane Collaboration, the National Academy of Medicine, and PRISMA guidelines [11,46,47]. This finding is even more staggering compared to a study that found that only one-third of systematic reviews published in MEDLINE in 2014 reported working from a previously developed protocol [48]. Detailed review protocols, developed a priori, are crucial to ensure that the chosen research designs/methods are appropriate, and that the findings of the reviews are adequately reported. Registering protocols in a database for systematic reviews enables researchers to better communicate their research aims and methods, prevents the unintentional reproduction of reviews addressing the same topic, and allows researchers to reproduce and validate findings or properly update the review with new evidence. Further, 80% of the systematic reviews failed to use an appropriate technique to assess the risk of bias of the included studies, and more than half of the reviews did not apply a comprehensive literature search strategy when conducting their review. Such methodological gaps may have resulted in reviews including studies with design flaws and prevented reviews from adequately capturing available evidence. Future systematic reviews should therefore strictly adhere to the established guidelines to minimize their risk of bias and avoid magnifying “waste” from primary studies.
To our knowledge, this is the first umbrella review to have focused on chemical profiles and toxicity of ECs. While this umbrella review provides a comprehensive and reproducible synthesis of the available literature, it has several limitations. Consistent with the first item on the PRISMA 2021 Checklist [11], our review identified systematic reviews as those that explicitly stated that they were “systematic reviews” in their title, abstract, methods, or keyword. This approach was selected as the most straightforward and transparent way to detect a systematic review. However, it may have led to the exclusion of well-designed literature reviews that did not self-identify as systematic, while at the same time allowing for the inclusion of reviews with methodological shortcomings, as indicated by the AMSTAR-2 checklist. We chose the AMSTAR-2 quality appraisal tool to be most suitable for our umbrella review among other existing tools [49]. However, in light of the heterogeneity in the methods and outcomes of the included reviews and due to the previously observed abundance of critically low ratings produced by the tool [50], we chose not to assign the overall quality ratings for systematic reviews. Instead, our approach was to narratively describe the areas of potential bias exhibited in our included reviews. Further, our literature search did not impose a restriction on publication date. Consequently, our analysis included seven reviews with data collection up to 2015, before new-generation ECs devices, such as Juul of Puff Bar, entered and subsequently dominated the U.S. market. Thus, their findings may be of limited relevance in the rapidly growing and diversifying EC market. Stratification of our findings by EC device generation or e-liquid type was not possible due to underreporting of their characteristics in most reviews. Further, given the abundance of chemical constituents present in e-liquids and aerosols, our reviews only focused on the constituents that were most commonly reported in the included systematic reviews. Due to the limited comparability of reviews, summarizing the evidence in quantitative terms was beyond the scope of this umbrella review. Finally, since we did not specifically seek to identify the overlapping original studies included in the reviews due to the large volume, some studies may be potentially overrepresented in our umbrella review.

5. Conclusions

This umbrella review examined evidence of the chemical composition and in vitro toxicity of ECs and highlighted methodological limitations of current research. It generally found evidence of toxicity of ECs, although to a lesser extent than CCs. It also identified the most common chemical constituents in ECs, but there was wide variation in the exact composition of EC aerosols and e-liquids within and across the reviews. This variation is mainly attributed to the differences in EC devices and e-liquids tested, and the lack of standardized protocols for methods of analyzing the chemical profiles of ECs. With EC products constantly evolving and their potential health effects taking years to assess, chemical and in vitro studies will continue to be essential for policymakers and regulatory agencies as they grapple with the issues of evaluating and regulating current and future EC products. Our study highlights the need for future systematic reviews on the composition and toxicity of ECs with better adherence to established reporting guidelines to adequately inform tobacco regulatory actions. The clinical significance of EC toxicity and the difference in toxicity between the types of e-cigarettes and in relation to combustible cigarettes should be the focus of future research.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/ijerph20031908/s1, Table S1: The search strategy for the PubMed database; Table S2: Characteristics of Sys-tematic Reviews Including Chemical Profiles of ECs (n = 20); Table S3: Characteristics of Systematic Reviews Including Toxicity of ECs (n = 14).

Author Contributions

Conceptualization, N.T. and D.T.L.; methodology, N.T., M.K., D.T.L. and C.J.C.; software, N.T. and M.K.; validation, N.T.; formal analysis, M.K. and N.T.; investigation, N.T. and M.K.; resources, N.T. and M.K.; data curation, N.T. and M.K.; writing—original draft preparation, N.T. and M.K.; writing—review and editing, N.T., M.K., D.T.L., S.C., C.J.C., H.O. and L.M.S.-R.; visualization, N.T and M.K.; supervision, N.T. and D.T.L.; project administration, N.T.; funding acquisition, D.T.L. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by National Cancer Institute (NCI) and Food and Drug Administration (FDA) grant number U54CA229974.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

  1. Collins, L.; Glasser, A.M.; Abudayyeh, H.; Pearson, J.L.; Villanti, A.C. E-cigarette marketing and communication: How E-Cigarette Companies Market E-Cigarettes and the Public Engages with E-cigarette Information. Nicotine Tob. Res. 2019, 21, 14–24. [Google Scholar] [CrossRef]
  2. Centers for Disease Control and Prevention. E-Cigarette, or Vaping, Products Visual Dictionary; Department of Health and Human Services: Washington, DC, USA, 2019. [Google Scholar]
  3. Chun, L.F.; Moazed, F.; Calfee, C.S.; Matthay, M.A.; Gotts, J.E. Pulmonary toxicity of e-cigarettes. Am. J. Physiol. Lung. Cell. Mol. Physiol. 2017, 313, L193–L206. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Centers for Disease Control and Prevention; National Center for Chronic Disease Prevention and Health Promotion; Office on Smoking and Health. How Tobacco Smoke Causes Disease: The Biology and Behavioral Basis for Smoking-Attributable Disease: A Report of the Surgeon General; Centers for Disease Control and Prevention: Atlanta, GA, USA, 2010. [Google Scholar]
  5. National Academies of Sciences, Engineering, and Medicine. Public Health Consequences of E-Cigarettes; The National Academies Press: Washington, DC, USA, 2018. [Google Scholar]
  6. Cheng, T. Chemical evaluation of electronic cigarettes. Tob. Control 2014, 23 (Suppl. S2), ii11–ii17. [Google Scholar] [CrossRef]
  7. Travis, N.; Knoll, M.; Cadham, C.J.; Cook, S.; Warner, K.E.; Fleischer, N.L.; Douglas, C.E.; Sanchez-Romero, L.M.; Mistry, R.; Meza, R.; et al. Health Effects of Electronic Cigarettes: An Umbrella Review and Methodological Considerations. Int. J. Environ. Res. Public Health 2022, 19, 9054. [Google Scholar] [CrossRef] [PubMed]
  8. Johnson, M.D.; Schilz, J.; Djordjevic, M.V.; Rice, J.R.; Shields, P.G. Evaluation of in vitro assays for assessing the toxicity of cigarette smoke and smokeless tobacco. Cancer Epidemiol. Biomarkers Prev. 2009, 18, 3263–3304. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  9. Belushkin, M.; Piade, J.J.; Chapman, S.; Fazekas, G. Investigating predictability of in vitro toxicological assessments of cigarettes: Analysis of 7 years of regulatory submissions to Canadian regulatory authorities. Regul. Toxicol. Pharmacol. 2014, 68, 222–230. [Google Scholar] [CrossRef] [Green Version]
  10. Aromataris, E.; Fernandez, R.; Godfrey, C.M.; Holly, C.; Khalil, H.; Tungpunkom, P. Summarizing systematic reviews: Methodological development, conduct and reporting of an umbrella review approach. Int. J. Evid.-Based Healthc. 2015, 13, 132–140. [Google Scholar] [CrossRef] [Green Version]
  11. Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021, 372, n71. [Google Scholar] [CrossRef]
  12. Shea, B.J.; Reeves, B.C.; Wells, G.; Thuku, M.; Hamel, C.; Moran, J.; Moher, D.; Tugwell, P.; Welch, V.; Kristjansson, E.; et al. AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017, 358, j4008. [Google Scholar] [CrossRef] [Green Version]
  13. Yang, I.; Sandeep, S.; Rodriguez, J. The oral health impact of electronic cigarette use: A systematic review. Crit. Rev. Toxicol. 2020, 50, 97–127. [Google Scholar] [CrossRef]
  14. Zhao, D.; Aravindakshan, A.; Hilpert, M.; Olmedo, P.; Rule, A.M.; Navas-Acien, A.; Aherrera, A. Metal/metalloid levels in electronic cigarette liquids, aerosols, and human biosamples: A systematic review. Environ. Health Perspect. 2020, 128, 036001. [Google Scholar] [CrossRef] [Green Version]
  15. Pisinger, C.; Dossing, M. A systematic review of health effects of electronic cigarettes. Prev Med. 2014, 69, 248–260. [Google Scholar] [CrossRef] [Green Version]
  16. Gaur, S.; Agnihotri, R. Health Effects of Trace Metals in Electronic Cigarette Aerosols-a Systematic Review. Biol. Trace Elem. Res. 2019, 188, 295–315. [Google Scholar] [CrossRef]
  17. Armendariz-Castillo, I.; Guerrero, S.; Vera-Guapi, A.; Cevallos-Vilatuna, T.; Garcia-Cardenas, J.M.; Guevara-Ramirez, P.; Lopez-Cortes, A.; Perez-Villa, A.; Yumiceba, V.; Zambrano, A.K.; et al. Genotoxic and Carcinogenic Potential of Compounds Associated with Electronic Cigarettes: A Systematic Review. BioMed Res. Int. 2019, 2019, 1386710. [Google Scholar] [CrossRef] [Green Version]
  18. Glasser, A.M.; Collins, L.; Pearson, J.L.; Abudayyeh, H.; Niaura, R.S.; Abrams, D.B.; Villanti, A.C. Overview of Electronic Nicotine Delivery Systems: A Systematic Review. Am. J. Prev. Med. 2017, 52, e33–e66. [Google Scholar] [CrossRef] [Green Version]
  19. Kaur, G.; Pinkston, R.; McLemore, B.; Dorsey, W.C.; Batra, S. Immunological and toxicological risk assessment of e-cigarettes. Eur. Respir. Rev. 2018, 27, 170119. [Google Scholar] [CrossRef]
  20. Farsalinos, K.E.; Gillman, G. Carbonyl emissions in e-cigarette aerosol: A systematic review and methodological considerations. Front. Physiol. 2018, 8, 1119. [Google Scholar] [CrossRef]
  21. Farsalinos, K.E.; Polosa, R. Safety evaluation and risk assessment of electronic cigarettes as tobacco cigarette substitutes: A systematic review. Ther. Adv. Drug Saf. 2014, 5, 67–86. [Google Scholar] [CrossRef] [Green Version]
  22. Zulkifli, A.; Abidin, E.Z.; Abidin, N.Z.; Amer Nordin, A.S.; Praveena, S.M.; Syed Ismail, S.N.; Rasdi, I.; Karuppiah, K.; Rahman, A.A. Electronic cigarettes: A systematic review of available studies on health risk assessment. Rev. Environ. Health 2018, 33, 43–52. [Google Scholar] [CrossRef]
  23. Harrell, P.T.; Simmons, V.N.; Correa, J.B.; Padhya, T.A.; Brandon, T.H. Electronic nicotine delivery systems (“e-cigarettes”): Review of safety and smoking cessation efficacy. Otolaryngol. Head Neck Surg. 2014, 151, 381–393. [Google Scholar] [CrossRef]
  24. Ioakeimidis, N.; Vlachopoulos, C.; Tousoulis, D. Efficacy and Safety of Electronic Cigarettes for Smoking Cessation: A Critical Approach. Hellenic. J. Cardiol. 2016, 57, 1–6. [Google Scholar] [CrossRef] [Green Version]
  25. Burstyn, I. Peering through the mist: Systematic review of what the chemistry of contaminants in electronic cigarettes tells us about health risks. BMC Public Health 2014, 14, 18. [Google Scholar] [CrossRef] [Green Version]
  26. Bozier, J.; Chivers, E.K.; Chapman, D.G.; Larcombe, A.N.; Bastian, N.A.; Masso-Silva, J.A.; Byun, M.K.; McDonald, C.F.; Crotty Alexander, L.E.; Ween, M.P. The Evolving Landscape of e-Cigarettes: A Systematic Review of Recent Evidence. Chest 2020, 157, 1362–1390. [Google Scholar] [CrossRef]
  27. Bjurlin, M.; Matulewicz, R.; Roberts, T.; Dearing, B.; Schatz, D.; Sherman, S.; Gordon, T.; El Shahawy, O. Carcinogen biomarkers in the urine of electronic-cigarette users and implications for potential bladder cancer: A systematic review. J. Urol. 2020, 203 (Suppl. S4), e1009. [Google Scholar] [CrossRef]
  28. Lee, S.J.; Rees, V.W.; Yossefy, N.; Emmons, K.M.; Tan, A.S.L. Youth and Young Adult Use of Pod-Based Electronic Cigarettes From 2015 to 2019: A Systematic Review. JAMA Pediatr 2020, 174, 714–720. [Google Scholar] [CrossRef]
  29. Salam, S.; Saliba, N.A.; Shihadeh, A.; Eissenberg, T.; El-Hellani, A. Flavor-Toxicant Correlation in E-cigarettes: A Meta-Analysis. Chem. Res. Toxicol. 2020, 33, 2932–2938. [Google Scholar] [CrossRef]
  30. Sharma, H.; Verma, S. “Vaping”—A Trojan Horse Against Fight Toward Tobacco Use and Cancer: A Systematic Review of the Existing Evidence. Indian J. Med. Paediatr. Oncol. 2020, 41, 321–327. [Google Scholar] [CrossRef]
  31. Ward, A.M.; Yaman, R.; Ebbert, J.O. Electronic nicotine delivery system design and aerosol toxicants: A systematic review. PLoS ONE 2020, 15, e0234189. [Google Scholar] [CrossRef]
  32. Flach, S.; Maniam, P.; Manickavasagam, J. E-cigarettes and head and neck cancers: A systematic review of the current literature. Clin. Otolaryngol. 2019, 44, 749–756. [Google Scholar] [CrossRef]
  33. Wills, T.A.; Soneji, S.S.; Choi, K.; Jaspers, I.; Tam, E.K. E-cigarette use and respiratory disorders: An integrative review of converging evidence from epidemiological and laboratory studies. Eur. Respir. J. 2021, 57, 1901815. [Google Scholar] [CrossRef]
  34. Wilson, C.; Tellez Freitas, C.M.; Awan, K.; Ajdaharian, J.; Geiler, J.; Thirucenthilvelan, P. Adverse Effects of E-cigarettes on Head, Neck, and Oral Cells: A Systematic Review. J. Oral. Pathol. Med. 2022, 51, 113–125. [Google Scholar] [CrossRef] [PubMed]
  35. Bravo-Gutiérrez, O.A.; Falfán-Valencia, R.; Ramírez-Venegas, A.; Sansores, R.H.; Ponciano-Rodríguez, G.; Pérez-Rubio, G. Lung Damage Caused by Heated Tobacco Products and Electronic Nicotine Delivery Systems: A Systematic Review. Int. J. Environ. Res. Public Health 2021, 18, 4079. [Google Scholar] [CrossRef] [PubMed]
  36. White, A.V.; Wambui, D.W.; Pokhrel, L.R. Risk assessment of inhaled diacetyl from electronic cigarette use among teens and adults. Sci. Total Environ. 2021, 772, 145486. [Google Scholar] [CrossRef] [PubMed]
  37. Romagna, G.; Allifranchini, E.; Bocchietto, E.; Todeschi, S.; Esposito, M.; Farsalinos, K.E. Cytotoxicity evaluation of electronic cigarette vapor extract on cultured mammalian fibroblasts (ClearStream-LIFE): Comparison with tobacco cigarette smoke extract. Inhal. Toxicol. 2013, 25, 354–361. [Google Scholar] [CrossRef] [PubMed]
  38. Scheffler, S.; Dieken, H.; Krischenowski, O.; Forster, C.; Branscheid, D.; Aufderheide, M. Evaluation of E-cigarette liquid vapor and mainstream cigarette smoke after direct exposure of primary human bronchial epithelial cells. Int. J. Environ. Res. Public Health 2015, 12, 3915–3925. [Google Scholar] [CrossRef] [Green Version]
  39. Kim, S.; Goniewicz, M.L.; Yu, S.; Kim, B.; Gupta, R. Variations in label information and nicotine levels in electronic cigarette refill liquids in South Korea: Regulation challenges. Int. J. Environ. Res. Public Health 2015, 12, 4859–4868. [Google Scholar] [CrossRef] [Green Version]
  40. Iskandar, A.R.; Gonzalez-Suarez, I.; Majeed, S.; Marescotti, D.; Sewer, A.; Xiang, Y.; Leroy, P.; Guedj, E.; Mathis, C.; Schaller, J.P.; et al. A framework for in vitro systems toxicology assessment of e-liquids. Toxicol. Mech. Methods 2016, 26, 389–413. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  41. Perkins, K.A.; Karelitz, J.L. A Procedure to Standardize Puff Topography During Evaluations of Acute Tobacco or Electronic Cigarette Exposure. Nicotine. Tob. Res. 2020, 22, 689–698. [Google Scholar] [CrossRef]
  42. Gonzalez-Jimenez, N.; Gray, N.; Pappas, R.S.; Halstead, M.; Lewis, E.; Valentin-Blasini, L.; Watson, C.; Blount, B. Analysis of Toxic Metals in Aerosols from Devices Associated with Electronic Cigarette, or Vaping, Product Use Associated Lung Injury. Toxics 2021, 9, 240. [Google Scholar] [CrossRef] [PubMed]
  43. Halstead, M.; Gray, N.; Gonzalez-Jimenez, N.; Fresquez, M.; Valentin-Blasini, L.; Watson, C.; Pappas, R.S. Analysis of Toxic Metals in Electronic Cigarette Aerosols Using a Novel Trap Design. J. Anal. Toxicol. 2020, 44, 149–155. [Google Scholar] [CrossRef]
  44. Havermans, A.; Krüsemann, E.J.Z.; Pennings, J.; de Graaf, K.; Boesveldt, S.; Talhout, R. Nearly 20 000 e-liquids and 250 unique flavour descriptions: An overview of the Dutch market based on information from manufacturers. Tob. Control 2021, 30, 57–62. [Google Scholar] [CrossRef] [Green Version]
  45. Krusemann, E.J.Z.; Boesveldt, S.; De Graaf, K.; Talhout, R. An E-Liquid Flavor Wheel: A Shared Vocabulary Based on Systematically Reviewing E-Liquid Flavor Classifications in Literature. Nicotine Tob. Res. 2019, 21, 1310–1319. [Google Scholar] [CrossRef]
  46. Henderson, L.K.; Craig, J.C.; Willis, N.S.; Tovey, D.; Webster, A.C. How to write a Cochrane systematic review. Nephrology 2010, 15, 617–624. [Google Scholar] [CrossRef]
  47. Institute of Medicine. Finding What Works in Health Care: Standards for Systematic Reviews; National Academies Press (US): Washington, DC, USA, 2011. [Google Scholar]
  48. Page, M.J.; Shamseer, L.; Altman, D.G.; Tetzlaff, J.; Sampson, M.; Tricco, A.C.; Catala-Lopez, F.; Li, L.; Reid, E.K.; Sarkis-Onofre, R.; et al. Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study. PLoS Med. 2016, 13, e1002028. [Google Scholar] [CrossRef] [Green Version]
  49. Ma, L.L.; Wang, Y.Y.; Yang, Z.H.; Huang, D.; Weng, H.; Zeng, X.T. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: What are they and which is better? Mil. Med. Res. 2020, 7, 7. [Google Scholar] [CrossRef] [Green Version]
  50. De Santis, K.K.; Lorenz, R.C.; Lakeberg, M.; Matthias, K. The application of AMSTAR2 in 32 overviews of systematic reviews of interventions for mental and behavioural disorders: A cross-sectional study. Res. Synth. Methods 2022, 13, 424–433. [Google Scholar] [CrossRef]
Ijerph 20 01908 g001 550
Figure 1. PRISMA flow diagram of the study selection process for the umbrella review.
Figure 1. PRISMA flow diagram of the study selection process for the umbrella review.
Ijerph 20 01908 g001
Table
Table 1. Characteristics of Included Systematic Reviews.
Table 1. Characteristics of Included Systematic Reviews.
Characteristics Citations
Funding Sourcen = 25
Federal11[6,14,18,19,22,23,27,28,29,33,35]
Non-Tobacco Affiliated Organization4[19,28,31,36]
Tobacco Affiliated Organization3[18,20,25]
No Specific Grant from Any Funding Agency5[13,15,21,30,34]
Not Stated5[16,17,24,26,32]
Number of Studies Includedn = 25
Not stated1[36]
<101[22]
10–5015[6,13,14,16,17,20,23,24,27,28,29,30,32,33,34]
51–1007[15,18,19,21,25,31,35]
100+1[26]
Number of Databases Searchedn = 25
111[17,18,20,21,22,24,25,26,29,30,35]
22[23,33]
35[13,14,15,16,36]
4+7[6,19,27,28,31,32,34]
Sample Type
Chemical Profiles
E-liquid/EC Aerosol		n = 20
18[6,14,15,16,17,18,19,20,21,22,23,24,25,26,28,29,30,31]
Biosample8[13,14,15,16,18,19,26,27]
Toxicityn = 14
Human Cells13[13,15,18,19,21,23,26,28,32,33,34,35,36]
Animal Cells 6[15,19,21,26,32,35]
Not stated1[30]
A detailed summary of the included reviews can be found in Supplementary Tables S2 and S3.
Table
Table 2. AMSTAR-2 Quality Assessment Scores; Chemical Profiles and Toxicity Outcomes.
Table 2. AMSTAR-2 Quality Assessment Scores; Chemical Profiles and Toxicity Outcomes.
Research Question Included PICOReview Methods Established Prior to Review *Explanation of Study Design SelectionComprehensive Literature Search Strategy *Study Selection Performed in DuplicateData extraction Performed in DuplicateList of Excluded Studies with Justification *Describe Studies in Adequate DetailSatisfactory Technique for Assessing Rob RCT/NRSI *Report Sources of FundingUse appropriate Methods for Statistical Combination of Results (for meta-Analysis) RCT/NRSI *Assess potential Impact of RoB in Individual Studies on the Results (for Meta-Analysis)Account for RoB in Individual Studies When Interpreting Results *Explanation for and Discussion of Heterogeneity in ResultsInvestigate Publication Bias and Discuss Impact (for Quantitative Synthesis) *Reported Any Conflict of Interest
Farsalinos and Polosa, 2014 [21] NANA NA
Burstyn, 2014 [25] NA NANA NA
Cheng, 2014 [6] NANA NA
Harrell et al., 2014 [23] NANA NA
Pisinger and Dossing, 2014 [15] NA NANA NA
Ioakeimidis et al., 2016 [24] NANA NA
Glasser et al., 2017 [18] NANA NA
Zulkifli et al., 2018 [22] NA NANA NA
Farsalinos and Gillman, 2018 [20] NANA NA
Kaur et al., 2018 [19] NANA NA
Flach et al., 2019 [32] NA NANA NA
Gaur and Agnihotri, 2019 [16] NANA NA
Armendáriz-Castillo et al., 2019 [17] NANA NA
Bjurlin et al., 2020 [27] NANA NA
Bozier et al., 2020 [26] NANA NA
Lee et al., 2020 [28] NANA NA
§ Salam et al., 2020 [29]NA NA
Sharma and Verma, 2020 [30] NANA NA
Ward et al., 2020 [31]NA NA NANA NA
Zhao et al., 2020 [14] NA NANA NA
Yang et al., 2020 [13] NANA NA
Bravo-Gutierrez et al., 2021 [35] NANA NA
White et al., 2021 [36] NANA NA
Wills et al., 2021 [33] NA NANA NA
Wilson et al., 2022 [34] NANA NA
* Critical domains. § Meta-analysis. Green = yes; red = no; orange = partial yes. PICO = Population or participants, and conditions of interest; Interventions or exposures; Comparisons or control groups; Outcomes of interest; RCT = randomized controlled trials; NRSI = non-randomized studies of interventions; RoB = risk of bias; NA = not applicable. One review cautioned that several of their included studies heated e-liquids at a power level beyond the capability of most modern EC systems, potentially attributing to high levels of carbonyls [25]. Potentially higher carbonyl levels were also reported in e-liquid or EC aerosol samples that used vegetable glycerin solvent compared to propylene glycol solvent in one review [20].
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Travis, N.; Knoll, M.; Cook, S.; Oh, H.; Cadham, C.J.; Sánchez-Romero, L.M.; Levy, D.T. Chemical Profiles and Toxicity of Electronic Cigarettes: An Umbrella Review and Methodological Considerations. Int. J. Environ. Res. Public Health 2023, 20, 1908. https://doi.org/10.3390/ijerph20031908

AMA Style

Travis N, Knoll M, Cook S, Oh H, Cadham CJ, Sánchez-Romero LM, Levy DT. Chemical Profiles and Toxicity of Electronic Cigarettes: An Umbrella Review and Methodological Considerations. International Journal of Environmental Research and Public Health. 2023; 20(3):1908. https://doi.org/10.3390/ijerph20031908

Chicago/Turabian Style

Travis, Nargiz, Marie Knoll, Steven Cook, Hayoung Oh, Christopher J. Cadham, Luz María Sánchez-Romero, and David T. Levy. 2023. "Chemical Profiles and Toxicity of Electronic Cigarettes: An Umbrella Review and Methodological Considerations" International Journal of Environmental Research and Public Health 20, no. 3: 1908. https://doi.org/10.3390/ijerph20031908

APA Style

Travis, N., Knoll, M., Cook, S., Oh, H., Cadham, C. J., Sánchez-Romero, L. M., & Levy, D. T. (2023). Chemical Profiles and Toxicity of Electronic Cigarettes: An Umbrella Review and Methodological Considerations. International Journal of Environmental Research and Public Health, 20(3), 1908. https://doi.org/10.3390/ijerph20031908

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop