Next Article in Journal
Microbial Steroid Production Technologies: Current Trends and Prospects
Previous Article in Journal
What Is the Role of Gut Microbiota in Obesity Prevalence? A Few Words about Gut Microbiota and Its Association with Obesity and Related Diseases
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Microorganisms
Volume 10
Issue 1
10.3390/microorganisms10010054
microorganisms-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Identification of the Bacterial Pathogens in Children with Otitis Media: A Study in the Northwestern Portuguese District of Braga

by
Maria Daniela Silva
1,2,
António Lima
3,
Nuno Marçal
4,
Luís Dias
3,
Miguel Gama
1 and
Sanna Sillankorva
2,*
1
CEB—Centre of Biological Engineering, LIBRO—Laboratório de Investigação em Biofilmes Rosário Oliveira, University of Minho, 4710-057 Braga, Portugal
2
INL—International Iberian Nanotechnology Laboratory, Avenida Mestre José Veiga, 4715-330 Braga, Portugal
3
Department of Otolaryngology, Hospital de Braga, 4710-243 Braga, Portugal
4
Department of Otolaryngology, Trofa Saúde Hospital, 4715-196 Braga, Portugal
*
Author to whom correspondence should be addressed.
Microorganisms 2022, 10(1), 54; https://doi.org/10.3390/microorganisms10010054
Submission received: 4 December 2021 / Revised: 19 December 2021 / Accepted: 26 December 2021 / Published: 27 December 2021
(This article belongs to the Section Medical Microbiology)
Browse Figures
Versions Notes

Abstract

:
Understanding the bacterial etiology of otitis media (OM) is important when designing and evaluating the best course of treatment. This study analyzed middle ear fluid (MEF) and nasopharynx (NP) samples collected from 49 children with OM undergoing myringotomy in the northwestern Portuguese district of Braga. A correlation between species in the NP and MEF was observed following pathogen detection by culture and quantitative polymerase chain reaction (qPCR) methods. Bacterial identification using culturing methods showed that Moraxella catarrhalis was the most representative in NP and MEF, followed by Streptococcus pneumoniae. However, qPCR of MEF showed a higher prevalence (61%) of Haemophilus influenzae. S. pneumoniae was not the most frequently identified species, but it still remains one of the leading causes of OM in this region despite 93.9% of the children being vaccinated with the pneumococcal conjugate vaccine. Furthermore, 46% of the samples analyzed by qPCR identified more than two bacterial species. M. catarrhalis and S. pneumoniae were the most frequent combination identified in NP and MEF samples by culturing methods. Additionally, a few NP and MEF samples simultaneously presented the three main otopathogens. These results point out that polymicrobial infections play an important role in OM. Further studies characterizing the serotypes of the strains isolated, their resistance profile, and their biofilm forming ability would help in the development of more targeted strategies against otitis media.

Graphical Abstract

1. Introduction

Otitis media (OM) is a significant public health problem in childhood. Middle ear inflammation is the most frequent reason children visit their doctor, take antibiotics, and undergo surgery [1]. Acute otitis media (AOM) and otitis media with effusion (OME) are the two major sub-classifications of OM. In AOM, the presence of middle ear fluid (MEF) is accompanied by otalgia, otorrhea, and fever, among other signs and symptoms of an acute infection. In OME, which can be developed as a sequel to AOM or as a new onset after viral infection, MEF is present in the absence of signs and symptoms of infection, with the most common alteration detected being hearing loss [2]. Recurrent AOM (RAOM) are very common, as well as persistent OME (chronic OME, COME), with myringotomy and the insertion of ventilation tubes often being performed for their treatment [1].
As well as its clinical relevance, OM also has a significant economic impact associated with medical and nonmedical costs. For instance, in Portugal, the mean costs per doctor-diagnosed OM episode are estimated at 334 euros divided into three categories. The highest share is related to direct medical costs (59.6%), including emergency room visits, pediatrician visits, and antibiotic prescriptions. However, there are also direct nonmedical costs (1.5%), and either indirect nonmedical or travel-related costs (38.9%), e.g., due to the loss of productivity at work, absence from a paid job, or loss of leisure time, among others. Thus, the authors estimated an economic burden of 72 million euros per year to the Portuguese society [3]. In the USA, the annual direct costs associated with OM have been estimated at 3–5 billion US dollars, with indirect nonmedical costs further increasing this value. The cost of one episode can vary between USD 108 and USD 1330 [4].
The nasopharynx (NP) is a key element in the pathogenesis of otitis media, being a reservoir of bacteria that can cause a middle ear infection. A viral upper respiratory tract infection usually triggers the progression of the bacteria from commensal to pathogenic. Genetically similar bacteria have been isolated both from MEF and NP samples [5,6] and from adenoid tissue and MEF [7], suggesting a correlation between the nasopharynx/adenoids and the middle ear microenvironments.
The three major bacterial pathogens associated with OM are Streptococcus pneumoniae, Haemophilus influenzae, and Moraxella catarrhalis. According to a systematic review, S. pneumoniae is the most predominant in AOM patients, while H. influenzae is the most frequently identified in patients with COME, RAOM, and AOM with treatment failure. This result is consistently obtained using microbiological culturing techniques and polymerase chain reaction (PCR) [8]. However, the use of molecular methods with improved sensitivity compared to bacterial culture allows the identification of other bacterial pathogens. For instance, Alloiococcus otitis and Turicella otitidis have been detected by PCR or 16S rRNA gene sequencing with higher frequency from the MEF samples of OM patients [9,10,11,12,13]. Other bacterial species identified in OM cases with lower prevalence include, for instance, S. aureus and P. aeruginosa [8,11,14].
Studies on the identification and prevalence of the bacterial species associated with OM are critical to developing and evaluating therapeutic approaches and the impact of vaccination programs [8,15]. Bacterial culture has been the standard method for identifying the AOM bioburden. However, the percentage of culture-positive results decreases in MEF samples from children with RAOM, AOM with treatment failure, and, particularly, OME. The reduced growth of some bacterial species, the presence of inhibitory substances, previous antibiotic therapy, and biofilms are some factors that might affect the detection by culture methods [11,16,17]. PCR improves the sensitivity of pathogens’ detection in samples from middle ear infections, increasing the prevalence of positive culture findings and allowing the detection of new bacterial species, as described above [18,19].
In Portugal, to our knowledge, only one study evaluating the bacterial etiology of OM has been performed. This study focused on children with AOM with spontaneous otorrhea. Of the 113 samples from ear discharge analyzed, 48.7% were culture positive for bacteria. Among these, S. pneumoniae was present in 50.9% (multiple serotypes), Streptococcus pyogenes in 30.9%, M. catarrhalis in 27.3%, and H. influenzae in 20.0%. Furthermore, 25.4% of the samples presented two or more of these species [20].
This study aimed to determine the prevalence of otopathogens in children with OM undergoing myringotomy in the northwestern Portuguese district of Braga. MEF samples were studied with conventional culture and quantitative PCR (qPCR) techniques to detect S. pneumoniae, H. influenzae, M. catarrhalis, S. aureus, and P. aeruginosa. Additionally, bacterial culture of nasopharyngeal swabs was performed and compared with the results obtained with MEF samples from the same patient to establish a possible relationship between them.

2. Materials and Methods

2.1. Patients and Sample Collection

This study included children (<12 years of age) with a diagnosis of acute otitis media (AOM) or otitis media with effusion (OME) and indication for myringotomy attending the Public Hospital of Braga between January 2018 and February 2020. Patients with a history of malignancy, organ transplantation, systemic infectious diseases, and immune deficiency were excluded.
Samples from the nasopharynx (NP) and the middle ear fluid (MEF) were collected during the surgery. Patients’ relevant data were registered in a form by the physician responsible for the surgery collecting different data including age, gender, type of otitis media (acute, with effusion, recurrent, chronic), laterality, myringotomy (first intervention or reintervention), antibiotics prescribed prior to surgery, PCV vaccination status. NP samples were collected by gently rotating a sterile cotton wool swab in the nasopharynx, then placed on Amies transport media (Deltalab 300284 or VWR 710-0438P). Before the surgical procedure, the external ear was disinfected with povidone-iodine for 2 min, then a washing step with saline (NaCl 0.9%) was performed to eliminate the antiseptic agent. After myringotomy, MEF was aspirated using a suction device coupled to a collector (Vygon 5534.06). The ethical committee of the hospitals approved the study, and written consent was obtained from all patients involved.

2.2. Bacterial Culture

Samples were stored at 4 °C temperature and transported to the laboratory. Once at the lab, NP swabs were immediately cultured (as described below). At the same time, MEF samples were divided into two fractions: one was immediately used for bacterial culture, and another was stored at −80 °C for subsequent bacterial DNA extraction.
All NP swabs and MEF specimens were examined for specific bacterial species using conventional culture methods. Briefly, samples were streaked on 5% (v/v) sheep blood agar and chocolate agar and incubated at 37 °C with 5% CO2 for 24 to 48 h. They were also streaked on 5% (v/v) sheep blood agar and MacConkey agar and incubated at 37 °C, aerobically, for the same period.
Colonies were inoculated on the following selective media for identification of: (i) S. pneumoniae—sheep blood agar (Tryptic Soy Broth (TSB) + 1.2% (w/v) agar with 5% sheep blood) supplemented with colistin (5 µg/mL) and oxolinic acid (2.5 µg/L); (ii) H. influenzae—chocolate agar (TSB + 1.2% (w/v) agar with 5% horse blood heated at 55 °C) supplemented with vancomycin (5 µg/mL), clindamycin (1 ug/mL), and bacitracin (300 µg/mL); (iii) M. catarrhalis—chocolate agar with vancomycin (5 µg/mL), trimethoprim (3 µg/mL), amphotericin B (2 µg/mL) and acetazolamide (10 µg/mL); (iv) Staphylococcus aureus—mannitol salt agar (MSA); (v) Pseudomonas aeruginosa—Pseudomonas isolation agar (PIA).
Species identification was also based on specific tests, such as catalase and oxidase tests. S. pneumoniae was specifically identified by the optochin disk test and H. influenzae by the V and X factor growth requirements.

2.3. Specific Quantitative PCR of MEF Samples

Detection of specific individual bacterial pathogens in MEF samples was also evaluated by quantitative PCR.
Calibration curves were performed using DNA extracted from individual strains. Briefly, DNA was extracted from cultures of S. pneumoniae R6st, H. influenzae C894248, M. catarrhalis U225012, S. aureus ATCC 6358, and P. aeruginosa PAO1 (Table 1) using the E.Z.N.A. Bacterial DNA Kit (OMEGA bio-tek) following the manufacturer’s instructions. The concentrations and purity ratios were measured (Nanodrop), and the DNA integrity was confirmed by agarose gel electrophoresis (2% agarose gel). Initially, primers described by Sillanpaa et al. (2016) were tested by conventional PCR. Amplification was performed using primers (Table 2) [11]. Three pairs of primers targeting the 16S rRNA genes of S. pneumoniae, H. influenzae, and M. catarrhalis, originally reported by Hendolin et al. (1997), were also used [17]. The multiplex-PCR reaction described by the authors was adapted for KAPA-Taq DNA Polymerase conditions. The other three pairs of primers target the lytA gene of S. pneumoniae, the nucA gene of S. aureus, and the oprL gene of P. aeruginosa. Genes were amplified by PCR using KAPA-Taq DNA Polymerase following the manufacturer’s instructions, and amplification confirmed by agarose gel electrophoresis (Figure S1). Then, calibration curves by quantitative PCR were performed using 1:10 DNA dilutions for each bacterial species and Xpert Fast SYBR Mastermix (Grisp, Portugal) following the manufacturer’s instructions (Figures S2 and S3).
MEF samples (approximately 250 µL) were defrosted and spiked with 1 µL of a plasmid (empty pET-28a(+)) as an exogenous internal control. The DNA was extracted using the E.Z.N.A. Universal Pathogen Kit (OMEGA bio-tek) following the manufacturer’s instructions. The detection of the specific pathogens was performed by individual qPCR reactions using the primers described above and Xpert Fast SYBR Mastermix (Grisp, Portugal), in duplicate. The exogenous internal control was detected using the universal T7 primers (Forward (5′ → 3′): TAATACGACTCACTATAGGG; Reverse (5′ → 3′): GCTAGTTATTGCTCAGCGG).

3. Results

3.1. Study Population

From January 2018 to February 2020, 49 samples were collected from the nasopharynx (NP) and the middle ear fluid (MEF) of children with OM. The patients’ sex was well-distributed, and the mean age of the individuals was 3.5 years (±1.7), ranging from 1 to 9 years old, and distributed as in Figure 1.
Patients’ history questionnaires were filled in by the responsible doctors and are compiled in Table S1. Seven of the cases were classified as RAOM, while the other 42 were due to OME, of which 15 were COME. Moreover, 29 children had bilateral OM, while the remaining 20 had unilateral OM on the right (15) or left (5) ear. The myringotomy was performed for the first time, except in two cases where these were reinterventions (Table S1, patients 1 and 2). Furthermore, in only two cases, adenoidectomy was not performed at the same time as the myringotomy. Only three of these children had not been vaccinated with a pneumococcal conjugate vaccine (PCV). Moreover, 57% of the children had not taken antibiotics in the last six months. Most of the children taking antibiotics one week to six months before the myringotomy received amoxicillin/clavulanic acid, while a few received penicillin (2), ceftriaxone (1), or cefixime (1) six months prior to the surgery.

3.2. Identification of Bacterial Species

The presence of bacterial pathogens in NP and MEF of 49 children with OM was detected by culture, and MEF samples were further evaluated by qPCR.
Bacteria were isolated from 85.71% of NP samples using culture-based methods. M. catarrhalis was the most commonly isolated species (61.22%), followed by S. pneumoniae (51.02%), H. influenzae (38.78%), and S. aureus (32.65%) (Figure 2a). P. aeruginosa was not found in any NP samples. The presence of M. catarrhalis was often accompanied by S. pneumoniae (18.37%) (Figure 3a).
Concerning the MEF samples, otopathogens were identified in 93.88% of them, either by culture or qPCR, with 57.14% or 89.80% of the samples being Culture+ or PCR+, respectively, for at least one pathogen. The most common bacteria isolated by culture from the MEF (Figure 2b) was again M. catarrhalis (40.82%), followed by S. pneumoniae (38.78%), with H. influenzae and S. aureus being isolated from the same number of samples (14.29%). M. catarrhalis was once more frequently found in co-occurrence with S. pneumoniae in the MEF (Figure 3b). However, by qPCR methodology, H. influenzae was detected in higher numbers (61.22%) (Figure 2b). Both M. catarrhalis and S. aureus were detected in 32.65% of samples, while S. pneumoniae was detected in 28.57% of the individuals. Regarding P. aeruginosa, its presence was detected in 10.20% of samples. Overall, combining the two methods (samples Culture+ or PCR+), H. influenzae was the most commonly identified pathogen (67.35%), followed by S. pneumoniae (57.14%), M. catarrhalis (51.02%), S. aureus (42.86%), and, lastly, P. aeruginosa (10.20%) (Figure 2b).
H. influenzae was detected alone by qPCR in 28.57% of the total MEF samples (Figure 3c). The presence of two or three different species was identified in 20.41% and 22.45% of samples, respectively, while in 4.08% of samples, four pathogens were detected.
More than half of the NP Culture+ isolates were also detected in the MEF sample of the same patient by culture or qPCR (Figure 4). For instance, 76.00% of the S. pneumoniae Culture+ NP samples were positive for S. pneumoniae in the MEF sample, of which 48.00% were culture positive (36.00% Culture+/PCR+ plus 12.00% Culture+/PCR+) and 40.00% qPCR positive (28.00% Culture−/PCR+ plus 12.00% Culture+/PCR+) (Figure 4b).

4. Discussion

The prevalence of bacterial pathogens in children with otitis media has been investigated in different European countries [21,22,23,24], but only one study (which analyzed samples from AOM cases with spontaneous otorrhea) has been performed in Portugal [20]. The implementation of the PCV has been reported to impact OM etiology. This vaccine was first licensed in Europe in 2001, and, since then, countries have been introducing it in their national immunization programs. It started as a 7-valent vaccine (PCV-7), protecting against seven pneumococcal serotypes, later substituted by the PCV-10, and then by the PCV-13 [25]. In Portugal, the PCV-13 was introduced in the routine childhood immunization program in 2015, with the different doses being administered at 2, 4, and 12 months of age [26], similarly to most European countries [27]. Before that, it was recommended and free to children at increased risk for pneumococcal disease [28]. The percentage of children immunized in Europe is high (80%) [29], as were those from which samples were collected (93.88%). After the implementation of PCV, studies reported a change in pathogen dominance from S. pneumoniae to H. influenzae (in France and Greece) [21,22], as well as the replacement of S. pneumoniae serotypes covered by the vaccine by others (in France and Spain) [21,24,30]. Moreover, the high use and misuse of antibiotics may have impacted the prevalence of some bacterial species and serotypes [22]. Amoxicillin–clavulanic acid, the gold standard of otitis media treatment, is the antibiotic most commonly used in Portugal, including in the pediatric population [31]. Importantly, 42.86% of the children included in this study had taken amoxicillin-clavulanic acid up to six months before the myringotomy. Despite this, the antibiotic consumption rate in Portugal is below (but near) the average of European countries [31].
In this study, M. catarrhalis was the most frequent pathogen isolated by culture from NP samples, being in agreement with studies that report this pathogen as the most common NP colonizer in both children with OM or who are healthy [16,32,33]. M. catarrhalis was frequently associated with S. pneumoniae. While colonization by M. catarrhalis alone is not reported to be associated with an increased risk of OM development, this is significantly enhanced when it co-colonizes the NP with S. pneumoniae [33]. The co-infection with M. catarrhalis affected the middle ear ascension of S. pneumoniae in mice and chinchilla models [34]. Moreover, the co-colonization with M. catarrhalis may negatively impact the treatment since β-lactamase-producing M. catarrhalis can confer passive protection to S. pneumoniae, promoting resistance to the first-line antibiotic amoxicillin [34].
Most of the samples analyzed are from OME cases (85.71%). Therefore, the identification of H. influenzae as the most prevalent pathogen in the MEF agrees with other studies that report H. influenzae as the predominant pathogen in OME [8,14,35]. However, the reported prevalence rates are highly variable due to different health care programs, particularly vaccination, in various geographical locations [36]. The fact that most children were immunized with a pneumococcal conjugate vaccine possibly contributed to a lower incidence of S. pneumoniae. The identification of S. aureus in 42.86% of samples may look somewhat surprising, but a recent study also reports a high prevalence (27.78%) of S. aureus in MEF samples from OME cases [14]. Nevertheless, differences in pathogens’ prevalence in studies conducted in the same country have been observed. For instance, two studies conducted in Iran reported a prevalence of H. influenzae of 95.2% in the city of Shiraz [37] and 11% in the city of Tehran [38] using PCR methods. The diversity of the results available may also result from variability in sample collection, the population studied, specific disease stage, and, of course, the detection techniques used (culture, PCR, next-generation sequencing) [36].
PCR methods usually improve the detection of bacteria in MEF samples to a magnitude that varies among the available reports [8,11,39]. Indeed, in the present study, the detection of H. influenzae was largely improved using qPCR compared to culturing (Figure 2b) by 4.3 times. This is in agreement with previous studies, where, for instance, increased detection of H. influenzae of five [40] or ten [19] times using PCR methods compared to culture was reported. H. influenzae is a fastidious organism that is difficult to culture, which could have affected its detection by this approach. Moreover, the fact that PCR methods do not depend on bacterial viability could also contribute to their increased sensitivity [19]. However, it has been shown that the DNA of nonviable bacteria does not persist in the MEF and cannot be detected by PCR 3 days after inoculation in the ME of chinchillas [41]. In contrast, antibiotic-treated bacteria persist in the MEF for weeks and can be detected by PCR but not by culture [41]. Furthermore, the presence of bacterial biofilms has been extensively pointed to being one of the main reasons for Culture−/PCR+ results [42,43].
However, qPCR did not increase the detection of S. pneumoniae and M. catarrhalis, which might be related to the DNA extraction method. It should be noted that, in some cases, only a few colonies (one or two) were isolated from the MEF samples, and the levels of bacteria in the sample might not have been enough for successful DNA extraction. While this hypothesis could be valid, we used an exogenous internal control, consisting of empty plasmid pET-28a(+), to safeguard against the loss of DNA during the extraction procedure or the presence of PCR inhibitors in the sample. This control was successfully detected in 91.84% of the samples. The nature of the sample, such as the viscosity, or the presence of inhibitors, such as blood, might have interfered in the DNA extraction process and may explain the lack of success in detecting the exogenous internal control in 8.16% of samples.
In 28.57% of the MEF samples, H. influenzae was detected alone. However, two different species were present in 20.40% of the samples (4.08% S. pneumoniae and S. aureus, 4.08% H. influenzae and M. catarrhalis, 2.04% S. pneumoniae and H. influenzae, 2.04% S. pneumoniae and M. catarrhalis, 2.04% H. influenzae and S. aureus, 2.04% H. influenzae and P. aeruginosa, 2.04% M. catarrhalis and S. aureus, 2.04% S. aureus and P. aeruginosa). Furthermore, 22.45% of samples presented three different bacterial species (6.12% S. pneumoniae, H. influenzae, and M. catarrhalis; 4.08% S. pneumoniae, H. influenzae, and S. aureus; 4.08% S. pneumoniae, M. catarrhalis, and S. aureus; 4.08% H. influenzae, M. catarrhalis, and S. aureus; 2.04% H. influenzae, M. catarrhalis, and P. aeruginosa; 2.04% H. influenzae, S. aureus, and P. aeruginosa). In 4.08% of samples, four pathogens were detected (2.04% S. pneumoniae, H. influenzae, M. catarrhalis, and S. aureus; 2.04% S. pneumoniae, H. influenzae, S. aureus, and P. aeruginosa). These show that polymicrobial infections play an important role in OM. Since most of the MEF samples correspond to OME cases (42 of 49), of which at least 15 are chronic, this is in accordance with the disease continuum hypothesis. According to this theory, S. pneumoniae is often involved in an initial AOM episode, which causes mucosal damage and predisposes children to subsequent more complex episodes of RAOM and COME. These most frequently involve non-typeable H. influenzae alone or a mixed-species infection that can be present in the form of biofilms [44].
NP colonization is a usual first step in OM development [5]; therefore, a correlation between species present in the NP and MEF was already expected. However, recent studies suggest that the NP reservoir theory may not explain all etiologies of MEF production [36]. Next-generation sequencing studies have reported similarities [45] and dissimilarities [46] between the MEF and the adenoidal microbiome in children with OME. Therefore, more extensive investigations are needed to validate or disprove the existing theory.
This study presented some limitations. The bioburden could not be quantified with accuracy since the volume of MEF varied between samples, and some of them were diluted in saline during the collection procedure, while others were not. Therefore, qPCR was used only as a qualitative method instead of quantitative. Moreover, using the universal SYBR Green fluorescent dye instead of specific probes for each bacterial pathogen could have reduced the specificity of the qPCR reactions. Furthermore, the presence of other pathogens, such as A. otitis, which may play an important role in both AOM and OME [47,48], was not assessed. Indeed, novel species, particularly Alloiococcus and Turicella, are being identified at significant relative abundances in MEF samples using metagenomic DNA sequencing, which captures all available DNA in a non-discriminatory manner [36].

5. Conclusions

To our knowledge, this is the first study investigating the prevalence of bacterial pathogens in the middle ear fluid of children with otitis media undergoing myringotomy in Portugal. The majority of the cases had a known diagnosis corresponding to otitis media with effusion, with H. influenzae being the most prevalent causative agent. qPCR was substantially more sensitive than culturing to detect this pathogen, showing the importance of using molecular methods. However, this was not valid for S. pneumoniae and M. catarrhalis, which were more frequently detected by culture. The development of H. influenzae vaccines, which currently only target H. influenzae type B, will be important to reduce the burden of the disease. Moreover, a better understanding of the prevalence of these pathogens and others not included in this study could be achieved by using next-generation sequencing methods, which would give information about the whole middle ear microbiome.

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/microorganisms10010054/s1, Figure S1: Agarose gel electrophoresis of amplified genes, Figure S2: Standard curves and melting curves for qPCR quantification of the internal exogenous control (pET-28a(+)), Figure S3: Standard curves and melting curves for qPCR quantification of middle ear fluid bacterial pathogens, Table S1: Demographic information (age, sex) and clinical characteristics of the study subjects.

Author Contributions

Conceptualization and methodology: M.D.S., S.S.; investigation: M.D.S., A.L., N.M., L.D.; data curation and formal analysis: M.D.S., S.S.; writing—original draft preparation: M.D.S.; writing—review and editing: M.G., S.S.; funding acquisition, resources, and supervision: S.S. All authors have read and agreed to the published version of the manuscript.

Funding

This study was supported by the Portuguese Foundation for Science and Technology (FCT) under the scope of the strategic funding of UID/BIO/04469/2020 unit. MDS acknowledges support from the doctoral fellowship from the Portuguese Foundation for Science and Technology (FCT), reference SFRH/BD/128825/2017. SS acknowledges funding by FCT, through the individual scientific employment program contract (2020.03171.CEECIND).

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the ethical committee of Hospital of Braga (28.11.2017). Demographic data and medical history of patients prior to surgery were collected from their medical records.

Informed Consent Statement

Patient consent was waived since they had consented the surgery and the study does not identify the participating patients.

Data Availability Statement

Not applicable.

Acknowledgments

The authors acknowledge the Department of Otolaryngology of Hospital de Braga for collecting the nasopharyngeal swabs and the middle ear fluid samples from children with otitis media.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Silva, M.D.; Sillankorva, S. Otitis media pathogens–A life entrapped in biofilm communities. Crit. Rev. Microbiol. 2019, 45, 595–612. [Google Scholar] [CrossRef]
  2. Schilder, A.G.M.; Chonmaitree, T.; Cripps, A.W.; Rosenfeld, R.M.; Casselbrant, M.L.; Haggard, M.P.; Venekamp, R.P. Otitis media. Nat. Rev. Dis. Prim. 2016, 2, 16063. [Google Scholar] [CrossRef] [PubMed]
  3. Speets, A.; Wolleswinkel, J.; Cardoso, C. Societal costs and burden of otitis media in Portugal. J. Multidiscip. Healthc. 2011, 4, 53. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Rovers, M.M. The burden of otitis media. Vaccine 2008, 26, G2–G4. [Google Scholar] [CrossRef]
  5. Stol, K.; Verhaegh, S.J.C.; Graamans, K.; Engel, J.A.M.; Sturm, P.D.J.; Melchers, W.J.G.; Meis, J.F.; Warris, A.; Hays, J.P.; Hermans, P.W.M. Microbial profiling does not differentiate between childhood recurrent acute otitis media and chronic otitis media with effusion. Int. J. Pediatr. Otorhinolaryngol. 2013, 77, 488–493. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. Chonmaitree, T.; Jennings, K.; Golovko, G.; Khanipov, K.; Pimenova, M.; Patel, J.A.; McCormick, D.P.; Loeffelholz, M.J.; Fofanov, Y. Nasopharyngeal microbiota in infants and changes during viral upper respiratory tract infection and acute otitis media. PLoS ONE 2017, 12, e0180630. [Google Scholar] [CrossRef]
  7. Emaneini, M.; Gharibpour, F.; Khoramrooz, S.S.; Mirsalehian, A.; Jabalameli, F.; Darban-Sarokhalil, D.; Mirzaii, M.; Sharifi, A.; Taherikalani, M. Genetic similarity between adenoid tissue and middle ear fluid isolates of Streptococcus pneumoniae, Haemophilus influenzae and Moraxella catarrhalis from Iranian children with otitis media with effusion. Int. J. Pediatric Otorhinolaryngol. 2013, 77, 1841–1845. [Google Scholar] [CrossRef] [PubMed]
  8. Ngo, C.C.; Massa, H.M.; Thornton, R.B.; Cripps, A.W. Predominant Bacteria Detected from the Middle Ear Fluid of Children Experiencing Otitis Media: A Systematic Review. PLoS ONE 2016, 11, e0150949. [Google Scholar] [CrossRef] [Green Version]
  9. Xu, Q.; Gill, S.; Xu, L.; Gonzalez, E.; Pichichero, M.E. Comparative Analysis of Microbiome in Nasopharynx and Middle Ear in Young Children With Acute Otitis Media. Front. Genet. 2019, 10, 1176. [Google Scholar] [CrossRef] [Green Version]
  10. Sillanpää, S.; Kramna, L.; Oikarinen, S.; Sipilä, M. Next-Generation Sequencing Combined with Specific PCR Assays To Determine the Bacterial 16S rRNA Gene Profiles of Middle Ear Fluid Collected from Children with Acute Otitis Media. mSphere 2017, 2, e00006-17. [Google Scholar] [CrossRef] [Green Version]
  11. Sillanpää, S.; Oikarinen, S.; Sipilä, M.; Kramna, L.; Rautiainen, M.; Huhtala, H.; Aittoniemi, J.; Laranne, J.; Hyöty, H.; Cinek, O. Moraxella catarrhalis Might Be More Common than Expected in Acute Otitis Media in Young Finnish Children. J. Clin. Microbiol. 2016, 54, 2373–2379. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  12. Chan, C.L.; Wabnitz, D.; Bardy, J.J.; Bassiouni, A.; Wormald, P.; Vreugde, S.; Psaltis, A.J. The Microbiome of Otitis Media With Effusion. Laryngoscope 2016, 126, 2844–2851. [Google Scholar] [CrossRef] [PubMed]
  13. Holder, R.C.; Kirse, D.J.; Evans, A.K.; Whigham, A.S.; Peters, T.R.; Poehling, K.A.; Swords, W.E.; Reid, S.D. Otopathogens detected in middle ear fluid obtained during tympanostomy tube insertion: Contrasting purulent and non-purulent effusions. PLoS ONE 2015, 10, e0128606. [Google Scholar] [CrossRef] [Green Version]
  14. Niedzielski, A.; Chmielik, L.P.; Stankiewicz, T. The Formation of Biofilm and Bacteriology in Otitis Media with Effusion in Children: A Prospective Cross-Sectional Study. Int. J. Environ. Res. Public Health 2021, 18, 3555. [Google Scholar] [CrossRef]
  15. Jervis-bardy, J.; Rogers, G.B.; Morris, P.S.; Smith-vaughan, H.C.; Nosworthy, E.; Leong, L.E.X.; Smith, R.J.; Weyrich, L.S.; De Haan, J.; Carney, A.S.; et al. The microbiome of otitis media with effusion in Indigenous Australian children. Int. J. Pediatr. Otorhinolaryngol. 2015, 79, 1548–1555. [Google Scholar] [CrossRef]
  16. Wiertsema, S.P.; Kirkham, L.-A.S.; Corscadden, K.J.; Mowe, E.N.; Bowman, J.M.; Jacoby, P.; Francis, R.; Vijayasekaran, S.; Coates, H.L.; Riley, T.V.; et al. Predominance of nontypeable Haemophilus influenzae in children with otitis media following introduction of a 3 + 0 pneumococcal conjugate vaccine schedule. Vaccine 2011, 29, 5163–5170. [Google Scholar] [CrossRef]
  17. Hendolin, P.H.; Markkanen, A.; Ylikoski, J.; Wahlfors, J.J. Use of multiplex PCR for simultaneous detection of four bacterial species in middle ear effusions. J. Clin. Microbiol. 1997, 35, 2854–2858. [Google Scholar] [CrossRef] [Green Version]
  18. Hendolin, P.H.; Paulin, L.; Ylikoski, J.; Hendolin, P.H.; Paulin, L. Clinically Applicable Multiplex PCR for Four Middle Ear Pathogens. J. Clin. Microbiol. 2000, 38, 125–132. [Google Scholar] [CrossRef]
  19. Yatsyshina, S.; Mayanskiy, N.; Shipulina, O.; Kulichenko, T.; Alyabieva, N.; Katosova, L.; Lazareva, A.; Skachkova, T.; Elkina, M.; Matosova, S.; et al. Detection of respiratory pathogens in pediatric acute otitis media by PCR and comparison of fi ndings in the middle ear and nasopharynx. Diagn. Microbiol. Infect. Dis. 2016, 85, 125–130. [Google Scholar] [CrossRef]
  20. Rodrigues, F.; Morales-aza, B.; Turner, K.M.E.; Sikora, P.; Gould, K.; Hinds, J.; Finn, A. Multiple Streptococcus pneumoniae Serotypes in Aural Discharge Samples from Children with Acute Otitis Media with Spontaneous. J. Clin. Microbiol. 2013, 51, 3409–3411. [Google Scholar] [CrossRef] [Green Version]
  21. Dupont, D.; Mahjoub-Messai, F.; François, M.; Doit, C.; Mariani-Kurkdjian, P.; Bidet, P.; Bonacorsi, S.; Carol, A.; Bingen, E. Evolving microbiology of complicated acute otitis media before and after introduction of the pneumococcal conjugate vaccine in France. Diagn. Microbiol. Infect. Dis. 2010, 68, 89–92. [Google Scholar] [CrossRef]
  22. Stamboulidis, K.; Chatzaki, D.; Poulakou, G.; Ioannidou, S.; Lebessi, E.; Katsarolis, I.; Sypsa, V.; Tsakanikos, M.; Kafetzis, D.; Tsolia, M.N. The Impact of the Heptavalent Pneumococcal Conjugate Vaccine on the Epidemiology of Acute Otitis Media Complicated by Otorrhea. Pediatr. Infect. Dis. J. 2011, 30, 551–555. [Google Scholar] [CrossRef] [PubMed]
  23. Grevers, G.; Wiedemann, S.; Bohn, J.-C.; Blasius, R.-W.; Harder, T.; Kroeniger, W.; Vetter, V.; Pirçon, J.-Y.; Marano, C. Identification and characterization of the bacterial etiology of clinically problematic acute otitis media after tympanocentesis or spontaneous otorrhea in German children. BMC Infect. Dis. 2012, 12, 312. [Google Scholar] [CrossRef] [Green Version]
  24. Pumarola, F.; Salamanca de la Cueva, I.; Sistiaga-Hernando, A.; García-Corbeira, P.; Moraga-Llop, F.A.; Cardelús, S.; McCoig, C.; Gómez Martínez, J.R.; Rosell Ferrer, R.; Iniesta Turpin, J.; et al. Etiología bacteriana de la otitis media aguda en España en la era de la vacuna neumocócica conjugada. An. Pediatr. 2016, 85, 224–231. [Google Scholar] [CrossRef] [PubMed]
  25. Tin Tin Htar, M.; Christopoulou, D.; Schmitt, H.-J. Pneumococcal serotype evolution in Western Europe. BMC Infect. Dis. 2015, 15, 419. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  26. Portuguese Directorate-General of Health. Programa Nacional de Vacinação. 2017. Available online: https://www.sns.gov.pt/wp-content/uploads/2016/06/programa_vacinacao_sns.pdf (accessed on 17 December 2021).
  27. European Centre for Disease Prevention and Control. Pneumococcal Disease: Recommended Vaccinations. Available online: https://vaccine-schedule.ecdc.europa.eu/Scheduler/ByDisease?SelectedDiseaseId=25&SelectedCountryIdByDisease=-1 (accessed on 17 December 2021).
  28. Tendais-Almeida, M.; Ferreira-Magalhães, M.; Alves, I.; Tavares, M.; Azevedo, I. Vacinação Contra Infeções por Streptococcus pneumoniae em Crianças e Adolescentes de Alto Risco para Doença Invasiva Pneumocócica. Acta Med. Port. 2015, 28, 583. [Google Scholar] [CrossRef] [Green Version]
  29. Stewart, C. Share of Children Who Received the Pneumococcal Conjugate Vaccine* (PCV) in Selected European Countries in 2018. 2020. Available online: https://www.statista.com/statistics/1122694/pcv-immunization-in-europe/ (accessed on 17 December 2021).
  30. Pumarola, F.; Marès, J.; Losada, I.; Minguella, I.; Moraga, F.; Tarragó, D.; Aguilera, U.; Casanovas, J.M.; Gadea, G.; Trías, E.; et al. Microbiology of bacteria causing recurrent acute otitis media (AOM) and AOM treatment failure in young children in Spain: Shifting pathogens in the post-pneumococcal conjugate vaccination era. Int. J. Pediatr. Otorhinolaryngol. 2013, 77, 1231–1236. [Google Scholar] [CrossRef] [PubMed]
  31. Silva, A.; Oliveira, R. Antibióticos em Portugal. 2016. Available online: https://www.infarmed.pt/documents/15786/2219894/Antibi%ff%ffticos%2bEspectro%2blargo%2be%2bestreito%2bem%2bambulat%ff%ffrio%2b%282014-2016%29/5817b82c-345c-4e7d-aea8-4b4072937a77 (accessed on 17 December 2021).
  32. Smith-Vaughan, H.C.; Binks, M.J.; Marsh, R.L.; Kaestli, M.; Ward, L.; Hare, K.M.; Pizzutto, S.J.; Thornton, R.B.; Morris, P.S.; Leach, A.J. Dominance of Haemophilus influenzae in ear discharge from Indigenous Australian children with acute otitis media with tympanic membrane perforation. BMC Ear Nose Throat Disord. 2013, 13, 12. [Google Scholar] [CrossRef] [Green Version]
  33. Ruohola, A.; Pettigrew, M.M.; Lindholm, L.; Jalava, J.; Räisänen, K.S.; Vainionpää, R.; Waris, M.; Tähtinen, P.A.; Laine, M.K.; Lahti, E.; et al. Bacterial and viral interactions within the nasopharynx contribute to the risk of acute otitis media. J. Infect. 2013, 66, 247–254. [Google Scholar] [CrossRef]
  34. Perez, A.C.; Pang, B.; King, L.B.; Tan, L.; Murrah, K.A.; Reimche, J.L.; Wren, J.T.; Richardson, S.H.; Ghandi, U.; Swords, W.E. Residence of Streptococcus pneumoniae and Moraxella catarrhalis within polymicrobial biofilm promotes antibiotic resistance and bacterial persistence In Vivo. Pathog. Dis. 2014, 70, 280–288. [Google Scholar] [CrossRef] [Green Version]
  35. Jörissen, J.; van den Broek, M.F.L.; De Boeck, I.; Van Beeck, W.; Wittouck, S.; Boudewyns, A.; Van de Heyning, P.; Topsakal, V.; Van Rompaey, V.; Wouters, I.; et al. Case-Control Microbiome Study of Chronic Otitis Media with Effusion in Children Points at Streptococcus salivarius as a Pathobiont-Inhibiting Species. mSystems 2021, 6, e00056-21. [Google Scholar] [CrossRef]
  36. Nogues, J.C.; Pérez-Losada, M.; Preciado, D. Review of otitis media microbiome studies: What do they tell us? Laryngoscope Investig. Otolaryngol. 2020, 5, 936–940. [Google Scholar] [CrossRef]
  37. Shishegar, M.; Faramarzi, A.; Kazemi, T.; Bayat, A.; Motamedifar, M. Polymerase chain reaction, bacteriologic detection and antibiogram of bacteria isolated from otitis media with effusion in children, Shiraz, Iran. Iran. J. Med. Sci. 2011, 36, 273–280. [Google Scholar] [PubMed]
  38. Khoramrooz, S.S.; Mirsalehian, A.; Emaneini, M.; Jabalameli, F.; Aligholi, M.; Saedi, B.; Bazargani, A.; Taherikalani, M.; Borghaei, P.; Razmpa, E. Frequency of Alloicoccus otitidis, Streptococcus pneumoniae, Moraxella catarrhalis and Haemophilus influenzae in children with otitis media with effusion (OME) in Iranian patients. Auris Nasus Larynx 2012, 39, 369–373. [Google Scholar] [CrossRef]
  39. Mills, N.; Best, E.J.; Murdoch, D.; Souter, M.; Neeff, M.; Anderson, T.; Salkeld, L.; Ahmad, Z.; Mahadevan, M.; Barber, C.; et al. What is behind the ear drum? The microbiology of otitis media and the nasopharyngeal flora in children in the era of pneumococcal vaccination. J. Paediatr. Child Health 2015, 51, 300–306. [Google Scholar] [CrossRef]
  40. Ueyama, T.; Kurono, Y.; Shirabe, K.; Takeshita, M.; Mogi, G. High incidence of Haemophilus influenzae in nasopharyngeal secretions and middle ear effusions as detected by PCR. J. Clin. Microbiol. 1995, 33, 1835–1838. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  41. Post, J.C.; Aul, J.J.; White, G.J.; Wadowsky, R.M.; Zavoral, T.; Tabari, R.; Kerber, B.; Doyle, W.J.; Ehrlich, G.D. PCR-based detection of bacterial DNA after antimicrobial treatment is indicative of persistent, viable bacteria in the chinchilla model of otitis media. Am. J. Otolaryngol. 1996, 17, 106–111. [Google Scholar] [CrossRef]
  42. Post, J.C. Direct Evidence of Bacterial Biofilms in Otitis Media. Laryngoscope 2001, 111, 2083–2094. [Google Scholar] [CrossRef] [PubMed]
  43. Hall-Stoodley, L.; Hu, F.Z.; Gieseke, A.; Nistico, L.; Nguyen, D.; Hayes, J.; Forbes, M.; Greenberg, D.P.; Dice, B.; Burrows, A.; et al. Direct detection of bacterial biofilms on the middle-ear mucosa of children with chronic otitis media. JAMA 2006, 296, 202. [Google Scholar] [CrossRef] [Green Version]
  44. Dagan, R.; Pelton, S.; Bakaletz, L.; Cohen, R. Prevention of early episodes of otitis media by pneumococcal vaccines might reduce progression to complex disease. Lancet Infect. Dis. 2016, 16, 480–492. [Google Scholar] [CrossRef]
  45. Chan, C.L.; Wabnitz, D.; Bassiouni, A.; Wormald, P.J.; Vreugde, S.; Psaltis, A.J. Identification of the bacterial reservoirs for the middle ear using phylogenic analysis. JAMA Otolaryngol.-Head Neck Surg. 2017, 143, 155–161. [Google Scholar] [CrossRef] [Green Version]
  46. Johnston, J.; Hoggard, M.; Biswas, K.; Astudillo-García, C.; Radcliff, F.J.; Mahadevan, M.; Douglas, R.G. Pathogen reservoir hypothesis investigated by analyses of the adenotonsillar and middle ear microbiota. Int. J. Pediatr. Otorhinolaryngol. 2019, 118, 103–109. [Google Scholar] [CrossRef] [PubMed]
  47. Harimaya, A.; Takada, R.; Hendolin, P.H.; Fujii, N.; Ylikoski, J.; Himi, T. High Incidence of Alloiococcus otitidis in Children with Otitis Media, Despite Treatment with Antibiotics. J. Clin. Microbiol. 2006, 44, 946–949. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  48. Korona-Glowniak, I.; Wisniewska, A.; Juda, M.; Kielbik, K.; Niedzielska, G.; Malm, A. Bacterial aetiology of chronic otitis media with effusion in children-risk factors. J. Otolaryngol.-Head Neck Surg. 2020, 49, 24. [Google Scholar] [CrossRef] [PubMed]
Microorganisms 10 00054 g001 550
Figure 1. Distribution of samples based on patients’ sex (a) and age (b).
Figure 1. Distribution of samples based on patients’ sex (a) and age (b).
Microorganisms 10 00054 g001
Microorganisms 10 00054 g002 550
Figure 2. Number of bacterial isolates (percentage of total patients/samples) identified by (a) culture in nasopharyngeal samples and (b) culture and qPCR in middle ear fluid samples from children with OM. * The total corresponds to the sum of Culture+ and PCR− with Culture− and PCR+ and Culture+ and PCR+ samples.
Figure 2. Number of bacterial isolates (percentage of total patients/samples) identified by (a) culture in nasopharyngeal samples and (b) culture and qPCR in middle ear fluid samples from children with OM. * The total corresponds to the sum of Culture+ and PCR− with Culture− and PCR+ and Culture+ and PCR+ samples.
Microorganisms 10 00054 g002
Microorganisms 10 00054 g003 550
Figure 3. Species combinations in NP and MEF: (a) Species in NP identified by culture; (b) Species in MEF identified by culture; (c) Species in MEF identified by qPCR.
Figure 3. Species combinations in NP and MEF: (a) Species in NP identified by culture; (b) Species in MEF identified by culture; (c) Species in MEF identified by qPCR.
Microorganisms 10 00054 g003
Microorganisms 10 00054 g004 550
Figure 4. Correspondence between the nasopharynx and the middle ear fluid: (a) Number of bacterial isolates (percentage of total patients/samples) detected in the nasopharynx and middle ear fluid of the same patient; (b) Percentage of bacterial isolates detected in the middle ear fluid that were isolated from the nasopharynx.
Figure 4. Correspondence between the nasopharynx and the middle ear fluid: (a) Number of bacterial isolates (percentage of total patients/samples) detected in the nasopharynx and middle ear fluid of the same patient; (b) Percentage of bacterial isolates detected in the middle ear fluid that were isolated from the nasopharynx.
Microorganisms 10 00054 g004
Table
Table 1. Bacterial strains and growth conditions.
Table 1. Bacterial strains and growth conditions.
SpeciesStrainOrigin or SourceGrowth ConditionsCulture Media
S. pneumoniaeR6stFélix d’Hérelle Reference Center for Bacterial Viruses37 °C, 5% CO2Todd Hewitt Broth + 2% (w/v) yeast extract
H. influenzaeC894248Sputum, Hospital de Braga37 °C, 5% CO2Brain Heart Infusion Broth + 10 µg/mL NAD + 10 µg/mL Hemin
M. catarrhalisU225012Ocular, Hospital de Braga37 °C, 5% CO2Brain Heart Infusion Broth
S. aureusATCC 6358Human lesion, American Type Culture Collection37 °CTryptic Soy Broth
P. aeruginosaPAO1 (DSM 22644)Infected wound, DSMZ—German Collection of Microorganisms and Cell Cultures GmbH37 °CTryptic Soy Broth
Table
Table 2. List of primers for specific PCR detection of target pathogens.
Table 2. List of primers for specific PCR detection of target pathogens.
TargetPrimer NamePrimer Sequence (5′ → 3′)Product Size
S. pneumoniae, oralis, mitis and infantis 16S rRNA geneSpomi_FWAAGGTGCACTTGCATCACTACC484
Common_RVCTACGCATTTCACCGCTACAC
H. influenzae 16S rRNA geneHi_FWCGTATTATCGGAAGATGAAAGTGC525
Common_RVCTACGCATTTCACCGCTACAC
M. catarrhalis 16S rRNA geneMc_FWCCCATAAGCCCTGACGTTAC237
Common_RVCTACGCATTTCACCGCTACAC
S. pneumoniae lytA geneSp_FWACGCAATCTAGCAGATGAAGCA76
Sp_RVTCGTGCGTTTTAATTCCAGCT
S. aureus nucA geneSa_FWCATCCTAAAAAAGGTGTAGAGA85
Sa_RVTTCAATTTTMTTTGCATTTTCTACCA
P. aeruginosa oprL genePa_FWGGGTTCATTAGGAGTTACATGA544
Pa_RVGGGCATAACGACTTCTTACTTC
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Silva, M.D.; Lima, A.; Marçal, N.; Dias, L.; Gama, M.; Sillankorva, S. Identification of the Bacterial Pathogens in Children with Otitis Media: A Study in the Northwestern Portuguese District of Braga. Microorganisms 2022, 10, 54. https://doi.org/10.3390/microorganisms10010054

AMA Style

Silva MD, Lima A, Marçal N, Dias L, Gama M, Sillankorva S. Identification of the Bacterial Pathogens in Children with Otitis Media: A Study in the Northwestern Portuguese District of Braga. Microorganisms. 2022; 10(1):54. https://doi.org/10.3390/microorganisms10010054

Chicago/Turabian Style

Silva, Maria Daniela, António Lima, Nuno Marçal, Luís Dias, Miguel Gama, and Sanna Sillankorva. 2022. "Identification of the Bacterial Pathogens in Children with Otitis Media: A Study in the Northwestern Portuguese District of Braga" Microorganisms 10, no. 1: 54. https://doi.org/10.3390/microorganisms10010054

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop