Next Article in Journal
Quantifying Grain-Size Variability of Metal Pollutants in Road-Deposited Sediments Using the Coefficient of Variation
Previous Article in Journal
Combined Effects of High-Speed Railway Noise and Ground Vibrations on Annoyance
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Multiyear Model of Influenza Vaccination in the United States

1
Information Systems and Operations Management Department, Sawyer Business School, Suffolk University, Boston, MA 02108, USA
2
Alivia Technology, Boston, MA 02114, USA
3
Lexington High School, Lexington, MA 02421, USA
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2017, 14(8), 849; https://doi.org/10.3390/ijerph14080849
Submission received: 19 May 2017 / Revised: 24 July 2017 / Accepted: 25 July 2017 / Published: 28 July 2017
(This article belongs to the Section Global Health)

Abstract

:
Vaccinating adults against influenza remains a challenge in the United States. Using data from the Centers for Disease Control and Prevention, we present a model for predicting who receives influenza vaccination in the United States between 2012 and 2014, inclusive. The logistic regression model contains nine predictors: age, pneumococcal vaccination, time since last checkup, highest education level attained, employment, health care coverage, number of personal doctors, smoker status, and annual household income. The model, which classifies correctly 67 percent of the data in 2013, is consistent with models tested on the 2012 and 2014 datasets. Thus, we have a multiyear model to explain and predict influenza vaccination in the United States. The results indicate room for improvement in vaccination rates. We discuss how cognitive biases may underlie reluctance to obtain vaccination. We argue that targeted communications addressing cognitive biases could be useful for effective framing of vaccination messages, thus increasing the vaccination rate. Finally, we discuss limitations of the current study and questions for future research.

1. Introduction

Vaccine avoidance is a serious problem, but its severity is not yet recognized by many adults. Their refusal to vaccinate contributes to increased incidence of preventable diseases, such as measles, putting population health in danger [1,2]. Influenza, which the majority of adults survive without serious side effects, can result in hospitalization or death, particularly among the elderly [3].
The United States’ Centers for Disease Control and Prevention (CDC) describes the benefits of influenza vaccines for vulnerable populations, such as children, the elderly, the chronically ill, and pregnant women. Influenza vaccination reduces the risk of influenza-related hospitalization for children and the elderly, while preventing chronic disease-based hospitalization for people with health conditions such as diabetes or chronic lung disease. It can also help prevent cardiac events in people with cardiac disease. Influenza vaccination reduces the risk of respiratory infections for pregnant women, and may protect their infants through antibodies passed via breast milk. Finally, the CDC points out that influenza vaccination helps not only the recipient but also the community through herd immunity.
The CDC recommends influenza vaccination for everyone 6 months of age and older. The only contraindications are allergies to the influenza vaccine, eggs, or any of its ingredients, and a Guillain-Barré Syndrome (GBS) occurrence in the six weeks prior to influenza vaccination. Those who experience only hives as an allergic reaction to eggs are recommended to nevertheless receive the vaccine. Individuals with non-life threatening allergic reactions other than hives should be vaccinated under medical supervision. GBS is a rare autoimmune disorder associated with rapid onset muscle weakness, and sometimes paralysis [3]. Individuals who have a recent history of GBS, who are also at high risk for influenza complications, should nevertheless receive the influenza vaccination in some cases. Considering the rarity of GBS and egg allergies in adults, as well as the sub-populations within these groups who should nevertheless receive influenza vaccination, the vast majority of adults should obtain it annually [3].
There are many contributing factors to the problem of failure to vaccinate. The common factors include the following: vaccine efficacy vs. the unpleasantness of the vaccination process or side-effects, the perceived adequacy of one’s natural immunity against viruses, the cost in time or money, or a distrust in the medical establishment or pharmaceutical companies [1]. Many people may be complacent about influenza, believing their good current health is a strong enough defense against viruses [4]. Some may not be persuadable, even with the prospect of a severe influenza season and a steady campaign of appeals to personal health, public health, and the desire to protect one’s family. Other people, conversely, will obtain a vaccination readily, with only one or two reminders from their doctor [5]. Finally, there are in-between people, those who need several reminders from various sources.
A review of the literature on influenza vaccination determined the most common reasons given for (not) obtaining it. Descriptively, the top three reasons for seeking influenza vaccinations: self-protection, protection of family or community, and protecting the patients of health care workers. The top three reasons for declining influenza vaccination: perception that one is healthy and not at risk [6], skepticism regarding vaccine efficacy [7], and unpleasant side-effects. In sum, there are many factors contributing to the public’s beliefs, attitudes, intentions, and ultimately whether a vaccination is obtained.
Given so many plausible contributing factors, it would be useful to model rigorously the variety of factors contributing to the likelihood of receiving a seasonal influenza vaccination. To better communicate with the segment of the population who may be persuaded, we need to understand the variety of factors weighing on the decision to vaccinate, including demographics (education level, income level, age, and gender), behavior (nutrition, exercise, smoking, etc.), health care coverage, and current health status.
In this paper, we present a classification model about whether a person receives a flu vaccination. The data is from the CDC’s Behavioral Risk Factor Surveillance system (BRFSS) annual survey, a nationwide phone-based survey about health-related behaviors on adults [8]. The response variable is the binary variable Influenza Vaccination, answering the question “During the past 12 months, have you had either a flu shot or a flu vaccine?” We review our methods in Section 2 and show the results in Section 3. In Section 4, we discuss the results, some implications, possible communication strategies, and some ideas for further research. Section 5 concludes with a summary of the paper.

2. Methods

The BRFSS data is publically available online [8]. It is a health-related phone survey of adults (at least 18 years of age), and conducted in fifty states, the District of Columbia, and three U.S. territories. State health departments use in-house interviewers or contract with universities or telephone call centers to administer the surveys throughout the year. It is the largest continuous health survey in the world, and is offered in both English and Spanish. Participants are administered a standardized core questionnaire, as well as optional modules and state-specific questions. The survey is conducted using Random Digit Dialing on cell phones and land lines.
We used three recent years’ data of BRFSS: 2012 to 2014. First, the 2013 dataset was used to build the model. Specifically, a logistic regression was developed on 60% of the data from 2013, validated on a holdout sample of 20% of the data, and tested on the final 20% of the data. This is consistent with standard data mining practices to avoid model overfitting [9]. The best 2013 model was tested further on the 2012 and 2014 BRFSS datasets to assess whether predictors of vaccination changed over time. All data (sub)sampling was random, and there was no evidence of significant collinearity among the nine predictors in any model. Ultimately, the model was stable during the 2012–2014 time period.
We initially gathered data from the 2013 BRFSS dataset (n = 491,773) and performed transformations to clean and standardize the data. Some non-Gaussian numerical variables were transformed to be more Gaussian with a mathematical function, e.g., logarithm. Almost all the variables had a small fraction of Don’t Know/Not Sure, Refused to Answer, or Missing/No Answer/Blank. In our model, we grouped those responses into one category: Missing. For the variable Pneumococcal Vaccination, the proportion of Don’t Know/Not Sure was almost 10%, so we left that category as is. Ultimately, out of 359 possible variables, using knowledge of the domain, 75 variables were deemed to be clean, complete, and usable for modeling, i.e., having significant variability.
We conducted a standard logistic regression, starting with the initial set of 75 usable predictor variables, and the binary response variable Influenza Vaccination [10]. The response variable is nearly balanced, e.g., 46% Yes vs. 54% No in the 2013 data. The fitted model predicted True if the fitted probability exceeded 0.5 and False otherwise. Our methodology for variable selection was stepwise: Forward Selection (Conditional), maximizing adjusted R-Squared, until the final model was obtained. The final model of nine predictors maximized predictive power (adjusted R-Squared), contained only statistically significant predictors, and contained little collinearity (VIF < 5 for each predictor). The response variable and the nine predictors are found in Table 1.

3. Results

Our final model to predict Influenza Vaccination from the data consists of nine predictor variables: Age, Pneumococcal Vaccination, Time Since Last Checkup, Highest Education Level Attained, Employment, Health Care Coverage, Number of Personal Doctors, Smoking Status, and Annual Household Income. Table 2 shows the frequency statistics of all the categorical predictors in the 2012, 2013, and 2014 datasets.
In Figure 1, the resulting Confusion Matrix and graph of Area Under the receiver operating Characteristic (AUC) curve show the predictive accuracy of the models. Figure 1b shows that sixty-seven percent of the cases were classified correctly as receiving influenza vaccination (27%) or not (40%), based on the nine predictors. The red line shows a smooth tradeoff between the false positive rate and the true positive rate. The gray diagonal line represents chance classification. The AUC for 2013 is 72% [12]. The same nine-predictor model applied to the 2012 and 2014 datasets yielded similar confusion matrices and AUC Curves (Figure 1a,c) [5,6]. The overall trend in vaccination is shown by the three confusion matrices. From 2012 to 2014, people receive an influenza vaccination at approximately the same rate, 44% (2012) to 46% (2013) to 47% (2014), whereas the model correctly classifies at a level of 67–68% across the three years.
Overall, Table 3 shows the nine predictors, ranked by mean analysis of deviance, across the three years. This shows the relative importance of the nine predictors in a given year as well as relative importance from year to year.
We see stability among the predictors, with some variation in the rankings. Age, Pneumococcal Vaccination, and Time Since Last Checkup are the top three predictors, and they never change rank. The middle three predictors are Number of Personal Doctors, Employment, and Annual Household Income. The bottom three predictors are, Smoking Status, Health Care Coverage, and Highest Education Level Attained. Overall, the nine predictors constitute a stable model of influenza vaccination in the United States 2012–2014.
Table 4 shows the details of the logistic models for the three years. We consider each of the nine predictors and interpret the results. For each categorical variable, the referent category is the first one, which has a beta coefficient of zero in the table. We have omitted the Missing category. We did so because even as an aggregate category, the category’s proportion of the data for each variable was small, and the result would be less interesting to interpret.
Age: Age is the only numerical variable in the model. For every additional year of age, the odds ratio is 1.015, meaning the odds increase by 1.5% for having received an influenza vaccination.
Pneumococcal Vaccination: The referent category is Yes, received a pneumococcal vaccination. If the respondent instead receives no pneumococcal vaccination, the odds of an influenza vaccination decrease strongly with an odds ratio of 0.365, which means a decrease of 1 − 0.365 = 63.5% in the odds. If the respondent does not know/is not sure whether a pneumococcal vaccination was received, the odds ratio is 0.456 compared to the referent category.
Time Since Last Checkup: The referent category is having a checkup less than one year ago. The other categories show odds ratios (each compared to the referent case) steadily decreasing as the time since last checkup grows longer. If the last checkup was between 1 and 2 years ago, the odds decrease strongly compared to the referent case with an odds ratio of 0.716, which means a decrease of 1 − 0.716 = 28.4% in the odds. If the last checkup was between 2 and 5 years ago, the odds decrease further with an odds ratio of 0.579 compared to the referent case. If the last checkup was greater than 5 years ago, the odds decrease further (odds ratio = 0.452 compared to the referent case). The exception to this trend is the case of never having received a checkup, which is rare (< 1% of respondents). In this case, the odds ratio (0.637 compared to the referent case) is between that of a checkup 1–2 years ago and a checkup 2–5 years ago.
Number of Personal Doctors: The referent category is having exactly one personal doctor. If the respondent has more than one personal doctor, the difference is not statistically significant. If the respondent has no personal doctor, the odds of an influenza vaccination decrease strongly with an odds ratio of 0.657, which means a decrease of 34.3% over having exactly one personal doctor.
Annual Household Income: The categories are ordinal, and income of $10,000 or less was used as the referent category. If the respondent has an annual household income of $10,000–$15,000, the difference is not statistically significant. Starting at the next bracket (category of income >$15,000), the odds of an influenza vaccination gradually increase as the income increases: from an odds ratio of 1.054 for the income level of $15,000–$20,000, up to an odds ratio of 1.447 for the income level of greater than $75,000 (all odds ratios are compared to the referent case; see Table 4 for all categories).
Smoking Status: The referent category is smoking Every Day. If the respondent only smokes Some Days, the odds of an influenza vaccination increase with an odds ratio of 1.134, which means an increase of 13.4% over smoking every day. If the respondent is a former smoker (ex-smoker), the odds of an influenza vaccination increase more strongly with an odds ratio of 1.416 compared to the referent case, which is an additional 28.2% over smoking only some days. If the respondent has never smoked, the odds increase is not as much as the former smokers, with an odds ratio of 1.372 compared to the referent case.
Health Care Coverage: The referent category is Yes, having some kind of health care coverage, private or public. If the respondent answers No, the odds of an influenza vaccination decrease strongly with an odds ratio of 0.635, which means a decrease of 36.5% over having some kind of coverage.
Employment: The referent category is Employed for Wages. If the respondent is Self-employed, the odds of an influenza vaccination decrease strongly with an odds ratio of 0.629 compared to the referent case, which means a decrease of 1 − 0.629 = 37.1% in the odds. If the respondent is out of work for more than 1 year, the odds decrease with an odds ratio of 0.847 compared to the referent case, which means a decrease of 1 − 0.847 = 15.3% over being Employed for Wages. The cases of being out of work for less than 1 year and being a homemaker show odds decrease quite similar to being out of work for more than 1 year, with odds ratios at 0.865 and 0.870, respectively, compared to the referent case. The only category more likely to have an influenza vaccination is Student. If that is the category, the odds increase slightly with an odds ratio of 1.077, which means an increase of 7.7% over being Employed for Wages. If the respondent is Retired, the odds decrease slightly with an odd ratio of 0.972, which means a decrease of 2.8% over being Employed for Wages. The category of Unable to Work is not statistically different from being Employed for Wages.
Highest Level of Education Attained: The referent category is Did Not Graduate High School. If the respondent is a high school graduate, the odds of an influenza vaccination decrease slightly with an odds ratio of 0.964 compared to the referent case, with a marginally significant p-value between 0.01 and 0.05. If the respondent has attended but not graduated from college or technical school, the difference is not statistically significant from the referent case. If the respondent is a College or Technical School Graduate, the odds increase strongly with an odds ratio of 1.275 compared to the referent case.

4. Discussion

We revisit the nine predictors and interpret the results broadly.
Age: Our results show an overall increase in the odds of influenza vaccination as one ages. Elderly individuals have more medical problems, e.g., stroke, myocardial infarction, pneumonia, chronic respiratory tract infections; thus they see a doctor more frequently, thereby receiving messages that they are at greater risk of complications from influenza [13,14,15,16]. Note: regarding influenza between 2010 and 2013, 54–70% of hospitalizations and 71–85% of deaths occurred among adults aged ≥65 years, adding to the significance of doctor-patient communications [17]. The elderly are thus reminded, perhaps frequently, that an influenza vaccination is effective at reducing and preventing morbidity and mortality [18]. The age effect is stable from 2012 to 2014, inclusive.
Pneumococcal Vaccination: The Pneumococcal Vaccination effect is the second most statistically significant of the nine predictors and it correlates positively with influenza vaccination. This could indicate convenience, obtaining both vaccinations in the same medical/clinical session. It could also indicate the lack of needle-phobia [19]. Note: the CDC’s policy for adults is that it recommends pneumococcal vaccination for all adults 65 years or older, adults who smoke, and others with certain medical conditions. The Pneumococcal Vaccination effect is stable from 2012 to 2014, inclusive.
Time Since Last Checkup: Time since one’s last checkup is a strong and consistent predictor of influenza vaccination. In general, the more recent a checkup, the greater the odds of an influenza vaccination. This could be a result of the reminder frequency or a recency effect. The one exception is adults who have never had a checkup. An adult in that category is somewhat more likely to receive an influenza vaccination than someone who has gone more than five years without a checkup. The Time Since Last Checkup effect is stable from 2012 to 2014, inclusive.
Number of Personal Doctors: Those with no personal doctor are less likely to receive influenza vaccination than those with one or more personal doctors. One does not need a personal doctor to obtain an influenza vaccination, however [20]. Many clinics, local government offices, and pharmacies provide the vaccination at little to no cost. This may indicate that someone without a personal doctor may be disinclined to seek medical attention or may be misinformed about the benefits of vaccination [21]. The effect is stable from 2012 to 2014, inclusive.
Annual Household Income: The models show the trend that greater income corresponds to greater odds of influenza vaccination, particularly at the higher income levels. This can be interpreted simply, that greater annual household income may indicate greater educational attainment and thereby obtaining information from reliable sources [22]. The Annual Household Income effect is stable from 2012 to 2014, inclusive, in most of the categories.
Smoking Status: The models show that in general, the less one smokes, the more likely it is that an influenza vaccination is obtained. It is interesting to note the exception: that an ex-smoker is even more likely to obtain vaccination than someone who has never smoked. A possible explanation may be that someone who quits smoking has consciously achieved a difficult health goal, whereas someone who has never smoked has not achieved that difficult goal. The ex-smokers are thus taking greater deliberate care of themselves, whereas those who have never smoked may be complacent [4]. The Smoking Status effect is stable from 2012 to 2014, inclusive.
Health Care Coverage: Those with any form of Health Care Coverage, private or public, are much more likely to receive influenza vaccination. A straightforward explanation is that the cost of the vaccination would not be a concern for those having coverage. The Health Care Coverage effect is stable from 2012 to 2014, inclusive.
Employment: The results in this variable can be interpreted from both an infection and a communication perspective. Employed for Wages, the most common category, is one in which adults are usually exposed to many people at work. There are therefore two possible factors that increase the odds of vaccination in those adults: verbal reminders (to vaccinate) and viral sources (if they fail to do so). All the other categories expose an adult to less of these two sources, except perhaps for the Student category. Students are usually exposed to even more reminders and viruses, when on a university campus, than someone working for wages, but the Student category is not statistically significant in 2012 or 2014. Nevertheless, this pattern may suggest that availability of information and social interaction/contagion, are important considerations. The Employment effect is stable from 2012 to 2014, inclusive, in most of the categories.
Highest Level of Education Attained: The categories of High School Graduate and College or Technical School Attendee both have odds ratios close to 1 and p-values larger than 0.01. It is reasonable to consider these two categories combined with the referent case, so that this variable is simplified to two levels: College or Technical School Graduate vs. lower level of education attained. It can be seen that college or technical school graduates are significantly more likely to receive influenza vaccination. The reasons may be that they are more receptive to medical advice, more information literate [23], or less swayed by misinformation found on the Internet [21]. The Highest Level of Education Attained effect is stable for College or Technical School Graduate from 2012 to 2014, inclusive.

Implications and Future Research

We address some implications of our findings, as well as communication strategies based on cognitive bias, limitations of this research, and ideas for future vaccination research.
Our results contain several findings that are not surprising. One is more likely to receive an influenza vaccination if one is older, has health care coverage, or has more frequent checkups, etc. Some readily apparent implications are suggested. For example, health communications in physician offices and pharmacies, e.g., posters or pamphlets, need to be appropriate to the age of the people likely to encounter them. Also, they should be designed for a lower education level to reach those who are less likely to vaccinate [24]. Some implications are less readily apparent. For example, one could tailor communications differently to the various employment categories: those who are Employed for Wages, Students, Homemakers, Unemployed, Self-Employed, Retired, or Unable to Work. Adults in those categories are exposed to different physical and social environments. The overall idea is to immunize the overall population and thereby achieve herd immunity [25,26] by optimizing communications to different subpopulations of the herd. Future research could investigate the tailoring of messages to these categories, and more generally, to people claiming to be too busy to vaccinate [27] or who postpone it repeatedly [28].
This paper shows that graduates of college or technical school are more likely to receive influenza vaccination, but even they can fail to receive a vaccination. When uninformed or uncertain, people—whether highly educated or not—may rely on cognitive biases. Cognitive biases are mental rules-of-thumb, i.e., heuristics that help people to make adequate decisions with a low level of effort. They are not optimal, systematic, or complete in their information processing, but they are usually adequate. For example, there are the Availability Heuristic and the Bandwagon Effect [29].
Availability Heuristic. The CDC, under its 2016 message strategy, referred to influenza as a serious disease that may result in hospitalization and possibly death [3]. This is scientifically accurate, but many people are ignorant or skeptical about the true risk posed by influenza. This may be an example of the Availability Heuristic, the estimation of likelihood based on the retrievability of cases from an individual’s memory, which is influenced by social communications. The United States population is currently greater than 324 million, and the upper limit of influenza deaths 1976–2007 is 49,000 in one season; less than 0.001 percent of the population has died. People are therefore much more likely to not know a person who died from influenza than to know a person who has died from it.
Bandwagon Effect. For another example of cognitive bias, if a friend, family member, or coworker says that an influenza vaccination is not efficacious, the Bandwagon Effect may result. The Bandwagon Effect compounds the problem of someone being misinformed/skeptical, because such a person may be surrounded by likeminded people. This would be herd immunity against messages about the importance of influenza vaccination.
Rather than consider cognitive biases to be irrational shortcuts to systemic, logical thinking, or ignore them completely, they could be used as a basis for decision guidance toward normative decision-making. The research question would be how to identify the cognitive bias in the given situation, or given subpopulation, and how to nudge it toward the normative vaccination behavior. Future research could investigate different cognitive biases that may need to be anticipated in order to help public health communication campaigns, whether aimed broadly at the overall population or narrowly at different subpopulations. Which populations can be persuaded to reconsider and rethink their cognitive biases if sent well-framed messages?
Although the BRFSS datasets have been shown to be reliable and valid [30], it is important to keep several limitations in mind. One, this study, although examining 2012–2014 datasets, is multiple-snapshot, cross-sectional research. Cross-sectional designs do not allow for causal inferences. Two, data self-reported in response to a telephone survey could be underreported or biased in some variables, especially in ones with social stigma, e.g., smoking or low household income [3,31]. Finally, institutionalized individuals, including those living in hospitals, nursing homes, or prisons are excluded.
Despite these limitations, our large sample size, rigorous data sampling, and model development with multiple validation and test samples, gives us a stable, multiple-year model of those who participated. There will always be some bias in survey response data. We assumed that any such bias is of low magnitude, given the rigorous reputation of BRFSS [30], and not specific to any particular demographic or behavioral category. There also may be timing issues, such as people not having a checkup and therefore not receiving vaccination reminders, in the year after a mild influenza season. The stability of the model over three years, 2012–2014, helps to guard against such timing issues.
To make our model more predictive, new research questions could be investigated. What is the impact of a person’s health information source preference: doctor, online consumer health websites, or social networks [32,33]? How influential are the beliefs about the health of a person’s online or offline community? How influential is a person’s trust in information from the health care system, personal doctors, pharmaceutical companies, government, or social media? How significant a factor is a person’s anxiety about vaccination infection [34] or fear of hypodermic needles [19]?

5. Conclusions

An influenza-induced fever is common and usually lasts two to four days [3,34]. Millions of people fall ill to influenza every year in the United States and recover in a few days [15]. Almost everyone has experienced it and knows others who have suffered similarly [16]. This commonly experienced episode needs to be counterbalanced with scientifically correct and appropriately communicated messages about influenza vaccination risk and the importance of vaccination.
The Internet has enabled the distribution of medical information, empowering patients to treat themselves or at least ask their doctors better, more informed questions. The Internet has also been getting noisier. The empowered patient has increased access to opinion or myth (noise) in addition to scientific information from the medical establishment (signal) [35,36]. Access to a variety of “medical beliefs, scientific and nonscientific evidence, and emotionally arousing stories of other patients” is growing [21]. Patients are questioning the legitimacy and recommendations of scientific authorities, allowing unscientific views the same legitimacy and weight as scientific ones [7]. The movement to greater patient engagement and empowerment may be unintentionally creating greater distrust of traditional authorities [1].
This article offers a stable, multiple-year model for predicting which adults received influenza vaccination in the United States from 2012 to 2014, inclusive. We also present some practical implications of the results and how health communications could be targeted. Our model improves our understanding of influenza vaccination in the United States, and how we might intervene. Outside the United States, there may be other challenges, i.e., vaccine shortages causing the prioritization of at-risk individuals and health care workers over other groups. In such environments, the ecosystem of vaccine supply, as well as social and viral contagion, may differ.

Author Contributions

All three authors contributed equally to this paper. Arnold Kamis and Tamara Kamis conceived and designed the study. Arnold Kamis and Yuji Zhang analyzed the data. All three authors wrote and revised the paper.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Gangarosa, E.J.; Galazka, A.; Wolfe, C.; Phillips, L.; Miller, E.; Chen, R.; Gangarosa, R. Impact of anti-vaccine movements on pertussis control: The untold story. Lancet 1998, 351, 356–361. [Google Scholar] [CrossRef]
  2. Majumder, M.S.; Cohn, E.L.; Mekaru, S.R.; Huston, J.E.; Brownstein, J.S. Substandard vaccination compliance and the 2015 measles outbreak. JAMA Pediatr. 2015, 169, 494–495. [Google Scholar] [CrossRef] [PubMed]
  3. Influenza (Flu). Available online: https://www.cdc.gov/flu/index.htm (accessed on 26 July 2017).
  4. Schmid, P.; Rauber, D.; Betsch, C.; Lidolt, G.; Denker, M.-L. Barriers of influenza vaccination intention and behavior—A systematic review of influenza vaccine hesitancy, 2005–2016. PLoS ONE 2017, 12, e0170550. [Google Scholar] [CrossRef] [PubMed]
  5. Anderson, K.K.; Sebaldt, R.J.; Lohfeld, L.; Goeree, R.; Donald, F.C.; Burgess, K.; Kaczorowski, J. Practice and physician characteristics associated with influenza vaccination delivery rates following a patient reminder letter intervention. J. Prim. Prev. 2008, 29, 93–97. [Google Scholar] [CrossRef] [PubMed]
  6. Weinstein, N.D.; Kwitel, A.; McCaul, K.D.; Magnan, R.E.; Gerrard, M.; Gibbons, F.X. Risk perceptions: Assessment and relationship to influenza vaccination. Health Psychol. 2007, 26, 146–151. [Google Scholar] [CrossRef] [PubMed]
  7. Larson, H.J.; Cooper, L.Z.; Eskola, J.; Katz, S.L.; Ratzan, S. Addressing the vaccine confidence gap. Lancet 2011, 378, 526–535. [Google Scholar] [CrossRef]
  8. Centers for Disease Control and Prevention (CDC). Behavioral Risk Factor Surveillance System Survey Data, 2012–2014. Available online: http://www.cdc.gov/brfss/ (accessed on 26 July 2017).
  9. Ng, A.Y. Preventing “Overfitting” of Cross-Validation Data. Proccdings of the International Conference on Machine Learning, San Francisco, CA, USA, 8–12 July 1997. [Google Scholar]
  10. Fitzmaurice, G.M.; Laird, N.M. Binary response models and logistic regression. In International Encyclopedia of the Social & Behavioral Sciences; Elsevier: Amsterdam, The Netherlands, 2015; pp. 587–595. [Google Scholar]
  11. Behavioral Risk Factor Surveillance System Survey Questionnaire. Available online: http://www.cdc.gov/brfss/questionnaires/index.htm (accessed on 26 July 2017).
  12. Molinari, N.A.; Ortega-Sanchez, I.R.; Messonnier, M.L.; Thompson, W.W.; Wortley, P.M.; Weintraub, E.; Bridges, C.B. The annual impact of seasonal influenza in the US: Measuring disease burden and costs. Vaccine 2007, 25, 5086–5096. [Google Scholar] [CrossRef] [PubMed]
  13. Nichol, K.L.; Nordin, J.; Mullooly, J.; Lask, R.; Fillbrandt, K.; Iwane, M. Influenza vaccination and reduction in hospitalizations for cardiac disease and stroke among the elderly. N. Engl. J. Med. 2003, 348, 1322–1332. [Google Scholar] [CrossRef] [PubMed]
  14. Andrew, M.K.; McNeil, S.; Merry, H.; Rockwood, K. Rates of influenza vaccination in older adults and factors associated with vaccine use: A secondary analysis of the Canadian Study of Health and Aging. BMC Public Health 2004, 4, 36–44. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  15. Smeeth, L.; Thomas, S.L.; Hall, A.J.; Hubbard, R.; Farrington, P.; Vallance, P. Risk of myocardial infarction and stroke after acute infection or vaccination. N. Engl. J. Med. 2004, 351, 2611–2618. [Google Scholar] [CrossRef] [PubMed]
  16. Nagata, J.M.; Hernández-Ramos, I.; Kurup, A.S.; Albrecht, D.; Vivas-Torrealba, C.; Franco-Paredes, C. Social determinants of health and seasonal influenza vaccination in adults >65 years: A systematic review of qualitative and quantitative data. Public Health 2013, 13, 1–25. [Google Scholar] [CrossRef] [PubMed]
  17. Grohskopf, L.A.; Sokolow, L.Z.; Broder, K.R.; Olsen, S.J.; Karron, R.A.; Jernigan, D.B.; Bresee, J.S. Prevention and Control of Seasonal Influenza with Vaccines Recommendations of the Advisory Committee on Immunization Practices—United States, 2016-17 Influenza Season; Recommendations and Reports 2016. Available online: https://www.cdc.gov/mmwr/volumes/65/rr/rr6505a1.htm (accessed on 3 July 2017).
  18. Jefferson, T.; Rudin, M.; di Pietrantonj, C.; Rivetti, D.; Rivetti, A.; Demicheli, V. Efficacy and effectiveness of influenza vaccines in elderly people: A systematic review. Lancet 2005, 336, 1165–1174. [Google Scholar] [CrossRef]
  19. Hamilton, J.G. Needle phobia: A neglected diagnosis. J. Fam. Pract. 1995, 41, 169–175. [Google Scholar] [CrossRef] [PubMed]
  20. Key Facts About Seasonal Flu Vaccine. 2016. Available online: http://www.cdc.gov/flu/protect/keyfacts.htm (accessed on 26 July 2017).
  21. Kata, A. Anti-vaccine activists, Web 2.0, and the postmodern paradigm—An overview of tactics and tropes used online by the anti-vaccination movement. Vaccine 2012, 30, 3778–3789. [Google Scholar] [CrossRef] [PubMed]
  22. Venkatesh, V.; Morris, M.G. Why don’t men ever stop to ask for directions? Gender, social influence, and their role in technology acceptance and usage behavior. MIS Quart. 2000, 24, 115–139. [Google Scholar] [CrossRef]
  23. Paris, L.A.H. Stopping Behavior: User Persistence in Online Searching and Its Relation to Optimal Cutoff Points for Information Retrieval Systems. Ph.D. Thesis, The University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, 1998. [Google Scholar]
  24. Davis, T.C.; Crouch, M.A.; Wills, G.; Miller, S.; Abdehou, D.M. The gap between patient reading comprehension and the readability of patient education materials. J. Fam. Pract. 1990, 31, 533. [Google Scholar] [PubMed]
  25. Brisson, M.; Edmunds, W.J. Economic evaluation of vaccination programs: The impact of herd-immunity. Med. Decis. Mak. 2003, 23, 76–82. [Google Scholar] [CrossRef] [PubMed]
  26. Anderson, R.M.; May, R.M. Vaccination and herd immunity to infectious diseases. Nature 1985, 318, 323–329. [Google Scholar] [CrossRef] [PubMed]
  27. Bridges, C.B.; Thompson, W.W.; Meltzer, M.I.; Reeve, G.R.; Talamonti, W.J.; Cox, N.J.; Lilac, H.A.; Hall, H.; Klimov, A.; Fukuda, K. Effectiveness and cost-benefit of influenza vaccination of healthy working adults: A randomized controlled trial. JAMA 2000, 284, 1655–1663. [Google Scholar] [CrossRef] [PubMed]
  28. Salmon, D.A.; Dudley, M.Z.; Glanz, J.M.; Omer, S.B. Vaccine hesitancy: Causes, consequences, and a call to action. Vaccine 2015, 33, D66–D71. [Google Scholar] [CrossRef] [PubMed]
  29. Kahneman, D.; Tversky, A. Prospect theory: An analysis of decision under risk. Econometrica 1979, 47, 263291. [Google Scholar] [CrossRef]
  30. Pierannunzi, C.; Hu, S.S.; Balluz, L. A systematic review of publications assessing reliability and validity of the Behavioral Risk Factor Surveillance System (BRFSS), 2004–2011. BMC Med. Res. Methodol. 2013, 13. [Google Scholar] [CrossRef] [PubMed]
  31. MacDonald, R.; Baken, L.; Nelson, A.; Nichol, K.L. Validation of self-report of influenza and pneumococcal vaccination status in elder outpatients. Am. J. Prev. Med. 1999, 16, 173–177. [Google Scholar] [CrossRef]
  32. Kana'iaupuni, S.M.; Donato, K.M.; Thompson-Colón, T.; Stainback, M. Counting on kin: Social networks, social support, and child health status. Soc. Forces 2005, 83, 1137–1164. [Google Scholar] [CrossRef]
  33. Smith, K.P.; Christakis, N.A. Social networks and health. Annu. Rev. Sociol. 2008, 34, 405–429. [Google Scholar] [CrossRef]
  34. Cheung, K.W.; Mak, Y.W. Association between psychological flexibility and health beliefs in the uptake of influenza vaccination among people with chronic respiratory diseases in Hong Kong. Int. J. Environ. Res. Public Health 2016, 23, 155. [Google Scholar] [CrossRef] [PubMed]
  35. Nyhan, B.; Reifler, J. Does correcting myths about the flu vaccine work? An experimental evaluation of the effects of corrective information. Vaccine 2015, 33, 459–464. [Google Scholar] [CrossRef] [PubMed]
  36. Silver, N. The Signal and the Noise: Why So Many Predictions Fail—But Some Don't; Penguin Books: London, UK, 2015. [Google Scholar]
Figure 1. Confusion matrix and area under the receiver operating characteristic curve.
Figure 1. Confusion matrix and area under the receiver operating characteristic curve.
Ijerph 14 00849 g001aIjerph 14 00849 g001b
Table 1. Predictor variables.
Table 1. Predictor variables.
VariableDefinition/Question Asked [11]
Influenza Vaccination (response)During the past 12 months, have you had either a flu shot or a flu vaccine?
Age[What is your] age in years?
Pneumococcal VaccinationA pneumonia shot (or pneumococcal vaccine) is usually given only once or twice in a person’s lifetime and is different from the flu shot. Have you ever had a pneumonia shot?
Time Since Last CheckupAbout how long has it been since you last visited a doctor for a routine checkup? [A routine checkup is a general physical exam, not an exam for a specific injury, illness, or condition.]
Highest Level of Education Attained[What is your highest] level of education completed?
EmploymentAre you currently Employed for wages, Self-employed…?
Health Care CoverageDo you have any kind of health care coverage, including health insurance, prepaid plans such as Health Maintenance Organizations, or government plans such as Medicare, or Indian Health Service?
Number of Personal DoctorsDo you have one person you think of as your personal doctor or health care provider? (If “No” ask “Is there more than one or is there no person who you think of as your personal doctor or health care provider?”)
Smoking StatusFour-level smoker status: Everyday smoker, Someday smoker, Former smoker, Non-smoker
Annual Household Income[What] is your annual household income from all sources…: (If respondent refuses at any income level, code “Refused”.)
Table 2. Frequency statistics.
Table 2. Frequency statistics.
201220132014
VariableCategoryFreq.Perc.Freq.Perc.Freq.Perc.
Pneumococcal VaccinationYes159,55734.40161,78435.61163,28236.97
No261,72856.43245,51254.03239,08954.13
Don’t Know/Not Sure40,0188.6344,3189.7537,1658.41
Refused25430.5527700.6121840.49
Time Since Last CheckupWithin past year341,89771.87356,25372.44340,58773.30
Within past 2 years56,16511.8157,33811.6653,66811.55
Within past 5 years33,6037.0633,6596.8430,7646.62
5 or more years ago33,2676.9933,7176.8629,2936.30
Don’t know/Not sure56771.1957431.1756011.21
Never46550.9845230.9240470.87
Refused4220.095400.117020.15
Highest Level of Education AttainedDid not graduate High School42,3518.9042,1328.5737,0037.96
Graduated High school139,50129.33142,95329.07131,32528.26
Attended College/Technical School128,40426.99134,24227.30125,63527.04
Graduated College/Technical School163,51034.37170,17334.60166,97235.93
Don’t know/Not sure/Missing19210.4022730.463,7290.80
EmploymentEmployed for wages196,28941.31202,16941.17189,21940.93
Self-employed38,8638.1839,8538.1139,1028.46
Out of work for more than 1 year14,4093.0314,0812.8710,7592.33
Out of work for less than 1 year12,3122.5912,2412.4997552.11
A homemaker31,5876.6531,6546.4529,0286.28
A student12,2252.5712,6642.5811,0212.38
Retired132,13827.81138,31128.16136,63929.56
Unable to work35,2607.4237,4197.6234,0217.36
Refused20530.4327130.552,7030.58
Health Care CoverageYes419,32888.15434,62788.38425,19891.51
No54,80111.5255,24211.2337,6428.10
Don‘t Know/Not Sure9320.2010230.219340.20
Refused6240.138810.188900.19
Number of Personal Doctor(s)1 Personal Doctor365,48376.83369,08475.05354,62376.32
2+ Personal Doctors38,8548.1741,3068.4037,1998.01
No Personal Doctor69,85514.6979,58716.1870,89115.26
Don’t know/Not Sure10880.2311760.2412710.27
Refused4060.096200.136790.15
Smoking StatusEvery Day54,94011.5555,15711.2247,12210.14
Some Days21,1604.4521,4554.3619,2424.14
Former Smoker135,42628.47138,21828.11128,62927.68
Never254,49253.50261,62153.20248,50053.48
Don’t know/Refused/Missing96692.0315,3223.1221,1714.56
Annual Household Income$10,00025,2375.3125,4115.1821,1994.60
$10,000–$15,00026,4065.5626,7595.4522,9434.98
$15,000–$20,00034,0817.1834,8857.1130,5116.63
$20,000–$25,00040,3848.5041,7388.5037,5318.15
$25,000–$35,00047,3529.9748,8639.9544,3159.62
$35,000–$50,00059,87712.6161,50612.5357,41812.47
$50,000–$75,00063,95113.4765,23813.2962,17513.50
$75,000111,65423.51115,98223.63117,17625.45
Don’t know/Not sure33,0346.9634,9467.1231,6226.87
Refused32,9616.9435,5597.2435,5537.72
Influenza VaccinationYes202,64043.61208,38745.79205,32846.41
No258,08855.55242,93953.38233,76952.84
Don’t Know/Not Sure13990.309590.2111580.26
Refused25190.5427910.6121790.49
Table 3. Analysis of deviance.
Table 3. Analysis of deviance.
2012 Logistic Regression Deviance2012 Logistic Regression Deviance Rank2013 Logistic Regression Deviance2013 Logistic Regression Deviance Rank2014 Logistic Regression Deviance2014 Logistic Regression Deviance Rank2012–2014 Logistic Regression Mean Deviance Rank
Age17,739.3116,378.7115,230.711.0
Pneumococcal Vaccination13,599.2210,199.2210,833.822.0
Time Since Last Checkup8280.737738.637577.933.0
Number of Personal Doctors1659.552014.341910.744.3
Employment2244.641577.361749.855.0
Annual Household Income1543.962013.351681.465.7
Smoking Status680.78785.37772.387.7
Health Care Coverage837.17709.8870198.0
Highest Education Level Attained616.59707.69773.278.3
Table 4. Logistic Regressions for 2012–2014.
Table 4. Logistic Regressions for 2012–2014.
2012 Logistic Regression2013 Logistic Regression2014 Logistic Regression
Chi−Square p−Value:0.0000.0000.000
Pseudo R−Square (Optimistic):0.4010.3890.383
EstimateStd. Error Odds RatioOdds Ratio 95%Z ValuePr(>|z|) EstimateStd. ErrorOdds RatioOdds Ratio 95%Z ValuePr(>|z|) EstimateStd. ErrorOdds RatioOdds Ratio 95%Z ValuePr(>|z|)
(Intercept)−0.5620.0340.570(0.533, 0.610)−16.432<2 × 10−16***−0.6130.0360.542(0.505, 0.581)−17.269<2 × 10−16***−0.4780.0360.620(0.578, 0.666)−13.229<2 × 10−16***
Ageyears0.0140.0001.014(1.013, 1.014)38.508<2 × 10−16***0.0150.0001.015(1.014, 1.016)39.791<2 × 10−16***0.0130.0001.013(1.013, 1.014)35.685<2 × 10−16***
Pneumococcal Vaccinationyes (ref. cat.)0.000 0.000 0.000
no−1.1200.0100.326(0.320, 0.333)−113.618<2 × 10−16***−1.0080.0100.365(0.358, 0.372)−101.101<2 × 10−16***−1.0250.0100.359(0.352, 0.366)−103.598<2 × 10−16***
Don’t Know/Not Sure−0.7390.0170.478(0.463, 0.493)−44.737<2 × 10−16***−0.7850.0160.456(0.442, 0.471)−48.287<2 × 10−16***−0.6110.0170.543(0.525, 0.561)−36.362<2 × 10−16***
Time Since Last Checkup<1 year ago (ref. cat.)0.000 0.000 0.000
1–2 years ago−0.3740.0140.688(0.670, 0.706)−27.702<2 × 10−16***−0.3340.0140.716(0.697, 0.735)−24.365<2 × 10−16***−0.3470.0140.707(0.688, 0.726)−25.427<2 × 10−16***
2–5 years ago−0.5730.0180.564(0.544, 0.585)−30.987<2 × 10−16***−0.5470.0190.579(0.558, 0.600)−29.125<2 × 10−16***−0.5800.0190.560(0.539, 0.581)−30.627<2 × 10−16***
5+ years ago −0.7610.0210.467(0.449, 0.487)−36.933<2 × 10−16***−0.7940.0210.452(0.434, 0.471)−37.771<2 × 10−16***−0.7650.0210.465(0.446, 0.485)−35.590<2 × 10−16***
Never −0.6260.0500.535(0.485, 0.589)−12.622<2 × 10−16***−0.4510.0500.637(0.578, 0.702)−9.064<2 × 10−16***−0.6260.0540.535(0.481, 0.594)−11.701<2 × 10−16***
Highest Education Level Attained<High School Grad. (ref. cat.)0.000 0.000 0.000
High School Graduate0.0280.0171.029(0.995, 1.064)1.6680.0953.−0.0370.0180.964(0.931, 0.998)−2.0800.0375*−0.0610.0180.941(0.908, 0.974)−3.4090.0007***
College/Tech. Sch. Attendee0.0930.0171.098(1.061, 1.136)5.342<0.0001***0.0290.0181.030(0.994, 1.067)1.6130.1068n.s0.0030.0181.003(0.968, 1.040)0.1730.8625n.s.
College/Tech. Sch. Graduate0.2760.0181.317(1.272, 1.365)15.371<2 × 10−16***0.2430.0181.275(1.229, 1.322)13.121<2 × 10−16***0.2270.0191.255(1.210, 1.302)12.106<2 × 10−16***
EmploymentEmployed for wages (ref. cat.)0.000 0.000 0.000
Self−employed−0.5390.0170.583(0.564, 0.603)−31.471<2 × 10−16***−0.4630.0170.629(0.608, 0.651)−26.805<2 × 10−16***−0.4980.0170.608(0.588, 0.628)−29.488<2 × 10−16***
Out of work for >1 year−0.2650.0280.768(0.727, 0.811)−9.452<2 × 10−16***−0.1660.0290.847(0.801, 0.896)−5.774<0.0001***−0.2210.0310.802(0.755, 0.852)−7.120<0.0001***
Out of work for <1 year−0.2560.0300.774(0.730, 0.822)−8.472<2 × 10−16***−0.1450.0310.865(0.815, 0.919)−4.728<0.0001***−0.1400.0320.870(0.816, 0.926)−4.343<0.0001***
A homemaker−0.1640.0180.849(0.819, 0.880)−9.007<2 × 10−16***−0.1390.0190.870(0.839, 0.903)−7.443<0.0001***−0.1530.0190.859(0.828, 0.891)−8.152<0.0001***
A student−0.0420.0300.958(0.904, 1.016)−1.4290.1531n.s.0.0740.0301.077(1.016, 1.142)2.4810.0131*0.0300.0301.030(0.971, 1.094)0.9850.3247n.s.
Retired−0.0440.0140.957(0.932, 0.983)−3.2530.0011**−0.0280.0140.972(0.946, 0.999)−2.0610.0393*−0.0240.0140.976(0.950, 1.002)−1.7910.0733.
Unable to work−0.0070.0180.993(0.957, 1.029)−0.3920.6947n.s.−0.0130.0190.987(0.951, 1.024)−0.7170.4731n.s0.0050.0191.005(0.969, 1.043)0.2620.7936n.s.
Health Care Coverageyes (ref. cat.)0.000 0.000 0.000
no−0.4810.0170.618(0.598, 0.639)−28.287<2 × 10−16***−0.4540.0170.635(0.614, 0.657)−26.069<2 × 10−16***−0.5150.0200.598(0.575, 0.621)−25.739<2 × 10−16***
Personal Doctor(s)1 Personal Doctor (ref. cat.)0.000 0.000 0.000
2+ Personal Doctors0.0430.0151.044(1.013, 1.075)2.8380.0045**0.0040.0151.004(0.974, 1.034)0.2540.7996n.s−0.0130.0150.987(0.958, 1.017)−0.8400.4007n.s.
No Personal Doctor−0.4340.0150.648(0.629, 0.667)−29.049<2 × 10−16***−0.4210.0140.657(0.639, 0.675)−29.366<2 × 10−16***−0.4310.0140.650(0.632, 0.668)−29.831<2 × 10−16***
Smoking StatusEvery Day (ref. cat.)0.000 0.000 0.000
Some Days0.0930.0241.097(1.046, 1.151)3.8040.0001***0.1260.0251.134(1.080, 1.191)5.057<0.0001***0.0680.0251.070(1.018, 1.124)2.6830.0073**
Former Smoker0.3130.0161.367(1.326, 1.410)20.136<2 × 10−16***0.3480.0161.416(1.372, 1.461)21.796<2 × 10−16***0.3190.0161.376(1.333, 1.421)19.711<2 × 10−16***
Never0.2820.0151.325(1.288, 1.364)19.269<2 × 10−16***0.3160.0151.372(1.332, 1.413)21.103<2 × 10−16***0.3160.0151.372(1.331, 1.413)20.684<2 × 10−16***
Annual Household Income$10,000 (ref. cat.)0.000 0.000 0.000
$10,000–$15,0000.0700.0261.073(1.019, 1.130)2.6590.0078**0.0030.0271.003(0.951, 1.058)0.1200.9047n.s0.0070.0281.007(0.953, 1.065)0.2530.8005n.s.
$15,000–$20,0000.0840.0251.088(1.035, 1.143)3.3250.0009***0.0530.0261.054(1.002, 1.109)2.0250.0429*0.0490.0271.050(0.996, 1.107)1.8220.0684.
$20,000–$25,0000.0890.0251.093(1.041, 1.147)3.5960.0003***0.0830.0251.087(1.034, 1.142)3.3010.0010***0.0430.0261.044(0.992, 1.099)1.6430.1004n.s.
$25,000–$35,0000.1540.0241.167(1.113, 1.224)6.355<0.0001***0.0970.0251.102(1.050, 1.157)3.9150.0001***0.0760.0261.079(1.026, 1.135)2.9660.0030**
$35,000–$50,0000.1300.0241.139(1.087, 1.194)5.489<0.0001***0.1410.0241.152(1.098, 1.208)5.819<0.0001***0.1140.0251.120(1.067, 1.177)4.531<0.0001***
$50,000–$75,0000.1900.0241.210(1.154, 1.268)7.969<0.0001***0.1970.0241.218(1.161, 1.278)8.093<0.0001***0.1570.0251.170(1.114, 1.229)6.254<0.0001***
$75,0000.3370.0231.401(1.339, 1.466)14.527<2 × 10−16***0.3690.0241.447(1.381, 1.515)15.592<2 × 10−16***0.3120.0241.366(1.302, 1.432)12.811<2 × 10−16***
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1.

Share and Cite

MDPI and ACS Style

Kamis, A.; Zhang, Y.; Kamis, T. A Multiyear Model of Influenza Vaccination in the United States. Int. J. Environ. Res. Public Health 2017, 14, 849. https://doi.org/10.3390/ijerph14080849

AMA Style

Kamis A, Zhang Y, Kamis T. A Multiyear Model of Influenza Vaccination in the United States. International Journal of Environmental Research and Public Health. 2017; 14(8):849. https://doi.org/10.3390/ijerph14080849

Chicago/Turabian Style

Kamis, Arnold, Yuji Zhang, and Tamara Kamis. 2017. "A Multiyear Model of Influenza Vaccination in the United States" International Journal of Environmental Research and Public Health 14, no. 8: 849. https://doi.org/10.3390/ijerph14080849

APA Style

Kamis, A., Zhang, Y., & Kamis, T. (2017). A Multiyear Model of Influenza Vaccination in the United States. International Journal of Environmental Research and Public Health, 14(8), 849. https://doi.org/10.3390/ijerph14080849

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop