*4.4. Questionnaire: Vaccine Hesitancy towards COVID-19 Vaccinations*

Our study is the only one to date to incorporate a questionnaire alongside the exploration of sentiment analysis on Twitter towards COVID-19 vaccinations. Most respondents (90.1%) had or would accept a COVID-19 vaccine, a view that is in line with conclusions drawn by other studies [87,88] whilst others have reported less public support for COVID-19 vaccinations [89].

The identification of factors that might predict hesitancy towards COVID-19 vaccines was investigated. A positive correlation between intensity of concern regarding vaccines and their uptake was established, suggesting that participants with higher levels of (or more intense) concern are less likely to accept the vaccine, whereas those with low levels (less intense) or no concern are more likely to accept the COVID-19 vaccine.

Additional predictors of vaccine hesitancy were explored by considering whether age, vaccine history, level of vaccine understanding and usage of social media were likely to influence an individual's decision to take a COVID-19 vaccination. No association was established between vaccine refusal and age, despite the Pew Research Group (2017) finding younger adults (<30 years) were less likely to consider beneficial aspects of the MMR vaccine outweighed the risks, compared to older age groups [90]. The same study found individuals with higher levels of understanding considered the risk of vaccine side effects as low, whereas there was no association found between vaccination understanding and vaccination uptake in our study. Survey research on COVID-19 vaccine hesitancy corroborated our results by also finding no association between age and vaccine refusal [91] although Bendau et al. (2021) did establish an association between vaccine hesitance and concern [92]. Interestingly, 17.2% of respondents in the present study somewhat or strongly agreed that "vaccine safety and effectiveness data are often false", suggesting a significant proportion of the general public have concerns trusting this information as evidenced previously [9]. Anecdotal evidence from the questionnaire suggests that participants are more likely to write negative comments. This view is supported by the literature where it is understood that negative emotions (such as anger, frustration, sadness and disappointment) motivate individuals to articulate their views [93,94].

Reports suggest that the acceptance of vaccines in emergency situations (such as a pandemic) differs to that of routinely administered vaccines in non-crisis situations [87]. However, contrastingly, public concerns surrounding safety are higher with the uncertainties that come with novel vaccines and new emerging infectious diseases [87,95–97]. For example, in the UK, France, Greece, America and Australia, only 17% to 67% of the general

public was willing to accept the vaccine for the H1N1 pandemic in 2009 [95–102], highlighting public concern in this area and also likely variable uptake figures. Chaudhri et al. (2021) established the public had a weakly positive sentiment towards receiving a COVID-19 vaccine [73]. Vaccination history has previously been identified as a major predictor of vaccine uptake [95,98,101,103], a view also identified in the present study which established an association between vaccine history and acceptance. Individuals with full previous vaccination history were more likely to accept a COVID-19 vaccine, further confirming the idea of the echo chamber effect.

The present study has confirmed the idea that vaccine compliance remains inconsistent with negative opinions and hesitancy still widespread [91,92] and the inclusion of a questionnaire provided a greater picture of overall sentiment towards vaccines. The questionnaire revealed generally positive sentiment, whereas more negative sentiment was found online, alongside positive and neutral views. The questionnaire revealed that concerns about vaccines typically centred around trust in safety and effectiveness.

#### *4.5. Limitations and Further Work*

As part of the pilot work for the present study, we manually categorised the sources (Twitter accounts) as 'personal', 'accredited medical', 'news' or 'government/public health'. It would have been helpful if we could have extended this into the main study to facilitate a better understanding of the most common sources of misinformation. However, with the large dataset in the main study, this was unrealistic, and we seek an automated approach to this for future studies.

The data were collected over a short period in July 2021 and so it would be interesting to extend this study to look at historical and future tweets to further understand whether public opinion regarding COVID-19 vaccinations changed during the course of the pandemic. It would also be interesting to compare the dates of specific events in the media with daily sentiment analysis to determine whether they are closely related.

The questionnaire was distributed via social media and so responses were limited to people with access and were typically in the authors' extended networks. Future studies should endeavour to distribute the questionnaire more widely and in particular to reach public without access to social media. Concern exists in the UK that certain groups are more susceptible to vaccine misinformation and we would like to reach those communities with future research. This is also the case with the sentiment analysis which only collected tweets in English and therefore had the potential to miss the view of non-English speaking groups in the UK.

A simplified interface would benefit this research as the low accuracy of Microsoft Azure and the complexity of using data mining and analysis tools such as Python requires specific computing expertise. Thus, a simplified graphical interface is in development that would benefit future projects seeking to collect datasets for analysis without a need for an understanding of Python or the VADER algorithm.

Sentiment analysis is a popular and rapidly developing area. An interesting avenue for further research would be to compare our approach using VADER to other languageencoder-based approaches (such as using Bert or GPT), in particular exploring whether these could be useful developments that would work with NLTK.

#### **5. Conclusions**

This study established that machine learning and lexicon-based sentiment analysis methods yielded different frequencies of sentiment results. Negative sentiment was found to be most frequent online, with a higher intensity of negativity within the neutral tweets. There was no significant change in sentiment towards COVID-19 across the three-week data collection period. Positive correlations were established between COVID-19 vaccine acceptance with full vaccination history and low levels of concern.

Sentiment analysis provides evidence to assess public perception about various topics [104], allowing officials in charge of managing the impact of COVID-19 and health

policy makers insight into how the public feel about vaccination safety and efficacy so they can identify areas and misconceptions that need to be addressed [93,94].

The identification of frequently occurring negative terms and of predictors that influence vaccine hesitancy can be utilised to deploy effective strategies such as educational campaigns to increase public confidence in the COVID-19 vaccines and improve vaccine uptake. To ensure vaccination uptake targets are met, this requires continued attention.

**Author Contributions:** Conceptualisation, C.M., M.L. and C.R.; methodology, C.M., M.L., C.R. and B.W.; analysis, C.M., C.R. and B.W.; writing—original draft preparation, M.L. and C.R.; writing—review and editing, C.M., C.R. and B.W.; supervision, C.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of the School of Life Sciences, University of Lincoln. Reference: BGY9013M15568465, 17 June 2021.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data available on request due to ethical restrictions. The data presented in this study are available on request from the corresponding author. The data are not publicly available due to conditions of ethical approval.

**Acknowledgments:** The authors would like to thank Jonathan Roe and Jamie Smith.

**Conflicts of Interest:** The authors declare no conflict of interest.
