**HPV-Vaccines**

Editor

**Gloria Calagna**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editor* Gloria Calagna University of Palermo Italy

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Topical Collection published online in the open access journal *Vaccines* (ISSN 2076-393X) (available at: https://www.mdpi.com/journal/vaccines/topical collections/HPV-Vaccines vaccines).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-5917-9 (Hbk) ISBN 978-3-0365-5918-6 (PDF)**

© 2023 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**


#### **Zixin Wang, Yuan Fang, Paul Shing-fong Chan, Andrew Chidgey, Francois Fong, Mary Ip and Joseph T. F. Lau**


## *Case Report* **Juvenile Membranous Nephropathy Developed after Human Papillomavirus (HPV) Vaccination**

**Haruna Arakawa 1, Shohei Yokoyama 1, Takehiro Ohira 1, Dedong Kang 2, Kazuho Honda 2, Yoshihiko Ueda <sup>3</sup> and Akihiro Tojo 1,\***


**\*** Correspondence: akitojo@dokkyomed.ac.jp; Tel.: +81-282-86-1111

**Abstract:** A 16-year-old girl with no history of renal disease had a fever of 38 ◦C after her second HPV vaccination and was identified as positive for proteinuria. As she maintained urinary protein of 3.10 g/gCr and 5–9 urinary red blood cells/HPF, a renal biopsy was performed and small spikes on PAM staining with the granular deposition of IgG1++ and IgG3+ on the glomerular capillary wall were discovered by immunofluorescence, although PLA2R immunostaining was negative. Analysis by electron microscope showed electron density deposition in the form of fine particles under the epithelium. The diagnosis was secondary membranous nephropathy stage II. Immunostaining with the anti-p16 INK4a antibody was positive for glomerular cells, and Western blot analysis of urinary protein showed a positive band for p16 INK4a. However, laser-microdissection mass spectrometry analysis of a paraffin section of glomeruli failed to detect HPV proteins. It is possible that the patient was already infected with HPV and administration of the HPV vaccine may have caused secondary membranous nephropathy.

**Keywords:** human papillomavirus; vaccination; membranous nephropathy; p16 INK4a; mass spectrometry

#### **1. Introduction**

SARS-CoV-2 vaccination was promoted during the COVID-19 pandemic; however, the effectiveness and side effects of the mRNA vaccines for SARS-CoV-2 were not fully elucidated. Gross hematuria in patients with IgA nephropathy and recurrence of nephrotic syndrome in patients with minimal change nephrotic syndrome (MCNS) were reported as side effects following SARS-CoV-2 vaccination [1–4]. Some young people are concerned about adverse reactions to the SARS-CoV-2 vaccination, especially in Japan, where young people have a distrust of the vaccine due to cases of side effects caused by the HPV vaccine [5].

Human papillomavirus HPV is a double-stranded DNA virus that can be divided into more than 100 types from the gene sequence of the surface capsid protein L1. High-risk HPV16, 18, 31, 33, 45, 51, 52, 56, 58, and 66 cause cervical cancer, and low-risk HPV6, 11, etc., cause condyloma acuminata [6–8]. In Japan, the recombinant divalent human papillomavirus vaccine Cervarix for high-risk HPV 16/18 was sold in 2009, and regular vaccination started in 2013 for 6th-grade elementary school to 1st-grade middle high school girls [8]. However, Cervarix was discontinued due to reports of adverse reactions, including chronic pain, motor impairment, and other symptoms after HPV vaccination. The recombinant precipitated 4-valent HPV-like particle vaccine has been on the market since 2011, and the recombinant precipitated 9-valent HPV-like particle vaccine has been on the market since 2021, although the Ministry of Health, Labor, and Welfare in Japan announced a suspension of the proactive recommendation for routine use of the HPV vaccine in the

**Citation:** Arakawa, H.; Yokoyama, S.; Ohira, T.; Kang, D.; Honda, K.; Ueda, Y.; Tojo, A. Juvenile Membranous Nephropathy Developed after Human Papillomavirus (HPV) Vaccination. *Vaccines* **2022**, *10*, 1442. https://doi.org/10.3390/ vaccines10091442

Academic Editor: Gloria Calagna

Received: 2 August 2022 Accepted: 30 August 2022 Published: 1 September 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

national immunization program in June 2013 [9] until November 2021. In Cervarix, the capsid protein of the recombinant precipitated divalent human papillomavirus is atomized and aluminum hydroxide and monophosphoryl lipid A (MPL) derived from the cell membrane of Salmonella are used as an adjuvant. These adjuvants destroy cells at the administration site, and the destroyed autologous cell DNA and proteins are recognized by the DAMPs in the innate immune response system [10] and cause not only fever and local swelling but also Guillain–Barré syndrome (GBS), systemic lupus erythematosus (SLE), autoimmune hepatitis, and cerebral vasculitis [11]. We experienced a case of secondary membranous nephropathy after the second injection of the divalent HPV vaccine Cervarix, and here we discuss the mechanism of membranous nephropathy after HPV vaccination.

#### **2. Case Presentation**

A 16-year-old girl had a fever of 39.0 ◦C for 4–5 days after the second HPV vaccination, even though she had no fever following the first HPV vaccination one month prior. She was prescribed acetaminophen, and she developed proteinuria for the first time 5 days after receiving the second vaccination. Her fever was resolved spontaneously, and proteinuria was not checked until the next school year in a urine checkup, which showed proteinuria 3+ and occult blood. She was hospitalized for examination by renal biopsy. She had a history of allergy to pollen. Her family history included cerebral hemorrhage in her maternal great-grandfather, multiple system atrophy in her grandfather, and hypertension in her grandmother. Physical findings on admission were blood pressure 120/82 mmHg, heart rate 115 bpm, body temperature 36.2 ◦C, SpO2 98% (room air) without tonsillitis, lymphadenopathy, or neurological dysfunction. A urinalysis showed urinary protein, 1.46 g/gCr, 5–9 RBC/HPF, 1–4 WBC/HPF, cast (-), NAG 2.7 U/L, and selectivity index 0.121. A blood test showed WBC 8100 × 106/L, (neutrophil 68.2%, eosinophil 1.7%, basophil 0.7%, monocyte 8.3%, lymphocyte 21.1%), Hb11.9 g/dL, and platelet 39.8 × 1010/L. Biochemistry data revealed TP 5.5 g/dL, Alb 3.5 g/dL, UN 11.0 mg/dL, Cr 0.47 mg/dL, eGFR 147.8 mL/min/1.73 m2, and CRP 0.01 mg/dL. Immunological tests found ANA (-), anti-dsDNA antibody (-), anti-SM antibody (-), IgG 544 mg/dL, IgA 112.4 mg/dL, IgM 134.7 mg/dL, IgE 34.3 mg/dL, C3 141.9 mg/dL, C4 23.2 mg/dL, ASO 14 IU/mL, lupus anticoagulant negative, anti-CLβ2GP1 antibody 1.2 U/mL, anti-SS A/B antibody (−/−), MPO-ANCA (-), PR3-ANCA (-), anti-GBM antibody (-), HBs antigen (-), HCV antibody (-), and D dimer 0.3μg/mL. These data indicate that there were no possibilities of lupus nephritis, other autoimmune diseases, or antiphospholipid syndrome.

#### **3. Renal Biopsy and Laser-Microdissection Mass Spectrometry**

#### *3.1. Renal Biopsy*

Two samples of cortex were collected, including a total of 25 glomeruli without global sclerosis, mild mesangial cell proliferation in 10 glomeruli (40%), and normal tubulointerstitium and vascular system. PAM staining showed mild spikes with subepithelial immune complexes by AZAN staining, which were granularly stained along the capillary wall by fluorescent immunostaining for IgG (Figure 1).

Among the IgG subclasses, IgG1 was the strongest, IgG3 was mildly stained (Figure 2), and immunofluorescence for phospholipase A2 receptor (PLA2R) was negative (Figure 3A), suggesting secondary membranous nephropathy. Electron microscopy revealed an electrondense deposit in the subepithelial membrane and partly in the basement membrane, indicating stage II–III membranous nephropathy (Figure 4).

**Figure 1.** Renal biopsy. PAM staining (**A**,**C**), PAS staining (**B**), Azan staining (**D**), and immunofluorescence of IgG. The bars indicate 50 μm (**B**,**C**,**E**) and 25 μm (**D**).

**Figure 2.** Immunofluorescence of IgG subclass staining. The bars indicate 50 μm.

**Figure 3.** Immunofluorescence of PLA2R (**A**) and immunostaining for the p16-INK4a antibody (**B**). The bars indicate 50 μm.

**Figure 4.** Electron microscopy. The bars indicate 2 μm and 0.5 μm.

#### *3.2. Surrogate Marker for HPV Infection*

Immunohistochemistry was performed with the Leica auto-immune stain system using the antibody against the anti-p16-INK4a antibody (CINtec® Histology, Roche Diagnostics KK, Tokyo, Japan), which is typically used as a surrogate marker of HPV infection in cervical cancer and oropharyngeal cancer [12,13]. The anti-p16-INK4a antibody showed significant staining in the intra-glomerular cells (Figure 3B).

Furthermore, Western blot analysis identified p16 protein in the urinary protein at the time of renal biopsy. On the other hand, it was not found in the urine of patients with secondary membranous nephropathy related to cancer or primary membranous nephropathy—it was specific to this case (Figure 5). Therefore, it was suggested that this case could have already been infected with HPV before the time of renal biopsy.

**Figure 5.** Western blot of the urinary protein at the time of renal biopsy for p16-INK4a. The urine of the present case showed a band at MW16kD, whereas the urine of other cases of membranous nephropathy did not show a band for p16-INK4a.

#### *3.3. Laser Microdissection Mass Spectrometry*

We performed laser microdissection mass spectrometry (LMD-MS), as mentioned previously [14], to detect HPV viral capsule proteins. LMD-MS failed to detect HPV protein or antigens for membranous nephropathy, including THSD7A, EXT1/2, NELL1, Sema 3B, PCDH7, HTRA1, Contactin 1 [15,16], which were negative, except for a small amount of PLA2R in one sample (Table 1). The cytokeratin-related proteins were increased, whereas podocyte proteins such as nephrin, podocin, and podocalyxin were decreased compared to the control glomeruli from the renal transplantation at 1 h biopsy (Table 1, Figure S1).

**Table 1.** Laser microdissection mass spectrometry analysis of glomeruli of this patient and glomeruli from the 1 h renal biopsy after renal transplantation as control.



**Table 1.** *Cont.*

#### *3.4. Clinical Course*

The angiotensin receptor blocker losartan 25 mg was administered to reduce the urinary protein, decreasing it from 3.1 g /gCr to 0.19 to 0.65 g /gCr. When the dose of losartan was reduced to 12.5 mg after 6 months, urinary protein increased slightly to 0.41 g/gCr (Figure 6).

**Figure 6.** Time course of urinary protein after HPV vaccination and treatment with an angiotensin receptor blocker, losartan, as indicated by the yellow bar.

#### **4. Discussion**

This study focuses on a report of membranous nephropathy in an adolescent girl after HPV vaccination, and the pathogenic mechanism is investigated.

#### *4.1. Characteristics of Adolescent Membranous Nephropathy*

Membranous nephropathy is a major cause of nephrotic syndrome in middle-aged and elderly people but is rare in adolescents (1–2% of renal biopsies) [17]. Histopathologically, adolescent membranous nephropathy is found in a relatively early phase of Ehrenreich– Churg classification stages 1 and 2, in which patients are frequently positive for IgA, IgM, and C1q, in addition to IgG and C3 [18]. In the IgG subclass, IgG4 was 87.5%, whereas IgG1 was 46.9% and IgG3 was 56.3% were positive in half of the cases [19], and the frequency of PLA2R positivity was slightly lower than that in adults [20]. These data indicate that it is important to investigate secondary membranous nephropathy, such as SLE and HBV infection, in adolescent cases [18]. The present case was secondary membranous nephropathy with predominant IgG1 and IgG3 deposition but was negative for PLA2R. Most of the well-known causes of secondary membranous nephropathy were ruled out from clinical findings and laboratory tests. Therefore, HPV vaccination is presumed to be the cause of membranous nephropathy.

#### *4.2. HPV Virus Infection and Kidney Disease*

HPV is a sexually transmitted disease that is persistently transmitted locally to the cervix, and high-risk HPV types 16 and 18 cause about 70% of cervical cancers [8]. Interestingly, mother-to-child transmission can occur at birth [21], and 22.8% of newborns have HPV detected in the oral cavity at birth [22]. It is possible that even young people who have never had sexual intercourse have already been infected with HPV. In addition, 30.3% of 122 patients with renal cell carcinoma were found to be positive for HPV-DNA in the renal cell carcinoma site of the paraffin specimens by PCR method, 20.3% were found to be positive by immunostaining for p16-INK4a, and 45% were found to be positive for HPV by the in situ hybridization method [23]. This indicates that HPV was infected and latent in the kidney. In this case, p16-INK4a protein, a surrogate marker of HPV infection [12,13], was found in glomerular cells (Figure 4B) by immunohistochemistry and also in the urinary protein by Western blotting analysis (Figure 5). Therefore, it is possible that the patient was infected with HPV, having an antibody against it, and administration of viral protein as a vaccine formed circulating immune complexes and developed proteinuria soon after vaccination. The p16-INK4a is also detected in the glomerulus and tubules of the aging kidney and in kidneys with chronic allograft rejection [24,25], even though this was not the case with this young patient. HPV envelope proteins, such as E1 proteins, induce overexpression of a set of genes associated with proliferation and differentiation processes and downregulation of immune response genes [26]. LMD-MS, in this case, showed an increase in cytoskeletal proteins and epithelial junctional proteins as well, as downregulation of nephrin, podocin, and podocalyxin may reflect HPV infection in the podocytes. Unfortunately, we could not detect HPV envelope proteins by LMD-MS analysis. The possibility of primary membranous nephropathy still remains, as LMD-MS analysis detected a small amount of PLA2R protein in one glomerular sample.

#### *4.3. Kidney Disease associated with Vaccination*

In this study, the urinary protein was observed with a fever of 39 ◦C after two doses of a recombinant precipitated divalent HPV-like particle vaccine.

As shown in Table 2, a case of acute kidney injury and nephrotic syndrome due to membranous nephropathy was reported after influenza vaccination [27]. It has been reported that nephrotic syndrome could be caused by the HBV vaccine [28], pneumococcal vaccine [29], and COVID-19 vaccine [1,30]. However, there has never been a report of nephrotic syndrome caused by the HPV vaccine, and to the best of our knowledge, this study presents the first report of such a case.


**Table 2.** Nephrotic syndrome and nephritis associated with vaccination.

The HPV vaccine Cervarix contains added aluminum hydroxide and monophosphoryl lipid A (MPL) derived from the cell membrane of Salmonella as an adjuvant. Aluminum oxyhydroxide (alum) binds to viral DNA fragments and lipid A derivatives to form alumnanoparticles, which are taken up by macrophages and form a granulomatous lesion called macrophagic myofasciitis (MMF) [40,41]. MMF in the vaccine injection site induces cell death, and adjuvants conjugate with degraded nuclear DNA and proteins, producing autoantibodies, which may cause Guillain–Barré syndrome, SLE, autologous immune hepatitis, and cerebrovascular inflammation. These are so-called adjuvant diseases, an autoimmune/autoinflammatory syndrome induced by adjuvant (ASIA) [40,42,43]. In this case, the possibility of lupus was ruled out from clinical findings and immunological data.

#### **5. Conclusions**

We reported a case of young-onset IgG1-dominant and PLA2R-negative secondary membranous nephropathy after HPV vaccination. Since p16-INK4a was positive in glomerular and urinary proteins, she may have been infected with HPV, and administration of HPV envelop protein vaccines could be implicated in the development of secondary membranous nephropathy.

#### **6. Take-Home Message and Lessons Learned**

It is important to check proteinuria after HPV vaccination. Secondary membranous nephropathy could occur after HPV vaccination with viral proteins. If it is possible to check the plasma antibody titers for HPV before vaccination, this may help to prevent the occurrence of membranous nephropathy.

**Supplementary Materials:** The following supporting information can be downloaded at: https://www. mdpi.com/article/10.3390/vaccines10091442/s1, Figure S1 shows the whole proteins identified by LMD-MS analysis.

**Author Contributions:** Conceptualization, A.T.; methodology, A.T., Y.U., D.K. and K.H.; software, S.Y.; validation, Y.U., K.H. and A.T.; formal analysis, A.T.; investigation, H.A., S.Y. and D.K.; resources, A.T., Y.U. and K.H.; data curation, A.T. and D.K.; writing—original draft preparation, H.A., T.O. and A.T.; writing—review and editing, A.T. and K.H.; visualization, H.A., D.K. and A.T.; supervision, A.T., K.H. and Y.U.; project administration, A.T., K.H. and Y.U.; funding acquisition, A.T. and K.H. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was partially supported by a research donation from Naohiko Kobayashi, Director of Kobayashi Medical Clinic in Yasuzuka, Mibu, Japan (#2020-9, #2022-7).

**Institutional Review Board Statement:** The renal biopsy and sample analysis were performed with the approval of the research ethics committee of Dokkyo Medical University (R-2-1).

**Informed Consent Statement:** Written informed consent was obtained from the patient at the renal biopsy to use samples and data in anonymized reports or publications.

**Data Availability Statement:** A table showing all of the data from the LMD-MS analysis is available upon Supplementary Material.

**Acknowledgments:** We thank Kinichi Matsuyama and Mihoko Ishikawa from the Department of Pathology, Dokkyo Medical University, for their excellent technical help concerning immunohistochemistry and electron microscopy. We also thank Yasuko Mamada from the Clinical Research Support Center for assistance with the Western blotting.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **HPV-Vaccine Hesitancy in Colombia: A Mixed-Methods Study**

**Veronica Cordoba-Sanchez 1,\*, Mariantonia Lemos 2, Diego Alfredo Tamayo-Lopera <sup>1</sup> and Sherri Sheinfeld Gorin 3,\***

	- Medellín 050022, Colombia; mlemosh@eafit.edu.co

**Abstract:** In Colombia, the uptake rate of the HPV vaccine dropped from 96.7% after its introduction in 2013 to 9% in 2020. To identify the behavioural components of HPV-vaccine hesitancy in females aged 15 and under and their families, we conducted a convergent mixed-methods study in which 196 parents/caregivers responded to an online questionnaire and 10 focus groups were held with 13 of these parents/caregivers, and 50 age-eligible girls. The study is novel as it is the first to explore the factors influencing HPV-vaccine hesitancy alongside the COVID vaccine within an integrative model of behaviour change, the capability-opportunity-motivation-behaviour (COM-B) model. We found that COVID-19 has had an impact on the awareness of HPV and HPV vaccination. Lack of information about the vaccination programs, concerns about vaccine safety and the relationship between HPV and sexuality could be related to vaccine hesitancy. Trust in medical recommendations and campaigns focused on the idea that vaccination is a way of protecting daughters from cervical cancer could improve HPV vaccine uptake.

**Keywords:** vaccination; human papilloma virus; health behaviour; vaccine hesitancy; cervical cancer

**Citation:** Cordoba-Sanchez, V.; Lemos, M.; Tamayo-Lopera, D.A.; Sheinfeld Gorin, S. HPV-Vaccine Hesitancy in Colombia: A Mixed-Methods Study. *Vaccines* **2022**, *10*, 1187. https://doi.org/10.3390/ vaccines10081187

Academic Editor: Silvio Tafuri

Received: 5 June 2022 Accepted: 13 July 2022 Published: 27 July 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**1. Introduction**

Cervical cancer is the fourth-most-common cancer among women worldwide. In 2020, there were an estimated of 604,127 new cases and 341,831 deaths [1]. In less-developed countries, its incidence is higher and it ranks second for mortality, after breast cancer [2]. In Colombia, cervical cancer has a crude incidence rate of 18.3 per 100,000 women per year (95% UI: 4.311–5.216) and a mortality rate of 9.61 (95% UI: 2.316–2.677) [3]. In 2020, 7.9% of all new cancer cases in Colombian women were of cervical cancer, the equivalent of 4742 new cases [4]. Population-based studies in this country report that women diagnosed with cervical cancer are younger, of lower incomes, and more often live in nonmetropolitan/rural areas than those diagnosed with breast cancer [5]. The highest mortality rates are observed in the most deprived regions (along the main rivers, harbours, and cities along the country's borders). The low impact of cervical cancer screening programs in the country is attributed to the poor quality of pap smears; low coverage, especially of women at high risk; and a lack of or partial follow-up of women with abnormal cytology [6].

Almost all cervical cancers are caused by persistent infections with oncogenic, or highrisk, types of human papillomavirus [7,8]. Rates of cervical cancer have declined worldwide in countries with successful HPV-vaccination strategies, cervical cancer screening programs, and the treatment of cervical intraepithelial neoplasia (CIN) earlier in the pathogenesis pathway to invasive cervical cancer [9–21]. By contrast, countries without cervical cancer prevention and control strategies—or where the strategies are not effective—have seen a rapid increase in early mortality due to this pathology [2], especially in low- and middleincome countries [22]. To encourage wider implementation of successful cervical cancer

prevention and control strategies worldwide, on 17 November 2020, after the closure of the 73rd World Health Assembly, the WHO Global Strategy for Cervical Cancer Elimination was formalized [23]. WHO set the following goals for 2030: 90% of girls fully vaccinated with the HPV vaccine by the age of 15, 70% of women screened using a high-performance test by the age of 35, and again by the age of 45, and 90% of women diagnosed with pre-cancer and invasive cancer treated [24].

While HPV vaccines are effective against many of the oncogenic types of HPV, vaccination uptake and completion are low among young Colombian females. In 2012, Colombia was among the first countries in South America to implement HPV vaccination among age-eligible girls, reaching 97.5% uptake of first doses and 96.7% uptake of second doses. In 2013, Colombia's HPV vaccination rate was one of the highest in the world [25]. In 2014, however, a group of young women in a Colombian coastal town who had been vaccinated experienced a mass psychogenic response including fainting spells, weakness, limb paraesthesia, chest pain, tachycardia, and headaches (the "Carmen de Bolivar" event). The purported vaccine side effects were videoed by various media outlets and shared widely on social networks [26]. While the Instituto Nacional de Salud Colombiano (Colombian National Health Institute) conducted a rigorous epidemiological study of this event and did not find any organic association between the vaccine and symptoms described [27], the lingering effects of the event continued to lower HPV vaccine rates in 2016 to 14% for first doses and 5% for the second. By 2021, however, the rate of HPV uptake had risen to 39.4% for first doses and 11.8% for second doses, according to the Colombia Health Ministry [28].

This crisis was similar to others in countries such as Denmark, Japan, and Austria. Denmark experienced a rapid decline in vaccination in 2014 following negative public attention coinciding with increased suspected adverse event reporting to the Danish Medicines Agency. This negative public attention included stories of the supposedly harmful effects of HPV vaccination that were widely shared on social media [29,30]. In Japan, active recommendations for HPV vaccination were suspended in June 2013 following media reports of girls having various symptoms such as chronic pain and motor impairment after vaccination [31]. The suspension remained until November 2021, despite large-scale epidemiologic studies showing the effectiveness and safety of the vaccine in Japan and worldwide, and the scientific community repeatedly calling for the resumption of active recommendations by the Japanese government [31]. In Austria, the HPV vaccine was licensed in 2006, comparatively early relative to other countries, but was not free of debate in a nation with a historical scepticism towards vaccination. To retain the HPV vaccine, policy makers and scientific experts disassociated the vaccine from gender, vaccine manufacturers, and youth sexuality, ultimately making the vaccine a strength of the Austrian Immunization Program [32].

These crises have led to lowering the rates of HPV vaccination worldwide, with the current rate of HPV vaccination far below the 90% goal proposed by the WHO [23]. Factors associated with vaccine hesitancy in South America are a lack of information and doubts about its safety and effectiveness [33]. These results were similar to those found in a qualitative study of young females in Colombia that highlighted that parents/caregivers of girls eligible for the first dose of the HPV vaccine had little knowledge about the aetiology of cervical cancer, had not received information about vaccination benefits, and feared its adverse events [34].

Since HPV vaccination requires parental consent, parental hesitancy may be one of the strongest influences on uptake among adolescent females [35]. Parental HPV-vaccine hesitancy is a complex phenomenon that is influenced by lack of information, the opinions of important others, the attitudes/beliefs of healthcare providers, news about adverse effects, religious beliefs, and, as HPV is a sexually-transmitted infection, attitudes towards adolescent sexual behaviour [36].

The aim of this study is to identify the behavioural components of HPV-vaccine hesitancy among girls aged 15 and under and their families at educational institutions in Colombia. These factors are best understood within an integrative model of behaviour change. This study relies on the capability-opportunity-motivation-behaviour (COM-B) model of behaviour [37]. To our knowledge, the study is the first to explore the factors influencing HPV-vaccine hesitancy alongside the COVID vaccine in Colombia within an integrative model of behaviour change.

In accord with a phased approach to behavioural research [38], the findings from this study are intended to undergird a subsequent efficacy trial of a behavioural intervention to decrease HPV-vaccine hesitancy.

#### **2. Materials and Methods**

#### *2.1. Study Design*

This study used a convergent mixed-methods design with multiple data collection strategies to ensure completeness and integration of results, and to describe the perspectives of participants at different levels in a system (Figure 1).

**Figure 1.** Convergent mixed-methods design.

#### *2.2. Setting*

Medellín is in the Metropolitan Area of the Aburra Valley; this is the second-largest urban area of Colombia with more than four million people. The city has 229 public schools (from first to eleventh grade of basic education) and 337 private (with tuition) schools (including kindergarten only) [39]. Healthcare in Colombia is both private, or contributory, and public, with a government subsidy, depending on household income levels. The HPV-vaccine is administered in schools, hospitals, and health centres in a two-dose series (0, 6 months) with no cost for girls between 9 and 17 years old and a three-doses series (0, 6, 60 months) for women 18 years and older [40].

#### *2.3. Study Model*

We used the COM-B model (capability, opportunity, motivation, and behaviour [38]), to select the study's quantitative measures and for qualitative data collection, and as a guide for analysing the data. COM-B posits that behaviour change is influenced by three factors: (1) capability, having the physical and psychological abilities to engage in the behaviour; (2) opportunity, having the physical or social opportunities to engage in the behaviour; and (3) motivation, psychological processes that energise and direct behaviour such as the belief that a behaviour is important and/or socially desirable [41].

#### *2.4. Participants*

All age-eligible female students (9–15 years) with an incomplete vaccination schedule (0 or 1 dose) and their parents from schools in Medellín and surrounding cities were recruited to participate in the study. The convenience sample was comprised of 196 parents who consented to participate in the online questionnaire and 50 girls who participated in the focus groups after their parents' consent was given.

#### *2.5. Study Measures*

Based on the COM-B model, we developed an online questionnaire that was administered to the sample of parents (see items in Appendix A). The quantitative measures were: perceived susceptibility, vaccine awareness, trust, beliefs about vaccine safety and efficacy, likelihood of vaccinating, factors influencing the decision to vaccinate or to complete the vaccination schedule, and intention to complete the vaccination schedule.

The same questions were asked of both the parents and the young women who participated in 10 focus groups, alongside prompts to encourage them to explain their responses more fully, to gain a deeper understanding of their thinking patterns within their social contexts [42]. An example is: "If you are worried about the safety of the HPV vaccine, tell us—what are your concerns?" By matching the qualitative focus-group questions to the quantitative survey items, we were able to integrate and interpret findings from the two approaches in a joint display.

#### *2.6. Procedure*

A convenience sample of three public and one private school was recruited; within each school, invitations were sent to parents to complete the online questionnaire and to participate in focus groups. In addition, to reach the largest possible number of participants in the defined geographical area, the online questionnaire was also shared in the socialmedia and instant-messaging groups of parents from different schools. Parents/caregivers who attended the focus groups were also asked for permission for us to contact their daughters to participate in focus groups with other girls from their own school and of the same approximate age. Groups with parents/caregivers were conducted using video-call platforms (Microsoft Teams and Google Meet) and lasted an average of 34.7 min. They were coordinated by a male psychologist and two research assistants. The girls' groups lasted an average of 28 min and were coordinated by a female psychologist and two research assistants using a room and schedule provided by the schools.

#### *2.7. Ethics*

Before starting, both parents and girls gave their consent and assent, respectively, in writing. All focus groups were recorded and then transcribed verbatim for subsequent analysis. Data were collected between September and November 2021. This project was approved by the Universidad EAFIT Ethics Board.

#### *2.8. Data Analysis*

The sample was characterized by descriptive statistics. Associations were established between categorical variables using Fisher's chi-square, and correlations were established using Spearman's rho. A binary logistic regression analysis of the probability of HPV vaccination was conducted, with the following independent variables: socioeconomic position, religion, factors associated with vaccination uptake such as cost, and medical recommendation, in accord with the COM-B model.

The qualitative data were analysed with thematic analysis [43], using AtlasTi version 7.5.4 for data management. Thematic analysis included: (1) Familiarization: Transcribing the entire dataset, and reading it twice to gain familiarity with the data; (2) Coding: generating initial codes, attaching codes to significant quotes, refining codes through two rounds to collate or split them if necessary; (3) Generating initial themes: examining the codes and looking for patterns and connections to cluster similar codes and create

candidate themes; (4) Developing and reviewing themes, using visual maps and reading the codes of each theme; (5) Refining and naming themes; and (6) Writing the synthesis of each theme and attaching illustrative quotes.

The data were integrated by assessing the concordance between the quantitative and qualitative results according to the components of the COM-B model. As is common in mixed-methods analyses, quantitative data were elaborated and further explained with the qualitative data through a joint display table, a strategy to organize and integrate the data and show how they were mixed [44]. Our joint display showed both the quantitative factors influencing HPV-vaccine hesitancy for parents/caregivers and daughters based on the COM-B model and the qualitative information to support each component.

#### **3. Results**

#### *3.1. Description of the Sample*

One hundred and ninety-six adults, with an average age of 42.2 years (SD = 6.39), 89.3% of whom were female, responded to the online questionnaire (see Table 1). Of the respondents, 13 parents/caregivers agreed to participate in the focus groups. Fifty girls aged 9–15 (average age of 11.5 years, SD = 1.9) who were students from schools in Medellín and surrounding cities, with an incomplete vaccination schedule (0 or 1 dose), participated in the focus groups (see Table 2).


**Table 1.** Sociodemographic characteristics of online questionnaire participants (N = 196).

#### *3.2. Analysis Based on the COM-B Model*

In the sections that follow, we present both quantitative and qualitative findings aligned with the COM-B model. Qualitative themes were determined by matching content of the focus groups with constructs from the model after analysis (see Table 3). In the COM-B component capability, the themes that we established were 'lack of information' and 'the relationship between HPV and cervical cancer.' The only theme describing opportunity was trust in traditional institutions. The association of HPV with sexuality and the vaccine as an act of care was related to automatic motivation, while respect for the personal decision to be vaccinated was related to reflective motivation. One theme that was transversal to each component was the impact of COVID-19 on the conception of HPV.


**Table 2.** Sociodemographic characteristics of girls and adolescents in focus groups (N = 50).

**Table 3.** A joint display of factors influencing HPV-vaccine hesitancy to parents/caregivers and daughters based on the COM-B model.


#### 3.2.1. Capability

Most participants perceived a susceptibility to cervical cancer and HPV, and were aware that their daughters should be vaccinated, but 69.4% stated they had not received any information about the vaccine. Likewise, there was a relationship between the likelihood of getting their daughter vaccinated and perceived susceptibility to HPV, as well as receiving positive information about the vaccine. Qualitative analyses confirmed that parents did not have enough information about the vaccine and vaccine programs, and some cited misinformation about the Carmen de Bolivar event. Additionally, there is a recognition that cancer is a severe disease that could be treatable if it is diagnosed early. However, participants were not completely aware of the link between HPV and cancer because they thought that, given the age of the girls, they were not at risk.

#### Lack of Information

Participants reported a lack of information about HPV; further, many still remembered the images of the women and girls who fainted en masse in Carmen de Bolivar. For example, one of the parents said: "*I saw on the news when the girls from the coast were vaccinated and they fainted, so there is no one in my family who has received those vaccines*" (mother of unvaccinated girl, 45 years old, private school). The legacy of this event has been distorted, with misinformation highly prevalent. Participants stated that the correct information needs to be provided to individual parents by doctors, but this approach is limited to one parent at a time. Thus, the larger responsibility for increasing public awareness and public education rests with the health and educational systems.

"*I think that there has been a need for an education campaign, that is, promulgation of what papilloma is and what the vaccine does, because normally when a vaccine is talked about repeatedly by many media and people get educated, people have more awareness and then they do it conscientiously, but I have not heard that there has been much of a campaign or education on the subject*". (mother of unvaccinated girl, 51 years old, private school)

For some parents, however, the first mention of the HPV vaccine is as a requirement for school: "*In my home we realized, first at school because they were asking about the vaccination card, and then with the prepaid healthcare provider; there the doctor talked about the vaccine*" (mother of girl with one dose, 41 years old, private school).

Some parents had gained limited and biased knowledge from their physicians, only in the context of their own diagnoses.

"*I was diagnosed with HPV 10 or 12 years ago (* ... *) they explained to me that even the nuns get it [HPV], even if they haven't had sex. I am in this meeting because I have many gaps [in my knowledge] and ignorance on the subject*". (mother of unvaccinated girl, 45 years old, private school)

#### Relationship between HPV and Cervical Cancer

Among all the focus groups of girls, cancer meant severe disease, with difficult treatments, and death as a possible outcome. However, this was not specific to cervical cancer but to the concept of cancer. "*For me, the worst thing that can happen to a person is the treatments they have to do, because they need quick treatment or they can die, because it is a very serious cancer, and it affects women*". (girl, one dose, 10 years old, public school).

For parents, cervical cancer is treatable when detected early. "*They do a surgery on that, don't they? If it is detected in time, they operate on it, then people do not metastasize or something like that. That's the only thing I know*" (mother of girl with one dose, 41 years old, private school).

Girls recognize the value of the vaccine for reducing HPV susceptibility and severity. The participants imagine (incorrectly) that the vaccine puts a little bit of the virus into the body; thus, if they get the disease, it will not be serious. They know that several doses are required for it to be effective, and one of them believed that the effect of the vaccine gets lost over time. "*They put bits of the virus into the vaccine so that when the virus reaches your body the virus doesn't hit you so hard, so that's what I know about vaccines*" (girl, one dose, 10 years old, public school).

#### 3.2.2. Opportunity

In focus groups of parents, participants mentioned that healthcare providers and schools were trusted information sources. Girls tended to trust their families, especially their mothers.

#### Trust in Traditional Institutions

The family and the health system are most trusted sources from which to receive information about the vaccine, and to make recommendations. For example, one of the girls said, "*If your mother tells you [to get vaccinated] it is because you mother wants the best for your life*" (unvaccinated girl, 9 years old, private school).

These two institutions are so important that they can alter previous negative attitudes and beliefs toward the vaccine. The healthcare provider can influence HPV vaccine uptake, particularly when other vaccines are routinely administered. Paradoxically, even in a country like Ecuador, which has higher reported rates of HPV vaccination than Colombia according to the WHO [45], one participant stated that she had never heard of the vaccine.

"*I wanted to say that my ignorance is even bigger because my children were born in Ecuador and there nobody talks about HPV (* ... *). I was one of those mothers who said that I would not give my daughter that vaccine, until recently, when I went to get a booster for my other 10-year-old son and I spoke with a doctor who explained to me that it was absolutely safe, so I began to think differently*". (mother of unvaccinated girl, 45 years old, private school)

Parents tend to trust in their doctors even when they have doubts about the vaccine. There is little reported discomfort with the vaccination-card requirement at school or with vaccination campaigns at schools, supporting the idea that institutions take care of people.

Quantitative data corroborated this finding, since 87.8% of respondent trusted the information they received about vaccines. We also found a bivariate relationship between the likelihood of getting vaccinated and receiving positive information about the vaccine (Spearman r = 0.338, *p* < 0.001). For parents, a binary logistic regression showed that the recommendation of the doctor was the most important factor related to the probability of the daughter getting vaccinated (See Table 4).

**Table 4.** Logistic regression results for predicting the reasons to vaccinate daughters using socioeconomic position (SEP), religion, and other influences as independent variables.


Note. SEP = Socioeconomic position.

#### 3.2.3. Motivation

Regarding motivation, survey data revealed that some parents were concerned about the effectiveness, adverse effects, and safety of the vaccines. In focus groups, parents also revealed a conflict between their desire to protect their daughters from the harmful virus and their concerns about encouraging early sexual activity through vaccination. Often, this led to parents avoiding talking about the HPV vaccine.

#### The Vaccine as an Act of Care

Most parents considered that vaccination is a form of protection and care for their daughters, which leads them to demand clear information about its effectiveness and safety. "*I did it [vaccinate the daughter] first of all, to protect my daughter's life*" (mother of girl with one dose, 41 years old, public school).

"*It is a matter of avoiding the disease; one does not know when it could happen, God willing it does not happen, but it is a way of caring for and protecting them*". (mother of girl with one dose, 41 years old, private school)

Often, parents would mention the adverse event at Carmen de Bolívar, leading to uncertainty about the impact of the vaccine.

"*I have heard many mothers who did not want to vaccinate their daughters; I do not know why. A long time ago there was a problem with some vaccine, after which many mothers believe in these things, and they are scared to vaccinate their daughters*". (mother of unvaccinated girl, 43 years old, private school)

#### Respect for the Personal Decision to Be Vaccinated

Some of the participants reported that vaccination is an individual act and at the same time, an act of collective responsibility. The decision cannot be forced, however. "*I think there must be free will there. Not only from parents, since at a certain age, young people make decisions regarding their bodies, and I think that it should not be mandatory*" (father of girl with one dose, 53 years old, private school).

Girls understood that to access to the vaccine they must go through two gatekeepers: first their parents and then their healthcare provider, acknowledging that, as minors, the decision rests with the parents. Some of the girls think that the vaccination is their decision; however, most of them think that this is a parental decision. "*Well, it depends, my mom talks about it with me; I tell her there is this vaccine, then she tells me yes, and the decision is mine, as she says: your body is yours*" (girl, one dose, 10 years old, public school).

"*I never had the power to decide if I wanted to get vaccinated or not; it was not a subject that I could get into*". (Girl, one dose, 15 years old, public rural school)

#### Association of HPV with Sexuality

As HPV is a sexually transmitted infection, some parents state that prevention can be conducted through education in values, not talking about sexuality openly.

"*I consider that in the subject of the infection with the human papilloma virus, it is possible to take a look from the biological point of view as the subject of the disease; let's say that it is detached from the subject of sexuality and that is perfectly well because one can approach the subject with one's daughter like that*". (Father of girl with one dose, 53 years old, private school)

Some parents stated that HPV infection occurred because people start sexual life early; they considered it an adult issue and did not consider their daughters susceptible to infection at their young ages. "*I have an acquaintance who had this cancer because she began her sexual life very young. That person passed away. She always warned us about the dangers of such a crazy sex life*" (mother of girl with one dose, 40 years old, public school)

This is also perceived by the girls, who state that their parents delegate the issue of sexuality to others, and that at school the information is provided superficially or not at all. "*Because it is a sexually transmitted disease, people avoid talking about it*" (girl with one dose, 15 years old, private school).

#### 3.2.4. The Impact of COVID-19 on HPV Hesitancy

As a result of the pandemic, people understand more about how vaccines work and what happens with a virus, and the analogy with COVID-19 allows them to understand that to be fully protected, they need several doses of a vaccine. Some of the young participants incorrectly think that the vaccines contain live viruses, however. In Colombia, greater understanding of COVID has allowed certain anti-vaccine discourses to be demystified (relatives of participants who were anti-vaccine but were now vaccinated) and negative ideas about vaccines to be discredited.

"*I don't know exactly how the virus spreads, but I suppose that, like a normal virus, like when you get vaccinated against the coronavirus, you prevent yourself and others, that you don't spread it, nor are you infected*". (girl with one dose, 15 years old, public school)

Yet, especially in younger girls, different forms of HPV and COVID transmission, prevention, vaccination components and dosage, and side effects can be confused. "*Well, I think it is transmitted like when you cough on someone or if you are with that person a lot, the virus passes to you or something like that*" (unvaccinated girl, 9 years old, private school).

Importantly, regarding opportunities, the pandemic further limited access to the HPV vaccines and reduced the importance of health campaigns focused on HPV vaccines.

#### **4. Discussion**

The aim of this study was to understand the psychological predictors of HPV vaccine hesitancy in girls aged 15 and under and their parents/caregivers who live in and around Medellín. In this sample, more than half (57.1%) of participants have not initiated vaccination despite quantitative data showing that parents and caregivers who participated in the study perceived a high level of susceptibility to HPV and cervical cancer in girls and adolescents. They also had high awareness of the need for the vaccine. However, qualitative data revealed some unrealistic parental views of their daughters' susceptibility to HPV; many parents thought that since their daughters are young, they were far from starting their sexual life. There is limited information about organized HPV vaccination programs in this sample, nor does it reveal how physical and social opportunities affect access to vaccines. Additionally, despite Colombia Health Ministry-led public health communications about the safety of the HPV vaccine, some participants remained concerned about the vaccine's effectiveness and safety, and the side effects that the vaccine might cause.

Parents also were not especially aware of the vaccine's benefits. This finding was also reported in an earlier study in a Colombian population [34]. Limited information and low vaccination rates are clear challenges for public health, given the prevalence of cervical cancer in Colombia and worldwide [2,5]. These findings point to the need for interventions that provide information about HPV, its relationship to cervical cancer, and the country's national vaccination program to increase the public's vaccine uptake. However, simply providing information does not cause a change in vaccination behaviour [46]. Perceived susceptibility to HPV is a fruitful target for future intervention approaches. Multiple studies have reported that perceived susceptibility is associated with a greater likelihood of getting vaccinated [47]. Educational interventions that also focus on susceptibility, by pointing out that HPV causes genital warts and that vaccination prevents diseases attributable to this virus have demonstrated effectiveness [48,49]. The WHO communication guidelines include educational interventions designed to increase HPV-vaccine uptake to increase the perception of susceptibility and the consequences of HPV infection [50].

The COVID-19 pandemic has highlighted the threat of an infectious disease and has increased awareness of the vaccine-development process, since many parents had never seen the devastating consequences of an infectious disease before 2020 [51]. This creates an opportunity to generate awareness about HPV and its relationship with cervical cancer, but also to promote the use of the vaccine as the best way of preventing future complications.

This study also showed how physical and social opportunities have an influence on vaccine-access behaviour. It is crucial that barriers to vaccine access be reduced, by offering inoculations at schools [52], at times when parents can give consent, since that is needed for vaccination. Educational institutions may be optimal settings for HPV-vaccine interventions, as natural gathering sites for students, parents, healthcare professionals, and teachers. School-based interventions may help to change attitudes and beliefs, as well as increase knowledge to increase HPV-vaccine uptake [53].

We found that people trust their healthcare professionals (HCPs) and the information they provide. This element is crucial when we consider that HCPs can promote vaccination by sharing accurate information and offering counselling to parents to facilitate decision making. Nonetheless, previous studies have found that the percentage of HCPs who speak with parents about HPV vaccines for their children is very low [54]. However, HCPs are critical to HPV-vaccine uptake, so provider-based interventions also offer considerable promise [55]. A provider-focus has also been recommended by the HPV Prevention and Control Board of Colombia, so broader implementation is a next step [56].

Lastly, it is important to remember the motivational component of this behaviour. Paradoxically, although most participants in this study trusted the vaccine information received and the healthcare system providing it, some of them were distrustful about vaccine safety, efficacy, and potential adverse effects. This aligns with studies in other countries, in which fears about side effects and vaccine safety, distrust in the vaccine, fear of the possibility of death, and negative comments from neighbours or acquaintances about the HPV vaccine were paramount in parents who did not want to vaccinate their children [57]. In Colombia, these fears were enhanced with the widespread coverage of the 2014 mass psychogenic response [26] that still remains in the public's memories.

The thematic analysis showed that although the act of vaccine is seen as an individual decision, mothers have a crucial influence on their daughters. Generally, parents' motivation was to protect and take care of their daughters. To increase the vaccination rates, parents should be involved, as has been suggested in other studies [36,48].

As HPV is primarily a sexually transmitted disease, the HPV vaccine is different from other vaccines for children. In Colombia, it is intended especially for girls [58], although HPV-vaccination should be gender neutral. Both men and women can develop cancer caused by HPV; further, it is easier to ensure a high coverage to achieve herd immunity when both sexes are vaccinated [59].

Several studies have reported that parents and caregivers may associate the vaccine with fears of compromising fertility [60] or giving children permission to become sexually active [61]. While the perceived risks are nearer term, the potential benefits of HPV vaccination are much longer term, since it takes 15 to 20 years for cervical cancer to develop in women with normal immune systems [62] and the estimated median time from HPV acquisition to cancer detection ranges from 17.5 to 26 years [63]. Educational interventions could highlight both the near-term and longer-term benefits of the HPV vaccine.

Creating opportunities for public discussion, led by trusted healthcare providers, for example, could help to allay these parental fears and increase HPV-vaccine uptake. Parents and educational institutions should be involved in the co-design of these public forums. Currently, governments should take clear actions to improve vaccination rates. Denmark shows how a nation grappled with negative media coverage of HPV vaccination with a parent information campaign designed to boost confidence in the safety of the vaccine [64]. Austria launched a renewed national discourse on vaccination, changing the message of vaccination from "save women's lives" to "save lives" to disassociate the vaccine from gender [32].

The US Centers for Disease Control and Prevention (CDC) has recommended ten strategies for healthcare providers to increase the HPV vaccine. These range from giving active recommendations, recommending the HPV vaccine the same day and the same way as all other vaccines, preparing to answer more common questions, learning the reasons for vaccine refusal, and implementing systems to ensure that an opportunity to vaccinate is never missed [65].

Before concluding, we want to highlight the limitations of the study. It was conducted in the middle of the COVID-19-pandemic health emergency. This meant that all parent focus groups were remote, and their participation was limited by connection capability and speed. However, remote data-collection strategies may increase participation among those who may not otherwise be able to participate due to logistical or time constraints [66,67]. Use of a convenience sample from a geographically-limited set of participants may bias the findings and limit generalizability. For the quantitative analysis, we did not include variables that have predicted HPV-vaccine hesitancy in other studies, such as the level of parental education [68], so the findings may be confounded. Importantly, the use of

mixed methods combines the strengths of both quantitative and qualitative research, by allowing the researcher to add insights that could be omitted when only a single method is adopted [69].

#### **5. Conclusions**

To summarize, this study improves the understanding of the capabilities, opportunities, and motivations associated with the HPV vaccine to reduce HPV-vaccine hesitancy for parents, girls, and adolescents in Medellín and nearby cities. It is important to provide information about the national vaccination program and HPV's association with cervical cancer. Likewise, it is important to make vaccination access easier by returning to the strategy developed in 2012 to bring the program to schools, thereby reducing barriers to access. Lastly, parents' concerns about vaccine safety and side effects should be acknowledged; healthcare leaders should create spaces for discussion so that the ideas behind the distrust can be dismantled.

**Author Contributions:** Conceptualization, V.C.-S. and M.L.; Data curation, S.S.G.; Formal analysis, V.C.-S. and M.L.; Funding acquisition, V.C.-S., M.L., D.A.T.-L. and S.S.G.; Investigation, V.C.-S., M.L., D.A.T.-L. and S.S.G.; Methodology, V.C.-S., M.L. and D.A.T.-L.; Project administration, D.A.T.-L.; Supervision, S.S.G.; Writing—original draft, V.C.-S., M.L. and D.A.T.-L.; Writing—review and editing, S.S.G. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Institucion Universitaria de Envigado (V.C.-S. and D.A.T.-L.), COD\_00-202 and Universidad EAFIT (M.L.). S.S.G. was supported in part by Michigan Institute for Clinical and Health Research grant UL1TR002240. The study sponsors had no role in the design of the study.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki, and approved by the Ethics Committee of Universidad EAFIT on 21 August 2021.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data supporting reported results can be found at https://osf.io/ n396v/?view\_only=6f04d370460a419bbeceba5600ce91c3 (accessed on 31 May 2022).

**Acknowledgments:** We thank the IBTN for introducing us during IBTN Summer School 2021 and the Colombian schools that helped us to recruit participants. Thanks also to Melissa Dejonckheere and Tim Guetterman for their helpful comments on the paper.

**Conflicts of Interest:** The authors declare no conflict of interest and the funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **Appendix A. Online Questionnaire**

Perceived susceptibility, vaccine awareness, and intention to complete the vaccination schedule:


Trust, beliefs about vaccine safety and efficacy: These items were answered on a Likert scale ranging from 0 (Not at all concerned) to

3 (Very concerned).


These items were from the Parent Attitudes about Childhood Vaccines (PACV) questionnaire, developed by Opel et al. (2011) and adapted into Spanish by Cunningham et al. (2019). This questionnaire has a Cronbach's alpha of 0.74 for the safety and efficacy domain, 0.84 for the general attitudes domain, and 0.74 for the behaviour domain [59].

Factors influencing the decision to vaccinate or to complete the vaccination schedule:

• "If a vaccine against HPV was available for your daughter, how likely is it that she would have it?" 0 (Extremely likely) to 3 (Not at all likely).

The following questions covered factors related to the vaccination decision from previous literature [60], which were answered on a Likert scale ranging from 0 (Very much) to 3 (Not at all).


#### **References**


## *Article* **What Factors Are Associated with Attitudes towards HPV Vaccination among Kazakhstani Women? Exploratory Analysis of Cross-Sectional Survey Data**

**Gulzhanat Aimagambetova 1,\*, Aisha Babi 1, Torgyn Issa <sup>1</sup> and Alpamys Issanov 2,3**

<sup>1</sup> Department of Biomedical Sciences, School of Medicine, Nazarbayev University,

Nur-Sultan 010000, Kazakhstan; aisha.mukushova@nu.edu.kz (A.B.); tshokanbaeva@nu.edu.kz (T.I.) <sup>2</sup> Department of Medicine, School of Medicine, Nazarbayev University, Nur-Sultan 010000, Kazakhstan; alpamys.issanov@nu.edu.kz


**Abstract: Background.** The high prevalence of HPV infection among Kazakhstani women and the absence of an HPV vaccination program are directly reflected in increasing rates of cervical cancer incidence and mortality. Kazakhstan made its first attempt at introducing the HPV vaccine in 2013, but was unsuccessful due to complications and low public acceptance. The attitudes of Kazakhstani women towards the vaccine were never measured. Therefore, this study aims to investigate the attitudes of women towards the HPV vaccine and determine factors associated with positive, negative, or neutral attitudes. **Methods.** A 29-item survey consisting of 21 demographic and contextual questions and 8 Likert-scale questions was distributed among women attending gynecological offices in four major cities of Kazakhstan from December 2021 until February 2022. Attitudes of women were measured based on their answers to the eight Likert-scale questions. Ordinal logistic regression was built to find associations between demographic characteristics and attitudes of women. **Results.** Two hundred thirty-three women were included in the final analysis. A total of 54% of women had positive attitudes towards the vaccine. The majority of women did not trust or had a neutral attitude towards the government, pharmaceutical industry, and traditional and alternative media. However, the trust of women was high in medical workers and scientific researchers. Women's age, education, number of children, effect of the 2013 HPV program, and trust in alternative medicine were included in the ordinal logistic model. Women with a low level of education, a high number of children, who believe in alternative medicine, and who were affected by the failed 2013 vaccination program were less likely to have a positive attitude towards the vaccine. **Conclusions.** Contrary attitudes towards HPV vaccination exist among Kazakhstani women, with approximately half having positive and almost half having negative or neutral attitudes towards the vaccine. An informational campaign that takes into consideration women's levels of trust in different agencies, as well as targets those who are the most uninformed, might help in a successful relaunch of the HPV vaccination program. However, more studies that cover a higher number of women are required.

**Keywords:** HPV; HPV vaccine; HPV vaccine knowledge; HPV vaccine awareness; Kazakhstan; cervical cancer prevention

#### **1. Introduction**

High-risk human papillomavirus (HR-HPV) infections cause a wide variety of benign and malignant conditions, including cervical cancer [1–4]. More than 90% of cervical cancer cases are attributed to HR-HPV infections, with HPV-16 and HPV-18 being reported to cause 70–75% of cases [3,4]. The knowledge that persistent HR-HPV infection is causally associated with cervical cancer has resulted in the development of prophylactic vaccines to prevent HPV infection and, thus, decrease the rates of HPV-related diseases [4]. HPV

**Citation:** Aimagambetova, G.; Babi, A.; Issa, T.; Issanov, A. What Factors Are Associated with Attitudes towards HPV Vaccination among Kazakhstani Women? Exploratory Analysis of Cross-Sectional Survey Data. *Vaccines* **2022**, *10*, 824. https:// doi.org/10.3390/vaccines10050824

Academic Editor: Gloria Calagna

Received: 26 March 2022 Accepted: 5 May 2022 Published: 23 May 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

vaccination programs have been implemented successfully in many high-income countries and have led to a decline in cervical cancer incidence rates [5–7]. However, the situation in low-income countries remains deplorable—cervical cancer is the leading cause of cancerrelated death in women, as primary and secondary prevention interventions are insufficient or lacking [4].

The association of HPV infection with anal, penile, head, and neck cancers is also well documented [8–10]. Therefore, immunization with the HPV vaccine can protect from the development of other HPV-related cancers.

The prevalence of HPV infection varies among countries worldwide, with the highest prevalence of HR-HPV infection among women in the developing regions of Southern and Eastern Asia (44.4% and 36.3%, respectively) [11]. A high prevalence of HPV infection was also identified among Northern American and Eastern European populations (41.1% and 28.9%, respectively) [11]. The lowest prevalence of HPV infection is reported in Middle Eastern and North African countries (7–16%) [12]. According to the available epidemiological data, HR-HPV prevalence in Kazakhstan is high among women attending gynecological offices, ranging from 39% to 43% [13–15].

The high prevalence of HR-HPV strains in Kazakhstan is contributing to increasing cervical cancer rates. Over the past 10 years (2009–2018), the crude rate of cervical cancer incidence in Kazakhstan has increased from 16.3 ± 0.4 to 19.5 ± 0.5 per 100,000 female population (*p* < 0.001) [16]. The national cervical cancer screening program was updated in 2017 [17], and eligible women received the screening free of charge. However, according to reports, the state-funded coverage reached only 50% of the target population [18]; thus, the screening coverage was low (around 46%) [18–20].

In 2013, the HPV vaccination campaign was introduced in Kazakhstan as a pilot program in four large regions [21]. Bivalent and tetravalent HPV vaccines were approved for the campaign, targeting 11–12-year-old girls [18]. However, the lack of an HPV vaccination awareness campaign that should have preceded and accompanied the vaccination program and the influence of social media's negative content on the program have led to a negative public reaction and unwillingness of parents to vaccinate their children against HPV [18,21,22]. Parents refused to vaccinate their daughters due to concerns about the HPV vaccine's safety and efficacy [23]. The vaccination program was discontinued in 2015 indefinitely. In 2020, the Ministry of Healthcare of the Republic of Kazakhstan announced the intention to relaunch the HPV vaccination program; however, due to the COVID-19 pandemic, it has been postponed.

Given the importance of the reintroduction of HPV immunization to prevent new cervical cancer cases, it is essential to understand what factors are associated with parents' willingness to vaccinate their children. Studies have shown that attitudes towards HPV vaccination are associated with HPV vaccination uptake [24]. Multiple factors play a role in shaping attitudes towards vaccination [25,26]. Education, social status, income, and cultural and religious preferences have a substantial impact on people's beliefs, thus affecting their understanding of different aspects of HPV as a sexually transmitted infection (STI) and its related diseases [26]. As reported by studies, attitudes and intentions to receive HPV vaccination vary from 15% to 95% in different societies, depending on the multiple factors involved [26–28].

As one of the tasks announced by the World Health Organization (WHO) in the Global Strategy to Accelerate the Elimination of Cervical Cancer [29,30] is to increase HPV vaccination coverage up to 90% among the target group, it is very important to investigate HPV knowledge, awareness, and attitudes towards HPV vaccination. For the successful relaunching and implementation of the HPV vaccination program in Kazakhstan, the investigation of attitudes to HPV vaccination among the general population has become even more essential, especially among women, the population most affected by HPV-related cancers. Thus, by surveying Kazakhstani women across the country, this pilot study aimed to evaluate attitudes towards HPV vaccination and explore factors potentially associated with differing attitudes towards the vaccine.

#### **2. Materials and Methods**

#### *2.1. Study Design and Subjects*

This is a cross-sectional study, which was conducted from December 2021 until February 2022 in four major cities (Nur-Sultan, Almaty, Aktobe, Oskemen, Kazakhstan) representing the central, southern, western, and eastern regions of Kazakhstan. Women aged from 18 to 70 years old visiting gynecological offices were recruited for the study. A total of 399 women agreed to participate in the study, and 347 reported full demographic characteristics. Out of those 347 women, 321 (80.45%) fully answered the HPV vaccine attitude questions. Almost 27% were excluded due to incomplete answers to the contextual questions. In total, 233 women were included in the complete-case analysis (Figure 1).

**Figure 1.** Flow chart on participants' inclusion.

#### *2.2. Study Instrument*

Two questionnaires were used in the study. The first collected data on demographic characteristics of women. The second questionnaire was adapted from the French Survey Questionnaire for the Determinants of HPV Vaccine Hesitancy (FSQD-HPVH) [31]. The questionnaire was adapted to the context of Kazakhstan (Supplementary Material). Based on experts' opinions, some questions were removed due to being irrelevant, and equivalent questions were added. For example, statements such as "Most of my friends get their daughters vaccinated against HPV" were removed, as the HPV vaccine is not readily available in Kazakhstan. The statement "Since the controversy over the H1N1 flu vaccination, I have less confidence in the French vaccination recommendations" was replaced with a statement of COVID-19, as it fits the context. All questions were translated and back-translated to Kazakh and Russian languages (official languages of the country) by experienced independent bilingual translators. The questionnaire included 29 items asking about contextual factors (historical factors, policies, and mandates), trust in different agencies, and beliefs and attitudes towards HPV vaccination.

#### *2.3. Study Variables*

#### 2.3.1. Independent Variables

Independent variables were socio-economic and demographic characteristics such as age, ethnicity, level of education, city, family income, marital status, and number of children. According to age, women were categorized into four major groups: 18–27 years old, 28–33 years old, 34–43 years old, and 44 years old and older. Age of women was categorized according to the quartiles. Ethnicity was categorized into 2 groups: Kazakh and other ethnic groups. There were three levels of education: high school or below, college, and university. In January 2022, the average income in Kazakhstan was 269 149 tenge (USD ≈ 520) [32]. Income lower and higher than the average was categorized as lower and upper income, while average income was categorized as middle income.

In addition, contextual factors such as historical factors, policies, and mandates were used as independent variables. This section consisted of 6 Likert-scale questions. The major topics of the questions were the impact of COVID-19 vaccination on trust in HPV vaccine recommendations (COVID-19 effect), the 2013 HPV vaccination program in Kazakhstan (2013 HPV program), the relationship of HPV vaccination with sexual promiscuity, compulsory vaccine mandated by the government, freedom of vaccine choice for children, and alternative medicine strengthening body immune system (alternative medicine beliefs). Statements about trust in different agencies were also used as independent variables. The Likert scale was used to measure trust in agencies and health systems such as the pharmaceutical industry, government, medical workers, scientific researchers, traditional media, and alternative media.

#### 2.3.2. Outcome Variable

The attitude of the participants was measured using 3 positive statements and 5 negative statement items on the Likert scale. Negative statements were reversed, and the mean attitude score of the participants was calculated. The attitude of the women was categorized as <3—negative attitude, 3—neutral attitude, and >3—positive attitude to separate women who have a positive or negative opinion about the vaccine from those who are hesitant. It is important to differentiate those who refuse vaccination from those who do not have an opinion and lack confidence in vaccination [33].

#### *2.4. Statistical Analysis*

Statistical analysis was performed using STATA 16 [34]. Cronbach's alpha was calculated for attitude items to measure internal reliability. The scale included both positive and negative items. Positive items had a Cronbach's alpha coefficient of 0.82. Negative statements had a Cronbach's alpha coefficient of 0.75, and the overall Cronbach's alpha was 0.79. Factor analysis was performed to test the questionnaire's consistency and validity. The Kaiser–Meyer–Olkin measure of sampling adequacy (0.752) and the Bartlett test of sphericity (*p* < 0.001) were performed before the factor analysis. All factors had a uniqueness lower than 0.6, except for the statement "Majority of my friends vaccinate their daughters from HPV", which had a uniqueness of 0.81. The item was dropped from the attitude scale, as it was not relevant to the Kazakhstan context due to the HPV vaccine not being readily available in the country.

Descriptive statistics consisting of mean values, standard deviations, and frequencies were obtained using univariable and bivariable analysis. Chi-square and Fisher's exact tests with a significance value of <0.005 were used to analyze the relationships between categorical variables.

Ordinal logistic regression was performed to explore factors associated with attitude towards HPV vaccination among women. Variables that showed significance in the bivariable analysis and were important as epidemiological factors were included in the final model. Among participants' characteristics, variables such as age, education, and number of children were included in the final model. Although participants' age was not a statistically significant factor, it was included in the final model due to its importance

for the epidemiological picture. Two contextual factors such as distrust in the healthcare system due to complications of the HPV vaccination program in Kazakhstan in 2013 and trust in alternative medicine (alternative medicine strengthening the body's defenses) were considered for the final model. The model assumptions were checked. The goodness of fit of the model was checked with Hosmer–Lemeshow, Pulkstenis–Robinson Chi-square, and deviance tests, as well as the Lipsitz likelihood-ratio test.

#### *2.5. Ethical Considerations*

The study was conducted in compliance with the Declaration of Helsinki and was approved by the Institutional Research Ethics Committee of Nazarbayev University (NU IREC) on 23 April 2019 (IREC Number: 146/4042019). Before inclusion in the study, all potential participants were informed about the aims, methods, risks, and benefits of the study. Verbal consent was received from participants after an explanation of the voluntary and anonymous nature of the study.

#### **3. Results**

#### *3.1. Participants' Characteristics*

The social and demographic characteristics of women are shown in Table 1. The mean age of the study participants was 36.46 ± 11.18 years. More than 80% of women represented the Kazakh ethnicity. Most women (55%) had a university degree, which is in line with the official government statistics that reported the gross enrollment rate in higher education in Kazakhstan in 2020 to be 64.07%, of which 70.5% were women [32]. The majority were either married or were in a committed relationship (80%) and had one to three children (69%). There was almost an equal distribution of the place of residence among the participants, except for Aktobe city, where the number of participants was 29%. Almost half of women (48%) reported a family income lower than the national average level.


**Table 1.** Social and demographic characteristics of the study participants (N = 347).

**Table 1.** *Cont.*


\* Agency for Strategic Planning and Reforms of the Republic of Kazakhstan. Bureau of National Statistics.

#### *3.2. Contextual Factors*

Table 2 demonstrates to which extent women agreed with the statements about contextual factors. Most women (48%) were neutral about COVID-19 s effect on trust in the HPV vaccine and about the 2013 HPV program's effect on their trust in the healthcare system. More than half of women (56%) disagreed with the statement that vaccinating teenage girls against HPV encourages them to have sex. The majority of women (42%) were not in favor of the compulsory vaccines for children mandated by the government of Kazakhstan. The majority of women (44%) were neutral to the statement that everyone should be able to decide which vaccines are needed for their children. More than half of women (61%) agreed that alternative medicines strengthen the body's defenses, thus leading to a complete cure.

#### **Table 2.** Contextual factors, N = 233.


#### *3.3. Trust in Sources about Vaccination*

The least trusted information sources among women were alternative media, the government, and the pharmaceutical industry. The most trusted were scientific researchers, closely followed by medical workers (Figure 2).

Table 3 demonstrates the extent to which women trust the sources to tell the truth about the vaccine. The most trusted sources to tell the truth about the vaccine, associated with a positive attitude towards vaccination, were healthcare professionals (60%) and scientific researchers (61%). The high level of distrust of the pharmaceutical industry was associated with a positive attitude towards HPV vaccination (58%). The government was also not viewed as a trustworthy source and was associated with a positive attitude towards HPV vaccination (61%). Both alternative (WhatsApp Messenger, Instagram, Telegram, YouTube, etc.) and traditional media (TV, newspapers) had lower trust levels among the participants with both negative and positive attitudes towards HPV vaccination. All the sources were statistically significantly associated with attitude towards HPV vaccination.

**Table 3.** Association between attitude towards HPV vaccination and the trust to the sources, N = 218.



**Table 3.** *Cont.*

\* *p* value < 0.05, Chi-square test. \*\* *p* value < 0.05, Fisher's exact test.

#### *3.4. Attitude towards HPV Vaccination*

#### 3.4.1. Bivariable Analysis

Bivariable analysis between attitude towards HPV vaccination and patient characteristics and contextual factors is shown in Table 4. A positive attitude towards HPV vaccination was reported by 52% of the respondents, with the highest proportion in the youngest age group and the lowest in the oldest age group. There was no statistical difference in ethnicities regarding attitude towards HPV vaccination. The highest proportion of positive attitudes (58%) was observed among women with a university degree and the lowest (14%) among women with only high school or unfinished high school education. Negative attitude was the lowest among women with middle income (7%) and highest among women with low income (27%), while women with an upper level of income had the highest proportion of positive attitudes (62%). The majority of women without children (72%) had a positive attitude towards the vaccine, while only 16% of women with 4 and more children had the same attitude.

The Chi-square test showed that all contextual factors were statistically significantly associated with the level of attitude towards HPV vaccination. Among women who had no prejudice that HPV vaccination encourages teenage girls to have sex, 65% had a positive attitude. Among those who disagreed with compulsory vaccination, a majority (54%) had a positive attitude to the HPV vaccine, and among those who agreed with compulsory vaccination, a majority (58%) had a positive attitude to the HPV vaccine. Similarly, a majority of women who agreed (52%) or disagreed (71%) with freedom of vaccine choice for their children were women with a positive attitude towards the HPV vaccine. The majority of women (73%) who disagree with the statement that alternative medicine strengthens the body's defenses had a positive attitude towards the HPV vaccine.


**Table 4.** Attitude towards HPV vaccination among women including social and demographic characteristics and contextual factors, (N = 233).


**Table 4.** *Cont.*

\* *p* value < 0.05, Fisher's exact test. \*\* *p* value < 0.05, Chi-square test. \*\*\* OR adjusted for age, education, number of children, the 2013 HPV vaccination program in Kazakhstan, alternative medicine strengthening the body's defenses.

#### 3.4.2. Ordinal Logistic Regression

Women aged between 34 and 43 were 1.23 times more likely to have a positive (in relation to neutral or negative) attitude towards HPV vaccination in comparison to women aged between 19 and 27 years, adjusting for other variables (Table 4). Having a college or university degree increased the odds of a positive attitude by 1.81-fold and 2.38-fold, respectively, compared to incomplete/complete high school education, holding other variables constant. Women with 1–3 children or 4+ children were 0.45 and 0.26 times the odds of a positive attitude towards HPV vaccination in comparison with women with no children, adjusting for other factors.

Women who were neutral about their confidence in the healthcare system were 0.34 times the odds of a positive attitude towards HPV vaccination. Agreeing that alternative medicines strengthen the body's defenses or being neutral about the statement decreased the odds of a positive attitude by 0.41-fold and by 0.48-fold compared with women who disagreed with the statement, adjusting for other variables (Table 4).

#### **4. Discussion**

To date, there are no published studies investigating attitudes toward the HPV vaccine among Kazakhstani women. Thus, this is the first study that aimed to examine attitudes toward HPV vaccination and explore factors associated with a positive attitude toward HPV vaccination in Kazakhstani women. Since HR-HPV prevalence is high among women in Kazakhstan [13–15,21], and cervical cancer incidence has increased in the past decade [16], becoming the fourth leading cause of death from cancer among women [35], it is important to implement primary prevention of HPV infection and its related diseases in the country. However, the successful relaunching of the HPV vaccination program largely depends on HPV vaccine attitudes; therefore, studies investigating society's attitudes towards the vaccine are an essential part of facilitating the process.

Some historical contextual factors were taken into consideration in this study, such as previous HPV vaccination program complications/failure in Kazakhstan. The situation in Japan closely resembles the failure of the HPV vaccination program in Kazakhstan. Reports of side effects in mass media and prompt cancellation of the program had a long-lasting effect in Japan, causing the vaccine coverage to fall to less than 1% [36]. In Kazakhstan, the pushback intensified after media coverage of two 11–12-year-old girls having trouble breathing, hallucinations, and being hospitalized in the ICU immediately after administration of the HPV vaccine. As was later clarified, such a reaction resulted from inappropriate actions of medical personnel, who panicked at an allergic rash to the vaccine and administered a mixture of sedatives and painkiller medications. Nevertheless, the damage was done. The media quoted government officials and medical workers saying "73 percent of participants in [HPV] vaccine trials acquired new diseases—similar facts have been recorded around the world" and "Who knows what will happen to the reproductive function of [HPV] vaccinated girls in ten years?" [37].

Among our respondents, those who were affected by the failed 2013 HPV vaccination program had a 79% lower likelihood of having a positive attitude towards the vaccine, adjusting for other variables. Overall, 16% of women indicated that the failure of the 2013 HPV vaccination program in Kazakhstan had affected their confidence in the healthcare system, and the majority of them (40%) had a negative attitude towards the HPV vaccine. Thus, an unsuccessful attempt to implement the HPV vaccination program in Kazakhstan in 2013 had a significant impact. This finding is similar to the study reporting data from Romania and France, where the initial HPV vaccination program coverage reported a poor response and required specific action plans to improve the situation [38,39].

Rapid introduction of the COVID-19 vaccine is another factor considered in this study, as it has been reported previously that the level of COVID-19 hesitancy is high in Kazakhstan [40,41], and the media coverage of the vaccine is politicized worldwide [42,43], which might carry over to other domains in healthcare, such as the HPV vaccine [44].

The ongoing debates around the COVID-19 vaccine, however, did not have the same effect. Among women who disagreed with having less confidence in HPV vaccination recommendations since the controversy over the COVID-19 vaccination, 69% still had a positive level of attitude towards HPV vaccination. Overall, 14% of women indicated that COVID-19 controversies have affected their trust in HPV vaccination, and their distribution among the attitude groups was even. This finding falls in line with previous studies that show a lack of hesitancy overlap between HPV and COVID-19 vaccines [45].

Unfortunately, this study revealed a low trust of the respondents in the government (including the Ministry of Healthcare) as the main policymaker in Kazakhstan and in pharmaceutical companies. The majority of the respondents (44%) did not trust the government; only 13% did, the rest remaining neutral. This evidence shows that the Ministry of Healthcare and other governmental agencies in Kazakhstan should reinforce the information campaign in trustworthy ways. Similarly, participants had low levels of trust in alternative media and mostly low or neutral levels of trust in traditional media. At the same time, the fact that in our study, researchers and healthcare professionals are the most trusted source of information about the vaccine gives hope and identifies directions that the Ministry of Healthcare can employ to increase HPV vaccine awareness and improve attitudes. Engaging and strengthening the relevant healthcare workforce in preparation for HPV vaccine introduction could be achieved through training/seminars, information about the evidence and necessity of HPV vaccination and cervical cancer, and communication training to address parents' concerns and make recommendations. Our findings of trust are

comparable with studies in European countries (Sweden, Hungary, France, and the UK), where mistrust of health authorities was reported by 47% to 55% of the respondents [46]. On the other hand, the results of one Italian study showed high trust in doctors (85.3%) and teachers (90.4%) [47]. Similar to our finding, the study by Karafillakis et al. (2019) also revealed mistrust of pharmaceutical companies due to their underlying profit-making motives reported in many other European countries (Bulgaria, Romania, Sweden, Ireland, the Netherlands, Spain, and the UK) [46]. The results are also somewhat comparable to those in the USA, where levels of trust were generally high for healthcare providers and low for pharmaceutical companies. The trust for governmental organizations, however, showed mixed results in varying studies. Unlike in our study, the factor of ethnicity was significant for trust levels in the USA [48]. This could be explained by a more homogenous ethnical composition of Kazakhstan.

In this study, a positive attitude towards HPV vaccination was reported by 52% of the respondents. Education, place of residence, level of income, and number of children were the factors that were found to be significantly associated with the level of attitude towards HPV vaccination. Women with high income levels had the highest proportion of positive attitudes toward the HPV vaccine (62%). On the other hand, among women with high-school-level education and those who did not finish high school, 62% were neutral towards HPV vaccination. Thus, the level of education directly correlates with positive attitudes toward the HPV vaccine. Similar results were obtained in an Italian study, where 49.7% of the respondents had a positive attitude toward the HPV vaccine [47], and in a Romanian study, where 50.7% of women have a positive attitude toward the vaccine [38]. Both cited studies, and our study, confirmed that level of awareness and attitudes were directly linked to education [38,47]. This was also confirmed in the study by Özdemir et al. (2020), where positive attitudes towards HPV and the HPV vaccine increased in employed women and those who had high education and economic levels [49].

However, having such a huge proportion of neutral respondents in our study indicates the direction of further actions. Informational interventions to increase awareness of the HPV vaccine and the advantages of being vaccinated should be developed, as a positive attitude toward the HPV vaccine is associated with a higher level of intention to receive the HPV vaccine [26,50].

Unfortunately, only 16% of mothers with four or more children had a positive attitude towards the HPV vaccine, and the majority (64%) had a neutral attitude. This is an important group of the population, who requires more information on the vaccine, as their attitude towards the vaccine has a direct impact on the HPV vaccination program coverage among the target group of 9–11 years old girls. The situation could be improved if relevant and effective informational programs are employed to explain the advantages of the HPV vaccine in HPV infection and cancer prevention, as has been demonstrated by a study among parents in Japan [51].

As was confirmed by many studies, a significant reduction in the incidence of precancerous cervical lesions and cervical cancer was observed after the introduction of HPV vaccination programs [6,52–54]. Moreover, HPV vaccination is associated with a substantially reduced risk of invasive cervical cancer at the population level [52]. This knowledge, together with the knowledge of HPV as the most common STI, could improve the HPV infection spread [55]. However, very few studies have reported the association between knowledge of STI and sexual behavior and HPV status [56,57]. Thus, more research on HPV infection and HPV vaccine knowledge, behavior, and attitudes is required.

Study strengths and limitations. This was the first study to assess attitudes towards HPV vaccination among Kazakhstani women. Another strength of this study is covering the female population in the big cities of four Kazakhstani regions (central, southern, eastern, and western) and investigating the factors associated with attitudes towards HPV vaccination. However, since we approached only the population of big cities, but not these regions' rural areas, this study cannot fully represent the whole country's female population. Moreover, in this study, the proportions of participants with university-level

and college-level education were 55% and 34%, respectively. These indicators could be different if we include participants from rural areas. These are the major limitations that we hope to overcome in our future studies to obtain a more precise picture in terms of participants' residence and education. Another limitation of this study is its cross-sectional nature and resulting lack of causality. The study employed convenience sampling with self-reporting methods. This limits the representativeness of our findings due to selection and nonresponse biases. The descriptive nature of our statistical model, as well as the low number of respondents, prompts for expansion of the study beyond the pilot investigation. Moreover, in future studies, we should investigate parents'/mothers' attitudes towards HPV vaccination further, as they are the main decision makers if the vaccination of the target group is planned.

There are many issues related to cervical cancer prevention that require solving in Kazakhstan in the near future [18]. Although it requires attention and effort from healthcare policymakers, social media could help to improve the situation with overall societal awareness. Evidence from recent studies has indicated that social media has good potential to reach adolescents and young adults with information about HPV [58], and instead of creating a negative view, it could help to promote the HPV vaccination campaign.

#### **5. Conclusions**

This study shows contrary attitudes toward HPV vaccination exist among Kazakhstani women, with approximately half of women having positive and almost half having negative or neutral attitudes towards the vaccine. We found that women's age, education, number of children, confidence in the healthcare system, and belief in alternative medicine were associated with attitudes towards HPV vaccination. These factors, as well as high levels of trust towards medical and science workers, should be taken into consideration when planning context-specific health educational interventions to form positive attitudes towards HPV vaccination in Kazakhstani women. In addition, sharing accurate information regarding the safety, effectiveness, and benefits of HPV vaccination to prevent HPV infection and related diseases could potentially improve women's attitudes. HPV educational interventions are warranted to successfully relaunch and include HPV vaccination in the national immunization program in Kazakhstan.

**Supplementary Materials:** The following supporting information can be downloaded at: https: //www.mdpi.com/article/10.3390/vaccines10050824/s1, HPV vaccine questionnaire.

**Author Contributions:** Conceptualization, G.A.; data curation, A.B. and T.I.; formal analysis, T.I. and A.I.; funding acquisition, G.A.; methodology, G.A.; project administration, G.A.; software, A.I.; supervision, G.A.; validation, A.I.; visualization, A.B. and A.I.; writing—original draft, A.B. and T.I.; writing—review and editing, G.A. All authors have read and agreed to the published version of the manuscript.

**Funding:** This study was supported by the Faculty Development Research Grant Program 2019–2021 (Funder Project Reference: 110119FD4528, title: A molecular epidemiological study to determine the prevalence of oncogenic HPV strains for cervical cancer prevention in Kazakhstan). G.A. is the PI of the project.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and was approved by the Institutional Research Ethics Committee of Nazarbayev University (NU IREC) on 23 April, 2019 (IREC number: 146/4042019).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The study questionnaires are available per resonable request form the study PI via email gulzhanat.aimagambetova@nu.edu.kz.

**Acknowledgments:** The authors thank the Nazarbayev University School of Medicine for the continuous support that enabled the completion of this study. The authors would like to express gratitude to Brett Craig, WHO, Regional Office for Europe, for comments that greatly improved the manuscript. **Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Article* **Immunity after HPV Vaccination in Patients after Sexual Initiation**

**Dominik Pruski 1,2,\*, Małgorzata Łagiedo-Zelazowska ˙ 3,4, Sonja Millert-Kali ´nska 1,4, Jan Sikora 3, Robert Jach <sup>5</sup> and Marcin Przybylski 1,2**


**Abstract:** Vaccinations against human papillomavirus (HPV) are included in the primary prevention of precancerous intraepithelial lesions and HPV-related cancers. Despite the undeniable effectiveness of vaccination in the juvenile population, there is still little research on the effect in patients after sexual initiation. Our study aims to assess anti-HPV (L1 HPV) antibodies in healthy patients and diagnosed cervical pathology after 9-valent vaccination. We provide a prospective, ongoing 12-month, non-randomised pilot study in which 89 subjects were enrolled. We used an enzyme-linked immunosorbent assay to determine IgG class antibodies to HPV. We noted significantly higher levels of antibodies in vaccinated individuals than in the unvaccinated control group. The above work shows that vaccination against HPV might be beneficial in patients after sexual initiation as well as in those already diagnosed with HPV or SIL infection.

**Keywords:** HPV serum antibodies; L1 HPV; 9-valent vaccination; squamous intraepithelial neoplasia

#### **1. Introduction**

Squamous intraepithelial lesion (SIL) and cervical cancer are among the most common oncological diagnoses in women globally; therefore, they constitute a significant health problem. Cervical cancer remains the fourth most frequent cancer in women worldwide [1] unless it is theoretically preventable. The most critical risk factor for the development of cervical cancer is a persistent infection caused by highly oncogenic types of human papillomavirus (HPV). Neoplastic transformation begins with integrating HPV DNA into the genome of a typical epithelial cell. This situation may occur when the circular form of HPV DNA breaks, and then chromatin shifts within the chromosomal DNA of the host cells. Vaccination against HPV prevents infections with specific HPV types and, consequently, cervical cancer development due to infection with a given type [2–4]. Generally, during the human immune response to HPV, B cells detect the viral antigens and exhibit them to T helper type 2 cells, promoting the production of high-affinity antibodies (IgG, IgA, and IgM) against HPV antigens by B cells. It has already been demonstrated that anti-HPV IgG might be a reliable marker for past HPV exposure [5]. Studies have shown that the median seroconversion time was about 8.3–11.8 months. These data suggest that the development of IgG antibodies at a detectable level after a natural infection can be a slow process, and it does not necessarily occur in every woman. Following a human papillomavirus (HPV) vaccine, type 16 virus-like particles (VLPs), according to the authors, appear within 8.3 months and remain for approximately 36 months [6]. Antibodies could

**Citation:** Pruski, D.; Łagiedo-Zelazowska, M.; ˙ Millert-Kali ´nska, S.; Sikora, J.; Jach, R.; Przybylski, M. Immunity after HPV Vaccination in Patients after Sexual Initiation. *Vaccines* **2022**, *10*, 728. https://doi.org/10.3390/ vaccines10050728

Academic Editor: Gloria Calagna

Received: 28 March 2022 Accepted: 4 May 2022 Published: 6 May 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

persist for a long period of time if the initial antibody levels were high or if there was continued antigenic exposure. At the same time, IgM may be detected in acute or current exposure, typically after one month following initial immunisation, as Harro et al. claim [7]. Vaccinations against HPV are included in the primary prevention of precancerous lesions mainly SIL and cervical cancer. The other cancers associated with HPV infections affect the genital organs (vulva, vagina, and penis), anal canal, oral cavity, and upper respiratory tract [8,9]. Vaccination against HPV significantly reduced the incidence of HPV-related lesions in New Zealand and the United States [10,11]. In the European countries, and thus in Poland, vaccination against HPV has been introduced into the vaccination calendar. Local governments organise vaccination programmes in many provinces of our country. It is recommended that both girls and boys are vaccinated before sexual initiation. After identifying an HPV infection, many patients decide to vaccinate after sexual initiation due to the fear of developing intraepithelial neoplasia of the cervix, vagina, vulva, or HPVdependent changes in the respiratory tract. After the treatment of HPV-related lesions, such as intraepithelial neoplasia of the cervix or genital warts, some patients decide to vaccinate to develop anti-HPV antibodies that can protect against re-infection and the formation of HPV-related lesions. However, there is still very little research into post-vaccination antibody levels (VLP), so it seems to us that this is a topic worth exploring [12–14].

The 9-valent vaccine contains the purified proteins of nine types of HPV, namely 6, 11, 16, 18, 31, 22, 45, 52, and 58. The vaccine is usually administered according to a three-dose schedule. Studies focusing on the presence of HPV genotypes in large populations may contribute to the development of further protective vaccinations [15].

Considering this, we aim to assess the level of anti-HPV (L1 HPV) antibodies in healthy patients and with diagnosed cervical pathology after vaccination. The introduction of tests for the detection of anti-HPV (L1 HPV) antibodies may, in the future, facilitate the assessment of the effectiveness of vaccine programmes. Moreover, it might be helpful in the identification of patients with immune disorders in whom infection with oncogenic types of HPV persisted, resulting in intraepithelial neoplasia. Analysing specific types of immune disorders will facilitate the identification of groups of women with the highest risk of developing high-grade squamous intraepithelial lesions and, consequently, malignancy.

The following meta-analysis compares the effectiveness of the vaccine administration in the population of patients before and after sexual initiation in either healthy individuals or those with diagnosed cervical pathology.

#### **2. Materials and Methods**

#### *2.1. Study Design*

We provide a prospective, ongoing 12-month, non-randomised pilot study to assess the level of anti-HPV (L1 HPV) antibodies in healthy patients and those with diagnosed cervix pathology after the 9-valent HPV vaccine. The Bioethical Committee of the Poznan University of Medical Sciences, Poland, approved the study protocol (597/19). We obtained written consent for the study from all patients. We included patients who met the following criteria: (i) only adult women, (ii) non-pregnant subjects, postpartum, (iii) patients not treated with immunosuppressive drugs, (iv) not previously vaccinated with other HPV vaccines, (v) expressing informed and written consent to participate in the study, (vi) agreeing to the proposed surgical diagnostics in the case of indications and possible surgical treatment, (vii) had taken three doses of the 9-valent vaccination against HPV according to the 0–2–6 months scheme, and (viii) provided blood samples after at least six months from the last dose of vaccination. The exclusion criteria were: (i) refusal of possible treatment of squamous intraepithelial lesions, and (ii) failure to complete the full vaccination schedule. A total of 61 women met the above criteria.

All subjects from the study group were undergoing a verification diagnostic of abnormal Pap-smear results by punch biopsy. We examined the status of HPV infection and looked for the presence of pre-neoplastic lesions, such as low-grade squamous intraepithelial lesions (LSIL) or high-grade squamous intraepithelial lesions (HSIL). All patients with

histopathologically confirmed HSIL (CIN 2, CIN 3) were treated with the LEEP conization and curettage of the cervical canal.

The control group 1 covers 20 healthy, unvaccinated patients, in whom we excluded an infection with hrHPV or squamous intraepithelial lesions confirmed through punch biopsy. Control group 2 includes eight subjects both infected with highly oncogenic types of HPV and diagnosed with pre-neoplastic lesions who decided not to receive the HPV vaccine. Figure 1 presents the process of recruiting patients for the study, and Table 1 shows the basic division into study groups.

**Figure 1.** Flow chart. SIL—squamous intraepithelial lesion, NILM—negative for intraepithelial lesion or malignancy, HPV—human papillomavirus.

**Table 1.** Basic characteristics of study and control groups.


HPV—human papillomavirus, *n*—number.

All examination and follow-up groups are under regular oncogynaecological care. Patients diagnosed with HSIL (CIN 2, CIN 3) underwent proper treatment—the removal of the lesions according to the current recommendations of the Polish Colposcopic Society and then subjected to close cytological and molecular control every six months.

#### *2.2. Specimen Collection and Handling*

Blood was drawn aseptically to the serum collection tubes (S-Monovette). The blood was collected at least six months after receiving the last vaccination dose. After that, the samples were centrifuged at 2000 rpm for 20 min. Supernatants (sera) were collected and frozen at −20 ◦C for further assays.

#### *2.3. HPV Serological Measurements*

We used an enzyme-linked immunosorbent assay to determine IgG class antibodies to human papillomavirus (Creative Diagnostics, New York, USA). The sera were diluted 1:101 into properly defined dilution tubes for the test. An ELISA microtiter plate was coated with recombinant VLP derived from HPV types 6, 11, 16, and 18. After incubation and washing, we added the diluted samples and quality control specimens to the microtiter plates along with a peroxidase-conjugated anti-human polyclonal antibody. Following incubation and washing, an enzyme substrate and chromogen were added to allow colour development. Reactions were stopped, and optical density (OD) was read at 450 and 620 nm, with the background measured at 620 nm and subtracted from the OD reading at 450 nm. A calculated formulation from the manufacturer determined the seropositive cut points. The cut points were set at 0.303 for HPV seronegative and >0.303 for HPV seropositive patients. We calculated the quantitative results of the assay as instructed in the kit insert (OD/CUT-OFF).

#### *2.4. Statistical Analysis*

We performed an analysis in SPSS, version 27. All tests were two-tailed, with α = 0.05. The normality of the variables was validated based on the Shapiro–Wilk test. All three groups were characterised by reporting median with quartiles 1 and 3, or mean and standard deviation for quantitative variables, or *n* value and percentage for qualitative variables. The values of variables with normal distributions were compared between the experimental and control group 1 or 2 with the Student's *t* test. Variables without normal distributions were compared with the Mann–Whitney U test. Dependencies between the group and other variables were measured using Fisher's exact test. Odds ratios or median differences (experimental group–control group) with 95% confidence intervals were given when the results of the analyses were significant. Median differences were calculated using the Hodges–Lehmann estimator. The correlation was calculated with Pearson's r coefficient.

#### **3. Results**

As shown in Table 1, the experimental and control groups did not differ significantly in age or regarding pregnancies. Comorbidities were observed in 38% of women from the experimental group, 30% from control group 1, and 12.5% of women from control group 2. The dependency between the group and Pap-smear results was insignificant for the experimental vs. control group 2. The most common Pap-smear result in the experimental and control group 2 was LSIL, which accounted for 33% and 38%, respectively. A more significant proportion of women with positive HPV tests was found in the experimental group than in control group 1—nine times more. There was no considerable dependency between the groups and HPV test results in the case of the experimental group vs. control group 2. All women from control group 1 were histopathologically confirmed to have no pathology (NILM). An NILM result was observed in 19% of women in the experimental group, which is statistically significant. We did not find any dependency between the experimental/control group 2 and biopsy results. The most common histopathological result for women from the experimental group and control group 2 was HSIL (57% of women from both groups).

Figures 2 and 3 show the graphical arrangement of the levels of antibodies in individual research groups. Antibody levels were significantly higher in the experimental group than in both control group 1 and control group 2. The antibody level divided by the cut-off value (0.303) was also significantly higher in the experimental group than both of the control groups. There were significant dependences between group and sample being reactive (*p* < 0.001 for both analysis—experimental group vs. control group 1 and

experimental group vs. control group 2). The sample was reactive for all women from the experimental group, 16% of women from control group 1, and one-fourth of women from control group 2, as presented in Table 2.

**Figure 2.** Antibody level.

**Figure 3.** Antibody level divided by the cut-off value.


#### **Table 2.** Detailed group characteristics.

NILM—negative for intraepithelial lesion or malignancy; ASCUS—atypical squamous cells of undetermined significance; ASC-H—atypical squamous cells cannot exclude HSIL; LSIL—low-grade squamous intraepithelial lesion; HSIL—high-grade squamous intraepithelial lesion; AGC—atypical glandular cells; Q1—first quartile; Q3—third quartile; *n*—number, *p*1—*p*-value for comparison between experimental group and control group 1; *p*2—*p*-value for comparison between the experimental group and control group 2. Comparisons were made with Student's *t* test <sup>1</sup> or Mann–Whitney U test <sup>2</sup> for quantitative variables and Fisher's exact test for qualitative variables.

Age was not significantly correlated with the antibody level, as seen in Table 3. The patients with LSIL diagnosis and those with HSIL or cancer diagnosis did not vary considerably in terms of antibody level and antibody level divided by cut-off value, as shown in Table 4.

**Table 3.** Correlation between age and antibody level.


r—Pearson's r correlation coefficient. *p*—*p*-value.

**Table 4.** Comparison of antibody level and antibody level/cut-off between groups with different diagnoses.


*p*—*p*-value; LSIL—low-grade squamous intraepithelial lesion; HSIL—high-grade squamous intraepithelial lesion; *n*—number.

Samples were reactive for 93% of women who received an LSIL diagnosis and 87% of women who received either an HSIL or cancer diagnosis.

Comparison made with Student's *t* test.

#### **4. Discussion**

Our study aimed to assess the level of anti-HPV antibodies in patients with diagnosed cervical pathology and in healthy patients after vaccination. Our work supports the practice of vaccinating HPV-infected patients after sexual initiation by showing the level of antibodies persisting after vaccination. Despite the proven and indisputable effectiveness of the 9-valent HPV vaccine as primary prevention in juveniles before sexual initiation, its efficacy has not yet been demonstrated in women with diagnosed cervical pathology.

As expected, in all patients vaccinated with the 9-valent vaccine, the samples turned out to be reactive. Additionally, the analysis confirmed a relationship between levels of antibodies and vaccination status. We noted significantly higher levels in vaccinated patients than in those of the unvaccinated control groups: 1.77 vs. 0.09 and 0.13, respectively. These results are consistent with the work published by Mirte Scherpenisse et al. Naturally induced HPV-specific antibodies from single-positive sera were genotype-specific and neutralising.

In contrast, the antibodies of multi-positive sera were less genotype-specific, crossreactive, and tended to be non-neutralising. Post-vaccination antibody avidity was approximately three times higher than after HPV infection [16]. Post-vaccination antibody status assessment may help analyse the effectiveness of HPV preventative vaccination programmes. Vaccine efficacy against HPV16 and 18 infections were sustained over eight years post-vaccination [17]. HPV-specific IgG antibody levels and its neutralising activity remained well above the antibody levels induced by HPV infection [17,18]. Additionally, HPV vaccines offer cross-protection against several non-vaccine HPV types in patients without a previous HPV infection [19]. Antibodies that were also capable of neutralising non-vaccine HPV types were most frequently found to be directed against HPV31 and 45. Cross-neutralising antibody levels against HPV31, 33, 35, and 45 were significantly associated with their phylogenetically related vaccine-type antibody levels [20]. HPV genotypes

frequently detected in cervical cancer are as follows: 31, 33, 45, and efficacious vaccines against these HPV types might further reduce malignancies. However, vaccine efficacy against non-vaccine HPV types decreased rapidly over time [19].

Interestingly, our results indicate that a current or persistent infection with human papillomavirus gives a lower antibody level percentage than in vaccinated patients, which is consistent with the reports presented by other researchers. In our study group, 18% of women infected with HPV had clinically significant levels of L1-HPV antibodies, and for comparison, we observed the antibodies in 100% of vaccinated women. Investigators argue that the rate of seroconversion associated with the vaccines is high, namely, >99% in women and men [21–24]. In contrast, the seroconversion results from natural infection are an estimated 50–70% in women [25] and men [26].

Although most of the patients in the control groups had antibody levels below the cut-off, we observed 3/20 reactive samples in control group 1 and 2/8 in control group 2. It is worth noting that only one woman with a reactive sample had no burden. In other cases, we observed the presence of infections with HR HPV and histopathologically confirmed HSIL or a history of Hashimoto's disease. This observation may provide new insight into the factors modulating the immune system. It is possible that comorbidities, dysfunctions of the immune system, or infection with HPV genotype 16 strongly stimulate the immune system to produce antibodies. Data provided by Aubin et al. suggest that autoimmune inflammatory diseases (AIID) and the drugs used to treat them are associated with an excess risk of genital HPV infection. Although this excess risk has not been specifically evaluated, the available data indicate a need for close monitoring of patients with AIID, regardless of their treatment, to ensure the prevention and treatment of benign and premalignant lesions [27].

Petter et al. [28] indicated that serological assays for HPV could help identify patients at risk of HPV-related cancers. In addition to strategies connected with antibody detection, DNA sequencing or the PCR method are also widely used to detect the viral DNA of HPV in tissue samples [29]. Therefore, antibody- and DNA- based assays can complement each other for the reliable identification of HPV-infected patients.

Researchers from Mexico presented somewhat similar work. Their study aimed to assess type-specific cervical HPV prevalence and their association with HPV-specific antibodies in a cohort of female university students. The observed study group was similar in terms of number. HPV genotyping was performed by amplifying and sequencing a fragment of the L1 protein. In addition to sexual behaviour, it was observed that the presence of serum-specific IgG antibodies against HPV can impact the prevalence of the virus. Alexander Pedroza-Gonzalez et al. suggest that seropositivity to HPV-16 and HPV-18 was associated with a lower prevalence of HPV-16, but not for other HPV types. Of note, there was a lower proportion of HPV-specific seropositivity in women who had the presence of the same HPV type in a cervical specimen, suggesting an immunoregulatory mechanism associated with the viral infection [30].

Efforts towards the detection of HPV antibodies as a tool to monitor and assess vaccine efficacy have increased significantly in recent years. In the study conducted by Bhatia et al., a standardised ELISA test developed for anti-HPV16L1 antibodies was validated against the WHO's international positive serum standard for HPV16. This assay was amenable to both venous blood and dried blood spots. The researchers also admit that the sample size used for the study was small; however, the presented technique has promise for widespread use in epidemiological and field studies of antibody prevalence and, coupled with the avidity measurement, may be of use in individual cases for monitoring vaccine responses such as failures [14].

A relatively small research group limited our methodological choices. However, in the future, we will be able to expand the group and test the level of antibodies over the next few years to assess the trend of changes. Fortunately, we observe an increasing awareness of patients and their partners and a growing number of vaccinations against HPV in both adults and those at the pre-contraceptive age.

#### **5. Conclusions**

The results of our study may indicate that high levels of antibodies are maintained after HPV vaccination. This further suggests that vaccinations are also effective in subjects after sexual initiation. These conclusions might help identify patients with immune disorders who have survived the infection with HR HPV, resulting in changes in the intraepithelial neoplasia. Analysing specific immune disorders might help identify groups of women with the highest risk of developing HSIL (CIN 2, CIN 3) lesions and, consequently, malignancies.

**Author Contributions:** Conceptualisation, D.P. and M.P.; methodology, D.P. and M.Ł.-Z.; software, ˙ D.P. and S.M.-K.; validation, D.P. and M.Ł.-Z.; formal analysis, D.P. and S.M.-K.; investigation, D.P. ˙ and M.P.; resources, D.P., M.Ł.-Z. and J.S.; data curation, D.P. and M.P.; writing—original draft ˙ preparation, D.P., S.M.-K. and M.Ł.-Z.; writing—review and editing, R.J., J.S. and D.P.; visualisation, ˙ D.P. and S.M.-K.; supervision, R.J. and J.S.; project administration, D.P. and M.P.; funding acquisition, D.P. and M.P. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee of the Poznan University of Medical Sciences, Poland (95/2021).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Adherence to the Recommended HPV Vaccine Dosing Schedule among Adolescents Aged 13 to 17 Years: Findings from the National Immunization Survey-Teen, 2019–2020**

**Chinenye Lynette Ejezie 1,2,\*,†, Ikponmwosa Osaghae 3,4,†, Sylvia Ayieko <sup>1</sup> and Paula Cuccaro <sup>1</sup>**


**Abstract:** The 9-valent human papillomavirus (9-vHPV) vaccine uptake rate among adolescents has improved over the years; however, little is known about the adherence to the recommended dosing schedule. This study examines the prevalence and factors associated with adherence to the recommended 9vHPV vaccination dosing schedule among adolescents aged 13 to 17 years. The crosssectional study was conducted using the 2019–2020 National Immunization Survey-Teen. The parents of 34,619 adolescents were included in our analyses. The overall up-to-date (UTD) prevalence was 57.1%. The UTD prevalence was 60.0% among females and 54.2% among males. Adolescents aged 16 years had the highest UTD prevalence of 63.0%. The UTD prevalence was 61.6% among Hispanics and 54.7% among non-Hispanic Whites. Overall, compared to females, males had 14% lower odds of UTD. The odds of UTD were 1.91 times, 2.08 times, and 1.98 times higher among adolescents aged 15–17 years, respectively, compared to those aged 13 years. Moreover, region, poverty, insurance status, mothers' educational level, and provider recommendation were associated with UTD. Our findings show that adherence to the recommended 9vHPV vaccine schedule is low in the US. Targeted public health efforts are needed to improve the rates of adherence to the recommended 9vHPV dose schedule.

**Keywords:** HPV vaccination; dosing schedule; dosing interval; 9-valent; low adherence

#### **1. Introduction**

In the United States, the Advisory Committee on Immunization Practices (ACIP) and the Centers for Disease Control and Prevention (CDC) recommend that adolescents prophylactically receive the 9-valent human papillomavirus (HPV) vaccine (9vHPV) [1–4]. Toward the end of 2016, the 9vHPV vaccine became the only HPV vaccine available in the US and the administration of other types of HPV vaccines were permanently discontinued [5]. Clinical trials have shown that 9vHPV is about 90% effective in preventing HPV-related cancers when administered on the recommended dosing schedule [6–9]. The dose schedule of 9vHPV differs across age categories: the ACIP and CDC recommends two doses administered at a schedule of 5 to 12 months for persons younger than 15 years [10,11]. The CDC recommends a third dose for people younger than 15 years who receive the second dose less than 5 months following the first dose [10,11]. Additionally, the CDC recommends three doses (second dose administered at a schedule of 1 to 2 months after the first dose;

**Citation:** Ejezie, C.L.; Osaghae, I.; Ayieko, S.; Cuccaro, P. Adherence to the Recommended HPV Vaccine Dosing Schedule among Adolescents Aged 13 to 17 Years: Findings from the National Immunization Survey-Teen, 2019–2020. *Vaccines* **2022**, *10*, 577. https://doi.org/ 10.3390/vaccines10040577

Academic Editor: Gloria Calagna

Received: 24 March 2022 Accepted: 6 April 2022 Published: 8 April 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

and the third dose administered 6 months after the first dose) for persons who initiate the vaccine series on or after their 15th birthday, as well as those with a compromised immune system [11].

Despite the recommendation by the ACIP and CDC, the number of adolescents adhering to the recommended dosing schedule remains relatively low. Adherence to the recommended dosing schedule is crucial to optimize immune response to the HPV vaccine [12,13]. According to the CDC, only about half of adolescents aged 13 to 17 years adhere to the recommended HPV vaccine dosing schedule [1]. A possible contributing factor of low adherence to the recommended dosing schedule is little or no emphasis on the dosing schedule in reporting the vaccine series as complete [14–16]. Specifically, the vaccine is often categorized as complete based on the uptake of the required number of doses (two or three doses depending on age category) without considering the interval between doses [14–16]. For example, in Munn et al.'s study conducted to examine the uptake and completion of HPV vaccine series among adolescent users and nonusers of school-based health centers, completion was defined as the uptake of the required number of doses per age category, with no information about the interval between doses [15]. Similarly, another study conducted by Simons et al. to estimate and examine predictors of HPV vaccine series completion defined completion based on the receipt of three doses of HPV vaccine within one year without considering adherence to the recommended dosing schedule [14]. Given that adherence to the recommended number of doses and interval between doses is necessary for efficacy [17,18], considerations should be made to note both requirements when categorizing HPV vaccination as complete.

Observational studies have identified several factors that influence the uptake of the HPV vaccine among adolescents [13–15]; however, the factors that influence adherence to the recommended dosing schedule remain unexplored. It is plausible that the factors associated with HPV vaccine uptake are also associated with adherence to the recommended HPV dosing schedule. For example, sociodemographic characteristics, including age, race/ethnicity, income level, insurance status, census region, and sex; and factors such as provider recommendation, parental hesitancy, structural barriers, and knowledge influence HPV vaccine uptake [13–16,19] and, therefore, may affect adherence to the recommended dosing schedule.

Receiving two or three doses of 9vHPV is not enough to offer the expected protection from HPV-related diseases [10,18]. Complying with the recommended interval between doses is also essential to ensure HPV vaccine efficacy [10,17,18]. Since adherence to the recommended dosing schedule seems necessary for protection from HPV-related infections [10,17,18], the efficacy of the 9vHPV vaccine may be reduced when the intervals between doses are not according to the recommended schedule. Adverse effects associated with not receiving the HPV vaccine, such as risk of HPV-related cancers [10], may be associated with non-adherence to the recommended HPV vaccine dosing schedule. Although previous research has shown that HPV vaccine uptake among adolescents aged 13 to 17 years has improved over the years, limited research on HPV vaccine completion has examined the adherence to the recommended dosing schedule. Thus, the goal of this research is to examine the prevalence and factors associated with adherence to the recommended HPV vaccination dosing schedule among adolescents aged 13 to 17 years. Findings could help researchers and policy makers to improve adherence to the recommended dosing schedule by highlighting the importance of dosing intervals when creating programs aimed at improving HPV vaccination.

#### **2. Materials and Methods**

Study Design, Setting, and Participants

In this secondary analysis, we utilized the 2019–2020 National Immunization Survey Teen (NIS-Teen) conducted by the CDC [20]. The NIS-Teen is conducted annually with samples of parents or caregivers of adolescents who are aged 13 to 17 years and reside in the US. The sampling frame includes a representative sample of eligible parents with landlines or cell phones. The survey consists of provider-verified data on vaccines from adolescents aged 13 to 17 years in the 50 states, the District of Columbia, and territories.

The survey occurs in two phases: in phase 1, parents or caregivers are contacted via telephone to provide information pertaining to their adolescents' sociodemographic characteristics, contact information, and vaccination history. Additionally, parents or caregivers are asked to provide consent for the adolescents' healthcare provider to be contacted. In phase 2, healthcare providers are contacted through a mailed survey to verify the accuracy of the information obtained from parents or caregivers. Detailed information about the methods for the NIS-Teen study are available elsewhere [20].

#### **3. Measures**

#### *3.1. Dependent Variable*

#### HPV Vaccine Uptake

The dependent variable assessed whether an adolescent adhered to the required dosing schedule when completing the HPV vaccine series. Specifically, the variable assessed whether adolescents were HPV vaccine up-to-date (UTD) in line with the required dosing and interval between HPV vaccine shots. This variable was first included in the NIS-teen survey in 2016. UTD was defined as 3+ human papillomavirus shots (9V, 4 V, UV, CV, or HP) or 2+ human papillomavirus shots, with the first shot received before age 15 and an interval between first and second shots of at least 5 months (minus 4 days), excluding any vaccinations after the random digit dialing interview date [20]. This variable was binary, i.e., "UTD" versus "Not UTD".

#### *3.2. Independent Variables*

#### Provider Recommendation

To assess provider recommendation, respondents were asked, "Had or has a doctor or other health care professional ever recommended that teen receive HPV shots?" Responses were either "Yes", "No", "Don't Know", or "Refused". This variable was operationalized as binary, with only responses "Yes" or "No" retained for our analysis.

#### *3.3. Sociodemographic Characteristics*

Sociodemographic characteristics assessed based on previous literature were adolescent's age (categorical variable 13–17 years), gender (male or female), race/ethnicity (non-Hispanic White, non-Hispanic Black, Hispanic, and non-Hispanic Other), region (West, Midwest, Northeast, and South), and insurance status (any Medicaid, private insurance only, other insurance, and uninsured). Other sociodemographic characteristics assessed were poverty status, defined as percentage of poverty line (categorized as above poverty ≤USD 75 k, above poverty >USD 75 k, and below poverty), and mother's education status (categorized as college graduate, some college, high school only, and less than high school).

#### *3.4. Statistical Analysis*

All analyses were conducted using Stata/IC V.15.1. The analyses accounted for the complex study design and survey sampling weights used in the NIS-Teens survey. The inclusion criterion was the presence of adequate provider data for UTD and not UTD adolescents. We excluded all "Don't Know" and "Refused" responses for provider recommendation, and "Unknown" responses for poverty status. A complete case analysis was conducted; as such, respondents with missing data for provider recommendation (1.79%) and region (0.84%) were excluded from the analysis. Weighted percentages were reported and are, therefore, representative of the general population. Descriptive statistics were presented in the overall population and in populations stratified by gender using simple proportions and chi-square test. Furthermore, we presented the weighted prevalence of UTD by sociodemographic characteristics and provider recommendations in the overall population and stratified by gender. Bivariable and multivariable logistic regression analyses were used to estimate the association between sociodemographic characteristics

and provider recommendation with HPV UTD among adolescents aged 13–17 years in our overall study sample and separately among females and males. Each model was adjusted for sociodemographic characteristics, provider recommendation, and survey year. Statistical significance was defined as a two-sided *p*-value < 0.05 for all comparisons.

#### **4. Results**

#### *4.1. Overall Population*

A total of 34,619 adolescents were included in our final analyses. Among those who were UTD in the overall population, most were females (51.8%), 16 years old (22.0%), non-Hispanic White (50.7%), resided in the southern region (36.5%), had private insurance only (55.0%), had income above poverty >USD 75 k (51.0%), had mothers who were college graduates (45.8%), and had received a provider recommendation (88.9%) (Table 1).

The overall UTD prevalence among all adolescents was 57.1% (95% CI: 56.0–58.1%). In the overall population, the UTD prevalence was 60.0% among females and 54.2% among males. Adolescents aged 16 years had the highest UTD prevalence of 63.0%. The UTD prevalence was also highest (61.6%) among Hispanics and highest (63.9%) among adolescents residing in the Northeast region. Moreover, adolescents who had any Medicaid had the highest (60.7%) UTD prevalence, while adolescents below the poverty line had the highest (60.8%) UTD prevalence. The UTD prevalence was highest (61.7%) among adolescents with mothers having less than high school education and highest (62.8%) among adolescents who received a provider recommendation. Additionally, the UTD prevalence was 54.6% in 2019 and 59.5% in 2020 (Table 2).

In the overall population, results of multivariable regression analysis showed that, compared to uninsured adolescents, adolescents who had any Medicaid had over twofold higher adjusted odds of UTD (AOR: 2.13; 95% CI: 1.64–2.76). Moreover, compared to adolescents below the poverty line, those living above poverty <= USD 75 k had 22% lower adjusted odds of UTD (AOR: 0.78; 95% CI: 0.66–0.92). Adolescents residing in the Midwest and Northeast regions had 1.19- and 1.49-times higher odds of UTD, respectively, compared to those residing in the Southern region (AOR: 1.19; 95% CI: 1.08–1.31, AOR: 1.49; 95% CI:1.33–1.66, respectively). Additionally, adolescents who received a provider recommendation had about twofold higher adjusted odds of UTD compared to those who did not (AOR: 1.85; 95% CI: 1.74–1.97). We also found 22% higher odds of UTD in 2020 compared to 2019 (Table 2).



**Table 1.** *Cont.*


humanpapillomavirus shots, age (minus days), excluding any random digit dialing interview date.


**Table 2.** Prevalence and association between sociodemographic characteristics and provider recommendation among teens aged 13–17 years and UTD in the overallsample, National Immunization Survey—Teen (2019–2020).





after the random digit dialing interview date. a Model adjusted for

sociodemographic

characteristics,

 provider

recommendation,

 and survey year.

#### *4.2. Female Adolescents*

Following stratification by gender, among females, the UTD prevalence was 60.0% (95% CI: 58.5–61.5%). Moreover, UTD prevalence was 63.2% among female adolescents on any Medicaid and 44.8% among uninsured females. Female adolescents below the poverty line had a UTD prevalence of 62.9%, while those above the poverty line ≤USD 75 k had a UTD prevalence of 56.4%. Female adolescents who received a recommendation from a provider had a UTD prevalence of 64.8%, while those who received no provider recommendation had a UTD prevalence of 34.2% (Table 3).

Among females, results of multivariable regression analysis showed that, compared to female adolescents that are uninsured, those with any Medicaid, private insurance only, and other insurance had 124%, 68%, and 62% higher adjusted odds of UTD, respectively. Moreover, females who had received a provider recommendation had 92% (AOR: 1.92; 95% CI: 1.75–2.11) higher adjusted odds of UTD compared to those who received no recommendation (Table 3).

#### *4.3. Male Adolescents*

Following stratification by gender, among males, the UTD prevalence was 54.2% (95% CI: 52.7–55.7%). Male adolescents who had any Medicaid insurance had the highest (58.3%) UTD prevalence, while those who were uninsured had the lowest (39.6%) UTD prevalence. Male adolescents below the poverty line had a UTD prevalence of 58.8%, while those above the poverty line >USD 75 k had a UTD prevalence of 54.0%. Male adolescents who received a recommendation from a provider had a UTD prevalence of 60.8%, while those who received no provider recommendation had a UTD prevalence of 31.9% (Table 3).

Furthermore, among male adolescents, results of multivariable logistic regression analysis were mostly similar but slightly attenuated compared to what was seen among female adolescents. Male adolescents with any Medicaid had 102% higher adjusted odds of being UTD compared to those who were uninsured. In terms of poverty status, compared to male adolescents below the poverty line, those living above poverty ≤USD 75 k had 27% lower adjusted odds of UTD (AOR: 0.73; 95% CI: 0.58–0.92) (Table 3).



by




after the random digit dialing interview date. a Model adjusted for

sociodemographic

characteristics,

 provider

recommendation,

 and survey year.

65

#### **5. Discussion**

To our knowledge, our study is among the first to employ the 2019–2020 NIS-Teen dataset to examine the prevalence and factors associated with the adherence to the ACIP recommended 9vHPV vaccine schedule among teenagers aged 13 to 17 years. Nationally, only about half of females and males completed the vaccine series with adherence to the recommended schedule. Our finding is consistent with previous research, which shows that only 58.6% of teenagers adhered to the recommended dosing schedule [1]. Without adherence to the recommended dosing schedule, immune response imparting expected protection from the HPV vaccine is uncertain [18]. It is possible that expected immune protection might be reduced or absent outside of the recommended vaccination schedule [9].

Our low adherence finding could be related to a number of possible reasons, including providers not scheduling follow-up doses at the time of the initial dose, or few or no reminders for follow-up dose scheduling. Since lack of knowledge is a factor associated with low HPV vaccine uptake [21–23], it could also be a reason for low adherence to the recommended vaccination schedule. Based on previous studies, knowledge is an important predictor of HPV vaccine uptake, and improvement in knowledge results in improvements in HPV vaccine behaviors [21–23]. Providing parents with information about the importance of adhering to the appropriate dosing interval may encourage parents to pay attention to the time points when vaccinating their adolescents. Moreover, creating intervention programs aimed at increasing knowledge on adherence to the recommended dosing may improve adherence to the recommended vaccination schedule.

In our provider recommendation analysis, we found that provider recommendation was consistently associated with HPV vaccine completion with adherence to the recommended dosing schedule. Our results are in agreement with other studies that have shown that parents are more likely to vaccinate their adolescents when they receive a recommendation from a healthcare provider [24]. How providers introduce and recommend vaccines is robustly associated with vaccine behaviors [25,26]. Specifically, "strong" provider recommendation (which encompasses the recommendation of same-day vaccination, emphasizing the completion of the vaccine series, and specifying the recommended dosing schedule) is associated with nine times the odds of HPV vaccine uptake compared to a weak recommendation [26]. Moreover, providing parents with information regarding the differences in dosing schedules and requirements by age provides an incentive for on-time vaccination [27]. Our provider-recommendation-related finding is possibly because parents who are rule followers adhere to the recommended schedule, or because parents who go to their primary care provider for vaccination are more likely to have a follow-up appointment scheduled and get a reminder call/postcard/text about their upcoming appointment. Although parental hesitancy discourages providers from having HPV-vaccine-related conversations with parents [25], it is important that providers understand that even hesitant parents are willing to change with provider encouragement. These findings offer early evidence for the need for provider education to improve the quality of provider recommendations, which could increase adherence to the recommended dosing schedule.

In our income analysis, we found that the odds of complying with the dosing schedule were higher among those below the poverty line. We found that adolescents who had any Medicaid had higher odds of adhering to the recommended HPV vaccine dosing schedule. According to Hoff et al. [28], Medicaid expansion resulted in a significant increase in HPV vaccine uptake among people living below the poverty level. Under the Vaccines for Children program (VFC), adolescents enrolled in Medicaid are eligible to receive all vaccines recommended by the ACIP at no cost to them or their families [29–31]. These factors may contribute to greater compliance with the vaccination schedule. Because VFC also covers uninsured adolescents, it is surprising that adherence to the recommended dosing schedule is lower in uninsured adolescents than in adolescents with only Medicaid. Therefore, the lower rates of adherence to the dosing schedule among uninsured adolescents should be investigated further.

In our census region analysis, we found that, in the South (where HPV vaccine completion is the lowest in the nation: Mississippi 28.8% and Wyoming 30.9%) [32], parents had lower odds of adhering to the recommended dosing schedule compared to other regions. This finding has important public health implications. If lower adherence to the recommended dosing schedule in the South census region persists, this region could face a disproportionately higher burden of HPV-related infections in future decades compared to regions with higher adherence to the recommended dosing schedule. Further research is needed to explore possible factors impacting adherence to the recommended HPV vaccine dosing schedule in the South census region and to determine how best to design interventions aimed at improving the adherence to the recommended HPV vaccine dosing schedule in the South census region.

While we found that the odds of vaccine completion with adherence to the recommended dosing schedule was higher in 2020 than in 2019, rates remain relatively low. Structural barriers, such as transportation difficulties, could make complying with the recommended vaccination schedule difficult for adolescents and their parents [33]. Thus, there is a need for improving vaccine accessibility for adolescents with parental consent by making the vaccine available in alternative settings, such as schools without the need for parents to be present. Increasing access to vaccination through systems-level interventions, such as school-based vaccination, can improve vaccination uptake with adherence to the vaccine schedule.

A limitation of this study is that the respondent is one parent/guardian of the adolescent and may not be the most knowledgeable about the adolescent's health status and vaccinations. However, we addressed this limitation by including only respondents with adequate provider data. Another limitation is that our dependent variable was designed to account for adolescents who received the 9vHPV vaccine as well as those who may have received other types of HPV vaccines (4 V, UV, CV, or HP) prior to the discontinuation in 2017. While this limitation did not affect the goal of our study, which is to examine the adherence to the recommended dose schedule, there is a need for the NIS-Teen researchers to include a variable that focuses on the adherence to the dosing schedule of the 9vHPV vaccine. This strategy will allow HPV-vaccine researchers to better examine adherence to the recommended dosing schedule of the only HPV vaccine currently administered in the US. Our study may be prone to residual confounding from father's educational status, as this variable is unavailable in the NIS-Teens dataset and was not accounted for in all our analyses. Other limitations, such as social desirability bias and recall bias, were also addressed by using respondents with adequate provider data. Strengths of the study include its large sample size and nationally representative data, and use of provider verified data.

Our study contributes to previous research by examining the sociodemographic factors associated with adherence to the HPV vaccine dosing schedule. Our findings suggest that adherence to the recommended dosing schedule remains relatively low. Adherence to the recommended dosing schedule is important for adequate immune response and expected protection from HPV infection. Therefore, findings from this research are important for the improvement in adherence to the HPV vaccine schedule.

#### **6. Conclusions**

It is important to investigate nonadherence to the recommended dosing schedule for HPV vaccination. Our cross-sectional analysis depicts salient factors, including provider recommendation, that can be targeted through interventions to improve adherence to the 9vHPV dosing schedule to offer better protection from HPV-related cancers. More research is needed to examine correlates of adherence to the recommended schedule for 9vHPV. Multilevel interventions to increase knowledge of the 9vHPV dosing schedule, improve the quality of provider recommendation, and remove access barriers to vaccination are warranted to improve the overall rates of adherence to the recommended 9vHPV dose schedule across races, age categories, and census regions.

**Author Contributions:** C.L.E. and I.O. are co-first authors, had full access to all of the data in the study, and take responsibility for the integrity of the data and the accuracy of the data analysis. *Study concept and design*: C.L.E., P.C., I.O. *Acquisition, analysis, and interpretation of data*: C.L.E., I.O., P.C. *Drafting of the manuscript*: C.L.E., I.O., S.A. *Critical revision of the manuscript for important intellectual content*: P.C., C.L.E. *Statistical analysis*: I.O., C.L.E. *Obtained funding*: NA. *Administrative, technical, or material support*: S.A., C.L.E. *Study supervision*: P.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This study was supported in part by: the authors did not receive any funding support for this study.

**Financial Disclosure:** None reported.

**All Financial Interests (Including Pharmaceutical and Device Product(s):** The authors declare no financial interest.

**Institutional Review Board Statement:** Ethical review and approval for this study was waived because the NIS-Teen data is de-identified and publicly available.

**Informed Consent Statement:** All participants in the NIS-Teen survey provided informed consent to participate in the survey.

**Data Availability Statement:** Data supporting reported results can be found at https://www.cdc. gov/vaccines/imz-managers/nis/datasets-teen.html, accessed on 5 April 2022.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Real-Life Safety Profile of the 9-Valent HPV Vaccine Based on Data from the Puglia Region of Southern Italy**

**Antonio Di Lorenzo, Paola Berardi, Andrea Martinelli, Francesco Paolo Bianchi, Silvio Tafuri \* and Pasquale Stefanizzi**

> Department of Biomedical Science and Human Oncology, Aldo Moro University of Bari, 70124 Bari, Italy; antoniodilorenzo95@gmail.com (A.D.L.); paola.berardi28@libero.it (P.B.); dott.a.martinelli@gmail.com (A.M.); frapabi@gmail.com (F.P.B.); pasquale.stefanizzi@uniba.it (P.S.)

**\*** Correspondence: silvio.tafuri@uniba.it

**Abstract:** Human Papillomavirus (HPV) is responsible for epithelial lesions and cancers in both males and females. The latest licensed HPV vaccine is Gardasil-9®, a 9-valent HPV vaccine which is effective not only against the high-risk HPV types, but also against the ones responsible for non-cancerous lesions. This report describes adverse events following Gardasil-9® administration reported in Puglia, southern Italy, from January 2018 to November 2021. This is a retrospective observational study. Data about the adverse events following immunization (AEFIs) with Gardasil-9® were collected from the Italian Drug Authority database. AEFIs were classified as serious or non-serious accordingly to World Health Organization guidelines, and serious ones underwent causality assessment. During the study period, 266,647 doses of 9vHPVv were administered in Puglia and 22 AEFIs were reported, with a reporting rate (RR) of 8.25 per 100,000 doses. The most reported symptoms were neurological ones (7/22). A total of 5 (22.7%) AEFIs were classified as serious, and 2 of these led to the patient's hospitalization. In one case, permanent impairment occurred. Following causality assessment, only 2 out of 5 serious AEFIs were deemed to be consistently associated with the vaccination (RR: 0.750 per 100,000 doses). The data gathered in our study are similar to the pre-licensure evidence as far as the nature of the AEFIs is concerned. The reporting rate, though, is far lower than the ones described in clinical trials, likely due to the different approach to data collection: in our study, data were gathered via passive surveillance, while pre-marketing studies generally employ active calls for this purpose. Gardasil-9®'s safety profile appears to be favorable, with a low rate of serious adverse events and a risk/benefits ratio pending for the latter.

**Keywords:** HPV; vaccines; AEFIs; causality assessment

#### **1. Introduction**

Human Papillomaviruses (HPVs) are responsible for multiple epithelial lesions and cancers in both males and females. They are the etiological agent of cutaneous and anogenital warts which may progress to carcinoma depending on the virus' subtype. Subtypes 16 and 18 are the ones most commonly associated with pre-cancerous and cancerous lesions of the cervix in females and of the anogenital and oropharyngeal areas in both sexes [1].

In Italy, the prevalence of high-risk HPV subtypes has been estimated at around 8% of the female population, with no significant geographical differences, while the incidence of cervical cancer is lower in the south of Italy. This uneven distribution of the incidence of HPV-related cancers has been explained by an increase in high-risk HPV prevalence in younger generations, thus predicting an increase in the burden of cervical cancer in the south of Italy in the coming decades [2].

The HPV vaccine is effective in reducing the risk of HPV infection, pre-cancerous lesions and cancer. Without HPV-vaccination, 7 out of 10 women and men will be infected with HPV, 1 in 10 women and men will get genital warts, 1 in 10 women will get precancerous lesions, 1 in 100 women will develop cervical cancer and 1 in 500 women will

**Citation:** Di Lorenzo, A.; Berardi, P.; Martinelli, A.; Bianchi, F.P.; Tafuri, S.; Stefanizzi, P. Real-Life Safety Profile of the 9-Valent HPV Vaccine Based on Data from the Puglia Region of Southern Italy. *Vaccines* **2022**, *10*, 419. https://doi.org/10.3390/ vaccines10030419

Academic Editor: Gloria Calagna

Received: 11 February 2022 Accepted: 7 March 2022 Published: 10 March 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

die of cervical cancer. In men, HPV is the cause of cancer of the penis, anus, oral cavity and throat, and it has been estimated that about 1 in 200 men will get cancer caused by HPV [3–5]. Anti-HPV vaccination is therefore an essential means of prevention both in HPV-naïve subjects and in subjects with high-risk-HPV-related pre-cancerous lesions.

Nowadays, three vaccines against HPV are available. Gardasil® (Sanofi Pasteur MSD)/Silgard® (Merck Sharp & Dohme, Kenilworth, NJ, USA), a quadrivalent recombinant vaccine against the HPV types 6, 11, 16 and 18 (qHPVv), was licensed by the Food and Drug Administration (FDA) in 2006, whereas Cervarix® (GlaxoSmithKline Biologicals, Brentfortd, Middlesex, UK), a bivalent recombinant vaccine for immunization against HPV types 16 and 18 (bHPVv), was licensed in 2007. Both vaccines contain non-infectious inactivated subunits, and protect against the high-risk HPV types 16 and 18, responsible for more than 70% of cervical cancer cases. The qHPVv also protects against subtypes 6 and 11, which cause most cases of genital warts [6,7].

The latest licensed HPV vaccine is Gardasil-9®, a 9-valent HPV vaccine (9vHPVv) which is effective against all of the HPV types covered by the qHPVv (6, 11, 16, and 18), as well as five additional oncogenic types (31, 33, 45, 52, and 58), showing strong protection against cervical infections caused by these HPV types as well as condylomas and some HPV-related cancers, including oropharyngeal, vaginal, vulvar, penile, and anal cancers [8].

The 9vHPVv was approved by the FDA on 10 December 2014, for use in females aged 9 to 26 years and males aged 9 to 15 years. For these recommendations, the Advisory Committee on Immunization Practices (ACIP) reviewed additional data on 9vHPVv in males aged 16 to 26 years. 9vHPVv and 4vHPVv are currently licensed for use in both females and males. 2vHPVv, on the other hand, is licensed for use in females only [9].

Immunization strategies in Italy are designed by the Ministry of Health and described in the National Immunization Plan (NIP). Each of the 20 Italian regions must follow the guidelines stated by the NIP, but may also offer other vaccines to target populations not covered by the National Plan itself. Furthermore, since 2012 the Ministry of Health has promoted the "Vaccination schedule for life", an immunization schedule that follows every phase of an individual's life with the objective of protecting them for the whole duration of their life [10].

The Italian Vaccination schedule for life, contained within the National Vaccine Prevention Plan 2017–2019, provides for two or three doses of HPV vaccine, according to the vaccine and to the patient's age. Two doses are recommended for subjects aged from 9 to 14 years (both males and females), with a 6- to 12-month interval between the doses, while three doses are recommended for patients aged from 15 to 26 years, with administration at 0, 1 to 2, and 6 months (only for women) [10].

Puglia is a region in the south of Italy (around 4 million inhabitants); the 2018 Apulian edition of the Vaccination schedule for life follows the same immunization schedule as the national one, while also offering the HPV vaccination to adult women considered at higher risk for cervical cancer due to high-risk sexual behaviors; in this case, the HPV vaccine may be requested by the patient herself via co-payment, or offered during periodic screening for cervical pre-cancerous lesions [11].

Three years after the implementation of the universal mass-vaccination program using 9vHPVv, it is useful, from a public health perspective, to assess the real-life safety profile of this vaccine. The World Health Organization (WHO, Geneva, Switzerland) recommends surveillance of Adverse Events Following Immunization (AEFIs) during the post-marketing life of new vaccines as a mean to better understand the safety profile and effectiveness of new drugs [12]. Thanks to the revision of AEFIs' reporting rates, post-marketing surveillance is indeed capable of detecting rare adverse events which pre-licensure studies could not observe, as well as studying the vaccine's safety profile in subgroups that were not represented in pre-marketing trials [13,14]. The WHO has recommended the application of a standardized causality assessment methodology in order to grant a more ordinate approach to the surveillance of AEFIs, as well as to surpass the

"emotional" approach, which may increase vaccine hesitancy while decreasing the quality of the surveillance data [15].

This report describes the adverse events following Gardasil-9® administration reported in Puglia from 2018 to 2021, focusing on serious AEFIs and taking into consideration the causality assessment. We aim to design the product's safety profile in a real-life scenario and compare it with the safety profile highlighted in phase-three clinical studies.

#### **2. Materials and Methods**

This is a retrospective observational study. Data were collected from the list of AEFIs recorded following 9vHPVv (Gardasil-9®) administration from January 2018 to November 2021, which was obtained from the Italian Drug Authority (AIFA, Rome, Italy) database. Reporting AEFIs is indeed mandatory for all healthcare workers in Italy, and reports must be submitted to the National Pharmacovigilance Network (RNF), an online platform managed by AIFA itself. AEFIs may also be reported by the person experiencing them or by their legal representative. The overall number of Gardasil-9® doses administered during the study period in Puglia was extrapolated from the regional online immunization database (GIAVA).

For every subject who suffered from one or more AEFIs, a form was completed which included information about date of birth, gender, date of vaccine administration, and other vaccines administered at the same time. AEFIs were described by providing the following data: date of onset and date of computing in RNF, clinical characteristics, duration, treatment, final outcome, hospitalization or emergency room access, and a description of the case.

An Excel spreadsheet was used to build the database and perform the required analyses. The total reporting rate was calculated as the total number of reported AEFIs divided by the number of Gardasil-9® administrations during the study period, while the annual reporting rate was calculated as the number of AEFIs that occurred in a year divided by the number of doses administered in the same year.

AEFIs were classified as "serious" or "non-serious" following WHO guidelines, that define an AEFI as serious if it results in death, is life threatening, requires in-patient hospitalization or prolongation of existing hospitalization, results in persistent or significant disability/incapacity, results in a congenital anomaly/birth defect, or requires intervention to prevent permanent damage or impairment. Additionally, in 2016, AIFA published a list of particular health conditions that must be considered as serious AEFIs when occurring after vaccination. This list is the Italian version of the European Medicines Agency's important medical events list [16,17].

For serious AEFIs, the WHO's causality assessment algorithm was applied in order to classify AEFIs as having a "consistent causal association", having an "inconsistent causal association", "indeterminate", or "non-classifiable". In particular, for AEFIs requiring hospitalization, the patient's medical records were examined for a better understanding of the event's characteristics [18]. Causality assessment was carried out by two different physicians with expertise in vaccinology and results were compared; in cases of divergent results, the literature was reviewed and a third physician was consulted in order to decide how to classify the adverse event.

#### **3. Results**

A total of 266,647 doses of 9vHPVv (Gardasil-9®) were administered in Puglia from January 2018 to November 2021. During the same period, 22 adverse events following Gardasil-9® administration were reported in Puglia (reporting rate (RR): 8.25 per 100,000 doses). Reporting rates were higher during the first two years after the vaccine's authorization, significantly decreasing over the following years (Table 1).


**Table 1.** AEFIs reporting rates during 2018–2021.

The overall male/female ratio of AEFIs was 0.833 (10 males vs. 12 females). The majority of AEFIs were reported in subjects aged from 10 to 18, with 15 reports out of 22 (68.2%) for subjects between 11 and 12 years of age. In detail, 17 AEFIs (77.3%) were reported in subjects aged from 10 to 14, an AEFI (4.50%) was reported for a 17-year-old subject and the remaining 4 AEFIs (18.2%) were observed in subjects over 25 years of age.

Table 2 describes the prevalence of specific signs and symptoms reported in the AEFI data. Neurological symptoms were the most common, followed by local events of pain, tenderness, oedema and/or swelling and allergic reactions.

**Table 2.** Prevalence of specific signs and symptoms described in the AEFI reports, and reporting rates (RR) per 100,000 doses.


Out of 22 AEFIs, 5 (22.7%) were classified as serious and 17 (77.3%) as non-serious, according to the latest WHO guidelines. A total of 2 out of 5 (40.0%) serious AEFIs led to the patients' hospitalization, and one of them (20.0%) caused impairment. The RR for serious AEFIs was 1.87 per 100,000 doses.

Out of the five serious AEFIs, two (40.0%) were deemed to be consistently associated with the vaccine's administration, while for another two of them (40.0%), no consistent causal association was found between the adverse event and the vaccination. For the fifth adverse event, the causality assessment outcome was undetermined.

The results show that 1 of the 2 AEFIs with consistent causal association caused the patient's hospitalization, but in both cases the subjects had fully healed by the time the report was completed. The RR for vaccine-related serious AEFIs was 0.750 per 100,000 doses.

The final outcome for 15 out of 22 AEFIs (68.2%) was the patient's complete recovery, while for three of them (13.6%) only partial improvement occurred. In total, 2 out of 22 patients (9.10%) had still not healed from the reported adverse events, while for another 2 (9.10%) the AEFI's outcome was not known.

We will now focus on the five serious AEFIs. We will describe these adverse events based on the data provided by the reporting subjects, and causality assessment will be taken into consideration in order to better understand the outcomes of the AEFIs' evaluations.

#### *3.1. AEFI 1*

The first case was reported in a female subject, aged 33 at the time of onset of the adverse event. The subject was already known to have had mild allergic reactions to food allergens. About six hours after Gardasil-9® administration, the patient manifested a skin rash localized to the chest and abdomen, followed by glottis oedema on the next day. The subject was therefore hospitalized, and the adverse event was treated by intravenous corticosteroid infusion and intramuscular injection of an antihistaminic (active ingredients were not reported), which were gradually discontinued over the following days. The adverse event healed completely. The reaction was deemed to be consistently associated with the vaccine's administration, as anaphylaxis has been observed as a rare serious adverse event following vaccination with Gardasil-9®.

#### *3.2. AEFI 2*

The second case was reported in a female subject, aged 11 at the time of onset of the adverse event. The subject was administered with Gardasil-9®, injected in the left deltoid, and anti-meningococcal serotype ACW135Y conjugated vaccine Menveo®, injected in the right deltoid. On the following day, the patient reported a syncopal episode with mild hypotension and significant oedema and hematoma were observed on the right arm. The subject required neither hospitalization nor pharmacological therapy, and the AEFI healed completely. The reaction was deemed to be non-associated with the vaccine's administration, as it happened more than 12 h after the vaccine's administration.

#### *3.3. AEFI 3*

The third case was reported in a male subject, aged 11 years at the time of onset of the adverse event. About one hour after Gardasil-9® administration, the subject lost consciousness for approximately one minute, falling and reporting mild cranial trauma. Witnesses reported that the patient appeared pale and sweating shortly before the event. The subject was put in the Trendelenburg position and an ice-bag was placed on his head, leading to complete recovery. Hospitalization was not needed, as the symptoms resolved in a few hours. The reaction was deemed to be consistently associated with the vaccine's administration.

#### *3.4. AEFI 4*

The fourth case was reported in a female subject, aged 25 at the time of onset of the adverse event. Ten days after Gardasil-9® administration, the subject reported prolonged asthenia of the lower limbs, muscle stiffness, fatigue and persistent paresthesia, with such intensity that ordinary activities were impeded. Myorelaxants were administered, followed by duloxetine, and the patient was diagnosed with fibromyalgia. It is interesting to note that this AEFI was reported only three years after its occurrence, mainly due to the symptoms' persistence and their significant impact on the patient's quality of life. The reaction was deemed to be non-associated with the vaccine's administration.

#### *3.5. AEFI 5*

The fifth case was reported in a female subject, aged 11 at the time of onset of the adverse event. The subject was administered with both Gardasil-9® and anti-meningococcal serotype B recombinant absorbed vaccine Trumenba® during the same immunization session. Following vaccination, the patient manifested headache, vertigo and extrasystoles with bigeminal rhythm (time of onset after vaccination was not noted in the report), and was therefore hospitalized. In hospital, laboratory analysis showed that their troponin level was 3 pg/mL, while the creatinine-kinase level was 106 U/L. No further action was taken, and the subject's health condition improved. The adverse event was defined as undetermined following causality assessment, due to the co-administration of two different vaccines and the lack of more specific information.

#### **4. Discussion**

Our study describes data referring to the safety profile of Gardasil-9®, which is currently the most used anti-HPV vaccine. Gardasil-9® is offered in Puglia to subjects aged

9 and older as part of the region's routine vaccination schedule. The vaccine was offered actively and free-of-charge, and 266,647 vaccine doses were administered in Puglia from 2018 to 2021.

Data from the passive surveillance of AEFIs showed that more than 8 subjects out of 100,000 receiving Gardasil-9® suffered from one or more adverse events. Serious AEFIs were reported in five patients, or less than 2 cases out of 100,000 administered doses, and the most common adverse events were neurological symptoms, reported in seven patients, local events of pain, tenderness, oedema or swelling, reported in six patients, and allergic reactions, reported in six patients.

Following causality assessment, a significant causal association between the adverse event and the vaccination was found for 2 out of 5 serious AEFIs. One of them was a case of loss of consciousness which occurred one hour after the vaccine's administration, and which spontaneously resolved in a few seconds; other symptoms, such as pallor and sweating, disappeared in less than a day. The episode did not determine any permanent damage to the patient. The second AEFI which was deemed related to the vaccine was an allergic reaction with mild glottis oedema and erythema located on the chest and abdomen, which occurred in a subject with a previous history of anaphylaxis, and which was successfully treated with corticosteroid and antihistaminic drugs after hospitalization.

Permanent impairment was reported in one case, but causality assessment ruled out the hypothesis of a causal correlation between the disability and the vaccine's administration; in this case, the patient reported asthenia, muscular stiffness, fatigue to the lower limbs and the formication of both hands and feet ten days after the vaccination, and was later diagnosed with fibromyalgia. While muscular symptoms are a common side effect of Gardasil-9® [19], no data exist regarding a plausible biological association between this vaccine and fibromyalgia, which was therefore defined as a non-vaccine-related AEFI. No cases of death were reported.

Comparing these data with pre-licensure evidence, reporting rates in our study are significantly lower. The United States Food and Drug Administration (FDA), for instance, reported local events of pain, swelling and erythema, and headache in more than 10% of patients treated with Gardasil-9® [19]. This discrepancy is likely due to the different surveillance methods employed in these studies: whereas pre-licensure evidence is gathered via active call, our data were collected through a passive surveillance network. Passive surveillance is in fact undermined by the risk of under-reporting, as Italian patients and healthcare professionals tend not to report adverse events, especially when mild and selflimiting. This phenomenon has already been documented by other studies of our research group [18].

On the other hand, our data are similar to pre-licensure evidence as far the symptoms' distribution is concerned. In fact, neurological symptoms and local reactions were the most reported symptoms both in FDA authorization studies and in ours [19].

A 2018 review on HPV vaccines' safety profile highlighted that injection-site reactions are fairly common with 9vHPVv, likely due to the greater amount of adjuvant contained in this product. Headache too was reported commonly, thus confirming the trend that emerged in our study. The higher reporting rates in this review are likely related to the active surveillance protocol employed in many of the considered studies. The review also focused on various adverse events of special interest, such as allergic reactions and anaphylaxis; the latter's reporting rate was 0.17 per 100,000 doses in the United States, about 0.30 per 100,000 doses in Australia and Canada and 1 per 100,000 doses in the United Kingdom. These data are slightly lower than the ones regarding allergic reactions we extrapolated from the RNF, which indicate an RR of 2.25 per 100,000 doses. Therefore, despite being flawed by under-reporting, Italian data may be considered a good estimate of the real incidence rate of AEFIs as far as serious reactions are concerned [20].

A 2017 review of clinical trials and case series regarding AEFIs after HPV vaccination included a clinical trial in which systemic adverse events were reported by 59.7% of patients who were administered with 9vHPVv. However, the difference between this group and the control one was not significant. Another study included in this review mentioned significant differences between 9vHPVv and 4vHPVv, with the 9-valent vaccine causing severe injection-site reactions and systemic adverse events more frequently than the 4-valent product. Systemic AEFIs were especially frequent, having been reported in nearly 30% of patients [21].

Our study's main strength is the high numerosity of the reference population: The Apulian population is over 4 million, and 266,647 doses of Gardasil-9® were administered over the course of four years. This is a significantly larger population than the ones examined in pre-licensure clinical trials. Moreover, Gardasil-9® has been licensed for use in Italy since 2018. During the immediate post-licensure period, the attention on new vaccines is higher from both the physicians' and the patients' perspectives. Studies that focus on the first stage of a drug's post-marketing life are fundamental not only to identify rare AEFIs and to better understand these new products' safety profile, but also to discern adverse events which are related to the vaccine from those that are not, building the public's trust towards vaccinations. In addition to this, our study focuses on causality assessment for serious AEFIs, an aspect that is often overlooked by post-marketing studies [15].

On the other hand, as already stated, our study is affected by our data collection method: passive surveillance carries a high risk of under-reporting and tends to alter the serious/non-serious AEFIs ratio. One of the reported adverse events, moreover, had not undergone causality assessment, thus reducing the available information regarding serious AEFIs.

The safety profile of vaccines is currently one of the main points of argument with anti-vaccination groups. This is also one of the most important reasons for vaccination skepticism among the general public, thus representing an essential building block of effective medical information [22]. As already demonstrated by the recent withdrawal of the AstraZeneca ChAdOx-1S anti-SARS-CoV-2 vaccine from the market, failures in communication between the scientific community and the public can cause a significant distrust of the general population towards vaccination practices [23].

As far as HPV vaccines are concerned, gaining optimal vaccination coverage is especially important. According to a 2020 modelling study, the recent crisis of HPV vaccinations in Japan is expected to result in 24,600–27,300 cases of cervical cancer in 1994–2007 birth cohorts, predicting at least 5000 deaths in the next few years; a further increase in HPV-related cancers might occur if vaccination coverages are not restored in the immediate future [24].

#### **5. Conclusions**

The risk of AEFIs is conclusively very low in subjects both under and over 18 years of age (<0.1‰ of administered doses), reinforcing the available evidence about the favorable risks/benefits ratio for Gardasil-9® [25]. Since the beginning of Gardasil-9®'s marketing, only one of the reported AEFIs led to permanent impairment, and it was not deemed to be consistently associated with the vaccine's administration. Furthermore, the outcomes of all the remaining serious AEFIs were at least a partial recovery.

Effective communication between the scientific world and the public is therefore imperative in order to ensure that people keep trusting vaccination practices and understand their importance.

**Author Contributions:** A.D.L. and P.B. conceived the study, collected data and drafted the manuscript. A.M. and F.P.B. contributed to the analysis of data. S.T. and P.S. reviewed the paper and carried out the causality assessment of serious AEFIs detected. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data regarding the total number of Gardasil-9® doses administered in Puglia during the study period were extrapolated from the regional online immunization database (GIAVA). Data regarding adverse events following immunization with Gardasil-9® were gathered from the Italian Drug Authority (AIFA) database.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Availability of the HPV Vaccine in Regional Pharmacies and Provider Perceptions Regarding HPV Vaccination in the Pharmacy Setting**

**Jill M. Maples 1,\*, Nikki B. Zite 1, Oluwafemifola Oyedeji 2, Shauntá M. Chamberlin 3, Alicia M. Mastronardi 1, Samantha Gregory 1, Justin D. Gatwood 4, Kenneth C. Hohmeier 4, Mary E. Booker 1, Jamie D. Perry 1, Heather K. Moss <sup>1</sup> and Larry C. Kilgore <sup>1</sup>**


**Abstract:** There is increasing support for HPV vaccination in the pharmacy setting, but the availability of the HPV vaccine is not well known. Additionally, little is known about perceptions of medical providers regarding referring patients to community pharmacies for HPV vaccination. The purpose of this study was to determine HPV vaccine availability in community pharmacies and to understand, among family medicine and obstetrics–gynecology providers, the willingness of and perceived barriers to referring patients for HPV vaccination in a pharmacy setting. HPV vaccine availability data were collected from pharmacies in a southern region of the United States. Family medicine and obstetrics–gynecology providers were surveyed regarding vaccine referral practices and perceived barriers to HPV vaccination in a community pharmacy. Results indicated the HPV vaccine was available in most pharmacies. Providers were willing to refer patients to a community pharmacy for HPV vaccination, despite this not being a common practice, likely due to numerous barriers reported. Pharmacist-administered HPV vaccination continues to be a commonly reported strategy for increasing HPV vaccination coverage. However, coordinated efforts to increase collaboration among vaccinators in different settings and to overcome systematic and legislative barriers to increasing HPV vaccination rates are still needed.

**Keywords:** HPV; HPV vaccine; pharmacy; vaccine referral; family medicine; obstetrics and gynecology; HPV vaccine barriers; provider perspectives

#### **1. Introduction**

Human papillomavirus (HPV) is the leading cause of most anogenital (i.e., anal, vulvar, vaginal, cervical, penile) and oropharyngeal cancers in both men and women, and the number of cancer cases linked to HPV has increased significantly over the past 15 years [1–3]. HPV-related cancers are the only cancers that can be prevented by a highly effective and safe vaccine [1,3]. In 2020, 75.1% of adolescents received at least one dose of an HPV vaccine, and 58.6% of adolescents were up to date with the entire series [4]. However, recent evidence suggests the COVID-19 pandemic resulted in fewer adolescents initiating the HPV vaccine series [4]. Among adults, the rates of HPV vaccination have increased moderately among certain populations over the past decade. For example, HPV vaccination

**Citation:** Maples, J.M.; Zite, N.B.; Oyedeji, O.; Chamberlin, S.M.; Mastronardi, A.M.; Gregory, S.; Gatwood, J.D.; Hohmeier, K.C.; Booker, M.E.; Perry, J.D.; et al. Availability of the HPV Vaccine in Regional Pharmacies and Provider Perceptions Regarding HPV Vaccination in the Pharmacy Setting. *Vaccines* **2022**, *10*, 351. https:// doi.org/10.3390/vaccines10030351

Academic Editor: Gloria Calagna

Received: 29 January 2022 Accepted: 22 February 2022 Published: 24 February 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

coverage among males aged 19–26 years and Hispanic females aged 19–26 years increased, but approximately 50% of females aged 19–26 years and 70% of males aged 19–26 years remained unvaccinated [5]. Disparities in HPV vaccination coverage and HPV-cancer incidence exist in geographic locations in the United States [6–8]. Specifically, those living in rural areas [8] and in the Southern United States [6,7] have lower HPV vaccination rates and higher HPV-related cancer incidence rates.

To address vaccination shortfalls, the President's Cancer Panel (2018) [9] and National Vaccine Advisory Committee (2016) [10] released statements supporting HPV vaccination utilizing community pharmacies. Pharmacists are ideally positioned to overcome some of the barriers to HPV vaccination initiation and series completion [11]. HPV vaccine administered in a pharmacy is a frequently mentioned strategy for increasing HPV vaccination coverage. Pharmacies are conveniently located for many families with easier access, especially in rural communities. In fact, most U.S. residents (91%) live within five miles of a community pharmacy [12]. Additionally, compared to doctors' offices, pharmacies are open for longer hours and on weekends, which potentially facilitates improved access to vaccines [13]. Pharmacists in most states are authorized to administer vaccines and provide a convenient option for patients [14]. The extent to which pharmacies stock and administer the HPV vaccine, particularly in rural settings, is unknown.

While pharmacies could potentially serve a powerful role in providing HPV vaccine access, administration, and series completion, relatively little is known about the willingness of physicians to refer patients to a pharmacy setting for HPV vaccination. Campos-Outcalt et al. (2010) reported that 34.2% of family physicians referred adult patients to a pharmacy for routinely recommended vaccines, but the authors did not report if they would refer adolescents and/or adult patients to a pharmacy setting for HPV vaccination specifically [15]. A recent study by O'Leary et al. (2020) reported that among obstetrician–gynecologists that routinely assess for patient vaccination status, 92% screened for HPV vaccination status [16]. However, not all of these providers stocked and/or administered the HPV vaccine [16]. For obstetrician–gynecologists that are unable to stock and administer vaccinations onsite, the American College of Obstetrics and Gynecology (ACOG) has created guidance to develop immunization referral systems [17]. This guidance was developed with the goal of future pharmacy–physician practice collaboration. However, to our knowledge, there are no studies reporting the willingness of and perceived barriers to referring patients for HPV vaccination in a pharmacy setting among family medicine and obstetrics–gynecology providers.

The purpose of this study was to determine HPV vaccine availability in community pharmacy settings in a southern region of the United States. Additionally, this study aimed to understand, among family medicine and obstetrics–gynecology providers, the willingness of and perceived barriers to referring patients for HPV vaccination in a community pharmacy setting. This was accomplished by surveying family medicine and obstetrics– gynecology providers at a single academic medical center in the same geographic region.

#### **2. Materials and Methods**

#### *2.1. Pharmacy Data Collection*

The availability of the HPV vaccine in the pharmacy setting was determined by the percentage of pharmacies that administer and stock the HPV vaccine. To accomplish this, a list of all pharmacies in a southern region of the United States (consisting of 17 counties in Eastern Tennessee) was generated. First, a list of zip codes for the target counties was generated and then an online search for pharmacies located in those zip codes was performed. Each pharmacy was called by a research team member acting as a "secret shopper", where the team member acted as a customer inquiring about pharmacy services from October 2020 through April 2021. A standardized script for data collection was developed primarily based on study outcomes, including HPV vaccine availability and other measures of accessibility (if the pharmacy stocked the HPV vaccine, how soon one could get the HPV vaccine, and if an appointment was needed to get the HPV vaccine). The script was discussed and confirmed by consensus among coauthors (J.M.M., O.O., S.G., J.D.G., K.C.H.). Using this standardized script, data regarding HPV vaccine availability were collected. The pharmacy staff member was first asked if the pharmacy administered the HPV vaccine. If the pharmacy stocked the HPV vaccine, they were then asked how soon one could get the HPV vaccine at their location, and if an appointment was required. Pharmacies were deemed "successfully contacted" if a research team member was able to call the pharmacy and get a definitive answer to any of the HPV vaccine-related questions. All pharmacies were considered in analyses. Up to three attempts were made to contact each pharmacy.

To better understand the impact of pharmacy type on HPV vaccine availability, pharmacies were categorized into the following groups: single independent, multiple independent, regional chain, grocery store chain (e.g., Kroger, Publix), national chain (e.g., Walgreens, CVS), and mass merchandiser (e.g., Wal-Mart, Target). To interpret the impact of pharmacy geographic location, categorizations of rural versus nonrural were created for each pharmacy location's zip code based on Health Resources and Services Administration's Federal Office of Rural Health Policy (HRSA FORHP) designations [18].

#### *2.2. Family Medicine and Obstetrics–Gynecology Provider Surveys*

An electronic survey was adapted from a previously published survey on adult immunization and preventative care practices by Hurley et al. [19]. The survey asked providers if their practice administered and stocked the HPV vaccine. Questions about Advisory Committee on Immunization Practices (ACIP)-recommended vaccine referral practices and then specifically about referral and prescription practices for the HPV vaccine across three different age categories for patients (age 11–18 years, age 19–26 years, and age 27–49 years) were included. Additionally, the survey asked about perceived barriers to HPV vaccination in a community pharmacy setting. The survey was emailed, via REDCap, to family medicine and obstetrics–gynecology providers (including physicians and midwives) at a single academic medical center. Research Electronic Data Capture (REDCap) is a secure, HIPAA-compliant, web application hosted by the University of Tennessee Graduate School of Medicine for building and managing online surveys and databases.

#### *2.3. Statistical Analysis*

Data analysis was primarily descriptive, using categorical data, means, and frequencies to determine vaccine availability and accessibility. Chi-squared analyses determined if HPV vaccine availability differed by pharmacy type and geographic location and if pharmacy type differed by geographic location. To determine HPV vaccine availability in the family medicine and obstetrics–gynecology providers' practices and to understand HPV vaccine referral practices, survey data analysis was primarily descriptive, using categorical data, means, and frequencies. Data were analyzed using Microsoft Excel (Office 365) and SPSS (Released 2019. IBM SPSS Statistics for Windows, Version 26.0. IBM Corp, Armonk, NY, USA).

#### **3. Results**

#### *3.1. Pharmacy Data Collection*

A total of 233 out of 240 (97.1%) pharmacies were successfully contacted. Characteristics of the pharmacies contacted are presented in Table 1 along with measures of HPV vaccine accessibility and availability. Among the pharmacies contacted, the majority were national chain pharmacies (35.2%), followed by single independent pharmacies (27.5%). More than half of the pharmacies administered the HPV vaccine (60.1%), and among those that administered it, over two-thirds (67.1%) reported having it "in stock". Ninety percent of the locations did not require an appointment to receive the vaccine, and 76.4% reported that the vaccine could be received the same day as the call of inquiry.


**Table 1.** Pharmacy characteristics.

HPV vaccine availability differed significantly (*p* < 0.01) across pharmacy type (Figure 1). A larger percentage of mass merchandisers (24 out of 25, 96.0%), national chain pharmacies (76 out of 82, 92.7%), and grocery store pharmacies (23 out of 32, 71.9%) provide the HPV vaccine compared to single independent pharmacies (13 out of 64, 20.3%), multiple independent pharmacies (2 out of 24, 8.3%), and regional chain pharmacies (2 out of 6, 33.3%). HPV vaccine availability in pharmacies stratified by geographic location (rural vs. nonrural) was not significantly different (*p* = 0.704). Additionally, there were no significant differences in pharmacy type when stratified by geographic location (*p* = 0.704).

**Figure 1.** Reported HPV vaccine availability by pharmacy type. The total number of pharmacies, stratified by type, is indicated by each bar. Each bar is stacked to display the proportion of pharmacies, within that pharmacy type, that have the HPV vaccine available and those that do not have the HPV vaccine available. Chi-squared analysis revealed a significant difference in HPV vaccine availability across pharmacy type (*p* < 0.001).

#### *3.2. Family Medicine and Obstetrics–Gynecology Providers*

#### 3.2.1. Provider Characteristics

The survey was distributed to 70 family medicine and Ob/Gyn providers; 60 providers completed the survey (85.7% response rate). Among the respondents, 34 (56.7%) were affiliated with the Department of Family Medicine at the University of Tennessee Graduate School of Medicine and 26 (43.3%) were affiliated with the Department of Obstetrics and Gynecology at the same institution. The majority identified as female (*n* = 38, 63.3%), and the age of the respondents ranged from 26 to 69 years of age (mean = 35.9 ± 10.6 years). In terms of role, 55.0% (*n* = 33) were resident physicians, 41.7% (*n* = 25) were attending physicians, and 3.3% (*n* = 2) were midwife practitioners.

#### 3.2.2. HPV Vaccination Availability and Referral Practices

Most respondents indicated their practice administered and stocked the HPV vaccine. Nearly all (98.3%, *n* = 59) respondents reported their practice administers the HPV vaccine, and 95.0% stock the vaccine. When asked if they had ever referred a patient to receive an ACIP-recommended vaccine outside of their practice, 81.7% (*n* = 49) reported they had referred a patient for vaccination. When asked specifically if they have ever referred a patient for HPV vaccination outside of their practice, 20.0% (*n* = 12) had referred a patient age 11–18 years, 21.7% (*n* = 13) had referred a patient age 19–26 years, and 20.0% (*n* = 12) had referred a patient age 27–49 years. Among those that had ever referred a patient for HPV vaccination outside of their practice, over 90.0% referred patients to a local health department for HPV vaccination and did so across all patient age groups. Regarding referrals to a community pharmacy for HPV vaccination, 50% of those ever referring a patient for HPV vaccination referred a patient age 27–49 years, 53.8% referred a patient age 19–26 years, and 33.3% referred a patient age 11–18 years.

When asked if they had ever prescribed the HPV vaccine for a patient to be received at a community pharmacy location, 6.7% (*n* = 4) reported they had prescribed the HPV vaccine for patients age 19–26 years, followed by 5.0% (*n* = 3) and 3.3% (*n* = 2) for patients age 27–49 and 11–18 years respectively. Most providers (>90%) were willing or very willing to refer an eligible patient to receive an HPV vaccination at a community pharmacy across all age groups (Table 2).


**Table 2.** Willingness of provider to refer an eligible patient for HPV vaccination in a community pharmacy, stratified by patient age group.

Perceived barriers to referring a patient for HPV vaccination in a pharmacy setting are shown in Figure 2. The most-reported barrier (61.7% of respondents) indicated that providers felt it made the most sense that their patients receive the HPV vaccine in their office (Figure 2).

**Figure 2.** Perceived barriers to providers referring patients for HPV vaccination at community pharmacies. Each bar presents the percentage of respondents that indicated the barrier listed was a barrier for referring patients for HPV vaccination at a community pharmacy. The total percentage summed from all bars exceeds 100% because this survey question was a "mark all that apply" format question.

When asked what percentage of pharmacies in their region administer and stock the HPV vaccine, 88.3% (*n* = 53) of providers answered, "I do not know". Among the few respondents (11.7%, *n* = 7) that provided a numeric answer to the question regarding the percentage of pharmacies that administer the HPV vaccine, answers ranged from 25% to 85%, which was similar to the answers provided in response to the question of percentage of pharmacies that stock the HPV vaccine (ranging from 10% to 90%).

#### **4. Discussion**

The purpose of this study was to determine HPV vaccine accessibility in pharmacy settings in a southern region of the United States and to understand, among family medicine and obstetrics–gynecology providers, the willingness of and perceived barriers to referring patients to community pharmacies for HPV vaccination. The primary findings indicated that the majority of pharmacies in Eastern Tennessee (a state in the Southern United States) administer and stock the HPV vaccine. Family medicine and obstetrics–gynecology providers surveyed in this study were willing to refer patients to a community pharmacy for HPV vaccination, despite this not being a common practice. Most providers did not know the extent to which community pharmacies in their region administered and/or stocked the HPV vaccine. Providers also reported several perceived barriers to referring patients for HPV vaccination in a pharmacy setting. The most common barriers expressed included their desire for patients to be vaccinated in their office; concern that patients would not complete the HPV vaccine series if referred outside their practice for vaccination; providers not knowing or having documentation of vaccination if performed in a pharmacy setting; and potential financial burdens for patients, associated with lack of insurance coverage, for the HPV vaccine in a pharmacy setting.

In the current study, a larger proportion of pharmacies (60.1%) reported administering the HPV vaccine (Table 1) compared to previous reports [13,20]. A study by Hastings et al. (2017) reported that only 11.7% of community pharmacies in the state of Alabama, which is another state located in the Southern United States, had the HPV vaccine in their inventory [20]. A study by Westrick et al. (2018) reported that, among a nationally representative sample of pharmacies in the United States, 38.9% of pharmacies offered the HPV vaccine [13]. However, 44.2% of the sample consisted of independently owned pharmacies, and the percentage of pharmacies offering the HPV vaccine was not stratified by pharmacy type [13]. In the current study, mass merchandisers (96.0%), national chain pharmacies (92.7%), and grocery store pharmacies (71.9%) reported administering the HPV vaccine, while fewer single independent pharmacies (20.3%), multiple independent pharmacies (8.3%), and regional chain pharmacies (33.3%) offered the HPV vaccine (Figure 1). This is consistent with previous reports that describe barriers to HPV vaccination among independent pharmacies [21,22]. These include organizational barriers to HPV vaccine administration including lack of space and staff.

Reports indicate HPV vaccination coverage is lower in rural areas, despite these areas having higher rates of HPV-associated cancers [4,8]. In the current study, we surveyed pharmacies in a region where approximately 38% of the population lives in a rural area [23]. We found that HPV vaccine availability in pharmacy settings was not different between those in rural and nonrural locations. Additionally, there were no significant differences in pharmacy type when stratified by geographic location, which suggests that, in the region surveyed, the HPV vaccine is likely available in a chain, mass merchandiser, and/or grocery pharmacy setting even in rural locations. This is consistent with the concept that pharmacies in rural locations could potentially serve a powerful role in providing HPV vaccine administration [12,22]. However, the benefit of the HPV vaccine in the rural pharmacy setting alone is clearly not enough to increase HPV vaccination coverage as HPV vaccination uptake and accessibility is impacted by legislative, social, and environmental factors unique to HPV vaccination, nor would availability overcome other obstacles for vaccination including medical literacy, socioeconomics, and personal choice [11].

In the current study, family medicine and obstetrics–gynecology providers were willing to refer patients to a community pharmacy for HPV vaccination, despite this not being a common practice. A study by Hurley et al. (2014) reported on vaccination referral practices by alternate vaccinators, including general internists and family medicine providers, where financial barriers made them less likely to stock and administer vaccines [19]. Although the report was not specific to the HPV vaccine, the providers surveyed indicated they "always, often or sometimes" referred patients to a pharmacy for vaccination [19]. Additionally, the study reported that most providers agreed it was helpful to have pharmacists share a role

in patient vaccination. However, one-third of providers had reservations about pharmacists as vaccinators. Some of these reservations were related to communication, or lack thereof, between physicians and pharmacists and concerns about inadequate documentation of patient vaccines. They disagreed that it was more convenient for their patients to get vaccines at a pharmacy compared to their office, and the majority reported their patients preferred to receive vaccines at their office, rather than a pharmacy. The results of the current study suggest similar themes presented in the Hurley et al. study. Providers support the idea of pharmacists as vaccinators (Table 2) but would prefer to vaccinate their patients in their offices (Figure 2). Perhaps this is tied to concern about fragmented care and the importance of patients receiving care in their medical home.

It is worthwhile to mention that among the providers surveyed in the current study, none of them perceived HPV vaccination as outside the scope of a pharmacist. This suggests the training and knowledge of a pharmacist surrounding the HPV vaccine may not be problematic for many family medicine and obstetrics–gynecology providers. Overall, these findings are in contrast to a study by Welch et al. (2003) that found most family medicine physicians were neither very knowledgeable about nor supportive of pharmacists as vaccinators [24]. It is possible the providers surveyed in the current study are inherently different compared to the providers surveyed in the Welch et al. study. For example, the providers included in the current study were from a single academic institution where the majority work in clinics that administer and stock the HPV vaccine. Similarly, it is possible that providers in clinical settings that are unable to or that chose not to administer or stock the HPV vaccine may have different HPV vaccine referral practices and/or attitudes towards referring patients to a pharmacy setting for HPV vaccination. In terms of generalizing the findings of the current study more broadly, several of the providers surveyed indicated they treat primarily adult patients, which could have influenced their willingness and comfort to refer patients to a pharmacy setting for HPV vaccination.

While the majority of pharmacies in the current study had the HPV vaccine available, the majority of providers in the current study were not actively referring patients to a pharmacy setting for HPV vaccination. Providers reported concerns their patients may not complete the HPV vaccine series if referred outside of their office for vaccination and expressed concern they would not be informed whether their patients received vaccines in a pharmacy setting (Figure 2). The lack of coordinated up-to-date immunization registries has been cited as a barrier in previous studies [21]. Promoting the wider use of immunization registries could be a strategy to overcome some provider concerns related to referring patients outside the office setting for vaccination.

In the current study, we did not specifically ask providers to distinguish their opinions and/or practices regarding HPV vaccine series initiation versus booster. For example, it is possible some providers prefer to administer the first HPV vaccine dose that a patient receives in their clinic, but would be willing to refer patients to a pharmacy setting for HPV vaccine series completion. Additional studies are needed to determine if providers might be more willing to refer patients to a pharmacy for HPV vaccine series completion and to determine the rate of follow-up for HPV vaccination among patients that are referred to a community pharmacy for HPV vaccination series completion. A recent study by Douchette et al. (2019) described a coordinated delivery of an HPV vaccine program using a clinic–pharmacy partnership. In their study, less than 50% of patients referred to a pharmacy received an HPV vaccine [25]. However, 100% of the patients that received the HPV vaccine in the pharmacy setting, after referral to the pharmacy for vaccination, completed the series. Further evidence of successful vaccine series completion in a community pharmacy setting is described in a recent report by Frederick et al. (2020). In this study, they describe the use of "nudge"-based clinical decision support embedded within a community pharmacy's software system to improve vaccine series completion in a community pharmacy setting [26].

This study has limitations. These results may not be generalizable beyond this specific region in the Southern United States, as HPV vaccination uptake and accessibility are impacted by several factors that are state- and region-specific. These may include legislation and policies regarding HPV vaccination, social and environmental factors, religiosity, political ideology, and vaccine hesitancy. However, these data suggest that more research is needed to better understand perceived barriers and opportunities for HPV vaccination among providers in a variety of clinical settings. The providers included in the current study were from a single academic institution, working in clinics that administer and stock the HPV vaccine. It is possible that providers working in different settings would not be as willing to refer patients to a pharmacy setting for the HPV vaccine. Future work should include providers that are not affiliated with an academic medical center and those that do not work in practices that administer and stock the HPV vaccine. Finally, this study did not attempt to evaluate pharmacists as facilitators of HPV vaccines or their comfort in counseling patients about HPV vaccination. Nor did this study attempt to evaluate perceptions of facilitators to HPV vaccination in the pharmacy setting among family medicine and obstetrics–gynecology providers. Additional studies are needed to understand barriers and overcome obstacles to HPV vaccination, particularly in the community pharmacy setting.

#### **5. Conclusions**

Findings suggest that the HPV vaccine is commonly available in nonindependent pharmacies in Eastern Tennessee. Family medicine and obstetrics–gynecology providers report they are willing to refer patients for HPV vaccination in a pharmacy setting; however, several barriers were reported that might limit this practice. While pharmacist-administered HPV vaccination continues to be a commonly reported strategy for increasing HPV vaccination coverage and the availability of the HPV vaccine in the pharmacy setting is common, coordinated efforts to increase collaboration among vaccinators in different settings and overcome systematic and legislative barriers to increasing HPV vaccination rates are still needed.

**Author Contributions:** Conceptualization, L.C.K., N.B.Z., J.M.M., O.O., S.G., J.D.G. and K.C.H.; Methodology, L.C.K., N.B.Z., J.M.M., O.O., S.G., A.M.M., J.D.G., K.C.H., S.M.C., J.D.P. and H.K.M.; Validation, J.M.M., O.O., S.G. and A.M.M.; Formal Analysis, J.M.M., M.E.B. and A.M.M.; Investigation, L.C.K., N.B.Z., J.M.M., O.O., S.G., A.M.M., M.E.B., J.D.G., K.C.H. and S.M.C.; Writing—Original Draft Preparation, J.M.M.; Writing—Review and Editing, L.C.K., N.B.Z., J.M.M., O.O., S.G., A.M.M., J.D.G., K.C.H., S.M.C., J.D.P., M.E.B. and H.K.M.; Visualization, J.M.M. and A.M.M.; Supervision, L.C.K. and J.M.M.; Project Administration, J.M.M., O.O., S.G. and S.M.C.; Funding Acquisition, L.C.K., N.B.Z. and J.M.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was funded by a University of Tennessee Medical Center Cancer Institute Research Grant awarded to L.C.K., N.B.Z. and J.M.M.

**Institutional Review Board Statement:** This study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board of the University of Tennessee Graduate School of Medicine (IRB# 4691 approved 26 October 2020; IRB# 4884 approved 2 December 2021).

**Informed Consent Statement:** For the pharmacy data collection portion of this study, respondent consent was waived due to the "secret shopper" nature of the study design. No identifiable information was collected from those contacted in the pharmacy setting and all pharmacy-related data were presented in aggregate. For the provider survey, informed consent was obtained from all respondents involved in the study.

**Data Availability Statement:** These data are not publicly available, but the data presented in this study are available upon reasonable request from the corresponding author.

**Acknowledgments:** The study team would like to acknowledge Mara Walters, Zachary Shelton, Emma Mitchell, Allison Zisko, and Cara Rubin for their assistance in collecting pharmacy data.

**Conflicts of Interest:** Kenneth C. Hohmeier reports vaccine-related funding from Genentech, Merck & Co., Kenilworth, NJ, USA and GlaxoSmithKline, Philadelphia, PA, USA. Justin D. Gatwood reports vaccine-related funding from Merck & Co., Kenilworth, NJ, USA and GlaxoSmithKline, Philadelphia, PA, USA. The funders, the University of Tennessee Medical Center Cancer Institute, had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

#### **References**


## *Article* **Parents' Knowledge and Attitude towards HPV and HPV Vaccination in Poland**

**Katarzyna Smolarczyk 1,\*, Anna Duszewska 2, Slawomir Drozd <sup>3</sup> and Slawomir Majewski <sup>1</sup>**


**Abstract:** HPV is one of the diseases of civilization that causes cervical cancer, among other diseases. For this reason, a vaccination program has been introduced worldwide for preadolescent, sexually inactive seronegative girls. However, the decision to vaccinate young girls must be made by the parents. In Poland, vaccinations are recommended but not financed by the government, which affects their choices, and there is insufficient knowledge of the diseases caused by genital HPV types. In addition, there are cultural, social, and even religious factors to be considered. Therefore, the aim of the study was to analyze the state of knowledge about HPV and HPV vaccines among parents. Two hundred and eighty-eight parents participated in the study, but only 180 of them declared that they had ever heard of HPV (62.5%). Therefore, only these parents completed the entire questionnaire consisting of 34 questions. The parents' answers were analyzed with the Fisher's and chi-squared tests. The study showed that parents' knowledge of HPV and HPV vaccination in Poland is low (49.4% of correct answers). Parents' attitudes were only influenced by knowledge and education and not by other parameters such as age, gender, place of residence, and the number of children. This study indicates that parents need to be educated about the threats of HPV and the possibilities of prophylactic vaccination.

**Keywords:** HPV; HPV vaccine; human papillomavirus

#### **1. Introduction**

Human papillomaviruses (HPVs) belong to the family *Papillomaviridae* and are nonenveloped icosahedral, circular, dsDNA viruses [1]. HPVs infect the cutaneous and mucosal epithelia, hence its wide spectrum of occurrence (skin, oral cavity, oropharynx, larynx, anogenital tract) [2]. There are more than 207 types of HPV [3], most of which do not cause any symptoms, lesions, or warts and are referred to as low-oncogenic (types). However, some types of HPVs are highly oncogenic and can induce intraepithelial neoplasia or cancers [2,4]. Among other diseases, cervical cancer is associated with types 16, 18, 31, 33, 35, 39, 45, 51, 52, 56, 58, 68, 73, 82, and especially 26, 53, and 66 [2]. In the European population, eight types are of particular importance (16, 18, 31, 33, 35, 45, 56, and 58), with 16 and 18 being responsible for 70 percent of all cases of cervical cancer [5].

In 2012, there were 530,000 cases of cervical cancer related to HPV, of which 370,000 were caused by HVP16/18 (71%) [6]. In addition, more than two-thirds of the cases are diagnosed in less developed countries [7,8]. Interestingly, the highest number of cases of HPV cervical cancer occurs in Asia (India 120,000) and Sub-Saharan Africa (93,000), and one of the lowest in Australia/New Zealand (940) [6]. The age-standardized mortality rate for cervical cancer is lower in developed nations at 2.2 per 100,000 compared with 4.3 per 100,000 in the developing nations [4].

**Citation:** Smolarczyk, K.; Duszewska, A.; Drozd, S.; Majewski, S. Parents' Knowledge and Attitude towards HPV and HPV Vaccination in Poland. *Vaccines* **2022**, *10*, 228. https://doi.org/ 10.3390/vaccines10020228

Academic Editor: Gloria Calagna

Received: 18 January 2022 Accepted: 31 January 2022 Published: 2 February 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

In the European Union, about 34,000 new cases of cervical cancer related to HPV are diagnosed every year. The highest prevalence of cervical cancer associated with HPV infection are observed in Latvia (25.0/100,000 women), Bosnia and Herzegovina (23.9/100,000), and Estonia (22.5/100,000). On the other hand, the lowest prevalence is found in Malta (3.5/100,000), Switzerland (3.8/100,000), and Finland (4.7/100,000). In Poland, the prevalence is 9.4/100,000 women [9].

Currently, three vaccines are used in the world: 1. Cervarix (GlaxoSmithKline), a bivalent (2-V) vaccine targeting HPV16 and HPV18, the two most carcinogenic types; 2. Gardasil (Merck Inc., Meguro City, Tokyo), a quadrivalent (4-V) vaccine targeting HPV16/18 and the low risk types, HPV6 and HPV11, that cause genital warts; and 3. Gardasil 9 (Merck Inc.), a nonavalent (9-valent, 9-V) vaccine targeting HPV6/11/16/18 and the next five most carcinogenic types (HPV31/33/45/52/58).

Most recommendations, including from the WHO, recommend routine HPV vaccination in girls aged 9–14 years before becoming sexually active. The secondary target group is girls over the age of 15 years and young women. HPV vaccination of males is currently not recommended as a priority [10,11]. However, the CDC recommends HPV vaccination at the age of 11 or 12 years (but it can start from the age of 9 years) and for men up to the age of 26 years, if not vaccinated already [12].

HPV vaccination was first introduced in 2007 in Australia and, since then, many countries have joined the vaccination program. In 2020, HPV vaccination has been introduced in 107 (55%) of the 194 WHO Member States. In the Americas and Europe, 85% and 77% of the countries have already introduced HPV vaccination. A lower number of countries with a HPV vaccination programme was observed in Oceania (56%), Asia (40%), and Africa (31%) [13].

The implementation of an HPV vaccine programme has created many controversies. The two most frequently cited sources of negative knowledge about HPV vaccination are the Canadian article "Guinea pigs" and the Danish paradocumentary "De Vaccinerede Piger" (The Vaccinated Girls—Sick and Abandoned). The article describes the vaccination programme as "the biggest Canadian science experiment in decades". The paradocumentary tells the story of some girls who allegedly had a reaction to a HPV vaccination and suffered from POTS—postural orthostatic tachycardia syndrome. The paradocumentary presents the HPV vaccine in a very negative light, although no studies confirmed the described symptoms.

The vaccination coverage level depends on at what age the first dose of vaccination is being administered (higher coverage level in younger groups). However, in general, a high level of vaccination coverage was observed in Australia (78.6%) and in the United Kingdom (81%), in contrast with countries with low vaccination coverage level such as Georgia (36.2%), Lithuania (29%) [13], and Poland (7.5–10%) [14].

Until now, Poland was one of the EU countries where vaccination against HPV was recommended but not government-funded [14]. There are no restrictions to access of any type of HPV vaccine in Poland, but they are expensive. Difficulties limiting or even preventing the implementation of HPV vaccination in Poland include:

	- a. lack of recommendations from the National Health Fund (NHF) and doctors,
	- b. "bad attitude" towards vaccination in anti-vaccine environments,
	- c. lack of support and conversations with parents about sexuality.
	- a. availability of vaccination,
	- b. the price of the vaccine,
	- c. the need to repeat vaccination (compliance).
	- a. sexual promiscuity,

b. negative information about the vaccine in the media (ineffective, not very well studied, dangerous).

Of these many barriers, the lack of support and conversations with parents about sexuality deserves special attention. This issue is critical because prophylactic vaccination of girls against HPV should be carried out in the period preceding sexual initiation. Therefore, the decision to vaccinate girls is the responsibility of parents who, by giving their consent, give them the chance to avoid a disease that is dangerous to their health and even their life when they become adults.

The attitudes of parents and adolescents, using many research models [15–17], have already been the subject of many studies globally, including those conducted in Europe, and a review in this field was undertaken by Lopez et al. [18]. However, the situation in Poland was not considered in that review, so it is well-justified to present the results of studies that aim to assess: 1. The state of knowledge of parents about HPV infections and HPV vaccines; and 2. Parents' attitude to vaccination, including HPV vaccines.

Cervical cancer remains high in incidence and mortality rankings in Poland. Despite this, HPV vaccines are not covered by the government. Many local governments in Poland organize free HPV vaccination programs, but HPV immunization remains low.

This study, the first of its kind to be undertaken in Poland, undertook an analysis to determine the barriers to vaccination acceptance.

This study aimed to assess parents' knowledge about HPV infections, assess parents' knowledge about HPV vaccines, and assess the impact of parents' knowledge about HPV infection and HPV vaccines on their attitude to primary prevention, i.e., vaccinations.

#### **2. Materials and Methods**

#### *2.1. Design of the Study*

An observational cross-sectional descriptive study was undertaken.

All subjects gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Ethics Committee of the Medical University of Warsaw (AKBE/123/16).

#### *2.2. Data Collection*

The survey was conducted in 2018 in Warsaw, Poland, and included parents of the children admitted to the Department of Pediatric Dermatology at a multidisciplinary regional hospital.

Data was collected using questionnaires presented on electronic devices (such as tablets, laptops, mobile phones) during the admission of the patients to the Department of Pediatric Dermatology. A researcher who monitored the study and helped with technical problems was present to ensure that only one parent was involved in the study.

Four parents refused to fill in the questionnaire, answering "no time" as the reason. The study was completed when questionnaires from a minimum of 200 parents were obtained. None of the children of the surveyed parents were diagnosed with HPV-related lesions as a reason for admission to the department.

#### *2.3. Sample Size*

A goal of the study was to include as many participants as possible. At the initial stage, no formal calculation of sample size was carried out. The final number of 288 parents ensured precision (measured at half the length of the 95% confidence interval) of 4 percentage points for assessing a trait whose true prevalence was 50% (for which 50% is needed in the most significant sample to reach a particular precision value).

#### *2.4. Statistical Analysis*

The analysis was conducted mainly with the use of descriptive statistics. The results are presented in the form of frequency tables and cross tables. Statistical inference was performed using the chi-squared test or, in the case of low frequencies of the analyzed features, Fisher's exact test.

Fisher's exact test is used in the case of samples that are too small—when the observed values calculated with the chi-squared test are below 5. The calculations were performed using the statistics program R 3.5.1. All tests were performed at a significance level *p* = 0.05.

The parents' responses regarding their knowledge about HPV were presented depending on their age, sex, their place of residence, and education. The parents' knowledge about vaccination was also analyzed. The parents' responses to the questions regarding their knowledge about vaccinations were classified as correct or incorrect. Total test result and results regarding the parents' knowledge about HPV and HPV vaccine was expressed as a percentage of available points.

#### *2.5. Questionnaire*

The questionnaire survey (Appendix A) for parents consisted of 34 questions and was designed by the authors. It was preceded by preliminary information, which consisted of explaining the purpose of the study, details on how to contact the author, and information about the voluntary and anonymous nature of the survey.

The survey consisted of both single-choice and multiple-choice questions. The age question was an open-ended question. The rest of the questions were closed questions. Seven questions related to their knowledge about the HPV virus, and seven questions related to their knowledge about the HPV vaccine. The rest of the questions were about the parents' attitudes to vaccination and demographic data. One hundred and eighty of the surveyed parents declared that they had heard about the HPV virus and filled in the entire questionnaire. The rest of the respondents answered "no" or "I do not know" and were asked to provide their age, gender, place of residence, and education.

#### **3. Results**

#### *3.1. Group Characteristics*

The study included the parents of the children admitted to the Department of Pediatric Dermatology in Warsaw, Poland. Two hundred and eighty-eight parents participated in the study. Most of the respondents were women (78.8%). Parents between the ages of 30 and 40 years accounted for 61% (n = 155) of the respondents, parents between the ages of 20 and 30 years accounted for 24% (n = 61), and parents between the ages of 40 and 70 years accounted for 15% (n = 38).

Furthermore, 38 parents (13.2%) lived in the countryside, 26 (9%) lived in a city with up to 20,000 inhabitants, 33 (11.5%) lived in a city with 20,000–100,000 inhabitants, 36 (12.5%) lived in a city with 100,000–500,000 inhabitants, and 155 (53.8%) lived in a city with more than 500,000 inhabitants.

Most of the respondents had higher education (71.2%, n = 205). Sixty-eight people had secondary education (23.6%), 13 people had basic vocational education (4.5%), and 2 people had primary education (0.7%). A summary of these data is presented in Table 1.


**Table 1.** Characteristic of the group.

**Table 1.** *Cont.*


Among those surveyed, 180 declared that they had heard of the HPV virus and filled in the entire questionnaire. The characteristics differentiating these two groups are presented in Table 2. Most people declaring knowledge about HPV were women. The numbers of men who declared that they had heard about the virus (n = 33) and had not heard about the virus (n = 28) were similar.

**Table 2.** Characteristics of the group of parents divided into two groups, declaring knowledge about HPV and HPV vaccination and declaring the lack of knowledge.


#### *3.2. Test Results*

Table 3 provides a summary of the total test scores and the number and percentage of parents with a particular test score. The first division presents all the scores together with the frequency and percentage of parents who achieved them. The highest score was 86.5% of correct answers, and the lowest score was 21.6% of correct answers.

From an analysis of the second part of the table, 2.8% of the parents achieved a score of up to 30% of correct answers, 47.8% scored in the range of 31–60 (n = 86), and 49.4% scored in the range of 61–100 (n = 89). Therefore, as many as 50.6% of parents would not pass the test if the pass mark for this test was a score above 61% of correct answers.

Table 4 provides summary statements for the percentage of the parental test for HPV vaccination knowledge. The first division presents the test scores together with the frequency and percentage of parents who achieved them. The highest score was 92.3% of correct answers, and the lowest score was 23.1% of correct answers.

From an analysis of the second part of the table, 2.2% of parents achieved a score of up to 30% of correct answers, 32.8% scored in the range of 31–60, and 65.0% scored in the range 61–100. Therefore, 35% of parents would not pass the test if the pass mark for this test was a score above 61% of correct answers.

Table 5 contains the summary result of the parents' test of knowledge about HPV infections. The first division presents the test scores together with the frequency and percentage of parents who achieved them. The highest test score was 84% of correct answers, and the lowest score was 16% of correct answers.

From an analysis of the second division, 6.7% of parents achieved a score of up to 30% of correct answers, 56.7% scored in the range of 31–60 (n = 102), and 36.7% scored in the range 61–100 (n = 66). Therefore, 63.4% of parents would not pass the test if the pass mark for this test was a score above 61% of correct answers.


**Table 3.** Summary test results.

Total test result = overall knowledge points scored in test by parents. N = number of parents who achieved this test score; % = percentage of parents who achieved this test score.


**Table 4.** Summary test results—knowledge about HPV vaccine.

Total test result = overall knowledge points scored in test by parents; N = number of parents who achieved this test score; % = percentage of parents who achieved this test score.


Total test result = overall knowledge points scored in test by parents; N = number of parents who achieved this test score; % = percentage of parents who achieved this test score.

#### *3.3. Parents' Knowledge—Correct Answers*

Table 6 contains a summary list of answers to questions concerning parents' knowledge about the HPV virus and HPV vaccinations. In this comparison, only 39.4% correctly indicated the association of HPV with cancers of the genitourinary organs, and 42.8% correctly indicated an association of HPV with papillary lesions of the genital organs. In addition, only 8.9% of parents indicated the answer "children" as a group exposed to HPV infection.


**Table 6.** Knowledge about the HPV virus and the HPV vaccine—correct answers to component questions.

N = number of parents who answered the question.

#### 3.3.1. Knowledge Regarding HPV

There were no significant statistical differences between the groups identified by the place of residence and the number of children. Considering the remaining characteristics (gender, age, education), parents with a higher education and women had more correct answers, which was statistically significant. Parents with a higher education had an overall higher test score and statistically more correct answers to questions about vaccine funding and vaccine availability in Poland and identified girls over the age of nine years as the target group for the vaccine. The number of correct answers decreased with the age of the parents.

#### 3.3.2. Knowledge Regarding Vaccine

Women scored a statistically significant higher percentage of correct answers regarding the availability of the vaccine in Poland and a statistically significant higher percentage of correct answers regarding vaccine funding and identified young girls before sexual initiation as the target group for the vaccine.

There were more statistically significant differences for incorrect vaccine responses between the education groups. People with a higher education had overall higher test scores and statistically answered more questions correctly about preventing or reducing the possibility of an HPV infection and about identifying the routes of transmission of the virus (through sexual intercourse, during natural childbirth). Women correctly identified most of the cervical cancer risk factors. The number of correct answers decreased with the age of the parents.

#### 3.3.3. Attitude towards Vaccination

The only factors differentiating the attitude towards vaccinations were knowledge and education. Parents with a test score in the third quartile indicated that the high effectiveness of the vaccine might influence their decision to vaccinate their children with the recommended vaccine. The remaining characteristics of the parents (sex, age, place of residence) did not significantly affect the attitude to vaccination. It was interesting that 55% of the parents would have vaccinated their children if the vaccination was covered by the government.

#### **4. Discussion**

This research is the first in Poland to assess the knowledge and attitudes of parents towards HPV vaccination. HPV vaccination may prevent diseases related to papillomaviruses, including cervical cancer. Research in this field has been conducted worldwide for many years [19–33], and an interesting analysis was conducted by Lopez et al. [18].

It should be emphasized, however, that the attitude of parents may be influenced by many factors, and one of the key influencing factors in Poland is the lack of recommendations and financing by the National Health Fund.

According to a study by Gerend et al., people recommended by the NHF to be vaccinated against HPV were forty times more likely to get vaccinated [34]. This is also confirmed by other studies [35–38].

The lack of recommendation and funding may have a negative impact on parents' attitudes, especially when the price of the vaccine is high. In these studies, over one-third of the respondents indicated the high price of the HPV vaccine as a possible reason for refusing to vaccinate their children.

However, it should be emphasized that in countries with recommendations and a reimbursement scheme, the problem of HPV vaccination is much more complex. As many as 28% of parents refuse to vaccinate in the USA, and 8% delay vaccination. The motivation of both groups for this type of behavior differs significantly. Parents who refused vaccinations gave reasons such as fear of promiscuity, lower vaccine efficacy expected, and higher expected side effects. Parents who delay vaccination do not rule out later vaccination after learning more about the vaccine [39]. In the study by Brabin et al., friends and school nurses (35% each) and teachers (20%) also influenced the views of girls about the HPV virus and HPV vaccination [40].

The state of parental knowledge has a crucial influence on the immunization of children. The study by Fishman et al. proved that mothers with more knowledge about HPV were not willing to vaccinate themselves or their daughters [41]. The present study found that the only factors that affect attitudes to vaccination are knowledge and education. The remaining characteristics of parents do not significantly affect the attitude to vaccination.

Lack of support from parents due to their fear of encouraging girls to engage in risky sexual behavior is also emphasized in the literature, reducing the motivation to vaccinate. Factors increasing parental acceptance of vaccination include HIV testing in the past, having an older daughter, having had more sexual partners, and having a family member with cancer [42]. The most common concerns indicated by parents in the available studies are related to vaccine safety, post-vaccination sexual promiscuity, moral problems related to sexuality, denial that the daughter is at risk, conservative and religious views, lack of knowledge, and unknown side effects [43].

An interesting aspect is a conversation with their children about HPV vaccinations as an introduction to sexuality. In the study by Marlow et al., mothers of the girls stated the age of their daughters at which they would like to start discussions with them about vaccinations, sex, cervical cancer, sexually transmitted diseases, HPV, and HPV vaccination (Table 7). The study shows that the age of the girls at which mothers would like to talk to them about HPV vaccines is statistically higher than the age of the girls at which mothers talk to them about general immunization, sex, and even higher for general discussions about HPV vaccines, the HPV virus, and sexually transmitted diseases [44].

**Table 7.** Average age of the discussion between parents and children on HPV-related topics (76).


The price of the vaccine is often discussed in the aspect of barriers that reduce vaccination. In the analysis, the low price of the vaccine was not a determinant for vaccinating children (0 answers). Most parents decided to vaccinate their child after a doctor recommended the vaccine or because the vaccine was very effective. However, in the case of HPV vaccination, more than one-third of the respondents indicated the high price of the HPV vaccine as a possible reason for not taking up vaccination (over two-thirds indicated possible side effects).

In the literature, concerns about the cost of the vaccine include costs to be borne by the parents and by healthcare professionals, and costs of the vaccine to be borne by the healthcare insurer [45]. Therefore, to increase vaccination availability in developing countries, vaccine companies have negotiated to lower the price of a single dose to 4.50–4.60 USD [46,47]. For comparison, in Poland, the price of a single dose of Cervarix and Silgard varies around 440 zloty for a single dose (around 110 USD). The price of the vaccine also affects the choice of the vaccine. Often, patients opt for a bivalent vaccine due to the lower cost.

This research is the first attempt in Poland to identify the most important barriers to the effective implementation of an HPV vaccination program, and thus to the prevention of diseases connected with this virus. The presented analysis may be of assistance in the implementation of HPV vaccination programs in Poland.

#### **5. Conclusions**

Parents' knowledge of the HPV virus is insufficient and depends on sex, age, and education and is independent of place of residence and the number of children. The parents' knowledge about the HPV vaccine is low and independent of the place of residence, the number of children, and depends on age, sex, and education. Further studies need to be carried out to provide information regarding pro-vaccine motivation tools.

**Author Contributions:** Conceptualization, K.S. and S.M.; methodology, S.M.; formal analysis, K.S.; investigation, K.S., A.D., S.D. and S.M.; resources, K.S. and S.M.; writing—original draft preparation, K.S., A.D., S.D. and S.M.; writing—review and editing, K.S. and A.D.; supervision, S.M.; project administration, K.S. and S.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** This study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Ethics Committee of the Medical University of Warsaw (AKBE/123/16).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interests.

#### **Appendix A**

*Questionnaire*

#### **1. Have you ever heard of the human papillomavirus (HPV) before?** *Select one answer*


#### **2. How did you learn about HPV?** *Select one answer*


#### **3. How can you get infected with HPV?**

*You can tick several options*


#### **4. Who is at risk of HPV infection?**

*You can tick several options*

• Only women


## **5. HPV infection predisposes to:**

*You can tick several options*


#### **6. Does HPV infection always lead to the manifestation of the disease?** *Select one answer*


#### **7. Do you know what the purpose of the Pap smear test is?** *Select one answer*


#### **8. What factors increase the risk of developing cervical cancer?**


#### **9. How can HPV infection be prevented, or the risk of HPV infection be reduced?** *You can tick several options*


#### **10. Is there a vaccine against HPV?**

*Select one answer*


#### **11. Is the HPV vaccine available in Poland?**

*Select one answer*


**12. For which sex are HPV vaccines registered in Poland?** *Select one answer*


#### **13. The target groups for the vaccine in Poland are:** *You can tick several options*


#### **14. The scientifically proven AEFI of HPV vaccination include:** *Select one answer*


#### **15. Does HPV vaccination give 100 percent protection against cervical cancer?** *Select one answer*


#### **16. Is the cost of the vaccine in Poland covered by the government?** *Select one answer*


## **17. What do you think about childhood vaccinations:**


#### **18. Do you have a child/children?**

*Select only one answer*


#### **19. Please enter the age of the child/children:**

*If you have more than one child, please state the age of all your children*


**20. Please select the gender of your child (ren)** *Please select only one answer per line*


#### **21. Have you vaccinated your child (ren) with obligatory vaccinations?** *Select only one answer*


#### **22. Why did you not vaccinate your/their child/children?** *Select only one answer*


#### **23. Did you vaccinate your child (s) with the vaccines recommended but optional (additionally payable)?**

*Select only one answer*


#### **24. What influenced your decision to vaccinate your child/you with the vaccine recommended but optional?**

*Select multiple answers*


## **25. Please select the opinion with which you agree.**

*Select multiple answers*


**26. Have you vaccinated your children against HPV?** *Select only one answer*


**27. What influenced your decision to vaccinate your child/you with the vaccine against HPV?**

*Select multiple answers*


#### **28. Are you willing to vaccinate your children against HPV? Select only one answer**


#### **29. What factors may influence your decision not to vaccinate yours? Children against HPV?**

*Select multiple answers*


#### **30. Would you vaccinate your child against HPV if vaccination was covered by the government?**

*Select only one answer*


#### **31. Please enter your gender:**

*Select only one answer*


**32. Please enter your age: 33. Please provide your education:** *Select only one answer*


#### **34. Size of the place of residence** *Select only one answer*


**35. How old are you? 36. Please select your gender:**


#### **References**


## *Review* **The Efficacy of Therapeutic DNA Vaccines Expressing the Human Papillomavirus E6 and E7 Oncoproteins for Treatment of Cervical Cancer: Systematic Review**

**Ayazhan Akhatova 1, Chee Kai Chan 2,3, Azliyati Azizan 2,4 and Gulzhanat Aimagambetova 2,\***


**Abstract:** Cervical cancer is recognized as a serious public health problem since it remains one of the most common cancers with a high mortality rate among women despite existing preventative, screening, and treatment approaches. Since Human Papillomavirus (HPV) was recognized as the causative agent, the preventative HPV vaccines have made great progress over the last few years. However, people already infected with the virus require an effective treatment that would ensure long-term survival and a cure. Currently, clinical trials investigating HPV therapeutic vaccines show a promising vaccine-induced T-cell mediated immune response, resulting in cervical lesion regression and viral eradication. Among existing vaccine types (live vector, protein-based, nucleic acid-based, etc.), deoxyribonucleic acid (DNA) therapeutic vaccines are the focus of the study, since they are safe, cost-efficient, thermostable, easily produced in high purity and distributed. The aim of this study is to assess and compare existing DNA therapeutic vaccines in phase I and II trials, expressing HPV E6 and E7 oncoproteins for the prospective treatment of cervical cancer based on clinical efficacy, immunogenicity, viral clearance, and side effects. Five different DNA therapeutic vaccines (GX-188E, VGX-3100, pNGVL4a-CRT/E7(detox), pNGVL4a-Sig/E7(detox)/HSP70, MEDI0457) were well-tolerated and clinically effective. Clinical implementation of DNA therapeutic vaccines into treatment regimen as a sole approach or in combination with conservative treatment holds great potential for effective cancer treatment.

**Keywords:** cervical cancer; cervical intraepithelial neoplasia; HPV; E6 oncoprotein; E7 oncoprotein; therapeutic vaccine; DNA vaccine; DNA therapeutic vaccine

#### **1. Introduction**

Cervical cancer is a largely preventable cancer of the cervix, which is the narrow part of the lower uterus that connects to the vagina. According to the World Health Organization (WHO) statistics, cervical cancer became the fourth most frequent cancer in women in 2018, with 570,000 cases, which represent 6.6% of all female cancers worldwide [1]. Similarly, cervical cancer is the 2nd most common type of cancer among females, and the 4th most common cause of cancer-related deaths (8.5%) among women in Kazakhstan [1].

Previous studies have established the strong causative association between persistent infection with certain high-risk Human Papillomavirus (HPV) types and the development of cervical cancer [2]. HPV is a small, non-enveloped deoxyribonucleic acid (DNA) tumor virus, which primarily affects human vaginal and oral mucosa [2]. There are more than

**Citation:** Akhatova, A.; Chan, C.K.; Azizan, A.; Aimagambetova, G. The Efficacy of Therapeutic DNA Vaccines Expressing the Human Papillomavirus E6 and E7 Oncoproteins for Treatment of Cervical Cancer: Systematic Review. *Vaccines* **2022**, *10*, 53. https:// doi.org/10.3390/vaccines10010053

Academic Editor: Gloria Calagna

Received: 6 December 2021 Accepted: 27 December 2021 Published: 31 December 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

100 HPV subtypes, which differ by less than 3% of their genome [3]. The most prevalent oncogenic subtypes in both symptomatic (50–70%) and asymptomatic (20–30%) women diagnosed with invasive cervical cancer are HPV-16 and HPV-18 [3].

About 90% of deaths from cervical cancer occurred in low- and middle-income countries largely due to the lack of proper prevention, early diagnosis, and effective screening [1]. In 2018, WHO started a new campaign to decrease the incidence rate, with the aim to eventually eradicate cervical cancer [4]. The campaign included three key steps, which are vaccination of 90% of girls by 15 years of age, screening provision of 70% of women by 35 years of age and again by 45, and treatment of 90% of women with diagnosed cervical neoplasia [5]. Ideally, if all countries accomplished the requirements of the campaign by 2030, it is expected to decrease the incidence of new cases by 40% and 5 million related deaths by 2050 [1]. However, it is now estimated that the annual number of new cases of cervical cancer would be increasing to 700,000, and the number of deaths would reach 400,000 by 2030 [4]. Such an increase is explained by the uneven provision of screening and vaccination among countries, since these actions have occurred mostly in high-income settings [4]. In high-income countries, screening programs cover 60% of the female population, while, in lower-middle-income countries, the figure is only 20% [4]. Although preventative measures are expected to be effective in the elimination of cervical cancer in the long term, the situation now requires a short-term solution for those already in need of better treatment and care.

Nowadays, early-stage cervical intraepithelial lesions (CIN) are treated by means of surgical resection of cancerous tissue, which include conization, loop electrical excision procedure (LEEP), and radical hysterectomy [6]. These already traumatizing procedures can be coupled with radiotherapy or chemotherapy for the purpose of treatment enhancement and prevention of relapse [6]. Since radiotherapy and cytotoxic chemotherapy target not only cancerous tissue but surrounding tissues as well, patients often suffer constitutional side effects such as fatigue, loss of appetite, nausea, hair loss, or adverse events (AEs) that negatively impact patient's quality of life like anemia, neutropenia, thrombocytopenia, neuropathy, nephro-/hepatotoxicity, premature menopause, and infertility [6]. Therefore, it is necessary to provide less toxic and traumatic treatment options, especially for patients with comorbidities. After surgical excision, quadrivalent HPV vaccination could be used for CIN2+ cervical lesions to reduce the risk of recurrent disease [7].

Therapeutic vaccines such as TheraCys, PROVENGE, and IMLYGIC used for the treatment of urothelial carcinoma in situ, prostate cancer, and advanced melanoma respectively with promising results [8]. These cancer vaccines showed greater median overall survival compared to the conservative chemotherapy approaches, resulting in the United States Food and Drug Administration (FDA) approval [8]. The development of effective therapeutic vaccines for precancerous cervical lesions and cervical cancer treatment and their implementation into clinical practice would be a huge improvement in gynecologic oncology.

Currently, HPV therapeutic vaccines under investigation include live vector vaccines (bacterial and viral vectors), subunit vaccines (peptides and protein-based vaccines), plant peptide/protein-based vaccines, nucleic acid vaccines (DNA and ribonucleic acid (RNA) replicon-based vaccines), and cell-based vaccines (dendritic cell-based vaccines and adoptive cell transfer) [2]. Among these subtypes, we are particularly interested in DNA therapeutic vaccines, since they are safe, cost-efficient, thermostable, easily produced in high purity, and distributed [2]. Unlike live vector vaccines, DNA therapeutic vaccines do not evoke neutralizing antibody production, thus allowing for repeated vaccination [9]. T cell-mediated immune response is achieved by targeting HPV E6 and E7 proteins, as they are solely responsible for the malignant transformation of cervical tissue [9]. Moreover, in a study by Daayana et al., strong adaptive immune responses to E6 and E7 were reported, and it was shown to be greater than previously reported immune responses to therapeutic HPV vaccines [2,10]. Thus, the immunogenicity of DNA vaccines expressing the HPV E6 and E7 oncoproteins needs to be further investigated. Currently, the majority of DNA therapeutic vaccines are undergoing clinical trials to evaluate their safety and stability. Therefore, the aim of this article is to assess and compare existing DNA therapeutic vaccines, which are evaluated in phase I and II trials, expressing HPV E6 and E7 oncoproteins for the prospective treatment of cervical cancer based on clinical efficacy, immunogenicity, viral clearance, and side effects.

#### **2. Materials and Methods**

This systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews (PRISMA) statement [11]. The study was registered in the PROSPERO database and confirmed with a registration code of CRD42021251476.

#### *Systematic Literature Search and Eligibility Criteria*

Articles were manually searched using databases as PubMed/MEDLINE, Google Scholar, and clinicaltrials.gov published in English from the year 2010.

The search was performed using the following keywords: "cervical cancer", "cervical intraepithelial neoplasia", "HPV", "HPV-positive", "E6 and E7 oncoproteins", "therapeutic vaccine", "DNA vaccine", and "DNA therapeutic vaccine". We used the medical subject heading (MeSH) term "Uterine Cervical Neoplasms" (MeSH Unique ID D002583) as major topic and "Vaccines" (MeSH Unique ID D014612), "E6 protein, HPV type 18" (MeSH Unique ID C052603) and "E7 protein, HPV type 16" (MeSH Unique ID C059731).

The search was narrowed by using "Cervical cancer OR Cervical Intraepithelial Lesion AND DNA therapeutic vaccines", "DNA therapeutic vaccines AND E6 OR/AND E7 oncoproteins". The selected studies were independently reviewed for inclusion eligibility by two reviewers (Akhatova and Aimagambetova) using standardized data collection forms. The following data were collected from the studies: the author, year of publication, number of study participants, vaccine administration strategies, and the main outcomes (clinical efficacy, viral clearance, immunogenicity, adverse events). Any discrepancy in the assessment of articles was resolved by discussion and consensus, as well as input from the third and fourth reviewers (Chan and Azizan).

The articles were selected to meet the following eligibility requirements to be included in the study: (1) research article, (2) human subject research, and (3) the study of DNA therapeutic vaccines targeting HPV E6 and E7 oncoproteins. The presence of the following did not allow for the study to be included: (1) reviews and case reports, (2) irrelevance to cervical cancer or CIN, (3) mouse model studies, (4) articles on preventative HPV vaccines, and (5) the use of the inappropriate methodology. Abstracts lacking full information about predefined criteria were excluded without further review.

The types of studies included were phase I and phase II clinical trials that were initiated and completed between 2003 and 2017, studying the clinical efficacy of DNA therapeutic vaccines expressing HPV E6 and E7 oncoproteins for the treatment of cervical intraepithelial lesions of grades 2 and 3 both newly diagnosed and recurrent malignancies. The treatment of the lesion may or may not be followed by conization or loop electrosurgical excision procedure. The study population was female patients aged 18 or older with histopathologically diagnosed CIN of grades 2 and 3, known to be caused by HPV 16 and/or HPV 18 based on polymerase chain reaction (PCR) amplification results. The main outcomes of the review were clinical efficacy based on the lesion regression, viral load reduction, immunogenicity, in particular, HPV E6 and E7 specific CD8+ T cell response, and AEs after vaccination.

#### **3. Results**

#### *3.1. Study Identification and Selection*

During this study, 120 articles were identified through PubMed/MEDLINE and Google Scholar searching platforms (Figure 1).

**Figure 1.** Flow-chart diagram of study selection.

Eighty-three articles were excluded based on the abstract, representing literature reviews. From the remaining 37 articles, 24 articles were excluded at this stage: 11 studies were mouse model-based, 9 studies were not addressing the study question, 2 studies were regarding the preventative vaccines, and 2 studies included additional oncogenes in the development of therapeutic vaccines. We included 6 studies performed between 2003 and 2017 in our systematic review after the exclusion of 7 articles studying the effects of therapeutic vaccines with either viral, bacterial, or peptide vectors [12–17]. These six studies represented the work completed in the United States of America, Korea, Estonia, South Africa, India, Canada, Australia, and Georgia [12–17] (Table 1).

*Vaccines* **2022**, *10*,53


**Table1.**Clinicaltrials.

*Vaccines* **2022**, *10*,53



dermal delivery—PMED; cervical intralesional—IL; positron emission tomography—PET; computer tomography—CT; calreticulin—CRT; heat shock proteins—HSPs; squamocolumnar junction—SCJ; viral load—VL; chemoradiotherapy—CRT; squamous cell carcinoma—SCC; adenocarcinoma—AC; adenosquamous cell carcinoma—ASCC; loop electrosurgical excision procedure—LEEP; external beam radiation therapy—EBRT.

Therapeutic vaccines were evaluated based on clinical efficacy (histopathological regression of the lesion to CIN < 1), viral clearance, immunogenicity, and adverse events after the vaccination. Subjects, female patients aged 18 or older with histopathologically diagnosed cervical intraepithelial neoplasia of grades 2 and 3, known to be caused by HPV 16 and/or HPV 18, received DNA therapeutic vaccines in different dose formulations and were evaluated generally within 20 weeks (36 weeks in extended trial groups) for the effects of vaccines mentioned above. Study populations ranged from 10 patients to 167, median age mostly being 21–30, except for Hasan et al. (2020) study, where the median age of participants was 51.50 years old due to more advanced stages of cancer [17].

#### *3.2. Outcomes*

#### 3.2.1. Clinical Efficacy

Clinical efficacy was evaluated according to the histopathological regression to CIN ≤1, which is less than one-third of the thickness of the cervical epithelium, on a colposcopyguided biopsy 15, 20, or 36 weeks after the first injection. All six studies [12–17] report tumor size decrease to some extent (Table 2).

GX-188 E in phase I trial by Kim et al. [13] showed a 78% success rate of complete response both histologically and virologically. The same vaccine in the phase II trial by Choi et al. (2019) resulted in histopathological regression to CIN < 1 in 52% of patients 20 weeks and 67% of patients 36 weeks after the first dose [12]. MEDI0457 by Hasan et al. showed 87.5% of complete response to the vaccine and 1 patient had a partial response to the treatment [17]. pNGVL4a-CRT/E7(detox) [15] and pNGVL4a-Sig/E7(detox)/HSP70 [16] vaccines both showed a similar response rate of around 30%.

#### 3.2.2. Viral Load Clearance

Viral load was measured by means of PCR amplification to assess the clearance of HPV DNA from the cervical biopsy after vaccination. Choi et al. established that HPV clearance was associated with the histopathologic regression as 77% of regressors had no trace of HPV DNA, while only 12% of non-regressors had no viral load in the tissue biopsy [12]. Kim et al. results show that GX-188E takes time to clear off the virus [13]. MEDI0457 [17] and VGX-3100 [14] report the association between viral clearance and tumor size reduction, whilst pNGVL4a-CRT/E7(detox) [15] did not result in any difference between pre- and post-treatment viral load.

#### 3.2.3. Immunogenicity

Immunogenicity is one of the key features of the therapeutic vaccines as it represents the potential of the vaccine to induce virus-specific T cell response, in particular HPV E6 and E7 specific CD8+ T cell immune response. IFN-γ response was measured by means of *ex vivo* ELISpot assay with cryopreserved and thawed peripheral blood mononuclear cells (PBMCs) at pre- and post-treatment stages. The vaccine response is considered positive when the increase in T-cell frequency was at least three times greater compared to the study entry measurement. GX-188E both in phase I [13] and phase II [12] studies showed a significant increase in IFN-γ response, which was correlated with the histopathologic regression and viral clearance. Moreover, an E6 specific response was more pronounced than E7 specific [12,13]. VGX-3100 induced 9.5 times greater IFN-γ response in the treatment group compared to the placebo, which lasted as long as 24 weeks post-vaccination [14]. On the contrary, MEDI0457 induced a greater response to E7, particularly in newly diagnosed cohort 1 that persisted up to 48 weeks [17]. In cohort 1, 4 of 7 patients exhibited IFNγ-producing spots exceeding 100 SFU/106 PBMC, whereas no patients produced similar responses in cohort 2 [17]. pNGVL4a-CRT/E7(detox) and pNGVL4a Sig/E7(detox)/HSP70 showed minimal dose-dependent immune response, which was remarkable from the unvaccinated group [15,16].




dysplastic tissue in 85–90% of women.

*Vaccines* **2022**, *10*, 53


biopsy—CB; maximum tolerated dose—MTD;

 cervical

intralesional—IL;

calreticulin—C-RT;

 heat shock

proteins—HSPs;

squamocolumnar

junction—SCJ;

 viral load—VL.

#### 3.2.4. Toxicity/Adverse Events

Overall, all vaccines were well-tolerated without vaccine-related serious adverse events. The most common adverse events were injection site pain and erythema, as well as constitutional symptoms (malaise, myalgia, and headache) [12–17]. No serious adverse events (Grade 3/4) related to the vaccination were reported. No dose-limiting toxicities were observed.

#### **4. Discussion**

This systematic review summarizes the findings of phase I and phase II clinical trials investigating the treatment of patients with histopathologically diagnosed CIN associated with HPV 16 or/and HPV 18 with DNA therapeutic vaccines. Six studies have demonstrated immunologic response in the form of lesion size regression, viral clearance, and increased T cell response of five different DNA vaccines–GX-188E (phase I and phase II), VGX-3100, pNGVL4a-CRT/E7 (detox), pNGVL4a-Sig/E7 (detox)/HSP70, MEDI0457. Vaccines were plasmid DNA encoding for either non-oncogenic E6/E7 or both, and chaperonin proteins such as HSP 70 and Calreticulin for the enhancement of the uptake by antigen-presenting cells, and MHC class I processing and presentation. MEDI0457 [17] had the same plasmid formulation as VGX-3100 [14] combined with plasmid encoding IL-12. All vaccines were well tolerated by patients, leading to only grade 1 or less systemic and local side effects.

Previous reviews have studied various existing therapeutic vaccines including live vectors, plant-based, protein, whole cell, and combinatorial vaccines [18]. This is the first systematic review of DNA therapeutic vaccines against cervical cancer expressing HPV16 and HPV18 E6 and E7 oncogenes. The feasibility of production, storage, and transportation, cost-effectiveness, the capability of multiple immunizations, and targeting different costimulatory genes provided the rationale for the study of DNA therapeutic vaccines [18]. However, comparatively weak immunogenicity and the risk of integration into the host genome are the main concerns, which could be addressed by modification of E6 and E7 to abolish its transformative capacity [18]. There are approaches of boosting the potency of DNA vaccines, such as increasing the number of antigen-expressing dendritic cells (DCs) by using a gene gun delivery method, enhancing antigen processing and presentation in dendritic cells via codon optimization, and improving the DCs and T-cell interaction [18]. These strategies were used in our selected studies, which led to increased antigen-specific, activated CD8+ T cell response in all of them. Patients with CIN2/3 were more likely to induce E6 and E7 specific CD8+ immune response, according to the IFNgamma ELISPOT results, compared to the invasive cervical cancer [17]. According to Hasan et al., diminished immune response in more advanced disease stages is associated with immune exhaustion, the effect of chemoradiation and selection of patients with diminished immunity against HPV [17]. The strongest evidence of the immunogenicity of DNA therapeutic vaccine VGX-3100 was observed by the increased intensity of CD8+ infiltrates in histopathologically regressed patients compared to the placebo group with regressed lesions [14].

DNA therapeutic vaccines were also assessed based on their clinical efficacy, i.e., the ability to induce cervical lesion regression. The regression to ≤CIN1 among study participants was observed in all studies with significantly varying degrees. The study of VGX-3100 vaccine with both treatment and placebo groups showed a response rate of 49.5% vs. 30.6%, respectively [14]. Meanwhile, GX-188E vaccine has resulted in histopathological regression in 67% of patients in both phase I study and phase II studies [12,13]. Choi et al. [12] have observed an enhanced response to GX-188E over time up to 83% among those with cervical lesions <50%, probably due to the enhanced memory T cell-driven therapeutic effect. The difference in clinical benefit between VGX-3100 and GX-188E could be explained with the recruitment of CIN3 HPV-positive patients only, the lack of placebo group, and the small number of participants in the latter. pNGVL4a-CRT/E7 (detox) and pNGVL4a-Sig/E7 (detox)/HSP70 had the lowest clinical efficacy of approximately 30% response rate among

all [15,16]. However, the effect of these two vaccines on the lesion regression is questionable, as this rate is similar to spontaneous remission rate over a 15-week period [15].

It was established that women, after excision of the cervical lesion, are more likely to have a relapse; therefore, viral clearance is a key factor of vaccine efficacy [18]. VGX-3100, GX-188E, and MEDI0457 effectively cleared detectable HPV DNA, which was significantly associated with histopathological regression [12–14,17]. In contrast, pNGVL4a-CRT/E7 (detox) has not resulted in viral load reduction [15].

Nevertheless, there are several limitations to this study. Firstly, limited data exist on the topic of DNA therapeutic vaccines, as not a single therapeutic vaccine against cervical cancer was approved. All these clinical trials were in either the phase I or phase II stage of assessing the efficacy and safety in humans. Secondly, the majority of studies enrolled a small number of participants without masking, stratification, or the control group, which poses a potential risk for bias. As vaccines investigated in this study had different structural designs, it was not feasible to make a statistical analysis of vaccine outcomes; therefore, qualitative analysis was performed overall.

#### **5. Conclusions**

As it was stated by WHO, a global strategy to accelerate the elimination of cervical cancer, 90–70–90 targets for prevention, screening, and treatment are the key to success. Preventative bivalent and quadrivalent HPV vaccines have undergone significant advancement in development and implementation. However, these preventative vaccines do not elicit a therapeutic effect but could be used as an adjuvant to surgical treatment. DNA therapeutic vaccines represent a potentially safe and novel approach to cervical cancer treatment. The main goal of this review was to discuss the effectiveness of existing DNA therapeutic vaccines against cervical cancer expressing HPV 16/18 oncoproteins E6 and E7. The idea of DNA therapeutic vaccines is inducing an adaptive immune response and immunologic memory via the expression of tumor antigens and activation of antigen-presenting cells.

DNA therapeutic vaccines are currently undergoing clinical trials to improve the potency of therapeutic vaccines and clinical efficacy using strategies as a modifying route of administration, adjuvant therapy, prime-boost regimen, and co-administering with other drugs for a synergistic effect. Nowadays, despite the treatment of locally advanced disease with chemoradiation, patients have a high recurrence rate and a poor 5-year survival rate, estimated at 50% and 70%, respectively. In contrast, the MEDI0457 vaccine, which contained VGX-3100 plasmid coupled with an IL-12 expression plasmid to promote T-cell function, evaluated the disease progression-free survival (PFS) at 12 months, which was estimated as 88.9% overall. These findings strengthen the hypothesis that DNA therapeutic vaccines could effectively induce *de novo* or boost existing immune responses. Moreover, studies have shown that using femtosecond laser treatment could also improve transfection efficiency administered intradermally and into the lesion *in vivo*. Thus, continuous efforts to improve the efficacy of DNA therapeutic vaccines and implementation of therapeutic vaccines into a treatment regimen as a sole approach or in combination with conservative treatment may greatly improve the current situation.

**Author Contributions:** A.A. (Ayazhan Akhatova)—data collection; A.A. (Ayazhan Akhatova) and G.A.—compiled, analyzed, and reviewed the data. A.A. (Ayazhan Akhatova) and G.A. prepared the manuscript draft. C.K.C. and A.A. (Azliyati Azizan) reviewed the final manuscript. All authors reviewed and approved the final article. All authors have read and agreed to the published version of the manuscript.

**Funding:** There is no funding source to declare.

**Institutional Review Board Statement:** Due to the nature of the study (systematic review), ethical approval is not required.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The datasets used and/or analyzed during the current study available from the corresponding author on reasonable request.

**Acknowledgments:** The authors would like to acknowledge the Nazarbayev University School of Medicine for the support that enabled the completion of this review.

**Conflicts of Interest:** The authors have no conflict of interest to declare.

#### **References**


## *Systematic Review* **Pharmacists' Perceived Barriers to Human Papillomavirus (HPV) Vaccination: A Systematic Literature Review**

**Oluwafemifola Oyedeji 1, Jill M. Maples 2,\*, Samantha Gregory 2, Shauntá M. Chamberlin 3, Justin D. Gatwood 4, Alexandria Q. Wilson 5, Nikki B. Zite <sup>2</sup> and Larry C. Kilgore <sup>2</sup>**


**Abstract:** About 45:000 cancers are linked to HPV each year in the United States alone. The HPV vaccine prevents cancer and is highly effective, yet vaccination coverage remains low. Pharmacies can play a meaningful role in increasing HPV vaccination access due to their availability and convenience. However, little is known about pharmacists' perceived barriers to HPV vaccination. The objective of this systematic review was to summarize existing literature on perceived barriers to administering HPV vaccination reported by pharmacists. Barriers identified from selected studies were synthesized and further grouped into patient, parental, (pharmacist's) personal, and system/organization barrier groups. Six studies were included in this review. The cost of the HPV vaccine, insurance coverage and reimbursement were commonly reported perceived barriers. Adolescent HPV vaccination barriers related to parental concerns, beliefs, and inadequate knowledge about the HPV vaccine. Perceived (pharmacist's) personal barriers were related to lack of information and knowledge about HPV vaccine and recommendations. At the system/organization level, barriers reported included lack of time/staff/space; difficulty in series completion; tracking and recall of patient; perceived competition with providers; and other responsibilities/vaccines taking precedence. Future strategies involving pharmacy settings in HPV-related cancer prevention efforts should consider research on multilevel pharmacy-driven interventions addressing barriers.

**Keywords:** Human Papillomavirus; barriers; pharmacists; vaccination

#### **1. Introduction**

Human Papillomavirus (HPV) is estimated to be the cause of 70% oropharynx, vaginal, and vulvar cancers, 60% of penile cancers, and 90% of anal and cervical cancers [1]. HPVassociated cancers are the only known cancers that can be prevented by receiving a vaccine. Yet, there are over 45,000 newly diagnosed HPV-related cancers in the United States each year [1]. The HPV vaccine has been shown to be safe and highly effective [2]. The Centers for Disease Control and Prevention (CDC) established HPV vaccination as a public health priority, yet vaccination coverage rates are less than the Healthy People 2020 goal that 80% of adolescents complete the vaccine series [3,4]. Nationally in 2017, less than 66% of adolescents received the first dose of the HPV vaccine, and only about 49% completed the series [5]. The HPV vaccine dosing schedule consists of a series of two or three doses, depending on the age of the patient at the start of the schedule [6]. For example, a patient that is under the age of 15 is recommended to receive two doses of the HPV vaccine

**Citation:** Oyedeji, O.; Maples, J.M.; Gregory, S.; Chamberlin, S.M.; Gatwood, J.D.; Wilson, A.Q.; Zite, N.B.; Kilgore, L.C. Pharmacists' Perceived Barriers to Human Papillomavirus (HPV) Vaccination: A Systematic Literature Review. *Vaccines* **2021**, *9*, 1360. https:// doi.org/10.3390/vaccines9111360

Academic Editor: Gloria Calagna

Received: 1 November 2021 Accepted: 16 November 2021 Published: 19 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

administered 6–12 months apart. A patient starting the vaccine series on or after the 15th birthday is recommended to receive three doses of the HPV vaccine [6]. In this dosing schedule, the second vaccine should be administered 1–2 months after the first dose, and the third dose should be administered 6 months after the first dose [6].

Community pharmacies have the potential to play a meaningful role in increasing HPV vaccination rates. The President's Cancer Panel [7] and National Vaccine Advisory Committee [8] released statements urging the increase in uptake of HPV vaccination rates by using pharmacies as a strategic site. Pharmacies are conveniently located for most families, including those in rural communities. In fact, most residents in the United States (91%) live within five miles of a pharmacy [9]. Additionally, pharmacies are typically open for extended hours and on weekends. Given the expanding scope of practice and evolving role of the pharmacist and pharmacy as playing a vital role in public health, due to their accessibility, cost-efficacy, and ability to provide education and shared responsibility, along with patient acceptance, they have been identified as crucial partners in vaccination administration [10].

During the ongoing COVID-19 pandemic, the U.S. Department of Health and Human Services (HHS) declared that pharmacists are allowed to administer childhood vaccines for children that are three years and above [11]. Most state laws also allow pharmacists to administer HPV vaccine to patients, with some states having varying age restrictions and/or prescription requirements (only two states do not allow pharmacists to administer the HPV vaccine at all) [12]. Pharmacists provide a convenient option for patients to receive the HPV vaccine. However, little is known about the current perceptions and barriers to administrating HPV vaccination among pharmacists. Therefore, this review aims to systematically examine the literature on pharmacists' perceived barriers to administering the HPV vaccine as a mechanism for improving HPV vaccination uptake.

#### **2. Methodology**

This systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (PRISMA) [13].

#### *2.1. Search Strategy*

Electronic searches were created and completed by a research team member and health sciences librarian (A.W.) experienced with systematic searches in the following databases: PubMed [NLM], Web of Science [Thomson Reuters], Cochrane Central Register of Controlled Trials [Wiley], and Scopus [Elsevier]. The search strategy was created first in PubMed (Appendix A), and then translated for each database platform as applicable. MeSH terms and keywords were used to search concepts related to Human Papillomavirus, vaccines, barriers, and pharmacists. The searches were performed on 18 September 2020. Results were restricted to English language. There were no restrictions set on the year of publication for inclusion. Duplicate references were removed using Rayyan software [14]. The references cited in the included articles were reviewed for additional relevant articles.

#### *2.2. Study Selection & Data Extraction*

The titles and abstracts were first screened independently for eligibility by two researchers (S.G., O.O.). Then, the same researchers reviewed the full text of selected abstracts for further eligibility. Conflicts were resolved by a third researcher (J.M.M.). Study inclusion criteria include: (1) primary research article including qualitative, quantitative, and mixed-method studies, (2) reported outcome included perceived barrier to administering HPV vaccine (3) study population included pharmacists, pharmacy students, pharmacy technicians, and pharmacist representatives. Studies were excluded if the study objectives were unrelated to HPV vaccination. Two co-authors (O.O., J.M.M.) abstracted and synthesized all findings. Barrier type categorization was adapted from the findings reported in the included studies [15–17]. Some barriers extended, at least partially, across other barrier types. All barrier type classifications were discussed among co-authors, and a

consensus was made for the final classification. Another co-author (S.M.C) reviewed the findings independently for accuracy. Any conflicts were resolved by deliberations among the co-authors.

#### **3. Results**

#### *3.1. Literature Search and Study Characteristics*

The electronic database searches yielded 444 articles; 292 titles and abstracts were selected, and 13 full-text studies were reviewed for inclusion. In the end, 6 out of 13 studies were selected for this review using the exclusion and inclusion criteria [15–20]. Figure 1 illustrates the PRISMA diagram detailing the search and selection process. Four of the studies [15,16,18,20] examined barriers related to HPV vaccine alone, while the remaining two studies [17,19] included other vaccines in addition to HPV. All included studies were carried out in the United States; however, geographical location and pharmacy setting differ across studies (Table 1). Table 2 presents an overview of the study designs, methodologies, and data reporting. While all the studies utilized a cross-sectional study design, the methodologies and results reporting varied widely across studies (Table 2). Table 3 presents a summary of the key findings for the included studies. These findings were further grouped by level of barrier into patient, parental, (pharmacist's) personal, and system/organization barrier groups, which were adapted from the findings reported in the included studies (Table 4) [15–17]. Some barriers extended, at least partially, across other barrier types. For example, "financially-related" (or cost) barriers were reported across multiple barrier types [15,17–20]. To the patient a "financially-related" (or cost) barrier may be the out-of-pocket cost of the vaccine, which is distinct from the "financially-related" (or cost) barrier at the systems/organizational level, which may be more related to the financial burden of stocking the vaccine.

**Figure 1.** PRISMA diagram showing search and selection.


#### **Table 1.** Characteristics of included studies.

Td-Tetanus/Diphtheria, Tdap-Tetanus/Diphtheria/Pertussis, MMR- Measles/Mumps/Rubella, Other- Typhoid, Yellow fever, Japanese Encephalitis and not otherwise stated.

#### **Table 2.** Overview of included study methodologies and data reporting.



#### **Table 3.** Key findings of included studies.


#### **Table 3.** *Cont.*


#### **Table 4.** Summary of key barriers grouped by level of barrier.



**Table 4.** *Cont.*

#### *3.2. Barrier Levels*

Patient: Four out of the six included studies reported patient-related barriers [15–17,19]. The most commonly reported barriers at the patient level were inadequate demand [15,16] and patient refusal [17]. Reasons for patient refusal reported in one of the studies included perceived safety concerns and financial reasons [17]. Other barriers included patients lacking insurance coverage [17] and difficulty overcoming financial barriers for an adolescent [19].

Parental: Three studies included parent-related barriers [15,18,20]. Across these studies, certain parental concerns and perceptions were reported as pharmacists perceived barriers to HPV vaccination. This included safety and efficacy concerns about the HPV vaccine [15], concerns that agreeing to vaccination means they are condoning premarital sex [15], and concerns that their children will engage in riskier sexual behaviors if they receive the vaccine [15]. Parental beliefs that their children are not at risk for HPV infection and that children are not old enough for the HPV vaccine were also reported as barriers [15]. Other key parent-related barriers reported include inadequate demand from parents [20], parental believe the cost is too high [15], perceived stigma about vaccination among parents of adolescents [18], parental consent [18], lack of knowledge [20], and inadequate education/understanding about HPV infection [15].

(Pharmacist's) Personal: Three studies reported pharmacist's personal barriers to HPV vaccination. [16,18,19]. In one of the studies, lack of information was reported as a barrier [16]. This included inadequate knowledge to educate and recommend the HPV vaccine and also a lack of information on subjects such as cost and storage of the vaccine [16]. Misinformation among pharmacists regarding HPV vaccination and administration was also reported as a barrier [16]. For instance, one pharmacist believed (incorrectly) that the HPV vaccination coverage in their (rural) area was good [16]. Another example of misinformation was evidenced by a pharmacist stating that Medicaid does not allow pharmacists to provide the HPV vaccine to those under 18, even with a prescription [16]. A third example of misinformation was evidenced by a pharmacist stating that adolescents are supposed to receive the HPV vaccine only at their doctor's office [16]. The perception that HPV infection/mode of transmission is a sensitive subject was also cited as a barrier [16]. Other personal barriers include safety concerns about the potential adverse reaction after receiving the HPV vaccine [16,18], and the belief that the Advisory Committee on Immunization Practices (ACIP) recommendations are controversial [19].

System/Organization: Five out of the six studies reviewed reported system/organization level barriers [15–18,20]. Three of these studies reported lack of time and staffing as barriers to HPV vaccine administration [16,17,20]. For example, often times only one pharmacist may be present, which could result in major interruptions in work flow to stop and administer an HPV vaccine [16]. Two studies reported that a lack of physical space was also a barrier [16,20]. For example, a lack of consulting space to accommodate both parent and adolescent for HPV vaccine counseling and/or administration is a barrier [16]. Financial barriers related to vaccination cost, insurance reimbursement, and compensation were also frequently cited [15,17,18,20]; this included lack of insurance coverage of vaccine costs [15], difficulty in knowing whether insurance will reimburse for vaccine [17], and adequate compensation for vaccine purchase/supplies purchase/vaccine administration [17]. Other cost-related barriers included upfront cost of buying vaccine/supplies and lack of demand to justify cost of stocking vaccines [17].

At the organization level, other responsibilities/vaccines taking precedence over HPV vaccination was also reported as a barrier [17,20]. Other reported barriers were related to series completion which included lack of incentives for series completion [15,17], tracking and recall of patients [18], and remembering to screen patients for vaccine [17].

In one study, concerns that it may appear that pharmacists are in competition with health care providers to administer the vaccine was seen as a barrier [16]. Liability issues relating to adverse effects after vaccination [16], lack of educational materials to provide to parents [20]; and inadequate vaccine promotion/education were also reported as barriers [18].

#### **4. Discussion**

Community pharmacies could potentially play an important role in improving HPV vaccination rates due to their availability and accessibility. This review presents a summary of the literature on pharmacists' perceived barriers to providing the HPV vaccine in a pharmacy setting. Based on the findings from the six included studies, pharmacists' perceived barriers to HPV vaccination uptake exist at the patient, parental, pharmacist's (personal), and system/organization levels. The existence of reported barriers at different levels suggests that efforts at improving HPV vaccination rates among pharmacies should be targeted at these levels.

Barriers related to the cost of the HPV vaccine and insurance coverage were frequently cited in the articles reviewed for this paper [15,17–20]. This is in concert with previous studies where vaccine cost and lack of insurance coverage have also been reported as specific barriers to HPV vaccination among health care providers [23,24]. Other studies have also reported that financial reasons may be a parental barrier for HPV vaccination for their children [25]. In fact, one study found that a lower proportion of parents were willing to vaccinate their children if the vaccine was not covered by insurance [25]. A number of options are available to cover the cost of HPV vaccine. Under the Affordable Care Act (ACA), most private insurance plans in the United States are required to cover immunizations recommended by ACIP without consumer cost-sharing [7]. Additionally, the Vaccine for Children (VFC) program is a federally funded program in the United States that covers vaccine costs for eligible children below 19 years of age (eligibility criteria include Medicaid-eligible, uninsured, underinsured, and American Indian or Alaska Native) [26]. A study among physicians reported that those participating in VFC program were less likely to indicate cost as a barrier to HPV vaccination [24,27]. Previous research showed that the VFC program led to increased vaccination for certain vaccines [28]. However, barriers such as state laws (only 34 states allow pharmacies to participate in VFC), inadequate reimbursement to cover actual costs, administrative burden and low demand from eligible persons may affect pharmacies' participation in VFC or willingness to carry all vaccines [7,29]. Other sources that cover HPV vaccine cost in the United

States include Merck assistance programs [30], Medicaid and Children's Health Insurance Program (CHIP) [31]. Promotion and public education on these little or no-cost options for HPV vaccination could improve access.

Parents play an important role in HPV vaccination for their adolescent children. Results from this review showed that pharmacists perceived parental concerns, beliefs, and lack of knowledge as barriers to HPV vaccination [15,18,20]. Similar findings were reported from a study that examined perceived barriers to HPV vaccination among health care providers; the authors reported parental beliefs and misconceptions as a major barrier to HPV vaccination [32]. About one-third of parents in previous research reported that they were willing to allow their children to get the HPV vaccine in a pharmacy setting [33]. Likewise, another study showed that 44% of parents are willing to get an HPV vaccine from pharmacies for their children [34]. Improving parental awareness and clarity that pharmacists offer HPV vaccines for children may facilitate series completion [33]. Also, strategies to educate parents on the importance of the HPV vaccine and demystify misconceptions may positively influence parental decision to vaccinate their children. This may include additional training for pharmacists on ways to educate and provide effective HPV vaccine recommendations to parents [15].

Partnerships and collaboration agreements between health care providers and pharmacists, including strong provider recommendation and referral to pharmacies for subsequent HPV vaccine doses may increase parental awareness and vaccine uptake [33]. Physicians could help improve parental awareness as parents may prefer to learn about the availability of pharmacists to provide the HPV vaccination from their children's physicians [33]. However, there is a lack of uniformity across states in the US and diverging opinions regarding authority to administer vaccines to children and adolescents that complicates the theoretical partnerships and collaboration agreements between health care providers and pharmacists. For instance, laws and regulations granting pharmacists the authority to administer HPV vaccine to all or certain adolescent age groups vary across states [12]. Also, there is a lack of consensus among professional organizations, like the American Academy of Pediatrics (AAP), and the US Department of Health and Human Services regarding whether pharmacists should be authorized to administer vaccines to children and adolescents [11,35]. In 2020, AAP released a statement that their organization believes that children should receive vaccines from a pediatrician, which opposes HHS authorization that allows pharmacists to administer childhood vaccines [36].

Three studies included in this review reported personal barriers among pharmacists to HPV vaccine administration [16,18,19]. The personal barriers were not religious or moral, but related primarily to inadequate knowledge to provide recommendations for HPV vaccination, or educate patients about the HPV vaccine (considering the sensitivity of HPV-related subject) [16,19]. This is promising as additional training on HPV vaccine recommendation strategies, coverage, and administration for practicing pharmacists, as well as student pharmacists, could help improve the pharmacists' knowledge and comfort level in educating and administering the HPV vaccine [37].

At the organization level, remembering to screen [17] as well as tracking and recall of patients were identified as barriers to HPV vaccination [18]. In one of the studies reviewed, only 33 percent of participants reported that they utilized their state's Immunization Information Systems (IIS) [18]. IIS can ensure timely immunization, clinical decision support, records consolidation, and data exchange among health care providers [38]. However, the operating and reporting for IIS requirements vary by state [39]. Another key organization level barrier reported was other responsibilities/vaccines taking precedence [17,20] over HPV vaccination and lack of time/staff/space [16,17,20]. The decision to prioritize and offer HPV vaccines by pharmacies may depend on the goals of the organization/management and may differ between independent versus chain pharmacies.

While the purpose of this review was to summarize existing literature related to the perceived barriers to administering HPV vaccination reported by pharmacists, some of the studies reported facilitators of HPV vaccination suggested by study participants [16,18,20]. Most frequently suggested facilitator was education about HPV vaccine [16,18,20]. As reported in one of the studies, education may involve providing education to patient/parent, pharmacists, and health care providers [20]. Another suggested way to improve HPV vaccination is advertising specifically through mass/social media [16,20]. Pharmacist providing stronger recommendation to patients about getting HPV vaccination was also reported as potential facilitator of HPV vaccination [20]. Partnership involving health care providers, schools [16,20] and public health organization [16] was reported as a facilitator. For instance, doctors may recommend that their patient get the HPV vaccine in a pharmacy or schools holding vaccine clinics [20]. Other facilitators reported by pharmacists include ease of accessibility of community pharmacies [16], clear guidelines from pharmacy/corporate management [18], vaccine promotion within and outside the pharmacy [18], adequate insurance coverage for vaccination [20], and state legislative authority to provide vaccination [18].

Evidence-based strategies to improve HPV vaccination rates that have previously been implemented in other settings, such as in primary care, may help to improve HPV vaccination in pharmacies. For instance, previous studies suggest that reminder and recall systems [40] and decision support systems [41] may be effective in improving HPV vaccination, which potentially could facilitate series completion. Alternatively, successful evidence-based strategies implemented in pharmacy settings to improve other vaccines could also be adapted to improving HPV vaccination rates. In one study, pharmacy-based interventions, including newspaper press releases, use of flyer advertisement, and personalized letters about herpes zoster infection and vaccination were shown to improve herpes zoster vaccination rates [42]. In another study, individuals who received a phone call intervention from pharmacists were more likely to receive their second dose of the recombinant zoster vaccine, thereby facilitating series completion [43]. Further research is needed to explore interventions unique and specific to improving HPV vaccination and series completion in pharmacies.

To our knowledge, this is the first study to review previously reported perceptions of barriers to HPV vaccine administration among pharmacists. Numerous commentaries and professional organizations suggest that pharmacies could play an important role in improving HPV vaccine uptake. The studies included in this review varied greatly in terms of study design, which made the synthesis of findings challenging. The search strategy employed to identify eligible studies for this review, encompassed publications on studies conducted worldwide. However, all the eligible studies included in this review were conducted in the United States. The perceived barriers that pharmacists experience may vary geographically within and outside of the United States. Therefore, generalizability of results may be limited. Nevertheless, many of these barriers are, to some extent, universal and therefore may provide direction for those investigating HPV vaccination barriers in different health care delivery systems. This literature search yielded only a few studies, which indicates that additional research is needed to explore pharmacist's perceived barriers, especially at the system/organization levels. Because most of the studies included in this review relied primarily on quantitative survey-based data collection methods, it limits the ability to thoroughly understand pharmacists' perceived barriers to HPV vaccine administration. A more comprehensive understanding of barriers to HPV vaccine administration is needed, especially to better understand how these barriers may impact various pharmacies in utilizing differing immunization delivery practices. Future work should include in-depth qualitative analyses of barriers among pharmacists practicing in a wide variety of settings.

In conclusion, pharmacies present an opportunity to increase HPV vaccination rates. Targeted public health efforts to increase HPV vaccination among pharmacies should consider a multilevel approach. Future strategies involving pharmacy settings in HPV-related cancer prevention efforts should consider research on pharmacy-driven interventions that addresses the barriers to administering the HPV vaccine at various levels.

**Author Contributions:** Conceptualization, J.M.M. and O.O.; methodology, A.Q.W., O.O. and J.M.M.; formal analysis, O.O., S.G. and J.M.M.; writing—original draft preparation, O.O., S.G. and A.Q.W.; writing—review and editing, J.M.M., S.M.C., J.D.G., N.B.Z. and L.C.K.; visualization, O.O.; supervision, J.M.M., L.C.K. and N.B.Z.; project administration, J.M.M.; funding acquisition, L.C.K., N.B.Z. and J.M.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the University of Tennessee Medical Center Cancer Institute Research Grant Awarded to L.C.K., N.B.Z., J.M.M.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** Justin Gatwood reports vaccine-related research funding provided by Merck & Co. and GlaxoSmithKline.

#### **Appendix A. PubMed Search Strategy**

(Papillomavirus Infections[Mesh]) OR (HPV[tw] OR "human papilloma virus"[tw] or "human papillomavirus"[tw])

AND

(Immunization[Mesh] OR "Vaccination"[Mesh] OR "Immunization Programs"[Mesh] OR "Mass Vaccination"[Mesh] OR "Papillomavirus Vaccines"[Mesh]) OR (vaccin\*[tw] OR immunization[tw])

AND

(Pharmacies[Mesh] OR "Pharmacy Technicians"[Mesh] OR "Pharmacy Residencies"[Mesh] OR "Students, Pharmacy"[Mesh] OR "Pharmacists"[Mesh] OR "Community Pharmacy Services"[Mesh]) OR (pharmac\*[tw])

AND

barrier\*[tw] OR challenge\*[tw] OR constrain\*[tw] OR difficult\*[tw] OR interfer\*[tw] OR obstruct\*[tw] OR problem\*[tw] OR restrain\*[tw] OR restrict\*[tw]

#### **References**


## *Article* **Effectiveness of a Community-Based Organization—Private Clinic Service Model in Promoting Human Papillomavirus Vaccination among Chinese Men Who Have Sex with Men**

**Zixin Wang 1,\*, Yuan Fang 2, Paul Shing-fong Chan 1, Andrew Chidgey 3, Francois Fong 4, Mary Ip <sup>1</sup> and Joseph T. F. Lau 1,\***


**Abstract:** This study evaluated the effectiveness of the community-based organization (CBO)-private clinic service model in increasing human papillomavirus (HPV) vaccination uptake among unvaccinated men who have sex with men (MSM) in Hong Kong during a 12-month follow-up period. A CBO-private clinic model was implemented to promote HPV vaccination among Chinese MSM. A CBO with good access to MSM approached MSM aged 18–45 years who had never received an HPV vaccination, invited them to receive an online health promotion, and referred them to receive HPV vaccination at gay-friendly private clinics. A baseline survey and a follow-up evaluation at Month 12 were conducted. A total of 350 participants completed the baseline survey. Among 274 participants who were followed up at Month 12, 46 (16.8%) had taken up at least one dose of HPV vaccination. After adjusting for significant baseline characteristics, the perceived susceptibility (AOR:1.25, *p* = 0.002) and perceived severity (AOR:1.21, *p* = 0.003) of HPV and HPV-related diseases, perceived benefits (AOR:1.16, *p* = 0.03), self-efficacy to receive HPV vaccination (AOR:1.37, *p* = 0.001), and behavioral intention to take up HPV vaccination at baseline (AOR:6.99, *p* < 0.001) significantly predicted HPV vaccination uptake. The process evaluation of the program was positive. The CBO-private clinic service model was helpful in increasing HPV vaccination uptake among MSM.

**Keywords:** HPV vaccination; men who have sex with men; online health promotion; outcome and process evaluation

#### **1. Introduction**

Across countries, men who have sex with men (MSM) have a much higher risk of contracting human papillomavirus (HPV) and its related diseases (e.g., genital warts and penile cancers) than the general male population [1–4]. As reported by a meta-analysis, the overall prevalence of genital HPV infection was very high among both human immunodeficiency virus (HIV)-negative (63.9%) and HIV-infected MSM (92.6%) [5]. Moreover, MSM's risk of genital warts and anal cancers were much higher than the general population [6,7]. The prevalence of genital warts was 13.2–58.6% among MSM [6,7]. The overall prevalence of high-grade anal histological lesions was 23.9% among HIV-infected MSM and 15.2% among HIV-negative MSM [5]. The HPV-related cancer risk was the highest among HIV-infected MSM, which accounted for 9.9% of the MSM population in China in 2016 [8]. Slow clearance and increased persistence of high-risk HPV might explain why HIV-infected MSM are more susceptible to HPV and its related diseases compared with HIV-negative MSM [9–13]. A previous study showed that men with HIV infection had lower high-risk HPV clearance [9]. In-vitro studies showed increased expression of HPV E1

**Citation:** Wang, Z.; Fang, Y.; Chan, P.S.-f.; Chidgey, A.; Fong, F.; Ip, M.; Lau, J.T.F. Effectiveness of a Community-Based Organization— Private Clinic Service Model in Promoting Human Papillomavirus Vaccination among Chinese Men Who Have Sex with Men. *Vaccines* **2021**, *9*, 1218. https://doi.org/10.3390/ vaccines9111218

Academic Editor: Gloria Calagna

Received: 27 September 2021 Accepted: 15 October 2021 Published: 20 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

and L1 viral gene in the presence in HIV trans-activator of transcription (tat) protein [10]. HIV-tat protein was shown to enhance HPV transcription during HPV replication [11]. Moreover, uncontrolled HIV infection might increase the persistence of high-risk HPV through altered cell-mediated immunity, local molecular interactions, and reduced tight junction function [12,13]. In Hong Kong, among MSM with an experience of HPV screening, 25% were diagnosed with HPV infection [14].

HPV vaccination is highly effective in preventing vaccine-type genital warts and cancers among MSM [15,16] and other HPV-related diseases [17]. HPV vaccination provides maximum benefit if a person receives it before he/she is sexually active, and its efficacy is lower for MSM aged up to 45 years who have an HPV infection [18,19]. However, the Advisory Committee on Immunization Practices of the United States Centers for Disease Control and Prevention (CDC) recommends that people having an HPV infection should still receive HPV vaccination if they are in the appropriate age group because vaccination may protect them against high-risk HPV types that they have not yet acquired [20].

The CDC recommends MSM aged ≤45 years to receive the HPV vaccination [21,22]. In the United Kingdom, national programs provide free HPV vaccination for MSM aged up to 45 years attending sexual health and HIV clinics [23,24]. In 2017, the Victorian Government of Australia rolled out a free time-limited HPV vaccination catch-up program for MSM aged up to 26 years [25]. Relatively high HPV vaccination uptake was observed among MSM in national programs in developed countries, ranging from 37.6% in the United States [26], 42.6% in Australia [25], 49.1% in England [27], to 63.7% in Scotland [24]. However, in Hong Kong, there is no free or subsidized HPV vaccination program for MSM or other male populations. HPV vaccination uptake among the male population is very low in Hong Kong [28–30].

A randomized controlled trial (RCT) was conducted between 2017 and 2019 among MSM in Hong Kong to evaluate online intervention promoting HPV vaccination [31]. The results showed that watching a 5-min online video based on the Health Belief Model [32] and receiving brief motivational interviewing (MI) through telephone was effective in improving HPV vaccination uptake over a 24-month study period [31].

Community-based organization (CBO) in Hong Kong has a good position of promoting HPV vaccination among local MSM. First, half of MSM usually received HIV prevention services provided by local CBO (e.g., 45.5–51.5% of the HIV testing and counseling services) [29,33]. Second, it is easier to promote HPV vaccination by CBO staff, as it is naturally for MSM to receive HIV/sexually transmitted infections (STI)-related information from them. Participants may feel less stigmatized and embarrassed. Third, we found that a cue to action, a construct of the HBM [32], was strongly associated with the acceptability of HPV vaccination among local MSM.

Suggestion from peer CBO workers can serve as a strong cue to action [29]. Moreover, these CBOs have been working closely with gay-friendly private clinics providing STI screening and treatment for MSM. A CBO having good access to local MSM adapted the online intervention developed by the aforementioned RCT and initiated a CBO-private clinic service model promoting HPV vaccination for MSM in August 2019. The CBO approached MSM using its network, provided online intervention promoting HPV vaccination, and referred interested participants to gay-friendly private clinics in its service network for vaccination uptake. This study evaluated the effectiveness of the CBO-private clinics service model in increasing HPV vaccination uptake among unvaccinated MSM in Hong Kong during a 12-month follow-up period. Baseline factors predicting HPV vaccination uptake were investigated and process evaluation was performed.

#### **2. Materials and Methods**

#### *2.1. Study Design*

This longitudinal study was conducted from August 2019 to April 2021. All participants completed a telephone survey before they received online interventions promoting HPV vaccination, and completed another telephone survey 12 months after the baseline survey. There was no control or comparison group. The flowchart diagram was shown in Figure 1. The study was registered at ClinicalTrial.gov (NCT04815837).

**Figure 1.** Flowchart diagram.

#### *2.2. Participants*

Inclusion criteria were: (1) Hong Kong Chinese speaking males aged 18–45 years old, (2) self-reported having had oral or anal intercourse with at least one man in the last six months, (3) never received HPV vaccination, and (4) willing to complete a follow-up telephone survey 12 months after the baseline survey.

#### *2.3. Sampling and Data Collection*

Trained staff of a CBO providing HIV-related services recruited participants through multiple sources, including outreaching in gay bars or saunas in Hong Kong, online recruitment, and referrals made by peers and other CBOs. Detailed recruitment procedures were reported in a published paper [14]. Interested participants contacted fieldworkers through WhatsApp, telephone, email, or other means. Participants were guaranteed anonymity during the study, and had the right to end participation in the study at any time. Their refusal or withdrawal from the study would not affect their access to any future services. Verbal instead of written informed consent was obtained due to maintaining anonymity, and the fieldworkers signed a form pledging that the participants had been fully informed about the study. Multiple forms of contact information were obtained to make appointments for conducting the baseline and Month 12 telephone interview. A HK\$25 (approximately US\$3.2) supermarket or café coupon was mailed to participants as compensation of their time after they completed each survey. Ethics approval was obtained from the Survey and Behavioral Research Ethics Committee of the Chinese University of Hong Kong (reference number: KPF18HLF22).

#### *2.4. Baseline Telephone Survey and Health Promotion*

By appointment, the CBO staff conducted the 15-min baseline survey by telephone. All participants received the following health promotion components:

(1) Watching an online video promoting HPV vaccination for MSM: The online video developed by our team in a previous RCT was modified for this study [31]. The modifications included: (i) cost of HPV vaccination was updated based on the recent market rate, and (ii) information of collaborative clinics (e.g., location and contacts) were added. The video was guided by the HBM [32].

In the video, a peer MSM talked about the risk of having penile or anal cancers among MSM, the severe consequences of HPV-related diseases, high efficacy, and long protection duration of HPV vaccination. The video also contained flashes of scary images of genital warts and penile and anal cancers to increase the perceived severity. The peer MSM also emphasized that HPV vaccination was a worthy long-term investment for their health. He demonstrated the procedures for receiving HPV vaccination in one of the collaborative private clinics, which was convenient, caring, privacy guaranteed, and non-judgmental. Participants could not fast-forward or skip any part of the video.

(2) Each participant was sent a discount coupon through SMS/WhatsApp and could enjoy a 10% discount for taking up three doses of HPV vaccines at a collaborative private clinic.

(3) Participants could access the project webpage by scanning the Quick Response (QR) code on the discount coupon. Through the webpage, participants could: (i) watch the aforementioned online video; (ii) read the description of the project, knowledge, and benefits of HPV vaccination, information about HPV/HPV-related diseases that increased the perceived susceptibility and perceived severity; (iii) access a discussion forum containing positive feedbacks of peers who have taken up HPV vaccination. These testimonials provided cue to action supporting HPV vaccination; and (iv) obtain the contact information of the project staff and information about the collaborative private clinics.

(4) Five reminders were sent to participants by SMS/WhatsApp at Month 1, 2, 4, 6, and 8.

#### *2.5. Facilitating HPV Vaccination Uptake in Collaborative Private Clinics*

During the project period, two types of HPV vaccines were available for males (Gardasil 4-valent and 9-valent vaccines). For participants who decided to take the vaccine in the collaborative private clinics, they could sign up on the project webpage by filling up their contacts and preferable timeslots. The CBO staff would contact them by telephone or WhatsApp to facilitate appointment making to take up HPV vaccination at collaborative private clinics at a 10% discount from the market rate (about HK\$4200 or US\$542 for three doses). Participants could also directly contact the collaborative clinics to make an appointment. The project did not limit participants' choice for taking up HPV vaccination in other places.

#### *2.6. Month 12 Follow-up Telephone Survey*

Participants were invited to complete a follow-up telephone survey 12 months after the baseline survey. Up to five calls were made at different timeslots during weekdays and/or weekends before considering a participant as a dropout.

#### *2.7. Measurements*

#### 2.7.1. Primary Outcome

The primary outcome of the study was the validated uptake of any doses of HPV vaccination within a 12-month follow-up period. To validate HPV vaccination uptake, the participants were requested to send the project staff an image of their receipt without personal identification after they received each dose of HPV vaccines. The same approach to validate HPV vaccination uptake was used in a previous study [31]. Vaccinated participants were asked about the location and cost of HPV vaccination uptake, whether they experienced side-effects after vaccination, and the severity of those side-effects.

#### 2.7.2. Baseline Characteristics

Various baseline information was collected, such as socio-demographics, sexual orientation, HIV or STI prevention service utilization, history of HIV and other STIs, and queried sexual behaviors, which included anal intercourse with male regular and non-regular sex partners and male sex workers, condomless anal intercourse (CAI) with men, multiple male sex partnerships, sexualized drug use (SDU), and so on. Regular male sex partners (RP) were defined as lovers and/or stable boyfriends, while non-regular male sex partners (NRP) were defined as casual sex partners. In this study, SDU was defined as the use of any psychoactive substance before or during sexual intercourse [34,35].

Six items measured knowledge related to HPV and HPV vaccination. A composite variable was constructed by counting the number of correct responses (ranged from 0 to 6). Five scales assessed perceptions based on the HBM, including a three-item Perceived Susceptibility Scale, four-item Perceived Severity Scale, four-item Perceived Benefit Scale, five-item Perceived Barrier Scale, two-item Cue to Action Scale, and two-item Perceived Self-Efficacy Scale. These scales were modified from those used in a previous study among MSM in Hong Kong [31].

#### 2.7.3. Process Evaluation

At Month 12, participants were asked to evaluate the health promotion (e.g., whether the contents were clear and attractive). Participants who have taken up HPV vaccination were asked additional questions rating their satisfaction of the services provided by the collaborating clinics.

#### *2.8. Statistical Analysis*

The difference in baseline characteristics between participants who completed Month 12 follow-up and dropouts were compared by using chi-square tests (for categorical variables) or Mann–Whitney U tests (for continuous variables). The subsequent analysis was performed with data from those who had completed both surveys. We used the validated uptake of any doses of HPV vaccination during the follow-up period as the dependent variable and baseline background characteristics (e.g., socio-demographics, sexual orientation, and history of HIV and other STIs) as independent variables.

Crude odds ratios (OR) predicting that the dependent variable were obtained using logistic regression models. After adjustment for those variables with *p* < 0.05 in the univariate analysis, the association between knowledge and perceptions related to HPV and HPV vaccination and the dependent variable were then assessed by adjusted odds ratios (AOR). Each AOR was obtained by fitting a single logistic regression model, which involved one of the perceptions and the significant background variables. SPSS version 21.0 (Chicago, IL, USA) was used for data analysis, and *p* < 0.05 (two-sided) is considered as statistically significant.

#### **3. Results**

#### *3.1. Baseline Characteristics*

Four hundred of the 565 prospective participants were screened to be eligible, 50 of whom refused to participate in the study for time or other logistic reasons, and 350 (87.5%) completed the baseline survey and received health promotion. At Month 12, 274 (78.3%) completed the follow-up survey.

At baseline, about half of the participants were 18–30 years old (50.3%), and with a monthly personal income at least HK\$20,000 (US\$2565) (57.4%). The majority of the participants were currently single (75.7%), had attained tertiary education or above (86.6%), were employed full-time (82.9%), and identified themselves as homosexuals (91.1%). Among the participants, 1.7%, 8.9%, and 16.9% reported a history of HIV, HPV infection/genital warts, and other STI infection, respectively. In the past six months, 50.3% and 54.6% reported CAI with men and multiple male sex partnerships.

Regarding knowledge related to HPV or HPV vaccination, 48.9–95.7% gave correct responses to different items. The Cronbach alpha for the five scales based on the HBM were acceptable (0.61–0.85). Single factors for these scales by exploratory factor analysis, which explained 56.2–82.5% of the total variance. Apart from self-reported HIV serostatus (*p* = 0.02), anal intercourse with NRP (*p* = 0.02), and male sex workers (*p* = 0.01), no significant difference was found between participants who completed the Month 12 follow-up and dropouts (Table 1).


**Table 1.** Baseline characteristics of the participants.

**All Participants (***n* **= 350) Being Followed Up at Month 12 (***n* **= 274) Dropouts (***<sup>n</sup>* **= 76)** *<sup>p</sup>* **Values** *n* **(%)** *n* **(%)** *n* **(%) Lifestyles** Smoking in lifetime No 257 (73.4) 203 (74.1) 54 (71.1) 0.60 Yes 93 (26.6) 71 (25.9) 22 (28.9) Drinking in the past year No 86 (24.6) 64 (23.4) 22 (28.9) 0.32 Yes 264 (75.4) 210 (76.6) 54 (71.1) **History of HIV and other STIs and service utilization** Self-reported HIV sero-status Negative 311 (88.9) 251 (91.6) 60 (78.9) 0.02 Positive 6 (1.7) 4 (1.5) 2 (2.6) Refuse to disclose 11 (3.1) 7 (2.6) 4 (5.3) Had never tested for HIV antibody 22 (6.3) 12 (4.4) 10 (13.2) History of HPV infection and/or genital warts (Yes) 31 (8.9) 27 (9.9) 4 (5.3) 0.21 History of other STIs (Yes) 59 (16.9) 47 (17.2) 12 (15.8) 0.78 Utilization of other HIV/STI prevention services (e.g., receiving free condoms, peer education and pamphlets, and attending seminars) (Yes) 81 (23.1) 69 (25.2) 12 (15.8) 0.09 **Sexual behaviors in the past six months** Anal intercourse with regular male sex partners (Yes) 293 (83.7) 230 (83.9) 63 (82.9) 0.83 Anal intercourse with non-regular male sex partners (Yes) 170 (48.6) 124 (45.3) 46 (60.5) 0.02 Anal intercourse with male sex workers (Yes) 9 (2.6) 4 (1.5) 5 (6.6) 0.01 Condomless anal intercourse with men (Yes) 176 (50.3) 142 (51.8) 34 (44.7) 0.27 Multiple male sex partnerships (Yes) 191 (54.6) 143 (52.2) 48 (63.2) 0.09 Sexual intercourse with female sex partners (Yes) 6 (1.7) 3 (1.1) 3 (3.9) 0.09 Condomless sex with female sex partners (Yes) 3 (0.9) 2 (0.7) 1 (1.3) 0.62 Sexualized drug use (use of psychoactive substances before or during sexual intercourse) (Yes) 20 (5.7) 17 (6.2) 3 (3.9) 0.45 **Knowledge related to HPV or HPV vaccination** Both males and females could be affected by HPV Yes <sup>a</sup> 335 (95.7) 264 (96.4) 71 (93.4) 0.11 No 5 (1.4) 2 (0.7) 3 (3.9) Do not know 10 (2.9) 8 (2.9) 2 (2.6) HPV infection could cause STI Yes <sup>a</sup> 317 (90.6) 248 (90.5) 69 (90.8) 0.79 No 8 (2.3) 7 (2.6) 1 (1.3) Do not know 25 (7.1) 19 (6.9) 6 (7.9) HPV infection could cause cancers among males Yes <sup>a</sup> 258 (73.7) 200 (73.0) 58 (76.3) 0.54 No 27 (7.7) 20 (7.3) 7 (9.2) Do not know 65 (18.6) 54 (19.7) 11 (14.5) HPV could be totally cured by available treatment Yes 62 (17.7) 48 (17.5) 14 (18.4) 0.93 No <sup>a</sup> 171 (48.9) 133 (48.5) 38 (50.0) Do not know 117 (33.4) 93 (33.9) 24 (31.6) Availability of effective HPV vaccination for males in Hong Kong Yes <sup>a</sup> 288 (82.3) 229 (83.6) 59 (77.6) 0.12 No 8 (2.3) 4 (1.5) 4 (5.3) Do not know 54 (15.4) 41 (15.0) 13 (17.1) Number of shots required to prevent HPV infection in males 3 <sup>a</sup> 176 (50.3) 135 (49.3) 41 (53.9) Other answers/Do not know 174 (49.7) 139 (50.7) 35 (46.1)

**Table 1.** *Cont*.

Number of correct responses, mean (SD) 4.4 (1.3) 4.4 (1.3) 4.4 (1.2) 0.91

**All Participants (***n* **= 350) Being Followed Up at Month 12 (***n* **= 274) Dropouts (***<sup>n</sup>* **= 76)** *<sup>p</sup>* **Values** *n* **(%)** *n* **(%)** *n* **(%) Perceptions related to HPV or HPV vaccination based on the HBM** Perceived susceptibility to HPV (high/very high) Perceived risk of contracting HPV in lifetime 89 (25.4) 65 (23.7) 24 (31.6) 0.16 Perceived risk of contracting genital warts in lifetime 72 (20.6) 50 (18.2) 22 (28.9) 0.04 Perceived risk of having penile/anal cancers in lifetime 42 (12.0) 32 (11.7) 10 (13.2) 0.73 Perceived Susceptibility Scale b, mean (SD) 8.2 (2.6) 8.1 (2.6) 8.8 (2.7) 0.35 Perceived severity of HPV-related diseases (agree/strongly agree) HPV infection would increase risk of HIV acquisition 106 (30.3) 81 (29.6) 25 (32.9) 0.58 HPV infection would cause penile or anal cancers 127 (36.3) 98 (35.8) 29 (38.2) 0.70 Genital warts would have severe harms on your health 182 (52.0) 140 (51.1) 42 (55.3) 0.52 Penile or anal cancers would have severe harms on your health 263 (75.1) 207 (75.5) 56 (73.7) 0.74 Perceived Severity Scale c, mean (SD) 13.4 (3.1) 13.3 (3.2) 13.8 (2.7) 0.43 Perceived benefits of HPV vaccination (agree/strongly agree) HPV vaccination is highly effective in preventing HPV infection 273 (78.0) 210 (76.6) 63 (82.9) 0.24 HPV vaccination is highly effective in preventing genital warts 240 (68.6) 181 (66.1) 59 (77.6) 0.054 HPV vaccination is highly effective in preventing penile/anal cancers 214 (61.1) 166 (60.6) 48 (63.2) 0.68 HPV vaccination can protect you for a long time 193 (55.1) 149 (54.4) 44 (57.9) 0.59 Perceived Benefit Scale d, mean (SD) 15.3 (2.5) 15.2 (2.6) 15.6 (2.1) 0.20 Perceived barriers of receiving HPV vaccination (agree/strongly agree) It is not worthy speeding HK\$6000–7000 (US\$774–903) to receive HPV vaccination 107 (30.6) 88 (32.1) 19 (25.0) 0.23 You would have severe side-effects after receiving HPV vaccination 37 (10.6) 30 (10.9) 7 (9.2) 0.66 Others would think you are having high-risk behaviors if you receive HPV vaccination 45 (12.9) 37 (13.5) 8 (10.5) 0.49 You would be stigmatized when you receive HPV vaccination 34 (9.7) 26 (9.5) 8 (10.5) 0.79 If you already infected with HPV, HPV vaccination could not protect you 111 (31.7) 88 (32.1) 23 (30.3) 0.76 Perceived Barrier Scale e, mean (SD) 12.4 (3.1) 12.4 (3.1) 12.2 (3.5) 0.76 Perceived cue to action related to HPV vaccination (agree/strongly agree) Mass media suggest males to receive HPV vaccination 62 (17.7) 46 (16.8) 16 (21.1) 0.39 People who are important to you would suggest you to receive HPV vaccination 83 (23.7) 66 (24.1) 17 (22.4) 0.76 Cue to Action Scale <sup>f</sup> , mean (SD) 4.9 (2.0) 4.8 (2.0) 5.2 (2.0) 0.14 Perceived self-efficacy related to HPV vaccination (agree/strongly agree) You are confident to receive HPV vaccination in the next year if you want 160 (45.7) 129 (47.1) 31 (40.8) 0.33 Receiving HPV vaccination in the next year is easy for you if you want 182 (52.0) 148 (54.0) 34 (44.7) 0.15 Perceived Self-efficacy Scale g, mean (SD) 6.8 (2.0) 6.9 (2.0) 6.7 (2.0) 0.37

**Table 1.** *Cont*.


<sup>a</sup> Correct response. <sup>b</sup> Perceived Susceptibility Scale: three items, Cronbach's alpha: 0.85, one factor was identified by exploratory factor analysis, explaining for 77.2% of total variances. <sup>c</sup> Perceived Severity Scale: four items, Cronbach's alpha: 0.72, one factor was identified by exploratory factor analysis, explaining for 54.9% of total variances. <sup>d</sup> Perceived Benefit Scale: four items, Cronbach's alpha: 0.79, one factor was identified by exploratory factor analysis, explaining for 62.4% of total variances. <sup>e</sup> Perceived Barrier Scale: five items, Cronbach's alpha: 0.62, one factor was identified by exploratory factor analysis, explaining for 57.9% of total variances. <sup>f</sup> Cue to Action Scale: two items, Cronbach's alpha: 0.61, one factor was identified by exploratory factor analysis, explaining for 56.2% of total variances. <sup>g</sup> Perceived Benefit Scale: two items, Cronbach's alpha: 0.79, one factor was identified by exploratory factor analysis, explaining for 82.5% of total variances. HK: Hong Kong; HIV: human immunodeficiency virus; STI: sexually transmitted infection; and HPV: human papillomavirus.

#### *3.2. HPV Vaccination Uptake*

At Month 12, 46 participants self-reported having had taken up at least one dose of HPV vaccination (three doses: *n* = 13; two doses: *n* = 24; and one dose: *n* = 9). All of them were able to provide receipts for verification. The location for receiving HPV vaccination included the collaborative private clinics (28/46, 60.9%) and other private hospitals/clinics (18/46, 39.1%). The participants self-paid HK\$2000–6800 (US\$ 258–877; median: HK\$4200 or US\$542) to receive the vaccination. Most vaccinated participants reported no side effects (37/46, 80.4%). The reported side effects included pain at the injection site (5/46, 10.9%), fatigue (4/46, 8.7%), and dizziness (1/46, 2.2%). Most of these side effects were very mild or mild (8/9, 88.9%).

#### *3.3. Factors Predicting HPV Vaccination Uptake*

Among 274 participants who had completed both baseline and Month 12 surveys, a history of HPV infection and/or genital warts (OR: 6.02, 95%CI: 2.60, 13.94, *p* < 0.001) and utilization of HIV/STI prevention services other than HIV testing at baseline (OR: 1.98, 95%CI: 1.01, 3.89, *p* = 0.046) were associated with higher HPV vaccination uptake during the follow-up period (Table 2).

After adjusting for these significant baseline characteristics, four constructs of the HBM were associated with the dependent variable in expected directions. They were: (1) perceived higher risk of contracting HPV and HPV-related diseases (perceived susceptibility) (AOR: 1.25, 95% CI: 1.09, 1.43, *p* = 0.002), (2) perceived consequences of HPV-related diseases to be severer (AOR: 1.21, 95%CI: 1.07, 1.37, *p* = 0.003), (3) perceived benefits of HPV vaccination (AOR: 1.16, 95% CI: 1.01, 1.33, *p* = 0.03), and (4) perceived self-efficacy of taking up HPV vaccination (AOR: 1.37, 95%CI: 1.14, 1.65, *p* = 0.001). In addition, behavioral intention to take up HPV vaccination at baseline was also associated with higher uptake during the follow-up period (AOR: 6.99, 95% CI: 3.34, 14.60, *p* < 0.001) (Table 3).


**Table 2.** Baseline background characteristics predicting human papillomavirus vaccination uptake (among participants who completed Month 12 follow-up, *n* = 274).

OR: crude odds ratios; CI: confidence interval; HK: Hong Kong; HIV: human immunodeficiency virus; STI: sexually transmitted infection; HPV: human papillomavirus; and N.A.: not applicable.


**Table 3.** Factors predicting human papillomavirus vaccination uptake (among participants who completed Month 12 follow-up, *n* = 274).

HPV: human papillomavirus; HBM: health belief model; OR: crude odds ratios; CI: confidence interval; and AOR: adjusted odds ratios, odds ratios adjusted for significant baseline background characteristics (i.e., history of HPV infection and/or genital warts, and utilization of other HIV/STI prevention services).

#### *3.4. Reasons for Not Taking up HPV Vaccination*

At Month 12, 228 unvaccinated participants were asked about their reasons for not taking up HPV vaccination. The most commonly mentioned reason was the high cost of HPV vaccination (128/228, 56.1%), followed by feeling unnecessary to receive such vaccination (79/228, 34.6%), concerns about COVID-19 transmission (73/228, 32.0%), and no time to do so (60/228, 26.3%).

#### *3.5. Process Evaluation*

Among 274 participants who completed the Month 12 survey, 72.3% and 38.7% believed the health promotion video was clear and attractive respectively. The participants thought the video was helpful in the following aspects, such as increasing awareness of benefits of HPV vaccination (73.4%), reducing barriers to receive HPV vaccination (46.0%), and increasing self-efficacy (56.2%) and intention (40.5%) to receive HPV vaccination.

Among 46 vaccinated participants, 50–95.7% rated "satisfactory" to the following services/features of the collaborating clinics: (1) convenience of the location of clinic (89.1%), (2) convenience of the arranged time slot for vaccination (95.7%), (3) waiting time between making appointment and receiving the first dose (84.8%), (4) level of privacy (69.6%), (5) staff's acceptance of MSM's subculture (50.0%), (6) explanation made by the staff of private clinics (73.9%), and (7) professionalism of the staff of private clinics (76.1%) (Table 4).

**Table 4.** Process evaluation of the health promotion and the procedures to receive human papillomavirus vaccination at private clinics.


HPV: human papillomavirus; and MSM: gay, bisexual, other men who have sex with men.

#### **4. Discussion**

It is important to transform research findings into regular services. We involved CBO and private clinics as key stakeholders at the planning stage. The implementation was smooth and concurred with routine services of CBO and private clinics. The evaluation results showed that the CBO-private clinic service model was helpful in increasing HPV vaccination uptake among MSM in Hong Kong, as 16.8% of participants being followed up at Month 12 had taken up at least one dose of HPV vaccination. We expected most of these participants would complete all three required doses in near future, as all of them settled the payment for the entire package when receiving the first dose and made appointments for receiving the remaining doses.

Although there was an increase in the cost of HPV vaccines (from HK\$3800 or US\$490 to HK\$4200 or US\$542) and there was no MI, the uptake rate was comparable to the intervention group (17.3%) in previous RCT [31]. However, the HPV vaccination uptake in this study was lower than those in the United States (19.4–45%) [36–39]. The difference could be partially explained by the cost of HPV vaccination, as it was free in the United States while it was charged at market price with a slight discount in this study. It was likely that COVID-19 had a negative impact on HPV vaccination uptake in this study, as one third of unvaccinated participants mentioned concerns about COVID-19 transmission as a reason of not taking up HPV vaccination at Month 12.

The findings also had some implications to health service and health care policy. Our results supported that HPV vaccination was accepted by local MSM at the market rate. Providing subsidized or free HPV vaccination would further increase the uptake rate. MSM in Australia and the United Kingdom responded well to pilot programs providing free HPV vaccination [23–25]. Hong Kong should consider a similar pilot program for MSM, which would largely increase HPV vaccination coverage in this group and reduce HPV-related disease burdens.

The planning of future programs can be facilitated by the experiences of the present study. We found that those with history of HPV infection and/or genital warts were more responsive to the health promotion as compared to MSM without such history. It is possible that MSM with such history would perceive a stronger need to protect themselves by taking up HPV vaccination. Future studies are needed to explore whether different strategies should be applied for MSM with and without prior experience of HPV infection/genital warts. We also found that the baseline measurement of perceived susceptibility, perceived severity, perceived benefit, perceived self-efficacy, and behavioral intention related to HPV vaccination significantly predicted HPV vaccination uptake during the project period.

Since our health promotion was standard and one-off, such findings suggested that some follow-ups should be implemented to modify these perceptions. Meta-analysis suggested that interventions tailored to one's stage of change (SOC) were more effective than non-stage-tailored, especially among less motivated individuals [40]. Studies also showed that people might move forward to a higher SOC, go back to a lower SOC, or stay in the same SOC after exposing to the health promotion [41]. Therefore, future programs should consider stage-tailored health promotion with multiple sessions, which might be helpful to strengthen the perceived threat of HPV and perceived benefit of HPV vaccination.

Moreover, asking people to provide where, when, and how they want to perform a behavior could increase the perceived self-efficacy [42]. The results of the process evaluation also provide insights for improving health promotion and procedures to receive HPV vaccination in future program. Future programs should consider crowdsourcing, which allows experts and target audience to share solutions together to increase the attractiveness of the contents [43]. Half of the MSM were not satisfied about HPV vaccination service providers' acceptance of their subculture. Interventions targeting service providers to enhance their knowledge about MSM's subculture is needed, and there were effective interventions in the literature [44]. Since the amount of HPV vaccination service providers would be relatively small, training to improve the situation should be feasible.

This study had the strengths of a relatively low dropout rate, validated primary outcome, and good process evaluation. However, it had several limitations. First, there was no control or comparison group. The aim of this study was to evaluate the effectiveness of the service model in real-world setting. Second, a selection bias existed, as we were not able to collect information from MSM who refused to join the project. They might have different characteristics comparing to participants. Third, attrition bias existed. The dropouts were more likely to be HIV positive or with unknown sero-status and had anal intercourse with NRP and male sex workers at the baseline. However, the bias should be limited as these baseline characteristics did not significantly predict HPV vaccination uptake during the project period.

Fourth, we did not ask for details about how COVID-19 influenced the participants' HPV vaccination. However, a previous study exploring the difficulties to access HIVrelated services among MSM in Hong Kong during the same period provided some insights. Worrying about being exposed to a potentially COVID-19 infectious environment, experiencing disruptions in work due to COVID-19 and its control measures, and reduced connection to the MSM community during the pandemic were associated with increased difficulties in accessing HIV-related services in general [45]. Some of these factors might have negatively affected HPV vaccination among our participants. Finally, the study was based on a convenient sample of MSM, cautions should be taken when generalizing the results to MSM in Hong Kong.

#### **5. Conclusions**

The CBO-private clinic service model was helpful in increasing HPV vaccination uptake among MSM in Hong Kong. A larger-scale pilot program should be considered based on the experience of this project.

**Author Contributions:** Conceptualization: Z.W., A.C., F.F., J.T.F.L.; Methodology: Z.W., J.T.F.L., Data curation: Z.W., M.I., A.C. and F.F.; Formal analysis: Z.W. and Y.F.; Project administration: Z.W., M.I. Resources: Z.W., M.I., A.C., and F.F.; Supervision: Z.W. and M.I.; Writing—original draft preparation: Z.W., Y.F. and P.S.-f.C.; Writing—review and editing: Z.W., Y.F., P.S.-f.C. and J.T.F.L.; Funding acquisition: Z.W., and J.T.F.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Knowledge Transfer Project Fund, the Chinese University of Hong Kong (grant number: KPF18HLF22).

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Survey and Behavioral Research Ethics Committee of the Chinese University of Hong Kong (reference number: KPF18HLF22 and date of approval: 18 March 2019).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available from the corresponding authors upon request. The data are not publicly available as they contain sensitive personal behaviors.

**Acknowledgments:** The authors would like to express their gratitude to all the participants for their engagement in this study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **A Long-Term Observation on the Possible Adverse Effects in Japanese Adolescent Girls after Human Papillomavirus Vaccination**

**Akiyo Hineno 1,2 and Shu-Ichi Ikeda 1,3,\***


**Abstract:** In Japan, a significant number of adolescent females noted unusual symptoms after receiving the human papillomavirus (HPV) vaccination, of which the vast majority of them were initially diagnosed with psychiatric illnesses because of the absence of pathologic radiological images and specific abnormalities in laboratory test results. Later these symptoms were thought to be adverse effects of HPV vaccination. However, a causal link between HPV vaccination and the development of these symptoms has not been demonstrated. Between June 2013 and March 2021, we examined 200 patients who noted various symptoms after HPV vaccination. In total, 87 were diagnosed with HPV vaccination-related symptoms based on our proposed diagnostic criteria. The clinical histories of these 87 patients were analyzed. The age at initial vaccination ranged from 11 to 19 years old (mean ± SD: 13.5 ± 1.5 years old), and the age at the first appearance of symptoms ranged from 12 to 20 years old (mean ± SD: 14.3 ± 1.6 years old). The patients received an initial HPV vaccine injection between May 2010 and May 2013, but the first affected patient developed symptoms in October 2010, and the last affected developed symptoms in October 2015. A cluster of patients with a post-HPV vaccination disorder has not appeared in Japan during the last five years. Our study shows that, in Japan, the period of HPV vaccination considerably overlapped with that of a unique post-HPV vaccination disorder development. This disorder appears as a combination of orthostatic intolerance, chronic regional pain syndrome, and cognitive dysfunction, but its exact pathogenesis remains unclear.

**Keywords:** human papillomavirus vaccination; adverse effects; orthostatic dysregulation; chronic regional pain syndrome; cognitive dysfunction

#### **1. Introduction**

The human papillomavirus (HPV) infection plays a crucial role in the development of uterine cervical cancers [1]. Therefore, in May 2010, HPV vaccines, Cervarix® (Glaxo-SmithKline, Brentford, UK), a papillomavirus recombinant bivalent vaccine, and Gardasil® (Merck & Co, Inc., Kenilworth, NJ, USA), a papillomavirus recombinant quadrivalent vaccine, were widely introduced to Japanese female teenagers [2,3]. Beginning April 2013, female adolescents aged 13–16 years were legally required to receive this vaccination. Soon after this vaccination program began, a significant number of the vaccinated females complained of a unique disorder that was composed of violent tremulous involuntary movement, chronic pain, and weakness in the limbs. The Japanese mass media largely reported that a combination of these symptoms was previously unexperienced, suggesting that this disorder was a possible adverse reaction to HPV vaccination. Repeated presentations of suffering vaccinated females on television had a strong impact on Japanese society,

**Citation:** Hineno, A.; Ikeda, S.-I. A Long-Term Observation on the Possible Adverse Effects in Japanese Adolescent Girls after Human Papillomavirus Vaccination. *Vaccines* **2021**, *9*, 856. https://doi.org/ 10.3390/vaccines9080856

Academic Editor: Gloria Calagna

Received: 10 June 2021 Accepted: 30 July 2021 Published: 4 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

forcing the Japanese Ministry of Public Health, Labour and Welfare to withdraw the recommendation for the use of HPV vaccination at the end of June 2013 [4]. Simultaneously, a special committee was organized to investigate the affected Japanese females, and our institution has been functioning as one of the investigation centers for the past eight years.

In our previous two reports [5,6], we described the clinical features and diagnostic criteria of the involved Japanese females with post HPV vaccination disorder. This disorder seems to include orthostatic dysregulation, chronic regional pain syndrome (CRPS), and cognitive dysfunction [5–7]. Post-vaccination abnormal autoimmune reactions are surmised to be responsible for this disorder [8,9], but a causal link has not been established between HPV vaccination and the appearance of these symptoms. Therefore, in this study, we attempted to clarify the temporal relationship between HPV vaccination and the development of this peculiar disorder on the basis of our single center's long-term observation of the affected Japanese females.

#### **2. Materials and Methods**

Between June 2013 and March 2021, we examined the symptoms and objective findings of 200 HPV vaccinated female patients. According to our proposed diagnostic criteria [6], we obtained the necessary patient information, paying special attention to the duration between vaccination and the development of the first symptoms suspected to be related to the vaccine. The patients underwent physical and neurological examinations and routine laboratory tests. Skin temperature and a digital plethysmogram were recorded, and if necessary, the Schellong test was conducted. Moreover, neuropsychological tests and functional brain imaging were performed in patients with cognitive dysfunction. The details of these methods are described in our previous reports [5,6]. The study protocol was approved by the Institutional Review Board (approval nos. 4128 and 4150) of Shinshu University School of Medicine, Matsumoto, Japan.

#### **3. Results**

During the past eight years, 200 female patients visited our hospital with the suspicion of HPV vaccine-related adverse effects (33 patients in 2013, 43 in 2014, 38 in 2015, 49 in 2016, 25 in 2017, 8 in 2018, 4 in 2019, 0 in 2020, 0 in 2021). Of these, we excluded 19 patients who had symptoms before vaccination and 5 who received the HPV vaccine after 30 years of age. An additional 28 patients whose symptoms or disorders were explained by known diseases or who had abnormal laboratory data were also excluded, specifically, eight with epilepsy, six with psychiatric or anxiety disorders, three with systemic lupus erythematosus, one with juvenile idiopathic arthritis, one with anti-SGPS antibody-positive polymyositis, and nine with other diseases. For the remaining 148 patients, the clinical manifestations and objective findings were analyzed. The results showed that 32 patients were diagnosed with definite vaccine-related symptoms, and 55 were diagnosed with probable vaccine-related symptoms. The patient's symptoms and signs of the 87 patients diagnosed are summarized in Table 1. The most frequent symptom was prolonged general fatigue, which led to an inability to wake up and subsequently go to school in the morning. Severe headache, widespread pain involving the limbs and trunk, and dysautonomic symptoms including orthostatic fainting and bowel dysfunction were also responsible for markedly decreased daily activity in the patients. Further, widespread pain typically appeared as migratory joint pain without any signs of inflammation, and intermittent neuralgic pain in the chest or abdominal wall was common. Motor dysfunction showed variable patterns, but the distal dominant weakness of the limbs, which was mimicking that of polyneuropathy, was predominant. Abnormal sensations were mainly observed in the thighs or lower legs where dysesthesia or allodynia was frequent. As compared with these symptoms, learning impairment and sleep disorder developed later. The patients complained of a lack of mental clarity. Objective findings that were frequently observed were orthostatic dysregulation, including postural orthostatic tachycardia syndrome (POTS), abnormal

digital plethysmogram recordings, and abnormalities on brain SPECT images. The details of these findings have been described in a previous report [6].


**Table 1.** Frequency of symptoms and signs in the 87 patients studied.

The temporal distribution of the period of initial vaccination and the appearance of the first symptom in the diagnosed 87 patients is shown in Figure 1. Note that the initial vaccination period ranged from May 2010 to May 2013, and the age at initial vaccination ranged from 11 to 19 years old (mean ± SD: 13.5 ± 1.5 years old). Meanwhile, the first symptom appeared from October 2010 to October 2015, and the age at the appearance of the first symptoms ranged from 12 to 20 years old (mean ± SD: 14.3 ± 1.6 years old). Thus, the time from the first vaccine dose to symptom onset ranged from 0 to 1532 days (median: 199 days). The interval between the onset of symptoms and our initial examination ranged from 0 to 85 months (median: 31 months), indicating the illness duration in the patients before they visited our center.

The temporal relationship between the HPV vaccination and the development of the symptoms was as follows: the first HPV vaccine injection was in May 2010 and the last was in May 2013 (Figure 1a). The first affected vaccinated female developed symptoms in October 2010, and the latest appearance of symptoms occurred in two patients in October 2015; the peak period of the first injection of HPV vaccine seems to be between July 2011 and September 2012, and that of the development of unique post-vaccinated symptoms appeared between September 2011 and August 2013 (Figure 1a,b). Over the previous five years, we did not examine any patients who were newly affected by these unique symptoms (Figure 1b).

**Figure 1.** Temporal relationship between HPV vaccination and the development of symptoms in the patients diagnosed as having HPV vaccine-related symptoms. The period presented here ranged from May 2010 to March 2021. (**a**) Number of patients who received the first injection of HPV vaccine each month. The arrow indicates the time when the Japanese Ministry of Public Health, Labour and Welfare stopped the recommendation of HPV vaccination. (**b**) Number of patients who developed symptoms each month. (**c**) Number of patients who visited our institution and were diagnosed as having a post-HPV vaccination disorder each month.

#### **4. Discussion**

HPV vaccine safety has been reported in HPV vaccination-predominant countries [10–12]. Especially in Australia, although syncope occasionally occurs after HPV vaccination, the frequency of other serious adverse effects including POTS, CRPS, primary ovarian insufficiency, Guillain-Barré syndrome, autoimmune diseases, and venous thrombosis is very low, suggesting no causal association [13]. However, the potential risk of HPV vaccination and dysautonomia, CRPS, and chronic fatigue syndrome has been identified based on a series of case reports from different countries [14–19]. Thus, safety concerns regarding HPV vaccines remain controversial [20].

According to the reports of a Japanese special committee [21,22], 3.39 million Japanese females received HPV vaccinations between May 2010 and November 2016, and 2024 recipients were reported to have adverse reactions, of which 673 experienced serious symptoms. However, the incidence of adverse reactions in this vaccination period was determined to be low and insignificant, even though similar symptoms were not observed as a result of other vaccines.

Variable clinical manifestations in the post-HPV vaccination disorders can be explained by a combination of orthostatic dysregulation, mainly appearing as POTS, CRPS, and/or cognitive dysfunction [5,6,23]. Recent research has found that among POTS, CRPS, and myalgic encephalomyelitis/chronic fatigue (ME/CFS), some conditions overlap [24–26]; especially for cognitive dysfunction, slow thinking, difficulty in focusing, lack of concentration, forgetfulness, and confusion are commonly observed in all three disorders, and correspond to haziness in thought process, which is currently called "brain fog" [27]. Thus, the cognitive dysfunction observed in patients with post-HPV vaccination disorders may be a secondarily induced pathological condition following the long-lasting POTS and/or CRPS. Furthermore, POTS, CRPS, and ME/CFS seem to share similar autoimmune abnormalities [28], and a few preliminary studies [29–33] and case reports [34–37] have shown that the presence of serum autoantibodies against autonomic nerve receptors may be a critical determinant in the pathogenesis of these three disorders. In relation to this hypothesis, we investigated the autoantibodies against autonomic nerve receptors in the serum of the affected patients and revealed that the serum levels of autoantibodies against the adrenergic receptors and muscarinic acetylcholine receptors were significantly elevated in patients with HPV vaccination, as compared with those in the controls [38]. However, there was no statistically significant association between the clinical symptoms and elevated serum levels of these autoantibodies. Thus, further studies are required to consider the possibility of HPV vaccination-related abnormal autoimmune reactions.

In our previous report [6], we described a close temporal relationship between HPV vaccine administration and the appearance of possible adverse symptoms in 72 Japanese patients on the basis of four years of observation. In this study, we extended this observation period to nearly eight years, and the number of patients diagnosed was increased to 87, reaffirming that the period of HPV vaccination considerably overlapped with that of a unique post-vaccination disorder development in our country. In Japan, HPV vaccine coverage for females aged 12 to 16 years has dropped to less than 1% after the termination of the government's recommendation [39], and during the previous three years, few females visited us for evaluations regarding a suspected post-HPV-vaccination disorder. These observations indicate that intensive injections of HPV vaccines between May 2010 and May 2013 induced a cluster of Japanese patients with a unique post-HPV vaccination disorder. Japan is not the only exceptional country for an extremely lowered rate of HPV vaccination in recent several years; Latin American countries, such as Columbia, followed a similar pattern [40]. Adverse reactions to HPV vaccines seem to be influenced by different genetic backgrounds, cultural, and/or religious conditions. These conditions with no evidence of abnormal radiological images or laboratory data are often difficult to diagnose, easily leading to a pitfall of making the diagnosis of psychiatric illness.

Nevertheless, while there is a possible occurrence of adverse effects after HPV vaccination, these results do not necessarily signal the negation of the usefulness of this vaccine for the prevention of uterine cervical cancer [41]. If the information reported in this study is provided and is widely available at the induction of HPV vaccines, a social distaste for HPV vaccination (All Japan Coordinating Association of HPV Sufferers) would likely not occur in Japan. HPV vaccines are prophylactic and are not therapeutic, and thus, serious adverse effects are not acceptable, even if their incidences are low. Wide monitoring and an open discussion are recommended to ensure the safe announcement of HPV vaccines [42].

**Author Contributions:** Conceptualization, A.H. and S.-I.I.; designed this study, A.H. and S.-I.I.; analyzed the data, A.H.; writing the manuscript, A.H. and S.-I.I. Both authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the grant from a Health and Labour Science Research Grant on Emerging and Re-emerging Infectious Diseases (Establishment of Diagnosis and Treatment System on Symptoms after HPV vaccination, grant number 19HA1006) to S.-I.I. from the Ministry of Public Health, Labour and Welfare, Japan.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board (approval nos. 4128 and 4150) of Shinshu University School of Medicine, Matsumoto, Japan.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest directly relevant to the content of this study.

#### **References**


## *Review* **A Systematic Review of Interventions to Improve HPV Vaccination Coverage**

**Edison J. Mavundza 1,\*, Chinwe J. Iwu-Jaja 2, Alison B. Wiyeh 3, Blessings Gausi 4, Leila H. Abdullahi 5, Gregory Halle-Ekane <sup>6</sup> and Charles S. Wiysonge 1,4,7**

	- Francie van Zijl Drive, Tygerberg, Cape Town 7505, South Africa

**Abstract:** Human papillomavirus (HPV) infection is the most common sexually transmitted infection worldwide. Although most HPV infections are transient and asymptomatic, persistent infection with high-risk HPV types may results in diseases. Although there are currently three effective and safe prophylactic HPV vaccines that are used across the world, HPV vaccination coverage remains low. This review evaluates the effects of the interventions to improve HPV vaccination coverage. We searched the Cochrane Central Register of Controlled Trials, PubMed, Web of Science, Scopus, and the World Health Organization International Clinical Trials Registry Platform and checked the reference lists of relevant articles for eligible studies. Thirty-five studies met inclusion criteria. Our review found that various evaluated interventions have improved HPV vaccination coverage, including narrative education, outreach plus reminders, reminders, financial incentives plus reminders, brief motivational behavioral interventions, provider prompts, training, training plus assessment and feedback, consultation, funding, and multicomponent interventions. However, the evaluation of these intervention was conducted in high-income countries, mainly the United States of America. There is, therefore, a need for studies to evaluate the effect of these interventions in low-and middle-income countries, where there is a high burden of HPV and limited HPV vaccination programs.

**Keywords:** human papillomavirus; vaccination coverage; recipient-oriented interventions; provideroriented interventions; systematic review

#### **1. Introduction**

Human papillomavirus (HPV) infection is the most common sexually transmitted infection worldwide [1]. It is estimated that 75% of sexually active men and women will acquire HPV infection in their lifetime. HPV infections are most prevalent in young adults, as sexual risk behaviors are greatest in this age group. Sexually active young women, in particular, carry the highest risk of infection, with studies documenting rates as high as 68–71% [2]. To date, more than 200 HPV types have been identified and classified into two groups: high-risk and low-risk types [3]. Although most HPV infections are transient and asymptomatic, persistent infection with high-risk HPV types may result in cancers, including cervical, anal, vulvar, vaginal, penile, and oropharyngeal cancers [4–6], and genital warts [6]. High-risk HPV types, including HPV-16, -18, -31, -33, -35, -39, -45, -51,

**Citation:** Mavundza, E.J.; Iwu-Jaja, C.J.; Wiyeh, A.B.; Gausi, B.; Abdullahi, L.H.; Halle-Ekane, G.; Wiysonge, C.S. A Systematic Review of Interventions to Improve HPV Vaccination Coverage. *Vaccines* **2021**, *9*, 687. https://doi.org/10.3390/ vaccines9070687

Academic Editor: Gloria Calagna

Received: 6 April 2021 Accepted: 24 May 2021 Published: 23 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).


Despite its effectiveness, safety, and recommendations, HPV vaccination coverage remains low. Numerous barriers to HPV vaccination have been identified, including lack of health care provider recommendations, concerns about safety, concerns about side effects, and a general lack of awareness and knowledge about HPV vaccination [12]. There is, therefore, an urgent need for effective interventions to improve HPV vaccination coverage and reduce the burden of HPV-associated infections and cancers. Several reviews have assessed interventions to improve HPV vaccination coverage. However, the reviews assessed the effectiveness of interventions among adolescents [13], young adults [14], adolescents and young adults [15], the effectiveness of practice- and community-based interventions [6], and communication technology interventions [16]. A comprehensive systematic review on interventions to increase HPV vaccination coverage was published in 2016 [17]. However, the review included only studies conducted in the United States of America. Therefore, this review's findings may not be applicable to low- and middleincome countries, where the burden of HPV is high, and vaccination coverage is very low. In addition, the review included only studies up to 2015, while there have been numerous potentially eligible studies published since then. To the best of our knowledge, there is no comprehensive systematic review that has assessed interventions to improve HPV vaccination coverage across all country income categories. These limitations justify the need for a comprehensive systematic review on the interventions to improve HPV vaccination coverage.

#### **2. Materials and Methods**

The protocol for this review was registered in the International Prospective Register of Systematic Reviews (PROSPERO) (CRD42019138971) [18], and the review was prepared according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline [19].

#### *2.1. Criteria for Considering Studies for This Review*

We included randomized trials, non-randomized trials, interrupted time-series studies, and controlled before–after studies that met the quality criteria used by the Cochrane Effective Practice and Organization of Care (EPOC) [20]. We only included cluster-randomized controlled trials with at least two intervention and two control clusters. Interrupted timeseries studies were only included if their outcomes were measured during at least three points before and after the intervention. We also included controlled before–after studies only if they had at least two intervention groups and at least two comparable control groups. We included studies conducted among all individuals eligible for HPV vaccines and their parents/legal guardians or healthcare providers. Included studies evaluated

recipient-oriented, provider-oriented, legislative, health system, and multi-component interventions. Eligible studies compared the interventions to standard HPV vaccination practices, alternative interventions, or similar interventions implemented with different degrees of intensity. Our primary outcome of interest was HPV vaccination coverage, while our secondary outcomes were adverse effects and the cost of the intervention.

#### *2.2. Search Methods for Identification of Studies*

We developed a comprehensive search strategy with the help of an information specialist. We searched the following databases: the Cochrane Central Register of Controlled Trials (CENTRAL), PubMed, Web of Science, and Scopus. We searched databases from inception until the day of the search. We searched for published articles with no language restriction. We provided the search strategies for databases searched (Appendix A, Table A1). We also searched the WHO International Clinical Trials Registry Platform for ongoing trials and the reference lists of included studies and related reviews for other relevant studies. In addition, we searched the abstracts of the latest conferences of relevant scientific societies related to vaccination and HPV virology for new or pending information not yet published in peer-reviewed journals.

#### *2.3. Selection of Studies*

Two review authors (Edison Mavundza [EM] and Chinwe Iwu-Jaja [CI]) independently screened the titles and abstracts to identify potentially eligible studies. Disagreements between the two authors were resolved by discussion and consensus. We obtained the full texts of all potentially eligible studies. Two authors independently screened the full texts and identified included studies, resolving discrepancies through discussion and consensus. Excluded studies are described in the table of excluded studies alongside their reasons for exclusion.

#### *2.4. Data Extraction and Management*

Two review authors (EM and CI) independently extracted data from each included study using a structured and standardized data extraction form. Extracted data included study setting, type of study, type of participants, type of intervention, type of comparator, and type of outcomes measured. Differences between the two review authors were resolved by discussion and consensus.

#### *2.5. Assessment of Risk of Bias in Included Studies*

Two review authors (EM and CI) independently assessed the risk of bias within each included study by addressing seven specific domains, namely, random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting, and "other issues" [21]. For each included study, the two review authors independently described what the study authors reported that they did for each domain and then made a decision relating to the risk of bias for that domain by assigning a judgement of "low risk" of bias, "high risk" of bias, or "unclear risk" of bias. The review authors compared the results of their independent assessments of risk of bias and resolved any discrepancies by discussion and consensus.

#### **3. Results**

#### *3.1. Results of the Search*

The search yielded 3936 records. After removing 1078 duplicates, 2858 titles and abstracts were screened, and 2764 were not relevant. We reviewed the remaining 94 potentially eligible full-text articles for inclusion; 49 met our inclusion criteria, and we excluded 45 articles. The 49 included publications reported data on 35 studies. The 45 excluded articles reported data on 38 studies. The process used for the search and selection of studies for this review is described in Figure 1.

**Figure 1.** PRISMA flow diagram showing the study search and selection process.

#### *3.2. Description of Studies*

The characteristics of the included studies are summarized in Appendix A, Table A2.

#### 3.2.1. Study Design and Setting

Thirty-two studies were randomized trials [22–53], two studies were controlled before– after studies [54,55], and one study was a non-randomized trial [56]. Thirty-two studies were conducted in the USA [22–35,37,39–48,50–56]. The remaining three studies were carried out in the UK [36], the Netherlands [38], and Australia [49].

#### 3.2.2. Participants

Seven studies were conducted among females only [22,24,27,33,36,37,51]; one study was conducted among males only [41]; one study was conducted among males and females [43]; thirteen studies were conducted among parents/ guardians [25,31,32,34,38–40,42,44,45,47,49,56]; nine studies were conducted among providers [23,28–30,46,52–55]. The remaining four studies were conducted among mixed participants: adolescents and parents/guardians [35,48,50] and young adults and parents/guardians [26].

#### 3.2.3. Interventions and Comparators

Twenty-six studies assessed recipient-oriented interventions [22,24–27,31–45,47–51,56]. The remaining nine studies assessed provider-oriented interventions [23,28–30,46,52–55]. Comparators ranged from the standard of care in each setting to alternative interventions.

#### 3.2.4. Outcome Measures

All included studies reported data on our primary outcome, HPV vaccination coverage. Twenty-two studies reported data on the initiation of the HPV vaccine series [22,23,25,27–37,39,41,44,48,53–56]. Nineteen studies reported data on the completion of HPV vaccine series [22,24,25,27,30,34,36,40–44,46,48,51,52,54–56]. Four studies reported data on the receipt of any HPV vaccine dose [26,38,49,50].

Only four studies reported data on our pre-specified secondary outcomes. Three studies reported data on the cost of the intervention strategies [25,45,47], and one study reported data on adverse effects of the intervention [44].

#### 3.2.5. Excluded Studies

Thirty-eight studies were excluded for reasons described in the characteristics of excluded studies (Appendix A, Table A3).

#### 3.2.6. Risk of Bias in Included Studies

The risk of bias in the included studies is summarized in Appendix A, Table A4. Below, we briefly describe the risk related to sequence generation, allocation concealment, blinding, completeness of outcome data, selective reporting, and other potential biases.

The risk of bias linked to the adequacy of the generation of the randomization sequence was low for twenty-two studies [22,23,25–28,32–34,36,37,39–41,44–49,51,52], unclear for ten studies [24,29–31,35,38,42,43,50,53], and high for two studies [55,56].

The risk of bias resulting from the adequacy of allocation concealment was low for five studies [22,27,37,46,47], unclear for fourteen studies [23,24,26,28–31,35,38,41–43,50,53], and high for fifteen studies [25,32–34,36,39,40,44,45,48,49,51,52,55,56].

The risk of bias linked to the adequacy of blinding of participants and research personnel was low for thirteen studies [22,26,32,39,42,43,45,47–49,51–53], unclear for thirteen studies [24,25,30,31,33,35–37,40,46,50,55,56], and high for eight studies [23,27–29,34,38,41,44].

The risk of bias related to the blinding of outcome assessors was low for four studies [24,26,32,40], unclear for twenty-six studies [22,25,27–31,33,36,37,39,42–53,55,56], and high for four studies [23,34,38,41].

The risk of bias linked to the completeness of outcome data was low for twenty-five studies [23,24,27–37,40,41,43–46,48–53], unclear for three studies [25,47,55], and high for six studies [22,26,38,39,42,56].

We did not find evidence of reporting bias or other biases beyond the ones reported above.

#### *3.3. Effects of Interventions*

3.3.1. Recipient-Oriented Interventions

Comparison 1: Tailored Education Compared to Standard of Care

Three studies assessed the effect of HPV-tailored education compared to the standard of care on the initiation of the HPV vaccine series. The studies showed that HPV-tailored education had no effect on the initiation of the HPV vaccine series (RR 1.00, 95% CI 0.86 to 1.17; 1350 participants) [22,27,48]. We judged the certainty of the evidence as very low because of concerns regarding the risk of bias in the included studies and serious imprecision in the findings.

Three studies assessed the effect of HPV-tailored education compared to the standard of care on the completion of the HPV vaccine series. Meta-analysis of data from these three studies showed that tailored education improved the completion of HPV vaccination series (RR 1.35, 95% CI 1.03 to 1.77; I<sup>2</sup> = 27%; 880 participants) [22,27,51]. We downgraded the certainty of the evidence to low because of study limitations (i.e., a high risk of bias in all studies).

Two studies assessed the impact of tailored education compared to the standard of care on receipt of any dose of the HPV vaccine. The study showed that tailored education had no effect on uptake of HPV vaccine (RR 1.01, 95% CI 0.98 to 1.04; 8931 participants) [26,38]. We judged the certainty of the evidence as very low because of concerns regarding the risk of bias in the included studies and serious imprecision in the findings.

The studies reported no relevant secondary outcomes.

#### Comparison 2: Tailored Education Compared to Untailored Education

One study assessed the effect of tailored education compared to untailored education on receipt of any dose of HPV vaccine. The study showed untailored education had a slight effect on uptake of HPV vaccine compared to the tailored education intervention (RR 0.97, 95% CI 0.80 to 1.19; 855 participants) [26]. We downgraded the certainty of the evidence to very low because of concerns regarding the risk of bias in the included study and serious imprecision in the findings.

The study reported no relevant secondary outcomes.

#### Comparison 3: Narrative Education Compared to Non-Narrative Education

Two studies showed that narrative education improved the initiation of the HPV vaccination series compared to non-narrative education (RR 1.38, 95% CI 0.95 to 2.00; I <sup>2</sup> = 24%; 728 participants) [33,35]. We judged the certainty of the evidence as very low because of concerns regarding the risk of bias in the included studies and very serious imprecision in the findings.

The studies reported no relevant secondary outcomes.

#### Comparison 4: Multicomponent Education Compared to Standard of Care

A study showed that a multicomponent HPV education led to a very small decrease in the uptake of HPV vaccine compared to the standard of care (RR 0.98, 95% CI 0.87 to 1.11; 2912 participants) [50]. We downgraded the certainty of the evidence to low because of concerns regarding the risk of bias in the included study and serious imprecision in the findings.

The study reported no relevant secondary outcomes.

#### Comparison 5: Outreach Plus Reminders Compared to Standard of Care

One study assessed the impact of outreach plus reminders compared to the standard of care on the initiation of the HPV vaccine series. The study showed that the intervention improved the initiation of the HPV vaccine series (RR 1.28, 95% CI 1.02 to 1.60; 1624 participants) [31]. We judged the certainty of the evidence as moderate because of an unclear risk of bias in the included study.

The study reported no relevant secondary outcomes.

#### Comparison 6: Outreach Plus Education Compared to Standard of Care

A study assessed the impact of education and outreach compared to the standard of care on the initiation of the HPV vaccine series. The study reported that 84% of participants in both groups (Brochure only and *Entre Madre e Hija* (EMH)) initiated HPV vaccination, and no differences were observed between EMH program and brochure-only participants [56]. We downgraded the certainty of the evidence to moderate because of study limitations (i.e., non-randomized study).

One study assessed the impact of education and outreach compared to standard of care on the completion of the HPV vaccine series. The study showed that the intervention improved the completion of the HPV vaccine series (RR 1.70, 95% CI 1.30 to 2.22; 288 participants) [56]. We downgraded the certainty of the evidence to moderate because of study limitations (i.e., non-randomized study).

The study reported no relevant secondary outcomes.

#### Comparison 7: Education Plus Reminders Compared to Standard of Care

A study assessed the effect of education plus reminders compared to the standard of care on the initiation of the HPV vaccine series. The study showed that the intervention improved the initiation of the HPV vaccine series (RR 1.74, 95% CI 1.10 to 2.76; 150 participants) [41]. We downgraded the certainty of the evidence to low because of study limitations, as the included study had a high risk of bias.

Another study assessed the impact of HPV education plus reminders compared to the standard of care on the initiation of the HPV vaccine series. The study showed that the intervention was significantly associated with HPV vaccine uptake (RR: 0.84; 95% CI: 0.31–2.28) [37]. We judged the certainty of the evidence as very low because of concerns regarding the risk of bias in the included study and serious imprecision in the findings.

Three studies assessed the impact of HPV education plus reminders compared to the standard of care on the completion of the HPV vaccine series. A meta-analysis of data from these three studies showed that the intervention improved the completion of the HPV vaccine series (RR 1.18, 95% CI 0.92 to 1.51; I<sup>2</sup> = 28%; 6711 participants) [41–43]. We downgraded the certainty of the evidence to very low because of concerns regarding the risk of bias in the included studies and serious imprecision in the findings.

The studies reported no relevant secondary outcomes.

#### Comparison 8: Reminders vs. Standard of Care

Three studies assessed the effect of a reminder compared to the standard of care on the initiation of the HPV vaccine series. Two studies showed that the intervention improved the initiation of the HPV vaccine series (RR 1.16, 95% CI 1.13 to 1.18; I<sup>2</sup> = 40%; 166,264 participants) [25,39]. We judged the certainty of the evidence as low because of study limitations, as the included studies had a high risk of bias.

Suh (2012) [45] reported that 26.5% of female adolescents initiated HPV vaccine series in the intervention group compared to 15.3% in the control group. We judged the certainty of the evidence as low because of study limitations, as the included studies had a high risk of bias.

Four studies assessed the effect of reminders compared to the standard of care on the completion of the HPV vaccine series. The study showed that intervention improved the completion of the HPV vaccination series (RR 1.23, 95% CI 1.18 to 1.29; I<sup>2</sup> = 63%; 175,743 participants) [24,25,40,48]. We downgraded the certainty of the evidence to very low because of concerns regarding the risk of bias and serious inconsistency in the included studies.

Tull (2019) [49] assessed the effect of reminders compared to the standard of care on the uptake of any HPV dose. The study found that the intervention had no effect on the uptake of HPV vaccine (RR 1.03, 95% CI 1.01 to 1.05; 5912 participants). We judged the certainty of the evidence as moderate because of a high risk of bias in the included study.

Three studies measured the costs of the intervention [25,45,47]. Coley (2018) [25] calculated the reminder mailing and vaccination costs. The mailing costs were \$13,698 for address verification, \$44,312 for printing, and \$57,991 for postage. The vaccination cost was \$30.95 per adolescent who initiated the HPV vaccine series. Szilagyi (2013) [47] measured the cost of the intervention on pertussis, meningococcal, and HPV vaccination among adolescents. The delivery cost of the intervention was \$18.78 for mailed and \$16.68 for phone reminders per adolescent per year, respectively. The cost per additional fully vaccinated

adolescent was \$463.99 for mailed and \$714.98 for telephone reminders. Suh (2012) [45] calculated the total operating cost of reminder/recall intervention per additional adolescent who received tetanus-diphtheria-acellular pertussis, meningococcal conjugate, or a first dose of human papillomavirus vaccine in four practices. The total operating cost, which included personnel and supply costs, ranged between \$1087 and \$1349.

#### Comparison 9: Educational Reminders Compared to Plain Reminders

Hofstetter (2017) [32] showed that educational reminders improve the initiation of the HPV vaccination series compared to plain reminders (RR 0.53, 95% CI 0.27 to 1.06; 90 participants). We downgraded the certainty of the evidence to very low because of concerns regarding the risk of bias in the included study and serious imprecision in the findings.

The study reported no relevant secondary outcomes.

#### Comparison 10: Financial Incentives Plus Reminders Compared to Standard of Care

One study assessed the impact of financial incentives plus reminders compared to the standard of care on the initiation of the HPV vaccine series. The study showed that intervention improved the initiation of the HPV vaccine series (RR 1.73, 95% CI 1.34 to 2.24; I2 = 64%; 1000 participants) [36]. We judged the certainty of the evidence as very low because of concerns regarding the risk of bias in the included study and serious inconsistency.

A study assessed the impact of financial incentives plus reminders compared to the standard of care on the completion of the HPV vaccine series. The study showed that intervention improved the initiation of the HPV vaccine series (RR 1.82, 95% CI 1.26 to 2.63; I<sup>2</sup> = 0%; 1000 participants) [36]. We downgraded the certainty of the evidence to low because of a high risk of bias in the included study.

The study reported no relevant secondary outcomes.

#### Comparison 11: Brief Motivational Behavioral Intervention Compared to Standard of Care

One study assessed the impact of the brief motivational behavioral intervention compared to the standard of care on the initiation of the HPV vaccine series. The study showed that intervention improved initiation of the HPV vaccine series (RR 1.10, 95% CI 0.85 to 1.43; 200 participants) [34]. We downgraded the certainty of the evidence to very low because of concerns regarding the risk of bias in the included study and serious imprecision in the findings.

A study assessed the impact of the brief motivational behavioral intervention compared to the standard of care on the completion of the HPV vaccine series. The study showed that intervention improved the completion of HPV vaccine series (RR 1.73, 95% CI 0.66 to 4.59; 200 participants) [34]. We judged the certainty of the evidence as very low, because of concerns regarding the risk of bias in the included studies and serious imprecision in the findings.

The study reported no relevant secondary outcomes.

#### Comparison 12: Brief Health Messaging Using Different Formats

One study assessed the effect of brief health messaging on the initiation of the HPV vaccine series. The study reported that rhetorical questions did not increase the initiation of the HPV vaccine series (RR = 1.15, CI 0.89, 1.50). One-sided and two-sided messages also had no effect on the initiation of the HPV vaccine series [44]. We downgraded the certainty of the evidence to very low because of concerns regarding the risk of bias in the included study and serious imprecision in the findings.

A study assessed the effect of brief health messaging on the completion of the HPV vaccine series. The study reported that rhetorical questions and message sidedness had no significant effect on the completion of the HPV vaccine series [44]. We judged the certainty of the evidence as very low because of concerns regarding the risk of bias in the included studies and serious imprecision in the findings.

Rickert (2015) evaluated the adverse events of the intervention, but none occurred.

#### 3.3.2. Provider-Oriented Intervention

Comparison 13: Prompts Compared to Standard of Car

One study assessed the impact of provider prompts compared to the standard of care on the initiation of the HPV vaccine series. The study showed that provider prompts improved the initiation of the HPV vaccine series (RR 1.36, 95% CI 1.20 to 1.54; 925 participants) [53]. We downgraded the certainty of the evidence to moderate because of study limitations, as the included study had an unclear risk of bias.

Two studies assessed the effect of provider prompts compared to the standard of care on the completion of the HPV vaccine series. The study showed that intervention improved the completion of the HPV vaccine series (RR 1.12, 95% CI 1.06 to 1.19; I <sup>2</sup> = 72%; 3056 participants) [46,52]. We downgraded the certainty of the evidence to very low because of concerns regarding the risk of bias in the included studies and serious inconsistency.

The studies reported no relevant secondary outcomes.

#### Comparison 14: Provider Training Compared to Standard of Care

A study assessed the effect of provider announcement and conversation training compared to the standard of care on the initiation of the HPV vaccine series. The study reported that clinics that received announcement training had increases in HPV vaccine initiation coverage that exceeded control clinics' increases (5.4% difference, 95% CI 1.1 to 9.7). Clinics that received conversation training did not differ from the control arm on uptake for HPV vaccine initiation (all *P*s > 0.05) [23]. We judged the certainty of the evidence as very low because of concerns regarding the risk of bias in the included studies and serious imprecision in the findings.

The study reported no relevant secondary outcomes.

Comparison 15: Provider Training Plus Assessment and Feedback Compared to Wait List Control

One study assessed the impact of provider training plus assessment and feedback intervention compared to wait list control on the initiation of the HPV vaccine series. The study showed that initiation of the HPV vaccine series rates increased by 10.2 percentage points in the intervention arm and 6.9 percentage points in the control arm [29]. We downgraded the certainty of the evidence to very low because of concerns regarding the risk of bias in the included studies and very serious imprecision in the findings.

The study reported no relevant secondary outcomes.

#### Comparison 16: Assessment and Feedback Compared to Standard of Care Series

Irving (2018) [54] evaluated the effect of assessment and feedback intervention compared to the standard of care on the initiation of the HPV vaccine series among adolescent boys and girls aged 11–17 years. The study reported that there was no significant difference in the initiation of the HPV vaccine series between intervention and control clinics. We downgraded the certainty of the evidence to very low because of study limitations (i.e., before–after study).

One study evaluated the effect of assessment and feedback intervention compared to the standard of care on the initiation of the HPV vaccine series among adolescent boys and girls aged 11–17 years [54]. The study found that the completion of the HPV vaccine series between the intervention and control clinics was not significantly different. We downgraded the certainty of the evidence to very low because of study limitations (i.e., before–after study).

The study reported no relevant secondary outcomes.

Comparison 17: Provider Consultation Compared to Standard of Care

A study assessed the effect of in-person and webinar-delivered Assessment, Feedback, Incentives, and eXchange (AFIX) consultations compared to standard of care on the initiation of the HPV vaccine series. The study reported that participants served by clinics in the in-person arm had uptake that exceed those in the control arm for HPV vaccine initiation (1.5% (95% CI: 0.3 to 2.7)). Participants served by clinics in the webinar versus control arms also had larger coverage increases for HPV vaccine initiation (1.9 (95% CI: 0.7 to 3.1)) [28]. We downgraded the certainty of the evidence to very low because of concerns regarding the risk of bias in the included study and very serious imprecision in the findings.

The study reported no relevant secondary outcomes.

#### Comparison 18: Funding Compared to Training and Technical Assistance

One study compared the effect of \$90,000 (2-year grant fund), \$10,000 (3-month grant fund), and training and technical assistance on the initiation of the HPV vaccine series among patients aged 11–12 years. The study found that initiation of the HPV vaccine series rates increased by 18.4, 14.6, and 11.1 percentage points in the \$90,000 grant fund, training and technical assistance, and \$10,000 grant fund, respectively [30]. We judged the certainty of the evidence as low because of concerns regarding the risk of bias in the included study and serious imprecision in the findings.

A study compared the effect of \$90,000 (2-year grant fund), \$10,000 (3-month grant fund), and training and technical assistance on the completion of the HPV vaccine series among patients aged 11–12 years. The study reported that completion of HPV vaccine series rates increased only in the \$90,000 grant fund by 5 percentage points and decreased by 4.5 and 1.7 percentage points in the \$10,000 grant fund and training and technical assistance arm, respectively [30]. We judged the certainty of the evidence as low because of concerns regarding the risk of bias in the included study and serious imprecision in the findings.

The study reported no relevant secondary outcomes.

Comparison 19: Multicomponent Intervention Compared Standard of Care

One study assessed the impact of a multicomponent intervention compared to the standard of care on the initiation of the HPV vaccine series among adolescents aged 11–12 and 13–17 years. Among adolescents aged 11–12 years, HPV vaccine series initiation rates increased by 18.7 percentage points in the intervention arm and 12.6 percentage points in the control arm, whereas, among adolescents aged 13–17 years, the rates increased by 8.7 percentage points in the intervention arm and 7 percentage points in the control arm [55]. We downgraded the certainty of the evidence to very low because of study limitations (i.e., before–after study).

A study assessed the impact of a multicomponent intervention compared to the standard of care on the completion of the HPV vaccine series among adolescents aged 11–12 and 13–17 years. HPV vaccine series completion rates among adolescents aged 11–12 years increased by the same 20.7 percentage points both in the intervention and control arms, whereas, among adolescents aged 13–17 years, the completion rates increased by 12.5 percentage points in the intervention and 11.9 percentage points in the control arms [55]. We downgraded the certainty of the evidence to very low because of study limitations (i.e., before–after study).

The study reported no relevant secondary outcomes.

#### **4. Discussion**

Our study found that recipient-oriented interventions that improved the initiation of the HPV vaccine series were narrative education, reminders, outreach plus reminders, education plus reminders, financial incentives plus reminders, and brief motivational behavioral interventions. We also found that the recipient-oriented interventions that improved the completion of the HPV vaccine series were tailored education, outreach

and education, education plus reminders, reminders in general, financial incentives plus reminders, and brief motivational behavioral interventions. Tailored education, outreach and education, and brief health messaging were recipient-oriented interventions that had no effect on the initiation of the HPV vaccine series. Brief health messaging was also found to be a recipient-oriented intervention that had no effect on the completion of the HPV vaccine series. The provider-oriented interventions that improved the initiation of the HPV vaccine series were prompts, training, training plus assessment and feedback, consultation, funding**,** and multicomponent interventions. Prompts, funding and multicomponent were also found to be provider-oriented interventions that improved the completion of HPV vaccine series. Assessment and feedback were provider-oriented interventions that had no effect on both the initiation and the completion of the HPV vaccine series. With regards to the improvement of uptake of any HPV vaccine dose, all assessed recipient-oriented interventions, tailored education, untailored education, multicomponent education, and reminders did not have any effect.

Our systematic review was comprehensive. We included all known types of interventions, including recipient- and provider-oriented interventions, and all country settings. Our comprehensive search resulted in 35 studies that met our inclusion criteria. However, all studies were conducted in high-income countries, mainly the USA, where the burden of HPV is relatively low. None of the included studies were conducted in low-income countries, where the burden of HPV is very high. Therefore, the findings of these studies may be applicable only in the settings of the high-income countries. Another limitation is that there is very small number of studies that reported data on our secondary outcomes. Among the included studies, there were only one and three studies that reported data on the adverse effects and the cost of the interventions, respectively. However, because of variations in the measures of costs between the three studies, we were unable to conduct a meta-analysis. One study that reported on the adverse effects of the intervention stated that there were no effects documented in the study. Given that there is insufficient data on adverse effects and costs of the interventions, there is an urgent need for more studies to address these gaps. In addition, these studies should be well-designed and should evaluate outcomes and report results in ways that will allow the clear assessment of the cost and adverse effects of the interventions.

Thirty-five studies were excluded in this review mainly based on the methods used to conduct them. In addition, most of these studies were published after 2015, the period in which a previous similar review by Smulian (2016) [17] included studies up to. We may therefore have missed important findings from these studies. Well-designed studies that assess the effect of the interventions on HPV vaccination coverage are needed.

We used the Grading of Recommendations, Assessment, Development and Evaluations (GRADE)approach to assess the certainty of the evidence on the effects of the included interventions on HPV vaccination coverage. Among the recipient-oriented interventions that improved HPV vaccination coverage, we judged the certainty of the evidence as moderate for outreach plus reminders, low for reminders, and very low for education, financial incentives plus reminders, and brief motivational behavioral interventions. Regarding provider-oriented interventions that improved HPV vaccination coverage, we judged the certainty of the evidence as moderate for provider prompts, low for funding, and very low for training, consultation, training plus assessment and feedback, consultation, and multicomponent interventions. Overall, the certainty of evidence of interventions that improved HPV vaccination coverage was very low to moderate. Our main concerns with the evidence related to study limitations: risk of bias, indirectness, and imprecision in the studies. There is, therefore, an urgent need for well-designed, well-implemented, and well-reported studies to increase the certainty of the current evidence. We minimized potential biases in the review process by adhering to the Cochrane guidelines for conducting a systematic review [21]. We conducted comprehensive searches of both peer-reviewed and grey literature, without limiting the searches to a specific language. Two review authors

independently assessed study eligibility, extracted data, and assessed the risk of bias in each included study. We are not aware of any biases in the review process.

Several systematic reviews have assessed the effectiveness of interventions for improving HPV vaccination coverage [6,13–17,57]. Smulian (2016) [17] evaluated the effectiveness of the interventions for improving HPV vaccination coverage in USA. The review found that many types of intervention strategies (targeting recipients, providers, and the health system) increased HPV vaccination coverage in different settings. Contrary to our review, which included 35 studies, this similar comprehensive review, which searched five databases for studies published between 2006 to 2015, resulted in 34 eligible studies. Like their review, all the studies included in our review were conducted in high-income countries. Of the 35 studies included in our review, 32 were conducted in the USA and the remaining three were from Australia, the Netherlands, and the UK. Acampora (2020) [13] and colleagues evaluated the effectiveness of interventions for improving HPV vaccination coverage among adolescents. The authors found that reminder-based interventions, either alone or in combination with other interventions, had a positive effect on vaccination coverage [13]. In another review, the effectiveness of intervention for improving HPV coverage among college students was assessed. The authors reported that the educational intervention that utilized a joint peer and medical provider message was the only intervention in their review that significantly increased HPV vaccine uptake [14]. The effectiveness of communication technology interventions on HPV vaccination coverage was assessed by Francis (2017) [16] and found that usage of computer, mobile, or internet technologies as the sole or primary mode for intervention delivery increased vaccination coverage. Niccolai (2015) [6] conducted a systematic review to assess the effectiveness of practice- and community-based interventions on improving HPV vaccination coverage. The review reported that several interventions including reminder and recall systems, physician-focused strategies (e.g., audit and feedback), school-located programs, and social marketing have improved vaccination coverage. The effectiveness of the interventions that applied new media to improve vaccination coverage was assessed by Odone and colleagues. The authors reported that text messaging, accessing immunization campaign websites, using patient-held web-based portals and computerized reminders, and standing orders increased vaccination coverage rates [57]. Walling and colleagues compared the effectiveness of the informational-, behavioral-, and environmental-based interventions on improving HPV vaccination coverage among adolescents and young adults aged 11 to 26 years. The authors found that environmental interventions, particularly school-based vaccination programs were most effective in increasing vaccination coverage [15].

#### **5. Conclusions**

Although several interventions improved HPV vaccination coverage, the certainty of the evidence varied from moderate to low. Although many studies were included in our review, all of them were conducted in high-income countries. There is, therefore, a need for further high-quality studies in low- and middle-income countries. At the same time, many studies assessing the effect of different interventions on improving HPV vaccination coverage were excluded because of the way they were conducted. As a result, well-designed, well-implemented, and well-reported studies are needed. In addition, given that there is limited information from existing studies on the cost of the tested interventions, further studies are needed to address this challenge.

**Author Contributions:** Conceptualization, E.J.M., C.J.I.-J. and C.S.W.; methodology and analysis, E.J.M. and C.S.W.; writing—original draft preparation, E.J.M.; writing—review and editing, E.J.M., C.J.I.-J., A.B.W., B.G., L.H.A., G.H.-E. and C.S.W. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the South African Medical Research Council.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors would like to thank Elizabeth Pienaar at Cochrane South Africa for assisting with the search strategy.

**Conflicts of Interest:** The authors declare no conflict of interest.

**Appendix A**

**Table A1.** Search strategies (search date: 9 July 2019).



**Table A2.**

Characteristics

 of included studies.










#### **Table A4.** Risk of bias summary.




#### **References**

1. Loke, A.Y.; Kwan, M.L.; Wong, Y.-T.; Wong, A.K.Y. The Uptake of Human Papillomavirus Vaccination and Its Associated Factors Among Adolescents: A Systematic Review. *J. Prim. Care Community Health* **2017**, *8*, 349–362. [CrossRef]

2. Perez, G.K.; Cruess, D.G.; Strauss, N.M. A brief information–motivation–behavioral skills intervention to promote human papillomavirus vaccination among college-aged women. *Psychol. Res. Behav. Manag.* **2016**, *9*, 285–296. [CrossRef]

3. Fontes, A.; Andreoli, M.A.; Villa, L.L.; Assone, T.; Gaester, K.; Fonseca, L.A.; Duarte, A.J.; Casseb, J. High specific immune response to a bivalent anti-HPV vaccine in HIV-1-infected men in São Paulo, Brazil. *Papillomavirus Res.* **2016**, *2*, 17–20. [CrossRef]

4. Holman, D.M.; Benard, V.; Roland, K.; Watson, M.; Liddon, N.; Stockley, S. Barriers to human papillomavirus vaccination among US adolescents: A systematic review of the literature. *JAMA Pediatr.* **2014**, *168*, 76–82. [CrossRef]


#### *Review*

## **Internal and External Validity of Social Media and Mobile Technology-Driven HPV Vaccination Interventions: Systematic Review Using the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) Framework**

**Matthew Asare 1,\*, Braden Popelsky 1, Emmanuel Akowuah 1, Beth A. Lanning <sup>1</sup> and Jane R. Montealegre <sup>2</sup>**


**Abstract:** Social media human papillomavirus (HPV) vaccination interventions show promise for increasing HPV vaccination rates. An important consideration for the implementation of effective interventions into real-world practice is the translation potential, or external validity, of the intervention. To this end, we conducted a systematic literature review to describe the current body of evidence regarding the external validity of social media HPV vaccination-related interventions. Constructs related to external validity were based on the reach, effectiveness, adoption, implementation, maintenance (RE-AIM) framework. Seventeen articles published between 2006 and 2020 met the inclusion criteria. Three researchers independently coded each article using a validated RE-AIM framework. Discrepant codes were discussed with a fourth reviewer to gain consensus. Of these 17 studies, 3 were pilot efficacy studies, 10 were randomized controlled trials (RCTs) to evaluate effectiveness, 1 was a population-based study, and 3 did not explicitly state which type of study was conducted. Reflecting this distribution of study types, across all studies the mean level of reporting RE-AIM dimensions varied with reach recording 90.8%, effectiveness (72.1%), adoption (40.3%), implementation (45.6%), and maintenance (26.5%). This review suggests that while the current HPV vaccination social media-driven interventions provide sufficient information on internal validity (reach and effectiveness), few have aimed to gather data on external validity needed to translate the interventions into real world implementation. Our data suggest that implementation research is needed to move HPV vaccination-related interventions into practice. Included in this review are recommendations for enhancing the design and reporting of these HPV vaccination social media-related interventions.

**Keywords:** HPV; HPV vaccine; social media; mobile phone; HPV vaccine intervention; RE-AIM Framework

#### **1. Introduction**

The human papillomavirus (HPV) vaccine protects against HPV-associated cancers, including most cervical cancer, as well as vulvar, vaginal, anal, penile, and oropharyngeal cancer. Cervical cancer is the fourth most common cancer among women worldwide, with approximately 570,000 new cervical cancer cases reported in 2018, representing 6.6% of female cancers [1]. The incidence of oral and anal cancers is increasing [2,3]. The HPV vaccine is recommended for adolescents aged 11–12 years, with catch-up vaccination through age 26 and FDA approval for adults up to age 45 years [4]. As part of the Global Strategy for the Elimination of Cervical Cancer as a Public Health Problem, the World Health Organization's (WHO) goal is for 90% vaccination of girls age 15 by 2030. The Healthy People 2030 goal is to increase the proportion of adolescents who receive

**Citation:** Asare, M.; Popelsky, B.; Akowuah, E.; Lanning, B.A.; Montealegre, J.R. Internal and External Validity of Social Media and Mobile Technology-Driven HPV Vaccination Interventions: Systematic Review Using the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) Framework. *Vaccines* **2021**, *9*, 197. https://doi.org/10.3390/ vaccines9030197

Academic Editor: Gloria Calagna

Received: 27 January 2021 Accepted: 23 February 2021 Published: 26 February 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

recommended doses of the HPV vaccine with a target goal of 80% [5]. Although integrated programs and efforts to increase the HPV vaccination have occurred in many countries over the last 14 years, HPV vaccine rates remain low [6]. For instance, in 2019, only 54.2% of adolescents in the US [7] and 15% of adolescents globally [8] were current on HPV vaccinations.

Multiple factors including lack of opportunity for vaccination, parental attitudes or perceptions towards vaccination, lack of recommendations from healthcare providers, concerns about the vaccine's effect on sexual behavior, religious objection, low perceived risk of HPV infection, social influences, irregular preventive care, and vaccine cost have contributed to the low vaccination rates [9–11]. In efforts to address these barriers, researchers have included mobile technology-related media including text-message, e-mail, phone calls, and private Facebook messages, in their interventions to increase vaccination awareness, uptakes, and dose completion [12,13]. These interventions have been efficacious in increasing HPV vaccination uptake and completion [12–15]. However, the prospect of translating these efficacious interventions into regular clinical practice is unknown due in part to the lack of reported external validity [16]. External validity is the ability to generalize an evidence-based study to different measures, persons, settings, and times [17]. Several translational researchers have argued that reporting detailed components and processes of evidence-based studies would increase studies' generalizability (external validity) and the ability to translate those interventions into practice [18–20].

To address the research-practice issue, Glasgow and colleagues developed the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework [21] with a set of metrics critical for evaluating the generalizability of an evidence-based intervention into routine practice. They proposed that the translatability of an intervention is best evaluated through the five dimensions (RE-AIM). The reach dimension is designed to assess the proportion of potentially eligible individuals who participate in the intervention study. Efficacy/effectiveness is the function of the intended positive impact of the intervention. Adoption reflects the potential settings and intervention agents that participate in a study. Implementation refers to the quantity and quality of delivery of the intervention's various components. Finally, the maintenance dimension is the longer-term efficacy/effectiveness of an intervention on an individual (see Table 1 for the definitions and indicators for each RE-AIM dimension). This framework can be used to organize and evaluate threats to the transferability of research to practice. Further, by evaluating a study through the five-dimension lens of the framework, researchers are able to assess internal and external validity equally [16,21].

The RE-AIM framework has been used recently in several systematic reviews to evaluate the internal and external validities of health intervention studies such as weight management intervention [18], physical activity intervention [22,23], worksite health behavior interventions [24], community settings [20], school-based health promotion [25,26], childhood obesity prevention [27–29], children dietary interventions with parents [30], injury prevention strategies [31], faith-based intervention [32], mobile phone-based intervention for diabetes self-management [33], and HIV prevention intervention [34]. While encouraging, there is little reporting on its potential use for translating HPV vaccination social media-driven intervention methods into regular practice settings, specifically at the population level. The two most recent comprehensive literature reviews on HPV vaccination mobile technology-related interventions evaluated the effectiveness of interventions in increasing the HPV vaccine [35], and the effectiveness of communication technology interventions on HPV vaccine [36]. However, the scope of these two previous literature reviews was narrow, focusing primarily on internal validity with limited information on the external validity of the HPV vaccination intervention studies [18,19].


**Table 1.** Average scores for the RE-AIM dimensions and the scores for each 33-item indicator as used in the current


**Table 1.** *Cont.*

Our study aimed to use the RE-AIM framework to evaluate HPV vaccination intervention studies that included mobile technology to increase HPV vaccination completion (i.e., receiving all the recommended doses: two doses for 9–14 years old and three doses for those between 15 and 45 years old) and/or vaccination uptake (defined as receiving at least one dose of vaccine). To our knowledge, this is the first review using the RE-AIM framework to evaluate the HPV vaccination mobile technology-related (Facebook, text messaging, and mobile health (miHealth)) interventions. Unlike the two previous reviews on HPV vaccine interventions [35,36], our current systematic review was structured to determine the translation potential or external validity of published HPV vaccination intervention studies by determining the extent to which those studies reported information across all five of the RE-AIM framework dimensions. We further provide recommendations for future research based on these findings.

#### **2. Materials and Methods**

#### *2.1. Search Strategy and Selection Criteria*

We conducted an extensive literature search to identify research articles related to HPV vaccination technology-based interventions. We searched nine databases (PubMed, EMBASE, Medline, ERIC, CINAHL, Academic Search Complete, Web of Science, PsycINFO, Cochrane Library) using the following terms: human papillomavirus OR human papillomavirus\* OR HPV and social media OR social medium OR Web 2.0 OR twitter messaging OR Instagram OR Facebook OR WhatsApp OR Tito OR text message OR mobile technology AND intervention OR RCT. An article was included in the review if it met the following inclusion criteria: published in English, between 2006 and 2020, and in peer-reviewed journals; outcome variables included HPV vaccination completion and/or vaccination uptake; intervention study; the intervention's mode of delivery included social media (WhatsApp, Facebook) and text messages. Articles were excluded if they were cross-sectional studies and included assessment of only participants' knowledge, attitude, and intention (see Figure 1).

**Figure 1.** Flowchart of the search strategy.

#### *2.2. RE-AIM Criteria*

A modified version of the RE-AIM 30-item data extraction tool (https://www.reaim.org, accessed on 10 February 2021) was used to code eligible articles on the degree to which internal and external validity indicators were reported. The RE-AIM dimensions and corresponding indicators are listed in Table 1.

#### 2.2.1. Reach Dimension

Seven indicators were used to evaluate the reach dimension of the study. They included the description of the target population, description of the participants' HPV vaccination behavior, recruitment strategies, inclusion and exclusion criteria, description of the sample size determination, and participation rate.

#### 2.2.2. Effectiveness Dimension

Eight metrics for effectiveness included the efficacy of the intervention in changing vaccination behavior, measurement of primary and/or secondary outcome (i.e., vaccination completion, or uptake), a short-term assessment, intent-to-treat assessment, description of imputation procedure, the measure of robustness across subgroups, and short-term attrition assessment.

#### 2.2.3. Adoption Dimension

The seven indicators used for adoption were a description of intervention location, intervention delivery staff, the method used to identify delivery staff, inclusion and exclusion criteria for the staff, and rate of staff participation.

#### 2.2.4. Implementation Dimension

The four metrics used for the implementation dimension were intention frequency, duration, the extent to which the intervention was implemented as planned, completion rates, and measurement of cost of implementing the intervention.

#### 2.2.5. Maintenance Dimension

The four metrics for maintenance included follow-up (3 and 6 months) assessments, attrition rates, continuation, and institutionalization of the program.

#### *2.3. Coding and Analysis*

Articles that met the inclusion criteria for this review were independently coded by three graduate research assistants (B.P., N.T., and C.M.) and supervised by the principal investigator (PI) of the research team (M.A.). Each reviewer coded a "yes" indicating the presence or "no," indicating the absence of the RE-AIM indicators outlined above. Following the individual coding, the PI and the three research assistants met to discuss articles and coding results, resolve uncertainty, and gain consensus in coding. The articles that had "yes" for any indicator were scored as 1 and "no" was scored as 0. Each of the 17 articles was tabulated and scored with a column totaling individual dimension scores for each article. Analyses included providing count and percentage data across RE-AIM indicators. Row percentages were calculated to display the proportion of articles addressing each of the dimension indicators. Finally, column totals, averages, and average percentages were computed to summarize the number of articles reporting each of the five dimensions. To determine the overall quality of RE-AIM reporting, we also examined the number of articles that included the 33 indicators from the data extraction tool (see Table 1). Based on the 33-item RE-AIM indicators, articles that scored between 0 and 11 indicated fewer reporting of RE-AIM indicators, between 12 and 22 indicated moderate reporting, and between 23 and 33 indicated high reporting of RE-AIM indicators.

#### **3. Results**

The literature search yielded 414 total articles. After removing duplicate articles and conducting a preliminary screening process based on title and abstract, a total of 53 eligible articles remained. The secondary screenings restricted articles to HPV vaccine interventions and social media-driven interventions. The complete article identification strategy produced 17 articles that met the inclusion criteria and were analyzed in this review [15,37–52]. Of these 17 studies, 3 were pilot efficacy studies [39,44,51], 10 were randomized controlled trials (RCTs) to evaluate effectiveness [15,38,43,45–50,52], 1 was a population-based study [41], and 3 did not explicitly state which type of study was conducted [37,40,42]. The target population for the reviewed articles includes parents with adolescents (boys and girls) [15,45,46,50], Young Sexual Minority Men [51,52], college students [37,38,44,47–49], adolescents (boy and girls) [40–42], young women (19–26 years) [39,43]. The vaccination uptake rates in the review ranged from 6.6% to 89% and the completion rates range from 17% to 88%, indicating successful implementation of many of the interventions. A total number of 189,877 participants were reached in the reviewed HPV vaccine interventions. Out of 189,769, (we excluded 108 participants in Ortiz et al.'s study because they did not provide absolute numbers for vaccination uptake) participants enrolled in the re-viewed articles, pooled estimate of 19,294 participants in studies received at least one dose of HPV vaccine representing a 10.2% vaccination rate among the participants in those reviewed articles. Mohanty et al. [41] and Chodick [50] used Facebook to deliver the intervention and they reached the largest target population or had highest penetrations (155,110 and 21,592 participants, respectively) and text messaging interventions. The most common social media used were mobile phone text messaging [39,46,49,52], combination of text messaging and email systems [15,43,47,48], Facebook [41,42,50], and mobile web technology [38,40]. Other studies mentioned that they used social media but did not mention specific social media [37,44,51]. Supplementary Table S1 shows the details on study design, outcome, demographic characteristics, social media used, and RE-AIM indicators used.

#### *3.1. RE-AIM Reporting Scores*

Using RE-AIM rating procedures, we found that 14 (82.35%) articles moderately reported RE-AIM indicators [15,37,39–46,48,50–52], and three articles (17.65%) had high reporting of RE-AIM indicators [38,45,47,49]. The three studies that were rated as high quality addressed between 69.70% (23 out of 33) and 72.73% (24 out of 33) of the indicators. The 14 medium reporting quality articles addressed between 42.42% (14 out of 33) and 66.67% (22 out of 33) of the indicators. The reach and effectiveness dimensions were the most addressed domains with an average score of 90.8% and 85.8%, respectively. The adoption, implementation, and maintenance domains were the least addressed domains with average scores of 38.2%, 45.6%, and 26.48%, respectively (see Table 1).

#### 3.1.1. Reach Dimension

The indicators with the greatest reporting scores under the reach dimension were description of the target population, including race/ethnicity and other demographic information (100%), behavioral information (100%), inclusion/exclusion criteria (94.1%), and recruitment strategies (94.1%). The least reported reach indicator was the method to identify the target population with an average score of 76.5% (see Table 1). Overall, ten articles [15,38,41,42,46–49,51,52] reported all seven indicators under the reach dimension, four studies [37,40,44,45] reported the least (5 out of 7) of the indicators in the reach dimension (see Figure 2).

**Figure 2.** The scores for the reviewed articles on the RE-AIM Dimensions (*n* = 17). Note: Reach scale 0–7; Efficacy scale 0–10; Adoption scale 0–8; Implementation and Maintenance scales 0–4.

#### 3.1.2. Efficacy/Effectiveness Dimension

The most common efficacy/effectiveness dimension indicators reported by the reviewed articles include intervention effectiveness in changing behavior (100%), measurement of the primary outcome (100%), short-term assessment (94.1%), and description of intervention design/conditions (100%). Indicators such as intent-to-treat, imputation procedure, quality of life, unintended consequence measurement, a measure of robustness across subgroups, and measures of short-term attrition were scarcely reported in the articles (see Table 1). Three articles reported eight out of 10 effectiveness dimension indicators [38,43,47]. Fontenot et al. [51] and Mohanty et al. [41] reported the least amount, 4 out of 10 indicators (see Figure 2).

#### 3.1.3. Adoption Dimension

For the adoption dimension, the most reported indicators were the description of intervention location (88.2%) and the staff who delivered intervention (64.7%). Organizational spread and measures of the cost of adoption were two adoption indicators that received the lowest reporting score of 5.9% (see Table 1). Only Gerend et al.'s [37] article reported six out of the eight adoption dimension indicators. The remaining articles reported a few of the adoption dimension indicators with scores ranging from 1 to 5 (out of 8 indicators) (see Figure 2).

#### 3.1.4. Implementation

Per our inclusion criteria for this review, all 17 articles included in the review used some form of technology including social media (Facebook), mobile phone (text messages), and emails to deliver the intervention. The most addressed implementation dimension indicators were intervention frequency and duration (88.2%) and participation/completion rates (82.4%) (see Table 1). The extent to which the protocol was delivered as intended was the least reported indicator (11.8%) and none of the studies reported the total cost of implementing the intervention. However, eight reported incentives given to each participant, and one reported that vaccines were given to participants at no cost (see Figure 2).

#### 3.1.5. Maintenance

The most common maintenance dimension indicator addressed in the reviewed articles was the 3-month and 6-month follow-up assessment (70.6%). The long-term attrition rate was addressed in 35.3% of articles. None of the articles addressed whether the programs were institutionalized or were still in place (see Table 1).

#### **4. Discussion**

#### *4.1. Summary*

This systematic review utilized the RE-AIM framework to evaluate the impact of HPV vaccine intervention studies that incorporated mobile technology to increase vaccine uptake and recommended dose completion. Eleven articles' outcome of measure included vaccination uptake [37–39,41,44,46,47,49–52], and six articles' outcome measure was vaccination completion [15,40,42,43,45,48]. The vaccination uptake rates in the review ranged from 6.6% to 89% and the completion rates range from 17% to 88%, indicating successful implementation of many of the interventions.

Our review of 17 articles showed the emphasis on reporting internal validity (i.e., reach and efficacy), and the collective absence of reporting external validity per Glasgow et al.'s (1999) reporting criteria. This finding is consistent with other RE-AIM literature reviews that found that the most common indicators reported in studies are reach and efficacy/effectiveness [23,25–28,32,53]. On the other hand, the external validity dimensions which include adoption, implementation, and maintenance were underreported. Therefore, making the translation of those intervention studies into the practice setting difficult.

#### 4.1.1. Reach Dimension

A total number of 189,877 participants were reached in the reviewed HPV vaccine interventions. Lee et al.'s [39] article reached the lowest number of participants (*n* = 30) while Mahanty et al.'s [41] Facebook intervention reached the highest number of participants (*n* = 155,110). The reporting for the reach dimension indicators ranges from 77% to 100%. Many of the articles included in this review provided detailed descriptions of the target population which is consistent with the literature [25,27–30,34,54,55]. Several articles targeted college students [37,38,43,44,48] which is not surprising given that the majority of those within the college-age groups are social media consumers. A few of the articles reviewed reached or targeted parents with adolescents (boys and girls) [15,45,46,50]. Parents are critical target audience for any HPV vaccination interventions because parental knowledge, positive attitude, affordability, and willingness are precursor to successful HPV vaccination programs [56–59]. The reach dimension indicator rarely discussed is the target population denominator. While reporting of this indicator tends to be challenging [55], by not reporting the target population denominator, there is no context given to help determine the sample sizes. In the efficacy studies the concern is recruiting enough participants to provide the necessary power to detect effect size; therefore, understanding the target population that was exposed to recruitment materials can provide an estimate of the likely reach the program will achieve [18].

#### 4.1.2. Efficacy/Effectiveness

Out of 189,769 participants enrolled in the reviewed articles, a pooled estimate of 19,294 received at least one dose of HPV vaccine, representing a 10.2% vaccination rate among the participants in those reviewed articles. Tull et al.'s [49] study reported the highest vaccination rates (motivational arm 88%, self-regulatory arm 89%, and control arm 86%). Our systematic review showed that four indicators in the Efficacy/Effectiveness dimension

including design/conditions, efficacy or effectiveness, the measure of the outcome with or without comparison to the vaccination goal, and short-term assessments, were regularly reported which is consistent with the literature [25,27–30,34,54,55]. Across all 17 articles, less than half the authors described their chosen method of analysis for missing variables and/or attrition whether to use intent-to-treat or per-protocol analysis (analysis by treatment administered) approach. The intention-to-treat principle states that all randomized participants are included in the statistical analysis and analyzed according to the group they were originally assigned, regardless of what treatment (if any) they received [60]. Whenever treatment groups are not analyzed according to the group to which they were originally assigned, the risk of bias increases [61]. Similar to the conclusion drawn by Hollis and Campbell [58] the intention to treat approach is often insufficiently described and inadequately applied. This lack of reporting can diminish the accurate (unbiased) conclusions regarding the effectiveness of an intervention [61]. Authors should explicitly describe the handling of any deviations from randomized allocation and discuss missing responses and their potential effect on the studies' outcomes [62]. However, the use of randomization in some of the studies [15,38,40,42,43,45,52] may attenuate the possible confounding bias. Although most of the articles reported short-term pre-post assessments, only four articles [38,43,45,53] calculated short-term attrition rates. Attrition prevents a full intention to treat analysis and it can occur when participants have missing data and/or loss to follow-up. We argue that researchers need to be more explicit about the loss to follow-up, especially if rates are high [63].

#### 4.1.3. Adoption Dimension

The average score of the adoption dimension was 38% with observed scores for the adoption indicators ranging from 6% to 88%. The description of the location of the intervention score (88%) for our reviewed articles is higher than the previous reviews scores of 48% to 60% [18,54], but lower when compared with the scores in Allen et al.'s [16] review findings. The description of the delivery staff received the second-highest in reporting (65%). The skill sets of the intervention staff, the description of the setting of the intervention staff, and information about the staff can help determine the translating potentials of the study in another setting. Less than half of the articles discussed the methods used to identify target delivery agents and the level of expertise. Crucial to the implementation of an intervention is the selection of the target delivery agents but indicators such as inclusion and exclusion criteria for selecting delivery staff, delivery staff participation rate, organizational spread, and measure of the cost of adoption were underreported. The underreporting of selection criteria of delivery agents has been debated by other reviews [16,27]. By not reporting a level of expertise or a method to select the interventionists, it becomes difficult not only to measure the intervention effectiveness but also to translate those studies to other settings to achieve the same level of success. Rarely reported is the organizational spread dimension indicator, which measures if the intervention used a multi-level approach. This is an important indicator because of the focus on collaboration and communication between multiple levels of an organization which can increase the overall impact of the intervention and the probability of the behavior being adopted and/or maintained. The full support of adopting agents directly influences implementation fidelity and program sustainability [64].

#### 4.1.4. Implementation

In the implementation dimension, we were interested in intervention studies that incorporated social media technology to deliver the intervention. Although the authors did not evaluate whether using a specific social media type was effective in delivering the intervention, based on our review we found that interventions that utilized Facebook [41,50] and text messages [15,45,49] reported significant improvements in HPV vaccination. The most frequent technology used was text messaging [15,37,39,43,45–49]. Additionally, only eight articles reported the underlying theoretical framework used in intervention development [38,39,41,42,49–52]. Theories provide a systematic view of phenomena by specifying the relationship between program inputs (resources), program activities (how the program is implemented), and their outputs or outcomes [65,66]. Reporting the theoretical framework used in intervention could facilitate replication and implementation of HPV intervention studies in several different settings.

The HPV vaccine intervention studies included in this review required multiple visits and multiple contacts (reminders) with the study population. Omitting information about the number of contacts or the exact reminder messages introduces a bias. The types of reminders and the difficulty in reaching vaccination sites can greatly influence vaccination uptake. Additionally, the impact of the intervention is weakened by the lack of reporting on the extent to which the protocol was delivered as intended, including sending reminders to participants. Furthermore, none of the articles included the costs incurred during the implementation phase of the interventions, which once again eliminates a monetary reference point to consider when designing future interventions. While many researchers incorporated incentives for participation and offered vaccines at reduced or no costs, there is minimal discussion of the cost incurred in implementing the interventions. Reporting cost is essential in understanding how resources are utilized in both effective and noneffective interventions. Cost is a critical aspect of interventions designed for low resource areas, as the success of those interventions is partly dependent on study participants' ability to pay for the cost of the vaccination.

#### 4.1.5. Maintenance

Our review showed the maintenance dimension was by far the most underreported indicator, with only 68.75% of articles reported conducting follow-up assessments. The lack of reporting could be due to researchers' financial and time limitations [20]. For HPV vaccines, it requires 12 months or more to complete the recommended dosing. HPV intervention studies should include a measurement of long-term effects and address the issue of maintenance. Without such measures, it is difficult to determine if the intervention strategies affect vaccination completion rates.

#### *4.2. Limitations*

There were several limitations to our study. First, this review focused on increasing the uptake of the HPV vaccine series. Because the vaccine comprises two to three doses administered at varying intervals in large catchment periods and populations, follow-up and series completion can be inherently difficult to ensure. Second, the RE-AIM framework served as the guide for evaluating the effectiveness of public health interventions. However, we utilized a modified or shortened version of the RE-AIM criteria in each dimension. Third, it is possible that our scope for article selection was significantly narrowed because we concentrated on reviewing recent articles involving a mobile technology and social media component through electronic messages/reminders, and/or social media campaigns. Fourth, while the interventions targeted several different populations, (e.g., college students, Appalachian women, adolescents, urban, rural, etc.) three of the articles in the review took place outside the United States with different contextual factors. Public health attitudes, perceptions, resources, and procedures may differ across cultures. Finally, researchers of the reviewed articles may argue that their studies were intended to demonstrate the efficacy of the intervention and that the scope of reporting might preclude effectiveness or generalizability information [20,67]. Further, researchers may collect data on external validity, but due to article length restriction may not report the date. However, even in efficacy trials, it would be beneficial to document and report adoption and implementation dimensions so that future researchers can replicate the study [20,67].

#### *4.3. Strengths*

Despite the above limitations, this review has several strengths. First, the use of the RE-AIM framework as a guiding metric for evaluating HPV vaccination interventions. The RE-AIM framework allows for the inclusion of the internal and external validity criteria that are important for evaluating the possibilities of translating interventions to other settings instead of evaluating just the efficacy of the interventions. Second, RE-AIM offers a systematic and structured examination of system-level considerations to the adoption of efficacious interventions. The RE-AIM framework has shown utility in assessing multiple criteria related to prevention research, namely, elements of efficacy, effectiveness, efficiency, and other implications for public health decision making such as quality of life and safety [64].

#### *4.4. Implications and Recommendations*

#### 4.4.1. Implication for Future Publication

The application of the RE-AIM framework to evaluate the effectiveness of HPV vaccine social mobile interventions is limited, yet our review demonstrates the utility of RE-AIM to evaluate HPV interventions and highlights the potential transferability of selected HPV intervention programs to a broader audience. To increase the potential to translate social media related HPV vaccine research findings to practice, researchers should place a greater emphasis on obtaining and reporting external validity information, such as adoption, implementation, and maintenance dimensions. Providing external validity information enhances other researchers' and practitioners' ability to judge the generalizability of effects and the comparative utility of interventions [20,67]. All stakeholders, including researchers, reviewers, editorial board, and funders, should emphasize the need for external validity information [20].

#### 4.4.2. Implications for Future HPV Vaccine Intervention


#### **5. Conclusions**

In conclusion, our findings show that social media-related HPV vaccination intervention studies demonstrated some effect on vaccination uptake (at least one dose of vaccination rate of 10.2% of the study population), reached larger study participants, and demonstrated that college students and college-aged groups are the targets of most social media intervention studies. While most articles in our review met the Consolidated Standards of Reporting Trials (CONSORT) metrics for reporting, specifically for internal validity reporting, the adoption, implementation, and maintenance dimensions for the RE-AIM framework were underreported. To ensure that these successful community-based interventions can be translated into practices, stakeholders should not only embrace the reporting of all the RE-AIM dimensions but should encourage researchers to adhere to external validity reporting standards similar to CONSORT internal validity reporting.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/2076-3 93X/9/3/197/s1, Table S1: RE-AIM Dimensions and selected indicators reported by the reviewed articles (*n* = 17).

**Author Contributions:** Conceptualization, M.A. and B.P.; methodology, M.A. and J.R.M.; validation, B.P., M.A., E.A., and B.A.L.; formal analysis, M.A. and B.P.; data curation, M.A. and B.P.; writing original draft preparation, B.P. and M.A.; writing—review and editing, M.A., B.P., E.A., B.A.L., and J.R.M.; supervision, M.A.; All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Acknowledgments:** We are thankful to Vedana Vaidhyanathan at the Baylor library for helping in the databases search during the literature review.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Vaccines* Editorial Office E-mail: vaccines@mdpi.com www.mdpi.com/journal/vaccines

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel: +41 61 683 77 34

www.mdpi.com

ISBN 978-3-0365-5918-6