Next Article in Journal
The Right to Oncological Oblivion: A Legislative Response to Cancer Survivor Discrimination in Italy
Previous Article in Journal
Determining the Best Noninvasive Test for Peripheral Arterial Disease Diagnosis to Predict Diabetic Foot Ulcer Healing in Patients Following Endovascular Revascularization
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Healthcare
Volume 12
Issue 16
10.3390/healthcare12161661
healthcare-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Needs-Assessment for an Artificial Intelligence-Based Chatbot for Pharmacists in HIV Care: Results from a Knowledge–Attitudes–Practices Survey

by
Moustafa Laymouna
1,2,3,
Yuanchao Ma
2,3,4,5,
David Lessard
2,3,4,
Kim Engler
2,3,
Rachel Therrien
6,
Tibor Schuster
1,
Serge Vicente
1,2,3,7,
Sofiane Achiche
5,
Maria Nait El Haj
8,
Benoît Lemire
4,
Abdalwahab Kawaiah
2,3,4 and
Bertrand Lebouché
1,2,3,4,*
1
Department of Family Medicine, Faculty of Medicine and Health Sciences, McGill University, Montreal, QC H3S 1Z1, Canada
2
Centre for Outcomes Research and Evaluation, Research Institute of the McGill University Health Centre, Montreal, QC H4A 3S5, Canada
3
Infectious Diseases and Immunity in Global Health Program, Research Institute of McGill University Health Centre, Montreal, QC H4A 3S5, Canada
4
Chronic Viral Illness Service, Division of Infectious Diseases, Department of Medicine, McGill University Health Centre, Montreal, QC H4A 3J1, Canada
5
Department of Biomedical Engineering, Polytechnique Montréal, Montreal, QC H3T 1J4, Canada
6
Department of Pharmacy and Chronic Viral Illness Service, Research Centre of the University of Montreal Hospital Centre, Montreal, QC H2X 0A9, Canada
7
Department of Mathematics and Statistics, University of Montreal, Montreal, QC H3T 1J4, Canada
8
Faculty of Pharmacy, University of Montreal, Montreal, QC H3C 3J7, Canada
*
Author to whom correspondence should be addressed.
Healthcare 2024, 12(16), 1661; https://doi.org/10.3390/healthcare12161661
Submission received: 21 July 2024 / Revised: 14 August 2024 / Accepted: 15 August 2024 / Published: 20 August 2024
Browse Figure
Versions Notes

Abstract

:
Background: Pharmacists need up-to-date knowledge and decision-making support in HIV care. We aim to develop MARVIN-Pharma, an adapted artificial intelligence-based chatbot initially for people with HIV, to assist pharmacists in considering evidence-based needs. Methods: From December 2022 to December 2023, an online needs-assessment survey evaluated Québec pharmacists’ knowledge, attitudes, involvement, and barriers relative to HIV care, alongside perceptions relevant to the usability of MARVIN-Pharma. Recruitment involved convenience and snowball sampling, targeting National HIV and Hepatitis Mentoring Program affiliates. Results: Forty-one pharmacists (28 community, 13 hospital-based) across 15 Québec municipalities participated. Participants perceived their HIV knowledge as moderate (M = 3.74/6). They held largely favorable attitudes towards providing HIV care (M = 4.02/6). They reported a “little” involvement in the delivery of HIV care services (M = 2.08/5), most often ART adherence counseling, refilling, and monitoring. The most common barriers reported to HIV care delivery were a lack of time, staff resources, clinical tools, and HIV information/training, with pharmacists at least somewhat agreeing that they experienced each (M ≥ 4.00/6). On average, MARVIN-Pharma’s acceptability and compatibility were in the ‘undecided’ range (M = 4.34, M = 4.13/7, respectively), while pharmacists agreed to their self-efficacy to use online health services (M = 5.6/7). Conclusion: MARVIN-Pharma might help address pharmacists’ knowledge gaps and barriers to HIV treatment and care, but pharmacist engagement in the chatbot’s development seems vital for its future uptake and usability.

1. Introduction

In Canada, the HIV epidemic is on the rise, with a 24.9% increase in reported cases across the country in 2022, amounting to 1833 new diagnoses [1]. The national rate of new HIV diagnoses was 4.7 per 100,000 people, with Québec slightly exceeding this average at 4.9 per 100,000 [2]. Montréal saw a particularly alarming spike, with a 120% increase in HIV diagnoses between 2021 and 2022, resulting in approximately 310 new cases [3]. This surge is attributed in part to the resumption of HIV testing post-pandemic and the arrival of immigrants from regions where HIV is more prevalent [4].
Managing HIV requires interprofessional efforts, and pharmacists play a significant role in the care of people with HIV (PWH) [5,6,7]. They often serve as first, regular, and trusted points of contact for people seeking medical treatment and advice [8,9,10,11,12], especially in community pharmacies [13]. They support PWH, notably, by providing crucial assistance in the regular refill and management of their antiretroviral therapy (ART).
Generalist pharmacists in Canada can struggle to stay current with HIV treatments and make informed decisions [14]. They can face challenges related to rapidly evolving drug landscapes, constrained time resources, and expanded care roles [15]. These difficulties can be compounded by infrequent interactions with PWH and significantly impair their ability to acquire specialized knowledge and expertise in HIV care, which can, in turn, impact the quality of care they provide.
A range of digital tools are available to support pharmacists in medication management, patient education, and professional communication, including mobile health applications and online clinical references [16,17,18]. These resources provide access to drug information, clinical guidelines, and patient care tips [16,17,18]. Nevertheless, the abundance of online health information complicates the retrieval of specific, reliable data [19], which can make it difficult for pharmacists to provide informed care and navigate treatment options effectively.
Chatbots that leverage artificial intelligence (AI) offer a promising solution to these challenges by facilitating the retrieval of medical information [20,21,22,23]. Chatbots, also known as “conversational agents”, are computer programs designed to mimic human conversation through various platforms, including websites, texting apps, and computer software [22,23,24,25,26,27,28,29,30,31,32,33]. As such, they could provide pharmacists with instant and streamlined access to the most current, validated information on HIV treatments, potentially outperforming web- or app-based tools in these functions. Chatbots could generate personalized advice, aligning with demands from healthcare providers for comprehensive information resources on medication, including, for example, drug–drug interactions and dietary supplements [34,35].
Given pharmacists’ challenges in HIV care delivery, our overarching goal is to adapt an existing AI-based chatbot and connect it with a constantly updated and verified source of information to help pharmacists serve PWH. Since 2020, a team of experts, including physicians, pharmacists, PWH, researchers, and engineers at the McGill University Health Centre (MUHC), developed an innovative AI chatbot called MARVIN for Minimal AntiretRoViral Interference [36]. Running on Facebook Messenger, MARVIN was created as a retrieval-based independent component analysis (ICA) trained to chat with PWH on various aspects of self-management of HIV in both English and French [36]. To configure MARVIN for pharmacists (MARVIN-Pharma), a formal assessment of local pharmacists’ needs is required. With this needs-assessment survey, our objectives are to describe the pharmacists’ knowledge, attitudes, involvement, and barriers in HIV care, as well as their perceptions of MARVIN-Pharma’s usability. Ultimately, this needs assessment aims to inform the adaptation of MARVIN-Pharma to ensure it effectively addresses the identified gaps and challenges in HIV care, enhancing pharmacists’ ability to deliver high-quality care and improving the overall usability and acceptance of the chatbot in clinical practice.

2. Methods

2.1. Study Design

For this study, we utilized a cross-sectional needs-assessment survey.

Conceptual Underpinnings

The Knowledge–Attitudes–Practices (KAP) model underscores the importance of considering the knowledge, attitudes, and practices of a given population, notably, to identify problems, design targeted interventions, and evaluate their effectiveness [37]. Knowledge refers to a person’s understanding of a given topic [38], or their capacity to receive, retain, and utilize information, and may require education [39]. Attitudes reflect a person’s feelings towards a specific topic, and they encompass how individuals react to a given situation [40]. Practice refers to how individuals demonstrate their knowledge and attitudes [38], namely, how they act on these [40]. Based on cognitive, affective, and behavioral theories, the model hypothesizes that relationships exist between these concepts [41]. Specifically, it suggests that knowledge influences attitudes, which in turn influence practices. Further supporting its relevance to our study, the KAP model is commonly used in health research to gather data from patients and practitioners with structured questions in cross-sectional surveys [37,42]. In this study, we assessed all three concepts of this model.
Usability concerns a technology’s ability to be used with effectiveness, efficiency, and satisfaction [43]. The related concepts we measured were perceived acceptability, compatibility, and self-efficacy. Acceptability concerns the degree to which a technology meets its users’ needs, preferences, and expectations; compatibility refers to the extent to which it is consistent with their habits, workflows, and digital environments; and self-efficacy relates to its users’ confidence in their ability to effectively use it.
A composite framework of the KAP and usability-related concepts guiding this study is shown in Figure 1.

2.2. Selection and Enrollment of Participants

2.2.1. Sample Size

Participant recruitment and data collection took place between December 2023 and December 2023. During this time, there were 9385 licensed practicing pharmacists in Québec [44]. Given our desired margin of error (15%), confidence level (95%), and assumed response distribution (50%), we determined our minimum required sample size to be 43 participants. This calculation, performed using an online sample size calculator [45], ensures that our study achieves a minimum precision level of 15 percentage points or less for the half-width of a 95% confidence interval, adequately representing estimated proportions within our population.

2.2.2. Eligibility Criteria

We recruited licensed practicing pharmacists working part- or full-time in the Canadian province of Québec, regardless of their specialized field. We adopted an inclusive approach that acknowledges that all pharmacists may encounter PWH in their practice.

2.2.3. Recruitment Strategy

We employed a two-fold recruitment approach utilizing convenience [46,47,48] and snowball sampling [48], starting with affiliates of the National HIV and Hepatitis Mentoring Program (Programme National de Mentorat sur le VIH et les Hépatites, or PNMVH). The PNMVH sent out invitations to their mailing list of affiliated pharmacists in Québec that described the study and provided a QR code to the survey as well as the email address of a research team member. The PNMVH also advertised the survey through multiple platforms (PNMVH newsletters, mailing lists, and postings on PNMVH social media accounts).
Snowball sampling proceeded as follows. Pharmacists who completed the survey were sent an email inviting them to forward the survey link to other pharmacists. The survey also included a prompt for participants to provide the email addresses of other potential respondents [49]. Referred pharmacists were emailed information about the study, including the email address of a study team member.

2.3. Data Collection and Analysis

2.3.1. The Survey

Sociodemographic and Professional Characteristics

The survey contained items on practice location (city), age, ethnic background, sexual orientation, and gender, as well as years of experience as a pharmacist, type of pharmacy practice setting, and served populations. Additionally, information was collected on Facebook Messenger use and the frequency of health/work application usage on mobile devices.

KAP Model Measures

To measure knowledge, we used a set of questions assessing perceived (subjective) knowledge related to HIV and its pharmaceutical treatment. Perceived knowledge was measured with 5 items on pharmacists’ perceived “preparedness” to advise patients extracted and adapted from a survey that underwent face validity testing with four Arizona pharmacists [50]. Items were answered on a 6-point Likert scale from “Very Unprepared” to “Very Prepared”. Cronback alpha for the 5 items was reported as 0.84.
Attitudes towards PWH and HIV care were measured with 13 attitudinal items also from the previous study’s survey [50]. Respondents recorded their answers on a 6-point Likert scale from “Strongly Disagree” to “Strongly Agree”.
Practice was assessed with three sets of items. First, 9 questions that measured involvement in HIV care were adapted from a questionnaire that was evaluated for face and content validity, tested for reliability in ten community pharmacies, and used among community pharmacists in Nigeria [51]. These questions were answered with a 5-point Likert scale from “Not Involved” to “Very Much Involved”. Barriers to providing HIV care were measured with 8 items from the same questionnaire. However, we replaced the original multiple-choice response options with a 6-point Likert scale from “Strongly Disagree” to “Strongly Agree”. Competence was evaluated with 4 items from a questionnaire used among Australian pharmacists on dispensing ART [52]. Participants responded with a 6-point Likert scale from “Strongly Disagree” to “Strongly Agree”.

Usability-Related Measures

To limit respondent burden, all three usability-related measures were answered on a 7-point scale of agreement from “Strongly Disagree” to “Strongly Agree”, which differed from the agreement scales of two of the original measures. Acceptability was assessed using the 4-item Acceptability of Intervention Measure (AIM), which has undergone elaborate psychometric testing (Cronbach alpha: 0.85) [53]. Note that the original response scale has 5 points. Compatibility was evaluated with a 4-item subscale of an “Instrument to Measure the Perceptions of Adopting an Information Technology Innovation”, which has demonstrated reliability and evidence of content and construct validity (Cronback alpha: 0.81–0.88) [54]. We slightly modified the original 7-point scale’s labels. To measure self-efficacy to use Internet health services, we used the 4-item Healthcare Technology Self-Efficacy Web subscale [55] with the original response options. It was created with adapted items from existing instruments and submitted to reliability and validity testing (global scale Cronbach alpha: 0.89) [55].

2.4. Technical Procedures and Survey Administration

We developed the survey in English, which was then translated into French using the forward–backward translation method. A pharmacy student in the research team, who is a native French speaker, performed the initial translation [56]. Subsequently, we retranslated the survey back into English to check for any discrepancies. To ensure linguistic and cultural appropriateness, the translation underwent rigorous pretesting. Two members of the research team, specifically a physician and a pharmacy student, carefully reviewed the French survey for clarity, comprehensibility, and coherence. The pretesting phase aimed to enhance the quality, interpretation, and suitability of the survey.
Between December 2022 and December 2023, pharmacists completed the survey online through a secure online platform, REDCap© (version 11.1.19 until 15 June 2023, and version 13.1.30 thereafter) [57], which ensured ease of administration and data storage. Participants were offered compensation of 50 CAD to complete the survey.

2.5. Statistical Analysis

We calculated descriptive statistics and reported them in tables. We provide the means and ± standard deviations (SDs) of continuous variables. We summarize categorical variables with frequencies and percentages. Statistical analyses were performed with R statistical software (version 4.2.1) [58].

2.6. Research Ethics Board (REB)

REB review exemption was obtained from the MUHC REB on 8 September 2022. The project does not qualify as “research” based on the definition outlined in Article 2.1 of the Tri-Council Policy Statement (TCPS2, 2018).

3. Results

We recruited 41 pharmacists, and all participants provided complete data.

3.1. Sociodemographic and Professional Characteristics

Table 1 presents descriptive sociodemographic information on respondents. Over two-thirds (68%) worked in the Montreal area. The average age of respondents was 36. In terms of ethnic backgrounds, approximately one half (49%) identified as French Canadian while the other half (51%) identified as East Asian, South Asian, Arab, or North African. The gender distribution showed that slightly more women than men were recruited. As to sexual orientation, most (85%) identified as heterosexual.
Professionally, respondents had an average of 10 years of experience. Over two-thirds (68%) worked in community pharmacies, with the remainder working in hospital settings. As to their patient groups, approximately half served women (54%) and LGBTQ+ community members (46%). Over a third provided care to people who use injection drugs (39%) and migrants from HIV-endemic countries (37%). Fewer attended to such groups as the elderly (12%), Indigenous people (12%), and sex workers (10%). Respondents’ patients were approximately 8% PWH, with an additional 12% deemed at high risk of acquiring HIV.
Considering technology usage, on average, respondents used Facebook Messenger regularly for over 10 years, indicating familiarity with the MARVIN chatbot’s current communication platform. Moreover, pharmacists reported, on average, “sometimes” using health or work-related applications on their mobile devices.

3.2. Knowledge

Perceived Knowledge

Our results indicate a moderate level of perceived knowledge on HIV and its treatment among respondents (Table 2). The average score across the items was 3.74 on 6, almost equivalent to being “somewhat prepared” to advise PWH in the areas evaluated. The highest average item scores (over 4.0) were for preparedness to counsel PWH on HIV transmission and HIV prevention. Meanwhile, the lowest individual item scores (with means between 3.5 and 3.6) concerned adverse drug effects and interactions; HIV treatments; and HIV symptoms and disease progression. These values corresponded to being “somewhat unprepared” to “somewhat prepared” to address these topics.

3.3. Attitudes

For the most part, pharmacists indicated positive attitudes towards PWH and HIV care in the context of their professional work, with a global mean of 4.02 on 6, where a higher mean indicates more favorable attitudes (Table 3). They strongly disagreed with stigmatizing statements (e.g., PWH deserve to have HIV or HIV treatments are not worthwhile) and agreed or strongly agreed with inclusive statements (e.g., I am willing to work with an HIV-positive colleague or providing information about HIV is a professional responsibility). However, pharmacists’ mean attitude toward mandatory continuing education on HIV/AIDS was tepid (M = 3.66; between “Somewhat Disagree” and “Somewhat Agree”).

3.4. Practice

3.4.1. Involvement

On average, respondents indicated being “little” involved in HIV care across the services considered (M = 2.08) (Table 4) and were most involved in ART adherence counseling, refilling ART, treatment and monitoring of ART, and referring patients for postexposure prophylaxis (PEP), with mean involvement ratings ranging from 2.3 to 2.8 on 5 (which refer to being “little” to “moderately” involved in these areas).

3.4.2. Competence

Pharmacists’ global mean score on items related to their perceived competence in HIV care delivery was 3.78/6 (Table 5). On average, they agreed they were sometimes hesitant to dispense ART to PWH and somewhat agreed they were unable to spend enough time counseling PWH. They also somewhat disagreed that they were able to confidently manage the complexities faced by PWH and that they were satisfied with the amount of care they provided them.

3.4.3. Barriers

Table 6 shows perceived barriers to providing HIV care. Most mentioned barriers were lack of time, of staff resources, of clinical tools, and of information or training on HIV/AIDS services. On average, they at least “somewhat” agreed that they experienced these barriers.

3.5. Usability

Table 7 presents the results on acceptability, compatibility, and self-efficacy. On average, pharmacists were “undecided” about the acceptability of the proposed MARVIN-pharma chatbot (M = 4.34/7) as well as about its compatibility with their work (M = 4.13). In addition, pharmacists showed tempered agreement with their self-efficacy to use Internet health services (M = 5.65).

4. Discussion

Drawing on a conceptual framework and existing instruments, this study aimed to better understand the needs of pharmacists in HIV care and their perceptions of the MARVIN-Pharma chatbot (in development) as a tool to reduce their challenges in HIV care. Specifically, it sought to characterize their knowledge, attitudes, and practices tied to HIV care, as well as the chatbot’s usability, in terms of its acceptability, its compatibility, and pharmacist self-efficacy.
Our findings indicate that pharmacists perceive their HIV knowledge as moderate and face significant barriers, such as lack of time and resources, which hinder their involvement in HIV care. Despite these challenges, there is a generally favorable attitude towards providing HIV care, suggesting a readiness to engage more deeply if provided with the right support tools. However, pharmacists were generally undecided about the acceptability and compatibility of MARVIN-Pharma. Nevertheless, this study highlights the potential of the chatbot to address the identified gaps and barriers, but also the importance of involving pharmacists in its development to ensure its future uptake and usability.

4.1. Knowledge

Our results showed that pharmacists perceived their knowledge to be moderate across the various aspects of HIV care evaluated. As found in past research [50,52], pharmacists felt most prepared to advise patients about HIV transmission and prevention. However, they did not feel fully prepared to advise on HIV treatments [50,52,59]; symptoms and progression of HIV; and adverse drug effects and interactions with ART. Pharmacists’ moderate HIV knowledge, as observed in our study, given their generally limited specialization in this area, reflects the broader trend towards increased HIV knowledge among healthcare professionals in general [50,52,59,60,61,62,63,64,65].
Efficiently bridging pharmacists’ knowledge gaps is essential, as their guidance may significantly enhance PWH’s understanding of their treatment [66], potentially boosting ART adherence and lowering HIV transmission rates [67]. MARVIN-Pharma could therefore prioritize content that helps pharmacists provide informed HIV care, such as HIV treatment guidelines and ART regimens, drug interactions, management of medication side effects, and considerations during pregnancy. The chatbot’s connection to a regularly updated ART regimen database like the Canadian HIV/AIDS Pharmacists Network (CHAP) or the Canadian AIDS Treatment Information Exchange (CATIE) seems essential to address the challenge of keeping pace with current information and to streamline access to it. Indeed, Ada Health, Buoy Health, Woebot, and Your.MD are real-world examples of chatbots linked to constantly updated medical databases that demonstrate the feasibility and benefits of such an approach in healthcare [68].
Additionally, MARVIN-Pharma’s eventual ability to deliver just-in-time training and educational content could further enhance pharmacists’ confidence and competence, addressing knowledge gaps as they arise during patient interactions. This practical application ensures the chatbot not only fills existing knowledge gaps but also supports continuous learning, directly linking back to the study’s objective of improving pharmacists’ knowledge in HIV care.

4.2. Attitudes

Pharmacists’ attitudes towards PWH and aspects of HIV care were favorable and non-stigmatizing, consistent with previous studies among pharmacists [50,51,62,64]. On average, the participants had been pharmacists for 10 years, which may partially explain these results. Healthcare personnel with greater work experience, especially related to PWH care, are found to have fewer stigmatizing attitudes towards PWH [69].
Our results show that some pharmacists were reluctant towards mandatory continuing education on HIV. This finding increases the relevance of a tool like MARVIN-Pharma for practice [70]. Indeed, AI-driven educational tools have been shown to improve healthcare outcomes by keeping practitioners updated with the latest guidelines and best practices in a user-friendly manner [71,72]. The chatbot, however, is not intended to teach pharmacists the latest guidelines directly. Rather, it will be designed to answer specific questions and provide responses based on these guidelines. Thus, pharmacists will be able to easily retrieve up-to-date information and apply it to real-life situations, enhancing their ability to make informed decisions quickly and effectively during patient care.
This focus on integrating the chatbot seamlessly into pharmacists’ workflows, rather than adding additional educational burdens, directly supports the objective of understanding and addressing pharmacists’ attitudes towards HIV care and ensuring that MARVIN-Pharma aligns with their professional practices.

4.3. Involvement

Our findings indicate that pharmacists, overall, were only a “little” involved in HIV care. Their most common activities concerned ART counseling, monitoring, and refilling, as well as PEP referrals. Notwithstanding, pharmacists’ delivery of HIV care remains crucial for extending services beyond specialized settings; it is both feasible and well-received and can help reach underserved populations [73], to enhance antiretroviral adherence, viral suppression, and retention in care [74]. MARVIN-Pharma could assist pharmacists in their important HIV care roles and help compensate for any limited involvement in this area. For example, an adherence counseling module could enable them to perform detailed, interactive assessments of patient adherence, providing personalized advice and interventions based on patients’ responses. Studies have shown that pharmacist-led adherence interventions significantly improve medication adherence and clinical outcomes in HIV patients [75,76,77,78,79]. Additionally, a PEP referral feature could streamline and simplify the referral process, complete with checklists and easily submittable forms to health services. This approach is supported by research indicating that pharmacist-initiated post-exposure prophylaxis (PEP) services increase timely access to PEP and improve adherence to PEP regimens [80,81,82].
These practical features demonstrate how MARVIN-Pharma could directly enhance pharmacists’ involvement in HIV care, fulfilling the study’s objective of assessing and improving their engagement in this area.

4.4. Competence

Our results indicate that pharmacists experience limited competence in HIV care delivery when managing complex issues faced by patients living with HIV, dispensing ART, and trying to provide sufficient care, underscoring the need for enhanced support and training.
The competence challenges identified align with those highlighted by past research [52]. Given their findings, some investigators [83] have emphasized the importance of continuous professional development to address knowledge gaps and improve clinical confidence among pharmacists, as well as the need for comprehensive training programs [84] to enhance their ability to manage complex patient needs and improve overall care delivery.
In addition to real-time support for pharmacists’ questions, MARVIN-Pharma could significantly enhance pharmacists’ competence in HIV care by offering tailored interactive learning modules, such as on ART management, drug interactions, and counseling techniques. Connected to a database like CHAP or CATIE, it could incorporate real-time updates on treatment guidelines and emerging data, thus reducing hesitation when dispensing ART. Additionally, it could facilitate just-in-time training, offering educational content and clinical support as pharmacists encounter specific challenges, thereby building confidence and improving the overall quality of HIV care [70].
By addressing these competence challenges through tailored features, MARVIN-Pharma aligns with the study’s ultimate objective of informing the development of tools to enhance pharmacists’ ability to deliver high-quality HIV care, ensuring the practical application of the findings.

4.5. Barriers

Mitigating barriers to integrating HIV services into pharmacy practice is crucial for enhancing care. Our study suggests that the pharmacists’ most prominent challenges were limited time, clinical tools, and staff resources, which are known to impact the provision of comprehensive HIV care [51,73,85,86,87]. Importantly, pharmacists also somewhat agreed that they lacked information or training on HIV/AIDS services, which can hinder effective patient counseling and care [14,88,89]. Lesser barriers, including a lack of collaboration with other healthcare professionals, inadequate pharmacy design, patient disinterest in preventive services, and the absence of financial incentives for HIV service provision, have also been reported in prior literature [74,85,90,91].
MARVIN-Pharma could be designed to help address the key barriers identified. By incorporating such features as educational modules [92,93,94,95,96,97], time-management tools [94,96,98,99,100,101], resource access [94,96,100,101,102], collaboration facilitation [99,102,103,104], patient engagement strategies [95,97,100,105], and incentive guidance [95,96,100,101], MARVIN-Pharma could significantly enhance HIV care delivery in pharmacy practice. The literature supports the potential of chatbots to incorporate these features effectively, demonstrating their capacity to overcome various barriers in healthcare.
Addressing these barriers directly relates to the objective of identifying challenges in HIV care and informing the design of MARVIN-Pharma to overcome them, ensuring the chatbot is practically useful in real-world settings.

4.6. Usability

Concerning the acceptability and compatibility of MARVIN-Pharma, pharmacists demonstrated a relatively neutral stance. This may be attributed in part to the hypothetical nature of the chatbot at this stage. Nevertheless, these findings emphasize that, to foster adoption, pharmacists must be engaged in the chatbot’s design to ensure it meets their specific needs and resonates with their professional responsibilities and workflow requirements. Stakeholder involvement and co-design are crucial for the success of mHealth tools. Studies have shown that involving end-users in the design process leads to higher acceptance and more effective implementation [106,107]. By collaborating with pharmacists during the development of MARVIN-Pharma, the tool can be tailored to their circumstances, enhancing its usability and integration into daily practice.
Encouragingly, our study also points to a level of comfort with digital technologies among Quebec pharmacists, aligned with broader research showing openness to digital tools among pharmacists [16,17,18,108,109,110,111,112,113,114]. The sample’s self-efficacy with using Internet health services, their average 10 years of using Facebook Messenger, and their occasional use of health or work-related applications on their mobile devices indicate a potential facility to take up a tool like MARVIN-Pharma.
Emphasizing the importance of user involvement and the need for further development based on pharmacists’ feedback highlights how the findings align with the objective of assessing MARVIN-Pharma’s usability to ensure its successful implementation in practice.

5. Limitations of This Study

This study has several limitations. Despite employing multiple recruitment strategies, its sample size was small (n = 41 pharmacists). We were thus unable to examine the effects of potential confounders (e.g., age, gender, and professional experience level). While snowball sampling may have helped reach a broader range of participants, self-selection and sampling bias may still affect the generalizability of the obtained results. For instance, the predominance of pharmacists from Montreal may not accurately represent the broader population of pharmacists in the province, and the sample’s average of 10 years of pharmacy experience and slight overrepresentation of females suggest a relatively experienced and gender-skewed group. We did, however, largely attain our sample size target to help ensure adequate statistical precision.
While this study relied on participant self-report, potentially introducing risks of response, recall, social desirability, and confirmation bias, the consistency of results obtained with respect to previous literature is encouraging in this regard.
This study’s quantitative design and limited scope mean the nuances and depth of pharmacists’ experiences and needs were not explored. A more comprehensive perspective will be achieved with the analysis of qualitative interview data collected as part of the larger mixed methods research project in which this study is embedded.

6. Conclusions

Our needs-assessment survey of Québec pharmacists indicates that while they have moderate HIV-related knowledge and positive attitudes towards providing HIV services in pharmacy practice, their involvement in HIV care remains limited. Despite pharmacists’ apparent capacity to adopt digital tools like MARVIN-Pharma, knowledge gaps and barriers in HIV treatment and management were identified. Although MARVIN-Pharma has the potential to address these challenges, its success will likely depend, in part, on achieving adequate perceived acceptability and compatibility with pharmacists’ work, on which they were undecided in this study. The findings suggest that buy-in for MARVIN-Pharma is not assured and that it must be developed with continuous pharmacist feedback and responsiveness to their specific needs.

Author Contributions

M.L. designed this study with the MARVIN team led by Y.M., supervised by B.L. (Bertrand Lebouché), and in collaboration with R.T., S.A. and M.L. developed the survey and its framework through discussions with K.E. and Y.M., under B.L. (Bertrand Lebouché)’s supervision. M.N.E.H. translated the survey. M.N.E.H., R.T., B.L. (Benoit Lemire), A.K. and D.L. assisted with recruitment. M.L. analyzed and interpreted the results. M.L. wrote the manuscript with K.E., T.S., K.E. and B.L. (Bertrand Lebouché) reviewed the first draft. T.S. and S.V. provided statistical advice throughout this study. K.E. conducted five cycles of manuscript and data presentation reviews. All authors have read and agreed to the published version of the manuscript.

Funding

This study is part of a wider project aiming to inform the adaptation of the MARVIN chatbot for pharmacists, led by Rachel Therrien (pharmacist) and Bertrand Lebouché and funded by the Canadian Institutes of Health Research Strategy for Patient-Oriented Research Québec Support Unit—Methodological Developments (grant M006, PI: BLeb), the Partnerships to Improve HIV Outcomes and Treatments (PIHVOT) program from ViiV Healthcare (grant 2020–1976-PIHVOT, PI: BLeb).

Institutional Review Board Statement

A REB review exemption was obtained from the MUHC REB on 8 September 2022. The project does not qualify as “research” based on the definition outlined in Article 2.1 of the Tri-Council Policy Statement (TCPS2, 2018).

Informed Consent Statement

Informed consents were obtained from all subjects involved in this study.

Data Availability Statement

The data sets generated and analyzed for this study are available from the corresponding author (B.L.) on reasonable request.

Acknowledgments

We extend our gratitude to the National HIV and Hepatitis Mentoring Program (PNMVH) for their support in recruitment and promotion of the survey through their mailing list, newsletters, member emails, and social media. B.L. is supported by 2 career awards: a Senior Salary Award from Fonds de recherche du Québec–Santé (FRQS) (#311200) and the LE 250 from Quebec’s Ministry of Health for researchers in Family Medicine, and holds a Canadian Institutes for Health Research Strategy for Patient-Oriented Research (SPOR) Mentorship Chair in Innovative Clinical Trials. M.L. is supported by a Graduate Excellence Fellowship (GEF) Award from the Department of Family Medicine at McGill University and McGill Centre for Viral Diseases (MCVD) studentship. Y.M. is supported by the Postgraduate Scholarship—Doctoral program (PGS D) and the OPSIDIAN program from the Natural Sciences and Engineering Research Council (NSERC), and a doctoral research award from the Fonds de recherche Nature et Technologies (FRQNT) in partnership with the Unité de soutien au système de santé apprenant (SSA), Québec.

Conflicts of Interest

B.L. has received research support and consulting fees and speaker fees from ViiV Healthcare, Merck, and Gilead, all outside of the submitted work.

Abbreviations

AIArtificial intelligence
ARTAntiretroviral therapy
ICAIndependent component analysis
KAPKnowledge–Attitudes–Practices
MARVINMinimal AntiretRoViral Interference
MARVIN-PharmaMinimal AntiretRoViral Interference for Pharmacists
MUHCMcGill University Health Centre
NNumber
PNMVHProgramme National de Mentorat sur le VIH et les Hépatites (National HIV and Hepatitis Mentoring Program)
PWHPeople with HIV
REB Research Ethics Board
SD Standard deviation
TCPSTri-Council Policy Statement

References

  1. Public Health Agency of Canada. HIV in Canada: 2022 Surveillance Highlights. Government of Canada, 1 December 2023. Available online: https://www.canada.ca/en/public-health/services/publications/diseases-conditions/hiv-2022-surveillance-highlights.html (accessed on 14 August 2024).
  2. The Canadian Press. HIV Epidemic on the Rise Again, with More Cases in Quebec than National Average. CTV News, 12 February 2024. Available online: https://montreal.ctvnews.ca/hiv-epidemic-on-the-rise-again-with-more-cases-in-quebec-than-national-average-1.6765242#:~:text=The%20Public%20Health%20Agency%20of,with%20a%20rate%20of%204.9 (accessed on 14 August 2024).
  3. Morris, E. Montreal Saw a 120% Increase in HIV Diagnoses Last Year. CBC News, 7 December 2023. Available online: https://www.cbc.ca/news/canada/montreal/montreal-hiv-increase-1.7050753 (accessed on 14 August 2024).
  4. McGill University. Expert: HIV Epidemic on the Rise, with More Cases in Quebec than National Average. McGill University Newsroom, 20 March 2024. Available online: https://www.mcgill.ca/newsroom/channels/news/expert-hiv-epidemic-rise-more-cases-quebec-national-average-356224 (accessed on 14 August 2024).
  5. Smith, M.; Bates, D.W.; Bodenheimer, T.S. Pharmacists belong in accountable care organizations and integrated care teams. Health Aff. 2013, 32, 1963–1970. [Google Scholar] [CrossRef]
  6. Smith, M.; Bates, D.W.; Bodenheimer, T.; Cleary, P.D. Why pharmacists belong in the medical home. Health Aff. 2010, 29, 906–913. [Google Scholar] [CrossRef]
  7. Schafer, J.J.; Gill, T.K.; Sherman, E.M.; McNicholl, I.R.; Hawkins, B. ASHP guidelines on pharmacist involvement in HIV care. Am. J. Health-Syst. Pharm. 2016, 73, 468–494. [Google Scholar] [CrossRef] [PubMed]
  8. World Health Organization. The Role of the Pharmacist in the Health Care System: Preparing the Future Pharmacist: Curricular Development: Report of a Third WHO Consultative Group on the Role of the Pharmacist, Vancouver, Canada, 27–29 August 1997; World Health Organization: Geneva, Switzerland, 1997. [Google Scholar]
  9. Vorobjov, S.; Uusküla, A.; Abel-Ollo, K.; Talu, A.; Jarlais, D.D. Should pharmacists have a role in harm reduction services for IDUs? A qualitative study in Tallinn, Estonia. J. Urban Health 2009, 86, 918–928. [Google Scholar] [CrossRef] [PubMed]
  10. Bluml, B.M. Definition of medication therapy management: Development of professionwide consensus. J. Am. Pharm. Assoc. 2005, 45, 566–572. [Google Scholar] [CrossRef]
  11. Park-Wyllie, L.Y.; Kam, D.; Bayoumi, A.M. The Adherence Support Gap: The ‘Ideal’ Versus ‘Reality’of Antiretroviral Adherence Support Provided by HIV Health Providers in Clinical Practice. Ann. Pharmacother. 2009, 43, 1036–1044. [Google Scholar] [CrossRef]
  12. Darin, K.M.; Klepser, M.E.; Klepser, D.E.; Klepser, S.A.; Reeves, A.; Young, M.; Scarsi, K.K. Pharmacist-provided rapid HIV testing in two community pharmacies. J. Am. Pharm. Assoc. 2015, 55, 81–88. [Google Scholar] [CrossRef]
  13. Smith, Y.B. Community Pharmacy. Available online: https://www.news-medical.net/health/Community-Pharmacy.aspx (accessed on 27 February 2019).
  14. Kibicho, J.; Pinkerton, S.D.; Owczarzak, J. Community-based pharmacists’ needs for HIV-related training and experience. J. Pharm. Pract. 2014, 27, 369–378. [Google Scholar] [CrossRef] [PubMed]
  15. Iwanowicz, S.L.; Marciniak, M.W.; Zeolla, M.M. Obtaining and providing health information in the community pharmacy setting. Am. J. Pharm. Educ. 2006, 70, 57. [Google Scholar] [CrossRef]
  16. Crilly, P.; Kayyali, R. A systematic review of randomized controlled trials of telehealth and digital technology use by community pharmacists to improve public health. Pharmacy 2020, 8, 137. [Google Scholar] [CrossRef]
  17. Crilly, P.; Hassanali, W.; Khanna, G.; Matharu, K.; Patel, D.; Patel, D.; Rahman, F.; Kayyali, R. Community pharmacist perceptions of their role and the use of social media and mobile health applications as tools in public health. Res. Soc. Adm. Pharm. 2019, 15, 23–30. [Google Scholar] [CrossRef] [PubMed]
  18. mHealth: Use of Mobile Health Tools in Pharmacy Practice 2019. Available online: https://www.fip.org/files/content/publications/2019/mHealth-Use-of-mobile-health-tools-in-pharmacy-practice.pdf (accessed on 8 February 2024).
  19. Curley, S.P.; Connelly, D.P.; Rich, E.C. Physicians’ use of medical knowledge resources: Preliminary theoretical framework and findings. Med. Decis. Mak. 1990, 10, 231–241. [Google Scholar] [CrossRef]
  20. Brixey, J.; Hoegen, R.; Lan, W.; Rusow, J.; Singla, K.; Yin, X.; Artstein, R.; Leuski, A. Shihbot: A facebook chatbot for sexual health information on HIV/AIDS. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, Saarbrücken, Germany, 15–17 August 2017; pp. 370–373. [Google Scholar]
  21. Garett, R.; Young, S.D. Potential application of conversational agents in HIV testing uptake among high-risk populations. J. Public Health 2023, 45, 189–192. [Google Scholar]
  22. Kocaballi, A.B.; Berkovsky, S.; Quiroz, J.C.; Laranjo, L.; Tong, H.L.; Rezazadegan, D.; Briatore, A.; Coiera, E. The personalization of conversational agents in health care: Systematic review. J. Med. Internet Res. 2019, 21, e15360. [Google Scholar]
  23. Luo, T.C.; Aguilera, A.; Lyles, C.R.; Figueroa, C.A. Promoting physical activity through conversational agents: Mixed methods systematic review. J. Med. Internet Res. 2021, 23, e25486. [Google Scholar]
  24. Laranjo, L.; Dunn, A.G.; Tong, H.L.; Kocaballi, A.B.; Chen, J.; Bashir, R.; Surian, D.; Gallego, B.; Magrabi, F.; Lau, A.Y.; et al. Conversational agents in healthcare: A systematic review. J. Am. Med. Inform. Assoc. 2018, 25, 1248–1258. [Google Scholar]
  25. Vaidyam, A.N.; Wisniewski, H.; Halamka, J.D.; Kashavan, M.S.; Torous, J.B. Chatbots and conversational agents in mental health: A review of the psychiatric landscape. Can. J. Psychiatry 2019, 64, 456–464. [Google Scholar] [PubMed]
  26. Milne-Ives, M.; de Cock, C.; Lim, E.; Shehadeh, M.H.; de Pennington, N.; Mole, G.; Normando, E.; Meinert, E. The effectiveness of artificial intelligence conversational agents in health care: Systematic review. J. Med. Internet Res. 2020, 22, e20346. [Google Scholar] [CrossRef]
  27. Griffin, A.C.; Xing, Z.; Khairat, S.; Wang, Y.; Bailey, S.; Arguello, J.; Chung, A.E. Conversational agents for chronic disease self-management: A systematic review. In AMIA Annual Symposium Proceedings; American Medical Informatics Association: Bethesda, MA, USA, 2020; Volume 2020, pp. 504–513. [Google Scholar]
  28. Tudor Car, L.; Dhinagaran, D.A.; Kyaw, B.M.; Kowatsch, T.; Joty, S.; Theng, Y.L.; Atun, R. Conversational agents in health care: Scoping review and conceptual analysis. J. Med. Internet Res. 2020, 22, e17158. [Google Scholar] [PubMed]
  29. Vaidyam, A.N.; Linggonegoro, D.; Torous, J. Changes to the Psychiatric Chatbot Landscape: A Systematic Review of Conversational Agents in Serious Mental Illness. Can. J. Psychiatry 2021, 66, 339–348. [Google Scholar]
  30. Geoghegan, L.; Scarborough, A.; Wormald, J.C.; Harrison, C.J.; Collins, D.; Gardiner, M.; Bruce, J.; Rodrigues, J.N. Automated conversational agents for post-intervention follow-up: A systematic review. BJS Open 2021, 5, zrab070. [Google Scholar] [CrossRef]
  31. Allouch, M.; Azaria, A.; Azoulay, R. Conversational agents: Goals, technologies, vision and challenges. Sensors 2021, 21, 8448. [Google Scholar] [CrossRef] [PubMed]
  32. Bibault, J.E.; Chaix, B.; Nectoux, P.; Pienkowski, A.; Guillemasé, A.; Brouard, B. Healthcare ex Machina: Are conversational agents ready for prime time in oncology? Clin. Transl. Radiat. Oncol. 2019, 1, 55–59. [Google Scholar] [CrossRef]
  33. Abd-Alrazaq, A.A.; Alajlani, M.; Alalwan, A.A.; Bewick, B.M.; Gardner, P.; Househ, M. An overview of the features of chatbots in mental health: A scoping review. Int. J. Med. Inform. 2019, 1, 103978. [Google Scholar] [CrossRef]
  34. Daniel, T.; de Chevigny, A.; Champrigaud, A.; Valette, J.; Sitbon, M.; Jardin, M.; Chevalier, D.; Renet, S. Answering Hospital Caregivers’ Questions at Any Time: Proof-of-Concept Study of an Artificial Intelligence–Based Chatbot in a French Hospital. JMIR Hum. Factors 2022, 9, e39102. [Google Scholar] [CrossRef] [PubMed]
  35. Koman, J.; Fauvelle, K.; Schuck, S.; Texier, N.; Mebarki, A. Physicians’ perceptions of the use of a chatbot for information seeking: Qualitative study. J. Med. Internet Res. 2020, 22, e15185. [Google Scholar] [CrossRef]
  36. Ma, Y.; Tu, G.; Lessard, D.; Vicente, S.; Engler, K.; Achiche, S.; Laymouna, M.; de Pokomandy, A.; Lebouché, B. An artificial intelligence-based chatbot to promote HIV primary care self-management: A mixed method usability study. Ann. Fam. Med. Novemb. 2023, 21, 5267. [Google Scholar]
  37. Vandamme, E. Concepts and challenges in the use of knowledge-attitude-practice surveys: Literature review. Dep. Anim. Health Inst. Trop. Med. 2009, 1, 1–7. [Google Scholar]
  38. Kaliyaperumal, K.I. Guideline for conducting a knowledge, attitude and practice (KAP) study. AECS Illum. 2004, 4, 7. [Google Scholar]
  39. Rwanamiza, E. Knowledge, education, learning and teaching: Meanings and relationships. J. Am. Assoc. Adv. Curric. Stud. 2009, 5, 4–6. [Google Scholar] [CrossRef]
  40. Badran, I.G. Knowledge, attitude and practice the three pillars of excellence and wisdom: A place in the medical profession. East. Mediterr. Health J. 1995, 1, 8–16. [Google Scholar] [CrossRef]
  41. Bano, R.; AlShammari, E.; Fatima, S.B.; Al-Shammari, N.A. A comparative study of knowledge, attitude, practice of nutrition and non-nutrition student towards a balanced diet in Hail University. J. Nurs. Health Sci. 2013, 2, 29–36. [Google Scholar]
  42. Launiala, A. How much can a KAP survey tell us about people’s knowledge, attitudes and practices? Some observations from medical anthropology research on malaria in pregnancy in Malawi. Anthropol. Matters 2009, 11. [Google Scholar] [CrossRef]
  43. ISO 9241-11:2018; Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts. International Organization for Standardization (ISO): Geneva, Switzerland, 2018.
  44. National Association of Pharmacy Regulatory Authorities (NAPRA). National Statistics 2023. Available online: https://www.napra.ca/resources/national-statistics/ (accessed on 14 August 2023).
  45. Sample Size Calculator by Raosoft, Inc. 2022. Available online: http://www.raosoft.com/samplesize.html (accessed on 5 May 2022).
  46. Etikan, I.; Musa, S.A.; Alkassim, R.S. Comparison of convenience sampling and purposive sampling. Am. J. Theor. Appl. Stat. 2016, 5, 1–4. [Google Scholar] [CrossRef]
  47. Stratton, S.J. Population research: Convenience sampling strategies. Prehospital Disaster Med. 2021, 36, 373–374. [Google Scholar] [CrossRef] [PubMed]
  48. Emerson, R.W. Convenience sampling, random sampling, and snowball sampling: How does sampling affect the validity of research? J. Vis. Impair. Blind. 2015, 109, 164–168. [Google Scholar] [CrossRef]
  49. Goodman, L.A. Snowball sampling. Ann. Math. Stat. 1961, 32, 148–170. [Google Scholar] [CrossRef]
  50. Katz, M.D.; Draugalis, J.R.; Lai, R.P. HIV infection and AIDS: Attitudes and knowledge of Arizona pharmacists. Ann. Pharmacother. 1995, 29, 1218–1223. [Google Scholar] [CrossRef] [PubMed]
  51. Oseni, Y.O.; Erhun, W.O. Assessing community pharmacists’ involvement and clients’ opinion on HIV/AIDS services in community pharmacies in Nigeria: A cross-sectional survey. Int. J. STD AIDS 2021, 32, 538–550. [Google Scholar] [CrossRef]
  52. Singleton, J.A.; Lau, E.T.; Nissen, L.M. Exploring Australian pharmacists’ perceptions and attitudes towards dispensing HIV medicines in the community setting. J. Pharm. Pract. Res. 2021, 51, 238–246. [Google Scholar] [CrossRef]
  53. Weiner, B.J.; Lewis, C.C.; Stanick, C.; Powell, B.J.; Dorsey, C.N.; Clary, A.S.; Boynton, M.H.; Halko, H. Psychometric assessment of three newly developed implementation outcome measures. Implement. Sci. 2017, 12, 108. [Google Scholar] [CrossRef]
  54. Moore, G.C.; Benbasat, I. Development of an instrument to measure the perceptions of adopting an information technology innovation. Inf. Syst. Res. 1991, 2, 192–222. [Google Scholar] [CrossRef]
  55. Rahman, M.S.; Ko, M.; Warren, J.; Carpenter, D. Healthcare Technology Self-Efficacy (HTSE) and its influence on individual attitude: An empirical study. Comput. Hum. Behav. 2016, 58, 12–24. [Google Scholar] [CrossRef]
  56. Yu, D.S.; Lee, D.T.; Woo, J. Issues and challenges of instrument translation. West. J. Nurs. Res. 2004, 26, 307–320. [Google Scholar] [CrossRef]
  57. Harris, P.A.; Taylor, R.; Thielke, R.; Payne, J.; Gonzalez, N.; Conde, J.G. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J. Biomed. Inform. 2009, 42, 377–381. [Google Scholar] [CrossRef] [PubMed]
  58. R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2021. [Google Scholar]
  59. Dapar, M.L.; Joseph, B.N.; Damun, P.A.; Okunlola, C.R.; Alphonsus, P.N.; Aya, B.M. Assessment of knowledge and competencies of community pharmacists for differentiated HIV care and services in Jos, Nigeria. J. Pharm. Res. Int. 2019, 29, 1–10. [Google Scholar] [CrossRef]
  60. Passannante, M.R.; Louria, D.B.; French, J. How Much Do Health Care Provid. Know About AIDS? Am. J. Prev. Med. 1993, 9, 6–14. [Google Scholar] [CrossRef]
  61. Knowledge and Training Needs of Health Professionals Working with People with HIV: With Consideration for Mixed Urban/Rural Care Settings Question. 2010. Available online: https://www.ohtn.on.ca/Pages/Knowledge-Exchange/Rapid-Responses/Documents/RR16-2010-HC-Professionals-Training-Needs.pdf (accessed on 7 April 2023).
  62. Ma’aji, H.U.; Ilyas, O.S. Assessment of knowledge, attitude and practice of community pharmacists towards pharmaceutical care in Kaduna State, Nigeria. Int. J. Pharm. Teach. Pract. 2014, 5, 972–976. [Google Scholar]
  63. Sianturi, E.I.; Latifah, E.; Pane, M.; Perwitasari, D.A.; Satibi Kristina, S.A.; Hastuti, E.B.; Pavlovich, J.; Taxis, K. Knowledge, empathy, and willingness to counsel patients with HIV among Indonesian pharmacists: A national survey of stigma. AIDS Care 2022, 34, 21–28. [Google Scholar] [CrossRef]
  64. Balfour, L.; Corace, K.; Tasca, G.A.; Best-Plummer, W.; MacPherson, P.A.; Cameron, D.W. High HIV knowledge relates to low stigma in pharmacists and university health science students in Guyana, South America. Int. J. Infect. Dis. 2010, 14, e881–e887. [Google Scholar] [CrossRef]
  65. Letshwenyo-Maruatona, S.B.; Madisa, M.; Boitshwarelo, T.; George-Kefilwe, B.; Kingori, C.; Ice, G.; Bianco, J.A.; Marape, M.; Haile, Z.T. Association between HIV/AIDS knowledge and stigma towards people living with HIV/AIDS in Botswana. Afr. J. AIDS Res. 2019, 18, 58–64. [Google Scholar] [CrossRef]
  66. Kim, G.S.; Choi, J.P.; Yi, J.M.; Shim, M.S. Development of a question prompt list for patients living with HIV and assessment of their information needs. J. Assoc. Nurses AIDS Care 2019, 30, 575–583. [Google Scholar] [CrossRef]
  67. Govender, S.; Naidoo, P.; Esterhuizen, T. Impact of pharmacists’ intervention on the knowledge of HIV infected patients in a public sector hospital of KwaZulu-Natal. Afr. J. Prim. Health Care Fam. Med. 2011, 3, 1–8. [Google Scholar] [CrossRef]
  68. Gilbert, S.; Fenech, M.; Idris, A.; Türk, E. Periodic Manual Algorithm Updates and Generalizability: A Developer’s Response. Comment on “Evaluation of Four Artificial Intelligence–Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study”. J. Med. Internet Res. 2021, 23, e26514. [Google Scholar] [CrossRef]
  69. Srithanaviboonchai, K.; Khemngern, P.; Chuayen, J.; Siraprapasiri, T. Increased Work Experience Associated with Less Stigmatizing Attitudes towards People Living with HIV among Thai Healthcare Personnel. Int. J. Environ. Res. Public Health 2021, 18, 9830. [Google Scholar] [CrossRef]
  70. Royal College of Physicians and Surgeons of Canada. AI in Medicine: Supporting Learning and Practice. 2023. Available online: https://www.royalcollege.ca/en/newsroom/posts/ai-in-medicine--supporting-learning-and-practice-.html (accessed on 8 February 2024).
  71. Moorhead, S.A.; Hazlett, D.E.; Harrison, L.; Carroll, J.K.; Irwin, A.; Hoving, C. A new dimension of health care: Systematic review of the uses, benefits, and limitations of social media for health communication. J. Med. Internet Res. 2013, 15, e1933. [Google Scholar] [CrossRef]
  72. Topol, E.J. High-performance medicine: The convergence of human and artificial intelligence. Nat. Med. 2019, 25, 44–56. [Google Scholar] [CrossRef]
  73. Kuo, A.P.; Roche, S.D.; Mugambi, M.L.; Pintye, J.; Baeten, J.M.; Bukusi, E.; Ngure, K.; Stergachis, A.; Ortblad, K.F. The effectiveness, feasibility and acceptability of HIV service delivery at private pharmacies in sub-Saharan Africa: A scoping review. J. Int. AIDS Soc. 2022, 25, e26027. [Google Scholar] [CrossRef]
  74. Schommer, J.C.; Garza, O.W.; Taitel, M.S.; Akinbosoye, O.E.; Suzuki, S.; Clay, P.G. Work system and process designs for community pharmacy-medical clinic partnerships to improve retention in care, antiretroviral adherence, and viral suppression in persons with HIV. Pharmacy 2020, 8, 125. [Google Scholar] [CrossRef]
  75. Ma, A.; Chen, D.M.; Chau, F.M.; Saberi, P. Improving adherence and clinical outcomes through an HIV pharmacist’s interventions. AIDS Care 2010, 22, 1189–1194. [Google Scholar] [CrossRef]
  76. Foisy, M.M.; Akai, P.S. Pharmaceutical care for HIV patients on directly observed therapy. Ann. Pharmacother. 2004, 38, 550–556. [Google Scholar] [CrossRef] [PubMed]
  77. Tseng, A.; Foisy, M.; Hughes, C.A.; Kelly, D.; Chan, S.; Dayneka, N.; Giguère, P.; Higgins, N.; Hills-Nieminen, C.; Kapler, J.; et al. Role of the pharmacist in caring for patients with HIV/AIDS: Clinical practice guidelines. Can. J. Hosp. Pharm. 2012, 65, 125–145. [Google Scholar] [CrossRef] [PubMed]
  78. Tailor, S.A.; Foisy, M.M.; Tseng, A.; Beardsall, A.; Ostrop, N.; Khaliq, Y.; Shevchuk, Y.; Chin, T.; Gin, A.; Dayneka, N.; et al. The role of the pharmacist caring for people living with HIV/AIDS: A Canadian position paper. Can. J. Hosp. Pharm. 2000, 53. [Google Scholar] [CrossRef]
  79. McCree, D.H.; Byrd, K.K.; Johnston, M.; Gaines, M.; Weidle, P.J. Roles for pharmacists in the “Ending the HIV Epidemic: A Plan for America” initiative. Public Health Rep. 2020, 135, 547–554. [Google Scholar] [CrossRef] [PubMed]
  80. Khosropour, C.M.; Backus, K.V.; Means, A.R.; Beauchamps, L.; Johnson, K.; Golden, M.R.; Mena, L. A pharmacist-led, same-day, HIV pre-exposure prophylaxis initiation program to increase PrEP uptake and decrease time to PrEP initiation. AIDS Patient Care STDs 2020, 34, 1–6. [Google Scholar] [CrossRef] [PubMed]
  81. Southern State Directors Work Group. Southern States Manifesto: HIV/AIDS and STD’s in the South; National Alliance of State and Territorial AIDS Directors: Washington, DC, USA, 2003. [Google Scholar]
  82. Scarnati, K.; Esser, K.; Sahloff, E.G.; Duggan, J. The role of community pharmacies in providing access to HIV post-exposure prophylaxis (PEP). J. Community Health 2024, 49, 222–228. [Google Scholar] [CrossRef]
  83. Blenkinsopp, A.; Bond, C.; Raynor, D.K. Medication reviews. Br. J. Clin. Pharmacol. 2012, 74, 573–580. [Google Scholar] [CrossRef]
  84. Cohen, M.S.; Chen, Y.Q.; McCauley, M.; Gamble, T.; Hosseinipour, M.C.; Kumarasamy, N.; Hakim, J.G.; Kumwenda, J.; Grinsztejn, B.; Pilotto, J.H.; et al. Antiretroviral therapy for the prevention of HIV-1 transmission. N. Engl. J. Med. 2016, 375, 830–839. [Google Scholar] [CrossRef]
  85. Laliberté, M.C.; Perreault, S.; Damestoy, N.; Lalonde, L. Ideal and actual involvement of community pharmacists in health promotion and prevention: A cross-sectional study in Quebec, Canada. BMC Public Health 2012, 12, 192. [Google Scholar] [CrossRef]
  86. Ajagu, N.; Anetoh, M.U.; Nduka, S.O. Expanding HIV/AIDS care service sites: A cross sectional survey of community pharmacists’ views in South-East, Nigeria. J. Pharm. Policy Pract. 2017, 10, 34. [Google Scholar] [CrossRef]
  87. Maja, L.; Polile, R.; Khoarai, N.; Ramathebane, M. Pharmacists’ perspective on hiv testing services in community pharmacies in maseru, lesotho. Int. J. Curr. Res. Life Sci. 2018, 7, 1594–1598. [Google Scholar]
  88. Adeola, O.A. The opinions of pharmacists in Ondo state, Nigeria, towards the inclusion of HIV services into community pharmacies. S. Am. J. Public Health 2014, 2, 476–488. [Google Scholar]
  89. Henderson, K.C.; Hindman, J.; Johnson, S.C.; Valuck, R.J.; Kiser, J.J. Assessing the effectiveness of pharmacy-based adherence interventions on antiretroviral adherence in persons with HIV. AIDS Patient Care STDs 2011, 25, 221–228. [Google Scholar] [CrossRef]
  90. Kielly, J.; Kelly, D.V.; Hughes, C.; Day, K.; Hancock, S.; Asghari, S.; Gahagan, J.; Marra, C.; Nguyen, H. Adaptation of POCT for pharmacies to reduce risk and optimize access to care in HIV, the APPROACH study protocol: Examining acceptability and feasibility. Pilot Feasibility Stud. 2018, 4, 59. [Google Scholar] [CrossRef]
  91. Babigumira, J.B.; Castelnuovo, B.; Stergachis, A.; Kiragga, A.; Shaefer, P.; Lamorde, M.; Kambugu, A.; Muwanga, A.; Garrison, L.P. Cost effectiveness of a pharmacy-only refill program in a large urban HIV/AIDS clinic in Uganda. PLoS ONE 2011, 6, e18193. [Google Scholar] [CrossRef] [PubMed]
  92. Clark, M.; Bailey, S. Chatbots in Health Care: Connecting Patients to Information. Can. J. Health Technol. 2024, 4, 1–22. [Google Scholar] [CrossRef]
  93. Prochaska, J.J.; Vogel, E.A.; Chieng, A.; Kendra, M.; Baiocchi, M.; Pajarito, S.; Robinson, A. A therapeutic relational agent for reducing problematic substance use (Woebot): Development and usability study. J. Med. Internet Res. 2021, 23, e24850. [Google Scholar] [CrossRef]
  94. Radziwill, N.M.; Benton, M.C. Evaluating quality of chatbots and intelligent conversational agents. arXiv 2017, arXiv:1704.04579. [Google Scholar]
  95. Inkster, B.; Sarda, S.; Subramanian, V. An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: Real-world data evaluation mixed-methods study. JMIR Mhealth Uhealth 2018, 6, e12106. [Google Scholar] [CrossRef]
  96. Chow, J.C.; Wong, V.; Li, K. Generative Pre-Trained Transformer-Empowered Healthcare Conversations: Current Trends, Challenges, and Future Directions in Large Language Model-Enabled Medical Chatbots. BioMedInformatics 2024, 4, 837–852. [Google Scholar] [CrossRef]
  97. Ni, L.; Lu, C.; Liu, N.; Liu, J. Mandy: Towards a smart primary care chatbot application. In International Symposium on Knowledge and Systems Sciences; Springer: Singapore, 2017; pp. 38–52. [Google Scholar]
  98. O’Leary, K.; Tanghe, D.; Pratt, W.; Ralston, J. Collaborative health reminders and notifications: Insights from prototypes. In AMIA Annual Symposium Proceedings; American Medical Informatics Association: Bethesda, MA, USA, 2018; Volume 2018, p. 837. [Google Scholar]
  99. Calvaresi, D.; Eggenschwiler, S.; Mualla, Y.; Schumacher, M.; Calbimonte, J.P. Exploring agent-based chatbots: A systematic literature review. J. Ambient Intell. Humaniz. Comput. 2023, 14, 11207–11226. [Google Scholar] [CrossRef]
  100. Fadhil, A.; Villafiorita, A. An adaptive learning with gamification & conversational UIs: The rise of CiboPoliBot. In Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization; Association for Computing Machinery: New York, NY, USA, 2017; pp. 408–412. [Google Scholar]
  101. Singh, J.; Sillerud, B.; Singh, A. Artificial intelligence, chatbots and ChatGPT in healthcare—Narrative review of historical evolution, current application, and change management approach to increase adoption. J. Med. Artif. Intell. 2023, 6. [Google Scholar] [CrossRef]
  102. Office of the National Coordinator for Health Information Technology. Patient Engagement Playbook. 2023. Available online: https://www.healthit.gov/playbook/patient-engagement/ (accessed on 23 June 2024).
  103. Suppadungsuk, S.; Thongprayoon, C.; Miao, J.; Krisanapan, P.; Qureshi, F.; Kashani, K.; Cheungpasitporn, W. Exploring the Potential of Chatbots in Critical Care Nephrology. Medicines 2023, 10, 58. [Google Scholar] [CrossRef] [PubMed]
  104. Yang, H.S.; Wang, F.; Greenblatt, M.B.; Huang, S.X.; Zhang, Y. AI Chatbots in Clinical Laboratory Medicine: Foundations and Trends. Clin. Chem. 2023, 69, 1238–1246. [Google Scholar] [CrossRef]
  105. Kadariya, D.; Venkataramanan, R.; Yip, H.Y.; Kalra, M.; Thirunarayanan, K.; Sheth, A. KBot: Knowledge-enabled personalized chatbot for asthma self-management. In Proceedings of the 2019 IEEE International Conference on Smart Computing (SMARTCOMP), Washington, DC, USA, 12–15 June 2019; IEEE: Piscataway, NJ, USA; pp. 138–143. [Google Scholar]
  106. Van Velsen, L.; Wentzel, J.; Van Gemert-Pijnen, J.E. Designing eHealth that matters via a multidisciplinary requirements development approach. JMIR Res. Protoc. 2013, 2, e2547. [Google Scholar] [CrossRef]
  107. O’connor, S.; Hanlon, P.; O’donnell, C.A.; Garcia, S.; Glanville, J.; Mair, F.S. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: A systematic review of qualitative studies. BMC Med. Inform. Decis. Mak. 2016, 16, 120. [Google Scholar] [CrossRef]
  108. Food and Drug Administration. What Is Digital Health? Available online: https://www.fda.gov/medical-devices/digital-health-center-excellence/what-digital-health (accessed on 22 September 2020).
  109. Crilly, P.; Jair, S.; Mahmood, Z.; Moin Khan, A.; Munir, A.; Osei-Bediako, I.; Samir, M.; Kayyali, R. Public views of different sources of health advice: Pharmacists, social media and mobile health applications. Int. J. Pharm. Pract. 2019, 27, 88–95. [Google Scholar] [CrossRef] [PubMed]
  110. Benetoli, A.; Chen, T.F.; Aslani, P. The use of social media in pharmacy practice and education. Res. Soc. Adm. Pharm. 2015, 11, 1–46. [Google Scholar] [CrossRef]
  111. Cain, J.; Romanelli, F.; Fox, B. Pharmacy, social media, and health: Opportunity for impact. J. Am. Pharm. Assoc. 2010, 50, 745–751. [Google Scholar] [CrossRef]
  112. Kayyali, R.; Crilly, P. Digital media in pharmacy public health. Pharm. Pharmacol. Int. J. 2016, 4, 00069. [Google Scholar] [CrossRef]
  113. Shcherbakova, N.; Shepherd, M. Community pharmacists, Internet and social media: An empirical investigation. Res. Soc. Adm. Pharm. 2014, 10, e75–e85. [Google Scholar] [CrossRef] [PubMed]
  114. Lam, A. Practice innovations: Delivering medication therapy management services via videoconference interviews. Consult. Pharm. 2011, 26, 764–774. [Google Scholar] [CrossRef] [PubMed]
Healthcare 12 01661 g001
Figure 1. Guiding composite conceptual framework.
Figure 1. Guiding composite conceptual framework.
Healthcare 12 01661 g001
Table 1. Participating pharmacists’ sociodemographic and professional characteristics (n = 41).
Table 1. Participating pharmacists’ sociodemographic and professional characteristics (n = 41).
Characteristicn (%)Mean ± SD (Range)
City of work
 Montreal28 (68%)
 Other cities13 (32%)
Age-36 yrs. ± 8 (23–52)
Ethnicity
 French Canadian 20 (49%)
 Other21 (51%)
Sexual orientation
 Heterosexual 35 (85%)
 Homosexual 3 (7%)
 Bisexual 3 (7%)
Gender
 Female 25 (61%)
 Male 16 (39%)
Years of pharmacy experience-10 yrs. ± 8 (1–30)
Main setting of pharmacy practice
 Community 28 (68%)
 Hospital 13 (32%)
Patient groups seen by pharmacists
 Women22 (54%)
 People from the LGBTQ+ community19 (46%)
 People who use injection drugs16 (39%)
 People from countries where HIV is endemic15 (37%)
 HIV-negative partners of PWH8 (20%)
 Indigenous people5 (12%)
 Elderly5 (12%)
 Sex workers4 (10%)
 Incarcerated people1 (2%)
 Dialysis patients with frequent transfusions1 (2%)
Percentage of patients with HIV-8 ± 15% (0–75%)
Percentage of patients at high risk of HIV-12 ± 15% (0–66%)
Years of using Facebook Messenger regularly-11 yrs. ± 3 (1–20)
Frequency of using applications related to health/work on pharmacists’ mobile devices-2 ± 1 (0–4) *
* Frequency of use was rated on a five-point scale from 0 = “Never” to 4 = “Very Frequently”.
Table 2. Pharmacists’ perceived knowledge to counsel people with HIV in various areas of HIV care.
Table 2. Pharmacists’ perceived knowledge to counsel people with HIV in various areas of HIV care.
AreaResponse * nMean ± SD
123456
HIV prevention1310111244.02 ± 1.20
HIV transmission138141144.05 ± 1.17
HIV/AIDS symptoms and disease progression161612153.51 ± 1.21
HIV treatments142010153.51 ± 1.17
Adverse effects and drug–drug interactions with HIV treatments14198453.61 ± 1.23
Global average3.74 ± 1.20
* 1 = Very Unprepared; 2 = Unprepared; 3 = Somewhat Unprepared; 4 = Somewhat Prepared; 5 = Prepared; 6 = Very Prepared.
Table 3. Pharmacists’ attitudes toward people with HIV and their related care.
Table 3. Pharmacists’ attitudes toward people with HIV and their related care.
StatementResponse * nMean ± SD
123456
Providing information about HIV/AIDS is part of a pharmacist’s professional responsibility002714185.17 ± 0.88
I have time while at work to answer questions about HIV/AIDS2889773.78 ± 1.49
People who are educated about how HIV is transmitted sexually will change their risky behavior116191134.15 ± 1.00
People who are educated about how HIV is transmitted by contaminated needles will change their risky behavior22919543.85 ± 1.16
Some people who have HIV/AIDS deserve to have the disease **34520005.78 ± 0.52
Drugs to treat HIV/AIDS are too expensive for the pharmacy to keep in stock16168643.59 ± 1.25
Approved treatments for HIV/AIDS simply delay the inevitable and may not be worthwhile **29921005.61 ± 0.69
I would provide information about a patient’s HIV status to healthcare providers other than the prescribing physician22575112.05 ± 1.34
I might become infected with HIV from contact with an HIV/AIDS patient when I am working **171082312.90 ± 1.14
Many of my patients are at high risk for HIV/AIDS141653302.15 ± 1.18
I might be sued if I unknowingly advise an HIV/AIDS patient with incomplete medication information17861274.02 ± 1.44
I am willing to work with a coworker who has HIV/AIDS00049285.59 ± 0.66
Continuing education on HIV/AIDS should be mandatory3411111023.66 ± 1.28
Global mean4.02 ± 1.64
* 1 = Strongly Disagree; 2 = Disagree; 3 = Somewhat Disagree; 4 = Somewhat Agree; 5 = Agree; 6 = Strongly Agree. ** The reversal process for this item converts the original score x to the reversed score 7 − x to calculate the global mean.
Table 4. Pharmacists’ involvement in HIV care services.
Table 4. Pharmacists’ involvement in HIV care services.
HIV Care ServiceResponse * nMean ± SD
12345
Pharmaceutical care to HIV patients9234142.22 ± 1.12
Adherence counseling for HIV/AIDS patients9159352.51 ± 1.25
Referral of patients for postexposure prophylaxis (PEP)101512312.27 ± 0.99
Refill of ART1597822.34 ± 1.28
Stocking of ART19710412.05 ± 1.15
Social responsibility to HIV patients18126322.00 ± 1.15
Treatment and monitoring of ART131210242.32 ± 1.24
Provision of a PEP test for a patient3181011.34 ± 0.75
Home delivery of ART24103311.71 ± 1.04
Global mean2.08 ± 1.12
* 1 = Not Involved; 2 = Little Involvement; 3 = Moderately Involved; 4 = Very Involved; 5 = Very Much Involved. PEP = post-exposure prophylaxis; ART = antiretroviral therapy.
Table 5. Pharmacists’ perceived competence in the delivery of HIV care.
Table 5. Pharmacists’ perceived competence in the delivery of HIV care.
StatementResponse * nMean ± SD
123456
I believe I am able to confidently manage the complex issues and experiences faced by patients living with HIV1010114242.76 ± 1.53
I feel I am unable to spend sufficient time counseling patients living with HIV **571114133.81 ± 1.31
I am satisfied with the amount of care I provide to patients living with HIV412118243.10 ± 1.39
I sometimes feel hesitant to dispense HIV medicines to patients living with HIV **27833005.44 ± 0.91
Global mean3.78 ± 1.31
* 1 = Strongly Disagree; 2 = Disagree; 3 = Somewhat Disagree; 4 = Somewhat Agree; 5 = Agree; 6 = Strongly Agree. ** The reversal process for this item converts the original score x to the reversed score 7 − x to calculate the global mean.
Table 6. Pharmacists’ perceived barriers to providing HIV care.
Table 6. Pharmacists’ perceived barriers to providing HIV care.
BarrierResponse * nMean ± SD
123456
Lack of clinical tools (i.e., HIV test kits)036161154.22 ± 1.07
Lack of collaboration with other healthcare professionals37914533.49 ± 1.31
Lack of information or training on HIV/AIDS services06519564.00 ± 1.19
Lack of staff resources141141654.34 ± 1.18
Absence of financial compensation2988863.71 ± 1.49
Lack of space/inadequate physical design of the pharmacy56691053.68 ± 1.57
Lack of time311914134.68 ± 1.39
Patients are not interested in preventive services391213313.17 ± 1.15
* 1 = Strongly Disagree; 2 = Disagree; 3 = Somewhat Disagree; 4 = Somewhat Agree; 5 = Agree; 6 = Strongly Agree.
Table 7. Pharmacists’ perceived acceptability and compatibility with the chatbot and self-efficacy to use Internet health services.
Table 7. Pharmacists’ perceived acceptability and compatibility with the chatbot and self-efficacy to use Internet health services.
Response * nMean ± SD
1234567
Acceptability
MARVIN Pharma Chatbot meets my approval111313314.15 ± 0.95
MARVIN Pharma Chatbot is appealing to me101268414.37 ± 0.92
I like MARVIN Pharma Chatbot101307114.20 ± 0.80
I welcome MARVIN Pharma Chatbot100237824.63 ± 1.10
Global mean4.34 ± 0.95
Compatibility
Using MARVIN Pharma Chatbot is compatible with all aspects of my work102277314.27 ± 0.94
Using MARVIN Pharma Chatbot is completely compatible with my current situation105283404.07 ± 0.90
I think that using the MARVIN Pharma Chatbot fits well with the way I like to work107263314.07 ± 0.90
Using MARVIN Pharma Chatbot fits into my work style113276214.12 ± 0.97
Global mean4.13 ± 0.95
Self-efficacy
It is easy for me to use internet health services0100151695.76 ± 0.96
I feel uncomfortable using internet health services **1116404515.27 ± 1.81
I am very confident in my abilities to use internet health services01001514115.81 ± 0.99
I would be able to use internet health services without much effort00121413115.76 ± 0.98
Global mean5.65 ± 1.24
* 1 = Strongly Disagree; 2 = Disagree; 3 = Somewhat Disagree; 4 = Undecided; 5 = Somewhat Agree; 6 = Agree; 7 = Strongly Agree. ** The reversal process for this item converts the original score x to the reversed score 8 − x to calculate the global mean.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Laymouna, M.; Ma, Y.; Lessard, D.; Engler, K.; Therrien, R.; Schuster, T.; Vicente, S.; Achiche, S.; El Haj, M.N.; Lemire, B.; et al. Needs-Assessment for an Artificial Intelligence-Based Chatbot for Pharmacists in HIV Care: Results from a Knowledge–Attitudes–Practices Survey. Healthcare 2024, 12, 1661. https://doi.org/10.3390/healthcare12161661

AMA Style

Laymouna M, Ma Y, Lessard D, Engler K, Therrien R, Schuster T, Vicente S, Achiche S, El Haj MN, Lemire B, et al. Needs-Assessment for an Artificial Intelligence-Based Chatbot for Pharmacists in HIV Care: Results from a Knowledge–Attitudes–Practices Survey. Healthcare. 2024; 12(16):1661. https://doi.org/10.3390/healthcare12161661

Chicago/Turabian Style

Laymouna, Moustafa, Yuanchao Ma, David Lessard, Kim Engler, Rachel Therrien, Tibor Schuster, Serge Vicente, Sofiane Achiche, Maria Nait El Haj, Benoît Lemire, and et al. 2024. "Needs-Assessment for an Artificial Intelligence-Based Chatbot for Pharmacists in HIV Care: Results from a Knowledge–Attitudes–Practices Survey" Healthcare 12, no. 16: 1661. https://doi.org/10.3390/healthcare12161661

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop