1. Introduction
The use of electronic patient-reported outcome (ePRO) measures in the care of oncologic patients is being increasingly studied. ePROs can collect real-time data about the patients’ general condition using digital surveys and, therefore, enable a fast and personalized monitoring strategy for various domains, such as symptoms, side effects of therapies, functioning, and/or quality of life (QoL). As physicians often underestimate the severity of symptoms and side effects [
1,
2], collecting information directly from patients might improve the accuracy of this subjective data and potentially improve the efficiency of clinical care [
3]. Digital tools such as apps facilitate this process [
4,
5].
Patients with metastatic cancer are often affected physically and mentally because of the diagnosis itself, its treatment, or coexisting conditions [
6]. Basch et al., used an intensified digital PRO-monitoring on 766 cancer patients under chemotherapy, and detected clinical benefits such as improved QoL and a reduction in unplanned hospitalization and ER visits [
7], as well as a statistically significant increase in overall survival [
8]. Another study by Denis et al. also examined an electronic PRO-monitoring with alerts in advanced-staged lung cancer patients, and found improvements in overall survival, as well as better performance status at the first relapse compared to standard care [
9,
10].
While designing the PRO B study [
11] which aims to evaluate the effects of an app-based PRO-monitoring with alerts on metastatic breast cancer patients’ fatigue levels and QoL, we modified an existing digital symptom monitoring tool specifically for the project. The adapted system measures ePROs weekly using an app, then graphically displays the results for clinicians in a web tool and generates alerts in the case of deteriorating ePROs. The alert notifies the treating physician to get in contact with the patient to determine if further interventions are necessary.
In preparation for the PRO B study, prior testing of the ePRO system was conducted and evaluated by breast cancer patients. The study team wanted to include patients’ perspectives on the planned PRO B study, since patient-centered care is a critical component of care quality.
Although prior testing of such systems is often recommended [
12,
13], usability evaluation of eHealth apps has been an under-represented topic in publications [
14].
2. Materials and Methods
2.1. Aim of the Study
The pretest aimed to explore the feasibility of the planned PRO B study. Therefore, the main areas of interest were the patient’s experience with the app used in the study, the patient’s experience with the PRO B study questionnaires, and the feasibility of the PRO B study from the patient’s perspective.
Insights and conclusions from the pretest will serve as suggestions for adjustments before starting the main phase of the PRO B study.
2.2. Study Design
This pretest was a single-arm pilot study and took place at the Breast Center of Charité—Universitaetsmedizin Berlin between February and March 2021. The study was approved by the Charité Ethics board (No. EA1/318/20). Breast cancer patients were invited to participate. They were instructed to download the PRO application “PatientConcept” on their smartphone or tablet after giving written informed consent. The app then automatically generated an ID, which served as a pseudonym in the study and was communicated to the study team so that no identifying personal data was included in the ePRO system.
The ePRO system assigned questionnaires to the IDs in conjunction with push notifications on the patients’ smartphones. Questionnaires were available for 48 h and disappeared after the 48 h period. The survey was performed at baseline and one week later. At baseline, anamnestic, socio-demographic, and PRO questionnaires were sent from the ePRO system to the app. A week later, patients were asked to complete a second PRO assessment and an evaluation questionnaire inquiring about the pretest.
Afterwards, we also conducted semi-structured telephone interviews with the participants to further assess their feedback on the pretest to explore additional information that might not have been represented in the evaluation questionnaire (
Figure 1).
2.3. Participants
Female breast cancer patients of legal age were eligible for participation. Moreover, the participants were required to have access to the internet via a smartphone or tablet and be able to consent to participation. Patients visiting the breast cancer unit at Charité were invited to participate. However, unlike the PRO B study, inclusion in the pretest included, not only metastatic breast cancer patients, but also, patients with early breast cancer.
2.4. Study Instruments
Baseline questionnaires collected data on medical history and socio-demographics [
15].
PRO questionnaires for the PRO B study were compiled from items of the EORTC (European Organisation of Research and Treatment of Cancer) CAT (computerized adaptive testing) item bank [
16]. Depending on the week, the number of questions asked ranged from 51 to 54 items concerning the domains QoL, physical, role, cognitive, emotional, and social functioning as well as the symptom scales of fatigue, pain, nausea/vomiting, dyspnea, insomnia, appetite loss, constipation, diarrhea, and financial difficulties. In the pretest, the questionnaires were assigned twice on a week interval.
The pretest evaluation questionnaire consisted of two parts: opinion about the weekly PRO questionnaire and opinion about the smartphone application. Both parts used closed questions with a Likert-scale and a few open questions (
Supplementary File: Table S1). The first part was developed by the study team and consists of five questions with a 5-Likert scales. Internal consistency using Cronbach’s alpha was 0.77 (acceptable). The second part was partly derived from the mHealth App Usability Questionnaire (MAUQ) [
17] with a 7-point Likert-scale: 1 = strongly disagree to 7 = strongly agree) and the Cronbach’s alpha was 0.87 (good internal consistency). Technical problems were assessed using open questions with a free text response option.
A semi-structured telephone interview guide was developed in a multi-stage consensus process by the study team, which included a senior breast surgeon/gynecologist, biostatisticians, a study nurse, and a psychologist pursuing a deductive and inductive strategy. Questions were developed attempting to verify hypotheses generated from the study teams’ clinical expertise as well as being explorative in nature in order to freely capture the patients’ perspective and gather information in addition to the quantitative assessment (
Supplementary File: Table S2). The aim was to identify additional challenges that might not have been represented with the evaluation questionnaire.
2.5. Semi-Structured Interviews
The telephone interviews were conducted by two physicians (A.M. and F.K.) from the PRO B team using the developed guide. The physicians were not involved in the clinical care of the participants.
Participants were then interviewed individually by phone in German language. The interview questions are presented in the
Supplementary data.
Since the interviews were not recorded and were only documented in the form of notes by the interviewing physicians, transcriptions were not available and, therefore, analysis was limited to a descriptive summary of the results.
2.6. Analysis
Exploratory statistics, such as median, minimum, maximum, and absolute and relative frequencies were used to describe quantitative data using Stata IC15 (StataCorp, 2017, College Station, TX, USA).
A biostatistician sorted the notes from the interviews into content domains (perception of workload and duration, perception of the PRO questionnaires, perception of using the application, general suggestions, and using the application in the future and recommendation) and summarized the results using frequency counts.
3. Results
Sixteen female patients were recruited. Four did not answer any questionnaires and were, therefore, excluded. Eleven patients answered questionnaires in the app at least once, and one patient received the questionnaires via e-mail due to technical problems. Mean age was 47 years (SD 11, range 31–70 years), two participants were diagnosed with metastatic disease, while 10 were diagnosed with early breast cancer.
Only six of the twelve participants answered the evaluation questionnaire at the end of the pretest.
Due to major technical problems, the app, in some cases, did not display notifications on the mobile phone, leaving the participants unaware that there were new questionnaires available, and the questionnaires expired unanswered after 48 h.
Nevertheless, we were able to include eleven patients in the semi-structured interviews to further assess the experience with the ePRO tool.
3.1. Results of the Evaluation Questionnaire
3.1.1. Patients’ Experiences with the PRO B Study Questionnaires
Five out of six patients who answered the evaluation questionnaire were satisfied or very satisfied with the PRO questionnaires and the time spent on them. Four of the patients reported that the questions were very easy to understand and that they were able to concentrate the entire time. However, five patients reported that the questions were only moderately relevant to them (
Table 1).
Two patients reported that some questions seemed very similar, e.g., regarding fatigue and physical functioning. Three patients answered that the survey lacked important aspects of chemotherapy-related side effects, e.g., polyneuropathy. One patient wished for more questions regarding psycho-oncology, e.g., on worry and anxiety.
3.1.2. Patients’ Experiences with the Application
The results show that patients who were able to answer the questionnaires were overall very satisfied with using the app (median score of 7 points (7 = highest score)) (
Table 2).
3.2. Results of the Telephone Interview
The study team reached fourteen of the sixteen participants for the semi-structured telephone interview. Three of the participants did not participate in the telephone survey because they either did not receive any questionnaires (n = 2) or felt too ill to fill them out (n = 1). Overall, eleven patients were interviewed, of whom ten received and answered PROs at least once, and one participant reviewed the questionnaires after receiving them via e-mail instead of in the app (due to the technical problems).
Out of eleven participants, all considered it feasible to answer all items once a week for a year as planned in the intervention group of the PRO B study. Most of the patients completed the questionnaires in around 5–20 min.
After being asked if the PRO-questionnaires covered the most relevant aspects, patients responded positively, seven patients gave minor suggestions about what they would like to additionally see represented in the PRO-questionnaire. Side effects of treatment were mentioned most frequently (n = 4), and digestion, mental health, sexual and social aspects were also mentioned to better represent the patients’ burdens. Four participants mentioned they felt that some questions seemed redundant and sounded too general. A participant remarked that the amount of housework varies for everyone and is, therefore, not a comparable term. One patient had a concern about emotional and financial problems related to the corona-virus pandemic, as it could have aggravated these problems. Another patient reported that it could be difficult to complete the quality-of-life questionnaire when a private event occurs that is not related to breast cancer. Moreover, one patient said that she was confused by the used ranking scale and remarked that she would feel more familiar with German school grades (1–6). However, all of the patients thought that the questions were helpful for the doctors to assess the patient’s health status.
Multiple patients reported having technical problems using the application, especially with getting push notifications. Therefore, they were left unaware of receiving questionnaires in the app. They had to check the app for the questionnaires frequently by themselves, otherwise the questionnaire would expire. No patient mentioned any problems in the app while answering the questionnaires.
All of the patients considered the system useful. One would have preferred a different color scheme, as the color blue seemed too sterile and reminded her of a hospital. A more personalized greeting would have been nice. Another patient would have liked to receive an additional notification on the second day to be reminded about the availability of questionnaires. It was also suggested to add a short explanation or instruction about the questions concerning health status. One patient would have liked to see the results of their health state assessment graphically displayed in the app and, if possible, receive a weekly report. She also suggested including a short explanation and information on each question in the app. Moreover, the patient would have liked to receive some information on the latest research in the app.
All but one of the patients interviewed (n = 10) would like to use the system and concept in the future and would recommend it to other patients. One patient reported that she already felt adequately treated and, therefore, would not need to use the application further in the future.
The control group in the PRO B study will answer questionnaires only quarterly, and alerts will not be generated in the case of deteriorating PROs [
9]. The treating physicians will not have access to their answers during the study but will receive an overview at the end of the trial. As there is no primary benefit to answering the questionnaires for the control group, we wanted to examine if we should expect a high drop-out or non-compliance rate. Therefore, we asked the participants for suggestions on how to motivate the patients in the control group. Two participants mentioned that the control group could be questioned even more frequently. However, most of them suggested using a reminder notification and felt that the opportunity to contribute to research was incentive enough. For high compliance, the study team should explain the potential advantages of the project for future patients afflicted with metastatic breast cancer.
4. Discussion
In summary, the results show positive experiences in both the evaluation questionnaire and the telephone interview among participants who used the PRO B tool. Since interviews were not recorded and transcribed, the analysis is limited to descriptive summaries. Nevertheless, we could collect valuable insights from patients’ perspectives and confirm the overall acceptability of the questionnaire and the PRO B study design. Participants were generally satisfied and found that the app could be useful for tracking their HRQoL and symptoms. The study team was previously concerned that the time spent on answering the questionnaires might be overwhelming. However, the pretest allayed those concerns. From the patients’ points of view, answering the questionnaire on a weekly basis in the upcoming PRO B study is a tolerable burden that takes up a reasonable amount of time. Accordingly, after finding solutions to the severe technical problems, the app is feasible and possible to use for monitoring PROs in breast cancer patients. Moreover, downloading the
PatientConcept application is free of charge and has been used to support communication between physicians and patients [
18,
19].
The pretest revealed a need for technical improvement to the ePRO system and app. Therefore, a function documenting the delivery of the questionnaires to the participants was established after the pretest. Most participants did not answer the full set of questionnaires during the pretest, mostly because they did not receive any or were not aware of them because of missing push-notifications. Receipt of push-notifications was improved in a subsequent app-update.
To ensure compliance in the upcoming PRO B study, non-responding participants will receive push notifications as a reminder after 24 h.
The limitations of this pretest are the small number of patients and short study period. Due to a longer than expected time for implementation of the ePRO system and the upcoming start of the PRO B study, the duration of the pretest was restricted. Also, most of the participants did not suffer from metastatic breast cancer—unlike the inclusion criteria of the PRO B study. Therefore, the patients’ points of view and needs might differ.
Due to major technical problems with the app, we cannot draw conclusions about the response rate. Also, there was no testing of the alarm function in case of worsening scores because of the short study period. Moreover, qualitative assessment was not possible since the interviews were not recorded and transcribed. Nevertheless, we were still able to get an impression of the patients’ views, and we consider their suggestions valuable.
5. Conclusions
As every medical procedure should primarily focus on the patient on whom it is performed, the inclusion of the patient’s perspective in the development of new forms of care such as ePRO systems is crucial.
Unfortunately, in this pretest, findings from this study are limited to our specific study and context due to the low number of patients and the major unexpected technical problems with the app. Nevertheless, we note that none of the participants raised major concerns regarding the time spent answering the questionnaires via app or the content of the questionnaires. Therefore, we conclude that the PRO B study design and questionnaires are feasible and adequate to implement as planned. Major technical issues regarding push notifications in the app were identified and rectified after the pretest, but considerably delayed the start of enrollment for the PRO B study. When including new technologies in clinical studies, technical difficulties cannot be underestimated. Therefore, calculating a sufficient amount of time from pretest to enrollment is crucial.
Author Contributions
Conceptualization, U.G. and M.M.K.; methodology, F.K., T.P., U.G., P.G. and M.M.K.; formal analysis, P.G.; writing—original draft preparation, A.M.H. and P.G.; writing—review and editing, F.K., T.P., U.G. and M.M.K. All authors have read and agreed to the published version of the manuscript.
Funding
This research was funded by a public grant from the Innovation Committee of the Federal Joint Committee (“Innovationsauschuss des Gemeinsamen Bundesausschusses”, grant number 01NVF19013). The funder has no further role in the conduct of the study but must approve the changes of the protocol and financing.
Institutional Review Board Statement
The study was conducted according to the guidelines of the Declaration of Helsinki and was approved by the Institutional Review Board (or Ethics Committee) of Charité Universitätsmedizin Berlin (application number: EA1/318/20).
Informed Consent Statement
Informed consent was obtained from all subjects involved in the study. Written informed consent has been obtained from the patients to publish this paper.
Data Availability Statement
The data analyzed during this study are not permitted to be made publicly available.
Acknowledgments
We acknowledge the participants in the pretest, study nurses and the administrative and technical support from NeuroSys, the company working on the PatientConcept application. The authors would like to thank Felix Fischer and Alizé Rogge for their intensive reviews and suggestions.
Conflicts of Interest
The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.
References
- Fromme, E.K.; Eilers, K.M.; Mori, M.; Hsieh, Y.C.; Beer, T.M. How accurate is clinician reporting of chemotherapy adverse effects? A comparison with patient-reported symptoms from the Quality-of-Life Questionnaire C30. J. Clin. Oncol. 2004, 22, 3485–3490. [Google Scholar] [CrossRef] [PubMed]
- Laugsand, E.A.; Sprangers, M.A.; Bjordal, K.; Skorpen, F.; Kaasa, S.; Klepstad, P. Health care providers underestimate symptom intensities of cancer patients: A multicenter European study. Health Qual. Life Outcomes 2010, 8, 104. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Basch, E.; Stover, A.M.; Schrag, D.; Chung, A.; Jansen, J.; Henson, S.; Carr, P.; Ginos, B.; Deal, A.; Spears, P.A.; et al. Clinical Utility and User Perceptions of a Digital System for Electronic Patient-Reported Symptom Monitoring During Routine Cancer Care: Findings From the PRO-TECT Trial. JCO Clin. Cancer Inform. 2020, 4, 947–957. [Google Scholar] [CrossRef] [PubMed]
- Bae, W.K.; Kwon, J.; Lee, H.W.; Lee, S.C.; Song, E.K.; Shim, H.; Ryu, K.H.; Song, J.; Seo, S.; Yang, Y.; et al. Feasibility and accessibility of electronic patient-reported outcome measures using a smartphone during routine chemotherapy: A pilot study. Support. Care Cancer 2018, 26, 3721–3728. [Google Scholar] [CrossRef] [PubMed]
- Benze, G.; Nauck, F.; Alt-Epping, B.; Gianni, G.; Bauknecht, T.; Ettl, J.; Munte, A.; Kretzschmar, L.; Gaertner, J. PROutine: A feasibility study assessing surveillance of electronic patient reported outcomes and adherence via smartphone app in advanced cancer. Ann. Palliat. Med. 2019, 8, 104–111. [Google Scholar] [CrossRef] [PubMed]
- Reilly, C.M.; Bruner, D.W.; Mitchell, S.A.; Minasian, L.M.; Basch, E.; Dueck, A.C.; Cella, D.; Reeve, B.B. A literature synthesis of symptom prevalence and severity in persons receiving active cancer treatment. Support. Care Cancer 2013, 21, 1525–1550. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Basch, E.; Deal, A.M.; Kris, M.G.; Scher, H.I.; Hudis, C.A.; Sabbatini, P.; Rogak, L.; Bennett, A.V.; Dueck, A.C.; Atkinson, T.M.; et al. Symptom Monitoring With Patient-Reported Outcomes During Routine Cancer Treatment: A Randomized Controlled Trial. J. Clin. Oncol. 2016, 34, 557–565. [Google Scholar] [CrossRef] [PubMed]
- Basch, E.; Deal, A.M.; Dueck, A.C.; Scher, H.I.; Kris, M.G.; Hudis, C.; Schrag, D. Overall Survival Results of a Trial Assessing Patient-Reported Outcomes for Symptom Monitoring During Routine Cancer Treatment. JAMA 2017, 318, 197–198. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Denis, F.; Lethrosne, C.; Pourel, N.; Molinier, O.; Pointreau, Y.; Domont, J.; Bourgeois, H.; Senellart, H.; Tremolieres, P.; Lizee, T.; et al. Randomized Trial Comparing a Web-Mediated Follow-up With Routine Surveillance in Lung Cancer Patients. J. Natl. Cancer Inst. 2017, 109, 1–8. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Denis, F.; Yossi, S.; Septans, A.L.; Charron, A.; Voog, E.; Dupuis, O.; Ganem, G.; Pointreau, Y.; Letellier, C. Improving Survival in Patients Treated for a Lung Cancer Using Self-Evaluated Symptoms Reported Through a Web Application. Am. J. Clin. Oncol. 2017, 40, 464–469. [Google Scholar] [CrossRef] [PubMed]
- Karsten, M.M.; Kuhn, F.; Pross, T.; Blohmer, J.U.; Hage, A.M.; Fischer, F.; Rose, M.; Grittner, U.; Gebert, P.; Ferencz, J.; et al. PRO B: Evaluating the effect of an alarm-based patient-reported outcome monitoring compared with usual care in metastatic breast cancer patients-study protocol for a randomised controlled trial. Trials 2021, 22, 666. [Google Scholar] [CrossRef] [PubMed]
- Friis, R.B.; Hjollund, N.H.; Mejdahl, C.T.; Pappot, H.; Skuladottir, H. Electronic symptom monitoring in patients with metastatic lung cancer: A feasibility study. BMJ Open 2020, 10, e035673. [Google Scholar] [CrossRef] [PubMed]
- Warrington, L.; Absolom, K.; Holch, P.; Gibson, A.; Clayton, B.; Velikova, G. Online tool for monitoring adverse events in patients with cancer during treatment (eRAPID): Field testing in a clinical setting. BMJ Open 2019, 9, e025185. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Maramba, I.; Chatterjee, A.; Newman, C. Methods of usability testing in the development of eHealth applications: A scoping review. Int. J. Med. Inform. 2019, 126, 95–104. [Google Scholar] [CrossRef] [PubMed]
- Bundesamt, S. Demographische Standards, Ausgabe 2016; Eine gemeinsame Empfehlung des ADM Arbeitskreis Deutscher Markt-und Sozialforschungsinstitute e.V., der Arbeitsgemeinschaft Sozialwissenschaftlicher Institute e.V. (ASI) und des Statistischen Bundesamtes. 6. überarbeitete und erweiterte Auflage. Statistik und Wissenschaft Band 17; Statistisches Bundesam: Wiesbaden, Germany, 2016. Available online: https://www.statistischebibliothek.de/mir/servlets/MCRFileNodeServlet/DEMonografie_derivate_00001549/Band17_DemographischeStandards1030817169004.pdf (accessed on 10 December 2020).
- Petersen, M.A.; Aaronson, N.K.; Arraras, J.I.; Chie, W.C.; Conroy, T.; Costantini, A.; Dirven, L.; Fayers, P.; Gamper, E.M.; Giesinger, J.M.; et al. The EORTC CAT Core-The computer adaptive version of the EORTC QLQ-C30 questionnaire. Eur. J. Cancer 2018, 100, 8–16. [Google Scholar] [CrossRef] [PubMed]
- Zhou, L.; Bao, J.; Setiawan, I.M.A.; Saptono, A.; Parmanto, B. The mHealth App Usability Questionnaire (MAUQ): Development and Validation Study. JMIR mHealth uHealth 2019, 7, e11500. [Google Scholar] [CrossRef] [PubMed]
- Lang, M.; Mayr, M.; Ringbauer, S.; Cepek, L. PatientConcept App: Key Characteristics, Implementation, and its Potential Benefit. Neurol. Ther. 2019, 8, 147–154. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Lang, M.; Ringbauer, S.; Mayr, M.; Cepek, L. How to improve disease management of chronically ill patients? Perception of telemetric ECG recording and a novel software application. Clin. Res. Trials 2019, 5, 1–6. [Google Scholar] [CrossRef]
| Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).