A Quality Assessment of Information Available on Renal Cancer on YouTube

Abstract

Objectives: Many people are turning to alternatives to the conventional doctor-patient relationship, such as web- based search engines and video forums for their health care information. We undertook this study to investigate the quality of videos and information on renal cancer available on the streaming platform YouTube. Methods: We completed a search of YouTube (www.YouTube.com) in September 2021 with the term “kidney cancer.” The first 120 videos found which met the inclusion criteria (English speaking, duration greater than one minute, greater than 500 views, renal cancer addressed) were selected. We recorded information including duration, view count, likes, dislikes, comments, publisher, and author. The modified DISCERN tool and Global Quality Score (GQS) questionnaire were used to assess the quality of the included videos. The level of misinformation was assessed using a Likert 5-point scale. Descriptive statistics were used to analyse the collected data. A 2-sample t test was used to further analyse the quality assessment tool results before, during, and after 2016. Results: Most videos were published during or after 2016 (63.3%), were predominantly created in North America (77.5%), and were presented by health care professionals (60%). The median length of the videos was 4.23 (1.01 to 65.55) minutes, and the median number of views was 3087 (514 to 228,152). The median number of likes and dislikes was 24 and 5, respectively. The median modified DISCERN score was 3, the median GQS score was 3, and the grading for overall level of misinformation was moderate. Conclusion: The quality of information accessed from YouTube on kidney cancer is of a low to moderate overall standard with significant levels of misinformation. YouTube should not be used alone for educational purposes on renal cancer by patients or the public. It is best used in conjunction with information and advice from a medical practitioner and the health care system.

Introduction

Kidney cancer is the second most common urological cancer worldwide, with 430 000 new cases recorded per year[1,2]. Studies have found that 9.8% to 15.1% of patients with renal cancers report a lack of appropriate information as a major source of frustration in their care[3]. Increasingly, the internet has become a primary information source used by patients. A review of Google Trends shows there has been a 400% increase in the number of searches on kidney cancer-related terms over the last 12 months[4]. Surveys have shown that approximately 8 out of 10 internet users have searched online about health topics, and 25% have watched an online video about health or medical issues[5,6]. The video-sharing platform, YouTube, is the most accessed streaming platform in the world with over 2 billion views per day[7]. YouTube has shown to be a commonly used resource for patients seeking medical information, including on urological diseases[8,9,10,11].

We believe video platforms such as YouTube can enable patients to develop an improved level of health literacy and a greater understanding of their health care needs. However, there has been no objective analysis of the accuracy of the information available. Web-based resources have been shown to have variable levels of quality and accuracy when compared with guidelines[12,13,14]. As distinguishing informative videos from those with significant inaccuracies may be challenging for the lay person, it is important that clinicians be aware of the quality of content and veracity of the resources patients may be using. The objective of our study was to use validated instruments to analyse the quality of renal cancer information available to patients on YouTube.

Methods

Search Strategy

The term “kidney cancer” was used to search YouTube (www.YouTube.com) in September 2021. Google Trends indicated this was the most frequently used search term relating to renal cancer during the preceding 12 months. The default search setting on YouTube was used, which automatically sorts the videos by relevance. We included all videos that were in English, had over 500 views, and covered information on renal cancer. Studies were excluded if they were duplicates, advertisements, instructional surgical videos, lacked audio, were not published in the English language, or did not contain information on renal cancer. The first 120 videos meeting these criteria were included in the study and saved to a “playlist” after the initial search.

Information was collected on various characteristics, including year of publication, place of publication by continent, number of views, length (minutes), number of likes (thumbs up), number of dislikes (thumbs down), number of comments, and quality of audio and visual (

Table 1. Video characteristics of 120 YouTube videos on renal cancer.

). Content characteristics were also recorded including subject, type of provider, author, and intended audience (

Table 2. Content characteristics of 120 YouTube videos on renal cancer.

).

Video Analysis

Two authors (J.S. R.S.) independently reviewed the 120 videos included for further analysis. If there was any discrepancy, a third author (D.A.) was consulted held the deciding vote. Information on the accuracy of information was assessed on the basis of consistency with either the latest European Association of Urology or American Urological Association guidelines[15,16].

If a video contained information on alternative therapies, then a literature search was conducted to assess the most current information on the topic.

Quality Analysis

The quality analysis was completed using a modified DISCERN tool[17], Global Quality Score (GQS)[18] and misinformation tool which was assessed on a Likert 5-point scale[19]. The modified DISCERN tool is scored based on 5 questions with 1 point for each answer, as shown in

Table 3. Modified DISCERN evaluation tool for reliability and quality of renal cancer information on YouTube.

. With 1-2 points indicating low quality, 3 points indicating moderate quality and 4-5 points indicating high overall quality. This tool has been validated to assess the reliability of online resources and has been used in similar studies[20,21,22]. The GQS was based on a 5-point scale including poor quality (not useful for patients), generally poor quality (very limited use), moderate quality (somewhat useful), good quality (useful for patients), and excellent quality (very useful for patients). The GQS is used to better assess the quality of online resources and indicates usefulness of the information to the intended audience. Misinformation was assessed using a 5-point Likert scale. On this scale, a score of 1 point indicated good quality information with low levels of misinformation, and a score of 5 points indicated high levels of misinformation. The videos were also assessed for inclusion of commercial bias. This was defined as a personal judgment in favour of a specific product or service of a commercial interest. The presence of disclosures or conflict of interest, either in the video or written description, was recorded. Video popularity was assessed using the Video Power Index (like ratio × view ratio [views per day ÷ 100]) and the like ratio (like ÷ (like + dislike) × 100).

Data Analysis

This study was completed with the use of descriptive statistics for analysis of outcomes and demographic variables. Continuous variables were expressed as a mean, range, and standard deviation. Categorical variables were expressed as a number and percentage. The 2-sample t test was used to assess the quality analysis and misinformation tools, and a P value of < 0.05 was considered statistically significant.

Results

The YouTube search for the term “kidney cancer” identified over 4000 videos. Two hundred fifty videos were saved into a playlist for further review, and the first 120 that met the criteria were included in the study.

The majority of included videos were published during or after 2016 (63.3%) and created in North America (77.5%); 13.3% were created in Europe. The median length of the videos was 4.23 minutes (1.01 to 65.55), and the median number of views was 3087 (514 to 228 152). The median number of likes and dislikes was 24 and 5, respectively (Table 1).

The most common content category was “treatment” and an “overview of renal cancer” with 3 out of the 4 included criteria (anatomy, symptoms, diagnosis, and treatment). Approximately 39% of the videos were created by hospitals or health clinics, 18% by health channels, and 18% by foundations or advocacy groups. Most of the videos (60%) were presented by health professionals including doctors or nurses, 20% by patients, and 16% were animations. Most of the videos were intended to be accessed by patients (62.4%) followed by the general public (37.5%) and then health care providers (14.7%).

The modified DISCERN indicated a high number of videos scoring as moderate quality of information with an overall median score of 3; 74.1% of all videos in the study were of moderate to high (≥ 3) reliability. However, a significant portion of the videos (25.9%) were of low or very low quality. The GQS tool indicated that 83.4% of the videos had moderate to excellent levels of quality; 24.2% of the videos scored as excellent for the content they provided. A proportion of the studies (16.6%) were scored either poor or very poor for levels of information. Misinformation was found in 49.7% of the videos analysed, and 22.5% contained moderate to high levels of misinformation. The other 50.3% were scored as having no misinformation.

A subgroup analysis was done of videos published within the last 5 years compared with videos published prior to this period (pre and post 2016). The median scores for all the quality assessment tools were equal between the 2 groups including the modified DISCERN (3), GQS (moderate) and misinformation scores (2). The GQS and misinformation tools were found to have statis tically difference P values between the videos published in 2015 or before and those published during or after 2016. Quality analysis information is shown in

Table 4. Quality and misinformation analysis.

alongside the supplementary analysis comparing videos submitted in 2015 or before with those submitted during or after 2016. Less than 6% of the videos had disclosures or COI documented, and 24.2% were deemed to have commercial bias (Figure 1).

Discussion

The aim of this present study was to analyse the quality of renal cancer information available to patients on the video-sharing platform YouTube, using validated instruments. We found over 7000 videos relating to renal cancer on YouTube. Most of the included videos were created by medical organisations (39%) and were intended primarily for patient viewing (62.4%).

To our knowledge, this is the first study to analyse the quality of renal cancer information available on YouTube. Readily available, good quality, unbiased medical information can be beneficial for patients who want to inform themselves about specific health conditions[23,24]. There are evidence-based videos on YouTube that have been found to have high-quality information when created or reviewed by a legitimate source[25]. However, many available videos and web-based sources have been shown to be of poor overall quality and to offer outdated advice that may detrimentally affect patient care and well-being[26,27,28]. At the same time, videos from high-quality sources such as universities and health care organisations are viewed less on social media and web-based browsers[29], further contributing to greater levels of misinformation accessible to patients and the public[30].

Our analysis found that there was a significant proportion of videos with misinformation (49.2%). Of these, 22.5% scored between moderate and extreme levels of misinformation. The overall level of quality was low to moderate, highlighted by the quality assessment tools used. On the modified DISCERN questionnaire, the videos had a median score of 3 out of 5, indicating moderate quality of information. Approximately one quarter of the overall total scored as having low levels of reliability. On the GQS, the videos had a median score of 3, indicating the information was somewhat useful to patients. However, 16.6% of the videos were found to have poor or generally poor information that would have minimal benefit for the viewer. Quality analysis studies within the field of urology have found that information available to patients on YouTube contains moderate to high levels of misinformation[31]. They also found that the included information was of low to moderate quality and reliability[32,33].

A contributing factor to the high levels of misinformation is the ongoing development of the medical field and our understanding of the conditions within it. A subgroup analysis was completed to review the quality of the information published from 2016 to 2021. A statistically significant difference was found when comparing the GQS and the level of misinformation with data published before 2016. This can be attributed to the evolution of alternate therapies that may have been disproved or to changes in treatment methods, such as surgical intervention techniques or radiotherapy and chemotherapy[34,35].

The role of the doctor-patient relationship in educating patients on diagnostic and management options is important to the patient-centred care model. The concern with web-based resources is that the quality of information varies significantly, as does the level of health literary expected in the viewer. As the videos are not reviewed for quality, to increased levels of misinter-pretation is both plausible and found to be common.

Only 34.2% of the included videos gave an overview of renal cancer, with many of the remaining studies missing key information such as symptoms, diagnosis, and treatment. The number of videos produced by health professionals is increasing. In our study, 60% were produced by health professionals, and 39.2% promoted the use of treatment methods. Health professionals and patients are increasingly being used in videos (16.7%) to promote newer treatment options, including immunotherapy and robotic surgical options. The popularity of robotic surgery has led to an increase of the use of intra- operative images and videos. Intraoperative scenes were included in 19% of videos made during or after 2016, compared with 12% of those made during or before 2015. Only 29% of the videos on treatment discussed side effects or postoperative complications.

The increasing use of YouTube videos that focus on diagnostic methods or treatment options may have been influenced by commercial interests and the potential for gain. In this study only 5.8% of the videos included conflict of interest disclosures. Videos on YouTube with greater advertising and promotion of medical treatments have higher levels of popularity. We found that videos with commercial bias have a higher standard of audio-visual quality. Loeb et al.[33] found that 17% of the YouTube videos included in their study on bladder cancer showed commercial bias and reached 324 287 viewers.

The comments sections of the videos were reviewed to gauge viewer response. Overall, there was a low number of comments per video (median 5), which may indicate poor patient engagement.

A significant portion (21%) of the publishers were found to have removed the ability of the public to comment. In 27% of the videos, the comments included patients/members of the public asking for health-related advice. In only one video did the publisher respond to the request and subsequently advised the person to discuss this concern further with a local health practitioner. The number of commenters asking for further clarification suggests the need for more access to reliable information.

Limitations

The study is limited by multiple factors including the inclusion of only YouTube videos. There are many web-based platforms for patients to use including other search engines such as Google, which is the largest web-based search engine. Further analysis is required to better understand the effectiveness of other search media engines. This study is also limited by design, this is a cross sectional study and YouTube changes rapidly, which makes reproducing the information and study design challenging.

Conclusion

The use of YouTube as a readily available source of health information on renal cancer has yet to be evaluated thoroughly. The information on YouTube assessed in this study was found to be low to moderate quality. There were significant levels of misinformation due to the ongoing development of treatment methods and understanding of the disease. YouTube should not be the sole source of information on renal cancer for patients or the public, and is best used in conjunction with information and advice from a medical practitioner and the health care system.