Digital Information Credibility: Towards a Set of Guidelines for Quality Assessment of Grey Literature in Multivocal Literature Review

Abstract

Credibility, in general, can be interpreted as a sense of trust in someone. The credibility of information remarkably influences the public’s willingness to do or not to do some things. In this research study, the credibility of digital news stories can be interpreted as the sense of confidence a person has in a source of available information that affects their behavior. Humans spread less credible information instead of more credible information very quickly because humans take an interest in fear, disgust, and surprise. Less credible news may affect individuals as well as economies. Therefore, there is a dire need in the current digital era to find out what affects the credibility of digital news stories. This study aims to review the published literature and the grey literature to determine the factors affecting digital news credibility and the factors that build credibility in digital news stories. In this paper, we have developed a multivocal literature review protocol to assess the credibility of digital news stories. The multivocal literature review is an advanced version of the systematic literature review that searches for grey literature in addition to the published literature. The expected outcomes after implementing our protocol will be a list of credibility factors and their practices that can play a vital role in ensuring the credibility of digital news stories. Based on this protocol, we formulated guidelines for the quality assessment of grey literature. The future direction of this research is to analyze the factors through multi-criteria decision-making (MCDM), i.e., analytical hierarchical process.

1. Introduction

The World Wide Web (WWW) is the medium of information used by institutions [1]. The information about these institutions is continuously growing. Regardless of the physical location, the World Wide Web quickly shares information. The increase in the consumption of this information rapidly changes with the addition of new Web pages, with a wide range of indexing each day [2]. The rapid changes in technologies and the short lifespan of digital objects compel the researcher to find ways out of this predicament. Hence, there is a dire need to preserve the information on the WWW for future generations. The preservation of information on the WWW is known as digital preservation [3,4,5,6,7]. The information to be stored for future generations may be of any nature, i.e., less-credible information or only information in the literature. We investigated several articles and found that these studies begin with the definition of the term credibility. According to the dictionary, “credible” is defined as “is worthy of confidence, plausible, or believable”. This word is derived from the Latin word “cedere”. Its meaning is to “believe” [8]. Therefore, it can be simply defined as truthfulness or believability [9]. It is closely related to authority, accuracy, quality, and reliability [10]. The general definition of the credibility of information is the believability of information [1]. The believability depends on the individual’s perception, which cannot always separate less credible information from more credible information. Therefore, a major question arises. What content of whom do you trust? [11]. Information in the form of news makes the situation more interesting. Less credible news affects individuals and economies [12,13]. These rumors have huge impacts on the rise and fall of stock prices and large-scale investments. The inaccurate information has disrupted our responses to everything from terrorist attacks to natural disasters [10,14]. We studied many multivocal literature reviews (MLRs) protocols and their guidelines [15,16,17] to develop the protocol presented in this paper.

Motivation and Novelty

News information credibility is as important as general information credibility and becomes more important if the news is generated from multiple sources instantly and continuously. It is necessary to examine the reliability of news articles based on different criteria. The criteria need to be devised using an enhanced systematic approach. One of the primary motivations of this research is to find out the factors that affect the credibility of digital news stories. Mainstream news reflects the traditions, ethics, and skills of society. Preserving credible news will help future generations understand the culture and society of our era. However, such preservation is difficult in the current digital era as we are constantly being bombarded with both credible and non-credible information. To determine which sources are credible and therefore need to be preserved, we need reliable metrics. This study aims to develop these metrics. The contributions of this study are as follows:

• In this study, we propose a set of guidelines for the assessment of unpublished sources, which will help the researchers to adopt a multivocal literature review.
• To evaluate the authenticity of digital news stories, we have devised a multivocal literature review protocol.

2. Literature Review

Educators, librarians, and application providers are interested in spreading information as much as possible. They are often not concerned with its nature and whether or not it is credible [18]. The nature of the information is essential. We are interested in promoting the credibility of information instead of spreading rumors. However, this does not receive much attention as it is the end user’s problem [19]. Finding credible information has become a challenging computational task in an atmosphere with so much non-credible information available. Researchers are now focusing on the nature of information rather than just the information itself. This different thought process is a huge step forward in research into credible information. Digital information media, such as social technologies, play a vital role in spreading less credible information. These technologies facilitate the information providers because they are for the most part not peer-reviewed [20].

News makes the situation more interesting. Whether political, social, sports, entertainment, or satirical news, the news can be less credible or more credible. The distribution of news generally provides its sources. Often, people do not understand the context of the stories they read, but they keep forwarding the information to others.

To fully overcome the lack of credibility in news stories, we first need to determine what is less credible and more credible. This can happen if we examine the diffusion in-depth after differentiating the credible information from the less credible information. This study aims to determine how to distinguish these factors and model best-practices for their implementation. The increase in the number of websites yields an overwhelming variety and volume of content. However, content credibility is a significant question in many situations. The importance of content is based on the increase or decrease of the credibility of the content. The credibility assessment can be incorrect without making the groups of contents, i.e., Article, Discussion Source, Help, etc. [21].

With the emergence of Web 2.0 and social media, the contents of the internet have become the primary source of spreading information. This information boom leads to the tsunami of Web pages having so many news stories, audio, and videos. This information is both less credible and more credible. Therefore, the trend of research from different aspects of content switched to the “credibility of the contents”. The words “credibility” and “Trust” are used interchangeably [12]. Many researchers called the word credibility also believe-ability. Thus, Believable information is credible information. People understand credibility by employing more than one dimension concurrently. All the reviewed research [22,23,24,25] highlights the need for powerful credibility assessment abilities among the users. For example, how do people assess information on the Web? Are they aware of potential bias and vested pursuit of content writers as they determine whether the information is appropriate, correct, or plausible? The problem is particularly vital as, with the exponential proliferation of information on any subject matter over the Web, users emerge as the arbiters of records accuracy in a domain wherein they are informed and in environments in which they may be no longer.

The digital information viewer is facing a significant challenge in the credibility of content enhancement. The Researchers from STUP- Stanford University Persuasive Technology Lab have published a series of publications related to Web content credibility [9,26]. In his article [14], Fog B investigates how different links and domains affect the perception of people’s in-terms of credibility with specific implications. Some of the research studies are based on the effect of the credibility of a business or E-Commerce websites [26]. According to [20,27,28], several studies were reported from the literature on the credibility of advertisement in the field of marketing. These research articles analyze the contents and a few attributes of promotion. One of our research’s major primary goals is the factors that ensure credibility. In the literature, we found that the perception regarding the credibility of the advertisement includes the reputation of the company and its experience. The credibility of the information is measured via certain factors, i.e., the message contents, source credibility, source bias, and source reputation [29]. Medical information has received widespread attention due to its credibility issue. Almost 50% of Internet users seek health information on the Web. Due to the critical nature of the information, many researchers emphasize the need for a credibility assessment of health-related information on the Web [19,30].

The credibility of news can be measured by detecting less-credible news stories. Therefore, Less-Credible news detection can be defined as the provision of specific news articles, i.e., editorial, news report, expose, etc., intentionally deceived [31]. According to Chen, 55% of readers do not read the full article. They only click on the link and go through it without reading it. These types of readings can be considered “click-bait”. This click-bait plays a vital role in spreading less-credible information [32]. The popularity of social media platforms such as Twitter and Facebook pushed data to consumers in a speedy way. The share feature in these platforms makes the information available for all [33]. The standard-issue leads the consumer to pay less attention. While attention to detail becomes, the contents are de-contextualized from their source, and facts are mixed with fiction or even personal bias.

3. Methodology

We have adopted a multivocal Literature Review for this study. A comprehensive literature review (MLR) involves evaluating all available literature, including formal academic publications like journals and conference papers, as well as practitioner literature such as blog posts and white papers. Its purpose is to identify, analyze, and interpret phenomena of interest. MLR studies are particularly useful in Computer Science and Engineering where new advancements are taking place and academic research is limited, as they can provide significant advantages. The work presented in this document is motivated by a set of research questions that form the basis of our study:

RQ 1: What factors are identified in the literature (as well as in the grey literature) to be considered for ensuring digital information credibility of digital news stories?

RQ 2: What are the practices, as identified in the literature (as well as in the grey literature), for implementing the factors to ensure digital information credibility of digital news stories?

3.1. Constructing Search Terms

Population: Digital News Stories

Intervention: Credibility Factors, Characteristics, Practices.

Outcomes of relevance: To determine the Factors and Solutions to ensure credibility in digital news stories.

Experimental Design: Multivocal Literature Review.

3.2. Search Strategy

3.2.1. Trial Search

The search string was utilized to conduct a trial search on Google Scholar and Springer Link.

(“Digital news stories ” OR “information credibility”) AND (“credibility factors” OR “credibility indicators” OR “believability factors”)

We tried the search string in the aforementioned digital libraries and retrieved research publications. After testing the search string, we finalized it. This string will be further used to validate and develop practical search terms.

3.2.2. Identifying Search Terms

We used the following search approach to construct the search terms.

• Use the Research Questions to derive significant terms by identifying population, intervention, and outcome.
• Find alternative spellings and synonyms for the significant terms.
• Verify the keywords in any relevant sources.
• If your database allows it, use the ‘OR’ operator to concatenate alternate spellings and synonyms and the ‘AND’ operator to concatenate significant terms.

3.2.3. Result for the Search String

RQ: What are the factors, as identified in the literature as well in grey literature, to be considered for ensuring digital information credibility of digital news stories?

We formulated the search string as per the syntax/limitations of each digital library. These search strings are given in

Table 1. Search Strings and Databases.

S.NO	Database	Search String
1	GOOGLE SCHOLAR	(“Information Credibility” OR “information believability” OR “news credibility”) AND (“News Stories” OR “Digital News archives”) AND (“credibility factors” OR “credibility indicators” OR “Practices” OR “Solutions”)
2	ACM	[[All: “information credibility”] OR [All: “digital information believability”] OR [All: “information believability”] OR [All: “news credibility”] OR [All: “news believability”]] AND [[All: “digital news stories”] OR [All: “news stories”] OR [All: “digital news archives”]] AND [[All: “credibility factors”] OR [All: “credibility indicators”] OR [All: “success factors”] OR [All: “practices”] OR [All: “solutions”]]
3	IEEE XPLORE	(“All Metadata”: “Information Credibility”) OR (“All Metadata”: “information believability”) OR (“All Metadata”: “news credibility”) AND (“All Metadata”: “News Stories”) OR (“All Metadata”: “News archives”) AND (“All Metadata” : “credibility factors”) OR (“All Metadata”: “credibility indicators”) OR (“All Metadata”: “information credibility Practices”)
4	SCIENCE DIRECT	(“Information Credibility” OR “news credibility” OR “news believability”) AND (“News Stories” OR “Digital News archives”) AND (“credibility factors” OR “credibility indicators” OR “Practices” OR “Solutions”)
5	SPRINGER LINK	(“Information Credibility” OR “information believability” OR “news credibility”) AND (“credibility factors” OR “credibility indicators” OR “Practices” OR “Solutions”)

.

3.3. Resources to Be Searched

3.3.1. Search through Relevant Digital Libraries (Published Sources)

• IEEE XPLORE
• SCIENCE DIRECT
• ACM
• GOOGLE SCHOLAR (SEARCH ENGINE)
• SPRINGER LINK

3.3.2. Search for Grey Literature

To locate documents in the grey literature, we used non-specialized internet search engines. Grey literature is a broad category of publications that includes a variety of report types, such as technical, research, project, and annual reports, as well as working papers, government documents, white papers, evaluations, and videos, among others. These materials are typically not published by commercial publishers and are not subject to their control. As recommended by Garousi et al. [17], grey studies can be identified by exploring search strings on search engines. We have studied a number of MLR protocols while designing this protocol [15,17,34,35,36,37,38,39,40]. We found MLR guidelines proposed by Garousi et al. [17] in the literature. The guidelines suggested two different strategies for searching, i.e., Automatic Search and Manual Search. Various search engines can do automatic Searches like Google search engine (https://www.google.com, accessed on 1 March 2023), Thesis Global databases and ProQuest Dissertations. Manual Search methods consist of an informal pre-search. The detail is given below.

• Google search engine: A sophisticated search will be conducted on a typical Google search engine to obtain suitable master thesis/Ph.D. thesis, as proposed by Garousi et al. [17].
• Digital database: An advanced search will be conducted on the ’ProQuest Dissertations and Thesis Global database to obtain the appropriate PhD./master thesis, as Garousi et al. [17] indicated.
• Methods’ creator websites: A review must include practitioners’ websites releasing papers (e.g., Web pages, reports) on the relevant research field, according to MLR criteria given by Garousi et al. [17]. As recommended by Garousi et al. [17], an informal pre-search will be undertaken to retrieve a list of available sources related to our crucial terms to retrieve the practitioners’ website publishing papers on digital news stories. The following are some of the links:
  • GOOGLE SEARCH ENGINE
  • SOCIAL NETWORKS WEBSITES
  • RELEVANT BLOGS
  • WEBSITES OF RELEVANT ORGANIZATIONS (NOT LIMITED TO THE FOLLOWING LINKS)

    –

    https://libguides.pvcc.edu/evaluatinganewsstory

    –

    https://tinyurl.com/3823y2w2

    –

    https://www.americanpressinstitute.org/publications/reports/survey-research/trust-news/single-page/

    –

    https://library.piercecollege.edu/c.php?g=598055&p=4140227

    –

    https://semanticsholar.org

Google provides numerous studies, and the sources should be limited to a specific size that can provide the best records and be easily readable. Only the first few pages are relevant to the study, while the rest are mostly irrelevant. This relevance can be achieved by using a page rank algorithm [41]. We will stop proceeding with more results once the page does not yield relevant items. Following the MLR guidelines [17], we will combine the inclusion and exclusion criteria for grey literature with quality assessment criteria.

3.4. Search Documentation

The data will be stored in a document to extract the final results. The search results will be recorded in the document provided in

Table 2. MLR Search Results Record table.

S.NO	DIGITAL LIBRARY	Initial Search	Title/Abstract Based Selection	Introduction/ Discussion Based Selection	Full Text Based Selection
1.	ACM	-	-	-	-
2.	IEEE XPLORE	-	-	-	-
3.	SCIENCE DIRECT	-	-	-	-
4.	GOOGLE SCHOLAR (Search Engine)	-	-	-	-
5.	SPRINGER LINK	-		-	-
		-	-	-	-
TOTAL RESULTS		-	-	-	-

.

3.5. Search Result Management

The references for each search string result will be stored in separate folders. Screenshots of the search query results will also be taken and stored in the aforementioned folder to organize the records better. Additionally, the format shown in

Table 3. Search Result Management Format.

S.NO	SOURCE NAME	PAPER ID	PAPER TITLE
1	IEEEXPLORE	IEEE-01	Preliminary Analysis On The Indicators Affecting Islamic Information Credibility In Social Media
2	SCIENCEDIRECT	SD-15	Fake News and its Credibility Evaluation by Dynamic Relational Networks: A Bottom up Approach
3	SPRINGER LINK	SL-03	Credibility-Based Fake News Detection

will be used for the primary selection. Specifically, we will use the SD number of the article for Science Direct, the SL number of the article for SpringerLink, and the IEEE number of the article for IEEEXplore.

The ’paper ID’ is the number assigned to a particular paper by the search query after applying the search string to the digital library. The process of removing duplicated occurrences involves eliminating studies from one of the digital libraries if the same paper is found in more than one library.

4. Publication Selection of Published Sources

The selection of papers from a large number of papers is a very challenging task. We have adopted inclusion and exclusion criteria to select the most relevant articles and exclude irrelevant studies. These criteria specify that the paper should be in English and that the paper’s title should match our research title. Furthermore, if the paper’s title only partially matches, we will review the abstract and include it if it is relevant; otherwise, we will exclude it.

4.1. Inclusion Criteria for Published Sources

When applied to digital libraries, the search string will yield many studies. The data will be extracted from the sources once checked against the inclusion criteria. The sources will only be included if they are deemed relevant to our research questions. We will only consider sources related to the credibility of digital information in digital news stories. The criteria are listed below:

Sources That Explain:

• The factors/motivators/success factors for digital information credibility.
• The practices/solutions for the factors in digital information credibility.
• The context of digital news stories for credibility purposes.
• The context of digital news archives for credibility purposes.

4.2. Exclusion Criteria for Published Sources

Sources that are irrelevant to our research questions will not be included. The decision to exclude them is based on the exclusion criteria, which are explained below.

• Sources that are not relevant to our research questions.
• Sources that do not describe digital information credibility.
• Sources that do not describe credibility in digital news stories.

4.3. Selecting Primary Sources

We are utilizing the tollgate approach proposed by Afzal et al. [42] to select the publications. The first phase of our source selection process involves reviewing the title, keywords, and abstract of primary sources. The purpose of this phase is to identify sources that are most relevant to our research questions and exclude irrelevant sources. The selection is further refined by reviewing the full text of studies and checking against the inclusion and exclusion criteria mentioned above. The sources will be sent to a secondary reviewer for further assessment against the inclusion and exclusion criteria to eliminate any remaining uncertainty. Records will be kept for each source selection to explain why a primary source was included or excluded from the final evaluation. Therefore, this section will be helpful in explaining the rationale behind the selection or removal of primary sources.

5. Publication Selection of Unpublished Sources

In the first step, the search string will be applied to all three sources specified previously in Section 3.3.2, namely Google, ProQuest, and a manual search for appropriate sources. After gathering studies using automatic and manual searches, the titles and abstracts will be thoroughly evaluated using the inclusion and exclusion criteria outlined in the following sections to eliminate irrelevant studies. If we encounter any difficulties in adding or removing a particular report, we will consult with our research supervisors to devise the best possible solution. The inclusion and exclusion criteria will be applied to the entire text of the pre-selected studies in the third stage.

5.1. Inclusion Criteria for Grey Literature

• Ph.D./Master’s thesis, key consulting firms, and digital news agencies who have accompanying support resources on their websites (e.g., case studies, blogposts).
• The study is relevant to the search terms.
• The study is written in the English language.

5.2. Exclusion Criteria for Grey Literature

• Studies that are not explicitly focused on digital news stories.
• Studies that do not address the trustworthiness/credibility of news stories.
• Studies for which the entire text is unavailable.
• Duplicate research items (the same studies can be found in other journals).
• Duplicate research items (the same studies in order, with only the most relevant or significant study included).

6. Publication Quality Assessment

Publication Quality Assessment for Published Literature

Quality assessment is a crucial activity that enhances the strength of a study. To improve the quality of our work, we have designed a short questionnaire to assess the quality of candidate papers. The questionnaire will be distributed via email among senior researchers and experts. After filling out the questionnaire, a scoring plan will be applied as elaborated below. This scoring plan is presented in columns A, B, C, D, E, and Score in

Table 4. Classification Scheme.

Paper-ID	Publication Channel	Publication Year	Research Method	Quality Assessment
				A	B	C	D	Score
1.	Conference	20xx	Case Study	0.5	1	0.5	0
2.	Journal	20xx	Experiment	0	1	1	1
3.	Symposium	20xx	SLR	0	1	0.5	0.5
4.	Workshop	20xx	Ordinary Literature Review	1.5	1	1	1
5.	-	-	-	-	-	-	-	-
6.	-	-	-	-	-	-	-	-

. The mechanism for assessing the literature was suggested by Khan et al. [43] and [44]. We have added a few more checklists to this and proposed the following mechanism for evaluating the quality of the literature:

Has the work been accepted for publication in a reputable journal or conference? This question will be ranked using the Computing Research Education (CORE) 2018 and Journal Citation Reports (JCR) 2018 lists of computer science conferences. Additionally, all seminars, workshops, and symposiums have been categorized as follows:

• To assess the ranking of a paper, it is necessary to consider whether it has been published in a recognized and stable journal or conference. The ranking will be determined by referring to the 2018 Computing Research Education (CORE) and Journal Citation Reports (JCR) lists. The conferences, seminars, workshops, and symposiums have been categorized into four groups based on their ranking: A1, A, B, and C. A conference that is not ranked is given a score of 0. The JCR provides rankings based on Impact Factor (IF), with Q1 denoting the top 25%, Q2 the top 50% to top 25%, Q3 the top 75% to top 50%, and Q4 the bottom 25%. A journal not included in the JCR list is given a score of 0.
• The primary focus of the paper is to investigate the factors that contribute to the credibility of digital news stories. The score awarded is 1 for ”Yes”, 0.5 for ”Partially”, and 0 for ”No.”
• The paper evaluates an approach to deal with the credibility factors of digital news stories. If the paper presents a new approach, it is given a score of 1 for ”Yes” 0.5 for ”Partially” if it offers an existing approach, and 0 for ”No” if it does not present a solution.

Figure 1 depicts the process of the classification scheme. The classification scheme is presented in Table 4.

Systematic reviews, therefore, emphasize the importance of evaluating the quality of the study selection. Quality assessment is an essential step in gaining a comprehensive understanding of the paper’s impact on the subject.

Publication Quality Assessment for Grey Literature

The SADACO (Significance, Authority, Date, Accuracy, Coverage, and Objectivity) checklist is used to assess grey literature. The Fourth International Conference on Grey Literature, which took place in Washington, DC, in October 1999 [45], defined grey literature as “that which is produced on all levels of government, academia, business, and industry in print and electronic formats, but which is not controlled by commercial publishers”. Grey literature encompasses a range of written materials, such as theses and dissertations, which are scrutinized by subject experts during review; conference papers, which are frequently peer-reviewed or presented by individuals with specialized knowledge; and diverse reports created by professionals working in the field. The SADACO approach proposes a mechanism for assessing the quality of grey literature. Other researchers have used a similar approach [34,35,36,37,38]. The SADACO checklist consists of 19 questions, each of which is assigned a score of 1. If the sum of the scores exceeds 50%, the source’s quality will be considered acceptable; otherwise, it will be considered a failure. Figure 2 depicts the SADACO approach.

The detailed process is explained in the following

Table 5. SADACO and relevant questions.

Significance	1. Does it enrich or add something unique, meaningful/ informative to the research context?
	2. Would the research area be lesser without it?
	3. Is it integral and representative and influences the work/behaviour of others?
Accuracy	4. Does the item have a clearly stated aim or brief? If so, is this met?
	5. Does it have a stated methodology? If so, is it adhered to?
	6. Has it been peer-reviewed?
	7. Supported by authoritative, documented references or credible sources?
	8. If the item is secondary material (e.g., a policy brief of a technical report), refer tothe original.
Date	9. Does the item have a clearly stated date related to the content?
	10. Check the bibliography: have key contemporary material been included?
Authority	11. Associated with a reputable organization?
	12. Professional qualifications or considerable experience?
	13. Cited by others? (use Google Scholar as a quick check)
	14. Higher degree student under “expert” supervision?
	15. Is the organization reputable? (e.g., Microsoft Corporation, IBM, PSEB etc.)
	16. Is the organization an authority in the field?
Coverage	17. Are any limits clearly stated?
Objectivity	18. It is important to identify bias, particularly if it is unstated or unacknowledged.
	19. Opinion, expert or otherwise, is still opinion: is the author’s standpoint clear?
	20. Does the work seem to be balanced in presentation?

.

7. Data Extraction Strategy

7.1. Primary Study Data Extraction

The objective of this study is to collect information from pertinent publications to respond to the research questions posed in the review. The data relating to the topic under consideration will be extracted from each publication.

• Information related to the publication, including its title, authorship, journal or conference title, and other relevant details.
• Information or data that pertains to the research question at hand.
  The information to be recorded in the data extraction form is outlined in

  Table 6. Data extraction form.

  Date of review

  Title

  Authors

  Database/search engine for grey literature

  Methodology (interview, case study, report, survey, etc.)

  Sample Population

  Target Population

  Publication Quality Description

  Organization Type (Media Cell, News Agency, research institute etc.)

  Company size (small, medium, large)

  Country/location of the Analysis

  Year

  Critical Success Factors (CSFs)/Motivators/ Factors for ensuring credibility in digital news stories

  Solutions of the CSFs for ensuring credibility in digital news stories.

  News Type: political, sports, crime, cultural, entertainment, accidents, economy, science & technology etc.

  Domestic/Foreign

  Medium: tv, blogs, video, YouTube, social networks, etc.

  .

7.2. Data Extraction Process

The primary reviewer is responsible for data extraction and will approach the secondary reviewer if any issues arise. After completing the extraction process, the primary reviewer will conduct an inter-reliability test. For this test, the secondary reviewer will randomly extract data from some sources, which will be compared to the primary reviewer’s extracted data. If the results are similar, the test will be considered positive. Otherwise, the primary reviewer will review the extracted data again.

Primary Reviewer: MUHAMMAD FAISAL ABRAR.

Secondary Reviewers: DR. MUHAMMAD SOHAIL KHAN.

7.3. Data Storage

The data extraction form in the word document will be stored for each selected source. The SPSS will further be used for the Analysis of the data.

8. Data Synthesis and Analysis

Although the existing SLR guidelines [46] briefly discussed the inclusion of grey literature sources in SLR studies, most SLRs published to date have not included the quality assessment criteria for grey literature in their studies. While guidelines for SLR studies, such as [47] and SM studies [47,48] may help carry out MLRs, they do not provide adequate directions on how to assess the quality of the grey literature, in particular, because grey literature sources should be evaluated differently. When presenting the MLR process, more recent works have cited previously published MLR studies such as [49] and [50]. Our guidelines cover a much broader range of MLR literature than any previous MLR study for evaluating the quality of grey literature. To summarize a lack of MLR guidelines, there are no systematic guidelines for the quality assessment of grey literature. In comparison with previous literature [11,22,35,36,37,42], this paper aims to give a set of guidelines for the assessment of the quality of grey literature. For published literature, we found some guidelines [46,47,48] for quality assessment, but we did not find any criteria and methods for the quality assessment of grey literature. This paper gives the approach SADACO through which we assess the quality of the grey literature. The Significance, Authority, Date, Accuracy, Coverage, and Objectivity (SADACO) checklist evaluates the grey literature.

The data will be synthesized by constructing a single summary table with the columns (S.No, Credibility Factors, Frequency/Occurrences, Percentages) at the top of all credibility factors and practices with their frequencies and percentages. The data extraction form has been designed and reviewed by the authors. Most of the data are extracted from the published literature and analyzed in the following sections. The MLR protocol is currently in the implementation phase. We have got results for some of the areas mentioned earlier in the protocol. After applying the search strategy mentioned earlier to the specified digital libraries, we found 1249 papers from the five digital libraries. The preliminary and final selection information for each digital library is given in

Table 7. DataBases Search Results.

S.NO	Digital Library	Initial Search	Title/Abstract	Introduction & Discussion	Full Text
1.	ACM	25	24	24	24
2.	IEEE XPLORE	87	77	61	59
3.	SCIENCE DIRECT	34	30	28	28
4.	GOOGLE SCHOLAR (Search Engine)	1030	705	385 + 113 + 31	385
5.	SPRINGER LINK	73	29	14 + 2	14
TOTAL RESULTS		1249	865	658	514 + 25

. In order to identify relevant studies for our research, we employed a snowballing technique that involved considering the references cited in the paper as well as the references in which a selected paper was cited. By performing both backward and forward snowballing, we were able to expand our initial set of relevant papers and extract 50 potential studies. Using a tollgate technique, we carefully reviewed and evaluated these 50 studies to identify the most relevant and informative papers. After this careful selection process, we ultimately considered 25 papers that met our rigorous criteria and were the most suitable for our research. With these 25 papers selected from the snowballing technique, we expanded our set of relevant studies even further by identifying additional articles using other search methods. This led to a total of 539 studies that were ultimately chosen for the data extraction process as shown in Table 7. To make it clear which studies were identified using the snowballing technique, we labeled them as ”SB” in our paper. By carefully considering the cited references and using snowballing, we were able to identify a comprehensive set of relevant studies for our research, ensuring that our findings are grounded in the most up-to-date and authoritative research in our field.

After applying the tollgate approach to our study, we selected 514 studies. We removed the duplicate studies counted as 33 in different digital libraries. Hence these studies were omitted from the final list of papers to remove duplication. At this stage of our protocol, we performed a few analyses on the finally selected studies. The first analysis is based on the location where the study is being fulfilled. Figure 3 shows the countries where research was conducted for the studies included in our MLR study. In total, 215 studies were carried out in the USA, 29 in the UK, 27 in China, 16 in India, 15 in Malaysia, 12 in Canada, 11 in Japan, and 9 in Germany, respectively.

Figure 4 shows the continents where research was conducted for the papers included in our MLR study. 48% of the study studies were achieved in North America, 26% were undertaken in Asia, 19% were conducted in Europe, 3% were conducted in Australia, and 3% were achieved in Africa.

Figure 5 shows the study-wise analysis included in our MLR study. 74% of the considered studies were published in journals, 21% belonged to Conferences, 3% were books, and 2% were dissertations.

We have categorized the journals according to impact factor and quartile. We set a threshold of five occurrences for the studies and identified the top 15 journals from the list of all journals.

Table 8. Quartile Based Analysis.

S.NO	Journal Name	Freq	Total Cites	Journal Impact Factor	Eigenfactor Score	Quartile
1	COMPUTERS IN HUMAN BEHAVIOR	10	45,035	6.829	0.05973	Q1
2	EXPERT SYSTEMS WITH APPLICATIONS	5	55,444	6.954	0.04053	Q1
3	IEEE ACCESS	6	105,968	3.367	0.15396	Q1
4	MASS COMMUNICATION AND SOCIETY	7	2.312	3.309	0.00334	Q1
5	AMERICAN BEHAVIORAL SCIENTIST	5	7.962	2.558	0.00725	Q1
6	JOURNALISM PRACTICE	18	2.426	2.537	0.0038	Q1
7	DIGITAL JOURNALISM	14	2.998	7.986	0.00793	Q1
8	JOURNAL OF COMPUTER-MEDIATED COMMUNICATION	9	6.427	5.41	0.00549	Q1
9	JOURNALISM STUDIES	11	3.892	3.741	0.00658	Q1
10	NEW MEDIA SOCIETY	8	10,325	8.061	0.02192	Q1
11	JOURNALISM	7	3.851	4.436	0.00607	Q1
12	JOURNAL OF APPLIED COMMUNICATION RESEARCH	7	1.526	2.855	0.00126	Q1
13	JOURNALISM MASS COMMUNICATION QUARTERLY	5	3.435	4.128	0.00427	Q1
14	JOURNAL OF COMMUNICATION	8	11,437	7.27	0.00963	Q1
15	INTERNATIONAL JOURNAL OF COMMUNICATION	5	4.282	1.802	0.01176	Q1

shows the details. Out of all the sources, including journals, books, conferences, and theses, we drew a graph that showed that 74% of studies were carried out in journals, 21% at conferences, and 5% in books and theses, respectively. In the future, we plan to follow this review protocol to identify the factors that ensure the credibility of digital news stories. We will rank the factors using the Analytic Hierarchy Process (AHP) method.

9. Conclusions and Future Directions

The world today is bombarded with a vast amount of information from various sources, making it challenging to distinguish between credible and unreliable information. Among these sources, news plays a critical role in shaping our understanding of current events and issues. However, the credibility of news information is equally important as that of general information, especially when news is generated from multiple sources in real-time. To ensure the reliability of news articles, it is crucial to develop systematic criteria that can be used for evaluation. This research aims to identify the factors that influence the credibility of digital news stories. It is essential to recognize that news reflects a society’s traditions, ethics, and skills. Therefore, preserving it with credible information can benefit future viewers greatly. The study focuses on digital news stories since they have become a primary source of information in the current digital age. With hundreds of articles published daily, it is challenging to distinguish between credible and less credible information. Therefore, preserving news on the Web with reliable information is imperative. The Multivocal Literature Review protocol was designed to address the issue of credibility in news stories in this study. During the protocol design, the researchers found no guidelines for the quality assessment of grey literature. Grey literature refers to literature that is not commercially published, such as reports, conference proceedings, and theses. Therefore, the researchers designed guidelines for the quality assessment of grey literature, named it SADACO, which was one of the major objectives of this research. The future direction of this study is to implement the protocol and retrieve all possible credibility factors from the mentioned sources. These credibility factors will then be validated by experts through a questionnaire survey. In the last step, the researchers will prioritize the credibility factors through a Multi-Criteria Decision Making Algorithm. This approach will ensure that only the most critical factors influencing the credibility of digital news stories are considered. The importance of credible news sources cannot be overstated, especially in today’s world, where fake news and misinformation are rampant. The role of credible news sources is to inform the public, hold leaders accountable, and promote democratic values. Therefore, any study aimed at improving the credibility of digital news sources should be encouraged and supported. The systematic evaluation of news articles will also help to build trust between news organizations and their readers, further strengthening the credibility of news sources. In conclusion, the credibility of news information is of utmost importance, and systematic criteria for evaluation must be established to ensure its reliability. This research aims to identify the factors that influence the credibility of digital news stories and prioritize them using a Multi-Criteria Decision Making Algorithm. The implementation of the study’s findings will help preserve news on the Web with reliable information, making it easier for readers to distinguish between credible and unreliable news sources.

10. Validation of the Review Protocol

Our collaborator, Dr. Faisal Nadeem, reviewed the protocol at Informatics Complex, H-8, Islamabad.