*Article* **Methodological Proposal for the Detection of the Composing Elements of Vulnerability Regarding Disinformation**

**Belén Puebla-Martínez, Nuria Navarro-Sierra \* and Gema Alcolea-Díaz**

Departament of Communication and Sociology, Faculty of Communication, Campus of Fuenlabrada, Universidad Rey Juan Carlos, 28942 Madrid, Spain; belen.puebla@urjc.es (B.P.-M.); gema.alcolea@urjc.es (G.A.-D.) **\*** Correspondence: nuria.navarro.sierra@urjc.es

**Abstract:** We live in a hyper-informed society that is constantly being fed with information stimuli. That information may not be correct, and society may be vulnerable to it. We present a methodological proposal with a mixed approach that allows the learning of the characteristics and weaknesses of news consumers in the face of disinformation. Said methodology moves away from the traditional model, and with it a new, much more complete and complex way of conducting discussion groups is carried out. The qualitative approach is carried out through the creation of an online community in which subjects are encouraged to participate in different activities and tests. On the other hand, in order to obtain quantitative data, a quasi-experimental survey where respondents are exposed to various stimuli created ad hoc, which seeks to measure the interest and credibility of different news items through an orthogonal design, is carried out. The use of this methodology will allow for an expansive and intensive approach to the knowledge of societal vulnerability factors, and with the subsequent results, a solid basis of disinformation can be established, which will allow for the development of a series of strategies to combat disinformation.

**Keywords:** methodology; online community; questionnaires; fake news; disinformation; vulnerability

#### **1. Introduction**

Social science research evolves over the years, as have the discipline's own objects of study. The rise of social networks, new media, technological advances, and other forms of media consumption, among other transformations, means that the analysis techniques traditionally used have to be updated and adapted to the current moment.

Currently, one of the topics that is arousing most interest in the scientific community is the analysis of disinformation and fake news in its broad spectrum, from its production to the effects it produces on citizens. Numerous authors study the factors that condition a person to be more or less vulnerable to disinformation.

The concept of vulnerability in this context can be understood as the weakness of consumers to identify manipulation (intentional or unintentional) by the media in sharing false or incorrect information (for further development of the concept of vulnerability, please refer to the studies conducted by Menczer and Hills [1], the report from the Council of Europe [2] or the research of the Data & Society group, where it is possible to highlight the one by Paris and Donovan [3]). To achieve this, different researchers use quantitative techniques, such as surveys, and/or qualitative techniques, such as in-depth interviews or focus groups. These types of techniques, which have a long history and a continuous presence in communication studies, have shown over time that they comply with the scientific rigour they are supposed to have if they are used correctly. In the case of surveys, they make it possible to reach a large yet superficial number of the population in an extensive yet superficial way. In the case of interviews or discussion groups, they go deeper into the subject with small profiles of the sample in a significant yet not representative way. As Wimmer and Dominick explain [4], these techniques prevent us from getting to know the individual in his or her different facets, such as his or her values, feelings and emotions

**Citation:** Puebla-Martínez, B.; Navarro-Sierra, N.; Alcolea-Díaz, G. Methodological Proposal for the Detection of the Composing Elements of Vulnerability Regarding Disinformation. *Publications* **2021**, *9*, 44. https://doi.org/10.3390/ publications9040044

Academic Editor: Dangzhi Zhao

Received: 8 August 2021 Accepted: 21 September 2021 Published: 26 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

in the face of misinformation. It should not be forgotten that a survey is often presented with closed or scalar questions that prevent the individual's position from being developed more broadly. In the case of the focus group, where in less than two hours individuals have to present their opinions with the disadvantages that this entails, such as the negative effects of "Face-to-Face" group meetings, the monopolisation of the group by one of the members or the complexity of the analysis since, on many occasions it depends on the non-verbal reactions of the participants and also on their communication styles.

This article presents a methodological proposal that aims to make up for these shortcomings thanks to the structure of the tool, which through various aspects, such as anonymity, the iteration of different formulas when posing a problem so that decisions can be modified or argued, feedback with moderators and with the rest of the participants, intragroup homogeneity and intragroup heterogeneity of the participants, ensure the validity of the results. On the other hand, as will be seen below, different activities and processes are used to obtain information from individuals in a holistic way, covering a wide variety of fields and taking into account spatial and temporal dimensions (it lasts more than two months), so that chaos is not formed, and there are no individuals who monopolise the research.

Some works approach the study of vulnerability factors from inductive–deductive processes. Monteiro Borges and Rampazzo Gambarato [5] study the role of beliefs and behaviour on Facebook and their relationship with fake news, using a methodology that encompasses the qualitative conceptual study of Peircean semiotics, focusing on different concepts, such as reality or perception, to investigate the relationship between algorithms, fake news and transmedia journalism. Even within serious games, fake news and disinformation have been the subject of study by Gomez and Carillo [6].

Many authors deal with the study of cognitive skills, attitudes or cognitive biases, among other explanatory elements found at the base of vulnerability, from experimental or quasi-experimental research. This is the case for Saunders and MacLeod [7] in their search for the circumstances related to misinformation and its variation, depending on aspects such as memory, or that of the methodology followed by Dibbets and Meesters [8] to corroborate confirmation bias in children and young people. Although, these authors point out that, among the limitations of the experiment developed, the children who participated in it were forced to select one of the answer alternatives presented.

However, authors who have used the experimental method to study the effects of levels of political competence and media literacy on the detection of manipulated information point out its suitability, due to the need to test cause–effect hypotheses [9].

In these study processes, the use of tasks to assess different skills is common, as in Wineburg and McGrew [10], Wineburg et al. [11] or Nygren and Guath [12]. In some cases, participants are exposed to the evaluation of digital information in sessions conducted by the authors, such as Wineburg and McGrew [10], who recognize that any task that involves researchers creates an artificial environment that can distort what people usually do.

Other researchers combine the use of online and offline assessment tasks, such as McGrew et al. [13] in their evaluation of students' civic online reasoning. In other cases, the use of these tasks is carried out through an online survey, in which the test items are included. Hatlevik et al. [14], to detect the factors that can influence how people navigate new information, facts and digital environments, used a survey with a digital competence test and a self-administered questionnaire. Nygren and Guath [12], in trying to establish the difficulties and abilities in determining the credibility of digital news, used an online survey with evidence elements designed by researchers in education and psychology, in collaboration with active teachers and in line with previous works [13,15], to ensure its internal validity and reliability. Although, they point out as a limitation that the sample is not random, and the survey is collected by teachers interested in signing up for their classes in order to participate. Instead, Kelly [16] in his study of people as biased information processors, conducted a nationally representative survey experiment combining two powerful research tools.

This paper aims to provide a methodological proposal that can be applied in subsequent research to achieve conclusive and profuse results regarding the vulnerability factors to misinformation, combining the validity and scientific rigor of the reviewed research, while increasing the benefits of traditional techniques and reducing the disadvantages that these may present. This methodological proposal has already been applied to the national project, "Study of the conditioning factors of misinformation and proposal of solutions against its impact based on the degrees of vulnerability of the groups analysed", financed internationally by the Luca de Tena Foundation and the social network Facebook, within the framework of the public contest "Academic research on disinformation in Spain and approach to: anthropological, economic and sociological aspects that motivate it, history, expansion and current situation and proposals for solutions", which had an endowment of 62,000 euros.

#### **2. Contextualisation of the Proposal: What Does Disinformation Mean?**

In order the understand how the methodological proposal works, it is necessary to explain how it works and what effects disinformation generates.

Disinformation has become a global phenomenon, to the point of talking about the post-truth era and the post-factual world [17]. The rise of social networks and their preponderance in the reconfiguration of public information spaces has led to citizens being overexposed to disinformation. The digital ecosystem in general, and social networks in particular, favour disinformation by increasing the potential audience of these messages and allowing for their re-broadcast [18]. This, together with the increase in connections between individuals and the speed of information transmission, allows an essential factor in its proliferation: virality. In addition, the increasing demand for news focused on emotional aspects, which is in turn more likely to be shared, also benefits disinformation [9]. On the other hand, this context prioritizes the immediacy of the information over reliability, affecting journalistic production routines [19,20] and increasing the difficulty of communication media to contrast much of the information they produce.

The High-Level Expert Group on Fake News and Online Disinformation of the European Commission recommends the use of the term disinformation when referring to bad informative practices [21]. The widespread expression "fake news" usually refers indistinctly to three concepts with different nuances: misinformation, disinformation and malinformation. Malinformation refers to information built on real data that, regardless of its informative relevance, is used as a weapon to attack people, organizations or states [18]. Misinformation and disinformation are distinguished by the intention of the sender in spreading false information, which in the case of disinformation is carried out with the knowledge that this information is disinformation. Misinformation is increasingly recurring with forwarded messages, especially on social networks, which focus on, among other issues, non-existent false public health alerts or simply on erroneous information [22]. On the other hand, an example of disinformation would be the creation of deceptions known as hoaxes [23].

Although there are authors who had not at first considered intentionality as a disinformative characteristic, pointing out only the misleading nature of the content [24], this element was later added, making it a common consideration as an indispensable intrinsic factor [25–28]. Other authors broaden the meaning of the term by also including the lack of information on certain topics from citizens [29,30].

Fallis points out that disinformation continues to be a type of information, misleading and intentional, with the ability to create false beliefs about the world or reality, and characterizes it with, among other things, the following distinctive notes [31]: it is usually a government or military activity; it is often the product of a carefully planned and technically sophisticated deception process; it may not come directly from the source attempting the deception; it can be widely distributed or, conversely, targeted at a specific group or organization. Disinformation, which ultimately produces an abuse of power [32], ends up harming the population by eroding their trust in institutions and the media [31].

The importance of its potential effects has captured the attention of European institutions [33,34], as has the determination of the causes of vulnerability to disinformation, the effects of which differing, among other factors, by educational levels, democratic culture or trust in institutions [33].

The disappearance of the gatekeeping function that was traditionally associated with mass communication has generated as a secondary effect a great difficulty, on the part of the recipients, to discern what is trustworthy and what is not [35]. The ability to assess the credibility of information is related to cognitive skills and attitudes. The lack of knowledge of the digital environment is very high [36], detecting a digital divide that reflects levels of education and social inequalities [12]. Similarly, previous beliefs, the coherence of the message and cognitive ability [12], as well as ideologies, age, memory and personality traits, can affect the evaluation of information [37]. All of which can be increased by the consumption occasion in which subjects may find themselves, perhaps overworked, tired or stressed, making it difficult to undertake effective evaluations [38].

Likewise, the cognitive biases used for the extension of informational disorders [39], such as confirmation bias in the evaluation of fake news, a natural tendency to blindly second messages related to one's own beliefs [40], should be considered. This has been found to be true for all age groups [41]. The consonance of the falsehood with the beliefs and desires of the receiving subject reduces the possibility of questioning their falsehood [42], since beliefs are typically resistant to change. This is the case even in the face of data that contradict said beliefs, especially in cases with a strong ideological association [43,44].

On the other hand, indicators such as people's perception of how often they find false information and their confidence in identifying it show differences when observed through sociodemographic analysis, revealing, for instance, that it is the youngest (between 15 and 24 years) and respondents with the highest educational levels who say they more frequently find fake news [45]. However, this confidence clashes with the position of some authors who have determined that especially young people may find it difficult to assess whether online information is reliable and to recognize disinformation [41,46].

In order to investigate disinformation in Spain, a systematic research methodology was built and endowed with economic resources that allow for articulating this central concern on disinformation. It aspires to contribute significant data to a heated debate with undoubted social impact.

This work presents a methodological proposal for the study of the constituent elements of vulnerability to disinformation. This methodology has already been applied in a research project that analyses the conditioning factors of disinformation and has been partially replicated in other investigations with positive results, which guarantees its validity and replicability.

#### **3. Methodological Proposal**

The proposed methodology is capable of shedding light on the characteristics of disinformation, with a special emphasis on the detection of vulnerable audiences and the subsequent formulation of actions whose implementation can help combat the effects of the phenomenon effectively.

The data obtained from the application of this methodology can provide systematic and reliable support for the actions proposed to combat the phenomenon of disinformation. The research design was planned in a way that approaches the problem from complementary methodologies and presents sufficiently broad coverage.

In this way, a work system is proposed to investigate the degree of vulnerability to disinformation concentrated on the independent variable of content consumption. The aforementioned system methodologically triangulates between quantitative and qualitative techniques (surveys and discussion groups in the Sensors online community, respectively) to increase the reliability of the results by approaching the phenomenon studied from different perspectives.

It is understood that the results that will be obtained from the two techniques will not only complement each other but will suggest similar conclusions, as other research has shown [47].

Thanks to the different techniques of the methodology being based on an exploratory approach, they can respond to a wide variety of research objectives. The following are examples:

	- Sociodemographic variables: level of income, social status, area of residence.
	- Intermediate or psychosocial variables: political orientation, strategies and thought patterns, attitude components, social attribution phenomena, cognitive dissonance, stereotypes, vulnerability to rumors, etc.
	- Influence of the source, support and type of channel on the level of acceptance of disinformation.
	- Role of the information content and level of specificity of said information.
	- The effectiveness of disinformative content enhanced by the presence or lack thereof (along with the accumulation) of news values.

#### *3.1. Methodological Design*

A complex study with different phases has been designed to propose a method for a comprehensive study of disinformation. It is based on data production and cross-analysis tools. For this purpose, the qualitative and quantitative approaches are separated into two different studies, whose data can be pooled for validation and new results. Since, as suggested by Morse and Chung:

"[...] simultaneous or sequential triangulation of more than one qualitative method or combining qualitative and quantitative methods provides a more balanced perspective, moving toward holism [...] The use of multiple methods leans toward developing a systematic research program, with one study dictating the direction and nature of the next. In this way, the researcher may carefully identify and encompass the scope of the phenomena or project, with each study being complete in itself [...] With minimal overlap between these projects, but with each project validating and extending the previous, the results may be fit together to form an understanding of the concept" ([48], p. 18).

So that the results are confirmed by this comparison, as Hernandez Sampieri, Fernandez Collado and Baptista Lucio indicate, "the idea is that when a hypothesis or result survives the confrontation of different methods, it has a higher degree of validity than if it is tested by a single method" ([49], p. 789).

The proposed timeframe is as follows:


• Final project meeting: project closing workshop and presentation of the results report.

#### *3.2. Target Study Variables, Field Work and Research Phases*

Regarding the sample cross variables, the following decisions are made to better address the susceptibility to being disinformed:


Finally, the complex cross-sectional variable of "disinformation effectiveness" is, at least, threefold:


After the initial workshop, the qualitative phase and the sample design that will best adapt to the object of study are defined, using innovative methodologies, such as web squared (a hybrid device with online and offline moments).

More specifically, the Sensors community and its platform are used together with other ethnographic work environments (mobile groups via WhatsApp).

This device of an online community plus smartphones multiplies the possibilities of traditional qualitative research since it allows:

	- Firstly, the exploration of the "opinion" of the participants on the matter both individually and as a group, and to encourage the exchange of perspectives, the subsequent elaboration, the reflection (discourse), etc.
	- After a reasonable period of time (a fortnight), begin to present informants with various news items (true or false) for their consideration without clarifying whether the content is true or false, in order to verify de facto, the real vulnerability to disinformation on the part of the sample (the behaviour).

• This allows the contrast of the intentions and theoretical disposition of the sample to canards, errors, fakes, etc., with their actual coping behaviour, observing the inconsistencies between both variables, as well as subsequently guiding a purely experimental methodology in the quantitative phase to verify the first data obtained.

#### *3.3. Qualitative Phase Specifications*

The Sensors community is an online tool of reciprocal and multichannel impact that, creatively and continuously in the period that is established, launches different qualitative ways of doing research through different spaces, accessible through various interaction supports on/off: computer/tablet plus smartphone, etc. These spaces include:


#### 3.3.1. Implementation Process

The way of carrying out a global investigation involves a progressive blending of the different qualitative elements worked on, which is:

• Data production → continuous analysis of said data → input of learnings in subsequent steps → until the construction of the final quantitative phase.

In fact, the last tasks and qualitative analysis coincide in their final stretch with the beginning of the quantitative field, since they support from the base a strategic definition of the device, its clearly experimental approach, the definition of the variables to be manipulated and the news content tested with the questionnaire used.

Commencement of the qualitative phase is the beginning of field work on the Sensors platform and its continuity through different tasks, supports, channels and challenges:


#### 3.3.2. Sample Design of the Online Community

The relevant variables for the sample design of this methodology, which are inspired both by the proposed objectives and the considerations that arise in the initial workshop, are the following:

	- Adolescents + youths.
	- Young stable couples without children aged 28–34 and couples with young children up to 12, aged 30–45.
	- Couples with children over 12 years of age between 44–55 years old.
	- Empty nest + 55 years old. Different political attitudes: distribution according to subjective statement on political ideology (left, centre-left, centre, centre-right, right and undefined). For the adolescent segment, the distribution according to political ideology is not considered; a group of its own is configured with this segment.

were distinguished in the segments, as well as people who work and people who do not work.

#### 3.3.3. Selection of Participants

The selection of participants for the community was carried out in three phases:


From the beginning, participants are informed that there will be two or three dynamics per week.

#### 3.3.4. Activities and Examinations

Some details and elements of the different activities and examinations require further specification:


• Scale of attitudes on issues related to national security, the progress of the economy, confidence in the future, the vision of human morality, progress, amongst others are used in order to contrast the results with the responses and attitudes expressed towards news items throughout the experience.

#### *3.4. Quantitative Phase Specifications*

Disinformation is a broad, complex and multifaceted field of study. However, from a quantitative perspective, it is essential to have concrete, well-defined and operationalizable concepts. Following this premise, the first step is to establish what is possible and what is not possible to measure. This implies, necessarily, leaving out of the scope of the study certain complex (if not impossible to be addressed with guarantees through quantitative methodology) aspects or dimensions. That is, in this phase the field of vision of the study must be reduced to ensure adequate observation.

In order to determine which factors make disinformation more or less credible, it is essential, firstly, to define these factors in order to measure and quantify their effect. In this sense, the qualitative phase helps, among many other things, to establish an exhaustive list of elements that affect the greater credibility of information.

The first task is to specify the possible scope of the quantitative phase by selecting, from the qualitative list, those factors that it is possible to use. Two criteria are taken into account in this selection: importance and measurement. That is, the factors whose effect on credibility are clearer are selected (disregarding others with a more residual weight), and which in turn were transferable to a questionnaire. Analysing the list extracted from the qualitative phase, it is possible to group the factors into three large dimensions:


Under this scheme, the following factors are selected:

In total, ten different factors are measured that meet the two aforementioned selection criteria: they are relevant for the message to be more or less credible, and they are measurable.

The list of factors selected has, in turn, additional implications that affect the scope of the quantitative phase of the study. The items of (dis)information used in the measurement must fit said factors.

It is therefore necessary that the pieces used have a clearly identifiable source/origin. Consequently, canards such as the ones that arrive via WhatsApp and that, regardless of the sender, are anonymous, are left out.

In turn, it is necessary that they are (dis)informative pieces that have been (supposedly) published in some written medium (online or paper). Rumours that come from informal comments from the environment or false news from media such as television or radio are left out. To make the news items more realistic, respondents will read the headlines. Expressing television or radio news pieces in writing makes the situation too artificial. Consequently, it is decided to omit these media in order to guarantee a higher quality of

the data. In addition, to achieve certain realism, it will be necessary to create videos and audios of the different news items, which is excessively complex.

#### 3.4.1. Prospective Approach

Once the what (range) is defined, it is necessary to make decisions about the how (focus). From a methodological perspective, two possible paths are opened to face measurement:


After analysing the pros and cons of the two possible alternatives, it is more convenient to opt for the second path and confront the respondents with an experimental situation: they will be exposed to a series of stimuli (news pieces) and they will indicate, through the questionnaire, what their perception is of these and what behaviour they have before them.

#### 3.4.2. Design of Stimuli for the Questionnaire

The following step, once the approach is decided, is to establish the stimuli that will be used in the measurement. At this point it is important to recall that one of the objectives of this methodology is to learn what factors affect the greater or lesser credibility of a news item. For this, there is a defined list of ten factors with their corresponding levels (Table 1). The stimuli that are used, consequently, must be able to be characterized as univocally as possible in a level of each of these factors.

It is advisable not to resort to fake news already published, given the complexity of finding valid examples for all the necessary options. At the same time, the possibility that some respondents already know the news items is avoided and their spontaneous reaction to it is not measured. Therefore, it is necessary to use fake news ad hoc designed for the investigation. However, how many news items are necessary? All the possible combinations of the factors and the levels used suppose such a high number that it seems virtually impossible to handle them in a single investigation. In this situation, it is recommended to generate a fractional factorial design (Table 2) that, ultimately, reduces the number of possible combinations to a more manageable one.


**Table 1.** Dimensions, factors and levels used.


**Table 1.** *Cont.*

Source: own elaboration.

**Table 2.** Fractional factorial design example.



**Table 2.** *Cont.*


**Table 2.** *Cont.*

Source: own elaboration.

The generated design must meet the following characteristics:


For the remaindering factors, the information should be included in the most specific way possible and without identifying specific examples (Table 3). In this way, all respondents are provided with the same information. Hence, their interpretation is more controllable and less subjective than when talking about specific media.

**Table 3.** Final example of a stimulus.


(It is shared, retweeted, sent and commented a lot)

Source: own elaboration.

It is not possible, for instance, to know how each respondent ranks a specific newspaper in the factors that are being measured. However, if it is directly indicated that it is an online newspaper with a centre-right-wing ideological line and with few readers, the information has a much more univocal and less interpretable character.

On the other hand, it is concluded that exposing each of the respondents to all possible stimuli is unwise. The duration of the interview would be longer than is advisable, causing fatigue in the respondent and, ultimately, making the quality of the responses and the information obtained inadequate.

To avoid this problem, it is recommended to expose each interviewee to six different stimuli. Different message rotations are designed for this, ensuring that each of them is composed of different topics and that each respondent sees the greatest variability of messages in relation to the factors being measured.

Additionally, another objective is addressed: to learn what elements influence the degree of vulnerability that individuals have with respect to the (dis)informative messages. To respond to this objective, the characteristics of each of the respondents must be measured in the questionnaire, in order to see how they relate to the credibility granted by the news tested: sex, age, educational level, exposure to the Internet, degree of interest and information with the topics to which the news refers, ideological position and socioeconomic level.

#### 3.4.3. Weight of Factors According to Interest and Credibility

For every visualized stimulus, each respondent is asked their degree of interest, degree of credibility and their behaviour: would you expand on the information/click on the link/read it in full/share it on your social networks/send it to your private contacts/comment on it with your environment (friends, family, colleagues, etc.)?

The Conjoint procedure was used to determine the weight of each of the factors in the credibility and interest of the news. Conjoint analysis is a statistical procedure that allows the determination of the importance of a series of aspects or characteristics (in this case, the ten factors), without asking directly about those characteristics.

Initially, the Conjoint procedure was developed for its use in mathematical models of psychology and its application to marketing. In fact, its use was limited for many years, mainly for two reasons: the complexity when designing, executing and analysing a study of this type, and the low power of computers in the eighties and nineties, which prevented the analysis of moderately complex Conjoint studies.

In recent years, this technique has once again received the attention of researchers because, amongst other reasons, current computers can carry out complex analyses in minutes (or a few hours). The Conjoint procedure results in a utility score—called a partial contribution— for each level of the factors. These scores provide a quantitative measure that expresses the effect of each level on the credibility and interest of each news item.

#### 3.4.4. Design of the Online Questionnaire

Finally, the technique chosen for this phase is an online panel survey. The decision to use this tool responds, fundamentally, to the need to show stimuli to the interviewees. The online interview enables respondents to read all the information with some confidence. In the case of conducting a telephone survey, it is more complex for the participants to assimilate all the stimuli and give quality responses.

At the same time, compared to other alternatives that also allow the interviewees to read the stimuli, the online interview allows a greater geographical dispersion, guaranteeing, in this sense, a more representative sample.

For the design of the sample, the variables that may a priori have a greater influence on the object of study will be taken into account: age and level of studies, namely. The distribution by sex and geographical area has also been controlled.

Therefore, a multistage sample is recommended, based on proportional and stratified random conglomerates, taking into account the idiosyncrasy of the country where the research is carried out that meets the quotas for sex, age and level of studies. This is performed considering the sampling error standards, where a confidence level of 95.5% is sought, presenting a real error of ±5.0% for the entire sample.

#### **4. Discussion and Conclusions**

This work provides a methodological proposal for the study of the vulnerability factors contributing to misinformation. The research design approaches the problem from complementary methodologies. The system methodologically triangulates between quantitative and qualitative techniques providing, as indicated by Hernandez Sampieri, Fernandez Collado and Baptista Lucio ([50], p. 791), greater confidence and validity to the results and greater sensitivity to the degrees of variation not perceptible with a single method.

We have worked with a qualitative technique through the Internet, located between the qualitative panel and the focus group, using an ad hoc qualitative tool, the Sensors methodological instrument, developed by the Analysis and Research group, a digital community subject to different variables, which integrates and represents different social sectors.

At the same time, it is combined with quasi-experimental research, in line with previous work in this field, such as Saunders and MacLeod [7]. The proposed methodology is aligned with the methodological construct followed by many other previous works, such as McGrew et al. [13], Hatlevik, Guðmundsdóttir and Loi [14] or Nygren and Guath [12], in which the use of tasks to evaluate different aspects, such as skills, is common, combined with the use of the survey and, in our case, with a clear bet for its representativeness, such as Kelly [16]. Another fundamental line is that the activity that involves researchers in the process is reduced to a minimum, avoiding creating the artificial environment indicated by Wineburg and McGrew [10].

However, previously existing content is not used, rather it has been developed to offer an ex profeso stimulus in which the specific levels of the proposed study factors are determined, unlike other works, such as the one cited by Nygren and Guath [12], in which the evaluated variables are native ads, unknown comments and scientific evidence. In our case, we design a set of stimuli uniquely characterized at one level for each of the factors. In addition, the established factors have already been used in other work, and some of which has already been published [53].

In short, the application of the methodological proposal results in an integrated analysis whose specification of data, operations, calculations and theoretical models consulted show that what is important for the work process is the following:


This has been evidenced in its application to the aforementioned national project, "Study of the conditioning factors of misinformation and proposal of solutions against its impact based on the degrees of vulnerability of the groups analysed", financed internationally by the Luca de Tena Foundation and the social network Facebook, within the framework of a public contest.

In addition to the data reflected in the report presented, the amount of information collected during the investigation is allowing new results to be obtained by crossing specific information from the different variables that show a series of synergies that had not been contemplated in the initial objectives and hypotheses.

The results achieved, and those that are being achieved with this new approach, make it clear that this methodology allows a holistic study of a polyhedral reality, such as disinformation.

Finally, with the results that can be obtained by applying this methodological proposal, some lines of action with which to deal with disinformation can be proposed:

• Create news verification networks in order to neutralize disinformation campaigns between the different social agents.


**Author Contributions:** Conceptualization, B.P.-M.; methodology, B.P.-M., N.N.-S. and G.A.-D.; validation, G.A.-D.; formal analysis, N.N.-S.; investigation, B.P.-M., N.N.-S. and G.A.-D.; writing original draft preparation, B.P.-M. and N.N.-S.; writing—review and editing, G.A.-D.; supervision, B.P.-M., N.N.-S. and G.A.-D.; project administration, B.P.-M.; funding acquisition, B.P.-M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research is part of the national project "Study of the conditioning factors of disinformation and proposal of solutions against its impact according to the degrees of vulnerability of the groups analysed", financed internationally by the Luca de Tena Foundation and the social network Facebook in the framework of the "Academic research on disinformation in Spain and approach to: anthropological, economic and sociological aspects that motivate it, history, expansion and current situation and proposals for solutions".

**Data Availability Statement:** Data sharing not applicable. No new data were created or analyzed in this study. Data sharing is not applicable to this article.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

