*3.4. Methodology*

Data was analyzed in Qualtrics and Microsoft Excel. Responses for each question were analyzed to identify the comparative impact of each assessed demographic characteristic on key metrics including respondents' perceptions of their own attitudes, beliefs and predicted actions, respondents' perceptions of others' attitudes, beliefs and actions and respondents' perceptions of ideal attitudes, beliefs and actions.

To evaluate the results of the survey, the implications of the responses to each of these three questions is considered. Through their choices, a respondent may indicate that they believe that their own behavior needs to change—either that they use a metric more than what is ideal or less than what is ideal. A respondent may additionally express that the behavior of others should change to either use a particular metric more or less than they currently do. When these beliefs are not common between self-perception and the perception of others, the respondent may either believe themselves to be better or worse than other people in their use of that metric. These paradigms may indicate different levels of buy-in for possible change. These paradigms are summarized in Table 3.

**Table 3.** Paradigms of survey respondents' beliefs based on the level of self-usage, others' usage, and ideal usage reported for each metric.


When respondents believe that a current status of high matches the ideal, or that self-usage is not ideal and the metric should be used, label mechanisms and educational initiatives may be more likely to achieve buy-in from the public. For example, if respondents believe that they should be more concerned about the sponsors of an article, it is likely that they would be receptive to labeling news media with sponsor information. By contrast, if users indicate that they use a metric too much or the problem (deviation from ideal) is only others' usage levels, this can indicate a potential problem area where usage is not perceived as ideal, but there may be little incentive to change. For example, a problem with only others' usage may mean that most individuals consider themselves to be an exception and not in need of labels' assistance. Using these responses, thus, the categories of labels that could be implemented most immediately with positive reception can be identified. Categories that would not be well received, and those which may require educational support to build public understanding of the metric and how it can be used are also apparent.

For each question, the hypothesis that a logical correlation between the demographic characteristic and responses being measured existed was tested against a null hypothesis of no correlation existing. Each hypothesis was based on a particular type of information about the article, such as it's the identity of its creator, publisher or sponsors. The characteristics which were selected were based on their availability to news article consumers for decisionmaking and were selected as part of a prior study [62].

The data from each metric for each demographic are presented in Sections 4 and 5 and trends present in the data are assessed. In this trend analysis, a positive correlation means that the metric increased along with an increase in the demographic (e.g., more support with older age) and a negative correlation means that the metric decreases with an increase in the demographic (e.g., less support with older age). Qualifiers such as 'minimal' are used to indicate the magnitude of the trend and thus identify trends that may exist but not have practical significance.

#### **4. Analysis of the Impact of Article Title, Author, Publisher and Metadata on Perceptions of Content Trustworthiness**

This section analyzes the impact of several article characteristics and metadata elements on Americans' perception of content trustworthiness. These perceptions are integral to decisions that Americans make when consuming and otherwise using (e.g., posting or sharing) news content.

### *4.1. Article Title*

The assessment in this section begins with article titles. The article title is, arguably, the most prominent feature of many articles (a lead picture may be more prominent in some cases). It tells the prospective reader what they may be reading about, should they choose to read the article, and is typically one of the first things the reader sees about the article—in many cases, seeing the title on a page that links to the actual article.

Given this, respondents were asked about the impact of an article's title on their perceptions of its trustworthiness. This data is shown in Appendix A in Figure A1. The vast majority of respondents felt that it was important, with a minimal number of respondents indicating that it had no impact at all. The impact of the title is the greatest for the 40–44 age group. The 18–24, 25–29 and 30–34 age groups also show high interest in this aspect of the article, with over 50% of respondents in these three groups indicating that they place a "great deal" or "a lot" of importance on this characteristic.

The impact of the article's title shows a negative correlation with educational level. Excepting a small resurgence for master's degree holders, the number of individuals indicating that it matters "a grea<sup>t</sup> deal" declines steadily with additional education. The pattern, when considering the "a lot" responses is less clear, with this second group reaching a similar level (when combined with the "great deal" responses across educational levels).

The impact of gender on the article title's impact on perceptions of trustworthiness is minimal. Male and female respondents both indicated placing "a grea<sup>t</sup> deal" and "a lot" of focus on the title with similar frequency. Males were more likely to place no weight on the title altogether, while females were more likely to place "a moderate amount" of focus on it than males.

Respondents were also asked how they believed the title impacted the trustworthiness perceptions of others. A demonstrable trend again exists between the 18–24 age groups and the 50–59 age groups with the number of respondents indicating "a grea<sup>t</sup> deal" consistently falling (with a slight resurgence in the 30–34 age group) throughout this range. This general trend, albeit with more fluctuations, also exists for the combined number of "great deal" and "a lot" respondents.

Comparing the results from Figures A1a and A2a, it is notable that, in most cases, more individuals indicated that others give "a grea<sup>t</sup> deal" of focus to the title than said that they themselves did. In all instances but one, the 25–29 age group, more individuals indicated that others gave either "a grea<sup>t</sup> deal" or "a lot" of focus to the title than indicated that they themselves did.

The educational level data, presented in Figure A2b, shows a minor trend amongs<sup>t</sup> the some high school, completed high school and some college levels, with growth in the number of individuals indicating that others place both "a grea<sup>t</sup> deal" and "a lot" of focus on the title. Beyond this, there is a notable decline.

Comparing the self-perception and the perception of others, at most levels more respondents indicated that others placed either "a grea<sup>t</sup> deal" or "a lot' of focus on the title than they did themselves, with some exceptions. For example, the some high school level had nearly three times as many respondents indicating that they placed "a grea<sup>t</sup> deal" of focus on the title themselves, as opposed to others. Generally, though, respondents indicated that others paid more focus to the title than they themselves did.

Like with respondents' self-perceptions, there is minimal difference between male and female respondents regarding the level of focus others placed on the article title.

Respondents were asked what the ideal level of focus to place on the title would be. These results are presented in Figure A3. Notably, there is a very different trend present with this data than either the self-focus or perceptions of others' focus. The age group data shows (with the exception of the 35–39 age group) a gradual incline, up to the exception at the 35–39 level, and then a decline for the combined "great deal" and "a lot" responses. This is notably different than the trends visible in either Figure A1a or Figure A2a.

At many of the levels, more respondents indicated self-belief (considering "great deal" and "a lot" responses) in the title's importance than indicated it ideally being important. In all age groups except one (55–59), more respondents indicated others having a "great deal" or "a lot" of focus on the title than ideally would. The level of difference was particularly pronounced in the 18–24 age group, where 25% more respondents indicated others having focus on this (at the "great deal" or "a lot" levels) than ideally would.

No clear pattern is present between ideal title focus and educational level. There is also not a clear pattern between self-perception and ideal perception, with three educational levels having more self-perception than ideal, two having similar levels and two having less self than ideal perception of the importance of the article title. There is also minimal difference between male and females, with regards to ideal levels of focus on article title.

#### *4.2. Article Publisher*

The next characteristic assessed was the importance of the article's publisher. This data is presented in Figures A4–A6. In terms of age-related data, there is a notable drop in the level of importance of the publisher at the "great deal" level between the 18–24 and 35–39 age groups. It is also present in the combined "great deal" and "a lot" levels between the 18–24 and 30–34 age groups.

Comparing this to the data in Figure A4b is of particular interest as there is a demonstrable increase, at both the "great deal" and combined "great deal" and "a lot" levels between the some high school, high school completion and some college levels. This is the opposite of what might be expected, based on the age-related data. Of course, these levels also include individuals who reached that status sometime ago and are now older. Thus, these two trends—when juxtaposed—are of interest as they show that the age-associated trend is not attributable to education but instead to other factors and (similarly) that the education level-associated trend is not age-attributable but instead attributable to other factors.

While the gender data, in previous figures, has shown minimal difference, Figure A4c shows a notable difference between the two genders. While the "great deal" level is similar, males' "a lot" level is notably higher (40% as opposed to 30%). Thus, males seem to place more weight on the publisher of an article than females. Females also indicated no focus on the article publisher approximately twice as often as males.

Data regarding the respondents' perceptions of the importance that others place on the article's publisher is presented in Figure A5. No obvious trends are present in the age group data. The education level data shows a notable increase in publisher importance between the some high school, high school complete and some college respondents at both the "great deal" and "a lot" levels. This mirrors the trend shown in the self-perception data. The gender data shows that males have a perception of others' importance of the publisher of an article that is similar to the importance they place on it themselves. More males indicated a "great deal" and "a lot" of importance than females. There are, also again, about twice as many females indicating attributing no importance to an article's publisher, as compared to males.

Next, the respondents indicated the ideal level of focus to place on the publisher of an article. The decrease in focus with increasing age at the younges<sup>t</sup> age levels, that was present in the self-perception data, was also present in the ideal data, with a notable decrease in importance between the 18–24 and 30–34 levels, at both the "great deal" and "a lot" levels.

Comparing Figures A4a, A5a and A6a, it is notable that, in most cases, respondents feel that they and others are placing too much focus on the publisher. More respondents indicated placing personal focus at the "great deal" and "a lot" levels on the publisher than indicated these levels of ideal focus in seven of the ten age groups. Similarly, more respondents indicated believing others focused on the publisher, at the "great deal" and "a lot" levels, then indicated this as being ideal. Again, seven of the ten age groups indicated more focus than ideal.

### *4.3. Publication Date*

Next, focus turns to the publication date of the article. Data related to the level of focus paid to the publication date is presented in Figures A7–A9. A trend of declining importance being placed on the publication date with advancing age is present in the younges<sup>t</sup> age groups and starting at the 25–29 age group and continuing through the 65 and older group.

Looking at the education level data, there is a notable increase between the some high school and bachelor's degree focus at the combined "great deal" and "a lot" levels. The most educated individuals (master's and doctoral degree holders), though, placed less importance on the publication date (but not as little as the some high school group).

Comparing the genders, males tend to place less importance on the publication date than females, with more females reporting placing a "great deal" or "a lot" of importance on the publication date than males.

With regards to the perceptions of respondents regarding others' perceived importance of the publication date, what is perhaps most notable (in Figure A8a) is the significant fluctuation between age groups. While gradual shifts and trends were present in the self-perception data, the perception of others data lacks a notable trend at the "great deal" and "a lot" levels. There is not a notable pattern between the self-perception and others-perception data.

Comparing the self-perception and others-perception data by educational level finds similarities. The four middle education levels all have a similar level of perception of others' publication date importance to each other. They also are similar to the reported levels of self-importance, both at the "great deal" and "a lot" combined level. Both the self- and others-perception data also show a decline in perceived importance at the master's and doctoral degree levels at the "great deal" and "a lot" combined level. The some high school group is notably different, with no respondents reporting a "great deal" of importance to others, but having more "a lot" of importance responses than the combined "great deal" and "a lot" for the self-perception question.

Finally, the gender data shows more interest in the publication date at the combined "a lot" and "great deal" and combined "a lot", "great deal" and "moderate" levels, despite being similar at the "great deal" alone level. Similar to the self-perception data, it appears that females also perceive the publication date as being more important to others than males do.

In the perceived ideal levels of focus on the publication date of the article (shown in Figure A9), a general decline in importance is seen from the 25–29 age group and older at the combined "great deal" and "a lot" levels. This trend is similar to the decline in importance of the publication date seen in the self-importance and others-importance data presented in Figures A7a and A8a.

Similar trends are also present in the educational level and gender data. A similar level of perceived ideal importance is seen at the middle education levels, with lower importance being placed on the publication date by the some high school level and masters and doctoral degree holders. Females also place more importance on the publication date, mirroring the self- and others-importance data.

#### *4.4. Article Author*

Focus now turns to the perceived importance of the article's author. Data regarding this is presented in Figures A10–A12. The younges<sup>t</sup> age groups place the highest weight on the author. The 18–24 to 30–34 age groups have the highest levels of "great deal" and "a lot" responses (though the 40–44 age group has more combined "great deal" and "a lot" but not "great deal" responses alone). A related trend exists of giving more weight to authors at higher educational levels. This trend is present throughout all educational levels, with limited fluctuation, for "great deal" responses and up to the master's level for the combined "great deal" and "a lot" levels. No major differences are notable in the gender data.

The data in Figure A11 depicts respondents perception of others' focus on the importance of articles' authors. Few patterns in this data are notable. The age level data shows fluctuation and no clear trends. The education-level data largely mirrors the self-perception data. Interestingly, associate's degree holders reported higher concern for an article's author than those that have completed high school and those with some college completed and bachelor's degrees; however, they reported others having less interest in it. Like the self-perception data, females had less "a lot" responses and more "moderate" responses for others-perception and a slightly higher level of reporting no focus on the article's author.

Figure A12 presents respondents' perception of the ideal level of focus to place on articles' authors in assessing their trustworthiness. The age group data shows a gradual rise in the combined "great deal" and "a lot" responses from the 18–24 to 55–59 age groups. Notably, the inverse of this pattern appears to be present in the "great deal" responses, meaning that the middle age groups have the highest "a lot" levels while the younger and older age groups have the highest "great deal" levels (40–44 is an exception to this).

A similar pattern exists with regards to the educational level data, with a gradual rise and then fall. Notably, the two ends (some high school and doctorate holders) both have no individuals giving no focus to the author and thus have the highest combined "great deal"/"a lot"/"moderate"/"little" combined responses. They both also have the highest "great deal"/"a lot"/"moderate" response levels. No notable differences are present in the gender data.

## *4.5. Article Sponsors*

Next, focus turns to the impact of article sponsors on respondents' perception of article trustworthiness and credibility. Figures A13–A15 present data related to this topic.

In assessing the age range data, no clear pattern or trends are present. In assessing the educational level data, there is a general growing weight given to articles' sponsors with increasing educational level, for the "great deal" and "a lot" levels, starting at the some high school educational level and reaching the master's degree level. This trend is also present with the combined "great deal", "a lot" and "moderate" response levels through all educational levels. The gender data is relatively close to parity.

Figure A14 shows respondents' perception of the weight that others place on the article sponsor. Similar to the self-perception data, the others-perception data shows no clear pattern related to age groups. The pattern of growing interest in article sponsors with advanced education level is apparent at both the combined "great deal" and "a lot" level and the combined "great deal" / "a lot" / "moderate" level. Only slight differences again exist between the male and female respondents, in Figure A14c.

Finally, Figure A15 presents respondents' perception of the ideal level of focus to place on article sponsors. Again, no clear pattern exists in the age group data. The pattern of growing interest with advanced education is again present and most notable at the combined "great deal" and "a lot" level. Once again, the genders are close to parity.

#### *4.6. Author's Political Alignment*

Now, focus turns to the impact of the author's political alignment on respondents' perception of article credibility. Data related to this topic is presented in Figures A16–A18.

Figure A16a shows a general trend where the level of weight placed on the author's political alignment increases from the 18–24 age group, peaking at the 40–44 age group, before declining until the 55–59 age group, for the combined "great deal" and "a lot" levels. It climbs again at the 60–64 and 64 and older groups. A trend of growth in focus with higher education level is shown between the some high school and master's degree levels for the combined "great deal" and "lot levels". The doctorate holders have a lower level of focus than the master's degree holders. Males also, notably, give more credence to the author's political alignment than females.

Figure A17 shows respondents' perceptions of others' focus on the author of an article's political alignment. The age series data shows two general upward growth trends in the combined "great deal" and "lot data". The first trend starts in the 18–24 age group and ends at 44. The second starts in the 45–49 age group and continues to the 65 and older group, with a demonstrable drop between the 40–44 and 45–49 age groups. The educational level trend present in the self-perception data is also present in the others-perception data, in the combined "great deal" and "a lot" level data, with growth from the some high school level up through the doctorate holder levels. In the gender data, the greater self-perception of males' focus on authors' political alignment has a corresponding believed greater focus of others, at the "great deal" and "a lot" levels. However, the gender difference is corrected in the "moderate" level and the two genders have similar "little" and "none" response levels.

Figure A18 presents respondents' perspectives of the ideal level of focus to pay to articles' authors' political alignment. Two growth trends, with a decline between them, are present. A less pronounced trend is also present of growing ideal focus on authors' political affiliations with increasing educational level, from the some high school educational level up until the master's degree level. There is marginally more ideal focus on author's political affiliations amongs<sup>t</sup> males at the combined "great deal" and "a lot" level; however, the "moderate" level has slightly less male responses and the "little" and "none" levels are the same for both genders.

#### *4.7. Publisher's Political Alignment*

Next, the perception of the publisher's political alignment's impact on article trustworthiness is considered. Figures A19–A21 present data related to this topic.

Figure A19a shows two growth trends, where older age correlates with more focus on publisher's political alignment. A growth trend is also present, associated with increasing educational level. It starts at the some high school level and continues up to the master's degree level for the "great deal" and "a lot" levels. There is also a notably higher level of focus on article publishers' political alignment amongs<sup>t</sup> male respondents at all levels.

Figure A20 presents respondents' beliefs about the level of focus that others place on article publishers' political alignment. Here, only a trend between the 55–59 and 65 and above age groups is notable. A trend of increasing focus is present in the educational level data from the some high school level to master's degree level. Doctoral degree holders have a notably lower level of focus than master's level respondents (and the second lowest overall). Finally, the greater focus amongs<sup>t</sup> males self-perception is also present amongs<sup>t</sup> males perceptions of other's focus on article publishers' political alignment; however, it is not as pronounced of a difference.

Figure A21 presents respondents' perspectives regarding the ideal level of focus to place on the publisher's political alignment. No clear trend is present in the age group data. A less pronounced version of the education level-associated trend is present. It is notable that, across most educational attainment levels for the "great deal" and "a lot" of focus levels, the ideal level of focus is less than the focus reported for self-perception and others-perception, with two exceptions. Notably, more focus is desired by the some high school group and the doctorate holders group has approximately same level of focus as

ideal in both their self- and others-perception responses. Finally, the gender data shows that more males than females see a "great deal" of focus as ideal. However, the difference becomes less notable at lower levels of concern.

#### *4.8. Sponsor's Political Alginment*

Focus now turns to the impact of articles' sponsors' political alignment. Data related to this topic is presented in Figures A22–A24.

The age group data presented in Figure A22a shows no clear trends. The educational level data shows a general increase in the level of focus placed on articles' sponsors' political alignment along with education level, from some high school to doctoral degree holders at the combined "great deal" and "a lot" levels. The gender-based data shows a greater focus among male respondents on articles' sponsors' political alignment, as compared to female respondents.

Figure A23 presents the respondents' beliefs regarding others' perspectives as to the importance of articles' sponsors' political alignment in assessing article trustworthiness. Like with the self-perspective data, no clear trends are present in the age group data. There is a positive correlation between higher levels of education and additional focus on articles' sponsors' political alignment. There is, also similarly, a greater level of focus on articles' sponsors' political alignment attributed to others by male respondents.

Figure A24 presents what respondents believe to be the ideal level of impact of articles' sponsors' political alignment on perceptions of article trustworthiness. Like with the selfperception and others-perception data, no clear trends are present in the age group data. There is a similar trend of a positive correlation between higher level of education and higher ideal levels of focus on articles' sponsors' political alignment. The trend of males having more focus on articles' sponsors' political alignment is also present in the ideal data.

#### **5. Analysis of the Impact of Article Characteristics on Perceptions of Content Trustworthiness**

This section analyzes the impact of four article characteristics on Americans' perceptions of online article trustworthiness. The impact of the quantity of opinion statements, article virality, article controversy level and article reading level are considered.

#### *5.1. Opinion Statement Quantity*

In analyzing the data regarding opinion quantity (in Figure A25a), no clear patterns are visible. Conversely, the education level analysis shows much less variability and a slight positive correlation between additional education and focus on opinion statements when assessing content trustworthiness. Figure A25c shows that there is slightly more interest amongs<sup>t</sup> females, than males, in the quantity of opinion statements when assessing trustworthiness.

Figure A26 presents Americans' perceptions of others' focus on the quantity of opinion statements in an article when assessing its credibility. A small downward trend is visible in the age group data, albeit with fluctuations, where older age groups are less concerned about the quantity of opinion statements than those in younger groups. No significant trends are notable in the education associated data. There is also no pronounced difference in males versus females.

Figure A27 presents data regarding what Americans think the ideal level of focus on the number of opinion statements in an article should be for assessing article credibility. Again, in the age group data, no clear pattern is present. In comparing Figures A25a, A26a and A27a, the age groups have appreciable correlation across the three types of responses. The correlation (being higher or lower, as compared to adjacent groups) is present across most levels for the self-perception and others-perception data. However, it is only prevalent across the higher age groups, when comparing the self- and others-perception data to the ideal-perception data.

There is, again, no notable trend across educational levels for the ideal data. The ideal level of focus on opinion statements, across educational levels, seems to be closer to the self-perception than the others-perception data. Notably, many age groups indicate more others-perception than ideal perception. Finally, the differences between the genders are minimal, with slightly more females using opinion statement quantity when assessing news trustworthiness.
