Reliability and Validity of Widely Used International Surveys on the Environment

Abstract

Do existing public opinion surveys provide valid and reliable measures of attitudes towards environmental sustainability? This question is critical given the importance of public support for achieving sustainability. Starting with 28 survey items about the environment drawn from the World Values Survey Waves 5 and 6 and the 2010 International Social Survey Program, we assessed reliability by checking for significant correlations between similar or identical items on different surveys. Next, to assess validity, we evaluated correlations between survey items and 22 objective environmental indicators drawn from the Environmental Performance Index (EPI). As the level of economic development is a likely confound, we also performed partial correlation analyses controlling for GDP per capita. From the initial 28 items, we identified 23 sufficiently reliable items, but many of these were found to have low predictive power in the validity analysis. Items about air and water pollution were valid predictors of objective environmental conditions in these areas. Items asking about the relative importance of environmental problems compared to other social issues were also good positive predictors of progress on perceptible environmental issues. Items asking about general sentiment with no clear referent performed poorly. When controlling for GDP, country-level attitudes were more aligned with country-specific environmental conditions. Finally, nearly half of all EPI indicators were associated with few or no survey items, indicating the existence of ‘blind spots’ in public awareness. Our findings should offer guidance to both survey developers and users, as well as to policy makers responsible for conveying information about environmental sustainability to the wider public.

1. Introduction

The systemic degradation of the biosphere caused by human activities (air and water pollution, biodiversity decline, soil erosion, freshwater scarcity, ocean degradation, etc.) threatens humanity’s “safe operating space” [1]. While climate change is the most widely discussed aspect of the threat, several planetary boundaries associated with providing food and water along with ecological services that support life may already have been crossed [1,2]. Growing awareness of these problems [3] has led to discussions of policy responses conducted under the umbrella terms of ‘sustainability’ and ‘sustainable development’ [4]. The United Nations has established milestones at the international level by defining sustainable pathways for acting on climate change (Framework Convention on Climate Change in 1992, Kyoto Protocol in 1997 and the Paris Agreement in 2015), biodiversity (Convention on Biological Diversity in 1993, Nagoya protocol with the Aichi Biodiversity targets in 2010), and sustainable development (Bruntland report in 1987, Copenhagen Declaration on Social Development in 1995, the Millennium Development Goals (MDGs) in 2000, the Sustainable Development Goals (SDGs) in 2016). In addition, various international, national, regional, and local organizations have met regularly over the last decades to propose, debate, and implement frameworks, action plans, policies, and financing mechanisms to support sustainable goals [5].

Yet, progress in reaching sustainable targets has been incremental and uneven [6,7,8,9], and degradation of the planet’s ecosystems has intensified in recent decades [8,10,11]. Such lagging outcomes indicate the presence of considerable barriers to achieving sustainability, likely related to the massive structural changes required across all sectors of society [5,7,10,12]. Achieving sustainability will require extensive transformations in resource use, infrastructure, institutions, technologies, and social relations around the world within a short period, with profound, long-term implications for the countries involved [5,12].

The success of such transformations will likely require the engagement and support of an informed and committed public opinion [5,10,13], but we still lack a sufficient understanding of civil society attitudes and levels of support for sustainability initiatives, particularly at an international scale [5]. Valid and reliable measures of attitudes about sustainability are critically needed.

To address this need, this paper focuses on two widely used international surveys: the World Values Survey (WVS) and the International Social Survey Program (ISSP). These two surveys were selected because they are designed to be representative and international in scope and because they ask questions about the environment on a recurring basis and make the results publicly available. Taken together, they cover a wide range of countries around the world, and they have been widely used by researchers seeking to understand global or country-level trends in attitudes over time [14,15,16,17,18,19], along with the determinants and consequences of such trends [20,21]. Cross-national comparisons made possible by such international surveys are critical for identifying dynamics and causal relationships that are not visible when looking only at a single country. Thus, for example, one comparative study found that environmental concern and environmental behavioral intentions were strongly associated, but only in wealthier countries [19]. Another found that environmental concern is associated with more support for environmental taxes but only in countries with high governance quality [20]. It is clear, then, that comparative studies using international survey data are of great potential value.

However, the quality of the inferences drawn from such studies depends on the reliability and validity of the measures of public opinion used. Despite their widespread use, little information is available on the validity and reliability of social surveys. Validity refers to the extent to which items actually measure the underlying attitude or trait they were intended to measure or, put slightly differently, the extent to which evidence supports our interpretations of survey results [22]. Reliability is related to the precision and stability of measures, and reliable measurement is a necessary precondition for valid measurement [23]. Researchers have suggested that the assumption of validity is highly problematic for international surveys, given that respondents from different countries may interpret the same items differently [24,25].

To assess the reliability and validity of items from the WVS and ISSP surveys, we adopt the approach used in an earlier study of environmental surveys [26]. As measurements that are not reliable cannot be valid [23], we first focus on the reliability or stability of items by evaluating the correlations between similar or identical items that appear on surveys administered at different times. If we find that responses or rankings of country-level scores are significantly correlated, we can feel more confident about the reliability of such items. Next, we investigate validity. One source of validity evidence comes from the relationship between a variable and its relation to other variables with which we expect it to be related on theoretical grounds [27]. Here, we assess the relationships between social survey items and objective measures of environmental conditions. If items are associated with outcomes in theoretically predicted ways, then we can feel more confident that the items are measuring what we believe they are measuring.

To assess validity in this way, appropriate measures of the state of the environment are required. Although the UN offers a broad set of indicators to measure SDG progress, these metrics have been criticized for being incomplete [28], overemphasizing socioeconomic development, and inadequately measuring the planet’s objective environmental conditions [7,29]. Therefore, in this study, we have instead selected the Environmental Performance Index [9,30,31], the most comprehensive and one of the oldest international indexes. Designed to measure progress towards the UN Millennial Goals and subsequently the Sustainable Development Goals using a variety of concrete indicators, the EPI is global in scope and has been released every other year since 2007. In this study, we seek to investigate the validity of items from the WVS and ISSP surveys by looking for theoretically predicted associations between these surveys and EPI indicators.

To arrive at the predicted associations, we start with two broad hypotheses concerning the associations we should expect. First, the public may respond to perceptions of poor environmental conditions by showing high levels of concern about the environment. This so-called ‘degradation hypothesis’ has been supported by the findings of several studies [32,33,34]. In the current study, the degradation hypothesis suggests that survey respondents should express higher concern about the environment in countries making weaker progress towards objective sustainability targets. Concern would then reflect an accurate appraisal of the relatively poor environmental conditions in their country. Alternatively, it is also possible that, in countries with high environmental concern levels, the government might more actively pursue environmental protection measures, leading to better environmental outcomes. We refer to this as the ‘policy impact hypothesis’. At least for democratic countries, it has found considerable support from research. Differences in public opinion have been found to explain differences in observed environmental outcomes [35], and democracies are more likely to commit to environmental policies than non-democracies [36,37,38]. Further, countries whose citizens have pro-environmental attitudes tend to have more environmentally friendly policies [39,40,41] and vice-versa [42]. Additionally, Agnone [43] and Vandeweerdt et al. [44] found that politicians with environmentally concerned constituents tend to vote more pro-environmentally. Finally, it should also be noted that fossil fuel interests have behaved as if public opinion can influence policy, spending large sums on disinformation campaigns [8,12]. In this study, the policy impact hypothesis suggests that, in countries where concern is high, this popular concern should result in strong progress towards sustainability.

To summarize, negative correlations between items and environmental outcomes would be consistent with the degradation hypothesis, while positive correlations would be consistent with the policy impact hypothesis. Both would offer support for the validity of the survey items. In contrast, findings of no correlation would represent a failure to find validating evidence for the item. Of course, the item could still be a valid measure of some underlying subjective attitude towards the environment. However, it would still lack theoretical relevance if the attitude were not observably related in any way to environmental conditions.

Research has shown that both survey items [18,32,45,46] and EPI indicators [9,30,31] are correlated with economic development. We thus expect economic development to serve as a confound, potentially obscuring the relationships between survey items and environmental indicators. Therefore, we explore the relationship between economic development, as measured by GDP per capita, and our two sets of variables of interest by conducting our validity analysis twice: first, by directly looking at the correlations between survey items and environmental indicators, and second, through a partial correlation analysis controlling for per capita GDP.

Most discussions of sustainability have set targets ranging from 10 to 30 years. For example, the Aichi Biodiversity targets, initially set for 2020, were reset for 2030 and 2050 by the Kunming Declaration of 2020 [47], and the targets for achieving the goals of the Paris agreement and UN SDGs were set according to the 2030 Agenda [48]. This suggests the analytical strategy of working backward from this timeframe [5] to identify survey items that best predict current environmental conditions. As the best available longitudinal dataset of environmental sustainability indicators covers the 2007–2016 period [30,31], we selected surveys administered within this period. These surveys will be continually administered in the future, so they should be able to provide continued information about public attitudes as we move forward in the sustainability transition.

In this study, we ask two broad research questions:

• How reliable are survey questions appearing across more than one international survey? That is, how consistent are results across surveys?
• Do survey items offer valid measures of public opinion towards environmental sustainability? That is, do they actually measure what they are supposed to measure?

2. Materials and Methods

2.1. Materials

Social Surveys. We use the responses to questions about the environment on the World Values Survey Wave 5 from 2006 [49], the International Social Survey Program 2010 survey [50], and the World Values Survey Wave 6 from 2012 [51]. The years given are those in which the largest number of participants responded to the survey; see Supplementary File S1 Section S1 for a more detailed breakdown.

Environmental Sustainability Indicators. As EPI datasets from different years are not directly comparable due to changes in the particular indicators used, we selected the EPI backcasted dataset for the 2007 to 2016 period [30,31]. We removed all higher-level composite indicators and retained 22 indicators measuring aspects of risks to environmental health and ecosystem vitality. Environmental health indicators measure environmental risks to human health and focus on air and water quality. Ecosystem vitality, in turn, concentrates on the long-term sustainability of the environment with an emphasis on terrestrial and aquatic biodiversity, climate change and the ability of the ecosystems to provide ecosystem services. (One indicator, ‘ACCESS’, was not used in this analysis; it refers to access to sustainable energy and is not a direct measure of environmental health or ecosystem vitality.) All EPI indicator scores range from 0 to 100 based on how close each country is to an identified policy target, with 0 being the farthest from and 100 being the closest to the target. The targets are drawn from national or international policy goals [9,30,31]. These indicators, their location within the EPI framework and their relationship to the UN Sustainable Development Goals (SDGs) are shown in

Table 1. Environmental Performance Index categories and their relations to the UN SDGs.

EPI Categories	EPI Indicator	Corresponding SDGs	Overlapping SDGs
Air quality	HAP (H) HAPR (H) PM25 (H) PM25R (H) PM25EXBL (H) NO2 (H)	3—Healthy lives and well being	6—Sustainable water and sanitation
Water and Sanitation Drinking Water Quality and Access to Sanitation Wastewater Treatment	ACSAT (H) ACSATR (H) WATQ (H) WATSUP (H) WASTECXN (V)	6—Sustainable water and sanitation	3—Healthy lives and well being 14—Sustainable ocean, seas, marine resources
Trend in CO2 per kWh Trend in Carbon Intensity	CO2NEW (V) CO2KWHd1 (V)	7—Sustainable energy 13—Combat climate change	11—Sustainable Cities and Communities 12—Responsible Consumption and Production
Nitrogen Use Nitrogen Balance	NUE (V) NBALANCE (V)	2—Food security and sustainable agriculture	3—Healthy lives and well being
Fish Stocks Marine Protected Areas	FSCOPEN (V) MPAEEZ (V)	14—Sustainable ocean, seas, marine resources	2—Food security and sustainable agriculture 11—Sustainable cities and Communities
Terrestrial Habitat Protection Species Protection	PACOVD (V) PACOVW (V) FORCH (V) PSPU (V) PSPW (V)	15 - Life on land	3—Healthy lives and well being 11—Sustainable cities and Communities 13—Combat climate change

Notes: For the EPI Indicators, ‘H’ = Health Impacts of Environmental Conditions; ‘V’ = Ecosystem Viability. One indicator, ‘ACCESS’, has been removed as it is not a direct measure of environmental health or viability.

.

The international representativeness of our three social surveys and the Environmental Performance Index is indicated in

Table 2. Representation of countries at different levels of development.

World Bank Income Classification *	No. (%) of Countries *	Percent of 2016 Global Pop. *	Percent of 2016 Global GDP *	EPI 2016 Countries	WVS 5 Countries	ISSP 2010 Countries	WVS 6 Countries
Low	31 (14%)	8.0%	0.6%	29 (16%)	4 (7%)	0 (0%)	3 (5%)
Lower Middle	53 (24%)	41.3%	8.3%	50 (28%)	12 (21%)	1 (3%)	16 (27%)
Upper Middle	56 (26%)	34.5%	26.9%	50 (28%)	15 (26%)	7 (19%)	20 (33%)
High	78 (36%)	16.2%	64.3%	51 (28%)	26 (46%)	28 (78%)	21 (35%)
	218	100%	100%	180	57	36	60

* Sources: [31,49,50,51,52,53].

, which shows the distribution of participating countries by level of economic development. The EPI covers nearly all countries. The two WVS surveys cover a wide range of countries and are highly representative in terms of economic development. The ISSP covers a more homogeneous sample of relatively wealthy countries, with 97.1% of countries covered falling in either the ‘upper middle’ or ‘high’ income category.

2.2. Reliability Analysis

For each survey, we selected all items related to the environment that were similar or identical across surveys. This resulted in a total pool of 28 items, with 13 items from the WVS Wave 5 Survey, 10 items from the ISSP 2010 Survey, and 5 items from the WVS Wave 6 Survey. We recoded Likert-scale items as necessary so that higher numbers indicated higher levels of environmental concern or support for environmental sustainability. For each item, we calculated the country-level mean response for each country. As the resulting mean scores are based on Likert-scale items and cannot be considered continuous variables, we used Spearman rank order correlations in our analysis. We calculated the Spearman’s rho coefficients for each item and its matching item or items on the other two surveys. If the resulting correlation was significant, the item was considered sufficiently reliable and retained for the second part of the analysis.

2.3. Validity

To investigate the validity of the items found to be reliable in the first stage of the analysis, we examined correlations between country-level mean responses to these survey items and country scores on 22 indicators from the 2016 Environmental Performance Index [30,31].

For survey items related to air quality, water quality, and global warming, we assessed correlations with EPI indicators specifically related to these issues. The remaining survey items, however, were not issue-specific. Therefore, we evaluated them against all EPI indicators.

We performed our analysis twice. First, we calculated the Spearman rank correlations between country mean scores on the opinion survey items and the country scores on the 22 EPI indicators. Second, to control for the potential confounding effects of the level of economic development, we repeated the analysis using Spearman partial correlations with GDP per capita as the control variable (for correlations between GDP per capita and the variables used in this study, see Supplementary File S1 Section S2).

All significant correlations shown in the paper appear in bold font. Significant correlations below 0.4 are described as ‘weak’ (highlighted in light green for positive or light red for negative correlations); significant correlations of 0.4 and above but below 0.7 are ‘moderate’ (highlighted in moderate shades of green or red); and significant correlations of 0.7 and above are ‘strong’ (highlighted in darker green or red).

This study uses only 2016 EPI data, the most recent data available in the longitudinal dataset. In preliminary analyses, we also used other years from the EPI 2007–2016 dataset, but these did not fundamentally alter our findings. We provide results using EPI 2011 data in the Supplementary File S1 Section S3.

Recoding of variables was conducted in R [54]. Analyses were performed using IBM SPSS Statistics, Version 20. The R code used for variable recoding and the SPSS syntax used for the partial correlation analyses are provided in the Supplementary File S1 Sections S4 and S5.

3. Results

3.1. Item Reliability

We first present the selected items for eight item groups, aggregated by theme or topic. The variable names used in this paper, the original questions, and their original item number are provided in

Table 3. ISSP and WVS survey items measuring attitudes about the environment.

Variable Name	Original Question	Original Item Number
Air Quality
AirPltnCars (I)	In general, do you think that air pollution caused by cars is… (Response options for 14a and 14b: 1 Extremely dangerous for the environment, 2 Very dangerous, 3 Somewhat dangerous, 4 Not very dangerous, 5 Not dangerous at all for the environment; recoded so that 1 = Not dangerous at all for the environment, etc.)	ISSP 14a
AirPltnIndus (I)	In general, do you think that air pollution caused by industry is…	ISSP 14b
PoorAir L (W5)	“I am going to read out a list of environmental problems facing many communities. Please, tell me how serious you consider each one to be here in your own community. Is it very serious, somewhat serious, not very serious or not serious at all? Poor air quality.” (Response options: 1 Very serious, 2 Somewhat serious, 3 Not very serious, 4 Not serious at all. Recoded so that 1 = Not serious at all, etc.)	WVS 5 B019
Water Quality
Wtr C (I)	And do you think that pollution of COUNTRY’S rivers, lakes and streams is... (Response options: 1 Extremely dangerous for the environment, 2 Very dangerous, 3 Somewhat dangerous, 4 Not very dangerous, 5 Not dangerous at all for the environment; recoded so that 1 = Not dangerous at all for the environment, etc.)	ISSP 14d
PoorWtr L (W5)	[Same prelude as for B019 above.] “Poor water quality.” (Response options: 1 Very serious, 2 Somewhat serious, 3 Not very serious, 4 Not serious at all. Recoded so that 1 = Not serious at all, etc.)	WVS 5 B018
PoorSwg L (W5)	[Same prelude as for B019 above.] “Poor sewage and sanitation.” (Response options: 1 Very serious, 2 Somewhat serious, 3 Not very serious, 4 Not serious at all. Recoded so that 1 = Not serious at all, etc.)	WVS 5 B020
WtrPlltn W (W5)	“Now let’s consider environmental problems in the world as a whole. Please, tell me how serious you consider each of the following to be for the world as a whole. Is it very serious, somewhat serious, not very serious or not serious at all? Pollution of rivers, lakes, and oceans. (Response options: 1 Very serious, 2 Somewhat serious, 3 Not very serious, 4 Not serious at all. Recoded so that 1 = Not serious at all, etc.)	WVS 5 B023
Climate change
GW (I)	In general, do you think that a rise in the world’s temperature caused by climate change is... (Response options: 1 Extremely dangerous for the environment, 2 Very dangerous, 3 Somewhat dangerous, 4 Not very dangerous, 5 Not dangerous at all for the environment; recoded so that 1 = Not dangerous at all for the environment, etc.)	ISSP 14e
GW (W5)	“I am going to read out a list of environmental problems facing many communities. Please, tell me how serious you consider each one to be here in your own community. Is it very serious, somewhat serious, not very serious or not serious at all? Global warming or the greenhouse effect.” (Response options: 1 Very serious, 2 Somewhat serious, 3 Not very serious, 4 Not serious at all. Recoded so that 1 = Not serious at all, etc.)	WVS 5 B021
Environmental Movement
Conf EnvMv (W5)	I am going to name a number of organizations. For each one, could you tell me how much confidence you have in them: is it a great deal of confidence, quite a lot of confidence, not very much confidence or none at all? The Environmental Protection Movement. (Response options: 1 A great deal, 2 Quite a lot, 3 Not very much, 4 None at all. Reverse coded so that 1 = None at all, etc.)	WVS 5 E069_14
Conf EnvMv (W6)	Same as Conf EnvMv (W5) above.	WVS 6 E069_14
Mmbr EnvMv (I)	Are you a member of any group whose main aim is to preserve or protect the environment? (Response options: 1 Yes, 2 No; recoded so that “No” = 0 and “Yes” = 1).	ISSP 21
Mmbr EnvMv (W5)	Now I am going to read out a list of voluntary organizations; for each one, could you tell me whether you are a member, an active member, an inactive member or not a member of that type of organization? Environmental organization (Response options: 0 = Not a member, 1 = Inactive member, 2 = Active member)	WVS 5 A103
Mmbr EnvMv (W6)	Same as Mmbr EnvMv (W5) above.	WVS 6 A103
Environment versus Economy
Env vs. Econ (W5)	Here are two statements people sometimes make when discussing the environment and economic growth. Which of them comes closer to your own point of view? A. Protecting the environment should be given priority, even if it causes slower economic growth and some loss of jobs. B. Economic growth and creating jobs should be the top priority, even if the environment suffers to some extent. (Response options: 1 Protecting environment, 2 Economy growth and creating jobs; recoded so that ‘A’ = 1 and ‘B’ = 0).	WVS 5 & 6 B008
Env vs. Econ (W6)	Same as for Env vs Econ (W5) above.	WVS 5 & 6 B008
Env vs. Econ (I)	In order to protect the environment [COUNTRY] needs economic growth (Response options: 1 Agree strongly, 2 Agree, 3 Neither agree nor disagree, 4 disagree, 5 Disagree strongly; reverse coded so that 1 = Disagree strongly, etc.)	ISSP 11A
Importance of Caring for Environment
Concern (I)	Generally speaking, how concerned are you about environmental issues? (Response options: 1 Not at all concerned, 2, 3, 4, 5 Very concerned).	ISSP 6
Imprt CareEnv (W5)	Now I will briefly describe some people. Using this card, would you please indicate for each description whether that person is very much like you, like you, somewhat like you, not like you, or not at all like you? It is important to this person looking after the environment. (Reverse coded to: 1 Not at all like me, 2 Not like me, 3 A little like me, 4 Somewhat like me, 5 Like me, 6 Very much like me)	WVS 5 A197
Imprt CareEnv (W6)	Same as Imprt CareEnv (W5) above.	WVS 6 A197
Relative Salience of Environmental Issues
Salience C (I)	Which of these issues is the most important for [COUNTRY] today? Which is the next most important? (Response options: 1 Health care, 2 Education, 3 Crime, 4 The environment, 5 Immigration, 6 The economy, 7 Terrorism, 8 Poverty, 9 None of these. These two items were recoded so that listing the environment as the first choice was scored as ‘2’, as the second choice was scored as ‘1’ and not at all was scored as ‘0’.))	ISSP 1a ISSP 1b
Salience C (W5)]	Which of these problems do you consider the most serious one in your own country? And which is the next most serious in your own country? (Response options: 1 People living in poverty and need, 2 Discrimination against girls and women, 3 Poor sanitation and infectious diseases, 4 Inadequate education, 5 Environmental pollution. These two items were recoded so that listing the environment as the first choice was scored as ‘2’, as the second choice was scored as ‘1’ and not at all was scored as ‘0’.)	WVS 5 E240 WVS 5 E241
Salience W (W5)	Please indicate which of the following problems you consider the most serious one for the world as a whole? (Response options: 1 People living in poverty and need, 2 Discrimination against girls and women, 3 Poor sanitation and infectious diseases, 4 Inadequate education, 5 Environmental pollution). This item was recoded so that listing the environment as the first choice was scored as ‘1’, and not doing so was scored as ‘0’.))	WVS 5 E238
Salience W (W6)	Same as Salience W (W5) above.	WVS 6 E238
Willingness to Make Financial Sacrifices for the Environment
WTP Taxes (W5)	I am now going to read out some statements about the environment. For each one read out, can you tell me whether you agree strongly, agree, disagree, or strongly disagree? Increase in taxes if used to prevent environmental pollution. (Response options: 1 Strongly agree, 2 Agree, 3 Disagree, 4 Strongly disagree; reverse coded so that 1 = Strongly disagree, etc.)	WVS 5 B002
WTP Higher Taxes (I)	And how willing would you be to pay much higher taxes in order to protect the environment? (Response options: 1 Very willing, 2 Fairly willing, 3 Neither willing nor unwilling, 4 Fairly unwilling, 5 Very unwilling; reverse coded so that 1 = Very unwilling, etc.)	ISSP 12b
WTP Inc (W5)	I am now going to read out some statements about the environment. For each one read out, can you tell me whether you agree strongly, agree, disagree, or strongly disagree? Would give part of my income for the environment. (Response options: 1 Strongly agree, 2 Agree, 3 Disagree, 4 Strongly disagree)	B001/WVS 5
WTP Higher Prices (I)	How willing would you be to pay much higher prices in order to protect the environment? (Response options: 1 Very willing, 2 Fairly willing, 3 Neither willing nor unwilling, 4 Fairly unwilling, 5 Very unwilling; reverse coded so that 1 = Very unwilling, etc.)	ISSP 12a

.

Results appear in

Table 4. Reliability analysis of the selected environmental concern items.

	Item 1	Item 2	Number of Countries	Spearman’s rho	p
Air Quality	AirPltnCars (I)	PoorAir L (W5)	18	0.711	0.001
	AirPltnIndus (I)	PoorAir L (W5)	18	0.550	0.018
Water Quality	PoorSwg L (W5)	Wtr C (I)	18	0.723	0.002
	PoorWtr L (W5)	Wtr C (I)	18	0.730	0.001
	WtrPlltn W (W5)	Wtr C (I)	19	0.679	0.001
Climate change	GW (I)	GW (W5)	19	0.568	0.011
Environmental Movement	Conf EnvMv (W5)	Conf EnvMv (W6)	37	0.470	0.003
	Mmbr EnvMv (W5)	Mmbr EnvMv (W6)	37	0.669	<0.001
	Mmbr EnvMv (W5)	Mmbr EnvMv (I)	24	0.789	<0.001
	Mmbr EnvMv (I)	Mmbr EnvMv (W6)	18	0.707	0.001
Environment versus Economy	Env vs Econ (I)	Env vs Econ (W5)	23	0.547	0.007
	Env vs Econ (I)	Env vs Econ (W6)	18	0.243	0.332
	Env vs Econ (W5)	Env vs Econ (W6)	36	0.455	0.005
Importance of Caring for the Environment	Concern (I)	Imprt CareEnv (W5)	22	0.104	0.644
	Concern (I)	Imprt CareEnv (W6)	18	−0.019	0.448
	Imprt CareEnv (W5)	Imprt CareEnv (W6)	34	0.773	<0.001
Salience of Environmental Issues	Salience C (W5)	Salience C (I)	18	0.783	<0.001
	Salience W (W5)	Salience W (W6)	30	0.653	<0.001
Willingness to Make Financial Sacrifices for the Environment	WTP Taxes (W5)	WTP Higher Taxes (I)	20	0.311	0.182
	WTP Inc (W5)	WTP Higher Prices (I)	20	−0.102	0.668

. Items asking about concern for air quality, water quality, climate change, confidence and membership in the environmental movement, environment versus the economy, care for the environment, and salience of environmental issues showed moderate to high significant positive correlations across surveys. Therefore, all of these items were retained for further analysis.

Five items were not significantly correlated with their matching items, indicating a lack of reliability. The ‘general concern’ item on ISSP 2010 was not correlated with the “importance of looking after the environment” items on WVS 5 and 6. None of the items measuring willingness to make financial sacrifices (higher taxes, income donation, and higher prices) showed significant correlations across surveys and years. These five items were not included in the second part of the analysis.

To conclude, 82% of items analyzed (23 out of the initial 28) were found to be sufficiently reliable. We computed Spearman correlations for 20 pairs of similar or identical survey questions. In five pairs, the items were identically worded, while in the remaining fifteen pairs, the items were worded somewhat differently. Significant correlations, indicating reliability, were found for all five pairs of identically worded items. Significant correlations were found for ten of the fifteen pairs of similarly worded items, but not for the remaining five. This underlines the importance of identically worded items for tracking attitudes over time. The remaining 23 items cover seven topics: air and water quality, climate change, environmental movement, environment versus economy, caring for the environment, and salience of environmental problems.

3.2. Validity

For the first three item sets, related to air quality, water quality and global warming, we assessed correlations with indicators related to these issues. The remaining survey items, which are not issue-specific, were evaluated against all EPI indicators. Given the strength of the correlations between GDP per capita and several EPI indicators and survey items (see Supplementary File S1 Section S2), GDP per capita is likely to be a confounding variable, potentially distorting the relationships between survey items and EPI indicators. For this reason, the correlation analyses were conducted twice, with and without controlling for GDP per capita.

3.2.1. Air Quality

Weak to moderate negative correlations were found between survey items about air quality and HAP and HAPR, both of which refer to indoor air pollution from the burning of solid fuels (

Table 5. Item-indicator correlations: Air quality (Spearman’s rho).

	PoorAirL (W5)	AirPltnCars (I)	AirPltnIndus (I)
HAP	−0.243	−0.535 **	−0.433 **
HAPR	−0.321 *	−0.582 **	−0.606 **
NO2	0.395 **	0.315	0.129
PM25	0.174	−0.038	−0.185
PM25EXBL	0.104	−0.039	−0.195
PM25R	−0.335 *	−0.166	−0.352 *
CONTROLLING FOR GDP PER CAPITA	PoorAirL (W5)	AirPltnCars (I)	AirPltnIndus (I)
HAP	0.363 *	0.058	0.185
HAPR	0.298 *	0.039	−0.112
NO2	−0.113	−0.620 ***	−0.876 ***
PM25	0.192	−0.108	−0.311
PM25EXBL	0.133	−0.064	−0.286
PM25R	−0.136	0.232	−0.117

Notes: ‘ns’ = not significant; * p <= 0.05, ** p <= 0.01, *** p <= 0.001.Indicator definitions: HAP = Percentage of the population burning solid fuel (biomass such as wood, crop residues, dung, charcoal, and coal); HAPR = health risk posed by household air pollution; NO2 = Average exposure to NO₂, in ppb; PM25 = Population-weighted exposure to PM_2.5 in micrograms per cubic meter; PM25R = Health Risk Exposure to PM_2.5; PM25EXBL = Population-weighted exposure to PM_2.5 in micrograms per cubic meter. All indicators were rescaled to 0–100 based on proximity to the target.

). This finding is consistent with the degradation hypothesis. The remaining air quality indicators (PM25, PM25R, PM25EX and NO2) showed no significant correlations with the survey items. Difficulties in reducing air pollution from PM_2.5 and NO_x [9,30,31] and low perception of these invisible pollutants might explain this lack of correlations. In addition, the fact that these EPI indicators are measured by satellite and averaged for each country might also result in distorted measurements.

Particulate matter smaller than 2.5 microns (PM_2.5) and nitrogen dioxide (NO₂) are mainly produced by the burning of fossil fuels. Processes associated with urbanization and industrialization continue to emit life-threatening levels of air pollutants in both developing and developed countries, and poor air quality remains a critical public health issue globally [9]. High NO₂ levels are characteristic of more developed countries, and NO₂ is both less perceptible and less frequently discussed in the media. This may explain the positive correlation: lower-income countries tend to have higher mean levels of concern about the dangers of air pollution but do not suffer from high NO₂ levels. In more developed countries where it is a problem, it may not result in higher survey scores because it is less salient.

After controlling for GDP per capita, none of the previous correlations remain. Two weak positive correlations indicate that higher levels of concern for local air quality in the WVS5 survey are associated with healthier levels of household air pollution. There were two moderate-to-strong negative correlations between NO₂ levels and levels of concern due to industrial or car pollution in the ISSP survey. Among the relatively wealthier countries in the 2010 ISSP, the countries the most concerned with air pollution had poorer air quality results, consistent with the degradation hypothesis. This is possibly due to lagging progress in controlling NO₂ levels in cities with heavy traffic congestion in Argentina, Mexico, Russia, and Turkey, or with intense export-oriented manufacturing in countries such as South Korea, Taiwan, and Japan.

3.2.2. Water Quality

Turning to water quality, we found a series of negative, significant correlations of moderate strength between the perception of local- or country-level water quality and EPI water quality indicators. This suggests that people in less developed countries are responding to highly salient problems with water quality (

Table 6. Item-indicator correlations: Water quality (Spearman’s rho).

	PoorWtrL (W5)	PoorSwgL (W5)	WtrPlltnW (W5)	WtrC (I)
ACSAT	−0.554 **	−0.580 **	0.284	−0.210
ACSATR	−0.648 **	−0.660 **	0.176	−0.467 **
WASTECXN	−0.583 **	−0.583 **	0.258	−0.473 **
WATQ	−0.540 **	−0.530 **	0.310 *	−0.386 *
WATSUP	−0.683 **	−0.682 **	0.247	−0.498 **
CONTROLLING FOR GDP PER CAPITA	PoorWtrL (W5)	PoorSwgL (W5)	WtrPlltnW (W5)	WtrC (I)
ACSAT	0.137	0.069	0.097	0.705 ***
ACSATR	0.074	0.034	−0.193	0.329
WASTECXN	0.123	0.123	0.038	0.161
WATQ	0.126	0.149	0.147	0.262
WATSUP	−0.193	−0.19	0.029	0.059

Notes: ‘ns’ = not significant; * p <= 0.05, ** p <= 0.01, *** p <= 0.001. Indicator definitions: ACSAT = % of population with access to improved sanitation; ACSATR = health risk posed by unsafe sanitation; WASTCXN = percent of wastewater that is treated; WATQ = health risk of exposure to unsafe drinking water; WATSUP = proportion of population with access to improved drinking water source.

; Figure 1). In contrast, the item about global water pollution shows only one weak, positive significant correlation, possibly suggesting a more abstract level of concern in wealthier countries with issues for which the respondents presumably have little personal experience.

After controlling for GDP per capita, none of the previous correlations remain. Instead, a single robust and positive relationship shows that the more serious the concern for pollution of one’s country’s rivers, lakes, and streams among the 2010 ISSP countries, the better the access to improved sanitation in 2016. This relationship only holds for one water quality indicator, showing that survey questions about water pollution have little predictive power after controlling for economic development.

3.2.3. Climate Change

Correlations between survey items asking about climate change result in only one moderate negative correlation with an indicator of progress on improving CO₂ emissions (

Table 7. Item-indicator correlations: Climate change (Spearman’s rho).

	GW (W5)	GW (I)
CO2NEW	−0.134	−0.164
CO2KWHd1	−0.124	−0.488 **
CONTROLLING FOR GDP PER CAPITA
	GW (W5)	GW (I)
CO2NEW	−0.190	−0.047
CO2KWHd1	−0.097	−0.774 ***

Notes: ‘ns’ = not significant; ** p <= 0.01, *** p <= 0.001. Indicator definitions: CO2NEW = measures ability to reduce the intensity of carbon emissions per unit GDP from 2002 to 2012, relative to a country’s economic peers; CO2KWHd1 = Trend in CO2 Emissions per kilowatt hour (kWh) of electricity produced, determined for most countries as a trend from 2002 to 2012.

). After controlling for GDP per capita, we identified the same relationship, with an increase in strength and significance. Mediterranean countries (Spain, Croatia, Turkey, etc.) or countries exposed to drought and heatwaves (Chile) showed both higher levels of concern and lower levels of progress for their CO₂ emissions per kWh. More developed countries that can afford to take measures against climate change have relatively better performance but also show less concern, perhaps because their relative wealth will keep them more insulated from climate change impacts. More broadly, it suggests that survey questions about climate change have little predictive power.

As the remaining survey items were not issue-specific, we assessed their correlations with all EPI indicators. Only EPI indicators with at least one correlation are included in the tables; indicators not in a table were not significantly correlated with any of that table’s survey items.

3.2.4. Environmental Movement

Membership and confidence in the environmental movement were weakly correlated (with only one moderate correlation) with ten EPI indicators (

Table 8. Item-indicator correlations: Environment movement vs. All EPI (Spearman’s rho).

	Mmbr_EnvMv (W5)	Conf_EnvMv (W5)	Mmbr_EnvMv (I)	Mmbr_EnvMv (W6)	Conf_EnvMv (W6)
ACSATR	−0.169	0.074	0.392 *	0.099	0.118
FORCH	−0.360 *	−0.371 **	−0.037	−0.232	−0.295
HAP	−0.348 **	−0.017	0.015	−0.221	−0.135
HAPR	−0.401 **	−0.193	−0.03	−0.203	−0.137
PACOVD	−0.048	−0.082	−0.184	0.141	0.286 *
PACOVW	0.009	0.019	−0.164	0.153	0.298 *
PSPW	0.088	−0.063	0.007	0.268 *	0.322 *
WATQ	−0.373 **	−0.124	0.242	−0.075	0.096
WATSUP	−0.128	0.004	0.368 *	0.028	0.203
CONTROLLING FOR GDP PER CAPITA
	Mmbr_EnvMv (W5)	Conf_EnvMv (W5)	Mmbr_EnvMv (I)	Mmbr_EnvMv (W6)	Conf_EnvMv (W6)
ACSAT	−0.319 *	−0.101	−0.361 *	−0.236	−0.101
HAP	−0.423 ***	−0.053	−0.601 ***	−0.527 ***	−0.432 ***
HAPR	−0.533 ***	−0.329**	−0.777 ***	−0.556 ***	−0.486 ***
NO2	0.291 *	−0.065	0.625 ***	0.374 **	0.112
PACOVD	−0.027	−0.095	−0.424 *	0.086	0.234
PACOVW	0.033	0.011	−0.401 *	0.099	0.247
WATQ	−0.525 ***	−0.242	−0.347 *	−0.396 **	−0.143

Notes: * p <= 0.05, ** p <= 0.01, *** p <= 0.001. Indicator definitions: ACSAT = Percentage of population with access to improved sanitation; ACSATR = health risk posed by unsafe sanitation; FORCH = Forest loss of >30% tree cover, as compared to 2000 levels; HAP = Percentage of the population burning solid fuel (biomass such as wood, crop residues, dung, charcoal and coal); HAPR = health risk posed by household air pollution; NO2 = Average exposure to NO₂, in ppb; PACOVD = Percentage of terrestrial biome area that is protected, weighted by domestic biome area; PACOVW = Percentage of terrestrial biome area that is protected, weighted by domestic biome area; PSPW = Proportion of a species’ global habitat represented within a country’s protected areas (global stewardship weight); WATQ = Health risk from exposure to unsafe drinking water; WATSUP = Proportion of a country’s total population with access to improved drinking water.

). The correlations do not persist across survey administrations, and no strong interpretations appear warranted.

In contrast, partial correlations controlling for GDP per capita show that, at a similar level of economic development, countries making poor progress on air and water quality issues had more people reporting being members of an environmental organization. This is consistent with the degradation hypothesis. Questions about confidence in the environmental movement showed weak predictive power overall.

3.2.5. Environment Versus the Economy

Items about prioritization of the environment versus the economy correlated with ten EPI indicators (

Table 9. Item indicator correlations: Environment/economy vs. All EPI (Spearman’s rho).

	Env_vs_Econ (W5)	Env_vs_Econ (I)	Env_vs_Econ (W6)
ACSAT	0.192	0.408 *	0.069
ACSATR	0.223	0.620 **	0.053
HAP	0.123	0.376 *	0.019
PM25	0.274 *	0.068	0.063
PM25EXBL	0.321 *	0.162	0.038
PM25R	0.355 **	0.208	0.256
WASTECXN	0.209	0.514 **	−0.045
WATQ	0.212	0.445 **	0.049
WATSUP	0.282 *	0.600 **	0.198
NUE	−0.203	0.001	−0.270 *
CONTROLLING FOR GDP PER CAPITA
	Env_vs_Econ (W5)	Env_vs_Econ (I)	Env_vs_Econ (W6)
ACSAT	−0.114	−0.697 ***	−0.086
CO2KWHd1	−0.009	0.594 ***	−0.199
HAP	−0.146	−0.443 **	−0.119
HAPR	−0.209	−0.921 ***	−0.338 *
NO2	0.314 *	0.841 ***	0.051
PACOVD	−0.045	−0.450 **	0.048
PACOVW	−0.001	−0.436 **	0.058
PM25	0.297 *	0.147	0.067
PM25EXBL	0.334 *	0.254	0.037
PM25R	0.274 *	−0.153	0.067
PSPU	−0.134	−0.591 ***	−0.062
PSPW	−0.141	−0.503 **	−0.070
WASTECXN	−0.118	−0.537 ***	−0.331 *
WATQ	−0.063	−0.523 **	−0.111

Notes: ‘ns’ = not significant; * p <= 0.05, ** p <= 0.01, *** p <= 0.001. Indicator definitions: ACSAT = % of population with access to improved sanitation; ACSATR = health risk posed by unsafe sanitation; HAP = Percentage of the population burning solid fuel (biomass such as wood, crop residues, dung, charcoal and coal); HAPR = health risk posed by household air pollution; NO2 = Average exposure to NO₂, in ppb; PM25 = Population-weighted exposure to PM_2.5 in micrograms per cubic meter; PM25EXBL = Population-weighted exposure to PM_2.5 in micrograms per cubic meter; PM25R = Health Risk Exposure to PM_2.5; CO2KWHd1 = Trend in CO2 Emissions per kilowatt hour (kWh) of electricity produced, determined for most countries as a trend from 2002 to 2012; PACOVD = Percentage of terrestrial biome area that is protected, weighted by domestic biome area; PACOVW = Percentage of terrestrial biome area that is protected, weighted by domestic biome area; PSPU = Extent of a species’ range protected as a proportion of a country’s biome (national weight); PSPW = Proportion of a species’ global habitat represented within a country’s protected areas (global stewardship weight); WATQ = Health risk when individuals are exposed to unsafe drinking water; WATSUP = Proportion of a country’s total population with access to an improved drinking water source as a main source of drinking water; WASTECXN = % of collected, generated, or produced wastewater that is treated, normalized by the population connected to centralized wastewater treatment facilities, NUE = Nitrogen Use Efficiency.

), but the lack of stable findings across administrations suggests that responses to this item may fluctuate with economic conditions. For the ISSP 2010 results, the items were moderately and positively correlated with water and air quality indicators strongly associated with economic development. Participants in wealthier countries appear more likely to report prioritizing the environment.

Controlling for per capita GDP, partial correlations resulted in little change for the WVS items but strong reversals for the ISSP item. The negative correlations for this item are consistent with the degradation hypothesis and suggest that, controlling for the economic level, this ISSP item is a sensitive indicator.

3.2.6. Importance of Caring for the Environment

The items related to caring for the environment correlated with only 1 of 22 EPI indicators, and the correlations were weak (

Table 10. Item-indicator correlations: Caring for environment vs. All EPI (Spearman’s rho).

	Imprt_CareEnv (W5)	Imprt_CareEnv (W6)
WATSUP	−0.315 *	−0.283 *
CONTROLLING FOR GDP PER CAPITA
	Imprt_CareEnv (W5)	Imprt_CareEnv (W6)
NO2	0.209	0.261 *
NUE	−0.047	−0.279 *
WATSUP	−0.337 *	−0.387 **

Notes: ‘ns’ = not significant; * p <= 0.05, ** p <= 0.01. Indicator definitions: WATSUP = Proportion of a country’s total population with access to improved drinking water, NO2 = Average exposure to NO₂, in ppb; NUE = Nitrogen Use Efficiency.

). After controlling for GDP per capita, the same relationship was confirmed along with two others: a positive one with the exposure to nitrogen dioxide and a negative one with the nitrogen use efficiency. These correlations are both weak and marginally significant, suggesting that questions about caring for the environment have low predictive power.

3.2.7. Salience of Environmental Issues

Salience variables predict more EPI indicators than any of the other survey items in this study (

Table 11. Item-indicator correlations: Salience vs. All EPI (Spearman’s rho).

	Salience C (W5)	Salience W (W5)	Salience C (I)	Salience W (W6)
ACSAT	0.744 **	0.379 *	0.494 **	0.392 **
ACSATR	0.739 **	0.409 **	0.633 **	0.358 **
CO2NEW	0.147	0.105	0.080	0.287 *
HAP	0.553 **	0.303 *	0.392 *	0.312 *
HAPR	0.614 **	0.478 **	0.335 *	0.364 **
NO2	−0.627 **	−0.592 **	−0.385 *	−0.437 **
PM25	−0.200	−0.372 *	−0.132	−0.176
PM25EXBL	−0.147	−0.306 *	−0.081	−0.198
PSPU	0.313 *	0.121	0.028	0.151
PSPW	0.398 **	0.081	0.076	0.189
WASTECXN	0.630 **	0.558 **	0.428 **	0.303 *
WATQ	0.576 **	0.332 *	0.385 *	0.202
WATSUP	0.715 **	0.447 **	0.613 **	0.521 **
CONTROLLING FOR GDP PER CAPITA
	Salience C (W5)	Salience W (W5)	Salience C (I)	Salience W (W6)
ACSAT	0.384 **	0.068	−0.287	0.101
HAPR	0.144	0.270	−0.513 **	0.084
NO2	−0.209	−0.461 **	0.336 *	−0.216
PACOVD	0.099	−0.033	−0.441 **	0.040
PACOVW	0.108	−0.068	−0.404 *	0.086
PM25	−0.251	−0.393 **	−0.155	−0.177
PM25EXBL	−0.212	−0.337 *	−0.12	−0.217
PM25R	−0.209	−0.072	−0.348 *	0.030
PSPU	0.030	−0.059	−0.441 **	−0.021
PSPW	0.126	−0.128	−0.413 *	0.007
WATQ	−0.002	−0.005	−0.500 **	−0.235
WATSUP	0.306 *	0.199	0.017	0.348 **

Note: ‘ns’ = not significant; * p <= 0.05, ** p <= 0.01. Indicator definitions: ACSAT = Percentage of population with access to improved sanitation; ACSATR = health risk posed by unsafe sanitation; CO2KWHd1 = Trend in CO2 Emissions per kilowatt hour (kWh) of electricity produced, determined for most countries as a trend from 2002 to 2012; HAP = Percentage of the population burning solid fuel (biomass such as wood, crop residues, dung, charcoal and coal); HAPR = health risk posed by household air pollution; NO2 = Average exposure, in ppb; PM25 = Population-weighted exposure to PM_2.5 in micrograms per cubic meter; PM25EXBL = Population-weighted exposure to PM_2.5 in micrograms per cubic meter; PM25R = Health Risk Exposure to PM_2.5; PACOVD = Percentage of terrestrial biome area that is protected, weighted by domestic biome area; PACOVW = Percentage of terrestrial biome area that is protected, weighted by domestic biome area; PSPU = Extent of a species’ range protected as a proportion of a country’s biome (national weight); PSPW = Proportion of a species’ global habitat represented within a country’s protected areas (global stewardship weight); WATQ = Health risk from exposure to unsafe drinking water; WATSUP = Proportion of a country’s total population with access to improved drinking water; WASTCXN = Percentage of collected, generated, or produced wastewater that is treated, normalized by the population connected to centralized wastewater treatment facilities.

). Environmental salience—particularly country-level salience—is strongly associated with several indicators, and most correlations are positive (for an example, see Figure 2), suggesting that in higher-income countries, where salience scores are relatively high, most basic environmental health issues have been largely overcome. The three negative correlations that we see, for CO2KWHd1, PM25 and especially NO2, are related to environmental problems that are less perceptible and complex to resolve, even for developed economies, because they are directly tied to high levels of economic activity.

Partial correlations controlling for GDP per capita show fewer significant correlations and a dominance of negative correlations, especially within the more developed countries participating in the ISSP. These results suggest that in these countries, public opinion is quite sensitive to environmental degradation.

To conclude, as seen on the left-hand side of

Table 12. Summary Table: ISSP 2010 and WVS 5 and 6 survey items ranked by predictive power.

Bivariate Correlations		Partial Correlations Controlling for GDP Per Capita
Items	Percent (Number) of Significant Correlations with EPI Indicators	Items	Percent (Number) of Significant Correlations with EPI Indicators
ISSP		ISSP
Wtr C (I)	80% (4/5)	GW (I)	50% (1/2)
GW (I)	50% (1/2)	Env vs Econ (I)	50% (11/22)
AirPltnIndus (I)	50% (3/6)	Salience C (I)	36% (8/22)
Salience C (I)	36% (8/22)	Mmbr EnvMv (I)	32% (7/22)
AirPltnCars (I)	33% (2/6)	Wtr C (I)	20% (1/5)
Env vs Econ (I)	27% (6/22)	AirPltnIndus (I)	17% (1/6)
Mmbr EnvMv (I)	9% (2/22)	AirPltnCars (I)	17% (1/6)
WVS 5 & 6		WVS 5 & 6
PoorSwg L (W5)	100% (5/5)	PoorAir L (W5)	33% (2/6)
PoorWtr L (W5)	100% (5/5)	Mmbr EnvMv (W5)	23% (5/22)
Salience W (W5)	45% (10/22)	Env vs Econ (W5)	18% (4/22)
Salience C (W5)	45% (10/22)	Mmbr EnvMv (W6)	18% (4/22)
Salience W (W6)	36% (8/22)	Salience W (W5)	14% (3/22)
PoorAir L (W5)	33% (3/6)	Imprt CareEnv (W6)	14% (3/22)
WtrPlltn W (W5)	20% (1/5)	Salience C (W5)	9% (2/22)
Mmbr EnvMv (W5)	18% (4/22)	Conf EnvMv (W6)	9% (2/22)
Env vs Econ (W5)	18% (4/22)	Env vs Econ (W6)	9% (2/22)
Conf EnvMv (W6)	14% (3/22)	Salience W (W6)	5% (1/22)
Mmbr EnvMv (W6)	5% (1/22)	Conf EnvMv (W5)	5% (1/22)
Imprt CareEnv (W6)	5% (1/22)	Imprt CareEnv (W5)	5% (1/22)
Conf EnvMv (W5)	5% (1/22)	PoorSwg L (W5)	0% (0/5)
Imprt CareEnv (W5)	5% (1/22)	PoorWtr L (W5)	0% (0/5)
GW (W5)	0% (0/2)	WtrPlltn W (W5)	0% (0/5)
Env vs Econ (W6)	0% (0/22)	GW (W5)	0% (0/2)

summarizing our results, we identified limited predictive power for the 23 items found to be reliable in the first part of our analysis. Survey questions with specific, concrete referents (e.g., air quality, water quality) outperform more general questions. Questions that ask about the relative salience or importance of the environment versus other social problems show clear positive correlations with objective sustainability indicators. Items asking about membership in the environmental movement or about whether to prioritize the environment or the economy show very mixed relationships with objective indicators, changing across different survey years. Finally, items asking about confidence in the environmental movement or the general importance of caring for the environment have weak predictive power.

Turning to the right-hand side of Table 12, results differ markedly when controlling for per capita GDP. The items asking about directly perceptible forms of pollution show very little predictive power. Items asking about relative salience also perform less well, while items asking about membership in the environmental movement do better. Items asking about confidence in the environmental movement and the general importance of caring for the environment continue to perform poorly.

4. Discussion

This study assessed the reliability and validity of 28 items drawn from the World Values Survey Waves 5 and 6 and the 2010 International Social Survey Program. Reliability was investigated by looking for significant correlations between identical or similar items appearing in different years, while validity was evaluated by looking for significant correlations between the survey items and objective indicators of environmental sustainability drawn from the Environmental Performance Index. Our findings show a striking reduction in the pool of items that appear to provide reliable and valid results for survey users.

A total of 5 of our initial pool of 28 items were removed due to non-reliability. Several reasons might explain this lack of reliability. First, differences in wording, even where the meaning is similar, can lead to different response patterns, emphasizing the need for identically worded items administered at different points in time [26]. Second, the lack of reliability could be due to a problem with vaguely worded items (e.g., the item asking about general concern for the environment), as suggested by prior research [26]. Finally, inconsistency in questions involving either economy versus environment trade-offs or willingness to make personal financial sacrifices for the environment may be due to economic fluctuations between survey administrations (period 2006–2016), as public opinion on the environment may be affected by short-term economic conditions [15,16,17]. Previous research has found that pro-environmental attitudes [16] and support for environmental protection [17] decrease during unfavorable economic periods [14]. Therefore, we recommend caution when interpreting results associated with low-reliability items.

Our validity analysis further reduced the list of social survey items useful for survey users or policy makers. The set of most valid items contains items that refer to concrete and specific environmental issues (e.g., air and water pollution) than items measuring concern for more global environmental problems (e.g., climate change). This is consistent with earlier research [26,32,34] showing that people are well aware of perceptually salient environmental problems. Items with clear referents provide valid measures of that awareness and should be preferred for inclusion in future surveys.

Among the less concrete social survey items associated with the environment (e.g., environmental movement, environment versus economy, caring for the environment, and salience of environmental problems), the relative salience of items asking participants to rank the importance of the environment compared to other social issues outperformed the others in terms of their ability to predict a greater number of objective environmental indicators. Other items forcing participants to make choices with clear referents (have they ever joined an environmental group?; would they prioritize the environment over the economy?) had a rather mixed performance but still did considerably better than items asking about general expressions of sentiment (importance of “caring” for the environment or “confidence” in the environmental movement). Vague, non-contextualized items may not yield clear results because environmentalism and a concern for sustainability are likely to have different meanings in different developmental settings [18,32,34,45,55,56]. Overall, items that force participants to make clearly defined choices seem preferable for inclusion in surveys.

Results differ sharply when controlling for per capita GDP. The items asking about directly perceptible forms of pollution show less predictive power. Concern for the environment in less developed countries is associated with highly salient environmental threats to human health [32,34]. However, respondents in more developed countries are less concerned about these issues. Higher levels of development impart better civil infrastructure, pollution control technologies, greener energy sources pollution, and more rigorous environmental regulations [9]. Such investments promote public health and better performance across the EPI indicators. It is thus not surprising that, when controlling for level of development, these items are less strongly associated with these perceptible forms of pollution. Indeed, these items can almost be said to serve as measures of the level of economic development.

Items asking about relative salience also perform less well, while items asking about membership in the environmental movement and the environment versus the economy do better. Importantly, correlations were more likely to be negative for all three of these item types (especially for the ISSP results). This suggests that at a similar level of economic development, the ‘degradation hypothesis’ [32,33,34] appears more relevant, as higher levels of concern are associated with poor environmental conditions.

Finally, items asking about confidence in the environmental movement and the general importance of caring for the environment continue to perform weakly while controlling for GDP per capita. Such items seem to have little to recommend them.

We also found that 8 of the 22 EPI indicators (including FORCH, FSOCPEN, MPAEZZ, PACOVD, PACOVW, PSPW, PSPU, and WASTECXN) were seldom or never associated with survey items. These indicators show the existence of environmental ‘blind spots’. Most of them (aside from WASTECXN) focus broadly on biodiversity, and poor performance on these indicators is associated with the dire state of global fisheries, high levels of deforestation, lack of progress in water treatment, and critical loss of biodiversity in the last decades [8,9,10,11]. Our findings suggest low awareness of these issues. This is consistent with earlier research that also failed to find an association between actual levels of biodiversity and levels of environmental concern within a country [57,58]. In one study that did find an association, the author nonetheless concluded that most lay people are not aware of variations in biodiversity levels [34]. This mismatch between perception and reality represents a grave concern as these indicators track long-term environmental sustainability.

Several other objective indicators also had very few associations with survey items. For example, less salient forms of air pollution (e.g., NO₂ and PM_2.5 markers) were much less likely to correlate with questions about air quality than more salient indicators, such as levels of household air pollution (HAP, HAPR). Similarly, a limited number of survey items correlated marginally with nitrogen imbalances (NUE and NBALANCE) linked to industrial farming. However, such imbalances impact air and water quality, deplete the ozone layer, worsen climate change, and are believed to have already violated the planetary boundary for biogeochemical flows [1,30,31]. Some of these less visible or perceptible pollutants (CO₂, NO₂, PM_2.5, nitrogen fertilizers) are related to features of ‘advanced’ economies (e.g., mass production, industrial farming) that are difficult to change without considerable economic sacrifice [5,12]. Such problems are particularly challenging to manage as they attract less attention and will not simply be resolved by economic development.

The existence of these apparent “blind spots” in public awareness suggests the need for better tools to educate and inform the public on these issues. It also reinforces the importance of more survey items measuring public attitudes regarding these environmental issues.

There are several limitations to this study. First, the EPI indicators themselves have limitations. They do not provide measurements for freshwater quality, species loss, indoor air quality of commercial buildings, toxic chemical exposures, municipal solid waste management, nuclear safety, wetlands loss, agricultural soil quality and degradation, recycling rates, and adaptation, vulnerability, and resiliency to climate change [30,31]. More current versions of the EPI have started to add measures covering some of these areas but are unfortunately not backcasted yet [9]. Unfortunately, the current EPI indicators also do not assess trade-offs associated with the use of renewable energy or with other changes made for the sake of sustainability. Our study covered only three administrations of two international surveys, preventing us from commenting on other surveys on the environment. The surveys we selected also impose certain limitations. As with the EPI indicators, they cannot be said to exhaust the universe of possible environmental issues that could be asked about. Also, many interesting social survey items only appeared once on the surveys chosen and could not be tested for reliability. With Neumayer [26], we hope that future studies will re-use questions with identical wording from previous studies to allow for reliable measurements across years. Further, our study used country averages, and thus we are not able to talk about within-country differences. A final limitation is that our study was interested only in identifying correlations, and we are thus not able to make causal claims relating environmental indicators to responses to survey questions.

5. Conclusions and Recommendations

The importance of transitioning to sustainability cannot be overestimated, and public support is essential for reaching this goal. To build such support, we need better tools and measures specifically designed for measuring it, including items that work together to provide a more comprehensive view, taking in comprehension, concern, and willingness to act or make sacrifices to achieve sustainability.

Our findings have implications for survey developers, survey users and policy makers. For survey developers, our results show that it is possible to develop items that are both reliable and valid in terms of being associated with objective indicators of environmental sustainability. However, there are currently too few such items. The use of clear, concrete items that appear in identical or nearly identical form across surveys is critical. Items that ask about specific problems or that force respondents to make clear, concrete choices appear to outperform items that ask for expressions of general sentiment. In terms of specific environmental problems that might be addressed, our finding of numerous EPI indicators that are effectively ‘blind spots’ that are seldom or never associated with survey responses is also noteworthy. Further items able to better tap into perceptions of these environmental conditions could offer a better sense of public awareness of the environment. More such survey items are urgently needed.

For survey users, our findings have a clear message: not all items are created equal, and careful choices are required in choosing measures of public opinion. The reliability or stability over time and survey administrations of such items and the associations of these items with actual environmental conditions can differ widely. It is thus perhaps not surprising that studies relying on a small number of survey items have arrived at widely varying conclusions [18,32,55,56,59,60]. Additionally, the large differences that emerge when GDP per capita is controlled for should serve as a warning to users of survey results. Deliberate decisions need to be made in analyses about whether to control for economic level [9]. A final point is that items from the WVS and ISSP surveys, with their very different samples of participating countries, performed rather differently. While not surprising, it does highlight the need to carefully consider participating countries in using international survey results.

Finally, our findings have implications for policy makers. First, they should clearly be aware of the limitations of existing surveys as representations of public opinion. Second, from our results, it is clear that the views of the mass publics in most countries do not accurately reflect actual environmental conditions and thus progress in achieving the sustainable development goals (Table 1). It is of critical importance that policy makers help to make less perceptible pollutants more visible through education campaigns and readily accessible metrics. While people are well aware of highly perceptible pollutants, there are several ‘blind spots’ associated with less salient yet critical environmental problems, such as NO₂ or PM_2.5 levels. Citizens deserve accurate and understandable information about environmental conditions that will allow them to make informed and reasoned judgments.

Valid and reliable measures of consciousness and attitudes about sustainability are critically needed if we are to understand civil society attitudes towards sustainability. This study has identified a small pool of survey items that appear both reliable and valid. However, this pool is currently far too limited, and more items are urgently needed. We hope that our research will offer guidance to future researchers in their selection of survey items and environmental indicators and to survey developers in their construction of new items.