Longitudinal Predictors of Perceived Climate Change Importance and Worry among Italian Youths: A Machine Learning Approach

Round 1

Reviewer 1 Report

Thank you for the opportunity to review the manuscript titled Longitudinal Predictors of Perceived Climate Change Importance and Worry among Italian Youths: A Machine Learning Approach.

Authors have tapped an important topic concerning an important population. Longitudinal data on Italian youths’ socio-demographic factors, political perceptions, attitudes on a national and European level, efficacy beliefs, social well-being, political interest, and different forms of participation were analyzed as potential predictors of perceived importance of climate change and personal worry.

The study has an overall high merit. However, this reviewer has some concerns to be addressed.

1-   Part of the data were collected from six schools located in north Italy. There needs to be some contextual information about this region. In particular, how similar/different it is from the rest of Italy as well as European countries? Also, how were the six schools selected? Randomly? Why private schools were excluded? Same questions go for the university students. We need to know how the Italian university students are compared to the rest of youth.

2-   Was the survey anonymous? If yes, how did researchers identify under-age students who secured parental consents from those who did not?  

3-   How long the survey took on average? It seems to be like a long survey. In case some students were willing to participate while others did not, how the researchers handled the classroom?

4-   I understand paper as well as online surveys were used. I believe this needs to be acknowledged as a limitation. It is well documented in the literature that a difference in the delivery of a research questionnaire (method, time, setting) can make significant differences in the answers.

5-   How were missing data explored and handled?

6-   Were all statistical tests assumptions examined? If any not met, how was the analysis handled?

7-   Finally, authors could very briefly describe the outcomes of the pilot study. Particularly, those related to changes on the study questionnaires.

Author Response

Reviewer 1

Thank you for the opportunity to review the manuscript titled Longitudinal Predictors of Perceived Climate Change Importance and Worry among Italian Youths: A Machine Learning Approach.

Authors have tapped an important topic concerning an important population. Longitudinal data on Italian youths’ socio-demographic factors, political perceptions, attitudes on a national and European level, efficacy beliefs, social well-being, political interest, and different forms of participation were analyzed as potential predictors of perceived importance of climate change and personal worry.

The study has an overall high merit. However, this reviewer has some concerns to be addressed.

Response: Thank you very much for your comments on our manuscript. We have revised the manuscript in order to address these concerns.

1-   Part of the data were collected from six schools located in north Italy. There needs to be some contextual information about this region. In particular, how similar/different it is from the rest of Italy as well as European countries? Also, how were the six schools selected? Randomly? Why private schools were excluded? Same questions go for the university students. We need to know how the Italian university students are compared to the rest of youth.

Response: We have added a paragraph entitled “2.2. The Context” to provide some contextual information about this region. Moreover, we added information about the selection of the six schools as well the university (convenience). Finally, we explained why private schools were excluded.

2-   Was the survey anonymous? If yes, how did researchers identify under-age students who secured parental consents from those who did not? 

Response: We confirm that the survey was anonymous. All under-age students secured parental consents, so we did not have to take any measure to identify under-age students who secured parental consents from those who did not.

3-   How long the survey took on average? It seems to be like a long survey. In case some students were willing to participate while others did not, how the researchers handled the classroom?

Response: It was about 30 minutes on average. Questionnaires were completed under the supervision of a researcher and/or a teacher during a class hour. Participation to the research was on a voluntary basis and no personal incentives were provided. None of the students interrupted the participation to the study, although it was possible.

4-   I understand paper as well as online surveys were used. I believe this needs to be acknowledged as a limitation. It is well documented in the literature that a difference in the delivery of a research questionnaire (method, time, setting) can make significant differences in the answers.

Response. Following your request, we have acknowledged the difference in the delivery of a research questionnaire as a limitation.

5-   How were missing data explored and handled?

Response: Missing data analysis revealed minimal missing data (i.e., 4.35%). Following recommendations for dealing with missing data [89], given that missing data was minimal (less than 5%), we decided not imputing data.

6-   Were all statistical tests assumptions examined? If any not met, how was the analysis handled?

Response: We agree that the assumptions statistical tests should be examined. For tree-based models such as Decision Trees, Random Forest & Gradient Boosting there are no model assumptions to validate. Unlike OLS regression or logistic regression, tree-based models are robust to outliers and do not require the dependent variables to meet any normality assumptions. However, there are other considerations to keep in mind when developing Friedman’s gradient boosting machines (GBM) algorithm and Breiman’s random forest algorithm. For both GBM algorithm and random forest algorithm, we scaled the variables. We conducted different sensitivity to determine the optimal selection of the hyperparameters. Concerning the GBM algorithm, we tested 100 to 5,000 number of trees to fit with shrink-age rates between 0.01 and 0.001. We chose a shrinkage rate below 0.001 with a maximum of 5,000 trees to improve predictive performance. We randomly chose 20% of the data as holdout test data. The accuracy of the model was tested by twenty-fold cross-validation. Regarding the random forest algorithm, we optimized the prediction error on a validation data set with respect to a maximum of 5,000 possible decision trees to be considered and we set as 50 the percentage of training data that is used to train each individual tree. As data split preferences, we used 20% of the data as holdout test data and 20% for the purposes of training. We evaluated the performance of each model using the Mean Square Error (MSE), the coefficient of determination, and the root-mean-square error (RMSE).

7-   Finally, authors could very briefly describe the outcomes of the pilot study. Particularly, those related to changes on the study questionnaires.

Response: We have briefly described the outcomes of the pilot study

Reviewer 2 Report

First, I would like to send congratulations to authors on the choice of the topic of the manuscript and on the study as a whole.  The Article made a very good impression. Introduction is clearly written and perfectly leads to the heart of the issue, also authors used a good literature. The Methodology could be improved. The Authors could better explain study sample and sampling. They said that the six  secondary school included in the study, but they did not explain how they choose school or how many secondary school have... Also they wrote that all school were public and it is not clear also why the private school did not include or they do not have public school.  Authors could explain the readers what percentage of the youth were included in the research and in the limitation why they said that study lack of generalizability. Tre Result and Discussion are very well presented and written. In addition authors could write what are the implications for policymakers. 

Author Response

Reviewer 2

First, I would like to send congratulations to authors on the choice of the topic of the manuscript and on the study as a whole.  The Article made a very good impression. Introduction is clearly written and perfectly leads to the heart of the issue, also authors used a good literature. The Methodology could be improved. The Authors could better explain study sample and sampling. They said that the six  secondary school included in the study, but they did not explain how they choose school or how many secondary school have... Also they wrote that all school were public and it is not clear also why the private school did not include or they do not have public school.  Authors could explain the readers what percentage of the youth were included in the research and in the limitation why they said that study lack of generalizability. Tre Result and Discussion are very well presented and written. In addition authors could write what are the implications for policymakers.

Response: Thank you very much for your insightful comments on our manuscript. In the revised version of the manuscript, we have improved the methodology section, explained why chose these schools (convenience sample), explained the readers why we said that study lack of generalizability, reported that all the students agreed to take part in the study, added implications for policymakers.