A Prediction Model of Water In Situ Data Change under the Influence of Environmental Variables in Remote Sensing Validation

Round 1

Reviewer 1 Report

Overview:
This paper is a renewed version of a manuscript “Reliability Assessment of In Situ Data in the Validation of Waterbody Remote Sensing Products: A Case Study in Taihu Lake”. The authors revised the manuscript from the title to detail parts. I feel the aim of this study is much clearer than that in the previous and this version is well improved from the previous. From this study, it is clear that which kinds of EVs will be enough to predict future ChI-a and TSS from their correlation analysis, this is a good achievement in this study.

However, I think there are still large gaps to be published of this study, unfortunately. Most reason is that the author did not provide sufficient information about feasibility of their methodology in actual use. I mean I cannot determine whether their method is actually applicable for sites where regular buoy observation is not conducted.
More detail, I have a question how much the model parameters the authors got in this study are general ones or not even in Taihu Lake, or even nearby the buoy site the author used. Although the authors discussed the validity of their method in L 479 – 484 in terms of spatially (the prediction at a buoy site can be applicable at a nearby site), the authors did not provide any experimental evidence. So I cannot determine their statement is reasonable especially qualitatively. Moreover, if the statement in L 479 – 484 is correct, then time series of in-situ measurement at a specific site is enough for determining the representativeness of the nearby in-situ data, so why we need prediction?

From the authors’ statement in introduction (L81 - 88), I understand that the purpose of this study is to predict how much ChI-a and TSS will vary at a site where regular in-situ measurement is hard to conducted, and determine the representativeness of single or limited number of in-situ measurements done by, for instance, an experimental ship at such a site. So the predicted values should be strongly required at a site where only limited number of observations can be done, is it right? I am confused by the inconsistency between the concept and conclusion in this study.

Therefore, I feel it should be better to conduct one or two more observations at one or two different sites in Taihu Lake to prove their model’s performance. Thus, I recommend the authors to revise their manuscript as a major revision. My suggestions are listed below.

Main point:
1.
As I wrote in above overview, I am very unsure that the proposed method is actually feasible in actual use. The author got prediction parameters from one site in Taihu Lake, and applied the model only to the same site to check validity of their model prediction.
Unfortunately, the authors did not provide any result that can help readers to understand the proposed method is feasible in a more realistic case, considering a site measured by an experimental ship where regular (long-term) in-situ measurement is not available. The validation of in-situ measurement at other sites which is different from the buoy site is a key issue for the authors' purpose and model usability.
I recommend the authors to conduct one more experiment at a site in Taihu Lake (but different from the buoy site), measure ChI-a and TSS at several time, and then compare the measured values with the predicted ones.

Since Taihu lake is somewhat large lake (~50 km width), the authors can use weather parameters (such as, AWS, AWD can be obtained) which are given from weather forecasting that may improve the prediction accuracy for other sites.

2.
To predict ChI-a and TSS, regular in-situ measurement for other EVs is required? In the equations of (2) – (4), the authors used EVs at a target time when ChI-a and TSS will be predicted.
If the author assume such regular in-situ measurement of EVs for the prediction, I am again confused why the authors need predicted values of ChI-a and TSS to determine the representativeness of them. I guess such in-situ observation system can also obtain time series of the actual in-situ ChI-a and TSS measurement…
Of course, the proposed method is good for the case that a buoy observation system only obtains EVs, and not obtain ChI-a and TSS. Is it a special case that a buoy system obtains ChI-a and TSS?

3. Section 4.2.1
I found a methodological issue in training and testing of their model. The authors prepared 15608 data vectors from 1678 ChI-a observation records. Then they divided the “15608 dataset” randomly to 12807 for training and 2261 to testing. This means that several ChI-a records should be included into both training dataset and test dataset, so that test result may not be independent to the training dataset perfectly.
To avoid such concern, I recommend the authors to separate their data into training and testing datasets in terms of “date”. For instance, 80 % dates are used for training, and 20 % dates are used for only test purpose, so that test dataset is perfectly independent to the training dataset.
This concern is same as TSS case (section 4.2.2) and sunny-day experiments (section 4.3).

4. Line 291-292:
Considering that the range of Chl-a in the all-valid dataset is between 0.46μg/L and 8.19μg/L, we assumed that AE less than 10% is the target accuracy of this paper.

I cannot understand why 10 % target accuracy is enough for their purpose. Please explain more. I think the target accuracy should be determined from requirent of RSPs validation issue, that is the motivation of this study.

5. Results in Table 1 in response 3 for Reviewer #4
The results shown in the table is different from that I expected, and I feel the result may not be correct to understand actual model performance in terms of time. Maybe underestimate for longer Δt case. Although the authors set time range as <=1h, <=2h, … <=5h, the results should be summarized with separate time ranges of 30min < Δt < 1h, 1h < Δt < 2h, …, 4h < Δt < 5h. This should be suitable to evaluate the model performance at different Δt.
In addition, these results are worth to be shown in main text.

Minor points:

Lines 66-67 :
The representativeness and accuracy of in situ data are of significant [9].

In the context of this manuscript, the representativeness of in-situ data is considered to be not guaranteed. To avoid misleading, remove “representativeness”

Line 79:
and the representativeness of the in situ data will be better with more samples

This statement is not clear. Do you mean "more samples" in terms of spatial?

Lines 126 - 128:

Although the authors showed "frequency" of the instrument calibration, it is not enough. The more important issue is accuracy of the instument. Please add information how accurate the measurement,

Line 143:
"wrong" should be "invalid".

Line 146:
I think "of the same day " is not necessary.

Line 187:
All the methods have its own advantages and disadvantages

I missed the disadvantages of "GRNN" method, and I think that's important information for readers. Please describe.

Line 195:
and models were established to predict their changes

The model output can be countious one in tems of time? If so, it is convienent to compare not only RSP, but also any other measurement.

Table 2:
I feel this table is not easy to catch numbers. My suggestion is
- remove redundant nubmers
- Emphasize results for important parameter combinations

Same as Table 6

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

The presentation of the paper regarding the validation terminology was improved.
Specifics:
Comment 1:
- After a total of 2166 effective measured data, the data were reanalyzed and 1678 observations remained. How were the 134 outliers identified? Which method was used for? And the abrupt variations of the observation environment? How to know if an observation is from the instrument drift? Yet, how to know if one observation comes from uncontrollable factors?
From Response 1 and changes in the paper, the part of abrupt variations, instrument, and uncontrollable factors were removed and the information about the outliers identification was included. However, as shown from the authors in the next comment, the data is not normally distributed, and under asymmetry, using mean +- 3 standard deviation is not a good option. Usually, what happens in this case, is to identify many more outliers, and consequently, excluding important data. There are adequate measures for that, considering such as the medcouple which is a robust measure of skewness and conducts to an adequate outlier detection.
Comment 2:
- Did the authors check the data distribution before Pearson Correlation? Is it normally distributed?
From response 2, the authors verified that not all data is normal distributed and kept using Pearson correlation. When bivariate distribution is not normal, the Spearman or Kendall coefficient is more appropriate.
Comment 3:
- Based on what the correlation between 0.2 to 0.8 was considered moderate? References? It seems a very large range to have a moderate correlation. Is the correlation of 0.2 statistically significant?
Based on the Response 3, the authors are not using even the reference they included. In this reference, Fair or Moderate Correlation is indicated for coefficients between 0.35 and 0.50. Of course, there are texts on the internet without any foundation, such as the presented print, where states week correlation between -0.2 and 0.2. There are statistical hypothesis tests for Pearson, Spearman or Kendall, and you can check if the correlation in question is statistically significant under for instance a 95% confidence level.
- Contradictorily, on line 246, the authors refer to variables with correlations of -0.26 and 0.24 as independent variables
Although the authors improved this section, comment that the correlation of -0.26 between the variables Cond and AWD represents almost independence remains. I suggest an initial comment in this section telling that the explanatory variables that have a correlation coefficient larger than some critical value are going to be excluded to avoid multicollinearity. Don’t use strong, moderate, almost independent, etc mainly if it is not following the standard you are using (or should use).
Comment 5:
- I don`t understand why is the measure in table 2 (Section 4.1.2) autocorrelation?
The authors made the adequate correction.
- Figs 4 and 5 lack an explanation of the colored and black points. The same for the connected black line. The reader needs to read the text until the end of the section to understand these figures.
Comment 6:
The authors made the adequate correction.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

Manuscript ID: remotesensing- 1035170

 

First, I would like to thank the authors for providing a revised version of the previous manuscript. The revised version improved the content of the research. Second, here are few comments about this revised version.

 

Abstract

L14-15 – In my opinion, I do not agree that only the reliability and effective application of the in situ data of waterbody parameters are the core of validation. Validation also encompasses the atmospheric correction process and algorithms' performance for estimating optically active components by remote sensing.

L25-28 – In this sentence, it seems that you ended the abstract with part of the methodology. It is important to add a general conclusion of your research to complete the idea of your abstract.

 

Introduction

 L113 – "… pattern…"

 

Results

 L259-264 – Sorry, but I am a little bit confused now. In the original manuscript (L238-239), you said that FDOM is closely related to the distribution of organisms and has high spatial heterogeneity in lakes, which can't be used as one of the effective modeling variables. In the new manuscript (L261-263), you said that although FDOM has a moderate correlation with Chl-a, it was not considered as the input variable for the modeling due to its weak spatial representation. Was the FDOM not considered as an input variable for the modeling due to the lakes' high heterogeneity or weak spatial representation? An observation, the sentences in L334-337 (new manuscript) show the same explanation as to the original manuscript (L238-239): "Like FDOM, PC is related to the distribution of phytoplankton, and has strong spatial heterogeneity in Taihu Lake [48]. Therefore, both of FDOM and PC are not suitable to be selected as modeling variables."

L450 – "… are needed…"

 

Discussion

L510-511 – You have not added the explanation of comment 8 (answer 8 from the discussion part that you sent to me) in the text. It is important to show in the text which EVs were the ones that most influenced the variation of Chl-a, just as you showed for TSS in the new manuscript (L516-520).     

            My question was:

            Comment 8:

        L500 – How was the EVs' behavior/variation between 10:00-12:30             and 13:00-15:00? Which EVs most influenced the Chl-a variation from             13:00 to 15:00 in Taihu Lake?

            Your answer was:

            Response 8:

           Figure 4 shows the corresponding EVs that affect the changes in Chl-a.           AWS and AT are increasing from 10:00 (the first point in Figure 4) to             15:00 (the 11th point in Figure 4), while AP is slightly decreasing from             10:00 to 15:00. All three EVs contribute to the increasing of Chl-a.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The authors almost addressed my concerns I raised in last revision, thank you for much efforts for your revising. I agree the authors responses, and understand the model performance based on the experimental results which the authors showed in Response 1 (of course it should be better that the results in Response 1 will appear in this paper, but I understand another paper will show the results in detail).

 

Unfortunately, I have still one concern/discussion relating to my Comment 3, which is about the issues for data separation of training and testing datasets and the result.

 

The data separation and estimation that the authors conducted in Response 3 were reasonable for me, in which the testing dataset was perfectly separated from the training dataset. I feel the slight decreasing of the model performance is natural comparing to model performance from original version, because the test dataset in the original version could have several data records that were used for training, which basically leads overestimation of model performance. Not only number of dates they could use.

 

My recommendation is that the authors will show the results from the updated data separation in the main text. However, as the authors mentioned, the scatter plots in Figure 1 and Figure 2 in Response 3 were somewhat strange for me, too, in which several points were arranged in the form of a horizontal line (= similar magnitudes of Chl-a will be predicted, though the corresponding in-situ Chl-a are different). Especially in Chl-a results, the predicted Chl-a values seems to distribute with a discrete manner. Not only amount of date, I think there could be some numerical mistakes, such as effect from rounding error.

 

To polish this research more, I recommend the authors to find the reason of forming the horizontal lines in the scatter plots and modify it. Even if it is difficult, still I recommend the authors to use the updated data separation manner for this paper.

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The comments 2-6 were taken into account. Comment 1 about using a more adequate method to exclude outliers was not. All the other analysis is made after that exclusion. Unfortunately, important information from the data may be out of the analysis.

Additionally, there is no comment about residual analysis in the method presentation nor any evaluation in section 4 of Modeling. It is important to include at least the information if the usual assumptions of the residuals were fulfilled.  The results can be trusted only if this residual analysis is performed.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

Overall this is an interesting paper. It is well written but needs a little English review concerning mainly prepositions. The abstract and objectives should be improved, and some parts of the results and discussion need to be more described.

 

Abstract

L15 – chlorophyll-a (Chl-a), Total Suspended Solids (TSS)

L16 – (environmental variables – EVs)

L18 – Please remove: (environmental variables)

L19 – Which models? Please list them for Chl-a and TSS

L21 – Which best models? Please list them for Chl-a and TSS

L22 – Generalized Regression Neural Network (GRNN)

L23 - absolute average relative error (AE)

L23 – Please specify the Chl-a model with this result

L24 – Please specify the TSS model with this result

L12-25 – Please improve the abstract. The objective, methodology, important results, and conclusion of your research should be better described.

 

Introduction

L31-32 – Why is remote sensing playing an indispensable role in measuring and monitoring the water environment? It is important to add this, please.

L31-32 – Chlorophyll-a (Chl-a) and Total Suspended Solids (TSS)

L33 – Remote Sensing Products (RSPs)

L35 – Chl-a is also used to characterize water quality. Please be more specific in describing the importance of TSS.

L36 – Which advanced products?

L42 – National Aeronautics and Space Administration (NASA)

L43 –  Earth Observation System (EOS)

L43 –  Moderate Resolution Imaging Spectroradiometer (MODIS)

L45 –  European Space Agency (ESA)

L46 –  Validation of Land European Remote Sensing Instruments (VALERI) program

L48 –  Medium Resolution Imaging Spectrometer (MERIS)

L49 –  Committee Earth Observation Satellite (CEOS)

L50 – Land Product Validation (LVP)

L53 – Canada's Boreal Ecosystem Research and Monitoring Sites (BERMS)

L54 – Large Scale Biosphere-Atmosphere Experiment in Amazonia (LBA)

L56 – Watershed Airborne Telemetry Experimental Research (WATER)

L60 – Not only the reliability of validation. The in situ data accuracy also determines the model's calibration reliability to retrieve the Optically Active Components (OACs).

L84-85 – “Using resolution of 3×3-pixel box and ±3 hours are used as match-up window”: please review this part; the writing is not ok.

L88 – Sea-viewing Wide Field-of-view Sensor (SeaWiFS)

L92 – Chl-a concentration

L94 – Which validation activities?

L99 – environmental variables (EVs)

L99-100 – Please add a reference

L103 – Chl-a and TSS concentrations

L105 – Chl-a and TSS concentrations

L99-107 – Please improve this paragraph. In this part, the central objective and specific objectives of your research must be clear. Here it seems that you are summarizing the methodology of your research.

 

Data and Preprocessing

L119-123 – Were the data (WT, SpC, Cond, Sal, Chl-a, Phy, FDOM and TSS) measured in the water subsurface (or which depth)?

 

Method

L148-150 – How did you establish these limits (0.8 and 0.2)? Please add a reference.

L153 – Chl-a and TSS concentrations

L154-155 – Multiple Linear Regression (MLR), Back Propagation Neural Network (BPNN) and Generalized Regression Neural Network (GRNN)

L153-155 – Please review this sentence; the writing is not ok.

L168 – Please specify PM 2.5

L173 – Chl-a and TSS concentrations

L186-188 – Chl-a and TSS concentrations

L195 – coefficient of determination (R²), Root Mean Square Error (RMSE) and absolute average relative error (AE)

 

Results

L204-208 – This paragraph seems to be part of the methods, not of the results

L206-208 – It is important to be clear why you selected the variables with lower cross-correlation as the independent variables to build the multi-parameter forecasting models. Please, explain more about it.

L218 – Remove “&”, add “and”

L219 – Please add in the legend the name of the abbreviations: WT, SpC, Cond, Sal, Phy, FDOM, AWD, AWS, AT, AH and AP

L221-223 – Please describe more the results of Table 2. You only showed the results.

L224 – Please add in the legend the name of the abbreviations: WT, SpC, Cond, Sal, AWD, AWS, AT, AH and AP

L227-229 – Please add a reference

L229 – “Chl-a x Cond” is 0.47 in Table 1

L238-239 – Please explain more about it and add a reference

L281 – “…are 5.2% and 9.8%, respectively.”

L286 – “Then 85% of the data (9856)…”

L287 – “… and the remaining 15% (1740)...”

L300 – “… and its modeling and testing AE are 7.3% and 10.3%, respectively.”

L301-303 – It would be interesting to add a description at the beginning of the results about the range of Chl-a, TSS and EVs obtained from May 2019 to May 2020. So the reader can have an idea of the parameters variation in the lake. According to the Chl-a range, is a model with AE lower than 10% also sufficient to predict the changes of the measured Chl-a_t0 at other unobserved moments t?

L318-320 – Please explain more about it and add a reference

L346-348 – You should describe the range of Chl-a in all-valid data. In all-valid and sunny-day data sections, please describe the range of Chl-a, TSS and EVs obtained from May 2019 to May 2020. It is important to do a characterization of these variables to better understand the water quality behavior through the months.

L361-364 – This part is not so clear to understand. Please rewrite it.

L365 – “In Figure 4, Chl-a values of three prediction sets are around 1.5 (line (3)), 2.5 (line (2)) and 3.5 μg/L (line (1)), respectively….” Were data from lines (3), (2) and (1) measured on different dates?

L366 – “…and the prediction results of Chl-a (colored dots) on each polyline are very close to…”

L369-370 – “…can meet the requirements of the satellite validation on the ground truth value.”

L370-373 – Please rewrite this part. It is not clear enough.

L374-375 – What represents line (1) and (2)? It should be clear. Were data from lines (2) and (1) measured on different dates?

L376-381 – Please rewrite this part. It is not clear enough.

L365-381 – Why does Chl-a appear to be more stable? Why does TSS have a more significant variation? What is happening in the environment that generates this? I hope to find these answers in the discussion.

L383 – Please add in the legend what are (1), (2), (3) and the colored dots. The legend should be more explicative to readers.

L385 – Please add in the legend what are (1), (2) and the colored dots. The legend should be more explicative to readers.

L386-387 – This part should have been added on L360-361. I can not see the black ‘*’ in Figures 4 and 5.

 

Discussion

L392 – In my opinion, the relationships between EVs and Chl-a/TSS are not so deeply analyzed. Please improve it.

L396-397 – You should say that WT, Cond, AWS, AT and AP are moderately correlated with Chl-a in the all-valid dataset. In the sunny-day dataset, Cond is not moderately correlated with Chl-a (R _{Chl-a x Cond} = 0.12).

L408 – It is important to explain more how the wind field affects the changes in Chl-a concentration. Is it positive or negative?

L408-409 – Please explain more the impact of AP on Chl-a changes. What could happen to chl-a concentration when AP increase or decrease?

L408-410 – Please add a reference.

L396-411 – And the discussion about the impact of Cond on Chl-a changes? Your results showed that SpC was selected as one of the independent variables to build the Chl-a prediction model, not Cond (L240 of the manuscript – all-valid dataset).

L417-418 – Please explain more about it.

L421-422 – Please explain more about it and add a reference.

L476  – “…by screening data according to the limitation…”

L481  – “… such as MODIS, MERIS, etc., due to the large…”

L483-484  – “ The excessive distance of sampling points and complex water environment will lead to a large difference…”

L485 – “…cannot represent the EVs of the whole experimental area.”

L486 – “… sampling points within the observing time period…”

L487 – “…in situ data overtime on large spatial scales…”

L491 – “… changes in the in situ values of Chl-a and TSS on a daily time scale…”

L494 – “…Figure 6 gave an example of using multi-parameter forecasting models…”

L496 – “Most satellites…”

L496 – “For the change curve in Figure 6(a),…”

L500 – How was the EVs' behavior/variation between 10:00-12:30 and 13:00-15:00? Which EVs most influenced the Chl-a variation from 13:00 to 15:00 in Taihu Lake?

L501 – Why did you not consider between 10:00 and 12:30?

L504-505 – How was the EVs' behavior/variation between 10:00-12:00 and 12:00-15:00? Which EVs most influenced the TSS variation from 10:00 to 12:00 in Taihu Lake?

L505 – “It can be seen in Figure 6(b) that….”

L515 – “An example of in situ data screening using a multi-parameter forecasting model of (a) Chl-a and (b) TSS.”

 

 Conclusion

L521-532 – It is important to highlight here how your work contributes to the improvement of remote sensing research.

Comments for author File: Comments.pdf

Reviewer 2 Report

It is a relevant paper using environmental variables to predict changes of Chl-a and TSS at different times. Different models were compared, showing the Generalized Regression Neural Network prediction models presented the highest accuracy. However, some points need to be discussed.

General: The validation terminology that comes at the beginning of the paper creates an expectation that is not supported in the rest of the paper. Although the paper is very interesting, the focus is not well oriented and presented. Some important points raised in the introduction are not discussed in the experiment/results of the article.

Specifics:

- After a total of 2166 effective measured data, the data were reanalyzed and 1678  observations remained. How were the 134 outliers identified? Which method was used for? And the abrupt variations of the observation environment?  How to know if an observation is from the instrument drift?  Yet, how to know if one observation comes from uncontrollable factors?  

- Did the authors check the data distribution before Pearson Correlation? Is it normally distributed?

- Based on what the correlation between 0.2 to 0.8 was considered moderate? References? It seems a very large range to have a moderate correlation. Is the correlation of 0.2  statistically significant?

- Contradictorily, on line 246, the authors refer to variables with correlations of -0.26 and 0.24 as independent variables

- I don`t understand why is the measure in table 2 (Section 4.1.2) autocorrelation?

- Figs 4 and 5 lack an explanation of the colored and black points. The same for the connected black line. The reader needs to read the text until the end of the section to understand these figures.  

Reviewer 3 Report

The authors presented an interesting study on modelling the time-series waterbody indices based on in situ measurements. My major concern is would the title of the manuscript and the current introduction be appropriate since no analysis related to the validation of RS products was conducted?

Other major concerns include: 

Abstract:

1. lines19-22, it's not clear what exactly the two kinds of prediction that you made were. What's the difference?

2. You used both all data and data from sunny days. You should mention this in the abstract with corresponding accuracy.

Introduction:

The aim of this study was to model waterbody indices using environmental factors. But I didn't see any description of this. What are the feasibility and benefits of using environmental factors to predict waterbody indices? Is there any literature supporting this? I guess one of the benefits is addressing the problem that you talked about in the previous paragraph.

Method:

I really got confused with section 3.3. The description of the method is not clear. In lines 254-257, you said 15068 data vectors were formed based on the method in section 3.3. Why? Not clear.

Results:

1. It really needs careful revision. The current text is a mixture of methods, results and discussion!! For example, in lines 262-282, you are actually discussing the results. There are many ambiguous arguments (e.g. lines 276-280). They actually don't explain the results. Also, please avoid repeating what you have mentioned in the previous sections (e.g. lines 280-282). 
2. lines 227-231, Cond has the same R with WT. Why did you select WT rather than Cond?
3. section 4.4 is not clear. Which dataset was used for prediction?
4. Figures 4 and 5, you should explain what the different lines represent in the captions.

Reviewer 4 Report

Please check the attached file.

Comments for author File: Comments.pdf