**6. Experiments**

*6.1. Predictive Model Evaluation Index*

#### 6.1.1. Goodness of Fit

Goodness of fit refers to the degree of fit of the regression model to the observations, and its measurement statistic is the coefficient of determination *R*2. The value range of *R*<sup>2</sup> is [0, 1]. The larger the value in this range, the better the fitting effect of the regression equation to the training sample; on the contrary, the worse the fitting degree. When the goodness of fit is negative, It shows that the fitting effect of this regression model is too poor and has no practical significance. Suppose *y* is the value to be fitted, its mean value is *y*¯, the fitted predicted value is rounded to *y*ˆ, the total square sum (*SST*) is ∑*<sup>n</sup> <sup>i</sup>*=1(*yi* − *<sup>y</sup>*¯)2, the regression square sum (*SSR*) is ∑*<sup>n</sup> <sup>i</sup>*=1(*y*ˆ*<sup>i</sup>* <sup>−</sup> *<sup>y</sup>*¯)2, and the residual square sum (*SSE*) is <sup>∑</sup>*<sup>n</sup> <sup>i</sup>*=1(*yi* − *<sup>y</sup>*ˆ*i*)2, then *SST* = *SSR* + *SSE*, the calculation method of the determination coefficient is as follows:

$$R^2 = \frac{SSR}{SST} = 1 - \frac{SSE}{SST} \tag{2}$$

Generalization ability is an important indicator for detecting regression prediction performance. Therefore, when designing a regression model, it is necessary to consider not only the model's correct prediction of the required regression prediction object, but also the prediction effect of the model on the new data. The preprocessed data is divided into training data and test data in a ratio of 7:3. The evaluation index of the final model is the sum of the weights of the goodness of fit of the two, and the weight ratio is 3:7.

#### 6.1.2. Mean Absolute Error

The mean absolute error (MAE) refers to the average of the absolute value of the difference between multiple predicted values and the true value. In the HFMD epidemic prediction model, the average absolute error indicates the degree of deviation between the number of HFMD cases predicted by the model and the number of true cases when the number of HFMD cases is predicted in different regions or at different times. The smaller MAE, the more accurate the mode. The larger MAE, the worse the predictive ability of the model. Therefore, the magnitude of the average absolute error MAE reflects the pros and cons of the model, and its calculation formula is as follows:

$$MAE = \frac{1}{n} \sum\_{i=1}^{n} |y\_i - y| \tag{3}$$

where, *n* represents the number of predictions, *yi* represents the predicted value, and *y* represents the true value.

#### 6.1.3. Accuracy within Error

Similar to the accuracy rate in the classification problem, in order to avoid the influence of interference factors, the accuracy rate within the error is introduced. The accuracy within error (AWE) refers to the ratio of the number of samples correctly fitted by the regression model obtained through training to the total number of training samples in the regression analysis process of predicting integer dependent variables within a certain error tolerance. To a certain extent, AWE explains the generalization ability of the regression model. The higher the AWE, the stronger the fitting ability of the model, and the lower AWE, indicating the weak fitting ability of the regression model, which ranges between 0 and 100%. The calculation formula of AWE is as follows:

$$A\% E = \frac{n}{N} \times 100\% \quad \left(n := n + 1 \quad \text{when} \quad |f(X\_i) - y\_i| \le \varepsilon r r r\_{\text{ people}}\right) \tag{4}$$

where *n* represents the number of samples correctly predicted by the regression, and *N* represents the total number of samples. When the absolute value of the error between the predicted value and the true value is less than the specified error range, add 1 to the number of correctly fitted samples *n*, until all samples are trained, and the final number of samples correctly predicted is obtained.

When predicting the number of HFMD cases, the number of cases of the dependent variable is an integer number. Therefore, the predicted value must be rounded up, and then the predicted value after processing is compared with the true value. If the difference between the two is within the allowable range of the number of errors, it is considered that the trained model predicts the sample correctly, otherwise it is considered that there is a large error in the sample prediction. According to the analysis of infectious disease researchers and the regression model, the number of errors is five, i.e., when the difference between the predicted value and the true value does not exceed five, it is determined that the model fits this sample correctly.

#### *6.2. Early-Warning Model Evaluation Index*

#### 6.2.1. Warning Rate

The warning rate (WR) refers to the ratio of the number of samples that use the early warning model to send out early warning signals to the total number of samples, and its value is within the range of [0%, 100%]. Appropriate warning rate can reflect the difference of the model to different test data, and avoid the phenomenon of full warning and no warning. If the warning rate is too large or too small, it reflects the large error of the early warning model and the failure of correct warning.

#### 6.2.2. Accuracy Rate

Accuracy rate (ACR) refers to the ratio of the number of samples with the same early warning results of the model to the test data and the real early-warning results to the total sample, reflecting the accuracy of the HFMD epidemic early-warning model, and its value range is Between [0, 1].

The higher the accuracy rate, the better the prediction ability of the early-warning model, and the lower the accuracy rate, the worse the prediction ability of the early-warning model. Therefore, the accuracy rate can truly reflect the prediction effect of the HFMD epidemic early-warning model.

### *6.3. Comparison of Different Prediction Time and Space Accuracy*

To obtain a more accurate prediction model of the number of HFMD cases, the different temporal and spatial precisions were compared. From the finest time accuracy (day) and spatial accuracy (districts and counties) to a week and prefectures, respectively, linear regression, BP neural network and SVR are used to fit predictions. Because the average and variance of the number of cases in different time and space accuracy are very different, only the goodness of fit *R*<sup>2</sup> is used to compare the models. The experimental results are shown in Table 2, and the broken line graph is shown in Figure 11. With the expansion of time and space accuracy, the goodness of fit of the model doubles. At the same time, in the comparison of three different methods, the BP neural network prediction model has the largest *R*2, so the city-level weekly prediction model based on the BP neural network has the highest accuracy.

**Table 2.** Experimental results of different time precision and spatial precision prediction.


**Figure 11.** Comparison line chart of results of different prediction accuracy. The value represents results for R2, MAE/10, AWE for different parameters settings.

#### *6.4. Comparative Analysis of Forecasting Models*

In this paper, by training and forecasting numerical variables, based on the weekly incidence of HFMD in the city and the weekly weather data that affects the epidemic and the child population data, the relevant variable analysis of the influencing factors is carried out. After that, a multivariate joint feature selection method based on correlation analysis was used to screen out a subset of features suitable for building a linear model, including the weekly ordinal number of the statistical time, the incidence of the previous week, the weekly average air pressure, and the number of children aged 0–6. At the same time, a subset of features suitable for establishing a nonlinear model is obtained, including the weekly ordinal number of the statistical time, the incidence of the previous week, the weekly average air pressure, the weekly minimum temperature, the weekly air humidity, the wind level and the number of children aged 0–6. Three regression methods were used to establish a model to fit the weekly incidence. Under different evaluation indicators, the training results of each model are shown in Table 3 and Figure 12. The analysis shows that the *R*<sup>2</sup> of the BP neural network reaches the maximum and the MAE reaches the minimum. At the same time, when the prediction error does not exceed the MAE, the AWE reaches its maximum value. Therefore, the HFMD epidemic prediction model based on BP neural network performs best, and the fitting effect is relatively best.

**Table 3.** Training results of different machine learning regression algorithms.


**Figure 12.** Line chart comparing the training results of different machine learning regression algorithms. The value represents results for R2, MAE/10, AWE for different parameters settings.

#### *6.5. GA Tuning Parameter Analysis*

When optimizing the connection weights of the BP neural network prediction model with a good fit effect, it is necessary to adjust and compare the hyperparameters such as the number of individuals in the population, the number of generations, the crossover and mutation probability in the genetic algorithm. Therefore, in this section, the hyperparameters in Table 4 are adjusted from the default values, and the optimal value is selected as the result of this test after three tests under the same conditions. It is used to evaluate the comparison results of *R*2, MAE and AWE recording the changes of various hyperparameters, find the relevant hyperparameters of the model with the strongest generalization ability, and find the HFMD epidemic prediction model with the highest prediction accuracy based on these hyperparameters.


**Table 4.** Hyperparameters related to genetic algorithm in GA-BP neural network model.

6.5.1. Impact of the Number of Generations

The model evaluation results of adjusting the number of generations are shown in Table 5, and the line graph shown in Figure 13 is drawn accordingly.

According to Table 5 and Figure 13, when the number of generations reaches 90 times, the goodness of fit and the accuracy within error achieve the maximum value. When it is greater than or less than this value, these two indicators will become smaller and affect the prediction effect. At the same time, MAE reaches the minimum value, and increases whenever it is greater or less than this value. Therefore, the population evolution is iterated 90 times, and the model is optimal.


**Table 5.** Evaluation table for adjusting the number of generations of hyperparameters in GA-BP model.

**Figure 13.** Line chart of adjustment results of hyperparameters in GA-BP model. The value represents results for R2, MAE/10, AWE for different parameters settings.

#### 6.5.2. Effect of Population Size

The evaluation results of the model for adjusting the population size are shown in Table 6, and the line graph shown in Figure 14 is drawn accordingly.


**Table 6.** The population size adjustment evaluation table of the hyperparameters in the GA-BP model.

**Figure 14.** Line chart of adjustment results of hyperparameters in GA-BP model. The value represents results for R2, MAE/10, AWE for different parameters settings.

According to Table 6 and Figure 14, when the population size reaches 60, the goodness of fit and AWE take the maximum value. When it is greater than or less than 60, the value decreases. At the same time, MAE is the smallest, and when it is greater than or less than 60, its value will increase. Therefore, the model with a population size of 60 is optimal.

#### 6.5.3. Impact of Crossover Probability

The model evaluation results for adjusting the cross probability are shown in Table 7, and the line graph shown in Figure 15 is drawn accordingly.

It can be obtained from Table 7 and Figure 15 that when the crossover probability reaches 0.8, the goodness of fit and AWE achieve the maximum value, while MAE is the smallest. When the crossover probability is not 0.8, the relevant index values are not ideal. Therefore, the GA model with a crossover probability of 0.8 performs best.

**Figure 15.** Line chart of the adjustment results of the crossover probability of the hyperparameters in the GA-BP model. The value represents results for R2, MAE/10, AWE for different parameters settings.


**Table 7.** Adjustment evaluation table of cross probability of hyperparameters in GA-BP model.

#### 6.5.4. Impact of Gene Mutation Probability

The evaluation results of the model for regulating the probability of gene mutation are shown in Table 8, and the line graph shown in Figure 16 is drawn accordingly.

**Table 8.** Adjustment evaluation table of hyperparameter mutation probability in GA-BP model.


It can be obtained from Table 8 and Figure 16 that when the mutation probability is 0.04, the goodness of fit and AWE reach a good result, the maximum value is selected, and MAE is minimum. Higher or lower than this probability will make the relevant index

value worse. Therefore, the GA model with a mutation probability of 0.04 has the best performance and a better effect.

Through the adjustment and comparison of these four hyperparameters in GA, it is found that the individual population is 60, the number of evolutionary iterations is 90, the gene crossover probability is 0.8, and the gene mutation probability is 0.04. At this time, the neural network HFMD prevalence prediction model based on GA has the strongest generalization ability, the smallest error, and best results.

#### *6.6. Comparative Analysis of Early-Warning Models*

According to our experiment, we have built the GA-BP HFMD prediction model. We set the critical eigenvalue of HFMD outbreak and substitute the eigenvalue into the prediction model to obtain the first HFMD outbreak threshold. Then, the samples of test data were input into three early warning models, and the 80 percentile incidence of the region was calculated as the second HFMD outbreak threshold for the same period of 3 years and 2 weeks before and after the same period. Finally, the WR and ACR values of different warning models were counted, as shown in Table 9.

**Table 9.** Comparison of different early-warning models.


According to the comparison, we could see that although warning model based on adjustable parameters has the best ACR, its WR is too small, so it is not generalized. Therefore, we conclude that the warning-model based on threshold comparison should be the optimal one.

#### **7. Conclusions**

This paper proposes a prediction and early-warning model for HFMD and the model uses big data. Data used in this paper are patient data and weather data. We can obtain a more accurate early-warning effect by constructing integrated prediction and early-warning model based on big data.

This paper constructs the prediction model by using GA to optimize the BP neural network. The best prediction accuracy could be gain as 92.56%. Then we explores the various construction methods of early-warning model. Through the comparison of experiment results, it is found that the early-warning model based on the comparison of threshold has the highest accuracy. And the optimal accuracy of the early-warning method is around 87.28%.There are still many parts that can be optimized in the research of this paper. For example, we would want to add more factors to enhance the accuracy of the prediction model. We will continue to study in depth accordingly.

**Author Contributions:** Conceptualization, X.W. and Y.W.; methodology, Y.W.; software, X.D.; validation, L.J.; formal analysis, M.W.; investigation, M.W.; resources, H.G.; data curation, X.L.; writing original draft preparation, X.L.; writing—review and editing, X.Y.; visualization, X.L.; supervision, X.Y.; project administration, H.G.; funding acquisition, H.G., and X.Y. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work is supported by the National Science and Technology Major Project of China under Grant No. 2018ZX10201-002, and the National Natural Science Foundation of China under Grant No. 91846303.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable

**Conflicts of Interest:** The authors declare no conflict of interest.
