3.4.1. 10-Fold Validation Evaluation Method

The adopted ANN classification model was evaluated with a 10-fold cross validation, which divides the initial dataset to 10 equal sized parts. Then, in certain loops, it incorporates the first nine parts to train the classifier, and the remaining part to test the classifier. This process is repeated until each and every part has been used for training and testing.

#### 3.4.2. Prediction Accuracy Evaluation Metric

The effectiveness of the adopted classifiers was assessed by incorporating the prediction accuracy evaluation metric, *a* ∈ [0, 1], which is defined in Equation (1) as follows:

$$a = \frac{t\_P + t\_n}{t\_P + f\_P + t\_n + f\_n} \tag{1}$$

where, *tp*, indicates the instances correctly classified as positives, and *tn*, are the instances correctly classified as negatives. In addition, *fp*, signifies the instances which are falsely classified as positives, and *fn*, indicates the instances which are falsely classified as negatives. A low value of *a* implies that the classifier is weak, whereas a high value of *a* indicates that the classifier is efficient.

#### **4. Research Results**

Experiments were performed on the adopted dataset of 4.071 valid instances with the proposed Multi-Layer Perceptron ANN classification model. The ANN was finely tuned based on certain experimental parameters, which were evaluated with a 10-fold cross validation evaluation method. The results of the application of the ANN classification process for all the three class attributes were assessed with certain values of predictive accuracy. More specifically, the predictive values of accuracy are as follows: (1) C1 prediction accuracy a = 0.731761; (2) C2 prediction accuracy a = 0.900516; and (3) C3 prediction accuracy a = 0.803488. The results of predictive accuracy, *a*, are presented in Figure 3.

## **5. Discussion**

Every EU program undergoes extensive evaluation to assess its effectiveness and efficiency. Effectiveness could be related with the degree to which the predefined objectives of the program are met. Efficiency could be linked with the extent to which overall program outcomes relate to its costs, respectively (Michalek 2012). Program evaluations highlight useful lessons learned, which can be essential for the preparation of new initiatives; however, the existence of an ineffective monitoring system and the lack of modern evaluation techniques could make it difficult for the evaluators to conduct their assessments.

This paper utilizes an advanced AI technique related to ANNs in order to analyze the available data of a co-financed EU program. The main research objective is to predict the program's outcomes by means of its impact on the non-financial measures of the government body that materialized it. These measures are set within the context of the BSC and are related to the customers, the internal process, and the learning and innovation perspectives. Each and every perspective of the BSC operates with the following sequence: strategic objectives -> measures -> targets -> actions, as shown in Figure 1. For instance, the strategic objective that the government body set concerning the learning and innovation perspective was the digital transformation of the Greek SMEs. The action that contributed to the implementation of this strategic objective was the materialization of funding programs, such as those previously mentioned in the current study. The measure used to monitor the effectiveness of the aforementioned action was the number of Greek SMEs owing an e-shop.

Bearing in mind that the perspectives of the BSC are interconnected with each other (Kaplan and Norton 1992), it is presupposed that the results of one perspective have an impact on the others. As previously mentioned, the companies that participated in the program, and which are under study, underwent three evaluation phases: (1) close examination of the basic prerequisites for participation in the program; (2) approval for receiving funding; and (3) total disbursement of funding. As shown in Figure 1, these are also the actions that contributed to the realization of the financial strategic objectives that the government body had set. Consequently, data retrieved from the financial perspective are related to the materialization of the funding program, and have been utilized to predict the results of the remaining perspectives.

The research question is whether someone, knowing the results of the actions taken by the government body for the realization of the financial strategic objectives, could predict the level of achievement of the remaining three perspectives. The answer to this research question could improve the decision-making process and facilitate change management, by altering the funding criteria or the financing areas if needed.

AI which is utilized in the present research effort is an empirical science. This implies that the number of hidden layers and the number of nodes in each hidden layer are related to the predictive accuracy and the total performance of the model. More specifically, the greater the number of hidden layers and nodes within each layer, the greater the predictive accuracy, which increases until a certain convergence threshold. As previously mentioned, this is defined by trial and error according to the adopted experimental parameters; however, after a certain level, there is a decrease in predictive accuracy, no matter how many hidden layers and nodes continue to be incorporated. In addition, predictive accuracy not only depends upon the number of hidden layers and nodes of the ANN, but also upon the quality of the provided dataset. Moreover, a classification model is effective for certain data if it achieves high predictive accuracy concerning the fine tuning of the experimental parameters of the ANN model. Finally, this process depends not only on quality, but it is also connected to the quantity of the training data. Such data are preprocessed accordingly in order to remove outliers, missing values, and redundant data, which could affect the effectiveness of the adopted model.

The predictive accuracy of the presented model was 73% for customer satisfaction, 90% for submitting proposals to other programs, and 80% for owing an E-shop. These predictive accuracies are considered efficient, bearing in mind that the provided data were analyzed and produced by multimodal human activity in complicated business environments. Such environments are characterized by a high degree of volatility; thus, the final results are important and may have a significant impact upon the decision-making process. This study indicated that the government body which materialized the program could apply changes to the funding process and improve the predicted results of the customer perspective. The results for the remaining perspectives seem to satisfy the targets that were set in the initial BSC.

The limitations of the present study should also be noted. As mentioned in the introduction, this paper examines the program's outcomes and efficiency from the viewpoint of the government body that materialized it. Additionally, the strategic objectives of the BSC should be revised on a yearly basis. In the current study, it has been assumed, for experimental purposes, that the strategic objectives have remained the same over the years. Moreover, the available data for the experiment is concerned with a single measure for each and every strategic objective; however, the intricate relationships among the four perspectives could be analyzed by exploiting all the measures for all perspectives put forward. Finally, the provided dataset consists only of binary variables, which is also a limitation of the current study.

#### **6. Conclusions**

The evaluation of EU programs is essential because useful insights are provided on whether the programs met their initial objectives. This process also helps policy makers to redesign programs and make them more efficient by optimizing the absorption of the invested funds. The existence of an effective monitoring system and a modern evaluation technique should help the evaluators to conduct their assessments. This study integrated the BSC and AI to predict the outcomes of a co-financed EU program by means of its impact on the non-financial measures of the government body that materialized it. The predictive accuracy of the model developed in this research effort is considered efficient, taking into account the complexity of the business environment in which the provided data were produced. The results indicated that corrective actions could be addressed by the government body which implemented the funding program in order to improve the outcomes of customer satisfaction. The utilization of the proposed model could improve the decision-making process and initiate changes to administrational issues in the available funding programs. Future research will be centered upon predicting program results more holistically by incorporating more variables related to the BSC.

**Author Contributions:** Conceptualization, A.P. and I.S.; methodology, I.S. and L.V.; software, T.A.; validation, T.A. and Y.P.; formal analysis, T.A.; investigation, A.P.; resources, A.P.; data curation, T.A.; writing—original draft preparation, A.P.; writing—review and editing, T.A.; visualization, A.P.; supervision, I.S., Y.P. and L.V.; project administration, I.S., Y.P. and L.V.; funding acquisition, I.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the course of Advanced Quantitative Statistics offered by the Master of Business Administration (MBA) of the Department of Business Administration at the University of West Attica, Greece.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data presented in this research are part of a PhD study and are available upon request from the corresponding author.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**

The following abbreviations are used in this manuscript:


## **Appendix A**



#### **References**


Wang, Dayong, Aditya Khosla, Rishab Gargeya, Humayun Irshad, and Andrew H. Beck. 2016. Deep Learning for Identifying Metastatic Breast Cancer. *arXiv* arXiv:1606.05718.

Wirtz, Bernd W., Jan C. Weyerer, and Carolin Geyer. 2019. Artificial Intelligence and the Public Sector—Applications and Challenges. *International Journal of Public Administration* 42: 596–615. [CrossRef]
