Adaptive Neural Architecture Search Using Meta-Heuristics: Discovering Fine-Tuned Predictive Models for Photocatalytic CO₂ Reduction

Abstract

This study aims to contribute to the reduction of carbon dioxide and the production of hydrogen through an investigation of the photocatalytic reaction process. Machine learning algorithms can be used to predict the hydrogen yield in the photocatalytic carbon dioxide reduction process. Although regression-based approaches provide good results, the accuracy achieved with classification algorithms is not very high. In this context, this study presents a new method, Adaptive Neural Architecture Search (NAS) using metaheuristics, to improve the capacity of ANNs in estimating the hydrogen yield in the photocatalytic carbon dioxide reduction process through classification. The NAS process was carried out with a tool named HyperNetExplorer, which was developed with the aim of finding the ANN architecture providing the best prediction accuracy through changing ANN hyperparameters, such as the number of layers, number of neurons in each layer, and the activation functions of each layer. The nature of the NAS process in this study was adaptive, since the process was accomplished through optimization algorithms. The ANNs discovered with HyperNetExplorer demonstrated significantly higher prediction performance than the classical ML algorithms. The results indicated that the NAS helped to achieve better performance in the estimation of the hydrogen yield in the photocatalytic carbon dioxide reduction process.

1. Introduction

Today, a large proportion of energy is produced using fossil fuels. This results in high carbon emissions and environmental pollution. The high level of carbon dioxide (CO₂) in the atmosphere is one of the main causes of global warming. There is an urgent need to reduce dependence on fossil fuels and focus on renewable energy. In this context, hydrogen is considered to be a clean and renewable substance that can act as a perfect energy carrier. Photocatalysis is a technique that utilizes sunlight to aid chemical reactions, such as the conversion of CO₂ into renewable substances such as hydrogen. The technique aids in reducing the carbon levels in the atmosphere, while increasing the production of clean energy carriers, such as hydrogen. Hydrogen production through the photocatalytic reduction of CO₂ is an active research field. In a photocatalytic CO₂ reduction process, the amount of outputs (such as hydrogen yield) can be estimated using predictive methods, such as Machine Learning (ML). Although several machine learning methods have been implemented and tested for hydrogen production (such as in [1], Ren et al. [2], Mageed [3], Ramkumar et al. [4], and Yurova et al. [5]) and significantly good results were achieved using the regression-based approach, the predictive ability of these methods has remained limited in studies conducted with classification methods. On the other hand, artificial neural networks have very flexible architecture, which provides options for improvement. In this study, we explore whether ANN architectures, which provide high classification accuracy in the prediction of gas (hydrogen) yield as a result of photocatalytic CO₂ reduction process, can be discovered through an adaptive Neural Architecture Search (NAS). Experiments were conducted in our study in parallel with this aim, and the results of the experiments demonstrated that ANN architectures with significantly higher prediction performance than those of the classical ML algorithms can be discovered. The implementation of these newly discovered architectures in practice will aid in classifying the (hydrogen) yield as a result of the photocatalytic CO₂ reduction process with a high accuracy. The following section presents our findings on the literature review; this paper continues with a section on the methodology and concludes after the presentation of results and discussion.

Literature Review

Increasing amounts of production with industrial development cause rapid consumption of natural resources, while causing increased carbon emissions, environmental pollution, and the destruction of nature by production-oriented industries [6]. With the rapid increase in the world’s population, the energy demands of societies have grown exponentially, and the per capita energy consumption rate and the atmospheric CO₂ level have reached alarming levels. Today, about 85% of energy needs are met by fossil fuels, including oil, coal, and natural gas [7]. If energy consumption continues in this way, it is predicted that a serious energy crisis will be faced soon. To prevent this crisis, it is necessary to reduce dependence on fossil fuels [8]. Intense carbon emissions cause an increase in greenhouse gas emissions in the atmosphere, leading to global warming and adverse changes that pose a threat to ecosystems [9]. Production based on fossil resources is not sustainable and negatively affects independence [10]. The use of nonrenewable energy sources, which are used to meet energy needs, causes irreparable harm to the environment. As a result of the use of these energy sources, many negative consequences, such as air pollution, water pollution, global warming, and acid rains, occur. Furthermore, the reserves of these resources are limited. For this reason, it has become an inevitable necessity to develop renewable energy sources that are less harmful to the environment and techniques for obtaining energy from these sources. Energy is considered one of the most important indicators of development. In this changing and developing world, the importance of energy is increasing day by day. Energy resources are rapidly depleting, energy costs are increasing, and environmental awareness is developing in societies. To meet energy demand and protect the environment, developing and developed countries are moving to renewable energy sources [11]. Renewable energy resources, which represented 7.7% of total energy consumption in the world in 2005, are expected to increase to 8.5% in 2030 [12]. Among these energy sources, hydrogen energy, which is environmentally friendly and abundant in nature, has come to the fore.

Table 1. Average monthly global atmospheric CO₂ concentration measured in parts per million (ppm) [13].

Year	CO2 Concentration (ppm)
2014 (December)	399.08
2015 (December)	402.06
2018 (December)	409.27
2019 (December)	411.98
2022 (December)	418.99
2023 (December)	421.86
2024 (June)	426.91

shows the global atmospheric CO₂ concentration.

As shown in Table 1, CO₂ emissions increase year by year. Greenhouse gas emissions, i.e., CO₂ emissions caused by the use of fossil fuels, trigger risks of climate change [14]. According to the scenarios of the Intergovernmental Panel on Climate Change (IPCC), the rates of greenhouse gases, such as carbon dioxide and methane, in the world’s atmosphere have increased in parallel with human activities and social–economic development since the industrial revolution. Global warming will exceed the 1.5 °C limit between 2030 and 2052 if greenhouse gas emissions continue [10]. This clearly shows that it is necessary to change the current approaches to energy production and consumption. For this reason, hydrogen production through CO₂ reduction is an issue that should be taken seriously.

Hydrogen contains high energy and carries energy, is clean, renewable, and convertible to other forms of energy, and since it can be stored, it reduces the problem of energy storage [15,16]. The main energy sources in nature are primary energy sources. Secondary energy sources obtained by converting primary sources to different physical states are called energy carriers [17]. The fact that hydrogen carries energy indicates that it can be produced using primary energy sources [18]. Hydrogen is the main component of water that covers 60% of our planet. Hydrogen exists in different forms in living things and fossil fuels [15]. Hydrogen gas was produced by Henry Cavendish in 1766. Seventy-three years later, in 1839, William Robert Grove developed fuel cells using hydrogen [19]. Components in which hydrogen is used as a fuel and chemical energy is directly converted into electrical energy are called fuel cells [20]. The production process and the use of hydrogen are emission-free. Commercial hydrogen production is carried out with a high percentage of fossil materials. When we look at the world’s hydrogen production at the end of 2021, it can be seen that 48% is from natural gas, 30% is from refinery products, 18% is from coal, and 4% is from water [21].

Although water consists of two hydrogens and one oxygen, fossil fuels are composed of hydrogen and carbon. The greatest difficulty in obtaining hydrogen is its separation from other compounds [22].

On a global scale, although there are concerns about energy consumption, energy is actually transformed, rather than consumed. What has changed is the raw energy materials, which are formed in certain geological periods and are produced and consumed in different ways and used in energy production [23].

The increase in the processing power of computers has led to the emergence of concepts such as optimization, machine learning, and artificial intelligence. Hydrogen energy and photocatalytic CO₂ reactions with big data have become very suitable topics for optimization and machine learning. Machine learning is used in many different fields, such as civil engineering [24,25,26,27,28,29,30], machine vision [31], optics [32], and biomedicine [33]. There are many studies that have used optimization and artificial intelligence in hydrogen energy and photocatalytic CO₂ reactions.

Ren et al. [2] developed a model for photocatalysis to produce maximum hydrogen and used Gaussian Process Regression (GPR) to find the optimal parameters. In order to increase the success of the model, Whale Optimization (WO) and Simulated Annealing (SA) algorithms were applied, and successful results were obtained. Saadetnejad et al. [1] applied Random Forest to predict the band gap while investigating the gas phase of photocatalytic CO₂ reduction for hydrogen production and obtained low error values. The Decision Tree applied for total hydrogen production also showed 80% success for the gas phase. Mageed [3] utilized two different ANNs, the Support Vector Machine (SVM), the Nonlinear Regression Model (NLM), and the Response Surface Model (RSM) to model hydrogen production. According to the results, ANN had a better prediction rate (R = 0.998), compared with the other models. Yan et al. [34] associated g-C3N4 and the hydrogen production rate using machine learning and demonstrated the success of the ML models. Xu et al. [35] used machine learning to explore alternative copolymers. Haq et al. [36] used different machine learning models and the Genetic Algorithm (GA) to predict the hydrogen yield for supercritical water gasification. Based on the results obtained, this ML-based method achieved an accurate estimate of hydrogen yield (R² = 0.997). Ramkumar et al. [4] predicted the hydrogen production value by photolysis using ANN. As a result of their study, they reached an accuracy value of 91.66%. Yurova et al. [5] used the ML models Gradient Boosting, Multilayer Perceptron, and Random Forest to predict the efficiency of photocatalytic hydrogen production. The models showed high prediction success with R² values greater than 0.9.

In recent years, photocatalytic technology has started to become an emerging topic to tackle environmental pollution [37]. To understand photocatalytic systems, it is necessary to mention the concept of photocatalysis. Photocatalysis is a combination of the words photo and catalysis, which means accelerating a chemical reaction. In other words, photocatalysis is the acceleration of a chemical reaction (photoreaction) that occurs under the influence of light (UV rays or sunlight) in the presence of a catalyst [38]. The photocatalysis method is economical, highly efficient, practical, and easy to implement [39]. On the basis of this phenomenon, photocatalytic systems have been developed. Photocatalysis is a photocatalytic technique [40]. The photocatalysts used generally show activity under ultraviolet (UV) irradiation, and various strategies have been developed to reduce the bandwidth, as the high bandwidth makes them not active enough under the visible region or sunlight. Among these, techniques such as the addition of different metal or nonmetal additives, surface modification, formation of composites with another semiconductor, and modification with carbon-based materials are preferred [41]. With the photocatalytic technique, pollutants can be completely broken down. The removal of the catalyst from the reaction medium and poor chemical stability caused by the large surface area are serious problems that can occur. Better photocatalytic activity is obtained when a semiconductor with lower band-gap energy is selected [42]. Figure 1 represents the proposed mechanism for a photocatalytic reaction.

The dataset used in this study is related to the photocatalysis process. Many semiconductors, such as TiO₂, rG, SiC, GO, lnTaO₄, Ta₂O₅, CeO₂, ZrO₂, CaFe₂O₄, BaLa₄Ti₄O₁₅, CdS, ZnO, ZnTe, C₃N₄, TiO₂BP, Zn₂Ti₃O₈, PbBiO₂Br, CeO₂, Fe₂O₃, SrTiO₃, ZnS, SrNb₂O₆, TaON, BiVO₄, and ZnGa₂O₄, were the photocatalysts in the dataset (which will be explained in detail in the following section). One of the most used semiconductors in the dataset was TiO₂. TiO₂ has an appropriate band gap energy, high photocatalytic activity, is chemically and biologically inert, low-cost, mechanically durable, and non-toxic [44]. TiO₂ is a widely used semiconductor in many scientific studies [45,46].

2. Materials and Methods

2.1. The Dataset

In this study, the dataset provided by Saadetnejad et al. [1] was used for model evaluation. This dataset consisted of 549 data points. In the compilation of the dataset, the data were collected from 80 studies that focused on the photocatalysis (photocatalytic CO₂ reduction) process. The aim of this process was to produce hydrogen. The dependent variable in the dataset, namely total gas yield, indicated hydrogen yield as a result of photocatalysis. The aim of our study was to facilitate and improve the estimation and classification of the total gas yield (hydrogen production) of the photocatalysis process through a neural architecture search. Details of the dataset and the collected references to the data can be found in [1] for further exploration. At the beginning of our study, the source dataset [1] was divided into two datasets according to the gas and liquid phases of the photocatalysis process. The number of rows was 549 in the source dataset; after the separation of the source dataset into two new datasets, the gas phase dataset contained 268 records and the liquid phase dataset contained 281 records. The metadata of these two datasets (before the data preprocessing stage) are provided in

Table 2. Gas-phase dataset.

Gas-Phase Variables	Range
Light	Vis, UV, UVVis
Temperature (K)	278–473
Pressure (bar)	0.7–4
Feed Molar Ratio	0.2–97.7
Additive	None, Na2CO3, NaHCO3, KHCO3, K2CO3, H2SO4
Amount of Additive (mol/L)	0–0.5
Semiconductor Preparation Method	None, Thermal method, Sol gel, Hydrothermal, Chemical reduction, Commercial, Solid state, Ion Exchange
Co-catalyst Preparation Method	None, Thermal method, Sol gel, IW, Deposition-precipitation, Hydrothermal, Photodeposition, ALD, Solid state
CalcTemp (K)	298–1223
Co-catalyst 1 Loading, wt%	0–100
Co-catalyst 1	None, Pd, Cu, Ag, Pt, Co, V, Cr, Ru, Au, Ag, MgO, ln2O3, CuO
Co-catalyst 2 Loading, wt%	0–2
Co-catalyst 2	No, Fe, Cu
Dopant1/Semiconductor1 Ratio	0–90
Dop1	No, Fe, Mg, Cu, V, W
Dopant2/Semiconductor1 Ratio	0–5
Dop2	No, W
Semiconductor1/Semiconductor2 Ratio	2.3–100
Semiconductor 1	C3N4, CeO2, GO, SrTiO3, TiO2, ZnS, Zn2Ti3O8, ZrO2, GO
Semiconductor 2	None, SiO2, TCPP, Modified Zeolite, ZnTiO3, ZnFe2O4
Band Gap Energy (eV)	1.54–3.08
Total Gas Yield (µmol/gcat/h)	0.01–310

for the (new) gas-phase dataset and

Table 3. Liquid-phase dataset.

Liquid-Phase Variables	Range
Light S.	UV, Vis
Temperature (K)	278–973
Pressure (bar)	0.25–7.5
Feed Molar Ratio	No
Additive	No, K2CO3, NaOH, C3H8O, CH3CN, TEOA, NaHCO3, KHCO3
Amount of Additive (mol/L)	0–6
Semiconductor Preparation Method	Sol gel, Chemical, Solid state, Hydrothermal, None, Commercial, Polymerizable complex method, Thermal methods
Co-catalyst Preparation Method	None, Sol gel, Chemical, Photodeposition, IW, Freeze-drying, Thermal methods, Hydrothermal, Solid state, Deposition-precipitation
CalcTemp (K)	298–1473
Co-catalyst 1 Loading, wt%	0–175
Co-catalyst 1	None, NiO, Ag, Pd, Cu, Ru, CPDs, Au, Co, AgBr, Pt, Ru (II) dinuclear complex, Bi2S3, Eu
Co-catalyst 2 Loading, wt%	0–1
Co-catalyst 2	None, Co, Fe, Ni, Cu, Ru(bpy3)+2, Ru (II) dinuclear complex
Dopant1/Semiconductor1 Ratio	0–25.22
Dop1	None, S
Dopant2/Semiconductor1 Ratio	0
Dop2	No
Semiconductor1/Semiconductor2 Ratio	0–100
Semiconductor 1	TiO2, rG, SiC, lnTaO4, Ta2O5, CaFe2O4, BaLa4Ti4O15, CdS, ZnO, ZnTe, C3N4, BP, PbBiO2Br, CeO2, Fe2O3, NiO, SrNb2O6, TaON, BiVO4, rGO, ZnS, ZnGa2O4
Semiconductor 2	ACF, CuI, Ta2O5, SiO2, WO3, ZnTe, Cu2O, Fe2O3,NiO, SiO2, rGO, CdS, Modified Molecular Sieve, CeO2
Band Gap Energy (eV)	1.6–4.67
Total Gas Yield (µmol/gcat/h)	0.01–840

for the (new) liquid-phase dataset.

For both datasets, the dependent variable bins were chosen within the same ranges given in [1], and 3 classes were defined as A, B, and C to denote these bins. With this approach, the dependent variable (total gas yield) was transformed into a categorical variable instead of a continuous one. The ranges of values for each class of the dependent variable are given in

Table 4. Range and instance numbers of classes A to C for the dependent variable for the gas dataset.

Class	Total Gas Yield (µmol/gcat/h)
A	6.97–310
B	0.75–6.97
C	0.01–0.75

and

Table 5. Range and instance numbers of classes A to C for the dependent variable for the liquid dataset.

Class	Total Gas Yield (µmol/gcat/h)
A	27.9–840
B	4.05–27.9
C	0–4.05

.

2.2. Data Preprocessing

Real-world data from different sources can be noisy, inconsistent, and incomplete. Therefore, raw data need to be converted to a suitable format for analysis. In this step, the data are prepared for the analysis phase by cleaning, merging, transformation, and reduction. Operations such as data cleaning, data transformation, and data reduction used in data preprocessing may affect the success of the ML model. This step in ML is called data preprocessing and includes all actions taken before modeling [47]. As not applying preprocessing can cause unsuccessful results, it is important to increase data quality and prepare for data mining by preprocessing the data to obtain high accuracy and reliability [48]. The data preprocessing methods (data transformation and missing value imputation) used in this study are shown in Figure 2.

For the data preprocessing stage and for training the ML models (in the last stage) of this study, Python 3.10.11 [50] Anaconda3 [51] environment, Spyder 5.2.2. [52] interface, NumPy [53] and Pandas [54] libraries, and Scikit-Learn [55] libraries are used.

2.2.1. Data Transformation

Since machine learning algorithms do not work directly on categorical data (in the form of string), the data must be converted into numerical data. In this study, One Hot Encoding was used to convert categorical values into numerical values. Categorical data (non-numeric) are not used directly in machine learning algorithms. Before the data can be used, they must be expressed in numerical form. The One Hot Encoding (OHE) method is generally used for the numerical encoding of categorical data [56]. Each integer value is represented as a binary vector with all zero values except for the integer index marked with 1, as shown in Figure 3.

For the variables with the numerical data, Standard Scaler of the Scikit-Learn [55] was used to scale the variables.

2.2.2. Data Imputation

The reason for the missing values in the data set is that people do not share the information during observation, the relevant information is not entered when collecting the data, and there are systemic errors or data losses [58]. The Scikit-learn machine learning library of Python includes the SimpleImputer [55] class, which supports imputation by taking the most frequent or mean values of the variable (column).

We observed missing values in two fields of the dataset, which were “Band Gap Energy” for the gas- and liquid-phase datasets and “Feed Molar Ratio” for the gas-phase dataset. The data of both these fields were numerical. Due to two reasons, (i) the missing values of this variable being predicted using the other fields of the dataset [1] (i.e., the type of variable cannot be considered as a fully independent variable after this), and (ii) there being a high ratio of synthetically generated data (i.e., 1/3) [1], we chose to exclude this field from our input dataset. For the “Feed Molar Ratio” field, where the missing data ratio is 12.3%, it is advised by the literature that the data with a missing ratio of over 10% should be imputed; in addition, the data with a missing ratio between 5% and 10% can be considered or evaluated for imputation [59,60]. We wrote software code (available upon request) to test if the missing data followed the MAR, MCAR, or MNAR pattern, and we observed that the data followed a MCAR pattern; we chose to use the mean imputation for this field. In the Scikit-Learn SimpleImputer class, the default imputation strategy is mean if the values of a field are numerical, and this strategy is implemented for the “Feed Molar Ratio” field.

At the end of this method, all features in the data set were converted to numerical features.

Table 6. Sizes of one hot encoded input variables.

Gas Phase Dataset		Liquid Phase Dataset
Variable Name	Size	Variable Name	Size
Light S.	2	Light S.	1
Additive	5	Additive	7
Semiconductor Preparation Method	7	Semiconductor Preparation Method	7
Co-catalyst Preparation Method	9	Co-catalyst Preparation Method	9
Co-catalyst 1	12	Co-catalyst 1	13
Co-catalyst 2	2	Co-catalyst 2	6
Dop1	5	Dop1	1
Semiconductor 1	7	Semiconductor 1	23
Semiconductor 2	6	Semiconductor 2	13

presents all input variables and their sizes.

Following the data preprocessing stage, the size of the gas-phase dataset was 67 and the size of the liquid-phase dataset was 90. In the next stage, a new tool developed by the authors was utilized to discover the most accurate ANN architecture for the prediction of the total gas yield in a photocatalytic CO₂ reduction process.

2.3. Adaptive NAS and HyperNetExplorer

Neural Architecture Search (NAS) has emerged as a subfield of automated machine learning (AutoML) for discovering the most accurate ANN architecture by searching architectural hyperparameters, such as the number of layers, number of neurons, and activation functions. The search can be conducted with traditional methods, such as grid search or random search, or adaptive methods (i.e., methods that adjust search based on ongoing results), such as optimization algorithms. In this research, we first utilized HyperNetExplorer, developed by our research group. HyperNetExplorer is a web-based NAS tool that aims to find an ANN architecture that provides the most accurate classification/regression (ML) model for a given dataset. This section will detail how HyperNetExplorer was developed and operated. HyperNetExplorer depends on ANNs and uses several optimization algorithms from the MealPy [61] package for hyperparameter optimization. The following sections provide information on ANN architectures, hyperparameter optimization algorithms, validation, and performance evaluation strategies, and finally elaborate on the HyperNetExplorer architecture.

2.3.1. Artificial Neural Networks (ANNs)

Artificial neural networks are computational networks that try to completely mimic the nerve cell networks of the nervous system and take samples from the human brain [62]. Artificial neural networks, like humans, gain experiences, store them in memory, and use their experiences. Artificial neural networks started with detailed studies of the human brain by scientists in the late 1800s. The artificial neural network model was first introduced in 1943 by Warren McCulloch and Walter Pitts [63]. Artificial neural networks can work only with numerical information. The symbolic expressions used must be converted into numerical values [64]. The smallest units that form the basis of an ANN’s work are called artificial neural cells or processing elements. Figure 4 shows the simplest artificial neural cells of 5 main components: inputs, weights, sum function, activation function, and output.

Inputs (x₁, x₂, x₃, … x_n) are information entering the cell from other cells or external environments. These are determined by the examples the network is asked to learn. Weights (w₁, w₂, w₃, … w_n) are values that express the effect of an input set or another processing element in a previous layer on this processing element. Each input is aggregated through the sum function, multiplied by the weight that connects that input to the processing element [66]. Since artificial neural networks are required to learn non-linear states, the activation function is applied to the outputs of artificial nerve cells. Artificial neural networks work against nonlinear results when the output of artificial nerve cells is active [67]. Activation functions provide coupling between input and output units. When the activation function is selected correctly, network performance is severely affected [68]. Figure 5 shows activation function graphs.

Sigmoid/LogSigmoid Activation Function: The sigmoid activation function, which is frequently used in classification problems, transforms incoming inputs into outputs by generating between 0 and 1, as shown in Figure 5a [69]. The logistic sigmoid function (logSigmoid) is a type of sigmoid function, which produces outputs on scale (−∞, 0).

Tangent hyperbolic activation function (TanH): The tangent hyperbolic activation function is very similar to the sigmoid function, but the output range of the function is between −1 and +1. As can be seen in Figure 5b, the outputs with values in the [−1, 1] range are zero-centered, thus facilitating optimization, and this function is preferred more often than the sigmoid activation function [69].

Rectified Linear Unit Function (ReLU): The ReLU activation function converts the incoming inputs into outputs between 0 and +∞, as shown in Figure 5c. Therefore, it is called a rectified linear function and can be quickly calculated [69].

Leaky Rectified Linear Unit Function (Leaky ReLU): Since ReLU directly equates negative values to zero (Figure 5c), the units in the layers do not become active in cases where there are many negative values. To avoid this situation, the Leaky-ReLU activation function was proposed. As shown in Figure 5d, in order to add slope to the function at negative values, a small constant value, such as 0.01, is chosen for the leakage value in the Leaky-ReLU method [69].

Mish Activation Function: The Mish activation function (Figure 5e) presents a new and nonmonotonic approach to other functions. It performs well with the standard activation function [70].

Exponential Linear Unit (ELU): The exponential linear unit (Figure 5f) is similar to ReLU, except for the negative inputs. For negative input, the alpha parameter is usually taken [71].

2.3.2. Algorithms for Hyperparameter Optimization

In this study, different optimization algorithms were used for ANN hyperparameter optimization. These algorithms were the Flower Pollination Algorithm (FPA), Genetic Algorithm (GA), Harmony Search (HS), Jaya Algorithm (JA), Particle Swarm Optimization (PSO), and Teaching–Learning-Based Optimization Algorithm (TLBO). Information about these algorithms is given below. A comparison table for the algorithms is given in Table 7.

Flower Pollination Algorithm (FPA): FPA is an algorithm inspired by the flower reproduction process and can mimic pollination in plants to achieve the best result in the shortest time for solving global optimization problems. The main goal of flower pollination is to ensure optimal vigor and an optimal biological reproduction stage. Pollination and other factors interact optimally to reproduce plants [72].

Genetic Algorithm (GA): GA is a search and optimization algorithm in which evolutionary processes are imitated [73]. The genetic algorithm achieves the best results in the search space due to selection, crossover, and mutation, as in evolutionary processes [74].

Harmony Search (HS): The HS algorithm is a music-based algorithm and mimics the path musicians take to find the most compatible harmony. The fitness value is determined by assigning initial values to the solution parameters of the investigated problem. Good results are retained in the harmony memory, while the worst results are removed from the memory. This process continues until the best one is found [75].

Jaya Algorithm (JA): JA is an algorithm named Jaya, a Sanskrit word that means victory, because it tries to achieve victory by reaching the best solution [76]. The algorithm constantly tries to approach good solutions to achieve success and to stay away from bad solutions to avoid failure.

Particle Swarm Optimization (PSO): Particle Swarm Optimization (PSO) is a population-based algorithm inspired by the swarm behavior of bird flocks and fish schools during foraging and/or escaping from danger. In PSO, decision variables do not need to take a special initial value to solve the problem. Since the PSO is population-based, it is possible to search for global solutions in multiple directions [77].

Teaching–Learning-Based Optimization Algorithm (TLBO): The TLBO algorithm is a population-based algorithm. The algorithm is based on the teaching–learning process. In the proposed algorithm, each candidate solution is expressed by a set of variables representing the results of a student. The process in which students try to improve their results by receiving information from the teacher is called the teacher phase. It is also the student phase in which they improve their performance by interacting with other students [78].

Table 7. Comparison table for algorithms.

Algorithm	Advantages	Disadvantages	Types of Optimization Problems Best Suited for	Convergence Rates
FPA	Simplicity and flexibility	Inadequate optimization precision [79]	Large integer programming problems, high-complexity convergence problems	Poor
GA	Parallel capabilities and flexibility	Difficult interpretation of solutions	Timetabling and scheduling problems	Slow
HS	No need for derivative information [80]	Stuck at the local optimum	Multi-objective optimization problems	Fast
JA	Simplicity, efficiency, and no algorithm-specific parameters	Get stuck in local minima	Large -scale real-life urban traffic light scheduling problems	Medium
PSO	Flexible and easy to implement	Easy stuck to local optimum	Constrained and unconstrained (one or more objective) optimization problems	Slow
TLBO	Powerful exploration	Suboptimal performance	Complex optimization problems	Fast

Table 7. Comparison table for algorithms.

Algorithm	Advantages	Disadvantages	Types of Optimization Problems Best Suited for	Convergence Rates
FPA	Simplicity and flexibility	Inadequate optimization precision [79]	Large integer programming problems, high-complexity convergence problems	Poor
GA	Parallel capabilities and flexibility	Difficult interpretation of solutions	Timetabling and scheduling problems	Slow
HS	No need for derivative information [80]	Stuck at the local optimum	Multi-objective optimization problems	Fast
JA	Simplicity, efficiency, and no algorithm-specific parameters	Get stuck in local minima	Large -scale real-life urban traffic light scheduling problems	Medium
PSO	Flexible and easy to implement	Easy stuck to local optimum	Constrained and unconstrained (one or more objective) optimization problems	Slow
TLBO	Powerful exploration	Suboptimal performance	Complex optimization problems	Fast

The complexity of algorithms can be expressed mathematically in Big-O Notation. According to the Big-O Notation, the letter O (i.e., function’s order) indicates the asymptotic upper bound of a function’s growth rate [81]. The growth rate describes how quickly the runtime or space increases as the input size N grows.

Table 8. Complexity of algorithms (Big-O Notation).

Algorithm	Temporal Complexity	Spatial Complexity
FPA	O(T⋅N⋅f(D)) [82]	O(N⋅D)
GA	O(T⋅N⋅(f(D) + log N + D)) [83]	O(N⋅D)
HS	O(N⋅D + T⋅(D + f(D) + N)) [84]	O(N⋅D)
JA	O(T⋅N⋅f(D)) [85]	O(N⋅D)
PSO	O(T⋅N⋅f(D)) [86]	O(N⋅D)
TLBO	O(T⋅N⋅f(D)) [87]	O(N⋅D)

illustrates the equations for the temporal and spatial complexity of the algorithms.

In Table 8, T is the number of generations/iterations, N is the population/swarm size, and D is the problem dimensions. (For our case T = 50, N = 20, D = 7). The temporal complexity ranking of these algorithms is provided in Equation (1).

$$\mathrm{G}\mathrm{A}:\mathrm{O}(\mathrm{T}·\mathrm{N}·\mathrm{f}\left(\mathrm{D}\right)+\log \left(\mathrm{N}\right)+\mathrm{D}>\mathrm{H}\mathrm{S}:\mathrm{O}(\mathrm{N}·\mathrm{D}·+\mathrm{T}·(\mathrm{D}+\mathrm{f}\left(\mathrm{D}\right)+\mathrm{N}>\mathrm{F}\mathrm{P}\mathrm{A},\mathrm{T}\mathrm{L}\mathrm{B}\mathrm{O},\mathrm{J}\mathrm{A},\mathrm{P}\mathrm{S}\mathrm{O}:\mathrm{O}(\mathrm{T}·\mathrm{N}·\mathrm{f}\left(\mathrm{D}\right))$$

(1)

For hyperparameter optimization in ANNs, f(D) can be denoted as Equation (2).

$$\mathrm{f}\left(\mathrm{D}\right)=\mathrm{O}(\mathrm{E}\times \mathrm{B}\times \mathrm{M}\times \mathrm{W}\left(\mathrm{D}\right))$$

(2)

In Equation (2), E is the number of epochs, B is the batch number, M is the dataset size, and W(D) is the number of total connections in the ANN (which changes in every iteration of the algorithm, based on the selected combination of hyperparameters). Given these variables, the temporal complexities of the algorithms can be re-written as Equations (3)–(8):

$$\mathrm{F}\mathrm{P}\mathrm{A}:\mathrm{O}(\mathrm{T}·\mathrm{N}·\mathrm{E}·\mathrm{B}·\mathrm{M}·\mathrm{W}\left(\mathrm{D}\right))$$

(3)

$$\mathrm{G}\mathrm{A}:\mathrm{O}(\mathrm{T}·\mathrm{N}·(\mathrm{E}·\mathrm{B}·\mathrm{M}·\mathrm{W}\left(\mathrm{D}\right)+\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{N}+\mathrm{D}$$

(4)

$$\mathrm{H}\mathrm{S}:\mathrm{O}(\mathrm{N}·\mathrm{D}+\mathrm{T}·\left(\mathrm{D}+\mathrm{E}·\mathrm{B}·\mathrm{M}·\mathrm{W}\left(\mathrm{D}\right)+\mathrm{N}\right))$$

(5)

$$\mathrm{J}\mathrm{A}:\mathrm{O}(\mathrm{T}·\mathrm{N}·\mathrm{E}·\mathrm{B}·\mathrm{M}·\mathrm{W}\left(\mathrm{D}\right))$$

(6)

$$\mathrm{P}\mathrm{S}\mathrm{O}:\mathrm{O}(\mathrm{T}·\mathrm{N}·\mathrm{E}·\mathrm{B}·\mathrm{M}·\mathrm{W}\left(\mathrm{D}\right))$$

(7)

$$\mathrm{T}\mathrm{L}\mathrm{B}\mathrm{O}:\mathrm{O}(\mathrm{T}·\mathrm{N}·\mathrm{E}·\mathrm{B}·\mathrm{M}·\mathrm{W}\left(\mathrm{D}\right))$$

(8)

2.3.3. Validation and Performance Evaluation Strategies

k-Fold Cross-Validation

Cross-validation is a method used to reevaluate machine learning methods within the dataset used. The method is called k-fold cross-validation because a parameter named k is used in the method, which expresses the number of groups into which the selected data set will be divided. The selected k-value determines how many folds of cross validation in the model will be tested [88].

Since k = 10 in Figure 6, there were 10 folds. In the first round, 1 line in the yellow part is taken for testing and 9 lines are taken for training. The first round ends, so the success rate is achieved and the algorithm learns. The same process is applied to the other rounds so that all of the data enter both training and testing. As parameter k is selected as 10, the model was tested with 10-fold cross-validation.

Performance Evaluation

In classification problems, in order to measure the performance of the trained model, the prediction results of the model are transformed into the confusion matrix for a more explanatory analysis [90]. In a confusion matrix, as shown in

Table 9. Confusion matrix.

Confusion Matrix	Actually Positive (1)	Actually Negative (0)
Predicted Positive (1)	True Positives (TPs)	False Positives (FPs)
Predicted Negative (0)	False Negatives (FNs)	True Negatives (TNs)

TP: True Positive: The algorithm gave the result yes, and the actual result is yes; FP: False Positive: The algorithm gave the result yes, but the actual result is no; FN: False Negative: The algorithm gave the result no, but the actual result is yes; TN: True Negative: The algorithm gave the result no, and the actual result is no.

, values under the name True indicate the number of predictions made correctly, and values under the name False indicate the number of predictions made incorrectly. Each column contains the actual value, and each row contains the estimated value. Thus, the cells that continue diagonally from the top left to the bottom right give the accuracy of the prediction result, and the other cells give the error rate.

After the confusion matrices are generated, the classification metrics must be evaluated. For this, it is necessary to specify how much of the results from the first model are correctly predicted. For this purpose, the measurement of the accuracy rate is one of the scales used for classification methods. Accuracy is obtained by correctly dividing classified data into all data (Equation (9)) [91].

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}+\mathrm{T}\mathrm{N}}}$$

(9)

Other performance metrics, such as precision, recall, and F1 score, were calculated using TP, TN, FP and FN. Precision measures how many of the instances that the classification model predicts as positive are actually positive. A high precision value indicates that the positive predictions of the model are indeed positive. Precision is calculated by Equation (10) [85].

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(10)

Recall measures how many true positives the classification model is able to detect. A high recall value indicates that the model is less likely to miss true positives. Recall is calculated by Equation (11) [92].

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}$$

(11)

The F1 score is the harmonic mean of the precision and recall metrics of the classification model. With the F1 score, the performance of the model is evaluated in a single metric by combining precision and recall values in a balanced way. The F1 score value is calculated by Equation (12) [93].

$$\mathrm{F}1 \mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\times {\displaystyle \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$$

(12)

2.3.4. HyperNetExplorer: Architecture and Operation

HyperNetExplorer depends on ANNs and utilizes several optimization algorithms from the MealPy [61] package for hyperparameter optimization of ANN. The tool was developed with Python and Pytorch [94] (as the ANN framework) and utilized Streamlit as the GUI [95]. The parameters to be optimized and their ranges are provided in

Table 10. Parameters to be optimized.

Parameter Name	Lower Bound	Upper Bound	Options
Number of Hidden Layers (HL)	0	2	0: Single HL 1: Two HL 2: Three HL
Number of Neurons in HL = 1	0	6	0: 8 1: 16 2: 32 3: 64 4: 128 5: 256 6: 512
Number of Neurons in HL = 2	0	6	0: 8 1: 16 2: 32 3: 64 4: 128 5: 256 6: 512
Number of Neurons in HL = 3	0	6	0: 8 1: 16 2: 32 3: 64 4: 128 5: 256 6: 512
Activation Function of HL = 1	0	6	0: Leaky-ReLU 1: Sigmoid 2: Tanh 3: ReLU 4: LogSigmoid 5: ELU 6: Mish
Activation Function of HL = 2	0	6	0: Leaky-ReLU 1: Sigmoid 2: Tanh 3: ReLU 4: LogSigmoid 5: ELU 6: Mish
Activation Function of HL = 3	0	6	0: Leaky-ReLU 1: Sigmoid 2: Tanh 3: ReLU 4: LogSigmoid 5: ELU 6: Mish

.

In the default configuration of the tool, the learning rate is 0.001, and the number of epochs is 200, but this can change depending on the type of the problem. In the default configuration of the tool, CrossEntropyLoss is used as the loss function for the network for classification and MSELoss is used for the network for regression-type problems. The tool can utilize all the optimization algorithms provided by the MealPy package. In the current configuration of the tool utilized for this dataset, the Flower Pollination Algorithm (FPA), Genetic Algorithm (GA), Harmony Search (HS), Jaya Algorithm (JA), Particle Swarm Optimization (PSO), and Teaching–Learning-Based Optimization Algorithm (TLBO) are used. The population size for all algorithms was 20 and number of generations was 50 for all algorithms. Further information about the algorithm implementations can be found in [61,96,97,98]. The dataset could be uploaded to the system using the GUI; following this, once the FindBestNet command is given through the GUI, the objective function of the tool generates an ANN in every iteration of the optimizer based on a set of the hyperparameters provided by the optimizer. Once an ANN is generated, the accuracy of this ANN is calculated through 10-fold cross validation and the mean accuracy of this 10-fold validation is output as the output of the objective function. This output is consumed/evaluated by the optimizer and based on this, the objective function is called again with a new set of hyperparameters. The default parameters for the optimizer (i.e., not the main tool) is 20 epochs and a population size of 50, which generates a minimum of 20 × 50 = 1000 ANN architectures in every run. The values of the parameters for each generated network and accuracies are provided as a table in the Streamlit-based GUI. Once the training is finished, all ANN models are kept on the server and can be downloaded as (*.pt) files. The ANN model with the best accuracy could then be used in a production/testing environment.

2.4. Utilisation of Conventional Machine Learning Algorithms

Within the scope of the study, to validate that the accuracies achieved by ANNs discovered by HyperNetExplorer are significantly superior to the classical (well-known) ML algorithms, in the following stage, the Decision Tree, Gaussian NB, K-Nearest Neighbors (KNN), Linear Discriminant Analysis (LDA), Support Vector Machine, Random Forest, Gradient Boosting, and Histogram Gradient Boosting classifiers from Scikit-Learn package (v 1.3.0) [55] and CatBoost classifier from the CatBoost package [99] were utilized.

The validation and performance evaluation strategy was 10-fold cross-validation, similar to the strategy chosen for the HyperNetExplorer. The mean accuracy of 10-fold cross-validation is considered as the overall accuracy of the classification. The default parameters for the ML models are given in

Table 11. Default parameters of the ML models used in this study.

Algorithm	Parameters
DT	criterion = ‘gini’, splitter = ‘best’, max_depth = None, min_samples_split = 2, min_samples_leaf = 1, min_weight_fraction_leaf = 0.0, max_features = None, ran-dom_state = None, max_leaf_nodes = None, min_impurity_decrease = 0.0, class_weight = None, ccp_alpha = 0.0, monotonic_cst = None).
Gaussian NB	priors = None, var_smoothing = 1 × 10−9
KNN	n_neighbors = 5, weights = ‘uniform’, algorithm = ‘auto’, leaf_size = 30, p = 2, metric = ‘minkowski’, metric_params = None, n_jobs = None
LDA	solver = ‘svd’, shrinkage = None, priors = None, n_components = None, store_covariance = False, tol = 0.0001, covariance_estimator = None
SVM	C = 1.0, kernel = ‘rbf’, degree = 3, gamma = ‘scale’, coef0 = 0.0, shrinking = True, probability = False, tol = 0.001, cache_size = 200, class_weight = None, verbose = False, max_iter = −1, deci-sion_function_shape = ‘ovr’, break_ties = False, random_state = None
RF	n_estimators = 100, criterion = ‘gini’, max_depth = None, min_samples_split = 2, min_samples_leaf = 1, min_weight_fraction_leaf = 0.0, max_features = ‘sqrt’, max_leaf_nodes = None, min_impurity_decrease = 0.0, bootstrap = True, oob_score = False, n_jobs = None, random_state = None, verbose = 0, warm_start = False, class_weight = None, ccp_alpha = 0.0, max_samples = None, monotonic_cst = None
GB	loss = ‘log_loss’, learning_rate = 0.1, n_estimators = 100, subsample = 1.0, criteri-on = ‘friedman_mse’, min_samples_split = 2, min_samples_leaf = 1, min_weight_fraction_leaf = 0.0, max_depth = 3, min_impurity_decrease = 0.0, init = None, random_state = None, max_features = None, verbose = 0, max_leaf_nodes = None, warm_start = False, validation_fraction = 0.1, n_iter_no_change = None, tol = 0.0001, ccp_alpha = 0.
HGB	loss = ‘log_loss’, learning_rate = 0.1, max_iter = 100, max_leaf_nodes = 31, max_depth = None, min_samples_leaf = 20, l2_regularization = 0.0, max_features = 1.0, max_bins = 255, categori-cal_features = ‘from_dtype’, monotonic_cst = None, interaction_cst = None, warm_start = False, early_stopping = ‘auto’, scoring = ‘loss’, validation_fraction = 0.1, n_iter_no_change = 10, tol = 1 × 10−7, verbose = 0, random_state = None, class_weight = None
CatBoost	class CatBoostClassifier(iterations = None, learning_rate = None, depth = None, l2_leaf_reg = None, model_size_reg = None, rsm = None, loss_function = None, border_count = None, feature_border_type = None, per_float_feature_quantization = None, input_borders = None, output_borders = None, fold_permutation_block = None, od_pval = None, od_wait = None, od_type = None, nan_mode = None, counter_calc_method = None, leaf_estimation_iterations = None, leaf_estimation_method = None, thread_count = None, random_seed = None, use_best_model = None, verbose = None, logging_level = None, metric_period = None, ctr_leaf_count_limit = None, store_all_simple_ctr = None, max_ctr_complexity = None, has_time = None, allow_const_label = None, classes_count = None, class_weights = None, auto_class_weights = None, one_hot_max_size = None, random_strength = None, name = None, ignored_features = None, train_dir = None, custom_loss = None, custom_metric = None, eval_metric = None, bagging_temperature = None, save_snapshot = None, snapshot_file = None, snapshot_interval = None, fold_len_multiplier = None, used_ram_limit = None, gpu_ram_part = None, allow_writing_files = None, final_ctr_computation_mode = None, approx_on_full_history = None, boosting_type = None, simple_ctr = None, combinations_ctr = None, per_feature_ctr = None, task_type = None, device_config = None, devices = None, bootstrap_type = None, subsample = None, sampling_unit = None, dev_score_calc_obj_block_size = None, max_depth = None, n_estimators = None, num_boost_round = None, num_trees = None, colsample_bylevel = None, random_state = None, reg_lambda = None, objective = None, eta = None, max_bin = None, scale_pos_weight = None, gpu_cat_features_storage = None, data_partition = None metadata = None, early_stopping_rounds = None, cat_features = None, grow_policy = None, min_data_in_leaf = None, min_child_samples = None, max_leaves = None, num_leaves = None, score_function = None, leaf_estimation_backtracking = None, ctr_history_unit = None, monotone_constraints = None, feature_weights = None, penalties_coefficient = None, first_feature_use_penalties = None, model_shrink_rate = None, model_shrink_mode = None, langevin = None, diffusion_temperature = None, posterior_sampling = None, boost_from_average = None, text_features = None, tokenizers = None, dictionaries = None, feature_calcers = None, text_processing = None, fixed_binary_splits = None

.

2.4.1. Decision Tree (DT)

A decision tree is the process of dividing a set of large numbers of data into smaller subsets, following certain rules. Through this process, large amounts of data can be divided into small data groups, and ease of operation is provided [100]. The advantages of decision trees are that the generated rule is simple and understandable, and the operations can be performed on both categorical and numerical variables. As a disadvantage, it is a model that tends to overfit data [101].

2.4.2. Gaussian Naïve Bayes (Gaussian NB)

NB classification is an algorithm based on statistics and the Bayes rule. NB uses available and classified data and calculates the probability of which class the new incoming data belongs to [102]. The advantage of the algorithm is that it is fast to implement, can be used for both binary and multiclass classification problems, and can be used for continuous and discrete data. Its advantage is that it can be applied assuming that the variables in real data are independent of each other [103].

2.4.3. K-Nearest Neighbors (KNN)

In classification problems, the K-nearest neighbor algorithm works by determining the nearest neighbor of the hidden data point of a selected algorithm [104]. KNN is suitable for instances where multimodal classes exist and objects can have multiple labels. But the algorithm is adversely affected by noise and is also sensitive to irrelevant features [105].

2.4.4. Linear Discriminant Analysis (LDA)

The LDA algorithm classifies a complex data set by reflecting it to a space containing smaller clusters that also have the characteristics of that cluster [106]. LDA is a method of dimension reduction in the preprocessing step and is also highly advantageous for classification [107]. One of the major disadvantages of LDA is that, in binary classification problems, only one new feature is available, independent of the number of features in the dataset [108].

2.4.5. Support Vector Machine (SVM)

The support vector classification method tries to keep the margin width at its maximum. Examples belong to specific classes, and support vectors are used to distinguish between classes [109]. SVM has the advantage that it can classify without being affected by multidimensionality. However, it also has the advantage that it is difficult to interpret the parameters and cannot be used effectively with online parameters [107].

2.4.6. Random Forest (RF)

The RF algorithm is a combination of decision trees. Among the decision trees used in this algorithm, the trees with the highest accuracy and independence are preferred. Each tree branches according to the features and decision points in the data set. The random forest algorithm branches each node by using the best of the randomly selected variables at each node. Each dataset is created by selecting displacement from the original data. Then, tree structures are created by choosing random features. Pruning is not performed on trees created with random features [110]. The advantages of the RF algorithm are that it can work with large data sets and can be easily applied to high-dimensional problems. Its disadvantage is its slow training [111].

2.4.7. Gradient Boosting (GB)

In the gradient boosting classifier algorithm, tree models are added sequentially to the ensemble. Each tree model added to the community tries to correct the prediction errors made by the tree models already available in the community [112]. GB can handle missing data and does not require data preprocessing. However, the training time is long.

2.4.8. Histogram Gradient Boosting (HGB)

The histogram gradient boost classification algorithm is faster than the gradient boost algorithm and allows for categorical features to be included in training data [113]. HGB includes more features and performs better than other models [114]. But the processing time is long.

2.4.9. Category Boosting (CatBoost)

CatBoost is an algorithm for gradient boosting on decision trees developed by the researchers of Yandex and widely used in many research organizations including CERN. The CatBoost algorithm is a gradient boosting algorithm that aims to reduce the prediction drift that occurs during training by estimating gradients using a set of base models that do not include samples in the training sets [99]. CatBoost applies random permutation to prevent overfitting [115], but has complex parameter setting.

3. Results and Discussion

In this study, as mentioned above, we tested six optimizers from the MealPy [61] package for the gas- and liquid-phase datasets. The HyperNetExplorer was able to discover ANN architectures that provided accuracies in the range of 94.24–94.61% for the gas-phase dataset and in the range of 90.51–90.79% for the liquid-phase dataset.

Table 12. Performance metrics of best solutions for each optimizer (Gas phase).

Optimizer	Gas-Phase Accuracy	Gas-Phase Precision	Gas-Phase Recall	Gas-Phase F1 Score
FPA	94.32	0.9412	0.9386	0.9398
GA	94.36	0.9418	0.9557	0.9486
TLBO	94.49	0.9417	0.9446	0.9431
HS	94.24	0.9401	0.9475	0.9437
JA	94.61	0.9446	0.9460	0.9452
PSO	94.44	0.9424	0.9439	0.9431

provides the best accuracy rates achieved by ANN architectures discovered for the gas-phase datasets using all the optimizers and

Table 13. Performance metrics of best solutions for each optimizer (liquid phase).

Optimizer	Liquid-Phase Accuracy	Liquid-Phase Precision	Liquid-Phase Recall	Liquid-Phase F1 Score
FPA	90.67	0.9057	0.9025	0.9040
GA	90.79	0.9073	0.9075	0.9074
TLBO	90.71	0.9061	0.9062	0.9061
HS	90.51	0.9045	0.9047	0.9046
JA	90.79	0.9071	0.9075	0.9073
PSO	90.63	0.9052	0.9054	0.9053

provides the best accuracy rates achieved by ANN architectures discovered for the liquid-phase datasets using all the optimizers. The best accuracy rates achieved for each optimizer for the gas phase and liquid phase are given in Table 12 and Table 13.

For the gas-phase dataset the best mean accuracy of 10-fold CV achieved by ANNs generated with JA was 94.61%. The ANNs (ANN architectures) with the highest accuracies were discovered with JA, TLBO, and PSO optimizers. Metadata on the architecture of the best-performing ANN are given in

Table 14. The best-performing gas-phase ANN architecture discovered with JA.

Type of optimization method	JA
Number of layers in the network	3
Number of neurons in the input layer	5
Number of hidden layers	4
Numbers of hidden layer neurons	1024–16–16–512
Total number of iterations	1050
Accuracy	94.61

. One of the commonly used evaluation methods used in classification models is the confusion matrix. The confusion matrix for the best-performing ANN is presented in Figure 7, and it can be seen from these values that an accuracy rate of 94.61% is achieved.

The RoC curve drawn during the training of the best-performing ANN (JA) for the gas-phase dataset with 10-fold CV is given in Figure 8. For Class A, the mean AUC is 0.95; for Class B, the mean AUC is 0.87; and for Class C, the mean AUC is 0.94. The model performed very well in distinguishing Class A and Class C. The performance of the model for distinguishing Class B was slightly worse than that of the other two classes. For Class A and Class C, the variation in AUC scores in folds was very low, which indicated a stable performance. The AUC scores varied a bit more in folds for Class B, which indicated a less reliable classification performance. When we controlled the effect of the class imbalance, class imbalance was in the safe limits, with an Imbalance Ratio (IR) of 1.25:1 for the gas-phase dataset.

For the liquid-phase dataset, the best mean accuracy of 10-fold CV achieved by ANNs generated with JA and GA was 90.79%. The ANNs (ANN architectures) with the highest accuracies were discovered with JA, GA, and PSO optimizers. The metadata on architecture of the best-performing ANNs are given in

Table 15. The best-performing liquid-phase ANN architecture discovered with GA.

Type of optimization method	GA
Number of layers in the network	2
Number of neurons in the input layer	5
Number of hidden layers	4
Numbers of hidden layer neurons	16–64–1024–128
Total number of iterations	1050
Number of acc iterations	384
Accuracy	90.79

and

Table 16. The best-performing liquid-phase ANN architecture discovered with JA.

Type of optimization method	JA
Number of layers in the network	2
Number of neurons in the input layer	5
Number of hidden layers	4
Numbers of hidden layer neurons	1024–64–1024–1024
Total number of iterations	1050
Number of acc iterations	236
Accuracy	90.79

. The confusion matrix for the best-performing ANNs is presented in Figure 9 and Figure 10 and it can be seen from these values that an accuracy rate of 90.79% was achieved.

The RoC curve drawn during the training of the best-performing ANN (JA) for the liquid-phase dataset with 10-fold CV is given in Figure 11. For Class A, the mean AUC is 0.93; for Class B, the mean AUC is 0.87; and for Class C, the mean AUC is 0.94. The model performed very well in distinguishing Class A and Class C, similar to the results achieved in the gas-phase dataset. The performance of the model for distinguishing Class B was weaker, particularly for the first two folds, which achieved AUCs of 0.66 and 0.73, respectively. For Class A and Class C, the variation in the AUC scores in folds was very low, which indicated stable performance. The AUC scores varied greatly for some folds (e.g., between fold 1 and fold 9) for Class B which indicated a lower stability. When we controlled the effect of the class imbalance, class imbalance was in the safe limits, with an Imbalance Ratio (IR) of 1.13:1 for the liquid-phase dataset.

Figure 12 and Figure 13 show the confusion matrix for the worst-performing ANN (GA) for the gas-phase data set and the confusion matrix for the worst-performing ANN (PSO) for the liquid-phase data set. These two matrices are based on the results from the test folds.

For the gas-phase dataset, the worst-performing ANN was generated with the GA, and the best mean accuracy of 10-fold CV for this ANN was 58.96%. In this ANN, the majority of class A (0) was predicted as class C (2), and a considerable number of class A (0) and class C (2) were predicted as class B (1).The number of classes in the gas phase dataset were A:86, B:81, and C:101, the minor imbalance in classes where C > A,B may be the main reason behind the erroneous predictions.

For the liquid-phase dataset, the worst-performing ANN was generated with the PSO, and the best mean accuracy of 10-fold CV for this ANN was 34.52%. In this ANN, none of the predictions were class B (1); the majority of class A (0) was predicted as class C (2); similarly, the majority of class B (1) was predicted as class C (2). The numbers of classes in the gas-phase dataset were A:98, B:87, and C:96, and the minor imbalance in classes, where B < A,C, may have been the main reason behind the erroneous predictions.

In this study, to validate that the precision achieved by the ANNs discovered by HyperNetExplorer was significantly superior to the classical (well-known) ML algorithms, a series of training and tests was performed utilizing Scikit-Learn’s Decision Tree, Gaussian NB, K-Nearest Neighbors (KNN), Linear Discriminant Analysis (LDA), Support Vector Machine, Random Forest, Gradient Boosting, and Histogram Gradient Boosting algorithms, with 10-fold cross-validation. As explained in [116], despite the fact that deep-learning methods perform extremely well for classification on homogeneous data, the classification of tabular data is still a challenge for deep-learning models due to its heterogeneous nature, leading to a mix of dense numerical and sparse categorical features. This heterogeneity in tabular datasets, especially when the dataset contains lots of categorical features, each with lots of categories as in our dataset, results in a very weak correlation among the features when compared with the one introduced through spatial or semantic relationships in image or speech data. The reasons for failure of deep-learning models are strongly associated with the very nature of data, which mostly requires imputation, dealing with outlier and rebalancing of classes; furthermore, as there is often no spatial correlation between the variables, the dependencies between features are either complex or irregular. When working with tabular data, the learning algorithm has to learn the structure and relationships between its features from scratch. Thus, the reason behind the selection of classical (well-known) ML algorithms for comparison with the results obtained from HyperNetExplorer was the superior performance of the classical ML algorithms in tabular data classification problem when compared with modern deep learning methods.

Table 17. Machine-learning models.

ML Models	Gas-Phase Rank	Liquid-Phase Rank	Gas-Phase Accuracy (%)	Liquid-Phase Accuracy (%)
Decision Tree	2	5	59.90	31.30
GaussianNB	8	6	41.20	26.76
KNeighborsClassifier	10	4	33.56	32.03
LinearDiscriminantAnalysis	9	7	40.44	25.23
SVC	6	9	43.96	19.16
Random Forest	4	3	50.89	35.92
GradientBoostingClassifier	5	2	44.30	39.15
HistGradientBoostingClassifier	3	8	50.90	24.18
CatBoostClassifier	7	10	43.20	0.39
Decision Tree [1]	-	-	79	65
HyperNetExplorer (ANN)	1	1	94.61	90.79

below shows the accuracies obtained for each algorithm.

The results of the testing with classical (well-known) ML algorithms demonstrate that the ANNs discovered with the HyperNetExplorer demonstrated significantly (200–250%) higher performance than that of the best-ranking classical ML algorithms of the Scikit-Learn package.

Table 18. A summary of studies on the estimation of hydrogen production with ML.

Author	Aim	Methods	Results
Regression-Based
Ren et al. 2021 [2]	Photocatalysis to produce maximum hydrogen	Gaussian Process Regression (GPR)	Reported as successful results
Mageed [3]	Prediction of photocatalytic hydrogen production from ethanol using copper oxide (CuO) nanoparticles as a photocatalyst	Artificial Neural Networks (ANNs), Support Vector Machine (SVM), Nonlinear Regression Model (NLM), and Response Surface Model (RSM)	R2 > 0.99
Yurova et al. [5]	Prediction of the efficiency of photocatalytic hydrogen production	Gradient Boosting, Multilayer Perceptron, and Random Forest	R2 > 0.99
Yan et al. [34]	Prediction of hydrogen production rate	Reported as an ML model	R2 = 0.78
Haq et al. [36]	Prediction of hydrogen yield for supercritical water gasification	Support Vector Machine, Ensembled Tree, Gaussian Process Regression, and Artificial Neural Network	R2 = 0.99
Classification-Based
Ramkumar et al. [4]	Prediction of the hydrogen production value by photolysis	ANN	Accuracy 92%
Xu et al. [35]	Prediction of the hydrogen evolution reaction	Electronic property-based ML model	Accuracy 91%
Saadetnejad et al. [1]	Predict the band gap while investigating the gas phase of photocatalytic CO2 reduction for hydrogen production	Random Forest	Liquid-phase accuracy 65% Gas-phase accuracy 79%
This study	Prediction of the total hydrogen production of the photocatalysis process	Fine-tuned ANN/NAS	Liquid-phase accuracy 94.61% Gas-phase accuracy 90.79%

provides a summary of the studies in which hydrogen production is estimated using machine-learning methods.

Studies that used regression-based ML approaches provided significantly good results by achieving R² scores equal to or higher than 0.99 [3,5,36]. In fact, studies that used classification algorithms were unable to produce such successful accuracy rates. The highest classification accuracy that was observed in the literature was 92% [4]. In some datasets, the accuracy could be as low as 65% for a three-class classification problem. In ML, the accuracy rate is highly correlated with the nature of the dataset, and the number of classes, and to test if the accuracy can be improved, the same dataset should be used in comparisons of learning models.

In our study, we conducted our experiments with the same dataset as [1] to test whether we could improve the accuracy in datasets where low classification accuracy is persistent through the use of adaptive NAS and metaheuristics. In [1], the accuracy rates achieved with the implementation of a decision tree for the gas-phase dataset were 80% (for the training set) and 79% (for the test set); for the liquid-phase dataset, 77% (for the training set) and 65% (for the test set). Compared with [1], the accuracies achieved for the gas-phase dataset were 94% > 79%, and for the liquid-phase dataset, 90% > 65%, demonstrating that the discovery and use of fine-tuned ANN architectures through NAS greatly contribute to the classification accuracy of the total gas yield estimation in the photocatalytic CO₂ reduction process.

4. Conclusions

Today, the amount of carbon dioxide is rapidly increasing due to reasons such as excessive consumption of fossil fuels and population growth. If this increase in the amount of carbon dioxide continues, extreme events, such as very high temperatures, droughts, and excessive precipitation, are inevitable. For this reason, methods to reduce the amount of carbon dioxide have gained importance in recent years. In this context, our study aimed to facilitate the reduction of CO₂ (and the production of hydrogen) through a photocatalytic reaction. This study aimed to improve the estimation of the total gas yield (hydrogen production) of the photocatalysis process through NAS and hyperparameter optimization in ANNs. In this study, the dataset provided in [1] was used. At the beginning of our study, the source dataset was divided into two datasets according to the gas and liquid phases of the photocatalysis process. Later, data preprocessing (transformation, and imputation) was conducted on both datasets. Following this stage, we utilized HyperNetExplorer, a NAS tool to find the best ANN architecture that provides the most accurate prediction rates. The tests carried out with HyperNetExplorer resulted in the discovery of highly accurate ANN architectures in the estimation of the total gas (hydrogen) yield for the gas- and liquid-phase datasets. For the gas-phase dataset, we discovered an ANN architecture providing 94.61% classification accuracy, and for the liquid-phase dataset, an ANN architecture providing 90.79% classification accuracy was discovered. These values were significantly higher when compared with those of other well-known ML (such as decision trees and support vector machine) models and higher when compared with previous research [1]. The findings indicate that the discovery and use of fine-tuned ANNs greatly contributed to the prediction accuracy of the total gas (hydrogen) yield in the photocatalytic CO₂ reduction process. The key limitation of this study was the experiment being conducted with a single dataset [1], future research will focus on validating the findings of this study with more datasets. Another dimension of this study was the use of HyperNetExplorer for finding the most accurate algorithms for estimating the total gas yield (hydrogen production) of the photocatalysis process. The ANN architectures are considered as foundational architectures when compared with CNN-based, RNN-based, or transformer-based architectures. In fact, the classification or regression-based estimation problems of tabular datasets benefitted much better from foundational ANN architectures, as fully connected neural network layers with their flexible and easily tunable parameters provide much better accuracy rates. This study provides a new neural architecture search implementation method, where the best hyperparameters of an ANN, such as the number of layers, number of neurons per layer, and activation functions per layer, can be determined as a result of searching by nearly a limitless number of optimizers. The key contribution of this study is the removal of the barrier of the need for tightly coupled hyperparameter optimization architectures, where hyperparameter optimization can only be conducted with a limited set of optimization algorithms provided by each hyperparameter optimization package. Instead, our implementation a new optimizer can be easily added to the MealPy package and the new optimizer can easily be used for optimization with a very simple adaptation in the code base.

The findings of this study will facilitate the estimation of the total gas yield in a photocatalytic CO₂ reduction process. This will provide time and cost efficiency in the process, as the estimation of the total gas yield can be performed prior to the actual process, which will contribute to conducting feasibility studies without the need for conducting actual experiments, which are costly and time-consuming.