**2. Materials and Methods**

The initial input database of 150 synthetic chemicals included JWH synthetic cannabinoids, others synthetic cannabinoids and other substances of abuse. These designer drugs were divided into three classes referred to as "Class 1—JWH", "Class 2—non-JWH Cannabinoids" and "Class 3—Others". The group of positives contained 50 JWH synthetic cannabinoids, while the group of negatives included 100 compounds, i.e., 50 non-JWH cannabinoids and 50 other substances of forensic interest [5].

We used the quantitative structure–activity relationship (QSAR) method to estimate and predict the pharmacokinetics, drug-likeness and medicinal chemistry friendliness of each input compound by calculating a number of 300 molecular descriptors in terms of their physical and chemical properties, as well as 50 indices characterizing their chemical

**Citation:** Burlacu, C.M.; Burlacu, A.C.; Praisler, M. Sensitivity Analysis of Artificial Neural Networks Identifying JWH Synthetic Cannabinoids Built with Alternative Training Strategies and Methods. *Inventions* **2022**, *7*, 82. https:// doi.org/10.3390/inventions7030082

Academic Editor: Anastasios Doulamis

Received: 13 July 2022 Accepted: 29 August 2022 Published: 13 September 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

109

absorption, distribution, metabolism, excretion and toxicity activity (ADMET). The descriptors were selected from three blocks, i.e., topological, 3D-MoRSE (molecule representation of structure based on electron diffraction) and toxicity [6].

Only the first 150 most relevant descriptors were selected and used for the final computational and modelling stage. Hence, the input database was a matrix consisting of 150 samples × 150 variables. The shape, feature and target types of the data set, including the list of the computed and tested input molecular descriptors was presented in a previously published article [7].

The data set was divided into three subsets of samples: training, selection and testing. Hence, we used 90 training samples (60%), 30 selection samples (20%) and 30 testing samples (20%). To discover redundancies between the input variables, we used a correlation matrix, which represents a numerical value between −1 and 1 that expresses the strength of the relationship between two variables [8]. The types of layers the most frequently used in our classification model were the perceptron layer, the probabilistic layer and the scaling and bounding layers. The objective of the selection was to find the best-performing network architecture in terms of system sensitivity.

To avoid underfitting and overfitting, the neuron selection algorithm responsible for finding the optimal number of neurons in the networks was the Growing neurons algorithm [9]. The *Neural Designer* data science and machine learning platform was used to generate the mathematical expression represented by ANNs in order to export and incorporate them into the programming language *Python 3.10* in the so-called production mode.

Our general training strategy consisted of two different concepts, i.e., the loss index and the optimization algorithms. The error was the essential term in the loss expression. The most important errors that we estimated were the sum squared error, the mean squared error, the root mean squared error, the normalized squared error and the Minkowski error. We used the L1 and L2 regularization methods, which involve the sum of the absolute values of all parameters and the square sum of all the parameters in the ANN. The loss index was measured on the data set and could be represented as a hyper-surface with the parameters as coordinates (see Figure 1) [10].

**Figure 1.** Loss index viewed as a hyper-surface with parameters as coordinates.

In order to train the ANN, we generated a sequence of parameter vectors so that the loss index was reduced at each iteration of the algorithm.

#### **3. Results**

Five different optimization algorithms were used and compared, each with a variety of different calculation and storage requirements: gradient descent [11], conjugate gradient, quasi-Newton method, Levenberg-Marquardt algorithm [12] and adaptative linear momentum [13].

In order to scale the inputs, we calculated the following parameters: the minimum, the maximum, the mean and the standard deviation (see Table 1). The ANN architecture is presented in Figure 2 for version 1. The architectures of the following versions (2, 3 and 4) were also with one hidden layer perceptrons and had the same input and output layers as version 1. On the other hand, their hidden layers contained three (version 2), one (version 3) and six (version 4) nodes respectively.


**Table 1.** Values used for scaling the inputs for all ANN versions.

**Figure 2.** Network architecture, version 1: scaling layer (yellow), perceptron layer (blue), probabilistic layer (red).

We used the adaptive moment estimation (version 1), the Levenberg–Marquardt (version 2), the gradient descent (version 3) and the conjugate gradient optimization algorithms, as well as the growing neuron selection (all versions) method with the L1 (versions 2, 3 and 4) and L2 (version 1) regularization methods.

#### **4. Discussion**

The confusion matrices, calculated for each architecture and 30 testing samples, are presented in Tables 2–5 and the error results are highlighted in Table 6. The activation functions used were the hyperbolic tangent (version 1), the rectified linear (versions 2, 3 and 4) and Softmax (all versions).

**Table 2.** Confusion matrix for the analyzed ANN—version 1: 29 (96.7%) tested compounds were correctly classified and 1 (3.3%) was misclassified.



**Table 3.** Confusion matrix for the analyzed ANN—version 2: 27 (90.0%) tested compounds were correctly classified and 3 (10.0%) were misclassified.

**Table 4.** Confusion matrix for the analyzed ANN—version 3: 26 (86.7%) tested compounds were correctly classified and 4 (13.3%) were misclassified.


**Table 5.** Confusion matrix for the analyzed ANN—version 4: 27 (90.0%) tested compounds were correctly classified and 3 (10.0%) were misclassified.


**Table 6.** Data errors for the analyzed ANNs.


In order to test and compare the performances of the analyzed ANNs, we used the weighted average derived from the confusion matrix, i.e., the accuracy, the recall and the F1 score (see Table 7) [14].

**Table 7.** Classification metrics for the target variables of each analyzed ANNs.


#### **5. Conclusions**

In terms of the system performance, the results obtained for the four ANNs designed to recognize the class identity of JWH synthetic cannabinoids lead to the following conclusions:

1. Accuracy [(true positives + true negatives)/total instances]:

In comparison with the accuracy (93.3%) obtained for the initial ANN model presented in a previous article, the ANN—amended version 1—generated a higher score (96.7%), while the other three ANNs generated a lower score (86.7% for the amended version 3 and 90.0% for the amended versions 2 and 4).

2. Sensitivity, or true positive rate (true positives/positive instances):

All the ANN architectures (the initial ANN and its four amended versions), had an exceptional sensitivity in detecting JWH synthetic cannabinoids (class I); all the tested JWHs were recognized as such without exception, the rate of true positives (TP) being 100% in all cases.

3. Specificity, or true negative rate (specificity = true negative/negative instances):

Compared to the specificity (90.0%) of the initial ANN model, one of the four new architectures recorded a higher score (95.0% for the amended version 1); the other three were characterized by a lower specificity (81.8% for amended version 3, 86.3% for amended version 2 and 86.3% for amended version 4).

4. Error rate [accuracy = (false positives + false negatives)/total instances]:

While the initial ANN model was characterized by an error rate of 6.6%, its amended version 1 recorded a better (lower) error rate (3.3%), while the other three ANNs recorded a higher error rate (10.0% for amended version 2, 3 and 4).

Regarding the goodness-of-fit analysis, the best results were recorded for the ANN amended version 1 (99.97%), followed by the amended version 2 (99.94%), amended version 3 (98.41%) and amended version 4 (99.04%). Hence, we may conclude that the best performing ANN architecture was the one that included the following elements:


The results indicate that very good classification rates were obtained although the data set was complex. We intend to continue this study by applying various metaheuristic algorithms on these data sets and compare the results.

**Author Contributions:** Conceptualization, C.M.B. and A.C.B.; methodology, M.P.; software, A.C.B.; validation, M.P.; formal analysis, C.M.B.; investigation, C.M.B.; resources, A.C.B.; writing—original draft preparation, A.C.B.; writing—review and editing, C.M.B.; supervision, M.P.; project administration, M.P. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** An appreciation of the "Wiley Online Library" and the data science platform and machine learning "Neural Designer", important and useful tools, used in the construction of system architectures presented in this paper.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

