Understanding Machine Learning Principles: Learning, Inference, Generalization, and Computational Learning Theory

Abstract

Machine learning has become indispensable across various domains, yet understanding its theoretical underpinnings remains challenging for many practitioners and researchers. Despite the availability of numerous resources, there is a need for a cohesive tutorial that integrates foundational principles with state-of-the-art theories. This paper addresses the fundamental concepts and theories of machine learning, with an emphasis on neural networks, serving as both a foundational exploration and a tutorial. It begins by introducing essential concepts in machine learning, including various learning and inference methods, followed by criterion functions, robust learning, discussions on learning and generalization, model selection, bias–variance trade-off, and the role of neural networks as universal approximators. Subsequently, the paper delves into computational learning theory, with probably approximately correct (PAC) learning theory forming its cornerstone. Key concepts such as the VC-dimension, Rademacher complexity, and empirical risk minimization principle are introduced as tools for establishing generalization error bounds in trained models. The fundamental theorem of learning theory establishes the relationship between PAC learnability, Vapnik–Chervonenkis (VC)-dimension, and the empirical risk minimization principle. Additionally, the paper discusses the no-free-lunch theorem, another pivotal result in computational learning theory. By laying a rigorous theoretical foundation, this paper provides a comprehensive tutorial for understanding the principles underpinning machine learning.

1. Introduction

Machine learning is now the most successful and mainstream artificial intelligence (AI) approach. It is the most widely investigated and used approach in almost all disciplines and fields. There are numerous machine learning approaches and models for selection and integration, and the wide applications of machine learning have introduced far-reaching impacts on human kind.

Artificial neural networks (ANNs), encompassing multilayer perceptrons (MLPs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs), have gained substantial popularity and success. Inspired by biological neural systems, the development of ANNs can be traced back to early works such as McCulloch (1943) [1] and Rosenblatt (1958) [2]. The effectiveness of ANNs is based on representation theorems that guarantee their capacity to approximate complex functions with high accuracy when properly parameterized.

Spiking neural networks (SNNs) [3] provide a biologically inspired alternative to traditional ANNs, using components like dendrites and synapses to model neural dynamics. SNNs excel in unsupervised learning with mechanisms like spike-timing dependent plasticity (STDP) and Hebbian learning, enabling distributed and online learning. Their sparse activity reduces computational demands, making them ideal for edge intelligence and neuromorphic chip designs.

While ANNs are suited for computationally intensive tasks, SNNs are energy-efficient and effective for binary coding and sparse models. Brain-inspired computing [4], driven by SNNs, advances AI by improving models, algorithms, hardware, and applications. Neuromorphic computing, a core aspect of brain-inspired computing, uses very large scale integration (VLSI) technology to replicate biological systems.

The event-driven dynamics of SNNs enhance energy efficiency and biological plausibility, offering an alternative to ANN-based deep learning. As brain-inspired computing evolves with deep learning, it emerges as a promising, energy-efficient AI paradigm.

Machine learning always goes hand-in-hand with statistics, and the border between the two is blurred. Likewise, the boundary between machine learning and neural networks is also blurred. Neural networks are usually treated as models that model the human or animal neural systems, but many neural network models do not actually align with the actual neural systems. Thus, neural networks can be described as those models that can be represented by graphs and adaptively adjust their model parameters. Neural networking is a field of machine learning. Some traditional machine learning models, such as support vector machine (SVM) [5], clustering [6], k-nearest neighbors (k-NN) [7], and ensemble learning [3], can also be represented by graphs and adaptively adjusted. Traditionally, statistical models such as Gaussian process, principal component analysis (PCA), logistic regression, canonical correlation analysis (CCA), and nonnegative matrix factorization (NMF) can all be represented by graphs, and the model parameters can be solved by adaptation. Therefore, machine learning, statistics, and neural networks are sometimes deemed synonyms in the AI or machine learning community. The present AI era started with the successful implementations of deep learning, which belongs to neural networks.

The literature on machine learning is extensive, covering various models, their improvements, applications, and numerous survey and tutorial papers on specific areas. However, there remains a notable scarcity of works focusing on the fundamental principles and theoretical foundations of machine learning. Our thorough literature review reveals that no comprehensive tutorial or survey has systematically covered the foundational principles of machine learning in the last two decades. To the best of our knowledge, this paper is the first such comprehensive survey in twenty years, specifically dedicated to the key principles and foundational theories of the field, with a particular emphasis on neural networks.

This paper presents the core theories of machine learning, with a focus on neural networks, and explores various learning and inference methods, as well as classical and computational learning theories. While practical problem-solving applications are not the primary focus, the paper positions machine learning as a universal statistical and mathematical framework that models human brain functions across multiple levels, encompassing learning, generalization, and inference for tasks such as classification, regression, memory, and optimization.

The research papers referenced in this work were collected through a systematic process informed by the extensive academic expertise of the first author, who has authored three textbooks on machine learning since 2006 [3,7,8]. Over the years, the authors have diligently reviewed every issue of leading journals, including IEEE Transactions on Neural Networks and Learning Systems, Neural Networks, Journal of Machine Learning Research, Neurocomputing, Machine Learning, Neural Computation, IEEE Transactions on Cybernetics, Artificial Intelligence Review, and other top-tier machine learning publications. In addition, the authors consistently monitor proceedings from prominent conferences such as ICML, NeurIPS, AAAI, CVPR, IJCAI, and ICLR to identify influential and foundational contributions to the field. For this work, over one thousand papers were systematically collected and rigorously analyzed. Only those offering original contributions, significant theoretical advancements, or the latest developments were selected. These papers were then categorized by topic, forming the basis of this paper, which seeks to provide readers with a comprehensive overview of the machine learning discipline.

The structure of this paper is as follows. The first part focuses on outlining the fundamental methods and concepts related to learning and inference. In Section 2, we introduce many learning and inference methods that are related logical and scientific reasoning. In Section 3, many common criterion functions for machine learning are introduced, and those criterion functions for robust learning are also introduced. Section 4 treats a wide range of topics regarding learning and generalization. Section 5 is on model selection by cross-validation or using complexity criteria. Section 6 discusses the bias and variance trade-off. Section 7 deals with overparameterization and double descent. Section 8 discusses the universal approximation capability of various neural network models. The second part of this paper relates to computational learning theory. In Section 9, an introduction to computational learning theory is presented, where the no-free-lunch theory is introduced. From Section 10, Section 11, Section 12 and Section 13, PAC learning, the VC-dimension, Rademacher complexity, and the empirical risk-minimization principle, respectively, are introduced. Finally, Section 14 concludes this paper by presenting a few directions for future investigation.

2. Learning and Inference Methods

Learning plays a critical role in the survival and evolution of living organisms, and it represents a key function of neural networks. Learning rules serve as algorithms designed to identify appropriate weights $\mathbf{W}$ and/or other parameters for the network. The process of training a neural network involves solving a nonlinear optimization problem aimed at identifying the parameter set that minimizes a cost function based on provided examples. This parameter estimation process is commonly referred to as a learning or training algorithm.

Neural networks are generally trained over multiple epochs. An epoch refers to a full cycle where the network processes all the training examples using the learning algorithm exactly once. Upon completion of training, the neural network has captured a complex mapping and demonstrates generalization capabilities. To manage the training process, a stopping criterion is established to determine when to halt the process. The algorithm’s complexity is often represented as $O\left(m\right)$, where m refers to the order, representing the number of floating-point operations required.

Transduction, in both logic and statistical inference, refers to reasoning from observed specific instances (training data) to new, specific instances (test data). In contrast, induction involves generalizing from the training data to form general principles or rules that can be applied to new cases. Machine learning can be broadly categorized into inductive learning and transductive learning. Inductive learning is aimed at the standard machine learning objective of accurately classifying the entire input space. On the other hand, transductive learning targets a predefined set of unlabeled data, with the goal of labeling this particular set.

Inductive inference involves estimating a model function by examining the relationship between the data and the entire hypothesis space. This model is then used to predict output values for examples that are not part of the training set. Numerous machine learning techniques, including SVMs, neural networks, and neuro-fuzzy models, fall into this category.

In contrast, transductive learning, or transductive inference, focuses on predicting model functions specifically for test cases by incorporating supplementary information from the training dataset related to the new instances [9].

2.1. Scientific Reasoning

Scientific reasoning can generally be divided into three categories: deduction, induction, and abduction. Deduction is concerned with conclusions that are necessarily true, induction deals with conclusions that are likely true, and abduction involves conclusions that are plausibly true, though not certain.

Deductive reasoning moves from a cause to derive its effects or consequences. It involves analyzing premises, which consist of a general rule and a specific case, resulting in a necessary truth. Inductive reasoning, on the other hand, allows for inferring potential causes based on a given consequence, formulating a rule based on a specific instance and its result, though the conclusion is only likely, not guaranteed. Abductive reasoning, nevertheless, infers a cause based on a rule and an observation (result), producing a conclusion that is plausible but hypothetical.

Syllogistic reasoning serves as a formal framework for deduction. In syllogistic inference, the conclusion logically follows from both the major and minor premises. Reasoning can progress through successive syllogistic inferences, using transitive closure (deduction), generalization (induction), and experimental validation (abduction). These reasoning forms are interconnected, and their varying degrees of truth can be understood as different levels of belief [10]. The differences in these beliefs stem from how the premises are converted in syllogistic processing.

2.1.1. Deductive Reasoning

In deductive reasoning (also referred to as top-down logic), a conclusion is derived by applying universally accepted principles within a closed system of discourse. The reasoning process progressively narrows the scope of consideration, leaving only the conclusion. The outcome of a deductive argument is always certain.

One of the primary forms of deductive reasoning is the law of detachment, also called affirming the antecedent or modus ponens, which is a Latin term meaning “the way that affirms by affirming”. In propositional logic, modus ponens is a rule of inference that states: “If P implies Q ($P\to Q$), and P is asserted as true, then Q must also be true”.

The law of syllogism involves combining two conditional statements to form a conclusion by joining the hypothesis of one statement with the conclusion of another. For example, if $P\to Q$ and $Q\to R$, then it follows that $P\to R$.

In propositional logic, modus tollens (Latin for “the way that denies by denying”) is a valid form of reasoning that denies the consequent. It follows from the principle that if a statement holds true, its contrapositive must also hold.

The contrapositive law asserts that, in a conditional statement, if the conclusion is false, the hypothesis must also be false. For example, from $P\to Q$, if $\neg Q$ is true, then $\neg P$ must also be true.

Any application of modus tollens can be transformed into an application of modus ponens or into a transposition of the premise, which represents a material implication. Similarly, each instance of modus ponens can be transformed into modus tollens or a transpositional form.

2.1.2. Inductive Reasoning

In inductive reasoning (often referred to as bottom-up logic), conclusions are drawn by generalizing or extrapolating from specific instances to broader principles, which inherently involves epistemic uncertainty. However, it is important to distinguish inductive reasoning from mathematical induction, which is a form of deductive logic used in mathematical proofs.

Inductive learning is a subset of supervised learning methods, where the goal is to derive a hypothesis $h\left({\mathbf{x}}_i\right)$ that approximates $f\left({\mathbf{x}}_i\right)$ for all i, given a set of input–output pairs ${\mathbf{x}}_i,f\left({\mathbf{x}}_i\right)$. In inductive learning, the task is to create a concept that can accurately categorize most positive instances and exclude negative ones, based on a collection of training examples. This process necessitates a sufficient quantity of training instances to effectively develop a generalizable concept.

2.1.3. Abductive Reasoning

Abductive reasoning, sometimes referred to as abduction, abductive inference, or retroduction, is a type of logical reasoning that begins with an observed phenomenon and aims to identify the most straightforward and plausible explanation by tentatively accepting a hypothesis. The outcomes of the hypothesis can be subjected to empirical verification. Unlike deductive reasoning, the premises do not guarantee the conclusion in abduction, which makes it a process of inferring the most probable explanation.

The scientific approach to answering questions is depicted in Figure 1. This method consists of three distinct phases [11]. In the first phase, labeled question in search of answers, the analyst generates potential hypotheses as possible answers. The second phase, labeled hypothesis in search of evidence, involves testing these hypotheses to find supporting evidence. In the third phase, evidentiary assessment of hypotheses, the likelihood of each hypothesis is assessed based on the gathered evidence. As new evidence is uncovered, it may lead to revised answers to the question, prompting the discovery of additional evidence and the reassessment of probabilities. The process incorporates abductive, deductive, and inductive reasoning. Abductive reasoning is used in generating hypotheses, suggesting what might be true. Hypothesis-driven evidence discovery employs deductive reasoning, showing what must be true. Hypothesis assessment utilizes inductive reasoning to determine what is likely to be true.

2.1.4. Analogical Reasoning

In contrast to inductive learning, analogical learning can be based on just a single instance. For example, by knowing that the plural form of fungus is fungi, one can deduce that the plural of bacillus should be bacilli.

2.1.5. Case-Based Reasoning

Case-based reasoning, along with knowledge generalization, is a primary method that utilizes prior experiences. The focus of case-based reasoning is on retrieving the most relevant case(s) from a database of past experiences to assist with learning in the current task.

2.1.6. Ontologies

Ontologies are structured frameworks that categorize data objects into classes and define the relationships between these classes. They allow for flexible classification, where an object can belong to multiple classes simultaneously. The most basic form of ontology involves classification, where each class has a single parent class. The assignment of an object to a class is governed by rules, which are programmed into the system, defining the relationships of classes, subclasses, and superclasses.

2.2. Supervised, Unsupervised, and Reinforcement Learning

Learning techniques are commonly categorized into three types: supervised, unsupervised, and reinforcement learning. In these methods, ${\mathbf{x}}_p$ and ${\mathbf{y}}_p$ represent the input and output for the pth instance in the training data, ${\widehat{\mathbf{y}}}_p$ represents the predicted output by the neural network for the pth input, and E denotes the error function. Statistically, unsupervised learning aims to model the probability density function (pdf) of the input data, $p\left(\mathbf{x}\right)$, whereas supervised learning focuses on the conditional pdf, $p\left(\mathbf{y}\right|\mathbf{x})$. Supervised learning finds widespread use in areas like classification, regression, modeling, control systems, signal processing, and optimization. In contrast, unsupervised learning is typically employed in tasks such as vector quantization, clustering, feature extraction, signal encoding, and data analysis. Reinforcement learning, however, is predominantly used in control systems and various artificial intelligence applications

2.2.1. Supervised Learning

Supervised learning involves learning a mapping $f:\mathcal{X}\to \mathcal{Y}$ from a provided dataset $\{({\mathbf{x}}_i,y_i),i=1,2,\dots ,N\}$, where each input instance ${\mathbf{x}}_i\in \mathcal{X}$ is associated with a corresponding label $y_i\in \mathcal{Y}$.

In this learning paradigm, the parameters of the model are adjusted by comparing the actual output of the network with the desired target output. This process functions as a closed-loop system, where the feedback signal is represented by the error. The error metric quantifies the disparity between the output generated by the network and the expected result from the training data, guiding the learning process. Typically, the error is measured using the mean squared error (MSE) as follows:

$$E={\displaystyle \frac{1}{N}}\sum _{p=1}^N{∥{\mathbf{y}}_p-{\widehat{\mathbf{y}}}_p∥}^2,$$

(1)

where N denotes the total number of training patterns, ${\mathbf{y}}_p$ is the actual output for the pth training instance, and ${\widehat{\mathbf{y}}}_p$ is the network’s predicted output for the same instance. The error E is recalculated at the end of each epoch. The learning process concludes once E falls below a predefined threshold or when a failure condition is triggered.

To minimize E and drive it towards zero, gradient descent is commonly used. This method guarantees convergence to a local minimum within a region near the initial network parameter values. The most commonly used gradient-descent algorithms are the least mean squares (LMS) and backpropagation (BP) algorithms. Second-order methods, on the other hand, involve calculating the Hessian matrix for optimization.

2.2.2. Unsupervised Learning

Unsupervised learning operates without predefined target values, aiming to uncover meaningful patterns or structures in input data by leveraging relationships between features. This approach reduces data dimensionality or volume, and is particularly suited for biological systems due to its reliance on intuitive mechanisms such as neural competition and cooperation, without requiring external supervision.

A stopping criterion is crucial to control the learning process, ensuring it halts appropriately and does not continuously adapt to evolving patterns outside the training set. Common unsupervised learning methods include Hebbian learning, competitive learning, and the self-organizing map (SOM), with these approaches typically converging more slowly to stable states.

• Hebbian learning: A local process involving two neurons and a synapse, where weight adjustment is proportional to the correlation observed between pre- and postsynaptic activities. It underpins models for PCA and associative memory.
• Competitive learning: Neurons compete to be the most responsive to a given input. The SOM, a form of competitive learning, relates closely to clustering techniques.
• Boltzmann machine: employs stochastic training via simulated annealing, inspired by thermodynamics, to learn in an unsupervised manner.

These methods highlight unsupervised learning’s capacity to extract structure and adapt in complex environments.

2.2.3. Reinforcement Learning

Reinforcement learning encompasses computational techniques that guide an artificial agent, whether a physical or simulated robot, to select actions that maximize its cumulative expected reward over time [12]. The difference computed for this purpose, known as the reward prediction error, has been found to closely align with the activity of dopamine-releasing neurons in the substantia nigra of nonhuman primates [13].

This type of learning is a variation of supervised learning, where the exact desired outcome is not provided. Instead of detailed guidance, the agent receives feedback on whether its actions were successful or not. This approach is considered more aligned with human cognition, where fully specified correct answers are often unavailable to both the learner and the teacher. In reinforcement learning, the agent is evaluated based on whether the actual outcome is close to the expected result. The agent is rewarded for successful actions and punished for unsuccessful ones, with no need for explicit derivative computations. However, this makes reinforcement learning a slower process. In control systems, if the controller produces a correct output in response to an input, it is considered a good result; otherwise, it is deemed bad. The binary evaluation of this output, known as external reinforcement, serves as the error signal for the learning process.

2.3. Semi-Supervised Learning and Active Learning

In domains such as bioinformatics, text categorization, web and text mining, spam detection, face recognition, and video indexing, vast amounts of unlabeled data can be collected quickly and inexpensively. However, manually labeling these data are often labor-intensive, costly, and susceptible to errors. When the availability of labeled samples is limited, the inclusion of unlabeled samples can help mitigate the risk of performance degradation caused by overfitting.

Semi-supervised learning aims to leverage a vast amount of unlabeled data alongside a small set of labeled examples to enhance generalization. Some methods in this domain rely on assumptions connecting the probability distribution $P\left(x\right)$ with the conditional distribution $P(Y=1|X=x)$. This learning paradigm shares similarities with transductive learning. Semi-supervised learning is grounded in two central principles: the cluster assumption [9] and the manifold assumption [14]. The cluster assumption posits that points within the same cluster tend to have identical labels, with transductive SVM [9] serving as a significant illustration.

Universum data consist of unlabeled examples that do not belong to any of the classes of interest in a classification task. A contradiction occurs when two functions from the same equivalence class produce differing outputs on a Universum sample. Unlike semi-supervised learning and transductive learning [9], Universum learning operates with Universum data originating from a distribution distinct from that of the labeled training data. This approach balances the need to explain training samples (using large margin hyperplanes) and minimize contradictions in Universum data.

In active learning, also known as pool-based active learning, the learner selectively queries the most informative examples for labeling to optimize generalization while minimizing labeling effort [15]. Given a pool of unlabeled samples, active learning selects data points that enhance model performance. Its effectiveness depends on the model architecture and data characteristics. To address model misspecification, methods often rely on the conditional expectation of generalization error and weight training samples by importance [16]. Reinforcement learning can be considered a type of active learning, where a query mechanism actively seeks labels for some of the unlabeled data.

Active learning can be applied in situations such as web searching, email filtering, and relevance feedback systems for databases or websites. The first two applications typically involve induction, aiming to build a classifier that generalizes well to unseen examples. The third scenario illustrates transduction [9], where the learner’s performance is evaluated on a set of instances within the same database, instead of using an independent test set.

Query-by-committee is a widely used pool-based active learning approach for classification [17] and regression [18]. It builds a committee of learners from labeled data, often using bootstrapping or diverse algorithms, and selects unlabeled samples where committee disagreement is highest for labeling. Query-by-committee also employs a prior over hypotheses and processes unlabeled data streams, deciding label requests. For data sampled uniformly from a unit sphere in ${\mathbb{R}}^d$ with labels aligned to a homogeneous linear separator, it achieves generalization error $ϵ$ with $O({\displaystyle \frac{d}{ϵ}}log({\displaystyle \frac{1}{ϵ}}))$ samples and $O(dlog({\displaystyle \frac{1}{ϵ}}))$ labels, significantly improving on supervised learning sample complexity $O({\displaystyle \frac{d}{ϵ}})$. The method involves random sampling from intermediate hypothesis spaces, with update complexity growing polynomially as the number of updates grows.

An information-theoretic method for active data selection is proposed in [19]. In [20], a two-stage sampling method is presented to reduce both bias and variance, leading to the development of two active learning strategies.

In batch-mode active learning, as outlined in [21], a set of informative examples is selected for labeling in each iteration. The key aspect of this approach is minimizing redundancy among the chosen examples, ensuring that each example contributes distinct information to update the model. The method chooses unlabeled examples that can effectively reduce the Fisher information in the classification model [21].

2.4. Transfer Learning

Domain refers to the state of the world at a given time. Transfer learning, or meta-analysis in statistics [22], models rare events by leveraging data from related cases to improve performance in a target domain using information from a source domain with a different distribution. This concept, based on case-based and analogical learning, suggests that fewer labeled examples are needed compared to independent task learning [23].

When single-source domain knowledge is insufficient, multiple source domains with richer transferable information are used, where multisource domain adaptation considers the differing contributions of each source to the target [22]. Transfer learning, which can involve inductive (e.g., multitask), transductive (e.g., domain adaptation), and unsupervised (e.g., clustering) cases, transfers knowledge from labeled sources to an unlabeled target, involving both single and multiple sources/targets.

To address source–target discrepancies, features are transformed into a latent space, with deep learning methods such as pretrained CNNs [24] extracting robust representations for visual domain adaptation. Transfer learning can be homogeneous, where source and target domains share the same instance space but differ in distribution, or heterogeneous, where domains are from distinct but related spaces. It also includes instance transfer, feature representation transfer, parameter transfer, and relational knowledge transfer.

Knowledge transfer has two main directions [25]: representation transfer [26], which abstracts task characteristics for future problems by collecting shared knowledge across tasks, and parameter transfer [27], which directly transfers parameters between tasks. Strategic knowledge transfer, especially in multiagent reasoning frameworks like intra-agent transfer [28] and ad hoc teamwork [29,30], enables agents to cooperate with new teammates by selecting the best pre-learned policy.

Two challenges are tackled in [31]: transferring learned responses across varying opponent strategies, and reducing game-solving costs. Q-mixing addresses the transfer problem for value-based policies by averaging Q-values, while the mixed-oracles and mixed-opponents algorithms minimize costs by reusing responses and combining prior opponents into novel policies.

A semiparametric additive framework is proposed in [32] to predict target models by transferring parameter information from source models. Using K-fold cross-validation, this approach assigns data-driven weights for adaptive transfer, ensuring optimality and robustness even in misspecified settings, while addressing negative transfer without requiring prior source model knowledge.

Error bounds for a single-source parameter transfer in high-dimensional linear models are established in [33], extending to multi-source learning with minimax optimality under $L_q$-regularization in [34]. A multi-source framework for generalized linear models is introduced in [35], including methods to identify transferable sources. This framework is applied to Gaussian graphical models in [36], with edge detection procedures ensuring false discovery rate control. These methods rely on parameter similarity, auxiliary source models, and effective tuning parameters.

Transfer learning can harm performance when source models are unrelated to the target, known as negative transfer [37]. Transfer learning for linear models [34] and generalized linear models [35] address this by constructing auxiliary source models based on parameter similarity.

Domain generalization [38] tackles prediction on future unlabeled data with varying distributions. Ref. [39] frames it as a supervised learning problem by augmenting the feature space with marginal distributions, building on [40]. They propose two statistical models: one for arbitrary dependencies, and another for independent and identically distributed data, alongside a distribution-free kernel machine for consistent learning across frameworks.

Learning to learn (LTL), or meta-learning, uses labeled data from multiple tasks to design a meta-learner that selects optimal algorithms for future tasks [41]. Well-adapted hypothesis classes improve sample efficiency [42]. Knowledge transfer to new tasks has been quantified through structures like feature representations [43,44], priors on predictors [45], and dictionaries [46]. While both LTL and domain generalization aim for task generalization, LTL relies on labeled data and achieves Bayes risk, whereas domain generalization works with unlabeled data. Active learning variants of domain generalization were explored by [23,47], and focused on identifying a common feature space across tasks.

2.5. Other Learning Methods

2.5.1. Ordinal Regression and Ranking

Regression learns the relationship between input variables and continuous output values. Categorical data includes ordinal, with natural ordering, and nominal, with no ordering (e.g., hair color is nominal, service quality assessment is ordinal). Ordinal regression predicts ordinal variables, treating it as a multiclass problem with ordinal constraints, often using numeric values for ordinal scales [48].

Ranking problems, such as those in search engines and online advertising, maintain ordinal relationships through pointwise, pairwise, and listwise methods [49]. Pointwise methods predict relevance scores, pairwise methods (e.g., rankSVM [50]) classify preference pairs, and listwise methods (e.g., LambdaMART [51]) optimize ranking lists. Gradient-boosted decision trees (GBDT) [52] and LambdaMART [51] excel in web search ranking tasks.

Preference learning focuses on binary preference relations [53], ranking input points based on comparisons.

2.5.2. Manifold Learning

Representation learning enables machines to autonomously derive task-relevant representations from raw data, often as unsupervised preprocessing methods like PCA and cluster analysis. Manifold learning focuses on creating low-dimensional representations that preserve the intrinsic geometric structure of data, ensuring close data points in the original space remain close after reduction. Sparse coding [54] adds a sparsity constraint to these representations, while multilinear subspace learning derives low-dimensional representations directly from tensor-based models of multidimensional data.

In deep learning, hierarchical representations are formed, with higher-level features built from lower-level ones. Many dimensionality reduction techniques stem from manifold learning, including:

• Locally Linear Embedding (LLE) [55]: captures global structures of nonlinear manifolds, such as face image datasets.
• Laplacian eigenmaps [56]: maintains local structures using a graph-based representation of data.
• Orthogonal neighborhood-preserving projections [57]: a linear extension of LLE that preserves local geometric relationships.

These methods highlight the versatility of manifold learning in representing complex data geometries.

2.5.3. Multi-Task Learning

Domain refers to the input space $\mathcal{X}$ and its marginal distribution, while tasks refer to the output space $\mathcal{Y}$ and the conditional distribution of $\mathcal{Y}$ given $\mathcal{X}$. Typically, $\mathcal{X}$ and $\mathcal{Y}$ remain consistent across problems, with the joint distribution $P_{\mathcal{XY}}$ describing domains and tasks interchangeably. Multi-task learning improves task learning by leveraging distribution similarities [58,59,60], whereas domain generalization focuses on generalizing to new tasks.

Multiagent learning shares similarities with multi-task learning, where addressing each strategic situation is comparable to solving a distinct task, with the strategies of opponents representing distributions over a common set of tasks [61,62].

The multi-task learning community categorizes knowledge into task-relevant knowledge, specific to a task [63,64], and domain-relevant knowledge, shared across tasks [58,65,66,67]. Some approaches bridge these, such as using task-specific knowledge as a curriculum [68]. In task-relevant learning, a key method is abstracting irrelevant state information [63,64], with [31] focusing on learning responses to specific opponent policies.

2.5.4. Imitation Learning

Imitation learning, where an agent learns from demonstrations, used in sequential decision-making tasks without predefined reward functions, is applied when designing rewards is difficult or uncertain. It finds optimal policies based on expert demonstrations, with two main approaches: behavioral cloning [69], which suffers from error accumulation, and inverse reinforcement learning [70], which infers a cost function, often through adversarial learning [71]. To enhance sample efficiency, self-supervised representation learning can generate training signals from available data to enhance efficiency [72].

Earlier works frame imitation as independent and identically distributed classification [73] or no-regret online learning [74]. An online imitation learner for stochastic environments [75] eliminates the need for environment resets or stationary dynamics. This approach enables continual learning with finite error bounds, querying only when necessary, with a rapid decrease in query frequency and bounds on unlikely events.

Imitation-active learning ensemble (IALE) [76] proposes a batch-mode, pool-based imitation learning strategy integrating expert heuristics like uncertainty, diversity, and query-by-committee. Using DAgger [74], IALE encodes states and expert actions, balancing exploration and exploitation while distilling expert knowledge into an active learning sampling strategy.

2.5.5. Curriculum Learning

The idea of using learning progress as a reward originates from [77], and is linked to intrinsic motivation [78]. Curriculum learning, introduced by [79], has been applied in contexts like semi-supervised learning, where label propagation benefits from ordering samples from simple to hard [80]. Challenges in curriculum learning include ordering subtasks by difficulty, defining a “mastery” threshold [81], and balancing task difficulty to avoid forgetting [81]. The teacher aims to train the student for a final task using minimal steps, modeled as solving a partially observable Markov decision process (POMDP) [82], with reinforcement and supervised learning formulations. Exploration bonuses [83] help mitigate sparse rewards in Student algorithms.

Deep reinforcement learning has excelled in domains like video games [84], but random exploration’s sample complexity grows exponentially with the number of steps required [85]. Curriculum learning [79,81] addresses this by sequencing tasks by difficulty. Recent work includes [86], which proposed a two-agent system for goal generation, and [87], who used generative adversarial networks (GANs) for goal state creation. However, these methods do not explicitly improve learning of the final task.

Teacher–student curriculum learning [88] automates curriculum generation by having the teacher select tasks based on learning progress, prioritizing those with steep positive slopes and addressing forgetting with negative slopes. It outperforms handcrafted curricula in supervised (decimal addition with long short-term memory (LSTM)) and reinforcement learning (Minecraft navigation) tasks [88]. A similar approach is presented by [89] for supervised sequence learning and some reinforcement learning tasks.

2.5.6. Multiview Learning

Human perception is inherently multimodal, involving sight, hearing, touch, smell, and taste. Similarly, data are often collected from multiple sources, each providing unique, complementary information. For multi-view data, consistency across sources is crucial [90]. Multi-view (or multimodal) learning integrates complementary data from various sources to enhance performance.

Three primary multi-view learning methods are co-training, multiple kernel learning, and subspace learning. Co-training maximizes mutual agreement between two data views [91]. CCA [92] adapts PCA to maximize the correlation between two views in a shared subspace, with multiset CCA [93] and tensor decomposition [94] further enhancing joint analysis. Multiple kernel learning [95] combines kernels from different modalities to improve model performance. Subspace learning discovers a shared latent subspace across views.

Multitask learning leverages domain-specific knowledge across tasks to boost generalization [58], benefiting from additional training signals that act as an inductive bias, akin to multimodal learning.

2.5.7. Multilabel Learning

Multilabel learning encompasses multilabel classification and multilabel ranking tasks. In multilabel classification [96], the goal is to learn a function that assigns multiple labels to each data instance. On the other hand, multilabel ranking involves assigning a real value to each instance, representing its relevance to a label, and generating an ordered ranking based on these values.

A multilabel problem with q labels can be broken down into multiple single-label tasks using one of three common strategies: binary relevance, label powerset, and pairwise methods. Binary relevance applies a one-vs.-all approach, creating q binary classification tasks, though it may struggle with correlated labels. The label powerset method converts the problem into a multiclass classification with $2^q$ possible labels, accounting for correlations but potentially increasing label space. The pairwise method trains $q(q-1)/2$ classifiers for label pairs, combining them through majority voting.

2.5.8. Multiple-Instance Learning

Multiple-instance learning [97] extends traditional supervised learning, where examples are “bags” of feature vectors. The bag label is determined by a function applied to the labels of its individual instances, often using Boolean OR. A theoretical analysis and PAC-learning algorithm for this approach are presented in [98].

In this framework, bags are labeled positive or negative, with the instances inside lacking individual labels. The goal is to build a classifier that predicts unseen bag labels. Multiple-instance learning is applied in drug activity prediction, as well as image categorization and annotation, particularly in region-based tasks.

Formally, let $\mathcal{X}$ represent the space of instances and $\mathcal{Y}$ the set of possible labels. The goal of multiple-instance learning is to learn a mapping $f:2^{\mathcal{X}}\to \{-1,+1\}$ from a collection of pairs $\{({\mathbf{X}}_1,y_1),({\mathbf{X}}_2,y_2),\dots ,({\mathbf{X}}_m,y_m)\}$, where each ${\mathbf{X}}_i\subseteq \mathcal{X}$ is a set of instances $\{{\mathbf{x}}_1^{\left(i\right)},{\mathbf{x}}_2^{\left(i\right)},\dots ,{\mathbf{x}}_{n_i}^{\left(i\right)}\}$, with ${\mathbf{x}}_j^{\left(i\right)}\in \mathcal{X}$ for $j\in \{1,\dots ,n_i\}$, and $y_i\in \{-1,+1\}$ is the label of the corresponding bag ${\mathbf{X}}_i$.

2.5.9. Parametric, Semiparametric, and Nonparametric Classifications

Pattern classification methods for numerical inputs are divided into three categories: parametric, semiparametric, and nonparametric. Parametric and semiparametric classifiers rely on prior knowledge of the data’s structure. Parametric methods assume the probability density function (pdf) is known, with parameters estimated from the data, allowing good performance with smaller sample sizes. Parametric methods, such as SVM and logistic regression, are effective when the pdf is well-understood. Semiparametric methods involve models with a fixed, but typically larger, number of parameters that do not grow with sample size.

Nonparametric methods estimate the pdf without prior assumptions on its form, requiring larger sample sizes than parametric methods. These approaches are more flexible, as parametric models are limited by their fixed structure. A common example of a nonparametric method is the k-NN classifier.

Methods like density estimation using neural networks or SVMs also belong to nonparametric techniques. The Parzen window method [99] is a nonparametric technique used to estimate the pdf for a finite set of patterns; however, it can be computationally expensive, due to the large number of kernels needed for its representation. Another efficient nonparametric approach is the decision tree, such as C5.0 (http://www.rulequest.com/see5-info.html, accessed on 29 December 2024). This method uses a hierarchical structure and follows a divide-and-conquer strategy, making it suitable for both classification and regression tasks in supervised learning.

2.5.10. Learning from Imbalanced Data

Imbalanced data poses a challenge in machine learning, where one class dominates and the other is underrepresented. This imbalance causes the classifier’s decision boundary to favor the majority class, even though the minority class may be more important for the task.

To address this, solutions fall into two categories. Preprocessing methods, such as oversampling, undersampling, and synthetic minority oversampling technique (SMOTE) [100], balance the dataset by adjusting the number of instances in the minority or majority class. These techniques are independent of the classifier used. Algorithm-based approaches modify the learning algorithm itself, using methods like error cost algorithms [101] and class-boundary-alignment [102] to adjust how imbalanced data are handled, often by penalizing misclassifications of the minority class more heavily.

The one-class classification problem, also referred to as anomaly, outlier, or novelty detection, involves identifying patterns that belong exclusively to a target class, with all other patterns treated as nontarget.

2.5.11. Zero-Shot Learning

Zero-shot learning classifies objects from unseen classes using auxiliary information like attributes or word vectors to connect seen and unseen classes [103,104]. It relies on knowledge transfer, where models trained on seen classes generalize to unseen ones. Traditional zero-shot learning assumes only unseen classes in test data, but real-world scenarios include both seen and unseen classes [104]. Generalized zero-shot learning handles both, though it often biases toward seen classes [103]. Methods like direct attribute prediction [104] perform poorly, while frameworks [105] combine zero-shot, generalized zero-shot, and few-shot learning. Generative models [106] synthesize unseen class data, turning generalized zero-shot learning into a supervised task, though imbalanced accuracy remains [106].

Meta-learning supports few-shot learning by focusing on tasks rather than individual data points. It learns meta-knowledge across training episodes, enabling rapid adaptation to new tasks. The consistent meta-regularization method [107] enhances this meta-knowledge, improving model performance.

3. Criterion Functions

MSE is a commonly used metric that penalizes larger errors more heavily, and is optimal for maximum likelihood estimation when feature vectors follow a Gaussian distribution [108]. Alternative metrics, such as mean absolute error or maximum absolute error, may be preferred in specific cases.

A notable alternative to MSE is the logarithmic error function, derived from the Kullback–Leibler divergence. This function offers advantages over MSE [109]. For the tanh activation function, the error for pattern p is defined as

$$E_p\left(\mathbf{W}\right)={\displaystyle \frac{1}{2}}\sum _{i=1}^{J_M}\left[\left(1+y_{p,i}\right)ln\left({\displaystyle \frac{1+y_{p,i}}{1+{\widehat{y}}_{p,i}}}\right)+\left(1-y_{p,i}\right)ln\left({\displaystyle \frac{1-y_{p,i}}{1-{\widehat{y}}_{p,i}}}\right)\right],$$

(2)

where $y_{p,i}\in (-1,1)$ is the output of the ith node in the output layer (the Mth layer), and $J_M$ is the number of nodes in the output layer.

For logistic activation functions, it is expressed as [110]

$$E_p\left(\mathbf{W}\right)={\displaystyle \frac{1}{2}}\sum _{i=1}^{J_M}\left[y_{p,i}ln\left({\displaystyle \frac{y_{p,i}}{{\widehat{y}}_{p,i}}}\right)+\left(1-y_{p,i}\right)ln\left({\displaystyle \frac{1-y_{p,i}}{1-{\widehat{y}}_{p,i}}}\right)\right],$$

(3)

with $y_{p,i}\in (0,1)$. Here, $y_{p,i}$, ${\widehat{y}}_{p,i}$, $1-y_{p,i}$, and $1-{\widehat{y}}_{p,i}$ represent probabilities. These error functions are strictly positive unless $y_{p,i}={\widehat{y}}_{p,i}$ for all $i=1,\dots ,J_M$. A simplified version, omitting constants, is provided in [111],

$$E_p\left(\mathbf{W}\right)=-{\displaystyle \frac{1}{2}}\sum _{i=1}^{J_M}\left[y_{p,i}ln{\widehat{y}}_{p,i}+\left(1-y_{p,i}\right)ln\left(1-{\widehat{y}}_{p,i}\right)\right].$$

(4)

Training neural networks is computationally hard (NP-complete) [112,113], with no algorithm guaranteeing optimal solutions in polynomial time. A single neuron using the logistic function and the MSE criterion may exhibit up to ${\left({\displaystyle \frac{N}{J_1}}\right)}^{J_1}$ local minima, for N training patterns and an input dimension of $J_1$ [114]. In contrast, the entropic error function creates a convex error landscape with a single minimum, significantly reducing local minima [109,111].

Entropy-based error functions in backpropagation algorithms address flat spots in the error surface without increasing computational costs, shortening training time and lowering local minima density [110]. They are particularly effective for probabilistic data, enabling the learning of hypothesis probabilities from neuron outputs [109,111].

In classification, using hard 0/1 target vectors with backpropagation risks weight saturation and overfitting. Generalization depends more on weight magnitudes than hidden node count [115], and constraining weights helps mitigate overfitting [115].

The cross-entropy loss function, grounded in the maximum likelihood principle, is expressed as

$$E_{\mathrm{CE}}=-\sum _{p=1}^N\sum _{c=1}^C{t}_p^{c}ln\left(y_p^c\right),$$

(5)

where C is the number of classes, $t_p^c\in \{0,1\}$ is the target value of the pth pattern in the cth class, and $y_p^c$ is the predicted probability of the pth pattern belonging to the cth class.

Cross-entropy loss enhances convergence and reduces local minima due to its steep gradient [110,111]. Unlike the mean squared error, which exaggerates small output errors, cross-entropy effectively estimates probabilities for rare events [111,116,117]. Networks trained with MSE or cross-entropy approximate posterior probabilities, yielding Bayesian-optimal results in large datasets, though flat regions in the weight space may hinder misclassification minimization.

MSE, assuming Gaussian-distributed targets, is less suitable for discrete labels. However, with sufficient samples and a 1-out-of-C coding scheme, MSE-trained networks can approximate posterior probabilities [116]. Cross-entropy and entropy-based losses are preferred for classification, optimizing probabilistic outputs more effectively. Classification-specific error functions [118] focus on misclassification, propagating gradients only for errors, thereby reducing weight adjustments and mitigating overfitting and saturation risks.

The MSE criterion can be extended using the Minkowski-r metric [119],

$$E_p={\displaystyle \frac{1}{r}}\sum _{i=1}^{J_M}{\left|{\widehat{y}}_{p,i}-y_{p,i}\right|}^r.$$

(6)

For $r=1$, this becomes the city block metric, while $r=2$ corresponds to MSE. Smaller r values reduce sensitivity to outliers, while larger r values emphasize significant deviations, benefiting low-noise or well-clustered data.

A generalized error function, adaptable via a real-valued parameter, is proposed in [120]. Other robust criteria, such as correntropy [121], handle non-Gaussian noise, while robust statistics [122] and regularization [123] offer further adaptations.

In regression, square loss is common for both training and testing, while classification uses hinge or logistic loss for training and 0-1 loss for testing. Despite the similar performance between square and cross-entropy loss across architectures, datasets, and domains, cross-entropy remains preferred for training modern neural networks [124].

Robust Learning

When training data contains noise or outliers, traditional algorithms may fail due to the disproportionate impact of outliers on MSE, which can lead the network to overfit these points [122]. The Student-t distribution, with heavier tails, is less sensitive to such deviations. Robust statistics [122] provide methods like the M-estimator, which modifies maximum likelihood to handle unknown underlying distributions by replacing squared error with specialized loss functions, reducing the influence of outliers.

The cost function for robust learning algorithms is given by

$$E_r=\sum _{i=1}^N\sigma \left({ϵ}_i;\beta \right),$$

(7)

where $\sigma (·)$ represents a symmetric loss function that achieves its unique minimum at zero, $\beta >0$ is the cutoff parameter serving as a scale estimator, ${ϵ}_i$ is the error associated with the ith training sample, and N is the total number of samples in the dataset.

Common loss functions include the logistic function [122], Huber’s function [122], Talwar’s function [125], and Hampel’s tanh estimator [126]. The logistic function is given by

$$\sigma ({ϵ}_i;\beta )={\displaystyle \frac{\beta }{2}}ln\left(1+{\displaystyle \frac{{ϵ}_i^2}{\beta }}\right),$$

(8)

and Huber’s function is

$$\sigma ({ϵ}_i;\beta )=\left\{\begin{array}{cc}{\displaystyle \frac{1}{2}}{ϵ}_i^2,\hfill & \mathrm{if}\left|{ϵ}_i\right|\le \beta \hfill \\ \beta \left|{ϵ}_i\right|-{\displaystyle \frac{1}{2}}{\beta }^2,\hfill & \mathrm{if}\left|{ϵ}_i\right|>\beta \hfill \end{array}\right..$$

(9)

Using the gradient-descent optimization method, weight updates are computed as

$$\Delta w_{jk}=-\eta {\displaystyle \frac{\partial E_r}{\partial w_{jk}}}=-\eta \sum _{i=1}^N\phi \left({ϵ}_i;\beta \right){\displaystyle \frac{\partial {ϵ}_i}{\partial w_{jk}}},$$

(10)

where $\eta$ denotes the learning rate, and $\phi (·)$, referred to as the influence function, is expressed as

$$\phi \left({ϵ}_i;\beta \right)={\displaystyle \frac{\partial \sigma \left({ϵ}_i;\beta \right)}{\partial {ϵ}_i}}.$$

(11)

For the standard MSE function, $\sigma \left({ϵ}_i\right)={\displaystyle \frac{1}{2}}{ϵ}_i^2$ and the corresponding influence function is $\phi \left({ϵ}_i;\beta \right)={ϵ}_i$. To reduce the impact of large errors, robust loss functions are designed so that $\phi \left({ϵ}_i;\beta \right)$ behaves sublinearly.

The $\tau$-estimator [127] is a variant of the M-estimator that integrates an adaptive, bounded influence function $\phi (·)$. This function is a weighted average of a robust function ${\phi }_1(·)$ and an efficient function ${\phi }_2(·)$. This estimator provides a high breakdown point while maintaining efficiency under Gaussian errors.

When initial weights are poorly chosen, loss functions may fail to detect outliers effectively, and selecting the parameter $\beta$ can be challenging. One approach to define $\beta$ is using the median of absolute deviations (MAD),

$$\beta =c\times \mathrm{median}\left(\left|{ϵ}_i-\mathrm{median}\left({ϵ}_i\right)\right|\right),$$

(12)

where $c\approx 1.4826$ [122]. Other methods involve utilizing the median of all errors or designating a specific percentage of data points as outliers [122,126].

The C-loss function, or correntropy-induced loss, is a robust method for outlier handling [128]. It is bounded, smooth, differentiable, nonconvex, and consistent with Bayesian principles. The C-loss behaves similarly to the $L_2$-norm for small errors, approximates the $L_1$-norm for moderate errors, and closely resembles the $L_0$-norm for large errors.

4. Learning and Generalization

Learning reconstructs a hypersurface from examples, while generalization estimates values on that surface for unseen points, corresponding to nonlinear curve fitting, interpolation, and extrapolation.

Training neural networks aims to develop statistical models that capture the data’s generative process rather than replicate the training data exactly. The mapping reconstruction problem is well-posed if each input has a unique, continuous output, but learning is often an ill-posed inverse problem. Regularization stabilizes the solution with nonnegative functional constraints [123,129].

Overfitting happens when a model achieves high performance on training data but generalizes poorly to new, unseen data, often due to excessive parameters, examples, or epochs. Simpler networks with smoother mappings typically generalize better. Generalization depends on dataset size, problem complexity, and network architecture.

The training set size N must adequately represent the problem, with $N\gg {\displaystyle \frac{N_w}{N_y}}$ for effective generalization, where $N_w$ is the number of weights, and $N_y$ is the number of outputs [130]. Feature extraction reduces dimensionality, mitigating the curse of dimensionality and enhancing generalization.

4.1. Generalization Error

The generalization error of a trained neural network consists of two components: an approximation error, due to the finite number of parameters and inherent noise in the training data, and an estimation error, caused by the limited data points [131]. For a feedforward network consisting of $J_1$ input nodes and one output node, the generalization error can be bounded by the hypothesis parameters, $N_P$, and the number of training examples, N [131],

$$O\left({\displaystyle \frac{1}{N_P}}\right)+O\left({\left[{\displaystyle \frac{N_P{J}_{1}ln\left(N_{P}N\right)-ln\delta }{N}}\right]}^{1/2}\right),\mathrm{with}\mathrm{probability}p>1-\delta ,$$

(13)

where $\delta \in (0,1)$ represents the confidence level, and $N_P$ is the number of parameters. The first term reflects the approximation error, while the second term relates to the estimation error.

As $N_P$ grows, the approximation error reduces, but the estimation error increases due to overfitting. The optimal model size balancing both errors is $N_P\propto N^{{\displaystyle \frac{1}{3}}}$ [131]. The generalization error for feedforward networks is expected to be $O\left({\displaystyle \frac{1}{N_P}}\right)$, consistent with MLPs using sigmoidal functions [132].

Standard gradient methods [133] do not overfit on separable data. After T iterations on a dataset of size N, empirical risk and generalization error both decline at a rate of $O(1/{\gamma }^{2}T)$ until T approaches N, where $\gamma$ is the margin. Beyond this point, the generalization error stabilizes at $O(1/{\gamma }^{2}N)$. Non-asymptotic bounds on margin violations are also provided in [133].

4.2. Generalization by Stopping Criterion

Generalization can be managed by halting training before reaching the absolute minimum, thereby preventing overfitting. Neural networks trained iteratively incorporate features hierarchically, and stopping at the right moment avoids fitting high-frequency noise, while training error decreases monotonically, generalization error initially declines to a minimum before rising, emphasizing the importance of early stopping [134]. Cross-validation is commonly used to identify the optimal stopping point, with slower criteria offering marginal generalization improvements at the expense of longer training times [134].

For three-layer MLPs, satisfactory generalization occurs when $N>30N_w$, where $N_w$ is the number of weights [135]. If $N<N_w$, early stopping mitigates overfitting, although risks remain if $N<30N_w$. Cross-validation aids in selecting the stopping point in such scenarios.

Advanced methods, like the signal-to-noise ratio figure (SNRF) criterion, automatically detect overfitting based on training error alone [136]. Moreover, optimal stopping rules refine error bounds in iterative regularization, with early stopping shown to be equivalent to Tikhonov regularization when aligned with its penalties. Cross-validation-based rules effectively address unknown parameters, achieving comparable generalization errors to traditional methods [137].

4.3. Generalization by Regularization

Regularization enhances generalization by assuming the target function is smooth, where small input variations yield minor output changes. Regularization reduces variance by increasing bias, following the bias–variance trade-off [138]. Effective regularizers minimize variance while limiting bias impact [139].

A constraint term $E_c$ is added to the training cost function E, forming the total cost function,

$$E_T=E+{\lambda }_c{E}_c,$$

(14)

where ${\lambda }_c>0$ balances error minimization and smoothness. Unlike early stopping, regularization applies to both iterative gradient-based methods and direct optimization techniques, such as singular value decomposition (SVD).

Network-pruning techniques, like weight decay, enhance generalization by pruning near-zero weights while retaining significant ones [140,141]. Biases are excluded from penalties to ensure unbiased target mean estimates.

Early stopping also acts as a regularization method, especially with the MSE function [139]. Here, the term ${\displaystyle \frac{1}{\eta t}}$, where $\eta$ denotes the learning rate and t is the iteration index, serves as the regularization parameter, with effective non-zero weights increasing during training.

Introducing jitter to input data enhances generalization by serving as a form of smoothing regularization, with the noise variance acting as the regularization parameter [139,141]. This technique helps prevent overfitting in large networks that lack sufficient constraints. Noise injection [140] establishes an equivalence between ridge regularization and Gaussian noise augmentation, linking regularization strength to noise variance. Incorporating noise into training, as proposed by [142], creates an infinite supply of examples, enhancing generalization, accelerating backpropagation, and reducing local minima risks.

Another approach proposed by [143] reduced model complexity by encoding weights with shorter bit lengths, balancing squared error minimization with reduced weight information via Gaussian noise adjustment. Weight sharing ties multiple weights to a single parameter, reducing parameters and improving generalization [144]. Soft weight-sharing, introduced in [145], uses regularization to determine which weights to tie. Additionally, dropout introduces implicit regularization by randomly deactivating intermediate neurons, akin to enforcing sparsity regularization.

Implicit regularization describes how optimization algorithms shape generalization through their preferred solutions. Gradient descent induces $L_2$-norm regularization but is limited to specific settings. Mirror descent, a generalization of gradient descent, provides a unified framework for controlling implicit regularization in regression and classification [146].

In infinite-dimensional linear models with convex, differentiable loss functions, implicit regularization has been analyzed through high-probability bounds on the excess risk of batch gradient descent iterates [147]. This work shows that optimization itself regularizes learning without explicit penalties, complementing earlier studies addressing statistical and optimization challenges in deep learning [148,149,150].

Unsupervised pretraining acts as a regularizer for deep learning algorithms [151], with theoretical analysis interpreting it as part of Tikhonov-regularized batch learning methods [152]. The method jointly optimizes predictors and neural network parameters to produce Tikhonov matrices. Learning meaningful matrices through pretraining enhances stability and improves generalization as sample size increases [152].

4.4. Data Augmentation

Data augmentation enhances supervised and self-supervised learning by improving generalization. While traditionally linked to generating data from the same distribution [153,154], recent techniques like randomized masking [155] and cutout [156] challenge this notion by significantly altering distributions [155]. Adding Gaussian noise aligns with Tikhonov regularization [140] and vicinal risk minimization [157,158], linking augmentation to explicit regularization, particularly in underparameterized models, where it reduces variance and mitigates noise overfitting.

In [159], a framework is proposed to analyze augmentation’s effects on generalization for underparameterized and overparameterized linear models, drawing from overparameterized learning theory [160,161]. Two mechanisms are identified: modifying eigenvalue proportions in the data covariance matrix, and amplifying the spectrum via ridge regression. These explain generalization differences across model complexities and tasks.

In underparameterized models, augmentation reduces variance with minimal bias, enhancing generalization. Overparameterized regression, however, often faces significant bias and distributional shifts, degrading performance. These biases are mitigated under 0-1 loss, improving classification tasks [159].

Randomized augmentation techniques, including random masking [155], cutout [156], noise injection [140], and group-invariant augmentations [153], are examined in [159]. Noise injection induces constant spectral shifts, while masking, cutout, and distribution-preserving augmentations isotropize the data spectrum. Random feature rotation [159] achieves lower bias than least-squares estimators with variance reduction comparable to ridge regression. Stochastic data augmentation, implemented on-the-fly via augmented stochastic gradient descent (SGD) [153,154], addresses augmented empirical risk minimization.

Distinct robustness to augmentation hyperparameters between regression and classification tasks, as well as differences between precomputed [162,163] and online augmentation methods, is highlighted in [159].

Augmentations producing identically distributed samples generally enhance generalization. A group-theoretic framework introduced in [153] emphasizes variance reduction for minimally disruptive augmentations in underparameterized models. Building on this, the analysis of out-of-distribution (OOD) augmentations shows that their spectral effects and biases can either enhance or impair generalization in overparameterized models [159].

A Markov process framework linking Bayes-optimal classifiers to augmentation-dependent kernel classifiers is proposed in [154], identifying data-dependent regularization. Deterministic linear augmentations expand the training span and induce regularization in regression [163], while local augmentations reduce network rugosity [164]. Permutation-style augmentations in two-layer convolutional networks enhance feature learning in classification tasks [162]. Random mask [155], cutout [156], and random rotation augmentation [159] achieve comparable generalization errors across hyperparameters, often outperforming ridge regularization, with random mask [155] removing covariates.

4.5. Dropout

Dropout, introduced by [165], mitigates overfitting in deep networks by randomly removing units and connections during training. This prevents fragile co-adaptations, enhancing generalization. Dropout effectively samples from numerous thinned networks, approximating their combined effect at test time with a single network using scaled weights. Retaining units with a fixed probability p adds noise, acting as regularization. Empirical studies suggest optimal performance by dropping 20% of input units and 50% of hidden units.

Dropout can be viewed as a stochastic regularization method, with a deterministic variant achieved by marginalizing the noise. In the case of linear regression, this regularization is a modified version of $L_2$ regularization [165]. An extension of dropout, called DropConnect [166], goes further by randomly zeroing elements of the weight matrix $\mathbf{W}$, rather than dropping neuron outputs.

Dropout prevents overfitting by randomly masking neurons during training, with each neuron deactivated based on a Bernoulli distribution. The method is often linked to explicit regularization [167,168,169,170,171,172,173].

Several theories explain the success of dropout. Ref. [174] likens it to sexual reproduction in evolution, where offspring inherit a combination of genes, suggesting dropout is most effective at $p=0.5$. It controls network complexity by limiting weight co-adaptation, enabling simpler functions in deeper layers [175]. Ref. [168] interprets dropout as an ensemble method combining various network topologies. Ref. [176] frames dropout as approximate Bayesian inference in deep Gaussian processes, minimizing Kullback–Leibler divergence between the approximate and true posteriors.

Dropout with rectified linear unit (ReLU) activation and quadratic loss differs from weight decay regularization [177], preventing co-adaptation of weights. Dropout’s penalty grows exponentially with depth, whereas weight decay grows linearly. Additionally, dropout is less sensitive to feature, output, and weight rescaling, suggesting no isolated local minima.

Spectral dropout [178] regularizes by eliminating weak and noisy Fourier coefficients, outperforming traditional techniques. This approach can be seen as a convolutional layer using a fixed basis function for decorrelation.

Uncertainty estimation methods include Bayesian approximations, ensemble approaches, and parametric models. Monte Carlo dropout [176] is a popular Bayesian approximation, with a more efficient variant, last-layer dropout [179]. Parametric uncertainty Monte Carlo [179] uses dropout samples from a parametric model to aggregate predictions for Gaussian parameters. Ref. [176] reinterpret dropout as Bayesian variational inference approximating a deep Gaussian process prior.

Wasserstein dropout [180] is a non-parametric approach for regression, modeling heteroscedastic noise and aleatoric uncertainty by generating sub-network distributions through dropout and minimizing the Wasserstein distance between predicted and true data distributions.

Standard dropout’s fixed random activation does not replicate the adaptive neuron activation of the human cerebral cortex. Inspired by gene theory, an adaptive dropout method [181] uses a variational autoencoder to dynamically adjust dropout probabilities, preserving critical features like CNN edges while deactivating less essential neurons. This approach, building on [165], enhances robustness, reduces overfitting, and improves training efficiency by leveraging diverse sub-networks.

Non-asymptotic bounds for gradient descent with dropout in linear regression show that additional randomness persists with fixed learning rates. The gap between dropout and $L_2$-penalization vanishes in Ruppert–Polyak averages. A simplified dropout model converges to the least-squares estimator without regularization, providing insights into strategies for extending to randomized optimization methods [182].

For two-layer models, marginalizing dropout noise introduces a nuclear norm penalty in matrix factorization [169] and linear neural networks, which generalizes to $L_2$-path regularizers in deep linear [171] and shallow ReLU networks [167]. Dropout generalization bounds [166,183,184] associate dropout rates with a reduction in Rademacher complexity, while bounds on complexity for shallow ReLU networks are derived in [167]. PAC-Bayes bounds [170] highlight trade-offs between dropout rates.

Exponential convergence with marginalized dropout in shallow linear networks is shown [172], while dropout in gradient descent on logistic loss for shallow ReLU networks is analyzed [185], with misclassification error rates derived in the lazy regime for overparameterized networks.

In generalized linear models, dropout training is shown to be the minimax solution to a zero-sum game where an adversary corrupts covariates with a multiplicative errors-in-variables model. The least favorable distribution is dropout noise, with covariates deleted with probability $\delta$, providing out-of-sample loss guarantees for perturbed data distributions, and offering a method for selecting $\delta$ along with a parallelizable multi-level Monte Carlo algorithm for faster dropout training [186].

4.6. Fault Tolerance and Generalization

Noise introduction during training enhances both generalization and fault tolerance: input noise improves generalization [140], and synaptic noise strengthens fault tolerance [187]. These properties are interconnected; improving one often enhances the other [188,189]. Reducing weight magnitudes increases both fault tolerance [188,190] and generalization [190,191]. VC-dimension theory explains how redundancy improves these attributes [192].

Fault tolerance relies on distributing learning across neurons, a task not inherently optimized by the BP algorithm [189]. Weight perturbations during training improve fault tolerance in MLPs [187,189], similarly to how input noise enhances generalization. Lower saliency, derived from the Hessian matrix, indicates higher fault tolerance [187]. Metrics for MSE degradation under perturbations help refine understanding of noise immunity and generalization [190].

Online node fault injection, where hidden nodes output zero randomly during training, enhances fault tolerance in MLPs, with convergence proofs [193]. This method incorporates terms for MSE, regularization, and weight decay. Fault injection studies in RBF networks show that weight noise (additive or multiplicative) does not improve fault tolerance, although additive input noise does improve generalization by resembling Tikhonov regularization [194]. Stuck-at faults, where a node’s output is fixed, are tolerated by well-trained networks, demonstrating robustness [188,195].

4.7. Sparsity Versus Stability

Stability plays a crucial role in determining the generalization performance of an algorithm [196]. Sparsity and stability are both highly desirable characteristics for learning algorithms, as they both contribute to enhanced generalization. However, these two properties are inherently contradictory, as demonstrated by the no-free-lunch theorem [197,198]. Specifically, a sparse algorithm cannot also exhibit stability, and vice versa. Sparse algorithms may have multiple optimal solutions, making them ill-posed. For sparse algorithms, uniform stability is inherently constrained by a nonzero lower bound, reinforcing the conclusion that sparsity and stability are fundamentally incompatible. As a result, a balance must be struck between sparsity and stability when designing learning algorithms.

In [197,198], it is shown that an $L_1$-regularized regression (LASSO, least absolute shrinkage and selection operator) lacks stability, while $L_2$-regularized regression is stable but not sparse. Algorithms that promote sparsity include $L_1$-norm SVM, LASSO, sparse PCA, and deep belief networks.

5. Model Selection

Model selection identifies the simplest model that balances data fit and generalization, a key model quality measure. Selection strategies include cross-validation, complexity-based criteria, regularization, and pruning/growing. Generalization error can be estimated through cross-validation or bootstrapping. Cross-validation evaluates models of varying complexities on validation sets but is resource-intensive, while complexity-based criteria avoid validation sets yet remain computationally demanding. Regularization efficiently penalizes complexity, but may yield suboptimal representations. Pruning and growing, often combined with regularization, rely on assumptions that can restrict model flexibility.

5.1. Occam’s Razor

Occam’s razor, attributed to William of Occam, advocates for simplicity in model selection: entities should not be multiplied beyond necessity. It suggests that simpler models are preferable when they fit the data similarly to more complex models, particularly in noisy environments, as they often generalize better.

While often seen as a principle of simplicity, Occam’s razor implies that simpler models generalize better when training performance is equivalent. However, overfitting or poor generalization is argued to be more closely related to the number of models evaluated by the learner, rather than to model complexity [199]. Complex models typically require more evaluations, creating a confounding relationship between complexity and generalization.

Empirical evidence from [200] supports this view, showing that: (1) models chosen from larger candidate pools compared to those from smaller pools, provided their complexity remains unchanged; and (2) more complex models are not inherently more likely to overfit than simpler ones, as long as the size of the candidate pool being evaluated stays fixed.

5.2. Cross-Validation

Cross-validation, a common model selection method [201], splits the dataset into a training set and a validation set (typically 10–20% of the data). Leave-one-out cross-validation uses a single sample for validation, while providing accurate performance estimates [134], cross-validation is computationally expensive due to repeated training, despite reducing variance in estimates.

Consider a dataset partitioned into n subsets, with ${\mathcal{D}}_i$ and ${\overline{\mathcal{D}}}_i$ representing the training and testing subsets, respectively, for the ith partition. During cross-validation, the model is trained n times to minimize the log-likelihood,

$$E_{\mathrm{CV}}=-{\displaystyle \frac{1}{n}}\sum _{i=1}^{n}ln\left(L\left(\widehat{\mathbf{W}}\left({\overline{\mathcal{D}}}_i\right)|{\mathcal{D}}_i\right)\right),$$

(15)

where $\widehat{\mathbf{W}}\left({\overline{\mathcal{D}}}_i\right)$ are the maximum likelihood estimates from the test set ${\overline{\mathcal{D}}}_i$, and $L\left(\widehat{\mathbf{W}}\left({\overline{\mathcal{D}}}_i\right)|{\mathcal{D}}_i\right)$ is the likelihood evaluated on the training set ${\mathcal{D}}_i$.

Validation uses a separate dataset to estimate generalization error, helping avoid overfitting and identifying the best model. Cross-validation, especially for large training sets (N), supports generalization [202].

In K-fold cross-validation, the dataset $\mathcal{D}$ is divided into K partitions, where one partition serves as the validation set while the remaining partitions are used for training. The error is averaged across folds, and variance is influenced by data distribution [203]. Nearly unbiased estimators are obtained for smooth loss functions and absolute error loss [204], with a practical 25% data test split [204]. K-fold cross-validation can introduce bias in least-squares regression, but K-fold penalization corrects this bias [205]. A K value greater than 5 stabilizes variance, with $K=5$ commonly recommended [206].

Monte Carlo cross-validation replaces K with B random splits, with variance estimators studied [205]. Leave-many-out cross-validation outperforms leave-one-out, particularly for small samples, although resampling reduces variance in both [207]. Cross-validation can be inconsistent for model selection [208], and the moment approximation estimator [204] generally outperforms the Nadeau–Bengio estimator [209], though the latter is simpler.

Cross-validation and bootstrapping differ in resampling methods: bootstrapping resamples with replacement, while cross-validation does so without. Cross-validation estimates generalization error, while bootstrapping estimates error bars and confidence intervals. In small sample sizes, cross-validation has higher variance than bootstrapping, despite being more accurate and less biased [207].

5.3. Complexity Criteria

An effective generalization strategy involves compact network development through parsimonious model selection, using methods like final prediction error (FPE) [210], Akaike information criterion (AIC) [211], Bayesian information criterion (BIC) [212], and minimum description length (MDL) principle [213]. These combine training error measurement with a penalty for model complexity.

Model order is typically determined by minimizing the Kullback–Leibler (KL) divergence between the true and model pdfs, or equivalently, maximizing the relative KL information. Asymptotic approximations of this lead to minimizing the AIC, which is formulated by optimizing an asymptotically unbiased estimator of the relative KL information, I. The BIC rule, similarly derived, is a penalized maximum likelihood method [214].

AIC minimizes prediction error, while BIC focuses on model selection for inference. The formulations for AIC and BIC are given by

$$E_{\mathrm{AIC}}=-{\displaystyle \frac{1}{N}}ln\left(L_N\left({\widehat{\mathbf{W}}}_N\right)\right)+{\displaystyle \frac{N_P}{N}},$$

(16)

$$E_{\mathrm{BIC}}=-{\displaystyle \frac{1}{N}}ln\left(L_N\left({\widehat{\mathbf{W}}}_N\right)\right)+{\displaystyle \frac{N_P}{2N}}lnN,$$

(17)

where $L_N\left({\widehat{\mathbf{W}}}_N\right)$ is the likelihood for a dataset of size N with parameters ${\widehat{\mathbf{W}}}_N$, and $N_P$ is the number of model parameters.

These criteria can be reformulated in terms of empirical risk [214],

$$AIC\left(N_P\right)=R_{\mathrm{emp}}\left(N_P\right)+{\displaystyle \frac{2N_P}{N}}{\widehat{\sigma }}^2,$$

(18)

$$BIC\left(N_P\right)=R_{\mathrm{emp}}\left(N_P\right)+{\displaystyle \frac{N_P}{N}}{\widehat{\sigma }}^{2}lnN,$$

(19)

where ${\widehat{\sigma }}^2$ is the estimated noise variance and $R_{\mathrm{emp}}\left(N_P\right)$ is the empirical risk,

$$R_{\mathrm{emp}}\left(N_P\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-f({\mathbf{x}}_i,N_P))}^2.$$

(20)

For a linear model with $N_P$ parameters, the noise variance is estimated by

$${\widehat{\sigma }}^2={\displaystyle \frac{N}{N-N_P}}{\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2.$$

(21)

This leads to the final prediction error (FPE) form of AIC [215],

$$\mathrm{FPE}\left(N_P\right)={\displaystyle \frac{1+\frac{N_P}{N}}{1-\frac{N_P}{N}}}{R}_{\mathrm{emp}}\left(N_P\right).$$

(22)

The MDL principle formalizes Occam’s razor by seeking the most concise dataset description with minimal symbols [213,216]. The model’s description length includes the code cost for inputs, the model cost, and the reconstruction error, with the optimal model minimizing this total length. MDL applies in unsupervised learning methods like competitive learning and PCA [143], where it encourages good generalization by encoding weights with shorter bit-lengths. The MDL criterion is similar to a Bayesian approximation, with BIC derived from coding arguments related to MDL.

Generalization error $Err$ combines training error $err$ and optimism $OP$ in an estimate [217], with complexity criteria like BIC used to estimate $OP$.

6. Bias and Variance

The generalization error comprises two components: the squared bias and the variance [138]. Supervised learning algorithms commonly encounter the bias–variance trade-off [138], where the goals of minimizing bias and variance are often in conflict, requiring a balance between the two.

Let $f\left(\mathbf{x};\widehat{\mathbf{w}}\right)$ represent the optimal model within the model space, where $\widehat{\mathbf{w}}$ is independent of the training dataset. The bias and variance are expressed as [139]

$$\mathrm{bias}=E_{\mathcal{S}}\left[f\left(\mathbf{x}\right)\right]-f(\mathbf{x};\widehat{\mathbf{w}}),\mathrm{var}=E_{\mathcal{S}}\left[{\left(f\left(\mathbf{x}\right)-E_{\mathcal{S}}\left[f\left(\mathbf{x}\right)\right]\right)}^2\right],$$

(23)

where $f\left(\mathbf{x}\right)$ is the target function, and $E_{\mathcal{S}}$ is the expectation over all possible training datasets. Bias arises from an inappropriate level of model complexity when assuming an infinite number of training samples, while variance is caused by the finite size of the training data.

The generalization error can be expressed as

$$\begin{array}{ll}{E}_{\mathcal{S}}\left[{\left(f\left(\mathbf{x}\right)-f(\mathbf{x},\widehat{\mathbf{w}})\right)}^2\right]& =E_{\mathcal{S}}\left[{\left(\left(f\left(\mathbf{x}\right)-E_{\mathcal{S}}\left[f\left(\mathbf{x}\right)\right]\right)+\left(E_{\mathcal{S}}\left[f\left(\mathbf{x}\right)\right]-f(\mathbf{x},\widehat{\mathbf{w}})\right)\right)}^2\right]\end{array}$$

(24)

$$\begin{array}{lll}& =& {\left(\mathrm{Bias}\right)}^2+\mathrm{Var}.\end{array}$$

(25)

A network with too few parameters may underfit (high bias, low variance), while one with too many parameters may overfit (low bias, high variance). Balancing bias and variance is essential for optimal generalization, achievable through model complexity selection or regularization. Nonparametric methods aim to reduce variance. Increasing hidden units reduces bias but raises variance.

For three-layer feedforward networks containing $N_P$ hidden sigmoidal units, the bias is bounded by $O\left({\displaystyle \frac{1}{N_P}}\right)$, while the variance is bounded by $O\left({\displaystyle \frac{N_P{J}_{1}lnN}{N}}\right)$ [132]. Here, N denotes the training set size and $J_1$ is the feature vector dimensionality. A larger $N_P$ reduces bias, but may lead to overfitting, highlighting the bias–variance trade-off.

Unbiasedness in model selection is important, but controlling variance is crucial, as excessive variance can lead to overfitting [218], mitigated by regularizing the selection criterion [219].

Variance and bias reduction are vital in Monte Carlo simulation, with control variates and importance sampling as key techniques. The control variate method reduces variance by incorporating an auxiliary variate with a known mean [220]. Coefficients for control variates are typically estimated using least squares [221], and adding more control variates accelerates convergence [222], while regularized least squares enhances precision [223].

7. Overparameterization and Double Descent

SVM generalization error in overparameterized settings has been analyzed using the convex Gaussian min-max theorem for precise asymptotics [224,225,226]. Upper bounds for max-margin SVMs in overparameterized linear discriminant models emphasize anisotropy for better generalization [227]. Overparameterization in linear models with Gaussian features turns all training points into support vectors [161], with least-squares minimum-norm interpolation outperforming hard-margin SVM solutions under 0-1 loss. Overparameterized models generalize better in classification than regression under milder conditions.

Overparameterized models, including deep neural networks, achieve near-zero training loss and strong generalization, creating a ‘generalization paradox’ [148,228,229,230]. This is linked to double descent [228,229,231], as illustrated in Figure 2, where test error decreases, rises near the interpolation threshold, and declines again with increased overparameterization. The generalization paradox contrasts the classical bias–variance regime (first descent) with the modern regime (second descent) [115,148,228,230], with VC-dimension potentially differing from the number of parameters [9,232,233].

Double descent highlights overparameterization’s benefits, where effective interpolation relies on feature families aligning critical signal directions while absorbing noise, emphasizing the complex relationship between overparameterization and generalization. For deep ReLU networks, the adoption of ReLU function makes many neuron become inactive and output zero, and a large portion of the parameters do not work. This properly explains the double descent phenomenon in deep learning [234].

Test error variance in two-layer networks has been analyzed via analysis of variance (ANOVA), decomposing contributions from initialization, label noise, and training data [231], diverging from traditional theory [235] that attributes variance solely to the training set. Interaction between samples and initialization dominates variance, with algorithms contributing to implicit regularization, reducing complexity and enhancing generalization [231]. Overparameterized models exhibit implicit regularization, imposing structural constraints [236,237].

Theoretical studies show benign overfitting in regression tasks, with empirical risk minimization leading to good generalization despite noisy data [160,238,239,240]. Generalization error in two-layer networks has been studied via ANOVA, with interaction effects dominating variance under Gaussian initialization [241]. Variance is unimodal with respect to parameterization, influenced significantly by label noise and initialization [242,243].

Efforts to reconcile classical theory with modern overparameterized models reflect nonparametric statistics principles [244], such as kernel smoothing, which uses many parameters for strong performance. This aligns with the “overparametrize-then-regularize” strategy in machine learning, akin to techniques like basis pursuit and Lasso [54].

In overparameterized settings, SGD can lead to benign overfitting without explicit regularization. Sharp excess risk bounds for constant-stepsize SGD in linear regression link variance to an SGD-specific effective dimension and bias to the initial iterate’s alignment with the covariance structure [245]. Tail averaging in SGD improves risk bounds, and algorithmic regularization in unregularized SGD is contrasted with ordinary least squares and ridge regression.

Three mechanisms for controlling generalization include maintaining a small feature set relative to training size, minimizing VC-dimension for linearly separable data, and maximizing data compression efficiency when encoding the training set [233]. The double descent phenomenon in deep learning, where large networks achieve perfect training fit and strong generalization, challenges classical VC-theory, which requires finite VC-dimension for generalization [148,228,230,246,247,248,249].

Weight initialization, normalization, decay, and epochs influence VC-dimension, known as SGD’s ‘implicit bias’. Generalization is controlled by weight norms [236,237,246,250], with gradient descent maximizing the margin on linearly separable data, in line with VC-theoretical explanations [237]. High-dimensional data are controlled by weight norms [236,237,249,250,251,252,253], while low-dimensional data benefits from data compression or MDL. VC-theory concepts, including radius-margin and leave-one-out bounds, provide insights into generalization for second descent, with weight norms governing generalization [249].

Double descent is explained within the VC-theoretical framework using VC-dimension and structural risk minimization (SRM), with generalization curves aligning with classical VC bounds. This analysis provides insights into generalization for noisy or high-dimensional data and extends to transfer learning in pre-trained networks [254]. A conceptual interpretation of double descent is presented using VC-bound (56) [254], where minimizing this bound via data-dependent SRM is linked to fat-shattering dimension for large-margin hyperplanes [255]. During second descent, as the number of parameters increases, VC-dimension is controlled by the weight norm and decreases for larger networks [254]. However, results from noisy, high-dimensional datasets indicate that second descent does not always enhance generalization. In fact, memorizing noisy data can increase the VC-dimension which, in turn, leads to a higher VC-bound. The VC-bound-based theoretical explanation of double descent is further validated for classifiers such as SVMs, least squares, and MLPs [256].

The optimal generalization strategy depends on training data characteristics, with VC-bounds guiding the selection of an optimal approach. Empirical studies suggest second descent is effective for high-dimensional data, though the reasons remain unclear [254]. In under-parameterized networks, the VC-dimension increases linearly with the number of features $N_f$, which aligns with first descent. In contrast, for over-parameterized networks, the VC-dimension is influenced by the weight norm, leading to second descent. As $N_f$ grows, double descent curves emerge.

In transfer learning, pre-trained deep networks extract features for simpler domain-specific networks. VC-theory explains this in two steps: a pre-trained network learns $N_f$ non-linear features using general data, and a linear classifier is trained on these features with domain-specific data. Since the non-linear mapping does not depend on domain-specific data, VC-bound (56) applies to step 2 for generalization.

8. Neural Networks as Universal Machines

Neural networks excel at approximating complex functions. Feedforward neural networks can approximate any continuous function under suitable conditions, while RNNs with sigmoidal activations are Turing equivalent [257], meaning they can replicate any function computable by a Turing machine, matching the computational power of digital computers.

Reservoir computation models, a subclass of RNNs, have fixed inputs and dynamic coupling weights, with only the static readout being trainable, avoiding gradient propagation issues. These models serve as universal approximators for dynamic filters with time-invariant fading memory. Simple cycle reservoirs, with equal weight ring connectivity and binary input-to-reservoir weights, can universally approximate any unrestricted linear reservoir system and time-invariant fading memory filter over uniformly bounded input streams [258].

Differentiable neural computers enhance neural networks with external memory. The reservoir memory machine [259], an echo state network with external memory and convex optimization training, outperforms finite state machines and solves benchmark tasks for differentiable neural computers more effectively than traditional recurrent models, including deep networks.

Universal memcomputing machines (UMMs) integrate memory for both storage and processing, achieving Turing-completeness [260]. They are capable of simulating both liquid-state machines (LSMs) and quantum computers, making them liquid-complete and quantum-complete [261], thus offering a unifying framework for diverse computational models.

8.1. Boolean Function Approximation

Feedforward neural networks with binary neurons model Boolean functions by processing variables in binary ($0,1$) or bipolar ($-1,+1$) formats. For $J_1$ Boolean variables, $2^{J_1}$ combinations exist, yielding $2^{2^{J_1}}$ distinct Boolean functions. A linear threshold gate (LTG) separates two classes, and the function counting theorem [262,263] quantifies the linearly separable dichotomies for m points in a general position in ${\mathcal{R}}^n$, defining a single LTG’s classification capacity.

Theorem 1

(Function counting theorem). The count of distinct dichotomies that are linearly separable for m points, in general, position within ${\mathbb{R}}^n$, is expressed as

$$C(m,n)=\left\{\begin{array}{cc}2\sum _{i=0}^n\left({\displaystyle \genfrac{}{}{0pt}{}{m-1}{i}}\right),\hfill & \mathit{if}m>n+1,\hfill \\ 2^m,\hfill & \mathit{if}m\le n+1,\hfill \end{array}\right.$$

(26)

A set of m points in ${\mathbb{R}}^n$ is in general position if any subset of m or fewer points is linearly independent.

For m points, the total number of possible dichotomies is $2^m$. Assuming equiprobable dichotomies, the likelihood of an LTG with n inputs successfully separating m points in a general position is

$$P(m,n)={\displaystyle \frac{C(m,n)}{2^m}}=\left\{\begin{array}{cc}{\displaystyle \frac{2}{2^m}}\sum _{i=0}^n\left({\displaystyle \genfrac{}{}{0pt}{}{m-1}{i}}\right),\hfill & \mathrm{if}m>n+1,\hfill \\ 1,\hfill & \mathrm{if}m\le n+1.\hfill \end{array}\right.$$

(27)

The probability of a linear dichotomy, $P(m,n)$, equals 1 when ${\displaystyle \frac{m}{n+1}}\le 1$. For $1<{\displaystyle \frac{m}{n+1}}<2$, P approaches 1 as $n\to \infty$. At the critical ratio ${\displaystyle \frac{m}{n+1}}=2$, P drops to ${\displaystyle \frac{1}{2}}$. The value $m=2(n+1)$ is often employed to assess the statistical capacity of a single LTG. As $m/(n+1)$ further increases, P approaches 0.

A three-layer feedforward LTG network with a ($J_1$-$2^{J_1}$-1) structure can represent any Boolean function with $J_1$ inputs [264,265]. For a general function $f:{\mathbb{R}}^{J_1}\to \{0,1\}$ defined over N arbitrary points in ${\mathbb{R}}^{J_1}$, the minimum number of hidden nodes required is estimated as $O\left({\displaystyle \frac{N}{J_1{log}_2\frac{N}{J_1}}}\right)$ for $N\ge 3J_1$ and $J_1\to \infty$ [109,263]. When N points are, in general, position, this reduces to ${\displaystyle \frac{N}{2J_1}}$ as $J_1\to \infty$ [263]. Multi-hidden-layer networks are generally more size-efficient than single-hidden-layer designs [263].

Binary Radial Basis Function

In three-layer feedforward networks, a network using binary or generalized binary RBF activation functions in the hidden layer and LTG output neurons is called a binary or generalized binary RBF network, capable of modeling Boolean functions.

A generalized binary RBF neuron is defined by a center $\mathbf{c}\in {\mathbb{R}}^n$ and a non-negative radius $r\ge 0$, with an activation function $\varphi :{\mathbb{R}}^n\to \{0,1\}$, defined as

$$\varphi \left(\mathbf{x}\right)=\left\{\begin{array}{cc}1,\hfill & {\parallel \mathbf{x}-\mathbf{c}\parallel }_{\mathbf{A}}\le r\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(28)

where $\mathbf{A}$ is a real, symmetric, positive-definite matrix, and ${\parallel ·\parallel }_{\mathbf{A}}$ represents the Euclidean norm influenced by $\mathbf{A}$. If $\mathbf{A}$ is the identity matrix, the neuron acts as a binary RBF neuron.

A generalized binary RBF neuron, with greater computational power than LTGs, can compute any Boolean function of an LTG, allowing it to replace an LTG processing binary inputs without changing the network’s computational power [266].

8.2. Linear Separability and Nonlinear Separability

Definition 1

(Linear separability). Consider a set $\mathcal{X}$ consisting of N patterns ${\mathbf{x}}_i$ in a $J_1$-dimensional space, with each pattern assigned to either class ${\mathcal{C}}_1$ or ${\mathcal{C}}_2$. The problem is termed linearly separable if there is a hyperplane capable of distinguishing all points of class ${\mathcal{C}}_1$ from those of class ${\mathcal{C}}_2$.

An LTG can represent a linearly separable dichotomy using a linear separating hyperplane, defined by

$${\mathbf{w}}^T\mathbf{x}+w_0=0,$$

(29)

where $\mathbf{w}\in {\mathbb{R}}^{J_1}$ and $w_0$ represents the bias term. For any given pattern, if ${\mathbf{w}}^T\mathbf{x}+w_0>0$, the point is assigned to ${\mathcal{C}}_1$; if ${\mathbf{w}}^T\mathbf{x}+w_0<0$, it is assigned to ${\mathcal{C}}_2$.

Definition 2

($\phi$-separation). A dichotomy $\left\{{\mathcal{C}}_1,{\mathcal{C}}_2\right\}$ of a set $\mathcal{X}$ is called$\phi$-separable if there exists a transformation $\phi :{\mathbb{R}}^{J_1}\to {\mathbb{R}}^{J_2}$ that maps the data into a space where a separating hyperplane exists [262], such that

$${\mathbf{w}}^T\phi \left(\mathbf{x}\right)=0$$

(30)

with ${\mathbf{w}}^T\phi \left(\mathbf{x}\right)>0$ for points $\mathbf{x}\in {\mathcal{C}}_1$ and ${\mathbf{w}}^T\phi \left(\mathbf{x}\right)<0$ for points $\mathbf{x}\in {\mathcal{C}}_2$. In this case, $\mathbf{w}$ is a vector in ${\mathbb{R}}^{J_2}$.

A dichotomy that is not linearly separable can become separable when mapped to a higher-dimensional space. This is illustrated in Figure 3, where two initially inseparable dichotomies become $\phi$-separable after transformation. The figure also illustrates linearly separable and nonlinearly separable classes in a two-dimensional space.

For nonlinearly separable problems, the linear term in an LTG can be substituted by higher-order polynomial terms, creating a polynomial threshold gate. The function counting theorem still holds for polynomial threshold gates, provided the m points are, in general, position in the transformed $\phi$-space.

Higher-order neurons, or $\Sigma$-$\Pi$ units, improve linear neuron models by adding nonlinearity through monomials—products of input variables—in the hidden layer, enhancing the model’s ability to capture complex relationships.

8.3. Universal Function Approximation

Kolmogorov’s theorem, or the Kolmogorov–Arnold superposition theorem, resolving Hilbert’s 13th problem, states that any continuous function defined on a compact set can be represented through compositions and superpositions of univariate functions [267]. The Kolmogorov–Arnold representation decomposes a multivariate function into interior and outer functions, akin to a two-hidden-layer network [268,269]. Kolmogorov’s theorem guarantees that a feedforward network with two hidden layers and a sufficient number of hidden units is capable of approximating any continuous function to any desired degree of accuracy [270].

Theorem 2

(Kolmogorov). Any continuous real-valued function $f(x_1,\dots ,x_n)$, where $n\ge 2$ and the domain is ${[0,1]}^n$, can be represented by

$$f(x_1,\dots ,x_n)=\sum _{j=1}^{2n+1}{h}_j\left(\sum _{i=1}^n{\psi }_{ij}\left(x_i\right)\right),$$

(31)

where $h_j$ and ${\psi }_{ij}$ are continuous, and ${\psi }_{ij}$ are monotonically increasing functions, independent of f.

Hecht-Nielsen extended this result, showing neural networks can approximate complex functions with simpler, single-variable components [270].

Theorem 3

(Hecht-Nielsen). Any continuous function $f:{[0,1]}^n\to {\mathbb{R}}^m$ can be approximated with arbitrary accuracy by a feedforward neural network consisting of n input nodes, $2n+1$ hidden neurons, and m output nodes.

The Kolmogorov two-hidden-layer network model can exactly represent continuous, discontinuous bounded, and all unbounded multivariate functions, depending on whether the activation function in the second hidden layer is continuous, discontinuous bounded, or unbounded [268]. Recent adaptations [269,271] applied this to deep ReLU networks, enabling them to approximate the outer function while most layers focus on the interior function.

The Weierstrass theorem states that any continuous real-valued function of multiple variables can be approximated with arbitrary precision by a polynomial. The Stone–Weierstrass theorem [272], extending Weierstrass’ result, underpins the approximation capabilities of dynamic system models.

Establishing appropriate activation functions and universal approximation properties has been foundational for neural networks. The universal approximation theorem shows that a neural network with a single hidden layer is capable of approximating any continuous function defined on compact sets, making it suitable for tasks like regression and classification.

The approximation capabilities of fully connected feedforward neural networks, or MLPs, were rigorously analyzed before 2000. Key results include proofs for networks with continuous and discriminatory activation functions [273], nonconstant, bounded, monotone continuous functions [274], bounded generalized sigmoid functions with relaxed continuity [275], bounded and nonconstant functions [276], and locally bounded, piecewise continuous, nonpolynomial functions [277].

RNNs can be broadly categorized into globally recurrent networks and locally recurrent networks with global feedforward structures, both of which can serve as universal approximators for dynamical systems [278,279]. Echo state networks, as a special form of RNN, are universal approximators under $L^p$ criteria [280].

Extensions of the universal approximation theorem to single hidden-layer MLPs incorporate hypercomplex-valued networks, where inputs, outputs, and parameters are hypercomplex numbers. Examples include networks based on complex numbers, quaternions [281], and Clifford algebra, such as hyperbolic-valued [282,283], tessarine-valued [284], complex-valued [285], and vector-valued networks [286,287].

Vector-valued networks treat vector arrays as unified entities, with hypercomplex-valued networks forming a subclass defined over algebras with a multiplicative identity, including complex, quaternion, hyperbolic, tessarine, and Clifford-valued networks.

Extensions also demonstrate that neural networks with dropout retain universal approximation properties [288]. Additionally, other feedforward networks, such as functional link neural networks [289,290] and broad learning systems [291,292], also possess universal approximation capabilities.

Geometric deep learning models can approximate continuous target functions on compact sets, with curvature-dependent bounds on diameter and model depth [293]. However, locally defined models may fail to approximate continuous functions between non-degenerate compact manifolds [293]. Universal approximation has also been established for hyperbolic feedforward networks [294] and deep Kalman filter architectures [295].

Deep sets [296] act as universal approximators for continuous set functions, with their expressive capacity governed by latent space dimensionality [296,297]. These dimensionality conditions, proven optimal in [297], emphasize the critical role of latent space size. Deep sets implement the Janossy pooling paradigm, foundational to modern set-learning methods [297]. Despite advancements, open questions remain about Janossy pooling’s broader applicability, even as curriculum learning [79] and exploration bonuses [83] address specific training challenges.

Self-attention provides an alternative, aggregating information via input-dependent weights to support relational reasoning by processing element pairs. Originally developed for natural language processing [298], self-attention has been adapted to sets [299] and graph learning [300], serving as a k-ary Janossy pooling instance alongside deep sets [301].

Capacity of a Neural Network Architecture

The capacity of a neural network, defined as the binary logarithm of the total functions it can realize by adjusting synaptic weights, establishes an upper bound on bits (information) that can be derived from training data.

For fully connected networks with L layers of sizes $n_1,n_2,\dots ,n_L$, capacity is approximated in [302] by

$$C(n_1,\dots ,n_L)=\sum _{k=1}^{L-1}min(n_1,\dots ,n_k)n_k{n}_{k+1},$$

(32)

with smaller layers acting as bottlenecks. Techniques like multiplexing, enrichment, and stacking, along with bounds on finite set capacities, identify optimal architectures under constraints, introducing structural regularization.

While shallow networks compute more functions, deep networks yield more structured functions [302]. It is summarized in [234] that both deep and shallow models can achieve the same performance, which is validated by the no-free-lunch theorem [8].

For ReLU networks, a single unit’s capacity scales as $n^2(1+o\left(1\right))$, where n is the input dimension [303], and multilayer networks follow a similar trend. The VC-dimension of ReLU networks grows super-linearly with depth L [304].

8.4. Turing Machines

Turing-completeness, essential for neural networks learning algorithms from examples, means a network can simulate any Turing machine with unbounded memory. Neural networks are generally Turing-complete, with RNNs proven to be so even with bounded resources [257], leveraging internal representations. Extensions like neural Turing machines [305] maintain this completeness.

Self-attention architectures, such as the Transformer [298], dominate sequence-processing tasks like GPT-3 [306]. The capabilities of the Transformer are underpinned by its attention mechanism. Turing-completeness of the Transformer with hard attention has been demonstrated due to its ability to compute and access internal dense representations, with minimal elements for this completeness identified [307]. Unlike the fixed precision assumption in [308], fixed-precision Transformers are not Turing-complete, with differences between hard and soft attention highlighted [307].

The computational power of neural networks dates back to [1] for comparing neurons to Boolean formulae, and [309] for linking neural networks to finite automata. Turing-completeness of finite neural networks with linear connections was first demonstrated by [257,257]. However, practical learning requires more than Turing-completeness, prompting interest in enhancing RNNs with external memory.

Recent trends favor non-recurrent architectures like the Transformer, which excels in language tasks [298]. Enhancements for better generalization across input lengths have been proposed [308], and studies have explored the Transformer encoder as a language recognizer with limitations identified [310] and a universal approximator of continuous functions over strings [311].

Turing Machine Computations

Definition 3

(Turing machine). A (deterministic) Turing machine is formally defined as a tuple $\mathcal{M}=(\mathcal{Q},\Sigma ,\delta ,q_{\mathrm{init}},\mathcal{F})$, where $\mathcal{Q}$ represents a finite state set, Σ denotes a finite input alphabet, $\delta :\mathcal{Q}\times \Sigma \to \mathcal{Q}\times \Sigma \times \{1,-1\}$ specifies the state transition function, $q_{\mathrm{init}}\in \mathcal{Q}$ is the starting state, and $\mathcal{F}\subseteq \mathcal{Q}$ represents the set of halting states.

The machine operates on a single tape with infinitely many cells extending to the right. A special blank symbol $\#\in \Sigma$ marks empty positions on the tape. The machine uses a single head that moves left or right, reading and writing symbols from $\Sigma$.

For the Turing-completeness of sequence-to-sequence (seq-to-seq) neural network architectures, let $\mathcal{N}$ denote such a class, with ${\mathcal{L}}_{\mathcal{N}}$ as the set of languages recognized by its language recognizers. Turing-completeness for $\mathcal{N}$ is defined as follows.

Definition 4

(Turing-complete). A sequence-to-sequence neural network class $\mathcal{N}$ is considered Turing-complete if the language set ${\mathcal{L}}_{\mathcal{N}}$ includes all languages that are decidable, meaning those that can be recognized by a Turing machine.

One can formalize the Transformer architecture, abstracting away specific parameter or function choices.

Theorem 4

(Turing-completeness of Transformer networks [307]). The Transformer network architecture, when equipped with positional encodings, is capable of universal computation and thus Turing-complete, even under restricted conditions where positional embeddings pos(n) for $n\in \mathbb{N}$ are limited to $n,1/n,$ and $1/n^2$, with a single encoder layer and three decoder layers.

Transformers can recognize all recursively enumerable (semi-decidable) languages, extending beyond decidable ones. The undecidability of some problems in probabilistic language modeling with RNNs [312] suggests heuristic solutions, relying on the Turing-completeness of RNNs [257].

8.5. Winner-Takes-All

The winner-takes-all (WTA) competition, seen in artificial and biological systems, is a highly efficient computational module, outperforming threshold and sigmoidal gates [313]. A quadratic lower bound for computing WTA in feedforward circuits with threshold gates is established in [313], and circuits with a single soft WTA gate can approximate arbitrary continuous functions [313].

Theorem 5

(Maass, 1 [313]). For a winner-takes-all (WTA) with $n\ge 3$ inputs, any feedforward circuit C composed of threshold gates with arbitrary weights requires at least $\left({\displaystyle \genfrac{}{}{0pt}{}{n}{2}}\right)+n$ threshold gates.

Theorem 6

(Maass, 2 [313]). A two-layer feedforward circuit C with m binary or analog inputs and a binary output, constructed from threshold gates, can be emulated by a circuit utilizing a single k-winner-takes-all (k-WTA) gate, which operates on n weighted sums of the input variables with positive weights, except for a measure-zero set $S\subseteq {\mathbb{R}}^m$.

Any Boolean function $f:{\{0,1\}}^m\to \{0,1\}$ can similarly be realized by a single k-WTA gate that processes weighted sums of the input bits. When C is a polynomial-size circuit with integer weights constrained by a polynomial in m, n can also be polynomially bounded in m, ensuring the weights are natural numbers and the circuit size remains polynomial in m.

For inputs $(x_1,\dots ,x_n)$ with real values, a soft-WTA gate produces outputs $(r_1,\dots ,r_n)$, where $r_i$ reflects the relative position of $x_i$ in the input order. These soft-WTA functions mimic cortical circuits with lateral inhibition and, and when used individually as gates, they can approximate any continuous function with arbitrary precision.

Theorem 7

(Maass, 3 [313]). Consider a continuous function $h:D\to [0,1]$ defined on a bounded, closed domain $D\subseteq {\mathbb{R}}^m$. For any given $ϵ>0$ and any function g meeting these conditions, there exist natural numbers k, n, biases ${\alpha }_0^j\in \mathbb{R}$, and coefficients ${\alpha }_i^j\ge 0$ for $i=1,2,\dots ,m$, $j=1,2,\dots ,n$, such that the soft-WTA gate soft-${\mathit{WTA}}_{n,k}^g$, when applied to the sums ${\sum }_{i=1}^m{\alpha }_i^j{z}_i+{\alpha }_0^j$ for $j=1,2,\dots ,n$, computes a function $f:D\to [0,1]$ that satisfies the condition $\left|f\right(\mathbf{z})-h(\mathbf{z}\left)\right|<ϵ$ for all $\mathbf{z}\in D$. Consequently, circuits containing a single soft-WTA gate acting on positively weighted sums of input variables can universally approximate any continuous function.

9. Introduction to Computational Learning Theory

Machine learning algorithms predict an unknown model using training data from various hypotheses, with learning theory providing probabilistic performance bounds due to finite data. Computational learning theory explores how to achieve optimal generalization in supervised learning, utilizing frameworks such as VC-theory [314] and probably approximately correct (PAC) learning [315], which are both distribution-free and nonparametric.

VC-theory [314], also known as statistical learning theory, estimates dependencies using the empirical risk minimization (ERM) principle. It connects uniform convergence in function classes to capacity, measured by the VC-dimension [314], which quantifies classification complexity. VC-theory underpins SVMs [316] and provides model optimism bounds.

PAC learning [315] identifies hypotheses approximating target concepts with high probability. Linked to ERM, it minimizes empirical error to improve approximations. PAC principles support methods like Boosting [317], and neural network generalization is analyzed via VC-dimension.

Kolmogorov complexity, though foundational, is impractical to compute. The MDL serves as an approximation. Vapnik and Chervonenkis introduced measures like VC-entropy, the growth function, and VC-dimension [9]. Additionally, the fat-shattering dimension [318], which generalizes VC-dimension to real-valued settings, relates to Rademacher complexity [319].

Minimax label complexity assesses the worst-case label requests needed for active learning to achieve a target error rate under noise models. Distribution-free bounds on minimax label complexity for general hypothesis classes are detailed in [320].

Classical statistics assumes a true model exists to explain the observed data, while VC-learning theory, grounded in risk minimization, does not require such an assumption. These concepts converge in practical machine learning. For example, least squares (LS) minimization, often used for function estimation, can be derived from maximum likelihood under Gaussian noise assumptions and framed within a risk-minimization framework. SVMs, originally developed within VC-theory, were later adapted for function approximation and regularization [235]. VC-theory asserts that generalization from finite samples is possible even when exact function approximation is not achievable [321], unlike regularization approaches that may not guarantee good generalization in finite sample settings.

Function approximation is concerned with estimating an unknown target function in regression tasks or predicting the posterior probability $P\left(y\right|\mathbf{x})$ in classification problems. In contrast, VC-theory focuses on selecting the function that minimizes prediction risk and ensures generalization, accounting for the (unknown) input distribution. The margin concept, introduced in VC-theory and later adopted as a form of regularization in SVM, does not naturally extend to the broader regularization framework. Ultimately, SRM and SVM can be viewed as specific instances within the larger regularization paradigm.

No-Free-Lunch Theorem

Prior to the no-free-lunch theorem by Wolpert [8,322], it was believed that some algorithms could consistently outperform others across all search tasks. Researchers sought to identify these universally efficient algorithms. The theorem refutes this, asserting that no algorithm is universally superior. Averaged over all discrete functions, all search algorithms perform equivalently, akin to random enumeration.

Theorem 8

(No-free-lunch theorem). Let $\mathcal{F}$ denote the set of all functions, with ${\mathcal{F}}_1$ being a particular subset. If algorithm $A_1$ outperforms algorithm $A_2$ on average over ${\mathcal{F}}_1$, then $A_2$ must outperform $A_1$ on the complement of ${\mathcal{F}}_1$, i.e., $\mathcal{F}-{\mathcal{F}}_1$.

Algorithm performance depends on prior knowledge about the cost function, making evaluation meaningless without specifying assumptions. Real-world problems often involve priors like smoothness, symmetry, or independent and identically distributed samples, typically encoded by restricting the hypothesis class. While neural networks are effective for classification, they may not always be the most efficient solution, with other methods sometimes outperforming them.

The no-free-lunch theorem extends to areas like coding techniques, early stopping strategies [323], overfitting prevention, and noise prediction [324], emphasizing that no method suits all situations. In line with this, the limitations of leave-one-out cross-validation were demonstrated in [325], while for simple problems, strict leave-one-out cross-validation can still yield reliable results [326]. Nevertheless, statistical tests generally outperform cross-validation for model selection, both in linear and nonlinear settings [327].

10. Probably Approximately Correct (PAC) Learning

The PAC learning framework focuses on learning a target function, or concept, by selecting a suitable function from a hypothesis space, which consists of mappings from inputs to $\{0,1\}$. The objective is to identify a function that approximates the target concept. In supervised learning, PAC learning aims to produce a classifier that, with high probability ($1-\delta$), achieves an error rate not exceeding $ϵ$, making it probably approximately correct. That is, PAC theory provides bounds on the generalization error with a probability larger than a certain threshold. A key challenge in PAC learning is understanding sample complexity.

Definition 5

(PAC learnability). Let ${\mathcal{C}}_n$ and ${\mathcal{H}}_n$, for $n\ge 1$, represent sets of target concepts and hypotheses over ${\{0,1\}}^n$, where ${\mathcal{C}}_n\subseteq {\mathcal{H}}_n$. If a polynomial-time algorithm can achieve low error with high confidence for all concepts in $\mathcal{C}=\left\{{\mathcal{C}}_n\right\}$ using $\mathcal{H}=\left\{{\mathcal{H}}_n\right\}$ with sufficient data, $\mathcal{C}$ isPAC learnable by $\mathcal{H}$.

If the hypothesis space $\mathcal{H}$ contains $\left|\mathcal{H}\right|$ classifiers, and a classifier labels a random example correctly with probability less than $1-ϵ$, then the probability of correctly labeling m examples is bounded by $P\le {(1-ϵ)}^m$, which decays rapidly for typical values of $ϵ$ and m.

If an error rate above $ϵ$ is unacceptable, evaluating m training examples allows the selection of $k\le \left|\mathcal{H}\right|$ classifiers with no mistakes. The likelihood that the chosen k classifiers accurately classify the m examples satisfies the following inequality:

$$P\le k{(1-ϵ)}^m\le {\left|\mathcal{H}\right|(1-ϵ)}^m<\left|\mathcal{H}\right|e^{-mϵ},$$

(33)

where $1-ϵ<e^{-ϵ}$.

To ensure the probability is below the failure threshold $\delta$, the following inequality must hold:

$$\left|\mathcal{H}\right|e^{-mϵ}\le \delta .$$

(34)

This leads to the bound derived in [328],

$$m>{\displaystyle \frac{1}{ϵ}}\left(ln\left|\mathcal{H}\right|+ln{\displaystyle \frac{1}{\delta }}\right).$$

(35)

Equation (35) illustrates that m grows linearly with $1/ϵ$ and is relatively unaffected by $\delta$. This result represents a worst-case scenario, assuming $k=\left|\mathcal{H}\right|$, which could be a conservative estimate.

A class is deemed not PAC learnable if satisfying the $(ϵ,\delta )$-criteria requires an impractically large number of training examples.

Example 1.

General Boolean functions are not PAC learnable. When dealing with n Boolean variables, the instance space consists of $2^n$ possible examples. Consequently, the number of possible subsets is $2^{2^n}$, meaning the size of the hypothesis space is $\left|\mathcal{H}\right|=2^{2^n}$. Using Equation (35), the following inequality for the required number of examples m is derived:

$$m>{\displaystyle \frac{1}{ϵ}}\left(2^{n}ln2+ln{\displaystyle \frac{1}{\delta }}\right).$$

This lower bound increases exponentially with n, demonstrating that a general Boolean function is not PAC learnable.

PAC-Bayesian theory [329,330] analyzes the generalization of models like linear classifiers, SVMs, and neural networks by modeling predictions as samples from a posterior distribution with bounds on average risk. While effective for randomized classifiers, applying these bounds to deterministic models requires derandomization, addressed in [331] through disintegrating PAC-Bayesian bounds [332,333]. These bounds offer guarantees for individual hypotheses, simplify optimization, and provide insights for algorithm design.

Sample Complexity

Definition 6

(Sample complexity of a learning algorithm). The sample complexity, denoted $m_{\mathcal{H}}$, refers to the minimum number of samples required for a learning algorithm to learn a concept class $\mathcal{C}$ using a hypothesis class $\mathcal{H}$, ensuring that the approximation error is within ϵ with probability at least $1-\delta$.

For any consistent algorithm learning $\mathcal{C}$ with hypothesis class $\mathcal{H}$, the sample complexity is bounded by [328,334]:

$$m_{\mathcal{H}}(ϵ,\delta )\le {\displaystyle \frac{1}{ϵ\left(1-\sqrt{ϵ}\right)}}\left(2dln{\displaystyle \frac{6}{ϵ}}+ln{\displaystyle \frac{2}{\delta }}\right),\forall 0<\delta <1,$$

(36)

where d is the VC-dimension of $\mathcal{H}$. This bound ensures that with a confidence level of at least $1-\delta$, the algorithm produces a hypothesis whose error does not exceed $ϵ$, and the sample size scales linearly with the VC-dimension.

The sample complexity is upper-bounded in terms of the hypothesis space size $\left|\mathcal{H}\right|$ by [328],

$$m_{\mathcal{H}}(ϵ,\delta )\le {\displaystyle \frac{1}{ϵ}}\left(ln\left|\mathcal{H}\right|+ln{\displaystyle \frac{1}{\delta }}\right).$$

(37)

For Boolean hypothesis spaces, the second PAC bound is typically more advantageous, whereas for infinite hypothesis spaces over real-valued attributes, only the first bound can be applied. The PAC framework provides upper bounds on the number of samples required for learning, with linear threshold concepts like perceptrons being PAC learnable over both Boolean and real-valued input domains [328].

The PAC framework and VC-dimension can estimate sample sizes for neural networks, including sigmoidal neurons [335] and linear threshold gates [336]. Sample sizes scale with $ϵ$ and $\delta$, typically as $N=O\left({\displaystyle \frac{N_w}{ϵ}}\right)$ [337].

Sample complexity estimates in [338] help control empirical deviations from expected cost functions and apply to matrix factorization techniques [54], such as PCA, NMF, and clustering methods [6], with bounds proportional to ${\left({\displaystyle \frac{log\left(N\right)}{N}}\right)}^{1/2}$. A majority-vote classifier trained on independent datasets mitigates the logarithmic growth of sample complexity [339].

11. Vapnik–Chervonenkis Dimension

The VC-dimension, a combinatorial measure of a neural network’s expressive capacity, extends Cover’s concept of capacity [262]. It is key in VC-theory for estimating the training sample size needed for effective generalization.

Definition 7

(VC-dimension). A set $\mathcal{S}\subseteq \mathcal{X}$ is shattered by a neural network class $\mathcal{N}$ if, for every function $f:\mathcal{S}\to \{0,1\}$, there exists a corresponding function in $\mathcal{N}$. The VC-dimension of $\mathcal{N}$ is the size of the largest shattered set,

$${dim}_{\mathrm{VC}}\left(\mathcal{N}\right)=max\left\{\left|\mathcal{S}\right||\mathcal{S}\subseteq \mathcal{X}\mathit{is}\mathit{shattered}\mathit{by}\mathcal{N}\right\}.$$

The VC-dimension d of a function family $\left\{f\right(\alpha \left)\right\}$ measures the maximum number of points that the classifier family can shatter, though not all sets of size d may necessarily be shattered. For instance, a neural network with the function $f(\mathbf{x},\mathbf{w},\theta )=\mathrm{sgn}({\mathbf{w}}^T\mathbf{x}+\theta )$ is capable of shattering up to three points, giving it a VC-dimension of 3.

For linear classifiers, the VC-dimension equals the number of parameters in the model. The Boolean VC-dimension of a neural network, ${dim}_{\mathrm{BVC}}\left(\mathcal{N}\right)$, refers to the VC-dimension of the Boolean functions it can compute.

To demonstrate that ${dim}_{\mathrm{VC}}\left(\mathcal{H}\right)=d$, one must show that a set of size d can be shattered, while no set of size $d+1$ can. Neural networks with LTGs have a VC-dimension of $O(N_{w}log{N}_w)$ [340], when having $N_w$ weights. Networks with real-valued outputs have extended VC-dimensions [340].

For feedforward networks, those with threshold activations have a VC-dimension of $O(N_{w}ln{N}_w)$ [336], and those with logistic activations have a VC-dimension of $O\left(N_w^2\right)$ [341]. When dealing with higher-order neurons involving k monomials over n variables, the VC-dimension is at least $nk+1$ [342].

Consider a $J_1$-$J_2$-1 feedforward network with LTG output, where the hidden neurons are LTGs, binary RBF, and generalized binary RBF neurons in ${\mathcal{N}}_1$, ${\mathcal{N}}_2$, and ${\mathcal{N}}_3$, respectively. The VC-dimensions of these networks satisfy [266]

$${dim}_{\mathrm{BVC}}\left({\mathcal{N}}_1\right)={dim}_{\mathrm{BVC}}\left({\mathcal{N}}_2\right)\le {dim}_{\mathrm{BVC}}\left({\mathcal{N}}_3\right).$$

(38)

For $J_1\ge 3$ and $J_2\le {\displaystyle \frac{2^{J_1+1}}{J_1^2+J_1+2}}$, the lower bound for these networks is [266,343]

$${dim}_{\mathrm{BVC}}\left({\mathcal{N}}_1\right)=J_1{J}_2+1.$$

(39)

The VC-dimension estimates of a classifier set, though analytically impractical, can be experimentally derived by fitting formulas to error frequencies on artificial datasets of varying sizes [344]. Sampling variability was mitigated using a nonuniform design for better model complexity control and improved VC-generalization bounds [345]. An objective function is introduced for VC-dimension estimation [344], later optimized for regression model selection, showing consistency and competitive performance against established methods like AIC, BIC, and cross-validation [346].

Fully connected networks have VC-dimensions proportional to $WLlog(W/L)$, where W and L are the number of weights and depth, respectively [304,347]. Equivariant networks, such as CNNs, show lower VC-dimensions due to reduced dimensionality in equivariant matrices [348]. VC-dimension also depends on training methods like SGD [232,233].

CNNs exhibit translation equivariance [24], while group CNNs extend this to rotations, scales, and matrix symmetries [349]. Invariant kernels reduce sample complexity by the group size [350], with larger group volumes improving generalization [351]. The VC-dimension on group orbits relates to optimal sample complexity [352]. For simple architectures, precise VC-dimension estimates exist [353], though infinite groups with specific kernels can lead to infinite VC-dimension, even with group invariance.

Machine Teaching and Teaching Dimension

Machine teaching complements machine learning by selecting training data to optimize the learning process. Unlike machine learning, where a target concept ${\mathcal{c}}^{*}$ is inferred from random examples, machine teaching involves a teacher designing a labeled dataset $\mathcal{T}\left({\mathcal{c}}^{*}\right)$ to ensure the learner reconstructs ${\mathcal{c}}^{*}$. Applications include explainable AI, trustworthy AI, and pedagogy.

Teaching dimension is a theoretical concept that determines the smallest training set size necessary for a learner to effectively approximate a target model. This concept has been explored in works such as [354,355]. In this setting, the teacher possesses knowledge of both the target model and the learners algorithm. The teacher’s task is to select a training set that helps the learner approximate the target model accurately. Importantly, the training examples do not need to be independent and identically distributed, and the teacher has the freedom to select any points from the input space.

For a finite hypothesis space $\mathcal{H}$, the relationship between the VC-dimension and teaching dimension is given by [354] as follows:

$${dim}_{\mathrm{VC}}\left(\mathcal{H}\right)=log\left(\right|\mathcal{H}\left|\right)\le {dim}_{\mathrm{TD}}\left(\mathcal{H}\right)\le {dim}_{\mathrm{VC}}\left(\mathcal{H}\right)+\left|\mathcal{H}\right|-2^{{dim}_{\mathrm{VC}}\left(\mathcal{H}\right)}.$$

(40)

Initially introduced for version-space learners, the concept of teaching dimension has since been expanded to machine learning models that select hypotheses via optimization. Studies such as [356] examine its application to models like ridge regression, SVMs, and logistic regression, identifying optimal training sets for these cases.

A learner L infers a hidden concept using priors $P\left(\mathcal{c}\right)$ and $\mathcal{c}$-conditional likelihoods $P\left(\mathcal{z}\right|\mathcal{c})$, where $\mathcal{c}\in \mathcal{C}$ represents concepts and $\mathcal{z}\in \mathcal{Z}$ represents observations. L is a maximum a posteriori (MAP)-learner if it maximizes the posterior probability, or a maximum likelihood estimation (MLE)-learner if it maximizes the $\mathcal{c}$-conditional likelihood [357]. Sampling modes depend on whether $\mathcal{S}$ is ordered or unordered, and with or without replacement.

For a target concept ${\mathcal{c}}^{*}\in \mathcal{C}$, a teacher aims to find the smallest set of observations ensuring L selects ${\mathcal{c}}^{*}$, defining the MAP- and MLE-teaching dimensions [357]. These dimensions exhibit monotonicity, relate across sampling modes, and are characterized graph-theoretically for 0,1-labeled examples. The MLE-teaching dimension is at most one greater than the MAP-teaching dimension, bounded by combinatorial measures like the antichain number and VC-dimension, and computable in polynomial time.

12. Rademacher Complexity

In contrast to the VC-dimension, a global measure of complexity, Rademacher complexity [358], ref. [359] and other data-dependent metrics [255] offer more refined estimates by considering the data distribution, leading to tighter bounds and more accurate generalization estimates. Rademacher complexity, which measures a hypothesis space’s ability to fit random labels for a training sample of size N, provides “average-case” bounds and is generally more effective than VC-dimension. Its precision, however, depends on the distribution approximation and empirical evaluation cost. In general, Rademacher complexity offers more precise generalization error bounds than VC-dimension and cover numbers.

Definition 8

(Rademacher complexity [358]). Consider a family of functions $\mathcal{F}$ defined on a probability space $(\mathcal{Z},\mathcal{P})$, with N independent training samples ${\mathbf{z}}_1,\dots ,{\mathbf{z}}_N$ drawn from $\mathcal{P}$. Let ${\sigma }_1,\dots ,{\sigma }_N$ represent independent Rademacher variables, each taking values $\pm 1$ with equal likelihood. The Rademacher complexity of $\mathcal{F}$ is then defined as

$$R_N\mathcal{F}=\underset{f\in \mathcal{F}}{sup}{\displaystyle \frac{1}{N}}\sum _{i=1}^N{\sigma }_{i}f\left({\mathbf{z}}_i\right).$$

(41)

The expected Rademacher complexity is $E\left[R_N\mathcal{F}\right]$, and the empirical Rademacher complexity is

$$E_{\sigma }\left[R_N\mathcal{F}\right]=E\left[R_N\mathcal{F}|{\mathbf{z}}_1,\dots ,{\mathbf{z}}_N\right].$$

(42)

The above formulation defines the global Rademacher complexity, which estimates the complexity of a function class and is used to derive generalization error bounds in supervised learning.

Theorem 9

(Generalization error bound from Rademacher complexity [359]). For $\delta >0$, consider a hypothesis $\widehat{f}$ learned from N examples. With probability at least $1-\delta$,

$$E\left[\widehat{f}\right]\le \underset{f\in \mathcal{F}}{inf}E\left[f\right]+4R_N\mathcal{F}+\sqrt{{\displaystyle \frac{2log(2/\delta )}{N}}}.$$

(43)

The bound in Theorem 9 converges at a rate of $O\left(\sqrt{1/N}\right)$.

Local Rademacher complexity focuses on the subset of the hypothesis class likely selected by the learning algorithm, defined by the intersection of the hypothesis class with a ball around the target function,

$$E_{\sigma }\left[R_N\left\{f\in \mathcal{F}:E\left[f^2\right]\le r\right\}\right]\mathrm{or}E\left[R_N\left\{f\in \mathcal{F}:E\left[f^2\right]\le r\right\}\right].$$

(44)

The Rademacher complexity of a set of functions is upper-bounded by the growth function $G\left(N\right)$ and VC-dimension d [359,360],

$$R_N\mathcal{F}\le \sqrt{{\displaystyle \frac{2lnG\left(N\right)}{N}}},R_N\mathcal{F}\le c\sqrt{{\displaystyle \frac{d}{N}}},$$

(45)

where c is a constant. Thus, Rademacher complexity is of order $O\left(\sqrt{d/N}\right)$.

Local Rademacher complexity, dependent on r, excludes high-variance functions, is always less than or equal to the global complexity, and offers tighter error bounds when the variance-expectation relationship is satisfied [361].

Theorem 10

(Generalization error bound using local Rademacher complexity [361]). For a function $\widehat{f}$ learned from N samples, where $E\left[f^2\right]\le r$ for all $f\in \mathcal{F}$, and for $\delta >0$, the following holds with probability at least $1-\delta$,

$$E\left[\widehat{f}\right]\le \underset{f\in \mathcal{F}}{inf}E\left[f\right]+8L{R}_N(\mathcal{F},r)+\sqrt{{\displaystyle \frac{8rlog(2/\delta )}{N}}}+{\displaystyle \frac{3log(2/\delta )}{N}}.$$

(46)

By selecting a smaller subset ${\mathcal{F}}^{\prime }\subseteq \mathcal{F}$ that minimizes variance while ensuring $\widehat{f}\in {\mathcal{F}}^{\prime }$, the error bound in Theorem 10 achieves a faster convergence rate of $O\left({\displaystyle \frac{logN}{N}}\right)$ compared to Theorem 9. After determining the local Rademacher complexity, the error difference $E\left(\widehat{f}\right)-{inf}_{f\in \mathcal{F}}E\left[f\right]$ can be bounded using the fixed point of the local Rademacher complexity of $\mathcal{F}$.

Direct computation of Rademacher complexity is challenging. Dudley’s entropy integral [362] relates covering numbers to Rademacher complexities, a result extended to local Rademacher complexity [363].

In ERM-based multi-label learning, the Rademacher complexity is constrained by the trace norm of the multi-label predictors, providing a rationale for applying trace norm regularization in multi-label problems [364]. The generalization of RBF networks has been analyzed using local Rademacher complexities with $L_1$-metric capacity, yielding improved error bounds [365].

Annealed VC-entropy [366] provides a bound for generalization error in terms of empirical error. The connection between VC-entropy and Rademacher complexity [360] allows the use of Rademacher complexity in Vapnik’s general bound, yielding faster convergence rates from $O\left(N^{-1/2}\right)$ to $O\left(N^{-1}\right)$, surpassing the $O\left(N^{-1/2}\right)$ rate with Rademacher complexity alone.

Local VC-entropy [367], a refined adaptation of VC-complexity, provides a generalization bound for binary classifiers while reducing computational demands compared to local Rademacher complexity. It quantifies the number of functions in $\mathcal{F}$, defined by ${\mathcal{D}}_N$, capable of classifying at least one configuration of $\sigma ={({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_N)}^T\in \mathcal{S}$ within a Hamming distance of less than $Nr$. By eliminating improbable functions, this approach enhances Vapnik’s results. The connection between local VC-entropy and Rademacher complexity is established through the determination of their admissible ranges [367], with further insights gained via extensions to the geometric framework [360].

13. Empirical Risk Minimization Principle

Consider N independent and identically distributed samples $({\mathbf{x}}_i,y_i)$ drawn from an unknown distribution $p(\mathbf{x},y)$. A machine defines mappings $\mathbf{x}\mapsto f(\mathbf{x},\mathit{\alpha })$, where $\mathit{\alpha }$ are adjustable parameters. Upon selecting $\mathit{\alpha }$, the machine becomes a trained machine.

The expected risk, representing the trained machine’s generalization error, is defined as

$$R\left(\mathit{\alpha }\right)=\int L\left(y,f(\mathbf{x},\mathit{\alpha })\right)\mathrm{d}p(\mathbf{x},y),$$

(47)

Here, $L\left(y,f(\mathbf{x},\mathit{\alpha })\right)$ quantifies the discrepancy between the true output y and the model prediction $f(\mathbf{x},\mathit{\alpha })$, with its form tailored to the specific task, such as

$$L(y,f(\mathbf{x},\mathit{\alpha }))=\left\{\begin{array}{cc}0,\hfill & y=f(\mathbf{x},\mathit{\alpha })\hfill \\ 1,\hfill & y\ne f(\mathbf{x},\mathit{\alpha })\hfill \end{array}\right.\left(\mathrm{for}\mathrm{classification}\right),$$

(48)

$$L(y,f(\mathbf{x},\mathit{\alpha }))={(y-f(\mathbf{x},\mathit{\alpha }))}^2\left(\mathrm{for}\mathrm{regression}\right),$$

(49)

$$L\left(p\right(\mathbf{x},\mathit{\alpha }\left)\right)=-lnp(\mathbf{x},\mathit{\alpha })\left(\mathrm{for}\mathrm{density}\mathrm{estimation}\right).$$

(50)

The empirical risk $R_{\mathrm{emp}}\left(\mathit{\alpha }\right)$ represents the average loss on the training set and is defined as

$$R_{\mathrm{emp}}\left(\mathit{\alpha }\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}L\left(y_i,f\left({\mathbf{x}}_i,\mathit{\alpha }\right)\right).$$

(51)

The ERM principle minimizes the empirical risk (51) to approximate the true loss (47) with respect to model parameters. Common in statistics and machine learning, ERM supports models such as SVMs, linear regression, and logistic regression.

For a 0-1 loss function, the VC-bound [316] holds with confidence $1-\delta$,

$$R\left(\mathit{\alpha }\right)\le R_{\mathrm{emp}}\left(\mathit{\alpha }\right)+\sqrt{{\displaystyle \frac{d\left(ln\frac{2N}{d}+1\right)-ln\frac{\delta }{4}}{N}}},$$

(52)

where d is the model’s VC-dimension. The second term, the VC-confidence increases with d, and lowering d tightens the true error bound. VC-confidence depends on the function class, while empirical and true risks depend on the chosen function during training.

For regression problems, a useful form of the VC-bound is given by [9]

$$R\left(d\right)\le R_{\mathrm{emp}}\left(d\right){\left(1-\sqrt{q-qlnq+{\displaystyle \frac{lnN}{2N}}}\right)}^{-1},$$

(53)

where $q={\displaystyle \frac{d}{N}}$, with d as the VC-dimension and N as the training set size. Equation (53) is a specific instance of the general VC-bound [316], obtained by assigning suitable values to theoretical constants.

The SRM principle [316] extends ERM by incorporating VC-dimension to minimize the upper bound of true risk, improving generalization by selecting the function with the lowest worst-case risk. Empirical comparisons of AIC, BIC, and SRM in regression tasks [368] show SRM outperforms AIC and performs similarly to BIC in predictive accuracy. Excess risk bounds for loss functions that are unbounded, such as log loss and squared loss, are derived in [369], focusing on estimators like $\eta$-generalized Bayesian, MDL, and ERM, particularly for heavy-tailed distributions.

13.1. Generalization Error by VC-Theory

VC-theory enforces regularization via a hierarchy of admissible models, known as a structure, which ranks models based on their complexity, measured by the VC-dimension. For a training sample containing N samples, the uniform convergence bound for classification is given by [9,232,233]

$$R_{\mathrm{test}}\le R_{\mathrm{emp}}+{\displaystyle \frac{1}{2}}\sqrt{ϵ},$$

(54)

where

$$ϵ={\displaystyle \frac{a_1}{N}}\left(d\left[ln\left({\displaystyle \frac{a_{2}N}{d}}\right)+1\right]-ln{\displaystyle \frac{\delta }{4}}\right).$$

(55)

VC-theory also establishes a uniform relative convergence bound [9,232,233],

$$R_{\mathrm{test}}\le R_{\mathrm{emp}}+{\displaystyle \frac{ϵ}{2}}\left(1+\sqrt{1+{\displaystyle \frac{4R_{\mathrm{emp}}}{ϵ}}}\right).$$

(56)

VC-bounds (54) and (56) hold with probability $1-\delta$ for all models, including the one minimizing the training error ($R_{\mathrm{emp}}$), with $\delta$ typically set as $\delta =min\left({\displaystyle \frac{4}{\sqrt{N}}},1\right)$ [232,233]. Bound (56) includes the training error and a confidence interval that depends on $R_{\mathrm{emp}}$ and $ϵ$ (55), reflecting the model’s capacity.

The constants $a_1\in [0,4]$ and $a_2\in [0,2]$ in (55) are usually set to $a_1=4$ and $a_2=2$ in classical VC-theory [9,233], though these worst-case values may lead to looser bounds for test error estimation [232]. For double descent modeling, $a_1=3$ and $a_2=1$ are preferred [254].

The uniform relative convergence bound (56) is more accurate than bound (54), particularly when $R_{\mathrm{emp}}$ is small (or zero) in the second descent mode [254]. When $R_{\mathrm{emp}}$ is zero, bound (54) gives $O\left(\sqrt{d/N}\right)$, while (56) provides a sharper bound of $O(d/N)$.

VC-theory asserts that finite VC-dimension is both necessary and sufficient for generalization [9,233], and similar conditions apply to distribution-dependent complexity measures like VC-entropy, annealed VC-entropy, and Rademacher complexity [370].

13.2. Generalization Error by Rademacher Bound

Rademacher bounds on test error for classification, similar to the VC-bound (54), are widely discussed [359]. For binary functions $\mathcal{F}$ mapping $\mathbf{x}$ to $[-1,+1]$, with inputs $\mathbf{x}$ drawn from distribution ${\mathcal{P}}_x$, the following bound holds for n training samples, with probability $1-\delta$:

$$R_{\mathrm{test}}\le R_{\mathrm{emp}}+a{R}_N\mathcal{F}\left({\mathcal{P}}_x\right)+\sqrt{{\displaystyle \frac{ln(1/\delta )}{2N}}},$$

(57)

where $R_N\mathcal{F}\left({\mathcal{P}}_x\right)$ represents the Rademacher complexity, and a denotes a theoretical constant (typically $a=1$ or 2). Estimating Rademacher complexity for arbitrary function sets is computationally challenging, requiring either analytical estimation or a relation to VC-dimension.

Both the VC-bound (54) and Rademacher bound (57) provide the same excess error bound $R_{\mathrm{test}}-R_{\mathrm{emp}}<O\left(\sqrt{d/N}\right)$, confirming their conceptual equivalence [371]. However, VC-bound (56) offers a more accurate estimate. A bound on Rademacher complexity for linear functions with bounded weights is provided [359,372]. It also aligns with the radius-margin bound in VC-theory for large-margin classifiers [359].

13.3. Fundamental Theorem of Learning Theory

The foundation of learning theory establishes PAC learnability for binary classifiers based on their VC-dimension. A hypothesis class is considered PAC learnable if and only if its VC-dimension is finite, which also dictates the sample complexity required for PAC learning. Uniform convergence of empirical error to the true error across all inputs guarantees that any training algorithm with low training error is PAC learnable. In cases where learnability holds, uniform convergence is assured, and the ERM principle can be applied.

Theorem 11

(Fundamental theorem of learning theory [373]). Let $\mathcal{H}$ be a class of hypotheses, where each function maps from a domain $\mathcal{X}$ to $\{0,1\}$, and the loss function is the 0-1 loss. The following statements are equivalent:

• $\mathcal{H}$ exhibits uniform convergence.
• Any ERM rule can agnostically learn $\mathcal{H}$.
• $\mathcal{H}$ is agnostically PAC learnable.
• $\mathcal{H}$ has a finite VC-dimension.

Agnostic PAC learning broadens the concept of PAC learnability to situations where the realizability assumption might not be valid. If realizability is assumed, agnostic PAC learning leads to the same results as PAC learning. Without realizability, a learner cannot guarantee an arbitrarily low error, but it can still perform well if its error is close to the best achievable by any predictor from $\mathcal{H}$.

When the VC-dimension of $\mathcal{H}$ is infinite, PAC learnability is not guaranteed. In contrast, a finite VC-dimension ensures PAC learnability, establishing VC-dimension as a crucial element in determining PAC learnability. The concept of VC-dimension also allows for the analysis of learnability in continuous domains and determines the sample complexity, as shown in Equation (36). The sample complexity is proportional to the VC-dimension, with further insights on this topic available in [373].

The SRM framework aims to select a hypothesis that minimizes the upper bound on the true risk. When applied to countable hypothesis classes, SRM leads to the MDL approach. In the SRM framework, prior knowledge is incorporated by encoding preferences for hypotheses in $\mathcal{H}={\bigcup }_{n\in \mathbb{N}}{\mathcal{H}}_n$, where weights are assigned to each subclass ${\mathcal{H}}_n$.

Natarajan dimension generalizes the concept of VC-dimension to multiclass predictor classes, and the multiclass fundamental theorem is derived using Natarajan dimension [373].

14. Conclusions and Future Directions

The topics of machine learning is very wide, and in this paper, we have only introduced the fundamentals of machine learning that are related to neural networks. The contents are sufficient to support the understanding of the discipline of neural networks and deep learning. Interested readers can turn to some other review papers on related machine learning topics, such as kernel methods and SVMs [5], compressed sensing and matrix factorization [54], MLP and learning methods [374], and clustering [6], reinforcement learning, and ensemble learning, to name a few.

While this survey focuses on the foundational principles and theories of machine learning, several important topics were not fully covered due to space constraints. Challenges such as noisy data, overlapping data, highly dynamic data, and high-dimensional data, which directly impact learning and generalization, are critical in practical machine learning applications, but are primarily associated with data preprocessing and feature extraction, topics typically addressed within statistics. Additionally, emerging areas such as explainability, interpretability, uncertainty quantification, and robust learning are gaining significant importance in developing trustworthy machine learning systems. Although these topics are essential for the application of machine learning models, they were not the central focus of this paper. Similarly, while recent advancements such as transfer learning, deep reinforcement learning, and transformer-based architectures significantly influence learning, inference, and generalization, they were mentioned briefly within the paper, but not explored in-depth due to the broad scope and theoretical focus of this survey. We encourage readers to explore these critical areas in other specialized literature.

14.1. Future Directions

Deep learning, often synonymous with modern machine learning, has ushered in the era of AI and transformative advancements. Below, we outline several fundamental issues that merit future investigation, many of which are central to the ongoing development of deep learning.

14.1.1. Analyzing Transfer Learning

Transfer learning has emerged as a dynamic and increasingly influential field in modern AI research. A wide range of transfer learning methods have been proposed, such as reweighting instances, adaptating features, adaptating classifiers, fine-tuning deep networks, and using adversarial strategies, which go beyond the conventional semi-supervised and unsupervised frameworks [22]. It has become the cornerstone of modern deep learning, operating on the heuristic of analogical learning [234].

Despite its significance, the theoretical foundations of transfer learning remain underdeveloped. Most theoretical work assumes a similarity between the source and target domains, yet this similarity is difficult to quantify, and the resulting theoretical frameworks often lack rigorous validation. Given its central role in deep learning, it is crucial to establish a deeper understanding of the principles and mechanisms underlying transfer learning, moving beyond reliance on empirical observations.

Future research should focus on uncovering the fundamental principles of transfer learning, developing metrics to evaluate domain similarity, and creating robust theoretical models to validate its effectiveness and applicability.

14.1.2. Explaining Double Descent

Understanding how neural networks learn remains one of the core challenges in machine learning. Neural network weights evolve from random initialization to effectively perform specific tasks. Despite their empirical successes, a comprehensive theoretical framework for neural networks is still lacking. The double descent phenomenon, observed in overparameterized models within deep learning, contradicts traditional theory, which suggests that models with too many parameters would overfit and generalize poorly. A central question is why neural networks can generalize well to unseen samples, despite their large number of parameters, which should theoretically lead to overfitting [230,375].

Various explanations have been proposed, but they often yield conflicting results. One explanation is that most of the nodes in a neural network remain inactive, with only a subset of the parameters being active, thus preventing overfitting. This idea is analogous to the way the human brain operates [234]. Additionally, training techniques like dropout and architectures like ResNet, which resemble ensemble learning, have been shown to improve generalization [234]. However, while these heuristics offer insights, they remain inadequate without a rigorous theoretical foundation.

Addressing this gap in understanding would not only improve the interpretability of neural networks, but also provide a principled approach to designing architectures that can leverage these insights effectively.

14.1.3. Exploring SGD in Deep Learning Setting

SGD is fundamental to modern machine learning, forming the basis for many optimization methods. Its simplicity and low computational complexity make it ideal for large-scale tasks, ensuring its continued importance in the field. By iteratively updating parameters using small data batches, SGD achieves efficient convergence and scalability across diverse applications, from linear regression to deep learning. SGD is still being actively investigated. Despite its success, challenges remain, particularly in deep learning. For instance, the behavior of SGD with dropout—a regularization technique—is not fully understood, complicating theoretical analysis. Future research may address these gaps, enhancing optimization strategies and ensuring SGD’s relevance as machine learning evolves.

14.1.4. Understanding Data Augmentation

While data augmentation is widely recognized as a crucial technique for preventing overfitting, particularly in situations where datasets are small or labels are scarce, its implementation often lacks a consistent theoretical foundation. Practitioners apply data augmentation heuristically, relying on domain-specific insights or empirical validation to guide their choices. Despite its effectiveness, there is limited understanding of the underlying mechanisms that drive its success or the precise conditions under which it provides optimal benefits [376,377].

14.1.5. Delving into Transformers

Transformers, driven by the self-attention mechanism introduced by Vaswani et al. [298], have transformed modern AI, replacing recurrence in sequence processing to excel in tasks like language generation, image analysis, and speech recognition. Models like GPT-3 [306] leverage this innovation to achieve state-of-the-art performance in NLP by supporting long-range dependency modeling and efficient parallelization. Beyond NLP, Transformers have impacted vision (e.g., Vision Transformers for image classification) and bioinformatics, advancing protein analysis and genomics.

Key challenges include scaling for extreme data while addressing computational costs. Efficient variants, such as sparse Transformers, tackle resource efficiency, while methods like pruning and knowledge distillation adapt Transformers for low-resource and real-time applications. Questions remain on theoretical performance limits, minimal architectures for tasks, and generalization guarantees.

Integrating multimodal data (e.g., text, images, and audio) represents a promising but complex frontier. Ongoing research into scalability, efficiency, and multimodal learning ensures Transformers remain central to AI’s evolution across diverse domains.