Formula-Driven Supervised Learning in Computer Vision: A Literature Survey

Abstract

Current computer vision research uses huge datasets with millions of images to pre-train vision models. This results in escalation of time and capital, ethical issues, moral issues, privacy issues, copyright issues, fairness issues, and others. To address these issues, several alternative learning schemes have been developed. One such scheme is formula-based supervised learning (FDSL). It is a form of supervised learning, which involves the use of mathematically generated images for the pre-training of deep models. Promising results have been obtained for computer-vision-related applications. In this comprehensive survey paper, a gentle introduction to FDSL is presented. The supporting theory, databases, experimentation and ensuing results are discussed. The research outcomes, issues and scope are also discussed. Finally, some of the most promising future directions for FDSL research are discussed. As FDSL is an important learning technique, this survey represents a useful resource for interested researchers working on solving various problem in computer vision and related areas of application.

1. Introduction

Deep learning is a powerful approach for performing different types of computer vision tasks [1], such as object recognition [2,3,4,5], image segmentation [6,7,8,9], visual captioning [10], etc. Deep learning also enables other tasks, such as natural language processing (NLP) [11]. Deep networks pre-trained on large image datasets, e.g., ImageNet [12] have been used after fine-tuning for two important reasons [13]: First, the features learned by deep networks from large datasets help deep networks to generalize more effectively and rapidly; Second, pre-trained deep networks are successful at avoiding over-fitting during fine-tuning for smaller downstream tasks.

It is well known that the performance of deep networks depends on their architecture as well as their training [14,15,16,17]. A multitude of successful deep networks have been developed with a large number of parameters. To train these parameters, a very large number of training images are required; hence, the need for large-scale image datasets. Popular deep networks include AlexNet [18], VGG [19], GoogLeNet [20], ResNet [21], and DenseNet [22]. Popular large-scale image datasets include ImageNet [12] and OpenImage [23]. Deep networks have achieved state-of-the-art performance for many computer-vision applications [2,6,10,24,25,26,27,28].

In spite of their success, the training of deep networks has become expensive and time-consuming. This is due to the laborious collection and manual annotation of the high volume of data required with large-scale datasets. For example, ImageNet [12], which is a popular dataset, has about 1.3 M annotated images with 1 K classes. In ImageNet, every image has been manually annotated and the annotation process is, therefore, substantial. In addition to this, collecting and annotating video data is more expensive still due to its temporal aspect. For example, the Kinetics video dataset [29] includes 500 K human action videos with 600 classes. Every video in the dataset is around 10 seconds long. Several Turkish Amazon workers were involved in the collection and annotation of this large-scale dataset.

Bearing the above issues in mind, it is necessary to avoid lengthy and expensive manual annotation. To this end, different learning schemes have been proposed for training deep networks. These schemes include semi-supervised learning (SSL) [30,31,32,33,34,35,36,37,38,39], weakly supervised learning (WSL) [40,41,42,43,44], unsupervised learning (USL) [45,46,47,48,49,50,51,52,53,54,55,56,57], and self-supervised learning [58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76]. The advantage of these schemes is that they can be used to train deep networks without the use of labeled data; hence, the expensive manual annotation process can be avoided. This can save time as well as expense. A promising supervised learning scheme, which involves use of synthetic images with auto-generated labels, is formula-driven supervised learning (FDSL) [77]. This learning scheme automates the dataset creation process and has produced promising results when used for pre-training. FDSL represents a viable alternative to other learning schemes which offer training using unlabeled data. FDSL has the potential to alter the way in which large-data models are trained, i.e., without manual collection or manual annotation. This potentially critical development, opening a new area of research, motivated us to conduct a literature survey on FDSL. It is hoped that, through such surveys, improved strategies for addressing the critical issues associated with the training of large-data models will emerge and become widely applied.

The main contributions of the paper can be summarized as follows:

• A detailed review of recent supervised learning schemes with respect to formula-driven supervised learning is presented.
• Extensive manual data collection and annotation methods for training large-data models based on FDSL are discussed using synthetic datasets.
• The state-of-the-art, the advantages and disadvantages, and the limitations and future potential of the techniques are considered.

The remainder of the paper is structured as follows: In Section 2, we discuss different deep-learning schemes. This is followed by Section 3, wherein, we discuss formula-driven supervised learning. Section 4 considers issues associated with FDSL and its future potential. We present our conclusions in Section 5.

2. Deep-Learning Schemes

In this section, we discuss different deep-learning schemes which are grouped into four categories: supervised, semi-supervised, weakly supervised, and unsupervised learning.

2.1. Brief Discussion

2.1.1. Supervised Learning

In supervised learning, for a dataset X containing data denoted by $X_i\in X$, for every data entry there is a manually annotated label $Y_i$. If there are N labels for the training set $D={\left\{X_i\right\}}_{i=0}^N$, the loss function is given by:

$$loss\left(D\right)=\underset{\theta }{min}\frac{1}{N}\sum _{i=1}^{N}loss(X_i,Y_i)$$

(1)

where $\theta$ is defined as the model convergence error parameter in the loss equation.

When trained accurately with manually annotated labels, supervised learning schemes achieve state-of-the-art performance on various computer vision tasks [2,6,18,26]. In spite of this, there are burdens of collection of data and investment of the time and effort required for manual annotation, which, in turn, requires specific skills. To address these issues, semi-supervised, weakly supervised and unsupervised learning schemes have been proposed.

2.1.2. Semi-Supervised Learning

In the semi-supervised learning scheme [30,31,32,33,34,35,36,37,38,39], for a small labeled dataset X and a large unlabeled dataset Z, for every data $X_i\in X$, there exists a manually annotated label $Y_i$. Given N labels in the training set $D_1={\left\{X_i\right\}}_{i=0}^N$ and M unlabeled training set $D_2={\left\{Z_i\right\}}_{i=0}^N$, the loss function is given by:

$$loss(D_1,D_2)=\underset{\theta }{min}\frac{1}{N}\sum _{i=1}^{N}loss(X_i,Y_i)+\frac{1}{M}\sum _{i=1}^{M}loss(Z_i,R(Z_i,X))$$

(2)

where $R(Z_i,X)$ is a task-specific function relating every unlabeled data $Z_i$ with the labeled training set X. Semi-supervised learning can be helpful for a very large corpus of data where manual annotation is not extensive. However, its accuracy is not as good as that of traditional approaches.

2.1.3. Weakly Supervised Learning

In the weakly supervised learning scheme [40,41,42,43,44], for a dataset X having data $X_i\in X$, there exists a coarse-grained label $C_i$. For a training set $D_i={\left\{X_i\right\}}_{i=0}^N$, the loss function is given by:

$$loss\left(D\right)=\underset{\theta }{min}\frac{1}{N}\sum _{i=1}^{N}loss(X_i,C_i)$$

(3)

The costs associated with weakly supervised learning are much less than for supervised learning; hence, large sparsely labeled datasets are much easier to build. Many studies have proposed learning from images collected from the Internet with the use of hashtag labels [78,79]. Good performance has been observed after using these techniques, although, again, the performance may not be as good as that using traditional methods. Improving the efficacy of the approach requires further research.

2.1.4. Unsupervised Learning

In unsupervised learning [80,81,82,83], manually annotated labels are not used. In fact, these techniques do not use labels. Formulas can serve as effective model representations [84]. The deep networks are trained using auto-generated pseudo-labels; hence, there is no need for manual annotation. One type of unsupervised learning is the self-supervised learning scheme. Several self-supervised learning schemes have been proposed [58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76]. This type of learning scheme is referred to in some reports as unsupervised learning [45,46,47,48,49,50,51,52,53,54,55,56,57]. In comparison to supervised learning schemes, which require data paired with labels in the form $(X_i,Y_i)$, where $Y_i$ is the manually annotated label, self-supervised learning uses pseudo-label data pairs in the form $(X_i,P_i)$. Here the pseudo-label $P_i$ is auto-generated for the task without using any manual annotation. The pseudo-labeling is performed with the help of image attributes, such as the image context [45,85,86,87].

For a training set $D_i={\left\{P_i\right\}}_{i=0}^N$ having N labels, the loss function is given by:

$$loss\left(D\right)=\underset{\theta }{min}\frac{1}{N}\sum _{i=1}^{N}loss(X_i,P_i)$$

(4)

Recently, unsupervised learning schemes have become quite popular. However, the complexity of the process is greater than that of competing techniques, and performance might not necessarily be equivalent to the latter.

2.2. Recent Trends in Deep-Learning Schemes

Here, we indicate some recent trends in the development and application of deep-learning schemes referred to above. Figure 1 shows the number of related global publications over the past 10 years. Figure 2 shows the total number of related global publications (with breakdown) over the last 10 years. Figure 3 presents a breakdown of the numbers of related global publications for 2010 and 2021, respectively. In the figures, SL, SSL, WSL, and USL stand for supervised learning, semi-supervised learning, weakly supervised learning, and unsupervised learning, respectively.

In the next section, we discuss the formula-driven supervised learning scheme, which, as the name indicates, is a supervised learning scheme.

3. Formula-Driven Supervised Learning

The development of FDSL [77] was motivated by the need to find a technique for the automatic generation of pre-training datasets without taking images from nature. The authors who proposed the concept believed that FDSL would outperform other pre-training techniques by being more fair, private, and ethical. They also believed that it would reduce the burden of manual annotation and the massive downloading of images.

3.1. Background of FDSL

Current computer-vision deep-learning schemes use image datasets that have millions of images to train visual architectures. Although outstanding results have been achieved using such schemes, there are serious issues associated with them. These include a huge burden of manual annotation, the cost of image collection and labeling, privacy issues, copyright issues, ethical issues, and fairness concerns [77]. In [77], the authors propose pre-training of computer-vision models without the use of natural images to overcome these issues. They refer to their technique as formula-driven supervised learning (FDSL). The technique is used for the creation of pairs of images and labels using mathematical formulae.

FDSL can be mathematically expressed by

$$arg\underset{M}{max}({\mathbb{E}}_{y,s}\left[l(M\left(x\right),y)\right]\mathbf{s}.\mathbf{t}.x=F(\theta ,s),y=\theta )$$

(5)

In the above equation, $\mathbb{E}$ is the Euclidean space representation of the fractal, M is the classification network used for pre-training, l is the classification-loss, x is the image obtained by generation, and y is the image label. The FDSL images are mathematically synthesized using a formula F with a parameter $\theta$, which is an affine transformation parameter set related to shift or rotation, and a randomly generated seed s. The aim of the FDSL training is prediction of $\theta$, using which the image x was generated. It is assumed that the label y has a uniformly distributed value on a set of discrete values $\Theta ={\left\{{\theta }_k\right\}}_k^K$. This feature introduces a classification-loss l over K classes, e.g., a cross-entropy-based loss function. Figure 4 presents an overview of FDSL.

3.2. Learning Frameworks

Currently, supervised learning represents the state-of-the-art in computer vision [18,19,20,21,88,89,90,91]. Research has recently been undertaken to decrease the data volume for unsupervised, weakly supervised and self-supervised training to avoid the need for manual annotation. Self-supervised training has the potential to create pre-trained architectures cost-efficiently. This involves use of a basic, but relevant, ‘pre-text task’ [45,50,51,69,87,92]. Although earlier techniques [45,51,87] were not suitable as alternatives to manual annotation, new techniques, such as SimCLR [93], MoCo [94], and DeepCluster [53], are much better. Semi-supervised learning (SSL) [30,31,32,33,34,35,36,37,38,39] has the potential to replace human annotation, although there are significant issues with downloading, privacy and fairness. FDSL [77] is superior to these techniques because it generates new mathematically formulated images along with their respective labels.

3.3. Formula Based Projection of Images

Fractals are one of the most popular mathematical image projection techniques. Fractal theory research is extensive [95,96,97]. It is used to render an image pattern using a basic mathematical expression [98,99,100] and to reconstruct architectures for object recognition [101,102,103,104]. Although rendering a fractal pattern leads to potential loss of its infinite representation for 2D images, humans naturally recognize such renderings. Since fractals occur naturally [95,105], the founders of FDSL claim [77] that fractals may aid in the learning of natural-image-based scenes and objects. They also consider [77] other techniques, such as Bezier curves [106] and Perlin noise [107] for rendering purposes. These techniques have been implemented and evaluated experimentally [77].

3.4. FDSL Datasets

As work on FDSL has increased, some interesting databases have been developed. Their details are provided below.

3.4.1. Fractal DataBase

The fractal dataBase (FractalDB) [77] was developed for FDSL. It contains pairs of fractal images I and their respective category labels c [98], which are generated using an iterated function system (IFS) [98]. The IFS is defined over a metric space $\chi$ as:

$$\mathrm{IFS}=\{\chi ;w_1,w_2,\cdots ,w_N;p_1,p_2,\cdots ,p_N\},$$

(6)

where $w_i:\chi \to \chi$ is the transformation function, $p_i$ is probability with summation 1, and N is the aggregate of transformations.

In IFS, each fractal $S={\left\{x_t\right\}}_{t=0}^{\infty }\in \chi$ is randomly constructed [98] using a two-step algorithm by repeating it for $t=0,1,2,\cdots$ from a starting coordinate ${\mathbf{x}}_0$. First, predefined probabilities $p_i=p(w^{*}=w_i)$ are used to select a transformation $w^{*}$ from $\{w_1,\cdots ,w_N\}$ for determining the $i^{th}$ transformation. Next, a fresh point ${\mathit{x}}_{t+1}=w^{*}\left({\mathit{x}}_t\right)$ is generated.

2D fractals are constructed using a Euclidean space $\chi ={\mathbb{R}}^2$ by an affine transformation [98]. The transformation has six parameters ${\theta }_i=(a_i,b_i,c_i,d_i,e_i,j_i)$ relating to rotation or shift:

$$w_i(\mathit{x};{\theta }_i)=\left[\begin{array}{cc}{a}_i& b_i\\ c_i& d_i\end{array}\right]\mathit{x}+\left[\begin{array}{c}{e}_i\\ f_i\end{array}\right]$$

(7)

The fractal image is generated by drawing dots on a uniform background.

IFS has a set of parameters along with their probabilities, given by:

$$\Theta ={\left\{({\theta }_i,p_i)\right\}}_{i=1}^N$$

(8)

The authors of FDSL assume that every category has a unique $\Theta$. They generate 1k and 10k random categories for the FractalDB-1k and the FractalDB-10k datasets, respectively. In these datasets, N is chosen from the distribution $\mathbb{N}=\{2,3,4,5,6,7,8\}$. ${\theta }_i$ has bounds [−1, 1] for $i=1,2,\cdots ,N$. $p_i$ is of the form:

$$p_i=\frac{\det \left(A_i\right)}{{\sum }_{i=1}^N\det \left(A_i\right)}$$

(9)

where $A_i=(a_i,b_i,c_i,d_i)$ is the affine rotation. Finally, the new category $\Theta ={\left\{({\theta }_i,p_i)\right\}}_{i=1}^N$ is accepted after further inspections.

Figure 5 shows fractal 2D images from the FractalDB dataset.

3.4.2. MV-FractalDB

Moving beyond FractalDB, which is a 2D image database, the authors of [108] developed an autogenerated multi-view image dataset for FDSL. They used fractal geometry to construct the dataset. The dataset has been named the multi-view fractal database (MV-FractalDB). Figure 6 shows some fractal images (3D) from the database.

MV-FractalDB has been used for pre-training deep models and promising results have been obtained. Based on experimentation [108], the MV-FractalDB pre-trained deep models were found to perform better than other self-supervised methods, such as SimCLR and MoCo. In addition, the MV-FractalDB pre-trained deep models converged faster than those trained on ImageNet. A performance comparison of MV-FractalDB pre-trained models against other state-of-the-art models is shown in

Table 1. Performance in terms of %age accuracy on the ModelNet [109] and the MIRO [110,111] datasets, respectively [108]. (Note: SFSL = self-supervised learning, SL = supervised learning).

Pretraining	Learning	ModelNet40	ModelNet40	MIRO
Technique	Method	(12 Views)	(20 Views)	(20 Views)
—	—	84.0	91.5	91.7
ImageNet	SL	88.1	96.1	100.0
SimCLR	SFSL	88.1	95.1	100.0
MoCo	SFSL	86.4	95.3	100.0
FractalDB1k	FDSL	87.4	94.9	100.0
MV-FractalDB1k	FDSL	87.6	95.7	100.0

.

3.4.3. Other Notable FDSL Databases

Another recent FDSL dataset is TileDB [112], which contains patterns made with tiles. A tile is a group of wallpapers with 2D repetition and complicated textures. It is obtained by adding three operations for hexagonally shaped tiles: (i) moving vertices, (ii) deforming edges, and (iii) moving symmetrically in a specific direction. Using this basic technique, the TileDB dataset was created with 1000 classes and 1071 images for each class. The FractalDB, Kataoka et al. [91] proposed some formula-based datasets for the pretraining of computer-vision models, such as convolutional neural networks [14,15,16,113,114] and visual transformers [17,115]. These are the Perlin noise-based PerlinNoiseDB and the Bezier curve-based BezierCurveDB. The DeiT model [116] was pretrained and fine-tuned using these formula-based image datasets. This enabled determination that FractalDB was the best choice for FDSL of computer-vision models developed to date. This is confirmed by data presented in

Table 2. Performance comparison of visual transformer pretraining in terms of % age accuracy with FractalDB1k and other FDSL datasets for BezierCurveDB and PerlinNoiseDB [77].

Dataset	Cifar-10	Cifar-100	Cars	Flowers
—	78.3	57.7	11.6	77.1
PerlinNoiseDB	94.5	77.8	62.3	96.1
BeizerCurveDB	96.7	80.3	82.8	98.5
FractalDB1k	96.8	81.6	86.0	98.3

. Improvements in the percentage classification accuracy of up to +18.5, +23.9, +74.4, +21.2 on the Cifar-10, Cifar-100, Cars dataset, and Flowers dataset, respectively, using FractalDB-1k were observed.

The authors of FractalDB [77] investigated various adaptable parameters, including #category, #instance, filling rate, fractal weight, #dot, and image size. For further information about these parameters, readers may refer to [77]. The experimentation results reported in [77] are shown in

Table 3. Performance comparison in terms of % age accuracy of FractalDB1k [77], FractalDB10k [77], TileDB [112], DeepCluster10k (DC) [53], ImageNet100 [12], ImageNet1k [12], Places30 [117] and Places365 [117] on pretrained ResNet-50 for various datasets, as given in [77]. The datasets used were CIFAR10 (C-10) [118], CIFAR100 (C-100) [118], ImageNet1k (ImNt-1k) [12], Places365 (P-365) [117], PascalVOC-2012 (VOC-12) [119] and Omniglot (OG) [120]. (Note: SFSL=self-supervised learning, SL=supervised learning).

Technique	Image Type	Method	C-10	C-100	ImNt-1k	P-365	VOC-12	OG
—	–	–	87.6	62.7	76.1	49.9	58.9	1.1
DC	Natural	SFSL	89.9	66.9	66.2	51.5	67.5	15.2
Places30	Natural	SL	90.1	67.8	69.1	–	69.5	6.4
Places365	Natural	SL	94.2	76.9	71.4	–	78.6	10.5
ImageNet100	Natural	SL	91.3	70.6	–	49.7	72.0	12.3
ImageNet1k	Natural	SL	96.8	84.6	–	50.3	85.8	17.5
TileDB	Synthetic	FDSL	92.5	73.7	–	–	71.4	–
FractalDB1k	Synthetic	FDSL	93.4	75.7	70.3	49.5	58.9	20.9
FractalDB10k	Synthetic	FDSL	94.1	77.3	71.5	50.8	73.6	29.2

.

As it can be seen from Table 3, FDSL showed notable performance improvements when using FractalDB for pre-training and its potential was highlighted. From the experimental results presented in Table 1, Table 2 and Table 3, the performance of FDSL was generally equivalent to that of other competing methods. This performance was achieved using different models pretrained on synthetic data, in contrast to the huge datasets created by conventional methods.

4. Issues and Future Scope

4.1. Issues

Although FDSL is promising, there are issues associated with it. These include the limited number of FDSL formulae and their limited parameters [77]. This issue impacts on pre-training and final validation, leading to lower performance compared to natural-image- based pre-training, as is shown in Table 1, Table 2 and Table 3, where the performance is seen to be lower. Capturing the richness of natural images using mathematically generated images remains an issue. Capturing color variations and naturally occurring patterns, textures, etc., as found in natural image pretraining, is also an issue. The limitations in the number of mathematically generated patterns may also affect performance [77]. FDSL currently lags behind other competing techniques due to its semi-simplistic model approximation of the mathematical formulae [77]. Richer mathematical models can help build better pseudo-natural data, which can be even richer and more compact than natural data. This is, again, subject to experimental confirmation.

4.2. Future Scope

In spite of the issues associated with FDSL, it is hoped that, with more in-depth research, better results will be achieved. Colored fractals are also available [108], leading to better generalizations. Developing stronger mathematical formula for automatic image generation is a potential area of interest. Pattern generation identical to that for natural objects is another potential area of interest. In addition, moving beyond simple fractal and basic patterns [77] by generating richer artificial images can lead to much better results. The combination of FDSL with other training techniques is also a promising research area, which may lead to enhanced performance. With the development of stronger large-data models, e.g., visual transformers [17,115], FDSL performance can improve. Moreover, by developing large-data models, which are more specifically adaptable to mathematical models, more success may be achieved for FDSL as it relies on mathematical modeling.

5. Conclusions

In this survey paper, a gentle introduction to formula-driven supervised learning (FDSL) has been provided. Various aspects of FDSL were discussed including its mathematical background, methodology, experimental results, issues and future scope. It was observed that FDSL produced promising results for object recognition. It was also emphasized that FDSL addresses the issues associated with natural image pre-training on huge datasets making it a suitable candidate for computer-vision applications. These issues include manual annotation costs and time, privacy, and fairness. It is hoped that the readers will be encouraged to learn about and undertake research in this interesting and promising area of computer vision.