A Novel Progressive Image Classification Method Based on Hierarchical Convolutional Neural Networks

Abstract

Deep Neural Networks (DNNs) are commonly used methods in computational intelligence. Most prevalent DNN-based image classification methods are dedicated to promoting the performance by designing complicated network architectures and requiring large amounts of model parameters. These large-scale DNN-based models are performed on all images consistently. However, since there are meaningful differences between images, it is difficult to accurately classify all images by a consistent network architecture. For example, a deeper network is fit for the images that are difficult to be distinguished, but may lead to model overfitting for simple images. Therefore, we should selectively use different models to deal with different images, which is similar to the human cognition mechanism, in which different levels of neurons are activated according to the difficulty of object recognition. To this end, we propose a Hierarchical Convolutional Neural Network (HCNN) for image classification in this paper. HCNNs comprise multiple sub-networks, which can be viewed as different levels of neurons in humans, and these sub-networks are used to classify the images progressively. Specifically, we first initialize the weight of each image and each image category, and these images and initial weights are used for training the first sub-network. Then, according to the predicted results of the first sub-network, the weights of misclassified images are increased, while the weights of correctly classified images are decreased. Furthermore, the images with the updated weights are used for training the next sub-networks. Similar operations are performed on all sub-networks. In the test stage, each image passes through the sub-networks in turn. If the prediction confidences in a sub-network are higher than a given threshold, then the results are output directly. Otherwise, deeper visual features need to be learned successively by the subsequent sub-networks until a reliable image classification result is obtained or the last sub-network is reached. Experimental results show that HCNNs can obtain better results than classical CNNs and the existing models based on ensemble learning. HCNNs have 2.68% higher accuracy than Residual Network 50 (Resnet50) on the ultrasonic image dataset, 1.19% than Resnet50 on the chimpanzee facial image dataset, and 10.86% than Adaboost-CNN on the CIFAR-10 dataset. Furthermore, the HCNN is extensible, since the types of sub-networks and their combinations can be dynamically adjusted.

1. Introduction

With the development of computer vision technologies, many visual tasks, such as object detection, semantic segmentation, and image classification, have been widely applied in many fields [1,2,3]. Image classification is one of the most common and important visual tasks [4,5,6], and a large number of models have been proposed based on traditional machine learning methods and deep learning methods [7,8,9]. Recently, Convolutional Neural Network(CNN)-based image classification methods, such as AlexNet [10], Visual Geometry Group 16 (VGG16) [11], ResNet [12], and Densely Connected Networks (DenseNet) [13,14], were widely applied in many visual tasks. Generally speaking, the networks with fewer layers usually extract the low-level visual features, while the networks with more layers can extract the more abstract visual features.

These primary research works focus on how to extract distinguishable local features to improve the image classification performance. In practice, however, there are large numbers of various types of objects, and many images suffer from poor illumination conditions, varying degrees of occlusion, similarities between objects, and so on. It is difficult to accurately classify all images by a consistent model, which presents great challenges to image classification [15,16]. For human beings, different types of objects are recognized through different processes, and people tend to quickly make judgments on easy-to-recognize objects based on their own subjective and objective cognition or prior knowledge. Meanwhile, people need further analysis and understanding for relatively difficult-to-recognize objects, and may further perform information abstraction and knowledge reasoning. Therefore, we contend that there are meaningful differences between images, and various models encounter various difficulties when attempting to accurately classify them. For example, images with appropriate lighting conditions are more easily classified correctly by the model than those with strong or weak lighting conditions; it is easier to perform disease screening on medical images for prominent lesions [17,18]. Therefore, we should select the appropriate networks according to the particular tasks. However, in most traditional CNN-based methods, all images need to be sent to the same classification process, which neglects the differences in discrepant classification difficulties for different images [19,20,21].

Inspired by the mechanism of human cognition and the fact that different images present different levels of cognitive difficulty, we design a hierarchical integrated deep learning model named HCNN. The HCNN treats multiple CNNs as sub-networks and uses them progressively for feature extraction [22,23]. Specifically, the simple sub-networks are used to extract visual features for the images that are easy to classify accurately. Moreover, the complex sub-networks are used to extract the more abstract visual features, which are more suitable for the images which are more difficult to accurately classify. The final classification results are obtained by integrating the results of these sub-networks. Most existing models integrate multiple CNNs by fusing the high-level feature/decision of the CNNs to obtain a final result. Our HCNN selectively extracts the composite features of multiple sub-networks in different levels, which is more reasonable and complies with the process of human cognition.

Furthermore, the multi-class joint loss is designed to offer the features of the samples within the same category higher similarity, while the similarity between the features of different categories is made as low as possible. Gradient descent is used to train the entire network end-to-end. Finally, several experiments are conducted on a medical image dataset, two common image classification datasets (CIFAR-10, CIFAR-100 [24]), and a chimpanzee dataset [25]. The comparison experimental results show that the HCNN achieves superior performance to the existing related models. Moreover, ablation experiments prove that our model’s performance is superior to that of each single network and combinations of several sub-networks. In addition, it is worth noting that the HCNN has good scalability, since the types and combinations of CNN modules can be dynamically adjusted depending on the specific tasks involved.

The main contributions of this paper are as follows:

(1) We propose a progressive image classification model, named HCNN, which can progressively use its sub-network modules (with different depths of network layers) to extract different levels of visual features from images, while the classification results of different images are output by corresponding sub-network modules. In brief, the HCNN can use the sub-network modules with fewer network layers to quickly yield image classification results for the images that are easy to classify accurately, while the images that are difficult to classify accurately need to pass through more complex sub-network modules.

(2) A multi-class joint loss is designed to reduce the distance between the features of samples within the same category, while increasing the distance between the features of samples in different categories. In addition, gradient descent is used for the entire model training end-to-end.

(3) The performance and scalability of the HCNN are verified on four image classification datasets. The comparison and ablation experimental results show that the HCNN achieves significant performance improvements compared with existing models and combinations of several sub-networks.

This paper is organized as follows. In Section 2, we review the related image classification models, ensemble learning models, and metric learning models and describes their relationships with our model. In Section 3, we elaborate the basic framework and loss functions of HCNNs and give the model implementation process in the test stage. In Section 4, we compare HCNNs and eight related methods on our own ultrasonic image dataset and three public image datasets. We also perform validation experiments to further analyze the HCNNs. The final conclusion is given in Section 5.

2. Related Work

Image classification is one of the most important visual tasks in computer vision. Due to the rapid development of deep learning technologies and its superior performance in computer vision, image classification methods based on DNNs have become increasingly mature. To accurately classify images, various types of artificial visual features are designed, and the visual features are automatically learned by DNNs. Related classifiers are then used to distinguish the categories of the images. To date, a large number of deep learning-based image classification methods have been proposed [26,27] and have been widely used in different computer vision tasks. In addition, several improved models have been successively proposed to improve the image classification performance. Xi et al. [28] proposed a parallel neural network by combining texture features. This model can extract features that are highly correlated with facial changes, and thus achieves better performance in facial expression recognition. Hossain et al. [29] developed an automatic date fruit classification system to satisfy the interest of date fruit consumers. Goren et al. [30] collected the street images taken by roadside cameras to form a dataset, and then designed a CNN to check the vacancy in the collected dataset. To form an efficient classification mechanism that integrates feature extraction, feature selection, and a classification model, Yao et al. [31] proposed an end-to-end image classification method based on an aided capsule network and applied it to traffic image classification. An image classification framework for securing against indistinguishable plaintext attacks was proposed by Hassan et al. [32]. This framework performs a secure image classification on the cloud without the need for constant device interaction. To solve the multi-class classification problems, Vasan et al. [33] proposed a new method to convert raw malware binaries into color images, which are used by the fine-tuned CNN architecture to detect and identify malware families.

A single CNN may be impacted by gradient disappearance, gradient explosion, and other similar factors, while network models based on ensemble learning have better immunity to these adverse factors due to the cooperative complementation of multiple CNNs. For example, Ciregan et al. [34] designed a method by utilizing multiple CNNs, which are trained by using the same training datasets. These trained CNNs are then used to obtain multiple prediction results, which are in turn fused to obtain the final result. This method employs the simple addition of the predicted results of different CNNs, which it treats in isolation. Frazao et al. [35] assigned different weights to multiple CNNs; the CNNs with better performance have higher weights, and therefore have greater impacts on the final results. An integration of CNNs is used to detect polyps by Tajbakhsh et al. [36]; this approach can accurately identify the specific types of polyps by using their color, texture, and shape features. Ijjina et al. [37] proposed a human action prediction method, which combines several CNNs and uses the best predicted result as the final result. Although these methods use multiple neural network modules to carry out related classification tasks, the modules are independent of each other and the interactions between models are ignored. To solve these problems, Adaboost CNN models have been proposed. For example, Taherkhani et al. [38] combined several CNN sub-networks based on the Adaboost algorithm. These CNN sub-networks have the same network structure; thus, the transfer learning method can be used between adjacent layers, and the last CNN sub-network module outputs the final results. The model in [38] is unable to selectively and progressively use the CNN sub-networks for feature extraction, and the testing images need to go through all CNN sub-networks to obtain the final results.

A key problem with the semantic understanding of images is that of learning a good metric to measure the similarity between images. Deep metric learning-based methods have been proposed to learn the appropriate similarity measures between pairs of samples, while samples with higher similarities are classified into a single category according to the distances between samples. These approaches have been widely used for image retrieval [39], face recognition [40], and person re-identification [41]. For example, Schroff et al. [42] proposed a face recognition system named FaceNet, and a triplet loss was designed to measure the similarities between samples. Wang et al. [43] proposed a general weighting framework for a series of existing pair-based loss functions by fully considering three similarities for pair weighting, and then collecting and weighting the informative pairs. These metric learning methods focus on optimizing the similarity of image pairs. Furthermore, center loss is proposed by Wen et al. [44] to define a category center for each category, as well as to minimize the distance within one category. Wang et al. [45] proposed an angular loss, which considers the angle relationship to learn a better similarity metric, while the angular loss aims at constraining the angle at the negative point of triplet triangles.

The related works mentioned above mainly involve DNNs, ensemble learning, and metric learning. Meanwhile, there are intrinsic correlations between these fields. In general, ensemble learning needs to use multiple DNN models, and the design of both ensemble learning and DNNs should be on the basis of the theory of metric learning. Specifically, the proposed HCNN is an ensemble learning model based on DNNs for the image classification task, and the multi-class joint loss is designed for the HCNN according to the basic theory of metric learning.

3. The Proposed Hierarchical CNNs (HCNNs)

In order to classify different images in real life, we design a hierarchical progressive DNN framework, named Hierarchical CNNs (HCNNs), which consists of several sub-networks. The images need to go through one or more sub-networks so as to obtain a more reliable classification result. In this paper, we refer to the definitions of samples in self-paced learning methods [46]: the samples that are easy for models to identify are defined as easy samples, while the difficult-to-identify samples are denoted as hard samples. In this section, we will describe the overall structure of the HCNN and its loss function. Multiple CNNs are combined to form HCNNs, which can progressively carry out the sub-networks to classify the images; the cross-entropy loss and triple loss are combined for model training to more accurately extract the distinguishing features of the images.

3.1. The Model Framework of HCNNs

Based on the basic concept of ensemble learning, we try to aggregate multiple CNNs into a strong image classification model [1,47,48]. However, unlike traditional ensemble learning methods or Adaboost CNNs [38], which consist of the same type of sub-networks that are indiscriminately trained and tested, our HCNN consists of several different types of CNNs as the sub-networks, and these sub-networks are trained progressively in order. In this paper, we choose Alexnet [10], VGG16 [11], Inception V3 [49], Mobilenet V2 [50], and Resnet-50 [12] as the basic sub-networks (see Figure 1). In addition, there are no limits on the number of sub-networks and their types. At the training stage, images are assigned weights to express the difficulties encountered by models in accurately classifying them. If an image can not be accurately classified by a sub-network, its weight will be increased. Images with updated weights are then input into the next sub-network for extracting more abstract and effective visual features. In this section, we will elaborate on HCNNs in more detail.

Assume that HCNNs have M sub-networks, and they are trained one by one. Let $w_i^m$ be the weight of the i-th image for the m-th sub-network, and $D^m=\{w_1^m,w_2^m,\cdots ,w_n^m\}$. Here, $i\in \{1,2,\cdots ,n\}$, $m\in \{1,2,\cdots ,M\}$, while n is the number of all images in the training dataset.

First of all, the weights of all images need to be initialized. We therefore input all these training images with their initial weights into the first sub-network (Alexnet, $m=1$) for model training.

$$D^1=\{w_1^1,w_2^1,\cdots ,w_n^1\},$$

(1)

where $w_i^1=1/n,i=1,\cdots ,n$. The first sub-network is then trained through multiple iterations. The gradient descent is used to update its parameters in each iteration. Finally, the trained sub-network can give the predictions:

$$y_i^m=G^m\left(x_i\right),$$

(2)

where $G^m(·)$ represents the m-th sub-network, and $y_i^m$ is the predicted label of the i-th sample by the m-th sub-network $G^m$. Next, we select the samples that meet the condition of $y_i^m\ne t_i$, where $t_i$ is the ground truth of the category label of the i-th sample. We further use the following equation to calculate the weighted error rate ${\epsilon }^m$ of the m-th sub-network $G^m(·)$ on all selected samples in the training set:

$${\epsilon }^m=\sum _{i\_s=1}^{N_{i\_s}}{w}_{i\_s}^m,$$

(3)

where $w_{i\_s}^m$ is the weight of the $i\_s$-th selected samples for $G^m(·)$, and $N_{i\_s}$ is the number of selected samples. Subsequently, ${\epsilon }^m$ is used to obtain the weight coefficient ${\alpha }^m$ of $G^m$, which denotes the importance coefficient of $G^m$ in HCNNs:

$${\alpha }^m=\frac{1}{2}log\frac{1-{\epsilon }^m}{{\epsilon }^m}.$$

(4)

As Equation (4) shows, ${\alpha }^m$ is inversely proportional to ${\epsilon }^m$, i.e., with a smaller error rate ${\epsilon }^m$, the corresponding sub-network will have larger values of the importance coefficient throughout the whole model. Furthermore, ${\alpha }^m$ is used to update the weights of the samples to train the next sub-network.

For the images that meet the condition $y_i^m=t_i$, we have

$$w_i^{(m+1)}=w_i^{m}exp(-{\alpha }^m).$$

(5)

Otherwise,

$$w_i^{(m+1)}=w_i^{m}exp\left({\alpha }^m\right).$$

(6)

Then,

$$D^{(m+1)}=\{w_1^{(m+1)},w_2^{(m+1)},...,w_n^{(m+1)}\}.$$

(7)

Therefore, if the predicted results $y_i^m$ exhibit a high degree of agreement with the true labels $t_i$ of the images, then the weights of the images for the next sub-network decrease; otherwise, their weights increase. We then use the image samples with their updated weights to train the next sub-network for multiple iterations.

For a dataset containing a small number of samples, the initial and updated weights of the samples are applicable to training of HCNNs. However, if the dataset consists of a large number of samples, there is a risk of gradient explosion occurring during model training due to the loss values being too small (possibly even approaching zero); this means that the network parameters cannot be updated normally. To solve this problem, we use the weights of samples to obtain the category weights using Equation (8):

$$C_j^{(m+1)}=\sum _{k\_j=1}^{K_j}{w}_{k\_j}^{(m+1)},$$

(8)

where $C_j^{(m+1)}$ represents the weight of the j-th category for the $(m+1)$-th sub-network, and $w_{k\_j}^{(m+1)}$ is the weight of the $k\_j$-th sample belonging to the j-th category, which has $K_j$ samples. We then use $C_j^{(m+1)}$ as the weights of the samples belonging to the j-th category (Equation (9)).

$$w_{k\_j}^{{}^{\prime }(m+1)}=C_j^{(m+1)}.$$

(9)

Therefore, before training each sub-network, we need to update the weights of all samples according to the weights of their corresponding categories. The sub-network will then pay more attention to the samples with larger weights.

HCNN is a scalable model, and its architecture is illustrated in Figure 1. In addition, HCNN enhances the correlation between different sub-networks by transmitting the feature vectors and the sample weights in the previous sub-network to the next sub-network.

3.2. Multi-Class Joint Loss in HCNNs

During model training, we constantly updated the weights of the image categories and the images to express the difficulties encountered by the model. We then needed to design the loss function, which can guide the model to extract the specific visual features from different images. In addition, this loss function should attempt to make the difference in the visual features within the same category as small as possible, while the difference in the visual features in different categories should be as large as possible.

Cross-entropy loss with category weights. The cross-entropy function $L_C$ is a classic and commonly used loss function. In this paper, to enable the HCNN to select its corresponding sub-networks and therefore extract the visual features in different levels, a category weight is assigned to each image category; subsequently, the new cross-entropy loss with category weights can be expressed by the following equation:

$$L_C^{{}^{\prime }(m+1)}=C_j^m{L}_C^{(m+1)}.$$

(10)

Here, $L_C^{{}^{\prime }(m+1)}$ is the cross-entropy loss with category weights for the $(m+1)$-th sub-network, and $L_C^{(m+1)}$ is the traditional cross-entropy loss.

Weighted triplet loss. For image classification, the problem may arise that there may be less similarity between images within the same category, while there is more similarity between images in different categories; as a result, it is difficult to effectively improve the image classification performance. The triplet loss can guide models to learn the visual features to further cluster the samples within the same category and separate the samples of different categories. Therefore, we use a weighted triplet loss in each sub-network. This guides HCNNs to extract more discriminative visual features between the samples of different categories, as shown in Figure 2.

Assume that we have a series of image samples $\left\{x_1,x_2,...,x_n\right\}$, and $\left\{y_1,y_2,...,y_n\right\}$ are their true labels. We then define an anchor image $u^a$, a positive image sample $u^{+}$, and a negative image sample $u^{-}$. More specifically, $u^a$ is an image in one category, $u^{+}$ is another image in the same category with $u^a$, and $u^{-}$ is an image in another category that differs from the category of $u^a$. During model training, we can obtain a triplet set consisting of $U^a$, $U^{+}$, and $U^{-}$ in each batch, and then randomly select the corresponding samples to form a triple $S=\left\{u^a,u^{+},u^{-}\right\}$ as the input of each sub-network. We can then obtain the triplet loss $L_T^m$ for the m-th sub-network:

$$L_T^m=Max\left\{d(f^a,f^{+})-d(f^a,f^{-})+\alpha ,\beta \right\}.$$

(11)

Here, $f^a$, $f^{+}$, and $f^{-}$ represent the visual features extracted by the m-th sub-network from the images of $u^a,u^{+},u^{-}$, respectively, while $d(·)$ is the Euclidean distance. Moreover, $\alpha$ is a threshold parameter used to distinguish between the positive and negative samples of the anchor samples. $\beta$ is a parameter that is close to 0 without being equal to 0. Triplet loss is used to reduce the distance between the features of $u^a$ and $u^{+}$ and expand the distance between the features of $u^a$ and $u^{-}$, as shown in Figure 2. Then, triplet loss can be used to solve the following three situations in HCNNs.

Case I: If $d(f^a,f^{+})+\alpha <d(f^a,f^{-})$, then $L_T^m=\beta$. This situation shows that the current sub-network can accurately classify these three image samples; thus, there is no need to pay more attention to them in the subsequent sub-networks.

Case II: $d(f^a,f^{+})<d(f^a,f^{-})<d(f^a,f^{+})+\alpha$. This situation shows that high similarity exists among these three image samples, and the current sub-network finds it difficult to distinguish them. This triple S then needs to pass through the subsequent sub-networks with more complex network structures.

Case III: $d(f^a,f^{-})<d(f^a,f^{+})$. This situation shows that the current sub-network cannot distinguish these image samples, and that their more abstract features need to be extracted by the subsequent sub-networks.

Weighted multi-class joint loss function. HCNNs can progressively classify images and achieve visual feature learning at different levels. In addition, in each batch during model training, a weighted multi-class joint loss function is designed by combining cross-entropy loss with category weights and weighted triplet loss.

$$L^m=L_C^{{}^{\prime }\left(m\right)}+\gamma L_T^m.$$

(12)

Here, $L^m$ is the weighted multi-class joint loss for the m-th sub-network. $\gamma$ is a hyperparameter; in this paper, $\gamma =0.5$.

3.3. Model Testing

To test the proposed model, we need to provide a threshold $H^m$ for each sub-network so as to make the model output the final classification results. When image classification confidence in the m-th sub-network is higher than $H^m$, this prediction is reliable; otherwise, the credibility of the image classification results is lower. Generally speaking, the values of $H^m$ can be set larger, which ensures that the difficult-to-identify images can pass through the subsequent sub-networks with more complex network structures. Figure 3 shows the simple process of image classification of HCNNs.

In more detail, the testing process of HCNNs with M sub-networks can be described as follows.

• Step 1: The test image is input into the m-th sub-network for visual feature learning ($m=1$ for the first sub-network). The model then outputs the probability distribution of the image classification results $P^m=\{p_1^m,p_2^m,...,p_{N\_C}^m\}$, where $N\_C$ is the number of image categories.
• Step 2: A comparison is drawn between the maximum classification probability $p_k^m=Max\left(P^m\right)$ and $H^m$.
• Step 3: If $p_k^m\ge H^m$ or $m=M$, then the model outputs the classification results corresponding to $p_k^m$; otherwise, $m=m+1$, and return to Step 1.

4. Experimental Results and Analysis

To verify the effectiveness and superiority of the proposed HCNN, we implement our model on two challenging image classification datasets (our ultrasonic prostate image dataset and the chimpanzee dataset [25]) and two commonly used image classification datasets (CIFAR-10 and CIFAR-100 [24]). Furthermore, we also utilize several related existing DNN models for comparative experimental analysis. In addition, we conduct ablation experiments to verify the influences of different sub-networks.

4.1. Image Classification Datasets

(1)

Ultrasonic image dataset of prostate

There have been related works on medical image datasets [51,52,53,54], while there are few works for prostate cancer screening. The traditional method of prostate cancer screening usually uses prostate biopsy puncture to obtain the pathological results, which causes great pain for patients. Therefore, we have collected ultrasonic images of prostate, and attempted to design CNN-based models for prostate cancer screening. Our ultrasonic image dataset of prostate has 932 images, which were selected from a number of ultrasonic images according to doctors’ experience. We divided these ultrasound images into two categories: the ultrasound images of patients with prostate cancer and those of patients without prostate cancer.

(2)

The chimpanzee facial image dataset

The chimpanzee dataset is provided by Loos et al. in [25]. The chimpanzee facial images were captured at Zoo Leipzig in Germany and Taï National Park in Africa. There are large numbers of images with weak or highlight illumination, incomplete facial contours, partial occlusion by branches or leaves, and inconsistent image sizes. This image dataset is therefore very challenging for the image classification task.

Table 1. Datasets summary.

Datasets	Category	Train	Validation	Test
Ultrasonic image dataset	2	746	93	93
Chimpanzee facial image dataset	52	1689	292	540
CIFAR-10	10	40,000	10,000	10,000
CIFAR-100	100	45,000	5000	10,000

presents the details of the chimpanzee facial images used in this paper. We selected at least five images for each chimpanzee individual from the entire dataset as our test images.

(3)

CIFAR-10 and CIFAR-100

CIFAR-10 and CIFAR-100 contain 60,000 images each, where each color image has 32 × 32 pixels. CIFAR-10 comprises 10 categories (aircraft, car, bird, cat, deer, dog, frog, horse, boat, truck). As in previous works, we also use 50,000 images for model training and 10,000 images for testing, and there are no duplicate image samples. In addition, we select 20% of the training images as the validation image dataset. CIFAR-100 consists of the image samples of 100 categories, and we also divide these into 50,000 training images and 10,000 testing images. The training, validation, and testing image sets are allocated according to a ratio of 9:1:2 in the whole CIFAR-100 image dataset (as shown in Table 1).

4.2. Experimental Setup

The proposed HCNN is an image classification model based on ensemble learning. In this paper, we choose Alexnet [10], VGG16 [11], Inception V3 (I_V3) [49], Mobilenet V2 (M_V2) [50], and Resnet-50 [12] as the basic sub-networks. Of course, in specific tasks, the types of sub-networks can be changed, and we also can add or subtract sub-networks. We further implement several existing deep learning models and ensemble learning models as the comparison models. The experimental results of our model are compared with these basic sub-networks by using the same parameters. We further make comparisons of our model with Adaboost-CNN [38] on CIFAR-10, and with Wide-ResNet 40-2 [55], Wide-ResNet 40-2+CutMix [56], DenseNet-100 [57], and DenseNet-100+CutMix [56] on CIFAR-100. In addition, to verify the gradual improvements achieved by HCNNs, we draw comparisons of HCNNs with the single sub-networks and their different combinations.

For each image dataset, we set the batch size to 10, and the initial learning rate is 0.0001. We set $\gamma$ in Equation (12) to 0.5. The dimensions of $f^a$, $f^{+}$ and $f^{-}$ in triplet loss are set to 256 uniformly. All models used in this paper were implemented on three TITAN Xp GPUs.

4.3. Experimental Results

(1)

Experimental results on the ultrasonic image dataset

Prostate cancer screening based on ultrasound images is mainly used to distinguish whether the patients have prostate cancer or not according to their prostate ultrasound images. It can be regarded as a binary image classification problem. However, the prostate ultrasound images are fairly complex, and the lesions are not obvious, so it is difficult for professional doctors to diagnose prostate diseases only using ultrasound images. Therefore, there are challenges for the automatic screening of prostate cancer utilizing computer vision technologies from ultrasound images. In this paper, we try to use several DNN models to perform the binary image classification task, and the experimental results are shown in

Table 2. The ablation analysis on the ultrasonic image dataset.

Models	ACC	F1 Score	Recall	Precision
Alexnet [10]	0.6989	0.7200	0.8571	0.6207
VGG16 [11]	0.7634	0.7381	0.7381	0.7381
Inception V3 [49]	0.7500	0.7164	0.7273	0.7059
Mobilenet V2 [50]	0.6989	0.6499	0.6190	0.6842
Resnet50 [12]	0.8065	0.7805	0.7619	0.8000
Alexnet+VGG16	0.7361	0.7077	0.7188	0.6970
Alexnet+VGG16+Inception V3	0.7639	0.7385	0.7500	0.7273
Alexnet+VGG16+	0.7917	0.7693	0.7813	0.7576
Inception V3+Mobilenet V2
HCNN	0.8333	0.8125	0.8125	0.8125

. We can obtain the following points from the experimental results. First, all the models used in this paper fail to achieve perfect image classification performance, and the highest recognition accuracy is lower than 85%, which shows that there is difficulty in recognizing prostate cancer. Second, Resnet50 achieves better performance among the single deep network models. It has 4.31% higher accuracy than VGG16, and 5.65% higher than Inception V3. Third, the models combing multiple networks have increasing accuracies; for example, “Alexnet+VGG16+Inception V3+Mobilenet V2” has 2.83% higher accuracy than VGG16. Among these methods, the HCNN with five deep networks achieves the best performance, and it has 2.68% higher accuracy than Resnet50. Figure 4 shows the graph of Table 2; we can see that HCNN achieves obvious advantages over other models in most evaluation indicators. In addition, the performance would be further improved with the improvement or addition of the sub-networks.

(2)

Experimental results on the chimpanzee facial image dataset

There are a large number of challenging chimpanzee facial images in the chimpanzee facial image dataset [25]; it is therefore difficult for models to achieve good image classification performance. In this paper, we implement nine different models on this dataset; the experimental results are shown in

Table 3. The ablation analysis on the chimpanzee facial image dataset.

Models	ACC	F1 Score	Recall	Precision
Alexnet [10]	0.5532	0.5470	0.5428	0.5512
VGG16 [11]	0.6885	0.6836	0.6818	0.6854
Inception V3 [49]	0.7008	0.6976	0.6956	0.6996
Mobilenet V2 [50]	0.5737	0.5699	0.5701	0.5698
Resnet50 [12]	0.7336	0.7327	0.7321	0.7334
Alexnet+VGG16	0.7023	0.7010	0.6998	0.7023
Alexnet+VGG16+Inception V3	0.7234	0.7200	0.7199	0.7201
Alexnet+VGG16+Inception V3+Mobilenet V2	0.7349	0.7316	0.7288	0.7344
HCNN	0.7455	0.7435	0.7451	0.7419

, where we can see that these models, which perform well on some public image classification databases, do not perform well on this chimpanzee facial image dataset. The model with the single network that works best is Resnet50, with 0.7336% accuracy, while the HCNN achieves the highest image classification accuracy of 74.55%. Figure 5 shows the curves of these models’ performance in related evaluation indicators, and it can be seen that the HCNN has general advantages over models with single neural networks and other models with multiple sub-networks in terms of F1 score, recall, and precision, which clearly demonstrates its effectiveness and superiority.

(3)

Experimental results on CIFAR-10

Table 4. The ablation analysis on the CIFAR-10 dataset.

Models	ACC	F1 Score	Recall	Precision
Alexnet [10]	0.7881	0.7881	0.7894	0.7868
VGG16 [11]	0.8860	0.8810	0.8818	0.8802
Inception V3 [49]	0.8922	0.8921	0.8928	0.8914
Mobilenet V2 [50]	0.8704	0.8639	0.8647	0.8631
Resnet50 [12]	0.9080	0.9008	0.9011	0.9005
Alexnet+VGG16	0.8877	0.8878	0.8880	0.8877
Alexnet+VGG16+Inception V3	0.9023	0.9086	0.9087	0.9085
Alexnet+VGG16+Inception V3+Mobilenet V2	0.9104	0.9106	0.9107	0.9105
HCNN	0.9226	0.9221	0.9222	0.9221

presents the experimental results of nine different models on CIFAR-10; these models include the sub-networks in HCNNs and their different combinations. For each model, we carried out 20 epochs of model training. From the results shown in Table 4, we can see that Resnet50 [12] achieves the best performance among all models with a single network. Furthermore, the ensemble learning models with different sub-networks achieve general improvements over the corresponding single-network models, which proves the effectiveness of ensemble learning models. Therefore, HCNN achieves the final best performance, with test accuracy of 92.26% on CIFAR-10, and 1.46–13.45% higher classification accuracy than the other five basic sub-networks. In addition, HCNN also has advantages in terms of F1 score, recall, and precision. In Figure 6, the performance difference among these models can be illustrated more clearly. Although these models achieved better results on CIFAR-10 than on the ultrasonic image dataset and chimpanzee facial image dataset, the overall trends of the models’ performance are similar. The reasons may be that the weights of the training samples for each sub-network in HCNNs are updated according to the classification results of the previous sub-network. In this way, different sub-networks can learn the specific features from the images, and present various degrees of difficulty to the various models that attempt to accurately classify them. Therefore, HCNNs can progressively learn the visual features at different levels and gradually improve the image classification performance.

AdaBoost-CNN [38], proposed by Taherkhani et al., is also an ensemble learning model based on DNNs. Its test accuracy on CIFAR-10 reaches 81.40%, as shown in

Table 5. The experimental analysis of Adaboost-CNN and HCNN on the CIFAR-10 dataset.

Models	Acc
Adaboost-CNN [38]	0.8140
HCNN	0.9226

. By contrast, our HCNN has 10.86% higher accuracy. Adaboost-CNN creates a classification model with better performance by combining several simple convolutional sub-networks. However, multiple sub-networks in Adaboost-CNN use the same network structure, and each sub-network is only fine-tuned on the parameters of its previous sub-network. Therefore, it is difficult for the model to learn the specific abstract visual features from the images; this may be the reason for the limited performance of Adaboost-CNN.

(4)

Experimental results on CIFAR-100

CIFAR-100 contains more image categories than CIFAR-10 and is less able to achieve higher image classification accuracy for classification models. Fourteen different models are implemented in CIFAR-100, and the experimental results are shown in

Table 6. The ablation analysis on the CIFAR-100 dataset.

Models	ACC	F1 Score	Recall	Precision
Alexnet [10]	0.5347	0.5326	0.5329	0.5323
VGG16 [11]	0.6556	0.6548	0.6556	0.6540
Inception V3 [49]	0.7675	0.7686	0.7690	0.7682
Mobilenet V2 [50]	0.6601	0.6634	0.6645	0.6623
Resnet50 [12]	0.6031	0.6033	0.6034	0.6032
Alexnet+VGG16	0.6623	0.6640	0.6646	0.6635
Alexnet+VGG16+Inception V3	0.7742	0.7767	0.7778	0.7757
Alexnet+VGG16+Inception V3+Mobilenet V2	0.7798	0.7740	0.7746	0.7735
HCNN	0.7847	0.7846	0.7844	0.7848

and

Table 7. The experimental results of Adaboost-CNN and HCNN on the CIFAR-100 dataset.

Models	Acc
Wide-ResNet 40-2 [55]	0.7473
Wide-ResNet 40-2+CutMix [56]	0.7821
DenseNet-100 [57]	0.7773
DenseNet-100+CutMix [56]	0.7855
HCNN	0.7847

and Figure 7. From the test results shown in Table 6, it can be seen that the models combining different sub-networks achieve better performance than models with a single neural network, which is similar to Table 4. The HCNN, with a test accuracy of 78.47%, achieves accuracy that is 9.46% higher than that of Mobilenet V2 and 1.72% higher than that of Inception V3, which represents the best performance among the single neural network models.

Moreover, as shown in Table 7, HCNN achieves similar performance to DenseNet-100+CutMix [56], but is better than other existing network models with complex network structures. During the image classification process, each image (regardless of whether it is easy or difficult for the model to accurately classify) needs to go through these complex network models to extract visual features. In HCNNs, however, different images will pass through different levels of sub-networks, and the model will learn specific visual features from images at different levels. Therefore, HCNN achieves better effectiveness and efficiency for image classification.

5. Conclusions

At present, all the image classification models treat the images equally. However, there are meaningful differences between images, so different images should be treated differently by various models, which would comply with the basic mechanism of human cognition. Therefore, we propose HCNNs, which classify different images by different numbers of sub-networks. In HCNNs, the easy-to-identify images are recognized by simple sub-networks and output the results directly, while images that are more difficult to identify may need to go through multiple complex sub-networks to extract their more abstract visual features. Through this image classification mechanism, HCNNs achieve better image classification performance compared with existing single-network models and Adaboost CNN with its multiple simple sub-networks. In addition, the HCNN has better scalability and variability; that is, the number of sub-networks can be increased or decreased, and the types of sub-networks can be changed according to the specific visual tasks involved. Therefore, in the future, more detailed models similar to HCNNs may be constructed based on the complexity of the image classification task, which would gradually become closer to the basic mechanism of human cognition, and the models will have higher recognition accuracy and efficiency.