BWLM: A Balanced Weight Learning Mechanism for Long-Tailed Image Recognition

Abstract

With the growth of data in the real world, datasets often encounter the problem of long-tailed distribution of class sample sizes. In long-tailed image recognition, existing solutions usually adopt a class rebalancing strategy, such as reweighting based on the effective sample size of each class, which leans towards common classes in terms of higher accuracy. However, increasing the accuracy of rare classes while maintaining the accuracy of common classes is the key to solving the problem of long-tailed image recognition. This research explores a direction that balances the accuracy of both common and rare classes simultaneously. Firstly, a two-stage training is adopted, motivated by the use of transfer learning to balance features of common and rare classes. Secondly, a balanced weight function called Balanced Focal Softmax (BFS) loss is proposed, which combines balanced softmax loss focusing on common classes with balanced focal loss focusing on rare classes to achieve dual balance in long-tailed image recognition. Subsequently, a Balanced Weight Learning Mechanism (BWLM) to further utilize the feature of weight decay is proposed, where the weight decay as the weight balancing technique for the BFS loss tends to make the model learn smaller balanced weights by punishing the larger weights. Through extensive experiments on five long-tailed image datasets, it proves that transferring the weights from the first stage to the second stage can alleviate the bias of the naive models toward common classes. The proposed BWLM not only balances the weights of common and rare classes, but also greatly improves the accuracy of long-tailed image recognition and outperforms many state-of-the-art algorithms.

1. Introduction

At present, the application of deep neural networks [1,2,3,4] has been popularized in various fields [5,6,7,8,9,10], but the achievements of deep neural networks greatly rely on the availability of real-world annotated datasets [11,12,13,14]. Such data often follow a long-tailed distribution [15,16,17,18], where the quantity or cardinality of each type of data is severely imbalanced [14,19,20]. In fact, rare classes are equally important in recognition tasks. For example, in autonomous driving, vehicles and pedestrians are the common classes with more available samples, but animals crossing the road also need to be detected for safety reasons. [14,21]. Another sample is that the common and rare species should have the same importance in animal recognition. [22]. This has sparked in-depth research in the field of deep neural networks on long-tailed image recognition, with the aim of achieving high accuracy averaging across all classes [17].

Due to having much more training data for common classes than rare classes, the training loss is dominated by common classes [17]. Therefore, in the naive models without any strategy, the accuracy of common classes is always higher than rare classes. To alleviate the challenges posed by the training data in the long-tailed problem and balance various classes [23,24], existing research has mainly employed three types of strategies: resampling, loss function engineering, and stage-wise training. In resampling, many methods have been proposed to balance the data distribution of each class by upsampling rare classes, downsampling common classes, or both [25,26,27]. In loss function engineering, the loss function is influenced by assigning low weights to common classes or high weights to rare classes, which has been widely adopted before [28,29]. However, multiple research studies have shown that artificially dividing weights can lead to poor training performance [30]. Therefore, there have been methods for implicitly defining common and rare classes, focusing on effective samples, sample difficulty, etc. [31,32]. In stage-wise training, some methods use transfer learning to learn rare classifiers based on the features of common classes [33,34,35,36]. Research has shown that the decoupling of feature learning and classifier learning is superior to traditional integrated training models [2].

The estimation of the long-tailed data using ordinary loss functions may lead to bias toward common classes. The current balanced weighted loss function either focuses on common classes or rare classes, which prompts us to explore balancing loss functions that simultaneously reduce the bias of common classes and increase the bias of rare classes. In addition, the naive training model has a high weight bias towards common classes, prompting us to explore further balancing network weights between classes.

To better balance common classes and rare classes, we train the model in two stages, as shown in Figure 1. Considering the effect of feature learning and decoupling training of classifier learning on long-tailed image recognition, we decide to transfer the feature learning from the first stage to the second stage, which enables the second stage to inherit the weights from the first stage. In the second stage, we propose the Balanced Focal Softmax (BFS) loss function that considers both common and rare class weights by combining a loss function that reduces redundancy of the common classes with a loss function that increases attention to rare classes. In order to better balance network weights between classes, we also propose a Balanced Weight Learning Mechanism (BWLM) that balances network weights in the norm. We adopt the weight decay for “punishing large weights and learning small weights” to further balance the weights of common and rare classes in the BFS loss function. We summarize the main contributions as follows:

• We propose the BFS loss function that combines the weight balance characteristics of common and rare classes. By adjusting the weights between the loss functions, the balance proportion of common and rare classes can be adjusted, which correspondingly affects the accuracy of common and rare classes.
• We propose BWLM to balance network weights, and investigate the weight balancing effect of weight decay on multiple balanced loss functions. We prove that weight decay also learns smaller balanced weights to make the model more balanced and improve the accuracy of multiple balanced loss functions, especially for BFS; BWLM performs better when the Imbalance Factor (i.e., the ratio of common classes with the most instances to rare classes with the least instances) is large. We use five long-tailed distributions including the large-scale dataset ImageNet-LT with Imbalance Factor (IF) = 256 and the dataset CIFAR100-LT with IF = [10, 50, 100, 200] to prove that our method achieves optimal results with increasing IF. Therefore, it proves that our method BWLM can handle more complex long-tailed image recognition problems.
• We adopt two-stage learning, which can narrow the weight norms gap between common and rare classes in the native model.

The remainder of this paper is structured as follows: Section 2 introduces and analyzes existing methods. Section 3 defines the problem and introduces our proposed method. Section 4 introduces the experimental setup, dataset, and analysis of experimental results. Section 5 concludes the study and introduces future research.

2. Related Work

Data Re-Balancing. Most previous research on long-tailed imbalanced data have focused on re-balancing during the training. The re-sampling strategy includes over-sampling the rare image classes [15,37,38] and under-sampling the common image classes [15,39,40]. The purpose is to restore imbalanced training data to a regular and balanced distribution state. Usually, over-sampling has a significant effect on large datasets, but for small datasets, rare classes often exhibit over-fitting. Because the majority of the data in the common classes are discarded in cases of under-sampling, the deep model’s capacity for generalization declines. When the IF is too large, the under-sampling strategy is unreasonable. The core idea of reweighting [29,35,41,42] is to assign different weights to the losses of the classes. However, such a strategy often leads to unstable training when the IF is too large. When the assigned weight is too large, the gradient will also be abnormally large, leading to a deviation in the accuracy of the class.

Loss Function Engineering. For class-level reweighting methods, it is difficult to find suitable and effective weights by directly setting weights based on the number of classes. Thus, some methods do not clearly define the common and rare classes, shifting attention to the effective sample size, the sample difficulty, etc. By assessing each class’s degree of learning, Sinha et al. [43] determined loss weights and suggested a method called Class Wise Differentially Balanced Loss (CDB loss). Wang et al. [44] considered all classes as rare classes and proposed the loss function, which adaptively balances the negative gradient between different classes to improve the classifier’s ability to distinguish rare classes. Seesaw loss [45] dynamically adjusts the penalty for classes with fewer samples based on the ratio of the number of samples in rare classes to that in common classes as a mitigation factor. Li et al. [46] utilized the degree of imbalance between positive and negative samples of different classes to rebalance the contribution of their losses, and extended the idea of equalization loss to the object detector [47]. Cui et al. [31] proposed the concept of effective sample size, believing that there is information overlap between data, especially in common classes with large amounts of data. This reduces the marginal benefits extracted by the model from the data. Hence, they proposed class-balanced loss (CB), which adds the effective sample size as a weight to the loss function.Tan et al. [48] found that, because the positive samples of each class are negative samples for other classes, the gradient from the negative samples for rare classes is greater than that of positive samples. Therefore, the negative gradient of common classes suppresses the learning of rare classes. Therefore, they proposed equalization loss to ignore the gradient of rare classes in the softmax function to solve the problem of long-tailed rare classes. In Ren et al. [32], based on the Bayesian theorem, the label distribution bias leads to Softmax regression, which results in a classifier that leans more towards common classes. Therefore, they considered the changes in label distribution and proposed a balanced softmax loss.

Stage-wise Training. Dividing the training task into multiple stages to train the deep networks [8,49,50] is also a common strategy. The LFME [51] framework is a multi-stage knowledge distillation method that divides a long-tailed dataset into multiple balanced subsets and distills the knowledge in the trained expert models into the student models to guide the learning of the student model. Decoupling feature learning with the usual cross-entropy loss and classifier learning with the class balance loss, and treating them as two separate stages, was the proposal put forth by Kang et al. [2]. This study proves the important role of multi-stage training in long-tailed image recognition, as well as the importance of class balance loss in the second stage. On the basis of decoupling feature learning and classifier learning, Shaden et al. [52] used fine-tuning to balance the norm through parameter regularization in the second stage to achieve good results in long-tailed image recognition. In our paper, we drew inspiration from this stage-wise training concept, and the focus is to balance the network weights of long-tailed image recognition for different class-balanced loss functions.

3. Method

We provide a thorough explanation of our framework in this section. In Section 3.1, we first give a summary of the particular problems with long-tailed picture identification. Then, we introduce how to obtain a relatively balanced dataset and transfer the weights of stage 1 to stage 2 in Section 3.2. Finally, we propose the BFS loss function and BWLM in Section 3.3.

3.1. Preliminaries

For training datasets with long-tailed data distribution, long-tailed image recognition attempts to learn high-performance classification models. Formally, we define $D=\left\{x_i,y_i\right\},i\in \left\{1,2,\dots ,n\right\}$ as the long-tailed training set, where data sample $x_i$ is labeled as $y_i\in \left[1,2,\dots ,C\right]$, C is the total number of classes, and $n_j$ is the number of samples in class j, where ${\sum }_{j=1}^C{n}_j=n$. IF $=\frac{max\left(n_j\right)}{min\left(n_j\right)}$ measures the degree of imbalance in the long-tailed training set. For long-tailed image recognition, IF ≫ 1. The estimated class probabilities of the model are $P={\left[p_1,p_2,\dots ,p_C\right]}^{\top }$, where $p_i\in \left[0,1\right]\forall i$. We assume learning a classification network $f\left(·;\mathsf{\Theta }\right)$, where $\mathsf{\Theta }=\left\{{\theta }_{l,k}\right\}$ is the $k^{th}$ filter weights at layer-l. We denote ${\theta }_j$ as the classifier weights of class-j and optimize $\mathsf{\Theta }$ by minimizing the loss L throughout the entire training set D to train the network:

$${\mathsf{\Theta }}^{*}=arg\underset{\mathsf{\Theta }}{min}F\left(\mathsf{\Theta };D\right)\equiv \sum _{i=1}^{n}L\left(f\left(x_i;\mathsf{\Theta }\right),y_i\right)$$

(1)

The naive classifier is influenced by the long-tailed distribution of D, and the classifier weights ${\theta }_j$ leans more towards common classes. Therefore, our motivation is to learn a balanced classifier by transferring the weights of common classes to rare classes, balancing the performance of the classifier by combining the advantages of two losses and further balancing classifier weights of common and rare classes by regularizing classifier weights. The specific implementation process can be seen in Figure 2.

3.2. Weights Transfer

We divide the training into two stages and transfer the weights from the first stage to the second stage, and then use the proposed balanced loss function to balance the long-tailed problem.

Data Balanced Review. In order to obtain a relatively balanced dataset, we propose the data balanced review method in [53]. For class-j, $D_j$ is the set of all its training samples. We denote the middle number of classes as $m=C/2$ and obtain the first-stage training data $D_{stage1}={\sum }_{j=1}^m{D}_j$. In the second stage, we define the relatively balanced training set $\widehat{D}=\left\{\widehat{x_i},\widehat{y_i}\right\},\widehat{y_i}\in \left[1,2,\dots ,C\right]$ as:

$$\widehat{D}={\sum }_{j=1}^m{D}_j\left[sam\left(n_{m+1}\right)\right]+{\sum }_{j=m+1}^C{D}_j$$

(2)

where $sam(·)$ is a random sampling method [54]. Figure 1 provides a more intuitive display.

Two-stage Transfer. Knowledge transfer currently has a wide range of research and applications, and its main idea is to transfer knowledge-rich upstream tasks to knowledge-scarce downstream tasks [55]. In the imbalanced long-tailed training set, which includes classes with sufficient knowledge and classes with insufficient knowledge, we use the Data Balanced Review [53] method to balance the dataset of downstream tasks based on the idea of head-to-tail transfer. Therefore, we use two stages to train our framework: According to Figure 1, we denote the training network for stage 1 as:

$${\mathsf{\Theta }}^{*}=arg\underset{\mathsf{\Theta }}{min}F\left(\mathsf{\Theta };D_{stage1}\right)$$

(3)

At this stage, we use cross-entropy (CE) loss as the cost function denoted as $L_{CE}$. Based on the ${\theta }^{*}$ from stage 1 of training, we freeze the model parameters of stage 1 and add a new Fully Connected (FC) layer, and optimize FC layer weights ${\mathsf{\Theta }}_{fc}$ by minimizing the loss L throughout the entire training set $\widehat{D}$ to train the stage 2 network:

$$\widehat{\mathsf{\Theta }}=arg\underset{{\mathsf{\Theta }}_{fc}}{min}F{\left({\mathsf{\Theta }}_{fc};\widehat{D}\right)}_{{\mathsf{\Theta }}^{*}}$$

(4)

Through our exploration, the weight transfer method can improve the accuracy of knowledge-deficient classes, but the accuracy of knowledge-sufficient classes relatively decreases.

3.3. BWLM

We explain how BWLM works in our second stage starting by introducing the two important elements in BWLM: BFS loss function and weight decay.

BFS Loss aims to solve the problem of training from long-tailed data by focusing on a reweighting strategy that simultaneously focuses on common and rare classes.

We choose Class-Balanced (CB) [31] loss to focus on common classes. This method introduces a weight factor

$$\left(1-\beta \right)/\left(1-{\beta }^{n_j}\right)$$

(5)

We add it to the loss function to design a balanced loss function to solve the problem of data imbalance in the training. It is inversely proportional to the number of effective samples. As a hyperparameter $\beta \in [0,1)$, $n_j$ represents the total sample size contained in class j and the balanced focal loss is defined as:

$$L_{CB}(z,y)=-\frac{1-\beta }{1-{\beta }^{n_y}}{L}_{FL(z,y)}$$

(6)

The focal loss $L_{FL(z,y)}$[47] can reduce the weight of well-classified samples and make the model focus on difficult samples during training. $p_j^t=1/(1+exp(-z_j^t))$ is the estimated probability of class j, where t means the target. The focal loss is

$$L_{FL}(z,y)=\sum _{j=1}^C{(1-p_j^t)}^{\gamma }log\left(p_j^t\right)$$

(7)

where $\gamma$ is the focusing parameter that can reduce the weight of simple samples. The balanced focal loss is expanded as:

$$L_{CB}(z,y)=-\frac{1-\beta }{1-{\beta }^{n_y}}\sum _{j=1}^C{(1-p_j^t)}^{\gamma }log\left(p_j^t\right)$$

(8)

Overall, the idea can be summarized as follows: By estimating the number of effective samples and reweighting the focal loss, it captures the marginal decreasing effect of the data and reduces redundant information in the common samples.

To remove the discrepancy between the test set’s and the training set’s posterior distributions [56], we use a balanced softmax [32]

$${\varphi }_j=\frac{n_j{e}^{z_j}}{{\sum }_{i=1}^C{n}_i{e}^{z_i}}$$

(9)

In order to avoid bias in testing and training, it considers label distribution shifts and derives softmax suitable for the long-tailed problem using the exponential family form of Multinomial. This tends to focus on rare classes. Similarly, we obtain the Balanced Softmax (BS) loss:

$$L_{BS}\left(z,y\right)=-log\left({\varphi }_y\right)=-log\left(\frac{n_y{e}^{z_y}}{{\sum }_{i=1}^C{n}_i{e}^{z_i}}\right)$$

(10)

Based on the characteristics of class-balanced focal loss and balanced softmax loss, it can be analyzed that class-balanced focal loss pays more attention to selecting effective samples for the common classes. Balanced softmax loss reduces the preference of the classifier towards the common classes caused by the difference in posterior distributions between the training and testing sets. Therefore, we propose a balanced loss function BFS loss that can address both common and rare classes. BFS is denoted as:

$$\begin{array}{cc}\hfill L_{BFS}& =\alpha \frac{1-\beta }{1-{\beta }^{n_y}}{L}_{FL(z,y)}+[-(1-\alpha )log\left({\varphi }_y\right)]\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & =\alpha \frac{1-\beta }{1-{\beta }^{n_y}}{L}_{FL(z,y)}+(\alpha -1)log\left({\varphi }_y\right)]\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & =(\alpha -1)log\left(\frac{n_y{e}^{z_y}}{{\sum }_{i=1}^C{n}_i{e}^{z_i}}\right)-\frac{\alpha (1-\beta )}{1-{\beta }^{n_y}}\sum _{j=1}^C{(1-p_j^t)}^{\gamma }log\left(p_j^t\right)\hfill \end{array}$$

(13)

We discuss the selection of $\alpha$ as a weight and hyperparameter in Section 4.3.

Weight Decay is a relatively mature research [57,58] that constrains the networks by limiting the growth of the network weights [52], which can reduce network complexity, reduce over-fitting, and improve the generalization ability of the network. Weight decay [52] applies the L2-norm to the network weights:

$$\widehat{\mathsf{\Theta }}=arg\underset{{\mathsf{\Theta }}_{fc}}{min}F{\left({\mathsf{\Theta }}_{fc};\widehat{D}\right)}_{{\mathsf{\Theta }}^{*}}+\lambda \sum _j^{}{∥{\theta }_j∥}_2^2$$

(14)

We select the value of hyperparameter $\lambda$ based on [52]. Weight decay increases the penalty for larger weights to prevent them from growing larger while emphasizing the learning of smaller weights. However, we find that using weight decay after Weights Transfer can improve the overall performance of long-tailed image recognition, which is beneficial for the accuracy of rare classes. To further balance the weight of common and rare classes, we adopt weight decay to adjust the impact of the model complexity on the balance loss function. Adopting weight decay can narrow the weight gap between common and rare classes, which can be reflected in Section 4.3. Therefore, we attempt to further balance the weights for $L_{CE}$, $L_{BS}$, and our proposed $L_{BFS}$:

(1) By combining the encouragement of learning weight decay with $L_{CB}$, we obtain:

$$L_{CBW}(z,y)=-\frac{1-\beta }{1-{\beta }^{n_y}}\sum _{j=1}^C{(1-p_j^t)}^{\gamma }log\left(p_j^t\right)+\lambda {∥{\theta }_j∥}^2$$

(15)

which can prevent over-fitting while balancing the weight difference between the common and rare classes.

(2) By combining the encouragement of learning weight decay with $L_{BS}$, we obtain:

$$L_{BSW}\left(z,y\right)=-log\left({\varphi }_y\right)=-log\left(\frac{n_y{e}^{z_y}}{{\sum }_{i=1}^C{n}_i{e}^{z_i}}\right)+\lambda {∥{\theta }_j∥}^2$$

(16)

(3) Similarly, our proposed balanced weight learning of $L_{BFS}$ is denoted as:

$$\begin{array}{cc}\hfill L_{BWLM}& =(\alpha -1)log\left(\frac{n_y{e}^{z_y}}{{\sum }_{i=1}^C{n}_i{e}^{z_i}}\right)-\frac{\alpha (1-\beta )}{1-{\beta }^{n_y}}\sum _{j=1}^C{(1-p_j^t)}^{\gamma }log\left(p_j^t\right)+\lambda {∥{\theta }_j∥}^2\hfill \end{array}$$

(17)

which balances the attention to common and rare classes appropriately, and punishes larger weights more heavily, allowing for learning smaller balance weights, thus facilitating long-tailed image recognition tasks.

4. Experiments

In this part, we test the proposed BWLM using CIFAR100-LT and ImageNet-LT, two popular long-tailed benchmarks. In addition, we carry out several ablation studies to assess the significance of every BWLM component in detail.

4.1. Datasets

We obtain five long-tailed image recognition datasets based on CIFAR100 and ImageNet for training (

Table 1. The specific information of five long-tailed image recognition datasets.

Dataset	IF	Training Samples	Testing Samples	Classes	Max Class Size	Min Class Size
ImageNet-LT [34]	256	115,846	50,000	1000	1280	5
CIFAR100-LT [59]	200	9502	10,000	100	500	2
	100	10,847	10,000	100	500	5
	50	12,608	10,000	100	500	10
	10	19,573	10,000	100	500	50

). The testing sets used for evaluation are all balanced.

CIFAR100-LT. Following [59], we use the CIFAR100 [60] dataset by downsampling the training samples to generate five long-tailed datasets of CIFAR100-LT with IF $\in \left[200,100,50,10\right]$. CIFAR100-LT includes 100 classes with the class frequency distributions of four IFs (Figure 3—left).

ImageNet-LT. The authors of [34] artificially truncate the balanced ImageNet [11] into a long-tailed version. Imagenet-LT has 1000 classes with a maximum of 1280 and a minimum of five samples of training data (Figure 3—right).

Figure 3 summarizes the class frequency distribution of these datasets. The detailed information of all the datasets is shown in Table 1. ImageNet-LT is the long-tailed distribution data with the largest IF and data volume, while CIFAR100-LT contains multiple long-tailed distributions with different IFs and relatively small data volumes.

4.2. Implementation

We train the models by using a long-tailed training set and evaluate them by using a balanced test set. Following [34], we divide the training set into three types based on the sample size of each category to further report accuracy: Many (greater than 100), Medium (Between 20 and 100), and Few (less than 20).

We use the PyTorch toolbox [61] to train our models. We use ResNet32 [8] on CIFAR100-LT and ResNeXt50 [62] on ImageNet-LT. The models are trained with a batch size of 64 on a single NVIDIA GeForce RTX 3090 GPU for 320 epochs on CIFAR100-LT and 500 epochs on ImageNet-LT. The momentum of the SGD optimizer is 0.9. According to the findings of [31], when the data is imbalanced, loss and validation errors gradually increase after the learning rate drops. We use a cosine learning rate regulator [63] to gradually decay the learning rate from 0.01 to 0.

4.3. Ablation Study

Choose $\alpha$ of BWLM. In Formula (17), we set the hyperparameter $\alpha$ to adjust the usage weights of CB and BS. $\alpha$ represents the usage weight of CB, and $1-\alpha$ represents the usage weight of BS. Figure 4 shows the accuracy of various classes on the five datasets, and analyzes the impact of $\alpha$ on All, Many, Medium, and Few classes of five datasets with different IFs. From the impact of $\alpha$ on the accuracy of Many classes in Figure 4b, it can be seen that almost all datasets can achieve the optimal results when $\alpha$ reaches its maximum value. It can be proven that CB reduces the redundancy of Many classes and focuses more on the accuracy of Many classes. In Figure 4c,d, Medium and Few classes can achieve optimal results when the value of $\alpha$ is small. Therefore, BS focuses more on classes with less data volume, which can also be proven. For All classes in Figure 4a, as $\alpha$ changes, the accuracy of all classes shows a slow downward trend in five datasets with different IFs. Since the number of Few classes is much more than that of Many classes, the overall performance of All classes is aligned with Few classes, that is, the optimal performance can be reached when $\alpha$ is set to be small. Although the optimal value of IF = 100 is obtained when $\alpha =0.7$, the accuracy difference is only 0.01 when $\alpha =0.1$ and $\alpha =0.7$. Therefore, we suggest using BWLM by first setting $\alpha =0.1$ by default. In the comparison experiments, we select $\alpha$ based on the optimal value of BWLM on the different datasets in Figure 4a.

The effect of each component. To make our proposed balancing technique more suitable for application in the second stage, we also include weight decay in the first stage. This effective non-learning technique can perform post hoc scales on the classifier learned in the first stage. Our results are based on the ImageNet LT results in

Table 2. Ablation study on ImageNet-LT with respect to accuracy (%). “naive”: single model with cross-entropy loss; “+”: after weight transfer in the second stage; “CE”: cross-entropy; “CB”: Class-Balanced focal loss; “BS”: balanced softmax; “BFS”: Balanced focal and softmax; “CEW”: CE with weight decay; “CBW”: CB with weight decay; “BSW”: BS with weight decay. The best and the second-best results are shown in underline bold and bold, respectively.

	ImageNet-LT
	Many	Medium	Few	All
naive	57.6	32.5	13.4	39.6
weight transfer
+CE	55.5	48.5	19.4	44.6
+CB	51.6	45.2	18.0	41.5
+BS	52.9	47.7	25.3	44.6
+BFS	52.8	47.7	24.3	44.5
+CEW	62.8	51.6	13.1	46.4
+CBW	61.7	50.2	26.3	47.0
+BSW	58.2	50.1	29.7	47.3
+BWLM	60.0	50.8	26.9	47.5

, where the accuracy of different classes is listed for all algorithms. We divide it into two categories for comparison: CE, CB, BS, and BFS is the reweighting loss function; CEW, CBW, BSW, and BWLM is the method after adding weight decay to the reweighting loss function. Firstly, as the weight transfer CE for the first stage increases from 39.6% of the naive model to 44.6%, it indicates the importance of weight transfer. Secondly, it is also very important to use the balance loss function to learn classifiers based on weight transfer. Then we find that using only the equilibrium loss function without weight decay slightly improves the accuracy for only Few classes. Our method BFS balances out the accuracy of Many and Few of CB and BS, indicating that our idea is effective. This compensates for the lack of CB for Few classes and BS for Many classes. But when weight decay is used to further balance weights, the accuracy can be improved greatly. In particular, CEW reaches 46.4%. BWLM learns the weights of Few classes from BSW and Many classes from CBW, increasing the accuracy to 47.5%.

Classifier’s weight norms.Figure 5a, using the classifier weight norms of the ImageNet-LT validation set, further proves the effectiveness of weight learning. The classifier weight norms of the naive model have a relatively large numerical span from Many classes to Few classes and show a “top-heavy” downward trend. The addition of weight decay to other models has greatly shortened the weight gap between the Many and Few classes. Especially for our proposed BWLM, the classifier weight norm shows a small upward trend, which gives the Medium and Few classes a “fighting power” in the image recognition.

4.4. Long-Tailed Image Recognition

Comparison Methods. Considering the rapid development of long-tailed image recognition, we have compared it with multiple related methods, including the method of modifying the loss function in one stage and the transfer strategy method in two stages. For better comparison, the method we chose and our commonality is that we did not add feature enhancement methods such as Mixup. For example, BS [32] for loss reweighting, NCM [2], $\tau$-Norm [2], and cRT [2] split the training process into two stages: classifier and representation learning, while LFME [51] used transfer learning, etc.

Table 3. The accuracy (%) on ImageNet-LT. In terms of results, our method has achieved competitive results compared to existing loss methods and two-stage transfer methods. Although not as advanced as some recent methods, these training methods added additional techniques such as Mixup and data augmentation. The best and the second-best results are shown in underline bold and bold, respectively.

Method	One-stage	Many	Medium	Few	All
CE [2]	✓	65.9	37.5	7.7	44.4
CE+CB [31]	✓	39.6	32.7	16.8	33.2
KD [65]	✓	58.8	26.6	3.4	35.8
Focal [31]	✓	36.4	29.9	16.0	30.5
OLTR [34]	✗	43.2	35.1	18.5	35.6
LFME [51]	✗	47.1	35.0	17.5	37.2
Range Loss [30]	✓	35.8	30.3	17.6	30.7
Equalization Loss [48]	✓	-	-	-	36.4
BS [32]	✓	50.3	39.5	25.3	41.8
NCM [2]	✗	53.1	42.3	26.5	44.3
$\tau$-Norm [2]	✗	56.6	44.2	26.4	46.7
cRT [2]	✗	58.8	44.0	26.1	47.3
CE+CR [64]	✓	65.1	40.7	19.5	47.3
Our methods
naive	✓	57.6	32.5	13.4	39.6
Weights Transfer
+CE	✗	55.5	48.5	19.4	44.6
+CEW	✗	62.8	51.6	13.1	46.4
+CBW	✗	61.7	50.2	26.3	47
+BSW	✗	58.2	50.1	29.7	47.3
+BWLM	✗	60.0	50.8	26.9	47.5

and

Table 4. The accuracy (%) on CIFAR100-LT with different IFs. We can combine Figure 5b to analyze the rate of accuracy decline.The best results are shown in underline bold.

Imbalance Factor (IF)	10	50	100	200
CE [31]	55.71	43.85	38.32	34.84
CE+CB [31]	57.99	45.32	39.60	36.23
KD [65]	59.22	45.49	40.32	-
Focal [47]	55.78	44.32	38.41	-
LDAM [1]	56.91	-	39.60	-
Mixup [66]	58.02	44.99	39.54	-
Focal+CB [31]	57.99	45.17	39.60	35.62
CE+DRW [1]	58.10	46.50	41.00	36.90
CE+CR [64]	57.40	45.10	40.50	36.90
Focal+CR [64]	58.30	45.20	40.20	37.50
CB+CR [64]	59.20	46.80	40.70	38.50
Our methods
Weights Transfer
+CE	38.8	40.5	37.8	34.8
+CEW	50.0	43.8	41.1	38.3
+CBW	51.3	44.7	41.5	39.5
+BSW	51.5	44.9	40.9	39.1
+BWLM	51.6	45.1	41.5	39.6

, respectively, list the test results for the large-scale ImageNet-LT dataset and the different IFs distributions CIFAR100-LT datasets.

Results. Dividing the long-tailed image recognition task into two stages and transferring the model weights from the first stage to the second stage can effectively improve the performance of long-tailed image recognition and outperform many existing methods. This conclusion can be drawn from ImageNet-LT in Table 3. Our weights transfer achieved 44.6% using the naive model with normal CE loss, which is superior to the naive model with an accuracy of 39.6%. It even outperforms other multi-stage learning methods including OLTR (35.6%) [34] and NCM (44.3%) [2]. By learning the CE classifier to regularize with weight decay, we improve accuracy from 44.6% to 46.4%. Using the weight decay to regularize the balanced loss classifier, the accuracy of the Few classes is improved by 13–16%. The accuracy of the Many classes in CBW is better than that in BSW, but the accuracy of the Few classes is lower than that in BSW. Therefore, the BWLM proposes to combine the two balanced loss functions to better balance the accuracy of Many classes and Few classes, and then reach the highest overall accuracy. For some balanced losses, such as Focal [31], Range Loss [30], Equalization Loss [48], BS [32], etc., they often choose to focus on common or rare classes, resulting in addressing one side and ignoring the other. BWLM addresses both common and rare classes, providing a more comprehensive retention of its classification ability for Many classes, in the meantime improving its classification performance for Medium and Few classes.

In Table 4, this conclusion is also valid on other long-tailed distribution datasets with different IFs. On the four small CIFAR100-LT with different IFs, our model performs best at IF = 100 and IF = 200. We find that as IF increases, the accuracy of existing methods decreases faster than our BWLM. When the IF = [10, 100], the accuracy decreases by 17–25%, while our method controls the accuracy decrease to around 10%. In order to visually observe the descent speed, we have selected several comparison methods in Figure 5b to compare with our method. We can intuitively see that the accuracy of CE+CR [64], Focal+CR [64], and CB+CR [64] decreases significantly with the increase of IF. When IF < 100, they still have an advantage. When IF ≥ 100, our model surpasses other comparison methods, and our BWLM performs best among all IFs compared to other methods that add weight decay. To sum up, our method performs better on long-tailed datasets with large IF, which also proves that our model can solve more extreme long-tailed image recognition problems. Training curves on ImageNet-LT and CIFAR100-LT are shown in Figure 6. It can be seen that BWLM has a significant advantage in big data and has been proven through 150 generations of training.

Figure 7 shows the per-class accuracy for CIFAR100-LT (IF = 200). The impact of the long-tailed problem on the naive model is evident, as there is a significant decline in performance as the class IDs go from small to large. In contrast, the BWLM mitigates the steepness of this decline. It is visually apparent that BWLM substantially improves the classification results for Medium and Few classes without compromising the classification performance of Many classes. This improvement is particularly pronounced for Few classes, providing compelling evidence of the significant balancing capability of BWLM.

5. Conclusions

Long-tailed image recognition is a necessary challenge for the imbalanced data that often occurs in the real world. Based on our observations, the weight allocation strategy for common and rare classes on long-tailed data is divided into paying attention to common class weights or paying attention to rare class weights. We suggest focusing on both common and rare classes in the same balanced loss function by combining the balanced loss functions that focus on common class weights with the balanced loss functions that focus on rare class weights and learning the balanced weights through weight decay. Extensive experiments have shown that the correct use of regularization and balanced loss functions can greatly improve the performance of long-tailed image recognition. We introduce a relatively simple and easy-to-implement method that outperforms various existing balanced loss functions and is validated on multiple long-tailed datasets. Since our BWLM performs the best in large-scale long-tailed datasets, we believe that it can contribute to solving real-world problems related to large-scale long-tailed distributions.