An Improved Medical Image Classification Algorithm Based on Adam Optimizer

Abstract

Due to the complexity and illegibility of medical images, it brings inconvenience and difficulty to the diagnosis of medical personnel. To address these issues, an optimization algorithm called GSL(Gradient sine linear) based on Adam algorithm improvement is proposed in this paper, which introduces gradient pruning strategy, periodic adjustment of learning rate, and linear interpolation strategy. The gradient trimming technique can scale the gradient to prevent gradient explosion, while the periodic adjustment of the learning rate and linear interpolation strategy adjusts the learning rate according to the characteristics of the sinusoidal function, accelerating the convergence while reducing the drastic parameter fluctuations, improving the efficiency and stability of training. The experimental results show that compared to the classic Adam algorithm, this algorithm can demonstrate better classification accuracy, the GSL algorithm achieves an accuracy of 78% and 75.2% on the MobileNetV2 network and ShuffleNetV2 network under the Gastroenterology dataset; and on the MobileNetV2 network and ShuffleNetV2 network under the Glaucoma dataset, an accuracy of 84.72% and 83.12%. The GSL optimizer achieved significant performance improvement on various neural network structures and datasets, proving its effectiveness and practicality in the field of deep learning, and also providing new ideas and methods for solving the difficulties in medical image recognition.

1. Introduction

Medical image classification plays a crucial role in medical diagnosis. Accurate and efficient classification methods can help doctors quickly identify lesions and develop more precise treatment plans. In recent years, the rapid development of deep learning techniques has provided new possibilities for medical image classification. Optimization algorithms are crucial in deep learning and have achieved great success in image classification and medical image analysis. Lingyi Ouyang, Tao He, and Yiqiao Xing [1] summarized the changes in the structure and function of the retinal neurovascular unit (RNVU) in glaucoma and explored the potential pathogenesis of glaucoma, aiming to provide new ideas for the intervention and treatment of glaucoma. In some cases, doctors are not able to make a good judgment of whether the disease is present or not. Therefore, diagnosis with the help of computer-aided diagnosis [2] is particularly important in the treatment of glaucoma.

Commonly used optimization algorithms include Adam [3], SGD [4] and NAdam [5]. Chakrabarti et al. [6] systematically investigated the optimal hyperparameter settings for different gradient descent algorithms to improve their performance. Kuangyu Ding et al. [7] focused on the problem of insufficient weight attenuation in the Adam algorithm, and focused on the Adam series of optimization methods combined with decoupled weight attenuation to improve the performance of deep learning models. Adam⁺ [8] addressed the limitations of the Adam optimizer in variance handling, and proposed an improved Adam⁺ by adaptive variance reduction method. EAdam [9] optimizes the Adam optimizer for the deficiencies in convergence speed and stability, analyzes the design principle of EAdam in detail, and demonstrates its performance improvement in different application scenarios. Lu Xia et al. [10] proposed a fast adaptive gradient method called AdamL that combines a loss function, which investigates the limitations of the Adam optimizer in terms of gradient. Ran Tian and Ankur P. Parikh [11] proposed an optimizer called Amos that combines Adam style optimization and adaptive weight decay to adapt to the scale of the model. Byeongho Heo et al. [12] proposed an optimization algorithm called AdamP that aims to mitigate the deceleration of momentum optimizers on scale invariant weights. The Radar proposed by Liu and Jiang et al. [13] introduces a learning rate correction mechanism to address the problem of unstable training caused by large fluctuations in Adam learning rate variance due to certain reasons. Yichuan Shao et al. [14] proposed an improved Adam’s algorithm based on cyclic exponential decay learning rate and gradient paradigm constraints. Liu Hailiang [15] introduced an adaptive gradient method with energy and momentum, which provides new ideas for the improvement of optimization algorithms. Sedjro S. Hotegni et al. [16] multi-objective optimization method for sparse deep multi-task learning also provides new perspectives and methods for optimization of multi-task learning. StochGradAdam [17] improves Adam’s algorithm for the performance degradation that may occur when dealing with large-scale data, and utilizes stochastic gradient sampling to accelerate neural network training. Can Zhang et al. [18] proposed the WuC-Adam algorithm by combining the Warmup and cosine annealing algorithm strategies for the instability problem in the Adam algorithm. Yichuan Shao, Jiantao Wang et al. [19] introduced a BGE-Adam optimization algorithm combining entropy weighting and adaptive gradient strategy improvement for the Adam optimizer in terms of handling sparse gradients and convergence speed.

This study provides insights into the challenges faced by the Adam algorithm in terms of convergence and robustness. This study find that the performance of the Adam algorithm may suffer when the dataset is more complex or when the learning rate parameter is not set properly. To address this problem, this study introduced three core techniques: a gradient trimming strategy and a periodic learning rate variation and linear interpolation [20] strategy. By using dynamic gradient pruning techniques to restrict gradients and solve the problem of gradient explosion, the total norm of all gradients is calculated. If the total norm exceeds the threshold, all gradients are scaled proportionally so that their total norm does not exceed the threshold. Periodic changes in learning rate can allow the model to escape from the current local optimal region and accelerate convergence, thereby finding better solutions. Linear interpolation technology combines the current updated value with the previous parameter value during the parameter update process, which can smooth the changes in parameters and avoid the drastic changes caused by a single update. These improvements increase the robustness and effectiveness of the model during training.

2. GSL Algorithm Design

2.1. Dynamic Gradient Trimming Strategy

Gradient explosion is a common problem in deep learning, especially when training deep neural networks. Gradient explosion leads to excessive gradient values, which in turn leads to excessive model parameter updates and even numerical overflow, making the training process unable to proceed normally. In order to effectively solve the gradient explosion problem, this study adopts the dynamic gradient trimming technique to limit the gradient.

First, the L2 normalization of the gradient is calculated. Among them, the L2 normalization refers to the Euclidean norm. The calculation formula is shown in Equation (1):

$$||{\nabla }_{\theta }|{|}_2=\sqrt{{\displaystyle \sum _p{(||\nabla p|{|}_2)}^2}}$$

(1)

In the formula $\left|\right|{\nabla }_{\theta }|{|}_2$ denotes the total number of paradigms of the gradient and $\left|\right|\nabla p|{|}_2$ denotes the L2 normalization of the gradient of the parameter $p$.

In order to control the size of the gradient of the parameters, a trimming factor ($clip\_coef$) is defined, which takes the total gradient paradigm as the denominator, and a maximum paradigm is preset $max\_grad\_norm$, if the total gradient paradigm exceeds the preset maximum paradigm, i.e., $clip\_coef$ < 1, then the gradient of each parameter is scaled proportionally, otherwise it remains at its original value. The purpose is to prevent gradient explosion caused by too large or too small gradient and to maintain the stability of gradient update. The calculation formula is shown in Equations (2) and (3):

$$clip\_coef={\displaystyle \frac{max\_grad\_norm}{\left|\right|{\nabla }_{\theta }|{|}_2+\epsilon }}$$

(2)

$$\nabla p\leftarrow \nabla p\times clip\_coef$$

(3)

In the formula ε is a very small value used to prevent the denominator from being zero; and $\nabla p$ denotes the gradient of the parameter $p$.

In this way, the gradients of all parameters are scaled so that the total gradient paradigm is equal to or less than the maximum paradigm $max\_grad\_norm$. Traditional gradient cropping usually restricts the total number of paradigms of all gradients to a predetermined threshold. That is, the total number of paradigms of all gradients is calculated, and if the total number of paradigms exceeds the threshold, all gradients are scaled in equal proportions so that their total number of paradigms does not exceed the threshold. In contrast, a more flexible approach to gradient cropping is used in this study. This means that gradients with different parameters may be cropped to different degrees, rather than just being scaled equally overall.

2.2. Periodic Adjustment of Learning Rate and Linear Interpolation Strategies

The learning rate determines the step size of each parameter update. Periodic increase of the learning rate allows the model to jump out of the current local optimal region and accelerate convergence to find a better solution as soon as possible. Decreasing the learning rate allows the model to more robustly converge to a new region and allows the model to make fine adjustments near the better solution, thus finding a better solution.

The linear interpolation technique combines the current update value with the previous parameter value during the parameter update process, which can smooth out the parameter changes and avoid the drastic changes brought about by a single update. This is useful for stabilizing the training process, especially when large gradient changes occur, i.e., when the learning rate is high or the gradient changes are large to help prevent the gradient explosion or gradient vanishing problem.

The learning rate is periodically adjusted according to a sinusoidal function, calculated as shown in Equation (4):

$$l{r}_{adjusted}=lr\times (1+lr\_factor\times \sin ({\displaystyle \frac{2\pi t}{lr\_period}}))$$

(4)

In the formula $lr$ represents the base learning rate; $t$ represents the current number of steps; $lr\_period$ represents the cycle length of the learning rate change; $lr\_factor$ represents the magnitude of the learning rate change; and $l{r}_{adjusted}$ represents the adjusted learning rate. This periodic change in the learning rate allows for a smoother training process, avoiding oscillations due to too large a learning rate, and avoiding too slow training due to too small a learning rate.

The standard Adam optimizer dynamically adjusts the learning rate during the training process while performing parameter updates. Its parameter update rule formula is shown in Equation (5):

$${\theta }_t={\theta }_{t-1}-\alpha {\displaystyle \frac{{\widehat{m}}_t}{\sqrt{{\widehat{v}}_t}+\epsilon }}$$

(5)

In the formula $\alpha$ denotes the learning rate; ε is a very small value used to prevent the denominator from being zero; deviation correction first-order moment estimation ${\widehat{m}}_t$ and deviation correction second-order moment estimation ${\widehat{v}}_t$.

Linear interpolation is a mathematical method of estimating unknown data points between two known data points. The underlying assumption is that the data changes in a linear fashion between these two known data points. Linear interpolation is the process of estimating new points in a straight line between two known points. Given two points $(x_0,y_0)$ and $(x_1,y_1)$, linear interpolation estimates the value of $y$ corresponding to $x$ between these two points. For two known points $(x_0,y_0)$ and $(x_1,y_1)$, and a point $x$ between $x_0$ and $x_1$, the linear interpolation formula is shown in Equation (6):

$$y=y_0+{\displaystyle \frac{(y_1-y_0)}{(x_1-x_0)}}(x-x_0)$$

(6)

In the process of parameter update, the changes of parameters are smoothed by adding the idea of linear interpolation, which combines the current update value and the last parameter value to reduce the drastic parameter fluctuations, so as to improve the stability of the training process and prevent overfitting. It combines the weights of the parameter values calculated from the current update with the previous parameter values to ensure a smooth and progressive parameter update. The calculation formula is shown in Equations (7) and (8):

$${\theta }_t^{new}={\theta }_{t-1}-l{r}_{adjusted}{\displaystyle \frac{{\widehat{m}}_t}{\sqrt{{\widehat{v}}_t}+\epsilon }}$$

(7)

$${\theta }_{tl}=(1-interp\_factor){\theta }_t^{new}+interp\_factor\cdot {\theta }_{t-1}$$

(8)

In the formula ${\theta }_{tl}$ is the parameter value after interpolation at step $t$; ${\theta }_{t-1}$ is the parameter value at step $t-1$; ${\theta }_t^{new}$ is the new parameter value calculated by the optimization algorithm without interpolation; and $interp\_factor$ denotes the interpolation factor, which controls the proportion of linear interpolation. When $interp\_factor$ = 0, it means no interpolation, i.e., all weights are on the current update value, implying that the parameter update is performed directly using the standard Adam update rule formula; when $interp\_factor$ = 1, it means full interpolation, i.e., all weights are on the last parameter value, implying that no update is performed; When 0 < $interp\_factor$ < 1 is partial interpolation, the size of the parameter set according to the interpolation factor indicates the proportional size of the linear combination of the current update value and the previous parameter value according to the weights, and this kind of interpolation can smooth out the update of the parameter to avoid drastic changes.

2.3. GSL Algorithm

An improved GSL algorithm based on Adam is proposed in this study, which combines a gradient trimming strategy with a periodic adjustment of the learning rate and a linear interpolation strategy in the following steps:

Initialization: set the initial learning rate $lr$, the period length of the learning rate change $lr\_period$, the amplitude of the learning rate change $lr\_factor$, the interpolation factor $interp\_factor$, the maximum number of paradigms $max\_grad\_norm$, the first-order momentum $m_t$ and the second-order momentum $v_t$ are all initialized to zero.

Gradient trimming step: calculate the L2 paradigm of the gradient, define a trimming factor ($clip\_coef$), i.e., preset a maximum paradigm $max\_grad\_norm$ as the numerator, take the total gradient paradigm as the denominator, calculate the size of the trimming factor, and if the trimming factor exceeds the threshold, scale the gradient in equal proportions, so as to keep it from exceeding the threshold size. The purpose is to prevent too large a gradient from causing a gradient explosion and to maintain the balance and stability of the parameter update.

Periodic learning rate adjustment step: the learning rate is periodically adjusted according to a sinusoidal function.

Linear interpolation parameter update step: by changing the value of the interpolation factor, the size of the ratio between the current update value and the last parameter value linearly combined by weights is adjusted to reduce drastic parameter fluctuations, thus improving the stability of the training process and preventing overfitting.

The GSL optimizer not only implements the periodic adjustment of the learning rate, combined with the idea of linear interpolation for parameter updating, but also introduces the concept of gradient trimming, which is performed by calculating the L2 paradigm of the gradient to set a threshold for trimming, preventing the gradient from exploding and increasing the stability of the algorithm. The specific algorithm is shown in Algorithm 1.

Algorithm 1: GSL Algorithm

1: Input: initial point ${\theta }_0$, the first-order momentum $m_t$, the second-order momentum $v_t$, first-moment decay ${\beta }_1$, second-moment decay ${\beta }_2$, avoid constants with zero denominator $\epsilon$. $2:\mathrm{Initialize}{m}_0=0$$\mathrm{and}{v}_0=0$$,lr$$,lr\_period$$,lr\_factor$$,interp\_factor$$,max\_grad\_norm$ $3:\mathrm{For}0\mathrm{to}lr\_period$ do $4:{\theta }_t=\nabla f_t({\theta }_{t-1})$ $5:m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)(\nabla {\theta }_t)$ $6:v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)(\nabla {\theta }_t{)}^2$ $7:\widehat{m_t}=m/(1-{\beta }_1^t)$ $8:\widehat{v_t}=v/(1-{\beta }_2^t)$ $9:||{\nabla }_{\theta }|{|}_2=\sqrt{{\displaystyle \sum _p{(||\nabla p|{|}_2)}^2}}$ $10:clip\_coef={\displaystyle \frac{max\_grad\_norm}{\left|\right|{\nabla }_{\theta }|{|}_2+\epsilon }}$ $11:\mathrm{If}clip\_coef$ < 1 $12:\nabla p=\nabla p\times clip\_coef$ $13:l{r}_{adjusted}=lr(1+lr\_factor\cdot \sin ({\displaystyle \frac{2\pi t}{lr\_period}}))$ $14:step\_size={\displaystyle \frac{l{r}_{adjusted}}{\sqrt{{\widehat{v}}_t}+\epsilon }}$ $15:{\theta }_t^{new}={\theta }_{t-1}-l{r}_{adjusted}{\displaystyle \frac{{\widehat{m}}_t}{\sqrt{{\widehat{v}}_t}+\epsilon }}$ $16:{\theta }_{tl}=(1-interp\_factor){\theta }_t^{new}+interp\_factor\cdot {\theta }_{t-1}$ End for $\mathrm{Return}{\theta }_{tl}$

3. Experimental

3.1. Experimental Dataset and Pre-Processing

In order to evaluate the effectiveness of the GSL algorithm, three public datasets were selected for image classification experiments. three datasets, the CIFAR10 dataset is a fixed-size color image dataset in which all images are 32 × 32 pixels. The dataset contains 10 different categories including airplanes, cars, birds, cats, deer, dogs, and trucks. These categories cover a wide range of real-world objects, allowing the model to learn a wide range of features. Gastroenterology [21] is a color medical image dataset containing eight classifications. Glaucoma [22,23,24] is a dataset of fundus images for glaucoma detection, containing two classifications: 1 (Glaucoma Present) and 0 (Glaucoma not Present), and 8621 images are taken for the experiment. The amount of data in the experimental dataset, and the division of the dataset are shown in

Table 1. Data volume of the experimental dataset, dataset partitioning.

Dataset	Number of Samples	Training Set	Test Set	Validation Set	Classes
CIFAR10	60,000	45,000	5000	10,000	10
Gastroenterology	1885	900	485	500	8
Glaucoma	8621	5000	1500	2121	2

.

The images are preprocessed using data augmentation operations in deep learning, including random rotation, center cropping, image enlargement, and normalization. The processed images are fed into deep learning network models (MobileNetV2, ShuffleNetV2) and experimented with different optimization algorithms (including SGD, Adagrad, Adam, NAdam, and GSL algorithm). Several experiments were conducted to comprehensively evaluate their performance. Taking one of the glaucoma presence images in the glaucoma dataset as an example, the experimental training diagram for the GSL algorithm glaucoma dataset is shown in Figure 1. The processed images are fed into the neural network model architecture for training, and the output of the obtained experimental results is used to evaluate the performance of the GSL algorithm. The experimental procedure flowchart is shown in Figure 2.

Taking one of the polyps images in the Gastroenterology dataset as an example, the image is enlarged by 1.25 times after random rotation and center cropping operations, and the enlarged image is normalized, normalize the mean and standard deviation of the R, G, and B channels in color images. The Gastroenterology image preprocessing process is shown in Figure 3.

3.2. Experimental Model Architecture

This study evaluated the performance of the GSL algorithm using MobileNetV2 and ShuffleNetV2 image classification model architectures. MobileNetV2 [25] is a lightweight convolutional neural network model proposed by Google that uses depth-separable convolution to decompose the standard convolution into deep and point-by-point convolutions, which significantly reduces the computational cost and the number of parameters, making it suitable for use on mobile and embedded devices that have limited computational resources and power consumption while maintaining a high level of accuracy in a variety of tasks. ShuffleNetV2 is a lightweight convolutional neural network architecture designed for efficient mobile computing that reduces the overall computational complexity by introducing fewer parameters and computational effort. This neural network improves the architecture of ShuffleNetV1 by introducing new unit structures. These units combine deep separable convolution and point-by-point convolution with channel shuffle operations to enhance information flow and maintain high accuracy and performance.

3.3. Experimental Evaluation Criteria

The performance of GSL algorithm was compared with other optimization algorithms from the following different aspects. It involves training the model and evaluating its accuracy, loss value on validation or test set. Observe its versatility ability by training on different datasets and different neural network architectures. The GSL algorithm is experimentally compared with other classical optimization algorithms such as SGD, Adam, and Adagrad algorithms. This allows for a clearer assessment of the strengths and weaknesses of the GSL algorithm relative to other algorithms. The following aspects are specifically considered:

(1)

Experimental setup:

Three typical datasets were selected, including CIFAR10, Gastroenterology and Glaucoma, and on each dataset this study used the same neural network architecture and hyperparameter settings to ensure a fair comparison.

(2)

Experimental comparison of different algorithms:

Compare the performance of GSL algorithm with classical optimization algorithms such as SGD, Adam, Adagrad, and NAdam algorithms on the target task. This study observe the performance of different algorithms within the same number of training iterations, including accuracy, loss value. Accuracy is a basic measure of the performance of a classification model, which indicates the number of correctly classified samples as a proportion of the total number of samples. The accuracy is calculated as shown in Equation (9):

$$\mathrm{Accuracy}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}}$$

(9)

In the formula TP denotes the number of true positives, correctly predicted as positive categories; TN denotes the number of true negatives, correctly predicted as negative categories; FP denotes the number of false positives, incorrectly predicted as positive categories; and FN denotes the number of false negative, incorrectly predicted as negative categories.

The loss function is used to measure the gap between the predicted and true values of the model. The loss value is calculated as shown in Equation (10):

$$H(p,q)=-{\displaystyle \sum _{i=1}^{n}p(x_i)}\log (q(x_i))$$

(10)

In the formula $H(p,q)$ represents the cross-entropy loss; $x_i$ represents different possible events that may occur; $p(x_i)$ is the expected output of the sample; $q(x_i)$ is the actual output of the sample, and $n$ represents the number of sample classes.

If the GSL algorithm achieves better performance within the same number of training iterations, then it can be considered as an effective improvement.

(3)

Versatility ability evaluation:

This study conduct experiments on different datasets (including CIFAR10, Gastroenterology, and Glaucoma) and different neural network architectures (two lightweight neural networks, MobileNetV2, and ShuffleNetV2) to observe the versatility ability of the GSL algorithm. By observing the performance of the GSL algorithm on different datasets and different neural network architectures, the versatility ability of the GSL algorithm as well as its strengths and weaknesses relative to other algorithms can be evaluated.

3.4. Experimental Results and Analysis

The appropriate choice of hyperparameters is very important for the performance of the GSL algorithm, and different hyperparameter values have different performance when training with different neural networks, and the same is true for the dataset. This study uses the open classical dataset CIFAR10 for experiments to illustrate this issue. The experiment was designed with three different combinations of the cycle length of the learning rate change $lr\_period$, the magnitude of the learning rate change $lr\_factor$ and the interpolation factor $interp\_factor$, and the experimental training was set to 20 epochs. The performance of the algorithm varies when the cycle length of the learning rate change $lr\_period$, the magnitude of the learning rate change $lr\_factor$ and the interpolation factor $interp\_factor$ are different. The results of experiments with different hyperparameters and different neural network architectures on the CIFAR10 dataset are shown in

Table 2. Experimental results with different hyperparameters and different neural network architectures on the CIFAR10 dataset.

	Neural Network	MobileNetV2		ShuffleNetV2
Hyperparameter 1
		Accuracy	Loss	Accuracy	Loss
1500, 0.5, 0.2		0.6986	0.9903	0.6533	1.4250
1500, 0.8, 0.2		0.7111	0.9481	0.6775	0.9875
2000, 0.8, 0.1		0.7184	0.8641	0.6902	1.0810

¹ The order of hyperparameter settings is $lr\_period$, $lr\_factor$,$interp\_factor$, Example 1500, 0.5, 0.2 means that $lr\_period$ is set to 1500, $lr\_factor$ is set to 0.5, and $interp\_factor$ is set to 0.2.

. The table shows the performance of the GSL algorithm for different combinations of hyperparameter settings on MobileNetV2 neural network and ShuffleNetV2 neural network under the CIFAR10 dataset, respectively.

According to the observations in Table 2, different hyperparameter settings have different experimental results when training on the same neural network is used, at which point the GSL algorithm performs differently. The same is true when training on different neural networks are used. Thanks to the hyperparameter settings, the best performance can be obtained by adjusting the values of the hyperparameters when facing different situations in order to achieve the desired experimental results.

In performing the following training, the hyperparameters were set to a set of data at 2000-0.8-0.1 when the GSL algorithm performed hyperparameter experiments on the CIFAR10 dataset.

Table 3. Experimental results of different algorithms and neural network architectures on the CIFAR10 dataset on the test set.

	Neural Network	MobileNetV2		ShuffleNetV2
Algorithm
		Accuracy	Loss	Accuracy	Loss
SGD		0.4832	1.4660	0.4623	1.5750
Adagrad		0.2987	1.9500	0.3398	1.9090
Adam		0.6849	1.2200	0.6419	1.4320
NAdam		0.6985	1.6850	0.6494	1.5040
GSL		0.7442	0.9232	0.6965	1.2350

shows the training results of the five algorithms on the CIFAR10 dataset. The experimental results of different algorithms and different neural network architectures on the CIFAR10 dataset are shown in Table 3, and the cases where the GSL algorithm performs well in the table are denoted in bold font.

The experimental results in Table 3 show that the GSL algorithm performs better when trained on the MobileNetV2 neural network using the CIFAR10 dataset, and its accuracy outperforms the remaining four algorithms. The algorithm improved the accuracy by 5.93% compared to the Adam algorithm and 4.57% compared the NAdam algorithm which is improvement of Adam’s algorithm; When trained on the ShuffleNetV2 neural network, the accuracy of the GSL algorithm improved by 5.46% compared to the accuracy of the Adam algorithm and by 4.74% compared to the NAdam algorithm, which was improvement of the Adam algorithm. The GSL algorithm has a better performance when facing different neural network model training tasks, which also reflects that the GSL algorithm has a good versatility ability.

Table 4. Experimental results of different algorithms and different neural network architectures on Gastroenterology dataset on the test set.

	Neural Network	MobileNetV2		ShuffleNetV2
Algorithm
		Accuracy	Loss	Accuracy	Loss
SGD		0.6000	1.0800	0.5560	2.0190
Adagrad		0.6740	1.0120	0.6620	1.0230
Adam		0.6980	1.0460	0.7040	0.9261
NAdam		0.7380	0.9189	0.7340	1.0400
GSL		0.7800	1.0300	0.7520	1.0410

shows the training results of the five algorithms on the Gastroenterology dataset. The experimental results of different algorithms and different neural network architectures on the Gastroenterology dataset are shown in Table 4, and the cases where the GSL algorithm performs well in the table are denoted by bold font.

Figure 4 and Figure 5 show the training results of the five algorithms for different neural network models on the Gastroenterology dataset, respectively.

The results in Figure 4 show that on the Gastroenterology dataset, the GSL algorithm is slightly more accurate than the other four algorithms as the number of training iterations increases, and the algorithm converges faster compared to the other algorithms. The results in Figure 5 show that although the accuracy of the GSL algorithm is slightly lower than that of Adam’s algorithm at the beginning of training, the GSL algorithm outperforms the remaining four algorithms as the training progresses. This advantage can be attributed to the gradient cropping strategy employed by the GSL optimizer. During the training process, the GSL algorithm restricts the size of the total gradient paradigm through the clipping coefficients to ensure the gradient update smoothness. By adjusting the change in the size of the parameter gradient, the GSL algorithm can effectively avoid drastic fluctuations in the gradient, thus improving the stability of model training.

Table 5. Experimental results of different algorithms and different neural network architectures on Glaucoma dataset on the test set.

	Neural Network	MobileNetV2		ShuffleNetV2
Algorithm
		Accuracy	Loss	Accuracy	Loss
SGD		0.7841	0.4751	0.6209	0.6562
Adagrad		0.7765	0.7450	0.7529	0.7160
Adam		0.8152	0.4510	0.8190	0.6830
Nadam		0.8147	0.4713	0.8223	0.8146
GSL		0.8345	0.7161	0.8312	0.5959

shows the training results of the five algorithms on the Glaucoma dataset. The experimental results of different algorithms and different neural network architectures on the Glaucoma dataset are shown in Table 5, and the cases where the GSL algorithm performs well in the table are denoted by bold font.

Figure 6 and Figure 7 show the training results of the five algorithms for different neural network models on the Glaucoma dataset, respectively.

According to the results in Figure 6, the GSL algorithm has better performance throughout the training process as the number of training rounds increases on the MobileNetV2 network. The GSL algorithm improves by 1.93% and 1.98% compared to the Adam algorithm and the NAdam algorithm which is improved based on the Adam algorithm. The results in Figure 7 show that the GSL algorithm improves the accuracy of the GSL algorithm by 1.22% compared to the Adam algorithm on the ShuffleNetV2 network. This advantage can be attributed to the periodic control of changing the learning rate strategy, where the learning rate determines the step size of each parameter update, the value of the learning rate is constantly changed during training, and linear interpolation is utilized to adjust the magnitude of the ratio of the current update value to the last parameter value linearly combined by weights to reduce drastic parameter fluctuations.

4. Discussion

Although research has shown significant progress in the performance of the GSL algorithm on these datasets and neural networks, there are also some limitations. One limitation is that when parameters have different scales and sensitivities, further research is needed to change the strategy of gradient clipping to achieve optimal performance. In addition, the performance of the GSL algorithm may be affected by the characteristics of the dataset or neural network, and additional hyperparameter adjustments may be required to achieve optimal performance.

5. Conclusions

In order to improve the problems and deficiencies of the Adam optimization algorithm, a new optimization algorithm, GSL algorithm, is proposed in this study. Experimental validation by combining different public datasets and neural networks confirms that the GSL algorithm has a better performance compared to traditional optimization methods. GSL algorithm provides a more flexible and efficient training approach by combining gradient cropping, periodic adjustment of the learning rate, and linear interpolation to adjust the weight proportions of the parameter update values. The results show that the GSL algorithm not only improves the performance of the model on the standard test set, but also provides a feasible path for performance improvement and stability enhancement of deep learning models in practical applications, such as helping doctors to diagnose whether a patient’s intestines are diseased as well as the type of the disease and to identify glaucomatous lesions efficiently, thus helping doctors to recognize whether the patient has glaucomatous features or not. In the future, we will investigate how the GSL algorithm can achieve optimal results when facing different datasets and models.