Optimization Strategies Applied to Deep Learning Models for Image Steganalysis: Application of Pruning, Quantization and Weight Clustering

Abstract

Image steganalysis methods aim at detecting whether there exist hidden messages in images. Deep learning (DL) models have been proposed to enhance steganography detection. These models occupy a large amount of memory and, for this reason, should be optimized when the scenario involves resource-limited devices and systems. This work addresses different deep learning model optimization strategies, namely model pruning, quantization and weight clustering, applied to a deep learning model that presents competitive accuracy results in image steganalysis and belongs to the family of DL models with smaller memory requirements. The results show that the use of optimization schemes can lead to similar or even better accuracy compared to the original model (without the use of optimization schemes), while requiring less memory to store the model. Different scenarios are simulated for each optimization technique, and, finally, quantization is combined with pruning. For dynamic range quantization (DRQ), we achieve models that can save approximately 72% of storage. For FP16 quantization, we obtain better accuracy results and a model with approximately 50% less memory consumption. By applying weight clustering, we also achieve compressed models that can save more than 72% of storage space and lead to better accuracy for some scenarios. Using the combination of pruning and quantization, smaller models in terms of memory requirements are obtained.

1. Introduction

Over the years, people have obtained greater access to communication devices and networks, resulting in more information sent over the internet, such as text messages, audio, images, and videos. Since these types of information can be changed and eavesdropped upon, encryption and data hiding techniques, such as steganography and watermarking, can be applied to achieve both malicious and good objectives [1].

Steganography is a subarea of data hiding with the objective of sending an occult message through a communication channel [2]. Steganalysis is a counter mechanism that focuses on discovering whether there is the presence of hidden information, thus counteracting steganography [3].

In the digital era, steganography methods have been used for malicious objectives, e.g., to leak information [4,5], for terrorism purposes [6,7], for cyber warfare [8], and for malware propagation [9,10,11]. As a countermeasure, steganalysis schemes have been employed to track criminal activities on the internet [12,13], to combat cyber warfare [14], in the detection of malware [15,16], and in forensics [13,17,18].

Steganography methods can be classified via the domain used to embed secret information. While spatial domain techniques consist of hiding information by changing the image pixel values, transform domain methods are based on embedding messages in the transform coefficients of images [19]. Among the steganography algorithms adopted to evaluate the performance of steganalysis in the spatial domain are Highly Undetectable steGO (HUGO) [20], wavelet-obtained weights (WOW) [21], spatial universal wavelet relative (S-UNIWARD) [22], and high-pass, low-pass, and low-pass (HILL) [23].

Steganalysis methods can be classified into specific steganalysis, which means that the technique is responsible for discovering whether there is information concealed in a specific steganography method, or universal/black-box/blind steganalysis, which can detect the existence of information embedded by any steganographic scheme, mostly through computational intelligence techniques [24]. These methods are used to discover if there is hidden communication in the most popular types of media [25], such as audio [26], video [27], text [28], and images [29].

In recent years, deep learning models for image steganalysis have been developed to improve the detection of hidden messages in images [30,31,32]. They have been proposed as an alternative to traditional methods, seeking to achieve better performance in terms of accuracy in detecting the presence of steganography. Different scenarios have been evaluated by changing the payload capacity of the chosen steganographic algorithm and the arrangement of the test, validation, and train image sets [33]. Reinel et al. [34] proposed a deep learning network applied to image steganalysis called GBRAS-Net, which achieves competitive results in terms of accuracy and belongs to the group of models that require less memory when compared to previously proposed models.

Deep learning models can have a structure with millions of parameters and many activation functions to be stored, resulting in concerns regarding the implementation of these models in resource-limited devices. To overcome this issue, efficient and smaller networks have been proposed recently by applying optimization techniques such as model pruning, quantization, and weight clustering, without a significant reduction in terms of accuracy [35,36,37].

This work aims to assess the effects of deep learning model compression strategies, namely quantization, pruning, and weight clustering, in a deep learning model applied to image steganalysis (GBRAS-Net). An evaluation in terms of accuracy and memory requirements is carried out for a wide variety of compression strategies.

The main scientific contribution of this research lies in the implementation of optimization strategies in a deep learning model for image steganalysis, specifically through the application of pruning, quantization, and weight clustering. These techniques not only significantly reduce the memory requirements of the model, making it more suitable for devices with limited resources, but, in some scenarios, also maintain or even improve the accuracy compared to the original model. Furthermore, this innovative approach combines model compression methods with the critical field of information security, offering an optimized solution for efficient steganalysis systems. The relevance of this research extends to areas such as cybersecurity, digital forensics, and data protection, providing valuable and original insights into how to apply optimization techniques in deep neural networks for image steganalysis.

The remainder of this paper is organized as follows. In Section 2, we address the model optimization techniques. In Section 3 and Section 4, we present the related works and methodology, respectively. In Section 5, we present the simulation results for the original and compressed models applied to S-UNIWARD and WOW. In Section 6, we present the conclusions and future work.

2. Model Optimization

2.1. Connection Pruning

Pruning is an optimization technique that aims to reduce the size of a model by removing connections, neurons, or parameters that are not relevant. Connection pruning consists of removing weights whose values are zero or close to zero to reduce the memory requirements of the entire network. The pruned model can be retrained and can even achieve superior accuracy compared to the original model [38]. Figure 1 shows a basic example of a model before pruning, i.e., with all neurons connected, and the resulting pruned model with fewer connections [39].

Magnitude-based pruning is one of the most common pruning techniques and has achieved great results in terms of reducing the memory requirements of machine learning models without compromising the accuracy of the scheme [38,40]. This technique consists of eliminating weights whose values are lower than a chosen threshold for a trained model. Pruning can also be implemented by defining the pruning ratio (PR) of the model, which refers to the percentage of weights to be zeroed in an entire network or the specific layers to be pruned by applying a pruning schedule [41].

2.2. Quantization

Quantization is a model optimization technique that has the objective of reducing the size of a DL model by converting weight values to a lower-precision bit representation, which is usually from 32 bits to 16 bits or 8 bits. Furthermore, it is possible to represent floating-point numbers in binary [42]. Quantization can be divided into two main categories: post-training quantization (PTQ) [43] and quantization-aware training (QAT) [44].

The PTQ technique is applied to a trained model with default values of representation—commonly 32-bit floating-point values. These values can be quantized, e.g., from 32-bit floating points (FP32) to 16-bit floating points (FP16) or 8-bit integers (INT8), without the need to retrain the model [45].

Figure 2 shows the FP32, FP16, FP8, and INT8 value representations on a computer system. It is observed that different representations use different numbers of bits. The precision is related to the number of bits used to represent a value. The more precise the representation is, the greater the amount of memory that has to be allocated to store the model. The floating-point representation consists of three parts: the sign, exponent, and mantissa. The sign, represented in red, is the most significant bit and can either represent a positive or negative value (0 means positive and 1 means negative). The set of binary digits represented in green in the image is the exponent; it is responsible for defining the power to which the base 2 is raised. Occasionally, the exponent is called the signed or unbiased exponent. The other bits, represented in blue, are the mantissa and define the fractional part of the number [46]. For 8-bit integer representations, there is no exponent and the values can be represented from −128 to 127. Figure 3 shows dynamic range quantization (DRQ) from FP32 to INT8 representations. The values after conversion are less precise, but this results in less information to be stored in a system.

The QAT strategy requires more computational effort since it is used while training the model [47].

2.3. Weight Clustering

Weight clustering or weight sharing is an optimization technique that consists of determining a representative value for a cluster or group of weights, thus decreasing the number of different values to be stored in the network. It means that many different neurons in a network share the same weight [48].

Figure 4 illustrates a basic weight clustering example with sixteen weights being clustered in four groups represented by a color. In this case, four reference values correspond to the mean of the values of each cluster, and each weight is quantized to one of the reference values mentioned above. The green cluster, for example, has five elements with distinct values. The mean of these elements is 0.15, which is the reference value for this cluster. After applying weight sharing, all elements represented in green now have 0.15 as the weight value. For this specific cluster, the distinct parameters are replaced with only one FP32 value and the reference index, which is an integer. Considering all clusters in the example, instead of having sixteen different FP32 values, the network after weight clustering replaces these unique values with four FP32 values, thus optimizing the memory storage.

The number of clusters chosen affects directly the performance and the memory requirements of a model. An important parameter to be considered is the centroid initialization. The strategy of weight clustering has been assessed for deep learning problems in recent works [49,50,51,52].

3. GBRAS-Net

GBRAS-Net is a steganalysis scheme based on convolutional neural networks (CNNs) and was proposed by Reinel et al. [34] in 2021. This deep learning scheme receives an image of 256 × 256 pixels. The model has a preprocessing layer, which consists of a 2D convolutional layer with 30 filters. This step has 780 non-trainable parameters with preset weights and a custom 3Tanh activation function. The feature extraction stage is composed of various layers: four 2D depthwise convolutional, four 2D separable convolutional, and nine 2D convolutional layers. Furthermore, the model uses batch normalization layers to improve the performance and avoid overfitting, followed by an average pooling layer to reduce the dimensionality. Moreover, the model uses skip connections. At the end of the model’s process, global average pooling is carried out, resulting in two values. For the classification stage, these values are input to a softmax activation function, which predicts whether the element is a cover or stego image.

GBRAS-Net was first experimented with in Python 3.8.1 and was developed using the TensorFlow framework version 2.2.0. The authors provided the code on GitHub. The metric used for the evaluation of the method was the accuracy, and the model was compared with seven other steganalytic methods. At the time, the network had achieved the best accuracy results. The proposed method was implemented and assessed with 10,000 pairs of cover and stego images of BOSSBASE 1.01 for the steganographic algorithm S-UNIWARD, with 73.6% accuracy for 0.2 bpp and 87.1% for 0.4 bpp, and for WOW, with 80.3% accuracy for 0.2 bpp and 89.8% for 0.4 bpp. The data distribution in the experiment was as follows: 4000 pairs of images for training, 1000 pairs for the validation set, and 5000 pairs for testing. In addition, the authors assessed the performance of the proposed model against spatial domain steganography algorithms such as HUGO, HILL, and MiPOD. In terms of model size, GBRAS-Net has one of the smallest storage requirements compared to other steganalysis deep learning models, with only 0.65 MB of occupied memory.

4. Methodology

We applied quantization, pruning, and weight clustering techniques for the optimization of GBRAS-Net and compared the different scenarios in terms of accuracy and occupied memory. We had to update and adjust some parts of the original GBRAS-Net model code to replicate the model and use updated optimization toolkits.

4.1. Computational Requirements

The tests were run on Google Colab with Python 3.10, Tensorflow 2.17.1, and Keras 3.5.0. The machine used was a Ubuntu 22.04.3 LTS, with a Intel(R) Xeon(R) CPU @ 2.20 GHz and RAM of 51 GB (Intel, Santa Clara, CA, USA). Two different GPUs were used, a Tesla T4 with RAM of 15 GB (Tesla Inc., Austin, TX, USA) and an NVIDIA A100-SXM4 with RAM of 40 GB (NVIDIA, Santa Clara, CA, USA).

4.2. Model Compression Framework

To implement quantization, model pruning, and weight clustering, we used the Tensorflow Model Optimization Toolkit (TFMOT) 0.8.0. The models were converted from the keras or hdf5 file format to tflite file formats for quantization.

4.3. Database

We performed our tests on the same database provided by Reinel et al. [34] for comparison purposes. The dataset contains BOSSBASE 1.01 with 10,000 pairs of grayscale images of 256 × 256.

5. Results

This section presents the results achieved in the simulations. The tables presented in this section summarize all implemented strategies in this work and point out the performance of the model in terms of accuracy and the model’s memory consumption for S-UNIWARD 0.4 bpp, S-UNIWARD 0.2 bpp, WOW 0.4 bpp, and WOW 0.2 bpp.

5.1. Dynamic Range Quantization (8 Bits)

This type of quantization means that the weight values will be converted from FP32 to INT8. The activation values and the output and input tensors are stored in FP32. For this implementation, we used the default optimization setting for the TensorFlow lite function.

Table 1. Dynamic range quantization applied to GBRAS-Net for S-UNIWARD at 0.4 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	87.16%	0.65 MB
Dynamic Range Quantization	84.80%	0.16 MB

and

Table 2. Dynamic range quantization applied to GBRAS-Net for S-UNIWARD at 0.2 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	73.60%	0.65 MB
Dynamic Range Quantization	70.88%	0.16 MB

show the results of this strategy applied to the S-UNIWARD steganography algorithm. The accuracy loss was 2.36% for S-UNIWARD at 0.4 bpp and 2.72% for S-UNIWARD at 0.2 bpp. The occupied memory achieved was 0.16 MB, corresponding to approximately one quarter of that of the original model.

Table 3. Dynamic range quantization applied to GBRAS-Net for WOW at 0.4 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	89.80%	0.65 MB
Dynamic Range Quantization	89.05%	0.16 MB

and

Table 4. Dynamic range quantization applied to GBRAS-Net for WOW at 0.2 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	80.30%	0.65 MB
Dynamic Range Quantization	79.32%	0.16 MB

show the results of the simulations for the WOW algorithm. This strategy applied to the original GBRAS-Net achieved less than a 1% accuracy decrease for the WOW dataset with either 0.4 bpp or 0.2 bpp, reaching 89.05% and 79.32% accuracy, respectively, while exhibiting approximately four times smaller memory requirements than the original model.

5.2. FP16 Quantization

The FP16 strategy is a quantization technique that applies a conversion from FP32 to FP16. When compared with 8-bit dynamic range quantization, more bits have to be stored after quantization. The weights are then represented in FP16.

Table 5. FP16 quantization applied to GBRAS-Net for S-UNIWARD at 0.4 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net (Original)	87.16%	0.65 MB
FP16 Quantization	87.29%	0.31 MB

,

Table 6. FP16 quantization applied to GBRAS-Net for S-UNIWARD at 0.2 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	73.60%	0.65 MB
FP16 Quantization	74.35%	0.31 MB

,

Table 7. FP16 quantization applied to GBRAS-Net for WOW at 0.4 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	89.80%	0.65 MB
FP16 Quantization	89.84%	0.31 MB

and

Table 8. FP16 quantization applied to GBRAS-Net for WOW at 0.2 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	80.30%	0.65 MB
FP16 Quantization	80.35%	0.31 MB

show the FP16 results for S-UNIWARD and WOW for both 0.2 and 0.4 bpp. The GBRAS-Net results are also presented in the aforementioned tables for comparison purposes.

Table 5 shows the results of the simulations for S-UNIWARD at 0.4 bpp. The quantized weights model achieved 87.29% accuracy, which represents an increase of 0.13% over the original model. The results for S-UNIWARD at 0.2 bpp, presented in Table 6, show that the accuracy increased by 0.75% when compared with GBRAS-Net, with 74.35%. The results for WOW at 0.4 and 0.2 bpp are shown in Table 7 and Table 8, respectively. The accuracy of the model increased slightly, by 0.04% for WOW at 0.4 bpp and 0.05% for WOW at 0.2 bpp, when compared with the original model.

Although the memory requirements are higher than those obtained with dynamic range quantization, they are reduced by approximately half compared to the original GBRAS-Net, with 0.31 MB.

5.3. Weight Clustering

To implement weight clustering, the cluster weight function was used from the TensorFlow optimization toolkit. All layers were clustered, except the first layer of GBRAS-Net. The maximum number of clusters was 32. We assessed the same optimization strategy with four different centroid initializations: density-based [53], K-means++ [54,55], linear, and random.

Table 9. Weight clustering applied to GBRAS-Net for S-UNIWARD at 0.4 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	87.16%	0.65 MB
Weight Clustering (Density-Based)	87.19%	0.18 MB
Weight Clustering (K-Means++)	86.83%	0.18 MB
Weight Clustering (Linear)	84.28%	0.15 MB
Weight Clustering (Random)	82.03%	0.14 MB

,

Table 10. Weight clustering applied to GBRAS-Net for S-UNIWARD at 0.2 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	73.60%	0.65 MB
Weight Clustering (Density-Based)	69.45%	0.18 MB
Weight Clustering (K-Means++)	67.00%	0.18 MB
Weight Clustering (Linear)	64.85%	0.15 MB
Weight Clustering (Random)	68.12%	0.15 MB

,

Table 11. Weight clustering applied to GBRAS-Net for WOW at 0.4 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	89.80%	0.65 MB
Weight Clustering (Density-Based)	87.34%	0.18 MB
Weight Clustering (K-Means++)	86.83%	0.18 MB
Weight Clustering (Linear)	85.96%	0.15 MB
Weight Clustering (Random)	89.20%	0.13 MB

and

Table 12. Weight clustering applied to GBRAS-Net for WOW at 0.2 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	80.30%	0.65 MB
Weight Clustering (Density-Based)	75.40%	0.18 MB
Weight Clustering (K-Means++)	78.05%	0.17 MB
Weight Clustering (Linear)	76.83%	0.15 MB
Weight Clustering (Random)	75.77%	0.14 MB

show all results for S-UNIWARD and WOW, for both 0.2 and 0.4 bpp.

Table 9 and Table 10 show the results for the S-UNIWARD algorithm at 0.4 bpp and 0.2 bpp, respectively. For S-UNIWARD at 0.4 bpp, when choosing the centroid initialization to be density-based, the model achieved a slight increase of 0.03% in terms of accuracy when compared to GBRAS-Net, while it could save approximately 73% of memory storage. When setting the centroid initialization as K-means++, the model experienced a slight accuracy decrease of 0.33%, with the same storage requirements as for the density-based centroid initialization. Table 10 shows that S-UNIWARD at 0.2 bpp experienced an accuracy decrease of more than 4% for all scenarios.

Table 11 and Table 12 show the results for the WOW algorithm at 0.4 bpp and 0.2 bpp, respectively. For the WOW algorithm at 0.4 bpp, when setting the centroid initialization to random, the model obtained accuracy of 89.20%, resulting in a decrease of 0.60% when compared with the original model. This configuration achieved memory storage of 0.13 MB, which was one fifth of that of GBRAS-Net. At 0.2 bpp, the best results in terms of accuracy were obtained when setting the centroid initialization as K-means++, with 78.05%, resulting in a decrease of 2.25% in accuracy and a memory requirement of 0.17 MB.

5.4. Low-Magnitude Pruning

To implement pruning, we used the prune low-magnitude function from the TensorFlow toolkit. We applied pruning for all layers and used 12 epochs for pruning and retraining. We defined the pruning to start from the first epoch with a 10% pruning ratio for all simulations. We assessed the results for eight different pruning ratios across the neurons of the network. The final pruning ratios chosen for the simulations were 20%, 30%, 40%, 50%, 60%, 70%, 80%, and 90%. The results show that the more sparse the network, the lower the occupied memory of the optimized model. The results of this strategy applied to GBRAS-Net for both S-UNIWARD at 0.4 bpp and 0.2 bpp and WOW at 0.4 bpp and 0.2 bpp are shown in

Table 13. Low-magnitude pruning applied to GBRAS-Net for S-UNIWARD at 0.4 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	87.16%	0.65 MB
Pruning ratio: 20%	86.40%	0.55 MB
Pruning ratio: 30%	86.12%	0.55 MB
Pruning ratio: 40%	86.98%	0.45 MB
Pruning ratio: 50%	87.12%	0.39 MB
Pruning ratio: 60%	85.66%	0.34 MB
Pruning ratio: 70%	85.03%	0.28 MB
Pruning ratio: 80%	84.46%	0.22 MB
Pruning ratio: 90%	82.64%	0.16 MB

,

Table 14. Low-magnitude pruning applied to GBRAS-Net for S-UNIWARD at 0.2 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	73.60%	0.65 MB
Pruning ratio: 20%	72.75%	0.54 MB
Pruning ratio: 30%	73.82%	0.50 MB
Pruning ratio: 40%	72.54%	0.45 MB
Pruning ratio: 50%	73.58%	0.39 MB
Pruning ratio: 60%	72.97%	0.34 MB
Pruning ratio: 70%	74.15%	0.28 MB
Pruning ratio: 80%	71.48%	0.22 MB
Pruning ratio: 90%	70.37%	0.16 MB

,

Table 15. Low-magnitude pruning applied to GBRAS-Net for WOW at 0.4 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	89.80%	0.65 MB
Pruning ratio: 20%	87.98%	0.54 MB
Pruning ratio: 30%	89.24%	0.50 MB
Pruning ratio: 40%	88.77%	0.45 MB
Pruning ratio: 50%	89.02%	0.39 MB
Pruning ratio: 60%	88.91%	0.34 MB
Pruning ratio: 70%	89.27%	0.28 MB
Pruning ratio: 80%	88.66%	0.22 MB
Pruning ratio: 90%	85.15%	0.16 MB

and

Table 16. Low-magnitude pruning applied to GBRAS-Net for WOW at 0.2 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	80.30%	0.65 MB
Pruning ratio: 20%	79.64%	0.54 MB
Pruning ratio: 30%	79.83%	0.50 MB
Pruning ratio: 40%	79.93%	0.45 MB
Pruning ratio: 50%	79.43%	0.39 MB
Pruning ratio: 60%	80.46%	0.34 MB
Pruning ratio: 70%	79.07%	0.28 MB
Pruning ratio: 80%	79.73%	0.22 MB
Pruning ratio: 90%	75.73%	0.16 MB

, respectively.

Table 13 shows the results for the S-UNIWARD algorithm at 0.4 bpp. None of the simulations resulted in accuracy better than that of the original GBRAS-Net. The best results in terms of accuracy were achieved by defining the pruning ratio as 40% and 50%, with a decrease of 0.18% and 0.04%, respectively. The respective model sizes were greater than the ones seen in the 8-bit dynamic range quantization, FP16 quantization, and weight clustering simulations, with 0.45 MB and 0.39 MB, respectively. At a 90% pruning ratio, accuracy of 82.64% was achieved, with corresponding memory storage of 0.16 MB. The results of the simulations for S-UNIWARD at 0.2 bpp are presented in Table 14. The best results were obtained by setting the pruning ratio as 50% or 70%, with 73.58% and 74.15% accuracy, respectively. In particular, by setting the pruning ratio to 70%, an increase of 0.55% in accuracy was obtained over the original model, with memory storage of 0.28 MB.

Table 15 shows the results of the simulations with this strategy applied to WOW at 0.4 bpp. The best result was achieved by setting the pruning ratio at 70%, with accuracy of 89.27%, while obtaining a model size of 0.28 MB. The results for this strategy applied to WOW at 0.2 bpp are presented in Table 16. By defining the pruning ratio as 60%, it was possible to obtain slightly better accuracy compared to GBRAS-Net. In particular, we observed an increase of 0.16% in accuracy when compared with GBRAS-Net, while the model was compressed by approximately two times.

5.5. Low-Magnitude Pruning + Quantization

We combined both pruning and quantization methods. We chose to combine the two better results in terms of accuracy and occupied memory from the previous pruning simulations with both dynamic range quantization and FP16 quantization.

Table 17. Low-magnitude pruning combined with quantization applied to GBRAS-Net for S-UNIWARD at 0.4 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	87.16%	0.65 MB
DRQ + PR: 40%	80.54%	0.14 MB
DRQ + PR: 50%	84.08%	0.13 MB
FP16 + PR: 40%	86.08%	0.24 MB
FP16 + PR: 50%	87.13%	0.21 MB

,

Table 18. Low-magnitude pruning combined with quantization applied to GBRAS-Net for S-UNIWARD at 0.2 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	73.60%	0.65 MB
DRQ + PR: 50%	71.85%	0.13 MB
DRQ + PR: 70%	67.87%	0.10 MB
FP16 + PR: 50%	74.19%	0.21 MB
FP16 + PR: 70%	75.19%	0.16 MB

,

Table 19. Low-magnitude pruning combined with quantization applied to GBRAS-Net for WOW at 0.4 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	89.80%	0.65 MB
DRQ + PR: 60%	87.08%	0.11 MB
DRQ + PR: 70%	88.25%	0.10 MB
FP16 + PR: 60%	87.44%	0.19 MB
FP16 + PR: 70%	89.49%	0.16 MB

and

Table 20. Low-magnitude pruning combined with quantization applied to GBRAS-Net for WOW at 0.2 bpp.

Strategy	Accuracy	Memory Requirement
GBRAS-Net	80.30%	0.65 MB
DRQ + PR: 60%	78.73%	0.11 MB
DRQ + PR: 80%	78.12%	0.08 MB
FP16 + PR: 60%	79.55%	0.19 MB
FP16 + PR: 80%	79.12%	0.13 MB

show the results of this strategy for S-UNIWARD and WOW, for both 0.2 and 0.4 bpp.

Table 17 presents the results for S-UNIWARD at 0.4 bpp. When combining pruning with a 50% pruning ratio and FP16 quantization, the model achieved accuracy of 87.13%, only 0.03% below that of the original model, while occupying three times less memory. Table 18 shows the results for S-UNIWARD at 0.2 bpp. Two different scenarios obtained better accuracy than the original model. At a 50% pruning ratio combined with FP16 quantization, the model exhibited 0.59% higher accuracy, achieving 74.19%, with a model size of 0.21 MB. A memory requirement that was about four times smaller than that of the original model was obtained, while achieving an increase of 1.59% in accuracy, when implementing a 70% pruning ratio, with the accuracy reaching 75.19%.

Table 19 presents the results when combining quantization and pruning for the WOW algorithm at 0.4 bpp. By setting the pruning ratio at 70%, combined with dynamic range quantization, accuracy of 88.25% was achieved, resulting in a decrease of 1.55% when compared with the original GBRAS-Net model. The corresponding compressed model could save approximately 85% of memory, with 0.10 MB. By combining the same pruning configuration with FP16 quantization, accuracy of 89.49% and a 0.16 MB memory requirement for the model’s storage were achieved.

Table 20 presents the results for WOW at 0.2 bpp. By setting the pruning ratio at 80% and combining it with dynamic range quantization, a less memory-demanding model was obtained, with a requirement of 0.08 MB, i.e., an approximately 88% smaller memory storage requirement when compared with the original model. Accuracy of 78.12% was obtained, resulting in a decrease of 2.18% when compared to the original GBRAS-Net.

5.6. Results Analysis

In this subsection, we compare the results obtained for each optimization strategy employed. Figure 5, Figure 6, Figure 7 and Figure 8 present bar graphs with the results obtained in terms of the accuracy and memory requirements of the optimized model for a selected algorithm at a fixed embedding rate. The accuracy is represented by the blue bars and the memory requirements by orange bars. The bars are arranged in descending order regarding the memory requirements. The left y-axis shows the accuracy value of the model, while the right y-axis shows the memory requirements in MB.

Figure 5 shows all results regarding the accuracy and memory requirements of the optimization strategies employed for S-UNIWARD at 0.4 bpp. The best accuracy results were achieved by applying FP16 quantization, FP16 quantization combined with a pruning ratio of 50%, and weight clustering with both density-based and K-means++ centroid initialization. The memory space required by the optimized model when applying FP16 quantization was approximately twice that required when applying each of the other three aforementioned techniques. Considering the relation between accuracy and occupied memory savings, the best results were obtained by applying weight clustering with both K-means++ and density-based centroid initialization and by combining FP16 quantization with a 50% pruning ratio. The weight clustering with density-based centroid initialization proved to be the best choice for this scenario, with a slight increase in accuracy of 0.03% and with approximately 72% less memory required when compared with the original model.

Figure 6 presents the achieved accuracy values and memory requirements of the optimized models when the scenario involved S-UNIWARD at 0.2 bpp. The accuracy of the original model (without the use of optimization strategies) was 73.60%. The results for five different optimized models showed accuracy values that were slightly better when compared with GBRAS-Net, along with savings in the memory required by the model. Accuracy results above 74% were obtained by applying FP16 quantization, a pruning scheme with a 70% ratio, and FP16 combined with a 50% pruning ratio. Considering the relation between the accuracy and the memory requirements of the model, the best results were obtained by applying FP16 quantization combined with pruning of 70%. The accuracy increased by 1.59% when compared to the original model. This strategy resulted in one of the five least space-demanding optimized models when the scenario involved S-UNIWARD at 0.2 bpp, with a size of 0.16 MB, which represents a saving of approximately 75% when compared with the original model.

Figure 7 shows all results regarding the accuracy and memory requirements of the optimization strategies employed for WOW at 0.4 bpp. The accuracy of the original model for this scenario was 89.80%. Six different scenarios presented accuracy values above 89.00%. There was no loss of accuracy when FP16 quantization was applied. The memory saving for this strategy was approximately 52%. The optimized models with dynamic range quantization (0.16 MB) and weight clustering with random centroid initialization (0.13 MB) presented accuracy values of 89.05% and 89.20%, respectively, which represent accuracy losses below 1% when compared with that of GBRAS-Net, while they exhibited smaller memory requirements compared with the optimized model when using FP16 quantization.

Figure 8 presents the achieved accuracy values and memory requirements of the optimized models for WOW at 0.2 bpp. In terms of accuracy, the best values obtained by an optimized model were achieved when applying the pruning ratio of 60%, followed by FP16 quantization. When the scenario involves no loss of accuracy in the model, the aforementioned techniques are recommended. To achieve greater compression, it is recommended to use dynamic range quantization (0.16 MB), which presents an accuracy decrease of 0.98%, or the combination of FP16 quantization with a pruning scheme with a ratio of 80% (0.13 MB), which presents an accuracy decrease of 1.18% when compared with GBRAS-Net.

Figure 9 presents the results obtained in terms of the memory requirements of all optimization strategies employed in this work. The storage requirements of the original model can be reduced from approximately 17% to, at most, 87% when using the considered strategies. The twelve different employed optimization schemes resulted in models with memory requirements that were one quarter of those of the original model, including dynamic range quantization, pruning, weight clustering, and the combination of both dynamic range quantization and FP16 quantization with pruning.

6. Conclusions

This work addressed a variety of deep learning model optimization techniques, namely model pruning, quantization, and weight clustering, applied to a deep leaning method for image steganalysis (GBRAS-Net). It was observed that these strategies can lead to the same or even better results in terms of accuracy, while reducing the memory requirements for the model’s storage by more than three times.

By applying FP16 quantization, we achieved accuracy results that were slightly better than those of GBRAS-Net, while reducing the memory requirements of the model by approximately two times for both the S-UNIWARD and WOW algorithms at either 0.4 bpp or 0.2 bpp.

By applying a weight clustering strategy, we achieved a model that was compressed by four times or more for all scenarios. For S-UNIWARD at 0.4 bpp, we achieved better accuracy results while reducing the memory requirements of the model by approximately four times compared to the original GBRAS-Net. For S-UNIWARD at 0.4 bpp and WOW at both 0.4 bpp and 0.2 bpp, we achieved accuracy results that were slightly worse than those of the original model. For WOW at 0.4 bpp, the decrease in accuracy was only 0.6%. The memory requirements of the compressed model were reduced by four times in this simulation.

We assessed model pruning applied with eight different pruning ratios for both the S-UNIWARD and WOW steganography algorithms at 0.4 bpp and 0.2 bpp. The simulations showed that the more pruned the network, the lower the memory requirements. We achieved slightly better accuracy results when compared with GBRAS-Net for both S-UNIWARD and WOW at 0.2 bpp at a 60% or 70% pruning ratio, respectively.

We combined both dynamic range and FP16 quantization with pruning to achieve even smaller occupied memory values. For S-UNIWARD at 0.2 bpp, accuracy of 75.19% was obtained, increasing by 1.59% when compared with the original GBRAS-Net, while the network was compressed four times. For the WOW algorithm, by applying this combination, we achieved the smallest network sizes. For WOW at 0.2 bpp, we were able to achieve a network with one eighth of the original GBRAS-Net’s memory requirement.

We compared the achieved results considering the accuracy impact and the memory savings of the model. For S-UNIWARD at 0.4 bpp, we highlighted the weight clustering strategy with K-means++ centroid initialization, which could maintain the accuracy while leading to a memory saving of 72% in comparison with the original model. For S-UNIWARD at 0.2 bpp, the best result was obtained by applying the combination of FP16 quantization with a pruning scheme at a 70% ratio.

When the scenario involved the WOW algorithm at an embedding rate of 0.4 bpp, FP16 quantization proved to be the best choice in terms of accuracy, while, for greater model compression with a small impact on the accuracy, the best optimized models were those optimized via dynamic range quantization and weight clustering with linear centroid initialization. For an embedding rate of 0.2 bpp, the best results in terms of accuracy and model compression were achieved by applying dynamic range quantization and the combination of FP16 quantization and a pruning scheme at a 60% ratio.

Although the implementation of the optimization techniques was focused on GBRAS-Net, there are other CNNs with greater network sizes. Future work could include addressing optimization strategies for these CNNs. It is important to highlight that there is room for improvement in weight clustering. In this work, four centroid initializations were considered. Other initialization strategies may be addressed or proposed. Future work could include the use of swarm intelligence approaches, e.g., [56], or modified versions of K-means or fuzzy K-means, e.g., [57,58,59,60], for the purpose of weight clustering. Another approach is to investigate different values of the maximum number of clusters and their impacts in terms of the accuracy and memory requirements of the model. Another future direction is to investigate the possibility of applying optimization strategies for Vision Transformers applied to steganalysis.