A Green AI Methodology Based on Persistent Homology for Compressing BERT

Abstract

Large Language Models (LLMs) like BERT have gained significant prominence due to their remarkable performance in various natural language processing tasks. However, they come with substantial computational and memory costs. Additionally, they are essentially black-box models, being challenging to explain and interpret. In this article, Persistent BERT Compression and Explainability (PBCE) is proposed, a Green AI methodology to prune BERT models using persistent homology, aiming to measure the importance of each neuron by studying the topological characteristics of their outputs. As a result, PBCE can compress BERT significantly by reducing the number of parameters (47% of the original parameters for BERT Base, 42% for BERT Large). The proposed methodology has been evaluated on the standard GLUE Benchmark, comparing the results with state-of-the-art techniques achieving outstanding results. Consequently, PBCE can simplify the BERT model by providing explainability to its neurons and reducing the model’s size, making it more suitable for deployment on resource-constrained devices.

1. Introduction

Over the last years, Large Language Models (LLM) like Bidirectional Encoder Representations from Transformers (BERT) [1] have achieved phenomenal results in a wide variety of natural language processing (NLP) tasks. These models, pre-trained on large amounts of data, are available for use by the scientific community, either as a semantic analytical tool or to build fine-tuned solutions for specific problems.

However, these models suffer from two weaknesses. On the one hand, they are computationally and memory-intensive, making it challenging to deploy them on devices with limited resources. As a result, these advancements run counter to the Green AI [2,3] paradigm, which is dedicated to developing AI technologies that minimize computational costs and place equal emphasis on efficiency and predictive accuracy. On the other hand, they are black-box models: due to the large number of neurons, layers, parameters, and data transformations (e.g., in Attention layers [4]), it is practically impossible to explain the internal state of the network at any given moment and provide interpretability to the final output.

In this article, Persistent BERT Compression and Explainability (PBCE) is proposed, a Green AI methodology based on homology theory to compress the BERT model and give insights about the importance of its neurons. BERT has been chosen as the reference model, given the fact that is one of the foundational models that laid the groundwork for LLM encoders. Several subsequent BERT-based encoder models, such as RoBERTa [5], DistilBERT [6], ALBERT [7], or TinyBERT [8], have been developed to address specific limitations of the original. PBCE is focused on the internal representation of the neural network, considering the individual role of each neuron when the model makes an inference. In particular, PBCE uses zero-dimensional persistent homology, extracting the topological characteristics of the neurons’ outputs and providing an assessment of their importance within the network. This way, it is feasible to identify neurons that contribute with more information to the overall computation of the network and pruning those that contribute less. As a consequence, the proposed technique becomes an effective method for compressing BERT.

To the best of our knowledge, this is the first approach that uses persistent homology to compress and give insights of the importance of the units of the BERT model. To measure the effectiveness of PBCE, an extensive experimentation has been designed based on natural language processing tasks proposed in General Language Understanding Evaluation (GLUE) benchmark [9], achieving results that outperform other state-of-the-art techniques for BERT compression. The main purpose of the experimentation is to provide an answer to the following research questions (RQ):

(RQ1)

How effective and robust is persistent homology at measuring the importance of neurons in a transformer encoder model such as BERT?

(RQ2)

Is it feasible to propose a practical methodology using persistent homology for simplifying BERT-based models?

(RQ3)

Can persistent homology be employed to enhance the explainability of language models like BERT?

The main contributions of the article can be summarized as follows:

• A methodology has been developed for compressing the BERT model by analyzing the topological features of the outputs of each neuron. This can be understood as explainability in terms of the individual role of each neuron.
• This methodology is applied to two versions of the BERT model, interpreting the topological characteristics as a tool to assess the importance of neurons, simplifying those that contribute less information, and generating a pruned version of the BERT model.
• The performance of the simplified models has been evaluated using the GLUE Benchmark and compared the results with other state-of-the-art compression techniques, demonstrating the effectiveness of PBCE for model explicability and compression.

The rest of this paper is structured as follows: In Section 2, the state of the art of different approaches using persistent homology and deep learning models to solve problems and BERT compression techniques is introduced. In Section 3, PBCE is described. In Section 4 the methodology is experimentally analyzed and discussed. Section 5 highlights the conclusions.

2. Previous Works

In this section, the most relevant articles from the state of the art related to PBCE are presented. To the best of our knowledge, there is no technique in the literature that uses persistent homology to prune neural networks. Therefore, techniques that utilize homology theory and deep learning to solve scientific problems are presented first. Finally, pruning techniques for LLMs are introduced, which will serve as a reference to validate the proposed methodology.

2.1. Persistent Homology Applied to Machine Learning Problems

In recent years, numerous machine learning methods based on persistent homology, which is a mathematical method used in topological data analysis to study features of data, have been proposed. Some of them involve feature extraction through the analysis of persistence diagrams (Birth–Death diagrams) and persistence barcodes. Additionally, there are methods which establish similarity metrics between barcodes [10]. From a mathematical perspective, solutions based on persistent homology have been presented to regularize the internal representation of deep neural networks applied to image classification [11,12].

Persistent homology, combined with machine learning, also finds applications in the fields of biology and chemistry, particularly in protein analysis. Deep neural networks are not directly applicable to molecular data, as they are characterized by complex three-dimensional structures. Therefore, the use of topological representations and features is essential for solving protein classification problems [13]. They are also highly useful for identifying structures and functions in a protein sequence [14], for automatic protein annotation [15] or, in chemistry, for tasks like simultaneous prediction of partition coefficients and aqueous solubility [16].

Getting closer to the topic of the article, homology theory has been used to analyze natural language. Specifically, in [17], TopoBERT is presented as a visual tool to explore the fine-tuning process of different language models from a topological perspective. Additionally, it facilitates the visualization of the shape of embedding spaces and the linguistic and semantic connection between the input dataset and the topology of its embedding space.

2.2. Brief Description of the BERT Model

BERT, an acronym for “Bidirectional Encoder Representations from Transformers”, Ref. [1] is a language model developed by Google that has revolutionized natural language processing. Its primary innovation lies in its ability to comprehend a word’s context within a text by analyzing all surrounding words, both to the left and right. Unlike previous language models, which were unidirectional and predicted words based on previous ones, BERT takes the entire context into account.

BERT is built upon the Transformer architecture, a deep neural network autoencoder. The Transformer architecture employs multi-head attention to process sequences. Before the network processes the information, the texts are tokenized, meaning texts are divided into a set of tokens that represent the fundamental semantic content of the word and are translated into their numerical position in the model’s vocabulary. Additionally, special tokens are added, such as [CLS], marking the start of a sentence; [SEP], marking the end, and [PAD], signifying the end of padding. Some research papers from the literature, such as [18], show that the [CLS] token, besides marking the beginning of a sequence, provides an aggregated representation of the input. In fact, it is used in text classification tasks and sentiment analysis. Figure 1 shows an example of how sequences are generated to be a valid input for BERT model.

To pre-train and fine-tune BERT, an extensive amount of text was used. During the training process, BERT learns to predict hidden words in complete sentences, enabling it to better understand how words relate to each other in a given context.

This pre-trained model can be fine-tuned for specific tasks such as text classification, entity tagging, machine translation, and more. Due to its context-awareness, BERT has outperformed many records in various natural language processing tasks and has become an essential tool in this field.

Each BERT Layer, as part of the Encoder of the Transformers architecture, consists of three components: the Multi-Head Attention layer, comprises the query (Q), key (K), and value (V) matrices, and Attention Output; the Intermediate layer; and the Output layer. Figure 2 shows the complete representation of the BERT architecture. Additionally, more detailed information can be found in [1,4].

Let L be the number of layers, H the hidden size, and A the number of self-attention layers of the model. There are two implementations of the BERT model: the BERT Base Cased model [19] ($L=12$, $H=768$, $A=12$, total number of parameters $=110$ M, and the BERT Large Cased model [20] ($L=24$, $H=1024$, $A=16$, total number of parameters $=340$ M).

2.3. BERT Model Pruning Methods

As mentioned earlier, to the best of our knowledge, there are no pruning methods for LLMs based on persistent homology. However, there are compression algorithms for the BERT architecture that provide competitive results [21]. These methods can be divided into structured and unstructured approaches [22].

Among the structured methods, some of them focus on pruning attention heads exclusively. In [23], a regularization method for BERT is proposed using reinforcement learning to prune attention heads. For text classification, a compression method based on adaptive truncation is presented in [24]. In [25], a forward and backward pass is used to calculate gradients, which are used as an importance score for each attention layer. Ref. [26] suggests constructing a loss function that minimizes both classification error and the number of heads used, pruning unproductive ones while maintaining performance. Ref. [27] presents a technique with three stages: in the first, n pruning strategies are generated with the same pruning ratio. In stage II, n candidates from the training set are evaluated, and after all iterations, the one that performs the best on a subnetwork is chosen as the best candidate. This best candidate undergoes fine-tuning to obtain a good subnetwork. In [28], a new sentence-level feature alignment loss distillation mechanism for BERT compression is introduced. It is guided by a mixture of experts to capture contextual semantic knowledge from the teacher model to the student model while reducing its parameters. In [29], a dynamic structure pruning method based on differentiable search and recursive knowledge distillation technique called DDK is presented. It is focused on pruning all feed-forward and self-attention layers. Additionally, a recursive knowledge distillation method is presented, designed to extract the most important features from the intermediate layers. Another uncertainty-driven knowledge distillation technique is presented in [30]. The uncertainty modeling guides the training process effectively, especially when there is a large gap in network performance between pretrained language models and compressed ones. In [31], a Layer-wise Adaptive Distillation (LAD) method is proposed for model compression. Specifically, an iterative aggregation mechanism is designed to distill layer-wise internal knowledge from the teacher model to the student model. Finally, in [32], a novel approach to compress BERT, called You Only Compress Once-BERT (YOCO-BERT), is introduced. YOCO-BERT constructs a search space with all the possible configurations for BERT model. Then, by using an stochastic nature gradient optimization method, an optimal candidate architecture is generated.

Among the unstructured methods, ref. [33] uses unstructured magnitude pruning to find subnetworks with sparsity levels between 40% and 90%. It concludes that those with 70% sparsity, using the masked language modeling task, are universal, and can solve other tasks without losing accuracy. Ref. [34] proposes a weight pruning algorithm that integrates reweighted L1 minimization with a proximal algorithm. More information on pruning algorithms for LLMs is available in [21].

Quantization algorithms have also been proposed, such as [35], which uses second-order Hessian matrix information for quantizing BERT models to low precision. Hardware-based techniques have also been proposed [36]. In [37], a sensitivity-aware mixed precision quantization method called SensiMix is proposed, which applies a 1-bit quantization to insensitive parts of the model. Finally, there are knowledge distillation-based algorithms, like the one presented in [30], which is based on parameter retention and feed forward network parameter distillation.

3. Our Proposal

In this article, Persistent BERT Compression and Explainability (PBCE) is proposed, a novel Green AI technique to compress BERT-based models through the application of homology theory, allowing us to derive a simplified yet effective version of the model. Specifically, using the BERT architecture as a reference, persistent homology is employed as a fundamental tool for analyzing the topological characteristics of the neurons, drawing conclusions to discard those that do not make a relevant contribution to the model. In this section, firstly, an intuitive geometric description of how PBCE applies zero-dimensional persistent homology to the vectors generated by hidden layers of a neural network is presented. Finally, the proposed methodology for compressing the model is presented.

3.1. An Intuitive Geometric Description of Persistent Homology Applied to LLM Explanations

Homology theory is a branch of algebraic topology increasingly used in data science. In particular, persistent homology is a powerful tool for studying patterns in data. Its mathematical foundation, summarized in Appendix A, is deep and complex with a significant algebraic burden. However, it can be given a much more intuitive geometric interpretation, which will be explained below.

In Figure 3, a complete example of data processing in BERT can be found. Starting with a set of elements from a corpus, after being tokenized, they are fed into the network. The output of any neuron in a hidden layer, which serves as input for the next layers, consists of vectors that, in an abstract sense, could be represented as points in a hyperspace. The geometric distribution of these points can provide meaningful information about the behavior and role that the corresponding neuron plays within the data flow of the network. Zero-dimensional persistent homology helps explain this role.

As can be seen in Figure 3, each BERT layer generates multidimensional vectors of size $N\times M\times NHU$, with N being the number of output rows, M the number of tokens in each sequence, and $NHU$ the number of hidden units. These vectors represent the internal state of texts used as inputs. For the sake of clarity, a simple two dimensional example is introduced in Figure 4. Two plots at three different time moments can be seen in that figure. The plot on the left depicts some vectors generated by any layer (as explained earlier), which are going to be converted into the centers of disks. The disks will evolve, growing uniformly with a common radius. The value of the radius is marked by the red line on the right (Birth–Death diagram). Notice that this red line will move upwards on the Y-axis of the graph on the right.

Since PBCE is based on zero-dimensional persistent homology, each output vector becomes a singleton connected component and all originate at time zero (value 0 on the Birth–Death diagram). In consequence, there are as many connected components as output vectors. As the radius of the disks grows, the connected components eventually collapse. When two connected components touch, they merge becoming a single area (a single component), and a blue point is marked on the Birth–Death plot at the radius value that led to the merger. This point on the Birth–Death plot implies the death of a connected component. Remarkably, the radius grows until the last two connected components touch, creating a large single connected component that includes all of the vectors. Let us call $r_f$ the radius value at which all connected components collapse. Please note that this is the smallest value for the radius at which all the connected components are glued together. Figure 5 shows an example of a persistence diagram in which $r_f=0.20229639$.

The $r_f$ value will be fundamental in the analysis because it provides information about the distribution of output vectors from each neuron and their variability. This helps assess whether the neuron has very uniform outputs (providing little information) or exhibit variability in its outputs (providing more information). Concretely, if $r_f$ is small, it means that the different outputs of a neuron are very similar to each other. However, if $r_f$ is large, then the outputs are less redundant. All in all, this procedure allows to “measure” the variability in the output distribution of a neuron. It can be described through a simple geometric procedure. Nevertheless, it has a founded mathematical background.

3.2. PBCE: Using Persistent Homology to Compress BERT

In this article, PBCE is proposed, a methodology aimed at compressing BERT models. PBCE uses zero-dimensional persistent homology to analyze the topological characteristics of neuron outputs for each layer, aiming to identify which of them play a more significant role in the information flow and, consequently, in the model’s decision-making process. This allows for the removal of neurons that provide less information, compressing the network and making it more efficient. Algorithm 1 outlines the proposed methodology. In summary, the methodology starts by selecting a corpus. Next, zero-dimensional persistent homology is used as a tool to measure the importance of neurons. Then, using the persistence diagram, it can be found which neurons can be removed. Finally, the simplified model is built and evaluated on the GLUE Benchmark.

Next, a detailed description of each of the steps outlined in Algorithm 1 can be found.

Algorithm 1 PBCE: BERT compression through zero-dimensional persistent homology

(1) Select the corpus to carry out the analysis. (2) Use the zero-dimensional persistent homology and the persistence diagram for the outputs of each unit and layer evaluated on the corpus. (3) Analyze the distribution of $r_f$ from the persistence diagram. Select the units to be removed. (4) Construct the simplified model. (5) Evaluate the simplified model with the GLUE Benchmark.

3.2.1. Corpus Selection

To measure the importance of each unit in the BERT model, an extensive text corpus is selected. Specifically, English Wikipedia [38] is chosen, consisting of more than 20 GB of sanitized text, including all entries from the Wikipedia in the mentioned language. To ensure that the texts, after tokenization, adhere to the input size constraint imposed by the model, the entries are split by periods, generating sentences, which will be the definite input for PBCE.

This corpus is suitable because it contains a large number of texts on diverse topics, all written with high quality. Moreover, it is not directly related to any of the tasks in the GLUE benchmark, which will be used to evaluate the performance of the simplified models.

3.2.2. Using Persistent Homology to Analyze BERT Layer Outputs

Once the corpus has been selected for analysis, persistent homology is used to collect and analyze the topological characteristics expressed by each neuron in the network. Considering the output vectors generated by each neuron, the connected components are constructed. These connected components represent the clusters of data points that are topologically connected in some way, and their persistence features helps reveal meaningful hidden structures in the data. This information is crucial for understanding the topological features that each neuron contributes to the model. As can be seen in Figure 4, the connected components evolve as the radius r grows, until all collapse into one. The lowest value of r that provokes the union of all the connected components, denoted as $r_f$, is crucial in the proposed methodology. The larger the $r_f$, the farther the neuron’s outputs are from one other, meaning that they exhibit a higher variability. Conversely, smaller values of $r_f$ suggest that the neuron is less relevant, as its outputs are very close in the metric space generated by the unit, and can thus be removed. In cases where the neuron is removed, to avoid losing all the information generated by the neuron, the mean plus the standard deviation of the outputs is taken and added to the bias of the layer. This way, the contribution of the eliminated neuron is implicitly considered in the network’s inference process.

As mentioned in the previous section, the persistence diagram is used to analyze the evolution of the connected components. Since PBCE focuses on zero-dimensional persistent homology, the points in the diagram align along a line parallel to the Y-axis, since all the points share the abscissa (given the fact that all connected components are born simultaneously). Therefore, the values on the Y-axis are only considered. $r_f$ can be readily identified in the persistence diagram as the value of r for which all connected components first merge.

3.2.3. Evaluation of $r_f$ Distribution and Selection of the Important Units

Once the value of $r_f$ for each neuron has been obtained, the analysis of the distribution of these values begins. This helps understand which unit can potentially be suppressed. To facilitate the simplification task, three levels of pruning are established:

• The first level, which is the lightest, involves calculating the first quartile (Q1) of the $r_f$ values and retaining neurons with $r_f$ values higher than Q1.
• The second level is slightly more severe, applying the same operation but with the second quartile (Q2).
• The most intense pruning is the third level, where only the neurons with $r_f$ values higher than the third quartile (Q3) are kept.

The more information a unit provides, the higher its chance to remain in the model. This way, the most influential neurons are retained and the less relevant ones are removed.

3.2.4. Evaluation of the Compressed Model Through the GLUE Benchmark

The evaluation of the simplified model through the General Language Understanding Evaluation (GLUE) benchmark involves assessing the model’s performance on a set of diverse natural language understanding tasks. GLUE is a benchmark that consists of multiple downstream NLP tasks, such as text classification, sentence similarity, and question answering. It serves as a standard evaluation suite for assess the generalization and performance of language models.

Here is how the evaluation process generally works:

• Fine-tuning process:The simplified model is fine-tuned on the GLUE benchmark tasks. The tasks are MNLI [39], QQP [40], QNLI [41], SST-2 [42], CoLA [43], STS-B [44], MRPC [45] and RTE [46].
• Model evaluation: Use the fine-tuned simplified model to make predictions on the GLUE benchmark tasks. For each task, the model will generate predictions.
• Evaluation metrics: Calculate task-specific evaluation metrics for each GLUE task. These metrics can vary depending on the task but often include accuracy, F1 score, or other relevant measures. GLUE provides a standard evaluation script for each task (

  Table 1. GLUE tasks, training and evaluation sizes, and metrics.

  Task	Train	Evaluation	Metric
  CoLA	10 K	1 K	Matthew’s Correlation
  SST-2	67 K	872	Accuracy
  MRPC	5.8 K	1 K	F1/Accuracy
  STS-B	7 K	1.5 K	Pearson–Spearman Correlation
  QQP	400 K	10 K	F1/Accuracy
  MNLI	393 K	20 K	Accuracy
  QNLI	108 K	11 K	Accuracy
  RTE	2.7 K	0.5 K	Accuracy

  ).
• Comparison: Compare the performance of the simplified model to the performance of the original, more complex BERT model and other simplified approaches from the literature. This will give an indication of how much simplification impacted the model’s ability to perform various NLP tasks.

The GLUE benchmark provides a standardized way to assess the trade-off between model complexity and task performance. It helps determine if the simplified model retains sufficient performance on a range of NLP tasks while being computationally more efficient than other BERT models.

4. Empirical Evaluation

To assess the performance of PBCE, a thorough empirical procedure has been designed that involves a wide range of natural language tasks included in the GLUE benchmark through their datasets and specific metrics (MNLI, QQP, QNLI, SST-2, CoLA, STS-B, MRPC, and RTE, as listed in Table 1); and both architectures of the BERT model: BERT Base and BERT Large. This represents a benchmark commonly used in the literature, as shown in [47,48,49]. Additionally, PBCE has been compared with the state-of-the-art BERT compression techniques.

This section presents and analyzes the experimental results. First, the distribution of $r_f$ values and their role in identifying significant neurons for prediction are examined. Subsequently, specific results for simplifying both BERT Base and BERT Large models are presented, addressing research questions RQ1 and RQ2.

4.1. Distribution of $r_f$ and Selection of the More Informative Neurons

The analysis of the distribution of $r_f$ is crucial for correctly identifying which neurons contribute the most information to a neural network’s predictions and which units are dispensable. The premise here is that a neuron with a high associated $r_f$ value implies that its outputs exhibit sufficient variability, making it important when the network computes outputs.

Bearing in mind that the mean can be heavily influenced by excessively high $r_f$ values in some situations, the median was chosen as the key metric to consider. Figure 6 shows the median of the $r_f$ values for each layer and component of BERT Base (upper image, Figure 6a) and BERT Large (lower image, Figure 6b). The Figure helps determine the pruning intensity to apply to each component and layer of the model. By analyzing the median of the $r_f$ values of the neurons within that component, PBCE quantifies the importance of each component and, consequently, the extent to which it should be pruned. The experimentation revealed that components with $r_f$ values above 0.2 will receive light pruning, values between 0.1 and 0.2 should receive medium pruning, and values below 0.1 should undergo intensive pruning. Nevertheless, it is crucial to asses the optimal cut-off value for each specific case to ensure effective pruning.

Before delving into the details of simplification for each architecture, some considerations that apply to both BERT Base and BERT Large should be taken into account. Firstly, PBCE starts the simplification process from the third layer on. This decision minimally reduces compression capacity, but does favor subsequent fine-tuning to meet the requirements of the GLUE benchmark tasks. In addition, Attention Output and Output components’ implementation involves a LayerNorm operation in which the state of the data and the original input are added together. As a consequence, these components cannot be simplified, and will be excluded of the analysis. Additionally, it is observed that the behavior of the Q and K components for each BertLayer are very similar, so the same pruning process will be applied to them. Additionally, it has been verified that the component that contributes the most information, according to our hypothesis, is the Intermediate component.

Table 2. Levels of pruning applied to each component and layer for the BERT Base architecture. The columns Q1, Q2, and Q3 indicate in which layers light, intermediate, or intense pruning is performed based on their $r_f$ values. Note that the V component tends to contribute less information (most of its layers are heavily pruned), while the Intermediate component provides a significant amount of information (none of its layers are pruned heavily, and most are pruned lightly). For more details, please refer to Figure 6.

BertLayer Component	Q1	Q2	Q3
Q	3, 4, 12	8–11	5–7
K	3, 4, 12	8–11	5–7
V	4	3, 11, 12	5–10
Intermediate	3, 4	5–12	-

and

Table 3. Levels of pruning applied to each component and layer for the BERT Large architecture. The columns Q1, Q2, and Q3 indicate in which layers light, intermediate, or intense pruning is performed based on their $r_f$ values. As in the case of BERT Base, the Intermediate component proves to be relevant for the network’s prediction in the BERT Large architecture as well. For more details, please refer to Figure 6.

BertLayer Component	Q1	Q2	Q3
Q	11,12, 14–17	4–10, 18	3, 13, 19–24
K	11, 14–17	4–10, 12, 18, 19	3, 13, 20–24
V	-	4, 8–12, 14–17	3, 5–7, 10, 13, 18–24
Intermediate	3, 4, 8–16, 18	5–7, 17, 19–24	-

present the distribution of levels of pruning to each component and layer. For example, for the Q component, layers 3, 4, and 12 are lightly pruned (retaining neurons with $r_f$ values higher than the first quartile), layers from 8 to 11 are pruned with medium intensity (retaining the second quartile of neurons), and layers 5, 6, and 7 are severely pruned (retaining the third quartile of neurons). Next, the main ideas derived from Figure 6 are explained.

In the case of BERT Base, there is an increasing trend in the contribution of neurons, with outputs becoming more variable as they approach the final layer. This trend experiences an abrupt disruption in layers 3 and 4, where all components exhibit a significant high value for the median of their $r_f$ values. In consequence, as shown in Table 2, the third and fourth layers are lightly pruned for all components, except for the V component, in which a medium pruned is applied to the third layer. In contrast, layers from 5 to 7 are the less relevant for each component. Additionally, the Intermediate component is the most meaningful one. As a result, none of its layers is heavily pruned.

In contrast, in the case of BERT Large, the median of the $r_f$ values across the layers does not exhibit a consistent growth pattern. As observed, between layers 3 and 11, all components show increasing values, contributing more information over time. However, there are significant drops in the median of $r_f$ for the Q, K, and V components between layers 12 and 13, as well as between layers 17 and 18. During these same layers, the Intermediate component experiences local maxima. In general, the values are not very high, and the Intermediate component continues to provide a significant amount of information compared to the others, except for layers 15, 16, and 17, where the Q and K components are more informative.

4.2. Results and Analysis on the BERT Base Model

The results for BERT Base are presented in

Table 4. Results of compressing BERT Base on GLUE Benchmark tasks. For each metric and task, the higher the value the better (Table 1). For the remaining parameters value (RP), expressed in millions of parameters, the smaller the value the better. Best results are highlighted in bold (excluding original BERT).

Method	MNLI-(m/mm)	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE	RP (M)
Original [1]	84.6/83.4	71.2	90.5	93.5	52.1	85.8	88.9	66.4	-
AUBER [23]	-	-	-	-	60.59 $\pm$ 0.73	-	85.62 $\pm$ 0.51	65.31 $\pm$ 1.30	-
AE-BERT [27]	-	-	88.7	-	-	86.1	89.5	69.7	-
ETbLSL [36]	82.9	90.7	88.2	89.3	52.6	84.6	88.3	63.9	-
LotteryTicketBert [33]	-	-	88.9	-	53.8	88.2	84.9	66	-
Michel et al. [25]	-	-	-	-	58.86 $\pm$ 0.64	-	84.22 $\pm$ 0.33	63.9	-
Voita et al. [26]	-	-	-	-	55.34 $\pm$ 0.81	-	83.92 $\pm$ 0.71	64.12 $\pm$ 1.65	-
QBERT [35]	77.02/76.56	-	-	84.63	-	-	-	-	-
YOCO-BERT [32]	82.6	90.5	87.2	91.6	59.8	-	89.3	72.9	67
SENSIMIX [37]	-	89.6	86.5	90.3	-	-	87.2	-	13.75
LAD [31]	81.01/81.47	87.56	89.24	91.74	-	-	88.71	67.15	52.5
DDK [29]	83.6	88.2	91.6	92.7	61.9	89.1	90.7	73.7	67
UEM [30]	-	86.2	86.4	87.5	46.8	-	86.7	-	66.8
MicroBERT [28]	80.3	86.6	-	89.6	-	-	88.7	62.8	14.5
PBCE	83.7/81.6	91.23	91.73	91.87	62.96	89.52	91.09	71.98	52

. In this table, each column corresponds to a task, and each row represents a compression technique. Thus, for each row, the result obtained by each technique is expressed in the metric corresponding to the task (Table 1). Results are reported according to the common practices in the literature.

Notice that most state-of-the-art techniques do not perform a complete evaluation on the tasks in the GLUE Benchmark. In this work, a thorough study of these tasks has been conducted, reducing the model to $47\%$ of the original parameters (from 110 M to 52 M). PBCE achieves better results in all the learning tasks compared to state-of-the-art techniques except for SST-2 and RTE, although it obtains competitive results. Even with a significant reduction in the number of parameters, it manages to improve the performance of the original model in most tasks, such as QQP (+20), QNLI (+1.2), CoLA (+10.86), STS-B (+3.72), MRPC (+2.19), and RTE (+5.58). This was accomplished with only 40 epochs in the fine-tuning process for each of the tasks. Regarding the number of parameters retained after pruning, PBCE is far from SENSIMIX (13.75 M) and MicroBERT (14.5 M), although the accuracy results indicate that PBCE achieves the best balance between accuracy and size.

4.3. Results and Analysis on the BERT Large Model

In contrast to BERT Base, there are far fewer experimental results reported in the state of the art for BERT Large. Once again, PBCE outperforms RPP [34] in all tasks except for MRPC and RTE, while improving upon the original model in MNLI (0.4/0.33), QNLI (0.45), and CoLA (1.28) (

Table 5. Results of compressing BERT Large on GLUE Benchmark tasks. For each metric and task, the higher the value, the better it is (Table 1). For the remaining parameters value (RP), expressed in millions of parameters, the smaller the value, the better it is. Best results are highlighted in bold (excluding original BERT).

Method	MNLI-(m/mm)	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE	RP (M)
Original [1]	86.7/85.9	72.1	92.7	94.9	60.5	86.5	89.3	70.1	-
RPP [34]	86.1/85.7	-	-	-	61.3	-	88.1	70.1	201
PBCE	87.1/86.2	71.9	93.2	94.7	62.1	85.2	88.6	71.7	146.2

) again, with only 40 epochs in the fine-tuning process for each of the tasks. In this case, the reduction in the number of parameters is significant, resulting in a compressed model with only $43\%$ of the parameters (from 340 M to 146.2 M). This can be explained by the fact that BERT Large is much larger than BERT Base, hence it exhibits greater redundancy.

5. Discussion

The current section elaborates on the key results detailed in the previous section. As evidenced in Table 4, PBCE demonstrates superior performance on six of the eight GLUE Benchmark tasks when applied to BERT Base. Notably, for the remaining tasks, while our approach may not achieve the highest performance, offers a substantial reduction in model size, with a parameter count that is more than 20% lower (15 M fewer parameters) compared to the DDK method. Moreover, when scaled up to BERT Large, Table 5 shows that PBCE consistently outperforms the state of the art, achieving a 27% reduction in model size (55 M fewer parameters).

In summary, given the results and methodology, it can be concluded that research questions RQ1 and RQ2 have been answered. It has been demonstrated that persistent homology is a useful tool for selecting the most important neurons in the predictive process. Furthermore, a methodology such as PBCE is proposed to simplify the BERT models.

The explainability of a machine learning is associated with its internal logic and the transformations that occur within it. The more explainable a model is, the greater the level of understanding a human can achieve in terms of the internal processes that take place when the model makes decisions [50]. As mentioned in [51], understanding the role of individual units in the inference and learning process is one possible way to endow explainability to a deep neural model. Given the fact that PBCE identifies the most significant units for each layer by analyzing the topological features of its outputs, it can be considered that PBCE provides explainability to highly complex neural networks, such as Transformer encoders like BERT, answering research question RQ3.

6. Conclusions

Large Language Models (LLMs) are revolutionizing a number of applications of artificial intelligence, especially in the field of natural language processing. However, due to their complex Transformer-based architecture, LLMs are black-box models with a large number of parameters. In this work, PBCE is presented, a zero-dimensional persistent homology Green AI methodology designed to compress the BERT models and give insights of the role of its units in the inference and learning process. In particular, PBCE analyzes the topological characteristics of neuron outputs, thereby identifying which neurons contribute more information to the inference process and which ones are dispensable. Even though it can be simply described through a geometric procedure, it has well founded mathematical background based on homology theory. Therefore, persistent homology is an effective tool for selecting key neurons, answering RQ1.

As a result of the proposed methodology, simplified versions of the BERT model are built that outperform state-of-the-art techniques, even surpassing the original model’s performance for most tasks included in the GLUE Benchmark. This allows to understand the topological behavior of LLMs like BERT, making them more explainable while maintaining performance and increasing efficiency. Therefore, RQ2 and RQ3 are answered by PBCE, which enhances model explainability.

As a line of research, it would be worthwhile to investigate the generalization of PBCE to other encoder architectures, particularly those based on the BERT family (e.g., RoBERTa, DeBERTa, DistillBERT). Furthermore, it would be interesting to explore the potential of applying PBCE to downstream tasks by fine-tuning these models on specific predictive objectives, such as sentiment classification, text summarization, and machine translation.