The Explainability of Transformers: Current Status and Directions

Abstract

An increasing demand for model explainability has accompanied the widespread adoption of transformers in various fields of applications. In this paper, we conduct a survey of the existing literature on the explainability of transformers. We provide a taxonomy of methods based on the combination of transformer components that are leveraged to arrive at the explanation. For each method, we describe its mechanism and survey its applications. We find out that attention-based methods, both alone and in conjunction with activation-based and gradient-based methods, are the most employed ones. A growing attention is also devoted to the deployment of visualization techniques to help the explanation process.

1. Introduction

Transformers are deep neural networks that exploit the self-attention mechanism to capture relationships between different portions of a text and have rapidly attracted interest in machine learning. Their applications span several domains [1], including natural language processing, Computer Vision [2], Audio and Speech signals [3], and Signal Processing [4]. Their popularity is making transformers become one of the most used deep learning architectures.

At the same time, the inherently opaque behavior of deep learning techniques has posed the problem of their explainability. Explainability (and other close terms such as interpretability [5,6]) concerns the attempt to overcome the inherent opacity of black-box models such as deep learning networks, but also ensemble methods such as those based on bagging (random forests) or boosting (e.g., AdaBoost or XGBoost). In general terms, we can define explainability as the capability to offer a detailed understanding of a machine learning (ML) model and its outputs. Such a need is present for all ML tasks, i.e., supervised tasks [7] such as classification, unsupervised tasks [8], and reinforcement learning [9]. The term XAI (Explainable Artificial Intelligence) has now come to describe this whole area of work. According to [10], that term was coined by Lent et al. [11] in 2004.

The need for explainability is strong for several reasons. This is not necessarily the case for all ML applications, as noted in [10]. Low-risk systems (where the consequences on human beings are minor) and well-studied high-trust systems (where we may dispense with explanations) are two examples where explainability is not an issue. Instead, Rudin herself focused on high-stakes decisions, where the use of ML models may have significant consequences on human lives, and mentioned healthcare and justice as two major fields where significant efforts have to be spent to achieve a transparent look at ML models’ outputs. In some cases, there are also legal requirements to enforce explainability [12].

Though the topic of the explainability of more established deep learning architecture has been widely addressed, the relative youth of transformer architecture has generated a more recent stream of studies on the explainability of transformers, whose number has, however, rapidly grown in parallel with the diffusion of the architecture itself. The establishment of a significant body of knowledge about the explainability of transformers suggests the need to organize it so as to identify the most relevant approaches, facilitate the approach of newcomers to the field, and highlight research gaps. Unfortunately, the few literature surveys (reviewed in Section 3) pre-date the explosion of interest for the topic and so do not account for the vast majority of the literature now existing.

We aim to fill that gap by proposing a survey of the current literature on the explainability of transformers by adopting a systematic approach based on a major scientific literature database (Scopus) and covering the whole production since the inception of the research efforts on this topic.

Our contributions are the following:

• We provide a taxonomy of explainability methods for transformers that relies on the transformer component that the method leverages;
• We describe each method and survey their applications;
• We identify the most favored methods;
• We identify the research gaps and provide some directions for future research.

This paper is organized as follows: We first describe the architecture of transformers in Section 2. Our literature review in Section 3 focuses on previous literature surveys on the same topic as ours. The literature proposing or applying explainability techniques for transformers is examined in Section 5, based on the dataset whose composition is reported in Section 4.

2. Transformers

Before dealing with the literature on the explainability of transformers in the next sections, we provide some background information about the architecture of transformers and their fields of application in this section.

The transformer architecture, introduced in [13], is a deep learning architecture, originally intended for sequences of text. Its structure is shown in Figure 1. It is composed of two different sections: an encoder and a decoder. The encoder part is responsible for compressing all the information derived from the input into a vector that is used by the decoder to build the next elements (e.g., the next pieces of text) in the sequence. The input to both sections is the embedding representation of each element of the sequence, combined with some form of positional encoding to represent the position of the element with respect to the whole sequence. The typical length of the embedding vector is 512. In most implementations, just as in the original paper, the positional encoding is carried out through a set of sine and cosine functions, whose frequency is modulated by the progressive index of the embedding vector element [14]. Each element of the embedding vector is added to the positional encoding vector. The specific procedure used to carry out embedding is not critical as long as it is the same for all the elements of the sequence.

Both the encoder and the decoder are composed of stacked layers of the same kind: a multi-head attention layer followed by some feed-forward linear layer. In the decoder modules, there is one more multi-head attention layer between the other two, which is used to map the input vector over all the sequences that make the output. There are also residual connections around each block, used to stabilize training. Each layer is the input to the next one.

Each multi-head attention block has to learn an attention-aware vector for each element of the sequence, based on the self-attention mechanism proposed in [15], which is in turn derived from the attention mechanism introduced in [16]. The attention-aware version of a vector in the sequence can be seen as the combination between the original vector and a weighted sum over all the vectors in the sequence, where the weights of the vectors are learned during training. Furthermore, the elements of the sequence are not the same in all the layers, but each of them learns a representation of them, also based on the representation used in the previous layer. The result of this stacked computation is a non-linear function of all the elements of the sequence combined.

Formally, we define the input of each head of each layer l of a transformer architecture as a sequence $h^{l-1}$ of vectors such that $h_i^{l-1}\in {\mathbb{R}}^d$ is the i-th vector in input, and we define the output as a sequence $h^l$ of the same dimensions. Each vector $h^l$ is computed by using a softmax function:

$$\begin{array}{cc}\hfill A_{i,j}^l& =\mathrm{softmax}\left(\frac{Q\left(h_i^{l-1}\right)·K{\left(h_i^{l-1}\right)}^T}{\sqrt{d}}\right)\in \mathbb{R}\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill h_i^l& =W_O^l·\left(\sum _j{A}_{i,j}^{l}V\left(h_j^{l-1}\right)+h_i^{l-1}\right)\hfill \end{array}$$

(2)

where $A^l$ is the attention matrix of the layer l; $W_O^l$ is the weights matrix learned by the feed-forward layer in output; and Q, K, and V are transformations defined as

$$Q\left(h\right)=W^Q·h,V\left(h\right)=W^V·h,K\left(h\right)=W^K·h,W^Q,W^V,W^K\in {\mathbb{R}}^{d\times d}$$

(3)

with $W^Q,W^V,W^K$ being weight matrices learned by the linear modules associated with the query, values, and keys branches, respectively. In each of the previous definitions, the bias term is dropped to simplify the notation. Furthermore, in the multi-head modules, all the head outputs are concatenated.

Since its introduction, the transformer has undergone many different experiments on variations of its attention mechanism and its architecture. One of the most influential variations of the transformer architecture is the transformer decoder presented in [17], where the encoder part of the transformer is completely dropped, leaving the decoder as the only component of the architecture. In this variation, both the input and the output are considered as part of the same sequence, with the model trained to just complete the sequence, given the first part that corresponds to the original input. Following the work conducted in [17], Radford et al. presented in [18] GPT-1, a decoder-only transformer trained using a self-supervised approach on a wide variety of texts, which was the first in a series of experiments whose last incarnation (at the current date) is GPT-4, presented by OpenAI in [19].

A different path of experiments began with the work presented in [20], where a different way was proposed to train an encoder-only transformer architecture. Indeed, they trained the model, randomly masking tokens of the input sequence and leaving it to the model to learn the correct missing tokens. Such an approach was called BERT (Bidirectional Encoder Representations from Transformers). Many different works have been derived from BERT, modifying some aspects to improve performances, such as RoBERTa, presented in [21], and DistilBERT, presented in [22].

A relevant branch of experiments focused on the choice of attention weights to be computed during training. Instead of computing the attention between all the combinations of input tokens, the attention may be computed over just a portion of the input sequence. Several proposals have been put forward, among which we can mention the following ones: Sparse Attention as presented in [23], Linearized Attention proposed in [24], and Low-rank self-attention suggested in [25].

Finally, though the architecture of transformers has initially been thought to process text, its application to non-text input is growing. Several experiments have considered the possibility of training multi-modal models, capable of understanding different types of input (e.g., text and images at the same time) and generating different types of output. It is worth mentioning VisualBERT described in [26], DALL-E by OpenAI, and Flamingo introduced in [27], which are all trained on both text and images. Furthermore, according to the Google Gemini Team in [28], this type of model has made it possible to achieve human expert performance on many different tasks.

Figure 1. Transformer architecture as shown in [29] (the image is available at https://www.ncbi.nlm.nih.gov/books/NBK597474/figure/ch6.Fig3/?report=objectonly (accessed on 1 March 2024) and is licensed under the terms of the Creative Commons Attribution 4.0 International License).

Figure 1. Transformer architecture as shown in [29] (the image is available at https://www.ncbi.nlm.nih.gov/books/NBK597474/figure/ch6.Fig3/?report=objectonly (accessed on 1 March 2024) and is licensed under the terms of the Creative Commons Attribution 4.0 International License).

3. Survey Literature

Since our aim is to provide a survey of the explainability techniques proposed for transformers, the rest of this paper is devoted to describing those techniques and reviewing the literature proposing them. In this section, we instead focus our attention on the survey literature, i.e., on those papers that have proposed a survey of explainability techniques in the past.

El-Zini and Awad claimed to provide the first survey of explainability issues in transformers [30]. Though their survey encompassed explainability for the larger class of deep learning models for natural language processing, they devoted a section to transformers. They analyzed three aspects: (1) visualization mechanisms, (2) the explainability of attention mechanisms, and (3) the explainability of BERT. As for visualization mechanisms, they analyzed papers (five in total) proposing interactive tools to help understand the inner workings of transformers. Two of those papers dealt specifically with BERT and one with GPT-2. The explainability of attention mechanisms was explored by examining eight papers, the majority of which (six out of eight) maintained that attention mechanisms are not inherently explainable. Finally, they analyzed six papers that scrutinize BERT. The grand total was largely lower than the number of papers we found through our literature search, due to the explosion of papers in 2023, which has made many previous surveys largely incomplete.

A more restricted field of analysis was opted for by Kashefi et al., who analyzed the explainability of vision (or visual) transformers, i.e., transformers designed for image understanding rather than NLP [31]. They introduced a taxonomy of models (from which we started to propose our more comprehensive one), built around five classes:

• Common-attribution methods;
• Attention-based methods;
• Pruning-based methods;
• Inherently explainable methods;
• Non-classification tasks.

As can be seen, their taxonomy relies on a mixture of criteria, making the taxonomy non-homogeneous, since they employed the type of task carried out by the transformer alongside the component leveraged by the explainability method. The number of papers examined (falling in at least one of the previous classes) was 35.

A more limited focus was taken by Vijayakumar, who considered explainability attained through the feed-forward layer (that follows the attention layer in the transformer architecture, as shown in Figure 1) just in the context of NLP [32]. Their limited focus leads to a very small number of papers surveyed, precisely just seven.

Beyond the survey of explainability methods, Brasoveanu and Andonie proposed instead a survey of papers dealing with the visualization of transformer operations [33]. Though they did not explicitly address explainability, visualization is tightly related to explainability since visualizing the operations of the neural network helps understand the role and relevance of the different contributions to the final outcome. They made a distinction between focused and holistic visualizers, where the former center on a single component of the machine, e.g., attention, while the latter examine the behavior of the transformer as a whole. Though they claimed to have examined 50 papers, they decided to restrict their attention to papers proposing NLP applications. Their final selection was made of 12 papers on focused visualizers and 6 papers on holistic visualizers. Again, the latest flurry of papers on the explainability of transformers urges the analysis of a wider body of literature.

Vision Transformers were, again, the focus of the survey conducted in [34]. The survey focused on post hoc XAI methods and proposed a classification of the taxonomies adopted for XAI methods. The authors distinguished between methods originally thought for CNNs (Convolutional Neural Networks) and methods specifically developed for Vision Transformers. The number of papers surveyed was 9 for the former category and 14 for the latter. Though the panorama was somewhat restricted, the survey differs significantly from the others in its effort to evaluate XAI methods on a common ground, employing the ImageNet Large-Scale Visual Recognition Challenge dataset [35] and adopting several metrics to carry out the evaluation using five criteria: faithfulness, complexity, randomization, robustness, and localization.

After examining the body of literature represented by surveys, we can conclude that ours differs from those that appeared so far in one or more of the following aspects:

• Time coverage;
• Extension, as for the number of papers analyzed;
• Extension, as for the type of application analyzed.

As for time coverage, all papers stopped their coverage at (or earlier than) 2022, missing the explosion of the literature in 2023, which we instead covered in full. The number of papers under analysis lay below 10 in [30,32] and stopped at 35 even in the most comprehensive survey [31], while we analyzed 95. The type of application is limited to NLP applications in [32] and to visual transformers in [31,33,34], while we considered all application domains.

4. Dataset

We extracted the literature of interest by querying the well-known Scopus database. The query used the terms explainability AND transformers. We obtained 279 papers (the database was accessed for the last time on 16 February 2024). We proceeded to clean that dataset using a systematic approach following the PRISMA guidelines [36,37]. The selection flowchart is shown in Figure 2. We removed first the items that represented whole conferences rather than individual papers. We then examined the abstracts to select just those papers actually dealing with explainability techniques applied to the deep learning architecture named transformer. Finally, after examining the full text of those papers, we removed another group of papers for two major reasons: they either did not deal with transformers, though they mention the architecture in the abstract, or they did not propose or analyze XAI techniques.

We can first examine the evolution of the papers satisfying the search criteria over time in Figure 3. The figure for 2024 up to the last time of access to Scopus is projected to the end of the year to make it comparable with the other years. The explosion of papers in the last two years is well visible and is a major reason to have an up-to-date survey of the subject.

Not all the papers introduced methodological innovations. Some papers introducing new XAI methods did so with no specific reference to transformers, though their methods could be applied to transformers. On the other hand, the vast majority of papers dealing with XAI for transformers analyzed the application of existing techniques, maybe with minor variants. We reported the innovative papers in Figure 4. Again, we projected the data for 2024, though the statistical reliability of such a projection is quite low since just a few innovative papers have appeared in 2024 so far. Though not explosive, the increase in 2023 is anyway visible here as well. The interest in the subject is growing.

The distribution of papers proposing innovative methods by country in Figure 5 shows some leadership by China and the U.S.A., but with European countries (Italy, Germany, and the UK) as good runner-ups.

If we consider the domain of application by type of input data in Figure 6, we can surely observe the large predominance of textual data and images. Though the leading role of the former type is well expected, since transformers have been initially proposed for NLP applications, image applications are now on par, marking the recent extension of transformers to visual data.

5. Methodologies for Explainability

In Section 3, we described the survey literature. That body of literature proposed a panorama of the existing explainability techniques. In this section, we go on to describe the literature proposing new explainability techniques for transformers. In doing so, we propose a classification of those techniques according to the architecture components they employ to explain the transformer’s output. Our classification modifies that introduced in [31], which included mixed criteria. Our classification employs the following classes and their combinations:

• Activation;
• Attention;
• Gradient;
• Perturbation.

Hereafter, for each class, we include the original formulation of the technique, though it was not expressly devised for transformers but later applied to them with some modifications, and describe the main papers that have adopted it.

5.1. Activation-Based Methods

One of the main ideas to identify the contribution of each input feature to the output is to use the activation of the neurons, going through the network back to the input. Developing this idea, Bach et al. presented in [38] the Layer-wise Relevance Propagation (LRP). This technique is based on the assumption that the activation of a neuron signals its relevance for the output. Also, the relationship between different layers is represented by the relevance of a neuron in a layer being a linear combination of the relevances of the neurons in the previous layer. This relationship allows us to track neuron relevances from the output back to the input. Furthermore, the basic method assumes that the network uses ReLU activations in all layers.

Using $a_i$ as the dummy variable describing the activation of neuron i, $w_{i,j}$ as the weight in the layer from the input i to the output j, and $R_i^l$ as the relevance of the neuron i in the layer l, we consider the following relation to hold, which allows us to compute the relevance of any neuron in a layer based on the relevances of all neurons in the subsequent layer:

$$R_i^{l-1}=\sum _j\frac{a_i·w_{i,j}}{{\displaystyle \sum _k}{a}_k·w_{k,j}}·R_j^l$$

(4)

We can apply this backtracking relationship starting from the output layer and going through the network back to the input layer, hence identifying the relevance of each input feature.

Even though the LRP method was developed to be applied to CNNs for images (which explains the assumption of ReLU activation), it has been explored and modified to be applied to other types of NNs. Ding et al. applied LRP for machine translation tasks in attention-based encoder–decoder RNNs [39]. Unlike the original method, they computed the relevance score for input vectors instead of single features. So, the relevance score for an embedding vector corresponds to the sum of the relevance scores of its features. Furthermore, they proposed to ignore the non-linear activation functions during LRP computations, based on the assumption that the choice of non-linear functions is irrelevant for LRP [38].

The application of LRP to transformers requires three more issues to cope with: different activation functions, skip connections, and matrix multiplications. Indeed, comparing a CNN architecture with that defined in Equations (1)–(3) allowed us to spot many differences; first of all, the multiplication $Q·V$ and the term $\cdots +h_i^{l-1}$ used to implement the skip connections. Following the implementation of [39], Voita et al. presented Partial-LRP to identify the most relevant heads in a transformer-based model and prune the least relevant ones [40]. In applying LRP to their models, they considered a value $R_{u\leftarrow v}$ as the relevance of neuron u for neuron v, which can be defined as

$$R_{u\leftarrow v}=\sum _{z\in \mathrm{OUT}\left(u\right)}{w}_{u\to z}·R_{z\leftarrow v}$$

(5)

where $\mathrm{OUT}\left(u\right)$ is the set of nodes directly connected to u in the next layer, and $w_{u\to v}$ is the weight ratio that measures the contribution of u to v, defined as

$$w_{u\to v}=\{\begin{array}{ll}\frac{W_{u,v}·u}{{\displaystyle \sum _{u^{\prime }\in \mathrm{IN}\left(v\right)}}{W}_{u^{\prime },v}·u^{\prime }}& \mathrm{if}v={\displaystyle \sum _{u^{\prime }\in \mathrm{IN}\left(v\right)}}{W}_{u^{\prime },v}·u^{\prime }\\ \frac{u}{{\displaystyle \sum _{u^{\prime }\in \mathrm{IN}\left(v\right)}}{u}^{\prime }}& \mathrm{if}v={\displaystyle \prod _{u^{\prime }\in \mathrm{IN}\left(v\right)}}{u}^{\prime }\end{array}$$

(6)

where $W_{u,v}$ is the weight learned between neurons u and v, and $\mathrm{IN}\left(u\right)$ is the set of nodes directly connected to u in the previous layer.

Chefer et al. in [41] proposed a variation of LRP where the issue of different activation functions is settled just considering the $a_i·w_{i,j}$ elements that turn out to be non-negative. Their analysis of the two other issues shows that both operations could be seen as Relevance Propagation through two different tensors instead of a single one. So, they expanded the definition of relevance for both tensors u and v as

$$\begin{array}{cc}\hfill R_i^{u^{l-1}}& =\sum _j\frac{u_i·v_{i,j}}{{\sum }_k{u}_k·v_{k,j}}·R_j^l\hfill \\ \hfill R_i^{v^{l-1}}& =\sum _j\frac{v_i·u_{i,j}}{{\sum }_k{v}_k·u_{k,j}}·R_j^l\hfill \end{array}$$

(7)

Finally, they introduced normalization:

$$\begin{array}{cc}\hfill {\overline{R}}_j^{u^{l-1}}& =R_j^{u^{l-1}}·\frac{\left|{\sum }_j{R}_j^{u^{l-1}}\right|}{\left|{\sum }_j{R}_j^{u^{l-1}}\right|+\left|{\sum }_j{R}_k^{v^{l-1}}\right|}·\frac{{\sum }_i{R}_i^l}{{\sum }_j{R}_j^{u^{l-1}}}\hfill \\ \hfill {\overline{R}}_k^{v^{l-1}}& =R_k^{v^{l-1}}·\frac{\left|{\sum }_j{R}_k^{v^{l-1}}\right|}{\left|{\sum }_j{R}_j^{u^{l-1}}\right|+\left|{\sum }_j{R}_k^{v^{l-1}}\right|}·\frac{{\sum }_i{R}_i^l}{{\sum }_j{R}_k^{v^{l-1}}}\hfill \end{array}$$

(8)

Even though this relevance score could be used to provide explanations for each attention layer (as in standard LRP), they used it as a construction block for a different score that computes both LRP and Attention Rollout, as described in more detail in Section 5.5.1.

In order to improve the distinction between the positive and negative relevance of neurons, Nam et al. in [42] proposed a variation of LRP called Relative Attributing Propagation (RAP), which uses normalization before the propagation of positive and negative relevances in each layer separately.

Since the results of LRP are class-independent, many methods have tried to differentiate the results of LRP to represent the classes. Gu et al. in [43] presented Contrastive Layer-wise Relevance Propagation (CLRP), which is based on the idea of computing LRP both for the class of interest and for the aggregation of the other classes, keeping only the positive differences between the two classes of relevances obtained.

In image classification, one of the most interesting aspects of prediction is understanding which image features were most influential in predicting the class. To estimate this type of influence, Zhou et al. introduced the Class Activation Mapping (CAM) technique in [44]. CAM considers a CNN with a global average pool just before the last fully connected layer (which could employ, e.g., a softmax function for classification and a ReLU activation function for regression). Its application is quite limited, given the large number of architectures other than CNN that have been proposed in the literature. The method consists of estimating the score for each area of the input image by multiplying the activation of each filter of the last convolutional layer (before the global average pool) and the weight learned for the average of the filter with respect to the neuron representing the class in the output layer. That multiplication outputs a matrix with the same size as the last convolutional layer for each filter. Summing them provides us with a relevance matrix for a single output class. The last step consists of visualizing this relevance matrix over the original input, resizing (usually with some form of interpolation) the relevance matrix to the same size as the original input image. Formally, given the last convolutional layer of a CNN architecture with the activation matrix $V^{n\times m\times F}$, composed of F filters of size $n\times m$, followed by a global average pool and a vector representing the output classes connected by a weight matrix W, we compute the relevance score $R_{x,y,c}$ for a super pixel at position $(x,y)$ for the class c as

$$R_{x,y,c}=\sum _f\left(W_{f,c}·V_{x,y,f}\right)$$

(9)

A super pixel is a segment of the matrix after reshaping by applying the filters of the architecture, basically corresponding to a patch of the input image.

A different idea was proposed by Ferrando et al. in [45], with a method called ALTI (Aggregation of Layer-wise Token-to-Token Interactions). The method computes the contribution of each component of a transformer block to the output of the block. Roughly speaking, the idea is to compute the difference between the component $A_{i,j}^{l}V\left(h_j^{l-1}\right)$ from Equation (2) and $h_i^l$ as the contribution of the j-th component to the i-th output. All the matrices composed using these differences are then combined using the same rules of Attention Rollout (explained in Section 5.2).

Focusing on the differences appearing in the output due to different inputs, Li et al. in [46] proposed a method (subsequently) called Input Erasure that masks part of the input with 0s to measure the contribution (the relevance) of this part to the same output (it could be even just a single embedding dimension).

A simple approach was proposed by Kim et al. in [47] with a method called Concept Activation Vectors (CAVs). They considered the activations of a layer for many input samples, both for the target class and the non-target class (in a binary problem). Afterward, they trained a linear model on the activations to distinguish between the target class and the non-target class. The linear model weights were the relevance scores for the class with respect to the features.

Muhammad and Yeasin in [48] proposed a method called Eigen-CAM, inspired by CAM, that is based on Single Value Decomposition. They combined the weight matrices from the first k convolutional layers, multiplying them by the input image matrix. The saliency map consisted of the projection of the matrix just computed on the first eigenvector.

Following the techniques described so far, many papers applied them in different contexts, often employing several techniques at the same time to look for the best one.

For example, Mishra et al. in [49] compared different methods (LRP and perturbation methods like LIME and SHAP, described in Section 5.4) to explain models for hate speech detection. Instead of straightforwardly evaluating models, Thorn Jakobsen et al. in [50] turned their attention to the datasets employed to evaluate explanability methods, proposing new datasets and using LRP and Attention Rollout (see Section 5.2). Other authors focused on LRP for several purposes. Yu and Xiang in [51] proposed a model for neural network pruning, visualising relevances using LRP. Chan et al. proposed a new method to perform early crop classification by exploiting LRP in [52].

CAM was instead the method chosen in [53] to explain remote sensing classification performed through a transformer-based architecture.

The ALTI method was instead chosen and slightly modified in [54] by Fernando et al., who proposed a variant called ALTI-logit, where each component $A_{i,j}^{l}V\left(h_j^{l-1}\right)$ is multiplied by the matrix just before the final softmax function in the network. They further proposed Contrastive ALTI-logit, where the difference between two different tokens is measured at the output, subtracting the results of ALTI-logit for the two tokens. They also compared their methods with Contrastive Gradient Norm and Contrastive InputXGradient.

CAVs were instead applied by Madsen et al. in [55] on an EEG classification model.

A comparison for (multi-modal) transformers was carried out in [56], namely between Optimal Transport, which considers activations of different input types, and Label Attribution, which is a variation of TMME (see Section 5.5.4). An even wider comparison was carried out by Hroub et al. in [57], where different models are employed for pneumonia and COVID-19 existence prediction from X-rays. The set of methods included Grad-CAM, Grad-CAM++, Eigen-Grad-CAM, and AblationCAM to produce saliency maps.

Another group of papers focused on visualization techniques. Alammar in [58] presented a tool (Ecco) to provide different visualization techniques for transformers. Each of them could be classified into one of the two classes: Gradient × Input or a function of activation (in this case, dimensionality reduction over responses is also performed). Van Aken et al. in [59] presented VisBERT, a visualization tool for BERT-like architectures, which is an activation-based method (they used the responses of each layer) followed by dimensionality reduction techniques (t-SNE, PCA, and ICA) to project input tokens on a 2D plane.

Gao et al. in [60] proposed a new architecture for table explanation, supplying three different methods of explanation: local explanation, global explanation, and structural explanation. The local explanation consists of computing output embeddings of text and extracting many subsequences, each of them being represented as the mean of the embeddings that compose it minus the embeddings of the CLS token. Afterward, they multiplied the matrix obtained by the weight matrix of the last layer just before applying a sigmoid activation. The results were the relevance scores of the subsequences. The global explanation consisted of computing the cosine similarity between the embeddings of the CLS (cross-lingual summarization) token of each sample in the dataset and assigning a relevance score to each of them accordingly. The structural explanation consisted of building a graph from the dataset, computing the embeddings of nodes and multiplying the embeddings by each other (CLS embeddings multiplied by its neighbors), and then (after multiplying the normalized version again for the neighbors’ embeddings) aggregating (by summing them) the scores of each neighbor and using the results as relevance scores.

5.2. Attention-Based Methods

Since the introduction of the attention mechanism in [16], attention weights have been one of the go-to indicators to estimate explanations. In [61], Abnar et al. introduced Attention Rollout and Attention Flow techniques. They share the same ground assumptions but differ in the information flow mechanism through the neural network. The assumption they share is that the attention in the last layer cannot be considered a proxy for explanation. Considering instead the attention in the first layers, we can use it to measure the contribution of each token to the result. They also underlined the importance of the residual connection with respect to the information flow, so instead of using just attention weights, they augmented the attention weights matrix with a layer l as

$$A^l=0.5·W_{att}^l+0.5·I$$

(10)

where $W_{att}^l$ is the average of all the attention weight matrices in all the heads of the layer l of the transformer.

In the Attention Rollout technique, a chain of cumulative attention matrices is formed by multiplying the attention matrix $A^l$ of the l-th layer by the attention matrices of the subsequent layers:

$${\tilde{A}}^l=\{\begin{array}{ll}{A}^l& \mathrm{if} \mathrm{it} \mathrm{is} \mathrm{the} \mathrm{last} \mathrm{layer}\\ A^l{\tilde{A}}^{l-1}& \mathrm{otherwise}\end{array}$$

(11)

The result of the multiplication can be easily shown as a heatmap of the relevance of each token in the input with respect to each token in the output.

In the Attention Flow technique, the neural network is seen as a graph whose edges are weighed by the attention weights $A^l$ of the pertaining layer. Considering the weights as capacities, the input tokens as source nodes (one at a time), and the output tokens as sink nodes (again one at a time), we can compute the max flow of the network and consider it as the relevance score for the pair (source and sink).

When we come to the application of attention-based techniques, we can recognize three streams: (1) the papers proposing the usage of attention weights; (2) the papers using the Attention Rollout that we described above; and (3) the papers exploiting visualization techniques for attention weights.

In the first group, we find Renz et al., who proposed two different models for route planning in complex environments using the sum of attention weights as relevance scores [62]. Feng et al. proposed a model for early stroke mortality prediction, using attention weights as relevance scores [63]. A more complex function of attention weights was employed by Trisedya et al. to explain the output in knowledge graph alignment [64]. Applications in the medical field were considered by Graca et al., who proposed a framework for Single Nucleotide Polymorphisms (SNPs) classification, using attention weights to explain the classification [65]; by Kim et al., who used attention weights to score text in input and explain decisions taken by a model dedicated to medical codes prediction [66]; and by Clauwaert et al., who focused on automatic genomics transcription, analyzing the attention weights of the trained model to prove the specialization of each head with respect to some input feature [67]. Sebbaq and El Faddouli proposed a new architecture to perform a taxonomy-based classification of e-learning materials, using attention weights for explainability [68]. In their model for sequential recommendation, Chen et al. also relied on attention weights for explainability [69]. An aggregation of attention weights was employed by Wantiez et al. to explain the results of their architecture for visual question answering in autonomous driving [70]. The context considered by Ou et al. to use attention weights was instead next-action prediction in reinforcement learning [71]. An aggregation over all attention weights of all heads in all layers was employed by Schwenke et al. to process time series data using symbolic abstraction [72,73]. Finally, Bacco et al. trained a transformer to perform sentiment analysis, using a function of the attention weights to select the input sentences in input that better justify the classification [74]. The same subject was more extensively dealt with in [75]. Humphreys et al. proposed an architecture for predicting defects, using the sum of attention weights over all layers [76]. Attention weights were employed for searching in a transformer-based model dedicated to multi-document summarization in [77].

The following six papers used the Attention Rollout technique. Di Nardo et al. proposed a transformer-based architecture for visual object-tracking tasks, using Attention Rollout for explainability [78]. Cremer et al. tested an architecture on 3 datasets for drug toxicity classification [79]. A variation of Attention Rollout was employed by Pasquadibisceglie et al. to generate heatmaps in a framework for next-activity prediction in process monitoring [80]. Attention Rollout was employed in conjunction with Grad-CAM (see Section 5.3) by Neto et al. to detect metaplasia in upper gastrointestinal endoscopy [81]. Both Attention Rollout and LRP were tested by Thorn Jakobsen et al. on new datasets in [50]. Finally, Komorowski et al. compared LIME, Attention Rollout, and LRP-Rollout (see Section 5.5.1) for a model dedicated to COVID-19 detection from X-ray images [82].

A large group of papers have addressed the use of the visualization of attention weights to help explain the outcome of transformers. Fiok et al. used both BertViz (a visualization tool for attention weights) and TreeSHAP (a variation of SHAP for tree-based models) [83]. Again, Tagarelli et al. employed BertViz after training a BERT-like model on the Italian Civil Code [84]. Lal et al. proposed a tool to explain transformers’ decisions by visualizing attention weights in many ways, including dimensionality reduction [85]. Dai et al. adopted the visualization of the attention weights to explain a classification model to infer personality traits based on the HEXACO model [86]. Gaiger et al. considered a general transformer-based model [87]. Zeng et al. used the visualization of attention weights to explain a new framework for DNA methylation sites prediction [88]. Textual dialogue interaction was instead the application of interest in [89]. Ye et al. employed attention weights visualization to classify eye diseases from medical records [90]. Neuroscience was the domain of application considered in [91], where a new architecture for brain function analysis was proposed, and [92], where a new architecture was proposed based on the graph representation of neurons from fMRI images to predict cognitive features of the brain. Sonth et al. trained a model for driver distraction identification [93]. Kohama et al. proposed a new architecture for learning action recommendations in [94]. Wang et al. proposed a new architecture for medical image segmentation, using visualizations of both attention weights and gradient values to explain the output [95]. Kim et al. proposed an architecture for water temperature prediction [96]. Monteiro et al. proposed an architecture for a 1D binding pocket and the binding affinity of drug–target interaction pairs prediction in [97]. Finally, Yadav et al. compared different explanability methods (LIME, SHAP, and Attention visualization) for hate speech detection models in [98].

A related research stream considered the use of transformers for images, i.e., visual transformers. Ma et al. computed an indiscriminative score for each patch of an image as a function of the attention weights in all the layers [99].

5.3. Gradient-Based Methods

Most of the training algorithms for neural networks are based on some form of gradient backpropagation from the loss function to the input. Many explainability methods are based on different functions of the gradient computed at different points in the neural network.

One of the first works using this approach was by Simonyan et al. in [100]. They presented a method (subsequently) called saliency, which computes the gradient of $Y^c$ with respect to the input x. Employing the gradient in the linear approximation of $Y^c$ in a neighborhood of x via its Taylor series is analogous to interpreting the coefficients in a linear regression model as a measure of feature importance. Furthermore, another work presented by Springenberg et al. in [101] introduced a class of methods that included Guided Backpropagation. This method consists of a forward pass through a CNN to reach a selected layer and then, after zeroing all the features but one, a backward pass to the input (filtering out all the non-positive pixels) to compute the relevance of the feature. After those papers, Kindermans et al. in [102] proposed to scale the scores obtained with saliency by multiplying them by the input in a method (subsequently) called InputXGradient.

Extending the work in [100], Yin and Neubig in [103] computed the gradient of the difference for two different outputs with the same input (formally $\frac{\partial (Y^c-Y^{c^{\prime }})}{\partial x}$) as the saliency score, and they called it the Contrastive Gradient Norm. They also used the same gradient to extend the work in [102], calling it the Contrastive InputXGradient.

To generalize the CAM technique, Selvaraju et al. introduced in [104] the Gradient-weighted Class Activation Mapping (Grad-CAM) technique. The most important advantage of Grad-CAM with respect to CAM is the compatibility of the method with any CNN-based architecture. The method consists of backpropagating the output until the last convolutional layer. The gradients we propagated back are averaged to obtain a vector with a size equal to the number of filters. So, we use this vector just like the learned weights in the CAM method, multiplying them by the activations of the filters in the last convolutional layer. As the last step, we apply ReLU over the results of the linear combinations to filter out negative scores. Formally, given the last convolutional layer of a CNN architecture with the activation matrix $V^{n\times m\times F}$, composed by F filters of size $n\times m$, and the final output for a class c represented as $Y^c$, we obtain:

$$R_{x,y,c}=\mathrm{ReLU}\left(\sum _f\left[\left(\frac{1}{F}\sum _{i,j}\frac{\partial Y^c}{\partial V_{i,j,f}}\right)·V_{x,y,f}\right]\right)$$

(12)

Grad-CAM++ presented in [105] is a variation of Grad-CAM. Unlike GRAD-CAM, the ReLU is moved to the partial derivative, and different coefficients are used for each combination of position, filter, and class. These coefficients are computed as a function of the gradients backpropagated from the last layer of the network (before the activation function).

We can now see the papers employing one or more of the techniques described so far. Grad-CAM alone was employed by Sobahi et al., who proposed a model to detect COVID-19 by using cough sound recordings [106]; Thon et al., who proposed a model to perform a 3-classes severity classification of COVID-19 from chest radiographs [107]; and Vaid et al., who trained a model for ECG 2D representation classification [108]. Wang et al. proposed a transformer-based architecture for medical 3D image segmentation, using Grad-CAM++ in [109].

More often we find Grad-CAM employed with other techniques in a comparative fashion. Wollek et al. compared TMME and Grad-CAM for pneumothorax classification from chest radiographs [110]. Neto et al. employed Grad-CAM and Attention Rollout to explain a model for metaplasia detection in upper gastrointestinal endoscopy [81]. Thakur et al. compared LIME and Grad-CAM for plant disease identification from leaves images [111]. Kadir et al. compared Soundness Saliency and Grad-CAM for image classification [112]. Vareille et al. employed a host of methods (SHAP, Grad-CAM, Integrated Gradients, and Occlusion) for multivariate time series analysis [113]. Hroub et al. compared different models for pneumonia and COVID-19 prediction from X-rays, using Grad-CAM, Grad-CAM++, Eigen-Grad-CAM, and AblationCAM to produce saliency maps [57].

A wider selection, not including Grad-CAM, was employed in other papers. Cornia et al. proposed a method to explain transformers’ decisions in visual captioning by applying three different gradient-based methods (saliency, Guided Backpropagation, and Integrated Gradients) [114]. Poulton et al. applied saliency, InputXGradient, Integrated Gradient, Occlusion, and GradientSHAP to explain the decisions of transformers concerning the automatic short-answer grading task [115].

Finally, visualization techniques were considered by Alammar, who presented a tool (Ecco) to provide different visualization techniques for transformers [58]. Wang et al. proposed a new architecture for medical image segmentation, using the visualization of both attention weights and gradient values to explain the output [95].

As an example of the results that can be obtained with such methods, a visualization dedicated to image classification can be seen in Figure 7. In this image, each pixel is colored according to the score assigned by the different methods (most of them already explained in this section) so as to compare them. This type of visualization, consisting just of a heatmap over the input image, is usually called a saliency map.

5.4. Perturbation-Based Methods

The perturbation approach identifies the relevance of a portion of the input by masking it and checking the consequences on the output.

Zeiler and Fergus in [117] introduced an approach (subsequently) called Occlusion, where part of the input is masked with 0s and the difference in the output is measured. So, moving the Occlusion, we can use the differences in the output as a measure of the relevance of the masked part of the input.

Comparing neural networks and linear models, it is known that even though neural networks provide better results, linear models are much more easily explainable. Thinking of this fundamental difference, Ribeiro et al. introduced Local Interpretable Model-agnostic Explanation (LIME) in [118], trying to build a linear model to explain the decision taken by the neural network by lightly perturbing the input and measuring the difference in the output. Formally, given a function of probability f with respect to a class (the model we are trying to explain), a class of explainable models G (it could be the class of linear models), a function ${\pi }_x$ that measures the proximity distance with respect to x, and a function $\Omega$ to measure the complexity of a model of the class G, we try to find a model M for the input x defined as

$$M\left(x\right)={argmin}_{g\in G}\mathsf{\ell }\left(f,g,{\pi }_x\right)+\Omega \left(g\right)$$

(13)

where the function ℓ returns a measure of how unfaithful the model g is in approximating f in the locality defined by ${\pi }_x$.

The definition is very general and could be arranged in many different ways. For the sake of clarity, we take the class of linear models as an example for G, defined as $g\left(z\right)=w_g·z$. Considering this type of model, we would define ${\pi }_x$ as

$${\pi }_x\left(z\right)=e^{\frac{-D{(x,z)}^2}{{\sigma }^2}}$$

(14)

with the function D defined as a distance function (e.g., cosine distance for text and L2 distance for images) and $\sigma$ as a weight factor. Furthermore, we would define the function ℓ as

$$\mathsf{\ell }(f,g,{\pi }_x)=\sum _{z,z^{\prime }\in \mathbb{Z}}{\pi }_x\left(z\right)·{(f\left(z\right)-g\left(z^{\prime }\right))}^2$$

(15)

with $\mathbb{Z}$ defined as the set of samples obtained by perturbing the initial input x.

It is important to notice that this type of technique guarantees a faithful local explanation but not a global one.

Lundberg et al. introduced SHAP in [119], a method based on the game-theory-based notion of Shapley values developed by Shapley in [120]. In their work, Lundberg et al. connected pre-existing methods (such as LIME and DeepLIFT) to Shapley values. They proved that LIME could return valid Shapley values after some variations. First of all, the perturbing method should introduce random masking over the features, replacing the missing ones with values sampled from a marginal distribution computed on the original input. The masking is represented by the function $h_x$. The proximity distance should be defined as

$${\pi }_x^{\prime }\left(z\right)=\frac{M-1}{\left(\genfrac{}{}{0pt}{}{M}{\left|z\right|}\right)\left|z\right|(M-\left|z\right|)}$$

(16)

where ${\mathbb{Z}}^{\prime }$ is the set of masked samples; $z\in {\mathbb{Z}}^{\prime }$, $\left|z\right|$ is the number of unmasked features; and M is the max number of unmasked features among the samples in ${\mathbb{Z}}^{\prime }$. The model to be used should be a weighted linear regression model defined as

$$g^{\prime }\left(z\right)=w_0+\sum _j{w}_j·z_j$$

(17)

Finally, the loss function ${\mathsf{\ell }}^{\prime }$ is defined as

$$\mathsf{\ell }(f,g^{\prime },{\pi }_x^{\prime })=\sum _{z\in {\mathbb{Z}}^{\prime }}{\pi }_x\left(z\right)·{[f\left(h_x\left(z\right)\right)-g^{\prime }\left(z\right)]}^2$$

(18)

A different approach, called Anchors, was proposed by Ribeiro et al. in [121]. They focused on the subset of input features that leaves the same output classification (with a high probability) even after the change in the remaining features.

Petsiuk et al. in [122] presented a method called RISE (Randomized Input Sampling for Explanation), which is suitable for any image classification model. It generates many random masks that are applied to the input image. The probability of classification is measured for each masked image, and then the relevance map is composed as a weighted sum of the masks (with respect to the probabilities measured in the output) after applying a normalization with respect to how many times a pixel was in a mask.

Gupta et al. in [123] propose a method (subsequently) called Soundness Saliency, which consists of learning a matrix (with the same size of the input image) used to mask the original input, such that the expectation of the negative logarithm computed on the classification output probability is minimum. Unlike other methods, the values used to replace the masked pixels are taken from an image randomly picked from the training set. The saliency map will correspond to the learned masking matrix.

A variation of CAM, called AblationCAM, was proposed by Desai and Ramaswamy in [124], also inspired by Grad-CAM but without using gradients. They predicted the probability of an output class for an input image, considering the last layer activation matrix, just before the classification layer. They computed the relative increase in the output probability for each pixel (of each filter) in the last layer, zeroing the corresponding activation and predicting again the output probability. After that, they computed a saliency map (reshaped with respect to the input image) composed by the activation of a pixel times the percentage computed for the same pixel.

When we examine the papers reporting the application of perturbation-based methods, we see that the great majority are papers using the two best-known methods: LIME and SHAP. In most cases, they use either alone.

As for LIME, Mehta et al. applied LIME to explain the decisions of BERT-like models in a hate speech detection task [125]. Rodrigues et al. extended LIME to meta-embedding input in a model for a semantical textual similarity task with a meta-embedding approach [126]. Janssens et al. employed LIME when comparing different models to detect rumors in tweets [127]. Chen et al. compared different models to perform Patient Safety Events classification using a proprietary dataset [128]. Collini et al. proposed a framework for online reputation and tourist attraction estimation, explaining the output with LIME [129]. Finally, Silva and Frommholz employed LIME as a model to perform multi-author attribution [130].

A similar-size group of papers have instead employed SHAP for explanation purposes. In the following, we report the task for which explainability was sought through SHAP. Upadhyay et al. proposed a new model for fake health news detection [131]. Abbruzzese et al. proposed a new architecture for OCR anomaly detection and correction [132]. Benedetto et al. proposed an architecture for emotional reaction prediction for social posts [133]. Rizinski et al. proposed a framework to perform a lexicon-based sentiment analysis [134]. Sageshima et al. proposed a method to classify donors with high-risk kidneys in [135].

Then came several papers that compared different explainability approaches, including LIME and SHAP, either alone or in combination. Most of them considered gradient-based approaches. El Zini et al. employed LIME, Anchors, and SHAP when introducing a new dataset to evaluate the performances of different models for sentiment analysis [136]. Lottridge et al. compared annotations from humans with respect to explanations provided by both LIME and Integrated Gradients within the scope of crisis alert identification [137]. Arashpour et al. compared a wide range of explainability methods falling into the classes of perturbation-based methods and gradient-based methods (Integrated Gradients, Gradient SHAP, Occlusion, the Fast Gradient Sign Method, Projected Gradient Descent, Minimal Perturbation, and Feature Ablation) for waste categorization in images [138]. Neely et al. compared LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, and Deep-SHAP to measure their degree of concordance [139]. Komorowski et al. compared LIME, Attention Rollout, and LRP-Rollout for a model to detect COVID-19 based on X-ray images [82]. Thakur et al. used LIME and Grad-CAM in [111] to compare different models for plant disease identification from leaf images. Tornqvist et al. proposed integrating SHAP and Integrated Gradients for automatic short-answer grading (ASAG) [140]. Vareille et al. compared different explainability methods (SHAP, Grad-CAM, Integrated Gradients, Occlusion, and different variations of them dedicated to the task) for multivariate time series analysis [113]. Mishra et al. compared LIME, SHAP, and LRP in explaining models for hate speech detection [49]. For the same task, Yadav et al. compared LIME, SHAP, and Attention visualization in [98]. Malhotra and Jindal used SHAP and LIME in models for depressive and suicidal behavior detection [141]. Abdalla et al. employed LIME and SHAP when introducing a dataset to be used as a benchmark for human-written papers classification [142]. Fiok et al. used both TreeSHAP (a variation of SHAP for tree-based models) and BertViz in [83].

Quite a smaller group of papers considered perturbation methods other than LIME or SHAP. Poulton et al. applied different methods (saliency, InputXGradient, Integrated Gradient, Occlusion, and GradientSHAP) to explain the decisions of transformers for the automatic short-answer grading task [115]. Kadir et al. compared Soundness Saliency and Grad-CAM with respect to image classification [112]. Tang et al. proposed a new method for explainability that consists of finding the relevance matrix that minimizes the difference between the loss computed on the original image and the perturbed version obtained by masking the image with the relevance matrix, applying the new technique to a new model for cancer survival analysis [143].

5.5. Hybrid

The methods described in Section 5.1, Section 5.2, Section 5.3 and Section 5.4 fall sharply into one of the categories we identified at the beginning of Section 5. However, some methods have been proposed that employ a combination of approaches (in most cases, a couple). In this section, we describe those methods, devoting a subsection to each combination.

5.5.1. Activation + Attention

As we said before, Chefer et al. in [41] introduced a form of a relevance score that is the combination of the Attention Rollout method and LRP method with the assumption of an architecture based on self-attention. For the sake of brevity, we will call it LRP-Rollout. Indeed, for each transformer block B, they compute a score matrix ${\overline{A}}^B$, multiplying them (equally to what is performed in Attention Rollout) to obtain a global score matrix C:

$$C={\overline{A}}^1·{\overline{A}}^2·\dots ·{\overline{A}}^N$$

(19)

where ${\overline{A}}^1$ is the score matrix for the last transformer block and ${\overline{A}}^N$ is the score matrix for the first transformer block. The difference with respect to the Attention Rollout methods is the definition of each score matrix. They are defined as

$${\overline{A}}^B=I+{\mathbb{E}}_h{\left(\nabla A^B\odot R^B\right)}^{+}$$

(20)

where $\nabla A^B$ is the gradient of the attention matrix $A^B$; ⊙ is the Hadamard product (element-wise product); $R^B$ is the relevance scores obtained with LRP over the softmax layer of the transformer block B; ${\mathbb{E}}_h$ is the average operation over all the heads in the transformer block B; and ${(\dots )}^{+}$ indicates only the positive values, leaving all the others as 0.

The number of applications of the combination of Activation and Attention is quite small.

Bianco et al. compared different architectures for food recognition, using LRP-Rollout for the explanation [144]. A combination of Attention Rollout with the activations of the last layer was proposed by Black et al. when detecting image similarities in the input of transformers [145]. Sun et al. proposed a method to estimate the relevance of each token in the input by masking all the attention matrices (in each layer) by the columns corresponding to the token we are computing, keeping also all the other values fixed in [146]. A comparison of LIME, Attention Rollout, and LRP-Rollout was carried out by Komorowski et al. in COVID-19 detection based on X-ray images [82].

5.5.2. Activation + Gradient

Gur et al. in [147] presented a method called Attribution-Guided Factorization (AGF), which combines the ideas behind both LRP and Grad-CAM. In their work, they tried to classify each neuron, with respect to the output class, as either in the background or in the foreground. The main idea behind the method is to use positive components of the gradients, which are then propagated back using variations of Relevance Propagation methods.

A similar approach is the basis of Softmax-Gradient Layer-wise Relevance Propagation (SGLRP), presented by Iwana et al. in [148]. They considered the relevance scores computed from the last layer, just as the gradients computed for the selected class, and then used classical LRP rules to propagate relevance to the input layer.

A method called FullGrad, thought for convolutional networks, was presented by Srinivas and Fleuret in [149]. They summed two terms: the InputXGradient term and the sum of the gradients with respect to each channel of each layer, each of them multiplied by its bias term. All the terms were rescaled and transformed according to the input image, just before the global sum. The result is a saliency map of the input image (with respect to the class of interest).

There also exists another method called Eigen-Grad-CAM, derived from both Eigen-CAM and Grad-CAM, that, as far as we know, has never been published, and it is only present in a Python library (https://github.com/jacobgil/pytorch-grad-cam (accessed on 1 March 2024)). The method consists of multiplying activations and gradients (like Grad-CAM) but then projecting on its first eigenvector (just like Eigen-CAM).

Ferrando et al. extended the ALTI method (see Section 5.1), multiplying each component $A_{i,j}^{l}V\left(h_j^{l-1}\right)$ by the matrix just before the final softmax computation in the network (supposing a next token prediction task) built by training and dropping the difference with respect to the module output. The method is called ALTI-logit. They also proposed to measure the difference between two different tokens in the output, subtracting the results of ALTI-logit for the two tokens. They called it Contrastive ALTI-logit. They also compared their methods with the Contrastive Gradient Norm and Contrastive InputXGradient in [54].

Hroub et al. compared different models for pneumonia and COVID-19 existence prediction from X-rays and used many methods (Grad-CAM, Grad-CAM++, Eigen-Grad-CAM, and AblationCAM) to produce saliency maps [57].

Arian et al. proposed a new architecture for human age and gender estimation from panoramic radiographs, using FullGrad to create saliency maps [150].

5.5.3. Activation + Perturbation

Sometimes, the difference in activations is measured when the input changes. The class of methods opting for this approach actually employs both techniques, perturbing the input and measuring the activation.

One of the main methods employing this approach is Deep Learning Important FeaTures (DeepLIFT), presented by Shrikumar et al. in [151]. It is based on the same ideas behind LRP, but it uses the differences in activations given by the input image and a perturbed one, which acts as the reference image, depending entirely on the task (for the MNIST dataset, they used a totally black image, while for DNA sequences, they used a reference input, which has expected frequencies for the base components in each position).

In the already cited work concerning SHAP, Lundberg et al. also introduced a modified version of DeepLIFT as an approximation of SHAP values [119]. The method consists of applying DeepLIFT propagation rules, defined in terms of SHAP values, recursively on small components of the input to obtain relevance scores for the input features.

Score-CAM, presented in [116], is a variation of Grad-CAM that does not use gradients but just the activations of the last convolutional layer, where each activation matrix for a filter is normalized, resized according to the original input size, and then passed again through the network to obtain the weights used to compute the linear combination of activation matrices, which will correspond to the saliency map.

Xie et al. in [152] presented ViT-CX, a technique dedicated to visual transformers. Considering the patches of the input image elaborated by the ViT, they used the output matrix of a module (usually the last one in the architecture) as the embeddings of the patches, building a new matrix composed of all the results of the same module for all the patches. All the slices of the matrix (each of them composed of all the patches) are then taken separately, upscaled according to the input image, and normalized between 0 and 1. All these slices become masks for the input. To reduce the cardinality of the problem, we aggregate the masks using some clustering technique and then we add noise to each mask. We consider the saliency map from a mask as the output of the model after we apply the mask, plus the difference in the output between the original input and the original input with noise added. The final saliency map is the sum of all the saliency maps, with each pixel component normalized with respect to the number of times the pixel is in a mask.

The papers employing these hybrid techniques typically compare several approaches to explainability. Neely et al. compared LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, and Deep-SHAP and evaluated their degree of concordance [139]. Englebert et al. presented a method called Transformer Input Sampling (TIS), which can be considered a variant of ViT-CX, but instead of masking the input image, they randomly picked a set of patches of the input and then computed the output for that set of embeddings. Furthermore, they compared TIS with ViT-CX, TAM, TMME, LRP-Rollout, Attention Rollout, BT, RISE, Integrated Gradients, and SmoothGrad [153]. Jourdan et al. proposed a method composed of three steps: (1) using Non-negative Matrix Factorization on the activation matrix from the last layer to obtain the topic matrix; (2) applying perturbation on the topic matrix; and (3) applying Occlusion to obtain the relevance scores [154].

5.5.4. Attention + Gradient

The method AttCAT, presented by Qiang et al. in [155], was inspired by Grad-CAM, but it was focused on transformers for text-related tasks. Considering the output of a single layer l of a transformer architecture as composed by the columns $h_1^l,\dots ,h_i^l,\dots ,h_n^l$, each of them representing the output for the i-th token, we can compute what they call Class Activation Tokens (CATs) for the i-th token in the l-th layer for the output class c as

$${\mathrm{CAT}}_{i,l}^c=\frac{\partial y^c}{\partial h_i^l}\odot h_i^l$$

(21)

After computing the CAT values, we can compute the relevance of each token i with respect to the class c as

$$R_i^c=\sum _l{\mathbb{E}}_h\left(A_i^l·{\mathrm{CAT}}_{i,l}^c\right)$$

(22)

$A_i^l$ is the attention weight in the l-th layer for the i-th token.

As an example of this class of explainability techniques, we can visually compare different methods, including AttCAT, dedicated to NLP tasks, in Figure 8. In this type of visualization, the color gradient represents the importance of each token (in this case, each word) with respect to the classification of the sentence.

Chefer et al. presented a method based both on attention weights and gradient computing in [156]. For the sake of brevity, we will call it TMME. It is compatible both with transformers with either text or image input and transformers with mixed input (text and image). Also, it can be applied both on self-attention-based architectures and co-attention-based architectures. Considering s and q, representing either a text or image, the relevancy maps are initialized as

$$\begin{array}{cc}\hfill R^{ss}& =\mathbb{I}\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill R^{sq}& =0\hfill \end{array}$$

(24)

where $s\ne q$, and also considering the dimensions of the matrices compliant with the input features. The matrices are then updated recursively as

$$\begin{array}{cc}\hfill R^{ss}& =R^{ss}+\overline{A}·R^{ss}\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill R^{sq}& =R^{sq}+\overline{A}·R^{sq}\hfill \end{array}$$

(26)

for self-attention layers, and it is updated recursively as

$$\begin{array}{cc}\hfill R^{sq}& =R^{sq}+{\left(R^{ss}\right)}^T·\overline{A}·R^{qq}\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill R^{ss}& =R^{ss}+\overline{A}·R^{qs}\hfill \end{array}$$

(28)

for co-attention layers. In all previous equations, we refer to $\overline{A}$, which is formally defined as

$$\overline{A}={\mathbb{E}}_h\left({\left(\nabla A\odot A\right)}^{+}\right)$$

(29)

where A represents the attention matrix of the layer, ${\mathbb{E}}_h$ is the average over all the heads in the layer, and the gradient is defined as

$$\nabla A=\frac{\partial Y_c}{\partial A}$$

(30)

A variant of TMME called Grad-SAS (Gradient-guided Self-Attention Summation) was proposed by Sun et al. in [157], where the relevance scores are just the sum of the $\overline{A}$ terms. another variant of TMME was proposed by Huang et al. in [158], where the updating rules for relevance values are modified as a weighted sum (with weights computing as a function of the gradient) for multi-modal models.

In [159], Liu et al. proposed a method to explain the classification of a text by a transformer-based model, composed of two methods: Attention Gradient (AGrad) and Relevance Propagation from Attention Gradient (RePAGrad). These methods return a relevance score for each token, which can be either positive or negative depending on the contribution given by the token for the classification. It computes the gradient of the loss to find if the contribution is positive (or negative), and it also backpropagates the relevance recursively using the Layer-wise Relevance Propagation (LRP) technique. Formally, the relevance score $R_j$ for the j-th input token is defined as

$$R_j=\sum _i\left[R_{i,j}^{\prime }·\left(-\frac{\partial L}{\partial A_j}·A_j\right)\right]$$

(31)

where $R_{i,j}^{\prime }$ is the relevance score for the i-th contextual embedding with respect to the j-th input token computed by using the LRP method (this is true only for the RePAGrad method), $A_i$ is the attention weight for the j-th input token, and L is the loss of the model. The variation in the AGrad method, with respect to RePAGrad, is the $R^{\prime }$ matrix, which is defined as equal to the identity matrix.

The product of the AGrad and RePAGrad scores was proposed by Thiruthuvaraj et al. in [160]. The combination of SHAP and Integrated Gradients was proposed by Tornqvist et al. for automatic short-answer grading (ASAG) [140].

A comparison of TMME and Grad-CAM was carried out by Wollek et al. for pneumothorax classification from chest radiographs [110]. Another comparison between Label Attribution (which is a variation of TMME) and Optimal Transport (the comparison between activations of different input types) was carried out by Ramesh and Koh in [56].

Sun et al. in [146] proposed a method to estimate the relevance of each token in the input by masking all the attention matrices (in each layer) by the columns corresponding to the token we are computing, keeping also all the other values fixed (we compute a forward pass on the original model, then we freeze the input of each layer, and we mask the attention matrices). The result of the last layer is extracted to represent the Shapley values (so the relevance of each token). They also proposed two variations of Grad-CAM based on attention: the first one consists of computing the gradient of attention multiplied by the attention, and the second one is the same but expanded to the second order.

5.5.5. Attention + Perturbation

For this combination, we do not have significantly innovative methods.

However, Setzu et al. explained the output of a transformer-based model for many NLP tasks by extracting structured triples from the input text (for instance: subject, verb, and object), then perturbing the triples (using WordNet), and finally assigning a score to each perturbed triple based on attention weights with respect to the original input [161]. Correia et al. proposed a method for the explainability of face detection tasks by computing a binary attention matrix (using a threshold), using that matrix to compute a set of masks to perturb the image, and finally measuring the difference between the outputs of the original image and the perturbed one, repeating the procedure to obtain values to be merged to compose a relevance matrix [162].

5.5.6. Gradient + Perturbation

The most relevant method for this combination is the Integrated Gradients method, presented by Sundararaja et al. in [163]. They proposed to compute the integral of all the gradients computed in each point from a baseline input to the actual input. Formally, we consider a function $F:{\mathbb{R}}^n\to [0,1]$ as the representation of our model (for a single output), x as the input, and $x^{\prime }$ as the baseline input (i.e., for images, it could be a black image, and for text, an embedding composed only by 0s). The relevance of each component i of the input is defined as

$$R_i=(x_i-x_i^{\prime })·{\int }_{\alpha =0}^1\frac{\partial F\left(x_i^{\prime }+\alpha \left(x_i-x_i^{\prime }\right)\right)}{\partial x_i}d\alpha$$

(32)

The integral can be approximated by using the Riemann sum method as follows:

$$R_i=(x_i-x_i^{\prime })·\sum _{k=1}^m\frac{\partial F\left(x_i^{\prime }+\frac{k}{m}\left(x_i-x_i^{\prime }\right)\right)}{\partial x_i}·\frac{1}{m}$$

(33)

where m is the number of steps in the Riemann approximation. The authors experimented and found that a fair approximation is obtained using values in the (20,300) range for m. Chambon et al. employed Integrated Gradients for a transformer-based model to detect COVID-19 presence from radiology reports [164].

A variation of Integrated Gradients called Discretized Integrated Gradients was proposed by Sanyal and Ren in [165]. The variation appears to be more suitable for text embeddings. The difference with respect to the original technique lies in the interpolation between the points x and $x^{\prime }$, which is a linear interpolation in Integrated Gradients, while in Discretized Integrated Gradients, it is a path composed of embedding vectors for existing words, chosen each time according to some distance with respect to the previous vector. Another variation of Integrated Gradients, though never published, has been implemented in the captum (https://github.com/pytorch/captum (accessed on 1 March 2024)) library. It is called Layer Integrated Gradients and is indeed equal to Integrated Gradients, though applied on a specific layer (to be chosen by the user) instead of using just the global output and input.

There also exists a commonly used method called Gradient SHAP, developed by the same authors of SHAP with the aim of extending Integrated Gradients to estimate shapely values. Again, it has never been published, but it is implemented inside the Python shap library (https://github.com/shap/shap (accessed on 1 March 2024)).

Smilkov et al. in [166] presented SmoothGrad, a method to compute saliency maps for images. The relevances are computed as the average over all the gradients computed on a set of input images, each of them created as the original image plus some Gaussian noise.

Several papers employed a variety of explainability techniques including a combination of gradient-based and perturbation-based techniques. Lottridge et al. compared annotations by humans with explanations provided by both LIME and Integrated Gradients in crisis alert identification [137]. Cornia et al. applied three different gradient-based methods (saliency, Guided Backpropagation, and Integrated Gradients), aggregating the scores for different zones in the image [114]. Poulton et al. employed saliency, InputXGradient, Integrated Gradient, Occlusion, and GradientSHAP for automatic short-answer grading [115]. Arashpour et al. employed an even wider choice, including Integrated Gradients, Gradient SHAP, Occlusion, the Fast Gradient Sign Method, Projected Gradient Descent, Minimal Perturbation, and Feature Ablation for waste categorization in images [138]. Neely et al. compared LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, and Deep-SHAP in [139]. Vareille et al. compared SHAP, Grad-CAM, Integrated Gradients, Occlusion, and different variations of them for a multivariate time series analysis [113]. Maladry et al. used LRP-Rollout, Discretized Integrated Gradients, and Layer Integrated Gradients when searching for bias in irony detection [167].

5.5.7. Attention + Gradient + Perturbation

This is the only combination of three techniques for which we have found relevant literature.

Yuan et al. presented a method called Transition Attention Maps (TAMs). This method considers the information flow inside the model as a Markov process, where the attention matrices represent the probabilities of state change. They accumulated attention matrices similarly to Attention Rollout, computing then the Integrated Gradients’ relevance with respect to the last attention block and finally obtaining the relevance map as the product of the accumulated attention and the IG relevance map [168].

Chen et al. proposed a method (subsequently) called Bidirectional Transformers (BTs) that produces a saliency map by the element-wise multiplication of two matrices: the relevance matrix obtained by applying Integrated Gradients to the last attention block and a matrix obtained similarly to Attention Rollout but after rescaling the columns of each attention matrix by using rescaling factors obtained as a function of each module’s output [169].

5.6. Overall Analysis

So far, we have described each technique according to the classification put forward at the beginning of Section 5. In this section, we analyze the relative extension and relevance of those techniques.

In

Table 1. References for the initial proposal of major techniques.

Class	References
Response	LRP [38], Partial-LRP [40], RAP [42], CLRP [43], CAM [44], ALTI [45], Input Erasure [46], CAV [47], Eigen-CAM [48]
Attention	Attention Rollout [61], Attention Flow [61]
Gradient	Saliency [100], Guided Backpropagation [101], InputXGradient [102], Contrastive Gradient Norm [103], Contrastive InputXGradient [103], Grad-CAM [104], Grad-CAM++ [105]
Perturbation	Occlusion [117], LIME [118], SHAP [119], Anchors [121], RISE [122], Soundness Saliency [123], AblationCAM [124]
Response + Attention	LRP-Rollout [41]
Response + Gradient	AGF [147], SGLRP [148], FullGrad [149], Eigen-Grad-CAM
Response + Perturbation	DeepLIFT [151], DeepSHAP [119], Score-CAM [116], ViT-CX [152]
Attention + Gradient	AttCAT [155], TMME [156], AGrad [159], RePAGrad [159]
Gradient + Perturbation	Integrated Gradients [163], Discretized Integrated Gradients [165], Layer Integrated Gradients, Gradient SHAP, SmoothGrad [166]
Attention + Gradient + Perturbation	TAM [168], BT [169]

, we report the major references for the techniques surveyed in Section 5.1, Section 5.2, Section 5.3, Section 5.4, Section 5.5, Section 5.5.1, Section 5.5.2, Section 5.5.3, Section 5.5.4, Section 5.5.5, Section 5.5.6 and Section 5.5.7. The references listed in Table 1 concern those papers where the technique was proposed first. As we mentioned earlier, it may well happen that the technique was proposed first in one context and later applied to transformers, possibly with slight modifications.

The papers specifically dealing with transformers are instead shown in

Table 2. Classification of papers on XAI for transformers.

Class	References
Response	[49,50,51,52,53,54,55,56,57,58,59,60]
Attention	[50,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99]
Gradient	[57,58,81,95,106,107,108,109,110,111,112,113,114,115]
Perturbation	[49,82,83,98,111,112,113,115,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143]
Response + Attention	[41,82,144,145,146]
Response + Gradient	[54,57,150]
Response + Perturbation	[139,153,154]
Attention + Gradient	[56,110,140,146,156,157,158,159,160]
Gradient + Perturbation	[113,114,115,137,138,139,164,167]
Attention + Perturbation	[161,162]

, again listed by the type of techniques they employ. The relative size of each category is shown in the heatmap of

Table 3. Number of papers by technique class.

	Attention	Gradient	Perturbation	Activation
Attention	38	10	2	5
Gradient	10	14	8	3
Perturbation	2	8	27	3
Activation	5	3	3	12

. Cells outside the main diagonal account for the combinations of two techniques. We see the dominance of attention-based techniques, followed by perturbation-based techniques. Hybrid methods are quite less represented.

However, if we turn our attention to the importance rather than sheer size, as embodied by the number of citations, the situation changes. As can be seen in the heatmap of

Table 4. Citations by technique class.

	Attention	Gradient	Perturbation	Activation
Attention	123	85	1	243
Gradient	85	57	26	1
Perturbation	1	26	51	2
Activation	243	1	2	42

, the dominant class is a hybrid one, namely Activation+Attention techniques, which have nearly twice as many citations as sheer Attention-based techniques. Another hybrid technique, Attention+Gradient, ranks third. Anyway, all the top three techniques include attention, either alone or as one of the components.

Finally, if we look at the papers proposing methodological innovation in

Table 5. Number of papers presenting new methods by category.

	Attention	Gradient	Perturbation	Activation
Attention	4	7	2	3
Gradient	7	2	1	1
Perturbation	2	1	2	2
Activation	3	1	2	5

, we find attention-based and activation-based methods at the top among the single-class methods, but the joint use of gradient and attention is attracting more research efforts overall.

6. Discussion and Conclusions

Our survey has highlighted an explosion of papers concerning the explainability of transformers. This trend accompanies the development of transformers’ applications. As their domain of application expands into image analysis from the original NLP context, the development of explainability methods follows suit. As transformer models are increasingly applied to multi-modal tasks (involving text, images, and audio), research should extend to develop and assess explainability methods that can handle the intricacies of multiple data types. Understanding how transformers integrate and prioritize information across modes is essential for comprehensive explanations. Also, as transformers continue to grow in size, developing explainability methods that can scale efficiently with model complexity will be crucial. A trade-off will have to be sought between fast and computationally efficient explanations and depth or accuracy, especially for real-time applications.

The favorite class of methods, as measured by the number of papers, still appears to be attention-based. That would appear as a natural choice since the introduction of attention layers has been a major feature of transformers. However, the literature itself warns that attention is not a synonym for explainability. If we look at citations as a measure of interest instead of the sheer number of papers, methods based on attention only are overcome by methods based on both attention and activation, i.e., that look at the output of neurons tracing the relevance of features from the network output back to its input. Also, if we just consider papers proposing methodological innovations, the joint use of gradient and attention is the topic appearing most in those papers.

Another relevant trend is the adoption of visualization tools to help explain the relevance of features. Though they do not introduce a real innovation in detecting the most relevant features, visual tools represent a step forward in the communication of explainability results to the layman or the professional in charge of using ML tools. In particular, developing interactive visualization tools that allow users to query, manipulate, and explore model decisions in real time could significantly enhance understanding. This is particularly true in the medical field, where ML tools are heavily used to support doctors in diagnosis tasks. Also, incorporating feedback from end-users, especially those without technical expertise, could guide the development of more user-friendly and accessible explainability methods. User studies and participatory design processes could help identify the most useful explanation approaches according to the context of usage.

Unfortunately, the diffusion of explainability methods and their introduction in ready-to-use libraries (e.g., in Python) may make it easier to apply such tools as black boxes without a clear awareness of their preferred field of application or limitations. In the long term, this behavior may lead to wrongly feeling self-confident and excessively relying on the method’s output, though it may not be appropriate.

An additional concern is the relative lack of literature on evaluating explainability methods. There is a need for standardized, objective metrics to evaluate the effectiveness of explainability methods. These metrics should assess not just the accuracy of explanations but also their comprehensibility to humans. Developing benchmarks that can compare different explainability approaches on a common ground would enable more direct comparisons and facilitate progress. Incorporating knowledge from fields such as cognitive science, psychology, and philosophy could also offer new perspectives on what constitutes a good explanation and how to evaluate the explainability of AI models from a human-centric viewpoint, aiming at explainability methods that are deeply aligned with human cognition and understanding and also recognize ethical and fairness issues. This topic is certainly the most compelling to investigate if we wish to identify the most effective ones.