No Fine-Tuning, No Cry: Robust SVD for Compressing Deep Networks

Abstract

A common technique for compressing a neural network is to compute the k-rank ${\ell }_2$ approximation $A_k$ of the matrix $A\in {\mathbb{R}}^{n\times d}$ via SVD that corresponds to a fully connected layer (or embedding layer). Here, d is the number of input neurons in the layer, n is the number in the next one, and $A_k$ is stored in $O\left(\right(n+d\left)k\right)$ memory instead of $O\left(nd\right)$. Then, a fine-tuning step is used to improve this initial compression. However, end users may not have the required computation resources, time, or budget to run this fine-tuning stage. Furthermore, the original training set may not be available. In this paper, we provide an algorithm for compressing neural networks using a similar initial compression time (to common techniques) but without the fine-tuning step. The main idea is replacing the k-rank ${\ell }_2$ approximation with ${\ell }_p$, for $p\in [1,2]$, which is known to be less sensitive to outliers but much harder to compute. Our main technical result is a practical and provable approximation algorithm to compute it for any $p\ge 1$, based on modern techniques in computational geometry. Extensive experimental results on the GLUE benchmark for compressing the networks BERT, DistilBERT, XLNet, and RoBERTa confirm this theoretical advantage.

1. Introduction

Deep learning revolutionized machine learning by improving the accuracy by dozens of percents for fundamental tasks in natural language processing (NLP), speech/image recognition, etc. One of the disadvantages of deep learning is that in many cases, the classifier is extremely large compared to classical machine learning models. A large network usually requires expensive and stronger resources due to: (1) slower classification time, which may be a serious limitation, especially in real-time systems such as autonomous cars or real-time text/speech translations; (2) a large memory requirement, which makes it infeasible to store the network on RAM or on a device such as IoT/smartphones; and (3) high energy consumption which is related to the CPU/GPU time of each classification and requires larger batteries with shorter lifespan.

Pipeline of network compression. Given training data P, a common pipeline to obtain a compressed network consists of the following stages:

(i)

Train a network N based on the training set P, starting from an initial random network.

(ii)

Compress the network N to a small network $\tilde{N}$. The input P may be not involved in this stage.

(iii)

Fine-tune the weights of $\tilde{N}$ by training it on P. This step aims to improve the accuracy of the network $\tilde{N}$ but does not change its size.

In this paper, our goal is to improve the compression step (ii) in order to avoid the fine-tuning step (iii) via suggesting a better and more robust compressing scheme. We suggest a novel low rank factorization technique for compressing an embedding layer of a given NLP model. This is motivated by the fact that in many networks, the embedding layer accounts for 20–40% of the network size. Indeed, the results are easily extended to fully connected layers.

1.1. Embedding Matrix

One of the most common approaches for compressing neural networks is to treat a layer in the network as a matrix operation and then to approximate this matrix by its compressed version. This is especially relevant in a fully connected layer. Specifically, in word embedding, this layer is called the embedding layer, which is defined by the following matrix.

The input of the embedding layer consists of d input neurons, and the output has n neurons. The $nd$ edges between these layers define a matrix $A\in {ℝ}^{n\times d}$. Here, the entry $A_{i,j}$ in the ith row and jth column of A is equal to the weight of the edge between the jth input neuron to the ith output neuron. Suppose that a test sample (vector) $x\in {ℝ}^d$ is received as an input. The corresponding output n-dimensional vector is thus $y=Ax$. To simply, a column from A during training is read, and a standard vector x (a column of the identity matrix) is used and is called a one-hot vector.

${\ell }_2$ k-rank approximation. One of the natural and common matrix approximations, including in the context of network compression, is the ${\ell }_2$ k-rank approximation (see, e.g., [1,2] and references therein). This is the matrix which minimizes the Frobenius norm, i.e., the sum of squared distances ${\parallel A-A_k\parallel }_F^2:={\sum }_{i=1}^n{\parallel A^{\left(i\right)}-A_k^{\left(i\right)}\parallel }_2^2$ between the ith row $A^{\left(i\right)}$ in A and its corresponding row $A_k^{\left(i\right)}$ in $A_k$, over every rank k matrix $A_k$. It can be easily computed via the singular value decomposition (SVD) in $O(min\{n{d}^2,d{n}^2\})$ time. Although $A_k$ has the same size as A, due to its low rank, it can be factorized as $A_k=UW$, where $U\in {ℝ}^{n\times k}$ and $W\in {ℝ}^{k\times d}$. We can then replace the original embedding layer that corresponds to A by a pair of layers that correspond to U and W, which can be stored using $O\left(k\right(n+d\left)\right)$ memory, compared to the $O\left(nd\right)$ entries in A. Moreover, the computation of the output $y_k:=A_{k}x$ takes $O\left(k\right(n+d\left)\right)$ time, compared to the $O\left(nd\right)$ time that it takes to compute $Ax$.

Handling other linear layers. The rank approximation technique can be also applied to a fully connected layer, where an activation function $f:{ℝ}^n\to ℝ$ is applied on the output $Ax$ or each of its coordinates (as Relu) to obtain $f\left(Ax\right)$. By approximating A, in a sense, $f\left(Ax\right)$ is also approximated by $f\left(A_{k}x\right)$. Then, $A_k$ is replaced by two smaller layers $U,W$, as explained above. Furthermore, it is known that convolutional layers (tensors) can be viewed as fully connected layers (matrix multiplication) applied to reshaped volumes of the input. Then, one can approximate the convolutional weights by approximating its corresponding weight matrix. Hence, the rank approximation technique can be also applied to a convolutional layer.

1.2. Motivation

In what follows, we explain the main motivation of this paper, which in sum, aims to eliminate the need for the fine-tuning step due to the reasons explained in Section 1.2.1. We also discuss the weaknesses of the known SVD factorization in Section 1.2.2, which in turn, give rise to the motivation behind our approach discussed in Section 1.2.3.

1.2.1. Fine-Tuning

The layers that correspond to the matrices U and W above are usually used only as initial seeds for a training process that is called fine-tuning, where the aim is to improve the initial results. Here, the training data are fed into the network, and as opposed to the ${\ell }_2$ error, the error is measured with respect to the final classification, i.e., in the fine-tuning step, the compressed network $\tilde{N}$ is trained using the input P, similar to Step (i). The goal of this step is to improve the accuracy of the compressed network without increasing its size. Hence, the structure of the data remains the same, but the edges are updated in each iteration.

To be or not to be fine-tuned? Fine-tuning is a necessary step to recover the generalization ability damaged by the model compression. Despite its widespread use, fine-tuning is vaguely understood, e.g., what fraction of the pre-trained weights are actually changing and why? [3].

In many cases, the fine-tuning cannot be applied:

• The original (large) training set is not necessarily available for us (e.g., for sake of data privacy) to apply the fine-tuning after compressing the model.
• For large datasets and complex tasks, the fine-tuning process takes a very long time and requires strong resources [4,5], even on the pruned networks. Hence, due to the limited computational power (and/or insufficient training time) of the end user device (e.g., smartphone, IoT), fine-tuning is not a viable choice.
• In the context of NLP, it is common to learn representations of natural language [6,7,8,9] via full-network pre-training followed by fine-tuning on a small dataset for the sake of learning a new task [10,11,12]. However such pre-trained models are very large. Thus, a natural coping mechanism would involve compression before the fine-tuning. After the compression, the model suffers from loss in its original learning capability, and unfortunately, the fine-tuning process is not sufficient to both retain the model’s quality and make the network learn a new task, since we may not be able to obtain enough tagged information that we can rely on to perform meaningful training from scratch, e.g., when compressing the embedding layer, we may lose the richness of the vocabulary, as it is responsible for representing each word from a vocabulary by a vector that reflects its semantic and syntactic information which can be extracted from the language.

Hence, some have attempted to prune each layer independently, by which a fine-tuning process can be done with a small number of epochs to avoid the excessive computational power required by the fine-tuning process [5]. Finally, it is worth mentioning that the fine-tuned parameters are not constrained to share any components with the pre-trained weights and thus are equally expensive to store and to compute per iteration [13].

In this paper, we replace the go-to method for compression models using matrix factorization by a more robust low rank approximation scheme, where the emphasis here is that the learning capability of the model after the compression is less affected.

1.2.2. Should We Use SVD?

Training the network and compressing it are natural steps. However, it is not clear that the last fine-tuning step, which may be a serious time consumer, is necessary. The goal of this work is to remove this step by improving the previous (compression) step via more involved algorithms that provably approximate the more robust ${\ell }_p$ rank approximation. We begin with geometric intuition.

The geometry behind SVD. Geometrically, each row of A corresponds to a d-dimensional vector (point) in ${ℝ}^d$, and the corresponding row in $A_k$ is its projection on a k-dimensional subspace of ${ℝ}^d$. This subspace (which is the column space of U) minimizes the sum of squared distances to the rows of A over every k-subspace in ${ℝ}^d$.

Statistically, if these n points were generated by adding a Gaussian noise to a set of n points on a k-dimensional subspace, then it is easy to prove that most likely (in the sense of maximum-likelihood) this subspace is U. The disadvantage of ${\ell }_2$ k-rank approximation is that it is optimal under the above statistical assumption, which rarely seems to be the case for most applications. In particular, minimizing the sum of squared distances is heavily sensitive to outliers [14] (see Figure 1). As explained in [15], this is the result of squaring each term, which effectively weights large errors more heavily than small ones.

This undesirable property, in many applications, has led researchers to use alternatives such as the mean absolute error (MAD), which minimizes the ${\ell }_1$ (sum of distances) of the error vector. For example, compressed sensing [16] uses ${\ell }_1$ approximation as its main tool to clean corrupted data [17] as well as to obtain sparsified embeddings with provable guarantees as explained, e.g., in [18].

In machine learning, the ${\ell }_1$-approximation replaces or is combined with the ${\ell }_2$ approximation. Examples in scikit-learn include lasso regression, elastic-nets, or MAD error in decision trees [19].

1.2.3. Novel Approach and Its Challenges

Novel approach: deep learning meets subspace approximation. We suggest generalizing the above ${\ell }_2$ approximation to ${\ell }_1$ k-rank approximation, or even ${\ell }_p$ approximation for more general $p<2$. Geometrically, we wish to compute the k-subspace that minimizes the sum of pth power of the distances to the given set of n points. This should result in more accurate compressed networks that are more robust to outliers and classification mistakes.

Unlike the case $p=2$, which was solved more than a century ago [20] via SVD and its variants, the ${\ell }_p$ low rank approximation was recently proved to be NP-hard even to approximate up to a factor of $1+1/\mathrm{poly}\left(d\right)$ (recall that d is the number of columns of A above) for $p\in [1,2)$ [21] and even for general (including constant) values of p (see Section 2). In the most recent decade, there was great progress in this area; however, the algorithms were either based on ad hoc heuristics with no provable bounds or impractical, i.e., their running time is exponential in k [21,22], and their efficiency in practice is not clear. Indeed, we could not find implementations of such provable algorithms.

This motivates the questions that are answered affirmably in this paper: (i) Can we efficiently compute the corresponding ${\ell }_p$ k-rank approximation matrix $A_k$, similar to SVD? (ii) Can we remove the fine-tuning step by using the ${\ell }_p$ low rank approximation, while scarifying only a small decrease in the accuracy of the compressed network? (iii) Can we obtain smaller networks with higher accuracy (without fine-tuning) by minimizing the sum of non-squared errors, or any other power $p\ne 2$ of distances, instead of the ${\ell }_2$ k-rank approximation via SVD?

1.3. Our Contribution

We answer these questions by suggesting the following contributions:

• A new approach for compressing networks based on ${\ell }_p$ k-rank approximation instead of ${\ell }_2$, for $p\in [1,\infty )$. The main motivation is the robustness to outliers and noise, which is supported by many theoretical justifications.
• Provable algorithms for computing this ${\ell }_p$ low rank approximation of every $n\times d$ matrix A. The deterministic version takes time $O(n{d}^{3}logn)$, and the randomized version takes $O(ndlogn)$. The approximation factor depends polynomially on d, is independent of n for the deterministic version, and is only poly-logarithmic in n for the randomized version.
• Experimental results confirming that our approach significantly improves existing results when the fine-tuning step is removed from the pipeline upon using SVD (see Section 5).
• Full open source code is provided [23].

Our results are based on a novel combination of modern techniques in computational geometry and applied deep learning. We expect that future papers will extend this approach (see Section 7).

To obtain efficient implementations with provable guarantees, we suggest a leeway by allowing the approximation factor to be larger than k, instead of aiming for ($1+\epsilon )$-approximation (PTAS). In practice, this worst-case bound seems to be too pessimistic, and the empirical approximation error in our experiments is much smaller. This phenomenon is common in approximation algorithms, especially in deep learning, when the dataset has a lot of structure and is very different from synthetic worse-case artificial examples. The main mathematical tool that we use is the Löwner ellipsoid, which generalizes the SVD case to general ${\ell }_p$ cases, inspired by many papers in the related work below.

To be part and not apart. Our technique can be combined with previous known works to obtain better compression. For example, DistilBERT [24] is based on knowledge distillation, and it reduces the size of the BERT [12] model by $40\%$, while maintaining $97\%$ of its language understanding capabilities and being $60\%$ faster. However, this result does not use low rank factorization to compress the embedding layer. We further compressed DistilBERT and achieved better accuracy than SVD.

2. Related Work

In the context of training giant models, some interesting approaches were suggested to reduce the memory requirement, e.g., [25,26]. However, those methods reduced the memory requirement at the cost of speed/performance. Later, [27] proposed a way to train large models based on parallelization. Here, the model size and evaluation speed are also still an obstacle. Hence, many papers were dedicated to the purpose of compressing neural networks in the field of NLP. These papers are based on different approaches such as pruning [28,29,30,31,32], quantization [33,34], knowledge distillation [24,35,36,37,38,39,40,41], weight sharing [42], and low rank factorization [42,43,44] (see the example table in [45] for compressing the BERT model). There is no convention for which approach from the above should be used. However, recent works, e.g., [42], showed that combining such approaches yields good results.

Subspace approximation. The ${\ell }_2$ k-rank approximation can be solved easily in $min\{n{d}^2,d^{2}n\}$ time, while a $(1+\epsilon )$ approximation can be computed deterministically in $nd{(k/\epsilon )}^{O\left(1\right)}$ time [46] for every $\epsilon >0$, and a randomized version takes $O\mathrm{nnz}\left(A\right)+(n+d)·{(k/\epsilon )}^{O\left(1\right)}$ time, where nnz(A) is the number of non-zero entries in A [47,48,49]. These and many of the following results are summarized in the seminal work of [21]. However, for $p\ne 2$, even computing a multiplicative $(1+\epsilon )$-approximation is NP-hard when k is part of the input [21]. Nevertheless, it is an active research area, where techniques from computational geometry are frequently used. The case $p\ge 1$ was introduced in the theory community by [50], and earlier, the case $p=1$ was introduced in the machine learning community by [51]. In [50], a randomized algorithm for any $p\ge 1$ that runs in time $nd{2}^{{(k/\epsilon )}^{O\left(p\right)}}$ was suggested. The state of the art for $p\in [1,2)$ in [21] takes $O\left(\mathrm{nnz}\left(A\right)+(n+d){(k/\epsilon )}^{O\left(1\right)}+2^{\left({(k/\epsilon )}^{O\left(1\right)}\right)}\right)$ time.

Approximation algorithms for the ${\ell }_p$ low rank approximation were suggested in [52] for any $p\ge 1$, which we also handle. Although the obtained approximation, in some cases, is smaller than the approximation achieved in this paper, the running time in most cases (depending on k) is much larger than that of ours.

Regardless of the approximation, [52] suggests a polynomial time algorithm (one of many) as long as $k\in \Theta \left(\frac{logn}{loglogn}\right)$. Similar to the discussion in [52], our ${\ell }_1$ low rank approximation allows us to recover an approximating matrix of any chosen rank, while the robust PCA [53] returns some matrix of unknown rank. Although variants of robust PCA have been proposed to force the output rank to be a given value [54,55], these variants make assumptions about the input matrix, whereas our results do not. The time complexity for $p=1$ was improved in [56] to $nd{(k/\epsilon )}^{O\left(1\right)}+(n+d)2^{{(k/\epsilon )}^{O\left(1\right)}}$, and later, for general p to $nd{(k/\epsilon )}^{O\left(1\right)}+2^{{(k/\epsilon )}^{O\left(p\right)}}$ [22]. The latter work, together with [57], also gives a coreset for subspace approximation, i.e., a way of reducing the number of rows of A so as to obtain a matrix $A^{\prime }$ such that the cost of fitting the rows of $A^{\prime }$ to any k-dimensional subspace F is within a $1+\epsilon$ factor of the cost of fitting the rows of A to F; for $p=2$, such coresets were known [47,58,59,60] and can be computed exactly $(\epsilon =0)$ [61,62].

Efficient approximations. The exponential dependency on k and hardness results may explain why we could not find (even inefficient) open or closed code implementations on the web. To our knowledge, it is an open problem to compute larger factor approximations ($\epsilon \in O\left(1\right)$) in a time polynomial in k, even in theory. The goal of this paper is to provide such a provable approximation in time that is near-linear in n with practical implementation and to demonstrate our usefulness in compressed networks.

3. Method

Notations. For a pair of integers $n,d\ge 1$, we denote by ${ℝ}^{n\times d}$ the set of all $n\times d$ real matrices, by $I_d\in {\{0,1\}}^{d\times d}$ the identity matrix, and $\left[n\right]=\{1,\cdots ,n\}$. For a vector $x\in {ℝ}^d$, a matrix $A\in {ℝ}^{n\times d}$, and a real number $p>0$, the pth norm of x is defined as ${\parallel x\parallel }_p={\left({\sum }_{i=1}^d{\left|x_i\right|}^p\right)}^{1/p}$, and the ${\ell }_p$ entry-wise norm of A is defined as ${\parallel A\parallel }_{p,p}={\left({\sum }_{i=1}^d{\parallel A{e}_i\parallel }_p^p\right)}^{1/p}$, where $e_i\in {\{0,1\}}^d$ is a vector whose ith entry is 1 and 0 elsewhere. We say that the columns of a matrix $A\in {ℝ}^{n\times d}$ (where $n\ge d$) are orthogonal if $A^{T}A=I_d$. In addition, a matrix $F\in {ℝ}^{d\times d}$ is called positive definite matrix if F is a symmetric matrix, and for every $x\in {ℝ}^d$ such that ${\parallel x\parallel }_2>0$, we have $x^{T}Fx>0$. Furthermore, we say that a set $L\subseteq {ℝ}^d$ is centrally symmetric if for every $x\in L$, it holds that $-x\in L$. Finally, a set $L\subseteq {ℝ}^d$ is called a convex set if for every $x,y\in L$ and $\theta \in [0,1]$, $\theta x+(1-\theta )y\in L$.

3.1. ${\parallel ·\parallel }_p$-SVD Factorization and the Löwner Ellipsoid

In what follows, we intuitively and formally describe the tools that will be used in our approach. Definition 1 is based on Definition 4 in [63]. While the latter defines a generic factorization for a wide family of functions, Definition 1 focuses on our case, i.e., the function we wish to factorize is ${\parallel Ax\parallel }_p$ for any $p\ge 1$, where $A\in {ℝ}^{n\times d}$ is the input matrix, and x is any vector in ${ℝ}^d$.

Definition 1

(Variant of Definition 4 [63]). Let $A\in {ℝ}^{n\times d}$ be a matrix of rank d, and let $p\ge 1$ be a real number. Suppose that there is a diagonal matrix $D\in {(0,\infty )}^{d\times d}$ of rank d, and an orthogonal matrix $V\in {ℝ}^{d\times d}$, such that for every $x\in {ℝ}^d$,

$${\parallel D{V}^{T}x\parallel }_2^p\le {\parallel Ax\parallel }_p^p\le d^{\frac{p}{2}}{\parallel D{V}^{T}x\parallel }_2^p.$$

Define $U=A{\left(D{V}^T\right)}^{-1}$. Then, $UD{V}^T=A$ is called the ${\parallel ·\parallel }_p$-SVD of A.

Why ${\parallel ·\parallel }_p$-SVD? The idea behind using the ${\parallel ·\parallel }_p$-SVD factorization of an input matrix A is that we obtain a way to approximate the span of the column space of $A\in {ℝ}^{n\times d}$. This allows us to approximate the dot product $Ax$ for any $x\in {ℝ}^d$, which implies an approximation for the optimal solution of the ${\ell }_p$ low rank approximation problem.

For example, in the case of $p=2$, the ${\parallel ·\parallel }_2$-SVD of a matrix $A\in {ℝ}^{n\times d}$ is equivalent to the known SVD factorization $A=UD{V}^T$. This holds due to the fact that the columns of the matrix U are orthogonal, and for every $x\in {ℝ}^d$, we have ${\parallel Ax\parallel }_2^2={\parallel UD{V}^{T}x\parallel }_2^2={\parallel D{V}^{T}x\parallel }_2^2.$ As for the general case of any $p\ge 1$, [63] showed that the ${\parallel ·\parallel }_p$-SVD factorization always exists, and can be obtained using the Löwner ellipsoid.

Theorem 2

(Variant of Theorem III [64]). Let $D\in {[0,\infty )}^{d\times d}$ be a diagonal matrix of full rank and an orthogonal matrix $V\in {ℝ}^{d\times d}$, and let E be an ellipsoid defined as $E=\{x\in {ℝ}^d|x^{T}V{D}^{T}D{V}^{T}x\le 1\}.$

Let L be a centrally symmetric compact convex set. Then, there exists a unique ellipsoid E called theLöwner ellipsoidof L such that $1/\sqrt{d}E\subseteq L\subseteq E$, where $1/\sqrt{d}E=\{1/\sqrt{d}x|x\in E\}$.

Computing ${\parallel ·\parallel }_p$-SVD via Löwner ellipsoid. Intuitively speaking, for an input matrix $A\in {ℝ}^{n\times d}$, the ${\parallel ·\parallel }_p$-SVD $A=UD{V}^T$ aims to bound from above and below the cost function ${\parallel Ax\parallel }_p^p$ for any $x\in {ℝ}^d$ by the term ${\parallel D{V}^{T}x\parallel }_2^p$. Since ${\parallel Ax\parallel }_p^p$ is a convex continuous function (for every $x\in {ℝ}^d$), the level set $L=\{x\in {ℝ}^d|{\parallel Ax\parallel }_p\le 1\}$ is also convex. Having a convex set enables us to use the Löwner ellipsoid, which, in short, is the minimum volume enclosing ellipsoid of L. In addition, contracting the Löwner ellipsoid by $\sqrt{d}$ yields an inscribed ellipsoid in L. It turns out that $D,V$ of the ${\parallel ·\parallel }_p$-SVD represents the Löwner ellipsoid of L as follows: D is a diagonal matrix such that its diagonal entries contain the reciprocal values of the ellipsoid axis lengths, and V is an orthogonal matrix which is the basis of the same ellipsoid. Using the enclosing and inscribed ellipsoids (the Löwner ellipsoid and its contracted form) enables us to bound ${\parallel ·\parallel }_p$ using the mahalonobis distance. Although in traditional k${\ell }_2$-low rank factorization with respect to an input matrix $A\in {ℝ}^{n\times d}$, the optimal result is equal to the sum of the smallest $d-k$ singular values, we generalize this concept to ${\ell }_p$-low rank factorization. Specifically, the singular values of D (the reciprocal values of the ellipsoid axis lengths) serve as a bound on the “${\ell }_p$” singular values of A.

3.2. Additive Approximation for the ${\ell }_p$-Low Rank Factorization

In what follows, we show how to compute an approximated solution for the ${\ell }_p$-low rank factorization for any $p\ge 1$ (see Algorithm 1). This is based on the ${\parallel ·\parallel }_p$-SVD factorization (see Definition 1).

From ${\parallel ·\parallel }_p$-SVD to ${\ell }_p$-low rank factorization. For any $k\in [d-1]$ and any matrix $A\in {ℝ}^{n\times d}$ of rank d, the ${\ell }_p$-low rank factorization problem aims to minimize ${\parallel A-AX{X}^T\parallel }_{p,p}^p$ over every matrix $X\in {ℝ}^{d\times k}$ whose columns are orthogonal. As a byproduct of the orthogonality of X, the problem above is equivalent to minimizing ${\parallel AY{Y}^T\parallel }_{p,p}^p$ over every matrix $Y\in {ℝ}^{d\times (d-k)}$ whose columns are orthogonal such that $Y{Y}^T=I_d-X{X}^T$. By exploiting the definition of the entry-wise ${\ell }_p$ norm of $AY{Y}^T$, we can use ${\parallel ·\parallel }_p$-SVD to bound this term from above and below using the mahalonobis distance. Furthermore, we will show that by using the ${\parallel ·\parallel }_p$-SVD, we can compute a matrix $A_k$ of rank k such that ${\parallel A-A_k\parallel }_{p,p}^p$ depends on the ellipsoid axis lengths (see Algorithm 1 and Theorem 5).

Overview of Algorithm 1. Algorithm 1 receives as input a matrix $A\in {ℝ}^{n\times d}$ of rank d, a positive integer $k\in [d-1]$, and a positive number $p\ge 1$ and outputs a matrix $A_k$ of rank k, which satisfies Theorem 5. At Line 1, we compute a pair of matrices $D,V\in {ℝ}^{d\times d}$ such that the ellipsoid $E:=\{x\in {ℝ}^d|x^{T}V{D}^{T}D{V}^{T}x\le 1\}$ is the Löwner ellipsoid of $L:=\{x\in {ℝ}^d|{\parallel Ax\parallel }_p\le 1\}$, where D is a diagonal matrix of rank d, and V is an orthogonal matrix; we refer the reader to the Appendix A for computing the Löwner ellipsoid. At Line 2, we compute the matrix U from the ${\parallel ·\parallel }_p$-SVD of A (see Definition 1). At Lines 3–4, we set $D_k$ to be the diagonal matrix of $d\times d$ entries where the first k diagonal entries are identical to the first k diagonal entries of D, while the rest of the matrix is set to 0 (see Figure 2 for an illustrative description of our algorithm).

Algorithm 1:ℓρ-LOW-RANK (A, k, p)

Input : A matrix $A\in {ℝ}^{n\times d}$ of rank d, $p\ge 1$, a positive integer $k\in [d-1]$, and a positive real number $p\ge 1$.  Output : A matrix $U\in {ℝ}^{n\times d}$, a diagonal matrix $D_k\in {[0,\infty )}^{d\times d}$, an orthogonal matrix $V\in {ℝ}^{d\times d}$ where $U,V$ are from the ${\parallel ·\parallel }_p$-SVD of A, and a set of d positive real numbers $\{{\sigma }_1,\dots ,{\sigma }_d\}$. $(D,V):=\mathrm{L}\mathsf{Ö}\mathrm{WNER}(A,p)$ //$\phantom{(}$See Algorithm A1 at the Appendix A $U:=A{\left(D{V}^T\right)}^{-1}$ //$\phantom{(}$computing U from the ${\parallel ·\parallel }_p$-SVD of A with respect to the ${\ell }_p$-regression problem $\{{\sigma }_1,\dots ,{\sigma }_d\}:=$ the diagonal entries of D $D_k:=\mathrm{diag}({\sigma }_1,\dots ,{\sigma }_k,0,\dots ,0)$ //$\phantom{(}$A diagonal matrix in ${ℝ}^{d\times d}$ return $U,D_k,V,\{{\sigma }_1,\dots ,{\sigma }_d\}$

4. Analysis

Some of the proofs in this section were moved into the Supplementary Material due to space limitations.

4.1. Deterministic Result

In what follows, we present our deterministic solution for the ${\ell }_p$-low rank factorization problem.

Claim 3.

Let$D\in {[0,\infty )}^{d\times d}$be a diagonal matrix of rankd, and let$\sigma >0$be the lowest singular value ofD. Then, for every unit vector$x\in {ℝ}^d$,${\parallel Dx\parallel }_2\ge \sigma .$

Proof. 

Let $x\in {ℝ}^d$ be a unit vector, and for every $i\in \left[d\right]$, let $D_{i,i}$ denote the ith diagonal entry of D, and $x_i$ denotes the ith entry of x. Observe that

$$\begin{array}{c}\hfill {\parallel Dx\parallel }_2={\left(\sum _{i=1}^d{\left|D_{i,i}{x}_i\right|}^2\right)}^{\frac{1}{2}}\ge {\left(\sum _{i=1}^d{\left|\sigma x_i\right|}^2\right)}^{\frac{1}{2}}=\sigma {\parallel x\parallel }_2=\sigma ,\end{array}$$

where the first equality follows from the definition of norm, the inequality holds by definition of $\sigma$, and the last equality holds since x is a unit vector.    □

Lemma 4

(Special case of Lemma 15 [63] Let $A\in {ℝ}^{n\times d}$ be a matrix of full rank, $p\ge 1$. Then, there exist a diagonal matrix $D\in {[0,\infty )}^{d\times d}$ of full rank and an orthogonal matrix $V\in {ℝ}^{d\times d}$ such that for every $x\in {ℝ}^d$,

$${\parallel D{V}^{T}x\parallel }_2^p\le {\parallel Ax\parallel }_p^p\le d^{\frac{p}{2}}{\parallel D{V}^{T}x\parallel }_2^p$$

(1)

Proof. 

First, let $L=\left\{\tilde{x}\in {ℝ}^d|{A\tilde{x}}_p^p\le 1\right\}$, and put $x\in {ℝ}^d$. Observe that (i) since $p\ge 1$, the term ${\parallel A\tilde{x}\parallel }_p$ is a convex function for every $\tilde{x}\in {ℝ}^d$ which follows from properties of norm function. This means that the level set L is a convex set. In addition, (ii) by definition of L, it holds that for every $\tilde{x}\in L$, also $-\tilde{x}\in L$, which makes L a centrally symmetric set by definition. Note that (iii) since A is of full rank, then L spans ${ℝ}^d$.

Since properties (i)–(iii) hold, we obtain by Theorem 1 that there exists a diagonal matrix $D\in {[0,\infty )}^{d\times d}$ of full rank and an orthogonal matrix $V\in {ℝ}^d$ such that the set $E=\{\tilde{x}\in {ℝ}^d|{\tilde{x}}^{T}V{D}^{T}D{V}^T\tilde{x}\le 1\}$ satisfies

$$\frac{1}{\sqrt{d}}E\subseteq L\subseteq E.$$

(2)

Proving the right hand side of Equation (1). Let $y=\frac{1}{{\parallel D{V}^{T}x\parallel }_2}x$, and observe that

$$\begin{array}{c}\hfill {\parallel Ax\parallel }_p^p={\left(\sqrt{d}{\parallel D{V}^{T}x\parallel }_2\right)}^p{\parallel \frac{1}{\sqrt{d}}Ay\parallel }_p^p\le d^{\frac{p}{2}}{\parallel D{V}^{T}x\parallel }_2^p,\end{array}$$

(3)

where the equality follows from the definition of y, and the inequality holds since $\frac{1}{\sqrt{d}}y\in L$ follows from Equation (2).

Proving the left hand side of Equation (1). Since L spans ${ℝ}^d$, there then exists $b>0$ such that ${\parallel A\left(bx\right)\parallel }_p^p=1$. By Equation (2), $bx\in E$, which results in ${\parallel D{V}^{T}x\parallel }_2=\frac{1}{b}{\parallel D{V}^{T}bx\parallel }_2\le \frac{1}{b}$. Thus,

$${\parallel Ax\parallel }_p^p=\frac{1}{b^p}{\parallel Abx\parallel }_p^p=\frac{1}{b^p}\ge {\parallel D{V}^{T}x\parallel }_2^p$$

(4)

Since Equation (3) and Equation (4) hold for every $x\in {ℝ}^d$, Lemma 4 follows.    □

Theorem 5.

Let $A\in {ℝ}^{n\times d}$ be real matrix, $p\ge 1$; $k\in [d-1]$ be an integer; and ($U,D_k,V,{\sigma }_1,\dots {\sigma }_d$) be the output of a call to ℓ_ρ-Low-rank($A,k,p$). Let $A_k=U{D}_k{V}^T$. Then,

$$d{\sigma }_d^p\le {\parallel A-A_k\parallel }_{p,p}^p\le d^{1+\frac{p}{2}}{\sigma }_k^p.$$

Proof. 

First, we assume that $p\ne 2$; otherwise, the ${\parallel ·\parallel }_2$ factorization is the SVD factorization, and we obtain the optimal solution for the ${\ell }_2$ low rank approximation problem. For every $i\in \left[d\right]$, let $e_i\in {ℝ}^d$ be a vector of zeros, except for its ith entry, where it is set to 1. Observe that ${\parallel A-A_k\parallel }_{p,p}^p=\sum _{i=1}^d{\parallel A-A_k{e}_i\parallel }_p^p=\sum _{i=1}^d{\parallel AI-{D{V}^T}^{-1}{D}_k{V}^T{e}_i\parallel }_p^p$, where the first equality holds by definition of ${\parallel ·\parallel }_{p,p}^p$, and the second equality follows from the definition of $A_k$ (see Lines 3–4 of Algorithm 1).

Plugging $A:=A$, $D:=D$, $V:=V$, $x:=(I-{\left(D{V}^T\right)}^{-1}{D}_k{V}^T)e_i$ into Lemma 4 yields that for every $i\in \left[d\right]$,

$$\begin{array}{cc}\hfill {\parallel (D-D_k)V^T{e}_i\parallel }_2^p& \le {\parallel (A-A_k)e_i\parallel }_{p,p}^p\hfill \\ \hfill & \le d^{\frac{p}{2}}{\parallel (D-D_k)V^T{e}_i\parallel }_2^p.\hfill \end{array}$$

(5)

Observe that for every $i\in \left[d\right]$,

$$\begin{array}{c}\hfill {\parallel (D-D_k)V^T{e}_i\parallel }_2^p\le {\parallel D-D_k\parallel }_2^p{\parallel V^T{e}_i\parallel }_2^p\le {\parallel D-D_k\parallel }_2^p,\end{array}$$

(6)

where the first inequality holds by properties of the ${\ell }_2$ matrix induced norm, and the second inequality holds since V is an orthogonal matrix.

Since $V^T{e}_i$ is a unit vector,

$${\parallel (D-D_k)V^T{e}_i\parallel }_2^p\ge {\sigma }_d^p,$$

(7)

where the inequality holds by plugging $x:=V^T{e}_i$ and $D:=(D-D_k)$ into Claim 3.

In addition, we have that

$${\sigma }_d\le {\parallel D-D_k\parallel }_2={\sigma }_{k+1}$$

(8)

where both the inequality and equality hold since ${\sigma }_d$ is the lowest eigenvalue of D, D being a diagonal matrix.

By combining Equations (5)–(8), we obtain that for every $i\in \left[d\right]$,

$${\sigma }_d^p\le {\parallel (A-A_k)e_i\parallel }_p^p\le d^{\frac{p}{2}}{\sigma }_{k+1}^p.$$

(9)

Theorem 5 follows by summing Equation (9) over every $i\in \left[d\right]$.    □

Note that the set ${\left\{{\sigma }_i\right\}}_{i=1}^d$ denotes the reciprocal values of the ellipsoid E axis’s lengths, where E is the Löwner ellipsoid of $L=\{x\in {ℝ}^d|{\parallel Ax\parallel }_p\le 1\}$. As discussed in the previous section, these values serve to bound the “ ${\ell }_p$ singular values of A”.

4.2. Randomized Result

In addition to our deterministic result, we also show how to support a randomized version that computes an approximation in a faster time, which relies on the following result of [65].

Theorem 6

(Variant of Theorem 10 [65]). For any $A\in {ℝ}^{n\times d}$ of rank d and $p\ge 1$, one can compute an invertible matrix $R\in {ℝ}^{d\times d}$ and a matrix $U=A{R}^{-1}$ such that ${\parallel Rx\parallel }_2\le {\parallel Ax\parallel }_p\le d{(d^3+d^{2}logn)}^{\left|1/p-1/2\right|}{\parallel Rx\parallel }_2$ holds with a probability of at least $1-\frac{1}{n}$, where R can be computed in time $O(ndlogn)$.

Theorem 7.

Let $A\in {ℝ}^{n\times d}$ be real matrix, $p\ge 1$, and $k\in [d-1]$ be an integer. There exists a randomized algorithm which, when given a matrix $A\in {ℝ}^{n\times d}$, $k\in [d-1]$, in time $O(ndlogn)$, returns $(U,D_k,V,\{{\sigma }_1,\dots {\sigma }_d\})$, such that

$$d{\sigma }_d^p\le {\parallel A-A_k\parallel }_{p,p}^p\le d^{1+p}{(d^3+d^{2}logn)}^{\left|1-p/2\right|}{\sigma }_{k+1}^p,$$

holds with a probability of at least $1-\frac{1}{n}$, where $A_k=U{D}_k{V}^T$.

Proof. 

The algorithm is described throughout the following proof. Let $R\in {ℝ}^{d\times d}$ be as defined in Theorem 6 when plugging $A:=A$ into Theorem 6. Let $R=\tilde{U}D{V}^T$ be the SVD of R; $D_k\in {[0,\infty )}^{d\times d}$ be a diagonal matrix where its first k diagonal entries are identical to those of D, while the rest of the entries in $D_k$ are set to 0; and $\{{\sigma }_1,\cdots ,{\sigma }_d\}$ be the set of singular values of D. Note that since for every $x\in {ℝ}^d$, by Theorem 6 it holds that

$${\parallel Rx\parallel }_2^p\le {\parallel Ax\parallel }_p^p{d}^p{(d^3+d^{2}logn)}^{\left|1-p/2\right|}{\parallel Rx\parallel }_2^p.$$

From here, similar to the proof of Theorem 5, we obtain that

$$d{\sigma }_d^p\le {\parallel A-A_k\parallel }_{p,p}^p\le d^{1+p}{(d^3+d^{2}logn)}^{\left|1-p/2\right|}{\sigma }_{k+1}^p.$$

   □

Remark 8.

Note that in our context of embedding layer compression, the corresponding embedding matrix A has more columns than rows. Regardless, our ${\ell }_p$ norm of any $A-B$ such that $A,B\in d\times n$ enables us to have ${\parallel A-B\parallel }_{p,p}^p={\parallel A^T-B^T\parallel }_{p,p}^p.$ Hence, substituting $A:=A^T$ and $A_k:=A_k^T$ yields

$$d{\sigma }_d^p\le {\parallel A-A_k\parallel }_{p,p}^p\le d^{1+\frac{p}{2}}{\sigma }_{k+1}^p,$$

for our deterministic results, and similarly, we can obtain this for our randomized result.

5. Experimental Results

The compressed networks. We compress several frequently used NLP networks:

(i)

BERT [12]: BERT is a bidirectional transformer pre-trained using a combination of masked language modeling objective and next sentence prediction on a large corpus comprising the Toronto Book Corpus and Wikipedia.

(ii)

DistilBERT [24]: the DistilBERT model is smaller, faster, cheaper, and lighter than BERT. This model is a distilled version of BERT. It has $40\%$ less parameters than bert-base-uncased and runs $60\%$ faster, while preserving over $95\%$ of BERT’s performances as measured on the GLUE language understanding benchmark [24].

(iii)

XLNet [66]: XLnet is an extension of the Transformer-XL model [67] pre-trained using an autoregressive method to learn bidirectional contexts by maximizing the expected likelihood over all permutations of the input sequence factorization order.

(iv)

RoBERTa [68]: RoBERTa modifies the key hyperparameters in BERT, including removing BERT’s next-sentence pre-training objective, and training with much larger mini-batches and learning rates. This allows RoBERTa to improve on the masked language modeling objective compared with BERT and leads to better downstream task performance.

See full details on the sizes of each network and their embedding layer before compression in

Table 1. The sizes of the networks BERT, RoBERTA, XLNet, and DistilBERT, and their embedding layers.

Model		Embedding Layer Size	Parameters
BERT	base	30,522 $\times 768$	110 M
RoBERTa	base	50,265 $\times 768$	125 M
XLNet	base	32,000 $\times 768$	110 M
DistilBERT	base	30,522 $\times 768$	66 M

.

Implementation, Software, and Hardware. All the experiments were conducted on an AWS p2.xlargs machine with 1 GPU NVIDIA K80, 4 vCPUs, and 61 RAM [GiB]. We implemented our suggested compression algorithm (Algorithm 1) in Python $3.8$ using the Numpy library [69]. To build and train networks (i)–(iv), we used the suggested implementation in the Transformers https://github.com/huggingface/transformers (accessed on 15 July 2021) library from HuggingFace [70] (Transformers version $2.3$ and PyTorch version $\mathrm{1.5.1}$ [71]). Before the compression, all the networks were fine-tuned on all the tasks from the GLUE benchmark to obtain almost the same accuracy results as reported in the original papers. Since we did not succeed in obtaining close accuracy on the tasks QQP and WNLI (with most of the network), we did not include results from them.

Our compression. We compress each embedding layer (matrix) of the reported networks by factorizing it into two smaller layers (matrices) as follows. For an embedding layer that is defined by a matrix $A\in {ℝ}^{n\times d}$, we compute the matrices $U,D_k,V$ by a call to ℓ_ρ-Low-rank$A,k,1$ (see Algorithm 1), where k is the low rank projection we wish to have. Observe, that the matrix $D_k$ is a diagonal matrix, and its last $d-k$ columns are zero columns. We then compute a non-square diagonal matrix $D_k^{\prime }\in {ℝ}^{d\times k}$ that is the result of removing all the zero columns of $D_k$. Now, the ${\ell }_1$k-rank approximation of A can be factorized as $A_k=\left(U\sqrt{D_k^{\prime }}\right)\left({\sqrt{D_k^{\prime }}}^T{V}^T\right)$. Hence, we save the two matrices (layers): (i) $U\sqrt{D_k^{\prime }}$ of size $n\times k$, and (ii) ${\sqrt{D_k^{\prime }}}^T{V}^T$ of size $k\times d$. This yields two layers of a total size of $nk+kd$ instead of a single embedding layer of a total size of $nd$.

Reported results. We report the test accuracy drop (relative error) on all the tasks from the GLUE benchmark [72] after compression for several compression rates:

• In Figure 3, the x-axis is the compression rate of the embedding layer, and the y-axis is the accuracy drop (relative error) with respect to the original accuracy of the network. Each figure reports the results for a specific task from the GLUE benchmark on all the networks we compress. Here, all reported results are compared to the known ${\ell }_2$-factorization using SVD. In addition, in all the experiments, we do not fine-tune the model after compressing; this is to show the robustness and efficiency of our technique.
• Table 2. In the above, we present our compressed networks and their drop in accuracy based on the compression rate of the embedding layer. Specifically, each non-percentile value represents the accuracy drop achieved by our compressed model with respect to its original model (e.g., RE-RoBERTa is a compressed model of RoBERTa), while negative values present improvements in the accuracy upon the non-compressed version of the corresponding model. The last column is the average accuracy drop over all tested tasks. The “RE” here stands for “Robust Embedding”.

  Model		Embedding Layer Compression Rate	MRPC	COLA	MNLI	SST-2	STS-B	QNLI	RTE	Avg.
  RE-RoBERTa	base	$15\%$	$0.49$	$-2.16$	$0.01$	$0.045$	$0.013$	$0.018$	$1.08$	$-0.072$
  	small	$28\%$	$0.98$	$-2.01$	$0.08$	$0.68$	$0.87$	$1.33$	$2.52$	$0.63$
  	tiny	$41\%$	$2.69$	$3.82$	$2.18$	$2.17$	$3.10$	$3.58$	$2.16$	$2.81$
  RE-XLNet	base	$15\%$	$2.20$	$-0.43$	$-0.07$	$0.22$	$0.03$	$2.39$	$2.16$	$0.92$
  	small	$21\%$	$1.47$	$0.26$	$0.11$	$-0.34$	$0.03$	$3.42$	$4.33$	$1.32$
  	tiny	$28\%$	$1.96$	$3.19$	$0.47$	$-0.22$	$0.19$	$4.46$	$6.13$	$2.31$
  RE-BERT	base	$15\%$	$0.73$	$-0.54$	$0.48$	$-1.49$	$0.85$	$2.36$	$1.80$	$0.59$
  	small	$21\%$	$3.43$	$0.08$	$1.72$	$0.45$	$1.62$	$3.78$	$1.44$	$1.78$
  	tiny	$28\%$	$4.90$	$-0.94$	$3.48$	$1.49$	$2.66$	$7.65$	$1.80$	3
  RE-DistilBERT	base	$15\%$	$1.47$	$5.24$	$0.86$	$0.34$	$0.13$	$5.80$	$2.16$	$2.28$

  suggests the best compressed networks in terms of accuracy vs size. For every network from (i)–(iv), we suggest a compressed version of it with a very small drop in the accuracy and sometimes with an improved accuracy. Given a network “X”, we call our compressed version of “X” “RE-X”, e.g., RE-BERT and RE-XLNet. The “RE” here stands for “Robust Embedding”.
• Table 3. We evaluate our compression method against different compression techniques that do not use any fine-tuning steps on the RoBERTa model (or any usage of the training data after compression). The following table reports the drop in accuracy on the MRPC task after using these compression techniques.

  Compression Method		Embedding Layer Compression Rate	MRPC
  SVD	base	$15\%$	$5.70$
  	small	$28\%$	$6.25$
  	tiny	$41\%$	$9.51$
  L1PCA [73]	base	$15\%$	$3.04$
  	small	$28\%$	$4.94$
  	tiny	$41\%$	$18.48$
  Pruning [74]	base	$15\%$	$1.90$
  	small	$28\%$	$2.17$
  	tiny	$41\%$	$2.98$
  Random pruning	base	$15\%$	$1.36$
  	small	$28\%$	$3.06$
  	tiny	$41\%$	$4.38$
  SynFlow [75]	base	$15\%$	$0.81$
  	small	$28\%$	$1.60$
  	tiny	$41\%$	$2.75$
  Our compression	base	$15\%$	$0.49$
  	small	$28\%$	$0.98$
  	tiny	$41\%$	$2.69$

  reports a comparison between our approach and different compressionmethods that do not require fine-tuning or any usage of the training data after compression:

  (i)

  SVD.

  (ii)

  L1PCA [73].

  (iii)

  Pruning [74].

  (iv)

  Random pruning.

  (v)

  Syn flow [75].

6. Discussion

It can be seen by Figure 3 that our approach is more robust than the traditional SVD. In most of the experiments, our suggested compression achieves better accuracy for the same compression rate compared to the traditional SVD. Mainly, we observed that our compression schemes shine when either vocabulary is rich (the number of subword units is large) or the model itself is small (excluding the embedding layer). Specifically speaking, in RoBERTa, our method achieves better results due to the fact that RoBERTa’s vocabulary is rich (i.e., 50 K subword units compared to the 30 K in BERT). This large vocabulary increases the probability of having outliers in it, which is the main justification for our approach. In DistilBERT, the network is highly efficient. This can lead to a sensitive snowball effect, i.e., the classification is highly affected by even the smallest errors caused by the compression of the embedding layer. Since SVD is sensitive to outliers and due to the fact that the network is highly sensitive to small errors, the existence of outliers highly affects the results. This phenomenon is illustrated throughout Figure 3. Here, our compression scheme outperforms the SVD due to its robustness against outliers, which, in turn, achieves smaller errors. As for XLNet, the model encodes the relative positional embedding, which, in short, represents an embedding of the relative positional distance between words. In our context, this means that having outliers highly affects the relative positional embedding, which, in turn, affects the classification accuracy. Hence, this explains why we outperform SVD. Since none of the above phenomena hold for BERT, this may explain why SVD sometimes achieves better results. However, across most tasks, our compression scheme is favorable upon SVD.

Finally, for some tasks at low compression rates, the accuracy has been improved (e.g., see task SST-2 at Figure 3 when compressing BERT). This may be due to the fact that at low compression rates, we remove the least necessary (redundant) dimensions. Thus, if these dimensions are actually unnecessary, by removing them, we obtain a generalized model which is capable of classifying better.

7. Conclusion and Future Work

We provided an algorithm that computes an approximation for ${\ell }_p$k-rank approximation, where $p\ge 1$. We then suggested a new approach for compressing networks based on k-rank ${\ell }_p$-approximation, where $p\in [1,2]$ instead of ${\ell }_2$. The experimental results in Section 5 showed that our suggested algorithm overcomes the traditional ${\ell }_2$k-rank approximation and achieves higher accuracy for the same compression rate when there is no fine-tuning involved.

Future work includes: (1) Extending our approach to other factorization models, such as non-negative matrix approximation or dictionary learning; (2) experimental results on other benchmarks and other models; (3) suggesting algorithms for the ${\ell }_p$k-rank approximation for any $p\in (0,1)$, while checking the practical contribution in compressing deep networks for this case; and (4) combining this result with other compression techniques to obtain a smaller network with higher accuracy.