Energy-Constrained Deep Neural Network Compression for Depth Estimation

Abstract

Many applications, such as autonomous driving, robotics, etc., require accurately estimating depth in real time. Currently, deep learning is the most popular approach to stereo depth estimation. Some of these models have to operate in highly energy-constrained environments, while they are usually computationally intensive, containing massive parameter sets ranging from thousands to millions. This makes them hard to perform on low-power devices with limited storage in practice. To overcome this shortcoming, we model the training process of a deep neural network (DNN) for depth estimation under a given energy constraint as a constrained optimization problem and solve it through a proposed projected adaptive cubic quasi-Newton method (termed ProjACQN). Moreover, the trained model is also deployed on a GPU and an embedded device to evaluate its performance. Experiments show that the stage four results of ProjACQN on the KITTI-2012 and KITTI-2015 datasets under a 70% energy budget achieve (1) 0.13% and 0.61%, respectively, lower three-pixel error than the state-of-the-art ProjAdam when put on a single RTX 3090Ti; (2) 4.82% and 7.58%, respectively, lower three-pixel error than the pruning method Lottery-Ticket; (3) 5.80% and 0.12%, respectively, lower three-pixel error than ProjAdam on the embedded device Nvidia Jetson AGX Xavier. These results show that our method can reduce the energy consumption of depth estimation DNNs while maintaining their accuracy.

1. Introduction

As a classical computer vision problem, stereo image-based depth estimation has a wide range of applications such as autonomous driving, robotics, 3D scene understanding, etc. [1,2,3,4,5]. Many of these have to operate in highly energy-constrained environments. However, most of the existing deep-learning-based depth estimation methods focus on designing more powerful network architectures to obtain more accurate depth images. Thus, their strong performance often requires massive computing resources, making them too heavy to run on embedded devices.

There are several state-of-the-art studies [6,7,8,9,10,11] that attempt to address this problem by building lightweight networks, that is, they trade off computation and accuracy with low inference time. These “mini” models could be further reduced in size to save more energy while maintaining a similar performance. An open-source study [12] proposes an end-to-end DNN training framework that provides quantitative energy consumption guarantees via weighted sparse projection and input masking. The input mask enables the input sparsity to be controlled by a trainable parameter, and thus increases the energy reduction opportunities. They also present a projected version Adam (termed ProjAdam) to train the model under quantitatively estimated energy consumption. However, the method only applies to image classification tasks, and the network only includes fully connected layers and 2D convolution layers, which are different from many depth estimation networks.

In this paper, we also formulate the training process of depth estimation DNN under a given energy budget as a constrained optimization problem. Different from prior work, we model the energy consumption of the depth estimation network after analyzing its architecture and taking 3D convolution layers into consideration. Furthermore, we propose a projected adaptive cubic quasi-Newton optimizer (termed ProjACQN) to obtain a better model of such a complex optimization problem. Unlike the commonly used first-order projected optimizers [12,13] or proxy optimizers [14,15,16], which only utilize the gradient information, ProjACQN incorporates Hessian information with the norm of difference between the two previous estimates, and performs the projection operation onto the energy constraint after the parameter update.

We evaluate the efficiency of our method on KITTI-2012 [17] and KITTI-2015 [18] using the depth estimation DNN AnyNet [6]. Experiments show that our method can reduce energy consumption of AnyNet by 30% while improving its accuracy by 0.13% and 0.62% compared to ProjAdam [12] on KITTI-2012 and KITTI-2015, respectively. Our method is also compared with three existing pruning methods (including L1-norm pruning [19], BN pruning [20] and Lottery-Ticket [21]) under the same energy budget, and achieves the best result. In addition, we perform the models on the embedded device Nvidia Jetson AGX Xavier [22], and find that ProjACQN with a 70% energy budget is able to outperform a dense model without an energy budget in terms of three-pixel error and time consumption.

2. Related Work

Generally, research on depth estimation using DNNs mainly focuses on designing architectures with better performance. These methods can be classified into three classes: supervised, semi-supervised and unsupervised. Taking [23,24] as examples of supervised methods, ref. [23] performs prediction at adaptively selected locations, which are easier to be estimated accurately and can alleviate excessive computation. Ref. [24] proposes a depth refinement architecture using 3D dilated convolutions to predict geometrically consistent disparity images. These supervised stereo depth estimation methods have achieved great success, but they require per-pixel ground truth depth data, which are often hard to acquire. To resolve this issue, an alternative is the class of unsupervised methods, which uses the geometric constraints between stereo images as the supervisory signal instead of directly inputting ground truth disparity. Ref. [25] combines an unsupervised stereo disparity estimation network with a perceptual loss network, which enables it to refine the predicted disparity. Ref. [26] designs a Siamese autoencoder architecture to extract mutual information between the rectified stereo images. It also exploits the mutual epipolar attention, and uses the optimal transport algorithm to refine the depth image. In addition, there are very few works on semi-supervised stereo depth estimation using DNNs. Ref. [27] trains in a semi-supervised manner to combine information from LIDAR and photometric data. The model can achieve a better performance than those trained only with LIDAR.

The methods mentioned above often contain massive parameter sets to achieve outperformance, while they require very a high cost of computation and GPU memory to be put on embedded devices. Recently, many studies have focused on improving energy efficiency through artifical intelligence [6,7,8,9,10,11,28,29]. This has inspires many studies to explore lightweight DNNs for stereo depth estimation. Ref. [10] performs online adaptive stereo depth estimation through a self-supervised learning method, which helps to save computation and GPU memory. Ref. [11] is based on a Max-tree hierarchical representation of image pairs, and is able to identify matching regions along image scan-lines. Ref. [8] estimates depth via a series of binary classifications. Instead of obtaining an accurate depth map, it classifies objects according to their relative distance. Ref. [7] proposes an efficient neural network, in which its computation and latency saving mostly benefit from using the depth-wise separable convolution and network pruning. Refs. [6,30] proposes AnyNet to achieve a wide accuracy range of the disparity map according to the permitted reference time. In practice, AnyNet has four stages, and the higher the stage, the better the accuracy with more time cost.

3. Problem Formulation

In this section, we provide an energy model for a depth estimation DNN which consists of fully connected layers, 2D convolution layers and 3D convolution layers.

3.1. Energy Consumption Modeling

Generally, the energy cost of inferencing a typical depth estimation DNN after the popular systolic array hardware architecture [31] can be formulated as [12]

$$\underset{\mathbf{M},\mathbf{W}}{min}{L}_{{\mathbf{W}}_{dense}}(\mathbf{M},\mathbf{W})s.tE(\mathbf{M},\mathbf{W})\le E_{budget}$$

(1)

where $L(·)$ is the loss function, W the weight tensor of the sparse model, ${\mathbf{W}}_{dense}$ the weight tensor of the original dense model, M the input mask, $E_{budget}$ the given energy budget and E the total energy cost. The depth estimation DNN model mainly consists of a sequence of fully connected layers, 2D convolution layers and 3D convolution layers. Also, their energy consumption can be decomposed into two parts: computation energy and data access energy. Let $\mathbb{U},\mathbb{V},\mathbb{W}$ be the sets of the fully connected layers, 2D convolution layers and 3D convolution layers in a DNN, respectively; $E_{comp}^{(i)}$ the energy consumed by computation units in layer i; and $E_{data}^{(i)}$ the energy consumed in layer i when accessing data from the hardware memory. The total energy cost can be acquired through

$$E=\sum _{u\in \mathbb{U}}(E_{comp}^{(u)}+E_{data}^{(u)})+\sum _{v\in \mathbb{V}}(E_{comp}^{(v)}+E_{data}^{(v)})+\sum _{w\in \mathbb{W}}(E_{comp}^{(w)}+E_{data}^{(w)})$$

(2)

assuming $e_{\mathbf{M}AC}$ denotes the energy consumption of one Multiply-and-Accumulate (MAC) operation, ${\mathbf{X}}^{(i)}$ the input tensor of layer i and $supp(·)$ returns a binary tensor which labels a nonzero position. Then, the computation energy cost of a fully connected layer is

$$E_{comp}^{(u)}=e_{MAC}sum(supp({\mathbf{X}}^{(u)})·supp({\mathbf{W}}^{(u)}))$$

(3)

The computation energy cost of a 2D convolution layer is

$$E_{comp}^{(v)}=e_{MAC}sum(supp({\mathbf{X}}^{(u)})\ast supp({\mathbf{W}}^{(u)}))$$

(4)

Generally, to accelerate the loading speed, it is common to load data from the DRAM to the cache and then from the cache to the register file (RF) when DNN is inferencng [12,32]. Thus, each layer has to load their input and weight three times, once each from DRAM, RF and cache. Let $e_{DRAM}$, $e_{RF}$ and $e_{cache}$ be the unit energy costs of DRAM, RF and cache, respectively. Thus, the data access energy cost of each layer can be formulated as

$$\begin{array}{cc}\hfill E_{data}^{(layer)}=& e_{DRAM}(N_{DRAM}^{input}+N_{DRAM}^{weight})+e_{RF}(N_{RF}^{input}+N_{RF}^{weight})\hfill \\ \hfill & +e_{cache}(N_{cache}^{input}+N_{cache}^{weight})\hfill \end{array}$$

(5)

where $N_{DRAM}^{input}$, $N_{RF}^{input}$ and $N_{cache}^{input}$ denote the total numbers of DRAM, RF and cache accesses related to input, respectively, and $N_{DRAM}^{weight}$, $N_{RF}^{weight}$ and $N_{cache}^{weight}$ the total numbers related to weight. Ref. [12] only gives the detailed computation of the energy cost for the fully connected layer and 2D convolution layer.

3.1.1. Computation Energy for 3D Convolution Layer

Let ${\mathbf{W}}^{(w)}\in {\mathbb{R}}^{c_{out}\times c_{in}\times r\times r\times r}$ be the weight tensor, where $c_{in}$ and $c_{out}$ are the number of input channels and output channels, respectively, and let ${\mathbf{X}}^{(w)}\in {\mathbb{R}}^{c_{in}\times h\times w\times d}$ be the input tensor, where h, w and d are the height, width and depth of X, respectively. The computation of the 3D convolution operation can be formulated as

$${({\mathbf{X}}^{(w)}\ast {\mathbf{W}}^{(w)})}_{j,x,y,z}=\sum _{i=1}^{c_{in}}\sum _{x^{{}^{\prime }},y^{{}^{\prime }},z^{{}^{\prime }}=0}^{r-1}{X}_{i,x-\lfloor \frac{r}{2}\rfloor +x^{{}^{\prime }},y^{{}^{\prime }}-\lfloor \frac{r}{2}\rfloor +y^{{}^{\prime }},z^{{}^{\prime }}-\lfloor \frac{r}{2}\rfloor +z^{{}^{\prime }}}^{(w)}{\mathbf{W}}_{j,i,x^{{}^{\prime }},y^{{}^{\prime }},z^{{}^{\prime }}}^{(w)}$$

(6)

where x, y and z represent the corresponding position of the output element. Assume the convolution padding is p and the convolution stride s. Let $h^{{}^{\prime }}=\lfloor (h+2p-r)/s\rfloor +1$, $w^{{}^{\prime }}=\lfloor (w+2p-r)/s\rfloor +1$, $d^{{}^{\prime }}=\lfloor (d+2p-r)/s\rfloor +1$. Then, the size of the output tensor is $c_{out}\times d{}^{\prime }\times h^{{}^{\prime }}\times w^{{}^{\prime }}$. Thus, the number of MAC operations is $sum(supp({\mathbf{X}}^{(w)})\ast supp({\mathbf{W}}^{(w)}))\le h^{{}^{\prime }}{w}^{{}^{\prime }}{d}^{{}^{\prime }}\left|\right|{\mathbf{W}}^{(w)}{\left|\right|}_0$, and the computation energy cost can be approximated through

$$E_{comp}^{(w)}\approx e_{MAC}{h}^{{}^{\prime }}{w}^{{}^{\prime }}{d}^{{}^{\prime }}\left|\right|{\mathbf{W}}^{(w)}{\left|\right|}_0$$

(7)

3.1.2. Data Access Energy for 3D Convolution Layer

According to Equation (5), to acquire the data access energy for the 3D convolution layer, the access number of the cache, RF and DRAM are needed. Let $N_{W\_cache}^{(w)}$, $N_{W\_RF}^{(w)}$ and $N_{W\_DRAM}^{(w)}$ be the access numbers of the weight tensor, respectively, and $N_{X\_cache}^{(w)}$, $N_{X\_RF}^{(w)}$ and $N_{X\_DRAM}^{(w)}$ the access numbers of the input tensor. For simplification, we can unfold the input tensor as ${\overline{\mathbf{X}}}^{(w)}\in {\mathbb{R}}^{h^{{}^{\prime }}{w}^{{}^{\prime }}{d}^{{}^{\prime }}\times c_{in}{r}^3}$ according to the output channel, where each column represents all elements in ${\mathbf{X}}^{(w)}$ that operate with one element in ${\mathbf{W}}^{(w)}$. The access number can be formulated as

$$\begin{array}{cc}\hfill & N_{W\_cache}^{(w)}=\lfloor h^{{}^{\prime }}{w}^{{}^{\prime }}{d}^{{}^{\prime }}/s_h\rfloor \left|\right|{\mathbf{W}}^{(w)}{\left|\right|}_0\hfill \\ \hfill & N_{W\_RF}^{(w)}=h^{{}^{\prime }}{w}^{{}^{\prime }}{d}^{{}^{\prime }}/s_h\left|\right|{\mathbf{W}}^{(w)}{\left|\right|}_0\hfill \\ \hfill & N_{W\_DRAM}^{(w)}=\lfloor h^{{}^{\prime }}{w}^{{}^{\prime }}{d}^{{}^{\prime }}/s_h\rfloor max(0,||{\mathbf{W}}^{(w)}{\left|\right|}_0-k_W)+min(k_W,\left|\right|{\mathbf{W}}^{(w)}{\left|\right|}_0)\hfill \\ \hfill & N_{X\_cache}^{(w)}=\lfloor c_{out}/s_w\rfloor \left|\right|{\overline{X}}^{(w)}{\left|\right|}_0\hfill \\ \hfill & N_{X\_RF}^{(w)}=c_{out}\left|\right|{\overline{X}}^{(w)}{\left|\right|}_0+2h^{{}^{\prime }}{w}^{{}^{\prime }}{d}^{{}^{\prime }}\left|\right|{\mathbf{W}}^{(w)}{\left|\right|}_0\hfill \\ \hfill & N_{X\_DRAM}^{(w)}=\left|\right|X^{(w)}{\left|\right|}_0+N_{overlap}+c_{out}{h}^{{}^{\prime }}{w}^{{}^{\prime }}{d}^{{}^{\prime }}\hfill \end{array}$$

(8)

where $k_W$ represents the cache size for the weight matrix, and $N_{overlap}$ the number of overlapped elements due to the nature of the 3D convolution operation. $s_h$ and $s_w$ denote the width and height of the systolic array, respectively. According to Equations (5) and (8), the total data access energy consumption for a 3D convolution layer can be formulated as

$$E_{data}^{(w)}=e_{DRAM}(N_{X\_DRAM}^{(w)}+N_{W\_DRAM}^{(w)})+e_{RF}(N_{X\_RF}^{(w)}+N_{W\_RF}^{(w)})+e_{cache}(N_{X\_cache}^{(w)}+N_{W\_cache}^{(w)})$$

(9)

4. Optimization Algorithm

In this section, we first provide the formulation of our projected adaptive cubic quasi-Newton optimizer. Then, we utilize the optimizer to solve the constrained optimization problem (1).

4.1. Projected Cubic Quasi-Newton Optimizer

Consider the following optimization problem:

$$\underset{\mathbf{W}\in \mathbb{C}}{min}L(W)$$

(10)

where the constraint set $\mathbb{C}$ is convex and compact. We can find the optima through the following augmented second-order approximation:

$$\underset{\mathbf{W}\in \mathbb{C}}{min}L({\mathbf{W}}_k)+{\mathbf{g}}_k^T(\mathbf{W}-{\mathbf{W}}_k)+\frac{1}{2}{(\mathbf{W}-{\mathbf{W}}_k)}^T{\mathbf{B}}_k(\mathbf{W}-{\mathbf{W}}_k)+\frac{\rho }{6}\left|\right|\mathbf{W}-{\mathbf{W}}_k{\left|\right|}^3$$

(11)

where ${\mathbf{g}}_k$ represents the gradient at iteration k, ${\mathbf{B}}_k$ an approximation to the Hessian matrix at ${\mathbf{W}}_k$, and $\rho$ a positive constant. We can obtain a stationary point without restriction through setting the derivative of the object function to zero:

$${\mathbf{g}}_k+{\mathbf{B}}_k(x-x_k)+\frac{\rho }{2}\left|\right|\mathbf{x}-{\mathbf{x}}_k\left|\right|(\mathbf{x}-{\mathbf{x}}_k)=0$$

(12)

The update of the stationary point becomes

$${\mathbf{W}}_{k+1}={\mathbf{W}}_k-({\mathbf{B}}_k+\frac{\rho }{2}\left|\right|{\mathbf{W}}_k-{\mathbf{W}}_{k-1}{\left|\right|·\mathbf{I})}^{-1}{g}_k$$

(13)

To avoid inverse difficulty caused by matrix degradation, we constrain the absolute value of the diagonal elements to be greater than a given positive parameter $\theta$:

$${\mathbf{W}}_{k+1}=max(abs({\mathbf{B}}_k+\frac{\rho }{2}\left|\right|x_k-x_{k-1}{\left|\right|·\mathbf{I}),\theta ·\mathbf{I})}^{-1}{\mathbf{g}}_k$$

(14)

Then, the optima can be acquired through projecting the stationary point into the constraint set. This yields a novel update,

$${\mathbf{W}}_{k+1}^{\ast }=pro{j}_{\mathbb{C}}(max(abs({\mathbf{B}}_k+\frac{\rho }{2}\left|\right|{\mathbf{x}}_k-{\mathbf{x}}_{k-1}{\left|\right|·\mathbf{I}),\theta ·\mathbf{I})}^{-1}{\mathrm{g}}_k)$$

(15)

where the projection operation $pro{j}_{\mathbb{C}}$ is to project the stationary point into optimization constraint set $\mathbb{C}$. The detailed algorithm for ProjACQN is shown in Algorithm 1. We then use this algorithm to solve problem (1) in the following subsection.

Algorithm 1 ProjACQN

Require:  Mini-batch size $n_g$, stepsize $\eta$, exponential decay rates ${\beta }_1,{\beta }_2\in [0,1)$, positive parameters $ϵ,\rho ,\theta$, constrain set $\mathbb{C}$. Require:  ${\mathbf{x}}_0,{\mathbf{g}}_0,{\mathbf{B}}_0,{\mathbf{m}}_0,{\mathbf{V}}_0$        //Initialize variables Require:  $k\leftarrow 0,t\leftarrow 0$        //Initialize timestep 1: while${\mathbf{x}}_k$ not converged do 2:    $k\leftarrow k+1$ 3:    ${\mathbf{g}}_k\leftarrow L({\mathbf{W}}_k;n_{\mathbf{g}})$        //Stochastic gradient at timestep k 4:    ${\mathbf{s}}_k\leftarrow {\mathbf{x}}_k-{\mathbf{x}}_{k-1}$ 5:    ${\mathbf{g}}_k\leftarrow {\mathbf{g}}_k-{\mathbf{g}}_{k-1}$ 6:    ${\mathbf{B}}_k\leftarrow {\mathbf{B}}_{k-1}+\frac{{\mathbf{s}}_k^T{\mathbf{g}}_k-{\mathbf{s}}_k^T{\mathbf{B}}_{k-1}{\mathbf{s}}_k}{\left|\right|{\mathbf{s}}_k{\left|\right|}_4^4+ϵ}Diag({\mathbf{s}}_k^2)$        //Update diagonal Hessian 7:    ${\mathbf{D}}_k\leftarrow max(abs({\mathbf{B}}_k+\frac{\rho }{2}\left|\right|{\mathbf{x}}_k-{\mathbf{x}}_{k-1}\left|\right|·\mathbf{I}),\theta ·\mathbf{I})$ 8:    ${\mathbf{m}}_k\leftarrow \frac{(1-{\beta }_1^{k-1}){\beta }_1{\mathbf{m}}_{k-1}+(1-{\beta }_1){\mathbf{g}}_k}{1-{\beta }_1^k}$        //Update first moment 9:    ${\mathbf{V}}_k\leftarrow \frac{(1-{\beta }_2^{k-1}){\beta }_2{\mathbf{V}}_{k-1}+(1-{\beta }_2){\mathbf{D}}_k^2}{1-{\beta }_2^k}$            //Update high-order second moment 10:    ${\mathbf{W}}_{k+1}\leftarrow pro{j}_{\mathbb{C}}({\mathbf{W}}_k-\eta {\mathbf{V}}_k^{-\frac{1}{2}}{\mathbf{m}}_k)$        //Update parameters with projection operation 11: end while

4.2. Update Weight Tensor and Input Mask

The problem in Equation (1) has two variables, M and W. For simplification, it can be transformed into two sub-problems and solved through alternative updating of the following two equations:

$${\mathbf{W}}^{\ast }=\underset{\mathbf{W}\in \mathcal{W}}{argmin}L(\mathbf{M},\mathbf{W})$$

(16)

$${\mathbf{M}}^{\ast }=\underset{\mathbf{M}\in \mathcal{M}}{argmin}L(\mathbf{M},\mathbf{W})$$

(17)

where $\mathcal{W}=\left\{{\mathbf{W}}^{\prime }\right|E(\mathbf{M},{\mathbf{W}}^{\prime })\le E_{budget}\}$ and $\mathcal{M}=\left\{{\mathbf{M}}^{\prime }\right|\left|\right|{\mathbf{M}}^{\prime }|{|}_0\le q,{\mathbf{M}}^{\prime }\in [0,1]\}$. The sub-problems (16) and (17) are similar to (10), and can be optimized with the projected cubic quasi-Newton optimizer. According to Equations (15)–(17), the update of the weight tensor can be formulated as

$${\mathbf{W}}_{k+1}=pro{j}_{\mathcal{W}}({\mathbf{W}}_k-(max(abs({\mathbf{B}}_k+\frac{\rho }{2}\left|\right|{\mathbf{x}}_k-{\mathbf{x}}_{k-1}{\left|\right|·\mathbf{I}),\theta ·\mathbf{I})}^{-1}{\mathbf{g}}_k)$$

(18)

The update of the mask tensor can be formulated as

$${\mathbf{M}}_{k+1}=pro{j}_{\mathcal{M}}({\mathbf{M}}_k-(max(abs({\mathbf{B}}_k^{\prime }+\frac{\rho }{2}\left|\right|{\mathbf{x}}_k^{\prime }-{\mathbf{x}}_{k-1}^{\prime }\left|\right|·{\mathbf{I}}^{\prime }),\theta ·{\mathbf{I}}^{\prime }{)}^{-1}{\mathbf{g}}_k^{\prime })$$

(19)

According to [12], the projection problem (18) can be transformed into a knapsack problem, and the projection problem (19) is the well-known $L_0$ norm projection. To summarize, the complete algorithm for training an energy-constrained depth estimation DNN is given in Algorithm 2.

Algorithm 2 Energy-Constrained Depth Estimation DNN Training

Require:  Mini-batch size $n_g$, stepsize $\eta$, exponential decay rates ${\beta }_1,{\beta }_2\in (0,1)$, mask sparsity decay step $▵q$, positive parameters $ϵ,\rho ,\theta$. Require:  $q=\left|\right|{\mathbf{M}}_t{\left|\right|}_0-▵q$ Require:  $k\leftarrow 0,t\leftarrow 0$        //Initialize timestep 1: while True do 2:    while not converged do 3:      Update weight tensor ${\mathbf{W}}_k$ through solving (18) 4:      $k\leftarrow k+1$ 5:    end while          //If accuracy decreases, exit loop with ${\mathbf{W}}_k$ and ${\mathbf{M}}_t$; 6:    while not converged do 7:         Update input mask ${\mathbf{M}}_t$ through solving (19) 8:         $t\leftarrow t+1$ 9:    end while             Round values of ${\mathbf{M}}_{t+1}$ into {0,1}      //Decay the sparsity constraint $q=q-▵q$ 10: end while 11: return $x_{k+1},M_{t+1}$

5. Experiment

To validate the efficiency of our method, we perform extensive experiments on KITTI-2012 [17] and KITTI-2015 [18] using the depth estimation network AnyNet [7]. The experiments can be divided into three parts: First, we test the performance of AnyNet trained with a GPU under different energy-cost budgets, and the results are compared with the projected first order optimizer ProjAdam. Then, we apply three existing pruning methods (including L1-norm pruning [19], BN pruning [20] and Lottery-Ticket [21]) to AnyNet under the same energy budget for comparison. FInally, the performance of the trained models are also accessed on the embedded device Nvidia Jetson AGX Xavier [22].

Implementation Details We implement AntNet with four stages, and the output of higher stages have better performance with more time cost. The hyper-parameters of the network are set according to the default values. The DNN framework we experiment on is PyTorch1.10.0 with python3.6 and is GPU-accelerated through Cuda-1.13. The hardware is a single RTX 3090Ti with I9-10920X CPU, while the RAM is 32GB. Following AnyNet, we use the metric three-pixel error to evaluate the performance (lower is better). In addition, the predicted depth maps are also enhanced through histogram equalization [33] for comparison.

Dataset The training set of KITTI-2012 contains 194 image pairs, while the test set contains 195 image pairs. Both the training set and test set of KITTI-2015 have 200 image pairs.

Experiment Setup We perform a careful hyper-parameter tuning for optimizers in experiments as follows:

ProjACQN: We set $\theta =1$, ${\beta }_1=0.9$, ${\beta }_2=0.999$, $ϵ={10}^{-8}$, $\rho =1$. The learning rates for weight and input mask update are set to $\eta =5\times {10}^{-4}$ and $\eta =1\times {10}^{-4}$, respectively. Meanwhile, the weight decay for weight and input mask update are set to $\eta =1\times {10}^{-4}$ and $\eta =1\times {10}^{-5}$, respectively.

ProjAdam: The learning rates are searched among $\{a\times {10}^b\}$ where $a\in \{1,2,3,4,5,6,7,8,9\}$ and $b\in \{-5,-4,-3\}$, while the weight decay for weight update and input mask update are set to the same as that of ProjACQN. Other parameters are set as their default values in the literature.

5.1. Results on GPU under Different Energy Budgets

Here, we test the performance of AnyNet trained through GPU under 50%, 60% and 70% energy budgets. Quantitative results on KITTI-2012 and KITTI-2015 are shown in

Table 1. Three-pixel error (%) and resulting energy consumption of AnyNet with 50%, 60% and 70% energy budget on KITTI-2012 dataset. A lower three-pixel error is better.

	Energy Budget	Energy	Three-Pixel Error (%)
			Stage 1	Stage 2	Stage 3	Stage 4
Dense Model	100%	97.6%	14.65	9.27	6.21	5.60
ProjAdam	70%	67.4%	15.42	11.87	6.97	5.98
	60%	55.4%	16.12	13.83	10.40	11.22
	50%	49.5%	16.06	13.98	10.84	40.55
ProjACQN	70%	65.5%	16.11	9.38	6.48	5.85
	60%	55.8%	16.35	9.32	6.63	6.09
	50%	49.3%	17.15	12.24	11.13	13.65

and

Table 2. Three-pixel error (%) and resulting energy consumption of AnyNet with 50%, 60% and 70% energy budget on KITTI-2015 datases. A lower three-pixel error is better.

	Energy Budget	Energy	Three-Pixel Error (%)
			Stage 1	Stage 2	Stage 3	Stage 4
Dense Model	100%	97.6%	13.24	9.12	6.30	5.75
ProjAdam	70%	67.1%	13.20	9.01	6.50	5.74
	60%	55.3%	20.63	17.70	14.15	13.40
	50%	49.8%	16.06	13.98	10.84	40.55
ProjACQN	70%	67.2%	12.79	8.60	5.66	5.13
	60%	57.4%	13.64	8.83	6.29	5.64
	50%	49.0%	17.12	12.25	11.13	13.69

, respectively. On KITTI-2012, the four stage results of ProjACQN and ProjAdam are comparable to the dense model under a 70% energy budget, while ProjACQN achieves a much lower three-pixel error under a 60% energy budget. On KITTI-2015, the four stage results of ProjACQN achieve a lower three-pixel error under 60% and 70% energy costs than ProjAdam and even outperform the dense model. This may be due to the removal of redundant information. Furthermore, ProjACQN achieves a comparable result under a 50% energy budget. We also show the curves of training loss on KITTI-2012 and KITTI-2015 under a 70% energy cost in Figure 1 and Figure 2, respectively. From these figures, we can see that ProjACQN achieves the best convergence speed. Figure 3 gives some visual examples of the predicted disparity.

5.2. Comparison between Different Pruning Methods

To comprehensively compare our method with prior work, we also slim AnyNet through three existing pruning methods including L1-norm pruning, BN pruning and Lottery-Ticket to AnyNet on KITTI-2015 and KITTI-2012. Here, we list the results under a 70% energy cost in

Table 3. Three-pixel error (%) of different pruning methods using AnyNet under 70% energy budget on KITTI-2012 and KITTI-2015 datasets.

Pruning Method	Dataset	Three-Pixel Error (%)
		Stage 1	Stage 2	Stage 3	Stage 4
BN	KITTI-2012	27.29	16.27	10.73	10.97
L1-Norm		26.14	19.99	10.01	9.83
Lottery-Ticket		24.25	19.14	10.71	10.67
ProjACQN		16.11	13.98	6.48	5.85
BN	KITTI-2015	39.62	26.20	15.13	15.02
L1-Norm		29.42	20.63	11.92	12.54
Lottery-Ticket		28.28	22.48	13.03	12.71
ProjACQN		12.79	8.60	5.66	5.13

. Our method achieves an evidently lower three-pixel error than the other methods.

5.3. Results on Embedded Device

In this section, we perform the model trained through ProjAdam and ProjACQN on an Nvidia Jetson AGX Xavier under 50%, 60% and 70% energy budgets. Quantitative results on KITTI-2012 and KITTI-2015 are shown in

Table 4. Three-pixel error (%) of AnyNet with 50%, 60% and 70% energy budget on KITTI-2012 dataset using AGV. A lower three-pixel error is better.

	Energy Budget	Three-Pixel Error (%)
		Stage 1	Stage 2	Stage 3	Stage 4
Dense Model	100%	18.23	19.91	11.21	10.64
ProjAdam	70%	22.76	15.37	14.20	14.05
	60%	21.52	17.37	12.16	13.74
	50%	18.90	14.06	9.30	64.20
ProjACQN	70%	15.27	12.70	8.14	8.25
	60%	15.20	13.73	8.32	8.48
	50%	16.74	16.21	12.18	12.44

and

Table 5. Three-pixel error (%) of AnyNet with 50%, 60% and 70% energy budget on KITTI-2015 datases using AGV. A lower three-pixel error is better.

	Energy Budget	Three-Pixel Error (%)
		Stage 1	Stage 2	Stage 3	Stage 4
Dense Model	100%	14.11	17.10	10.41	9.03
ProjAdam	70%	16.66	14.42	9.97	8.50
	60%	17.20	13.63	9.06	10.22
	50%	21.76	20.40	17.17	40.63
ProjACQN	70%	15.30	13.92	9.87	8.38
	60%	16.28	14.39	9.24	9.32
	50%	18.15	16.71	13.86	13.91

, respectively. It can be found that the stage four result of ProjACQN has an obvious advantage on KITTI-2012 and KITTI-2015, while the performance of the other three stages are comparable. The FPS of stage four using the dense model, ProjAdam and ProjACQN is 11.5, 20.4 and 20.31, respectively. It should be noted that the three-pixel error increased mostly due to the Float16 data type of the embedded device, which could be solved through quantization. Figure 4 and Figure 5 present some visual examples of disparity predictions from stage four of AnyNet under 50%, 60% and 70% energy budgets on KITTI-2102 and KITTI-2015, respectively. We can see that the segmentation results using ACQN-H is more similar to the dense model than ProjAdam. It is worth noting that the results are noisy under 50% and 60% energy budgets, due to the input mask.

6. Conclusions

We have presented an approach to compress deep neural networks under a given energy constraint for depth estimation. The training process of the depth estimation DNN model is formulated as a constrained optimization problem, and can be solved through the proposed projected adaptive cubic quasi-Newton optimizer. Experiments show that our method can reduce the energy consumption of AnyNet by 30% while improving accuracy by 0.13% and 0.62% compared to the state-of-the-art method ProjAdam on KITTI-2012 and KITTI-2015, respectively. When comparing with existing pruning methods, ProjACQN also achieves the best three-pixel error. It is worth mentioning that, when performing the models on the embedded device Nvidia Jetson AGX Xavier, ProjACQN with a 70% energy budget is able to outperform the dense model without an energy budget in terms of three-pixel error and time consumption.