Animal Pose Estimation Based on 3D Priors

Abstract

Animal pose estimation is very useful in analyzing animal behavior, monitoring animal health and moving trajectories, etc. However, occlusions, complex backgrounds, and unconstrained illumination conditions in wild-animal images often lead to large errors in pose estimation, i.e., the detected key points have large deviations from their true positions in 2D images. In this paper, we propose a method to improve animal pose estimation accuracy by exploiting 3D prior constraints. Firstly, we learn the 3D animal pose dictionary, in which each atom provides prior knowledge about 3D animal poses. Secondly, given the initially estimated 2D animal pose in the image, we represent its latent 3D pose with the learned dictionary. Finally, the representation coefficients are optimized to minimize the difference between the initially estimated 2D pose and the 2D projection of the latent 3D pose. Furthermore, we construct 2D and 3D animal pose datasets, which are used to evaluate the algorithm’s performance and learn the 3D pose dictionary, respectively. Our experimental results demonstrate that the proposed method makes good use of the 3D pose knowledge and can effectively improve 2D animal pose estimation.

1. Introduction

In recent years, animal pose estimation has attracted more and more attention owing to its potential applications in biology, neuroscience, ecology, agriculture, and entertainment [1,2,3,4]. For instance, animal pose estimation [5,6,7] could be employed in markerless motion capture systems to capture the movements of wild animals. Animal pose estimation [8,9] would also support advances in entertainment, where animal animation is still largely performed manually. In neuroscience [3,10], tracking animals is fundamental for understanding behavior or relating motion to brain activity. In bio-inspired robotics [11,12], understanding how animals move can help to design more efficient robots.

Despite the promising applications in various fields, there are few works on animal pose estimation to date. This has much to do with the challenges posed by the following factors. Firstly, research on animal pose estimation is largely limited by data scarcity. Typical human pose estimation benchmarks, such as Human3.6M [13], capture data from 15 sensors and rely on small markers attached to the subject’s body to obtain motion. However, such data acquisition methods are hard to be applied to animals because it is impractical to ask animals to cooperate during data acquisition, especially in wild conditions. Secondly, due to occlusions, complex backgrounds, and unconstrained illumination conditions in the wild, the quality of the collected images is often uncontrollable, resulting in unreliable animal pose estimation. Thirdly, the variations in animal poses are far more complicated than in humans. The high deformations of animal pose (or body) make annotating and predicting animal pose quite challenging.

Most existing methods for animal pose estimation focus on addressing data scarcity by transferring knowledge from other more accessible domains such as synthetic animal data [6,14,15,16] or human data [17]. However, there are domain gaps [18,19] between synthetic images and real images, which prevent models trained on synthetic data from generalizing well to real-world images. Furthermore, 2D poses are the projection of 3D body configurations, during which the relationship between the joints in 3D poses is distorted. Consequently, the models that are trained with 2D annotations alone may not conform to the real configurations of animal poses. This problem can become severe, especially in wild-animal images with uncontrolled quality, for which large errors in animal pose estimation would be obtained using existing methods.

In general, the performance can be improved by preprocessing (such as image segmentation [20,21], image enhancement [22], etc.) or postprocessing. However, the preprocessing of wildlife images, such as denoising or enhancement, is a non-trivial task in itself. In this paper, we propose a method for estimating 2D animal poses with 3D priors, in which 3D priors are constructed with synthetic 3D poses and encoded in a 3D pose dictionary using sparse dictionary learning. As can be seen in Figure 1, taking tigers as an example, the proposed method can estimate more accurate 2D poses in poor-quality images.

A preliminary version of this work has been published in the 33rd British Machine Vision Conference (BMVC) [23]. This paper extends the BMVC paper in the following aspects: (i) We extensively compare the proposed algorithm with more algorithms. (ii) The proposed method is also evaluated on more datasets, which demonstrates its good generalization. (iii) This paper considers more widely distributed noise for more comprehensive and detailed performance evaluation and analysis. (iv) We implement the proposed method through both traditional optimization and deep learning. The contributions of this paper can be summarized as follows:

• We propose a novel method for refining 2D animal pose estimation using 3D priors, which can be easily incorporated into existing 2D pose estimation methods;
• We present both conventional optimization and learning-based neural networks to implement the proposed method;
• We build a 3D animal pose dataset and manually annotate a 2D pose dataset for animal pose estimation;
• Extensive experiments are conducted to evaluate the proposed method. The experimental results show that the proposed method is effective in improving 2D animal pose estimation accuracy.

The remainder of this paper is organized as follows: We discuss related work and motivation in Section 2. Section 3 introduces the establishment of the relevant datasets. Section 4 provides details of the proposed method. Our experimental evaluation results are presented in Section 5. Finally, we finish the paper with concluding remarks in Section 6.

2. Related Work

2.1. The 3D Human Pose Estimation Methods

2.1.1. Deep-Learning-Based Human Pose Estimation

Thanks to deep learning and large-scale datasets, significant progress has been made in 3D human pose estimation. The existing deep-learning-based human pose estimation methods can be divided into the following categories:

Full Supervision. Supervised approaches rely on large datasets that contain millions of images with corresponding 3D pose annotations. Martinez et al. [26] trained a neural network on 2D poses and corresponding 3D ground truth. Due to its simplicity, it can be quickly trained for a large number of epochs leading to high accuracy and serves as a baseline for many following approaches. Recently, Transformer [27] equipped with a global self-attention mechanism has also become popular in 3D pose estimation [28,29]. However, the major downside of all supervised methods is that they do not generalize well to images with unseen poses.

Weak Supervision. Weakly supervised approaches require only a small set of annotated 3D poses or unpaired 2D and 3D poses [30,31,32]. Wandt et al. [30] used a discriminator to represent the prior distribution of 3D poses. Meanwhile, in order to avoid the over-fitting problem that may be caused by using only the projection error, a camera network was also introduced to estimate the camera parameters. Based on [30], Li et al. [31] introduced a random variable to explicitly generate multiple 3D poses from a 2D input and select the optimal 3D pose from them. Wandt et al. [33] decomposed the observed 2D pose into a 3D pose and camera rotation using multi-view consistency constraints. Compared with completely supervised approaches, these weakly supervised methods generalize and transfer better to new domains. However, they still struggle with poses that are very different from the annotated training set.

Unsupervised. Drover et al. [34] proposed an unsupervised learning approach to monocular human pose estimation. They randomly projected an estimated 3D pose back to 2D. Chen et al. [35] extended [34] with a cycle consistency loss that was computed by lifting the randomly projected 2D pose to 3D and inversing the previously defined random projection. Yu et al. [36] further introduced a learnable scaling factor for the input 2D poses. Wandt et al. [37] improved their results by learning the camera distribution of the training set. The unsupervised methods do not generalize better than supervised ones, but they allow training models without 3D priors.

2.1.2. Dictionary-Based Human Pose Estimation

Dictionary-based methods are also widely used in human pose estimation [38,39]. In these methods, a 3D pose is defined by a set of joints and is assumed to be represented by a linear combination of predefined pose bases and sparse coefficients. Given the 2D correspondence of the joints in a single image, the calculation problem is to simultaneously estimate the coefficients of the sparse representation as well as the viewpoint of the camera. For example, the authors of [40] proposed a sparse representation-based approach to estimate a 3D human pose from 2D annotations in a single image. They presented a projected matching pursuit algorithm for reconstructing 3D poses and camera settings by minimizing the reprojection error. Wang et al. [41] proposed to estimate the 3D pose by minimizing an L1-norm penalty between the projection of 3D joints and 2D detections to reduce the impact of inaccurate 2D pose estimations. Zhou et al. [38] adopted an augmented 3D shape model to achieve a linear representation of shape variability in 2D and proposed to use spectral-norm regularization to penalize invalid cases caused by augmentation. Akhter et al. [42] integrated joint-angle limits into the sparse representation to reduce the possibility of invalid reconstruction. Such methods have achieved promising results in 3D human pose estimation, inspiring us to exploit a pose dictionary in animal pose estimation.

2.2. Animal Pose Estimation

Recently, deep-learning-based approaches [25,43,44,45] have made significant progress in human pose estimation. Attributed to these methods, the studies of [1] and [46] are the first to use convolutional neural networks [25,43,47] for animal pose estimation. Mathis et al. [1] developed a pose estimation model, which they called DeepLabCut, by modifying a previously published human pose model called DeeperCut [48]. The DeepLabCut model, like the DeeperCut model, was built on the popular ResNet architecture [47], which is one of the most advanced deep-learning models for image classification. Pereira et al. [46] implemented a modified version of a model called SegNet [49], which they called LEAP. Graving et al. [2] presented a multi-scale deep-learning model, called DeepPoseKit, which is based on the DenseNet [50].

It is appealing to directly estimate animal pose via extending the deep-learning algorithms designed for human subjects, such as DeepLabCut [1], LEAP [46], and DeepPoseKit [2]. These models have been widely used for laboratory animals (e.g., fruit flies or mice). However, they are hardly applicable for non-laboratory animals due to the higher cost of data acquisition and annotations. To solve data limitations, the authors of [17] proposed a cross-domain adaptation scheme to learn a shared feature space between human and animal images, so that their network can learn from existing human pose datasets. Furthermore, they also designed a “progressive pseudo-label-based optimization” (PPLO) to improve model performance by bringing the target domain data into training with pseudo-labels. Mu et al. [15] used the synthetic animal data generated from CAD models to train their model, which was then used to generate pseudo-labels for unlabeled real animal images. To handle noisy pseudo-labels, they designed three consistency constraints to evaluate the quality of the predicted labels. Li et al. [16] designed a multi-scale domain adaptation module (MDAM) to reduce the domain gap between the synthetic and real data. Furthermore, a coarse-to-fine pseudo-label updating strategy was further introduced to gradually replace noisy pseudo-labels with more accurate ones during training.

In addition, there are also several works focusing on shape reconstruction. Kanazawa et al. [51] learned the deformation and stiffness of 3D animal meshes from manually annotated 2D images. Zuffi et al. [14] introduced the skinned multi-animal linear model (SMAL), which is similar to the popular skinned multi-person linear model (SMPL) [52] for humans. Zuffi et al. [6] fit the SMAL model to several images of the same animal and then refined the shape to better fit the image data, so as to obtain shape and texture (SMAL with Refinement, SMALR). These algorithms [6,14,51] also require ground-truth pose to train the models.

Despite the improvements made by these approaches [1,2,15,16,17,46], animal pose estimation is still non-trivial. These methods are trained on images with 2D pose annotations, which may not follow the real distribution of animal poses and very likely degrade their performance, especially for poor-quality images. In contrast to these methods, this paper aims to explore the 3D knowledge of animals for refining estimated animal poses such that the misaligned key points caused by image noise can be amended toward their true positions. Inspired by [38,39,40,41,42], we used a 3D pose dictionary to encode 3D prior constraints, which is simple and effective. Note that previous studies [38,39,40,41,42] require large-scale real 3D human poses as the training data for the dictionary, but collecting real 3D animal pose data is very difficult if not impractical. To address the lack of 3D animal pose data, we collected and synthesized data for 3D pose dictionary learning.

3. Dataset Collection

3.1. Cat

We built a cat dataset by collecting images on the Internet, which contained more than 400 images. This dataset, called Cat, was used to learn a 3D pose dictionary. The details are as follows: First, we defined the key points with reference to [51] and annotated them manually (as illustrated in Figure 2 and

Table 1. Definition of key points in the proposed dataset, Cat.

Index	Definition	Index	Definition	Index	Definition	Index	Definition
1	forehead	11	left shoulder	21	right front ankle	31	right ankle
2	spine 0	12	left front thigh	22	right front toe	32	right toe
3	spine 1	13	left front shin	23	left thigh	33	left ear
4	spine 2	14	left front foot	24	left shin	34	left eye outer corner
5	spine 3	15	left front ankle	25	left foot	35	left eye inner corner
6	spine 4	16	left front toe	26	left ankle	36	right ear
7	root of tail	17	right shoulder	27	left toe	37	right eye outer corner
8	tail 1	18	right front thigh	28	right thigh	38	right eye inner corner
9	tail 2	19	right front shin	29	right shin	39	nose
10	end of tail	20	right front foot	30	right foot	40	chin

). Second, we used [51] to synthesize their deformations. Lastly, the joints on the 3D deformed shapes could form a 3D pose, which is explained below. These joints were defined with reference to ATRW [24]. As illustrated in Figure 3, there were 15 joints related to ears, nose, shoulders, paws, hips, knees, root of tail, and center (i.e., the midpoint of nose and root of tail).

3.2. Amur

To evaluate the effectiveness of our proposed method for low-quality images, we constructed another dataset by recycling the low-quality frames in the public ATRW dataset [24]. The ATRW was proposed for tiger detection, pose estimation, and reidentification. However, it provides ground-truth annotations only for good-quality images. Hence, we manually marked the key points for the low-quality Amur images in the dataset and called the obtained dataset Amur. This dataset was used for evaluating the performance of our proposed method in refining the key points detected in low-quality images.

4. Proposed Method

It is well known that 2D poses are projections of 3D configurations. From this 2D projection alone, human observers are able to effortlessly imagine its possible 3D pose and camera position. Therefore, we assumed that prior knowledge about 3D poses can be encoded by atoms in a 3D pose dictionary. In particular, when the image quality is poor, and the textural information is missing, the 3D pose dictionary can provide 3D structure information, which could effectively help 2D pose estimation. The overview of the proposed method is shown in Figure 4, which can be roughly divided into three modules:

1.

Pose dictionary learning: First of all, 3D poses were generated, as described in Section 3.1. These 3D poses were then used as the training data for dictionary learning to obtain the 3D pose dictionary (see Section 4.1);

2.

Initial pose estimation: The initial pose of the sample was estimated using existing pose estimation algorithms such as HRNet [25];

3.

Pose refinement: The initial pose p was used together with the 3D pose dictionary B to obtain the latent 3D pose P, which was then reprojected to obtain a more accurate 2D pose $p^{\prime }$ (see Section 4.3).

4.1. Pose Dictionary Learning

We used dictionary learning to find a good set of bases for 3D poses, and the dataset established in Section 3.1 was used as training data. We believe that a reasonable 3D configuration space of 3D poses can be generated on these bases, so as to make the sparse representation of any 3D pose possible. Our task of dictionary learning can be formulated as follows:

$$\begin{array}{c}\underset{B_{1,...,}{B}_K,C}{min}\sum _{n=1}^N{\displaystyle \frac{1}{2}}{∥P_n-\sum _{k=1}^K{C}_{k,n}{B}_k∥}_F^2+\lambda {∥C∥}_1\\ s.t.C_{k,n}\ge 0,{∥B_k∥}_F\le 1\\ k\in [1,K],n\in [1,N]\end{array}$$

(1)

where $\lambda$ is a non-negative parameter, K is the size of the dictionary, N represents the number of training samples, $P_n$ denotes a 3D pose in the collected dataset, $B_k$ is the base pose to be learned, and $C_{k,n}$ represents the n-th coefficient of the representation of $P_n$. The two terms in the cost function corresponding to the reconstruction error and the sparsity of representation, respectively. Examples of learned dictionary atoms are shown in Figure 4.

The problem in Equation (1) is locally solved by alternately updating C and $B_k$s via the projected gradient descent, an algorithm widely used in dictionary learning that converges to a local optimum [53]. The cost function can be rewritten as

$$f(\tilde{B},C)={\displaystyle \frac{1}{2}}{∥P_n-\sum _{k=1}^K{C}_{k,n}{B}_k∥}_F^2+\lambda \sum _{k,n}{C}_{k,n}$$

(2)

where $\tilde{B}$ is the concatenation of $B_k$s. These steps are summarized in Algorithm 1.

Algorithm 1 Pose Dictionary Learning

Input:$P_1,\cdots ,P_N$ Output:$B_1,\cdots ,B_K$ 1: initialize $\tilde{B},C,$ step size ${\delta }_1$ and ${\delta }_2$; 2: while not converged do 3:     while not converged do 4:         $C\leftarrow C-{\delta }_1{\nabla }_{c}f(\tilde{B},C)$ 5:         for $k=1$ to K do 6:            for $n=1$ to N do 7:                if $C_{k,n}<0$ then 8:                    $C_{k,n}\leftarrow 0$ 9: $\tilde{B}\leftarrow \tilde{B}-{\delta }_2{\nabla }_{\tilde{B}}f(\tilde{B},C)$ 10: for$k=1$ to K do 11:     if ${∥B_k∥}_F>1$ then 12:         $B_k\leftarrow B_k/{∥B_k∥}_F$

4.2. Initial Pose Estimation

It is worth emphasizing that the proposed method works as a plug-in postprocessing module and can be attached to existing animal pose estimation methods. Existing animal pose estimation methods can be divided into two categories: fully supervised learning and domain adaptation. In this paper, we chose state-of-the-art algorithms from the two categories for initial pose estimation. Mu et al. [15] proposed a novel consistency-constrained semi-supervised learning method (CC-SSL) to bridge the domain gap between real and synthetic images. Li et al. [16] designed a multi-scale domain adaptation module for unsupervised domain adaptation (UDA) on animal pose estimation. In addition, fully supervised methods such as Hourglass [43], ResNet [47], and HRNet [25], which perform well in human pose estimation, were also used for initial animal pose estimation.

4.3. Pose Refinement

4.3.1. Optimization-Based Pose Refinement

In this paper, we simplify the relationship between 3D and 2D poses as follows:

$$p\approx \Pi P$$

(3)

where $p\in {\mathbb{R}}^{2\times \mathrm{J}}$ and $P\in {\mathbb{R}}^{3\times \mathrm{J}}$ denote 2D and 3D poses, respectively. $\mathrm{J}$ is the number of joints or key points. $\Pi$ is usually defined based on the weak perspective camera model as

$$\Pi =\left[\begin{array}{ccc}a& 0& 0\\ 0& a& 0\end{array}\right]$$

(4)

where a is a scalar depending on the focal length and the distance to the object [38].

According to Equation (3), 2D pose p can be obtained if the latent 3D pose P is accessible. However, estimating a 3D pose from a 2D pose is an ill-posed problem. Fortunately, this pursuit is actually not in vain, as it seems, and attempts in the field of 3D human pose estimation have justified that lifting 2D poses to 3D poses is very fruitful [26,38]. We assume that the latent 3D pose has a combined representation:

$$P=\sum _{k=1}^K{c}_k{R}_k{B}_k$$

(5)

where $B_k\in {\mathbb{R}}^{3\times \mathrm{J}}$ for $k\in [1,K]$ represents a base pose in the learned dictionary. While $c_k$ denotes the weight of each base pose, $R_k$ is a rotation matrix. Then, we substitute Equation (5) into Equation (3) to obtain

$$\begin{array}{c}p\approx \Pi P=\Pi \sum _{k=1}^K{c}_k{R}_k{B}_k=\sum _{k=1}^K{M}_k{B}_k\\ s.t.M_k{M}_k^T=c_k^2{I}_2,k\in [1,K]\end{array}$$

(6)

where $M_k=c_k{\overline{R}}_k$, ${\overline{R}}_k$ is the first two row vectors of $R_k$, and $I_2$ is the unit matrix of size $2\times 2$.

Further, we use the following objective function to estimate the latent 3D pose:

$$\begin{array}{c}\underset{M_{1,...,}{M}_K}{min}{\displaystyle \frac{1}{2}}{∥p-\sum _{k=1}^K{M}_k{B}_k∥}_F^2+\alpha \sum _{k=1}^K{∥M_k∥}_2\\ s.t.M_k{M}_k^T=c_k^2{I}_2,k\in [1,K]\end{array}$$

(7)

where $\alpha$ is a predefined coefficient of the regularization. An auxiliary variable Z is introduced, and Equation (7) is rewritten as

$$\begin{array}{c}\underset{\tilde{M},Z}{min}{\displaystyle \frac{1}{2}}{∥p-Z\tilde{B}∥}_F^2+\alpha \sum _{k=1}^K{∥M_k∥}_2\\ s.t.\tilde{M}=Z,\tilde{M}=\left[\begin{array}{c}{M}_1\cdots M_K\end{array}\right],\tilde{B}=\left[\begin{array}{c}{B}_1\\ ⋮\\ B_K\end{array}\right]\end{array}$$

(8)

The steps are summarized in Algorithm 2. With ${\left\{M_k\right\}}_1^K$ we can finally obtain our refined 2D pose $p^{\prime }$, which is the projection of the estimated latent 3D pose P, i.e.,

$$p^{\prime }=\sum _{k=1}^K{M}_k{B}_k$$

(9)

Algorithm 2 Pose Refinement

Input:$p,\alpha$ Output:$M_1,\cdots ,M_K$ 1: initialize $Z=Y=0,\mu >0$; 2: while not converged do 3:     for $k=1$ to K do 4:         $Q_k^t=Z_k^t-{\displaystyle \frac{1}{\mu }}{Y}_k^t$ 5:         $M_k^t={\mathcal{D}}_{\frac{\alpha }{\mu }}\left(Q_k^t\right)$ 6:     $Z^{t+1}=(p{\tilde{B}}^T+\mu {\tilde{M}}^{t+1}+Y^t){(\tilde{B}{\tilde{B}}^T+\mu I)}^{-1}$ 7:     $Y^{t+1}=Y^t+\mu ({\tilde{M}}^{t+1}-Z^{t+1})$ 8: where ${\mathcal{D}}_{\frac{\alpha }{\mu }}$ is proximal operator [38].

4.3.2. Deep-Learning-Based Pose Refinement

We provide a deep-learning-based implementation, in which both the coefficients $c_k$ and the rotation matrix $R_k$ in Equation (6) are estimated using a neural network. Figure 5 shows the pipelines. The proposed network consists of the coefficient network and camera network. Given the initial pose p as network input, the coefficient network and camera network estimate coefficients $c_k$ and rotation matrix $R_k$ in Equation (6), respectively. The coefficients $c_k$ and the rotation matrix $R_k$ estimated via the network are combined with the learned 3D pose dictionary B to generate the 3D pose P corresponding to the initial pose p. Finally, the 3D pose P is reprojected to obtain the refined pose $p^{\prime }$.

Network Architecture. The 2D input vectors are connected to a fully connected layer to expand the dimensionality to 1024 and then fed into subsequent residual blocks. Afterward, the network splits into two paths that predict the coefficients and the rotation matrices. Each path has two consecutive residual blocks followed by a fully connected layer that downscales the features to the required scale.

5. Experiments

We evaluate the effectiveness of the proposed method in this section. As a pose refinement module, the proposed method can be regarded as the postprocessing of existing pose estimation algorithms. Therefore, we applied the proposed method to state-of-the-art pose estimation methods and evaluated its effectiveness on different datasets.

5.1. Datasets

Synthetic animal. The synthetic animal dataset in [15] consists of images of elephants, horses, hounds, sheep, and tigers. In this dataset, animal textures and backgrounds are randomly synthesized using the COCO dataset [54]. For each animal species, 5000 images were generated with random textures and 5000 images with textures derived from the original CAD model. We only used the tiger subset, denoted by SA-Tiger, in our experiments (see Figure 6a).

TigDog. The TigDog dataset [55] provided key-point annotations for horses and tigers, where the images were taken from YouTube (for horses) and National Geographic documentaries (for tigers). Each image was annotated with 19 key points. These key points were distributed as follows: two key points on the eyes, one key point on the chin, two key points on the shoulders, twelve key points on the legs, one key point on the hip, and one key point on the neck. Among them, 11 key points overlapped with those defined by the ATRW [24]. These overlapping key points were used for evaluation in this paper. For convenience, we call the subset of tiger images in this dataset TD-Tiger (see Figure 6b).

Amur. As described in Section 3.2, Amur is a tiger image dataset. The image quality in this dataset is poor due to factors such as complex backgrounds, unconstrained illumination, and four-limbed movement. We manually annotated these samples for experimental evaluation (see Figure 6c).

5.2. Experimental Setup

As in [56], we used the percentage of correctly localized key points (PCK) as the metric for evaluation. For the i-th sample in the test set, PCK defines the predicted position of the j-th key point, ${\tilde{y}}_i^j$, which would be correct if it falls within a threshold of its ground-truth position $y_i^j$.

$${∥y_i^j-{\tilde{y}}_i^j∥}_2\le \beta \mathcal{D}$$

(10)

where $\mathcal{D}$ is the reference normalizer, namely the maximum side length of the image bounding box for animals, and $\beta$ controls the threshold for correctness. $\beta$ was set to $0.05$ as in [56].

5.3. Results on the SA-Tiger Dataset

In order to analyze the robustness of the proposed method to noise, we added noise to ground truth, just as [57]. The corrupted dataset was then used to evaluate the performance of the proposed 3D constraints in correcting the corrupted 2D poses. As in [57], we estimate the scale s by finding the maximal length of the bounding box along $x,y-axis$. Then, we sampled zero-mean Gaussian noise with standard variance $\sigma =\delta \%\times s$, where $\delta$ were set to 3, 4, 5, and 6.

The SA-Tiger dataset had 10,000 samples, which were randomly divided into 6000 training samples and 4000 testing samples. We trained the neural networks in both fully supervised and self-supervised ways. Fully Supervised: The initial pose p was used as the network input. c and R were estimated using the coefficient network and camera network, respectively, and then combined with the learned dictionary B to obtain the refined pose $p^{\prime }$ ( $\Pi {\sum }_{k=1}^K{c}_k{R}_k{B}_k$). The network was trained by calculating the error between the refined pose $p^{\prime }$ and the ground-truth pose $p_{gt}$, as in Equation (11). Self-supervised: The loss function was the error between the input pose (i.e., the initial pose p) and the refined pose $p^{\prime }$ (see Equation (12)).

$${\mathcal{L}}_{reproj}^{full}={\parallel p^{\prime }-p_{gt}\parallel }_F^2$$

(11)

$${\mathcal{L}}_{reproj}^{self}={\parallel p^{\prime }-p\parallel }_F^2$$

(12)

The different scales of noise indirectly reflect the difficulty levels of the pose estimation tasks. GT+$\mathcal{N}(0,\delta )$ simulates different initial pose p and then evaluates the refinement performance of different coefficient solutions. It can be seen from

Table 2. [email protected] accuracy on SA-Tiger with different levels of Gaussian noise added to the ground truth(GT) pose (%). The best results are marked in bold.

Method	Ear	Nose	Shoulder	Front Paw	Hip	Knee	Back Paw	Tail	Center	Mean
GT+$\mathcal{N}(0,3)$	75.9	75.2	75.5	75.9	75.0	74.4	74.1	75.6	75.8	75.3
Ours (Optimization)	75.6	77.4	78.3	76.5	78.3	76.9	75.1	80.7	81.1	77.2
Ours (Fully supervised)	85.7	86.2	89.4	83.5	86.9	87.9	87.3	86.2	95.1	87.3
Ours (Self-supervised)	67.2	65.0	77.7	69.2	71.9	76.3	69.4	71.3	38.7	67.3
GT+$\mathcal{N}(0,4)$	55.0	54.1	53.8	53.5	55.2	54.5	54.0	54.8	53.0	54.2
Ours (Optimization)	55.2	57.4	61.4	54.3	60.3	59.9	56.0	59.8	64.5	58.0
Ours (Fully supervised)	73.8	71.7	83.0	71.0	79.0	78.9	77.3	73.3	90.6	77.3
Ours (Self-supervised)	49.0	52.8	61.2	50.4	61.5	58.3	49.9	55.2	36.7	51.9
GT+$\mathcal{N}(0,5)$	40.0	38.5	38.7	40.6	39.3	39.3	39.7	40.3	39.7	39.7
Ours (Optimization)	41.0	42.8	46.0	40.6	48.3	44.8	41.3	45.5	54.3	44.2
Ours (Fully supervised)	62.0	60.8	75.1	59.5	69.7	69.0	67.4	63.7	82.8	67.3
Ours (Self-supervised)	37.0	38.9	48.3	37.4	49.0	40.9	36.4	43.0	34.4	39.7
GT+$\mathcal{N}(0,6)$	29.2	29.1	29.5	29.0	29.5	30.3	29.0	28.5	30.5	29.4
Ours (Optimization)	29.8	32.5	38.5	30.2	40.3	35.7	30.3	34.8	43.6	34.2
Ours (Fully supervised)	51.0	50.0	67.6	50.8	61.3	58.4	58.0	54.1	74.2	57.8
Ours (Self-supervised)	26.8	31.3	36.7	27.3	41.5	31.5	25.8	31.0	27.8	30.1

that full supervision achieves the greatest improvement. However, in practical applications, it is difficult to obtain ground-truth data with the same distribution as the pose to be refined for training. In addition, there is no significant improvement in self-supervision. These indicate that the network heavily relies on training data. In contrast, optimization-based pose refinement has better practicability and generalization. Specifically, the improvement is 1.9%, 3.8%, 4.5%, and 4.8% respectively, and the top two gains are achieved on GT+$\mathcal{N}(0,5)$ and GT+$\mathcal{N}(0,6)$. Thus, when we mention the proposed method in the following sections, the optimization-based pose refinement is indicated unless specified.

5.4. Results on the TD-Tiger Dataset

The quality of animal data varies in a wide range, especially those collected in the wild. To investigate the performance of the proposed method on images with different qualities, we divided the images into different subsets according to the PCK values of the initially estimated poses. Specifically, we set three intervals of PCK values, $\left(0,45\right]$, $\left(45,65\right]$, and $\left(65,100\right]$. We evaluated the effectiveness of the proposed method for images in different intervals, respectively.

It is now known that in CC-SSL [15] and UDA [16], the trained model can be used directly. However, Hourglass [43], ResNet [47], and HRNet [25] are not currently designed for animal pose estimation on the TD-Tiger. Therefore, we trained these fully supervised networks according to the data settings of CC-SSL [15] and UDA [16]. Images in TD-Tiger were split into training and verification sets, with 6523 images for training and 1765 images for verification. The [email protected] accuracies obtained using the proposed method and different initial pose estimation methods on TD-Tiger are shown in

Table 3. [email protected] accuracy on TD-Tiger (%). The best results are marked in bold.

Method	$\left(0,45\right]$	$\left(45,65\right]$	$\left(65,100\right]$
Hourglass [43]	28.3	55.7	87.6
Ours	29.4	57.0	87.6
ResNet [47]	29.3	54.3	88.1
Ours	29.4	54.8	88.1
HRNet [25]	29.5	55.4	87.8
Ours	30.5	56.1	87.8
CC-SSL [15]	28.8	54.7	73.6
Ours	32.6	55.3	73.6
UDA [16]	30.0	54.7	75.0
Ours	34.7	56.4	75.0

. As can be seen from the results, while the proposed method maintained comparable accuracy for high-quality images, it obviously improved the accuracy for poor-quality images (i.e., in $\left(0,45\right]$). It should be noted that the initial pose for high-quality images was already fairly accurate (i.e., in $\left(65,100\right]$), and refinement may not be necessary. The success of the proposed method proves that the deviated key points can be effectively pulled back toward their true positions by exploiting 3D prior constraints on the animal poses.

5.5. Results on the Amur Dataset

HRNet [25] and ResNet [47] trained for tiger pose estimation have been provided by MMPose [58]. Following the MMPose [58], we trained the Hourglass [43] using 2193 images from the training set in the ATRW [24]. For a fair comparison, we retrained CC-SSL [15] and UDA [16]. The training strategy was as follows: The source domain was the Synthetic Animal [15], which was the same as CC-SSL [15] and UDA [16]; the target domain training data were the training set in the ATRW [24]. As can be seen, with the proposed method, all the initial models had improvements in performance or were comparable to initial results. For example, UDA [16] achieved a $4.4\%$ improvement in the interval $\left(0,45\right]$; Meanwhile, in the interval $\left(65,100\right]$, the proposed method was also comparable. On the whole, the proposed method had noticeably superior performance in challenging samples. This justifies the effectiveness of the proposed 3D constraints for 2D tiger pose estimation, particularly in challenging cases. The results of different methods on Amur are shown in

Table 4. [email protected] accuracy on Amur (%). The best results are marked in bold.

Method	$\left(0,45\right]$	$\left(45,65\right]$	$\left(65,100\right]$
Hourglass [43]	34.1	56.4	84.0
Ours	34.4	57.1	84.1
ResNet [47]	35.0	56.8	84.3
Ours	36.8	57.4	84.3
HRNet [25]	35.9	57.0	84.7
Ours	39.2	57.0	84.8
CC-SSL [15]	32.1	55.4	75.4
Ours	35.1	55.9	76.0
UDA [16]	34.4	55.2	76.5
Ours	38.8	56.1	77.2

and Figure 7.

6. Discussions and Conclusions

6.1. Discussions

As shown in Figure 7, the proposed method could effectively improve the animal pose estimation accuracy for poor-quality images due to occlusion, complex background, and uncontrollable illumination. In addition, we show two samples in which the animals were subject to the initial pose or the pose dictionary. In the first sample (Figure 8a), the initial poses completely violated the reasonable distribution of 2D poses, leading to irreparable errors. The second sample (Figure 8b) was not only limited by the initial pose but also by the pose dictionary. The training data for 3D pose dictionary learning were synthetic. If more real data can be used for training, a pose dictionary that is more robust to noise can be obtained. We will consider this in future work. Moreover, the dictionary encoded prior knowledge for specific classes of animals. It can be generalized between animal species with similar shapes, e.g., cats and tigers, but not between arbitrary animals. Therefore, obtaining a more diverse dictionary might be a potential way to further improve the estimation performance.

6.2. Conclusions

In this paper, we presented a method to estimate 2D animal poses with 3D constraints. The 3D constraints were constructed from synthetic 3D poses and encoded in the 3D pose dictionary using sparse dictionary learning. Extensive experiments were conducted to evaluate the proposed method. Experimental results showed that the constraints provided by the 3D pose dictionary produced more reasonable animal pose estimation. In addition, we constructed two datasets: Cat for 3D pose dictionary learning and Amur for algorithm evaluation. The findings revealed in this paper can promote the further development of animal pose estimation.