Making the Coupled Gaussian Process Dynamical Model Modular and Scalable with Variational Approximations ^†

Abstract

We describe a sparse, variational posterior approximation to the Coupled Gaussian Process Dynamical Model (CGPDM), which is a latent space coupled dynamical model in discrete time. The purpose of the approximation is threefold: first, to reduce training time of the model; second, to enable modular re-use of learned dynamics; and, third, to store these learned dynamics compactly. Our target applications here are human movement primitive (MP) models, where an MP is a reusable spatiotemporal component, or “module” of a human full-body movement. Besides re-usability of learned MPs, compactness is crucial, to allow for the storage of a large library of movements. We first derive the variational approximation, illustrate it on toy data, test its predictions against a range of other MP models and finally compare movements produced by the model against human perceptual expectations. We show that the variational CGPDM outperforms several other MP models on movement trajectory prediction. Furthermore, human observers find its movements nearly indistinguishable from replays of natural movement recordings for a very compact parameterization of the approximation.

1. Introduction

Two formidable problems that the human brain has to solve are planning and execution of movements of its body. As a means to simplify these problems while keeping a sufficient degree of control flexibility for a wide range of tasks, modular movement primitives (MP) have been suggested (see [1,2] for reviews). There is no universally accepted definition of the term “movement primitive”. For the purposes of this paper, an MP is a spatiotemporal component of a human (full-body) movement that may be produced by mapping a latent state onto observable variables, such as joint angles. The latent state can be generated by dynamical systems [3] or source functions [4,5]. “Modular” usually refers to the existence of an operation which allows for the spatial, temporal or spatiotemporal combination of (simple) primitives into (complex) movements.

Two prominent examples, where this operation is the linear combination of stereotypical time-courses or muscle-coactivations, are called temporal MP-models [6,7,8,9] or spatial MP-models [10,11]. While these models are inherently modular, the assumption of stereotyped MPs makes it difficult for a control system built out of these primitives to respond to perturbations. A type of MP which can be controlled on-line more easily is the dynamical MP (DMP) [3], which has been developed for robotics applications. In this approach, each primitive is encoded by a canonical second order differential equation with guaranteeable stability properties and learnable parameters. A DMP can generate both discrete (e.g., reaching) and rhythmic (e.g., walking) movements and drives the trajectory of one degree of freedom, e.g., a joint angle. Modularity arises because of the latter property, which might be viewed as an “extreme” form of the modularization that we investigate here, where one movement module might affect several degrees of freedom. Similarly, recent extensions of the DMP framework allow for the reuse of a DMP across end-effectors via kinematical mappings [12] or across tasks [13].

We describe a model that learns MPs composed of coupled dynamical systems and associated kinematics mappings, where both components are learned, thus lifting the DMP’s restriction of canonical dynamics. We build on the Coupled Gaussian Process Dynamical Model (CGPDM) by [14], which combines the advantages of modularity and flexibility in the dynamics, at least theoretically. In a CGPDM, the temporal evolution functions for the latent dynamical systems are drawn out of a Gaussian process (GP) prior [15]. These dynamical systems are then coupled probabilistically, and the result is mapped onto observations by functions drawn from another GP. One drawback of the CGPDM is its fully non-parametric nature, which results in cubic scaling (with the dataset size) of learning complexity and quadratic scaling of MP storage size, i.e., the CGPDM can not be learned from large data sets, and its effective parameterization is not compact. We improve both scalability and compactness with deterministic, sparse variational approximations [16]. In this sparse variational CGPDM, each MP is parameterized by a small set of inducing points (IPs) and associated inducing values (IVs), leading to a compact representation with linear scaling of the training complexity in the number of data points, and constant storage requirements. This compactness is important for real-world applicability of the model, since there might be more primitives than muscles (or actuators) across tasks, as pointed out by Bizzi and Cheung [17]: the motor “code” might be sparse and overcomplete, similar to the sparse codes in early vision [18].

Table 1. Overview of movement primitive models compared in this paper. (v)CGPDM, (variational) coupled Gaussian process dynamical model; (v)GPDM, (variational) Gaussian process dynamical model; TMP, temporal movement primitives; DMP, dynamical movement primitives. Modular, learns reusable MPs. Scalable, below cubic learning complexity with respect to the data set size; Compact, size of the effective parameterization does not grow with the data set size; Canonical dynamics, dynamics model specified before learning; Learned dynamics, dynamics model is a free-form function.

	Modular	Scalable	Compact	Canonical Dynamics	Learned Dynamics
vCGPDM	✓	✓	✓	✗	✓
CGPDM	✗	✗	✗	✗	✓
vGPDM	✗	✓	✓	✗	✓
GPDM	✗	✗	✗	✗	✓
TMP	✓	✓	✓	✗	✗
DMP	✓	✓	✓	✓	✗

provides an overview of the key MP models which we compare in this paper.

Our target application here is human movement modeling, but the vCGPDM could be easily applied to other systems where modularized control is beneficial, e.g., humanoid robotics [9].

We briefly review related work in Section 2 and introduce the vCGPDM in Section 3. The derivation of the variational approximation is outlined in Section 4. In Section 5, we first illustrate the vCGPDM on artificial data. Second, we benchmark the vCGPDM against other MP models. Third, we perform an experiment to quantify the degree of of human-tolerable sparseness in a psychophysics experiment. Fourth, we demonstrate modular movement composition with the vCGPDM. In Section 6, we propose future research directions based on our work.

This paper is a substantially extended version of our earlier conference publication [19].

2. Related Work

The Gaussian process (GP) is a machine learning staple for classification and regression tasks [15]. A GP is a prior on functions ${\mathbb{R}}^Q\to \mathbb{R}$ from a Q-dimensional input space to one-dimensional output. By drawing D times from the GP, functions from ${\mathbb{R}}^Q\to {\mathbb{R}}^D$ can be realized. Its advantages include theoretical elegance, tractability and closed-form solutions for posterior densities. Its main disadvantage is cubic runtime scaling with the number of data points. Several solutions have been proposed for this problem. Many of these involve a sparse representation of the posterior process via a small set of IPs, which may [20] or may not be a subset of the data points [21]. If the input space is unobserved, one obtains a GP latent variable model (GPLVM), for which sparse approximations have also been devised [22]. One problem with sparse GP approximations is their tendency to overfit [22], leading to incorrect variance predictions [23]. In that paper, it is also demonstrated that the problem can be alleviated by a variational approximation, which prompted us to develop a similar approach for the CGPDM: as in [24], we extend the sparse GPLVM in time, but we use an autoregressive dynamical system.

If the temporal evolution function of this dynamical system is also drawn from a GP, the resulting model is called Gaussian Process Dynamical Model (GPDM), which can be learned by maximum-a-posteriori approximation if the observed dimension D is greater than the latent dimension Q [25]. Figure 1 (left) shows a graphical model representation of the GPDM, and introduces the related notation which we use throughout the paper. Slices “:” indicate collections of variables along one integer index. Multiple slices refer to collections along multiple indices, e.g., ${\overrightarrow{x}}_{:}^{:}$ are the latent variables of all parts and time-steps. The GPDM can model the variability of human movements and has been used for computer animation with style control [26,27,28]. It has also been used with an additional switching prior on the dynamics for motion tracking and recognition [29] and deep variants have been devised [30]. However, with the exception of the coupled GPDM [14,19], all these approaches have a “monolithic” latent space and thus lack the modularity of MPs. One reason for this might be the fact that, for the maximum-a-posteriori approximation to work, the latent space has to be lower-dimensional than the observed space, $Q\ll D$. If, as explained above, we want a modular, possibly overcomplete (i.e., the effective $Q>D$) set of MPs, we need learning approaches that are robust to overfitting. The works of Frigola et al. [31] indicate that such approaches may be obtained with variational approximations. In the following, we therefore introduce a variational approximation to CGPDM learning and inference based on an approach similar to Frigola et al. [32], but, as in [30], we aim to obviate the need for sampling altogether to allow for fast, repeatable trajectory generation. While deriving a variational approximation is not trivial, we expect it to avoid overfitting and yield a good bound on the marginal likelihood [33]. Figure 1 (right) shows the graphical model of the GPDM augmented by IPs and IVs. This augmentation yields a tractable variational approximation to the GPDM’s posterior [30].

3. The Model

The basic building blocks, or “parts”, of the CGPDM are a number of GPDMs run in parallel. In the context of human movement modeling, e.g., they may be thought of as body parts. A part evolves in discrete time $t=0,\dots ,T$ and is endowed with a Q-dimensional latent space, a D-dimensional observed space and second-order autoregressive dynamics described by a function $\overrightarrow{f}({\overrightarrow{x}}_{t-2},{\overrightarrow{x}}_{t-1})$. The component functions ${\left(\overrightarrow{f}({\overrightarrow{x}}_{t-2},{\overrightarrow{x}}_{t-1})\right)}_q$ have a Gaussian process prior $\mathcal{GP}(\mu ,k(.,.))$ with mean function $\mu$ and kernel $k(.,.)$ Second-order dynamics seem to be a good choice for our target application of human movement modeling [34], but the order can be easily altered simply by concatenating previous states into one larger vector. Let ${\overrightarrow{x}}_t\in {\mathbb{R}}^Q$ the state of latent space of the part at time t (see Figure 1, left). This latent state produces observations ${\overrightarrow{y}}_t\in {\mathbb{R}}^D$ via the function $\overrightarrow{g}({\overrightarrow{x}}_t)$ as well as isotropic Gaussian noise with variance $\beta$. The components ${\left(\overrightarrow{g}({\overrightarrow{x}}_t)\right)}_d$ of this function are drawn from a Gaussian process prior, too. GPDMs can be learned from data via a combination of exact marginalization and maximum-a-posteriori learning of the latent dynamics [25].

While the GPDM is a very expressive model, it suffers from poor runtime and memory scaling with the data set size due to its non-parametric nature, which it inherits from the involved GPs. We remedied this problem by an approach pioneered in [23]: augmenting the GPs with inducing points (IPs, here: ${\overrightarrow{r}}_{:}$, ${\overrightarrow{z}}_{:}$) and associated inducing values (IVs, $\overrightarrow{v}$, $\overrightarrow{u}$) (see Figure 1, right). These IP/IV pairs might be thought of as prototypical examples of the mappings represented by the corresponding functions, e.g., ${\overrightarrow{v}}_k^i={\overrightarrow{g}}^i({\overrightarrow{r}}_k^i)$. Note that the IPs are not drawn from a prior, whereas the IVs are. Hence, the latter are model parameters, whereas the former are not: IPs are merely parameters of the approximation, or “variational parameters”. This augmentation allows for the derivation of a closed-form evidence lower bound (ELBO) on the marginal likelihood of the model.

In a CGPDM, the latent spaces of the parts are coupled to each other. We index parts by superscripts $i=1,\dots ,M$. The index notation in such models can be confusing for first-time readers, we provide a notation and index overview in the tables below Figure 2. In our target application, the coupling may reflect the influences which parts of an articulated body have on each other during the execution of a movement. The coupling is implemented by having the parts make Gaussian-distributed predictions ${\overrightarrow{x}}_t^{i,j}$ about each other’s latent states with means generated by $M\times M$ many mean coupling functions ${\overrightarrow{f}}^{i,j}({\overrightarrow{x}}_{t-2}^i,{\overrightarrow{x}}_{t-1}^i)$ and coupling variances ${\alpha }^{i,j}$. i indexes the origin part of a coupling, and j its target. Thus, ${\overrightarrow{f}}^{i,i}$ refers to the dynamics function for part i. The components of ${\overrightarrow{f}}^{i,j}({\overrightarrow{x}}_{t-2}^i,{\overrightarrow{x}}_{t-1}^i)$ are drawn from GPs. As described in [14], these predictions are combined with a product-of-experts construction (PoE [35]), including the predictions which a part makes about its own future. A product-of-experts construction forces the experts to agree on one prediction (Equation (1), left), which amounts to multiplying the individual predictions (Equation (1), right) and renormalizing (Equation (1), middle):

$$p({\overrightarrow{x}}_t^j|{\overrightarrow{f}}^{:,j}({\overrightarrow{x}}_{t-2}^i,{\overrightarrow{x}}_{t-1}^j),{\alpha }^{:,j})=\frac{exp\left[-\frac{1}{2{\alpha }^j}{\left({\overrightarrow{x}}_t^j-{\alpha }^j{\sum }_i\frac{{\overrightarrow{f}}^{i,j}({\overrightarrow{x}}_{t-2}^i,{\overrightarrow{x}}_{t-1}^j)}{{\alpha }^{i,j}}\right)}^2\right]}{{(2\pi {\alpha }^j)}^{\frac{Q^j}{2}}}\propto \prod _i\mathcal{N}\left({\overrightarrow{x}}_t^j|{\overrightarrow{f}}^{i,j}({\overrightarrow{x}}_{t-2}^i,{\overrightarrow{x}}_{t-1}^i),{\alpha }^{i,j}\right)$$

(1)

where ${\alpha }^j={\left({\sum }_i{({\alpha }^{i,j})}^{-1}\right)}^{-1}$.

To understand the function of the ${\alpha }^{i,j}$, consider the form of Equation (1): the smaller a given variance, the more important the prediction of the generating part. We optimize the ${\alpha }^{i,j}$ during learning, letting the model discover which couplings are important for predicting the data. In other words, whenever an ${\alpha }^{i,j}$ is small compared to ${\alpha }^{i^{\prime }\ne i,j}$, then part i is able to make a prediction about part j with (relatively) high certainty. Furthermore, the ${\alpha }^{i,j}$ can be modulated after learning to generate new movements, as shown below.

In the following, we denote all relevant timesteps before time t with subscript $-t$, e.g., ${\overrightarrow{x}}_{-t}^j=({\overrightarrow{x}}_{t-2}^j,{\overrightarrow{x}}_{t-1}^j)$ for a second-order dynamics model. We showed in [14] that the individual predictions of part i about part j, ${\overrightarrow{x}}_t^{i,j}$ can be exactly marginalized, leading to a GPDM-like model for each part with a dynamics kernel given by the ${\alpha }_{i,j}$-weighted mean of the individual coupling kernels:

$$k_f^j\left({\overrightarrow{x}}_{-t}^1,{\overrightarrow{x}}_{-t}^{1\prime },\dots ,{\overrightarrow{x}}_{-t}^M,{\overrightarrow{x}}_{-t}^{M\prime }\right)={{\alpha }^j}^2\sum _{i=1}^M\frac{k_f^{i,j}({\overrightarrow{x}}_{-t}^i,{\overrightarrow{x}}_{-t}^{i\prime },{\overrightarrow{x}}_{-t}^j,{\overrightarrow{x}}_{-t}^{j\prime })}{{{\alpha }^{i,j}}^2}$$

(2)

However, doing so results in a model which lacks modularity: after learning, it is difficult to separate the parts from each other, and recombine them for the production of new movements that were not in the training data. We facilitate this modular recombination by restating CGPDM learning such that we can keep an explicit, sparse representation of the coupling functions. Another reason for a sparse representation is that the CGPDM exhibits cubic run time scaling with the data points, which it inherits from the composing GPDMs. To remedy these problems, we follow the treatment in [16,30]: we augment the model with IPs ${\overrightarrow{r}}_{:}^i$ and associated IVs ${\overrightarrow{v}}_{:}^i$ such that $g^i({\overrightarrow{r}}_k^i)={\overrightarrow{v}}_k^i$ for the latent-to-observed mappings $g^i()$. Then, we condition the probability density of the function values of $g^i()$ on these IPs/IVs, which we assume to be a sufficient statistic. Likewise, we reduce the computational effort for learning the dynamics and coupling mappings by inducing them through ${\overrightarrow{z}}_{:}^{i,j}$ and ${\overrightarrow{u}}_{:}^{i,j}$ (also known as “dynamics IPs/IVs”). See Figure 2 for a graphical representation of the augmented model.

Besides introducing IPs, computing an ELBO requires a simplifying assumption about the latent state posterior, which is intractable. We choose a posterior distribution q over the latent states ${\overrightarrow{x}}_t^i$ that factorizes across time steps $0,\dots ,T$, parts $1,\dots ,M$ and latent dimensions $1,\dots ,Q^i$ within parts. Furthermore, we assume that the individual distributions are Gaussian:

$$q({\overrightarrow{x}}_0^1,\dots ,{\overrightarrow{x}}_T^M)=\prod _{t=0}^T\prod _{i=1}^M\prod _{q=1}^{Q^i}q({({\overrightarrow{x}}_t^i)}_q);q({({\overrightarrow{x}}_t^i)}_q)=\mathcal{N}({\mu }_{t,q}^i,{\sigma }_{t,q}^{2,i}).$$

(3)

While this approximation is clearly a gross simplification of the correct latent state posterior, with the right choice of kernels, an ELBO can now be computed. Our approximation assumption (Equation (3)) seems appropriate for human movement data, see Section 5. Whether it is also useful for other data remains to be determined.

As for a tractable kernel, we decided to use an ARD (automatic relevance determination) squared exponential kernel [36] for every part-i-to-j prediction GP:

$$k^{i,j}({\overrightarrow{x}}_{-t}^i,{\overrightarrow{x}}_{-t}^{i\prime })=exp\left(-\frac{1}{2}\sum _{t\in -t}\sum _q^{Q^i}\frac{{({({\overrightarrow{x}}_t^i)}_q-{({\overrightarrow{x}}_t^i{}^{\prime })}_q)}^2}{{\lambda }_q^{i,j,t}}\right).$$

(4)

and a radial basis function kernel for the latent-to-observed mappings. Next, we outline the key steps of the derivation of the ELBO.

4. Computing an Evidence Lower Bound for the vCGPDM: An Overview

In this section, we provide an overview of the derivation of the evidence lower bound (ELBO) for the vCGPDM; for details, the reader is referred to Appendix C. We construct a sparse variational approximation by augmenting each of the $M\times M$ dynamics and coupling mappings ${\overrightarrow{f}}^{i,j}()$ with IPs and IVs. The variational distribution of the latent variables, $q({\overrightarrow{x}}_{:}^{:})=q({\overrightarrow{x}}_1^1,\dots ,{\overrightarrow{x}}_T^M)$ factorizes according to Equation (3). We let $q({\overrightarrow{u}}_{:}^i)$ and $q({\overrightarrow{v}}_{:}^i)$ be unconstrained distributions, which will turn out to be multivariate Gaussians. In the following, we denote the coupling function values at t with ${\overrightarrow{f}}_t^{i,j}=f^{i,j}({\overrightarrow{x}}_{-t}^i)$ and likewise ${\overrightarrow{g}}_t^i=g^i({\overrightarrow{x}}_t^i)$. The factor structure of the joint density of the augmented model follows from the graphical model (see Figure 1 and Figure 2):

$$p({\overrightarrow{y}}_{:}^{:},{\overrightarrow{g}}_{:}^{:},{\overrightarrow{v}}_{:}^{:},{\overrightarrow{x}}_{:}^{:},{\overrightarrow{f}}_{:}^{:,:},{\overrightarrow{u}}_{:}^{:,:}|{\overrightarrow{z}}_{:}^{:,:},{\overrightarrow{r}}_{:}^{:})=p({\overrightarrow{y}}_{:}^{:}|{\overrightarrow{g}}_{:}^{:})p({\overrightarrow{g}}_{:}^{:}|{\overrightarrow{x}}_{:}^{:},{\overrightarrow{v}}_{:},{\overrightarrow{r}}_{:}^{:})p({\overrightarrow{v}}_{:}|{\overrightarrow{r}}_{:}^{:})p({\overrightarrow{x}}_{:}^{:},{\overrightarrow{f}}_{:}^{:,:}|{\overrightarrow{u}}_{:}^{:,:},{\overrightarrow{z}}_{:}^{:,:})p({\overrightarrow{u}}_{:}^{:,:}|{\overrightarrow{z}}_{:}^{:,:})$$

(5)

Note that we marginalized (most of) the functions $f^{:,:}()$ here, keeping only their values at the latent points ${\overrightarrow{x}}_{:}^{:}$ and at the IPs. Hence the dependence of ${\overrightarrow{g}}_{:}^{:}$ on ${\overrightarrow{r}}_{:}^{:}$. Likewise, ${\overrightarrow{f}}_{:}^{:,:}$ depends on ${\overrightarrow{z}}_{:}^{:,:}$. For easier notation, we omit spelling out the dependence of the IVs on the IPs in the following. Thus,

$$\begin{array}{cc}\hfill p({\overrightarrow{y}}_{:}^{:}|{\overrightarrow{g}}_{:}^{:})& =\prod _{i=1}^M\prod _{d=1}^{D_i}p({({\overrightarrow{y}}_{:}^i)}_d|{({\overrightarrow{g}}_{:}^i)}_d)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill p({\overrightarrow{g}}_{:}^{:}|{\overrightarrow{x}}_{:}^{:},{\overrightarrow{v}}_{:}^{:})& =\prod _{i=1}^M\prod _{d=1}^{D_i}p({({\overrightarrow{g}}_{:}^i)}_d|{\overrightarrow{x}}_{:}^i,{({\overrightarrow{v}}_{:}^i)}_d)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill p({\overrightarrow{v}}_{:}^{:})=\prod _{i=1}^{M}p({\overrightarrow{v}}_{:}^i)& ;p({\overrightarrow{u}}_{:}^{:,:})=\prod _{i=1}^M\prod _{j=1}^{M}p({\overrightarrow{u}}_{:}^{j,i}).\hfill \end{array}$$

(8)

where Equation (6) follows from the assumption of independent observation noise. Equation (7) is a consequence of the Gaussian process prior on the $g^i()$, which makes the components of ${\overrightarrow{g}}_t^i$ independent. The density of the latent variables and the individual parts’ predictions can be factorized as:

$$p({\overrightarrow{x}}_{:}^{:},{\overrightarrow{f}}_{:}^{:,:}|{\overrightarrow{u}}_{:}^{:,:})=p({\overrightarrow{x}}_{:}^{:}|{\overrightarrow{f}}^{:,:},{\overrightarrow{u}}_{:}^{:,:})p({\overrightarrow{f}}_{:}^{:,:}|{\overrightarrow{u}}_{:}^{:,:})$$

(9)

with

$$\begin{array}{cc}\hfill p({\overrightarrow{x}}_{:}^{:}|{\overrightarrow{f}}_{:}^{:,:},{\overrightarrow{u}}_{:}^{:,:})& =\prod _{t=2}^T\prod _{i=1}^{M}p({\overrightarrow{x}}_t^i|{\overrightarrow{f}}_t^{:,i},{\alpha }^{:,i})\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill p({\overrightarrow{f}}_{:}^{:,:}|{\overrightarrow{u}}_{:}^{:,:})& =\prod _{t=2}^T\prod _{i=1}^M\prod _{j=1}^{M}p({\overrightarrow{f}}_t^{j,i}|{\overrightarrow{f}}_{1:t-1}^{j,i},{\overrightarrow{x}}_{0:t-1}^j,{\overrightarrow{u}}_{:}^{j,i})\hfill \end{array}$$

(11)

where Equation (10) follows from the graphical model and the product-of-experts construction (Equation (1)). An empty slice ($t<2$ for a second-order dynamics model) implies no conditioning. The first two latent states at $t=0,1$ are drawn from independent Gaussians, ${\prod }_{i=1}^{M}p({\overrightarrow{x}}_0^i)p({\overrightarrow{x}}_1^i)$. Equation (11) is one possible way of factorizing the augmented Gaussian process prior on the coupling function values: when ${\overrightarrow{f}}^{i,j}()$ is marginalized, the function values at time t depend on all past function values and latent states. We use this particular factorization for analytical convenience. Note that the dependence of the right hand side of Equation (11) on ${\overrightarrow{x}}_{0:t-1}^j$ does not contradict the factorization order of Equation (9), because it depends only on latent variables from timesteps prior to t. Furthermore, we choose the following proposal variational posterior:

$$q({\overrightarrow{g}}_{:}^{:},{\overrightarrow{x}}_{:}^{:},{\overrightarrow{v}}_{:}^{:},,{\overrightarrow{f}}_{:}^{:,:},{\overrightarrow{u}}_{:}^{:,:})=p({\overrightarrow{g}}_{:}^{:}|{\overrightarrow{x}}_{:}^{:},{\overrightarrow{v}}_{:}^{:})q({\overrightarrow{v}}_{:}^{:})p({\overrightarrow{f}}_{:}^{:,:}|{\overrightarrow{u}}_{:}^{:,:})q({\overrightarrow{x}}_{:}^{:})q({\overrightarrow{u}}_{:}^{:})$$

(12)

with $p({\overrightarrow{g}}_{:}^{:}|{\overrightarrow{x}}_{:}^{:},{\overrightarrow{v}}_{:}^{:})$ given by Equation (7), $p({\overrightarrow{f}}_{:}^{:,:}|{\overrightarrow{u}}_{:}^{:,:})$ by Equation (11) and $q({\overrightarrow{x}}_{:}^{:})$ by Equation (3). The densities $q(\overrightarrow{v})$ and $q(\overrightarrow{u})$ are unconstrained except for normalization. With these distributions, we derive the standard free-energy ELBO [36], denoting $\Theta =({\overrightarrow{x}}_{:}^{:},{\overrightarrow{u}}_{:}^{:,:},{\overrightarrow{f}}_{:}^{:,:},{\overrightarrow{v}}_{:}^{:},{\overrightarrow{g}}_{:}^{:})$:

$$logp({\overrightarrow{y}}_{:}^{:})\ge \mathcal{L}(\Theta )=\int d\Theta q(\Theta )log\left(\frac{p({\overrightarrow{y}}_{:}^{:},\Theta )}{q(\Theta )}\right)$$

(13)

exploiting the assumption that the IPs ${\overrightarrow{r}}_{:}^i$ and IVs ${\overrightarrow{v}}_{:}^i$ are sufficient statistics for the function values ${\overrightarrow{g}}_{:}^i$. As we explain in detail in Appendix C, after canceling common factors in the variational posterior (Equation (12)) and the joint model density (Equation (5), cf. [16]), we find that the ELBO can be decomposed into one summand per part that describes the quality of the kinematics mapping (latent-to-observed) ${\mathcal{L}}_{kin}^i$, and one summand for the dynamics ${\mathcal{L}}_{dyn}$:

$$\mathcal{L}(\Theta )=\sum _{i=1}^M{\mathcal{L}}_{kin}^i+{\mathcal{L}}_{dyn}$$

(14)

where

$${\mathcal{L}}_{kin}^i=\sum _{d=1}^D\int d{\overrightarrow{x}}_{:}^{i}d{({\overrightarrow{v}}_{:}^i)}_{d}d{({\overrightarrow{g}}_{:}^i)}_{d}p({({\overrightarrow{g}}_{:}^i)}_d|{\overrightarrow{x}}_{:}^i,{({\overrightarrow{v}}_{:}^i)}_d)q({\overrightarrow{x}}_{:}^i)q({({\overrightarrow{v}}_{:}^i)}_d)log\frac{p({({\overrightarrow{y}}_{:}^i)}_d|{({\overrightarrow{g}}_{:}^i)}_d)}{q({({\overrightarrow{v}}_{:}^i)}_d)}.$$

(15)

is—up to the Shannon entropy of approximating posterior of the latent dynamics variables $H(q({\overrightarrow{x}}_{:}^i))=-\int d{\overrightarrow{x}}_{:}^{i}q({\overrightarrow{x}}_{:}^i)log\left(q({\overrightarrow{x}}_{:}^i)\right)$—equal to the Bayesian GPLVM ELBO of [16]. The remaining integral

$$\begin{array}{cc}\hfill {\mathcal{L}}_{dyn}& =\int d{\overrightarrow{u}}_{:}^{:,:}q({\overrightarrow{u}}_{:}^{:,:})\left[\sum _{t=2}^T\int d{\overrightarrow{x}}_{1:t}^{:}q({\overrightarrow{x}}_{1:t}^{:})\left(\int d{\overrightarrow{f}}_t^{:,:}p({\overrightarrow{f}}_t^{:,:}|{\overrightarrow{f}}_{2:t-1}^{:,:},{\overrightarrow{x}}_{0:t-1}^{:},{\overrightarrow{u}}_{:}^{:,:})logp({\overrightarrow{x}}_t^{:}|{\overrightarrow{f}}_t^{:,:},{\alpha }^{:,:})\right)\right]\hfill \\ & +\int d{\overrightarrow{u}}_{:}^{:,:}q({\overrightarrow{u}}_{:}^{:,:})log\frac{p({\overrightarrow{u}}_{:}^{:,:})}{q({\overrightarrow{u}}_{:}^{:,:})}+\int d{\overrightarrow{x}}_{0:1}^{:}q({\overrightarrow{x}}_{0:1}^{:})logp({\overrightarrow{x}}_{0:1}^{:})+H(q({\overrightarrow{x}}_{:}^{:}))\hfill \end{array}$$

(16)

is derived in detail in Appendix C. Briefly, we use the assumption that the IPs and IVs ${\overrightarrow{z}}_{:}^{j,i}$ and ${\overrightarrow{u}}_{:}^{j,i}$ are sufficient statistics for the function values ${\overrightarrow{f}}_t^{j,i}$. Optimizing with respect to $q({\overrightarrow{u}}^{:,:}:$) can be carried out in closed form using variational calculus and yields

$${\mathcal{L}}_{dyn}(\Theta )\ge log\int p({\overrightarrow{u}}_{:}^{:,:})exp(\mathcal{C}({\overrightarrow{u}}_{:}^{:,:}))d{\overrightarrow{u}}_{:}^{:,:}+H(q({\overrightarrow{x}}_{:}^{:}))$$

(17)

where $\mathcal{C}({\overrightarrow{u}}_{:}^{:,:})$ is given by Equation (A31). The inequality is due to the sufficient statistics assumption, which introduces another approximation step that lower-bounds ${\mathcal{L}}_{dyn}$. We now have all the ingredients to compute the ELBO for the whole model, and learn it.

5. Results

We used the machine-learning framework Theano [37] for automatic differentiation in Python 2.7 (Python Software Foundation. Python Language Reference, version 2.7. Available at http://www.python.org) to implement the model, and learned via optimization of the ELBO with the scipy.optimize.fmin_l_bfgs_b routine [38]. Latent space trajectories were initialized with PCA. We obtained the best ELBOs by first optimizing all parameters jointly, followed by a blocked optimization procedure. We optimize three groups of parameters: latent points and variances, kernel parameters and couplings, and IPs. The number of iterations of the blocked procedure depended on the application; we provide details in the sections below.

The advantage of the sparse approximations in the vCGPDM is that memory consumption of the model is greatly reduced. However, this approximation might also introduce errors, along with the fully factorized latent posterior (Equation (3)). We tried to quantify these errors in a cross-validatory model comparison, and in a human perception experiment.

5.1. Synthetic Data

We demonstrate the learning of coupled dynamical systems on a synthetic dataset. First, we draw two dynamics transition functions $g^1,g^2\in {\mathbb{R}}^2\to {\mathbb{R}}^2$ from a $\mathcal{GP}$ with an RBF kernel, and then we generate latent trajectories according to:

$$\begin{array}{cc}\hfill {\overrightarrow{x}}_t^1& =g^1({\overrightarrow{x}}_{t-1}^1)\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill {\overrightarrow{x}}_t^2& =0.1g^1({\overrightarrow{x}}_{t-1}^1)+0.9g^2({\overrightarrow{x}}_{t-1}^2)\hfill \end{array}$$

(19)

at $T=300$ timepoints. Thus, we get two first-order, coupled latent dynamical systems, each of dimensionality 2. The trajectory in Latent Space 1 is independent of Latent Space 2, whereas Latent Space 2 is weakly coupled to Latent Space 1. Then, for each of the two parts, we draw 10 observed trajectories from another two RBF GPs with inputs on the latent trajectories. The latent trajectories are shown in Figure 3A,C. Figure 3B,D displays the corresponding observed trajectories. We learned a second-order vCGPDM from these data, iterating the blocked optimization until convergence of the ELBO to machine precision. We chose a second order system for this learning example, because the human movement models in the following are second-order vCGPDMs, too.

The results are shown in Figure 3E–H. Plots on the left half were generated with four IPs, plots in the right half with ten IPs. Figure 3E displays the initial positions of the dynamics IPs (blue circles) at the beginning of learning, connected circles form one second-order IP. Green crosses are the kinematics IPs (latent-to-observed mapping). Initial latent points (dashed blue lines) were obtained from the first two PCA components of the training data. Blue and red line segments are examples of the dynamics mapping: the end-points of the blue segments are the inputs, the distal endpoint of the red segment is the output. As one might expect, the initial conditions do not describe a dynamics which can produce trajectories resembling those in the training data: the black line is the mean latent trajectory, and Figure 3G shows the corresponding observable trajectories.

After learning, the latent trajectories appear to have a limit cycle (black line, Figure 3F), which is required to reproduce the training data. Furthermore, note that the inducing points align with that cycle, and the example mappings (blue and red line segments) indicate clearly how the latent trajectory is formed by iterating the GP mapping. Cross coupling GP mappings omitted for clarity. Note that the vCGPDM with ten IPs can model more complex latent dynamics manifolds than the four-IP vCGPDM. The observable trajectories (Figure 3H) look very similar to the training data up to a small phase shift, particularly for the 10 IP model. This observation is confirmed by the reduced mean squared trajectory error (MSE) between generated and training data after learning, which was evaluated after dynamic time warping [39] of the generated trajectories onto the training data. The MSEs are listed in the table at the bottom of Figure 3, where “final” indicates the values after learning, while “initial” indicates the values at the onset of learning after the latent space trajectories had been initialized to the first two PCA components of the training data. That learning was successful is also indicated by the increased final ELBO, which is higher for the 10 IP model.

We also provided the learned coupling $\alpha$s in this table. Recall that a low(high) $\alpha$ means a large (small) influence of the corresponding part on the dynamics. The dependency structure between the latent spaces was correctly identified during learning: $a^{2,1}\gg {\alpha }^{1,1}$, i.e., Part 2 has almost no influence on Part 1. In contrast, ${\alpha }^{1,2}\approx 2{\alpha }^{2,2}$, which indicates that Part 1 weakly controls Part 2.

5.2. Human Movement Data

Model comparisons and psychophysical tests were carried out on human movement data. We employed a 10-camera PhaseSpace Impulse motion capture system, mapped the resulting position data onto a skeleton with 19 joints and computed joint angles in exponential-map representation, yielding a total of 60 degrees of freedom. Five walking-only and four walking + waving sequences each were used to train the models, as well as ten movements where the human participants were seated and passed a bottle from one hand to the other. Dynamical models were initialized with starting conditions taken from the training data. The blocked optimization was run for at most four iterations, which was enough to ensure convergence. It was terminated earlier if ELBO values did not change within machine precision between two subsequent iterations. Furthermore, we recorded another nine walking sequences for catch trials during the perception experiment, to rule out memorization effects. Generated and recorded sequences were rendered on a neutral avatar. Examples of stimuli, for different numbers of IPs, can be found in the movie example_stimuli.mov in the Supplementary Materials.

5.3. Variational Approximations are Better than MAP

We performed cross-validatory model comparison on the following datasets: walking, walking + waving and passing-a-bottle. Examples of these data are shown in the movies in the Supplementary Materials: S1_example_stimuli.mov and S4_pass_the_bottle.mkv. We performed four-, five- and ten-fold crossvalidation, the number of folds was dictated by the dataset size. We were trying determine how the sparsely parameterized vCGPDM performs in comparison to the full CGPDM, and several other MP models from the literature. Held-out data were always one complete trial. Models were trained on the remaining data and the generated trajectory was compared to the held-out one. Cross-validation score was the mean-squared error (MSE) of the kinematics after dynamic time warping [39] of trajectories generated by initializing the model to the first two frames of a held-out trial onto the complete held-out trial. We used dynamic time warping to compensate a slight phase difference in generated motions, which would otherwise lead to a much larger and uninformative MSE. We compared the following models:

• a GPDM with maximum-a-posteriori (MAP) estimation of the latent variables [25], called MAP GPDM in Figure 4;
• a fully marginalized two-part (upper/lower body) CGPDM with MAP estimation of the latent variables [14], called MAP CGPDM U+L;
• a three-part CGPDM model (left hand, right hand, and body) for the non-periodic “passing a bottle” dataset;
• their variational counterparts, vCGPDM 3-part, vCGPDM U+L and vGPDM;
• temporal MPs (TMP, instantaneous linear mixtures of functions of time) [9]; and
• DMPs [12].

All latent spaces were three-dimensional. We tried 4–30 latent-to-observed IPs and 2–30 dynamics IPs. The MSE optima were near 10–15 IPs for both the walking and the walking + waving datasets, and near eight IPs for the “passing a bottle”. MAP GPDM and MAP CGPDM learning do not use any approximations or inducing points; they are the full GPs with covariance matrices $\mathbf{K}\in {\mathbb{R}}^{T\ast T}$.

For the TMPs, we used up to 10 primitives; the MSE optimum was located at approximately six. For the DMPs, we used between 1 and 50 basis functions, and the lowest MSE was found around 15.

The results are plotted in Figure 4. Generally, the walking + waving movement is more difficult to reproduce for all models than walking only: the MSE of the latter is lower than that of the former, and the ELBO is higher. This indicates that the latter is a more complex movement, see also the movie modular_primitives.avi in the online Supplementary Materials. The two-part vCGPDM reaches the lowest MSE compared to all other models. Clearly, it is better than the full-capacity (no IPs) MAP models, which means that the extra effort of developing of a variational approximation which explicitly represents an approximation to the latent states’ posterior and needs to store only ≈10 IPs rather than ≈10${}^4$ data points was well spent. In addition, the best ELBO’s MSE (that is, the MSE at the maximum of the ELBO) is a fairly good predictor of the best MSE, which justifies our simple variational approximation for model selection.

The vCGPDM U+L outperforms the vGPDM particularly on the “walking + waving” dataset. This shows the usefulness of having modular, coupled dynamics models when the (inter)acting (body)parts execute partially independent movements.

The vCGPDM with three parts for “passing a bottle” does not show a clear advantage over the monolithic model in the cross-validation test, and is on par with the TMP model. However, dynamics factorization did not affect the performance either. This may be indicative for the strong coupling necessary to successfully pass an object from one hand to the other. Such a strong coupling is parsimoniously expressed by having a single latent dynamical system drive all observable degrees of freedom.

The timing results show that the training times of the vCGPDM are usually less than 15 min. Error bars are standard deviations, estimated across all numbers of IPs and cross-validation splits. The rather large training time for the TMP model is due to the implementation from [9] which optimizes a rather large covariance matrix between all MPs.

5.4. A Small Number of IPs Yields Perceptually Convincing Movements

We conducted a psychophysical experiment to quantify the perceptual validity of the generated movements. More specifically, we investigated the model complexity required for perceptually convincing movements.

Experiment: Thirty-one human observers (10 male, mean age: $23.8\pm 3.5$a) participated in a two-alternative forced-choice task to distinguish between natural and generated movements (see Figure 5 for an example of an experimental trial). Natural movements consisted of 15 walking movements. The artificial movements were generated by a two-part (upper/lower body) vCGPDM. We used 2–16 dynamics IPs and 4–16 latent-to-observed IPs. We chose these numbers based on pilot tests to span the range from clearly unnatural to very natural looking movements. To test whether participants simply memorized the 15 natural stimuli during the experiment, we added 10 catch trials in the last quarter of the experiment where previously unused natural movements were tested against the known natural stimuli. The trial sequence was randomized for every subject. All experimental procedures were approved by the local ethics commission.

Results: We computed the confusion rate, i.e., the frequency of choosing the model-generated movement as more natural across all participants as a function of the number of IPs for the dynamics and latent-to-observed mappings. Optimally, we might expect this rate to approach $\frac{1}{2}$ when the generated movements are indistinguishable from the natural ones. We investigated if the confusion rate approached this limit, how it depends on the mean-squared residual error on the training data, and how this error is connected to the ELBO. The results are plotted in Figure 6. We also fitted the confusion rate data with a logistic sigmoid $\frac{0.5}{1+exp(a·MSE+c)}$ (solid line in Figure 6A), and the MSE with an exponential function (solid line in Figure 6, right). Each data point represents one combination of dynamics/latent-to-observed IP numbers, indicated by width and height of the ellipses. Clearly, confusion rate increases fairly monotonically with decreasing MSE, as indicated by the good logistic sigmoid fit. Furthermore, models with more IPs also tend to yield higher confusion rates. A sufficient number (>10) dynamics IPs is more important than a large number of latent-to-observed IPs, which can be seen by the very narrow ellipses in the region with high MSE, and many wider ellipses in the lower MSE part of the figure. A similar observation can be made about the relationship between ELBO and MSE (Figure 6B). It indicates that ELBO is already a good predictor for the model performance. For a very small number of dynamics IPs, increasing the number of latent-to-observed IPs does not decrease the MSE as much as increasing the dynamics IPs does. Moreover, note that the relationship between MSE and ELBO becomes fairly monotonic when ELBO > 28,500, which is where human perceptual performance can be predicted from ELBO. While the confusion rate has not quite reached its theoretical maximum in our experiment, these results are evidence that human perceptual expectations can be nearly met with very compactly parameterized MP models (Figure 6C,D). Moreover, good dynamics models seem to outweigh precise kinematics. We found no evidence for stimulus memorization from the confusion rates of the catch trials.

5.5. Modularity Test

Next, we examined if the intended modularization of our model can be used to compose novel movements from previously learned parts. We trained a vCGPDM consisting of one part for the lower body (below and including pelvis), and a second part for the upper body. Twenty-five IPs for the latent-to-observed mapping of each part were shared across all movements. The walking MP, parameterized by 16 IPs for the lower-body dynamics and the lower-to-upper mappings, was also shared. We used a different set of 16 IPs for the upper body MPs between arm-swing and waving. Furthermore, the coupling ${\alpha }^{j,i}$ were learned anew for each combination of upper/lower MPs. The resulting latent space trajectories are plotted in Figure 7. All generated trajectories (solid lines) are on average close to the training data (dashed lines). While the walking trajectories for the lower body are very similar for the two movements, the upper body trajectories clearly differ. Movements generated from this model are very natural (see video S3_modular_primitives.mov in the Supplementary Materials). This is a first demonstration that the vCGPDM with non-marginalized couplings can be used to learn a library of compactly parameterized MPs, from which novel movements can be produced with little additional memory requirements (i.e., new coupling ${\alpha }^{j,i}$ only).

For a quantitative evaluation, we looked at the leave-one-out cross-validation MSEs of the vCGPDM trained on datasets separately and modular vCGPDM trained on both datasets (see Figure 7, bottom). Within the standard errors, MSEs are equal, indicating that modular re-use of previously trained components does not necessarily sacrifice accuracy, while reducing storage requirements. Training time for the modular vCGPDM is larger due to the learning of the combined dataset and optimizing the couplings afterwards. This time would be amortized if more compositional movements were learned, where previously learned parts could be reused.

6. Conclusions

The vCGPDM, a full variational approximation to the CGPDM, allows for learning a deterministic approximation of latent space trajectories, and compactly parameterizing dynamics and kinematics mappings. First, we showed that the sparsely parameterized vCGPDM outperforms the full-capacity, monolithic CGPDM employing MAP to approximate the latent dynamics posterior. It also surpasses other current MP models; we speculate that this is accomplished by its learnable dynamics.

Second, we demonstrated that our compact representation of the latent space dynamics, and of the latent-to-observed mapping, enables the model to generate perceptually convincing full-body movements with a fairly small number of IPs To our knowledge, a systematic investigation of the number of IPs needed for perceptual plausibility had not been done before, albeit more monolithic models were in the focus of earlier studies [27,34,40]. Moreover, we demonstrated that a high enough ELBO can be used to predict average human classification performance, which might allow for an automatic model selection process when training the model on large databases. Within the range of IPs which we tested, the ELBO was still increasing with their number. We chose that range because we wanted to see how few IPs would still lead to perceptually indistinguishable movements. Due to experimental time constraints, we did not investigate perceptual performance at the point where the ELBO begins to decrease with increasing IPs (i.e., the approximately optimal model), but we plan to do that in the future.

Third, we showed that the model can be employed in a modular fashion, using one lower-body dynamics model, and coupling it to two different models for the upper body. Note that the lower-to-upper coupling function was the same for the two upper-body models. Each of these models, including the coupling functions to the other model parts, may therefore be viewed as a modular MP that is parameterized compactly by a small number of IPs and values. This sparse parameterization allows us to infer modular MPs from a large collection of movements, and investigate their composition. To generate complex movement sequences, we will put a switching prior on top of the dynamical models, as in [29].

We are currently researching sensorimotor primitives, i.e., MPs that can be used to predict sensory input and be controlled by it via conditioning. This conditioning can take place on at least two timescales: a short one (while the MP is running), thus effectively turning the MPs into flexible control policies, such the probabilistic MPs described by Paraschos et al. [41], and a long timescale, i.e., the planning of the movement. This could be implemented by learning a mapping from goals and affordances onto the coupling weights, comparable to the DMPs with associative skill memories [42]. There is evidence that humans modulate the coupling between their MPs during the planning stage: whole-body posture changes have been observed in anticipation of reaching for a goal object in a known location, even if the object is currently invisible [43].

Lastly, we note that our CGPDM could be used as a flexible policy model for PILCO-style reinforcement learning (Probabilistic Inference for Learning Control, [44]). PILCO requires a dynamics model that can propagate uncertainties through time; the vCGPDM is able to do that. Thus, our model could be used as a lower dimensional dynamics model which can capture the dependencies between observable variables via latent space uncertainties.