An Iteratively Reweighted Importance Kernel Bayesian Filtering Approach for High-Dimensional Data Processing

Abstract

This paper proposes an iteratively re-weighted importance kernel Bayes filter (IRe-KBF) method for handling high-dimensional or complex data in Bayesian filtering problems. This innovative approach incorporates importance weights and an iterative re-weighting scheme inspired by iteratively re-weighted Least Squares (IRLS) to enhance the robustness and accuracy of Bayesian inference. The proposed method does not require explicit specification of prior and likelihood distributions; instead, it learns the kernel mean representations from training data. Experimental results demonstrate the superior performance of this method over traditional KBF methods on high-dimensional datasets.

1. Introduction

Bayesian filtering is a probabilistic approach that recursively estimates an unknown probability density function over time based on a mathematical model and observation process [1,2,3]. It involves updating the prior distribution to obtain the posterior distribution, which is the essence of Bayesian statistics. As a fundamental problem in probabilistic inference and sequential estimation, Bayesian filtering focuses on inferring the state of a dynamic system over time from noisy and incomplete observations while integrating prior knowledge about the system’s behavior. This problem finds wide-ranging applications in fields such as signal processing [4], robotics [5], finance [6,7], and more.

Traditional Bayesian filtering methods often encounter limitations when dealing with high-dimensional or complex data spaces. Kernel methods have emerged as a powerful tool for generalizing linear statistical methods to nonlinear settings, which is achieved by embedding samples into a high-dimensional feature space known as a reproducing kernel Hilbert space (RKHS). Kernel mean embedding plays a crucial role in this context, representing probability distributions as expectations of features in the RKHS [8,9,10].

By embedding distributions into the RKHS, kernel mean embedding facilitates various statistical analyses and machine learning tasks. The work of Smola et al. in 2007 [9] emphasized the significance of kernel mean embedding in Bayesian updates. The use of kernel means in characteristic RKHS has been widely proven successful in a number of statistical tasks, including two-sample problems [11], independence tests [12], and conditional independence tests [13]. Notably, these tests are applicable to any domain where kernels can be defined, showcasing the versatility of the kernel approach.

In the context of Bayesian filtering, it enables the representation of probability distributions in a space where operations such as inner products and norms can be easily computed. This not only enhances the scalability of Bayesian filtering algorithms but also allows for the incorporation of complex, structured data into the estimation process. Moreover, kernel mean embedding provides a framework for integrating diverse sources of information, including prior knowledge and domain expertise, into the filtering process.

The kernel Bayesian filtering procedure introduced in [14] is model-free and produces sample-derived estimates of the posterior embedding. This Bayesian filtering method involves the construction of a posterior distribution over the current state based on the sequence of noisy observations up to the present time, without making explicit modeling assumptions about the underlying dynamics. Unfortunately, it may cause instability during the calculating process. Fukumizu et al. [14] proposed a formulation that requires an unconventional form of regularization in the second stage, which adversely impacts the attainable convergence rates. Boots et al. [15] introduced one KBF where only a simple form of Tikhonov regularization is applied. Unfortunately, this method requires the inversion of the matrix, which is often indefinite, necessitating high regularization constants, which can degrade performance. Xu et al. [16] introduced an approach called importance-weighted KBF to avoid the instability problem. The importance weight is based on the density ratio. However, the calculation of the density ratio may cause high variance.

In this paper, we explore an innovative extension of the importance weighted kernel Bayesian filtering method proposed in [16] known as Iteratively Re-weighted importance Kernel Bayesian Filtering (IRe-KBF). The concept of iterated re-weighting, which has been successfully applied in other contexts (e.g., [17]), is now integrated into the filtering process, as explained in Section 3. This integration, which is also discussed in the Introduction, along with reference [17], brings about a significant enhancement by making the filter more robust and efficient in high-dimensional systems.

The IRe-KBF method learns kernel mean representations directly from training data, eliminating the need for explicit specification of prior and likelihood distributions. By harnessing importance weights and iterative re-weighting, our filter exhibits enhanced robustness and efficiency in the context of high-dimensional datasets.

The remainder of this paper is organized as follows. In Section 2, we review the basic concepts of kernel mean embeddings and the kernel Bayes rule. In Section 3, we first introduce the importance-weighted kernel Bayes rule proposed in [16]; then the iteratively re-weighted importance kernel Bayes rule (IRe-KBR) is proposed. We introduce the filtering problem in Section 4, which applies the IRe-KBR to the filtering problem. Experiments are reported in Section 5.

2. Background and Preliminaries

In this section, we review some basic concepts.

Kernel mean embeddings provide a way to represent distributions in reproducing kernel Hilbert spaces (RKHSs) using positive definite kernels. Given random variables $(X,Y)$ on $\mathcal{X}\times \mathcal{Y}$ with a joint distribution (P) and density functions $p(x,y)$, $k_x:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ and $k_y:\mathcal{Y}\times \mathcal{Y}\to \mathbb{R}$ are measurable positive definite kernel corresponding to scalar-valued RKHSs ${\mathcal{H}}_x$ and ${\mathcal{H}}_y$, respectively. The feature maps are denoted as $\varphi \left(x\right)=k_x(x,.)$ and $\psi \left(y\right)=k_y(y,.)$, and the RKHS norm is denoted as $\parallel ·\parallel$, with corresponding Gram matrices of ${\left(G_X\right)}_{ij}=k_x(x_i,x_j)$ and ${\left(G_Y\right)}_{ij}=k_y(y_i,y_j)$.

The kernel mean embedding of the marginal distribution ($P\left(x\right)$) is denoted by $m_{P\left(x\right)}$ and defined as follows:

$$\begin{array}{c}\hfill m_{P\left(x\right)}=E_P\left[\varphi \left(x\right)\right]\in {\mathcal{H}}_x.\end{array}$$

It always exists for kernels that are bounded. From the reproducing property, we derive the following for all $f\in {\mathcal{H}}_x,$:

$$\begin{array}{c}\hfill <f,m_{P\left(x\right)}>=E_P\left[f\left(x\right)\right]\end{array}$$

which is advantageous for estimating expectations of functions. Moreover, if a kernel $k_x$ such as the Gaussian kernel is characteristic, the embedding uniquely determines the probability distribution, which means that $m_{P\left(x\right)}=m_{Q\left(x\right)}$ implies $P=Q$. Additionally, we introduce the following (uncentered) kernel covariance operators:

$$\begin{array}{cc}\hfill C_{XX}^P& =E_P[\varphi \left(X\right)\otimes \varphi \left(X\right)],C_{XY}^P=E_P[\varphi \left(X\right)\otimes \psi \left(Y\right)]\hfill \\ \hfill C_{YY}^P& =E_P[\psi \left(Y\right)\otimes \psi \left(Y\right)],C_{YX}^P={\left(C_{XY}^P\right)}^{\ast }.\hfill \end{array}$$

In this context, ⊗ denotes the tensor product, and ∗ denotes the adjoint of the operator. Covariance operators extend the concept of finite-dimensional covariance matrices to the realm of infinite kernel feature spaces, and they are always defined for bounded kernels.

The kernel conditional mean embedding [10] extends the kernel mean embedding to conditional probability distributions. It is defined as follows:

$$\begin{array}{c}\hfill m_{P\left(X\right|Y)}\left(y\right)=E_P\left[\varphi \left(X\right)\right|Y=y]\in {\mathcal{H}}_x.\end{array}$$

This embedding captures the conditional expectation of the feature space representation of X given $Y=y$. Under the assumption of regularity, which requires that $E_P\left[f\left(X\right)\right|Y=y]\in {\mathcal{H}}_x$ for all $f\in {\mathcal{H}}_x$, an operator ($E_P\in {\mathcal{H}}_x\otimes {\mathcal{H}}_y$) can be identified such that

$$\begin{array}{c}\hfill m_{P\left(X\right|Y)}\left(y\right)=E_P^{*}\psi \left(y\right).\end{array}$$

The conditional operator ($E_P$) can be obtained by minimizing the loss function (${\mathcal{L}}^{\left(P\right)}$), which is

$$\begin{array}{c}\hfill {\mathcal{L}}^P\left(E\right)=E_P[\parallel \varphi \left(X\right)-E^{*}\psi \left(Y\right){\parallel }^2].\end{array}$$

The minimization of this loss function leads to an analytical solution in the form of the following closed-form expression:

$$\begin{array}{c}\hfill E_P={\left(C_{YY}^P\right)}^{-1}{C}_{XY}^P.\end{array}$$

The solution provides a means to estimate the conditional operator ($E_P$), which is crucial for various machine learning tasks that involve conditional inference or prediction. Obtaining an empirical estimate of the conditional mean embedding is a straightforward procedure. Given i.i.d samples (${\left\{(x_i,y_i)\right\}}_{i=1}^n\sim P$), we seek to minimize the sample estimate of the loss function (${\widehat{\mathcal{L}}}^{(P,\lambda )}$), which is defined as

$$\begin{array}{c}\hfill {\widehat{\mathcal{L}}}^{(P,\lambda )}={\displaystyle \frac{1}{n}}\sum _{i=1}^n\parallel \varphi \left(x_i\right)-E^{*}\psi \left(y_i\right){\parallel }^2+\lambda {\parallel E\parallel }^2.\end{array}$$

This is a sample estimate of ${\mathcal{L}}^P$ that incorporates a Tikhonov regularization parameter ($\lambda$) to mitigate overfitting. The solution to the minimization problem can be expressed as follows:

$$\begin{array}{c}\hfill {\widehat{E}}_{(P,\lambda )}={({\widehat{C}}_{YY}^P+\lambda I)}^{-1}{\widehat{C}}_{XY}^P,\end{array}$$

where $\widehat{C}$ is the empirical estimate of the covariance operators, i.e.,

$$\begin{array}{c}\hfill {\widehat{C}}_{YY}^P={\displaystyle \frac{1}{n}}\sum _{i=1}^n\psi \left(y_i\right)\otimes \psi \left(y_i\right){\widehat{C}}_{XY}^P={\displaystyle \frac{1}{n}}\sum _{i=1}^n\varphi \left(x_i\right)\otimes \psi \left(y_i\right).\end{array}$$

With respect to the kernel Bayes rule (KBR), in the context of Bayes’ theorem, our objective is to update the prior distribution ($\Pi$) with the density function ($\pi \left(x\right)$) to the posterior distribution (Q) of the state (X) given an observation (Y) with the corresponding density function ($q\left(x\right|y)$).

$$\begin{array}{c}\hfill q_{\left(X\right|Y)}\left(x\right|y)={\displaystyle \frac{p_{\left(Y\right|X)}\left(y\right|x)\pi \left(x\right)}{q_Y\left(y\right)}},q_Y\left(y\right)=\int p_{\left(Y\right|X)}\left(y\right|u)\pi \left(u\right)du\end{array}$$

where $p\left(y\right|x)$ is the density function of the likelihood function and $q_Y$ is the marginal probability density function.

The KBR aims to update the prior mean embedding ($m_{\Pi }$) to the posterior mean embedding ($m_Q$) according to the Bayes rule. Unlike traditional methods that require the explicit form of the likelihood function ($p_{Y|X}$), the KBR learns the relations between the latent variable (X) and the observable variable (Y) directly from the data. This is achieved by analyzing the dataset (${\left\{(x_i,y_i)\right\}}_{i=1}^n\sim P$), where the density of data distribution (P) shares the same likelihood ($p_{Y|X}$) with the density function ($p\left(x\right)$). In general, the density function ($\pi \left(x\right)$) is not equal to $p\left(x\right)$. One of the advantages of the KBR approach is its ability to perform Bayesian inference, even in the absence of a specific parametric model or an analytic form for the prior and likelihood densities. By making sufficient observations of the system, the KBR allows for the estimation of probabilities and the construction of probabilistic models, thereby enabling a more flexible approach to statistical analysis.

Similarly, the conditional operator $E_Q:{\mathcal{H}}_x\otimes {\mathcal{H}}_y$ satisfying $m_{Q\left(X\right|Y)}\left(y\right)=E_Q^{*}\psi \left(y\right)$ is a minimizer of the loss function.

$$\begin{array}{c}\hfill {\mathcal{L}}^Q\left(E\right)=E_Q[\parallel \varphi \left(X\right)-E^{*}\psi \left(Y\right){\parallel }^2].\end{array}$$

(1)

However, since the posterior distribution (Q) is unknown, we cannot sample the dataset (${\left\{(x_i,y_i)\right\}}_{i=1}^n$) from the posterior distribution (Q) directly. But we could still utilize the analytical form of $E_Q$ according to [14], which is given by

$$\begin{array}{c}\hfill E_Q={\left(C_{YY}^Q\right)}^{-1}{C}_{XY}^Q.\end{array}$$

(2)

In the context of vector-valued kernel regression, the covariance matrices are replaced by empirical estimators as follows:

$$\begin{array}{c}\hfill {\widehat{C}}_{XY}^Q=\sum _{i=1}^n{\mu }_i\varphi \left(x_i\right)\otimes \psi \left(y_i\right)\\ \hfill {\widehat{C}}_{YY}^Q=\sum _{i=1}^n{\mu }_i\psi \left(y_i\right)\otimes \psi \left(y_i\right)\end{array}$$

where each coefficient ($\mu$) is given by

$$\begin{array}{c}\hfill {\mu }_i=<\varphi \left(x_i\right),{({\widehat{C}}_{XX}^P+\eta I)}^{-1}{\widehat{m}}_{\Pi })>.\end{array}$$

(3)

In Equation (3), $\eta$ is another Tikhonov regularization parameter, and ${\widehat{m}}_{\Pi }$ is the prior mean, given as ${\sum }_{i=1}^n{\xi }_i\varphi \left(x_i\right)$, where ${\xi }_i$ represents the weights.

However, since the ${\mu }_i$ coefficient may not necessarily be positive, ${\widehat{C}}_{YY}^Q$ could not be positive semi-definite. Consequently, calculating Equation (2) causes instabilities when inverting the operator ($C_{YY}^Q$). To address this issue, Fukumizu et al. [14] proposed an alternative formulation of $E_Q$, which is expressed as follows:

$$\begin{array}{c}\hfill {\widehat{E}}_{Q,\lambda }={\widehat{C}}_{YY}^Q{({\left({\widehat{C}}_{YY}^Q\right)}^2+\lambda I)}^{-1}{\widehat{C}}_{XY}^Q\end{array}$$

(4)

In the next section, we introduce the iteratively re-weighted importance kernel Bayes rule for application to the filtering problem, a novel design for a KBR that does not require the problematic second-stage regularization. The essential idea is to use multiple weight functions.

3. Iteratively Re-Weighted Importance Kernel Bayes Rule

3.1. Importance Weighted Kernel Bayes Rule with Density Ratio

Here, we review the importance of the weighted kernel Bayes rule proposed in [16]. The method attempts to minimize the loss function (${\mathcal{L}}^Q$), which is estimated through importance sampling by applying a density ratio of $r\left(x\right)={\displaystyle \frac{\pi \left(x\right)}{p\left(x\right)}}$. The loss function can be reformulated as

$$\begin{array}{c}\hfill {\mathcal{L}}^Q\left(E\right)=E_{P}r\left(X\right){\parallel \varphi \left(X\right)-E^{*}\psi \left(Y\right)\parallel }^2\end{array}$$

(5)

The empirical loss function with added Tikhonov regularization can be constructed as follows:

$$\begin{array}{c}\hfill {\widehat{\mathcal{L}}}^{(Q,\lambda )}\left(E\right)={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\widehat{r}}_i\left(x_i\right)\parallel \varphi \left(x_i\right)-E^{*}\psi \left(y_i\right){\parallel }^2+\lambda {\parallel E^{*}\parallel }^2\end{array}$$

(6)

According to the Kernel-based unconstrained Least-Squares Importance Fitting (KuLSIF) estimator proposed in [16,18], ${\widehat{r}}_i$ is

$$\begin{array}{c}\hfill \widehat{r}\left(x_i\right)=max(0,{\mu }_i),\end{array}$$

where ${\mu }_i$ is the same as in Equation (3).

The minimizer of ${\widehat{\mathcal{L}}}^{Q,\lambda }\left(E\right)$ can be obtained analytically as

$$\begin{array}{c}\hfill {\widehat{E}}_{(Q,\lambda )}={({\widehat{C}}_{YY}^Q+\lambda I)}^{-1}{\widehat{C}}_{XY}^Q,\end{array}$$

(7)

where

$$\begin{array}{c}\hfill {\widehat{C}}_{XY}^Q={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\widehat{r}}_i\varphi \left(x_i\right)\otimes \psi \left(y_i\right){\widehat{C}}_{YY}^Q={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\widehat{r}}_i\psi \left(y_i\right)\otimes \psi \left(y_i\right)\end{array}$$

(8)

The importance sampling technique enables the estimation of expectations under the posterior distribution using samples from the prior distribution, which is crucial for Bayesian filtering in high-dimensional spaces. The density ratio ($r\left(x\right)$) serves as the importance weights, ensuring proper weighting of the samples from the prior distribution when estimating posterior expectations. By incorporating these weights into the loss function, the weighted importance kernel Bayes rule is able to accurately estimate the posterior distribution from the available prior samples. This approach results in superior numerical stability relative to the existing approach to KBR.

3.2. Re-Weighted Importance Kernel Bayes Rule

The re-weighted method is a technique designed to enhance the robustness of the estimation process. In traditional least squares regression, which aims to minimize the sum of squared differences between observed and latent values, outliers can significantly influence the results, as a few extreme data points can disproportionately affect the outcome. The re-weighted scheme addresses this sensitivity by assigning different weights to the residuals, leading to a more robust estimation that is less susceptible to the influence of outliers. Equation (5) can be regarded as a kind of least square regression that could employ the re-weighted method to enhance the robustness of the approach.

Similarly, the loss function with multiple weights is constructed as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}^Q\left(E\right)=E_P\left(\omega \right(\parallel \varphi \left(X\right)-E^{*}\psi \left(Y\right)\parallel )\left[r\left(X\right)\right]\parallel \varphi \left(X\right)-E^{*}\psi \left(Y\right){\parallel }^2)\end{array}$$

(9)

where $\omega$ is a weight function of the residual ($(\parallel \varphi \left(X\right)-E^{*}\psi \left(Y\right)\parallel )$). Additionally, the empirical loss function, augmented with Tikhonov regularization, can be recast as follows:

$$\begin{array}{c}\hfill {\widehat{\mathcal{L}}}^{(Q,\lambda )}\left(E\right)={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\omega }_i{\widehat{r}}_i\parallel \varphi \left(x_i\right)-E^{*}\psi \left(y_i\right)\parallel +\lambda \parallel E^{*}{\parallel }^2\end{array}$$

(10)

where ${\omega }_i=\omega (\parallel \varphi \left(x_i\right)-E^{*}\psi \left(y_i\right)\parallel )$.

Iteratively re-weighted least square (IRLS) is a common robust learning paradigm [17] that implements multiple weighted processes for better performance. It can be expressed by a sequence of successive minima of weighted $\mathcal{L}$ based theoretical regularized risk. The $k+1$th iteration problem can be written as follows:

$$\begin{array}{c}\hfill {\widehat{E}}_{Q,\lambda ,\omega ,(k+1)}=argmin{\displaystyle \frac{1}{n}}\sum _{i=1}^n\omega (\parallel \varphi \left(x_i\right)-{\widehat{E}}_{Q,\lambda ,\omega ,k}^{*}\psi \left(y_i\right)\parallel ){\widehat{r}}_i\parallel \varphi \left(x_i\right)-{\widehat{E}}_{Q,\lambda ,\omega ,(k+1)}^{*}\psi \left(y_i\right)\parallel \\ \hfill +\lambda \parallel {\widehat{E}}_{Q,\lambda ,\omega ,(k+1)}^{*}\parallel .\end{array}$$

Next, we write ${\widehat{E}}_{Q,\lambda ,\omega }$ as ${\tilde{E}}_Q$. The weight function ($\omega \left(x\right):\mathbb{R}\to [0,1]$) should satisfy the following three conditions:

1.

$\omega \left(x\right)$ should be a Borel measurement;

2.

$\omega \left(x\right)$ should be an even function; and

3.

$\omega \left(x\right)$ should be a differential, especially, ${\omega }^{\prime }\left(x\right)<0$ when $x>0$.

${\tilde{E}}_{Q,k+1}$ can be obtained analytically as

$$\begin{array}{c}\hfill {\tilde{E}}_{Q,k+1}={(\mathit{\omega }\left({\tilde{E}}_{Q,k}\right){\widehat{C}}_{YY}^Q+\lambda I)}^{-1}\mathit{\omega }\left({\tilde{E}}_{Q,k}\right){\widehat{C}}_{XY}^Q,\end{array}$$

(11)

where $\mathit{\omega }=$ diag $\left({\omega }_i\right)$ and ${\omega }_i\left({\tilde{E}}_{Q,k}\right)=\omega (\parallel \varphi \left(x_i\right)-{\tilde{E}}_{Q,k}^{*}\psi \left(y_i\right)\parallel )$, ${\widehat{C}}_{YY}$ and ${\widehat{C}}_{XY}$ are the same as in Equation (8). Note that

$$\begin{array}{c}\hfill {\tilde{E}}_Q={(\mathit{\omega }\left({\tilde{E}}_Q\right){\widehat{C}}_{YY}^Q+\lambda I)}^{-1}\mathit{\omega }\left({\tilde{E}}_Q\right){\widehat{C}}_{XY}^Q.\end{array}$$

(12)

According to Theorem 1 in [17], the iterative process converges such that ${\tilde{E}}_{Q,k+1}\to {\tilde{E}}_Q$, and [17] has already proven the robustness of the re-weighted scheme. This iterative re-weighted scheme not only reduces the impact of outliers but also adaptively refines the estimate based on the current residuals, leading to improved performance in handling complex, high-dimensional data.

Theorem 1.

(Convergence analysis) Suppose that kernels $k_x$ and $k_y$ are continuous and bounded. Additionally, assume the density ratio ($r_0$) and the conditional operator ($E_Q$) are smooth. Given data (${\left\{(x_i,y_i)\right\}}_{i=1}^n\sim P$) and estimated covariance operators such that $\parallel {\widehat{C}}_{YY}^{\left(Q\right)}-C_{YY}^{\left(Q\right)}\parallel \le O_p\left(n^{-\alpha }\right)$ and $\parallel {\widehat{C}}_{XY}^{\left(Q\right)}-C_{XY}^{\left(Q\right)}\parallel \le O_p\left(n^{-\alpha }\right)$, by setting $\lambda =O_p(n^{-{\displaystyle \frac{\alpha }{\beta +1}}})$ we have

$$\begin{array}{c}\hfill \parallel {\tilde{E}}_Q-E_Q\parallel \le O_p(n^{-{\displaystyle \frac{\alpha \beta }{\beta +1}}}).\end{array}$$

where $E_Q$ is Equation (2). The proof presented is in Appendix A.

The Weight Function to Choose

Many weight functions have been described in the literature, e.g., the Huber weight function [19] and the Hampel weight function [20], which do not satisfy the convergence conditions. In the following, we introduce three types of weight functions that have already satisfied the convergence conditions (1–3).

Logistic function: Debruyne et al. [17], introduced the following logistic weight function:

$$\begin{array}{c}\hfill \omega \left(x\right)={\displaystyle \frac{tanh\left(x\right)}{x}}.\end{array}$$

(13)

This function is a variant of the standard logistic function, which is commonly employed to model probabilities in logistic regression. The logistic weight function is symmetric around zero and non-negative, with a derivative that becomes negative as the residual increases, effectively down-weighting observations with larger residuals.

The ℓ_s-induced weight function: Dong and Yang [21] introduced the following weight function:

$$\begin{array}{c}\hfill \omega \left(x\right)={\displaystyle \frac{a}{2x}}{\displaystyle \frac{1}{(1+exp(-ax)-\frac{1}{2})}},\end{array}$$

(14)

where a is a constant parameter. The ℓ_s-induced weight function is also symmetric around zero and non-negative, and its derivative becomes negative as the residual increases, resulting in a progressive decrease in the weight assigned to larger residuals.

Tukey Biweight function: Tukey’s Biweight function, often used in robust statistics, is a weighting function that assigns weights to data points [22]. It is also known as Tukey’s bisquare function. It is defined as follows:

$$\omega \left(x\right)=\left\{\begin{array}{cc}{\left(1-{\left({\displaystyle \frac{x}{c}}\right)}^2\right)}^2\hfill & \mathrm{if}\left|x\right|<c\hfill \\ 0\hfill & \mathrm{if}\left|x\right|\ge c\hfill \end{array}.\right.$$

where c is a constant that determines the range of the function. The function assigns higher weights to data points closer to zero and rapidly decreases the weights for data points farther away. This makes it less sensitive to outliers than a simple average.

These weight functions are designed to assign lower weights to larger residuals, which effectively reduces their influence on operator estimation. This strategy contributes to the development of robust regression techniques that are less sensitive to outliers.

4. The Iteratively Re-Weighted Importance Kernel Bayes Filter

In this section, we describe the iteratively re-weighted importance kernel Bayes filter. Kernel Bayesian inference is a well-founded approach to non-parametric data in probabilistic graphical models, where probabilistic relationships between variables are learned from data in a non-parametric manner.

In the filtering problem, the states evolve according to a Markov process determined by the state transition model ($p\left(x_{t+1}\right|x_t)$) describing the conditional probability of the next state ($x_{t+1}$) given the current state ($x_t$). Observation $y_t$ at time t is generated depending only on the corresponding state ($x_t$) following the observation model ($p\left(y_t\right|x_t)$). When applying the kernel Bayes rule, we do not need to assume the conditional probabilities ($p\left(y_t\right|x_t)$ and $p\left(x_{t+1}\right|x_t)$) to be known explicitly, nor do we estimate them with simple parametric models. Rather, we assume that a sample $\{x_1,\cdots ,x_T,y_1,\cdots ,y_T\}$ is given for both the observable and hidden variables in the training phase.

The aim of the filtering method is to probabilistically estimate the state ($x_{t+1}$) at each time ($t+1$) using the new observation sequence (${\widehat{y}}_1,\cdots ,{\widehat{y}}_{t+1}$), i.e., to estimate $p\left(x_{t+1}\right|{\widehat{y}}_1,\cdots ,{\widehat{y}}_{t+1})$. The sequential estimate for the kernel mean of $p\left(x_{t+1}\right|{\widehat{y}}_1,\cdots ,{\widehat{y}}_{t+1})$ can be derived by employing the iteratively re-weighted kernel Bayes rule. This can be obtained by iterating the following two steps.

Prediction step

Assume that we have a posterior embedding of $m_{x_t|{\widehat{y}}_{1:t}}$ at time t. Then, we can compute the embedding of forward prediction ($m_{x_{t+1}|{\widehat{y}}_{1:t}}$) as follows:

$$\begin{array}{c}\hfill m_{x_{t+1}|{\widehat{y}}_{1:t}}={\left(E_{x_{t+1}|x_t}\right)}^{*}{m}_{x_t|{\widehat{y}}_{1:t}}\end{array}$$

where $E_{x_{t+1}|x_t}$ is the conditional operator for $P\left(x_{t+1}\right|x_t)$. Empirically, this is estimated based on $\{x_t,x_{t+1}\}$ as follows:

$$\begin{array}{c}\hfill {\widehat{E}}_{x_{t+1}|x_t}={({\widehat{C}}_{x_{T-1},x_{T-1}}+{\lambda }_{1}I)}^{-1}{\widehat{C}}_{x_{T-1},x_T},\end{array}$$

where ${\lambda }_1$ is another regularizing coefficient and

$$\begin{array}{c}\hfill {\widehat{C}}_{x_{T-1},x_{T-1}}={\displaystyle \frac{1}{T-1}}\sum _{i=1}^{T-1}\varphi \left(x_i\right)\otimes \varphi \left(x_i\right),\\ \hfill {\widehat{C}}_{x_{T-1},x_T}={\displaystyle \frac{1}{T-1}}\sum _{i=1}^{T-1}\varphi \left(x_i\right)\otimes \varphi \left(x_{i+1}\right)\end{array}$$

Update step

When a new observation (${\widehat{y}}_{t+1})$ is obtained, the mean embedding of the posterior distribution is expressed as follows:

$$\begin{array}{c}\hfill m_{x_{t+1}|{\widehat{y}}_{1:t+1}}=\left({\tilde{E}}_Q\right)\psi \left({\widehat{y}}_{t+1}\right)\end{array}$$

where ${\tilde{E}}_Q$ is ${(\mathit{\omega }\left({\tilde{E}}_Q\right){\widehat{C}}_{YY}^Q+\lambda I)}^{-1}\mathit{\omega }\left({\tilde{E}}_Q\right){\widehat{C}}_{XY}^Q$.

During the calculation, it is preferable to calculate the Gram matrices instead of covariance matrices. This approach simplifies the calculation and can improve computational efficiency. In general, we need to express the mean embedding in the form of a feature map with latent training points as follows:

$$\begin{array}{c}\hfill m_{x_{t+1}|{\widehat{y}}_{1:t+1}}=\sum _{i=1}^T{\xi }_i^{t+1,t+1}\varphi \left(x_i\right),\end{array}$$

(15)

where ${\xi }^{t+1,t+1}$ is the coefficient at time $t+1$. Notably, There is a rule ($[a\otimes b]c=a<b,c>$) that can be employed to simplify the covariance matrix in the Gram matrix. Hence, the coefficient is

$$\begin{array}{c}\hfill \xi ={(\mathit{\omega }\left(\widehat{\mathbf{r}}{G}_Y\right)+T\lambda I)}^{-1}\mathit{\omega }\left(\widehat{\mathbf{r}}{\widehat{G}}_Y\right)\end{array}$$

(16)

where Gram matrices are ${\left(G_Y\right)}_{ij}=k_y(y_i,y_j)$ and ${\left({\widehat{G}}_Y\right)}_{ij}=k_y(y_i,{\widehat{y}}_j)$, ${\left(\widehat{\mathbf{r}}\right)}_{ii}=<\varphi \left(x_i\right),T{(G_X+T\eta I)}^{-1}{\widehat{m}}_{\Pi }>$. The algorithm can be summarized as follows (see Algorithm 1).

In this algorithm, we use a trick to calculate the weight. In the iteration step, $E_{k+1}$ should be ${({\omega }_k\widehat{r}{G}_Y+T\lambda I)}^{-1}\left({\omega }_k\widehat{r}{C}_{XY}\right)$, but we use the $E_{k+1}\psi \left(y_i\right)$; then, we can calculate the Gram matrix ($G_Y$) instead of the covariance matrix ($C_{XY}$).

Kernel methods have some limitations because they rely on predefined features from the RKHS, which may not work well with complex or high-dimensional data. To address this, adaptive neural network features [16] refer to the features generated by neural networks that can automatically adjust and learn from the data during the training process. Unlike fixed features, these adaptive features evolve to better capture the underlying patterns in the data, especially in complex or high-dimensional scenarios. This adaptability makes them particularly useful in situations where traditional methods, like kernel methods with predefined feature maps, may struggle to represent the data effectively. Here, we briefly introduce this concept. We rewrite the feature map ($\psi$) as ${\psi }_{\theta }$, an adaptive feature map represented by a neural network parameterized by $\theta$. The optimal $\theta$ is obtained by minimizing the function as follows:

$$\begin{array}{c}\hfill \mathit{\ell }\left(\theta \right)=\mathrm{tr}\left(G_X\left(\widehat{\mathbf{r}}-\widehat{\mathbf{r}}{\Psi }_{\theta }^T{({\Psi }_{\theta }\widehat{\mathbf{r}}{\Psi }_{\theta }^T+\lambda I)}^{-1}{\Psi }_{\theta }\widehat{\mathbf{r}}\right)\right)\end{array}$$

(17)

where ${\Psi }_{\theta }=[{\psi }_{\theta },\cdots ,{\psi }_{\theta }]$.

The Gram matrix in Equation (16) is defined as ${\left(G_Y\right)}_{ij}=k_y(y_i,y_j)={\psi }_{\theta }{\left(y_i\right)}^T{\psi }_{\theta }\left(y_j\right)$.

In the following experiment, we use the linear kernel on the learned adaptive feature ($\psi \left(y\right)$) and finite-dimensional random Fourier feature approximation of $\varphi \left(x\right)$ to calculate the coefficient ($\xi$) [23]. We use a linear kernel of $\varphi \left(x\right)=x$ to estimate the posterior mean (${\widehat{m}}_Q=\sum {\xi }_i{x}_i$) in the latent space. This approach does not necessitate the same feature map used to compute the weight function, as demonstrated in [14]  

Algorithm 1: Iteratively Re-weighted Importance Kernel Bayes Filter Algorithm

Input: Training dataset ${\left\{(x_t,y_t)\right\}}_{i=1}^T$, regularization parameters $\eta ,{\lambda }_1,\lambda$,    and test sequence ${\widehat{y}}_1,\cdots ,{\widehat{y}}_{\tilde{T}}$. Initialize ${\xi }^{(0,1)}$. Compute Gram matrices $G_X,G_Y\in {\mathbb{R}}^{T\times T}$.    ${\left(G_X\right)}_{ij}=k_x(x_i,x_j)$, ${\left(G_Y\right)}_{ij}=k_y(y_i,y_j)$. For $t=1$ to $\tilde{T}$ do:       1. Compute the prior embedding ${\widehat{m}}_{\Pi }$ at conditioning point ${\widehat{y}}_t$ using training dataset          ${\left\{(x_t,y_t)\right\}}_{i=1}^T$:       ${\widehat{m}}_{\Pi }={\sum }_{i=1}{\xi }_i^{t-1,t}\varphi \left(x_i\right)$.       2. Compute the density ratio $\widehat{r}=({\widehat{r}}_1,\dots ,{\widehat{r}}_T)$ in ${\mathbb{R}}^T$ as:          $\widehat{r}=max(0,T{(G_X+T\eta I)}^{-1}{g}_{\Pi })$,          where ${\left(g_{\Pi }\right)}_i=<{\widehat{m}}_{\Pi },\varphi \left(x_i\right)>$.       3. For $k=1$ to K:             Compute weight ${\omega }_k$, where ${\omega }_{k,i}=\omega (\parallel \varphi \left(x_i\right)-E_k\psi \left(y_i\right)\parallel )$.                Update embedding $E_{k+1}\psi \left(y_i\right)$ as:                $E_{k+1}\psi \left(y_i\right)={({\omega }_k\widehat{r}{G}_Y+T\lambda I)}^{-1}\left({\omega }_k\widehat{r}{G}_Y\right)\varphi \left(x_i\right)$,                if $\parallel E_{k+1}\psi \left(y_i\right)-E_k\psi \left(y_i\right)\parallel \le ϵ$, stop.       4. Compute ${\xi }^{t,t}$ as:          ${\xi }^{t,t}={(\mathit{\omega }\widehat{\mathbf{r}}{G}_Y+T\lambda I)}^{-1}\left(\mathit{\omega }\widehat{\mathbf{r}}{\widehat{G}}_Y\right)$,          where $\mathit{\omega }$ is diag(${\omega }_K$), $\widehat{\mathbf{r}}$ is diag($\widehat{r}$), ${\widehat{G}}_Y=k_y(y_i,{\widehat{y}}_t)$.       5. Compute ${\xi }^{t,t+1}$ as:          ${\xi }_1^{(t,t+1)}=0$ and ${\xi }_{2:T}^{t,t+1}={(G_{X_{-1}}+(T-1){\lambda }_{1}I)}^{-1}{\tilde{G}}_{X_{-1}}{\xi }^{t,t}$,          where ${\left(G_{X_{-1}}\right)}_{ij}=k_x(x_i,x_j)\in {\mathbb{R}}^{T-1\times T-1}$ and ${\left({\tilde{G}}_{X_{-1}}\right)}_{ij}=k_x(x_i,x_j)\in {\mathbb{R}}^{T-1\times T}$. End For

5. Numerical Illustration

In this section, we explore the effectiveness of our method by applying it to two nonlinear dynamical models, with the code written in Python.

A Synthetic Problem:

Inspired by Fukumizu et al. [14] and Xu et al. [16], a simple, synthetic, nonlinear dynamic is introduced to illustrate the kernel Bayesian filtering methods. With a latent variable of $X_t={(u_t,v_t)}^T$, the synthetic dynamic is described as

$$\left[\begin{array}{c}{u}_{t+1}\\ v_{t+1}\end{array}\right]=\left(1+\beta sin\left(M{\theta }_t\right)\right)\left[\begin{array}{c}cos({\theta }_t+\omega )\\ sin({\theta }_t+\omega )\end{array}\right]+{ϵ}_X$$

(18)

where ${\theta }_t\in [0,2\pi ]$ and

$$cos\left({\theta }_t\right)={\displaystyle \frac{u_t}{\sqrt{u_t^2+v_t^2}}},sin\left({\theta }_t\right)={\displaystyle \frac{v_t}{\sqrt{u_t^2+v_t^2}}}$$

for given parameters of $(\beta ,M,\omega )$ and ${ϵ}_X\sim \mathcal{N}(0,{\sigma }_X^{2}I)$. The noise (${ϵ}_X$) is an independent process. Observation $Y_t$ is expressed as

$$\begin{array}{c}\hfill Y_t=X_t+{ϵ}_Y.{ϵ}_Y\sim \mathcal{N}(0,{\sigma }_Y^{2}I)\end{array}$$

where ${ϵ}_Y$ is independent noise. In the following experiments, the parameters in the dynamics are set as $\omega =0.2,\beta =0.2$, and $M=8$. Additionally, the noise levels are set to ${\sigma }_Y=0.2$ and ${\sigma }_X=0.2$, respectively. The $\eta$ and ${\lambda }_1$ parameters are tuned by Kernel-based unconstrained least squares importance fitting (KuLSIF) leave-one-out cross-validation procedure. The bandwidth Gaussian kernel employed for all KBF methods is given by the medians of pairwise distances among the training set.

In the following figures, “original” denotes the performance of original KBF estimator using the ${\widehat{E}}_{Q,\lambda }$ operator in Equation (4). IW and re-weight represent importance KBF using the ${\widehat{E}}_{(Q,\lambda )}$ operator in Equation (7) and the iteratively re-weighted importance kernel Bayes filter using the ${\tilde{E}}_Q$ operator in Equation (12), respectively.

Figure 1 illustrates the posterior approximation of latent variables u and v. The red line represents the given latent test sequence. The blue line, yellow line, and green line are the three KBF methods used to approach the latent variables. This experiment was conducted with a training sequence length of 100 and the regularized parameter ($\lambda$) set to ${10}^3$. The ℓ-induced weight function was employed in The IRe-KBF method in this experiment.

As observed in Figure 1, it is evident that the KBF method utilizing iterative re-weighting yields the closest approximation to the actual latent test values. This indicates that this particular method outperforms the other two techniques in this context. However, this kind of obvious situation is not always the case. Many times, the results are not obvious. Hence, we calculate the MSE, which is the mean square error between the posterior estimation of the latent variables and the latent test value to evaluate the performance of the methods, as shown in Figure 2.

First, we conducted some research about the kernel Bayes filter with three different weight functions described in Section 3 to find the optimize weight function. In this experiment, the parameters were $a=1$ in the ℓ_s-induced weight function and $c=1$ in the Tukey biweight function. Essential regularization ($\lambda$) is obtained by the hyperparameter tuning method. Figure 3 summarizes the mean squared error (MSE) over 10 runs when the conditional points (${\widehat{y}}_t$) are sampled. The length of test sequence is set to 50.

Figure 3 shows that the IRe-KBF with the ℓ-induced weight function leads to the fewest errors between the posterior estimation and the test sequence. Hence, in the following, we apply the ℓ-induced weight function withing the IRe-KBF method.

The MSE with varying training lengths is summarized in Figure 2. As depicted in the figure, the MSE decreases as the training length increases, which is expected. Furthermore, the “re-weight” method demonstrates comparable or superior performance compared to the other two KBF methods.

We also conducted an experiment employing the ensemble Kalman filter (EKF), which yielded a mean squared error (MSE) of 0.049. This value is notably lower than the MSE obtained by the other three filter methods when the training length is less than 200. This result is reasonable, since the ensemble Kalman filter is applied under an explicit model with a relatively low-dimensional system.

Lorenz 96 model:

The Lorenz 96 model is a simplified mathematical model used to study atmospheric and climate dynamics. It was proposed by Edward Lorenz in 1996 to capture key nonlinear characteristics of such systems in a more manageable form.

The standard form of the Lorenz 96 model is given by the following set of differential equations:

$${\displaystyle \frac{d{x}_i}{dt}}=(x_{i+1}-x_{i-2})·x_{i-1}-x_i+F,$$

(19)

where

• $x_i$ represents the ith variable of the system, i.e., $i=1,\cdots ,N$;
• F is an external forcing term that represents the energy input into the system; and
• The indices are treated cyclically, meaning $x_{N+1}=x_1$ and $x_0=x_N$.

If the Lorenz 96 model has four variables (i.e., $x_1,x_2,x_3,x_4$), the system of equations becomes

$${\displaystyle \frac{d{x}_1}{dt}}=(x_2-x_4)x_3-x_1+F$$

$${\displaystyle \frac{d{x}_2}{dt}}=(x_3-x_1)x_4-x_2+F$$

$${\displaystyle \frac{d{x}_3}{dt}}=(x_4-x_2)x_1-x_3+F$$

$${\displaystyle \frac{d{x}_4}{dt}}=(x_1-x_3)x_2-x_4+F$$

This system is a four-dimensional nonlinear system, since it involves four state variables ($X={(x_1,x_2,x_3,x_4)}^T$), and each variable’s evolution is influenced by nonlinear interactions with the other variables. The Lorenz 96 model can be generalized to any number of variables (N), but in this case, we are considering the specific case where $N=4$.

Observation Y is expressed as follows:

$$Y_t=H{X}_t+{ϵ}_Y$$

where ${ϵ}_Y$ is independent noise with ${ϵ}_Y\sim \mathcal{N}(0,{\sigma }_Y^{2}I)$, and $X_t$ is the state at time t.

In this experiment, the observation matrix (H) is expressed as follows:

$$H=\left(\begin{array}{cccc}1& 0& 0& 0\\ 0& 0& 1& 0\end{array}\right).$$

In the filtering problem, we have parameters of $F=1$ and $N=4$. The dynamical systems is discrete, with a time step $dt=0.1$. The latent variable noise is set to zero, while the observable noise is characterized by ${\sigma }_Y=0.1$. The test size for this experiment was set to 70. Additionally, in the filtering problem, the $\eta$ and ${\lambda }_1$ parameters are tuned by a kernel-based unconstrained least squares importance fitting (KuLSIF) leave-one-out cross-validation procedure. Essential regularization $\lambda$ is obtained by the hyperparameter tuning method, and the ℓ-induced weight function was employed in this experiment. The results presented in Figure 4 consist of graphs depicting the relationship in the whole filtering process between $x_1,x_2$, and $x_3$, which are the first three variables in the latent variable vector (X). It is clear to see that the KBF with “re-weighting” yields a better approximation of the actual test latent value, which is given by the dynamics, than the other two methods.

We also computed the MSE between the posterior estimation and the test latent sequence. The results are depicted in Figure 5. As illustrated in Figure 5, the “re-weight” method consistently performs better than the other two methods. The original Kernel Bayesian filter exhibits significant instability in performance. However, it is worth noting that the obtained results are sensitive to the $\lambda$ parameter. In Figure 6, we observed that the performance of IRe-KBF significantly declines when the $\lambda$ parameter is small.

Furthermore, we conducted an experiment using the extended Kalman filter (EKF), resulting in an MSE of 375.21. This value is notably larger when compared to the performance of the three kernel-based filtering methods. The limitations of the EKF are apparent, particularly in handling complex nonlinear systems. Conversely, the iteratively re-weighted importance kernel Bayes filter (IRe-KBF) demonstrates its ability to capture intricate nonlinear dependencies between observations and latent states.

6. Conclusions

In this paper, we proposed a robust approach to kernel Bayes filtering that does not depend on an explicit dynamical system. The kernel Bayes filter process only depends on a training dataset to calculate the transition and the observation operator. This method employs multiple weight functions, i.e., the density ratio and weight function related to residuals. Additionally, we employed the iteratively re-weighted process to enhance robustness and efficiency.

Two examples that represent high-dimensional dynamics, namely a nonlinear synthetic model and Lorenz 96, were applied to illustrate the performance of the IRe-KBF. Experimental results demonstrate the effectiveness of IRe-KBF in handling complex data, with superior performance over traditional KBF methods and the importance weighted KBF method on high-dimensional datasets.

In this paper, we also conducted some research on the weight function, although more research is needed in this area. Future work will focus on exploring and refining weight functions to optimize their performance in kernel Bayes filtering applications.