Empowering Advanced Driver-Assistance Systems from Topological Data Analysis

Abstract

We are interested in evaluating the state of drivers to determine whether they are attentive to the road or not by using motion sensor data collected from car driving experiments. That is, our goal is to design a predictive model that can estimate the state of drivers given the data collected from motion sensors. For that purpose, we leverage recent developments in topological data analysis (TDA) to analyze and transform the data coming from sensor time series and build a machine learning model based on the topological features extracted with the TDA. We provide some experiments showing that our model proves to be accurate in the identification of the state of the user, predicting whether they are relaxed or tense.

1. Introduction

While there have recently been considerable advances in self-driving car technology, driving still relies mainly on human factors. Even in self-driving mode, human drivers must often make decision in a fraction of a second to avoid accidents. Therefore, it is still of utmost importance to develop systems capable of discerning if the human driver is attentive or not to the road conditions. In general, the so-called advanced driver assistance systems (ADAS) [1,2] are systems that are able to improve the driver’s performance, among which, adaptive speed limiters, pedestrian detectors [3], and cruise controllers are some of the most popular systems. Fatigue alerting systems are among the most useful among ADAS systems, and the aim of this work is to contribute to the development of such a system based on a systematic analysis of drivers in actual driving conditions.

The estimation of the driver’s condition (degree of attention to the road, fatigue, etc.) is a very important factor to ensure safety in driving [4,5]. A recent review on the topic can be found in [6]. The goal of this work is to extract behavior patterns from car user data to be able to accurately estimate their state. We used data obtained by the laboratory of prof. Hyung Yun Choi at Hongik University in Seoul. His experiment involved the application of mechanical stimulation to people seated in an automobile.

Our main goal is to extract patterns of behavior from experimental data so as to allow us to learn the most relevant factors affecting driver’s attention to the situation of the road.

In the present work, we combine some tools from Morse theory [7] and topological data analysis (TDA) with all of the associated concepts and methods (e.g., Betti numbers, homology persistence, barcodes, persistence images, etc.) [8], most of them introduced and employed later in order to analyze and classify the experimental data. This allows us to introduce concepts as barcodes, that is, persistent and life-time diagrams in a similar way to how they are used in persistent homology. Our main goal is to predict car user behavior following a supervised approach [9]. Instead of considering an original sensor signal as the quantity of interest, we focus on its topological features. In this sense, the framework proposed in this paper allows us to unveil the true dimensionality of data or, in other words, the actual number of factors affecting driver’s performance. Thus, we model a sensor signal as a dynamical system, and, therefore, our approach seems to be better at describing its properties, or rather its variations, such as extrema, patterns, and self-similarity, than other approaches. We note that our approach is, in some senses, similar to that followed by Milnor and Thurston [10] in the study of the combinatorial properties of dynamical systems by combining tools from automata theory.

The structure of the paper is as follows: In Section 2, we describe the material and methods employed in this work. Particular attention is paid to the process of data acquisition and the description of time series and data curation. In Section 3, we present the main results of this work, and we discuss the main consequences in Section 4. As a complement, in Appendix A, we thoroughly illustrate the process of computing persistence images for the data of interest.

2. Material and Methods

In this section, we describe the collection and preprocessing of the experimental data. In Section 2.1, we describe the data acquisition, and in Section 2.2, we provide a description of the time series. Section 2.3 is devoted to data preprocessing. The mathematical tools used to describe the times series at a topological level are explained in Section 2.4. Finally, the image classification methodology is given in Section 2.5.

2.1. Data Acquisition

Our proposed predictor directly uses the data collected from the experiments. The data acquisition process involves measuring the response of human behavior when an excitation is applied to the seat. Figure 1 shows the location of the sensors in the experiments.

The excitation signal is an angular acceleration imposed on the seat of the user. This acceleration is an oscillating chirp function with a frequency range of 1 to 7.5 Hz on the X axis in rotation. The linear acceleration $\mathbf{a}=(a_x,a_y,a_z)$ and angular velocity $\mathit{\omega }=({\omega }_x,{\omega }_y,{\omega }_z)$ were measured in both the head and the seat by two IMU (Shimmer inertia measurement unit (IMU) sensors) at 256 Hz. By observing the floor excitation signals, we noted that the excitation is purely rotational around the X-axis—see Figure 2.

Several experiences were conducted by nine people by taking into account a set of six fixed states: driver, passenger, tense person, relaxed person, rigid seat, and SAV (sport activity vehicle seat). In particular, for each individual, eight experiments for the six available states were performed:

$$\left\{\begin{array}{cc}\mathbf{Class}& \mathbf{Label}\hfill \\ 1& \mathrm{SAVRelaxedPassager}\hfill \\ 2& \mathrm{SAVTensePassager}\hfill \\ 3& \mathrm{SAVRelaxedDriver}\hfill \\ 4& \mathrm{SAVTenseDriver}\hfill \\ 5& \mathrm{RigidRelaxedPassager}\hfill \\ 6& \mathrm{RigidTensePassager}\hfill \\ 7& \mathrm{RigidRelaxedDriver}\hfill \\ 8& \mathrm{RigidTenseDriver}\hfill \end{array}\right.$$

As a consequence, we worked with a sample of 72 experiences, each of them encoded in a time series (as we explain later). Our goal is to classify the behavior of a generic driver, assigning one of the two states (tense or relaxed) by using the sensor data.

2.2. Time Series Description

The data acquired from sensors (see Figure 3 and Figure 4) were stored into six-dimensional time series, for both linear acceleration and angular velocity of the head movement. The sampling frequency of the data was 256 Hz, and the duration of the experiment was 34 s; hence, the resulting data dimensionality is $256\times 34=8704$. For each times series, where $1\le t\le 8704$, we constructed three new times series called the sliding window, embedding a length of 5800. The first one is given by the times values from $t=1$ to $t=5800$, the second is given by the times values from $t=1450$ to $t=7250$, and, to conclude, the third time window is defined as from $t=2904$ to $t=8704$. Each element in the sample $(1\le i\le 72)$ was encoded by means of three six-dimensional time series representing each of the three sliding windows that we represent in matrix form as follows:

$$T{S}_{3(i-1)+1}=\left[\begin{array}{cccc}{a}_x^{\ell }\left(1\right)& a_x^{\ell }\left(2\right)& \cdots & a_x^{\ell }\left(5800\right)\\ a_y^{\ell }\left(1\right)& a_y^{\ell }\left(2\right)& \cdots & a_y^{\ell }\left(5800\right)\\ a_z^{\ell }\left(1\right)& a_z^{\ell }\left(2\right)& \cdots & a_z^{\ell }\left(5800\right)\\ {\omega }_x^{\ell }\left(1\right)& {\omega }_x^{\ell }\left(2\right)& \cdots & {\omega }_x^{\ell }\left(5800\right)\\ {\omega }_y^{\ell }\left(1\right)& {\omega }_y^{\ell }\left(2\right)& \cdots & {\omega }_y^{\ell }\left(5800\right)\\ {\omega }_z^{\ell }\left(1\right)& {\omega }_z^{\ell }\left(2\right)& \cdots & {\omega }_z^{\ell }\left(5800\right)\end{array}\right],T{S}_{3(i-1)+2}=\left[\begin{array}{cccc}{a}_x^{\ell }\left(1450\right)& a_x^{\ell }\left(1451\right)& \cdots & a_x^{\ell }\left(7251\right)\\ a_y^{\ell }\left(1450\right)& a_y^{\ell }\left(1451\right)& \cdots & a_y^{\ell }\left(7251\right)\\ a_z^{\ell }\left(1450\right)& a_z^{\ell }\left(1451\right)& \cdots & a_z^{\ell }\left(7251\right)\\ {\omega }_x^{\ell }\left(1450\right)& {\omega }_x^{\ell }\left(1451\right)& \cdots & {\omega }_x^{\ell }\left(7251\right)\\ {\omega }_y^{\ell }\left(1450\right)& {\omega }_y^{\ell }\left(1451\right)& \cdots & {\omega }_y^{\ell }\left(7251\right)\\ {\omega }_z^{\ell }\left(1450\right)& {\omega }_z^{\ell }\left(1451\right)& \cdots & {\omega }_z^{\ell }\left(7251\right)\end{array}\right]$$

and

$$T{S}_{3i}=\left[\begin{array}{cccc}{a}_x^{\ell }\left(2903\right)& a_x^{\ell }\left(2905\right)& \cdots & a_x^{\ell }\left(8704\right)\\ a_y^{\ell }\left(2903\right)& a_y^{\ell }\left(2905\right)& \cdots & a_y^{\ell }\left(8704\right)\\ a_z^{\ell }\left(2903\right)& a_z^{\ell }\left(2905\right)& \cdots & a_z^{\ell }\left(8704\right)\\ {\omega }_x^{\ell }\left(2903\right)& {\omega }_x^{\ell }\left(2905\right)& \cdots & {\omega }_x^{\ell }\left(8704\right)\\ {\omega }_y^{\ell }\left(2903\right)& {\omega }_y^{\ell }\left(2905\right)& \cdots & {\omega }_y^{\ell }\left(8704\right)\\ {\omega }_z^{\ell }\left(2903\right)& {\omega }_z^{\ell }\left(2905\right)& \cdots & {\omega }_z^{\ell }\left(8704\right)\end{array}\right].$$

Here, the matrices have a size of $6\times 5800$ and $1\le i\le 72$. This allows us to represent the information by using a third-order tensor, namely, $\mathcal{Z}\in {\mathbb{R}}^{216\times 6\times 5800}$ defined by

$${\mathcal{Z}}_{i,j,k}:={\left(T{S}_i\right)}_{j,k}$$

for $1\le i\le 216$, $1\le j\le 6$ and $1\le k\le 5800$. We can identify $Z_i=T{S}_i$ for $1\le i\le 216$.

2.3. Data Preprocessing

In order to obtain a single series for each observation, we concatenated all of the 6 time series (linear accelerations and angular velocities) for each observation horizontally and then created a data frame by stacking the 216 in sample observations.

The concatenation operation on the multidimensional time series collapsed the last two dimensions into one dimensional arrays with a length of $5800\times 6=\mathrm{34,800}$. The result is the two-dimensional table of concatenated time series

$$D=\left[\begin{array}{c}vec\left({\mathcal{Z}}_{1,:,:}\right)\\ \dots \\ vec\left({\mathcal{Z}}_{216,:,:}\right)\end{array}\right]\in {\mathbb{R}}^{216\times 34800}.$$

We chose not to filter the signals because the topological sub-level set method should filter the high-frequency features naturally. We also chose to keep working on acceleration signals in order to avoid signal deviations after two integrations in time so as to obtain positions, the sensors not always keeping a zero mean height. Thus, the approach is completely (topologically) data-based.

The six time series ${\mathcal{Z}}_i$ of each observation were collapsed into a single concatenated time series with a size of 34,800—see Figure 5. The concatenated time series for the 216 observations were then stacked to create the dataset D with a size of $216\times$ 34,800. We also used binary labels in the chained time series ${\mathcal{Z}}_i$ on the two target classes that we were interested in. In particular, we wrote ${\mathcal{Z}}_i^{\left(\alpha \right)}$ where $\alpha$ is "0" for a relaxed driver and “1” for a tense one.

2.4. Extracting Topological Features from a Time Series

The idea to extract the topological information regarding the times series is to consider each sample observation as a piecewise linear continuous map from a closed interval to the real line. Therefore, we used a construction closely related to the Reeb graph [11] used in Morse theory to describe the times series at the topological level.

To this end, we consider the time series $x_t$ for $0\le t\le N-1$$(N\ge 3)$ given by a vector

$$\mathbf{X}=(x_0,x_1,\dots ,x_{N-1})\in {\mathbb{R}}^N.$$

we can view $\mathbf{X}$ as a function also denoted by $\mathbf{X}:\{0,1,\dots ,N-1\}⟶\mathbb{R}$ defined by $\mathbf{X}\left(i\right)=x_i$ for $0\le i\le N-1$. Here, to study the topological features of $\mathbf{X}$ we use the sub-level set of a piecewise-linear function $f_{\mathbf{X}}:\mathbb{R}⟶\mathbb{R}$ associated with $\mathbf{X}$ satisfying that $f_{\mathbf{X}}\left(i\right)=\mathbf{X}\left(i\right)=x_i$ for $0\le i\le N-1$.

To construct this function, we consider the basis functions $\{{\phi }_0,\dots ,{\phi }_{N-1}\}$ of continuous functions ${\phi }_i:\mathbb{R}⟶\mathbb{R}$ defined by

$${\phi }_i\left(s\right):=\left\{\begin{array}{ccc}s-i+1& \mathrm{if}& i-1\le s\le i\\ i+1-s& \mathrm{if}& i\le s\le i+1\\ 0& \mathrm{if}& s\notin ]i-1,i+1[\end{array}\right.$$

where $i=1,\dots ,N-2$ and

$${\phi }_0\left(s\right):=\left\{\begin{array}{ccc}1-s& \mathrm{if}& 0\le s\le 1\\ 0& \mathrm{if}& s\in [0,1]\end{array}\right.$$

$${\phi }_{N-1}\left(s\right):=\left\{\begin{array}{ccc}s-N+2& \mathrm{if}& N-2\le s\le N-1\\ 0& \mathrm{if}& s\notin [N-2,N-1[\end{array}\right.$$

This allows us to construct a piecewise continuous map $f_{\mathbf{X}}:\mathbb{R}⟶\mathbb{R}$ by

$$f_{\mathbf{X}}\left(s\right)=\sum _{j=0}^{N-1}{x}_j{\phi }_j\left(s\right),$$

and also to endow ${\mathbb{R}}^N$ with a norm given by

$$\parallel \mathbf{X}\parallel :=\parallel f_{\mathbf{X}}{\parallel }_{L^2\left(\mathbb{R}\right)}={\left({\int }_{-\infty }^{\infty }{\left|f_{\mathbf{X}}\left(s\right)\right|}^{2}ds\right)}^{1/2}.$$

In particular, we prove the following result, which helps us to identify the time series given by the vector $\mathbf{X}$ in ${\mathbb{R}}^N$ with the function $f_{\mathbf{X}}$ in $L^2\left(\mathbb{R}\right)$.

Proposition 1.

The linear map $\Phi :({\mathbb{R}}^N,\parallel ·\parallel )⟶(L^2{\left(\mathbb{R}\right),\parallel ·\parallel }_{L^2\left(\mathbb{R}\right)})$ given by $\Phi \left(\mathbf{X}\right)=f_{\mathbf{X}}$ is an injective isometry between Hilbert spaces. Furthermore, $\Phi \left({\mathbb{R}}^N\right)$ is a closed subspace in $L^2\left({\mathbb{R}}^N\right)$.

Proof. 

The map is clearly isometric and injective because $\{{\phi }_0,\dots ,{\phi }_{N-1}\}$ is a set of linear independent functions in $L^2\left(\mathbb{R}\right)$. □

Here, we describe the maps $f_{\mathbf{X}}\in \Phi \left({\mathbb{R}}^N\right)$ at the combinatorial level using the connected components (intervals) associated with its $\lambda$ sub-level sets

$${\mathcal{LS}}_{\lambda }\left(f_{\mathbf{X}}\right):=\{x\in [0,N-1]:f_{\mathbf{X}}\left(x\right)\le \lambda \}$$

for $\lambda \in \mathbb{R}$. For this purpose, we introduce the following distinguished objects related to the $\mathrm{supp}\left(f_{\mathbf{X}}\right)=[0,N-1]\subset \mathbb{R}$ of $f_{\mathbf{X}}$:

• The nodes or vertices denoted by

  $$\mathcal{V}:=\left\{\right[0],[1],\dots ,[N-1\left]\right\}$$

  that represent the components of the vector $\mathbf{X},$;
• The faces denoted by

  $$\mathcal{F}:=\left\{\right[0,1\left]\right[1,2],\dots ,[N-2,N-1\left]\right\}$$

  that represent the intervals used to construct the connected components of the sub-level sets of the map $f_{\mathbf{X}}$. Recall that we consider

  $$[i,i+1]:=\{z\in \mathbb{R}:z=\mu x_{i+1}+(1-\mu )x_i,0\le \mu \le 1\}\subset \mathbb{R}.$$

Let

$${\lambda }_{max}=\underset{s\in [0,N-1]}{max}{f}_{\mathbf{X}}\left(s\right)=\underset{0\le i\le N-1}{max}\mathbf{X}\left(i\right),$$

and

$${\lambda }_{min}=\underset{s\in [0,N-1]}{min}{f}_{\mathbf{X}}\left(s\right)=\underset{0\le i\le N-1}{min}\mathbf{X}\left(i\right).$$

For each ${\lambda }_{min}\le \lambda \le {\lambda }_{max}$, we introduce the following symbolic $\lambda$ sub-level set for the map $f_{\mathbf{X}}$:

$$L{S}_{\lambda }\left(f_{\mathbf{X}}\right):=\left\{\sigma \in \mathcal{F}:f\left(\sigma \right)\le \lambda \right\}$$

For ${\lambda }_{min}\le \lambda \le {\lambda }^{\prime }\le {\lambda }_{max}$, it holds

$$L{S}_{\lambda }\left(f_{\mathbf{X}}\right)\subset L{S}_{{\lambda }^{\prime }}\left(f_{\mathbf{X}}\right).$$

Our next goal was to quantify the evolution of the above symbolic $\lambda$ sub-level with. To this end, we introduce the notion of feature associated with the $\lambda$ sub-level set $L{S}_{\lambda }\left(f_{\mathbf{X}}\right)$.

We define the set of features for functions in $\Phi \left({\mathbb{R}}^N\right)$ as

$$\mathfrak{F}(\Phi \left({\mathbb{R}}^N\right)):=\left\{[i,j]\subset \mathbb{R}:0\le i<j\le N-1\right\}.$$

We note that $L{S}_{\lambda }\left(f_{\mathbf{X}}\right)\subset \mathcal{F}\subset \mathfrak{F}(\Phi \left({\mathbb{R}}^N\right))$. Then next definition introduces the notion of features for a symbolic $\lambda$ sub-level set as the interval of $\mathfrak{F}(\Phi \left({\mathbb{R}}^N\right))$ constructed by a maximal union of faces of $L{S}_{\lambda }\left(f_{\mathbf{X}}\right)$.

Definition 1.

We suggest that $\mathbb{I}\in \mathfrak{F}(\Phi \left({\mathbb{R}}^N\right))$ is a feature for the symbolic λ sub-level set $L{S}_{\lambda }\left(f_{\mathbf{X}}\right)$ if there exists ${\mathbb{I}}_1,\dots ,{\mathbb{I}}_k\in L{S}_{\lambda }\left(f_{\mathbf{X}}\right)$ such that $\mathbb{I}={\bigcup }_{j=1}^k{\mathbb{I}}_k$ and for every $\mathbb{J}\in L{S}_{\lambda }\left(f_{\mathbf{X}}\right)$ such that $\mathbb{J}\ne {\mathbb{I}}_i$ for $1\le i\le k$ it holds that $\mathbb{I}\cap \mathbb{J}=\varnothing$. We denote by $\mathfrak{F}\left(L{S}_{\lambda }\left(f_{\mathbf{X}}\right)\right)$ the set of features for the λ sub-level set $L{S}_{\lambda }\left(f_{\mathbf{X}}\right)$.

A feature for a $\lambda$ sub-level set $L{S}_{\lambda }\left(f_{\mathbf{X}}\right)$ is the maximal interval of $\mathfrak{F}(\Phi \left({\mathbb{R}}^N\right))$ that we can construct by unions of intervals in $L{S}_{\lambda }\left(f_{\mathbf{X}}\right)$. To illustrate this definition, we give the following example:

Example 1.

Let us consider the time series

$$\mathbf{X}=(11,14,9,7,9,7,8,10,9).$$

This allows us to construct the map $f_{\mathbf{X}}$ as shown in Figure 6. Then, ${\lambda }_{min}=7$ and ${\lambda }_{max}=14$, and we have the following symbolic λ sub-level sets.

$$\begin{array}{cc}\hfill L{S}_{\lambda =7}\left(f_{\mathbf{X}}\right)& =\varnothing \hfill \\ \hfill L{S}_{\lambda =8}\left(f_{\mathbf{X}}\right)& =L{S}_{\lambda =7}\left(f_{\mathbf{X}}\right)\cup \left\{[5,6]\right\}\hfill \\ \hfill L{S}_{\lambda =9}\left(f_{\mathbf{X}}\right)& =L{S}_{\lambda =8}\left(f_{\mathbf{X}}\right)\cup \{[2,3],[3,4],[4,5]\}\hfill \\ \hfill L{S}_{\lambda =10}\left(f_{\mathbf{X}}\right)& =L{S}_{\lambda =9}\left(f_{\mathbf{X}}\right)\cup \{[6,7],[7,8]\}\hfill \\ \hfill L{S}_{\lambda =11}\left(f_{\mathbf{X}}\right)& =L{S}_{\lambda =10}\left(f_{\mathbf{X}}\right)\hfill \\ \hfill L{S}_{\lambda =12}\left(f_{\mathbf{X}}\right)& =L{S}_{\lambda =11}\left(f_{\mathbf{X}}\right)\hfill \\ \hfill L{S}_{\lambda =13}\left(f_{\mathbf{X}}\right)& =L{S}_{\lambda =11}\left(f_{\mathbf{X}}\right)\hfill \\ \hfill L{S}_{\lambda =14}\left(f_{\mathbf{X}}\right)& =L{S}_{\lambda =11}\left(f_{\mathbf{X}}\right)\cup \left\{[0,1]\right\}.\hfill \end{array}$$

This allows us to compute the available features for each λ-value:

	$\lambda =7$	$\lambda =8$	$\lambda =9$	$\lambda =10$	$\lambda =11$	$\lambda =12$	$\lambda =13$	$\lambda =14$
$\mathfrak{F}\left(L{S}_{\lambda }\left(f_{\mathbf{X}}\right)\right)$	∅	$[5,6]$	$[2,6]$	$[2,8]$	$[2,8]$	$[2,8]$	$[2,8]$	$[0,8]$

Let $\mathfrak{F}\left(f_{\mathbf{X}}\right)$ be the whole set of features for $f_{\mathbf{X}}$, that is,

$$\mathfrak{F}\left(f_{\mathbf{X}}\right)=\left\{\mathbb{I}:\mathbb{I}\in \mathfrak{F}\left(L{S}_{\lambda }\left(f_{\mathbf{X}}\right)\right)\mathrm{for}\mathrm{some}{\lambda }_{min}\le \lambda \le {\lambda }_{max}\right\}.$$

Example 2.

From Example 1, we obtain

$$\mathfrak{F}\left(f_{\mathbf{X}}\right)=\{[5,6],[2,6],[2,8],[0,8]\}.$$

We can represent the map $\lambda \mapsto L{S}_{\lambda }\left(f_{\mathbf{X}}\right)$ from $[{\lambda }_{min},{\lambda }_{max}]$ to $\mathfrak{F}\left(f_{\mathbf{X}}\right)$ as shown in Figure 7.

Let $\mathbb{I}\in \mathfrak{F}\left(f_{\mathbf{X}}\right)$; in order to quantify the persistence of this particular feature for the map $f_{\mathbf{X}}$, we use the map $\lambda \mapsto L{S}_{\lambda }\left(f_{\mathbf{X}}\right)$ from $[{\lambda }_{min},{\lambda }_{max}]$ to $\mathfrak{F}\left(f_{\mathbf{X}}\right)$. To this end, we introduce the following definition: the birth point of the feature $\mathbb{I}$ is defined by

$$a\left(\mathbb{I}\right)=inf\left\{\lambda :\mathbb{I}\in \mathfrak{F}\left(L{S}_{\lambda }\left(f_{\mathbf{X}}\right)\right)\right\}$$

and the corresponding death point by

$$b\left(\mathbb{I}\right)=sup\left\{\lambda :\mathbb{I}\in \mathfrak{F}\left(L{S}_{\lambda }\left(f_{\mathbf{X}}\right)\right)\right\}.$$

In particular, we note that $a\left([0,N-1]\right)={\lambda }_{max}$ (see Figure 7). Since $a\left(\mathbb{I}\right)\le b\left(\mathbb{I}\right)<\infty$ holds for all $\mathbb{I}\in \mathfrak{F}\left(f_{\mathbf{X}}\right),$$\mathbb{I}\ne [0,N-1],$ we call the finite interval $\left[a\right(\mathbb{I}),b(\mathbb{I}\left)\right]$ the barcode of the feature $\mathbb{I}\in \mathfrak{F}\left(f_{\mathbf{X}}\right)\backslash \left\{[0,N-1]\right\}$.

Example 3.

From Example 1 we consider the features $[5,6]\in L{S}_{\lambda =8}\left(f_{\mathbf{X}}\right)$, $[2,6]\in L{S}_{\lambda =9}\left(f_{\mathbf{X}}\right)$, and $[2,8]\in L{S}_{\lambda =10}\left(f_{\mathbf{X}}\right)$. Then, the feature $[5,6]$ has its birth point at $a\left(\right[5,6\left]\right)=8$ and its death point at $b\left(\right[5,6\left]\right)=9$; the feature $[2,6]$ has its birth point at $a\left(\right[2,6\left]\right)=9$ and its death point at $b\left(\right[2,6\left]\right)=10$. Finally, the feature $[2,8]$ has its birth point at $a\left(\right[2,8\left]\right)=10$ and its death point at $b\left(\right[2,8\left]\right)=14$. As a consequence, the set

$$\mathcal{B}\left(f_{\mathbf{X}}\right):=\{([5,6];8,9),([2,6];9,10),([2,8];10,14)\}$$

of features and its corresponding barcodes contain the relevant information of the shape of $f_{\mathbf{X}}$ (see Figure 7).

Thus, we define the set of barcodes for $f_{\mathbf{X}}$ by

$$\mathcal{B}\left(f_{\mathbf{X}}\right)=\left\{(\mathbb{I};a\left(\mathbb{I}\right),b\left(\mathbb{I}\right)):\mathbb{I}\in \mathfrak{F}\left(f_{\mathbf{X}}\right)\backslash \left\{[0,N-1]\right\}\right\}$$

and its persistence diagram as

$$\mathcal{PD}\left(f_{\mathbf{X}}\right):=\left\{(a\left(\mathbb{I}\right),b\left(\mathbb{I}\right))\in {\mathbb{R}}^2:\mathbb{I}\in \mathfrak{F}\left(f_{\mathbf{X}}\right)\backslash \left\{[0,N-1]\right\}\right\}$$

(see Figure 8). An equivalent representation of the persistence diagram is the life-time diagram for $f_{\mathbf{X}}$, which is constructed by means of a bijective transformation $T(a,b)=(a,b-a)$, acting over $\mathcal{PD}\left(f_{\mathbf{X}}\right)$, that is,

$$\mathcal{LT}\left(f_{\mathbf{X}}\right):=\left\{(a\left(\mathbb{I}\right),b\left(\mathbb{I}\right)-a\left(\mathbb{I}\right))\in {\mathbb{R}}^2:\mathbb{I}\in \mathfrak{F}\left(f_{\mathbf{X}}\right))\backslash \left\{[0,N-1]\right\}\right\};$$

see Figure 9.

In order to determine the grade of similarity between two barcodes from two different time series, we need to set a similarity metric. To this end, we construct the persistent image for $f_{\mathbf{X}}$ as follows: we observe that $\mathcal{LT}\left(f_{\mathbf{X}}\right)$ is a finite set of points, namely,

$$\mathcal{LT}\left(f_{\mathbf{X}}\right)=\{(a_1,b_1-a_1),\dots ,(a_k,b_k-a_k)\}$$

for some natural numbers $k\ge 1$ and such that $b_1-a_1\le b_2-a_2\dots \le b_k-a_k$. Then, we consider a non-negative weighting function $w:\mathcal{LT}\left(f_{\mathbf{X}}\right)⟶[0,1]$ given by

$$w(a_i,b_i-a_i)=\frac{b_i-a_i}{b_k-a_k}\mathrm{for}1\le i\le k$$

.

Finally, we fix M, a natural number, and take a bivariate normal distribution $g_u(x,y)$ centered at each point $u\in \mathcal{LT}\left(f_{\mathbf{X}}\right)$ and with its variance $\sigma id=\frac{1}{M}{max}_{1\le i\le k}(b_i-a_i)id,$ where $id$ is the $2\times 2$ identity matrix. A persistence kernel is then defined by means of a function ${\rho }_{\mathbf{X}}:{\mathbb{R}}^2\to \mathbb{R}$, where

$${\rho }_{\mathbf{X}}(x,y)=\sum _{u\in \mathcal{LT}\left(f_{\mathbf{X}}\right)}w\left(u\right)g_u(x,y).$$

(1)

We associate with each $\mathbf{X}\in \mathbb{R}$ a matrix in ${\mathbb{R}}^{M\times M}$ as follows: let $\epsilon >0$ be a non-negative real number that is sufficiently small, and then consider a square region ${\Omega }_{\mathbf{X},\epsilon }=[\alpha ,\beta ]\times [{\alpha }^{*},{\beta }^{*}]\subset {\mathbb{R}}^2$, covering the support of ${\rho }_{\mathbf{X}}(x,y)$ (up to a certain precision), such that

$${\int \int }_{{\Omega }_{\mathbf{X},\epsilon }}{\rho }_{\mathbf{X}}(x,y)dxdy\ge 1-\epsilon$$

holds. Next, we consider two equispaced partitions of the intervals

$$\alpha =p_0\le p_1\dots \le p_M=\beta \mathrm{and}{\alpha }^{*}=p_0^{*}\le p_1^{*}\dots \le p_M^{*}={\beta }^{*}.$$

Now, we put

$${\Omega }_{\mathbf{X},\epsilon }=\bigcup _{i=0}^{M-1}\bigcup _{j=0}^{M-1}[p_i,p_{i+1}]\times [p_j^{*},p_{j+1}^{*}]=\bigcup _{i=0}^{M-1}\bigcup _{j=0}^{M-1}{P}_{i,j}$$

The persistence image of $\mathbf{X}$ associated with the partition $\mathcal{P}=\left\{P_{i,j}\right\}$ is then described by the matrix given by the following equation:

$$PI(\mathbf{X},M,\mathcal{P},\epsilon )={\left({\int \int }_{P_{i,j}}{\rho }_{\mathbf{X}}(x,y)dxdy\right)}_{i=0,j=0}^{i=M-1,j=M-1}\in {\mathbb{R}}^{M\times M}.$$

(2)

2.5. Classification

Image classification is a procedure that is used to automatically categorize images into classes by assigning to each image a label representative of its class. A supervised classification algorithm requires a training sample for each class, that is, a collection of data points whose class of interest is known. Labels are assigned to each class of interest. The classification problem applied to a new observation is thus based on how close a new point is to each training sample. The Euclidean distance is the most common distance metric used in low-dimensional datasets. The training samples are representative of the known classes of interest to the analyst. In order to classify the persistence diagrams, we can use any state-of-the-art technique. In our case, we considered the random forest classification.

Recall that we conducted 9 different experiments, with 24 samples associated with each one of them corresponding to 3 samples for each of the different experimental conditions: relaxed rigid driver, relaxed rigid passenger, relaxed SAV driver, relaxed SAV passenger, tense rigid driver, tense rigid passenger, tense SAV driver, and tense SAV passenger. Their respective labels are $\{0,0,0,0,1,1,1,1\}$. Therefore, we designed the following training validation process: The model is trained over 144 samples and evaluated over the remaining unseen 72 experiments (two-to-one training-to-testing ratio). The split between training and sampling is achieved using random shuffling and stratification to ensure balance between the classes. In order to improve the evaluation of the model generalizability, we also performed a cross-validation procedure following a leave-one-out strategy, consisting of iteratively training over the full dataset except one sample that was left out and used to test and score the model. We used the accuracy metric to evaluate the classification model. We can represent the performance of the model using the so-called confusion matrix: a 2D entries table where elements account for the number of samples in each category, with the first axis representing the true labels and the second axis the predicted labels. We also computed the different classification metrics to obtain a more detailed reporting of the model performances.

3. Results

The trained random forest classifier model for the persistence images has a notably high accuracy score on the training dataset (144) for both approaches and high accuracy for the testing dataset (72 samples). This suggests strong differentiation of the images with the respect to their generating signals, see Figure 10. The scores on the training and testing are 93 and 83%, respectively. The leave-one-out cross-validation achieved a score of 81%, indicating a good variance–bias trade-off and good generalization potential of the model.

4. Discussion

The combination of Morse theory and topological data analysis allows us to extract information from real data without the need for smoothness or regularity assumption on the time series. In our case, input data for each experiment were reduced from six-sensor time series of measurements to one single image containing the persistent pattern for attention to the road. Using the obtained persistence images as the new inputs, supervised learning proved to successfully predict the attention state of the driver or passenger.

The procedure used and described in this paper does not involve any additional pre-processing of the sensor data; is robust to noise and degraded signals; and supports large quantities of data, which makes it efficient and scalable.

It is important to highlight the fact that while the proposed methodology based on the TDA (successfully applied in large datasets [9]) seems general and powerful and it was able to extract the main data features, the validity of the driver behaviors observed in the analyzed dataset should be carefully checked due to the overly reduced dataset employed (limited to nine individuals) that does not allow for the full validation of prediction robustness.