Deep LSTM with Dynamic Time Warping Processing Framework: A Novel Advanced Algorithm with Biosensor System for an Efficient Car-Driver Recognition

Abstract

The latest generation of cars are increasingly equipped with driver assistance systems called ADAS (advanced driver assistance systems) which are able to assist the car driver in different driving scenarios, and in the most advanced automation levels, able to take over driving the car if required due to dangerous situations. Therefore, it is essential to adapt the ADAS specifically to the car-driver’s identity in order to better customize the driving assistance. To this end, algorithms that allow correct recognition of the vehicle driver are fundamental and preparatory. In this context, an algorithm for car-driver identity recognition is proposed which allows, with an accuracy close to 99%, recognition of the driver by means of a properly designed pipeline based on the analysis of the car driver PhotoPlethysmoGraphic (PPG) signal. The proposed approach makes use of deep long short-term memory (LSTM) architecture for learning such PPG signal features needed to discriminate one car driver from another. The extended validation and testing of the proposed system confirm the reliability of the proposed pipeline.

1. Introduction

The automotive industry is constantly evolving to ensure reliability, durability, and safety for the car driver. The technology applied to the automotive market is also growing and evolving with increasingly advanced products designed to assist the drivers in preventing accidents [1]. Automotive design is turning to advanced driver assistance systems (ADAS), to address safety issues while driving. These ADAS inform the driver in case of danger related to reduced attention [1,2,3]. For this reason, further efforts are being made in the latest generation of cars to make the ADAS more efficient by trying to make them more and more adaptable to the driving needs of the individual driver [4]. In fact, each car driver has their own driving dynamics, level of perception, reflexes, readiness, and driving experience which, obviously, must be taken into consideration by the ADAS which are suitable to assist the driver. To this end, it is particularly useful to continuously identify the user while driving in order to apply the most suitable ADAS assistance setup [4]. There are several approaches in the literature for car-driver identity recognition.

In [5] the authors proposed using different Android smartphone sensors and classification algorithms in order to profile driver behavior. Generally, user profiling is a task that includes a preliminary identification of the profiled subject. Moreover, in this survey, the authors analyzed several methods based on machine learning approaches, i.e., artificial neural networks (ANN), support vector machines (SVM), random forest (RF), and Bayesian network (BN) [5]. They confirmed that the gyroscope and the accelerometer are the best sensors to detect the driving behavior (although they confirmed that as general rule, using all sensor axes perform better than using a single one), whereas in terms of machine learning architecture, RF is by far the best performing approach, followed by ANN even if the performance of both is satisfactory and equivalent, varying from 0.980 to 0.999 mean Area Under the ROC Curve (AUC) values [5]. In [6] the authors proposed a logging approach for car-driver profiling. By means of a simulator, the authors proposed a driver monitoring based on the analysis of the key-pressed dynamic during the simulation. They were able to identify such patterns for discriminating one driver with respect to others with good accuracy. In [7] the authors proposed a system named the Driver Adaptive Vehicle Interaction System which was composed of the following three modules: the profile management module, the driver management module, and the interaction management module. The profile management module manages the profile of each car driver through logging the driver’s personality, characteristics, behavior, and status information. On the basis of this profiling data, a custom user-adaptive interaction system is provided for each driver. In [8] the authors analyzed the main issues inherent in the task of understanding and modeling of human driving behavior, including an interesting discussion about the main human factors which have an impact on driving, that is, personality, age, anger, gender, distraction, mental stress, etc. The authors analyzed several pipelines which attempted to address the problems related to the aforementioned human factors with reference to the driver profiling. More details are available in [8]. In [9] the authors proposed an approach based on the use of smartphone’s sensors to profile the car driver. They proposed a pipeline named SenseFleet which was able to analyze the output of smartphone sensors while driving. This method performs well but as for previous mentioned approaches, requires the use of an external device to the driver and to the car (smartphone) and which in any case must be supplied to the user and on which obviously hypotheses cannot be made in relation to the physical, electrical, and dynamic characteristics of the sensors. In [10], the authors proposed a car-driver profiling algorithm based on multistate car following modeling, where a GMM (Gaussian mixture model) is used to model the internal stochasticity of an individual driver using a mixture of Gaussian kernels. The testing data reported by the authors confirmed that their approach was robust [10]. In this context, a novel approach for car-driver identity recognition based on the analysis of the so-called “cardiovascular imprint” is proposed. In this paper, cardiovascular imprint refers to a set of physiological features extracted from the PhotoPlethysmoGraphic (PPG) signal of the driver [11]. Basically, the proposed algorithm is assessed at Level II of the standard of autonomous driving [12] since driving control remains with the car driver while control of the engine, speed (current and maximum allowed), and other automotive services support are customized according to the driver identity recognized by the method herein described.

The contribution herein described is organized as follow: First, the implemented hardware framework for sampling the car-driver PPG signal is introduced. Next, the proposed PPG signal processing pipeline as well as the designed deep learning architecture suitable to recognize the driver identity from PPG data is provided. At the end of this manuscript, experimental results and future works are reported.

2. Materials and Methods

As introduced in the previous section, the proposed identity recognition pipeline is based on adaptive and efficient processing of the driver’s PPG signal. The PPG signal is considered to be a non-invasive physiological trace of the cardiovascular system, in particular, the heart pulse rate. Both heart pulse and respiratory rate of a subject can be monitored by means of ad hoc analysis of such PPG signal features [11]. Through the PPG signal, a non-invasive measurement of the blood volume dynamic can be done for several medical assessments [13,14]. A standard PPG waveform includes a pulsatile (“AC”) physiological waveform which is correlated to cardiac-synchronous changes in the blood volume superimposed with a slowly varying (“DC”) component containing lower frequency information correlated to respiration, thermoregulation, and skin tissues (in which the PPG sensor is adapted) [11]. For each cardiac cycle, the heart pumps blood to the periphery with a specific pressure (systolic phase) which is enough to stretch the arteries and arterioles in the subcutaneous tissue. By means of a sensing device composed of a light emitter and a detector attached over the skin, a pressure pulse can also be seen from the venous plexus, as a small secondary peak. The change in volume caused by the heart pressure pulse can be detected by illuminating the skin, and then by measuring the amount of light either transmitted or reflected to a detector. More detail about the PPG patter formation can be found in [11]. Figure 1 shows a compliant PPG waveform. As shown in Figure 1, the formation of the single waveform PPG is closely related to the cardiac activity of the subject as the features of the waveform are closely correlated to the cardiac phases, i.e., systole, diastole, and dicrotic notch [11]. For this reason, the features extracted from the PPG signal were considered to be a cardiovascular imprint of the subject.

The following Figure 2 shows the proposed PPG signal sensing device to be used for the car drivers identity recognition.

The implemented sensing device was comprised of the following two main components: a silicon photomultipliers (SiPMs) detector and two LED photon emitters. The SiPM detector used in our system was produced by STMicroelectronics (Catania, Italy) [11,12,13,14,15] and featured a total area of 4.0 × 4.5 mm² and 4871 square microcells with 60 µm of pitch. It had a geometrical fill factor of 67.4% and was packaged in a surface mount housing (SMD) of 5.1 × 5.1 mm² total area [15,16,17]. A Pixelteq dichroic bandpass filter (Bryan Dairy Rd, Largo, FL, USA) centered at 542 nm (full width at half maximum (FWHM) of 70 nm, and optical transmission higher than 90% in the pass band range) was glued on the SMD package using 352TM adhesive (Loctite^®, Milan, Italy). In this setup, in the driving range 0–3 V, the device had a maximum photon detection efficiency (PDE) of approximately 29.4% at 565 nm and approximately 27.4% at 540 nm (central wavelength in the filter pass band). Furthermore, a dichroic filter for reducing the absorption of environmental light by more than 60% in the linear operation range was included. As emitter, we proposed the use of two LT M673 LEDs (OSRAM, Milan, Italy) based on InGaN technology (in SMD package) emitting at 873 nm and used as optical light sources [11,12,13,14,15,16,17]. The LEDs had an area of 2.3 × 1.5 mm², a viewing angle of 120°, and typical power emission of a few mW in the standard operation range [15,16,17].

There are more details provided in [11,12,13,14,15,16,17,18]. Figure 3 depicts some electrical features of the proposed SiPM detector.

Figure 4 shows the general outline of the whole proposed car-driver identity recognition pipeline.

The subsections that follow provide a description of each block of the proposed pipeline, as reported in Figure 4.

2.1. The PPG Preprocessing Block

The aforementioned PPG sensing device was placed in the car steering in order to continuously sample the PPG signal of the driver both when getting behind the wheel and while driving in order to collect the so-called cardiovascular imprint of the vehicle driver. As soon as the driver gripped the steering and started the car, the PPG signal was immediately sampled from the palm of the driver’s hand on the steering wheel, allowing the driver to be identified by means of the pipeline (described below) having the capability to extract and analyze specific features of the collected PPG time series. Figure 5 shows the described hardware setup:

As shown in Figure 5, the PPG sensing devices were positioned at different points on the steering, statistically the most significant points, so that once stimulated by the driver who placed his hands above the devices (in the steering), the PPG signal was produced in raw format. Specifically, we distributed six PPG sensors on the car steering, equidistant from each other, in order to cover the most significant parts of the steering (i.e., those in which the main driving positions of a common driver are concentrated statistically), and therefore increase the probability that the driver would place a hand on the sensing devices. If the driver did not have either hand over the PPG sensors arranged on the steering, the last identification setup was maintained. Anyway, the first acquisition of the car driver PPG signal was carried out when the car was switched on, since the described sensing devices were placed both on the start button of recently produced cars and on the manual and automatic gear shift. These parts of the car were used by the car driver, providing enough samples to generate preliminary identity recognition. The sensing devices, as described above, were tested with a sampling frequency of 1 Khz. On average, the activities of starting the car and engaging the gear shift, cumulatively, took a few seconds (3 or 4 s), and therefore were sufficient to provide a quantity of samples of the driver’s PPG signal in order to allow recognition of the driver’s identity.

However, numerous profile changes were not made while driving, therefore the problem of missed PPG signal was not particularly felt in this application. Nevertheless, a recovery system was provided (based on vision data) for this issue, which is partially introduced in the Conclusions section and future works.

This phase was managed through a hardware setup in which an ADC with 24-bit precision of the National Instrument was used. More details on the hardware setup can be found in [1]. Figure 6 shows the details of the implemented motherboard which was integrated into the car in order to collect the raw signal from the sensing devices placed on the car’s steering wheel. The PPG sensing devices (as reported in Figure 2) placed on the car steering were plugged into the USB connectors of the developed motherboard reported in Figure 6 (which were part of the main car control unit).

The properly filtered PPG signals, coming from each of the sensing devices placed on the steering and activated by the driver’s hand, were monitored using LabView software. More details on the LabView pipeline designed to visually check the dynamics of each steering PPG sensor can be found in [19].

The collected raw PPG time series coming from the PPG sensing devices, were further post-processed by means of ad hoc dynamic time warping (DTW) processing (made by the DTW block) followed by deep learning blocks, respectively.

2.2. The DTW Processing Block

As described in the previous section, in this block, a set of PPG signals sampled from the car driver’s hand were received. Considering that there were more sensors placed on the car steering and therefore we could receive more valid PPG signals, the first operation carried out in this block was to validate the time series coming from each PPG sensing channel, i.e., the compliance with the PPG standard. To cover this aspect, a specific approach called PPG-Pattern Recognition System (PPG-PRS) patented by the author was proposed. The mentioned PPG-PRS pipeline has been described in a previous article [11]. Specifically, through this pipeline, it was possible to validate a compliant PPG signal, as well as stabilize the signal by eliminating noise components due to electronic components, movement artifacts, and so on. More details about PPG-PRS are available in [11].

At this point, the PPG signals coming from the various sensing devices placed on the steering were suitably filtered and stabilized, and ready to be used to characterize the driver’s cardiovascular imprint. Next, a further preprocessing of the collected PPG time series was proposed using dynamic time warping (DTW) techniques [19,20] with respect to a PPG reference signal generated by ad hoc reaction-diffusion model (RD-M) [11] which was able to generate fully compliant PPG patterns. It is well known that two time-varying signals can have similar features but appear very different because the temporal distribution of the features is different in each signal. The DTW technique made it possibe to create distortions in the temporal distribution of the features of a signal with respect to one considered reference until a distribution that maximized the temporal similarity (defined by specific metric) of the two signals, was found [20]. Formally, given two K-dimensional signals ($\zeta ,\vartheta )$:

$$\zeta =\left[\begin{array}{ccc}{\zeta }_{1,1}& \dots & {\zeta }_{1,M}\\ \dots & \dots & \dots \\ {\zeta }_{K,1}& \dots & {\zeta }_{K,M}\end{array}\right]$$

(1)

$$\vartheta =\left[\begin{array}{ccc}{\vartheta }_{1,1}& \dots & {\vartheta }_{1,N}\\ \dots & \dots & \dots \\ {\vartheta }_{K,1}& \dots & {\vartheta }_{K,N}\end{array}\right]$$

(2)

If we define with d($\zeta ,\vartheta$) an appropriate metric that represents the distance between the M-th samples of $\zeta$ as compared with the N-th samples of $\vartheta$, through the DTW method, we process the vectors $\zeta$ and $\vartheta$ onto a common set of instants such that a global signal-to-signal distance measure is the smallest. Basically, we are interested in searching a path through the lattice structure, parameterized by two sequences of the same length, i_x and i_y, such that the following distance (see Equation (3)) is minimum.

$$d\left(\zeta ,\vartheta \right)={\displaystyle \sum }_{\begin{array}{c}m ϵ i_x\\ n ϵ i_y\end{array}}{d}_{mn}\left(\zeta ,\vartheta \right)$$

(3)

Therefore, to characterize the level of similarity of two signals or patterns using the DTW technique, it is necessary to define a suitable metric. Among the possible adoptable, the symmetric Kullback–Leibler (SKL) metric (see Equation (4)) was proposed because, in our tests, it showed better performance than the classic Euclidean or absolute. More details about DTW and SKL metric are available in [19,20].

$$d_{mn}^{SKL}\left(\zeta ,\vartheta \right)={\displaystyle \sum }_{k=1}^K({\zeta }_{k,m}-{\vartheta }_{k,n})(\log {\zeta }_{k,m}-\log {\zeta }_{k,m})$$

(4)

The following is a brief recall of the hyper-filtering method proposed in [1]. If we set with x(k) the acquired raw PPG signal from a biosensing device placed on the car steering, for each frequency setup we obtain the following hyper-filtered time series:

$${\phi }_{H_{LP}}^i\left(k\right)={\mathsf{ℱ}}_{Butterworth}\left(f_L^i ,f_H ,x\left(k\right)\right) i=1,2,\dots 11 ;k=1,2,\dots .n$$

(5)

$${\phi }_{H_{HP}}^i\left(k\right)={\mathsf{ℱ}}_{Butterworth}\left(f_L ,f_{H }^i,x\left(k\right)\right) i=1,2,\dots 11 ;k=1,2,\dots .n$$

(6)

where ${\phi }_{H_{HP}}^i\left(k\right)$, and ${\phi }_{H_{LP}}^i\left(k\right)$ represent the set of hyper-filtered time series coming from the hyper-filtered high-pass and low-pass layers, respectively. The parameter “n” represents the length of the PPG signal. The function ${\mathsf{ℱ}}_{Butterworth}$ represents the filter processing performed by the configured Butterworth filter, while $f_L,f_H$ represent the fixed cutoff frequencies and $f_{H ,}^i{f}_L^i$ the variables ones as per

Table 1. Hyper low-pass filtering setup (in Hz).

Filter	Fc-pass-1	Fc-pass-2	Fc-pass-3	Fc-pass-4	Fc-pass-5	Fc-pass-6	Fc-pass-7	Fc-pass-8	Fc-pass-9	Fc-pass-10	Fc-pass-11
High pass	0.5	/	/	/	/	/	/	/	/	/	/
Low pass	1.15	1.21	2.30	2.65	3.14	3.24	4.25	4.56	5.80	5.95	6.50

,

Table 2. Hyper high-pass filtering setup (in Hz).

Filter	Fc-pass-1	Fc-pass-2	Fc-pass-3	Fc-pass-4	Fc-pass-5	Fc-pass-6	Fc-pass-7	Fc-pass-8	Fc-pass-9	Fc-pass-10	Fc-pass-11
High pass	0.55	1.4	2.7	2.91	3.75	3.90	4.4	4.85	5.75	5.65	6.90
Low pass	7	/	/	/	/	/	/	/	/	/	/

. From this hyper-filtered time series, the so-called signal-patterns (which shows how each PPG sample varies in the time according to the type of filtering) are generated. The following Equations (7) and (8) show how we obtain the signal patterns ${\zeta }_{HP}^k\left(s\left(t_k\right)\right)$ and ${\zeta }_{LP}^k\left(s\left(t_k\right)\right)$ for each sample $s\left(t_k\right)$ of the acquired PPG waveforms:

$${\zeta }_{HP}^k\left(s\left(t_k\right)\right)=\left[{\phi }_{H_{HP}}^1\left(t_k\right),{\phi }_{H_{HP}}^2\left(t_k\right),\dots {\phi }_{H_{HP}}^{11}\left(t_k\right)\right] k=1,2,\dots .n$$

(7)

$${\zeta }_{LP}^k\left(s\left(t_k\right)\right)=\left[{\phi }_{H_{LP}}^1\left(t_k\right),{\phi }_{H_{LP}}^2\left(t_k\right),\dots {\phi }_{H_{LP}}^{11}\left(t_k\right)\right] k=1,2,\dots .n$$

(8)

Therefore, the proposed system applies the DTW technique to the various pattern time series generated by the hyper-filtering technique applied to each PPG time series coming from the car steering and already illustrated by the author in the contribution reported in [1]. Basically, through a multi-frequency filtering of the PPG signal, an ad hoc analysis was proposed related to the modality with which each PPG sample varies over time as a function of the cutoff frequency of the applied filter. Considering that the hyper-filtering technique requires the setup of a certain number of cutoff frequencies both for high-pass filters and for low-pass filters, below in Table 1 and Table 2, the set of used cutoff frequencies are reported.

At this point, ad hoc further mathematical processing is needed to proceed with the characterization of the so-called cardiovascular imprint of the car driver. Specifically, the hyper-filtering technique is also applied to the PPG time series generated by the reaction-diffusion model (RD-M) which, as previously mentioned, is capable of generating a classic compliant PPG signal. The herein proposed RD-M is reported in the following Equation (9):

$$\begin{array}{c}\begin{array}{c}\frac{\partial x_1}{\partial t}=-x_1+\left(1+\mu \right)y_1-\beta y_2+{\rho }_1\\ \frac{\partial x_2}{\partial t}=-x_2+\left(1+\mu \right)y_2+\beta y_1+{\rho }_2\end{array}\\ y_j=\frac{1}{2}\left(\left|x_j+1\right|-\left|x_j-1\right|\right) ; j=1,2\\ \mu =0.5; {\rho }_1=-0.3; {\rho }_2=0.3; \beta =1; x_1\left(0\right)=0.1; x_2\left(0\right)=0.08\end{array}$$

(9)

where the variables x₁ and x₂, respectively, represent the reactive-diffusive dynamics which, in this case, have been associated with the two phases of the cardiac cycle, namely diastole (reaction) and systole (diffusion). Figure 7 shows the classic and compliant PPG time series generated by the variable x₂ (named as PPG_ref (k)) which is taken into account as the PPG reference signal [11,12,13,14,15,16,17].

The hyper-filtering pipeline is also applied to this PPG signal generated by the RD-M in order to generate signal patterns according to the filter setup used, and shown in Table 1 and Table 2. The following mathematical formalism indicates the signal patterns produced by the hyper-filtering analysis applied to the referring PPG signal (with sample s_r(t_k)) generated by the RD model:

$${\beta }_{HP}^k\left(s_r\left(t_k\right)\right)=\left[{\eta }_{H_{HP}}^1\left(t_k\right),{\eta }_{H_{HP}}^2\left(t_k\right),\dots \phi {\eta }_{H_{HP}}^{11}\left(t_k\right)\right] k=1,2,\dots .n$$

(10)

$${\beta }_{LP}^k\left(s_r\left(t_k\right)\right)=\left[{\eta }_{H_{LP}}^1\left(t_k\right),{\eta }_{H_{LP}}^2\left(t_k\right),\dots {\eta }_{H_{LP}}^{11}\left(t_k\right)\right] k=1,2,\dots .n$$

(11)

$${\eta }_{H_{LP}}^i\left(k\right)={\mathsf{ℱ}}_{Butterworth}\left(f_L^i ,f_H ,PP{G}_{ref}\left(k\right)\right) i=1,2,\dots 11 ;k=1,2,\dots .n$$

(12)

$${\eta }_{H_{HP}}^i\left(k\right)={\mathsf{ℱ}}_{Butterworth}\left(f_L ,f_{H }^i,PP{G}_{ref}\left(k\right)\right) i=1,2,\dots 11 ;k=1,2,\dots .n$$

(13)

Now, for each of the signal patterns $\left[{\zeta }_{HP}^k\left(s\left(t_k\right)\right),{\zeta }_{LP}^k\left(s\left(t_k\right)\right)\right]$ generated by the PPG signals coming from the car steering, the proposed approach includes the application of the DTW method with the SKL metric looking for the time distortion that minimizes the distance between the individual signal patterns generated by the PPG of the car driver and the signal patterns generated by the reference PPG signal $\left[{\beta }_{HP}^k\left(s_r\left(t_k\right)\right),{\beta }_{LP}^k\left(s_r\left(t_k\right)\right)\right]$. For each of the signal patterns generated by each of the PPG signals coming from the steering, a similarity value ${\delta }^k(.)$, therefore, is generated which represents the minimum of the SKL metric in the DTW procedure. Formally, these ${\delta }^k(.)$ values can be modelized as follows:

$$\begin{gathered} {\delta }_{xy}^k\left(k\right)=d_{mn}^{SKL}\left({\zeta }_x^k\left(s\left(t_k\right)\right),{\beta }_y^k\left(s_r\left(t_k\right)\right)\right)k=1,2...n; x=\left(HP,LP\right); y=\left(HP,LP\right) \\ \forall n,m \in {\mathsf{\mathbb{N}}}^{+} \end{gathered}$$

(14)

Figure 8 shows an example of the DTW processing through the SKL metric between a signal patterns of the car-driver PPG signal (coming from biosensing device on the steering) and the corresponding signal pattern generated from the reference RD-M:

The collected setup of similarity values ${\delta }_{xy}^k\left(k\right)$ are used as input for the deep long short-term memory (LSTM) model [20,21,22,23,24] described in the next subsection.

2.3. The Deep LSTM Processing Block

As described in the previous section, a set of similarity values ${\delta }_{xy}^k\left(k\right)$ are received as input from a deep learning classifier. The proposed deep learning pipeline is depicted in Figure 9.

The LSTM vanilla architecture was originally proposed by Hochreiter and Schmidhuber [22] for preventing the vanishing gradient problem which affects recurrent neural network (RNN). The LSTM cell (see Figure 9 detail) is able to select what information to discard or store. In order to produce effective results in real applications, this selective method requires three different mechanisms to read, store, and discard information by taking advantage of specific vectors called “gates” [23,24]. The mathematical model of the used LSTM cell is the following:

$$f_t=\sigma (W_f·\left[h_{t-1}, x_t\right]+b_f$$

(15)

$$i_t=\sigma (W_i·\left[h_{t-1}, x_t\right]+b_i$$

(16)

$${\tilde{C}}_t=\tanh (W_C·\left[h_{t-1}, x_t\right]+b_{C })$$

(17)

$$C_t=f_t\ast C_{t-1}+i_t\ast {\tilde{C}}_t$$

(18)

$$o_t=\sigma (W_o·\left[h_{t-1}, x_t\right]+b_o$$

(19)

$$h_t=o_t\ast \tan \mathrm{h}(C_t)$$

(20)

A fully connected layer with SoftMax and classification layer has been inserted in order to output the driver’s class identity. In the next section the testing and validation scenario of the described pipeline is reported.

3. Results

In order to test and validate the proposed pipeline, several PPG measurements of several subjects in different driving scenarios were collected. A team of physiologists supported the collection of this dataset. For each driving scenario, the physiologists stimulated different physiological states of the driver (careless, excited, stressed, etc.) in order to have a complete mapping of the PPG signal of the car driver. During the sampling phase of the PPG signal, the various induced physiological states were confirmed and monitored by means of the electroencephalographic (EEG) signal which was sampled simultaneously with the PPG signal.

More in detail, the performed experiments were conducted by generating various physiological levels by means of exercises suggested by physiologists who simultaneously examined the subject’s EEG signal path. In relation to physiological states, the physicians suggested the recruited subjects perform simple math exercises (additions, multiplications, etc.) or simple analytical problems and quizzes that they themselves had to read from the texts that were submitted. In this way, certain areas of the brain and the autonomic nervous system were properly stimulated. The protocol for this dataset setup is suitable to characterize such physiological states as confirmed by the EEG tracing which shown the presence of beta waves (aperiodic signal in the range 14–30 Hz) typically shown in attentive subjects, and intense brain activity or gamma waves in the range (28–40 Hz) generally shown in subjects with strong states of tension. Instead, alpha waves, in the range (8–15 Hz) typically associated with subjects in a state of relaxation and a decrease in neuronal activity, were, therefore, induced in the same subject, i.e., with a prevalence of the parasympathetic system. The latter physiological states were induced by closing the eyes, listening to slow music, etc. In this way, for each analyzed physiological state, the corresponding morphological and dynamic characteristics of the sampled PPG signal that was gradually acquired, was collected [3]. In order to test the proposed pipeline, it was necessary to sample physiological signals (PPG) from various subjects, therefore, authorization was required and requested from all subjects. The experiments were carried out on 75 subjects of both sexes (35 men and 40 women), aged between 20 and 79 years (specifically, 40% of the dataset in the range 20–45 years, 42% in the range 46–65 years, and 18% in the range 66–79 years). Of the 75 recruited subjects, 30 were smokers while 20 subjects suffered from moderate hypertension (and took specific drugs to keep blood pressure under control). All the recruited subjects had eaten before the acquisition session. None of the subjects had taken alcohol or other drugs that altered the physiological state. The same psychological tests indicated in the previous paragraph were presented to all 75 subjects recruited. None of the recruited volunteers were using drugs capable of changing cortical excitability. Volunteers gave informed consent to the procedures approved by the Ethical Committee Catania 1 (authorization no. 113/2018 PO), which were conducted in accordance with the Declaration of Helsinki. Participation criteria included the possession of a valid driving license for motor vehicles. We collected 10 min of PPG with biosensing devices as described in previous sections and based on coupled LED-SiPM technology with a sampling frequency of 1 Khz. The car steering was equipped with six LED-SiPM coupled sensing devices arranged equidistant from each other in order to cover the most common driving styles of a subject. For the software and hardware framework used to host the proposed PPG post processing pipeline, the following setup was used: MATLAB full toolboxes vers. 2019b running in a server having an Intel 16-Cores and NVIDIA GeForce RTX 2080 GPU. The DTW processing was covered by using the function “dtw” supported by MATLAB (for dynamic time warping) and specifying the usage of the SKL metric. A deep LSTM architecture was implemented which had 16 input channels and four hidden layers of 150 neurons each. In order to test the proposed pipeline, the collected PPG dataset was divided as follow: 70% of the dataset was used for training the proposed pipeline, while the remaining 30% was used for testing and validation. In order to train and test the proposed method in a robust way, artificial PPGs were constructed made up of fragments of waveforms from different recruited subjects, therefore, to verify, both in the training phase and in the testing phase, if the algorithm was able to discriminate the identity of one subject from another with respect to the reference generated by the RD-M. The RD-M was classified with identity Class 0 while the other identity was classified with identity Class 1. The benchmark diagrams reported in Figure 10 show the experimental results for both testing and training sessions of the proposed pipeline.

Moreover, further tests with multiple car-driver identities were checked in order to validate the robustness of the proposed pipeline in multiclass identity recognition.

Table 3. Multiclass driver identity testing scenario.

Number of Car-Driver Identities Included in the Synthetic PPG Signal	Overall Discrimination Accuracy
2	99.32%
3	99.20%
4	99.17%
5	99.13%

reports the overall accuracy of the proposed approach with different car-driver identities. For each identity recognition performance reported in Table 3, the proposed pipeline was tested using the PPG signal of the same subject but in different acquired physiological states. Therefore, the accuracy rates refer to the recognition of the identity of the subject in different physiological states.

The recognition of identity was tested in the ignition phase of the car (with start button equipped with PPG sensing device as described in this work) and when changing the gear lever. An average of 3 to 4 secs were employed by the recruited volunteers to perform the aforementioned operations (which involve positioning the driver’s hand or finger over these parts). The used sensing devices acquired the PPG signal at 1 KHz. In addition, in relation to this scenario, the collected accuracy was 98% in validation, confirming the effectiveness of the proposed method. The overall accuracy (both training and test) of the proposed approach was approximately 99.45% which confirmed the robustness of the pipeline described in this contribution.

4. Discussion and Conclusions

The experimental results reported in the aforementioned tables, confirm the capability of the proposed pipeline for recognizing driver identity of a motor vehicle by analyzing some physiological features. The proposed approach introduced the concept of cardiovascular imprint as a valuable replacement of the classic fingerprint to profile and correctly identify a subject. Specifically, through the proposed approach, the equipped ADAS system was able to constantly monitor the driver’s identity knowing exactly and at all times the person who was driving. In addition, by recognizing the driver identity, the proposed approach was able to provide suitable information to the car control unit to enable the services and ad hoc configurations for the detected identity, offering, as mentioned in the Introduction, the capability of tailor-made ADAS systems for each driver.

The author is bringing the algorithm described, in this study, into microcontrollers designed and certified for the automotive industry. Specifically, to manage the sampling and preprocessing part of the PPG signal by using devices of the SPCxx family of 32-bit automotive microcontrollers series produced by STMicroelectronics specifically for the ADAS sector [24] and equipped with ADC and ARM core(s) for implementing the preprocessing pipeline, as described in Section 2. The DTW technique and deep learning have been covered by the ST Accordo5 microcontroller [25,26]. Figure 11 shows the proposed hardware architectures:

As shown in Figure 11, the proposed pipeline is being ported to automotive grade devices normally used in the latest generation cars. The PPG sensing probes positioned on the steering wheel would then be plugged into the USB hub of the motherboard, as described in Figure 6, which would be included in the car control unit. The SPCxx series microntrollers embedded in the car control unit would sample and digitize the PPG signal, and through the PRG-PRS pipeline included as firmware, they would output a filtered and stable PPG signal. This would feed to the deep learning and DTW processing pipeline embedded as firmware in the system based on the ST Accordo5 platform which would, therefore, generate the classification of the driver identity.

Future developments of the proposed method should focus on the integration of vision algorithms always based on the use of deep learning and video saliency [27,28] for the characterization not only of the driver’s identity but also of the correlated anxiety and stress level in order to improve driving safety and assistance systems. It is interesting to highlight that the author has already patented and published a study in which, through an innovative low frame-rate motion magnification technique applied to the driver’s face (captured by a vision camera with which the latest generation cars are being equipped), it is possible to reconstruct some features of the corresponding car-driver PPG signal in order to address the problem that could occur if the driver does not put a hand on any of the sensors located on the steering. More details are available in [21]. Specifically, the two analyzed methods (sampling of the PPG signal in the car steering versus reconstruction of the PPG signal by the motion magnification technique described in [21]) allow, in a complementary way, to always have a car-driver PPG signal that can be used by the car’s control unit to correctly monitor the identity of the driver. Therefore, in the absence of a reliable PPG signal from the steering or from other parts where the PPG sensing devices are installed (start button and car gear lever), the car’s control unit would integrate the PPG signal reconstructed from the camera located on the base of the steering wheel, in order to apply the proposed identity recognition approach. The proposed hardware setup is reported in Figure 10 in which the author described the architecture including both PPG sampling from the car steering wheel, as well as from camera vision installed at the base of the steering. The camera vision used has 2.3 Mpx and 50 fps as frame rate.

Therefore, the proposed method has the ability to identify the driver quickly and correctly by an approach based on the subject’s so-called cardiovascular imprint, and therefore, differently from the recognition based on the classic fingerprint method, it has numerous advantages. For example, it cannot be easily hacked, because a subject with the same cardiovascular imprint (the physiological features correlated to cardiac activity of the subject, as illustrated in the previous paragraphs) would have be found. Notably, finding two people who show the same cardiovascular imprint is particularly complex, and therefore difficult to manipulate. The classic method based on fingerprint as it is known can be hacked more easily, and therefore several algorithms have been developed to cover this issue [29]. Future extensions of the proposed pipeline should evaluate the use of one-dimensional (1D) convolution neural networks (1D-CNNs) in place of LSTM-based architectures, as 1D-CNNs have shown promising performance in the area of signal processing [30].

Finally, preliminary and encouraging results are being found by analyzing the integration of the driver’s identity recognition through the fusion of information coming from both physiological signals (PPG) and vision signals. Specifically, through stereoscopy techniques and through some hand-crafted image features already successfully used by the author, a more robust acquisition of the driver identity data is being analyzed in order to integrate the physiological part and the vision signals both in visible spectrum and near-infrared spectrum [31,32,33,34].

5. Patents

F. Rundo, C. Spampinato, S. Conoci, Physio-Fingerprint System: A Novel Deep learning system with Adaptive Dynamic Time Warping framework for a robust car-driver identification and profiling, IT Patent Nr. 102019000015926, 9 September 2019;

F. Rundo, S. Conoci, Combined Hyper Butterworth Filtering and Machine Learning for Efficient High-speed Time-based Drowsiness Detection in Next-Generation Cars, IT Patent Nr. 102019000005868, 16 April 2018;

F. Rundo, F. Trenta, S. Battiato, S. Conoci, Advanced Motion-Tracking System with Multi-Layers Deep Learning Framework for Innovative Car-Driver Drowsiness Monitoring, IT Patent Nr. 102019000000133, 07 January 2019.