Smartphone User Identity Verification Using Gait Characteristics

Abstract

Smartphone-based biometrics offers a wide range of possible solutions, which could be used to authenticate users and thus to provide an extra level of security and theft prevention. We propose a method for positive identification of smartphone user’s identity using user’s gait characteristics captured by embedded smartphone sensors (gyroscopes, accelerometers). The method is based on the application of the Random Projections method for feature dimensionality reduction to just two dimensions. Then, a probability distribution function (PDF) of derived features is calculated, which is compared against known user PDF. The Jaccard distance is used to evaluate distance between two distributions, and the decision is taken based on thresholding. The results for subject recognition are at an acceptable level: we have achieved a grand mean Equal Error Rate (ERR) for subject identification of 5.7% (using the USC-HAD dataset). Our findings represent a step towards improving the performance of gait-based user identity verification technologies.

1. Introduction

Recent research has indicates that the number of smartphone users is expected to reach 2.08 billion in 2016 (Statista: www.statista.com). The advanced features and high performance characteristics allow smartphones to be used not only as tool of communication, but also in business applications, to store personal data (contacts, calendar, photos, location data, etc.) and to access personal data over the Internet (social networks, e-mail servers, etc.). The access to personal data provided by smartphones may raise various security problems for smartphone users related to privacy and personal or business data protection or the risk of being impersonated. Physical security of the device should be ensured as smartphones can be left, lost or stolen. According to consumer reports [1], about 10% of U.S. smartphone owners have been the victims of phone theft while 68% of theft victims were unable to recover their device. Therefore, the development of new reliable and robust authentication and verification methods against phone hijackers is urgently needed.

The most common authentication mechanisms now are still based on typing a password, PIN or assembling a graphical puzzle. Security of the password-based authentication systems mainly depends upon keeping the passwords secret. While these authentication methods each have their advantages such as high accuracy, they still require the users to memorize the password or a puzzle. The problem is that the required secret is either secure or easy-to-remember. When managing multiple passwords or puzzles for different systems or devices (one for each device), it is a significant overhead to users when trying to remember each password. In practice, users often use the same password for different applications and choose easy to remember passwords such as those that contain semantic information like birthdays, family names, pet names, etc. In order to use different and random looking passwords users usually write down their passwords. As a result, the search space for an attacker is decreased. These practices make the password based authentication mechanisms vulnerable to dictionary attacks.

Furthermore, entering a password may be time-consuming, error-prone and cumbersome, especially while using the phone on the go, and that is why many users are not using passwords. Drawing a graphical pattern on screen may reduce the burden, but it still requires explicit user interaction and is not very convenient in high mobility scenarios. Another target group of users who find passwords difficult are people suffering from memory loss or hand tremor. Finally, passwords are only artificially associated with users and cannot truly verify the identity of individuals. Consequently, they can be spied upon, guessed, lost or stolen, resulting in impersonation attacks and other security breaches. As a result, a significant part of users consider the password/PIN-based authentication as inconvenient and do not use it (see the results of a survey presented in [2]). According to a survey presented in [3], only 13% of the participants secure their phones with a PIN or visual code during standby, or deactivate the authentication methods of their mobile devices citing usability issues as the main reason for it.

Other authentication methods such as face recognition, speech recognition or fingerprint scans are not widely used. For face recognition, the main concerns are the restricted memory and computational power available, as well as the uncontrolled ambient environment. For a continuous authentication, the speech during phone calls is analyzed and the authentication is performed in the background, which also introduces considerable computational overhead and reduces battery life. Fingerprint scanning requires an extra high-cost sensor that is not needed by the average end-user. Capturing high-quality finger photos for fingerprint recognition using existing phone cameras is still a problem for current mobile phones.

Recently, biometric methods of authentication have started to be used, including face, voice or fingerprint recognition [4]. These authentication methods do not require memorization and depend upon unique biometric characteristics of a user. However, they heavily depend upon ambient conditions, e.g., poor lighting or ambient noise may prevent the device from correctly recognizing the face or voice of its user. Furthermore, after the user logs in, there are no further authentication procedures employed until the phone locks or switches off. A hijacker may gain access to the phone if the owner leaves it unattended. Another problem is if a hijacker steals the phone but does not try to log in, no active authentication measures are imitated by the device.

To overcome these problems, smartphone-based biometrics offers a wide range of possible solutions, which could be used to verify user’s identity and thus to provide an extra level of security and theft prevention. One of such solutions is the ability recognize human gait (a person’s walking style) using a set of in-built sensors such as accelerometers. Considering that each person has a unique way of walking containing user-distinctive patterns, inertial sensors embedded in smartphones can be applied to the problem of gait recognition in security-related applications [5]. Human gait has been widely acknowledged by researchers as a biometric trait that can be used for authentication purposes via recognizing individuals based on their behavioral or physiological characteristics [6]. One of the advantages of human gait is that it can be passively observable, unobtrusive, implicit, continuous and concurrent, and it is easily measured as a user carries his phone around. When the user is walking, the phone will be recognizing him based on his gait, so that he can directly use the services of the phone without any further authentication. Hence, in contrast to password or PIN based authentication, it incurs no extra effort for the user.

Gait recognition can be executed continuously in the background while the user is walking and during log in, no additional calculations are required, thus avoiding annoying delays. Furthermore, when several users use the same device, biometric authentication can be used to automatically personalize the services provided by a mobile device [7]. Only if the user is not recognized by his walk or is not walking, an active authentication via PIN or graphical puzzle may be activated [8]. Gait-based authentication is difficult to attack as it is very difficult to emulate the gait characteristics of a legitimate user, and by trying to do so the attacker will probably appear even more suspicious. The deployment of gait-based user identification is cost efficient as it does not require deploying additional hardware and increasing product cost (accelerator sensors are available) on most of smartphones.

On the other hand, gait-based authentication has its own weakness, which have been acknowledged many times before: gait can be affected by clothes, shoes, carrying objects, physical changes to the state of the user (injury, weight gain/loss, aging), phone placement and orientation, environmental context (conditions of walking surface), stimulants (drugs and alcohol), and psychological state (mood). Additional user time may be required for dealing with Failure To Acquire (FTA) errors. These weaknesses reduces the discriminating power of gait as a biometric, but still does not deny the use it as a complementary security mechanism or for providing access to low security data and resources (such as music files).

Summarizing, the authentication via accelerometer-based biometric gait recognition offers a user-friendly alternative to common authentication methods on smartphones. It has the great advantage that the authentication can be performed without user interaction. Furthermore, it can be used as one of security levels in a multi-level security system that allows to trade-off usability and accessibility. From 2003, implicit sensor-based gait recognitions are initially proposed to support existing authentication mechanisms that are obtrusive and inconvenient in frequent use on mobile phones and achieved promising results.

In 2003, Wang et al. [9] apply a maximum-based cycle extraction to the vertical acceleration measured using the data of a sensor attached to the back of the waist. The cycles are segmented based on identified extreme points, and different features, like relative time in cycle or slope of straight line between two endpoints, are extracted. Classification based on the Dynamic Time Warping (DTW) distance resulted in an EER of 5% for a dataset containing data of 24 subjects.

Ailisto et al. [10] propose the gait authentication based on wearable accelerometer. Acceleration data were analyzed to find individual steps, normalize, and align them with the template. Then, cross-correlation was applied as a measure of similarity, reaching 6.4% of EER.

Thang et al. [11] use accelerator data in time domain to construct gait templates and DTW to evaluate the similarity score. Features in frequency domain are classified using Support Vector Machine (SVM), achieving the accuracy of 79.1% and 92.7%, respectively.

Rong et al. [12] applied a simple cycle extraction method based on the number of zero-crossings. The average cycle was computed by normalizing the detected cycles to equal length using DTW. The gait template was constructed by concatenating average cycles for each acceleration direction. The same DTW algorithm is used for comparison, and an EER of 5.6% was obtained on a database of 21 subjects.

Pan et al. [13] attached accelerometers (Wii remote) to five different parts of the body, i.e., upper arm, wrist, thigh, ankle, and waist and reached recognition rates of 96.7% when using all five channels for a database containing 30 subjects.

Sprager [14] used a mobile phone (Nokia N95) attached to the hip and divided the recorded forward-backward and vertical acceleration data into cycles. Cumulants of order 1 to 4 were extracted from the signals and transformed into a feature vector. SVMs were used for classification, reaching a recognition rate of 92.9% for a dataset of six subjects.

Bachlin et al. [15] evaluated the influence of shoes, weight, and time on gait. Four different feature types are computed for different signal types: segments containing 64 data samples with and without normalization of step length, and FFT coefficients from a jumping window containing 256 samples starting at regular points or at the heel strike. Similarity between data was computed using one-way Analysis of Variance (ANOVA) and by determining the percentage of positions in the feature vector for which the ANOVA showed that they are not statistically significant different. EER of 2.8% was reported for the same-day recognition task, while EER increased to 21.3% when mixed data of several days was used using data of five subjects.

Trivino et al. [16] used a linguistic model based on the computational theory of perceptions. The perception of the signal was modeled by a Fuzzy Finite State Machine (FFSM) and the model was expressed via linguistic terms. The method was tested on a database containing same-day data of eleven subjects and an EER of 3% was obtained.

Frank et al. [17] used time-delay embedding models created from the acceleration data collected from 25 individuals via a smartphone placed in the trouser pocket. For each subject, the probe segments were mapped into the model-space. Considering four nearest neighbors, scores are calculated for each mapped test segment. Classification was based on the highest average score, resulting in a perfect classification result for the given test set.

Nickel et al. [18] use the Mel- and Bark-frequency cepstral coefficients (MFCC, and BFCC) and a SVM classifier. The proposed approach showed competitive recognition performance, yielding 5.9% false match rate (FMR) and 6.3% false non-match rate (FNMR).

Kwapisz [7] used J48 and Neural Net classifiers for classifying multiple activity data such as from walking, jogging, going up stairs, and going down stairs of 36 subjects. Forty-three features, were generated for each axis for each feature-type as follows: average, average acceleration value, standard deviation, average absolute difference, average resultant acceleration, time between peaks, and binned distribution. They were able to identify a person walking with a positive authentication rate of 82.1%–92.9%, respectively.

Kobayashi et al. [19] constructed a feature extraction model based on Fourier transform features derived from 58 subjects who held the phone in a hand while walking. The model resulted in accuracy between 45% and 50%.

Juefei-Xu et al. [20] used SVM, a time frequency spectrogram model and a cyclo-stationary model using data collected from 36 subjects. The best results were 99.4% verification rate with normal walking and 96.8% verification rate with fast walking using both accelerometer and gyroscope data.

Hoang et al. [21] used the gait template matching approach to compare data collected from 38 subjects on four consecutive gait cycles and reported EER of 3.5%.

Derawi and Bours [22] proposed a feature extraction method that used time interpolation to find the average cycle of a subject for authentication. The result of this study was an EER of 20.1% for a dataset of 10 subjects.

Wolff [23] used a Gaussian distribution model constructed using the variance in acceleration and orientation across the three dimensions (x, y, and z) and has achieved the subject classification accuracy of 83%.

Lu et al. [24] used a Gaussian Mixture Model—Universal Background Model (GMM-UBM) framework for gait verification. The authentication is done by comparing the likelihood score from a user gait model (representing the user’s specific gait pattern) and a universal background model of all human gait patterns. The gait model used the following features of signals: mean, variance, skewness, kurtosis, energy, mean crossing rate, energy ratio between vertical and horizontal components, spectrum peak, spectral entropy, ratio between low frequency band energy and high frequency band energy, compressed sub-band cepstral coefficients and coefficients of autocorrelation.

Lin et al. [25] propose a system for gait recognition analysis. The αβ filters were used to improve the recognition and, Empirical Mode Decomposition (EMD) was used to filter the noise. Then Linear Discriminant Analysis (LDA) was applied to the Fourier transform energy spectrum for training and recognition.

Johnston et al. [26] used smartwatches to collect gait data and classical induction algorithms from Weka package for subject authentication achieving the EER rate of 2.6% when using features derived from the accelerometer data and EER of 8.1% when using data derived from gyroscope sensor.

Attal et al. [27] have presented a comparative study of classifiers (k-Nearest Neighbors (kNN), Gaussian Mixture Models, Support Vector Machines (SVM), Random Forest, k-means and Hidden Markov Models (HMM)), which can be used to recognize human activities from wearable inertial sensor data. Best results were obtained with kNN classifier reaching accuracy of 96.53%.

Abidine et al. [28] have discussed feature extraction methods (Principal Component Analysis (PCA), Independent Component Analysis (ICA), Linear Discriminant Analysis (LDA)) and their relevance to improve the classification accuracy of the existing daily living activity recognition systems, and achieved 77.0% accuracy using Weighted Support Vector Machines (WSVM) classifier.

The related works are summarized in

Table 1. Summary of related work in human activity recognition domain.

Ref.	Features	Methods	Subjects	Results
Wang et al. [9]	Domain specific (e.g., relative time in cycle or slope of straight line between two endpoints)	Maximum-based cycle extraction, Dynamic Time Warping (DTW) distance	24	5% (EER)
Ailisto et al. [10]	Averaged x (forward), and z (vertical) acceleration signals	Template matching, cross-correlation	36	6.4% (EER)
Thang et al. [11]	Time and frequency domain features	Gait templates, Dynamic Time Warping (DTW), Support Vector Machine (SVM)	11	DTW: 79.1%; SVM: 92.7% (accuracy)
Rong et al. [12]	Acceleration	Zerocrossing-based cycle extraction, DTW	21	5.6% (EER)
Pan et al. [13]	Extrema in acceleration data space	Difference-of-Gaussian filtering, Nearest Neighbors	30	96.7% (accuracy)
Sprager [14]	Order 1–4 cumulants of acceleration data	Support Vector Machine (SVM)	6	92.9% (accuracy)
Bachlin et al. [15]	FFT coefficients	FFT, one-way Analysis of Variance (ANOVA)	5	2.8%–21.3% (EER)
Trivino et al. [16]	Vertical acceleration, lateral acceleration, and acceleration in the progress direction.	Fuzzy Finite State Machine (FFSM), linguistic model	11	3% (EER)
Frank et al. [17]	acceleration data	time-delay embedding models, k Nearest Neighbors	25	Perfect classification
Nickel et al. [18]	Mel- and Bark-frequency cepstral coefficients (MFCC, BFCC)	SVM classifier	48	5.9% (FMR); 6.3% (FNMR)
Kwapisz [7]	Average, average acceleration value, standard deviation, average absolute difference, average resultant acceleration, time between peaks, binned distribution	J48 and Neural Net classifiers	36	82.1%–92.9% (positive authentication rate)
Kobayashi et al. [19]	Cross-correlations of Fourier transform coefficients	Multi-class classification by nearest means in Fisher discriminant space and majority voting	58	45%–50% (accuracy)
Juefei-Xu et al. [20]	Accelerometer and gyroscope data	SVM, a time frequency spectrogram model and a cyclo-stationary model	36	96.8%–99.4% (accuracy)
Hoang et al. [21]	Magnitude of the acceleration forces acting on three directions (x, y and z)	Gait template matching	38	3.5% (EER)
Derawi and Bours [22]	Magnitude of the acceleration	Weighted moving average (WMA) filter, cycle detection, Manhattan distance metric, LibSVM	10	99.6%—same subject, 87.6%—cross-subject (accuracy)
Wolff [23]	Variance in acceleration and orientation across the three dimensions (x, y, and z)	Gaussian distribution model		83% (accuracy)
Lu et al. [24]	Mean, variance, skewness, kurtosis, energy, mean crossing rate, energy ratio between vertical & horizontal components, spectrum peak, spectral entropy, ratio between low and high frequency band energy, compressed sub-band cepstral coefficients, compressed sub-band cepstral coefficients of autocorrelation	Gaussian Mixture Model—Universal Background Model (GMM-UBM)	47	14% (EER)
Lin et al. [25]	Spectral energy diagrams of pitch, roll, acceleration X, acceleration Y, and acceleration Z	αβ filtering, Empirical Mode Decomposition (EMD), Fourier Transform, Linear Discriminant Analysis (LDA)	10	90% (recognition rate)
Johnston et al. [26]	Average sensor value, standard deviation, average absolute difference between the 200 values and the mean of these values, time between peaks (each axis), binned distribution, average resultant acceleration	Multilayer Perceptron (MLP), Random Forest, Rotation Forest, and Naive Bayes	59	2.6%–8.1% (EER)

.

The aim of this paper is to analyze the existing research on gait recognition using features derived from the acceleration and gyroscope sensors of a smartphone and propose a method for gait-based user identity verification. The novelty of the paper is the application of Random Projections for gait feature dimensionality reduction in the context of user identity verification.

The structure of the remaining parts of the paper is as follows. Section 2 describes the methodological background for human activity recognition and subject identification using smartphone or wearable sensors and the proposed method. Section 3 describes our experiments using the USC-HAD dataset and the obtained results. Section 4 presents the evaluation and discussion. Finally, Section 5 presents conclusions and considers future work.

2. Materials and Methods

2.1. Background

As a methodological background of our analysis we adapt the concept of Context Pyramid adopted from Pei et al. [29]. We describe the domain of Human Activity Recognition (HAR) using a six level Context Pyramid: Raw Sensor Data, Patterns, Activities, Actions, and Context (see Figure 1). In this paper, we focus on the three lowest levels of the Context pyramid: sensor data, features, and activities. Raw data from diverse sensors are the foundation of the Context. Based on the Raw Data, we can extract activity features such as spatial coordinates, orientation, movement direction, speed and acceleration. Activities define the higher levels of the Context Pyramid (the state, position and context of the person). Hereinafter we continue with the analysis of the HAR tasks (Section 3.2) and human activities (Section 3.3).

2.2. Tasks of HAR

We define four main tasks of HAR (see Figure 2):

Task 1. Basic activity recognition: Basic activities are low-level activities such as walking, sitting, standing, i.e., activities which can be characterized by statistical sequence of body motions or gestures, and which typically last between few seconds and several minutes. On an even smaller time scale, brief and distinct body motions such as taking a step are sometimes referred to as actions, movements, gestures, or motifs [30]. Low-level activities are only loosely defined since there is no generally accepted definition of these terms in the activity recognition community. Bobick [31] attempted to differentiate “action” as a higher semantic level than an “activity” based on the occurrences of movements and interactions, in which he has defined an “action”. In contrast, Govindaraju [32] defined an “action” as an atomic motion pattern that is often gesture-like and has a specific trajectory (e.g., wave arm), whereas an “activity” is a series of actions performed in an ordered sequence that is dependent on human motion patterns.

Task 2. Daily activity recognition. Activities of Daily Living (ADL) are a standard set of higher-level activities used by physicians and care-givers as a measure to estimate the physical well-being of elderly patients, as well as their need for assisted living. High-level activities are usually composed of a collection of low-level activities, and are longer-term typically lasting from several minutes to few hours. The ADLs include such activities as dressing, bathing, toileting, cleaning the room, cooking, eating, and washing dishes. A related set of complex activities, called Instrumental ADLs (IADLs) consist of using the phone, shopping, food preparation, housekeeping, doing laundry, transportation, taking medications, and handling finances that are dependent on memory and executive functioning of subjects. Apart from the activities mentioned so far, further activities that can be recognized with wearable sensors include sports activities such as cycling, rowing, running, Nordic walking. Recognizing the complete set of ADLs and IADLs using sensors is challenging, since some activities such as handling finances are only loosely defined and are difficult to detect. The recognition of high-level activities is important for the description of an individual’s daily routine.

Task 3. Unusual event recognition. Event recognition is domain aiming to provide convenience, safety and comfort for elderly by detecting potentially dangerous human activity events to reduce the risks for the elderly. The following types of unusual events have been analyzed in the literature:

• Sudden events can be defined as an abrupt, unintentional and unexpected change in the human body position that happens during a short period of observation, has not been observed before (i.e., was not present in the training dataset) and is unpredictable [33]. In the case of a home care assistance system, a sudden event refers specifically to a sudden fall by a patient or elderly person that requires immediate response. Detection and tracking of the position and movement of human body and parts thereof are useful features for early indication of a sudden fall event.
• Abnormal events are actions that are performed at an unusual location and at an unusual time [34]. This type of events can be characterized as temporal or spatial outliers, which deviate from normal events as represented in the training dataset or learned motion patterns and require a longer observation for identifying it.

Task 4. Biometric subject identification. The sensor readings registered during different human activities can be considered as a kind of physiological biometric characteristic that is further used to identify the subject [34]. The authentication process is an essential requirement so as to permit the genuine user (owner) to obtain access to the device. Behavioral biometrics is related to specific actions (e.g., walking, running, etc.) and the way that each person executes them [35]. Biometric-based authentication verifies that the genuine owner of the device is present in the immediate vicinity of the biometric sensor. An example of activity-based biometric characteristics is gait, which is a complex spatio-temporal motor-control behavior that allows biometric recognition of individuals at a distance [36].

Task 5. Prediction of energy expenditures. Energy expenditure estimation using wearable sensors seeks to find the relationship between the energy expenditure and the sensor outputs [37]. In this task, activity recognition is performed as a part of the energy expenditure prediction process. Energy expenditure is estimated using MET (Metabolic Equivalent Task), which is defined as the ratio of metabolic rate during a specific physical activity to a reference metabolic rate [38]. The measurement is useful for real-time physical activity monitoring.

All tasks of the HAR domain require correct identification of human activities from sensor data, which in turn requires that human activities must be properly categorized and described. In the following subsection, taxonomy of human activities is analyzed.

2.3. Taxonomy of Human Activities

The development taxonomies of human activities are important as the gained knowledge can be used in multi-layer (multi-step) classification systems such as described in [39]. There is a great deal of taxonomies (explicitly or implicitly formulated) found in the literature. In fact almost every author of a paper on this topic introduces and analyses his own set of human activities. The examples of such taxonomies are given hereinafter.

Zhu and Sheng [39] classified human daily activities as stationary (lying, sitting, standing, walking), motional and other activities. Motional activities include long term activities such as walking ant transitional activities such as sit-to-stand.

The taxonomy of Incel et al. [40] covers locomotion (walking, running, sitting, standing, lying), transportation (biking, riding, driving), exercise (bicycling, playing), health related activities (falls, rehabilitation procedures), and daily activities (shopping, sleeping, working on a PC, eating, etc.).

Lara and Labrador [41] provided taxonomy of activities recognized by the state-of-the art human activity recognition systems. Seven groups of activities are recognized: ambulation (walking, running, sitting, standing, stairs up/down, elevator up/down), transportation (riding a bus, cycling, driving), phone usage (messaging, calling), daily activities (eating, drinking, working with the PC, watching TV, reading, doing hygiene, cleaning), exercises (rowing, etc.), military (crawling, kneeling, etc.) and upper body activities (chewing, speaking, etc.).

Fleury et al. [42] presented the following classification: Sleeping, Preparing and having a meal, Dressing/undressing, Resting (including watching TV, listening to the radio, reading a book, sitting down on the sofa), Hygiene (tooth brushing and washing of the hands), Bowel movement, and Communication (using a phone).

Capela et al. [43] identified seven different meta-classes (or levels) of activities differing by the level of detail: Level 1: Mobile, and immobile (large movements and stairs labeled as mobile; sit, stand, lie, and small movements labeled as immobile); Level 2: Sit, and stand (not including small movements); Level 3: Sit, stand, and lie; Level 4: Large movements (going upstairs); Level 5: Ramp up, ramp down, large movements, stairs up, and stairs down; Level 6: Small movements (e.g., sitting, standing or lying); and Level 7: Transition states (transition between activities).

Atallah et al. [44] proposed the following classification of activities based on energy expenditure of a person: very low-level activities (e.g., lying down); low-level activities (e.g., eating, drinking, reading, and getting dressed); medium level activities (e.g., walking, vacuuming, and cleaning); high level activities (e.g., running and cycling); and transitional (transfer) activities (e.g., sit-to-stand, laying down-to-stand).

Several classifications of human activities include more complex activities related with the field of sports: lying, Nordic walking, outdoor bicycling, rowing with the rowing machine, running, sitting, soccer playing, standing, walking in [45], or for daily activities such as having lunch, breakfast or dinner, going to work, shopping, sleeping, using a computer, and working [46]. The most comprehensive classification of human activities is the Physical activity compendium [35], which has 21 categories of activities as follows: Bicycling; Conditioning Exercise; Dancing, Fishing and Hunting; Home Activities; Home Repair; Inactivity; Lawn and Garden; Miscellaneous; Music Playing; Occupation; Running; Self Care; Sexual Activity; Sports; Transportation; Walking; Water Activities; Winter Activities; Religious Activities; and Volunteer Activities.

The results of analysis of human activity domain are represented as a feature diagram [47] in Figure 3. The feature diagram notation has been adopted from the field of product line engineering to represent compactly a set of related entities in the domain of interest. The meaning of the elements of the feature diagram is explained below [48]. Feature diagram is a connected graph, where nodes represent features and edges represent relations among features. There are three types of features: mandatory (boxes with the black circle above), optional and alternative. Mandatory features express commonality of the concept, whereas optional and alternative features express variability. Features may appear either as a solitary feature or in groups. If all mandatory features in the group are derived from the same parent in the parent–child relationship, there is the “and” relationship among those features. An optional feature may be included or not if its parent is included in the model. If only one feature can be included from a set of child features, it is called an “alternative” feature.

2.4. General Scheme of Gait-Based User Identity Verification

The proposed method follows a typical architecture of biometric systems (see Figure 4) and consists of two continuous processes: training and verification. During training, the person registers with the systems and the capture of his gait characteristics using built-in sensors is performed. Data pre-processing is performed to segment the data into frames, and remove noise and outlier artifacts. Features are extracted and dimensionality reduced to construct a user’s gait model that characterizes the considered person while discarding irrelevant information. It is subsequently stored, for instance, on a memory card or on a cloud. As the user gait characteristics may drift from day to day due to his/her health condition or emotional state, the system has to be retrained at least once a day.

During verification, the first steps are the same as in case of training: sensing, data pre-processing, feature extraction and feature dimensionality reduction. The reduced set of feature values is, first, activity detector identifies a specific action the user is undertaking (e.g., walking or running). Then it is compared with the users gait profile stored by the system and the decision is taken based on the distance of the recorded profile from known user’s profile either to confirm user’s identity or to reject it and initiate pre-specified security measures.

2.5. Description of the Method

The proposed method for subject identification based on gait characteristics is based on feature dimensionality reduction using Random Projections [49] and classification using probability density function (PDF) estimate as a decision function.

When performing random projection, the original d-dimensional data is projected into a k-dimensional (k << d) subspace using a random k × d matrix R. The projection of the data onto a lower k-dimensional subspace is $X_{k\times N}^{RP}=R_{k\times d}{X}_{d\times N}$, where X_d×N is the original set of N d-dimensional observations. In the derived projection, the distances between points are approximately preserved, if points in a vector space are projected onto a randomly selected subspace of suitably high dimension (Johnson–Lindenstrauss lemma [50]). The random matrix R can be selected as follows:

$$r_{ij}=\{\begin{array}{c}+1,probability=1/6\\ 0,probability=2/3\\ -1,probability=1/6\end{array}$$

(1)

Given the low dimensionality of the target space, we can treat the projection of low-dimensional observations onto each dimension as a set of random variables for which the probability density function (PDF) can be estimated using the Parzen window method [51].

If x₁, x₂, …, x_N is a sample of a random variable, then the kernel density approximation of its probability density function is:

$${\widehat{f}}_h\left(x\right)=\frac{1}{Nh}K\left(\frac{x-x_i}{h}\right)$$

(2)

where K is some kernel function and h is the bandwidth (smoothing parameter). K is taken to be a standard Gaussian function with mean zero and variance 1 of the examined data features:

$$K\left(x\right)=\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}{x}^2}$$

(3)

For a two dimensional case, the bivariate probability density function is calculated as a product of univariate probability functions as follows:

$$\widehat{f}\left(x,y\right)=\widehat{f}\left(x\right)\cdot \widehat{f}\left(y\right)$$

(4)

where x and y are data in each dimension, respectively.

However, each random projection produces a different mapping of the original data points that reveals only a part of the data manifold in the higher-dimensional space. In case of the binary classification problem, we are interested in a mapping that separates data points belonging to two different classes best. As a criterion for estimating the mapping, we use the Jaccard distance metric between two probability density estimates of data points representing each class. The Jaccard distance is easily adaptable to multidimensional spaces where compared points show relations to different subsets. The Jaccard distance, which measures dissimilarity between sample sets, is complementary to the Jaccard coefficient and is obtained by subtracting the Jaccard coefficient from 1, or, equivalently, by dividing the difference of the sizes of the union and the intersection of two sets by the size of the union:

$$d_J\left(A,B\right)=1-J\left(A,B\right)=\frac{\left|A\cup B\right|-\left|A\cap B\right|}{\left|A\cup B\right|}$$

(5)

For classification the best random projection with smallest overlapping area is selected. In the case of multiple classes, the method works as a one-class classifier: recognizing instances of a positive class, while all instances of other classes are recognized as outliers of the positive class. The acception/rejection is demonstrated in Figure 5.

3. Results

3.1. Dataset

To evaluate the performance of the proposed approach for the smartphone data, we used the USC Human Activity Dataset [52] recorded using the MotionNode device (sampling rate 100 Hz, triaxial accelerometer range: ±6 g, triaxial gyroscope range: ±500 dps). The dataset consists of records recorded with 14 subjects (7 male, 7 female; Age: 21–49) of 12 activities, five trials each.

The following activities have been recorded: Walking Forward (WF), Walking Left (WL), Walking Right (WR), Walking Upstairs (WU), Walking Downstairs (WD), Running Forward (RF), Jumping Up (JU), Sitting (Si), Standing (St), Sleeping (Sl), Elevator Up (EU), and Elevator Down (ED).

3.2. Features

Sensor readings consist of six readings: acceleration along x-, y- and z-axes, and gyroscope along x-, y- and z-axes. Based on the extensive analysis of literature and features used by other authors (especially by Capela et al. [43], Mathie et al. [53], Zhang and Sawchuk [52]), we have extracted 99 features of data, which have been detailed in [54]. The feature ranking was performed using Kullback–Leibler divergence as class separability criterion on the human activity data from the USC-HAD dataset as described in [54].

We summarize the best ranked features for subject identification in

Table 2. Best ranked features for subject identification.

Rank	Feature	Description
1	Moving variance of 100 samples of gyroscope data along z-axis	$\mathrm{var}=\frac{1}{N\left(N-1\right)}\left(N{\displaystyle \sum _{i=1}^N{x}_i^2}-{\left({\displaystyle \sum _{i=1}^N{x}_i}\right)}^2\right)$, here $x=g_z$
2	Moving variance of 100 samples of acceleration intensity data	$\mathrm{var}=\frac{1}{N\left(N-1\right)}\left(N{\displaystyle \sum _{i=1}^N{x}_i^2}-{\left({\displaystyle \sum _{i=1}^N{x}_i}\right)}^2\right)$, here $x=\sqrt{a_x^2+a_y^2+a_z^2}$
3	First eigenvalue of moving covariance of difference between acceleration and gyroscope data	$E_{ag}=ei{g}_1\left(\mathrm{cov}(a_x-g_x,a_y-g_y,a_z-g_z)\right)$
4	Moving energy of gyroscope data along z-axis	$ME=\frac{1}{N}{\displaystyle \sum _{i=1}^N{x}_i^2}$, here $x=g_z$
5	Moving energy of difference between acceleration and gyroscope data along z-axis	$M{E}_{ag}=\frac{1}{N}{\displaystyle \sum _{i=1}^N{\left(x_i-y_i\right)}^2}$, here $x=a_z$, $y=g_z$
6	Moving variance of 100 samples of acceleration data along x-axis	$\mathrm{var}=\frac{1}{N\left(N-1\right)}\left(N{\displaystyle \sum _{i=1}^N{x}_i^2}-{\left({\displaystyle \sum _{i=1}^N{x}_i}\right)}^2\right)$, here $x=a_x$
7	First eigenvalue of moving covariance between acceleration data	$E_a=ei{g}_1\left(\mathrm{cov}\left(a_x(1:N),a_y(1:N),a_z(1:N)\right)\right)$
8	First eigenvalue of moving covariance between gyroscope data	$E_g=ei{g}_1\left(\mathrm{cov}\left(g_x(1:N),g_y(1:N),g_z(1:N)\right)\right)$
9	Moving energy of orientation vector of acceleration data	$MEA=\frac{1}{N}{\displaystyle \sum _{i=1}^N{\phi }_i^2}$, here $\phi =\frac{\arccos \left(a_x\cdot a_y\right)}{\left|a_x\right|\cdot \left|a_y\right|}$
10	Movement intensity of gyroscope data	$M{I}_g=\sqrt{g_x^2+g_y^2+g_z^2}$

.

3.3. Evaluation Metrics

To evaluate the effectiveness of the proposed method for user identity verification, we use four metrics widely used in the information security community:

• False Accept Rate (FAR) is the probability (or a portion of recognition attempts) that the identity verification system incorrectly identifies the hijacker (impostor) as the genuine user. For a user, the FAR is a measure of system security.
• False Reject Rate (FRR) is the probability (or a portion of recognition attempts) that the identity verification system incorrectly rejects the genuine user. For a user, the FRR measures the user inconvenience level.
• Equal Error Rate (EER) is the rate at which both FAR and FRR are equal. The lower the value of ERR is, the higher is the accuracy of the biometric system.
• Accuracy (or a true positive rate, TPR) is a proportion of all recognition attempts where subjects were identified correctly.

3.4. Results

We evaluate the effectiveness of the proposed method for user identity verification based on user gait parameters using the FRR, FAR and Accuracy metrics. For subject identification, the data from all physical actions is used to train the classifier. Here we consider 1-vs.-all subject classification. Therefore, the data of one subject are defined as a positive class, and the data of all other subjects are defined as a negative class. Five-fold cross-validation was performed using 80% of data for training and 20% of data for testing. The results of 1-vs.-all subject identification using all activities for training and testing are presented in Figure 6 and Figure 7. Grand mean FAR for all users is 0.0869, and FRR is 0.0763, with accuracy of 0.9171.

If an activity of a subject has been established, separate classifiers for each activity can be used for subject identification. Then, data from one action only are used for training as well as for testing. In this case, five-fold cross-validation was also performed, using 80% of data for training, and 20% of data for testing, and the results are presented in Figure 8. The grand mean accuracy is 0.7202, which is not a very high result (worst results provided by Elevator Up and Elevator Down actions). However, if we consider only top three walking-related activities (Walking Forward, Walking Left or Walking Right), the mean accuracy is 0.9444.

Finally, we can simplify the classification problem to binary classification (i.e., recognize only one subject against all other subjects). Then, the data from a pair of subjects performing a specific activity are used for classification and training. Separate classifiers are built for each pair of subjects, the results are evaluated using five-fold cross-validation, and the results are averaged. The results are presented in Figure 9. Note that the grand mean accuracy has increased to 0.9475, while for top three walking-related activities (Walking Forward, Walking Left or Walking Right), the grand mean accuracy is 0.9916.

In Figure 10, Figure 11 and Figure 12, we present the values of FAR, FRR and accuracy metrics of subject-vs.-subject identification using data from all types of physical activities. The grand mean achieved is: FAR is 0.056 ± 0.027, FRR is 0.053 ± 0.028, and accuracy is 0.945 ± 0.028 (mean ± standard deviation values given).

Finally, we have evaluated the ERR for each subject as the intersection point of probability density functions (PDFs) of FAR and FRR. We have modeled the PDFs of FAR and FRR using the Weibull distribution. Weibull distribution is often used in reliability analysis. With an appropriate choice of parameters, a Weibull distribution can take on the characteristics of many other types of distributions, including Gaussian. The results are presented in Figure 13. The grand mean ERR for all subjects is 0.057 ± 0.030. In addition, note the substantial variability between subjects (e.g., between S1 and S13). This means that some subjects have more distinctive characteristics of gait than other subjects.

4. Evaluation

Human gait is a complex spatiotemporal biometric characteristic. Gait cannot be considered as a distinctive biometric since it may change over a period of time because of age-related health problems, injuries, changes in clothing (shoes, etc.), terrain of walking, or changes in behavior or mood. However, it may be used for certain low security applications on the smartphone or together with other biometrics (e.g., speech recognition) as an additional layer of security.

To be practical in everyday scenarios, a gait-based user identity verification system must be able to cope with changing context of its use (different placement and orientation of the phone). The position of the phone relative to the user’s body is a key parameter for activity recognition, whereas activity recognition is a key to successful user identity verification. The gait-based user identity verification methods must be computationally simple to be able to run in the phone itself rather than on the remote cloud and must ensure continuous verification of user’s gait parameters.

The challenges these methods face include noisy data due to contamination or interference, high intra-class variability of user’s gait parameters, limited discriminative capability to recognize valid user from other users (some users are more easily recognizable than others), collecting consistent and reliable dataset for testing the algorithm as some activities may have been marked by users with wrong labels, dealing with transitionary and overlapping activities which have fuzzy borders (e.g., between fast walking and running), dealing with incorrect or unanticipated placing of the phone relative to the activity being measured, and ability to withstand spoof attacks when an impostor tries to mimic the gait of the valid user.

Finally, the evaluation of the proposed method using the criteria formulated by Jain et al. [4] and Wayman et al. [2] is presented in

Table 3. Evaluation using biometric system criteria.

According to Jain et al. [4]

According to Wayman et al. [2]

Distinctiveness: gait parameters are unique and difficult to mimic Permanence: human gait characteristics are sufficiently stable over a short (day-to-day) period of time Universality: all people able to walk have fairly distinctive gait characteristics Collectability: the gait characteristics can be measured quantitatively using sensors commonly available on most of smartphones Performance: the advantage of Random Projections is simplicity, scalability, robustness to noise and low computational complexity: constructing random matrix R and projecting the d × N data matrix into k dimensions is of order O(dkN). User matching is performed using a computationally low expensive Jaccard distance Acceptability: continuous gait monitoring is not as intrusive as other methods of authentication (e.g., face recognition) Circumvention: gait is difficult to mimic by an impostor

Cooperative/non-cooperative: does not apply. The method does not require explicit cooperation of user Overt/covert: the method is covert as the user may not be aware that one of his biometrics is being measured Habituation: habituated. The gait characteristic should be measured continuously both to ensure security monitoring and keep the user’s gait model up-to-date Attended/non-attended: non-attended. The method does not require direct supervision of the user Standard/nonstandard: nonstandard. Collecting enough data for user verification may require a longer period of walking, which may be ensured only in a nonstandard (outdoor) environment

.

5. Conclusions

We have presented a method for gait-based user identity verification using smartphone sensor data. Gait-based user identity verification relies on the biometric specificity of human activity traits. By continuously, implicitly and unobtrusively identifying the phone’s owner using accelerometer and gyroscope sensing, gait analysis has a great potential to improve user identify verification on the go.

The proposed method is based on selected statistical and heuristic gait features and application of Random Projections method for reduction of feature dimensionality. Estimate of probability density function (PDF) of low-dimensional feature vector is used to match the user in question with the PDF of the valid user.

The proposed method was tested with off-line data from the USC-HAD dataset. The results for subject recognition are at an acceptable level: the achieved grand mean Equal Error Rate (ERR) for all subjects is 5.7% (std = 3.0%). As gait-based verification technologies are being considered to be deployed as an additional (optional) security layer in smartphones, our findings represent a step towards improving the performance and usability of these systems.

Future work will include the implementation of the gait-based user identity verification system on the mobile (Android) platform and performing experiments in real-time with subjects.