UniMiB SHAR: A Dataset for Human Activity Recognition Using Acceleration Data from Smartphones

Abstract

Smartphones, smartwatches, fitness trackers, and ad-hoc wearable devices are being increasingly used to monitor human activities. Data acquired by the hosted sensors are usually processed by machine-learning-based algorithms to classify human activities. The success of those algorithms mostly depends on the availability of training (labeled) data that, if made publicly available, would allow researchers to make objective comparisons between techniques. Nowadays, there are only a few publicly available data sets, which often contain samples from subjects with too similar characteristics, and very often lack specific information so that is not possible to select subsets of samples according to specific criteria. In this article, we present a new dataset of acceleration samples acquired with an Android smartphone designed for human activity recognition and fall detection. The dataset includes 11,771 samples of both human activities and falls performed by 30 subjects of ages ranging from 18 to 60 years. Samples are divided in 17 fine grained classes grouped in two coarse grained classes: one containing samples of 9 types of activities of daily living (ADL) and the other containing samples of 8 types of falls. The dataset has been stored to include all the information useful to select samples according to different criteria, such as the type of ADL performed, the age, the gender, and so on. Finally, the dataset has been benchmarked with four different classifiers and with two different feature vectors. We evaluated four different classification tasks: fall vs. no fall, 9 activities, 8 falls, 17 activities and falls. For each classification task, we performed a 5-fold cross-validation (i.e., including samples from all the subjects in both the training and the test dataset) and a leave-one-subject-out cross-validation (i.e., the test data include the samples of a subject only, and the training data, the samples of all the other subjects). Regarding the classification tasks, the major findings can be summarized as follows: (i) it is quite easy to distinguish between falls and ADLs, regardless of the classifier and the feature vector selected. Indeed, these classes of activities present quite different acceleration shapes that simplify the recognition task; (ii) on average, it is more difficult to distinguish between types of falls than between types of activities, regardless of the classifier and the feature vector selected. This is due to the similarity between the acceleration shapes of different kinds of falls. On the contrary, ADLs acceleration shapes present differences except for a small group. Finally, the evaluation shows that the presence of samples of the same subject both in the training and in the test datasets, increases the performance of the classifiers regardless of the feature vector used. This happens because each human subject differs from other subjects in performing activities even if she shares with them the same physical characteristics.

1. Introduction

Nowadays, many people lead a sedentary life due to the facilities that the increasingly pervasive technologies offer.

Unfortunately, it is recognized that insufficient physical activity is one of the 10 leading risk factors for global mortality: people with poor physical activity are subjected to a risk of all-cause mortality that is 20% to 30% higher then people performing at least 150 min of moderate intensity physical activity per week [1]. Another important global phenomenon currently affecting our society is population aging: the decline or even decrease of the natural population growth rates due to a rise in life expectancy [2] and to a long-term downtrend in fertility (expecially in Europe [3]). Falls are a major health risk that impacts the quality of life of elderly people. Indeed, among elderly people, accidental falls occur frequently: 30% of the over 65 population falls at least once per year; the proportion increases rapidly with age [4]. Moreover, fallers who are not able to get up are more likely require hospitalization or, even worse, die [5].

Thus, research on techniques able to recognize activities of daily living (ADLs), also known as human activities (HA), and to detect falls is very active in recent years: the recognition of ADLs may allow to infer the amount of physical activity that a subject perform daily, while a prompt detection of falls may help in reducing the consequence (even fatal) that a fall may cause mostly in elderly people.

ADLs recognition and fall detection techniques usually accomplish their task by analizing samples from sensors, which can be physically deployed in the ambient surroundings (ambient sensors, e.g., cameras, vibration sensors, and microphones) or worn by people (wearable sensors, e.g., accelerometers, gyroscopes, and barometers) [6]. To train and evaluate their techniques, researchers usually build their own dataset of samples and rarely make it publicly available [7,8,9]. This practice makes difficult to compare in an objective way the several newly proposed techniques and implementations due to a lack of a common source of data [9,10,11]. Cameras (ambient sensors) and inertial sensors (wearable devices) are the mainly used sensors to record public publicly datasets containing kinematic data of human subjects. HumanEva [12] , CMU Mocap [13] , and those reported in the survey by Chaquet et al. [14] , are examples of datasets including samples from cameras, while the works by Wang et al. [15] and Urtasum et al. [16] are some examples of methods for learning dynamical models of human motion from camera data. Only very recently, Janidarmian et al. combined 14 publicly available datasets focusing on acceleration patterns in order to conduct an analysis on feature representations and classification techniques for human activity recognition [17]. Unfortunately, they do not make the resulting dataset available for downloading.

The few publicly available datasets can been primary divided into three main sets: acquired by ambient sensors, acquired by wearable devices, and a combination of the two. Recently, a lot of attention has been paid to wearable sensors because they are less intrusive, work outdoors, and often cheaper than the ambient ones. This is confirmed by the increasing number of techniques that are based on wearable sensors (see for example the survey by Luque et al. related to fall detection techniques relying on data from smartphones [18]).

Wearable sensors are divided in two main groups: ad-hoc wearable devices (e.g., SHIMMER sensor nodes), and smartphones (e.g., Android). Concerning fall detection, several studies concluded that, in order to be used, fall detection devices must not stigmatize people nor disturb their daily life [19,20,21]. Unfortunately, devices such as ad-hoc wearable devices and ambient sensors are not well accepted by elderly people because mostly of their intrusiveness. On the contrary, smartphones are good candidate devices for hosting fall detection systems: they are are widespread and daily used by a very large number of person, included elderly people. This, on the one hand, reduces costs, and on the other, eliminates the problem of having to learn how to use a new device. Moreover, studies demonstrated that samples from smartphones sensors (e.g., accelerometer and gyroscope) are accurate enough to be used in clinical domain, such as ADLs recognition [22]. This is also confirmed by the amount of publications that rely on the use of smartphones as acquisition devices for fall detection systems [18,23] and ADLs recognition.

For these reasons we concentrate our attention on smartphones as acquisition devices both for ADL recognition and fall detection. Thus, we searched the publicly available datasets acquired with smartphones in order to identify their strengths and weaknesses so as to outline an effective method for carrying out a new acquisition campaign. We searched the most common repository (IEEE, ACM Digital Library , Google, and Google Scholar) by using in our query the terms ADL dataset and Fall dataset in combination with the following words smartphone, acceleration, accelerometer, inertial, IMU (Inertial Measurement Unit), sensor, and wearable. We selected the first 100 results for each query. Removing duplicate entries, we obtained less then 200 different references. Then we manually examined the title, the abstract, and the introduction to eliminate references unrelated to ADL recognition and fall detection, and references that were based on ambient sensors such as camera, microphones, or RFID (Radio Frequency Identification) tags. We then read carefully the remaining references and discarded those that do not make publicly available the dataset used in the experimentation. Finally, we added the relevant references that we missed with our searches but were cited in the papers we selected. At the end of the process, we individuated 13 datasets with data from smartphones and 19 with data from wearable ad-hoc sensors. We then included only those datasets that have been recorded starting from 2012 (We considered the year in which the dataset has been made available. This year does not necessarily coincide with the year in which the related article has been published) mostly because the oldest dataset including samples from smartphones is dated 2012. This choice makes the datasets homogeneous with respect to the sensors technologies related to acquisition sensors which rapidly evolves year by year.

At the end of the process, we individuated 13 datasets with data from smartphones and 13 with data from wearable ad-hoc sensors. In the following, we will detail some relevant characteristics of the 13 datasets from smartphones since our aim was to build a new dataset containing acceleration patters from smartphone able to complement the existing ones. As it will presented in Section 2, the datasets from ad-hoc wearable devices have be examined with the aim of identifying the most common ADLs and falls.

Table 1. The publicly available datasets containing samples from smartphones sensors.

Dataset	Year	ADLs	Falls	Nr. of Subjects	Gender		Age	Height	Weight
					Female	Male	(Years)	(cm)	(Kg)
DMPSBFD [24]	2015	yes	yes	5	-	-	-	-	-
Gravity [23]	2016	yes	yes	2	-	-	26–32	170–185	63–80
							29 ± 4.2	178 ± 10.6	71.5 ± 12
MobiFall [11]	2014	yes	yes	24	7	17	22–47	160–189	50–103
							27 ± 5	175 ± 7	76.4 ± 14.5
MobiAct [25]	2016	yes	yes	57	15	42	20–47	160–193	50–120
							25 ± 4	175 ± 4	76.6 ± 14.4
RealWorld (HAR) [26]	2016	yes	no	16	7	8	16–62	163–183	48–95
							32 ± 12	173 ± 7	74.1 ± 13.3
Shoaib PA [27]	2013	yes	no	4	0	4	25–30	-	-
							-
Shoaib SA [28]	2014	yes	no	10	0	10	25–30	-	-
							-
tFall [29]	2013	yes	yes	10	7	3	20–42	161–184	54–98
							31 ± 9	173 ± 1	69.2 ± 13.1
UCI HAR [30]	2012	yes	no	30	-	-	19–48	-	-
							-
UCI HAPT [31]	2015	yes	no	30	-	-	19–48	-	-
							-
UCI UIWADS [32]	2013	yes	no	22	-	-	-	-	-
							-
UMA Fall [33]	2016	yes	yes	17	6	11	14–55	155–195	50–93
							27 ± 10	172 ± 9	69.9 ± 12.3
WISDM [34]	2012	yes	no	29	-	-	-	-	-
UniMiB SHAR	2016	yes	yes	30	24	6	18–60	160–190	50–82
							27 ± 11	169 ± 7	64.4 ± 9.7

shows the publicly available datasets recorded by means of smartphones and their characteristics. Table 1 also includes the dataset we realized in the last row, in order to ease the comparison.

The total number of datasets decreases to 11 because MobiAct and UCI HAPT are updated versions of MobiFall and UCI HAR respectively. Thus, in the following we will refer to 11 datasets overall, discarding MobiFall and UCI HAR.

The 11 datasets have been recorded in the period 2012 to 2016 (column Year). Only 5 datasets out of 11 contain both falls (column Falls) and ADLs (column ADLs).

The average number of subjects for dataset is 18 (column Nr. of subjects). The datasets that specify the gender of the subjects (which are MobiAct, RealWorld (HAR), Shoaib PA, Shoaib SA, tFall, and UMA Fall) contain in mean 6 women and 13 men (columns Gender-Female and Gender-Male respectively).

DMPSBFD, UCI UIWADS, and WISDM do not specify the age of the subjects (column Age). In the remaining 8 datasets, subjects are aged between 21 and 43 on average with a standard deviation of 4 and 11 respectively.

Finally, only Gravity, MobiAct, RealWorld (HAR), tFall, and UMA Fall datasets provide detailed information about the height and the weight of the subjects (columns Height and Weight respectively).

The detailed information reported in Table 1 have been collected from the web site hosting the dataset, the readme files of each dataset, and the related papers. It is remarkable to notice that in many cases such information get lost in the downloaded dataset. Grey cells in Table 1 indicate that samples are stored so that they can be filtered according to the information contained in the cell.

For instance, in all the datasets, with the exception of tFall, it is possible to select subsets of samples according to the specific ADL (column ADLs). For example, it is possible to select all the samples that have been labeled walking. tFall is an exception because the samples are simply labeled as generic ADL, thus not specifying which specific kind of ADL are.

Concerning falls (column Falls), all the datasets have organized samples maintaining the information related to the specific type of fall they are related to (e.g., forward).

As specified in column Nr. of subjects, the samples are linked to the subjects that performed the related activities and, where provided, falls. This means that in all the datasets (with the exception of Shoaib PA) it is possible to select samples related to a specific subject. However, this information is unhelpful if there is no information on the physical characteristics of the subject. Looking at the double column Gender, only MobiAct, RealWorld (HAR), Shoaib PA, Shoaib SA, and UMA Fall maintain information related to the gender of the subject. Finally, it is surprising that only Gravity, MobiAct, RealWorld (HAR), and UMA Fall allow to select samples according to age, height, and/or weight of the subjects (columns Age, Height, and Weight).

In view of this analysis, only MobiAct, RealWorld (HAR), and UMA Fall allow us to select samples according to several dimensions, such as the age, the gender, the weight of the subjects, or the type of ADL. MobiAct and UMA Fall allow us to select samples also according to the type of fall. Unfortunately, the other datasets are not suitable in some experimental evaluations. For example, the evaluation of the effects of personalization in classification techniques [35] taking into account the physical characteristics of the subjects, that is, operating leave-one-subject-out cross-validation [36].

To further contribute to the worldwide collection of accelerometer patterns, in this paper we present a new dataset of smartphone accelerometer samples, named UniMiB SHAR (University of Milano Bicocca Smartphone-based Human Activity Recognition). The dataset was created with the aim of providing the scientific community with a new dataset of acceleration patterns captured by smartphones to be used as a common benchmark for the objective evaluation of both ADLs recognition and fall detection techniques.

The dataset has been designed keeping in mind on one side the limitations of the actual publicly available datasets, and on the other the characteristics of MobiAct, RealWorld (HAR), and UMA Fall, so to create a new dataset that juxtaposes and complements MobiAct, RealWorld (HAR), and UMA Fall with respect to the data that is missing. Thus, such a dataset would have to contain a large number of subjects (more than the 18 on average), with a large number of women (to compensate MobiAct, RealWorld (HAR), and UMA Fall), with subjects over the age of 55 (to extend the range of UMA Fall (We do not consider RealWorld (HAR) since it contains ADLs only. Indeed, it is most difficult to recruit elderly subjects performing falls), with different physical characteristics (to maintain heterogeneity), performing a wide number of both ADLs and falls (to be suitable in several contexts). Moreover, the dataset would have to contain all the information required to select subjects or ADLs and falls according to different criteria, such as for example, all the female whose height is in the range 160–168 cm, all the men whose weight is in the range 80–110 Kg, all the walking activities of the subjects whose age is in the range 45–60 years.

To fulfil those requirements, we built a dataset including 9 different types of ADLs and 8 different types of falls. The dataset contains a total of 11,771 samples describing both activities of daily living (7579) and falls (4192) performed by 30 subjects, mostly females (24), of ages ranging from 18 to 60 years. Each sample is a vector of 151 accelerometer values for each axis. Each accelerometer entry in the dataset maintains the information about the subject that generated it. Moreover, each accelerometer entry has been labeled by specifying the type of ADL (e.g., walking, sitting, or standing) or the type of fall (e.g., forward, syncope, or backward).

We benchmarked the dataset by performing several experiments. We evaluated four classifiers: k-Nearest Neighbour (k-NN), Support Vector Machines (SVM), Artificial Neural Networks (ANN), and Random Forest (RF). Raw data and magnitudo have been considered as feature vectors. Finally, for each classification we performed a 5-fold cross validation and a leave-one-subject-out evaluation. Results show how much the proposed dataset is challenging with respect to a set of classification tasks.

The article is organized as follows. Section 2 describes the method used to build the datasets. Section 3 presents the dataset evaluation and Section 4 discusses the results of the evaluation. Finally, Section 5 provides final remarks.

2. Dataset Description

This section describes the method used to acquire and pre-process samples in order to produce the UniMiB SHAR dataset.

2.1. Data Acquisition

The smartphone used in the experiments was a Samsung Galaxy Nexus I9250 with the Android OS version 5.1.1 and equipped with a Bosh BMA220 acceleration sensor. This sensor is a triaxial low-g acceleration sensor. It allows measurements of acceleration in three perpendicular axes, and allows acceleration ranges from $\pm 2g$ to $\pm 16g$ and sampling rates from 1 KHz to 32 Hz. The Android OS both limits to $\pm 2g$ with a resolution of $0.004g$ the acceleration range, and takes samples at a maximum frequency of 50 Hz. However, the Android OS does not guarantee any consistency between the requested and the effective sampling rate. Indeed, the acquisition rate usually fluctuates during the acquisition. For the experiments presented in this paper, we resampled the signal in order to have a constant sampling rate of 50 Hz, which is commonly used in literature for activity recognition from data acquired through smartphones [28,29,30]. The accelerometer signal is for each time instant made of a triplet of numbers (x, y, z) that represents the accelerations along each of the 3 Cartesian axes.

We used also the smartphone built-in microphone to record audio signals with a sample frequency of 8000 Hz, which are used during the data annotation process.

The subjects were asked to place the smartphone in their front trouser pockets: half of the time in the left one and the remaining time in the right one.

Acceleration triples and corresponding audio signals have been recorded using a mobile application specially designed and implemented by the authors, which stores data into two separated files inside the memory of the smartphone.

2.2. ADLs and Falls

In order to select both the ADLs and the falls, we analyzed the datasets in Table 1 and the most recent publicly available datasets recorded with wearable ad-hoc devices. As discussed in Section 1, we considered the datasets acquired from 2012 to be compliant with the year of the older smartphone-based dataset. This set includes, sorted by year of creation from the oldest the most recent, the following datasets: DLR v2 [37], Ugulino [38], USC HAD [39], DaLiAc [10], EvAAL [40], MHEALTH [41], UCI ARSA [32], BaSA [42], UR Fall Detection [43], MMsys [9], SisFall [44], UMA Fall (UMA Fall contains samples from both smartphones and ad-hoc wearable devices.) [23], and REALDISP [45].

Regarding ADLs, Figure 1 shows the most common ones in the overall 24 datasets we analyzed (11 with samples from smartphones and sketched in Table 1 and 13 with samples from wearable ad-hoc devices listed above). The y axis represents the number of datasets that include the specified ADL. ADLs are grouped by category. The following categories have been identified by analizyng the datasets: Context-related, which includes activities that someway deal with the context (e.g., Stepping in a car), Motion-related, which includes activities that imply some kind of physical movement (e.g., Walking), Posture-related, which includes activities in which the person maintains the position for a certain amount of time (e.g., Standing), Sport-related, which includes any kind of activity that requires a physical effort (e.g., Jumping), and Others, which includes activities that are presented in one dataset only (e.g., Vacuuming in category Housekeeping-related). The Jogging and Running activities deserve a clarification. In all the datasets we analyzed, they are mutually exclusive, that is, datasets that contain Running, do not contain Jogging and vice versa. The datasets REALDISP and MHEALT are an exception because they include both the activities. These datasets, besides being realized by the same institution, are primarily oriented towards the recognition of physical activities (warm up, cool down and fitness exercises). Moreover, none of the datasets analyzed exactly specify what the Jogging and Running activities are related to. Thus, even though they may be considered very similar activities, we have decided to keep them separated in oder to do not loose their specificity. We classify Jogging as a sport-related activity (in the sense, for instance, of jogging in the park), and Running as a motion-related activity (in the sense, for instance, of running for the bus). For each category, the x axis shows all the ADLs we found and that are present in at least 2 datasets. Under the label Others fall all the ADLs for the corresponding category that have been included in one dataset only (e.g., Walking left-circle in category Motion-related).

Table 2. ADLs performed by the subjects In the UniMiB SHAR dataset.

Category	Name	Description	Label
Motion-related	Standing up from laying	From laying on the bed to standing	StandingUpFL
	Lying down from standing	From standing to lying on a bed	LyingDownFS
	Standing up from sitting	From standing to sitting on a chair	StandingUpFS
	Running	Moderate running	Running
	Sitting down	From standing to sitting on a chair	SittingDown
	Going downstairs	Climb the stairs moderately	GoingDownS
	Going upstairs	Down the stairs moderately	GoingUpS
	Walking	Normal walking	Walking
Sport-related	Jumping	Continuos jumping	Jumping

shows the 9 ADLs we have selected among the most popular included in the analyzed publicly available datasets. UniMiB SHAR includes the top 5 most popular Motion-related activities (i.e., Walking, Going upstairs, Going downstairs, Sitting down, and Running). Moreover, we detailed the generic Standing up, by including the Standing up from sitting and Standing up from laying activities. Finally, we included also the Lying down from standing.

In the Sport-related category, we did not included Jogging even if it is the most popular activity in its category because we included the Running activity in the Motion-related category. In Sport-related category, we chose the Jumping activity being the second most popular one.

Our dataset does not include Postural-related activities. Indeed, we were interested in acquiring acceleration data from activities related to movements both because from them it is possible to estimate the overall physical activity performed by a person, and because people are more likely to fall during movements [46].

We do not include ADLs belonging to categories such as Housekeeping-, Cooking-, or Personal care-related (those fall in the Others category in Table 1, because we are interested in low order activities of daily living, which include simple activities such as, Standing, Sitting down, Walking, rather than high order activities of daily living, which include complex activities such as, Washing dishes, Combing hair, Preparing a sandwich. The same holds for contex-related activities that are intended as high order activities. This choice was also motivated by the fact that these activities are scarcely present in the analyzed datasets (in particular, each activity belonging in the above mentioned categories is present in only one of the 24 analyzed datasets).

Finally, among the ADLs related to movements, we selected the ADLs most used in literature as demonstrated by the analysis we performed, which is also confirmed by Pannurat et al. in [47].

Regarding falls, we analized DMPSBFD, Gravity, MobiAct, tFall, and UMAFall datasets from smartphones (see Table 1), and DLR v2, EvAAL, MMsys, SISFall, UMA Fall, and UR Fall Detection datasets from wearable ad-hoc devices, since they are the only datasets that contain falls. From this set, we excluded DLR v2, EvAAL because they do not specify the type of fall.

Figure 2 shows the most common falls in the resulting 9 datasets we analyzed. The y axis represents the number of datasets that include the specified fall. Likewise ADLs, falls are grouped by category. Falling backward, Falling forward, and Falling sideward include back-, front-, and side-ward falls respectively. Sliding category can be further specialized so that to include Sliding from a chair, Sliding form a bed, and Generic sliding that not specifies details about the type of sliding. Finally, the category Specific fall includes different type of falls that have not been further specialized.

For each category, the x axis shows all the types of falls we found. The label Others includes all the falls for the corresponding category that have been included in one dataset only. The Specific fall category is an exception since it includes falls types not particularly present in the analyzed datasets.

Choosing which falls to include in the dataset was driven by the following considerations: the number of falls should have been comparable to that of the other datasets, and the dataset should have included a set of representative types of falls. Thus, having four categories (not considering Sliding, which includes only one type of fall that has been considered by two datasets only), we selected two falls from each of them. In each category, we selected the first two most popular falls. The category Falling sideward is an exception since we preferred to choose the two most specific falls instead of including the too generic Generic falling sideward.

Table 3. Falls performed by the subjects in teh UniMiB SHAR dataset.

Category	Name	Description	Label
Falling backward	Falling backward-sitting-chair	Fall backward while trying to sit on a chair	FallingBackSC
	Generic falling backward	Generic fall backward from standing	FallingBack
Falling forward	Falling with protection strategies	Falls using compensation strategies to prevent the impact	FallingWithPS
	Generic falling forward	Fall forward from standing, use of hands to dampen fall	FallingForw
Falling sideward	Falling rightward	Fall right from standing	FallingLeft
	Falling leftward	Fall right from standing	FallingRight
Specific fall	Hitting an obstacle in the fall	Falls with contact to an obstacle before hitting the ground	HittingObstacle
	Syncope	Getting unconscious	Syncope

shows the 8 falls that we selected according the adopted criterion.

Finally, studies on this topic confirm the falls we selected are common in real-life [29,48,49,50].

2.3. Subjects

Thirty healthy subjects have been involved in the experiments: 24 were women and 6 men. The subjects, whose data are shown in

Table 4. The characteristics of the subjects.

	Total	Female	Male
Subjects	30	24	6
age	18–60	18–55	20–60	min–max
	27 ± 11	24 ± 9	36 ± 15	mean ± std
height	160–190	160–172	170–190	min–max
	169 ± 7	166 ± 4	179 ± 6	mean ± std
weight	50–82	50–78	55–82	min–max
	64.4 ± 9.7	61.9 ± 7.8	74.7 ± 9.7	mean ± std

, are aged between 18 and 60 years (27 ± 12 years), have a body mass between 50 and 82 kg (64.4 ± 9.7 kg), and a height between 160 and 190 cm (169 ± 7 cm). Note that we included more women and older ages to compensate for the lack of MobiAct.

All the subjects performed both ADLs and Falls. The subjects gave written informed consent and the study was conducted in accordance with the WMA (World Medical Association) Declaration of Helsinki [51].

2.4. Protocols

To simplify the data annotation process, we asked each subject to clap her hands early before and after she performed the activity/fall to be recorded. Moreover, to reduce background noise, we asked each subject to wear gym trousers with front pockets.

Concerning ADLs, in order to avoid mistakes by the subjects due to too long sequences of activities, registrations have been subdivided in the three protocols showed in

Table 5. The protocols for ADLs aquisition.

Protocol	Action	Iteration
Protocol 1	Start the registration	1 time
	Put the smartphone in the pocket
	clap
	Walking for 30 s
	clap
	Running for 30 s
	clap
	Pull the smartphone from the pocket
	Stop the registration
Protocol 2	Start the registration	1 time
	Put the smartphone in the pocket
	clap
	Climb 15 steps
	clap
	Go down 15 steps
	clap
	Wait 2 s
	clap
	Jump 5 times
	clap
	Pull the smartphone from the pocket
	Stop the registration
Protocol 3	Start the registration	1 time
	Put the smartphone in the pocket
	clap	5 times
	Sitting down
	clap
	Standing up from sitting
	clap
	Wait 2 s
	clap	5 times
	Lying down from standing
	clap
	Standing up from laying
	clap
	Wait 2 s
	Pull the smartphone from the pocket	1 time
	Stop the registration

. Each protocol has been performed by each subject twice, the first one with the smartphone in the right pocket and the second in the left. Those smartphone positions were chosen because both they are the most natural ones and they are exactly the positions used in the analyzed references dealing with smartphones.

Protocol 1 includes Walking and Running activities. We opted for moderate walking and running so as to include even older people. Protocol 2 includes activities related to both climbing and descending stairs, and jumps. In our registration, we selected straight stairs ramps, and asked each volunteer to perform jumps with a moderate elevation, with little effort, and spaced each other about 2 s. Protocol 3 includes ascending and descending activities. The Sitting down and Standing up from sitting activities have been performed with a chair without armrests; the Lying down from standing and Standing up from laying have been performed on a sofa. The duration of the actives are on average similar to those reported in [52] that reviews a set of public datasets for wearable fall detection systems.

Falls have been recorded individually, always following the pattern of making a start and end clap (see

Table 6. Protocol for each fall.

Action	Iteration
Start the registration	6 times
Put the smartphone in the pocket
clap
fall
clap
Pull the smartphone from the pocket
Stop the registration

). In cases where the volunteer ended in a prone position, the clap has been performed by an external subject to avoid as far as possible any movements that might lead to recording events outside the study. To carry out the simulation safely, a mattress of about 15 cm in height was used. Each fall was repeated six times, the first three with the smartphone in the right pocket, the others in the left. Finally, falls have been simulated, started from a standing straight up position, and self-started.

2.5. Segmentation and Preprocessing

The audio files helped in the identification of the start and stop time instants for each recorded activity. From the labelled recorded accelerometer data, we extracted a signal window of 3 s each time a peak was found, that is, when the following conditions were verified:

• the magnitude of the signal $m_t$ at time t was higher than $1.5g$, with g being the gravitational acceleration;
• the magnitude $m_{t-1}$ at the previous time instant $t-1$ was lower than $1.5g$.

Each signal window of 3 s was centered around each peak and it is likely that several overlaps between subsequent windows may happen. We adopted this segmentation technique instead of selecting overlapped sliding windows because our dataset is mostly focused on motion-related recognition of ADLs and falls. The choice of taking 3 s window has been motivated by: (i) the cadence of an average person walking is within 90–130 steps/min [30,53]; (ii) at least a full walking cycle (two steps) is preferred on each window sample.

Figure 3 shows samples of acceleration shapes. For each activity, we displayed the average magnitude shape obtained by averaging all the subjects’ shapes.

Since the device used for data acquisition records accelerometer data with a sample frequency of 50 Hz, for each activity, the accelerometer data vector is made of 3 vectors of 151 values (a vector of size 1 × 453), one for each acceleration direction. The dataset is thus composed of 11,771 samples describing both ADLs (7759) and falls (4192) not equally distributed across activity types. This is because the activity of running and walking were performed by subjects for a time longer than the time spent for other activities. Originally, 6000 time windows of the running activity were found. In order to make the final dataset as much as balanced, we have deleted about 4000 samples related to running activities. The resulting samples distribution is plotted in Figure 4, where the samples related to running activities are about 2000. On our web site we release both datasets, the one balanced and the original one.

We preprocessed the acceleration signal $s\left(t\right)$ in order to remove the gravitational component $g\left(t\right)$. Since the gravitational force is assumed to have only low frequency components, we applied a Butterworth ($BW$) low-pass filter with a cut off frequency of 0.3 Hz [30]: $g\left(t\right)=BW\left(s\right(t),0.3)$. The accelerometer data without gravitational component is then obtained as: $\widetilde{s\left(t\right)}=s\left(t\right)-g\left(t\right)$. The dataset is freely downloadable at the following address: http://www.sal.disco.unimib.it/technologies/unimib-shar/.

3. Dataset Evaluation

We organized the accelerometer samples in order to evaluate four classification tasks:

• AF-17 contains 17 classes obtained by grouping all the 9 classes of ADLs and 8 classes of FALLs. This subset permits to evaluate the capability of the classifier to distinguish among different types of ADLs and FALLs;
• AF-2 contains 2 classes obtained by considering all the ADLs as one class and all the FALLs as one class. This subset permits to evaluate, whatever is the type of ADL or FALL, the classifier robustness in distinguishing between ADLs and FALLs;
• A-9 contains 9 classes obtained by considering all the 9 classes of ADLs. This subset permits to evaluate how much the classifier is capable to distinguish among different types of ADLs;
• F-8 contains 8 classes obtained by considering all the 8 classes of FALLs. This subset permits to evaluate how much the classifier is capable to distinguish among different types of FALLs.

We initially evaluated the classifiers by performing a traditional 5-fold cross-validation. This means that all the data have been randomly split into 5 folds. Each fold has been considered as test data and the remaining ones as training data. Results are computed by averaging the result obtained on each test fold. The folds have been obtained by applying the stratified random sampling that ensures samples of the same subject in both the test and the training folds. We refer to this evaluation also as subject-dependent.

To make the dataset evaluation independent from the effect of personalization, we conducted another evaluation by performing a leave-one-subject-out cross-validation. Each test fold is made of accelerometer samples of one user only, namely the test user, while the training folds contain accelerometer samples of all the other users except the samples of the test user. We refer to this evaluation also as subject-independent.

Previous studies have explored a wide range of feature vector representations for acceleration signals. Some examples are [48,54,55]: raw data, magnitude of the signal, frequency-domain features, signal moments, cepstral frequency, and other heuristic features.

Here we considered two feature vectors:

• raw data: the 453-dimensional patterns obtained by concatenating the 151 acceleration values recorded along each Cartesian direction;
• magnitude of the accelerometer signal, that is a feature vector of 151 values.

We experimented with four different classifiers:

• k-Nearest Neighbour (k-NN) with $k=1$;
• Support Vector Machines (SVM) with a radial basis kernel;
• Artificial Neural Networks (ANN). We set up a three-layers feed forward network with back propagation. The network architecture includes an input layer, a layer of hidden neurons and an output layer that includes a softmax function for class prediction. The number of hidden neurons n has been set in such a way that $n=\sqrt{m\times k}$, where m is the number of neurons in the input layer and k is the number of neurons in the output layer, namely the number of classes [56].
• Random Forest (RF): bootstrap-aggregated decision trees with 300 bagged classification trees.

All the classifiers have been implemented exploiting the MATLAB Statistics and Machine Learning Toolbox and the Neural Network Toolbox.

Evaluation Metrics

As shown in Figure 4, each of the 17 sets containing samples related to a specific activity is different in size. To cope with the class imbalance problem of the dataset we used as metric the macro average accuracy [57].

Given E the set of all the activities types, $a\in E$, $N{P}_a$ the number of times a occurs in the dataset, and $T{P}_a$ the number of times the activity a is recognized, MAA (Macro Average Accuracy) is defined by Equation (1).

$$MAA=\frac{1}{\left|E\right|}\sum _{a=1}^{\left|E\right|}Ac{c}_a=\frac{1}{\left|E\right|}\sum _{a=1}^{\left|E\right|}\frac{T{P}_a}{N{P}_a}.$$

(1)

MAA is the arithmetic average of the accuracy $Ac{c}_a$ of each activity. It allows each partial accuracy to contribute equally to the evaluation.

4. Results and Discussion

In the following, we discuss separately the results achieved with the traditional 5-fold cross-validation and the leave-subject-out cross-validation.

4.1. 5-Fold Evaluation

The k-fold evaluation is the most employed evaluation scheme in literature [58]. This evaluation considers a training set and a test set made of activity samples performed by all the human subjects. The resulting classifier is subject-dependent and usually exhibits a very high performance. Results of the k-fold evaluation (here we used k = 5) scheme are showed in

Table 7. Five-fold evaluation. Mean Average Accuracy for each classification task using raw data and magnitude as feature vectors. In bold are the best results for each classification task and feature vector employed.

5-fold
	Raw Data				Magnitude
Data	KNN	SVM	ANN	RF	KNN	SVM	ANN	RF
AF-17	82.86	78.75	56.06	81.48	65.30	65.71	41.95	65.96
AF-2	97.78	98.71	98.57	98.09	95.56	97.42	96.71	95.74
A-9	87.77	81.62	72.13	88.41	77.37	78.94	62.81	75.14
F-8	78.55	75.63	55.07	78.27	53.31	56.34	37.66	57.26

for raw data and magnitude. Overall, the performances achieved using raw data are better than the ones obtained using magnitude as a feature vector. This confirms a result already achieved in previous works [48,54].

The AF-17 recognition task is quite challenging with a MAA of about 83% in the case of raw data with KNN, and a MAA of about 66% in the case of magnitude with RF. This means that is quite difficult to distinguish among types of activities especially in the case when magnitude is adopted as feature vector. Figure 5 shows the confusion matrix of the k-NN experiment in the case of raw data.

The A-9 classification task is quite easy, the MAA obtained by raw data with RF is about 88% while the MAA obtained by magnitude with SVM is about 79%. This means that is quite easy to distinguish between types of activities. Looking at the confusion matrix in Figure 5, the most misclassified pairs of activities are Standing up from laying and Standing up from sitting, Lying down from standing and Sitting down, Going upstairs and Walking, Going downstairs and Walking, Jumping and Going downstairs.

The F-8 recognition task is quite challenging: the MAA is about 78% and 57% in the case of raw data with KNN and magnitude with RF respectively. This result suggests that distinguish among falls is very complicated The most misclassified pairs of falls are Falling with protection strategies and Generic falling forward, Syncope and Falling leftward, Generic falling backward and Falling backward-sitting-chair, Falling rightward and Falling with protection strategies, Falling rightward and Syncope.

In contrast, the AF-2 recognition task is very easy for all the classifiers and for both raw data and magnitude with an MAA of about 99% achieved with raw data and SVM. These results are similar to those obtained by previous researchers on a similar classification task performed on different datasets [29,54]. This means that it is very easy to distinguish between falls and no falls.

To summarize, the F-8 and AF-17 are quite challenging classification tasks. The use of this dataset for those tasks will permit researchers:

• to design and evaluate more robust feature representations as well as more robust classification schemes for human activity recognition.
• to study more robust features to deal with accelerometer samples of different types of falls.

4.2. Leave-One-Subject-Out Evaluation

Table 8. Leave-one-subject-out. Mean Average Accuracy for each classification task using raw data and magnitude of the signal as feature vectors. In bold the best results for each classification task and feature vector employed.

Leave-One-Subject-Out
	Raw Data				Magnitude
Data	KNN	SVM	ANN	RF	KNN	SVM	ANN	RF
AF-17	52.14	55.15	48.00	56.53	52.14	55.09	48.00	56.58
AF-2	92.90	97.57	95.41	97.02	92.90	97.57	96.07	97.05
A-9	63.79	63.32	63.63	73.17	63.79	63.36	63.63	72.67
F-8	43.66	48.84	38.50	45.88	43.66	49.35	38.50	45.26

shows the results obtained by performing the leave-one-subject-out evaluation. In this case the training set is made of activity samples of subjects not included in the test set. This evaluation is also known as subject independent evaluation and shows the feasibility of a real smartphone application for human activity recognition [35,59,60] where data of a given subject are usually not included in the training set of the classifier.

From the results it is almost evident the drop of performances with respect to the case of 5-fold evaluation. Human subject performs activities in a different way and this influences the recognition accuracy especially when it is necessary to distinguish between fine grained types of activities, that is in the case of AF-17, A-9 and F8 recognition tasks. In particular, in the case of AF-17 the best MAA is 56.58% using RF and magnitude. In the case of A-9 the best MAA is 73.17% using RF and raw data. In the case of F-8 the best MAA is 49.35% using SVM and magnitude. In contrast, distinguishing between coarse grained activities, such as falls vs no falls, is quite easy with a MAA of 97.57% with SVM for both raw data and magnitude. Overall the magnitude feature vector performs slightly better than the case of raw data. This suggests that using the magnitude as a feature vector in the case of subject-independent evaluations could be more reliable than raw data.

The low performance achieved in the case of subject-independent evaluation permits the researcher to investigate the following issues:

• the study of a more robust feature vector that is able to reduce as much as possible the performance gap between the subject-dependent and subject-independent evaluation;
• the study of on-line learning classification schemes that permit us, with the use of a few subject-dependent data, to improve the performance as much as possible.

5. Conclusions

Almost all publicly available datasets from smartphones do not allow the selection of samples based on specific criteria related to the physical characteristics of subjects and the activities (and/or falls) they performed. Of the 11 datasets containing smartphone measurements, only MobiAct, RealWorld (HAR), and UMA Fall are the exception. These three datasets include more men than women. Considering only datasets that include falls (MobiAct and UMA Fall), the maximum age of the subjects is 47 years. Our goal was therefore to create a new dastaset that would be complementary to the more complete ones and that also include falls. The result is UniMiB SHAR dataset that includes 9 ADLs and 8 falls performed by 30 humans, mostly female, with a huge range of ages, from 18 to 60 years.

The classification results obtained on the proposed dataset showed that the raw data performs somewhat better than magnitude as feature vector in the case of subject-dependent evaluation, and, on the opposite, the magnitude performs quite better than raw data in the case of subject-independent evaluation. The classification of different types of activities is simpler than the classification of different types of falls. It is very easy to distinguish between falls and no falls for both raw data and magnitude. The subject-independent evaluation showed that recognition performance strongly depends on the subject data.

As future work we plan to include other alternative artificial neural network architectures such as Recurrent Neural Networks (RNNs) and Deep Convolutional Neural Networks (CNNs). An RNN, especially its variant built on Long short-term memory (LSTM) units, is well suited to model temporal dynamics. Deep Convolutional Neural Networks (CNNs) are a class of learnable architectures used in many domains such as image recognition, image annotation, image retrieval etc. The combination of a CNN and RNN has demonstrated to be very effective for activity recognition [61].

UniMiB SHAR dataset will permit researchers to study several issues, such as: (i) robust features to deal with falls; (ii) robust features and classification schemes to deal with personalization issues.

We are planning to carry out an evaluation of the state-of-the-art techniques for ADLs recognition on both UniMiB SHAR and all the publicly available datasets of accelerometer data from smartphone to have and objective comparison. Moreover, we have planned to make experimentation on personalization by using those datasets that include information about the characteristics of the subjects. We want to investigate whether the training set containing samples acquired by subjects with similar characteristics to the testing subject may result in a more effective classifier. Finally, we are planning to check if and how data from smartwatches and smartphones can jointly improve the performances of the classifiers. To this end, we are improving the data acquisition application used for UniMiB SHAR.