A Survey on Ambient Sensor-Based Abnormal Behaviour Detection for Elderly People in Healthcare

Abstract

With advances in machine learning and ambient sensors as well as the emergence of ambient assisted living (AAL), modeling humans’ abnormal behaviour patterns has become an important assistive technology for the rising elderly population in recent decades. Abnormal behaviour observed from daily activities can be an indicator of the consequences of a disease that the resident might suffer from or of the occurrence of a hazardous incident. Therefore, tracking daily life activities and detecting abnormal behaviour are significant in managing health conditions in a smart environment. This paper provides a comprehensive and in-depth review, focusing on the techniques that profile activities of daily living (ADL) and detect abnormal behaviour for healthcare. In particular, we discuss the definitions and examples of abnormal behaviour/activity in the healthcare of elderly people. We also describe the public ground-truth datasets along with approaches applied to produce synthetic data when no real-world data are available. We identify and describe the key facets of abnormal behaviour detection in a smart environment, with a particular focus on the ambient sensor types, datasets, data representations, conventional and deep learning-based abnormal behaviour detection methods. Finally, the survey discusses the challenges and open questions, which would be beneficial for researchers in the field to address.

1. Introduction

1.1. Background

The share of the global population aged 65 years or above is predicted to grow from 10 percent in 2022 to 16 percent in 2050 [1]. These numbers imply a massive demand for healthcare, putting more pressure on health systems. Unsurprisingly, senior people prefer to live in a self-determined private home environment while ageing. Age-in-place could help improve the quality of elderly people’s daily life [2]. Hence, tracking daily life activities in a smart environment (SE) and detecting abnormal behaviour arising from health conditions can help to monitor the health conditions of elderly people [3,4,5].

An SE is a dwelling that is mounted with ambient sensors to enable monitoring of the occupants, capturing their behaviour and understanding their activities by using an array of different sensors such as motion, door switch, temperature, pressure, pulse rate, or blood glucose sensors [6]. In this way, ambient intelligence can predict normal or abnormal behaviour and inform about risky situations requiring further assistance or interference. The application areas of ambient intelligence involve fall detection, detection of dementia indicators, normal or abnormal behaviour (physical) detection, well-being monitoring, etc. [3,7,8,9,10,11,12].

Abnormal behaviour detection is the task of identifying daily living activities whose execution deviates from the expected or normal execution due to the health problems that elderly people may have (such as cognitive impairment). For example, people with dementia might have difficulties performing daily activities and cannot lead an independent life. An SE can be exploited to track the daily lives of elderly people and detect abnormal behaviour stemming from cognitive decline [3,4,5,13,14].

Given the importance of elderly healthcare in an SE, especially for people with cognitive impairment, providing a summary of the literature works and identifying the corresponding gaps and challenges within the field would be beneficial for researchers in this field. Henceforth, in this paper, we analyse the recent trends in abnormal behaviour detection in an SE, primarily focusing on healthcare applications using ambient sensors, such as item, door, motion sensors, etc.

Our survey goes beyond the prior research surveys [15,16,17,18,19,20,21,22] by: (1) focusing on healthcare applications for elderly people; (2) providing an updated assessment of the literature; (3) covering available simulation datasets as well as real-world datasets; and (4) identifying the research gaps in an SE for healthcare.

1.2. Recent Surveys on Abnormal Behaviour Detection

Table 1. Related surveys on abnormal behaviour detection.

Main Focus	Reference
Technologies and methodologies on indoor and outdoor context, based on multi-modality sensors	Cicirelli et al. [22]
Context-aware computing in healthcare for the elderly	Mshali et al. [17]
Abnormal behaviour detection with wearable and ambient sensors	Lentzas et al. [20]
Sensors, data, analysis, algorithms, reminder system and anomaly activity detection	Bakar et al. [16]
2D and 3D AbHAR based on RGB, skeleton, and depth	Dhiman et al. [18]
Sensors and communication platforms, along with artificial intelligence techniques, used for modeling and recognising activities	Amiribesheli et al. [15]
Activity recognition methods, but still covers abnormal behaviour detection methods	Patel et al. [19]
Abnormal behaviour identification for elderly care, using dense-sensing networks	Deep et al. [21]
Infrastructure systems and sensor technologies, activity recognition, and anomaly detection techniques	Dunne et al. [23]

summarises the earlier surveys on healthcare and their focuses. In [17], smart monitoring systems, their design and technologies, modeling, and development challenges were reviewed. Nevertheless, it does not address the approaches to detecting abnormal behaviour, the issues in dataset collection and the generation of artificial datasets when necessary. Dhiman et al. [18] provide a structured overview of abnormal human activity recognition (AbHAR) methods, with a focus on 2D and 3D AbHAR based on RGB, depth and skeleton evidence, for single-person and multiple-person-based methodologies. They do not discuss healthcare-related abnormal behaviour in the sensor-based environment.

In [24], outlier detection methods for temporal data are presented and summarised. Whilst the survey does not focus on abnormal activity recognition for an SE. In [16], the focus is on activity recognition in sensor-based data. The study categorises the anomaly detection methods in two ways: profiling and discriminating. The former learns normal behaviour and detects anomalies, and the latter learns anomalies from historical data and detects similar patterns from incoming data. Deep et al. [21] present a dense sensing network-based anomaly detection. The advantages and disadvantages are investigated, along with an overview of sensor fusion technology. In [19], the authors review sensors, data, analysis, algorithms, prompting reminder systems, and anomaly activity detection. The survey in [15] summarises ambient technologies, the sensors used, communication platforms, modeling techniques, and activity recognition methods, but it does not discuss the sensor-based abnormal behaviour detection for healthcare in an SE.

In a recent survey [23], infrastructure systems, such as body area networks, dense/mesh sensor networks, and microelectromechanical system (MEMS) sensors, are presented. Moreover, activity recognition and anomaly detection techniques for healthcare purposes are discussed. In addition, ethical issues such as anonymisation and privacy-preserving techniques are discussed, along with possible healthcare applications in an SE. Cicirelli et al. [22] provide a review on active and assisted living, covering possible application contexts for elderly people. They focus on different technologies and compare their pros and cons, including ambient sensors, smart everyday objects, wearable sensors, and socially assistive robots. They also discuss the different methodologies used for data processing, considering the context.

Overall, some of the studies in Table 1 skip a discussion about different definitions of abnormal behaviour for different tasks and contexts, while some others miss a discussion about the publicly available datasets and methods to artificially generate synthetic data reflecting abnormal behaviour. Thus, our survey will consider the healthcare domain, focusing on the recent methods as well as the issues in datasets.

1.3. Overview of the Survey

The references in this survey are obtained in the following way. First, the queries of “abnormal behaviour detection in an SE” and “activity recognition in a smart home” are run on Google Scholar, which covers most academic search engines such as IEEE, Springer, Elsevier, and ACM. Then, the papers are filtered based on their relations to the healthcare of elderly people and ambient sensors.

The organisation of our survey is shown in Figure 1. Section 2 discusses the different definitions of abnormal behaviour and categorises the purposes of abnormal behaviour detection. Section 3 presents different sensor types. Section 4 summarises the methods to segment sensor data and different ways to represent sensor readings in the format of the features. Different methods used to detect abnormal behaviour, including conventional machine learning and deep learning models, are presented and analysed in Section 5. The research gaps, challenges, and open questions are discussed in Section 6. Finally, Section 7 presents the conclusion of the paper.

2. Definition of Behaviour Changes in Elderly People

Changes of behaviour in the daily life routines of elderly people, can be indicators of some health-related problems, such as cognitive decline. Thus, understanding and defining these changes and providing a warning system for caregivers, is important. The definition of abnormal behaviour depends on the context of the research problems. For example, waking up and going to the toilet in the middle of the night might be an abnormal behaviour to predict dementia indicators [3,4,5,14,25], but can be regarded as normal behaviour for fall detection. Thus, defining abnormal behaviour for the given purposes would be more sensible. We provide a discussion about the definition of abnormal behaviour in the following, with a summary of the related work in

Table 2. Abnormal activity types.

Definition Rules	Abnormality Types	Reference
Time and location	Temporal	Paudel et al. [26]
	Spatial
	Behavioural
	Activity in unusual time	Novák et al. [27]
	Unusually long activity
	Unusually short activity
Deviation	Temporal deviation	Lundström et al. [9]
	Transitional deviation
	Transitional and spatial deviation
	Spatial, temporal, and transitional deviation
	Usual behaviour at a deviating time	Tran et al. [28]
	Anomaly in intensity or intra-variations in sensor firings
	Behaviour with a deviating duration
	Behaviour with a deviating spatial context
Behaviour or order changes	Point	Chandola et al. [29]
	Collective
	Contextual
	Order-based	Suresh et al. [12]
	Duration-based abnormal behaviour

.

A number of studies define abnormal behaviour based on the time and location where the behaviour occurs. In [26], three types of anomalies are defined, i.e., temporal, spatial, and behavioural anomalies. A temporal anomaly can be found in the duration of an activity that takes longer than usual or shorter, while spatial anomalies are ones which are performed in different places to usual. For instance, sleeping in the living room at night can be a spatial anomaly. On the other hand, a behavioural anomaly occurs in the sequence of sub-activities, when the person performs the activity in a way different to their usual patterns. The authors in [27] classify the anomalies based on the time spent on the activity as follows: activities at unusual times, e.g., rather than sleeping, the resident is spending time in the living room or kitchen during the night; unusually long activities, e.g., the resident spends more time on an activity than usual, which can be an indicator of falling off; unusually short activities, e.g., the user sleeps less at night, wakes up earlier than usual, or wakes up much earlier than usual, can suggest a health problem.

Some researchers define the abnormal activities according to deviations. For example, they are classified as in [9]: (1) temporal deviation, the activities take time at an unusual time, such as having breakfast at night; (2) transitional deviation, the activity is performed earlier or later than usual, such as getting up or leaving home at night; (3) transitional and spatial deviation, when the activity is performed at an unusual location but an unusual time, such as falling in the bathroom at night; (4) spatial, temporal and transitional deviation, the time and location of the activity are unusual while the activity itself is normal, such as watching television at night. In [28], the abnormalities are identified as: (1) known behaviour at a deviating time (e.g., waking up, getting dressed, eating, and leaving home); (2) changes in the intensity or intra-variations in sensor firings during an activity (e.g., falling in the bedroom going to the bathroom); (3) behaviour with a deviating duration (e.g., spending a longer time in the bathroom); (4) behaviour with a deviating spatial context (sleeping in the living room during the night).

For some applications, the completion of the activity might be enough, but the ordering of sub-activities could be important for the detection of indicators of dementia [3,5,14]. Certain authors classify the anomalies on the basis of the relationship of the activity with others, or changes to the order of activities [28]. There are three anomalies defined in [29]: point, collective, and contextual anomalies. Point anomalies consider each activity independent from the other and it does not depend on time or space. On the other hand, in collective anomalies, activities are considered dependent on each other. Contextual anomalies, such as time, visitors, or medications, are defined under some context. We can also consider sequential abnormal activities as a collective anomaly, which is defined as a sequence of events. An activity can be normal alone but abnormal when they come together with other events. For example, going to the toilet is a typical and expected activity, but a frequent repetition in a short time may indicate something anomalous [25]. The authors in [12] classify abnormal behaviour into duration-based and order-based. The duration-based abnormal behaviour looks at the unusual amount of time spent performing a behaviour, e.g., the elderly spend too much time in the toilet during the night. The order-based abnormal behaviour determines the irregularities and deviations in the order of activities. E.g., the elderly forget to take their medicine at the scheduled time. The abnormality detection, based on the above cases, can enable the early detection of risk related to specific health issues and enhance the elderly performing their daily routines independently.

Other abnormal activities, such as not opening the refrigerator all day or turning on the bathwater but not turning it off, can be risky in terms of the health of a resident and the caregiver should be notified [30]. In [3,5,7], abnormal behaviour is described in the context of early indicators of dementia. For instance, an elderly person suffering from dementia may forget to have dinner or have multiple dinners, as they have forgotten they have already eaten. On the other hand, not having dinner might be normal behaviour for a young person. In addition, we can see from Table 2 that the definition of abnormal behaviour can be interwoven, e.g., in [26], the definition involves both time-based and order-based rules. Hence, the definition of abnormal behaviour differs from context to context.

3. Ambient Sensors and Datasets

Figure 2 presents a typical pipeline of ambient sensor-based abnormal behaviour detection. First, activity-related data from the ambient sensors deployed in a house are acquired. The sensor measurements are then sliced into window chunks and mapped in different ways to attributes in the preprocessing stage. For conventional behaviour detection methods, the extracted features from the preprocessed data, including time, duration, frequency, or the quality of the activity being performed, are required for modeling. As for the deep learning-based methods, they can automatically learn the features instead. The abnormal behaviour detected can be used in the decision making for caregivers, community centres, or hospitals to trigger risk alarms, early diagnosis, and so on.

3.1. Sensor Types

In an SE, profiling and tracking user activities via cameras introduces user privacy and ethical problems. Meanwhile, tracking activities with the help of wearable sensors might be obtrusive for the residents or cause discomfort during long-term use [31]. One of the most popular ambient sensors used in an SE is passive infrared (PIR) sensors, also called motion sensors. Motion sensors are attached to a bed, a couch, around a table or the predefined places to track the resident’s motion [32,33]. Item sensors, another commonly-used sensor, are usually attached to kitchen utilities or medicine boxes to observe daily activities such as cooking or medicine usage [34,35].

A typical ambient sensor layout, named HomeSense [36,37], is displayed in Figure 3. HomeSense is a smart home project at the University of South Florida. It uses an ambient sensor network for health and wellness monitoring, to assist older adults living independently with routine healthcare. These deployed devices collect motion-related information, like the use of a microwave or TV, room entry/exit, electrical consumption, humidity, temperature, luminance, location of an occupant, toilet using, etc. It is noted that the number of sensing devices can vary along with the number of rooms in a real home; most smart environments deploy 16 to 20 sensing devices to cover $90\%$ of the home environment.

Sensor status is usually binary for motion, door, and item sensors, where $ON$ status is recorded as 1 and $OFF$ status is recorded as 0. These sensors give information about collecting objects’ status, such as doors opening or closing, lights turning on or off, or residents’ movement. Temperature sensors measure the temperature in an SE. Sensor activation information is recorded as a time tuple, where each sensor recording is a sequence of date, time, sensor identification number and status.

A range of ambient sensors that provide time-series measurements are used for behaviour detection; a summary of the related studies is provided in

Table 3. Commonly- used sensors in ambient sensor-based abnormal behaviour detection.

Sensor Types	Data Types	Reference
Pressure sensor	Numerical: continuous pressure measurements	[8,13,38,39]
Passive infrared (PIR) sensor	Categorical: binary	[9,26,32,33,37,38,40,41]
Temperature sensor	Numerical: continuous temperature measurements	[5,33,36,41,42]
Contact sensor	Categorical: binary	[8,13,35,37,39,40,43,44]
Humidity sensor	Numerical: continuous humidity measurements	[33,36,42,45]
Smoke, light, ultrasonic, water sensors, etc.	Categorical: binary	[13,36,44,45,46]
Item sensors	Categorical: binary	[30,34,39,47]
Motion sensors	Categorical: binary	[30,34,35,36,47]
RFID	Numerical: measurements of signal strength or frequency phase values	[48,49,50,51]
Radar	Categorical: binary	[52,53,54,55]
WiFi	Numerical: channel state information and signal strength	[56,57,58,59,60]
GPS	Numerical: speed, location, etc.	[61,62,63]

. Apart from the typical ambient sensors, we can also see WiFi, GPS, and RFID frequently used in an SE for behaviour detection. As a ubiquitous element in indoor environments, WiFi can deliver channel state information and signal strength signals for positioning, gestures, or other behaviours. RFID is also a commonly used technique, capable of automatically tracking and identifying the movement of humans based on the measurements of signal strength or frequency phase values received by the reader. The GPS modules built in smart devices are conveniently used for positioning. At the same time, they can also be used for behaviour inference in an SE by detecting the movment speed, the location, or the number of available satellites provided.

3.2. Real-World Data

This subsection summarises datasets that are constructed for tracking daily life activities. Although data collection is a tedious task and takes time, there are datasets [7,8,26,64] (see

Table 4. A summary of sensor-based SE datasets.

Dataset	Resident	Sensors	Publicly Available or Not
Dataset A,B,C[8]	(A) an autonomous and independent adult man, (B) an elderly woman diagnosed with Parkinson’s disease, (C) an autonomous, healthy elderly woman	Pressure mats, passive infrared sensors, contact switches for doors, cupboards, float sensors in the toilet, etc.	nPA
Domus [65]	A group of residents	Pressure detector, IR, door contacts, switch contacts, lamps, flow meter, etc.	PA, https://domus.recherche.usherbrooke.ca/ (accessed on 13 February 2023)
Kasteren [66]	26-year-old adult	Sensors on cupboards, doors, refrigerator, a toilet flush sensor, etc.	PA, https://www.uva.nl/~tlmkaste (accessed on 13 February 2023)
Kyoto [67]	400 residents	Infrared motion sensors, item, temperature, burner, magnetic door, hot and cold water sensors, etc.	PA, https://casas.wsu.edu/datasets/ (accessed on 13 February 2023)
Logan [68] Cook [34] Suresh [12]	Two married people	Motion, temperature, water, electrical energy sensors, etc.	PA, https://ailab.eecs.wsu.edu/casas (accessed on 13 February 2023)

PA: Publicly available; nPA: not publicly available.

) collected to monitor daily life activities in an SE. One of them is the MavHome (managing an intelligent versatile home) project [64], in which an SE that acts as an intelligent agent is designed. Sensors help perceive the state of the home, and the agent optimise the comfort and productivity of the residents.

The Kasteren [66] dataset is collected in 28 days from a house with 14 state-change sensors, where a 26-year-old man lives alone. In the house, digital sensors are placed on the cupboards, doors, refrigerator, etc. The activity labels are annotated using a blue-tooth headset and speech recognition software. In the dataset, there are 2120 sensor events, which are comprised of seven activities, such as using the toilet, leaving the house, sleeping, showering, preparing breakfast, preparing a beverage, and preparing dinner. This dataset covers a period of less than a month; thus it would not be sufficient for the application of deep learning methods that require a large training dataset. Conventional methods, such as SVMs, would be more applicable to this dataset.

In [8], three different datasets are collected. The first resident is an entirely autonomous adult man living independently; the second resident is an elderly female with Parkinson’s disease; the third resident is an autonomous elderly woman without life-threatening diseases. The sensors used include: pressure mats for measuring lying in bed or sitting on a couch, passive infrared sensors for detecting motions in a specific area, float sensors for measuring the toilet being flushed and contact switch sensors for the open–close status of cupboards and doors. In total, 12 sensors are used and the total number of days is 14, 25, and 21, respectively. Real data acquired from the sensors are analysed by experts; the abnormal sensor observations are manually labeled. The three datasets are suitable for detecting the abnormal behaviour of elderly people.

In the Domus dataset [65], six adults’ early morning routines (grooming, breakfast) are tracked. The data are collected in two different ways: (i) the user performs the early morning routine, and (ii) repeats the same routine with the introduction of a constraint. The dataset comprises 84 sequences during the grooming activity and 62 sequences gathered during the breakfast activity. The dataset involves five activities: waking up, using the toilet, preparing breakfast, having breakfast, and washing the dishes. In total, 36 binary sensors are used, such as lamps, PIR, switch contacts, pressure detectors, flow meters, and door contacts. The data were collected for resident identification.

In Placelab [69], research participants individually live in a smart apartment that contained a dining area, a bedroom, a living room, a small office and a kitchen. The apartment deployment facilitates data collection for multiple or individual inhabitants over a period of a couple of weeks. Conditions inside the apartment are detected using 10 humidity sensors, 34 distributed temperature sensors, one barometric pressure sensor and five light sensors. The PlaceLab dataset also featurs two gas flow meters, 37 electrical current sensors, and 11 water flow sensors. The PlaceLab dataset suits studies that focus on multi-week or multi-day observations of individuals living independently or as a couple.

The dataset Kyoto in [67] is collected from 400 participants by CASAS (Centre for Advanced Studies in Adaptive Systems) at Washington State University. The sensors used include item sensors for selected items in the kitchen, wide-area infrared motion sensors, temperature sensors, hot and cold water sensors, burner sensors and magnetic door sensors. Out of the 400 participants, there are 239 healthy participants and 3 participants previously diagnosed with cognitive impairment. In this dataset, temporal and spatial abnormal behaviour are included; other activities include dusting the living room, obtaining a set of medicines, filling the medicine dispenser, sweeping the kitchen, and so on.

Studies in [12,34,68] use the Tulum dataset to validate their work. The Tulum data is also from the CASAS smart home project, collected from two people living in a smart apartment that is instrumented with motion sensors, door switches, light sensors, temperature sensors, etc. However, different studies have selected different sensors and the associated activities from the dataset for their research. Apart from the Kyoto and Tulum datasets in CASAS, the Aruba dataset has also been frequently used in the tasks of ambient sensor-based behaviour detection [12,35,41]. Aruba consists of eleven kinds of activities, collected over seven months. A total of 40 sensors are used, including sensors attached to doors, motion sensors, and temperature sensors during data collection in a smart environment with a single elderly woman resident. It is worth noting that the samples from the same activity in the dataset have varied lengths of sensor firings, which makes detection tasks more challenging.

3.3. Artificial Data

Artificial data are sometimes created for comparing the performance of the anomaly detection methods, or as a complement, when real datasets are not available [70]. Synthetic data based on real datasets are generated in [8], following the equation $D*=D\pm \Delta$, where $\Delta$ is an offset, D is the true measurement, and $D*$ is the anomalous measurement. Using the mechanism above, the authors randomly introduced the synthetic anomalies into each dataset separately at the frequency of $5\%$.

The Domus dataset in [7] is modified in the following way. Given a sequence representing one typical day, the cycle “waking up, using the toilet” is initially reproduced to mimic an inhabitant who wakes up at midnight. For example, some sequences remove the events representing dish washing, some remove the events relating to breakfast, others remove the events relating to showers.

The synthetic behaviour in [27] is generated on the MavHome dataset [71], focusing on the times and locations of the events. For this purpose, the reference points that represent anomalies (e.g., firing in the bedroom at 4.00 a.m.) are generated manually. The synthetic data at a frequency of 5 min for both the start time and duration of an activity are generated based on the reference anomalies. The simulator is constructed as a Markov process, in which the probability of the next state is created only by the current state. The simulator moves forward through the pattern states and, among these transitions, generates noisy firings.

The authors in [72] use DSMC (direct simulation Monte Carlo) and HMM (hidden Markov model) to produce data for ADLs of an older resident’s behaviour. There are two levels in the simulator: the first level profiles the activities based on the movements, and the second level models the movement of each profile and the corresponding activity. Consequently, the simulator design comprises two phases. The first step simulates the duration of staying in a specific location and the sequential movements of the older resident; the second step ensures the simulated data is representative of the person’s behaviour.

In [73], the researchers alter the real-world dataset to synthetic health-related abnormal behaviour. Daily living activities, like waking up and sleeping, are chosen; the abnormal behaviour, like no exercise, frequent toilet visits and slept without dinner, are synthesised. In [9], more data are synthesised based on collected real data. The sensor events are modeled by the Poisson distribution and a combination of Markov chains to increase the data simulation realism. Nevertheless, in [9,73], the authors do not mention how the data synthesis is completed in detail. In [74], the authors modify a real-world dataset by converting the room’s occupation information into activities; they focus on eating, sleeping and walking, and samples are inserted manually.

The deviating behaviour [9] is detected through analysis of associations and data clustering between data vectors expressing adjacent time intervals and clusters. Three types of abnormal behaviour are generated: temporal, transitional, and spatial deviations. The temporal deviation anomaly is designed to test whether the activities are placed in clusters of appropriate activities, including getting dressed, waking up, preparing breakfast, getting out of bed and eating.

In [3,4,5,14], abnormal behaviour representing the early indicators of cognitive decline is generated by removing or repeating activity sequences or sensor readings from real activity sequences. For example, a sequence of sensor events representing eating activity is injected into sleeping activity, implying that the resident has a sleeping problem. The type of anomalies generated represents cognitive decline-related problems. Some studies that use artificial datasets are summarised in

Table 5. A summary of simulated abnormal behaviour.

Dataset	Reference	Application	Abnormality	Publicly Available or Not
Domus	Saives et al. [7]	Dementia evolution	Frequency	PA, https://domus.recherche.usherbrooke.ca/ (accessed on 13 February 2023)
Own data	Ordonez et al. [8]	Parkinson’s disease	Duration	nPA
Own data	Dahmen et al. [70]	General	Sensor events, sequence, time	nPA
MavHome	Novak et al. [27]	Elderly care	Time anomaly	PA, https://ailab.wsu.edu/mavhome/research.html (accessed on 13 February 2023)
Own data	Elbayoudi et al. [72]	Elderly care	Location anomaly	nPA
Own data	Lundstrom et al. [9]	Elderly healthcare	Temporal, transitional, spatial	nPA

PA: Publicly available; nPA: not publicly available.

.

4. Data Preprocessing and Sensor Representation

4.1. Sensor Data Segmentation

Different studies have been dedicated to addressing sensor data segmentation [66,75,76,77,78]. For example, sliding a time window over sequential sensor data is a popular method to deal with time dependency, as in Figure 4. An overlap can also be used in time windows, e.g., an overlap of $50\%$ is used to avoid information loss in [75]. There mainly exist two approaches to slide time windows: (i) using a static sliding window [66,75], where a window with a fixed length is used to slide over the sequence; or (ii) using a dynamic time window with varying window lengths [76,77,78]. For example, a window length of 1 min is used in [79]. The sensors are converted to a one-hot encoded feature representation, where the column length equals the number of sensors available in the SE.

The fixed time slices cause immense time complexity since the sliding window is applied even when no sensor is activated. In [80], a hidden Markov model is used to segment sequential sensor data via a variable time window. However, prior data and segmentation rules are required to decide the optimal time-window length in the work.

Another problem with sliding window-based segmentation is that sensor activation data and the correct activity labels are discretised using the same time-slice length. During the discretisation process, two or more activities occur within a single time slice. For example, an activity might end somewhere halfway through the time slice and another activity can start immediately after. In this case, the authors in [79] represent the time slice with the most dominant activity in the time window. On the other hand, the discretised correct labels might differ from the actual ground truth. The discretisation error is hence introduced to express the magnitude of this difference. The discretisation error represents the percentage of incorrect labels in the discretised ground truth.

4.2. Sensor Data Representation

For conventional detection methods, the raw sensor readings usually need to be mapped to features after a sliding window is applied to extract chunks of data. For this purpose, in earlier literature, a series of feature representations that are beneficial in identifying abnormal behaviour are introduced. One of the earlier sensor representations [79] (see Figure 5) has been explored widely in the literature.

• Raw sensor: in a given time slice, if a sensor is triggered, its representation is assigned to 1 and otherwise to 0 [4,5,79].
• Change-point: whenever a sensor changes its state, either going OFF from ON or vice versa, it is assigned to 1 and otherwise to 0.
• Last-fired: in this representation, the sensor that changed its state last is assigned to 1, and others are assigned to 0.

The raw features may fail to provide information about the activity being performed. The change point representation represents when a sensor changes state and thus indicates when an object is used. However, the location of the resident can be understood from the last-fired representation. Moreover, these features extracted from time-slice chunks represent sensor activation as a bag-of-words style model and ignore the relationship between sensors, such as their activation frequency and order. The abnormal activity recognition tasks, such as detecting early dementia, require more context-related feature representations, where the relation between sensor activation and their frequency is modeled better to understand the intrinsic substructures of activities. Thus, the studies [3,9,14,81,82] in

Table 6. Sensor representations.

Sensor Representation	Reference
Graph features represented as nodes and edges	Akter et al. [82]
Spatio-temporal features in a matrix	Lundstrom et al. [9]
Adjacency matrix	Twomey et al. [81]
Graph features	Arifoglu et al. [3]
Hierarchical features in an RAE tree	Arifoglu et al. [14]
Binary, change-point, last-fired	Kasteren et al. [79]
Activity image	Gochoo et al. [83]

model the sensors based on their intrinsic structures and their relationship with each other. The interaction between sensors and their order is more suitable for applications such as the detection of dementia indicators [3,14] that require a fine-grained level of detail about the sensors’ activation and their relations.

Instead, the authors in [83] use a deep convolutional neural network (DCNN) for activity recognition of the elderly living alone based on PIR motion and door binary sensors. The data are first split with varied sliding window sizes and then alerted into binary activity images that will be fed to their proposed DCNN model. The activity image is a 2D visual representation with a black background and white pixels, corresponding to each “ON” and “OFF” or “OPEN” and “CLOSE” of the binary sensory data from the motion and the door sensors, respectively. Figure 6 shows an activity image sample in [83] with a size of 50 × 40 converted from the segmented csv file, in which the two dimensions represent the temporal and intra-sensor patterns of the activities, respectively. The length equals the sliding window length L, while the height H is a parameter that needs to be determined.

5. Abnormal Behaviour Detection Methods

5.1. Methods Overview

Most abnormal behaviour (change) detection studies rely on the assumption that a person’s daily activities (ADLs) follow regular patterns. For example, it is assumed that people normally get up at approximately the same time in the morning or sleep for roughly the same amount of time every night. Based on this assumption, the existing studies try to detect abnormal activities by first modeling the regular patterns and then detecting the deviations from the regular ones. This section will cover the conventional machine learning models [40,84] and deep learning methods [85,86,87]. The conventional machine learning models include the generative [88], the probabilistic [89], the discriminative [90], the clustering [91], the graph-based [92] and the rule-based methods [93], etc. Please see Table 7 for a summary of the methods.

5.1.1. Generative Methods

A generative classifier learns the model to generate the data by estimating the assumptions and distributions of the model [73]. Models, like conditional random field and hidden Markov models, belong to this group.

Daily life activities occur in a temporal context following a sequential trend. Modeling activities based on temporal dependency makes detecting abnormal behaviour deviating from the cumulatively learned normal ones easy. For this purpose, sequence-based methods, such as hidden Markov models (HMMs) [38,73], are exploited. HMM generates hidden states from input data, assuming a Markov process with unobserved or missing states. The assumption involves event ordering, in which the probability of each event only relies on the previous adjacent event [94]. HMM is widely employed for anomaly detection, for it is a statistical method that works well on insufficient training data or small datasets. Sanchez et al. [38] use an HMM to model normal behaviour, including a person’s location, duration and posture. The HMM shows the ability to detect the deviation from the normal or usual pattern of the person, thereby predicting the abnormal behaviour in a person’s usual pattern.

Table 7. Typical studies in pervasive sensor-based abnormal activity detection.

Method	Target	Dataset (Publicly Available or Not)	Sensors	Performance	Ref. (Year)
Bayesian	Anomalous behaviour signs	Datasets consisting of raw observations of the activities of the corresponding user (nPA)	Pressure mats, passive infrared sensors, contact switches, installed in different home settings	Specificity: 0.98 Sensitivity: 0.73	[8] 2015
HMM	Abnormal behaviour from normal pattern of the person	An open dataset using binary sensors, gathered by two people living in their own homes (PA, https://archive.ics.uci.edu/ml/datasets/Activities+of+Daily+Living+28ADLs29+Recognition+Using+Binary+Sensors (accessed on 13 February 2023))	PIR sensors in rooms, magnetic sensors attached to the objects, pressure sensors under bed, seat, etc.	Accuracy: $72\%$	[38] 2019
Probabilistic spatio-temporal model	Anomalies that are different from the subject’s past pattern	Data collected from recruited senior subjects by using a commercial product (nPA)	Door, pressure and motion sensors mounted in different positions of the apartment	True positive rate for anomaly detection: 0.72	[13] 2016
Decision tree	Noisy patterns for older people	Dataset collected in a care home by thirteen senior residents during a long period (nPA)	Temperature sensor, humidity sensor, CO2 concentration sensor, PIR sensors, etc., installed in the environment	Accuracy for room occupancy: $98\%$ Accuracy for ventilation model: $93\%$	[33] 2019
Incremental decision tree	Activities of daily living (ADL)	MAVHome (PA, https://ailab.wsu.edu/mavhome/research.html (accessed on 13 February 2023))	Manually generated anomalies data	Accuracy: more than $90\%$	[95] 2021
Semantic rules	Anomaly from the normal behaviour	two public datasets, one dataset collected from one single-resident home, one dataset from a commercial provider (PA, https://ailab.eecs.wsu.edu/casas (accessed on 13 February 2023))	Floor presence mats, door and cabinet sensors, PIR sensors, chair and bed occupancy sensors, etc.	Reduces false positives and false negatives by at least $46\%$ and $27\%$, respectively	[43] 2015
SPARQL rules	Behaviours of daily living	Dataset including three defined activities collected twice per day and for 5 days (nPA)	Pressure, contact, ultrasonic sensors	Accuracy for getting dressed: $85\%$ Accuracy for taking shower: $90\%$ Accuracy for watching TV: $100\%$	[46] 2017
Association rules	Unexpected behaviour	Dataset from AGACY monitoring system (PA, http://hadaptic.telecom-sudparis.eu/ (accessed on 13 February 2023))	Smart lab equipped with beacons, motion sensors, thermometers, switches, etc.	Accurate and effective	[96] 2018
SVM	Abnormal conditions, like weakness, falls, altered mental status, etc.	Collected during approximately seven months from multiple subjects (nPA)	Infrared (IR) motion sensors	Positive predictive value: $90.5\%$	[32] 2011
Multivariate Gaussian via maximum likelihood estimation	Activities of daily living (ADL)	Twelve real-life activities from four subjects (nPA)	RFID sensor	Accuracy: $97.9\%$ Precision: $96.97\%$ Recall: $96.73\%$	[97] 2019
Graph-based approach	Temporal, spatial, behaviour anomaly	Kyoto dataset with 400 participants provided by Washington State University’s CASAS program (PA, https://ailab.eecs.wsu.edu/casas (accessed on 13 February 2023))	Infrared motion sensors, item sensor, burner sensor, hot and cold water sensor, etc., set up throughout the house	Anomalies are flagged in different scenarios	[26] 2018
CNN, autoencoder, convolutional autoencoder	Daily activities, including falling	1007 gait samples spanning 12 different classes collected from 11 subjects (nPA)	Radar	Convolutional autoencoder ranks best, with the overall accuracy of $94.2\%$	[26] 2018
NB, HMM, SVM, MLP, autoencoder, DBN, CNN, LSTM, etc.	Activities of daily living	Dataset from AGACY monitoring system (PA, http://hadaptic.telecom-sudparis.eu/ (accessed on 13 February 2023))	Binary environment sensors: reed switches, pressure mats, mercury contacts, passive infrared, float sensors, etc.	CNN1d obtains better results with or without NULL class	[39] 2020
Deep neural network	Health risk prediction	Patient’s physical data including EMR and PHR, and their environmental information including PHD and open API data (nPA)	Temperature, humidity, illumination, noise, position, date, time	Success in predicting patients’ risk of disease	[42] 2018
CNN, RNN	Abnormal behaviour related to dementia; activity patterns of elderly people with cognitive decline	Two datasets: Aruba and WSU, available publicly (PA, https://ailab.eecs.wsu.edu/casas (accessed on 13 February 2023))	Motion, door, and temperature sensors, etc.	Sensitivity: $98.67\%$ Sensitivity: $88.70\%$	[5] 2019
Deep belief networks (DBFs)	Behaviours and abnormalities within activities of daily living (ADLs) of the elderly	Dataset including a range of activities collected from the bathroom, bedroom, kitchen, and living room (nPA)	Motion sensor, door sensor, light sensor, smoke sensors, cameras, etc., installed inside living room, bedrooms, kitchen, etc.	F1 score: bedroom $89.1\%$, bathroom $90.1\%$, living room $80.8\%$, kitchen $83.6\%$	[44] 2020
Hybrid framework with 1D-CNN, 2D-CNN, and LSTM	Behaviours and abnormalities within activities of daily living	Data collected with FMCW radar, operating at 5.8 GHz C-band and 400 M bandwidth (nPA)	Frequency-modulated continuous wave (FMCW) radar	Overall accuracy: $93.391\%$ Accuracy of Falling: 100%	[55] 2021
Transformer with bidirectional GRU	Multi-resident ADLs	CASAS (PA, https://ailab.eecs.wsu.edu/casas), ARAS (PA, https://www.cmpe.boun.edu.tr/aras/ (accessed on 13 February 2023))	Temperature sensors, motion sensors, door sensors, PIR sensors, item sensors, cabinet sensors, water sensors, burner sensors, phone sensors, etc.	Accuracy: $91.44\%$ Precision: $90.64\%$ Recall: $91.15\%$ F-measure: $90.89\%$	[98] 2022

PA: Publicly available; nPA: not publicly available.

Table 7. Typical studies in pervasive sensor-based abnormal activity detection.

Method	Target	Dataset (Publicly Available or Not)	Sensors	Performance	Ref. (Year)
Bayesian	Anomalous behaviour signs	Datasets consisting of raw observations of the activities of the corresponding user (nPA)	Pressure mats, passive infrared sensors, contact switches, installed in different home settings	Specificity: 0.98 Sensitivity: 0.73	[8] 2015
HMM	Abnormal behaviour from normal pattern of the person	An open dataset using binary sensors, gathered by two people living in their own homes (PA, https://archive.ics.uci.edu/ml/datasets/Activities+of+Daily+Living+28ADLs29+Recognition+Using+Binary+Sensors (accessed on 13 February 2023))	PIR sensors in rooms, magnetic sensors attached to the objects, pressure sensors under bed, seat, etc.	Accuracy: $72\%$	[38] 2019
Probabilistic spatio-temporal model	Anomalies that are different from the subject’s past pattern	Data collected from recruited senior subjects by using a commercial product (nPA)	Door, pressure and motion sensors mounted in different positions of the apartment	True positive rate for anomaly detection: 0.72	[13] 2016
Decision tree	Noisy patterns for older people	Dataset collected in a care home by thirteen senior residents during a long period (nPA)	Temperature sensor, humidity sensor, CO2 concentration sensor, PIR sensors, etc., installed in the environment	Accuracy for room occupancy: $98\%$ Accuracy for ventilation model: $93\%$	[33] 2019
Incremental decision tree	Activities of daily living (ADL)	MAVHome (PA, https://ailab.wsu.edu/mavhome/research.html (accessed on 13 February 2023))	Manually generated anomalies data	Accuracy: more than $90\%$	[95] 2021
Semantic rules	Anomaly from the normal behaviour	two public datasets, one dataset collected from one single-resident home, one dataset from a commercial provider (PA, https://ailab.eecs.wsu.edu/casas (accessed on 13 February 2023))	Floor presence mats, door and cabinet sensors, PIR sensors, chair and bed occupancy sensors, etc.	Reduces false positives and false negatives by at least $46\%$ and $27\%$, respectively	[43] 2015
SPARQL rules	Behaviours of daily living	Dataset including three defined activities collected twice per day and for 5 days (nPA)	Pressure, contact, ultrasonic sensors	Accuracy for getting dressed: $85\%$ Accuracy for taking shower: $90\%$ Accuracy for watching TV: $100\%$	[46] 2017
Association rules	Unexpected behaviour	Dataset from AGACY monitoring system (PA, http://hadaptic.telecom-sudparis.eu/ (accessed on 13 February 2023))	Smart lab equipped with beacons, motion sensors, thermometers, switches, etc.	Accurate and effective	[96] 2018
SVM	Abnormal conditions, like weakness, falls, altered mental status, etc.	Collected during approximately seven months from multiple subjects (nPA)	Infrared (IR) motion sensors	Positive predictive value: $90.5\%$	[32] 2011
Multivariate Gaussian via maximum likelihood estimation	Activities of daily living (ADL)	Twelve real-life activities from four subjects (nPA)	RFID sensor	Accuracy: $97.9\%$ Precision: $96.97\%$ Recall: $96.73\%$	[97] 2019
Graph-based approach	Temporal, spatial, behaviour anomaly	Kyoto dataset with 400 participants provided by Washington State University’s CASAS program (PA, https://ailab.eecs.wsu.edu/casas (accessed on 13 February 2023))	Infrared motion sensors, item sensor, burner sensor, hot and cold water sensor, etc., set up throughout the house	Anomalies are flagged in different scenarios	[26] 2018
CNN, autoencoder, convolutional autoencoder	Daily activities, including falling	1007 gait samples spanning 12 different classes collected from 11 subjects (nPA)	Radar	Convolutional autoencoder ranks best, with the overall accuracy of $94.2\%$	[26] 2018
NB, HMM, SVM, MLP, autoencoder, DBN, CNN, LSTM, etc.	Activities of daily living	Dataset from AGACY monitoring system (PA, http://hadaptic.telecom-sudparis.eu/ (accessed on 13 February 2023))	Binary environment sensors: reed switches, pressure mats, mercury contacts, passive infrared, float sensors, etc.	CNN1d obtains better results with or without NULL class	[39] 2020
Deep neural network	Health risk prediction	Patient’s physical data including EMR and PHR, and their environmental information including PHD and open API data (nPA)	Temperature, humidity, illumination, noise, position, date, time	Success in predicting patients’ risk of disease	[42] 2018
CNN, RNN	Abnormal behaviour related to dementia; activity patterns of elderly people with cognitive decline	Two datasets: Aruba and WSU, available publicly (PA, https://ailab.eecs.wsu.edu/casas (accessed on 13 February 2023))	Motion, door, and temperature sensors, etc.	Sensitivity: $98.67\%$ Sensitivity: $88.70\%$	[5] 2019
Deep belief networks (DBFs)	Behaviours and abnormalities within activities of daily living (ADLs) of the elderly	Dataset including a range of activities collected from the bathroom, bedroom, kitchen, and living room (nPA)	Motion sensor, door sensor, light sensor, smoke sensors, cameras, etc., installed inside living room, bedrooms, kitchen, etc.	F1 score: bedroom $89.1\%$, bathroom $90.1\%$, living room $80.8\%$, kitchen $83.6\%$	[44] 2020
Hybrid framework with 1D-CNN, 2D-CNN, and LSTM	Behaviours and abnormalities within activities of daily living	Data collected with FMCW radar, operating at 5.8 GHz C-band and 400 M bandwidth (nPA)	Frequency-modulated continuous wave (FMCW) radar	Overall accuracy: $93.391\%$ Accuracy of Falling: 100%	[55] 2021
Transformer with bidirectional GRU	Multi-resident ADLs	CASAS (PA, https://ailab.eecs.wsu.edu/casas), ARAS (PA, https://www.cmpe.boun.edu.tr/aras/ (accessed on 13 February 2023))	Temperature sensors, motion sensors, door sensors, PIR sensors, item sensors, cabinet sensors, water sensors, burner sensors, phone sensors, etc.	Accuracy: $91.44\%$ Precision: $90.64\%$ Recall: $91.15\%$ F-measure: $90.89\%$	[98] 2022

PA: Publicly available; nPA: not publicly available.

5.1.2. Probabilistic Methods

Probabilistic methods make use of the distribution of the training data or features, to identify the location of the anomaly boundary [13]. Probabilistic-based methods work poorly when the data are insufficient. Probabilistic methods typically include methods based on data distribution, such as cumulative distribution [99], Bayesian methods [8], etc. Bayesian methods feature a mechanism of convening prior beliefs into posterior beliefs when unseen data arrives. Based on this, the behaviour features can be modeled using Bayesian statistics. Bayesian methods offer a powerful framework to construct modeling techniques that specifically consider uncertainties.

The authors in [33] propose a cumulative distribution function (CDF)-based probabilistic method to analyse the temporal and sequential information from volunteers; the aim is to identify the daily patterns for detecting abnormal behaviour. The study only uses contact sensors attached to the objects in a kitchen; each contact sensor provides two possible states (“0” or “1”) of the corresponding object. The CDF obtains the probabilities of abnormal behaviour regarding time and steps. The probabilistic analysis achieves accurate results for activities with a long duration and many steps. However, for activities with short time durations or fewer steps, how CDF can be used to identify abnormal behaviour is not studied in this paper. Probabilistic methods, e.g., cumulative distribution, determine the bounds of normal or abnormal behaviour using the distribution of the training data. The performance of such methods is less satisfactory, especially in a small data scenario, since reliable estimations cannot be obtained.

5.1.3. Discriminative Methods

Discriminative methods try to find a boundary between different classes instead of modeling how instances of the classes are generated [84]. SVM is one of the most popular discriminative methods used to detect abnormal activities by modeling a normal activity as one-class and then detecting deviations from this class.

In [100], the hand-crafted features are extracted and then autoencoders and one-class SVMs are used to first learn the normal behaviour and then detect the abnormal behaviour. Moreover, the abnormal activities are simulated in the dataset by introducing disruption in sleeping patterns, abnormal temperature changes and frequently leaving the room. The results show that one-class SVM achieves an accuracy of $98.6\%$, and autoencoders achieve a detection rate of $72\%$. The authors claim that both experiments have a high number of false positives. In [101], SVMs are used to predict the progression of mild cognitive impairment based on movement data using motion sensors. First, hand-crafted features, such as detrended fluctuation analysis, cyclomatic complexity, fractal index, entropy and room transition, are extracted and fed into the SVM. The results show that the onset is predicted six months earlier than the clinical diagnosis.

5.1.4. Clustering

Clustering-based abnormal behaviour detection [9,27,102] might be appealing because it can provide promising results in the detection of deviations and unsupervised modeling of human behaviour. Abnormal behaviour detection-based clustering methods, typically either cluster the data into different groups and then identify the data not belonging to the groups as an anomaly; or determine the distance or the threshold between the normal group and the abnormal ones. The clustering methods’ performance relies strongly on the assumption of the data distribution.

The authors in [9] detect abnormal behaviour by data clustering and analysing transitions between data vectors representing adjacent time intervals and clusters. First, they train a random forest to obtain clusters of activity patterns. The information from the random forest data, i.e., proximity matrix, is mapped onto the 2D space; data clusters are obtained by agglomerative clustering. A third-order Markov chain models the transitions between clusters. One advantage of the approach is that it does not make any assumptions about the position, type and relationship of the sensors. Experimental results show that temporal and spatial anomalies can be detected by analysing a 2D map of high-dimensional data. The authors state that such a map is independent of the number of clusters formed. In addition, the data clusters can be obtained by analysing the structure of the most representative tree and identifying the most important variables.

5.1.5. Graph-Based Methods

Graph-based methods [7,26,103] allow the relationship between sensors to be exploited. Long and Holder [104] perform an activity prediction by using three different graph-based approaches, representing time-based sensor data as a graph. In their case, none of the graph-based models outperforms the non-graph SVM models. It also shows that the graph-based approaches are able to correctly classify graphs that cannot otherwise be classified correctly. Akter and Holder [82] represent an SE as a graph, in which motion sensors are taken as vertices and movements as edges to perform activity recognition. They then extract the graph-based features as input for a support vector machine (SVM). This method outperforms conditional random fields (CRF), hidden Markov model (HMM) and Naive Bayes. However, their study fails to take temporal information into account.

The elderly can benefit from a reduced risk of falls by following the OTAGO exercise program. The research in [105] generated a weighted, directed multigraph from the deployed PIR motion sensor network in the flat where the resident lived. Then, the authors subtract the weights of one day from a baseline and use the difference to monitor the performance progress of older adults. They also find the adults who finish the OTAGO exercise program could slow their decline, compared with the older people who do not finish the exercises. Further clinical studies are required to verify their findings based on the pilot study.

5.1.6. Rule-Based Methods

Rule-based methods [7,43,46,96,106,107,108,109] require expert knowledge to provide rules to the proposed system for abnormal behaviour detection. Rule-based approaches are easily readable by humans, while they cannot handle noisy data. Although these rules help eliminate false alarms, they sometimes change from person to person, as well as from application to application.

In [110], entropy-based measures are used to detect anomalies in daily living when there are visitors. The authors first calculate the maximum entropy value in normal activities per hour and day, and then use it as a threshold to decide whether a new activity is abnormal. One advantage of this study is that the algorithm not only detects abnormal activities but also detects the cause of them, such as disruption in sleeping patterns or visitor presence. Moreover, the authors compare different types of entropy, like Shannon entropy, dispersion entropy, etc., on the accuracy of the results, and find that fuzzy entropy and multiscale-fuzzy entropy achieve the best accuracy rates.

5.2. Sub-Activities

Daily activities are typically composed of granular level units, called actions, steps, or sub-activities [25,111]. These interior structures are important to model the activities hierarchically and construct coarse-grained details. For example, the activity wash clothes involves the actions below: getting clothes from a basket, filling up the washing machine, turning the washing machine on and taking clothes out. Modeling daily life activities from sub-activities is important to detect abnormal behaviour for applications such as detecting indicators of cognitive decline [3,4,5,14]. The anomalies stemming from cognitive decline may be reflected in the repetition frequency of these steps and their relation with each other.

In [112], actions are defined as the simplest movements, and behaviours are described as the most complex ones. Behaviours are divided into two different types, intra-activity and inter-activity behaviours. The different elements of user behaviour are as follows.

• Actions are temporally short and conscious muscular movements made by the users (e.g., taking a cup, opening the fridge).
• Activities are temporally longer but finite, and are composed of several actions (e.g., preparing dinner, taking a shower, watching a movie).
• The intra-activity behaviours describe how a user performs a single activity at different times (e.g., while the user is preparing dinner, sometimes they may gather all the ingredients before starting, while on other occasions, the user may take them as they are needed). The inter-activity behaviours describe how the user chains different activities (e.g., on Mondays, after having breakfast, the user leaves the house to go to work, but on the weekends, they go to the main room).

5.3. Deep Learning Methods

The above conventional machine learning methods can model complex ambient sensor data, yet require extensive efforts in feature engineering. Deep learning (DL) approaches are the state-of-the-art methods also used in the detection of abnormal behaviour in an SE [44,89,113,114]. Deep neural network architectures (DNN) have become very popular for abnormal behaviour detection due to their characteristics of being able to automatically mine the nature of input data [4,5,44,114]. DNN techniques automatically learn hierarchical discriminative features from data. There is no need to develop manual features by expertise, compared with the conventional methods discussed above. Various DL-based methods, e.g., recurrent neural networks (RNNs) [114], long short-term memory networks (LSTM) [67], or graph neural networks [115], have also been deployed for abnormal behaviour detection.

5.3.1. RNN and CNN

RNNs have been exploited to detect anomalies where sequential data are present [4,5,114]. RNNs can be an alternative method to HMMs, where abnormal behaviour might occur in the sequential context of activities. For example, in [4], variants of RNNs, namely vanilla, long short-term memory and gated recurrent units, are exploited to detect abnormal behaviour to identify indicators of cognitive decline. The authors try to model sequential activities with RNNs, and then detect abnormal activities deviating from temporal patterns by utilising a threshold in the activity recognition confidence probabilities. Moreover, RNNs in [5] are combined with convolutional neural networks (CNNs) for the same task. While CNNs extract their own features, RNNs add temporal context to these features. The authors claim that RNNs and CNNs are promising for detecting abnormal behaviour in detecting cognitive decline indicators in an SE.

In [116], CNNs are used to detect anomalies in the path of wandering activity for elderly people suffering from Alzheimer’s. The authors in [55] focus on the multi-domain fusion of radar information with a novel hybrid neural network for human activity recognition. The network combines a 1D convolution neural network (CNN), a 2D convolution network and a recurrent neural network (RNN) to extract much richer attributes, so as to enhance the recognition performance effectively.

In [55,86,117], CNNs, LSTM, CNN-LSTM and autoencoder-CNN-LSTM are used to detect the abnormal activities of elderly people. Two datasets are used: simulated activities of daily living (SIMADL) and MobiAct. MobiAct is collected with a smartphone placed in the pocket while the participants performed activities such as jumping, sitting, walking, etc. As the number of abnormal activities is fewer, compared to the normal ones, the SMOTE statistical method is used to oversample the abnormal activities, thus solving the imbalanced class problem. The results show that the CNN-LSTM method achieves an accuracy of $93\%$ on the MobiAct dataset and $98\%$ on the SIMADL dataset, as they take both the temporal and spatial information into account.

5.3.2. Graph Neural Networks

Some researchers apply CNNs to extract the spatial dependencies between sensors and then deploy LSTM on the time dimension to learn temporal dependencies [118,119]. The data in the above-mentioned CNN or RNN methods are typically represented in Euclidean space. CNNs and RNNs only model the temporal dependencies of time series or the spatial dependencies between sensors; this somehow ignores the influence of other sensors. There are many applications in which data are generated from non-Euclidean domains, such as social networks, knowledge graphs or interaction networks. Motivated by graph embedding and CNNs, graph neural networks (GNNs) are proposed to learn a state embedding, relying on the feature information of a node through the historical attributes of itself and its neighbours [115].

In [3], the authors exploit GCNs (graph convolutional networks) building daily actions from their granular-level structures, thereby detecting abnormal behaviour from cognitive impairment. The authors in [14] use recursive autoencoders (RAE) to create a hierarchical tree structure, which can detect the abnormal behaviour of dementia sufferers by decomposing actions into sub-actions.

Spatial-temporal GNNs, recurrent GNNs and convolutional GNNs have been widely applied to data represented in graphical forms, such as drug chemical stability identification [120], traffic flow prediction [121], fault diagnosis [122], etc. The authors in [123] propose a hierarchical attention graph convolutional network (HAGCN) for remaining useful life (RUL) prediction. HAGCN aims to improve the accuracy of RUL predictions of ambient sensor-based abnormal behaviour by modeling the sensor network to spatial-temporal graphs. They use a regularised self-attention graph pooling layer in HGRL for graph representation. Two case studies are conducted to verify the performance of HAGCN in generating graphical representations. To the best of our knowledge, only a few studies using GNNs for ambient sensors-based abnormal behaviour detection (ASABD) have been performed. We believe there will be an increasing number of tasks for ASABD using GNNs, since the deployment of ambient sensor data are graphically presented in essence. Similarly, the other popular models, like attention or transformer-related frameworks [124], are also rarely accessed in the field.

6. Challenges and Open Questions

6.1. Sensor Representation

Representation of the sensor activation in time-slice chunks extracted with sliding windows is one of the critical steps in activity recognition and abnormal behaviour detection. Most of the studies treat each sensor activation independently from each other and ignore the relationship with each other (such as order of triggering) [4,5,66,79]. Whereas, abnormal behaviour detection applications, such as healthcare (supporting elderly, detecting indicators of dementia), require sensor representations which encode the granular-level details of sensor activation, such as frequency, order, etc. Although there are studies dedicated to this purpose, such as exploiting an adjacency matrix to model the relative locations of the sensors [81], modeling the sensors in a graph structure [3,82], or encoding the sensors in an adjacency matrix [9], there is still room for further analysis to exploit sequential sensor activation. The traditional sensor representations, such as binary, change-point and last-fired, treat the sequential information with a bag-of-words style where in-depth information is lost.

Moreover, sub-activities are critical when detecting abnormal behaviour related to dementia. Existing studies take each activity as an atomic unit and fail to model activities based on their sub-activities. Although there are a few studies [3,14] trying to model activities hierarchically from their sub-activities, future research should be dedicated to representing the sensor activation, frequencies and their interaction with each other to exploit the hidden, fruitful information lost in the bag-of-sensors approach. These representations will boost the success of abnormal behaviour detection methods, such as detecting the indicators of dementia, where the abnormality lies within the details and hierarchy of the sensor activation.

6.2. Multi-Resident

Abnormal behaviour detection methods learn the patterns of habits of a single user [8,66] in an SE and detect the abnormal behaviour deviating from these patterns. Understanding the resident’s patterns and differentiating abnormal behaviour is still challenging in the context of multi-resident homes. Further study is needed to detect abnormal behaviour in the case of multi-resident homes.

6.3. Dataset Issues

• Realness of the simulated instances

Although publicly available datasets can be used for daily life activity recognition tasks [8,65,66], there is still a gap for datasets addressing specific healthcare applications, such as dementia care. Collecting of real-world datasets is time-consuming and challenging, especially for diseases such as dementia. These datasets need to be collected over a period of years, since this type of disease might change slowly. When there are no real-world data available, simulation of abnormal behaviour might be a solution [3,4,5,7,8,9,27,70,72,73,74]. Future research should focus more on simulating and generating abnormal behaviour for specific cases of healthcare applications. Although some studies target the simulation of this kind of dataset, they may be lacking in their representation of real-world instances. Thus, methods are also required to check the real-world applicability of simulation methods.

• Imbalanced dataset

Daily life activity recognition datasets suffer from the following problems: (1) the frequency of some of the activities dominates the others (such as cooking and going to the toilet vs. sleeping); (2) some other activities take a longer time than others (e.g., sleeping takes hours, while leaving or entering the home is performed only a couple of times). This introduces a class instance imbalance problem that needs to be considered, in, for example, training-based methods, where the model learns the class labels based on the instances, especially discriminative methods and generative methods, that favour the most frequent classes [4,25,66]. In contrast, deep learning methods, like RNNs and CNNs [4,5], perform relatively better on the less frequent classes. The HMM is complex when there is a large set of transitions for all possible states. Neural network methods can be slow, expensive to train, and have a complex architecture. Thus, future studies should explore how to differentiate between the dominant classes and the less common ones, to provide better accuracy for activity recognition systems.

• Limited abnormal instances

Another challenge in abnormal behaviour detection is a dataset’s limited number of false positives. An abnormal behaviour is one that deviates from normal behaviour [9,26,27,28], but it is difficult to know its characteristics beforehand. Unfortunately, training-based methods need to learn and model normal and abnormal behaviour, and it is necessary to have both normal and abnormal instances in the training set. However, there are not enough instances of abnormal behaviour in the training set. As a result, the training-based studies in the literature follow a method first to learn what normal is (on a training set containing only normal instances) and then flag the abnormal behaviour based on activity recognition score. Studies that do not favour the frequent classes (normal behaviour) and model the least occurring ones well (abnormal behaviour) should be proposed and developed.

6.4. Ethical Issues

In recent years, the use of artificial intelligence involving ambient sensors for healthcare has proliferated. Data collection and its usage are critical in the associated applications; however, this can also bring ethical concerns. The collected data are recorded in various formats, including personal information and sensor readings. Inferences are then drawn from the data for specific uses. Each project involving data collection should carefully consider the questions, such as how to store and maintain the data, how or when the data should be accessed or used, etc. The legal requirements or ethical agreements should be approved to protect participants from unexpected disturbances or intrusions into their personal life.

7. Conclusions

This survey comprehensively reviews the literature on abnormal behaviour detection methods in healthcare. Specifically, we present varied definitions of abnormal behaviour. We survey popular ambient sensors, the datasets available in the fields and the methods for generating synthetic data. As one of the core focuses of this survey, we discuss different modeling methods to detect abnormal behaviour and suggest potential application contexts for them. Finally, we highlight the research gaps and challenges.

There exists a large body of methods for activity recognition or abnormal behaviour detection. However, the methods that have become dominant in the fields of computer vision or language processing, such as attention-based, graph neural networks, as well as transformer-based models, have yet to be commonly applied in abnormal behaviour detection using ambient sensors. This may imply that challenging questions exist in deploying these novel techniques in SEs. The other open questions that need to be addressed include abnormal behaviour detection in a multi-resident SE, data collection for dementia, tackling false positives in activity recognition, ethical concerns, and so on.