PIFall: A Pressure Insole-Based Fall Detection System for the Elderly Using ResNet3D

Abstract

Falls among the elderly are a significant public health issue, resulting in about 684,000 deaths annually. Such incidents often lead to severe consequences including fractures, contusions, and cranial injuries, immensely affecting the quality of life and independence of the elderly. Existing fall detection methods using cameras and wearable sensors face challenges such as privacy concerns, blind spots in vision and being troublesome to wear. In this paper, we propose PIFall, a Pressure Insole-Based Fall Detection System for the Elderly, utilizing the ResNet3D algorithm. Initially, we design and fabricate a pair of insoles equipped with low-cost resistive films to measure plantar pressure, arranging $5\times 9$ pressure sensors on each insole. Furthermore, we present a fall detection method that combines ResNet(2+1)D with an insole-based sensor matrix, utilizing time-series ‘stress videos’ derived from pressure map data as input. Lastly, we collect data on 12 different actions from five subjects, including fall risk activities specifically designed to be easily confused with actual falls. The system achieves an overall accuracy of 91% in detecting falls and 94% in identifying specific fall actions. Additionally, feedback is gathered from eight elderly individuals using a structured questionnaire to assess user experience and satisfaction with the pressure insoles.

1. Introduction

Falls are a major public health problem, accounting for an estimated 684,000 deaths worldwide each year [1]. They are the second leading cause of unintentional injury deaths and are especially common among adults over 60 years of age. Even non-fatal falls can lead to serious health issues, such as fractures, contusions, and head injuries, often necessitating medical care and potentially resulting in long-term health and quality-of-life impacts.

Given the severity and prevalence of falls in senior citizens, it is important to have effective methods to detect falls and provide timely assistance. Our goal is to develop effective fall detection strategies for the elderly that respect their independence and privacy, while also improving their health and quality of life. Fall detection is essential for providing timely assistance in the event of a fall and to help reduce the potential consequences of falls in the elderly. It can also help identify the root causes of falls and provide valuable information for developing fall prevention strategies.

Current traditional detection methods include image-based cameras and wearable sensors [2], and a large number of publicly available datasets at this stage. However, these established detection methods still have limitations, such as not guaranteeing the independence and privacy of the elderly, and the burden that additional wearable devices impose on the elderly. Therefore, several key factors must be considered when building fall detection solutions for the elderly:

1.

Protecting the privacy of the elderly: Protecting the independence and privacy of the elderly is crucial to avoid solutions that may infringe on their habits and sense of privacy. According to previous study [3], even the smallest infrared cameras can cause discomfort to the elderly and invade their privacy.

2.

Continuous detection: A fall detection system must be able to continuously detect falls without being obstructed or obscured by other objects in the environment. This is critical to ensure that the system is continuously active and ready to respond in the event of a fall.

3.

Portability: A fall detection system should be independent of its surroundings so that the individual does not need something else to utilize it. It is necessary to ensure that the system is available for user action detection in all situations.

4.

Cost-effective and low-maintenance: Traditional fall detection systems can be expensive and require a lot of maintenance, which can be a barrier for the elderly. Low-cost, low-maintenance fall detection solutions that respect the independence and privacy of senior citizens are needed.

To address the four problems mentioned above, this study proposes a solution. We design a pair of insoles with embedded pressure sensors to collect pressure data from the user, addressing the four problems outlined earlier, which are as follows:

1.

Protecting the privacy of the elderly: We develop a pair of soft insoles with embedded E-Textile pressure sensors, designed to be easily incorporated into regular footwear, which negates the necessity for extra wearable sensor devices. These insoles only collect plantar pressure data, thus preserving the user’s privacy without capturing any personal identifiers. This approach effectively addresses the discomfort and privacy issues associated with camera-based systems or wearable sensors technologies.

2.

Continuous detection: The insoles can be conveniently placed within everyday footwear to continuously monitor plantar pressure in real-time, with the pressure data being transmitted promptly via Wi-Fi. This method overcomes the challenges of discontinuous monitoring due to blind spots or inadequate lighting that are common with camera-based systems.

3.

Portability: When monitoring environmental changes, RF-based methods like Wi-Fi and infrared require data re-collection and model re-training. In contrast, the insole-based fall detection system exhibits environmental independence, ensuring stable operation despite changes in the usage environment.

4.

Cost-effective and low-maintenance: Compared to existing fall monitoring systems, the developed insoles significantly reduce manufacturing costs, requiring only a chip, fabric insole, and the circuitry integrated on the insole. The system has also been optimized for reduced maintenance complexity. A sleep feature is introduced where the insole enters a low-power state when no pressure is detected and reactivates upon sensing pressure, thereby conserving battery life and extending operational longevity. An MD5 checksum has been implemented to ensure the integrity of data transmission. To further ease maintenance efforts, the insoles are equipped with an OTA (Over-The-Air) update capability, allowing for automatic firmware upgrades without manual intervention, simplifying the user experience for the elderly.

In existing studies [4,5,6,7,8,9], off-the-shelf Force Sensitive Resistors (FSR), accelerometers, and gyroscopes are utilized to fabricate pressure insoles designed to protect user privacy and enable continuous monitoring capabilities. However, due to the size and shape constraints of FSRs, sensors can only be placed at critical positions on the sole of the foot, allowing for the detection of falls and gait but limiting the ability to capture a richer set of motion information from the insole. To acquire comprehensive gait information, study [10] expanded the number of pressure sensors in each insole to 96. However, customizing such a pair of insoles comes at a higher cost. To reduce the cost of pressure insoles, study [11] proposes a deep learning model capable of predicting the entire foot’s pressure distribution using data from a limited number of insole pressure sensors, offering potential applications in diagnosing foot deformities, pathological gait, falls, and pressure sores, especially in diabetic patients. It is evident that embedding a greater number of sensors in pressure insoles at a lower cost is crucial, as a dense array of sensors provides richer plantar pressure data for improved analysis of falls and activity. In this paper, we introduce a fall detection system based on pressure insoles, utilizing resistive film to manufacture pressure sensors densely distributed across the insole, thereby reducing production costs and simplifying fabrication. Our key contributions are as follows:

First, we propose PIFall, a Pressure Insole-Based Fall Detection System for the Elderly, utilizing the ResNet3D algorithm. We design and fabricate a pair of insoles equipped with resistive films to measure plantar pressure, arranging $5\times 9$ pressure sensors on each insole. Compared to capacitive pressure sensors, resistive films offer a cost-effective alternative, enabling the dense arrangement of sensors at a lower expense.

Second, we present a fall detection method by combining a unique insole-based sensor matrix with the ResNet(2+1)D architecture. Our system utilizes ‘stress video’, a time-series dataset from pressure maps, as network input. To address the challenges of low-resolution data, we employ upsampling. Interpolation makes the pressure map closer to real-world smoothness and enhances data granularity, thereby improving recognition accuracy.

Lastly, we collect data on 12 different actions from five subjects, including fall risk activities specifically designed to be easily confused with actual falls. These actions are efficiently classified using a ResNet(2+1)D neural network model, achieving an overall accuracy of 85.61% and a fall detection accuracy of 94%.

2. Related Works

2.1. Human Action Recognition

Human action recognition is an area of computer science and artificial intelligence that involves the development of algorithms and systems to recognize and classify human actions in digital videos and other media.

According to a recent review [12], mainstream data collection methods are visual-based, such as using video frames or images [13], as well as sensor-based methods that collect data on various modalities, such as acceleration [14], body metrics and pressure. Data can also be collected using wireless bands and infrared signals. For image-skeleton-based fall recognition [15], RGB data from the internet and multiple sources and formats are used to train recognition models. From a skeleton-based perspective, Ref. [16] adds classification based on inter-joint connection relationships and joint trajectory information to the ST-GCN algorithm for enhanced recognition. In the domain of non-image recognition, Ref. [17] employs accelerated data and compares the performance of three deep learning model architectures on a dataset for senior citizens fall detection, achieving improved results by using gender and age as auxiliary outputs. Furthermore, Ref. [18] utilizes ultra-wideband radar to detect falls by capturing wireless channel fluctuations and applying the ConvLSTM recognition algorithm for device-free fall detection. Recently, multimodal fusion methods have been proposed, where multiple features such as video features and acceleration features are combined for enhanced learning [19].

However, in the context of fall detection for senior citizens, ensuring sufficient independence and privacy remains a significant concern. Visual modalities not only present technical challenges but also raise ethical privacy issues. Wearable devices that rely on accelerometer sensors can cause stress and discomfort for seniors, who may then reject the device or forget to carry it consistently. Wireless channel-based signals may offer a promising approach, yet their effectiveness is limited by the structural layout of the room, and a single model may not perform well in different environments. These challenges must be addressed to improve fall detection for senior citizens.

2.2. Fall Detection for Senior Citizens

The primary target audience for fall detection is senior citizens, as falls are the second leading cause of accidental or unintentional injury deaths within this group. Effective fall detection can facilitate timely medical care and potentially save lives. In this case, falls are considered a subset of human actions, which means that methods used in motion detection, such as acceleration sensors and camera/depth cameras, can also be applied to fall detection.

There are currently several challenges in the field of senior citizen fall detection, such as privacy, user adoption and challenges in maintaining privacy. Wearable devices that collect personal motion and activity data may raise privacy concerns, especially when the data is shared with third parties. Moreover, seniors may exhibit reluctance towards employing wearable devices or technologies that are unfamiliar to them, potentially undermining the efficacy of fall detection systems.

To tackle these challenges, a novel direction distinct from normal human motion detection has been proposed in [20], which entails the use of environmental devices. This method involves installing a series of sensors near the person of interest, such as walls, floors, beds, etc. Data gathered from these sensors are analyzed by an algorithm to ascertain the occurrence of a fall. Subsequently, the event is reported to the caregiver. Since there is no need to wear any sensors, this becomes a solution to both of the challenges. For instance, Ref. [18] introduces a technique utilizing ultra-wideband (UWB) single-station radar for data acquisition and convolutional long short-term memory (LSTM) networks for detecting falls within a room. Additionally, Desai Kimaya’s study [21] presents a novel machine learning approach for fall detection using a wearable belt. Another study [22] details an improved threshold-based fall detection method applied to smartphones, which uses collected acceleration data to identify falls during everyday activities in four different directions.

2.3. Pressure-Sensing-Based HAR

In action detection of environmental devices, pressure data can also be used as a data that can be collected by environmental devices. Pressure-based action detection in the field of action recognition, ref. [23] proposed by Sundaram, presents an electronic textile glove as an electronic textile glove using pressure variable resistance film. Likewise, Ref. [24] develops a large-scale pressure-sensing carpet that utilizes the same mechanism to reconstruct the human skeletal structure from pressure distributions, then applies established skeleton-based recognition techniques for action classification. Concurrently, studies such as [25] involve extensive data gathering from a significant number of senior citizens via plantar pressure sensors, which assists in identifying falls and frailty through classification methods.

The effectiveness of pressure-based data for action recognition has been demonstrated, and this paper further distinguishes between fall actions and normal actions using plantar pressure data. This study identifies 12 actions, including 5 daily safety actions, 4 actions that pose a risk of falling, and 3 different directional fall actions.

In contrast to approaches that place pressure sensors directly on the sole of the foot, we present a uniquely crafted insole. It features a three-layer electrical circuit structure—comprising wire, piezoresistive film, and wire—arranged in both horizontal and vertical orientations to form a $5\times 9$ pressure sensing matrix. This matrix enables pressure detection at 45 distinct locations on the sole of the foot, aiding the model in learning the dynamics of the subject’s center of gravity shifts. Our design incorporates a greater number of sensors and a higher pressure data sampling frequency than similar studies [25,26] that classify actions based on plantar pressure data. Capable of detecting pressure changes at 50–125 Hz sampling rate across 45 points on the foot’s underside, our insole offers an enhanced range of detection capabilities.

3. Application Model

The application model is shown in Figure 1. It is a high-resolution and -sampling-rate pressure-detecting insole worn by senior citizen living alone to detect in real-time whether fall-related actions have occurred. When the elderly person falls while wearing the shoe with the pressure-detecting insole, the data during this time is transmitted to a computer/server via a wireless chip mounted on the insole. A convolutional neural network learning algorithm is used in the computer/server to classify the input data to determine if the action is a fall. In addition, the system detects and records daily actions, creating a life log of the elderly’ actions. These data can provide caregivers or medical professionals with valuable information to monitor the health and well-being of the senior citizen. The computer/server then analyzes these data with deep learning algorithms and calls for emergency medical assistance and notifies the guardian when the features of a fall are detected.

4. System Design

4.1. System Outline

To solve the falls problem, the fall detection system developed in this research uses pressure sensors installed in shoe insoles to detect falls for senior citizens living alone. The system process is shown in Figure 2, and the key processes are as follows:

1.

The user produces different voltages according to different pressures when wearing the shoes. This allows the system to indirectly obtain the pressure distribution of the user over time. The voltage is transmitted to the chip through a wire, connected to the pressure sensors in the insoles.

2.

The pressure distribution data are transmitted to a computer via wireless communication, using a Wi-Fi access point device. This allows the data to be analyzed in real-time, enabling the system to detect a fall as soon as it occurs.

3.

The computer analyzes the data to determine whether the user is likely to fall. The analysis is based on machine learning algorithms that have been trained on a dataset of falls and non-falls. If a fall is detected, the system provides timely assistance to the user, such as alerting a caregiver or emergency contact.

Figure 2. The overall flow of the system. The system detects the user’s pressure distribution over time using a pressure sensor in the shoe, which is then transmitted to a computer for analysis. To determine whether the user is likely to fall, the computer employs a machine learning algorithm trained on the dataset collected by the insole. If a fall is detected, the system responds quickly by alerting caregivers or emergency contacts.

Figure 2. The overall flow of the system. The system detects the user’s pressure distribution over time using a pressure sensor in the shoe, which is then transmitted to a computer for analysis. To determine whether the user is likely to fall, the computer employs a machine learning algorithm trained on the dataset collected by the insole. If a fall is detected, the system responds quickly by alerting caregivers or emergency contacts.

4.2. Plantar Pressure Collection Method

To measure the pressure, our insole has a three-layer structure, which measures the pressure through a pressure variable resistor and a path formed by horizontal and vertical lines, as shown in Figure 3. A GPIO pin that outputs 3.3 V is connected to the horizontal line of the first layer. Current flows through a pressure variable resistor layer in the second layer. A vertical line connected to an ADC pin forms the third layer.

Current flows horizontally through the first layer and vertically through the variable resistor layer being introduced into the third layer in order to measure pressure. The current flows into the ADC pin to measure the resistance of the variable resistor and determine the corresponding pressure value. The ADC pin has a 900 Ω sampling resistor.

Figure 4 shows the circuit design diagram, and the physical circuit board is shown in Figure 5. The circuit includes four modules, the description of each module is as follows:

(a)

5 ADCs and 9 GPIOs, with 900 $\Omega$ pull-down sampling resistors under each ADC.

(b)

ESP32C-12F chip: A microprocessor that supports high-speed computing and low power consumption with wireless and Bluetooth functions, can be written in code using Arduino IDE 2.2.1. ESP is described in detail in the next paragraph.

(c)

Charging and low power consumption circuit design: It can automatically switch between high/low power consumption mode through the code (high power consumption mode when downloading data and detecting the presence of pressure; low power consumption mode when no pressure is detected for a certain period of time, reducing the sampling frequency to save power).

(d)

USB to serial input: Before the implementation of OTA, there is a need to rely on this part of the input compiled binary file.

Figure 4. Circuit design diagram.

Figure 4. Circuit design diagram.

Figure 5. Physical circuit diagram.

Figure 5. Physical circuit diagram.

We use the ESP32 as the board’s processor, using five ADCs and seven GPIOs. Compared to other circuit devices that use serial communication, we have the following two advantages:

1.

The traditional circuit relies on serial communication for data transmission, as the maximum bandwidth of serial communication is 115,200 bps stable, resulting in the sampling frequency receives a bandwidth limit, while the bandwidth of wireless communication is far greater, so a higher sampling frequency can be achieved. The device can reach 45 pressure points up to 125 Hz sampling frequency.

2.

The device has a function called OTA. It can realize the function of version upgrade through Wi-Fi and compile upload without connecting any transmission line. Since wireless communication replaces serial communication, it means the pins of RX and TX can be multiplexed as GPIO to realize the pressure point detection of $5\times 9$ matrix. Also, OTA can solve the challenge of the unmanned maintenance of intelligent textiles.

4.3. Fall Recognition System

Figure 6 shows the flow chart of the fall detection process, including data collection, data preprocessing and implementation of machine learning model.

1.

Data collection: Incoming data from the insoles are collected and sent to a computer/server for analysis.

2.

Data pre-processing: The collected data is pre-processed to ensure that it is ready for analysis. This includes normalization, adding default values and removing outliers.

3.

Machine learning model: A ResNet(2+1)D neural network is used to predict the probability of various actions based on the pre-processed data and is used to classify the stress action data.

Figure 6. Recognition process of the fall detect system.

Figure 6. Recognition process of the fall detect system.

4.3.1. Data Collection

The ESP32 can either establish a wireless network by itself as an access point or connect to a wireless network as a client. In this research, we use a router to open a wireless network and let the ESP32 and the computer connect to this wireless network, with each of the two ESP32s connected to a different port of the computer. Since high-frequency network transmission wastes a lot of chip computing time, we let the ESP32 cache the measured data and send it as a batch when it reaches 1500 bytes in size.

4.3.2. Data Pre-Processing

We utilize a sewing machine to fix conductive sewing thread and resistive film with non-conductive cotton thread to make 45 pressure sensors. However, due to human factors, the 45 sensors are unevenly stressed, resulting in different pressure responses when no load is applied. To solve this problem, we calculate the average no-load pressure for each pressure sensor and then subtract this average to zero out the sensor reading.

For the arrangement of the pressure sensors, each insole contains sensors laid out in five columns and nine rows. When two insoles are combined, this layout doubles to 10 columns and 9 rows. To standardize the data format, we add an extra row of zeros, resulting in a final distribution of 10 columns by 10 rows, which gives us a uniform pressure image data matrix. We initially sample the pressure data at a frequency of 100 Hz within a 5-s timeframe for each action. To optimize data processing, we resample this data to 50 Hz. From this resampled data, we then generate a 250-frame grayscale video to visually represent the pressure changes over time.

4.3.3. Machine Learning Models

To take into account the possibility of more action types and more data volumes in the future, the choice of a suitable neural network model is important. Focusing on the purpose of preventing overfitting, we chose the ResNet, which utilizes residual connectivity to make it easier for the model to learn complex functions and reduce the risk of overfitting.

In our study, we analyze stress video data, which necessitates a model capable of interpreting both spatial and temporal dimensions. Therefore, we adopted ResNet(2+1)D [27], a variant of ResNet inspired by the work of Du Tran. Tran’s research demonstrated the effectiveness of replacing ResNet’s 2D convolutional layers with 3D convolutional layers. They decomposed the 3D convolution into two independent consecutive operations, a 2D spatial convolution and a 1D temporal convolution. This change brings an additional nonlinear correction (additional nonlinear correction for spatial convolution followed by spatial convolution), which is easier to optimize and thus reduce losses compared to full 3D convolution. So in this research, we are using ResNet(2+1)D as the neural network model for the classifier.

Given the small size of the data collected in this research and the lower resolution than traditional video data ($112\times 112$ for traditional video data, while we use $10\times 10$), we chose to use only a 10-layer ResNet(2+1)D model for training. If in the future we obtain more data and action types, it may be more effective to use a model with more complex layers. In addition, we use 75% of the original dataset for training and 25% for testing to ensure the robustness and accuracy of the model.

5. Implementation

We implement a pressure-based fall detection system in three key steps: design of the tactile insoles, data collection, and analysis.

5.1. Pressure Sensors Distribution in Insoles

We used the Japanese size standard for insoles, which is measured in cm with 0.5 cm intervals between each size. When designing the insoles for the fall detection system, the distribution of pressure points was a key consideration, because an uneven distribution of pressure points could affect the accuracy and reliability of the fall detection system. The irregular shape of the insole presented a challenge in this regard, as it is difficult to arrange the pressure points in a way that will ensure an even distribution.

To address this issue, two solutions were considered. The first solution involves giving up pressure points in the narrower part of the width and focusing on ensuring an even distribution in the wider part of the insole. This approach has the advantage of being relatively easy to implement and requiring no special sewing operations. However, it may result in some areas of the plantar being left uncovered by pressure points.

The second solution involves carefully controlling the direction of the stitches to ensure that each pressure point is evenly distributed on the horizontal line. This approach provides better coverage of the plantar, but requires more time and effort in the sewing process.

Ultimately, we chose the second solution, depicted in Figure 7, as it provides the best balance between coverage and accuracy. However, both solutions have their advantages and limitations, and further research may be needed to determine the optimal approach for distributing pressure points in insoles for fall detection systems.

5.2. Stitching Details

The stitching of the pins is also a key factor to consider when sewing the wires of the insoles. As shown in Figure 8, stitches are used to hold wires in place and prevent them from moving or shifting during use. However, if the stitches are placed too close together, a number of problems can arise.

First, deformation of the fabric can cause the wire to be pulled by the cotton thread to the other side of the fabric, possibly making direct contact with another crossed wire and causing a short circuit. A short will cause all pressure points on this crossed wire to bypass the resistor going through here, which affects the accuracy and reliability of the device. Second, the wire at the pin may be tied up in cotton thread, which may prevent the horizontal and vertical wires from crossing, resulting in poor contact between the wire and the pin. This can affect the system’s ability to accurately detect pressure changes and interpret them as drops.

To avoid these problems, we carefully considered the location of the pins at the crossover of the horizontal and vertical wires to ensure that they are spaced far enough apart at the crossover time to prevent short circuits and maintain good contact between the wires and pins. In our designs, we tried to avoid pins at the intersection of horizontal and vertical wires (as shown in the Figure 9, the pin spacing at the intersection is manually stretched to prevent problems). Ensuring that good contact is maintained between the wires results in more accurate and reliable data collection.

5.3. Conductive Wiring

However, the link between the circuit board and the insole became a challenge. The main problem is the flexibility of the conductive sewing wire, which made it impossible to use soldering on the conductive wire, as well as making it not easy to insert the wire into the terminals for connection. The wire itself is also very fragile and prone to breakage.

To solve this problem, we stripped the outer shell of the DuPont wire and twisted the hard wire of the DuPont wire to the conductive sewing wire, creating a strong, stable connection. The details are shown in Figure 10. To further ensure the integrity of the connection, we sealed and secured it with hot melt adhesive and heat shrink tubing. The other end of the wire was attached to the circuit board and secured with tape. The final design also takes into account that collecting data during use impacts the wiring connections, so we needed to ensure that the battery and wires do not come off during strenuous action.

5.4. Controller

The controller part of the code was written using C++ code for the Arduino platform. The workflow of controller is shown in Figure 11. The Arduino code provides two main functions, setup() and loop(). The setup function runs once after the controller is powered up, and then the code loops through in loop(). The controller’s setup handles some initialization functions, such as setting the state of each pin and connecting to the wireless network. In the loop function, two tasks are performed: running a check for code updates and executing a sleep task.

To improve the accuracy and stability of the sampling frequency, a special function called interrupter was implemented in the controller, which hangs the current process for a fixed period of time and then inserts a piece of code to execute it. The right-hand part of the controller performs the data collection and sending functions after the interrupt. Furthermore, in order to optimize the performance of the system, a global buffer was added in front of the saved data in order to transfer several rounds of data at once and to avoid the network response time affecting the computing time due to multiple high-frequency TCP transfers.

5.5. Classification Algorithm

We utilize the ResNet(2+1)D algorithm as the classifier for fall detection, as shown in Figure 12. These sensors capture a $10\times 10$ resolution pressure map for each frame, resulting in a unique form of time-series data, termed “stress video”.

We identify ResNet3D as a potential solution due to its proficiency in learning time-varying relationships in video data. Nevertheless, the standard 3D convolution operations in ResNet3D are computationally intensive and less effective at disentangling spatial and temporal features. Hence, we adopted the ResNet(2+1)D architecture, which introduces a novel (2+1)D convolution block. This block separates the 3D convolution into two distinct operations: a 2D spatial convolution to process each frame independently, followed by a 1D temporal convolution to model the temporal dynamics across frames. This decomposition not only simplifies the learning of spatial and temporal features but also reduces the model’s complexity.

Initially, our approach involved directly feeding the $10\times 10$ resolution data from the pressure maps into the network. However, preliminary training results plateaued at a modest 50% accuracy. We hypothesized that the application of the $1\times 3\times 3$ convolution kernel on the low-resolution data ($250\times 10\times 10$) might result in substantial information loss, hindering the model’s learning capability. To counter this, we employed an upsampling strategy, resizing the data to $250\times 112\times 112$. This transformation significantly enhanced the model’s performance, as the increased resolution provided a richer representation of the pressure patterns, allowing the convolutional networks to capture more nuanced spatial and temporal features. We know that the pressure of a foot stepping on an insole in the real world is gradual and smooth, so upsampling can actually restore the pressure distribution in the real world through an appropriate difference method, providing data closer to reality in the network.

6. Experiment and Evaluation

In this section, we first introduce the experimental scenario setup, followed by a description of the types of actions and the dataset utilized in the experiment. Subsequently, we present the analysis results of pressure values for two specific actions: falling and walking. Finally, an analysis of the system’s performance is conducted.

6.1. Experimental Setup

Figure 13 illustrates our indoor experimental setup. We invited five volunteers who wore the pressure insoles and performed specified actions within a $2\times 3$ m action area, as depicted in the diagram. Considering the interior floor was hard, for safety purposes, soft mats were utilized during the fall experiments to provide cushioning and ensure participant safety. The insoles collect plantar pressure data at a frequency of 100 Hz, which was wirelessly transmitted to a laptop via Wi-Fi device. A camera was used to record the actions as ground truth. The collected data was processed using Python 3.7.4.

6.2. Dataset

To assess the system performance, we designed twelve actions categorized into three types, as detailed in

Table 1. Details of actions.

Actions	Number of Samples	Description
Walking	65	Each participant starts from a standing position, takes two steps forward, and then returns to a stationary position at the end of the two steps, waiting for the test time to end.
Sweeping	65	Participants hold a stick to simulate a sweeping motion. They begin by leaning slightly forward and shifting their weight slightly, then perform a sweeping motion to simulate sweeping.
Seating	65	A chair is placed behind the participant. The participant starts standing still and then slowly and naturally sits in the chair until the test period ends, simulating a real-life sitting down situation.
Standing	65	A chair is used in this action. Participants initially sit in the chair, then slowly and naturally stand up from the chair and remain standing until the end of the trial period, simulating a real-life standing up situation.
Walking with a cane	65	The participant holds a cane and simulates an older person walking with a cane. The participant, holding the cane with the right hand, starts from a stationary position, walks with the cane ensuring that part of the pressure on the right foot is taken by the cane. The rest of the requirements are the same as the walking action.
Body leaning	65 × 4	The participant is instructed to perform leaning in one direction, exhibiting a posture with an unstable center of gravity. This action is divided into four categories corresponding to different directions of tilt: front, back, left, and right, to simulate real-life body leaning directions.
Falls	65 × 3	The participants perform falls in three different directions: forward, backward, and leftward. They have the freedom to perform the falls in any manner, as long as the direction of the fall aligns with the specified direction.

. The first type includes five daily activities: walking, sweeping, seating, standing, and walking with a cane. The second type involves four actions associated with a risk of falling, including body leaning forward, backward, left, and right. The third type encompasses three falling actions, specifically falling forward, backward, and to the left. Each action consists of 65 samples.

Participants performed each action for a duration of 5 s. To ensure data consistency and integrity, they were instructed to complete a full set of actions from a stationary position within 5 s and then return to a stationary position. Following this, a pause of 5–10 s was observed to prepare for the next action. To ensure participant safety during fall data collection, we used a mat and allowed participants to fall in any safe manner towards a designated direction, without restricting their landing posture or position.

6.3. Pressure Sensor Data Analysis

To illustrate the differences in plantar pressure distribution across various actions, we analyze the data using two actions: forward falling and walking. Figure 14 depicts the relationship between different phases of these actions and plantar pressure distribution. Figure 14a illustrates the action of forward falling, while Figure 14b depicts walking.

In Figure 14a, the forward fall is divided into four stages: (a) stationary standing, (b) forward-leaning with both forefoot and heel on the ground and center of gravity shifting towards the forefoot, (c) further leaning with only the forefoot touching the ground, and (d) falling, where both forefoot and heel are off the ground, resulting in zero plantar pressure. Each insole is equipped with 45 sensors, and we selected 3 sensors from both the forefoot and heel areas of each insole to plot the real-time pressure value change curves. The x-axis represents time, while the y-axis corresponds to voltage values. The data collected from the pressure sensors are outputted as a distribution of voltage values ranging from 0–4095, with a measurement range of 0–2.450 V. Upper curves in the figure indicate forefoot pressure, while lower curves represent heel pressure, with dashed lines for the left insole and solid lines for the right. The pressure curves reveal distinct trends across different stages. During stage (a), both the forefoot and heel exhibit stable, minor fluctuations in pressure. In stage (b), there is an increase in forefoot pressure accompanied by a decrease in heel pressure. Moving to stage (c), the forefoot pressure continues to rise, reaching a peak, while heel pressure drops to zero. By stage (d), pressure in both the forefoot and heel registers as zero.

In Figure 14b, the walking action is divided into five stages: (a) stationary standing, (b) right foot lifted with left foot on the ground, (c) right foot touching the ground while left foot is lifted, (d) left foot on the ground with right foot lifted, and (e) similar to stage (a). During each walking cycle, the center of gravity on the foot in contact with the ground shifts from the heel to the forefoot. Similarly, we select three sensors from the forefoot and heel of each insole to plot the curves depicting the change in pressure over time. From the figure, we can see the obvious changes in pressure at each stage. In stage (a), participants stand still with evenly distributed pressure on both feet. In stage (b), lifting the right foot reduces its pressure to zero. Stage (c) sees a similar pattern with the left foot, shifting weight to the right. Finally, stages (d) and (e) rebalance the pressure across both feet as walking continues. This sequence of alternating high and zero pressure illustrates a typical walking gait.

Each action is composed of multiple stages, each stage exhibiting distinct characteristics in terms of pressure distribution and variations. Data from 90 pressure sensors, distributed with 45 sensors per insole, are fed into the ResNet3D model. The ResNet3D model is trained to recognize distinct pressure patterns, improving its ability to differentiate between falling and other types of actions.

6.4. Evaluation

The experimental data from this study were used to train a neural network classifier and evaluate its performance. Figure 15 shows the curves of loss and accuracy for the classification of 10 classes of actions. In Figure 16, the classifier achieves an overall accuracy of 91% on the dataset when the three falls are grouped into one fall action. For the detection of falls, the accuracy is 94%, which is a very effective result.

We also trained the classification model for 12 actions, splitting falls into 3 categories (forward, backward, and leftward) for classification. Figure 17 shows the curves of loss and accuracy for the classification of 12 classes of actions. In Figure 18, the classification results achieved a recognition rate of 85.61%. This demonstrates that our system can accurately classify most of the actions in the dataset. However, it is also observed that the major incorrect classifications occur between the three fall categories.

It is notable that there are only 65 samples per action in the current experiment, which is a very small amount of data. This could affect the learning of the neural network and also have a significant impact on the classification of the actions. Despite this limitation, the results of this research are promising and suggest the potential for further development and improvement of this method. Future research should aim to increase the amount of data and test classifiers on larger and more diverse groups of people.

The results of the experimental data in this research show that the proposed approach to fall detection using pressure sensors in insoles and ResNet(2+1)D is promising. It provides a non-invasive, privacy-respecting and efficient solution for fall detection in the senior citizens. The results suggest that this approach has the potential to be further developed and improved to make it more accurate and reliable in detecting falls.

7. Discussion

To assess the system’s performance in real-world scenarios, we engaged eight volunteers (five females, three males) aged between 37 and 85, with an average age of 61, for testing. Participants were instructed to perform actions listed in Table 1 wearing shoes equipped with the smart insole. Foot pressure data were collected in real-time and visualized in MATLAB R2022a to depict pressure distribution. To gather authentic user feedback, a questionnaire was designed, with questions as presented in

Table 2. List of questions in the questionnaire.

Category	Question
Privacy	Q1-1 It protects your privacy when collecting and processing your foot pressure data.
	Q1-2 You would feel uncomfortable if your data were accessed by third parties (e.g., doctors or family members).
Utility	Q2-1 It effectively helps you monitor your foot pressure.
	Q2-2 It is helpful in preventing falls.
	Q2-3 You have a better understanding of your own gait and walking style after using it.
	Q2-4 It is helpful in improving your walking habits and preventing potential health issues.
Portability	Q3-1 It is easy to carry and wear.
	Q3-2 It meets your daily shoe needs in terms of weight and size.
	Q3-3 It is convenient for you to switch between multiple shoes.
	Q3-4 You would be willing to wear it for extended periods of time.
Usability	Q4-1 You are able to use it without any trouble.
	Q4-2 The feeling of using this insole is almost the same as using other insoles.
	Q4-3 Its user interface is intuitive and clear.
	Q4-4 It is easy to use.
Comprehensive	Q5-1 You would like to use such an insole in your daily life.
	Q5-2 It causes discomfort to the soles of your feet.
	Q5-3 You would like to recommend this insole to others who may be at risk of falling.

. We framed questions around five system characteristics: privacy, utility, portability, usability, and comprehensiveness. Responses were scaled from 1 to 5, with 1 indicating strong disagreement and 5 indicating strong agreement. In addition, in the actual questionnaire, the order of the questions is different from that in Table 2.

We designed this questionnaire to comprehensively assess the multifaceted impact of insole systems on the end user. Each question was carefully designed to gather insights into a specific dimension of user experience, as detailed below:

Privacy (Q1-1 and Q1-2): These two questions sought to understand users’ perceptions of protecting their privacy and how comfortable they are with third parties, such as medical personnel or family members, accessing their data. These questions are critical to assessing the level of trust in the system’s data management and identifying any privacy issues that may need to be addressed.

Utility (Q2-1 to Q2-4): These questions are integral to evaluating the functional benefits of a system. They explored the system’s effectiveness in monitoring foot pressure, helping to prevent falls, enhance understanding of gait, and help improve walking habits, allowing the practical application and value of the system to be evaluated.

Portability (Q3-1 to Q3-4): These questions assessed ease of carrying, ease of wearing, ease of integration of the insole into a variety of footwear, and user willingness to wear it for extended periods of time. This assessed whether product design meets user needs for mobility and convenience.

Usability (Q4-1 to Q4-4): Since user-friendliness is critical for product adoption, the questionnaire included inquiries about ease of use, comparison to other insoles, and intuitiveness of the user interface. These questions help us understand the user’s adaptation and comfort level with the insole.

Comprehensive (Q5-1 to Q5-3): Evaluated the overall fit and perceived benefit of the system to the user’s lifestyle with respect to questions such as likelihood of daily use, comfort during use, and willingness to recommend the product to others. Helped us measure overall user satisfaction and acceptance.

7.1. Questionnaire Results

We collected questionnaire responses from eight volunteers and calculated scores based on their answers to each question. Responses ranged from 1–5, with each option representing its respective score. The average score for each category was computed, and the survey results are depicted in Figure 19 and Figure 20.

In Figure 19, the boxplot illustrates the distribution of scores for the system characteristics. The x-axis denotes these characteristics, while the y-axis represents scores. The line within the boxplot indicates the mean score for each characteristic. With a maximum score of 5, the results are as follows: privacy scored 4.13, utility 4.53, portability 3.91, usability 4.09 and comprehensive 4.35. Utility and privacy receive high user ratings, highlighting their positive reception. Conversely, portability shows potential for improvement. The system’s usability and comprehensive characteristics score 4.09 and 4.35, respectively, underscoring its ease of use and users’ perception of it as a well-rounded and beneficial tool.

In Figure 20, the radar chart demonstrates that the system receives favorable feedback across all characteristics. Utility and privacy emerge as significant strengths, while portability represents a primary area for enhancement.

7.2. Privacy Analysis

In privacy assessment of the system, we queried eight participants with Q1-1 and Q1-2. Q1-1 evaluates system privacy perceptions, and Q1-2 probes attitudes on third-party data access. Figure 21 displays user ratings for system privacy, with the x-axis representing user IDs and the y-axis representing scores. Blue bars correspond to Q1-1 ratings, and orange bars to Q1-2. The chart reveals that six users strongly oppose third-party data access, whereas two find it acceptable. Six users give the system’s privacy a score of 4.17, and two rate it 4, underscoring user approval of the system’s privacy.

7.3. Utility Analysis

For system utility evaluation, we implement four questions (Q2-1 to Q2-4) that address foot pressure monitoring, fall prevention, gait understanding and walking habit improvement. Figure 22 shows that participants consistently rate the system’s utility highly, especially for foot pressure monitoring and walking habit improvement, underscoring the system’s effectiveness in fall prevention.

Additionally, of the eight participants surveyed, five have experienced falls while three have not. Categorizing participants based on fall experience, we analyze their ratings on system utility, as depicted in Figure 23. In the figure, orange bars represent ratings from users with fall experience, while blue bars denote those without. Participants with fall experience have an average rating of 4.5, whereas those without score an average of 4.58, further highlighting the system’s effectiveness in fall prevention.

7.4. Portability Analysis

For the system portability assessment, we introduced questions Q3-1 to Q3-4, asking participants to rate the convenience of carrying and wearing the insole, its weight and size, ease of swapping it between different pairs of shoes, and willingness for prolonged use. Figure 24 presents the results. Q3-2 scores the highest at 4.5, followed by Q3-1 and Q3-4, both at 3.75, with Q3-3 being the lowest at 3.43. The data suggest users appreciate the insole’s size and quality, yet its usability requires enhancement, pointing towards further miniaturization in hardware design.

7.5. Usability Analysis

We set up four questions to count the usability of the system, and users scored the system in terms of whether they encountered difficulties in using the system, whether the insole was the same as normal insoles, whether the user interface of the system was intuitive and clear, and whether it was easy to use. Figure 25 shows the scoring results, with Q4-4 scoring the highest at 4.63, followed by Q4-3 at 4.13, Q4-1 at 3.43, and Q4-2 at the lowest at 2.33. It is worth noting that the users’ ratings of Q4-2 are polarized, with some users scoring the insoles in terms of shape and size, believing that the shape is almost the same as that of ordinary insoles, and others scoring the insoles in terms of their functionality and considered the functionality to be completely different from that of a regular insole, thus leading to lower ratings.The ratings in Q4-4 illustrate that the users gave high ratings to the ease of use of the system.

7.6. Comprehensive Analysis

For the comprehensive evaluation of the system, we designed a total of three questions from Q5-1 to Q5-3, inviting users to rate the insole in terms of whether they would like to use this insole in their daily lives, whether the insole caused discomfort, and whether they would recommend this insole to a friend who is at risk of falling. Please note that question Q5-2 was a reverse question and we converted its rating. The results of the research are shown in Figure 26, where users gave a high score of 4.71 for the comfort of the insole, 4.13 for the willingness to use this insole in daily life, and 4.13 for the willingness to recommend this insole to a friend with a risk of falling, which further illustrates the user’s recognition of the system’s comprehensiveness.

8. Conclusions

In conclusion, this research proposes a fall detection system using a shoe insole with a pressure sensor. The experimental results show that our proposed system can accurately classify 12 different actions and detect falls with 94% accuracy. The overall recognition rate of the 12 actions is 85.61%. These results demonstrate the potential of our proposed system for fall detection in the senior citizens.

However, the major false identifications occurred between the three fall categories, suggesting that there is still room for improvement in fine-grained fall detection. In addition, the experiment used a small sample size, which may have an impact on the results.

In future work, we aim to enhance the performance of our system by utilizing larger sample sizes and applying more advanced deep learning algorithms. Additionally, we plan to explore the reduction of sensor quantity and optimization of sensor distribution in the insoles to lower the model complexity and computational time. For the production of insoles, we will use an embroidery machine to improve the accuracy of the distribution of sensors and the efficiency of the production, and to reduce the influence of human factors on the variability of each pair of insoles. At the same time, we will consider the effect of plantar temperature on the conductivity of the sensors, and add heat-insulating materials to the insoles. Finally, we intend to integrate the developed insoles with Wi-Fi devices to enable human activity recognition and location tracking within the context of smart home applications.