In-Home Evaluation of the Neo Care Artificial Intelligence Sound-Based Fall Detection System

Abstract

The Neo Care home monitoring system aims to detect falls and other events using artificial intelligence. This study evaluated Neo Care’s accuracy and explored user perceptions through a 12-week in-home trial with 18 households of adults aged 65+ years old at risk of falls (mean age: 75.3 years old; 67% female). Participants logged events that were cross-referenced with Neo Care logs to calculate sensitivity and specificity for fall detection and response. Qualitative interviews gathered in-depth user feedback. During the trial, 28 falls/events were documented, with 12 eligible for analysis as others occurred outside the home or when devices were offline. Neo Care was activated 4939 times—4930 by everyday household sounds and 9 by actual falls. Fall detection sensitivity was 75.00% and specificity 6.80%. For responding to falls, sensitivity was 62.50% and specificity 17.28%. Users felt more secure with Neo Care but identified needs for further calibration to improve accuracy. Advantages included avoiding wearables, while key challenges were misinterpreting noises and occasional technical issues like going offline. Suggested improvements were visual indicators, trigger words, and outdoor capability. The study demonstrated Neo Care’s potential with modifications. Users found it beneficial, but highlighted areas for improvement. Real-world evaluations and user-centered design are crucial for healthcare technology development.

1. Introduction

Falls among the elderly constitute a burgeoning public health crisis with far-reaching implications for both individuals and healthcare systems [1]. These incidents often result in a spectrum of adverse outcomes, ranging from minor abrasions and bone fractures to severe complications, such as loss of consciousness or even fatality [2,3]. Timely medical intervention is imperative, not only to alleviate the immediate physical and psychological trauma, but also to curtail the escalating burden on healthcare resources [4,5].

Monitoring falls, particularly within the domestic setting, presents unique challenges [6]. Conventional wearable technologies, like alert pendants, are limited in their efficacy. Their utility is predicated on the user’s ability to activate them, which is not possible if the individual is unconscious or incapacitated during the fall [6]. Additionally, the stigma associated with these devices or the fear of being a burden may deter their usage among the elderly [7].

Recent advancements in artificial intelligence (AI) and machine learning offer promising avenues for enhancing fall detection systems. Smartwatches, for instance, come equipped with sophisticated algorithms capable of autonomously alerting emergency services. However, these devices inherit the fundamental limitation of all wearables: they must be worn to function. Moreover, their algorithms are not foolproof and are susceptible to false positives and missed detections [8].

A prior systematic review [9] that sought to examine the existing literature on the use of AI techniques for analyzing data pertaining to postural control and fall risk assessment identified and included a total of 22 relevant studies. The findings showed that various AI techniques, including machine learning models (logistic regression, decision trees, random forests, boosted trees, and support vector machines) and deep learning models (convolutional neural networks [CNNs], long short-term memory [LSTM] networks, and associative skill memories) were used to develop predictive models for classifying the risk of falls [9]. The studies demonstrated the feasibility and promising accuracy of these AI approaches, with most achieving over 70% accuracy in fall risk prediction [9]. Techniques like XGBoost, random forests with feature engineering, CNNs, and LSTMs were found to be particularly effective [9]. Several studies compared different AI methods or combined multiple parameters, such as gait-related features and seated position data, to improve classification accuracy. The highest reported accuracy was 92.7% using CNNs to differentiate between normal gait and fall events [9]. While in a previous study, Kulurkar et al. [10] evaluated an “Internet of Things” (IoT) elderly fall detection system leveraging wearable accelerometers, big data analytics from the MobiAct dataset, cloud computing with Apache Flink, and LSTM machine learning models on an IoT gateway, achieving 95.87% real-time fall detection accuracy to improve elderly care and independence. Overall, the findings highlighted the potential for applying various AI techniques to the issues of falls. To date, the most common approaches are either through accelerometry-based wearables, or video-based systems. Overall, previous research has predominantly involved wearable sensors or accelerometers [11,12,13,14,15], studies utilizing tests or assessments in combination with camera-based systems [16,17], or studies relying solely on data from medical records [18]. The use of smart speakers equipped with AI models for audio-based fall detection could offer a more convenient and unobtrusive approach to monitoring fall risk, particularly in home settings.

Environmental monitoring systems, leveraging the IoT, offer an alternative approach. These systems monitor routine activities through smart appliances and can trigger alerts based on significant deviations or anomalies [19]. Another emerging solution utilizes smart speakers trained to recognize sounds indicative of emergencies, such as a thud or glass breaking. However, exploring the use of smart speakers equipped with AI models for audio-based fall detection could offer a more convenient and unobtrusive approach to monitoring fall risk, particularly in home settings.

The Neo Care Monitoring System, hereinafter referred to as “Neo”, is a compact device that employs AI and assistive technology to identify household sounds that may signify emergencies, including falls (Figure 1). The Neo operates similarly to popular smart speakers, like Google Home. Its main purpose is to listen for sounds that could signal potential issues, such as cries or the sound of breaking glass. It records 10 s audio clips of various events for analysis and can engage in voice interactions to assist occupants during emergencies. Upon detecting a concerning sound, Neo activates an AI-driven aged care assistant that converses with the user to assess the situation through a series of questions. If a crisis is confirmed, or there is no response, Neo attempts to contact a designated caregiver or relative’s mobile phone. The core algorithm driving Neo’s functionality is a machine learning model trained on a vast dataset of audio samples representing various household sounds and events, including potential emergency situations like falls (Figure 2). This algorithm continuously analyzes the incoming audio stream from Neo’s microphone, identifying patterns that may signify a concerning event, which then triggers further analysis and response workflows. Additionally, the algorithm leverages natural language processing capabilities to engage in contextual voice interactions with users, interpreting responses to accurately assess emergencies and determine appropriate actions. This system offers a potentially more cost-effective solution compared to other environmental monitoring systems, especially those requiring a multitude of smart devices and sensors. However, this innovation solution requires rigorous evaluation before it can be widely adopted.

The primary objective of this study was to assess the accuracy and responsiveness of the Neo in real-world emergency scenarios, focusing on falls. Specifically, we aimed to answer the following research questions:

• What is the technical accuracy of the Neo in detecting events of concern, measured in terms of sensitivity and specificity?
• How effectively is the Neo system’s post-detection response also measured in terms of sensitivity and specificity?

Additionally, we sought to understand users’ experiences and perceptions of the Neo system, focusing on its perceived effectiveness, ease of use, and areas for improvement. The study also sought to understand the factors that influence future user adoption, including their willingness to recommend the system, and to incur associated costs.

2. Materials and Methods

2.1. Study Design

This study utilized a sequential mixed-methodology design, where both quantitative and qualitative data were collected in two separate phases [20]. The first phase employed a quasi-experimental approach, involving participants in a 12-week in-home trial of the Neo. This phase was followed by a qualitative study that delved into user perspectives and experiences related to the Neo. The study was designed and reported according to the Transparent Reporting of Evaluations with Non-randomized Designs (TREND) [21] and Consolidated Criteria for Reporting Qualitative Research (COREQ) [22] statements, which guided the quasi-experimental and qualitative research components, respectively. Ethical approval for the study was obtained from the University of South Australia Human Research Ethics Committee (Application ID: #204274). Informed consent was obtained from all participants prior to participation in the study.

2.2. Context

This research was conducted as a joint venture between Neo Care, a medical technology startup, an Aged Care provider, LifeCare, and university researchers (CM, KD, RC, BS, and SB). The team collaborated under an innovation grant provided by the South Australian government. This funding scheme aimed to foster collaboration between the industry and the university sector to promote advancements in health and medical innovation.

2.3. Participants and Recruitment

Participants were individuals aged above 65 years old who met the following criteria: ability to speak English; experienced a fall within the last 12 months; living either in a residential home or in a LifeCare retirement village within Adelaide, South Australia; had a home-based wireless internet connection; access to a telephone or smartphone; and had no pre-diagnosed cognitive impairments (dementia) or significant hearing or speech impairments.

Various recruitment strategies were employed, including both purposive and snowball sampling. Information flyers and face-to-face sessions were conducted at retirement villages and associated allied healthcare providers (e.g., physiotherapy and occupational therapy clinics associated with the retirement villages). Interested individuals completed an online expression of interest form, after which a research team member confirmed eligibility, addressed questions, and obtained consent. Initially, we aimed to include 30 households in the study. This target was not determined by statistical power calculations but was considered a practical size for a pilot study. However, we had to halt recruitment after enrolling 18 households. This was due to a combination of factors: delays in device manufacturing, software development hurdles, and disruptions caused by the COVID-19 pandemic. All participants completing the in-home trial were invited for the qualitative study. They were also encouraged to refer family members experienced with the Neo, but not residing with them. Participants were recruited between July and October 2022.

2.4. Phase 1: In-Home Trial

2.4.1. Procedure

Participants were visited at home by two research team members for Neo installation. Typically, two Neos were installed: one in the living area and another near the bathroom. Participants were provided a logbook to record events of concern and interactions with the Neo. Weekly phone check-ins were conducted to collect logbook data and troubleshoot any issues. The logbook was collected at the end of the 12-week trial.

2.4.2. Data Management

Post-trial, logbook events were cross-referenced with server logs from Neo Care. Categories were created for (1) event detection and (2) response to potentially significant events, including:

(i)

Neo activated by a true event (fall or other event of concern);

(ii)

Neo activated by everyday sounds (i.e., Neo detected a sound that could indicate a potentially significant event, however a true event did not occur);

(iii)

Neo heard an everyday sound, and correctly determined that it was not significant, and so refrained from activating.

The following categories were created related to the Neo’s response to potentially significant events:

(i)

Correct alert: The Neo detected a potentially significant event, asked the user if they were ok, and correctly determined an alert was required (because the user responded that they were not ok);

(ii)

Correct no action: The Neo detected a potentially significant event, asked the user if they were ok, and correctly determined an alert was not required (because the user responded that they were ok);

(iii)

Incorrect alert: The Neo detected a potentially significant event, asked the user if they were ok, and incorrectly determined an alert was required (because the Neo either did not hear or did not understand the user’s response);

(iv)

Incorrect no action: The Neo detected a potentially significant event, asked the user if they were ok, and incorrectly determined no further action was required (because the user responded that they were not ok);

(v)

System error: The Neo detected a potentially significant event, but there was a system error resulting in a response failure (e.g., a looping response where the participant responded that they were ok, but the Neo asked again).

2.5. Analysis

Descriptive statistics (n and %) were used to describe the frequency of the event detection and response categories. The data were then used to calculate the sensitivity and specificity of fall detection and response. The sensitivity refers to the device’s ability to correctly identify participants who experienced a fall and was calculated as: [true positives/(true positives + false negatives)]. The specificity refers to the device’s ability to correctly identify participants not experiencing a fall, was calculated as: [true negatives/(true negatives + false positives)]. In particular, in the context of fall detection, sensitivity refers to the ability of the Neo device to correctly identify actual falls (i.e., how well the device detects falls when they do occur), while specificity refers to the ability of the Neo device to correctly identify non-fall events (i.e., how well the device avoids false alarms by not triggering when a fall has not occurred). In the context of fall response data, sensitivity refers to the ability of the Neo device to correctly alert a caregiver when a fall actually occurs (i.e., how well the device responds when intervention is needed), while specificity refers to the ability of the Neo device to correctly refrain from initiating an unnecessary response (such as alerting a caregiver) when a fall has not occurred (i.e., how well the device avoids false alerts in situations where no intervention is needed). Sensitivity and specificity with exact Clopper–Pearson confidence intervals [2] were calculated in STATA (version 17, StataCorp LLC, College Station, TX, USA).

2.6. Phase 2: User Experience

2.6.1. Procedure

Semi-structured interviews lasting between 30 and 60 min were conducted over the phone by a research assistant (KD; female; with prior experience in semi-structured interviews and analysis) from December 2022 to March 2023. These interviews were recorded and transcribed verbatim using the online transcription service Descript. No field notes were taken during the interviews. A pre-tested interview guide developed by the research team guided the conversations.

The research assistant conducting the interviews was already familiar with the participants due to involvement in recruitment and weekly check-ins during Phase 1. To mitigate any potential bias arising from this familiarity, the assistant adhered strictly to the interview guide, refrained from sharing personal anecdotes, and engaged in reflexivity. This involved regular debriefings and feedback sessions with the research team, as well as self-reflection on their role in the study. The interview questions focused on user experiences, troubleshooting, and future use. Participants were asked about their experiences with the system, such as “What did you like about the Neo Care Monitoring System?” and “What challenges did you face?” For troubleshooting, they were asked, “Did you experience any system issues when using Neo Care? If so, how did you go about finding a solution?” The future use section included questions like, “Would you continue using the Neo Care system?” and “Would you be willing to pay for Neo Care in the future? Why or why not?” Due to practical constraints, participants were not provided the opportunity to review or correct their interview transcripts.

2.6.2. Analysis

Data management and analysis were conducted using NVivo software (Version 10; QSR International Pty Ltd., Melbourne, Australia). We employed a thematic analysis to identify and report emerging themes in the data [23,24]. Our approach was guided by a qualitative descriptive methodology, which uses an inductive approach for data interpretation. This is particularly useful when the subject matter is not well-understood and helps to manage biases and assumptions [25,26]. The analysis followed six phases: familiarization with the data, initial code generation, theme searching, theme review, theme definition and naming, and report production [23]. One researcher (KD) conducted the coding while engaging in thorough discussions and receiving close oversight from team members CM and SB. A copy of the thematic analysis codebook and hierarchical map of codes within the main themes are shown in Table S1 and Figure S1, respectively.

To ensure the study’s credibility, dependability, transferability, and confirmability, we implemented rigorous strategies [20,27]. Credibility was maintained through investigator triangulation, involving three team members (KD, CM, and SB) in coding, interpretation, and feedback. Dependability was achieved by using rich codes and direct quotes to develop themes, supported by a diverse range of participant codes. Transferability was facilitated through regular team meetings and documentation of decisions. Confirmability was upheld by documenting assumptions, biases, and allowing for peer review during theme development. The research team independently reviewed transcripts, discussing any differences in the themes generated.

3. Results

3.1. Participant Characteristics

The study was completed in 18 households, involving 18 primary participants and 11 additional household members. The average age of primary participants was 75.3 years old (SD = 7.8), with a majority being female (n = 12, 67%). Most lived with a spouse or family member (n = 12, 67%), while the rest lived alone (n = 6, 33%). A significant majority resided independently in their own homes (n = 15, 83%), and the remaining participants (n = 3, 17%) lived in retirement villages. Most did not receive in-home care (n = 16, 89%).

3.2. Phase 1: In-Home Trial

Descriptive analysis of the in-home trial: Neo activations, events of concern, and technical performance. During the trial period, there were 28 falls or events of concern recorded in the logbooks. Of these, 12 occurred within the detectable zone of a Neo and were considered eligible for analysis. The remaining 16 were excluded due to the event occurring outdoors (n = 6) or technical problems (n = 10) resulting in the Neo device being disconnected at the time of the event, either because it was intentionally turned off by the participant (e.g., due to irritation from it being mistakenly triggered by everyday sounds) or WiFi disconnection.

Server logs showed that the Neos detected a sound 5300 times in total. Most activations (n = 4930) were triggered by everyday sounds, while only nine were triggered by actual events, all of which were falls. The Neos failed to detect 3 out of the 12 eligible falls.

The server logs showed that the most common sounds detected by the Neo were slapping/smacking (n = 1098), smoke detectors (n = 691), and thumps/thuds (n = 670,

Table 1. Overview of all the occurrences where the Neo was activated.

	Neo Activated by a True Fall (n = 9)	Neo Activated by Everyday Sounds (n = 4930)	Neo Detected an Everyday Sound and Refrained from Activating (n = 361)	Total Detected Sounds (n = 5300)
Banging	0	14	0	14
Something breaking	0	2	0	2
Coughing	0	0	105	105
Crying/ sobbing	1	664	0	665
Fire alarm	0	115	10	125
Glass breaking	0	41	0	41
Groaning	3	32	0	35
Screaming	0	655	0	655
Shattering	0	4	0	4
Shouting	0	56	0	56
Slapping/ smacking	2	1096	0	1098
Smoke detector	0	579	112	691
Throat clearing	0	0	134	134
Thump or thud	1	669	0	670
Wail or moan	1	96	0	97
Whimpering	1	633	0	634
Yelling	0	274	0	274

). Of the 9 correctly detected falls, the Neos detected these sounds as groaning (n = 3), slapping/smacking (n = 2), crying/sobbing (n = 1), thump/thud (n = 1), wail/moan (n = 1) and whimpering (n = 1). Reports from the logbooks suggested that, while the Neo detected the above sounds, these detections were often caused by other noises, such as dogs barking and the television.

An overview of the 4939 Neo activations is summarized in

Table 2. Overview of response outcomes.

Response Outcome	Description	n (%)
Correct Alert	The Neo detected a potentially significant event, asked the user if they were ok, and correctly determined an alert was required (because the user responded that they were not ok).	4 (0.1%)
Correct No Action	Correct No action: The Neo detected a potentially significant event, asked the user if they were ok, and correctly determined an alert was not required (because the user responded that they were ok).	1430 (29%)
Incorrect Alert	The Neo detected a potentially significant event, asked the user if they were ok, and incorrectly determined an alert was required (because the Neo either did not hear or did not understand the user response).	299 (6.1%)
Incorrect No Action	The Neo detected a potentially significant event, asked the user if they were ok, and incorrectly determined no further action was required (because the user responded that they were not ok).	1 (<0.1%)
System Error	The Neo detected a potentially significant event, but there was a system error resulting in a response failure (e.g., a looping response where the participant responded that they were ok but the Neo asked again).	3205 (64.9%)

. Out of these, the Neo device responded correctly 1434 times. This includes four instances where it appropriately alerted a caregiver following a fall and 1430 instances where it correctly took no action, as no caregiver alert was needed.

Among the nine falls that the Neo device successfully detected, it responded correctly in five cases. Specifically, it alerted a caregiver in four instances and took no action in one case, as the participant confirmed they were fine. However, there was one instance where Neo failed to alert a caregiver when it should have (i.e., when the participant indicated that they needed assistance after the fall). Additionally, technical issues interfered with the device’s performance for the remaining three falls, resulting in the device not asking if the participants needed assistance after a fall.

Table 3. Neo’s detection of potentially significant events.

	Neo’s Fall Detection, n=		Total, n=
Fall Status	Did Not Detect Fall	Detected Fall
No fall	361	4930	5291
Fall	3	9	12
Total	364	4939

presents the Neo’s performance in detecting potentially significant events, categorized by the actual occurrence of a fall, and detections. Similarly,

Table 4. Neo’s response to potentially significant events.

	Neo’s Response 1, n=		Total, n=
Alert requirement	Inappropriate response	Appropriate response
Alert not required	299	1430	1730
Alert required	3	5	8 2
Total	302	1436

¹ This table does not include system errors (n = 3205). ² Alert was only required in 8 of the 9 detected falls, since in 1 instance, the participant told the device they were ok.

presents the Neo’s performance in responding to potentially significant events, categorized by whether or not an alert was required, and whether or not an alert was sent. Finally, the sensitivity and specificity metrics for the Neo’s ability to detect and respond to falls are detailed in

Table 5. Accuracy of the Neo to detect and respond to falls.

	Value (95% CI)
Fall detection Sensitivity Specificity	75.00% (42.81% to 94.51%) 6.80% (6.16% to 7.54%)
Fall response Sensitivity Specificity	62.50% (24.49% to 91.48%) 17.28% (15.53% to 19.15%)

. The sensitivity for falls detection was 75.00% (95% CI: 42.81% to 94.51%) and specificity was 6.8% (6.16% to 7.54%). The sensitivity and specificity for the ability of the Neo to respond to falls was 62.50% (24.49% to 91.48%) and 17.28% (15.53% to 19.15%), respectively.

3.3. Phase 2: User Experience

Out of the 18 households involved in the in-home trial, eight participants agreed to participate in user experience interviews. Additionally, one family member, referred by a trial participant, also consented to join the study. Thematic analysis revealed six main themes: (1) Areas for Improvement, (2) Challenges, (3) Decision Making in Adopting a Monitoring System, (4) Future System Use, (5) Cost Considerations, and (6) Positive Anecdotes. Sub-themes were also identified within each main theme.

3.4. Theme 1: Areas for Improvement

Participants pointed out several aspects of the Neo system that could be enhanced. A recurring suggestion was the need for more specific and customizable sound detection to suit individual homes.

“I would suggest you probably have to come into an individual home and try and record and replicate those noises. So that it became specific to yours, to the person’s home, because there must be all sorts of other factors like how big the room is in which you make the noise or I’m sure there are lots of reasons why it would react to one noise in one house and not in another.” (NCTI117P1).

The home’s layout was also mentioned as a factor affecting the Neo’s accuracy. Participants noted that, while having multiple Neos could be beneficial, it also presents challenges related to power source availability.

“This house has a lot of rooms. So, unless I’ve got one of those things everywhere [Neos], which means I’m going to have to have a lot of plugs everywhere.” (NCTI15P1).

Sub Theme 1: Additional Features

Participants discussed the potential value of additional features that could be added to the Neo. The most common suggestions were visual in indicators for sound events, such as flashing lights for doorbell rings, to assist the hearing-impaired; auditory reminders for tasks, like taking medication; an activity monitor to notify family general movement; and an outdoor version suitable for the garden, which some participants describing as essential.

“I think that is a must. Anyone with a garden would want one.” (NCTI15).

Participants also suggested that specific trigger words, like “help” or “I need assistance”, could be programmed into the Neo. They noted that the system’s sensitivity levels needed refinement; it was sometimes too sensitive and other times not sensitive enough.

“So, I guess it needs to be developed more to the sounds of somebody calling out for help.” (NCTI04P1).

“Making it more sensitive to people and not animals, you know?” (NCTI19CP1).

Other valuable features mentioned included integrating the Neo with phone calendars for verbal appointment reminders and other health-related prompts, such as hydration reminders.

“It would be great if it was connected to your calendar on your phone so you don’t forget you’ve got an appointment with the doctor or you’ve got dinner or lunch today.” (NCTI13P1).

3.5. Theme 2: Challenges

Participants identified several issues with the Neo system, ranging from design limitations to functional concerns.

3.5.1. Sub Theme 1: Neo Issues

Participants criticized Neo’s constant need for a power source and its sensitivity to everyday noises, with some even considered unplugging the device due to frequent false alarms.

“I was having to always tell it I’m ok. If you’ve got it going off 3, 4, 5 times in an hour you know, you’re going to pull it out the wall.” (NCTI28).

This concern was echoed by a family member of another participant, who worried that frustration with false alarms could lead to the device being unplugged, compromising safety.

“If they were frustrated with it, I would worry that they would just find a way to unplug it.” (NCTF29).

The plug-in feature of the Neo was also criticized for limiting the device’s mobility within the home.

“I can’t see the point of having something like that plugged into my wall in a room unless I’m in that room and something happens to me in that room.” (NCTI15).

3.5.2. Sub Theme 2: Technical Issues

Some Neos experienced connectivity issues, going offline or disconnecting from the Wi-Fi, which rendered them inactive for periods.

“Sometimes we haven’t heard it for weeks and then all of a sudden, it starts up again.” (NCTI19).

Additionally, the Neo sometimes failed to hear or correctly interpret verbal responses from users.

“Well, sometimes the system [Neo] didn’t hear our response. It was saying, are you ok? We’d say yes, but we had to repeat it a few times for it to hear us.” (NCTI17P1).

3.5.3. Sub Theme 3: Responding to Everyday Sounds

Participants reported that the Neo was often triggered by non-concerning noises, such as pets or household activities.

“It was going off a lot whenever the dog would cry. It would say, I can hear crying. Are you ok? Which is great if I were crying, but it was the dog.” (NCTI28).

“I would be cutting vegetables or the microwave would beep and it would go off, I was usually in the middle of cooking so it was a little bit annoying.” (NCTI04P1).

3.5.4. Sub Theme 4: Missed Falls

Some participants experienced falls that the Neo failed to detect, leading to disappointment, surprise, or a loss of trust in the system.

“There were couple of times that he did have a fall and it didn’t respond. Well it was a bit disappointing, I suppose to think that was obviously one of the main reasons for it is that it’s an alarm for things like that. So that was, I guess, a bit disappointing.” (NCTI104P1).

3.5.5. Sub Theme 5: Tech Savvy

Some participants were apprehensive that older users might find the phone application daunting and may need assistance to navigate it.

“You’re forgetting a lot of older people don’t like using phones, like me. I don’t like using phones and an extra app or something is daunting.” (NCTI24).

However, this perspective was not universal. A family member countered that many older individuals are already accustomed to smartphones and regularly engage with various apps.

“I mean, apps and that, that’s the way it is these days. That’s what everyone does. So even the older generation generally have some sort of smartphone that would have an app on it.” (NCTF29).

3.6. Theme 3: Decisions to Acquire a Monitoring System

Participants shared their experiences and considerations regarding the use of monitoring devices or personal alarms. A variety of experiences were reported: some participants had used fall detection watches or pendants, while others had not used any monitoring systems despite experiencing near falls or minor fall events. Those who opted not to use a system often cited the presence of someone at home, like a spouse or family member, who could assist in the event of a fall. Some felt self-assured about their stability and did not see the need for a device.

“I feel confident enough to not to need one. I’ve had a lot of incidents of falls around the property. It’s actually too dangerous for me really. But it’s mainly myself, it’s clumsiness on my behalf.” (NCTI19).

Sub Theme 1: Personal Experiences

Participants described that their decision to use the Neo was often influenced by past experiences, either their own or those of people they knew. Factors such as having experienced a fall, knowing someone who had a severe fall, or recognizing their own risk of falling due to their current condition played a role in their decision making.

“I wanted one because I had a friend, and she wouldn’t wear one and she died.” (NCTI24).

3.7. Theme 4: Future Use of the Neo

Participants discussed their willingness to continue using the Neo in the future. Some indicated that they would be inclined to use the Neo if it were improved, whilst others stated that a portable system, like a watch, would better suit their current, busy lifestyle.

“Not at the moment because I think there’s still a bit to be sorted out. I think that down the track, if, if other things were incorporated, I think yes I would use it.” (NCTI04).

“Personally? At the moment, I would choose a watch because I’m a busy person. I’m in and out all day.” (NCTI19).

Sub Theme 1: Aged Care Facilities

When asked about potential beneficiaries of the Neo, most participants felt that the system could be a valuable addition to aged care facilities. They suggested that the Neo could be integrated into individual rooms to assist with staffing challenges and enhance resident safety.

“If they’re not checking on you all the time and if you fall over in the first 15 min, you are laying on the ground unconscious bleeding for four hours before someone finds you. The Neo would solve that problem.” (NCTI27P1).

3.8. Theme 5: Cost

Whilst participants were provided with two Neo devices free of charge for the trial, they were asked about their willingness to pay for the Neo system into the future, including whether they would prefer for monthly, quarterly, or upfront payments. Opinions were mixed, with some preferring monthly or quarterly payments and one participant favoring a lump sum. Regarding the price, some participants indicated they would be willing to pay any amount if the Neo met their needs, as long as the cost was not exorbitant. Others specified a range of AUD 100–200 per year.

“I think it’s an impossible question for us, because the reality is if we thought we needed a comprehensive system and it supplied that, then, you know, almost wouldn’t matter in a sense what it costs.” (NCTI17P1).

Sub Theme 1: Supported by Allied Health Providers

Participants expressed that they would be more likely to adopt a monitoring system if it came recommended by healthcare professionals, like doctors or occupational therapists, or if the cost was subsidized by an organization or provider.

“If you get a specialist in and they say this is what you need for your safety, for your wellbeing, and for your continued independence—and there’s the keyword, independence—they’ll pay for just about anything because they want people to be at home.” (NCTI27P1).

3.9. Theme 6: Positive Anecdotes

Despite encountering challenges during the in-home trial, the majority of participants reported positive experiences and held a favorable perception of the Neo. Many praised the Neo for its convenience, considering it a great alternative to wearable systems.

“It was convenient. I mean, you didn’t have to carry it around with you and you didn’t have to wear it.” (NCTI19).

3.9.1. Sub Theme 1: A Sense of Security

Some participants described that using the Neo enhanced their sense of security.

“It made you feel more secure. Like a guardian angel just sort of watching over us. We just really enjoyed knowing that we had that.” (NCTI13).

3.9.2. Sub Theme 2: Recommending to Others

The majority of participants were enthusiastic about recommending the Neo to friends and family, although some suggested that additional features and further testing would make it even more recommendable. The Neo was praised as an alternative to wearable systems, particularly for those who might forget to charge them or dislike wearing them.

“People have got the necklaces, but they don’t always wear them. They’ll leave them in the bedroom or in the bathroom, they take them off to have a shower and then they forget to put them back on.” (NCTI04P1).

However, one participant had reservations about recommending the Neo due to its performance issues.

“As it is now? No, I wouldn’t. If it develops to a stage where it works, yes. I think it would be beneficial.” (NCTI27P1).

3.9.3. Sub Theme 3: Neo Compared to Other Systems

Participants frequently compared the Neo to other monitoring systems, like wearable pendants, pointing out limitations such as forgetting to wear them or disliking the physical sensation of wearing them. The Neo was generally viewed more favorably because it does not require the user to remember to wear something.

“He doesn’t really need a pendant at the moment. Only when I’m not here. So, probably the Neo one would be better. Because I don’t like wearing a pendant. I don’t like something around my neck like that” (NCTI13P1).

Another participant appreciated the Neo’s flexibility in emergency response.

“The alarm system [Neo] I have now goes straight through to an ambulance. I would like to have one where I can ring up my family, instead to just get somebody to come in and say, well, look, I think you’ll be all right. That’s all you need when you get older.” (NCTI24).

4. Discussion

This study rigorously evaluated the Neo Care Monitoring System, examining both its accuracy in real-world settings and its user experience, including perceived benefits and drawbacks and future utility. The system was frequently activated, but showed only moderate accuracy, primarily due to difficulties in differentiating everyday sounds from noises that signified genuine concerns, such as falls. User interviews reinforced the need for better calibration to improve this accuracy. The Neo’s strengths included providing a sense of security and serving as a wearable-free safety monitoring alternative. However, the system also faced significant challenges, including misinterpreting everyday sounds, failure to detect some falls, and experiencing technical issues, like going offline. Users suggested several improvements, such as adding visual cues for hearing impaired users, voice-activated triggers, and an outdoor version. The majority of participants stated they would be willing to continue using the Neo after these adjustments and saw its potential utility in aged care facilities.

Our study found that the Neo had a relatively high sensitivity of 75% for detecting falls, but struggled with low specificity, leading to frequent false positives. While high sensitivity is essential for not missing critical events like falls, it often came at the expense of specificity, causing the system to frequently misinterpret normal activities as falls. These findings are notably similar to those of Aziz et al. [3], who studied a tri-axial accelerometer wearable in 19 older adults under free-living conditions. They reported an 80% sensitivity rate, detecting 8 out of 10 actual falls, but also had a high false positive rate of up to 0.3 false alarms per hour of wear. On the other hand, Chaudhuri et al.’s study with 18 participants using an accelerometer in free-living conditions reported a low sensitivity of 25% but a high specificity of 91% [28], underscoring the trade-off between sensitivity and specificity in monitoring devices. Two previous studies also evaluated fall detection using sound signals [29] or smart speakers [30]. A previous study that evaluated a novel system utilizing both passive and active acoustic signals, implemented with a multi-modal classification framework, achieved 99% accuracy in typical conditions and over 90% accuracy in noisy or non-line-of-sight environments [29]. Lin et al. [30] evaluated a wearable device worn on the chest and a smart speaker or IoT device for confirmatory purposes. The results demonstrated that the system exhibited high sensitivity (0.94–0.96), specificity (0.95–0.97), and accuracy (0.95–0.96) in accurately detecting falls that occurred during transitions from sitting to standing, standing to sitting, and while in a standing or seated position. However, this was a lab-based study and not conducted in a real-world home setting. Interestingly, our study observed a notably lower performance compared with previous research. For example, specificity in the present study was 6.8%, suggesting potential variations in device performance and user population. Interpreting this lower specificity rate may imply that the Neo could be more effective in accurately identifying fall-related events while minimizing unnecessary alerts, potentially improving user experience and caregiver burden.

Our study addresses the growing call for real-world testing of fall detection systems. Interestingly, most studies of fall detection systems are conducted in laboratory settings, typically with young participants who simulate falls. Highlighting this point, a systematic review of fall detection devices identified 92 studies, of which just 7% involved older adults being monitored in a real-world setting [31]. When tested under artificial conditions, the studies tend to reach far favorable results. For example, the same system described in the Chaudhuri et al. study (with sensitivity 25% and specificity 91% in real-world conditions) yielded a sensitivity range of 94.1–94.4% and specificity of 92.1–94.6% in a laboratory setting [28]. This underscores the critical need for real-world evaluations like ours, which offer a more accurate and comprehensive understanding of how fall detection systems perform in the environments where they are actually used.

The qualitative analysis in Phase 2 enriched our understanding of the Neo Care Monitoring System’s performance, revealing a complex interplay between user experience and system functionality. A number of participants suggested that, in addition to its auto-detection features, the Neo could be trained to recognize a trigger word (e.g., “Help!”) to allow users to manually call for help in the case of the system failing to trigger automatically. This could potentially improve the sensitivity performance (given that the Neo missed 3 out of 12 falls, which anecdotally occurred because the falls were quiet). Another common suggestion for improvement was a system customization feature (for example, reducing the Neo’s sensitivity to crying, screaming or whimpering noises in homes with dogs or young visiting children). Such a feature is likely to improve the device’s specificity, though as already acknowledged, actions to improve specificity will risk harming the device’s sensitivity. Participants’ suggestion that personalized calibration aligned with previous research advocating for machine learning algorithms in healthcare settings [32].

On the theme of future use, participants expressed varied preferences. While some were open to continuing with the Neo system if enhancements were made, and appreciated that it offered a non-wearable monitoring option, others leaned towards wearable devices, a trend noted in other studies [33]. The potential for Neo’s application in institutional settings, like aged care facilities, suggests a broader scope for its utility, consistent with recent calls for versatile monitoring solutions in such environments [34]. Cost considerations were diverse, but participants generally indicated a willingness to invest in a reliable and effective system. The influence of healthcare providers in shaping these choices was noteworthy and aligns with existing research emphasizing the role of clinicians in technology adoption [35]. The Neo system’s non-reliance on wearable technology offers a unique advantage, addressing common issues of forgetfulness or discomfort associated with wearable devices [31].

4.1. Strengths and Limitations

A key strength of the study was that it was undertaken in real-world conditions, addressing a notable gap in the existing literature on fall detection systems. This ensured that the findings have broader applicability and are reflective of genuine user experiences. Utilizing a mixed-methods approach, the research integrated quantitative metrics, such as sensitivity and specificity, with qualitative data from user interviews. This offered a multi-faceted understanding of the Neo system’s performance, usability, and potential areas for improvement. The study was conducted as a collaboration between academic researchers and industry professionals. This partnership brings together the methodological rigor commonly found in academic settings with the more agile development timelines typical of the tech industry. As a result, the study adheres to stringent research standards, while also being in line with industry objectives for the evidence-based improvement of the Neo system.

The limitations of the study warrant acknowledgment. First, the study was unable to meet its target sample size due to a confluence of factors, including COVID-19-related delays, manufacturing issues, and commercial and budgetary constraints. This limitation suggests that ongoing evaluation and algorithmic refinements could potentially improve the sensitivity and specificity values reported here. Second, technical challenges, such as devices going offline and frequent false activations, compromised data quality. Notably, 10 falls occurred while devices were offline, a missed opportunity given the rarity of fall events during the trial period. Third, despite measures like logbooks, weekly phone calls, and the exclusion of participants with known cognitive impairments, the study’s reliance on self-reported events raises the possibility of underreporting. Therefore, the findings should be interpreted with caution, given the limited sample size and the potential impact of technical problems on data integrity. A further limitation is the possibility of bias being introduced by interviewer-participant familiarity. Such familiarity may have led to social desirability bias, where participants could have altered their responses to appear more favorable or agreeable. To address this, the interviewer was trained to maintain a neutral stance and encouraged participants to provide honest feedback, emphasizing the importance of candid responses for the developmental process of the Neo system. Participants were assured that all types of feedback, especially critical insights, were valuable and necessary to refine and improve the technology. It is also important to consider potential selection bias, as the participants who agreed to be interviewed may have held more favorable views toward the Neo system compared to those who declined, potentially skewing the findings. Despite these limitations, the study’s findings contribute valuable insights into the Neo Care Monitoring System’s performance and user experience and offer a roadmap for its future development and implementation.

4.2. Clinical Implications and Future Directions

The study’s findings hold implications for both developers of in-home falls monitoring systems and their potential users. Key to the system’s success will be its ability to accurately detect falls while minimizing false positives. This highlights the ongoing need for refining the system’s sound analysis algorithms to better distinguish between fall-related and everyday noises. Addressing design, responsiveness, and technical issues is also essential for increasing user satisfaction and trust. The Neo’s design could be optimized for better handling of real-world complexities through personalized sound calibration tailored to individual home environments, multi-device integration for enhanced mobility coverage, and incorporation of visual/multi-modal alerts to improve accessibility. Empowering users with voice trigger capabilities and ensuring continuous operation during connectivity disruptions are also vital considerations. Ultimately, an adaptive, user-centric design approach that leverages machine learning and user feedback loops is crucial for refining the system’s accuracy and robustness in complex residential settings. The incorporation of user-suggested features, such as personalized sound detection and visual cues for the hearing impaired, could significantly elevate the Neo’s overall value and utility. This user-centric approach to system improvement aligns with the healthcare technology trend of prioritizing end-user needs, thereby enhancing outcomes and experiences.

Future research should aim to improve the system’s sensitivity and specificity by refining its sound analysis algorithms and adhering to user-centered design principles. Collaborative initiatives involving healthcare providers and end-users can inform the development of a falls monitoring solution that effectively balances accuracy, user-friendliness, and customization. As healthcare technology advances, systems like the Neo offer promising avenues for enhancing safety and independence for individuals at risk of falls, not just in home settings, but also potentially in institutional environments. Furthermore, the aim of this study was to assess the accuracy and responsiveness of the Neo in real-world settings. An in-depth analysis of the evaluation process would help identify potential biases and refine the process, making this an important topic for future research.

5. Conclusions

In summary, this study provides a comprehensive evaluation of the Neo Care Monitoring System, shedding light on its performance and user experience in real-world settings. The findings underscore the system’s moderate accuracy and the challenges it faces, such as false positives and missed fall events. The study also highlights the value of user feedback for system refinement, including the need for personalized calibration and additional features. The mixed-methods approach and the collaboration between academic and industry stakeholders add rigor and practical relevance to the study. Despite the limitations related to sample size and data quality, the insights gained offer a valuable foundation for the ongoing development of the Neo system and similar technologies. This work contributes to the broader discourse on healthcare technology, emphasizing the need for real-world evaluations and user-centered designs to improve the safety and independence of individuals at risk of falls.