Next Article in Journal
Psychometric Properties of Jebsen Taylor Hand Function Test in an Italian Population with Parkinson’s Disease
Previous Article in Journal
Comparison of Short-Term Effects of Different Spinal Manipulations in Patients with Chronic Non-Specific Neck Pain: A Randomized Controlled Trial
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Systematic Review

Effectiveness of Online Programmes for Family Carers of People with Intellectual Disabilities: Systematic Review of the International Evidence Base

by
Rachel Leonard
1,
Nathan Hughes
2,
Trisha Forbes
1,*,
Michael Brown
1,
Lynne Marsh
1,
Maria Truesdale
3,
Stuart Todd
4 and
Mark Linden
1
1
School of Nursing and Midwifery, Queen’s University Belfast, Belfast BT7 1NN, UK
2
Department of Sociological Studies, University of Sheffield, Sheffield S10 2TN, UK
3
College of Medical Veterinary and Life Sciences, University of Glasgow, Glasgow G12 8QQ, UK
4
School of Care Sciences, University of South Wales, Usk Way, Newport NP20 2BP, UK
*
Author to whom correspondence should be addressed.
Healthcare 2024, 12(13), 1349; https://doi.org/10.3390/healthcare12131349 (registering DOI)
Submission received: 20 May 2024 / Revised: 17 June 2024 / Accepted: 3 July 2024 / Published: 6 July 2024

Abstract

:
This systematic literature review examines the evidence base on the effectiveness of online programmes on the mental health and well-being of family carers of people with intellectual disabilities. Databases (ERIC, Medline, PsycINFO and CINAHL) were searched for intervention studies that considered online interventions for family carers of people with intellectual disabilities. Data were extracted using standardised data extraction tools. Bias was assessed using the Cochrane risk of bias tool for randomised trials (RoB 2). Screening, extraction and assessment of bias were completed independently by two members of the research team. Given the low number of included studies and different outcomes assessed within them, it was not possible to conduct a meta-analysis; therefore, data are presented narratively. Two studies met the criteria to be included in the review. Both studies utilised a feasibility randomised controlled trial methodology. One study found a significant decrease in parental stress, while the other found a significant increase in psychological well-being. Caution must be taken in drawing firm conclusions, given the small sample sizes and low retention rates in both studies. Online programmes seem to offer potential benefits to family carers of people with intellectual disabilities. However, further investigation is needed to examine these programmes, adopting a collaborative approach with family carers.

1. Introduction

Family carers of people with intellectual disabilities (IDs) provide ongoing care for their family members, which may include medication management, monitoring a physical condition and behaviour management [1]. Family carers play a crucial role in supporting their health and well-being; approximately 77% of people with IDs in England were living with their families in 2017/2018 [2]. The benefits of directly supporting carers were also highlighted by the UK’s Cabinet Office as part of their ‘Think Family’ policy initiative [3,4]. The report recognised the importance of services and supports that address the specific and independent needs of family members providing care to maintain or enhance the support they are then able to provide. This includes supports that only indirectly relate to their caring role, for example, to maintain their paid employment or address their mental well-being. Such services and supports, therefore, perceive the carer as a service user in their own right [5].
Family carers report positive experiences of having a family member with an ID [6]. A systematic review of 22 studies detailed positive increased personal growth, strength and development of parents while acknowledging that distress and stress may still be present for some [6]. There is recognition of the need to promote and support the mental and physical health of family carers of children with IDs. A number of factors have been shown to positively impact the quality of life of families, including informal support from family and friends, access to professional supports, social relationships and, for some, spiritual and religious support [7]. Other factors found to effect family carers’ health and ability to provide care and support include characteristics of the child and family and access to support and advice [8]. Family carers of people with IDs are twice as likely as other carers to experience stress and mental health challenges; however, there is a lack of research exploring suitable interventions [9].
While recognising the positive effects of having a family member with an ID, some family carers experience significant and ongoing stress and mental health difficulties [10,11]. A UK-based study interviewed 18 mothers who cared for children with IDs with challenging behaviours, finding they experienced social isolation, conflict, limited lifestyles and self-blame [12]. Mothers highlighted that they were surviving day to day and described conflict with professionals over a lack of service provision [12]. The mothers were also shown to possess higher levels of anxiety and depression in comparison to a group of mothers with typically developing children [13]. The demands of caregiving and the lack of appropriate support also make participation in employment difficult, particularly for mothers [14]. There is less available research on the impact of caregiving on fathers; however, a qualitative study described how fathers’ identities changed following diagnosis from that of ‘father’ to ‘father with a child with ID’; this unanticipated change would affect their relationship across the child’s lifespan [15]. A systematic review and meta-analysis suggested that fathers have lower mental health difficulties, possibly due to their more limited caring role [16]. A second review identified the emotional consequences of a diagnosis of ID, finding that fathers experienced feelings of grief, loss, denial and blame [17].
Programmes and interventions which provide training to family carers have been shown to have a positive impact on levels of stress and feelings of confidence [18], child socialisation [19] and quality of life [20]. However, parents face barriers in taking part in such interventions, including time, family pressures, cost and availability of services [21]. Online programmes offer greater flexibility of delivery and have been shown to offer comparable outcomes to more traditional treatment approaches in areas such as parenting styles [22], knowledge and self-efficacy [23]. By online programme, we refer to any intervention or training delivered via the internet for the purposes of improving the lives of carers of people with ID.
The COVID-19 pandemic made access to face-to-face support even more difficult for families, with attempts being made by some services to offer online alternatives [24,25]. While this has created challenges for family carers (e.g., access to reliable technology), it has also provided opportunities (e.g., reduced travel and delivery costs, improved access to programmes). In one study, mothers reported the need to stay healthy and how they made the changing pandemic situation work for their child with ID [26]. Families reported increased levels of stress and burden of care because of the lockdowns, in part due to the closure and changes to service provision, resulting in significant additional family pressures [27]. Family carers of adults with ID also reported receiving limited support from services and experienced feelings of powerlessness. However, support from family and friends was helpful and resulted in resilience [28]. A recent scoping review evidenced the importance of learning from the COVID-19 pandemic from the perspectives of people with ID and their family carers [29]. The review identified three core themes related to ensuring that preventative information is quickly available and accessible, that reasonable adjustments are made in response to need (e.g., access to technology to facilitate the provision of care and support) and that treatment and management needs are identified and enabled (e.g., access to acute hospitals and lifesaving interventions) [22].
The purpose of the current work is to review the evidence base on the effectiveness of online programmes on the mental health and well-being of family carers of people with ID.

2. Materials and Methods

The review follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [30].

2.1. Eligibility Criteria

To be included, studies needed to be randomised controlled trials (RCTs) of online interventions for family carers of people with IDs. Only studies that reported on carers of people with IDs were included. Where a study reported on a sample that included carers of people with IDs and carers of people with other conditions (autism, ADHD etc.), we followed the predominance rule. Namely, to be included, the sample must have predominately been made up of carers of people with IDs (i.e., over 50%). Studies had to report on carer-focused outcomes to be included. Publications reporting on peer-reviewed empirical studies were included. Only publications available in the English language were included due to resource restrictions.
The review excluded studies that presented interventions that were not delivered online, reported on carers of people with conditions that were not IDs or did not report on specific carer outcomes (e.g., where the outcome was in relation to the person with the ID, not the carers themselves). Feasibility and pilot studies that did not employ an RCT methodology were not included. Reviews, protocols and discussion articles were also excluded. Empirical studies published in languages other than English were excluded.

2.2. Information Sources

Searches were conducted using the electronic databases ERIC, Medline, PsycINFO and CINAHL.

2.3. Search Strategy

Database choices and search terms were refined through an iterative process including consultation with a subject specialist librarian. Searches used a combination of MeSH (medical subject heading) terms and free text terms, which were organised into categories representing randomised controlled trials, online delivery, programmes, family carers and IDs. These categories related to the research question, which sought to examine online interventions for family carers of people with IDs. Each category was broken down into search terms which represented keywords and phrases used in our searches. Searches were conducted on 18 January 2024. The search terms are presented in Table 1, and an example of the search string in Medline (Ovid) is available in the Supplementary Data File S1.

2.4. Study Selection

The online platform Covidence [31] was utilised for organising and selecting literature for the review based on the criteria outlined above. Titles and abstracts of all papers were independently screened against the eligibility criteria by two reviewers. Following this, full texts were retrieved for all potentially relevant studies, which were then screened independently by two reviewers at the full-text level. All conflicts between reviewers were resolved by discussion, with a third reviewer acting as the arbiter. The process is outlined in the PRISMA diagram in Figure 1 [30].

2.5. Data Extraction

Data were extracted in relation to country, study aim(s), study design, sample, intervention description, comparison, outcomes measured, findings for the primary outcome and conclusions. Data extraction was performed independently by two members of the review team using standardised data extraction tools.

2.6. Risk of Bias Assessment

Two reviewers independently reviewed the risk of bias of the included studies, discussing any discrepancies. The revised Cochrane risk of bias tool for RCTs (RoB 2) [32] was employed to assess bias in the included studies. The RoB 2 contains five domains through which bias is assessed: (1) bias arising from the randomization process; (2) bias due to deviations from intended interventions; (3) bias due to missing outcome data; (4) bias in the measurement of the outcome and (5) bias in selection of the reported result. Each domain contains signalling questions which elicit information relevant to an assessment of risk of bias. The responses include yes; probably yes; probably no; no and no information. These answers feed into algorithms which determine the bias within each domain and overall. The possible risk of bias judgements are: (1) low risk of bias; (2) some concerns and (3) high risk of bias.

2.7. Outcome of Interest

This review primarily focused on measures of carers’ mental health and well-being. Well-being and mental health were measured according to the mean overall change (from baseline to endpoint) assessed by any psychometrically validated tool. Examples of tools include the Warwick–Edinburgh Mental Well-being Scale (WEMWBS) [33] and the Parenting Stress Index—Short Form (PSI) [34].

2.8. Data Analysis and Synthesis

Due to the low number of studies and differences in measurements used, it was not possible to conduct a meta-analysis; therefore, the results are reported narratively [35]. This allowed key concepts to be identified and grouped together for ease of comparison. Concepts were grouped under subject headings or themes, which were then discussed by members of the research team for verification.

3. Results

As seen in Figure 1, our searches yielded a total of 718 results. After removal of duplicates (n = 111), screening titles and abstracts, we were left with 17 results. Of these 17 records, 15 were excluded. The reasons included the following: the intervention was not conducted online (n = 5); the study was not an RCT design (n = 4) and study did not predominantly consider IDs (n = 6). As such, two studies were included in this review.

3.1. Study Characteristics

Details of the two studies that met our inclusion criteria are displayed in Table 2. One study conducted a randomised feasibility trial [9], with the other conducting a randomised effectiveness trial [36]. Flynn et al.’s [9] study was conducted in the UK, while Grenier-Martin et al. [36] conducted their trial in Canada. Both studies utilised a comparator, one with a waitlist control (intervention received at week 4) [36] and the other compared against the same intervention but with an additional element (x3 30 min peer support telephone calls). Samples ranged in size from 42 [36] to 60 [9]. Follow-up rates of completed outcome measures were 83% (50/60) [9] and 69% (29/42) [36]. Both studies included family carers of children (<18 years) with IDs. In Flynn et al.’s [9] sample, most children had a diagnosis of ID (n = 54), with some having a dual diagnosis of ID and autism (n = 42). Carers themselves were predominantly female (n = 55 out of 60) and white British (n = 48 out of 60) [9]. Grenier-Martin et al. [36] did not report on the age or sex of their participants. Many of the children of the carers had a chromosomal abnormality or genetic syndrome (72.4%); the remaining children had an ID (24.1%) or cerebral palsy (3.4%) [36].

3.2. Intervention Description

Flynn et al. [9] tested the feasibility of the ‘Be Mindful+’ online intervention. Be Mindful is a publicly available online mindfulness programme which includes ten online sessions based on the elements of Mindfulness-Based Cognitive Therapy. The sessions are presented by two qualified mindfulness trainers. Participants are provided with twelve assignments to practice in daily life and six downloadable course handouts, and auto-generated supporting motivational emails were also included throughout [9]. The study aimed to test an additional element to the Be Mindful programme; a peer support element consisting of three 30 min telephone calls from a Peer Mentor (a carer who had been through the Be Mindful programme). Participants were given four weeks to complete the online intervention.
Grenier-Martin et al. [36] tested the effectiveness of an online training programme to support carers in managing problem behaviours at home. The online training was asynchronous (i.e., parents were able to learn at their own pace) and consisted of five separate modules. At the end of each module, quizzes evaluated parental comprehension. Modules included: how to document and assess the functions of problem behaviours, what an antecedent was, the different intervention strategies associated with antecedents, the influence of the consequences on the behaviour, how caregivers should react to problem behaviours, methods for teaching appropriate behaviour as an alternative to problem behaviour and practical considerations for the intervention. Participants were given two weeks to complete the online intervention.

3.3. Primary and Secondary Outcomes

While both studies included carer outcomes, only Flynn et al. [9] examined these as the primary outcome. The primary outcome of Grenier-Martin et al. [36] was to evaluate the effects of this online training on the intensity and frequency of problem behaviours observed in children by their caregivers, as measured by the Adaptive Behaviour Scale Assessment System—2nd Edition (problem behaviours in children) [37] and the Behaviour Problems Inventory-01 [38]. Secondary outcomes included the Parenting Stress Index—Short Form [34] and the Self-Assessment Questionnaire of the Parent’s Educational Competence [39]. Flynn et al. [9] primarily measured psychological well-being, as measured by the Warwick–Edinburgh Mental Well-being Scale (WEMWBS) [33]. Secondary outcomes included quality of life, psychological distress and family functioning.
Grenier-Martin et al. [36] completed measures over the phone or at home with two psychology PhD students, depending on location and the caregiver’s preference. Outcomes were assessed at four time points: Week 1, 6, 10 and 14 for the intervention group and week 1, 4, 10 and 14 for the control group [36]. Flynn et al. [9] assessed outcomes at three time points: baseline, 12 weeks and 6 months post-randomisation.
Grenier-Martin et al. [36] only provided data for the Parenting Stress Index—Short Form (PSI) [34] at T2 for the intervention and control groups. In their statistical analysis, they controlled for T1 results; however, they did not provide comparable data for T1, T3 or T4. It is also important to note that the time points for T2 were different for the intervention and control groups. Flynn et al. [9] compared scores on the WEMWBS (psychological well-being) 6 months post-randomization, adjusting for baseline scores. Data were not presented for outcomes at week 12.

3.4. Risk of Bias

3.4.1. Bias Arising from the Randomisation Process

Both studies utilised random allocation, with concealment of allocation until after participants were assigned to the intervention or control conditions. Grenier-Martin et al. [36] used block randomisation, with a spreadsheet automatically assigning participants to the experimental or waitlist groups. Flynn et al. [9] stated that participants were randomly allocated, but did not provide details on randomisation procedures. No significant differences were noted in baseline demographic information for either study. Both studies were rated as having a low risk of bias within this domain.

3.4.2. Bias Due to Deviations from Intended Interventions

With both studies, it was not possible to blind participants to allocation. However, the researchers were blinded throughout data collection in both studies [9,36]. Adherence to intervention was poor within Flynn et al. [9], with 13 out of 30 fully completing ‘Be Mindful’ and 14 out of 27 fully completing Be ‘Mindful+’ (52%). In addition, only 3 of the intervention group completed ‘Be Mindful+’ within the timeframe, with 23 out of 30 receiving peer support telephone calls. Grenier-Martin et al. [36] did not specifically discuss intervention adherence; however, they had a 69% (29/42) follow-up rate. Specifically, this was 16 out of 22 for the control group and 13 out of 20 for the intervention group. Neither Flynn et al. [9] nor Grenier-Martin et al. [36] accounted for this loss to follow up or lack of adherence to intervention within the analysis of data. It is important to note that Flynn et al. [9] were primarily concerned with feasibility outcomes as opposed to the establishing effectiveness of the intervention on carer well-being. Within this bias domain, Flynn et al. [9] and Grenier-Martin et al. [36] were rated as having ‘some concerns’.

3.4.3. Bias Due to Missing Outcome Data

Loss to follow-up was high in both studies, at 83% (50/60) [9] and 69% (29/42) [36]. Missing outcome data was largely due to participants withdrawing from the study. Reasons for withdrawal were provided by Flynn et al. [9]; however, no details were provided by Grenier-Martin et al. [36]. Neither study used analysis methods that mitigated the risk of bias from missing data (i.e., Intention to Treat (ITT) or analysis based on ITT principals). For example, participants were not excluded based on a minimum amount of the completed intervention. Flynn et al. [9] analysed 50 out of 60 participants who had provided outcome data. However, only 27 out of 60 fully completed the intervention (both control and intervention groups). Grenier-Martin et al. [36] did not provide details on intervention adherence, so it is not possible to determine whether they excluded participants from their analysis based on the minimum number of participants who completed the intervention. For both studies, the proportions of missing outcome data were similar across groups. Both studies were rated as having a low risk of bias within this domain.

3.4.4. Bias in Measurement of the Outcome

Both studies utilised self-reported questionnaires (PSI and WEMWBS). Outcome assessors remained blinded throughout data collection. In addition, the comparator in both studies was another active intervention (waitlist control and Be Mindful), which was facilitated by an independent researcher outside of their standard care. Both studies were rated as having a low risk of bias within this domain.

3.4.5. Bias in Selection of the Reported Results

Both studies either failed to report or partially reported outcomes that were measured. Flynn et al. [9] did not report on week 12 data, reporting only a subset of time points (baseline and 6 months post-randomisation). On this basis, Flynn et al. [9] was rated as having ‘some concerns’ for this domain. Grenier-Martin et al. [36] failed to report T1, T3 or T4 individual group data. The only data reported were a comparison of T2 scores; however, these were collected at different time points for the intervention and control groups (week 4 and week 6). For the other time points, Grenier-Martin et al. [36] reported on combined scores for control and intervention groups. It is unclear why the authors opted for this analysis, as it limits the conclusions that can be drawn regarding the effect of the invention. On this basis, Grenier-Martin et al. [36] was rated as ‘high risk of bias’ for this domain.

3.4.6. Overall Judgement of Bias

Due to having concerns around deviations from intended interventions and selection of reported results, Flynn et al. [9] was assessed as having ‘some concerns’ in relation to the overall risk of bias judgment. Grenier-Martin et al. [36] had concerns around deviations from intended interventions and a high risk of bias within the selection of the reported results domain. Overall, Grenier-Martin et al. [36] was assessed as having a ‘high risk’ of bias. Each domain and judgment is presented in Figure 2 and Figure 3.

3.5. Effectiveness and Impact of Online Interventions for Carers

The two studies offer some indicative evidence of the potential impact of online interventions for carers of people with ID, though caution must be taken in drawing firm conclusions. Grenier-Martin et al. [36] demonstrate that their online training for managing problem behaviours produced a significant decrease in parental stress from pre-training to follow-up (F (2,56) = 5.26, p < 0.01). This significant decrease was immediately apparent post-intervention (p = 0.024) and was sustained over time, with follow-up measures taken up to 2 months. However, the sample size was small, and the attrition rate was 30%.
Flynn et al. [9] indicated significant increases in psychological well-being (β = 1.15, p < 0.001) following the Be Mindful intervention. However, all participants in the study received the online intervention, and so the study design did not test the intervention against no online support. Instead, the study tested the value of an additional telephone support service, providing ‘space for reflection’ for carers undertaking the online programme. The study ‘was not powered to detect differences in outcomes between the two arms of the trial’, and found ‘small, but not significant, improvements’ for those who received the additional telephone support.
The two studies also offer some indication that online interventions may be suitable and attractive to carers of people with IDs. Flynn et al. [9] highlighted the difficulties for carers of people with IDs to prioritise their own mental well-being given competing commitments associated with their care. Online provision is seen to offer ‘a potential solution’, given that it can be completed alongside other commitments in a flexible way. Similarly, Grenier-Martin et al. [36] reported that participants valued the approach of online delivery, and that the content of such a programme was consistent with their needs. The authors also reflected on the potential for online support to provide immediate and timely support, accessible to all, at a time when families may wait months or even years for specialized intervention services.

3.6. Challenges in Delivering Interventions for Carers

The two studies suggest some challenges in the use of online support for carers of people with IDs. Most notably, both studies struggled with retention. Grenier-Martin et al. [36] suffered from a 30% attrition rate, and such rates are common among online programmes for parents.
Both studies suggest potential limitations in reliance on online content and value in additional human contact, whether with professionals or peer mentors. Flynn et al. [9] specifically examined the value of additional telephone support from a peer mentor, providing space for reflection about their completion of the course. They argue that this enables an intervention to become tailored to the individual carer or family in a timely and cost-effective manner. While Grenier-Martin et al. [36] did not include such a mechanism, they also argued for further research into the relevance of enabling carers to engage remotely with professionals during the online training.

4. Discussion

This review sought to synthesise robust experimental research regarding the effectiveness of online interventions on the well-being and mental health of family carers of people with ID. However, only two studies met the eligibility criteria, and both presented limitations in relation to dimensions of study quality and potential bias, including high attrition rates. This indicates a dearth of high-quality evidence in this field.
Both studies highlight some potential benefits of using online approaches, including the flexibility with which online material can be engaged with, so as to meet other commitments, and the efficacy of online delivery of the specific information needs of family carers. These are in keeping with the benefits noted in studies of online interventions for carers of people with other forms of support needs [40,41,42]. For example, other studies have also noted the benefits of being able to access online content when and where it most suits a carer’s other commitments [40,41], while the benefits to carers of video recordings that can be watched repeatedly and of video conferencing methods have also been [41,42]. Other benefits reported elsewhere include addressing inaccessibility due to geographical barriers, including for those in remote, rural or isolated settings [41,43,44,45], and reducing the participation costs of childcare or transportation for carers [41,46,47] or barriers to participation due to lack of access to transport [45].
Both studies also highlight some of the potential limitations to, or challenges with, online provision for carers of people with IDs. Recruitment and retention rates may be indicative of challenges in engaging carers sufficiently to complete online programmes. Both studies also suggest that there is value in additional personal contact alongside online content delivery, and it has been argued elsewhere that online provision might be most useful as a complimentary approach to traditional approaches to supporting carers, rather than simply a replacement [42,44]. While not prominent in the two studies reviewed here, other studies have suggested potential challenges in carers’ access to and difficulties with technology. Online programmes are clearly limited to those families and carers who own the necessary technology and sufficient internet connection [45,48,49], and the latter may be a particular issue for carers in rural areas [47]. Technical issues are also reported elsewhere, including web browser issues and problems with logging in [46,47], which have been seen to disrupt the fluidity of collecting and observing data [48]. However, as we become increasingly used to mobile devices and have better access to reliable digital networks, many of these difficulties will be overcome.
There is growing recognition of the utility of online approaches for addressing the needs of family carers, which may partly be due to the increasing digital literacy of carers and the ubiquity of technology. The COVID-19 pandemic has significantly increased familiarity with online platforms [25] and demonstrated that many services can be delivered online. Of course, some services will always need to be delivered face to face, and online platforms should be used where they can be most effective in addressing the needs of carers. Overall, our findings suggest merit in further research to establish an evidence base for the use of online support for carers of people with ID. Funding bodies should prioritise the development and testing of high-quality interventions to support family carers in their crucial role in providing care to their family members.

4.1. Implications for Policy

Arising from the findings of this review are implications for policy that need to be considered for the future, notably in relation to long COVID and the longer-term effects of lockdown. As the full implications and long-term effects of the COVID-19 pandemic are realised, policy makers need to ensure that the needs of families caring for people with ID are fully recognised and included. This is important due to the impact on the health and well-being and quality of life of families when caring for a family member with an ID [50,51]. There is scope for policy makers to ensure that the specific needs of families of people with ID are included in post-pandemic responses, thereby seeking to ensure that there is ongoing access to social support networks [52]. As the full extent of the effects of COVID-19 and lockdown restrictions during the pandemic on physical and mental health of family carers becomes apparent, primary care and mental health policies need to reflect the needs and concerns of families and their family members with IDs [53,54]. Online programmes for family carers may be one approach for inclusion in future health policies. Failure to recognise and respond to the needs of this population will negatively impact their long-term health and their capacity and capability to continue to care and provide support for their family members [55].

4.2. Potential Biases in the Review Process

We reduced bias in this review by following predetermined data extraction and risk of bias questions, and ensured our search terms were appropriate through consultation with a subject specialist librarian. We did not seek to include studies published in languages other than English. This was due to limitations in the availability of translation services and means that we may have missed some aspects of the international evidence-base.

4.3. Suggestions for Future Research

This review found only two RCTs amongst the available literature. This suggests a need for large-scale, evidence-based interventions that have been thoroughly tested prior to implementation. No studies in this review were conducted in low- or middle-income countries (LMIC), and there was limited participation from ethnically diverse carers. Online delivery of interventions to support family carers seems ideally placed for use in LMIC due to their relatively low cost and long reach. However, the reliability, or existence, of internet connections in certain regions may mean that efforts to introduce such programmes would be stymied. With improvements in technologies such as mobile internet or satellite-based services, such barriers could be overcome. Future research should seek to understand the barriers and facilitators to delivery in LMIC and create interventions which can address these whilst supporting family carers.
A related area for future investigation would be the exploration of rural versus urban differences in the delivery of interventions. Family carers living in rural locations in many developed regions of the world experience reduced services and a lack of access to care. Online programmes have been successfully used in remote locations across the world; however, few of these have been rigorously evaluated. Future research should seek to conduct RCTs of these programmes to determine whether they are effective.
In the post-COVID era, many services have returned to in-person delivery. Many family carers prefer this approach; however, for some, the demands of their caring role mean that regular attendance can be difficult to manage. In such cases, online delivery of services could be seen as a means to enhance, or compliment, existing services rather than a replacement. This blended approach to service delivery could be examined to identify successful components to better refine delivery and provide the optimal approach which best suits the needs of family carers.

5. Conclusions

More research focusing on online support programmes for family carers of those with IDs is needed. This work should be co-designed with family carers so that it addresses issues which directly impact their lives. Much of the work included in this review was conducted in high-income countries; however, the ubiquity of online platforms and mobile technology would mean that people living in low-income countries could also benefit from inclusion in such work. It may not be appropriate to deliver all types of support and services online; however, such programmes offer a low-cost opportunity to deliver care and support which is convenient and open to all who wish to take advantage of it.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/healthcare12131349/s1, Supplementary Data File S1. Medline (Ovid) search strategy.

Author Contributions

Conceptualization N.H. and M.L.; Methodology, R.L., M.L. and N.H.; Software, R.L. and N.H.; Formal Analysis, R.L. and N.H.; Data Curation, R.L. and M.L.; Writing—Original Draft Preparation, N.H., M.L., T.F. and R.L.; Writing—Review & Editing, T.F., M.B., S.T., L.M., M.T., M.L., N.H. and R.L.; Visualization, M.L., N.H., T.F. and R.L.; Project Administration, M.L., N.H. and R.L.; Funding Acquisition, M.L. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the UKRI Economic and Social Research Council ES/W001829/1. For the purpose of open access, the authors have applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising from this submission.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The original data presented in the study are openly available in Queen’s University Belfast PURE repository at 10.17034/ec9b2b37-8248-40b0-a08a-0f06c4a9bb14.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Yannamani, N.; Zia, A.; Khalil, N. Family carers of people with learning disabilities: Common themes across caring. Psychiatry 2009, 8, 441–444. [Google Scholar] [CrossRef]
  2. Nuffield Trust. Adults with Learning Disabilities Who Live in Their Own Home or with Their Family. 2021. Available online: https://www.nuffieldtrust.org.uk/resource/adults-with-learning-disabilities-who-live-in-their-own-home-or-with-their-family (accessed on 23 February 2022).
  3. Morris, K.; Hughes, N.; Clarke, H.; Tew, J.; Mason, P.; Galvani, S.; Lewis, A.; Loveless, L. Think Family: A Literature Review of Whole Family Approaches. Cabinet Office. 2008. Available online: https://core.ac.uk/download/pdf/4156933.pdf (accessed on 12 February 2024).
  4. Social Exclusion Task Force. Think Family: Improving the Life Chances of Families at Risk. 2009. Available online: https://www.drugsandalcohol.ie/17766/ (accessed on 21 March 2022).
  5. Hughes, N. Models and Approaches in Family-Focused Policy and Practice. Soc. Policy Soc. 2010, 9, 545–555. [Google Scholar] [CrossRef]
  6. Beighton, C.; Wills, J. How parents describe the positive aspects of parenting their child who has intellectual disabilities: A systematic review and narrative synthesis. J. Appl. Res. Intellect. Disabil. 2019, 32, 1255–1279. [Google Scholar] [CrossRef]
  7. Boehm, T.L.; Carter, E.W. Family Quality of Life and Its Correlates among Parents of Children and Adults with Intellectual Disability. Am. J. Intellect. Dev. Disabil. 2019, 124, 99–115. [Google Scholar] [CrossRef] [PubMed]
  8. Marquis, S.; Hayes, M.V.; McGrail, K. Factors affecting the health of caregivers of children who have an intellectual/developmental disability. J. Policy Pract. Intellect. Disabil. 2019, 16, 201–216. [Google Scholar] [CrossRef]
  9. Flynn, S.; Hastings, R.P.; Burke, C.; Howes, S.; Lunsky, Y.; Weiss, J.A.; Bailey, T. Online Mindfulness Stress Intervention for Family Carers of Children and Adults with Intellectual Disabilities: Feasibility Randomized Controlled Trial. Mindfulness 2020, 11, 2161–2175. [Google Scholar] [CrossRef]
  10. Cook, T.; Noone, S.; Thomson, M. Mindfulness-based practices with family carers of adults with learning disability and behaviour that challenges in the UK: Participatory health research. Health Expect. 2019, 22, 802–812. [Google Scholar] [CrossRef]
  11. Staunton, E.; Kehoe, C.; Sharkey, L. Families under pressure: Stress and quality of life in parents of children with an intellectual disability. Ir. J. Psychol. Med. 2023, 40, 192–199. [Google Scholar] [CrossRef]
  12. Johnson, R.F.; O’Reilly, M.; Vostanis, P. Caring for children with learning disabilities who present problem behaviours: A maternal perspective. J. Child Health Care 2006, 10, 188–198. [Google Scholar] [CrossRef]
  13. Rydzewska, E.; Dunn, K.; Cooper, S.; Kinnear, D. Mental ill-health in mothers of people with intellectual disabilities compared with mothers of typically developing people: A systematic review and meta-analysis. J. Intellect. Disabil. Res. 2021, 65, 501–534. [Google Scholar] [CrossRef]
  14. Ejiri, K.; Matsuzawa, A. Factors associated with employment of mothers caring for children with intellectual disabilities. Int. J. Dev. Disabil. 2017, 65, 239–247. [Google Scholar] [CrossRef]
  15. Marsh, L.; Brown, M.; McCann, E. The views and experiences of fathers regarding their young child’s intellectual and developmental disability diagnosis: Findings from a qualitative study. J. Clin. Nurs. 2020, 29, 3373–3381. [Google Scholar] [CrossRef]
  16. Dunn, K.; Kinnear, D.; Jahoda, A.; McConnachie, A. Mental health and well-being of fathers of children with intellectual disabilities: Systematic review and meta-analysis. BJPsych Open 2019, 5, e96. [Google Scholar] [CrossRef] [PubMed]
  17. Marsh, L.; Brown, M.; McCann, E. The Views and Experiences of Fathers of Children with Intellectual Disabilities: A Systematic Review of the International Evidence. J. Policy Pract. Intellect. Disabil. 2020, 17, 79–90. [Google Scholar] [CrossRef]
  18. Hudson, A.M.; Matthews, J.M.; Gavidia-Payne, S.T.; Cameron, C.A.; Mildon, R.L.; Radler, G.A.; Nankervis, K.L. Evaluation of an intervention system for parents of children with intellectual disability and challenging behaviour. J. Intellect. Disabil. Res. 2003, 47, 238–249. [Google Scholar] [CrossRef]
  19. MacDonald, E.E.; Hastings, R.P. Mindful parenting and care involvement of fathers of children with intellectual disabilities. J. Child Fam. Stud. 2010, 19, 236–240. [Google Scholar] [CrossRef]
  20. Masulani-Mwale, C.; Kauye, F.; Gladstone, M.; Mathanga, D. Development of a psycho-social intervention for reducing psychological distress among parents of children with intellectual disabilities in Malawi. PLoS ONE 2019, 14, e0210855. [Google Scholar] [CrossRef] [PubMed]
  21. Davis, C.; Hendon, F.; McDonald, K.; Blanco, S. Use of technology in facilitating remote caregiver training for token systems. Int. J. Dev. Disabil. 2020, 66, 330–338. [Google Scholar] [CrossRef]
  22. Sanders, M.R.; Dittman, C.K.; Farruggia, S.P.; Keown, L.J. A comparison of online versus workbook delivery of a self-help positive parenting program. J. Prim. Prev. 2014, 35, 125–133. [Google Scholar] [CrossRef]
  23. Corralejo, S.M.; Rodríguez, M.M.D. Technology in parenting programs: A systematic review of existing interventions. J. Child Fam. Stud. 2018, 27, 2717–2731. [Google Scholar] [CrossRef]
  24. Monaghesh, E.; Hajizadeh, A. The role of telehealth during COVID-19 outbreak: A systematic review based on current evidence. BMC Public Health 2020, 20, 1193. [Google Scholar] [CrossRef] [PubMed]
  25. Webster, P. Virtual health care in the era of COVID-19. Lancet 2020, 395, 1180–1181. [Google Scholar] [CrossRef] [PubMed]
  26. Embregts, P.; Heerkens, L.; Frielink, N.; Giesbers, S.; Vromans, L.; Jahoda, A. Experiences of mothers caring for a child with an intellectual disability during the COVID-19 pandemic in the Netherlands. J. Intellect. Disabil. Res. 2021, 65, 760–771. [Google Scholar] [CrossRef] [PubMed]
  27. Navas, P.; Verdugo, M.; Martínez, S.; Amor, A.M.; Crespo, M.; Deliu, M.M. Impact of COVID-19 on the burden of care of families of people with intellectual and developmental disabilities. J. Appl. Res. Intellect. Disabil. 2022, 35, 577–586. [Google Scholar] [CrossRef] [PubMed]
  28. Patel, V.; Perez-Olivas, G.; Kroese, B.S.; Rogers, G.; Rose, J.; Murphy, G.; Cooper, V.; E Langdon, P.; Hiles, S.; Clifford, C.; et al. The Experiences of Carers of Adults With Intellectual Disabilities during the First COVID-19 Lockdown Period. J. Policy Pract. Intellect. Disabil. 2021, 18, 254–262. [Google Scholar] [CrossRef] [PubMed]
  29. Taggart, L.; Mulhall, P.; Kelly, R.; Trip, H.; Sullivan, B.; Wallén, E.F. Preventing, mitigating, and managing future pandemics for people with an intellectual and developmental disability—Learnings from COVID-19: A scoping review. J. Policy Pract. Intellect. Disabil. 2022, 19, 4–34. [Google Scholar] [CrossRef]
  30. Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews and meta-analyses. BMJ 2021, 372, 71. [Google Scholar] [CrossRef] [PubMed]
  31. Covidence Systematic Review Software, Veritas Health Innovation, Melbourne, Australia. Available online: www.covidence.org (accessed on 26 January 2024).
  32. Sterne, J.A.C.; Savović, J.; Page, M.J.; Elbers, R.G.; Blencowe, N.S.; Boutron, I.; Cates, C.J.; Cheng, H.Y.; Corbett, M.S.; Eldridge, S.M.; et al. RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJ 2019, 366, l4898. [Google Scholar] [CrossRef] [PubMed]
  33. Tennant, R.; Hiller, L.; Fishwick, R.; Platt, S.; Joseph, S.; Weich, S.; Parkinson, J.; Secker, J.; Stewart-Brown, S. The Warwick-Edinburgh Mental Well-Being Scale (WEMWBS): Development and UK Validation. Health Qual. Life Outcomes 2007, 5, 63. [Google Scholar] [CrossRef]
  34. Abidin, R.R. Parenting Stress Index (PSI) Manual, 3rd ed.; Psychological Assessment Resources: Odessa, FL, USA, 1995. [Google Scholar]
  35. Popay, J.; Roberts, H.; Aj, S.; Petticrew, M.; Britten, N.; Arai, L.; Roen, K.; Rodgers, M. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews Final Report. J. Epidemiol. Community Health 2005, 59 (Suppl. S1), A7. [Google Scholar]
  36. Grenier-Martin, J.; Rivard, M.; Patel, S.; Lanovaz, M.J.; Lefebvre, C. Randomized Controlled Trial on an Online Training to Support Caregivers of Young Children with Intellectual and Developmental Disability managing Problem Behaviors at Home. J. Child Fam. Stud. 2022, 31, 3485–3497. [Google Scholar] [CrossRef]
  37. Harrison, P.L.; Oakland, T. Adaptive Behavior Assessment System, 2nd ed.; Wiley Online Library: Hoboken, NJ, USA, 2003. [Google Scholar]
  38. Rojahn, J.; Matson, J.L.; Lott, D.; Esbensen, A.J.; Smalls, Y. The behavior problems inventory: An instrument for the assessment of self-injury, stereotyped behavior, and aggression/destruction in individuals with developmental disabilities. J. Autism Dev. Disord. 2001, 31, 577–588. [Google Scholar] [CrossRef] [PubMed]
  39. Johnston, C.; Mash, E.J. A Measure of Parenting Satisfaction and Efficacy. J. Clin. Child Psychol. 1989, 18, 167–175. [Google Scholar] [CrossRef]
  40. Hinton, S.; Sheffield, J.; Sanders, M.R.; Sofronoff, K. A randomized controlled trial of a telehealth parenting intervention: A mixed-disability trial. Res. Dev. Disabil. 2017, 65, 74–85. [Google Scholar] [CrossRef] [PubMed]
  41. Lunsky, Y.; Albaum, C.; Baskin, A.; Hastings, R.P.; Hutton, S.; Steel, L.; Wang, W.; Weiss, J. Group Virtual Mindfulness-Based Intervention for Parents of Autistic Adolescents and Adults. J. Autism Dev. Disord. 2021, 51, 3959–3969. [Google Scholar] [CrossRef] [PubMed]
  42. De Luca, R.; Rifici, C.; Pollicino, P.; Di Cara, M.; Miceli, S.; Sergi, G.; Sorrenti, L.; Romano, M.; Naro, A.; Billeri, L.; et al. ‘Online therapy’ to reduce caregiver’s distress and to stimulate post-severe acquired brain injury motor and cognitive recovery: A Sicilian hospital experience in the COVID era. J. Telemed. Telecare 2023, 29, 451–460. [Google Scholar] [CrossRef]
  43. Lindgren, S.; Wacker, D.; Suess, A.; Schieltz, K.; Pelzel, K.; Kopelman, T.; Lee, J.; Romani, P.; Waldron, D. Telehealth and Autism: Treating Challenging Behavior at Lower Cost. Pediatrics 2016, 137 (Suppl. S2), S167–S175. [Google Scholar] [CrossRef] [PubMed]
  44. Siracusano, M.; Calsolaro, J.; Riccioni, A.; Gialloreti, L.E.; Benvenuto, A.; Giovagnoli, G.; Curatolo, P.; Mazzone, L. TrASDition Training: An online parental training for transition-age youth with autism spectrum disorder. Psychiatry Res. 2021, 300, 113930. [Google Scholar] [CrossRef] [PubMed]
  45. Antonini, T.N.; Raj, S.P.; Oberjohn, K.S.; Cassedy, A.; Makoroff, K.L.; Fouladi, M.; Wade, S.L. A pilot randomized trial of an online parenting skills program for pediatric traumatic brain injury: Improvements in parenting and child behavior. Behav. Ther. 2014, 45, 455–468. [Google Scholar] [CrossRef]
  46. Douglas, S.N.; Kammes, R.; Nordquist, E. Online Communication Training for Parents of Children With Autism Spectrum Disorder. Commun. Disord. Q. 2018, 39, 415–425. [Google Scholar] [CrossRef]
  47. Heitzman-Powell, L.S.; Buzhardt, J.; Rusinko, L.C.; Miller, T.M. Formative Evaluation of an ABA Outreach Training Program for Parents of Children With Autism in Remote Areas. Focus Autism Other Dev. Disabil. 2013, 29, 23–38. [Google Scholar] [CrossRef]
  48. Davis, N.L.; Gough, M.; Taylor, L.L. Online teaching: Advantages, Obstacles and Tools for Getting It Right. J. Teach. Travel Tour. 2019, 19, 256–263. [Google Scholar] [CrossRef]
  49. Vismara, L.A.; McCormick, C.E.B.; Wagner, A.L.; Monlux, K.; Nadhan, A.; Young, G.S. Telehealth Parent Training in the Early Start Denver Model: Results From a Randomized Controlled Study. Focus Autism Other Dev. Disabil. 2018, 33, 67–79. [Google Scholar] [CrossRef]
  50. Arora, S.; Goodall, S.; Viney, R.; Einfeld, S. The MHYPEDD team Health-related quality of life amongst primary caregivers of children with intellectual disability. J. Intellect. Disabil. Res. 2020, 64, 103–116. [Google Scholar] [CrossRef]
  51. Totsika, V.; Hastings, R.P.; Vagenas, D. Informal caregivers of people with an intellectual disability in England: Health, quality of life and impact of caring. Health Soc. Care Community 2017, 25, 951–961. [Google Scholar] [CrossRef] [PubMed]
  52. Willner, P.; Rose, J.; Kroese, B.S.; Murphy, G.H.; Langdon, P.E.; Clifford, C.; Hutchings, H.; Watkins, A.; Hiles, S.; Cooper, V. Effect of the COVID-19 pandemic on the mental health of carers of people with intellectual disabilities. J. Appl. Res. Intellect. Disabil. 2020, 33, 1523–1533. [Google Scholar] [CrossRef] [PubMed]
  53. McBride, E.; Arden, M.A.; Chater, A.; Chilcot, J. The impact of COVID-19 on health behaviour, well-being, and long-term physical health. Br. J. Health Psychol. 2021, 26, 259–270. [Google Scholar] [CrossRef] [PubMed]
  54. Theis, N.; Campbell, N.; De Leeuw, J.; Owen, M.; Schenke, K.C. The effects of COVID-19 restrictions on physical activity and mental health of children and young adults with physical and/or intellectual disabilities. Disabil. Health J. 2021, 14, 101064. [Google Scholar] [CrossRef]
  55. Lorenz-Dant, K.; Comas-Herrera, A. The Impacts of COVID-19 on Unpaid Carers of Adults with Long-Term Care Needs and Measures to Address these Impacts: A Rapid Review of Evidence up to November 2020. J. Long Term Care 2021, 124–153. [Google Scholar] [CrossRef]
Figure 1. PRISMA diagram.
Figure 1. PRISMA diagram.
Healthcare 12 01349 g001
Figure 2. ROB2 domains [9,36].
Figure 2. ROB2 domains [9,36].
Healthcare 12 01349 g002
Figure 3. ROB2 overall judgment.
Figure 3. ROB2 overall judgment.
Healthcare 12 01349 g003
Table 1. Search terms.
Table 1. Search terms.
RCTFamily CarerOnlineProgrammeDisability
(“adaptive clinical trial” or “clinical trial” or “clinical trial, phase i” or “clinical trial, phase ii” or “clinical trial, phase iii” or “clinical trial, phase iv” or “controlled clinical trial” or “equivalence trial” or “multicenter study” or “pragmatic clinical trial” or “randomized controlled trial”).pt. or double-blind method/or “adaptive clinical trials as topic”/or “clinical trials as topic”/or “clinical trials, phase i as topic”/or “clinical trials, phase ii as topic”/or “clinical trials, phase iii as topic”/or “clinical trials, phase iv as topic”/or “controlled clinical trials as topic”/or “equivalence trials as topic”/or “intention to treat analysis”/or “non-randomized controlled trials as topic”/or “pragmatic clinical trials as topic”/or “randomized controlled trials as topic”/or “multicenter studies as topic”/or evaluation studies/or “evaluation studies as topic” or program evaluation/or validation studies/or “validation studies as topic” or (effectiveness or intervention or (pre-adj5 post-) or (pretest adj5 posttest) or (program* adj6 (evaluate or evaluated or evaluates or evaluating or evaluation or evaluations or evaluator or evaluators)(Care* or Parent* or Famil*)(Online or Internet or E-learning or Web* or Virtual or Computer-based or Electronic or Tech* or Telemedicine)(Program* or Intervention or Training or Education*)(Intellectual disab* or Learning disab)
Table 2. Characteristics of included studies.
Table 2. Characteristics of included studies.
Author and Country of OriginAim(s)Design Participants Intervention Description Intervention Duration Comparison Primary Outcomes and Time Points Findings
Flynn (2020)
UK
To examine whether Be Mindful can be delivered successfully to family carers of children or adults with ID, and whether it would be feasible to conduct a later definitive randomized controlled trial of the effectiveness and cost-effectiveness of Be Mindful.Feasibility randomised control trial—parallel design. Family carers (n = 60) (50 analysed)
Female (n = 55)
Mean age 46.09 (SD7.71)
The people for whom the participants cared: mean age of 13.73 (SD 8.97)
Diagnoses of ID (n = 54) and/or autism (n = 41).
Be Mindful is a publicly available online mindfulness programme developed by the Mental Health Foundation, which has ten easy-to-follow online sessions based on the elements of Mindfulness-Based Cognitive Therapy. Be Mindful+ participants were offered a further two telephone mentoring sessions—two 30 min telephone calls.4 weeks plus three 30 min telephone callsBe Mindful—without telephone mentoring sessionsPsychological well-being was assessed using the Warwick–Edinburgh Mental Well-being Scale (WEMWBS).
Baseline, 12 weeks and 6 months post-randomisation
At 6 months post-randomisation, there was a greater increase in psychological well-being for Be Mindful+ compared with Be Mindful, but this was not statistically significant (β = 2.88, p = 0.32).
Grenier-Martin (2022)
Canada
To evaluate the effectiveness of online parent training for managing problem behaviours for families awaiting specialized service.Randomised control trial—wait list control design. Family carers (n = 42) (29 analysed)
Age not reported
Sex not reported
The people for whom the participants cared:
4.1 (SD 1.7)
Diagnosis of chromosomal
abnormality or genetic syndrome (72.4%)
ID (24.1%) cerebral palsy (3.4%).
The online training was asynchronous (i.e., parents were able to learn at their own pace) and consisted of five separate modules. At the end of each module, quizzes evaluated parental comprehension.2 weeks to complete the intervention Waiting list—control group received intervention in week 4Parenting Stress Index–Short Form Self-assessment
Week 1, 6, 10, 14
Parental stress was reduced for the experimental group as compared to the control group at T2, F(1,26) = 7.63), p < 0.05.
Across both groups, training produced a significant decrease in parental stress from pre-training to follow-up: F(2,56) = 5.26, p < 0.01).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Leonard, R.; Hughes, N.; Forbes, T.; Brown, M.; Marsh, L.; Truesdale, M.; Todd, S.; Linden, M. Effectiveness of Online Programmes for Family Carers of People with Intellectual Disabilities: Systematic Review of the International Evidence Base. Healthcare 2024, 12, 1349. https://doi.org/10.3390/healthcare12131349

AMA Style

Leonard R, Hughes N, Forbes T, Brown M, Marsh L, Truesdale M, Todd S, Linden M. Effectiveness of Online Programmes for Family Carers of People with Intellectual Disabilities: Systematic Review of the International Evidence Base. Healthcare. 2024; 12(13):1349. https://doi.org/10.3390/healthcare12131349

Chicago/Turabian Style

Leonard, Rachel, Nathan Hughes, Trisha Forbes, Michael Brown, Lynne Marsh, Maria Truesdale, Stuart Todd, and Mark Linden. 2024. "Effectiveness of Online Programmes for Family Carers of People with Intellectual Disabilities: Systematic Review of the International Evidence Base" Healthcare 12, no. 13: 1349. https://doi.org/10.3390/healthcare12131349

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Article metric data becomes available approximately 24 hours after publication online.
Back to TopTop