*Article* **Pilot Study to Develop and Test Palliative Care Quality Indicators for Nursing Homes**

**Charlèss Dupont 1,2,\*, Robrecht De Schreye <sup>1</sup> , Joachim Cohen 1,2, Mark De Ridder <sup>3</sup> , Lieve Van den Block 1,2 , Luc Deliens 1,2,4 and Kathleen Leemans 1,3,\***


**Abstract:** An increasingly frail population in nursing homes accentuates the need for high quality care at the end of life and better access to palliative care in this context. Implementation of palliative care and its outcomes can be monitored by using quality indicators. Therefore, we developed a quality indicator set for palliative care in nursing homes and a tailored measurement procedure while using a mixed-methods design. We developed the instrument in three phases: (1) literature search, (2) interviews with experts, and (3) indicator and measurement selection by expert consensus (RAND/UCLA). Second, we pilot tested and evaluated the instrument in nine nursing homes in Flanders, Belgium. After identifying 26 indicators in the literature and expert interviews, 19 of them were selected through expert consensus. Setting-specific themes were advance care planning, autonomy, and communication with family. The quantitative and qualitative analyses showed that the indicators were measurable, had good preliminary face validity and discriminative power, and were considered to be useful in terms of quality monitoring according to the caregivers. The quality indicators can be used in a large implementation study and process evaluation in order to achieve continuous monitoring of the access to palliative care for all of the residents in nursing homes.

**Keywords:** nursing homes; quality indicators; quality measurement; palliative care; quality of care; end of life care

#### **1. Background**

In the past decade, in many Western countries, an increasing number of elderly persons were admitted to nursing homes. Projection studies concerning numbers of deaths and place of death suggest that, by the year 2040, the majority of deaths will occur in nursing homes [1,2]. Moreover, prediction studies also indicate that the need for high quality care at the end of life will most likely double in the nursing home setting, because of an increasing prevalence of frailty and multimorbidity in the resident population [2,3], being linked to a strongly reduced average length of stay of residents in recent years [4]. The quality of care at the end of life in long-term care facilities is currently high on the agenda: WHO and other health care organizations have advocated for good palliative care for older people already for years [5,6]. Additionally, in research, emphasis in the past decade has been placed on the quality of advance care planning, autonomy of residents, and the implementation of palliative care in nursing homes [7,8]. Until now, palliative care has been insufficiently developed in nursing homes and international studies show the late initiation of palliative care and even mostly for residents with a cancer diagnosis. An urgent need arises for better access for all elderly persons to high quality palliative care provision in the nursing home context.

Cohen, J.; De Ridder, M.; Van den Block, L.; Deliens, L.; Leemans, K. Pilot Study to Develop and Test Palliative Care Quality Indicators for Nursing Homes. *Int. J. Environ. Res. Public Health* **2021**, *18*, 829. https:// doi.org/10.3390/ijerph18 020829

**Citation:** Dupont, C.; De Schreye, R.;

Received: 16 December 2020 Accepted: 15 January 2021 Published: 19 January 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

When implementing palliative care in a nursing home context, it is important to evaluate its success and measure its outcomes for residents. One way to do so is the use of quality indicators within a continuous cycle of implementing, monitoring, and improvement [9]. By continuously monitoring quality of care and its outcomes and conducting implementation trajectories that are based on the results of these measurements, care teams are able to optimize the quality of care based on information, patient experiences, and best practice examples [10–13]. Quality indicators can be used within this monitoring cycle to provide data on subjective and objective aspects of quality of care over time. They are defined as measurable aspects of care, calculated as a percentage with a predefined numerator and denominator [14,15]. These indicators give caregivers information on their performance in terms of care processes and outcomes and which elements of care may need improvement [16]. Several national health care monitoring programs have been started in Western Countries, including Belgium, based on quality indicators. However, they mainly focus on the hospital or home setting or circumstances surrounding death such as symptoms and place of death [17–25]. Although initiatives have been taken for improving palliative care in nursing homes [24,26,27], researchers were not yet able to validate and implement solid quality indicators for palliative care in this specific setting.

A previous program to develop and implement palliative care quality indicators, the Belgian Q-PAC study, used a rigorous development method combining literature review, expert consultation, and pilot testing, resulting in a core set of 31 quality indicators covering a broad range of aspects of palliative care. The set was meant to be used for all palliative care services and settings, including nursing homes [24,28,29]. However, because the nursing home setting appeared to be too different from the specialized palliative care services in terms of organization and structure of care (e.g., no dedicated palliative care teams), and in characteristics of the population cared for (e.g., specific population with frailty, dementia, cognitive decline), the quality indicator set was implemented into all specialized palliative care services, but not in the nursing homes from 2014 onwards [30]. Therefore, a need persisted to investigate which indicators can be used for monitoring the quality of palliative care in the nursing home context.

Because of the increasing need for development of palliative care and its monitoring in nursing homes, we started a project to develop a set of quality indicators for the quality of palliative care in nursing homes. Previous research already highlighted the importance of person centered care through autonomy and involvement of family, but also communication and advance care planning in nursing homes [31–34]; hence, we decided to develop quality indicators specifically targeting advance care planning, palliative care, and end of life care. The main aim for nursing home teams is to obtain insights in their care processes and outcomes, and further develop missing elements in the care for their residents. In this study, we develop and test a quality indicator set and measurement procedure for palliative care in nursing home context.

#### **2. Methods**

#### *2.1. Design*

We used a two-step approach with a mixed-method design based on a standardized indicator testing protocol for generic quality indicators in order to develop the quality indicator set and measurement procedure [29,35]. First, we developed the set of quality indicators using the Rand/UCLA appropriateness method in three phases: (1) literature review to develop a preliminary set of quality indicators, (2) interviews with experts to test face validity of the preliminary set of quality indicators, and (3) indicator selection by expert evaluation [36]. Second, we evaluated the face validity, feasibility, discriminative power, and usefulness of the quality indicators in a quantitative cross-sectional application of the quality indicators in combination with qualitative interviews. Figure 1 shows an overview of the development process and pilot testing.

an overview of the development process and pilot testing.

**Figure 1.** Overview of study phases and accompanying results. **Figure 1.** Overview of study phases and accompanying results.

*2.2. Step 1: Indicator and Questionnaire Development 2.2. Step 1: Indicator and Questionnaire Development*

2.2.1. Phase 1 and 2: Literature Study and Expert Interviews 2.2.1. Phase 1 and 2: Literature Study and Expert Interviews

In phase 1, we identified a comprehensive set of candidate quality indicators for palliative care in nursing homes. Therefore, we adapted the existing Belgium quality indicators for specialized palliative care listed in the Q-PAC (Quality indicators for Palliative Care) study [24] to the nursing home context. To do so, we searched literature in PubMed while using a snowball method starting with the reviews of Pasman et al. and De Roo et al. [37,38]. We searched for existing quality indicators, domains of quality of care for elderly and questionnaires or instruments for quality of palliative care, advance care planning, and end-of-life care in the nursing home context. Candidate quality indicators could be processed or outcome indicators and emphasis were placed particularly on subjective quality indicators in order to reflect the user perspective on quality of care. In phase 2, we performed interviews with relevant stakeholders (i.e., healthcare professionals, community-based organizations, and policy makers) that are involved in Flemish nursing home care, to test face the validity of the candidate quality indicators and gather additional indicators and domains for this specific context not found in literature. This way, an iterative process of literature search and interviews lead to the selection of a preliminary set of quality indicators representing all of the identified domains. Furthermore, to be able to calculate the quality indicators, we operationalized each of them into questions for residents, bereaved family and nursing home staff, accompanied with measurement instructions. This was done based on input that was provided by the experts and questionnaires In phase 1, we identified a comprehensive set of candidate quality indicators for palliative care in nursing homes. Therefore, we adapted the existing Belgium quality indicators for specialized palliative care listed in the Q-PAC (Quality indicators for Palliative Care) study [24] to the nursing home context. To do so, we searched literature in PubMed while using a snowball method starting with the reviews of Pasman et al. and De Roo et al. [37,38]. We searched for existing quality indicators, domains of quality of care for elderly and questionnaires or instruments for quality of palliative care, advance care planning, and end-of-life care in the nursing home context. Candidate quality indicators could be processed or outcome indicators and emphasis were placed particularly on subjective quality indicators in order to reflect the user perspective on quality of care. In phase 2, we performed interviews with relevant stakeholders (i.e., healthcare professionals, community-based organizations, and policy makers) that are involved in Flemish nursing home care, to test face the validity of the candidate quality indicators and gather additional indicators and domains for this specific context not found in literature. This way, an iterative process of literature search and interviews lead to the selection of a preliminary set of quality indicators representing all of the identified domains. Furthermore, to be able to calculate the quality indicators, we operationalized each of them into questions for residents, bereaved family and nursing home staff, accompanied with measurement instructions. This was done based on input that was provided by the experts and questionnaires identified in the literature.

#### 2.2.2. Phase 3: Expert Consensus

identified in the literature.

*Int. J. Environ. Res. Public Health* **2021**, *18*, x. https://doi.org/10.3390/xxxxx www.mdpi.com/journal/ijerph The preliminary set of candidate quality indicators was sent to 15 experts (see results infra). They were asked to score the quality indicators with "1" as "not appropriate" to "9" as "very appropriate" in order to measure the quality of palliative care in nursing homes. Experts were provided with the candidate indicator's description, rationale, numerator and denominator, question (per response type: residents, bereaved family, or nursing home staff) and literature source. They were also able to suggest missing domains or themes.

The median scores on appropriateness were calculated per candidate indicator. The quality indicators were then categorized based on the RAND/UCLA consensus method: accepted, to be decided, or rejected. Indicators were immediately accepted if they had a median score of 7 or more and if no more than two experts scored the indicator with a 1, 2, or 3 (strict positive consensus). Indicators with a median score of 3 or lower and for which no more than two experts scored the indicator 7, 8, or 9 were immediately rejected (strict negative consensus) [36]. All other to-be-decided indicators were discussed during the one-day plenary discussion until consensus was found regarding rejection or acceptance among the experts. During the discussion, additional selection criteria were to be considered as defined by the researchers: (1) a maximum of eight quality indicators per questionnaire (i.e., response type) was suggested to ensure feasibility of the quality monitoring in nursing homes, without overburdening nursing home staff, and (2) experts were encouraged to consider a good balance between process and outcome indicators, as well as objective and subjective quality indicators. As such, experts were asked to select the eight most important indicators for each response type.

#### 2.2.3. Questionnaires to Measure the Quality Indicators

After defining the quality indicators together with the experts, we developed four questionnaires to be able to calculate the performance score per quality indicator. These questionnaires were based on questions of validated scales as much as possible, or if no good question gathering the right information for a specific indicator existed in the literature, it was developed by the researchers, together with the experts. An overview of all indicators, accompanying questions and evidence can be found in the Supplementary Table S1.

Two questionnaires were developed to measure indicators of quality of care for residents who currently lived in the facility: one for the resident [1] and one for the most involved professional caregiver [2]. To be able to question every resident, we decided, in consultation with the experts, to create an adapted version of the questionnaire for residents who needed help to fill out a questionnaire because of physical or mental health issues. The questions in this version are the same as the questions in the standard resident questionnaire, but are reformulated from second to third person. They can be read to the resident or filled out by the resident's informal caregiver (or professional caregivers if no informal caregiver was noted in the patient record), preferably together with the resident.

A questionnaire was developed for the closest family caregiver (as noted in the health record) [3] and a separate one for the most involved professional caregiver [4] in order to measure indicators of quality of care for residents who passed away in the facility within the last six months. We performed a cognitive testing for all questionnaires in the corresponding responder group (i.e., residents, family, and professional caregivers). We tested the comprehensibility and response burden: recommendations resulted in minor linguistic changes for both residents and family caregivers.

#### *2.3. Step 2: Pilot Testing*

#### 2.3.1. Design

We used a mixed-method design, including a quantitative application of the quality indicators and qualitative interviews with the nursing home staff using the instrument, in order to evaluate the face validity, feasibility, discriminative power, and usefulness of the instrument.

#### 2.3.2. Setting and Participants

Nursing homes were recruited on a voluntary basis through a call for participation via involved community-based organizations. From the 24 candidates, we selected a purposive sample of nine nursing homes, while considering the number of beds (between 64 and 290 beds), the organizational structure (i.e., six profit and three non-profit) and the geographical location (every Flemish province was represented).

Nursing homes were able to measure the quality indicators via questionnaires through a cross-sectional inclusion design. This method allows for nursing homes to gather information on residents who were currently living in the nursing home as well as those who had passed away. Following inclusion criteria were used:

Residents who were currently living in the nursing home and:

• lived for a minimum of one month in the facility;

Residents who had passed away and:


#### 2.3.3. Measurement Procedure

All nursing homes followed the same measurement procedure based on a previously developed and tested method in order to measure the quality indicators via questionnaires [16]. Before the start of the pilot test, a coordinator per nursing home was appointed in consultation with the researchers. The researchers visited the coordinator (in the nursing homes) in order to explain the study, expectations, measurement procedure, how to work with the online questionnaires, going through the detailed instruction manual. The coordinator responsibilities include the supervision of the measurement procedure, communication within the nursing home (e.g., informing the staff about the instrument and procedure, announcing start date), drafting the list for including residents (in concordance with the researcher), and distribution of the questionnaire among residents, family caregivers and staff (Figure 2). The coordinator was also asked to keep a diary and note thoughts regarding the workload, setbacks, and/or other findings (Figure 2). *Int. J. Environ. Res. Public Health* **2021**, *18*, x FOR PEER REVIEW 6 of 17

**Figure 2.** Measurement procedure. **Figure 2.** Measurement procedure.

Nursing homes were asked to include minimum of 2/3 of all residents at random and all deceased residents who met the inclusion criteria. We developed an inclusion matrix, depending on the number of residents per nursing home. We used online questionnaires via Limesurvey because nursing home staff were responsible for the distribution of questionnaires and to ensure responders privacy. No IP addresses were saved to guarantee anonymity. Residents could fill out the questionnaire via portable computers or tablets available in the nursing homes; family members received a link to the questionnaire via email; inhouse caregivers accessed the online questionnaire via computers in the nursing homes or on their private computers. Nursing homes were asked to include minimum of 2/3 of all residents at random and all deceased residents who met the inclusion criteria. We developed an inclusion matrix, depending on the number of residents per nursing home. We used online questionnaires via Limesurvey because nursing home staff were responsible for the distribution of questionnaires and to ensure responders privacy. No IP addresses were saved to guarantee anonymity. Residents could fill out the questionnaire via portable computers or tablets available in the nursing homes; family members received a link to the questionnaire via email; inhouse caregivers accessed the online questionnaire via computers in the nursing homes or on their private computers.

#### 2.3.4. Feedback and Evaluation 2.3.4. Feedback and Evaluation

Per nursing home, a report was created, summarizing the individual and overall performance scores in a structured and standardized way. Nursing home coordinators were responsible for communicating the results to the nursing home staff (step 3 and 4 in Figure 2). After the report was sent to the nursing homes, the researchers visited the nursing Per nursing home, a report was created, summarizing the individual and overall performance scores in a structured and standardized way. Nursing home coordinators were responsible for communicating the results to the nursing home staff (step 3 and 4

> with open-ended questions. The coordinators kept a diary during the measurement and delivered it in advance to the researcher. During the interview, the workload of the coordinator and the nursing home staff was evaluated and barriers and facilitators in the use of the quality indicators were identified. Coordinators could also share their thoughts on future use of the instrument and wider implementation. The researcher kept a diary of evaluation points, remarks, and questions for further qualitative evaluation during the

> Data collection was closed after one month. Performance scores (non-adjusted mean) per quality indicator were calculated while using the defined numerators and denominators (range 0–100). In order to evaluate feasibility and discriminative power for individual quality indicator, we used descriptive and psychometric analyses in Microsoft Excel and SAS. Furthermore, the interviews were conducted with all coordinators in order to evaluate the face validity and usability of the indicators and the feasibility of the procedure. Together with the diaries of the coordinators and the field notes of the researcher, these interviews were analyzed while using a thematic framework approach, which was based on the barriers and facilitators for implementation framework of Grol and Wensing

*Int. J. Environ. Res. Public Health* **2021**, *18*, x. https://doi.org/10.3390/xxxxx www.mdpi.com/journal/ijerph

whole period of the pilot test.

2.3.5. Analyses

in Figure 2). After the report was sent to the nursing homes, the researchers visited the nursing homes for an evaluation interview with the coordinator, while using an interview guide with open-ended questions. The coordinators kept a diary during the measurement and delivered it in advance to the researcher. During the interview, the workload of the coordinator and the nursing home staff was evaluated and barriers and facilitators in the use of the quality indicators were identified. Coordinators could also share their thoughts on future use of the instrument and wider implementation. The researcher kept a diary of evaluation points, remarks, and questions for further qualitative evaluation during the whole period of the pilot test.

#### 2.3.5. Analyses

Data collection was closed after one month. Performance scores (non-adjusted mean) per quality indicator were calculated while using the defined numerators and denominators (range 0–100). In order to evaluate feasibility and discriminative power for individual quality indicator, we used descriptive and psychometric analyses in Microsoft Excel and SAS. Furthermore, the interviews were conducted with all coordinators in order to evaluate the face validity and usability of the indicators and the feasibility of the procedure. Together with the diaries of the coordinators and the field notes of the researcher, these interviews were analyzed while using a thematic framework approach, which was based on the barriers and facilitators for implementation framework of Grol and Wensing [10,39,40]. Table 1 presents an overview of all evaluation aspects, accompanying methods, and criterions.


**Table 1.** Overview of the evaluation and accompanying methods and criterions.

#### *2.4. Ethical and Language Issues*

This study is approved by the Ethical Review Board of Brussels University Hospital of the Vrije Universiteit Brussel (protocol: QPACWZC01 BUN: 143201838240). All of the respondents (i.e., residents, family, and nursing home staff) received an online questionnaire, including cover letter and informed consent. Only questionnaires with signed informed consent were used to calculate performance scores. No IP addresses, names, or other personal identifiers were saved in the online questionnaire system.

All of the indicators and questionnaires were developed and evaluated in Dutch. All of the interviews and trainings were performed in Dutch. The English translation was done specifically for this article. Dutch versions of the indicators or questionnaires are available on request.

#### **3. Results**

#### *3.1. Step 1: Indicator Development*

3.1.1. Phase 1 and 2: Literature Study and Expert Interviews

Based on the existing QPAC quality indicator set for specialized palliative care, the additional literature search (phase 1) and interviews with relevant stakeholders (n = 10) (phase 2), we identified 26 candidate quality indicators in eight domains of quality of palliative care for elderly persons in nursing homes. Table 2 shows the difference between the Q-PAC domains (specializes palliative care) and the domains for the nursing homes based on literature search and stakeholder interviews in phase 2.


**Table 2.** Eight thematic domains for quality indicators.

#### 3.1.2. Phase 3: Expert Consensus

Based on their individual evaluation of the 26 candidate quality indicators, seven quality indicators were immediately accepted and included. None were immediately rejected, so the remaining 19 quality indicators were debated in a one-day plenary discussion until consensus was found. Nine of 19 quality indicators were eventually accepted and three were newly developed during the meeting and added to the draft set. After the discussion, a set of in total 19 quality indicators were drafted and per email consented by all experts (Tables 3 and 4). In Supplementary Table S1, the full list of quality indicators, as was tested in the pilot phase, is presented with accompanying numerator, denominator, question, and source.

**Table 3.** Participants in expert consultation rounds.



#### **Table 4.** Quality indicators for palliative care in nursing homes.


#### **Table 4.** *Cont.*

PC = Palliative care; ACP = advance care planning; EOL = end of life.

#### *3.2. Step 2: Pilot Test*

#### 3.2.1. Responder Characteristics

Nine nursing homes tested the quality indicator set and measurement procedure. In total, 294 residents, 393 professional caregivers (345 for residents who currently lived in the facility and 48 for deceased residents), and 34 family caregivers completed the whole questionnaire and hence were included for the pilot study. We asked nursing homes to list the total number of inclusions, but four of them did not perform this assignment correctly; hence, we lack information on the response rates in this study. In total 214 of the residents were female and the majority (53%) of residents was between 85- and 94-years old. Table 5 presents an overview of characteristics.

**Table 5.** Characteristics per response type in the pilot test.


A: 5 missings for length of stay. B: Questioned only when the family caregivers completed the questionnaire in the name of the resident. C: Length of stay in months.

#### 3.2.2. Psychometric Analyses: Feasibility and Discriminative Power

None of the indicators had too many missing (>10%) answers. The quality indicators showed good discriminative power, as there were no indicators that had 95% or more answers in an extreme category (Table 3). Only two indicators had a variation range (minmax) smaller than 20 percentage points between different nursing homes, i.e., 'Recognizing the approaching death' and 'Satisfied by care delivered' and, hence, showed problems with sensitivity to change.

#### 3.2.3. Qualitative Analyses; Feasibility, Usefulness and Face Validity

We interviewed all nine coordinators of the included nursing homes. With regard to face validity, all of the coordinators confirmed that the appeared to reflect their practice and seemed valid. As indicated by one on the coordinators: "*The results indicate clear work points and results are recognizable*". They also agreed the results were easy to interpret and useful in terms of improving their service, but they indicated that they struggled in establishing concrete improvement goals that are based on the quality indicator scores. The coordinators evaluated the length of all four questionnaires as feasible, but four coordinators declared that they would prefer paper questionnaires for residents and family, as this may improve response rates. As indicated by one of the coordinators: "*We would prefer paper questionnaires* . . . *we* [staff in de nursing home] *don't have professional email addresses and I didn't want to send*

#### *the questionnaires to their private email. Also, our residents don't know how to use a computer or tablet and therefore some residents who normally could fill in a questionnaire alone, now couldn't*".

Moreover, all of the coordinators indicated that they would use the instrument again and evaluated the instruction manual as useful and sufficient and assumed the instrument could be executed without the researchers. One of the coordinators said: "*Training was okay, the manual is clear and I think we could have managed without* [the manual]". Additionally, all of the coordinators indicated that the workload was feasible and worthwhile, although they declared that preparing the list of respondents was time-consuming, as they had to acquire their own approach.

Based on field notes, the interviews with the coordinators, and diaries of the same coordinator, we made an overview of facilitators and barriers regarding the use of the instrument (Table 6) in general terms and per step of the measurement procedure, as described in Figure 2.

**Table 6.** Facilitators and barriers based on the interviews and dairies regarding the use of the instrument.


#### **4. Discussion**

In this study, we developed and evaluated a quality indicator set and a tailored measurement procedure consisting of 19 indicators to monitor the quality of palliative care in Belgian nursing homes. The composition of this indicator set is based on previously developed quality indicators for specialized palliative care, but, after adaptation to the nursing home context by experts and stakeholders, the themes differ somewhat: more emphasis is placed on autonomy and dignity of the nursing home residents. From this first pilot study, the quality indicators seem to be valid and the measurement procedure feasible for caregivers in nursing homes who are interested in improving the quality of end-of-life care within their center. From the psychometric analyses, we found that most of the quality indicators were feasible and they showed good discriminative power. The instrument appeared to reflect practice and hence confirmed face validity, according to coordinators during the qualitative interviews. The measurement procedure was evaluated by the interviewed coordinators as feasible and they indicated the measurement of the quality indicators could be performed based on the manual without extra help of the researchers. Overall, this study shows that the quality indicators are ready for further use in a large implementation study in Flemish nursing homes in order to further evaluate their feasibility, usefulness, discriminative power, and potential for quality improvement.

An evaluation of the quality of care with quality indicators best includes process as well as outcomes indicators of care in one monitoring cycle [16,41]. The quality indicator set for palliative care that we developed for nursing homes uses both types of indicators. We also included objective as well as subjective quality indicators. The psychometric analyses in this pilot study showed good results for all of the indicators on discriminative power expect for 'recognizing the approaching death' and 'satisfied by care delivered', which are both indicators subjectively measured by caregivers. Both of the indicators might have been influenced by response bias, due to social desirability or a tendency to overestimate their skills [42] and were discarded from the quality indicator set. From this finding we might conclude that, when using self-assessment instruments for quality monitoring, caregivers should report as much as possible on objective information of care, i.e., information that can be found in the patient file. Such biases can best be monitored by regularly evaluating the quality indicator set for psychometric criteria, in order to keep the quality data sensitive to changes in quality of care over time and between health care services.

An important strength of our study is the rigorous, systematic development method while using stakeholders and the mixed-method design, including the RAND/UCLA method for indicator development, quantitative analysis of data, and qualitative interviews with the coordinators in the nursing homes to evaluate the instrument. Hence, we were able to evaluate the instrument and its measurement procedure in terms of face validity, feasibility, discriminative power, and usefulness. Additionally, because the thoroughly follow-up with the involved coordinators before, during and after the pilot, barriers and facilitators influencing the course of the measurement were identified. The small database is one of the limitations of this study. Psychometric analyses were limited and a study on further implementation is necessary to evaluate and validate the instrument including the quality indicators. The absence of response rates is another limitation of our study. Although coordinators drafted a list of included residents, we were unable to match them with the questionnaires because of GDPR policies. Additionally, although we aimed to include as many residents as possible while using three versions of the questionnaire in the resident's evaluation, we have no insights regarding whether residents with cognitive problems, such as dementia, were sufficiently involved in the quality monitoring.

Worldwide initiatives have been taken to monitor and improve the quality of palliative care in different settings [43–47]. Several studies have pointed out that the quality of dying and end of life care is not optimal across Western countries [48–50]. Some of these studies also used quality indicators in order to evaluate quality of palliative care in this setting, albeit being mostly focused on cancer patients, hospital and home setting, and administrative data in order to gather information on care processes and patient outcomes. Therefore, these measures are labeled as objective indicators and, although they provide a good basis for quality monitoring, they are not enough to point out strengths and weaknesses in specific long-term care organizations. Additionally, user perspective needs to be considered through subjective quality measures [51]. With our instrument, we focused on nursing homes and tried to combine both objective and subjective measures into different stakeholder perspectives in order to reach a comprehensive picture on quality of palliative care. Only this way, important themes for elderly persons, such as dignity and autonomy, can be properly addressed in order to improve the quality of care in the light of also improving the quality of life for residents in nursing homes. According to our qualitative analysis in this pilot study, coordinators indeed found the results of their measurement recognizable for their nursing home, supporting the face validity of these indicators (i.e., they are measuring what they aim to measure) from a caregiver point of view. This is an important finding, because, in order to reach effective change in health care, the value of timely and recognizable feedback is a crucial incentive for caregivers in order to continuously engage in these monitoring and improvement processes [11,40,45,52–54].

In light of care improvement in the field of palliative care in a nursing home context, a large-scale research project, 'PACE steps to success', has recently been implementing a combination of tailored improvement initiatives focusing on communication, advanced care planning, and knowledge and skills on end of life care while using a train-the-trainer implementation model. Although the intervention did not show significant effect on their primary outcome (comfort in the last week of life for residents), the process evaluation showed that the implementation rate was highly variable between countries and teams, and several challenges arose, such as attitude and motivation of staff, and skills and expertise of the trainer appointed to the individual nursing homes [26,48]. Our previous implementation research in palliative care already showed that caregivers are willing to invest in quality improvement trajectories and learn from other teams, but they need support from their management and financial reimbursement or staff to engage in these activities [40]. In this pilot study, we found the same barriers and facilitators pointing out the importance of setting the right preconditions for implementation in the nursing home context, throughout research and policy. This might be done by primordially evaluating nursing home readiness in order to increase the use and correct application of the quality indicators [55].

#### **5. Conclusions**

In this study, we developed and evaluated a quality indicator set and a tailored measurement procedure consisting of 19 indicators to monitor quality of palliative care in Belgian nursing homes. We combined both objective and subjective measures into four questionnaires for different perspectives in order to reach a comprehensive picture on quality of palliative care, end-of-life care and advance care planning in nursing homes. Care teams in nursing homes are able to monitor themselves based on these indicator scores. We found, while using both quantitative as qualitative analyses, the developed instrument had good face validity, feasibility, discriminative power, and it is useful in terms of quality monitoring according to caregivers, though establishing concrete improvement goals based on quality indicator scores remains difficult for them. The quality indicators are ready for further use in a large implementation study and process evaluation in Flemish nursing homes in order to further evaluate their feasibility, usefulness, discriminative power, and potential for quality improvement.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/1660-460 1/18/2/829/s1, Table S1: list of quality indicators with numerators, denominators and questions.

**Author Contributions:** All authors have substantially contributed to the research. Conceptualization, J.C., L.V.d.B., L.D. and K.L. Methodology, J.C., L.V.d.B., L.D., M.D.R., R.D.S. and K.L.; Formal Analysis, R.D.S., C.D. and K.L.; Investigation, R.D.S., C.D. and K.L.; Writing—Original Draft Preparation, C.D., R.D.S. and K.L.; Writing—Review & Editing, J.C., L.V.d.B., L.D., and M.D.R.; Supervision, J.C., L.V.d.B., L.D., M.D.R. and K.L.; Project Administration, R.D.S. and C.D.; Funding Acquisition, J.C., L.V.d.B., L.D., M.D.R. and K.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This study was funded by the Flemish Agency of Healthcare (AZG/WEL/TPKO/2019/ 20190405).

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethical Review Board of Brussels University Hospital of the Vrije Universiteit Brussel (protocol: QPACWZC01 BUN: 143201838240)

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Acknowledgments:** We would like to thank all nursing homes and their residents who voluntarily participated in this pilot study for their time and effort. We would also like to thank all stakeholders and experts as well as the staff members of the Flemish Agency of Healthcare for their comments and expertise in this study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


International Journal of *Environmental Research and Public Health*
