Assessing Nursing Homes Quality Indicators’ between-Provider Variability and Reliability: A Cross-Sectional Study Using ICCs and Rankability
Abstract
:1. Introduction
2. Materials and Methods
2.1. Design and Sample
2.2. Variables and Measurements
2.3. Data Collection
2.4. Statistical Analysis
2.4.1. Between-Provider Variability: ICC1 and Rankability
2.4.2. Reliability: ICC2
2.5. Data Management and Ethical Considerations
3. Results
3.1. Sample and Quality Indicators Description
3.2. Between-Provider Variability: ICC1 and Rankability
3.3. Reliability: ICC2
4. Discussion
5. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
Appendix A
Selection Process of the Four Themes of the Swiss Quality Indicators
Appendix B
Assessment Instrument 1 | Nursing Home Resident Assessment Instrument | Planification Informatisée des Soins Infirmiers Requis 2 | BewohnerInnen-Einstufungs-und Abrechnungssystem 3 |
---|---|---|---|
Abbreviation | RAI-NH | PLAISIR/PLEX | BESA |
Distributor in Switzerland | Q-Sys | Eros | BESAcare |
Language availability | German, French, Italian | French | German, French, Italian |
QI variables integration | Updated version of the instrument | Additional module | Updated version of the instrument |
Data collection by | Healthcare staff | Healthcare staff or external evaluators (choice of each NH) | Healthcare staff |
Start of the data collection (month) | July 2016 | July 2016 | July 2016 |
Data export (month) | August 2017 | February 2017 | August 2017 |
Appendix C
Theme | ICC1 (95% CI) | ICC2 (95% CI) | Rankability (ρ) |
---|---|---|---|
Polypharmacy | 0.055 (0.037–0.068) | 0.898 (0.865–0.918) | 0.120 |
Self-reported pain | 0.119 (0.087–0.149) | 0.953 (0.931–0.962) | 0.437 |
Observed pain | 0.147 (0.113–0.177) | 0.963 (0.949–0.971) | 0.575 |
Physical restraint, trunk fixation or seating that prevents the resident from rising | 0.343 (0.235–0.405) | 0.988 (0.980–0.991) | 0.970 |
Physical restraint, bedrails | 0.245 (0.197–0.286) | 0.980 (0.973–0.983) | 0.783 |
Weight loss | 0.135 (0.095–0.165) | 0.959 (0.941–0.969) | 0.715 |
References
- Organisation for Economic Co-operation and Development. Improving Value in Health Care; OECD Publishing: Paris, France, 2010. [Google Scholar]
- Mainz, J. Defining and classifying clinical indicators for quality improvement. Int. J. Qual. Health Care 2003, 15, 523–530. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Lilford, R.; MMohammed, A.; Spiegelhalter, D.; Thomson, R. Use and misuse of process and outcome data in managing performance of acute medical care: Avoiding institutional stigma. Lancet 2004, 363, 1147–1154. [Google Scholar] [CrossRef]
- Anderson, J.; Hackman, M.; Burnich, J.; Gurgiolo, T.R. Determining hospital performance based on rank ordering: Is it appropriate? Am. J. Med. Qual. 2007, 22, 177–185. [Google Scholar] [CrossRef] [PubMed]
- Mor, V. Defining and measuring quality outcomes in long-term care. J. Am. Med. Dir. Assoc. 2007, 8 (Suppl. 2), e129–e137. [Google Scholar] [CrossRef]
- Arling, G.; Kane, R.L.; Lewis, T.; Mueller, C. Future development of nursing home quality indicators. Gerontologist 2005, 45, 147–156. [Google Scholar] [CrossRef] [Green Version]
- Van Dishoeck, A.M.; CLooman, W.N.; Lier, E.C.M.; Mackenbach, J.P.; Steyerberg, E.W. Displaying random variation in comparing hospital performance. BMJ Qual. Saf. 2011, 20, 651–657. [Google Scholar] [CrossRef]
- Normand, S.-L.T.; Glickman, M.E.; Gatsonis, C.A. Statistical Methods for Profiling Providers of Medical Care: Issues and Applications. J. Am. Stat. Assoc. 1997, 92, 803–814. [Google Scholar] [CrossRef]
- Normand, S.-L.; Shahian, T.D.M. Statistical and clinical aspects of hospital outcomes profiling. Stat. Sci. 2007, 22, 206–226. [Google Scholar] [CrossRef] [Green Version]
- Burke, R.E.; Werner, R.M. Quality measurement and nursing homes: Measuring what matters. BMJ Qual. Saf. 2019, 28, 520–523. [Google Scholar] [CrossRef]
- Du Moulin, M.F.; van Haastregt, J.C.; Hamers, J.P. Monitoring quality of care in nursing homes and making information available for the general public: State of the art. Patient Educ. Couns. 2010, 78, 288–296. [Google Scholar] [CrossRef]
- Ibrahim, J.E.; Chadwick, L.; MacPhail, A.; McAuliffe, L.; Koch, S.; Wells, Y. Use of quality indicators in nursing homes in Victoria, Australia: A cross-sectional descriptive survey. J. Aging Health 2014, 26, 824–840. [Google Scholar] [CrossRef] [PubMed]
- Zimmerman, D.R. Improving nursing home quality of care through outcomes data: The MDS quality indicators. Int. J. Geriatr. Psychiatry 2003, 18, 250–257. [Google Scholar] [CrossRef] [PubMed]
- Canadian Institute for Health Information. When a Nursing Home Is Home: How Do Canadian Nursing Homes Measure up on Quality? CIHI: Ottawa, ON, Canada, 2013. [Google Scholar]
- VLAAMS Indicatoren Project. Vlaams Indicatorenproject Woonzorgcentra: Kwaliteitsindicatoren 2019 Deel 1 [Flemish Indicators Project for Residential Care Centres: Quality Indicators 2019 Part 1]. 2020. Available online: https://www.zorgkwaliteit.be/sites/default/files/imports/wzc_20192.pdf (accessed on 19 April 2020).
- Article 59a of the Swiss Federal Act of 18 March 1994 on Health Assurance (KVG/LAmal; RS 832.10). 2017. Available online: https://www.admin.ch/opc/fr/classified-compilation/19940073/index.html (accessed on 14 November 2020).
- Zúñiga, F.; Blatter, C.; Wicki, R.; Simon, M. Fragebogenerhebung zur Zuverlässigkeit der Daten und Nützlichkeit aus Sicht der Anwender [National quality indicators in Swiss nursing homes: Questionnaire survey on data reliability and users’ view on the usefulness]. Zeitschrift fur Gerontologie und Geriatrie 2019, 52, 730–736. [Google Scholar]
- Bundesamt für Statistik. Bundesamt für Statistik, Schnittstellenkonzept Medizinische Qualitätsindikatoren Pflegeheime, Version 1.3 [Interface Concept of the Medical Quality Indicators in Nursing Homes, Version 1.3]; University of Basel: Basel, Switzerland, 2019. [Google Scholar]
- Snijders, T.A.; Bosker, R.J. Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling, 2nd ed.; SAGE: Los Angeles, CA, USA; London, UK, 2012. [Google Scholar]
- Austin, P.C.; Ceyisakar, I.E.; Steyerberg, E.W.; Lingsma, H.F.; de Mheen, P.J.M. Ranking hospital performance based on individual indicators: Can we increase reliability by creating composite indicators? BMC Med. Res. Methodol. 2019, 19, 131. [Google Scholar] [CrossRef] [PubMed]
- Lingsma, H.F.; Eijkemans, M.J.; Steyerberg, E.W. Incorporating natural variation into IVF clinic league tables: The Expected Rank. BMC Med. Res. Methodol. 2009, 9, 53. [Google Scholar] [CrossRef] [Green Version]
- Bliese, P.D. Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In Multilevel Theory, Research, and Methods in Organizations: FOUNDATIONS, Extensions, and New Directions; Jossey-Bass: San Francisco, CA, USA, 2000; pp. 349–381. [Google Scholar]
- Bundesamt für Statistik. Statistik der Sozialmedizinischen Institutionen 2017-Definitive Standardtabellen [Statistics of Socio-Medical Institutions 2017-Definitive Standard Tables]; Bundesamt für Statistik: Neuchâtel, Switzerland, 2017. [Google Scholar]
- Anderson, R.L.; Buckwalter, K.C.; Buchanan, J.L.; Maas, M.L.; Imhof, S.L. Validity and reliability of the Minimum Data Set Depression Rating Scale (MDSDRS) for older adults in nursing homes. Age Ageing 2003, 32, 435–438. [Google Scholar] [CrossRef] [Green Version]
- Paquay, L.; de Lepeleire, J.; Schoenmakers, B.; Ylieff, M.; Buntinx, O.F.F. Comparison of the diagnostic accuracy of the Cognitive Performance Scale (Minimum Data Set) and the Mini-Mental State Exam for the detection of cognitive impairment in nursing home residents. Int. J. Geriatr. Psychiatry 2007, 22, 286–293. [Google Scholar] [CrossRef] [Green Version]
- Jokanovic, N.; Tan, E.C.; Dooley, M.J.; Kirkpatrick, C.M.; Bell, J.S. Prevalence and factors associated with polypharmacy in long-term care facilities: A systematic review. J. Am. Med. Dir. Assoc. 2015, 16, 535e1. [Google Scholar] [CrossRef]
- Vettori, A.; von Stokar, T.; Petry, C.; Britt, D.; Saxer, H.G.S. Mindestanforderungen für Pflegebedarfserfassungssysteme (Schlussbericht) [Minimum Requirements for Care Needs Assessment Systems]; Final Report; Bundesamt für Gesundheit (BAG): Neuchâtel, Switzerland, 2017. [Google Scholar]
- Van Dishoeck, A.M.; Lingsma, H.F.; Mackenbach, J.P.; Steyerberg, E.W. Random variation and rankability of hospitals using outcome indicators. BMJ Qual. Saf. 2011, 20, 869–874. [Google Scholar] [CrossRef]
- Nakagawa, S.; Schielzeth, H. Repeatability for Gaussian and non-Gaussian data: A practical guide for biologists. Biol. Rev. Camb. Philos. Soc. 2010, 85, 935–956. [Google Scholar] [CrossRef]
- LeBreton, J.J.; Senter, J.L. Answers to 20 Questions About Interrater Reliability and Interrater Agreement. Organ. Res. Methods 2008, 11, 815–852. [Google Scholar] [CrossRef]
- Stoffel, M.A.; Nakagawa, S.; Schielzeth, H. An Introduction to Repeatability Estimation with rptR. 2013. Available online: https://cran.r-project.org/web/packages/rptR/vignettes/rptR.html) (accessed on 19 April 2020).
- Stoffel, M.A.; Nakagawa, S.; Goslee, H.S.S. rptR: Repeatability estimation and variance decomposition by generalized linear mixed-effects models. Methods Ecol. Evol. 2017, 8, 1639–1644. [Google Scholar] [CrossRef] [Green Version]
- Steyerberg, E.W. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating; Springer: New York, NY, USA, 2009. [Google Scholar]
- R Development Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2019. [Google Scholar]
- Vos, E.L.; Lingsma, H.F.; Jager, A.; Schreuder, K.; Spronk, P.; Peeters, M.V.; Koppert, S.S.L.B. Effect of Case-Mix and Random Variation on Breast Cancer Care Quality Indicators and Their Rankability. Value Health 2020, 23, 1191–1199. [Google Scholar] [CrossRef] [PubMed]
- Rantz, M.J.; Hicks, L.; Petroski, G.F.; Madsen, R.W.; Mehr, D.R.; Conn, V.M.; Zwygart-Staffacher, M.; Maas, M. Stability and sensitivity of nursing home quality indicators. J. Gerontol. A Biol. Sci. Med. Sci. 2004, 59, 79–82. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- UBC GmbH. Modellhafte pilotierung von indikatoren in der stationären pflege (MoPIP) (SV14-9015). In Abschlussbericht zum Forschungsprojekt [Model-Based Piloting of Indicators in Inpatient care (MoPIP) (SV14-9015)]; Final Report on the Research Project; Universität Bremen: Bremen, Germany, 2017. [Google Scholar]
- Morris, J.N.; Moore, T.; Jones, R.; Mor, V.; Angelelli, J.; Berg, K.; Hale, C.; Morris, S.; Rennison, K.M.M.M. Validation of Long-Term and Post-Acute Care Quality Indicators; Final Report; Brown University: Rockville, MD, USA; Abt Associates Inc.: Cambridge, MA, USA, 2003. [Google Scholar]
- Bates-Jensen, B.M.; Simmons, S.F.; Alessi, J.F.S.C. Evaluating the accuracy of minimum data set bed-mobility ratings against independent performance assessments: Systematic error and directions for improvement. Gerontologist 2005, 45, 731–738. [Google Scholar] [CrossRef] [Green Version]
- Bours, G.J.; Halfens, R.J.; Haalboom, M.L.J.R. The development of a national registration form to measure the prevalence of pressure ulcers in The Netherlands. Ostomy Wound Manag. 1999, 45, 28–33. [Google Scholar]
- Mor, V. Improving the quality of long-term care with better information. Milbank Q. 2005, 83, 333–364. [Google Scholar] [CrossRef] [Green Version]
- Fung, V.; Schmittdiel, J.A.; Fireman, B.; Meer, A.; Thomas, S.; Smider, N.; Selby, J.H.J.V. Meaningful variation in performance: A systematic literature review. Med. Care 2010, 48, 140–148. [Google Scholar] [CrossRef]
- Berwick, D.M.; James, B.; Coye, M.J. Connections between quality measurement and improvement. Med. Care 2003, 41, I:30–I:38. [Google Scholar] [CrossRef] [Green Version]
- Shieh, G. Choosing the best index for the average score intraclass correlation coefficient. Behav. Res. Methods 2016, 48, 994–1003. [Google Scholar] [CrossRef] [Green Version]
- Berner Fachhochschule. Nachweis der Qualitäts- und Leistungsfähigkeit für die Stationären Leistungserbringer in der Langzeitpflege [Proof of Quality and Efficiency for Inpatient Service Providers in long-Term Care]; Berner Fachhochschule: Bern, Switzerland, 2010. [Google Scholar]
- Wingenfeld, K.; Kleina, T.; Franz, S.; Engel, D.; Mehlan, S.; Engel, H. Entwicklung und Erprobung von Instrumenten zur Beurteilung der Ergebnisqualität in der Stationären Altenhilfe [Development and Testing of Instruments for Assessing the Quality of Outcomes in Inpatient Care for the Elderly]; Bundesministerium für Gesundheit: Bielefeld, Germany; Köln, Germany, 2011.
- National Quality Forum. National Voluntary Consensus Standards for Nursing Homes: A Consensus Report; National Quality Forum: Washington, DC, USA, 2011. [Google Scholar]
- Leichsenring, K. Ergebnisorientierte Qualitätsindikatoren für Alters-und Pflegeheime [Outcome-oriented quality indicators for care and nursing homes]. Soziale Sicherheit CHSS 2012, 19, 26–31. [Google Scholar]
- Fitch, K.; Bernstein, S.J.; Aguilar, M.D.; Burnand, B.; LaCalle, J.R.; Lazaro, P.; van het Loo, M.; McDonnell, J.; Vader, J.; Kahan, J.P. The RAND/UCLA Appropriateness Method User’s Manual; RAND Corporation: Santa Monica, CA, USA, 2001. [Google Scholar]
- Zúñiga, F.; Schneider, P.; Gehrlach, C.T. Grolimund. Medizinische Qualitäts-Indikatoren. Schlussbericht zu Handen von Curaviva Schweiz [Medical Quality Indicators]; Final Report at the Attention of CURAVIVA Switzerland; University of Basel: Basel, Switzerland, 2014. [Google Scholar]
- Wingenfeld, K.; Stegbauer, C.; Willms, G.; Woitzik, C.V.R. Entwicklung der Instrumente und Verfahren für Qualitätsprüfungen nach §§ 114ff. SGB XI und die Qualitätsdarstellung nach § 115 Abs. 1a SGB XI in der Stationären Pflege. aQua, Bielefeld (Abschlussbericht: Darstellung der Konzeptionen für das neue Prüfverfahren und die Qualitätsdarstellung) [Development of Instruments and Procedures for Quality Inspections According to §§ 114 ff. SGB XI and Quality Presentation According to § 115 para. 1a SGB XI in Inpatient Nursing care. aQua, Bielefeld]; Final report: Presentation of the Concepts for the new Inspection Procedures and Quality Presentation; Universität Bielefeld: Bielefeld, Germany, 2018. [Google Scholar]
Theme | Definition | Numerator | Denominator | Items Measured | Exclusion Criteria |
---|---|---|---|---|---|
Polypharmacy | Percentage of residents who took 9 or more active ingredients in the last 7 days | All residents who had taken 9 or more active ingredients in the last 7 days | All long-term care residents | Number of active ingredients in the last 7 days | No exclusion criteria |
Self-reported pain | Percentage of residents with daily moderate or higher pain intensity or residents with nondaily very strong pain intensity in the last 7 days | All residents who reported the following pain in the last 7 days:
| All long-term care residents, excluding those who did not give a valid answer regarding frequency or intensity of self-reported pain | Frequency and intensity of self-reported pain in the last 7 days | No valid answer to questions on frequency OR intensity of self-reported pain |
Observed pain | Percentage of residents who showed daily moderate or higher pain intensity or residents who showed nondaily very strong pain intensity in the last 7 days | All residents where the following pain was observed in the last 7 days:
| All long-term care residents | Frequency and intensity of observed pain in the last 7 days | No exclusion criteria |
Physical restraint, trunk fixation or seating that prevents the resident from rising | Percentage of residents with daily fixation of the trunk or with seating that prevented the resident from rising in the last 7 days | All residents who had daily in the last 7 days:
| All long-term residents, excluding those who wanted or agreed to the use of this measure | Frequency of use in the last 7 days and context of the measure | Residents capable of judgment who either requested or agreed to the measure |
Physical restraint, bedrails | Percentage of residents with daily use of bedrails or other devices on all open sides of the bed that did not allow the resident to leave the bed independently in the last 7 days | Residents with daily application of bedrails or other devices on all open sides of the bed, which does not allow the resident to leave the bed independently | All long-term residents, excluding those who requested or agreed to the use of this measure | Frequency of use in the last 7 days and context of the measure | Residents capable of judgment who either requested or agreed to this measure |
Weight loss | Percentage of residents with weight loss of 5% or more in the last 30 days or of 10% or more in the last 180 days | Residents with a weight loss of 5% or more in the last 30 days or 10% or more in the last 180 days | All residents, excluding those with a life expectancy estimated by the staff as lower than 6 months or residents who were last assessed at admission to the nursing home | Weight loss of 5% or more in the last 30 days or of 10% or more in last 180 days | Residents with:
|
Theme | Risk Adjustment Variables | Prevalence Rate, Mean %, SD 1 | Missing, % (n) |
---|---|---|---|
Polypharmacy |
| 43.0 (12.9) | 0.0 (0) |
Self-reported pain |
| 19.7 (11.8) | 13.4 (1525) |
Observed pain |
| 14.9 (10.4) | 0.7 (81) |
Physical restraint, trunk fixation or seating that prevents the resident from rising |
| 3.4 (5.2) | 0.0 (0) |
Physical restraint, bedrails |
| 13.0 (11.3) | 1.6 (132) |
Weight loss |
| 7.9 (6.8) | 0.1 (2) |
Theme | ICC1 1 (95% CI 2) | ICC2 3 (95% CI) | Rankability (ρ) |
---|---|---|---|
Polypharmacy | 0.068 (0.047–0.086) | 0.917 (0.889–0.935) | 0.144 |
Self-reported pain | 0.134 (0.104–0.166) | 0.896 (0.852–0.917) | 0.471 |
Observed pain | 0.223 (0.131–0.325) | 0.941 (0.879–0.965) | 0.692 |
Physical restraint, trunk fixation or seating that prevents the resident from rising | 0.396 (0.297–0.474) | 0.990 (0.985–0.993) | 0.976 |
Physical restraint, bedrails | 0.371 (0.297–0.425) | 0.989 (0.984–0.991) | 0.865 |
Weight loss | 0.137 (0.085–0.180) | 0.899 (0.856–0.922) | 0.720 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Favez, L.; Zúñiga, F.; Sharma, N.; Blatter, C.; Simon, M. Assessing Nursing Homes Quality Indicators’ between-Provider Variability and Reliability: A Cross-Sectional Study Using ICCs and Rankability. Int. J. Environ. Res. Public Health 2020, 17, 9249. https://doi.org/10.3390/ijerph17249249
Favez L, Zúñiga F, Sharma N, Blatter C, Simon M. Assessing Nursing Homes Quality Indicators’ between-Provider Variability and Reliability: A Cross-Sectional Study Using ICCs and Rankability. International Journal of Environmental Research and Public Health. 2020; 17(24):9249. https://doi.org/10.3390/ijerph17249249
Chicago/Turabian StyleFavez, Lauriane, Franziska Zúñiga, Narayan Sharma, Catherine Blatter, and Michael Simon. 2020. "Assessing Nursing Homes Quality Indicators’ between-Provider Variability and Reliability: A Cross-Sectional Study Using ICCs and Rankability" International Journal of Environmental Research and Public Health 17, no. 24: 9249. https://doi.org/10.3390/ijerph17249249