The Utility and Feasibility of Routine Use of a Patient-Reported Outcome Measure in Paediatric Dentistry

Abstract

Within healthcare services, there is increasing emphasis to incorporate patient-reported outcome measures (PROMs), rather than relying solely on clinical outcomes. A 12-item caries-specific measure (CARIES-QC) has been developed and validated for children aged 5–16 years. To date, the routine use of PROMs in paediatric dentistry new patient clinics (NPC) has not been reported. The aim was to conduct a pilot study to assess the feasibility, utility and validity of routine use of a PROM in paediatric dentistry NPC in a UK teaching hospital. Children attending NPC over a four-week period were asked to complete CARIES-QC with an additional free-text box. Interviews were held with members of staff to assess the feasibility of using a PROM routinely. CARIES-QC was completed by 99 children. Almost half of the participants had caries (n = 49, 49.5%). CARIES-QC demonstrated good internal consistency (Cronbach’s alpha = 0.9) and reliability with the global question (r = 0.75, p = 0.01). Clinical staff valued the information provided by the PROM. An electronic delivery method would be beneficial to both clinical and administrative staff. CARIES-QC was able to capture impacts for children with a range of oral conditions. Its use aided treatment planning and future studies should investigate the use of an electronic delivery system to reduce the administrative burden.

1. Introduction

Patient-reported outcomes (PROs) are “any report of the status of a patient’s health condition that comes directly from the patient without interpretation of the patient’s response by a clinician or anyone else” [1]. Patient-reported outcome measures (PROMs) are the measures used to gain these PROs, usually in the form of a self-completed questionnaire. Different aspects of a patient’s condition can be assessed, for example: symptoms from the disease or its treatment; functional outcomes or multidimensional constructs such as (oral) health-related quality of life or health utility [2]. PROMs used in clinical settings can aid clinicians’ understanding of how treatment and diseases impact upon those who experience them [3].

Communication between health care professionals and their patients can be greatly improved by clinicians’ awareness of patient-centred information. Clinicians’ interest in obtaining patients’ perspective on their health and wellbeing, through the use of PROMs, may make the patients feel that the medical team are more involved and invested in their individual care [3]. This of course relies on the ability of the clinical team to use the data provided and for it to be presented in a way that is useful to them.

PROMs can be either general in nature or disease-specific and are increasingly being used as organisational performance measures by clinicians and healthcare administrators [4]. A number of PROMs exist purely to investigate the impact of oral health conditions, and their treatment from the patient’s perspective [5]. Using PROMs also gives the patient the opportunity to write down answers that they may not feel comfortable discussing face to face with a clinician. The use of a PROM in rheumatology was found to increase patient satisfaction for those new to the service [5]. Specifically, in paediatric dentistry, it may help children express impacts using their own language, which otherwise may have been missed. This may be particularly pertinent in new patient consultations where children may feel shy in a new environment and gives parents the opportunity to discuss their child’s condition with them prior to the appointment.

The Caries Impacts and Experiences Questionnaire for Children (CARIES-QC) was developed with children in the UK, to measure the impacts of caries using language which children themselves use [6]. It has been validated for use in children with caries aged 5–16 years in the UK, New Zealand, Australia and the Netherlands [7,8,9,10,11]. The impacts it includes may be applicable to children with other oral conditions, however it has not been validated for use in this way.

Currently in the Department of Paediatric Dentistry in Sheffield, new patients are seen by clinicians and a history taken using a structured proforma. Input from children into their history varies depending on their age and how comfortable they feel discussing their condition with an unfamiliar adult. As PROMs can improve communication with children and their families, the aim of this pilot study was to evaluate the feasibility and usefulness of the routine use of CARIES-QC in a paediatric dentistry clinical setting. The objectives were therefore as follows: (1) examine the results from the CARIES-QC questionnaire and discuss their utility in relation to routine practice (2) to provide an initial evaluation of the reliability and validity of CARIES-QC in this setting, and (3) identify the benefits and potential barriers to the introduction of a such PROM onto paediatric dentistry clinics in the future.

2. Materials and Methods

2.1. Pilot of a PROM on Clinics

A convenience sample of children attending new patient assessment clinics on the Paediatric Department, Charles Clifford Dental Hospital (CCDH), Sheffield, between 7th July and 5th August 2019, were included in this pilot study. New patient appointments were chosen as it was felt that this would be where the information obtained would be most useful to clinicians and these patients could readily be identified. The project was registered with Sheffield Teaching Hospital Foundation Trust’s Clinical Effectiveness Unit (Project registration: Ref 9336).

Children were asked to complete the Caries Impacts and Experiences Questionnaire for Children (CARIES-QC) on arrival to the department. Parents were asked to help the child to complete CARIES-QC if necessary. Where parents or children did not complete the questionnaire, any reason given was noted by reception staff on the form. Questionnaires were given to clinicians and at the end of the appointment the CARIES-QC questionnaire was filed in the patient’s notes. Notes were retrieved by a member of the study team (R.K.) and data extracted from the questionnaires and patient record as follows:

(1)

Demographic details, including date of birth and postcode (used to calculate deprivation quintile);

(2)

CARIES-QC: responses to individual questions and total overall score;

(3)

Comments recorded in the free text box on the questionnaire.

2.2. CARIES-QC

CARIES-QC is a 12-item measure that seeks children’s assessment of the severity of their caries-related impacts and has previously been validated for use in 5–16-year-olds [7]. CARIES-QC contains 12 items relating to potential impacts of dental caries and one global question, which asks “How much of a problem are your teeth for you?”. The items are scored on a 3-point Likert scale (“Not at all”; “A bit” and “A lot”) and scored 0–2 (maximum score 24) with increasing score indicating increasing severity of the impact. The items are shown in the Supplementary Materials. This instrument was developed with children and has been shown to have good face, content and construct validity, responsiveness, reliability and acceptability [7]. Although this measure was designed for children with caries, it was thought that the items may be applicable to children with other oral conditions. An additional free text box was added to the measure to allow children to add any other impacts they may have experienced which were not included in CARIES-QC.

2.3. Missing Data

Where greater than two questions were unanswered in CARIES-QC, the participant was eliminated from further analysis. Where fewer than two missing values were present, the individual mean for that participant was used (i.e., values were filled with the computed mean of the participant’s completed items) with the exception of the global question where no substitution was made [12].

2.4. Staff Perspectives on the Pilot Introduction of a PROM

A qualitative approach, using semi-structured interviews, was used to obtain feedback from clinic staff on the use of a routine PROM on clinic. A purposeful sample of staff in different roles in the department were invited to participate. Semi-structured interviews were held with these members of the paediatric dentistry team (including consultants, nurses, dentists and reception staff) three weeks after data collection had ceased. All interviews were conducted by a member of the research team (R.K.). Questions were asked specifically about benefits, problems/challenges and suggested ways of improving the PROM or its administration. Interviews were recorded and transcribed verbatim.

2.5. Data Analysis

Quantitative data were entered into IBM SPSS Statistics (v25 IBM, Armonk, NY, USA) for simple descriptive analysis. The fit and function of the CARIES-QC items were examined using an item response theory Rasch model. In addition, the items were assessed to ensure they were free from differential item functioning (DIF). That is, that they function in the same way between subgroups (age, gender, ethnicity and caries status). The methods used were the unrestricted or partial credit model suggested by Tennant and Connaghan [13]. The Rasch analysis was undertaken using RUMM2030 (RUMM Laboratory PtyLtd, Perth, WA, Australia). Item fit to the model, if the data fit the Rasch model, each item and person fit residual should be within the range ±2.5 and the mean item and person fit statistics should be close to zero with a standard deviation of one [14]. Finally, the individual items and summary chi-square interaction statistics should be non-significant (p > 0.05), although these are subject to Bonferroni adjustment based on the number of items. Strict unidimensionality was then examined using an independent t-test on two subsets of items identified using principal component analysis of the item residuals. Missing data were not imputed for the Rasch analysis.

Qualitative data from the free text section of the questionnaires and the staff interviews were analysed using thematic analysis. The data from the free text information box on the questionnaires was used to identify any additional impacts which were not included in the CARIES-QC.

The interview transcripts were analysed independently by two researchers (R.K. and F.G.). Initially, notes were made on the general themes and these were then discussed to further refine the themes. Statements made by participants were highlighted and categorised into different themes. Analysis of data from the staff interviews focused on the perceived benefits and challenges for staff of using CARIES-QC routinely.

3. Results

3.1. Demographics

A total of 99 participants completed CARIES-QC. A further 26 questionnaires were filed in the notes but not completed. The most common reason for non-completion was due to language barriers where the family did not speak English (n = 10). Four parents declined to complete the questionnaire, two children were unable to participate as they had a learning difficulty, five children were deemed too young to participate and for a further five no reason was given. The mean age of those who completed the measure was 9.1 years (range 3–15 years). There were 52 (52.5%) females and the majority identified as White British (n = 63; 63.6%). The majority of participants were from the most deprived quintiles (quintiles 4 and 5; n = 55; 55.5%). Almost half of the participants had been referred for caries management (n = 49; 49.5%). Further demographic details and diagnoses are shown in

Table 1. Demographics and CARIES-QC scores.

	Number (%)	Mean (SD) CARIES-QC Score	Range CARIES-QC Score
Overall	99	6.39 (6.1)	0–24
Gender:
Male	47 (47.5%)	6.04 (5.91)	0–24
Female	52 (52.5%)	6.71 (6.25)	0–23
Ethnicity:
White British	63 (63.6%)	5.24 (4.88)	0–19
Other ethnicity	36 (36.4%)	8.42 (7.37) *	1–24
Deprivation:
Quintile 1	12 (12.1%)	5.25 (5.60)	0–18
Quintile 2	13 (13.1%)	4.08 (4.63)	0–16
Quintile 3	14 (14.1%)	6.36 (4.81)	1–18
Quintile 4	12 (12.1%)	4.25 (3.79)	0–12
Quintile 5	43 (43.4%)	8.00 (7.07)	0–24
Diagnosis:
Caries	49 (49.5%)	8.63 (6.68)	0–24
Enamel defect	24 (24.2%)	5.08 (4.40)	0–19
Trauma	13 (13.1%)	6.92 (6.56)	0–19
Tooth number/position	8 (8.1%)	3.38 (2.13)	1–7
Other	5 (5.1%)	3.60 (2.30)	1–6

Data on ethnicity was missing for 5 participants. Quintile 1 is least deprived and Quintile 5 most deprived. Other diagnoses included soft tissue lesions and orthodontic extractions. * statistically significant (p = 0.03; CI = −5.9 to −0.4).

.

3.2. CARIES-QC

Nineteen participants had omitted questions. Two participants had omitted two questions with the others missing one. Twelve omitted the global question. The most commonly missing item (n = 3) was related to schoolwork. This item was omitted by three participants who were aged 3 years, which we may assume was due to its lack of relevance in this age group. The only other question which had been omitted more than once (n = 2) was the item related to feeling cross. No participants weree omitted from analysis as there were no more than two missing answers for each individual.

Scores ranged from 0–24 with a mean of 7.26 (SD = 5.9). The mean score for those with caries was 8.88 (range = 0–24) compared with 5.68 (range = 0–19) for those without caries (p = 0.007; CI = −5.49 to −0.9). Only 6 children (6.1%) reported no impacts. Scores according to demographics and diagnosis are shown in Table 1. The most frequently reported impact for all participants was “Getting food stuck in teeth” (n = 80; 80.8%), followed by “Annoying” (n = 59; 59.6%) and “Hurting” (n = 57; 57.6%). For those with caries these impacts were also reported most frequently (“Getting food stuck in teeth” = 85.7%, n = 42; “Annoying” = 65.3%, n = 32; “Hurting” = 63.3%, n = 31) with the addition of item related to “Crying” (n = 32; 65%). Only one child with caries reported no impact on CARIES-QC.

In answer to the global question 52% (n = 51) responded that their teeth were “a bit” or “a lot” of a problem. For those with caries this was 66% (n = 31) compared to 51% (n = 20) of those with other diagnoses.

Additional impacts suggested by participants in the free text box were related to four themes:

(1)

Related to bleeding: “Tastes like blood in my mouth”, “It bleeds when I brush”

(2)

Related to pain: “It hurts to drink water”, “My gum hurts when I brush”, “My tooth is sharp”

(3)

Wobbly teeth: “They are wobbly”

(4)

Appearance: “I have a gap”, “My teeth are slanty”, “White spots on my teeth”, “My tooth is going black”, “My teeth don’t grow straight and it’s annoying” “They look bad”.

3.3. Reliability and Validity of CARIES-QC

Internal consistency of CARIES-QC in this population was excellent (alpha = 0.9). The correlation between the CARIES-QC total score and the global score was strong (r = 0.750, p < 0.01). Analysis using the Rasch model showed that CARIES-QC demonstrated unidimensionality similar to that found in the original validation study [7]. There was some DIF seen in relation to two questions. Older children (aged 11–16 years) appeared more likely to say their teeth made them feel cross than younger children (aged 3–7 years or 8–11 years) with the same level of impact, although this should be interpreted with caution as there were only 14 children in the 11–16 year group. Not surprisingly children with caries were more likely to have been kept awake by their teeth than children with other diagnoses. There was slightly more misfit to the Rasch model than in the original validation study which would be expected with a more diverse population. Comparison between the original validation study is shown in

Table 2. Comparison between this study and original validation study.

Analysis Name	Item Residual		Person Residual		Chi-Square		Reliability	Unidimensionality
	Mean	SD	Mean	SD	Value (df)	p		Percentage of Tests > 5%	95% CI
Original study	−0.37	0.86	−0.21	0.89	38 (24)	0.04	0.85	4.15%	2.07–13.93
Present study	−0.37	1.39	−0.21	0.81	40 (24)	0.02	0.85	6.32%	1.84–10.16
Ideal	0	1	0	1		>0.004 *	>0.7	<5%	LCI < 5

* Bonferroni adjusted for 12 items; SD = standard deviation; df = degrees of freedom; CI = confidence interval; LCI = Lower confidence interval.

.

3.4. Qualitative Findings

In total, seven staff members who had been involved in the use of CARIES-QC on the clinics gave their opinions on the introduction of a routine PROM on the clinic. Staff participants’ comments were generally positive and that they had found the addition of the PROM a useful adjunct to the new patient consultation. Suggestions were made as to how the delivery of the PROM could be improved to reduce staff and patient burden. The themes which emerged are discussed here.

3.4.1. Benefits of the PROM

Well received by Staff and Families

Participants discussed that children and families had found the measure easy to complete and that as they were completed in the waiting room there were no delays to starting the consultation.

“I found the forms quite easy to encourage the parent or child to engage, they were always happy to do so.”

(Consultant 1)

“… I got some feedback from patients where they found it quite easy to understand, because it was just simple words and tick yes or no. It was very understandable, it was clear. That’s it really.”

(Receptionist)

“They were easy because the patients were doing them in the waiting room”

(Nurse 1)

“I didn’t have any complaints about it taking up extra time, because we often keep them waiting because we are so busy, so they don’t … I don’t think they feel like the time is wasted.”

(Dentist 1)

Use as a Communication Tool

Staff participants commented on how having the information about the impacts the child was experiencing helped with communication both between parent and child and between clinician and the family.

“Generally, parents and children comment on how they are feeling that day. So, having a questionnaire gives them an opportunity to think a bit more. Then things come to the fore.”

(Consultant 1)

“Because the kids do them with the parents in the waiting room, it means the parents actually ask the children about how they feel about their teeth.”

(Dentist 1)

Improving Patient Care

It was recognised that having information from the patient’s perspective may help to improve patient care and that the evidence generated by such a measure could support safeguarding activities:

“I think it was really helpful. I think using it more routinely would help towards patient care. At least you have that information from the patient’s perspective.”

(Consultant 2)

“It would be really useful where we have kids that we are really concerned about. It could be good evidence. There aren’t many children like that, but we had a safeguarding meeting the other day and they were talking about evidencing what impact things are having on children, and it’s actually a really good document to have for if a child is not brought to subsequent appointments… another thing we can give to social workers or the paediatric liaison nurse to say this child is having problems’.”

(Dentist 1)

Aid with Treatment Planning

Some clinicians discussed that the information from CARIES-QC had had an impact on their treatment planning as they were more aware of the impacts that the children were experiencing.

“So I do think it’s important. Because it helps you with your treatment planning. Because if it’s something that’s bothering them… maybe you do that first. Or if you are thinking about how you will do the treatment, you have to think well... can you acclimatise them?”

(Dentist 1)

“I would always look at them. It was quite helpful to see where the starting point for the patient was… Bearing in mind the referrals we get don’t contain… they… vary in terms of the amount of information they contain. You’ve never met these patients before. So anything extra they can tell you, or what they might be like, or any anxiety or apprehension is quite helpful.”

(Consultant 2)

3.4.2. Concerns about the Questionnaires

Staff participants discussed some concerns they had relating to the PROM and how it could be implemented routinely. These related to the length of CARIES-QC, workload, and its suitability for some patients.

Length

“for me as a parent, I wouldn’t want anything more than two sides.”

(Consultant 1)

Adding to Workload

“It adds to the workload to read it.”

(Consultant 1)

“The thing is, because we give three forms out to patients, parents are like … that’s a lot to fill out…”

(Receptionist)

Suitability for Some Patients

“We do have a lot of patients who have interpreters. I don’t know if it’s ideal for that.”

(Nurse 1)

“I suppose the only thing is the parents who come without translators. But if they have chosen to come without a translator, I would assume they could understand the forms. And actually, because the kids are doing it themselves, the children’s English is actually usually pretty good.”

(Dentist 1)

3.4.3. Changes to Make Implementation Possible

Staff participants made a number of suggestions about how routine implementation could be achieved and what changes might be beneficial.

Changes suggested included sending the measure out to patients to reduce the burden on families when they came for an appointment.

Reducing Workload for Families and Staff

“Could you send it out through the post? Could they bring it with them? If that’s an option, I’d send it out in the post. You’ll get some who forget, but we could have some spare ones in our drawer. That would be doable.”

(Receptionist)

Changes to CARIES-QC

Alterations to the measure included ensuring it was available in different languages to reduce inequalities, shortening the measure and having different ways for children to respond.

[In relation to patients who do not speak English] “That’s my only apprehension, that we won’t be rolling it out to all patients.”

(Consultant 2)

“Is it possible to condense it?”

(Dentist 2)

“Maybe you could have a free bit, for maybe… draw a picture of how your teeth make you feel or something.”

(Consultant 1)

4. Discussion

This pilot study aimed to evaluate the usefulness and feasibility of routine use of a PROM in paediatric dentistry. The PROM (CARIES-QC) performed well and staff members expressed generally positive feelings regarding its introduction to clinics. The study objectives were addressed as follows:

• Examine the results from the CARIES-QC questionnaire and discuss its usefulness in relation to routine administration.

CARIES-QC was designed to be used for evaluation of the impacts of dental caries [7]. It was chosen for use in this study as it was developed with children and uses language which they are familiar with. It was also presumed that many of the items would have relevance for children with other oral conditions. CARIES-QC performed well and was found to have good reliability and validity in this population. Its performance in this population was similar to that found in the original validation study which was carried out in the same location. CARIES-QC currently has 12 items and a global question and, in this study, an extra question was added to ensure that impacts that were important to children were not overlooked. Some children did add other impacts that they were experiencing, and it would appear useful to allow children to respond in this way to improve communication about what is important to them. The PROM was able to elicit information from children which was not reported through routine questioning. For example, children often do not describe chronic pain such as that experienced in dental caries as “pain” but may instead say that their teeth annoy them [11,14]. The PROM allows them to consider what impacts they have experienced in language which is more familiar to them.

Only six children reported no impacts. Of concern is that over half (57.6%) the children reported experiencing pain from their teeth and 66% of those with caries reporting that their teeth were “a bit” or “a lot of a problem for them”. This highlights the impacts experienced by children while they are waiting for treatment, in common with other studies [15,16]. The results also indicated that children from Black, Asian and Minority Ethnic (BAME) backgrounds were more likely to have higher CARIES-QC scores than their White British counterparts. This has been found in previous studies investigating oral health-related quality of life in children with caries and is likely to be related to other socioeconomic factors affecting these populations [17,18,19]. The fact that children were in pain is particularly relevant in the present circumstances of the COVID-19 pandemic where many children have had their treatment delayed due to reduced capacity. The ability to send a PROM like this to patients would have been useful during this period to aid with triaging those most in need. It would have allowed us to fully assess the impacts children were experiencing as often children do not discuss the everyday impacts they are experiencing with their parents, tending to report more acute episodes [14].

2

To examine the reliability and validity of CARIES-QC in this setting.

Testing revealed that CARIES-QC demonstrated excellent internal consistency and good validity, which indicated this measure could be useful for routine introduction onto a Paediatric clinic. While CARIES-QC was originally designed as a caries-specific measure, these findings suggest it could also be used in more diverse situations, for children with a range of oral conditions.

3

Identify the benefits and potential barriers to the introduction of a PROM onto the clinics in the future.

It is recognised that for routine administration that shorter measures are desirable and this was discussed by clinical staff members in this study. Using the measure in a larger population would aid the development of a short-form or computer-adaptive Test (CAT). CAT allows the PROM to be administered electronically and ensures that questions are tailored to the individual rather than asking questions which the patient may find irrelevant. A study involving patients with cancer found that patients prefer this type of questionnaire as the questions are more appropriate to their individual concern [20].

Clinicians discussed the impact that the information from the PROM had on their treatment planning and communication. While there is evidence that the use of PROMs can enhance communication there is little evidence that it changes care management or treatment outcomes [21,22]. In the present study, clinicians valued the additional information the PROM provided and used it to help decide which treatment should be completed first. There is some evidence that clinicians are more likely to use the information from the PROM if they value it. As this PROM has been used in research in this department over a number of years, it may be that the clinicians in this study were accustomed to the measure and therefore were aware that it may contain additional information to aid their treatment planning.

However, studies have shown that clinicians may find it hard to engage with the information provided by PROMs as it adds to the administrative burden for already overwhelmed clinicians [23,24]. Indeed, a systematic review of qualitative studies reporting the experience of professionals using PROMs identified a number of barriers including the workload involved and that this may be reduced by ensuring the PROM is incorporated naturally into the consultation [25]. They also identified that PROMs data was less likely to be used by clinicians where the PROM is not user-friendly. In the present study, clinical staff reported that they felt that the way the PROM was delivered did not interfere with patient care and that it appeared to be user-friendly. However, administrative staff felt that an electronic delivery system where patients could complete the PROM prior to the appointment would reduce the administrative burden. An electronic delivery system for PROMs is already in use in other areas of the NHS Foundation Trust where this study was carried out and so adoption of this system may aid future delivery of the PROM [26].

It is worth considering the limitations of this work. As this was a pilot study to evaluate the feasibility of implementing a PROM for routine use, it was only performed over a limited period in one setting and therefore the findings may not be applicable to other settings. Unfortunately, the PROM is not available in a range of languages as yet, this limited the number of children who were able to complete it and was the most common reason for it not to be completed. A number of participants had omitted the global question, it is not clear why this had happened as it has not been a common feature when the PROM has been used previously. Its position on the page may have been a factor and future studies should discuss the layout with participants. Parents were asked to complete the PROM with their child and for the child’s views to be recorded. We did not record who completed the questions but it was felt by reception staff that the vast majority did comply with the instructions and that they could see families discussing their answers. In any future work it would be beneficial to record who was completing the PROM so that an accurate picture of how the PROMs are used by families can be explored. This study did not incorporate the views of children and their families, however as the measure was developed in this setting with children, we know that children find it easy to complete and that they enjoy being involved. Ideally future studies should seek to incorporate the views of parents and children using qualitative analysis. To reduce non-responses, future studies should ensure that the measure is available in a range of languages and that if an electronic delivery method is used, this should be evaluated not only with staff but with children and their families.

5. Conclusions

In summary, this study found that clinicians valued the information provided by the PROM and felt that it aided their treatment planning. The PROM used (CARIES-QC) demonstrated good reliability and validity in children with a range of oral conditions, and as such could be a useful measure for routine introduction in this setting. Although CARIES-QC appeared to be an appropriate PROM to be used in this setting although further investigation is required to ensure it is valid for use in younger children and a short form of CARIES-QC would be desirable. Future implementation of a routine PROM in this setting should consider the use of an electronic PROM to reduce administrative burden and to enable it to be targeted at children of an appropriate age.