Dynamic Assessment to Assess Mathematical Problem Solving of Students with Disabilities

Abstract

The importance of mathematical problem solving (MPS) has been widely recognized. While there has been significant progress in developing and studying interventions to support teaching and learning MPS for students with disabilities, the research on how to accurately and effectively assess the impact of those interventions has lagged, leaving a gap in understanding whether interventions are truly achieving their intended outcomes. The purpose of this mixed-method study was to explore how a dynamic assessment (DA) approach can be used in the context of an evidence-based MPS intervention, Enhanced Anchored Instruction, as an alternative means of assessing the MPS of students with disabilities. Our findings suggest that DA is an adequate assessment tool and can provide additional information for teachers to better understand the MPS strengths and challenges of students with disabilities such as MPS ownership transition. Study limitations, considerations for future research, and implications for practice are discussed, emphasizing the importance of rigorous evaluation of the DA approach to improve teaching and learning MPS for students with disabilities.

1. Introduction

Mathematical problem solving (MPS) is a complex mathematical task for which “the solution method is not known in advance” (National Council of Teachers of Mathematics, 2000, p. 52). The importance of MPS has been widely recognized, and MPS instruction and assessment have been a longstanding concern among educators, researchers, and policymakers (e.g., Schumacher et al., 2019; Woodward et al., 2012). Since Pólya’s (1945) introduction of the heuristic principles for problem solving, many efforts have been made to support teaching and learning MPS over the past 80 years. In special education, researchers have developed evidence-based MPS interventions to support teachers in more effectively teaching MPS and helping students with disabilities (SWDs) improve their MPS understanding and skills (A. Jitendra et al., 2002; J. E. Kong et al., 2021; Myers et al., 2023). The special education research on MPS has promoted the exploration of MPS assessment to determine the MPS understanding and skills of SWDs using different assessment approaches and strategies (e.g., B. A. Bottge et al., 2021; Stevens et al., 2024). However, “[r]esearch in assessment has not kept pace with research on intervention” (Talbott et al., 2023, p. 468). To address the gap, this paper explores the adequacy and the nature of an alternative assessment in the context of an evidence-based MPS intervention for SWDs.

1.1. Enhanced Anchored Instruction

Over the past three decades, Dr. Brian Bottge and his research team have developed and tested the efficacy of an MPS instructional method called Enhanced Anchored Instruction (EAI) specifically designed to improve the MPS of SWDs (B. A. Bottge et al., 2007, 2014, 2015; Choo, 2024). Based on a pedagogical approach labeled anchored instruction (Cognition and Technology Group at Vanderbilt, 1990, 1997), EAI is a multimedia- or video-based, contextualized instruction that enhances the use of anchored instruction by providing students additional opportunities to practice MPS in real-world situations. With this method, students typically work in small groups over several class periods to identify a solution path to the overarching mathematical problem posed in a video story. To enhance anchored instruction, Bottge and his team added hands-on applications (e.g., building hovercrafts) and explicit instruction of difficult mathematical concepts (e.g., computing fractions) to help remediate students’ basic MPS skills and prevent cognitive overload, which is often an issue in any multimedia-based instruction (e.g., Meyer’s cognitive load theory, Kennedy et al., 2014). The combination of explicit instruction, multimedia-based instruction, and project-based activities with instructional support for SWDs has been beneficial to students who struggle with comprehending the text of typical school-based mathematical word problems. In the EAI lessons, students can practice their MPS understanding and skills in real-world situations, thereby increasing the chances that they will transfer skills across contexts.

1.2. EAI Assessment

In the efficacy studies in both resource rooms and inclusive classrooms, B. A. Bottge et al. (2014, 2015, 2018) included two forms of MPS assessments (formative and summative) that were developed and validated from the previous development study. Frequent and planned formative assessments monitor the performance of students as they progress through the curriculum (Brookhart & Lazarus, 2017). Summative assessments, which are typically administered at the end of units, measure larger chunks of learned content. Both types are in paper–pencil or written format with open-ended questions matched to the mathematical concepts just taught in the EAI lessons. Figures, tables, and other graphics accompany each item to help students understand the content.

Based on the validated MPS assessments, B. A. Bottge et al. (2021) developed a technology-assisted test via tablet computers to explore the potential of assisting the MPS process of the SWDs. Students had tablet computers read aloud to them to help eliminate any reading barriers. Students could also select an option that showed animations of the mathematical problem, which they could replay as needed. For each test item, students needed to show their work and enter their answers in the item workspaces. If they wanted to start over or erase their work, they could click the CLEAR button. After students finished, their work and answers were stored on a server.

Through the iterative development process of the new technology-assisted test, B. A. Bottge et al. (2021) found that most students liked taking the test on the tablet computers. However, some students preferred the paper–pencil format, because it was easier to erase mistakes on paper; the tablet computer workspace was too small; or the animations were geared toward younger students. Other feedback included difficulties with using a stylus, complexities of navigating through screens, and remembering to save their work. B. A. Bottge et al. (2021) concluded that the best test format was to use a hybrid version with the added option of showing their work and answering the items on a paper copy of the test. Despite the efforts in constructing our measures, the results of a randomized study showed no difference between test formats (i.e., hybrid vs paper–pencil) in terms of performance.

The lack of improvement in student performance on the technology-assisted test was puzzling to our research team, because the 600-plus classroom observation reports and teacher interviews indicated that even the lowest-achieving students had demonstrated their MPS understanding throughout the study. For example, some of the low-achieving students took on a leadership role during group work by measuring and constructing hovercrafts (e.g., Measurement and Data). After discussing the findings with participating teachers, we posed the following question: Is the lack of improvement shown on the technology-assisted tests due to characteristics of the EAI tests or to students’ difficulties to learn the MPS concepts and skills in the EAI unit? Although answering this question is beyond the scope of this paper, the EAI MPS tests (whether on paper or tablet computers) might have underestimated what SWDs had learned because of the following difficulties that students might have encountered in our tests. First, SWDs often experience difficulties with the conventional test format and construction. For example, many SWDs lack the requisite decoding and/or comprehension skills for understanding what the mathematical “word” problems were asking of them (Spencer et al., 2014). Second, SWDs often have weak cognitive function, causing working memory difficulties in retrieving verbal information and mental images across storage systems (A. Baddeley, 2000, 2012).

The focus of this paper is on addressing these difficulties the SWDs might have encountered in the conventional assessment format in the context of EAI. Based on recommendations from professional organizations designed to ensure fair and valid assessment when working with special populations (e.g., American Educational Research Association et al., 2014; Council for Exceptional Children, 2014; National Association of School Psychologists, 2010) as well as suggestions made from systematic reviews of alternative assessment approaches and strategies (e.g., Dixon et al., 2023; Ukobizaba et al., 2021; Verdugo et al., 2020), evidence-based assessment in special education research (e.g., Talbott et al., 2023), and equity-related concerns of educational accommodations for students with disabilities (e.g., Lovett, 2021), we explore an alternative assessment approach that can address the difficulties SWDs often experience with conventional tests.

1.3. Dynamic Assessment

One of the predominant instructional approaches in the field of special education is providing positive and constructive feedback to guide students’ learning and behavior (e.g., High-Leverage Practices 8 and 22, Aceves & Kennedy, 2024). However, providing feedback during assessment has been limited to classroom-based formative assessment due to its potential risk (i.e., validity threat) of resulting in inaccurate or biased reflection of students’ actual knowledge and ability (Crooks et al., 1996). Despite an ongoing debate of providing feedback during assessment (e.g., Gu, 2021), the meaningful feedback students receive during assessment can improve learning effectiveness as well as increase motivation and self-esteem (Marriott et al., 2009).

One formative assessment approach that provides feedback during assessment is dynamic assessment (DA). DA is an umbrella term to describe an alternative assessment that includes common elements of instruction and feedback that are built into the testing process and are differentiated based on an individual’s performance (Elliott, 2003; C. Lidz & Elliott, 2000; Stringer, 2018). Grounded in Vygotsky et al.’s (1978) zone of proximal development theory, which conceptualizes the gap between what students can do without any assistance and what students can do with teacher guidance, the purpose of DA is to define that gap and determine how to maximize their learning potential with scaffolding (Campione & Brown, 1987; D. Fuchs et al., 2007; Spector, 1992). To identify the gap, researchers (e.g., L. S. Fuchs et al., 2008, 2011) often use an index technique to quantify responsiveness and compare the unassisted phase with the assisted phase (e.g., scoring an unassisted test following an assisted test). Although testing items and tasks in DA are similar to those of standardized tests, DA emphasizes the interaction between those who give a test and those who take the test (Le et al., 2023). DA researchers also use different styles of interaction. While some DAs use a fixed series of prompts (i.e., standardized), other DAs address a student’s specific learning difficulties as revealed by the student’s responses (i.e., individualized). Additionally, the nature of the skills assessed in DAs is different in DA research. Early DA research focused on domain general skills (e.g., cognitive ability), but more recent DA research includes more academic skills (e.g., reading, math).

Limited studies have utilized DA to assess the MPS of SWDs. Among early studies with DA in special education, for example, A. K. Jitendra and Kame’enui (1993) investigated the extent of using DA in the context of schema-based instruction for elementary students with and without disabilities to solve mathematical word problems. The overall results indicated that both the students with and without disabilities increased and maintained the improvement in MPS after receiving schema-based instruction. The specific findings related to DA suggest that (a) DA is a valuable tool in assessing MPS; and (b) SWDs require more prompts than those without disabilities. A. K. Jitendra and Kame’enui (1993) also suggested that future research would consider “specific, accurate feedback” (A. K. Jitendra & Kame’enui, 1993, p. 321), encouraging continuous research efforts with DA. L. S. Fuchs et al. (2008, 2011) utilized DA to predict how elementary SWDs learn novel mathematical content based on MPS. More specifically, their findings suggest that students’ MPS assessed by DA can be a strong predictor of their future success in algebra. Additional findings, aligned with the findings of A. K. Jitendra and Kame’enui (1993), indicated that SWDs required a greater amount of scaffolding to reach mastery level on mathematical skills.

According to a recent systematic review (Le et al., 2023), only four additional studies used DA for assessing MPS limited to mathematical word problems among elementary SWDs (J. E. Kong & Orosco, 2016; M. J. Orosco, 2014; M. J. Orosco et al., 2011; Seethaler et al., 2012). Although its potential and value have been recognized in educational settings (Elliott, 2003; Stringer, 2018), limited guidance or resources (other than a protocol to identify students at risk for reading disabilities by D. Fuchs et al., 2007) are available for special education teachers to utilize the DA approach for improving the MPS skills of SWDs to date. To address the paucity of DA for assessing MPS and the issue with limited teacher resources, we aim to provide conceptual and empirical directions for the use of DA in the MPS domain.

1.4. Study Purpose and Research Questions

The overarching goal of the current study was to address the research gap by exploring how the DA approach can be used in the context of EAI, one of the evidence-based MPS interventions for SWDs. To explore the adequacy of DA in EAI as a means of assessing MPS of SWDs, we followed D. Fuchs et al.’s (2007) and L. S. Fuchs et al.’s (2011) index technique to quantify the responsiveness of and compare the written assessment scores (i.e., unassisted phase) of the EAI MPS test to the DA scores (i.e., assisted phase) of the same test items. To understand the nature of DA in EAI, we conducted a qualitative analysis to explore the MPS process of SWDs as well as the interactions between the examiner and examinees.

Specific research questions (RQs) that guided our study were as follows: (RQ 1) Is the DA approach as adequate as the conventional approach to assess the MPS of SWDs before and after implementing EAI? (RQ 2) To what extent does the DA in EAI help SWDs express their MPS process? (RQ 3) What is the nature of the interaction between the DA examiner and examinees while assessing the MPS skills of SWDs? We anticipated that the answers to these questions could provide insights into the assessment approaches and strategies that might complement the conventional paper–pencil written assessment method for SWDs in the classroom setting and draw implications for future instruction and assessment practices.

2. Materials and Methods

2.1. Research Design

We sought to explore the possibility and value of using the DA approach in the context of an evidence-based EAI and to capture the unique process of MPS among SWDs that might not be revealed in conventional written tests. We documented our approach through a unique situation in a federally funded study when COVID-19 forced us to change the research plan. To answer our research questions, we conducted an exploratory study with both qualitative and quantitative components based on the mixed-method case study (J. W. Creswell & Clark, 2017; Love et al., 2022) with the explanatory sequential design (J. W. Creswell et al., 2003; Ivankova et al., 2006). We collected and analyzed the quantitative data first, and the qualitative data were collected and analyzed second in the sequence. The quantitative and qualitative data carried equal weight, but each analysis had a different focus. The focus of quantitative analysis was on comparing how students responded equally or differently between the original, written EAI MPS test and the DA version of the EAI MPS test. The main unit of qualitative analysis was the verbal interaction between the examiner and students in the DA, with individual students as the subunit of analysis. Since the DA data set included both quantitative (numeric scores) and qualitative (verbal interaction) data, however, we used the embedding integration approach (Fetters et al., 2013) to complement each other throughout the study.

2.2. Setting and Context

The goal of our original study was to test the feasibility of the DA approach using the validated EAI MPS test items by capturing what SWDs had learned from the EAI units but had not shown on the written test. Participating classrooms were recruited from a large public school district located in a suburb area of central Kentucky in the United States. We modified the study to account for the unusual circumstances caused by the COVID-19 pandemic in 2021, which forced us to conduct much of the study online with the participants.

As shown in

Table 1. Research timeline: in-person and online days of instruction and assessment.

Activities in Order	Days	Days per Activity
Pretest: written test	1	3 days pretest
Pretest: dynamic assessment	2
Kim’s Komet instructional days in classroom	13	22 days classroom instruction
Other instructional days in classroom	8
Holidays in classroom	1
Days out of school for COVID-19 non-traditional instruction planning	10	15 days break with no school
Days off for spring break	5
Kim’s Komet + Grand Pentathlon instructional days via Zoom	12	16 days online Zoom instruction
Other instructional days in classroom	4
Posttest: written test	1	4 days of posttest
Posttest: dynamic assessment	3
	60 days	12 weeks total

, one participating teacher taught the two EAI instructional units for 25 days. Midway through the study, instruction was suspended for school-wide COVID-19 planning and spring break. During the first days of online instruction via a virtual learning platform (i.e., Zoom), the teacher reviewed general math computation with students while planning the lessons to be delivered online. The teacher then taught the remainder of the first EAI MPS unit and the modified version of the second unit (see below for the EAI MPS units), which each student accessed via Zoom from home. During the online instruction, the teacher taught all students in one large group for one hour a day rather than teaching three small groups of students in resource rooms for three hours. This arrangement increased the class size from 4–5 students in the resource room setting to 14 students online.

2.3. Participants

The trained examiner, a member of the research team, had more than 12 years of experience coordinating the research activities of EAI including our previous large-scale efficacy trials. She held a doctoral degree in special education with multi-year teaching experience in both higher education and K–12.

The participating middle school special education teacher taught the EAI units. She had a master’s degree in special education and a bachelor’s degree in collaborative elementary education and special education. She had taught SWDs for 13 years and had participated in two previous EAI studies with our research team.

The participating students (9 boys, 5 girls) were in sixth grade (age 11–12 years old) and received special education services based on their Individualized Education Programs (IEPs) in resource room settings across three class periods. Disability classifications distributed across students were as follows: five with specific learning disabilities, three with mild intellectual disabilities, three with other health impairments, one with autism, and two with emotional or behavioral disabilities. Nine students were White, two were Hispanic/Latino, two were African American, and one was Middle Eastern. Additional demographic data such as standardized test scores and free/reduced lunch status were not available due to the pandemic-related school closure.

2.4. EAI Problem-Solving Units

The teacher taught two EAI MPS units: a video-based MSP unit called Kim’s Komet (KK) and a hands-on application of MPS concepts called the Grand Pentathlon (GP). The objective of both units was to help students develop their MPS understanding and skills in pre-algebraic concepts such as linear function, line of best fit, variables, rate of change (i.e., slope), reliability, and measurement error. Central to both KK and GP is a competition that requires students to time a model car at the beginning and end of a 20-foot-long straightaway after being released from several points on a 6-foot-high ramp (see Figure 1). Students learn that they need to time their cars on the straightaway, when speed is constant, rather than on the ramp, when the car is accelerating. As they view the KK video, students help the actors in the video time their cars from several release points on the ramp, plot this information on a graph, and draw a line of best fit. Their graphs inform them of how fast their car will be traveling at the end of the straightaway from every release point on the ramp. On the day of the competition, students learn the range of speeds the car must be traveling to successfully navigate each of the five tricks (i.e., banked curve, long jump, short jump, double hump, and loop-the-loop as shown in Figure 1) which had speed minimum and maximum parameters, attached to the end of the straightaway. Students consult their graph to help them locate how high on the ramp to release their car.

The GP unit is the hands-on application of the KK unit. We built a ramp, straightaway track, and tricks like those shown in Figure 1. We attached a timer that shows the time it takes for a car to travel from the beginning to the end of the straightaway. Students took turns releasing their cars from several points on the ramp, recorded their heights and corresponding times, and computed the rates. As they did with KK, students plotted each release point height by rate and then drew a line of best fit. The repeated computing and plotting are especially helpful for students with disabilities to practice these skills.

To prepare students for the MPS activities, the teacher taught the meaning of and relationship between independent and dependent variables. The teacher showed students a series of statements (e.g., elderly people’s reaction time as the dependent variable). The teacher asked students to give full verbal explanations of the relationship of the two kinds of variables in complete sentences. For example, students could say: “Age is the independent variable, and time is the dependent variable because reaction time depends on age.” Next, students indicated on what axis of the graph they should place the independent and dependent variables. Lastly, the teacher reviewed the elements of scale by displaying a graph with the scale of 2, 4, and 6 on the x-axis and then asking students where 3 and 5 were located on the graph, because the 3 and 5 did not appear as numbers on the graph. Students were eager to learn how to interpret graphs because they knew that they would have to make their own graph of a line of best fit to help a character in the KK video win the GP events (see Figure 2).

2.5. Mathematical Problem-Solving Written Test

We used the same 9-item MPS test that Bottge and his team used in the previous studies to assess students’ understanding of the concepts in the KK and GP units (e.g., B. A. Bottge et al., 2014, 2015, 2018, 2021). Figure 3 shows a snapshot of the test items (modified to fit on one page). The test assesses the concepts covered by the EAI MPS units that are closely aligned with the Common Core State Standards for Mathematics (CCSSI-M; National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010): Measurement and Data, Number and Operations—Fractions, and Ratios and Proportional Relationships and Geometry (e.g., calculate speed, draw and interpret a line of best fit).

2.5.1. Written Test Administrations

On Day 1 of the study, students took the math pretest in their resource rooms in the conventional format of a written test (WT). The trained examiner asked students to read the test items, show their work, and write their answers in the space provided. The examiner did not help students with reading or interpretation of the test items. The average time to complete the WT pretest was 24 min.

The original plan was to administer the WT posttests in the same way as the pretests, but COVID-19 forced changes. For the posttest, we mailed the paper test and test administration instructions to the students’ homes. On the scheduled testing day, during a Zoom class session, students removed the test from its envelope and followed live directions for taking the test. We observed the testing via Zoom and recorded the amount of time each student spent on the test (i.e., 25 min on average). Within two days after the test, members of the research staff visited the students’ homes to pick up the completed posttests and worksheets they used during the test.

2.5.2. Written Test Scoring

For the WT, as with our previous studies, we awarded students a total of up to 15 points: full credit (2 or 3 points per item) for the correct answer and partial credit (1 point) for showing they knew how to solve the mathematical problem but made a mistake in computing the answer. The readability level of the test was kept at or below fourth grade level, and students were allowed to use calculators. Internal reliability estimates are 0.90 at pretest and 0.94 at posttest. In this current study, the total of 14 students’ pretest score was M = 1.9 credits (SD = 3.2, range = 0–12) and posttest score was M = 6.4 credits (SD = 4.8, range = 0–15).

2.6. Problem-Solving Dynamic Assessment

Because no consensus on the definition of DA has been reached (C. S. Lidz, 2014) specifically in the field of special education (Le et al., 2023), we operationally defined DA as an alternative assessment form with assisted instruction and feedback for individualized scaffolding construction for the purpose of the study. We based our work to develop the DA administration guideline and scoring protocol (using the same test items of WT) on clinical models of dynamic assessment in assessing MPS (Nur et al., 2022; Rosas-Rivera & Solovieva, 2023) specifically for SWDs (e.g., L. S. Fuchs et al., 2008, 2011) and a sample scripted protocol in special education (D. Fuchs et al., 2007). Following the clinical models for guiding principles of instruction and feedback in assisted DA contexts as scaffolding construction (e.g., verbal prompting) is crucial to the validity of its implementation (Pearce & Chiavaroli, 2020).

Apart from the WT using binary assessment to evaluate the correctness of the MPS, DA administration and scoring protocol in our study contexts was used as an alternative assessment, providing individualized scaffolding for students with difficulties in MPS. In DA administrations, MSP discussions using prompts between examiners and examinees were used for individualized assistance. In addition, the DA scoring protocol includes differentiating their scores by deducting the independence of prompts that students made or providing opportunities to award the prompts of response, though they were given 0 credit. DA scoring protocol allows identifying the individual students’ level of support by differentiating the credits, which ranged from 0 to 40 points.

2.6.1. Dynamic Assessment Administrations

The trained examiner administered the DA (assisted) with individual students in person on Days 2 and 3 of the study after the WT (unassisted) pretest on Day 1. We audio-recorded the DA pretest sessions and transcribed them verbatim for analysis. Pretesting all students took 238 min overall, with an average of 17 min per student.

Students took the DA posttests after the WT posttests via Zoom. The teacher informed students of the day and time they were to attend class. All students scheduled for their designated day logged into class at the same time. The classroom teacher set up a breakout room in Zoom where the examiner conducted the DA posttests, and a second member of the research team observed and took notes. The examiner administered the DA to groups of two or three students, while the other students stayed in the main room with the teacher discussing topics other than mathematical problems. We recorded the DA posttest sessions using Zoom’s built-in recording with the audio transcript option. There were six DA posttest sessions over the course of three days immediately following the WT posttest after the last day of the GP unit. Post-testing all students took 197 min overall, with an average of 32 min per group.

During the DA posttests, the examiner projected each test item using the shared screen option in Zoom. Then, the examiner chose one student to read the mathematical problem aloud and describe how to solve it. The examiner ensured that each student participated in the MSP discussions using prompts in the same manner as on the DA pretests. Although tested together, students were asked to respond to the examiner with their own answers. For mathematical problems involving graphs or charts, the examiner used editing tools to make notations on the shared screen as the students directed. When plotting points for item 5 (see Figure 3), for example, students told the examiner exactly where on the graph to plot the data points and draw the line of best fit.

2.6.2. Dynamic Assessment Scoring

The first author trained two graduate research assistants (GRAs) to score the DA assessments. In the DA scoring training sessions, the first author explained the structure of the relevant data used for scoring, including distinguishing the time point (pre- or postest) for individual students’ data. GRAs familiarized themselves with the scoring flowchart, scoring sheet, and an annotated table that offered specific instructions for each test item. GRAs initiated scoring individual DA tests by selecting the student data scoring sheet and reviewing the transcript of dialogue between each student and the examiner to understand the student’s thought process and MPS approach, with the scoring sheet as a guide (see Appendix A).

GRAs first evaluated the correctness of the test items by awarding full credit (up to 40 points) for correct answers or partial credit for answers that demonstrated some understanding. Next, the credits were deducted based on the independence of the prompts that each student made. In other words, the more independently a student formed a correct response, the fewer prompts the student needed; thus, the student received a higher score. These prompts were categorized by the level of assistance: none (0 points), few (−5 points), more (−10 points), and many (−15 points). Each level reflects a different degree of assistance. If a student completed a task “independently” without any prompts, the student received no deductions, indicating full independence (i.e., 40 points). Minimal prompts, such as small hints or slight nudges that do not provide direct answers, resulted in a deduction of 5 points. Moderate prompts, which offered more direct guidance while still requiring the student’s input, led to a 10-point deduction. Extensive prompts involving detailed, step-by-step guidance and significantly reducing the student’s need for independent thought resulted in a 15-point deduction.

When a student answered incorrectly and got 0 points, possible award points were given based on the level of responsiveness to or engagement with the examiner’s prompts. In other words, the students “earned back” or received additional points by demonstrating their MPS understanding through a verbal interaction with the examiner: none (0 points), few (+5 points), more (+10 points), and many (+15 points). If a student did not respond correctly to any prompts, no additional points were awarded, reflecting a lack of understanding. If a student provided a few correct responses, assuming a small step toward MPS understanding, they were awarded 5 points. Students who showed some level of engagement, demonstrated partial MPS understanding, or completed several correct steps with guidance were awarded 10 points. Students who provided significant responses and made considerable progress toward solving the mathematical problem, even with assistance, received 15 points, reflecting a near-complete MPS understanding with minor errors.

Misconceptions that arose during the verbal interactions in the DA tests were noted on the scoring sheet, as this information could inform the level of students’ MPS understanding. After all these factors were considered, the total score for the DA tests was calculated, including any deductions based on the level of prompting required or additional credits based on the level of responsiveness. After all credits were entered in the scoring sheet, scorers reviewed the whole scoring process to ensure the accuracy of the evaluation. Finally, the first author assessed the inter-rater reliability (IRR) between the scoring of the two GRAs. IRR was calculated by dividing the number of matching credits assigned to each student’s responses by the total number of responses. The IRR was 95.2%, and any mismatched credits were resolved by 100% consensus. The maximum credit for each student was 280 points when multiplying the 7 items by 40 points per item. In this current study, a total of 14 students’ pretest score was M = 111.4 points (SD = 70.6, range = 0–215) and posttest score was M = 196.8 points (SD = 17.0, range = 175–225).

2.7. Data Analysis

2.7.1. Quantitative Analysis

To explore the adequacy of DA compared with WT (RQ 1), we analyzed pre- and posttest scores from both WT and DA applying the two-level multilevel models via the R package lme4 (Bates et al., 2015, package version 1.1-35.4, R version 4.3.2). The hierarchical linear modeling approach was used, as the outcome measurements were nested within students (14 participants, 56 measurements in total). We noted that the classroom was not treated as a separate level. Instead, we followed recommendations in (McNeish & Kelley, 2019) by dummy-coding Class as student-level predictors because only three classrooms were included. The null model yielded an intraclass correlation coefficient (ICC) of 0.06. In the final model, we included the predictors Time (0 = pretest, 1 = posttest), Test (0 = WT, 1 = DA), and their interaction in the level 1 equation. At level 2, we added the covariates Sex (0 = male, 1= female) and dummy-coded Class variables. To ensure convergence and avoid singularity issues in estimating variance components, we fixed the slopes for all level 1 predictors. The final model specifications are provided below.

$$\begin{array}{l}\mathrm{Level}1:\\ \\ {\mathrm{Y}}_{\mathrm{ij}}={\mathsf{\beta }}_{0\mathrm{j}}+{\mathsf{\beta }}_{1\mathrm{j}}{\mathrm{TIME}}_{\mathrm{ij}}+{\mathsf{\beta }}_{2\mathrm{j}}{\mathrm{TEST}}_{\mathrm{ij}}+{\mathsf{\beta }}_{3\mathrm{j}}{\mathrm{TIME}}_{\mathrm{ij}}\times {\mathrm{TEST}}_{\mathrm{ij}}+{\mathrm{e}}_{\mathrm{ij}},{\mathrm{e}}_{\mathrm{ij}}\text{~}\mathrm{N}(0,{\mathsf{\sigma }}_{\mathrm{e}}^2)\\ \\ \mathrm{Level}2:\\ \\ {\mathsf{\beta }}_{0\mathrm{j}}={\mathsf{\gamma }}_{00}+{\mathsf{\gamma }}_{01}{\mathrm{SEX}}_{\mathrm{j}}+{\mathsf{\gamma }}_{02}{\mathrm{CLASS}\text{\_}1}_{\mathrm{j}}+{\mathsf{\gamma }}_{03}{\mathrm{CLASS}\text{\_}2}_{\mathrm{j}}+{\mathrm{u}}_{0\mathrm{j}}\\ {\mathsf{\beta }}_{1\mathrm{j}}={\mathsf{\gamma }}_{10}\\ {\mathsf{\beta }}_{2\mathrm{j}}={\mathsf{\gamma }}_{20}\\ {\mathsf{\beta }}_{3\mathrm{j}}={\mathsf{\gamma }}_{30}\\ {\mathrm{u}}_{0\mathrm{j}}\text{~}\mathrm{N}(0,{\mathsf{\sigma }}_{\mathrm{u}0}^2)\end{array}$$

(1)

To get more accurate statistical inferences given the small sample size, we applied the Kenward–Roger approach to adjust the standard error and degrees of freedom estimates for fixed effects (Kenward & Roger, 1997). Following Nakagawa et al. (2017), we calculated R-squared (R²) to quantify the proportion of variation in the outcome explained by the predictors. Given the differences in test scoring scales, DA scores were transformed and thus comparable with WT scores. Specifically, DA scores were divided by 18.67, the ratio of the total DA score (280) to the total WT score (15). Sensitivity analyses using an alternative transformation method—dividing DA scores by 280 and WT scores by 15—yielded identical inferential results. We used the first transformation method in our analysis.

2.7.2. Qualitative Analysis

To answer qualitative RQs (RQs 2 and 3), we applied a qualitative case study approach in special education research (Leko et al., 2021) using a thematic analysis methodology (Peel, 2020; Yin, 2014) via qualitative data analysis software, MAXQDA 24 (VERBI Software, 2021). To achieve methodological triangulation in our qualitative analysis (Arias Valencia, 2022), we analyzed multiple data sources of pre- and post-DA transcripts, notes taken from the DA observer, and student work. In addition, multiple researchers independently analyzed the same data set, including two GRAs who were blind to the WT and thus unbiased regarding students’ WT responses.

We conducted our qualitative analysis in several rounds, owing to our choice of a combined deductive and inductive approach. For deductive coding, we adopted Pólya’s (1945) four-step problem-solving heuristic, which has been widely recognized for its theoretical strength in the problem-solving literature (e.g., Ho & Hedberg, 2005), as the framework to structure our codes. This step aimed to guide our thematic analysis of the MPS process (RQ 2) based on the existing problem-solving theories and assist with the latent analysis (Bengtsson, 2016) of the examiner–examinee interactions (RQ 3) with the scarcity of research in the context of DA to assess MPS for students with disabilities.

After completing the first coding round and gaining a deeper understanding of the various interactions between the examiner and students during the DA sessions, we proceeded to the second round of coding, during which latent themes within the analysis units (e.g., student–examiner dialogue) were extracted and categorized. Through iterative reviews of the coded meaningful units, we revised and rearranged the themes generated through the inductive approach and the parent-level ones informed by Pólya’s (1945) heuristic: (1) understand the problem, (2) devise a plan, (3) carry out the plan, and (4) look back or check the answer. This resulted in a final set of codes that characterized the stages of the MPS process in the context of DA, providing a basis for theory development. In addition to MPS stages, we created new codes categorizing corresponding meaningful units regarding rerouting in MPS through analyzing efforts. All coding procedures were conducted on the digital qualitative analysis tool MAXQDA without using any AI functions.

3. Results

3.1. Adequacy of DA (RQ 1)

Our quantitative analysis for RQ 1 revealed no statistical differences between DA and WT in terms of the evaluation of intervention effects. After accounting for level 2 covariates, the results showed a significant difference between pre- and posttest scores for the WT (${\gamma }_{10}$ = 5.09, SE = 1.07, t = 4.77, df = 38.41, p < 0.0001). However, the interaction effect for ${TIME}_{ij}\times {TEST}_{ij}$ was not statistically significant (${\gamma }_{30}$ = −0.52, SE = 1.49, t = −0.35, df = 38.27, p = 0.73), meaning that the difference between pre- and posttest scores for the DA was not significantly different from the WT. The R² for level 1 predictors was 0.62, meaning that they explained 62% of the variation in the outcome after controlling for covariates at level 2.

3.2. MPS Process of Students with Disabilities (RQ 2)

Our qualitative analysis revealed a four-stage MPS process in the context of DA: (1) understanding the problem’s context, (2) thinking through the problem while forming a plan, (3) carrying out the plan, and (4) looking back. Overall, our results share two overlapping stages with Pólya’s original heuristic: carry out the plan and look back. This indicates that Pólya’s (1945) heuristic has a moderate fit with the current MPS process of our study. The revision of the first two stages—understanding the problem’s context (stage 1) and thinking through the problem while forming a plan (stage 2)—was based on the distinct yet dominant patterns of interactions between the examiner and students shown in our data set. While carrying out the plan (during stage 3), we found another unique theme of the problem-solving ownership transition. Although it does not stand alone as a separate stage, it features the dynamic turning point in interactions between students and the examiner, as well as serving a critical signaling function for the stage shift in the dynamic MPS process. Looking back (stage 4) did not occur frequently nor consistently across the students.

3.2.1. Stage 1: Understanding the Mathematical Problem’s Context

Despite the question type in the DA, the examiner always initiated the assessment by asking the students to read the question out loud and familiarize themselves with any other assessment material (e.g., graphs) to understand the problem’s context before any attempt at addressing the question. For example, the examiner asked students to “read the first question out loud to me—in the yellow”. When the question item included a graph or other visual information, the examiner directed the students, saying, “that’s a lot of information, but let’s look at this graph, okay?” We found that the examiner’s prompts were generic and very similar across the students.

3.2.2. Stage 2: Thinking Through the Mathematical Problem While Forming a Plan

Following the first theme, or stage 1, in MPS, the examiner walked students through the information provided in the question and other assessment materials to identify important steps for solving the mathematical problem. During this stage of thinking through the problem while forming a plan, the examiner often posed a series of prompting questions to help students recognize differences between the known elements within the mathematical problem context (i.e., information provided) and the unknown problem (i.e., novel problem). From the students’ standpoint, this stage provided them with ample opportunities to anchor their MSP understanding and skills into the mathematical problem’s scenario; therefore, they were gradually forming MPS plans to solve the mathematical problem in the subsequent MPS stages. The examiner then included the MPS plans that each student formed while interacting with them through a compounded format of the examiner’s (a) explicit guidance (e.g., “she wants to get there by 6 o’clock and she’s going to leave by 4 o’clock, so how much time is that?”), (b) correction to students’ responses (e.g., “THIS means miles, right?”), and/or (c) subtle hints such as prompting questions (e.g., “What do we divide? Do we have a formula at all?”). The examiner also paused providing further directions or prompting questions if students were actively working although making constant mistakes. During the same stage of MPS, students were more passive, devising their formed MPS plans by responding to the examiner’s instructions; and the examiner acted as an active guide who monitored the students’ MPS understanding and skills closely to optimally help them get the plan “set up”. Unlike the previous stage, the examiner’s prompts remained highly idiosyncratic, meaning prompts were unique to and individualized for each student in this stage.

3.2.3. Stage 3: Carrying out the Plan

One unique finding is that the onset of this stage is almost always marked by the examiner’s brief summary of stage 2 planning and a prompt encouraging the students to carry out the plan (e.g., “So, now you have THAT information—and so she needs to go the distance you said—110 miles in 2 h.”). After summarizing the formed plans, the examiner conceded the MPS ownership to students, asking if they have the necessary skills for carrying out the formed plan from stage 2 (e.g., “Do you know how you would figure out how fast she would need to go—how many miles—each hour she would have to go?”). Students then proceeded in their MPS by carrying out the plan with or without the examiner’s prompts. While the examiner guided some students to carry out their formed plan (e.g., “Go ahead.”), the examiner waited for other students to start solving the mathematical problems. Our findings indicate that students always made an independent attempt regardless of the prompts.

The examiner’s responses were varied depending on the students’ answers. If a student’s independent answer was correct, the examiner called it to the end or moved on to the next stage. If a student remained silent or came up with a wrong answer, however, the examiner usually took the MPS ownership back and engaged the student in one of the three approaches: (a) scaffolded learning—step-by-step direct guidance to help the student arrive at the right answer; (b) preview solution—the examiner shows the student the steps to take without direct effort from the student; or (c) quit MPS or circle up to the carrying out the plan stage.

3.2.4. Stage 4: Looking Back

Different from the previous stages, looking back did not frequently occur in the MPS process. This result aligns with previous research findings that stage 4 occurs relatively infrequently in MPS contexts (Krawec et al., 2012; Montague et al., 2011). Our analysis revealed several riveting subcategories that resemble other extended theories of Pólya’s (1945) heuristic (e.g., Cai et al., 2024). Within our looking back framework, we found four sub-themes: (a) considering a different approach; (b) solution refinement; (c) solution review and reflection; and (d) applying the solution to a new scenario. Interestingly, we found the distribution of these sub-themes varied according to the timing of the assessments. Solution review and reflection occurred most often in pretests, while solution refinement and considering a different approach predominantly occurred in posttests. One related finding that potentially explains this pre–post assessment difference in stage 4 is the examiner’s expectation disparity toward the students between the pre- and posttests. For the same answer provided by the student (e.g., 55) to a given question, the examiner would count the answer as correct and focus on reflecting on the student’s MPS process (i.e., solution review and reflection) in stage 4 during the pretests. In the posttests, however, the examiner usually expected students to have a more precise answer and a deeper understanding of the applied scenario (e.g., 55 miles per hour), leading to more frequent occurrences of solution refinement and considering a different approach.

3.3. Interactions Between the Examiner and Examinees (RQ 3)

To better understand the highly individualized testing format of assisted DA, we looked closely at the nature of the interactions between the examiner and students. We found that the examiner played an active role in guiding the MPS process of students with disabilities by providing individualized verbal prompts and scaffolded instruction and feedback based on a student’s responses. However, the scaffolded guidance in DA fundamentally differs from grading as an authority on the WT assessment. Throughout the MPS process, the examiner constantly provided individual students with instruction and feedback by treating them as stakeholders in tackling a task rather than passive examinees. We found that the guidance the examiner provided to each student did not follow a one-size-fits-all approach. Instead, the examiner took account of various personal and situational factors through scaffolded learning phases for individual students.

When a student who demonstrated a solid understanding of the question showed a lack of confidence (e.g., “I think I got this one wrong”), for example, the examiner encouraged the student to keep trying and provide more extensive support to their attempts and praise afterward (e.g., “That’s okay, let’s see if we can figure it out!”, “That’s right!”). Conversely, if a student was reluctant to answer a question, constantly giving answers not germane to the question, or expressed a lack of endurance to an extent that interfered with their MPS engagement (e.g., yawning), the examiner would cease the MPS process for the sake of both the student’s preference and the assessment’s validity. Occasionally, we found that students showed extreme perseverance in insisting on solving a mathematical problem when the examiner showed signs of pause (e.g., “I’m waiting for the rest of the final answer”) or moving to the next step (e.g., “I’m going to tell you this one and then we’ll do the next one together”). In such situations, the examiner always appreciated the perseverance and followed the student’s willingness to continue MPS until finished.

4. Discussion

In this study, we explored the use of DA in the context of the evidence-based EAI. Aligned with previous research (L. S. Fuchs et al., 2008, 2011; A. K. Jitendra & Kame’enui, 1993), our findings suggest that DA not only is an adequate assessment tool for MPS skills, but it is also a complementary assessment approach that may be used to support teaching and learning MPS for SWDs. WT methods often rely on conventional formats (e.g., unassisted paper–pencil) that may not fully capture the MPS processes of SWDs. DA, on the other hand, can provide additional insights into how students approach, plan, and solve mathematical problems, offering educators a fuller picture of the students’ MPS strengths and challenges.

4.1. Adequacy of DA

Although research suggests the feasibility and usability of DA to assess MPS (Le et al., 2023) and the validity of the EAI MPS written assessment (B. A. Bottge et al., 2014, 2015, 2018), no known study has investigated the adequacy of EAI assessment in DA form, especially among middle school SWDs to date. Using D. Fuchs et al.’s (2007, 2011) index technique, we quantified the gap between what students can do with assistance (WT) and without assistance (DA) and compared the students’ WT scores (unassisted test) with their DA scores (assisted test) on identical EAI test items to evaluate the adequacy of the DA. Our finding that DA and WT showed no statistically significant differences in capturing the effects of the EAI intervention suggests that the assisted test can be a comparable measure of students’ MPS skills. Our finding documents that the assisted test in the context of EAI demonstrated adequacy in evaluating the effects of EAI for middle school SWDs and provides empirical evidence of using DA to assess the MPS skills of SWDs.

4.2. MPS Process

Through our qualitative analysis, we identified a four-stage MPS process in the DA context: understanding the problem’s context, thinking through the problem, carrying out the plan, and looking back. While partially aligning with Pólya’s heuristic, we found a unique theme of MSP ownership transition emerged while carrying out the MPS plan as a key point in examiner–student assessment exchanges. While some students independently carried out the plan, the examiner provided different responses or individualized prompts based on student performance with scaffolded guidance, previewing solutions, or disengaging as needed. These varied responses or scaffolded prompts align with previous research (A. K. Jitendra & Kame’enui, 1993; L. S. Fuchs et al., 2008, 2011) indicating that SWDs require greater scaffolding and individualized support. We suggest that DA can promote the MPS learning ownership when examiners provide individualized, scaffolded prompts.

We are encouraged that our findings concretize the concept of Vygotsky et al.’s (1978) zone of proximal development with respect to student learning potential. Responding to students’ personal needs at the moment of testing can also help the examiner avoid making attributional errors in explaining students’ failures on answer sheets (e.g., ability vs. motivation) by showing a more precise view of a student’s capacity and providing insights about optimal ways to support the student in the future. Despite the potential validity threat of assisting students in DA testing (e.g., Crooks et al., 1996; Gu, 2021), we suggest that the assisted testing approach can maximize the MPS learning potential of SWDs.

4.3. Instruction and Feedback of DA

Unlike WTs, the DA approach emphasizes meaningful interactions between examiners and examinees during assessment (Le et al., 2023), whether using standardized prompts or individualized prompts to address the specific learning difficulties of SWDs. In the highly individualized DA approach used in this study, the examiner actively guided SWDs through verbal prompts and scaffolded support tailored to their responses. The DA examiner treated students as active collaborators, providing personalized instruction and feedback throughout the MPS process. This scaffolded guidance was adapted to meet individual needs and situational factors, avoiding a one-size-fits-all approach (Elliott, 2003; C. Lidz & Elliott, 2000; Stringer, 2018). Consistent with the previous research in special education (A. K. Jitendra & Kame’enui, 1993), we suggest DA examiners should include “specific, accurate feedback” (A. K. Jitendra & Kame’enui, 1993, p. 321) that is systematically designed for individual SWDs. Aligned with High-Leverage Practices 8 and 22, recommending effective instructional feedback that is “clear, specific, explanatory, and timely” (Aceves & Kennedy, 2024, p. 133) based on student performance, we suggest that MPS assessment can include meaningful feedback to address a faulty interpretation of testing items and better support SWDs to make meaningful progress toward their learning goals (Aceves & Kennedy, 2024).

4.4. Limitations and Suggestions for Future Research

Potential research limitations include disruptions to the instructional delivery due to external factors such as school-wide COVID-19 planning and the shift to online instruction. Instruction was suspended for logistical adjustments and spring break, creating potential interruptions to the students’ learning. Additionally, the shift to online instruction via Zoom altered the teaching context, increasing the class size from small groups of students in a resource room setting to a whole group online. This change may have reduced the teacher’s ability to provide individualized support and replicate the small-group, intensified EAI instructional format. These changes may have impacted the fidelity of implementation for the EAI MPS units and potentially influenced student engagement and outcomes. Thus, the findings may not fully capture the effectiveness of the intervention, as it was originally designed and intended for smaller group instruction in the classroom.

Another limitation of this study is the small sample size included for the analyses, as only one teacher, three classes, and a limited number of students from each class participated. This could lead to low power to detect significant effects when using hierarchical linear modeling and limit the generalizability of the findings to broader populations of students and educators. To address these limitations and strengthen the validity of the DA approach, future research should involve larger and more diverse samples of teachers and students across multiple settings. Expanding the participant pool would allow for a more robust evaluation of the DA approach, ensuring it can be effectively implemented in varied educational contexts while maintaining consistent outcomes.

We also suggest that more studies should be replicated to ensure the findings of our study. As well, future studies should explore how teachers are able and willing to use the DA approach and further develop detailed guides and resources to implement the DA protocol used in the study.

4.5. Implications for Practice

Implementing DA within the EAI intervention offers several benefits for educators working with SWDs. By integrating individualized instruction and scaffolded support during the assessment process, DA provides a more equitable platform for students to fully express their MPS concepts and skills. Unassisted written assessments may fail to capture the depth of students’ MPS understanding due to disability-related difficulties and testing format or access barriers. Assisted DA, however, emphasizes meaningful interaction and scaffolded feedback, empowering students to actively engage in and take ownership of their MPS process. Additionally, DA requires no appreciably different time to administer compared with conventional WT, but it can provide richer and more detailed insights into students’ MPS process and understanding.

For educators, adopting DA within EAI enables a deeper understanding of individual students’ current MPS skills and specific learning needs, promoting instructional practices that are responsive and adaptive. When combined with unassisted written tests, administering assisted DA can help teachers identify the gap between what students can do with assistance and without assistance. This approach of combined tests not only ensures the validity of the assessment results but also fosters student confidence and autonomy in tackling complex MPS tasks. In practice, this means shifting the assessment focus from outcome-driven to process-driven, where assessments serve as an extension of teaching, enabling SWDs to demonstrate their MPS abilities in a supportive, guided environment.