A Theory of Impacts Model for Assessing Computer Science Interventions through an Equity Lens: Identifying Systemic Impacts Using the CAPE Framework

Abstract

With more recognition being given to the diverse and changing demographics in education, there is a need to understand how well computer science education is meeting the needs of all learners as it starts to infiltrate K-12 schools. The CAPE framework is a newer model for assessing the equitable delivery of computer science education and can be used to understand a school’s capacity to offer equitable computer science (CS) education, equitable student access to CS education, equitable student participation in CS, and equitable experiences of students taking CS. Since the CAPE framework is a new way to research CS education through an equity-lens, there are few, if any, frameworks that can be leveraged to explore research questions in a complex, multi-school intervention. To address this gap, we used a design-based research approach to create and determine the feasibility of a new model, Theory of Impacts, informed by the CAPE framework (the ToI-CAPE model), for evaluating a multi-school intervention. In this article, we provide a detailed explanation of creating and using the ToI-CAPE model for a specific intervention and the feasibility of using ToI-CAPE across factors based in experiences and how to use this model in other research and evaluation projects. Overall, the use of the ToI-CAPE model can be used to shed light on the critical subcomponents and agents at work in the intervention and the actions necessary across these components and agents to support intended outcomes.

1. Introduction

The expansion of computing education in primary and secondary schools across many countries requires the expansion of computing education research for finding promising practices. There remains much to be learned across international efforts focused on broadening participation in computer science (CS), including the research necessary for identifying promising practices for equity-focused, educational outcomes across a variety of schools and student population groups [1]. Though there are efforts underway to grow the available scholarship, the dearth of research is understood by considering the following: fewer than 3% percent of published papers in the field of K-12 (K-12 is used to reference primary and secondary schools within the United States). CS education needs to recognize students with disabilities in CS education communities to become more aware of the inequities that exist and to work to eradicate them. To advance both the field of CS education as well as the research within the field in terms of equity, we need a systemic framework to structure evaluation and reflection, and to assess the impact of the individual components within the framework. This article discusses the enactment of such a framework within the context of a CS educational program.

Due to the tumultuous 2020 with respect to race relations in the U.S., prominent CS researchers have been critical of the collective body of CS research, stating its lack of equity-focus [2,3,4]; however, it is important to note that equity is a term that often does not have a shared meaning in the field of CS. While the authors of the CAPE framework do not define equity, we use a definition of equity provided by UC Berkeley Initiative for Equity, Inclusion, and Diversity:

The guarantee of fair treatment, access, opportunity, and advancement while at the same time striving to identify and eliminate barriers that have prevented the full participation of some groups. The principle of equity acknowledges that there are historically underserved and underrepresented populations and that fairness regarding these unbalanced conditions is needed to assist equality in the provision of effective opportunities to all groups [5].

The complexities of equity-focused issues have spurred many calls for and commitments from within K-12 CS education communities to become more aware of the inequities that exist and to work to eradicate them through implementation practices. To advance both the field of CS education as well as the research within the field, the CS community can benefit from a systemic model to measure the impacts of interventions across the CAPE (Capacity, Access, Participation, and Experience) framework.

On its own, the CAPE theoretical framework represents a meaningful structure for understanding the most salient aspects of an (in)equitable CS education program; however, the components within each of these aspects are themselves dynamic and complex, impacting the outcomes of the program in different ways and to different degrees. It is essential that research is conducted to study these components and that their roles be analyzed and explained. Our creation of a formal method to carry out this process maps a Theory of Impacts (ToI) across each aspect of the CAPE Framework (ToI-CAPE) onto a medium-sized CS education research study.

This study is important to researchers and evaluators since the K-12 CS education is still in its early formation stages, and there remains a great deal to be studied to ensure the most promising practices for all children are discovered. This model’s contribution will provide a step-wise method for understanding an intervention and its likely impacts, which enables the creation of a set of hypotheses based on the agents, actions, and potential equityfocused outcomes. Researchers studying the intervention can then take these hypotheses, collect the evidence, and test them to determine how well the intervention achieves its equity-focused goals.

In this article, we first provide a review of extant literature on frameworks and models related to ToI-CAPE, which provides a starting point to develop informed research questions and hypotheses for this study. We then outline the methods, including the participants and stakeholders involved, the design-based approach to iterating the implementation of ToI-CAPE, and the sources of data which were used for evidence and evaluation of ToI-CAPE’s feasibility. Finally, evidence-based findings are provided as well as future research implications.

2. Review of Relevant Literature

The CAPE framework is one model for evaluating and assessing the impact (outcomes) of interventions within K-12 CS education. We discuss relevant methods and extant models here.

2.1. Approaches to Education Program Assessment and Evaluation

In general, approaches to assessment and evaluation of interventions involve the selection of a research design, a determination of appropriate measures to be used, analyses of multiple types of data, and conclusions or recommendations based on the outcomes [6,7]. Each of these pieces are dependent upon the goals of the evaluation as delineated by stakeholders and/or program designers [8]. The evaluation design may be exploratory in nature, wherein researchers collect and report on data in absence of previous research or with the goal of determining the most important avenues for subsequent evaluation; or, it may be confirmatory in nature, where hypothesis are made a priori [9]. The design may involve experimental research with intervention and control groups [10]. Researchers may also utilize case studies on a smaller scale. Across these designs, measures typically include quantitative data from participants (e.g., surveys, knowledge assessments) and qualitative data (e.g., interviews, artifacts from observations).

2.2. Logic Models in Educational Research and Evaluation

To engage in this process, we first turned to the standard logic model. A logic model is a descriptive way to delineate logical consequences (outputs) of inputs, describing the what of an intervention (Sharp, 2021) [11].

Effective logic models make an explicit, often visual, statement of the activities that will bring about change and the results you expect to see for the community and its people (definition modified from [12]). It provides an outlined structure for measuring an intervention’s outcomes [13,14]. The U.S. Department of Education encourages and enables programs to use logic models since they are useful for program development and evaluation planning [15]. Logic models (1) serve as a format for clarifying what the program hopes to achieve; (2) are an effective way to monitor program activities; (3) are useful for performance measurement or evaluation; (4) can help programs stay on track for the future; (5) are an excellent way to document intention and reality of an intervention [14]. A logic model not only specifies the outcomes, but also an intervention’s inputs, goals, and objectives (e.g., delivery methods and approaches to measurement). In the context of education research, logic models can be leveraged to help evaluate outcomes and approaches to determine when they are aligned to the overall program goals.

A commonly known logic model for evaluating interventions with specified desired outcomes is Theory of Change (ToC). A Theory of Impact (ToI), on the other hand, looks more broadly at the system of causes, effects, feedback, and stakeholders that lead some interventions to generate greater impacts [16]. In their discussion of logic models for program evaluation, McLaughlin and Jordan noted how the approach allows researchers to “focus on the important elements of the program and to identify what evaluation questions should be asked and why and what measures of performance are key” (p. 62).

2.3. Complex Systems Research in Education

By uncovering the most impactful variables, there can be a focus on an interactive complex systems view of the intervention rather than unidirectional, linear view. Complex systems research has gained traction in studies of educational interventions because it acknowledges the dynamicity of school-based work: “Emergent outcomes are more than the sum of their parts, meaning the complex behavior cannot be reduced to the components that make up the system” (p. 186 [17]). These components may include school teams (e.g., teachers, principals, counselors); students and their unique skills, interests, goals; settings and resources (e.g., classroom structure and availability, course offerings, technology); and other factors in the community (e.g., experiences of the group, perspectives). Research focused on systems-level approach to evaluation research or an intervention’s impacts recognize that systems are “...goal oriented...have inputs from their environment,...have outputs to achieve their goals, and there is feedback from the environment about the output” (p. 43 [18]).

2.4. Culturally Responsive Evaluation (CRE)

Another important approach to evaluation in educational research is culturally responsive evaluation (CRE). The focus on equity necessarily positioned considerations of culture at the forefront of developing ToI-CAPE. CRE places the onus on evaluators to center community needs and goals in assessing program effectiveness. In essence, CRE helps researchers redefine what counts as effective by delineating how it is envisioned by stakeholders. Incorporating tenets of CRE into our development of the ToI-CAPE reminded us of the responsibility of evaluators to use approaches valid within the given context: “To act on this responsibility requires one to be responsive by being aware and recognizing the centrality of culture and cultural context in our evaluative work and identifying the appropriate methods and tools that will best serve the community” (p. 309 [19]).

2.5. The CAPE Framework: Capacity, Access, Participation, and Experience

The CAPE framework was designed to help researchers dissect a computer science program, course, or intervention with regard to equity by using. Created by [20], it serves to disaggregate the capacity of schools to offer equitable computer science (CS) education, equitable access to this education, equitable participation of students in this education, and equitable experiences of the students who participate (Figure 1).

CAPE takes a systems-level approach to studying the complexities of CS educational environments. In this approach, not only are student or teacher level outcomes (elements) considered, but also considered are how those outcomes are situated within a larger initiative and policy level environment (system) [21]. It is understood that the system (whole) is greater than the sum of its parts (elements). The relationships between and among the elements and subsystems is continually changing in educational environments, which creates a perfect atmosphere for a systems-level approach to evaluate intended and unintended outcomes of this intervention [21]. Figure 1 shows the CAPE framework as described by [20] and used for state level CS education analysis.

2.6. Summary: Mapping a Theory of Impacts (ToI) onto CAPE in CS Education Research

The CAPE theoretical framework on its own represents a meaningful structure for understanding the most salient aspects of an (in)equitable computer science education program; however, the components within each of these aspects are themselves dynamic and complex, impacting the outcomes of the program in different ways and to different degrees. It is essential, therefore, that these components and their roles be explicated.

In the next sections, we present a way to carry out this process using a Theory of Impacts (ToI) across each aspect of the CAPE framework by mapping ToI-CAPE on to a large-scale computer science education research study. First, we define important terms, and review extant literature on frameworks and models related to ToI-CAPE, which are followed by informed research questions and hypotheses for this study. Then, we detail the methods utilized in this study, including the participants and stakeholders involved, the design-based approach to iterating our implementation of ToI-CAPE, and the sources of data that were used for evidence and evaluation of ToI-CAPE’s feasibility. Lastly, we offer our findings and conclusions based on this evidence and present directions for future research.

Thus, this research introduces a new model to use when investigating equity-focused impacts of interventions that encompass each component of CAPE. This new model is important for researchers and evaluators who perform or want to understand how to structure and perform equity-focused research (including evaluation research) across complex systems that range from a school’s (or district’s or state’s) capacity to offer computing related education to the students’ experiences engaging in that research. We offer perspectives on how to effectively design and implement this model, and discuss considerations of the practicality of this new approach.

3. Materials and Methods

3.1. Feasibility Study Design

Our main research question under investigation in this study was:

How feasible is it to create a Theory of Impacts model grounded in the CAPE framework (the ToI-CAPE model) and to use that model to study computer science education interventions?

To answer this question, we used a design-based research approach [22], an approach that uses “...methods that link processes of enactment to outcomes”, which then:...has power to generate knowledge that directly applies to educational practice” (p. 7 [22]). This approach centers on methodological alignment, which ensures “...that the research methods we use actually test what we think they are testing” (p. 203 [23]).

Further, we adapted Bowen et al.’s focus areas for feasibility studies Bowen et al. [24], zeroing in on Implementation and Practicality (see

Table 1. Focus areas (adapted from Bowen et al. [24]), outcomes, and hypotheses.

Focus Area	Outcomes of Interest	Hypothesis
	1. Types of resources needed to implement ToI-CAPE	H1. Personnel, Equipment, Software
Implementation	2. Amount of each resource needed to implement ToI-CAPE	H2. 3–6 months, 2–4 researchers, computers, software for communication, analyses
	3. Degree of difficulty in implementing ToI-CAPE	H3. Challenges in understanding the intervention, decisionmaking around research scope
	4. Efficiency of implementation (does the quality of the product(s) justify the time taken by the approach?)	H4. Using ToI-CAPE means slower execution (initially), but higher quality product
Practicality	5. Effects of using ToI-CAPE (positive and negative)	H5. Positive = gain a clear understanding of system components and their impacts, maintain focus on equity; Negative = initially complex
	6. Degree of ability to administer ToI-CAPE in evaluation research or other complex systems research	H6. Possible
	7. Cost analyses	H7. Reasonable expenditures on personnel and resources

). Implementation considers the degree of execution of the process, its success or failure, the amount and type of resources needed to implement, and the factors affecting implementation ease or difficulty. Practicality considers the efficiency, speed, or quality of implementation, the positive/negative effects of using the process, the ability to administer the process and a cost analysis.

Our hypotheses for each outcome were based on previous research and experience. We anticipated the resources necessary for implementing ToI-CAPE would include personnel, equipment (computers, phones, internet), and software for organizing and analyzing data. We also hypothesized challenges in terms of iterating and decision-making, complexities of the work, cost, and initial efficiency with a new process. Specifically, we considered how time and resources are important resources that can be drained when conducting evaluation research because of the complexities of the systems under investigation. Thus, we hypothesized that ToI-CAPE would be as efficient or more efficient than other approaches because there is an a priori structure (CAPE) in place for embedded systems evaluation.

3.2. Feasibility Test Design

3.2.1. Intervention

The larger study that we used to implement the ToI-CAPE model involved a multischool intervention that brings CS education to high schools in the United States with Air Force Junior Officer Reserve Training Corp (AFJROTC) programs across the U.S. The intervention’s goal is to scale-up evidence-based CS/Cyber education program through the reach of JROTC [25]. Data from the AFJROTC program indicate that there are over 500,000 AFJROTC cadets in over 3400 high schools in the U.S. and overseas [26]. Among the cadets, 55% are from underrepresented populations and 40% are female. Over 50% of the cadets are located at Title 1 Schools, which in the U.S. indicates that students are from families with low socio-economic status. Today, just 32% of JROTC high schools (and students) have access to an Advanced Placement (AP) CS course in their school. By leveraging the JROTC infrastructure and cadet population, the intervention can (1) significantly increase equitable participation in AP CS courses in the US, preparing students for pathways into critical need areas such as computing, cybersecurity, defense technologies, and artificial intelligence and (2) begin to bridge equity gaps in CS education through an existing and established point of entry.

The intervention engages school teams that include administrators, teachers, AFJROTC instructors, and counselors. CSforALL provides guidance and direction as well as monitors progress of the teams through workshops, community building, webinars, and recommending professional development programs for teachers. This concentrated effort also engages AFJROTC instructors, who work directly with the cadets in their high schools, to direct, encourage, and mentor their cadets to take relevant CS/Cyber courses.

We adapted the CAPE framework to ensure that it connected the experiences of students to the institutional environment of capacity, access, and participation in the intervention. Figure 2 shows our interpretation of this framework, which is slightly reframed to consider previously established curriculum as well as ongoing activities implemented as part of the intervention. It also shows our envisioning of the framework, highlighting Capacity’s unique foundational relationship with access, participation, and experience.

3.2.2. ToI-CAPE Development Process

The ToI-CAPE lays out a set of equity-focused claims about how an intervention will impact valued outcomes [20]. ToI-CAPE provides a framework that formalizes how similar interventions can generate hypothesized outcomes for various stakeholders. An important component of this ToI-CAPE model is a logic model that links program components and subcomponents to short-, mid-, and long-term outcomes. The outcomes inform a set of hypothetical claims that specify the action mechanisms which link an interventions program components and subcomponents and intermediate and ultimate outcomes. The claims are then supported with references to research findings, logical arguments, relevant literature, and general knowledge regarding how CS reform programs function in secondary education systems.

While creating the ToI-CAPE model, we completed a process for bounding and framing the inputs, impacts, and outcomes across all contexts using a design-based research approach. As outlined by Barab and Squire, “design-based research focuses on understanding the messiness of real-world practice, with context being a core part of the story and not an extraneous variable to be trivialized” (p. 3 [27]). Establishing bounds and frames is a matter of judgment within real-world school settings, but it is necessary in scoping the ToI-CAPE model to align with available resources and funding. Additionally, some components and outcomes that are desirable may not materialize immediately in practice; however, the ToI-CAPE model can identify unintended consequences or undesirable outcomes. By specifying a theory for how a CS reform program might lead to an undesirable outcome, it is possible to consider what kinds of actions might be needed to intervene to make desirable outcomes more likely.

Two linear steps were taken to build the Theory of Impacts (see Figure 3). First, we deconstructed each component of the intervention, identified the agents involved, identified the actions that the agents take, and identified the Intervention’s desired outcomes. This analysis was used to build hypotheses for each of the outcomes (see Table 1). Second, we identified the evidence needed to support or refute each hypothesis including the data sources for evidence and analysis of each source.

Step 1: Deconstruct components. The logic model for our intervention provides a highlevel overview of the following structural features of a ToI that include subcomponents, Agents, their actions, and the anticipated/desired outcomes for each component of CAPE. For the intervention we studied, some of the relevant subcomponents of the intervention included (but were not limited to) the Strategic CSforALL Planning Tool (SCRIPT) workshops (for capacity), CS/Cyber courses (for access), student recruitment strategies (for participation), and student counseling sessions (for experience).

Step 2: Identify evidence. Associated with each subcomponent are agents involved in the process of developing or using subcomponents. Examples of agents in the intervention include school administrators and curriculum designers (capacity), CS/Cyber instructors (access), families and community members (participation), and JROTC instructors, and cadets (experience). Agents are involved in actions of various kinds that constitute the mechanisms through which impacts are produced. The impacts of actions then unfold over time and directly or indirectly produce outcomes.

4. Results: Testing the ToI-CAPE Model against the JROTC-CS Intervention

This section provides a descriptive summary of each CAPE component. Within each, we provide a general description of the development process. We also include for each important items to consider when using the model for other interventions. We present the assessment of our feasibility study in the discussion section.

4.1. Capacity

Capacity pertains to a school’s ability to offer equity-focused training and resources for implementing CS/Cyber courses—essentially, the extent to which a school can be effectively prepared to implement equitable CS/Cyber courses and ECAs. Here, we provide a breakdown of the capacity component of CAPE in terms of its subcomponents, agents, actions, and outcomes, methods and evidence, and aspects related to data collection, including instrumentation and a timeline for collecting the relevant data.

4.1.1. Capacity Subcomponents, Agents, Actions, and Outcomes

Step 1 of component deconstruction for capacity is shown in Figure 4. To create this, we considered the known subcomponents for building capacity, including the CSforALL SCRIPT workshops, professional development activities, and regularly scheduled webinars.

Capacity agents were those entities or individuals involved with facilitating and/or engaging in those subcomponents, including the AFJROTC headquarters, administrators, teachers, AFJROTC instructors, and school guidance counselors. Capacity actions were those activities that the agents engaged in, including the process of defining community values and identifying ways to overcome their challenges in bringing an equitable CS/Cyber education to their schools. Capacity outcomes are focused on four groups of capacity actors, including administrators/principals, CS/Cyber teachers, JROTC instructors, and school guidance counselors. These were selected due to each actor’s agency in building capacity for their school.

4.1.2. Capacity Impacts, Methods, and Evidence

In Step 2, we first hypothesized several impacts for Capacity by linking relevant subcomponents, agents, actions, and outcomes. Figure 5 shows two of these hypotheses:

• Workshops are used by the project director and trained facilitators to help school principals and administrators increase their understanding of the value of equitable CS/Cyber courses and extracurricular offerings (ECAs), and plan and implement equitable CS/Cyber courses and ECAs.
• Teacher PD is used by independent PD providers and school leadership to help teachers feel more confident in implementing equity-focused instructional practices in basic and/or advanced CS/Cyber courses, and equity-focused strategies for advising students about CS/Cyber courses and ECAs.

Figure 5 also shows the methods and evidence we used to investigate our hypotheses for capacity impacts. For methods, we identified the data to be collected and when it will be collected, as well as the evidence or indicators that will be used to determine the extent to which the hypothesized capacity impacts are being achieved.

4.1.3. Capacity Considerations during Application of the Model

Capacity is a critical subcomponent and as such it requires care. When creating this model, we developed our access and participation hypotheses and evidence first, since these are subcomponents with more limited scope (by definition). Then we returned to capacity to define all of the moving pieces of the intervention and form the hypotheses.

Capacity subcomponents can include items related to:

• Human Resources.
• Funding.
• Policies.
• Standards.
• School environment.
• Community/parental involvement.

Working closely with the intervention specialists, we asked many questions related to each of the pieces of the intervention (subcomponents), who was delivering those pieces, what activities comprised those actions, and what they expected to achieve.

When using this model for capacity for studying a new intervention, we recommend focusing on those aspects of the intervention in capacity that will change as a direct or indirect impact of the intervention. We do not delineate between the two in our example since in this case they are all direct impacts within the intervention.

The outcomes are generated during discussions with the intervention specialists who are designing and implementing it. There are many outcomes that could be measured, so we recommend considering time and scope of the project and priorities of the specialists are imperative to ensuring mission creep is contained. We also recommend that each outcome be situated with equity to honor the intent of CAPE.

4.2. Access

Access pertains to students’ access to courses—essentially, whether or not courses are offered by a school.

4.2.1. Access Subcomponents, Agents, Actions, and Outcomes

Step 1 of component deconstruction for Access are shown in Figure 6. For this, we considered the known subcomponents for building capacity that are part of the intervention, including the CSforALL SCRIPT workshops, professional development activities, and regularly scheduled webinars. Access agents were those entities or individuals involved with facilitating and/or engaging in those subcomponents, including the JROTC Air Force headquarters, administrators, teachers, JROTC instructors, and school guidance counselors. Access actions were those activities that the agents engaged in, including the process of defining community values and identifying ways to overcome their challenges in bringing an equitable CS/Cyber education to their schools. Access outcomes are focused on four groups of capacity actors, including administrators/principals, CS/Cyber teachers, JROTC instructors, and school guidance counselors. These were selected due to each actor’s agency in building capacity for their school.

4.2.2. Access Impacts, Methods, and Evidence

For conducting Step 2, we hypothesized only one access impact by linking relevant subcomponents, agents, actions, and outcomes:

• The CS/Cyber course and ECA offering plan is used by school leadership and ECA coordinators to increase the number of CS/Cyber courses and ECAs offered to all students so that they are equitable across schools.

Figure 7 shows the methods and evidence we are using to investigate our hypotheses regarding access impacts. For methods, we identified the data to be collected, when it will be collected, and the evidence or indicators that will be used to determine the extent to which the hypothesized access impacts are being achieved.

4.2.3. Access Considerations during Application of the Model

When proceeding through steps 1 and 2 for access, we recommend considering the various forms of access for CS education. Access can include items such as curriculum offerings, school-based extracurricular activities, and community-based activities. Access is typically measured by the types of activities and the number (quantity) of offerings of these activities. Evidence will then focus on collecting data about these for the intervention. Equity is embedded to ensure that the quantity is equitable with other variables (e.g., student or school demographics).

4.3. Participation

Step 1 of decomposition of the participation component pertains to students’ awareness of and enrollment in courses and AP exams—essentially, whether students are aware of the courses and ECAs offered by their school and the extent to which they enroll in those courses and exams. Equitable participation pertains to the diversity of student enrollment in courses and exams matching the diversity of the host schools and being equitable across subgroups.

4.3.1. Participation Subcomponents, Agents, Actions, and Outcomes

Figure 8 shows the subcomponents, agents, actions, and outcomes we specified for the participation part of the CAPE ecosystem. To accomplish this, we considered the known subcomponents for building participation that are part of the intervention, including student recruitment and retention strategies, student counseling sessions, CS/Cyber course curriculum, and CS/Cyber ECAs.

Participation agents are those entities or individuals involved with facilitating and/or engaging in those subcomponents, including school guidance counselors, JROTC leaders and instructors, CS/Cyber teachers, the project director, community members, parents/guardians, CS/Cyber industry representatives, and extracurricular activity providers and coordinators. Participation actions are those activities that the agents engaged in, including implementing a recruitment and retention plan, implementing student counseling sessions, implementing CS/Cyber curricula and ECAs, developing school-industry partnerships, and codesigning and disseminating public relations and media outreach with community members and parents/guardians/families.

Participation outcomes are focused on the students and include increasing awareness of CS/Cyber and ECAs and improving diversity of student enrollment in CS/Cyber courses and ECAs.

4.3.2. Participation Impacts, Methods, and Evidence

For conducting Step 2, we hypothesized several impacts for participation by linking relevant subcomponents, agents, actions, and outcomes. For brevity, we present only one here:

• Counseling sessions are used by school guidance counselors to increase cadet and student awareness of CS/Cyber courses and ECAs so that it is equitable across schools.

Figure 9 shows the methods and evidence we are using to investigate our hypotheses regarding participation impacts. For methods, we identified the data to be collected, when it will be collected, and the evidence or indicators that will be used to determine the extent to which the hypothesized participation impacts are being achieved.

4.3.3. Participation Considerations during Application of the Model

Participation can include items such as student participation in curriculum offerings, school-based extracurricular activities, and community-based activities. Participation typically measured by student enrollment (number) of offerings of these activities and the demographics of the students, particularly when compared to the students who could potentially participate who do not. We recommend considering both the student who ultimately participate in the intervention as well as those who do not—particularly for interventions in which the students have choice in participating in it and how students are potentially recruited. Evidence will then focus on collecting data for the intervention.

4.4. Experience

Step 1 of decomposition of the experience component pertains to student outcomes, including the extent to which students are impacted student cognitive and noncognitive outcomes, as well as their interest in attending college and awareness of career options. Equitable experience pertains to cognitive and noncognitive learning gains, as well as college and career interest and awareness, being equitable across student subgroups.

4.4.1. Experience Components, Agents, Actions, and Outcomes

Figure 10 shows the subcomponents, agents, actions, and outcomes we specified for the experience part of the CAPE ecosystem. To accomplish this, we considered the known subcomponents for building experience that are part of the intervention, including CS/Cyber course curricula and ECAs, student academic and career counseling sessions, and school–industry mentorships.

Experience agents are those entities or individuals involved with facilitating and/or engaging in those subcomponents, including students, JROTC cadets and instructors, CS/Cyber teachers, school guidance counselors, and ECA providers and coordinators. Experience actions are those activities that the agents engage in, including implementing student counseling sessions, CS/Cyber courses and ECAs, and school–industry partnerships.

Experience outcomes are focused on students and include increasing and equitable cognitive and noncognitive performance, as well as increased interest and awareness in post-high school CS/Cyber academic and career opportunities.

4.4.2. Experience Impacts, Methods, and Evidence

For conducting Step 2, we hypothesized several impacts for experience by linking relevant subcomponents, agents, actions, and outcomes. We present only one of these hypotheses here:

• CS/Cyber curricula are used by JROTC instructors and CS/Cyber teachers to improve cadets’ and students’ CS/Cyber knowledge and skills, and self-efficacy, sense of identity, and engagement so that it is equitable across student subgroups.

Figure 11 shows the methods and evidence we are using to investigate our hypotheses regarding experience impacts. For methods, we identified what data to be collected and when it will be collected, as well as the evidence or indicators that will be used to determine the extent to which the hypothesized experience impacts are being achieved.

4.4.3. Experience Considerations during Application of the Model

Experience focuses on student outcomes such as student engagement in computer science learning, student interest in and career awareness of computer science, content knowledge, learning strategies, and social–familial influences. Equitable outcomes are important to understand how the intervention impacted different students and/or general outcomes from different schools. We recommend embedding the outcomes into the hypotheses, evidence, and indicators to ensure that appropriate data are collected to measure whether outcomes are equitable.

When establishing the hypotheses for experience, similar to capacity, we recommend carefully considering scope and align it with the time and resources allotted for the study. Although many aspects can be measured, we recommend focusing in on the most critical aspects of the intervention that is under investigation. We also recommend investigating embedding equitable outcomes.

5. Discussion

Our findings for each of the implementation and practicality hypotheses are presented in

Table 2. Implementation focus area outcomes, hypotheses, and findings.

Outcomes of Interest	Hypotheses	Findings
1. Types of resources needed to implement ToI-CAPE	H1. Personnel, Equipment, Software	The personnel were key in using their expertise in logic models and evaluation research studies to create the ToI-CAPE model and to implement it. No additional equipment or software was needed.
2. Amount of each resource needed to implement ToI-CAPE	H2. 3–6 months, 2–4 researchers, computers, software for communication, analyses	The two key personnel spent approximately 4–6 months part-time on building and implementing the model. In addition, the broader team of 3–4 people spend time reflecting on the work and providing valuable, iterative feedback.
3. Degree of difficulty in implementing ToI-CAPE	H3. Challenges in understanding the intervention, decision-making around research scope	Since we worked closely with the intervention team, we were able to understand the intervention as we went through each component. The broader team clarified our understanding through the iterative feedback process.

and

Table 3. Practicality focus area outcomes, hypotheses, and findings.

Outcomes of Interest	Hypotheses	Findings
4. Efficiency of implementation (does the quality of the product(s) justify the time taken by the approach?)	H4. Using ToI-CAPE means slower execution (initially), but higher quality product	The process to develop the model was intertwined with using it to create our evaluation research plan. Our product is comprehensive and for our particular scope and needs, it was highly efficient.
5. Effects of using ToI-CAPE (positive and negative)	H5. Positive: gain a clear understanding of system components and their impacts, maintain focus on equity; Negative: initially complex	Although the process of creating and implementing the model felt tedious as we went through it, we centered equity in each component and the outcomes and, despite the complexities, we gained a very thorough understanding of all nuances of the intervention (as well as their interdependencies).
6. Degree of ability to administer ToI-CAPE in evaluation research or other complex systems research	H6. Possible	We learned that it is possible to administer the ToI-CAPE model in our research.
7. Cost analyses	H7. Reasonable expenditures on personnel and resources	For our particular needs in a multi-year study, the cost of building this comprehensive logic model and using it in our research design was cost-effective.

and discussed in this section. Overall, our hypotheses were met in terms of our estimates on resources, effects of the process, efficiency, and costs.

Overall, leveraging the CAPE framework revealed important components across the JROTC-CS intervention. Using the ToI-CAPE model allowed us to explicate the qualities and characteristics of successful and unsuccessful components within the system, and chart out the relationships between them—all through an equity-focused lens.

5.1. H1 and H2: Resources Needed: Time, Expertise, and Tools

On our team, two researchers with multiple years of experience created the ToI-CAPE model and applied it to the Intervention. The ongoing cycle of discussion, reflection, and review with three members of the broader team continued throughout the four months, which provided time and space for deeper reflection on what we were producing and how we were producing it. Members of the broader team were more closely situated to the implementation of the intervention, which provided opportunities for us to clarify our understanding of the intervention on a continual basis. As typical in these projects, the team used shared digital files and folders to develop the model. No additional equipment or software was needed.

5.2. H3: Degree of Difficulty: Anticipated and Unanticipated Challenges

We chose one appropriately scoped intervention to evaluate the ToI-CAPE model and we recommend other researchers weigh this consideration when implementing ToI-CAPE in their work. Because of appropriate scope, most challenges we faced were anticipated. One of these was developing ToI-CAPE as we simultaneously gained a deeper understanding of the intervention. As we gained more understanding, it became easier for us to visualize the flow of our process; however, we did not fully anticipate the difficult nature of identifying agents, actions, outcomes, and indicators for impacts. This was particularly true with respect to capacity, as there were many complex components in play across multiple school sites. Future research with ToI-CAPE should be conducted across varying scopes and settings to determine feasibility generalization.

5.3. H4: Efficiency: Quality Products Are Worth the Time

The ToI-CAPE model took several months for two researchers to complete, one more familiar with the intervention and the other more familiar with logic model development. At several times throughout the process, we solicited feedback from the larger JROTC-CS research team. The resulting model was worth the time and resources, and we conclude that the ToI-CAPE model is highly efficient.

5.4. H5: Effects: Tedious but Thorough

The positive effects of the ToI-CAPE model included the structured development of a blueprint for engaging in research to study the impacts of the intervention. This blueprint or model enabled us to gain an understanding of the complexities of the intervention across the various aspects of CAPE, including insight into the intervention’s effect on building capacity to offer CS and cybersecurity courses across the various schools. The ToI-CAPE model and outcomes also kept a critical focus on equity, which is the underpinning of our entire research project.

Although we found no negative effects, we acknowledge that engaging in this process can be tedious. We continually engaged in a cycle of discussion–reflection–review that repeated several times within each CAPE component, in part because the ToI-CAPE model was new, the CAPE framework was new, and the intervention was new. As each of these become more familiar to us and to other researchers and evaluators, use of ToI-CAPE to study equity-focused interventions will streamline.

5.5. H6: Administration: ToI-CAPE Is Feasible in Research

Since we engaged in developing the ToI-CAPE model, we know that the ability to use it is possible. We have provided detailed information in this article to enable others to administer it in studying interventions across multiple schools. Future research should also explore the ToI-CAPE model in smaller settings such as individual schools or CS classrooms.

5.6. H7: Cost: Reasonable Cost for a Comprehensive Model

Like any project, a cost analysis is mapped to the amount and type of resources needed to implement a given process. In the context of our research, we weighed the demand on resources against the value of the resulting ToI-CAPE model as it pertained to the goals of stakeholders. We found the process to be impactful and comprehensive at this early stage of use.

5.7. Reconstructing the ToI-CAPE Model

Throughout this process, we experienced the nonlinearity of engaging in ToI-CAPE. We revised the original model to reflect this nature of the process more accurately. When implementing the model, it is expected that researchers will gain a deep understanding of the intervention and its goals through a feedback cycle with the intervention specialists; therefore, we recommend that Step 1 be implemented for each CAPE component before proceeding to Step 2. It may be challenging to tease apart the actions, agents, and outcomes for each of the components, and defining these prior to moving to Step 2 will save time.

5.8. Limitations and Challenges

Limitations of this feasibility study include the fact that the ToI-CAPE model has been vetted in one central study, though several person-months were dedicated to this process and experienced researchers constructed the model. Future applications of the process will enable us to determine its broader applicability; however, as a logic model, we expect extensions and adaptations to the ToI-CAPE model as time progresses and based on needs of individual researchers and their projects. As the investigation of the intervention using this model progresses, we expect to learn more about how it can be potentially tweaked to further meet our needs.

Another limitation is the fact that the CAPE framework is still relatively new. As such, we relied on the framework’s authors to help us understand its nuances and were able to integrate that knowledge within the ToI-CAPE model. Though we were comprehensive in our use of the model for establishing our hypotheses and identifying what and when we should collect data, we are aware that capacity and experience have numerous factors attached to them that can influence success. In future research, we intend to provide a more comprehensive list of factors across each component that can ultimately influence student experiences.

6. Conclusions

Because of the nascence of the field of computer science education and thus computer science education research, few evaluative frameworks have themselves been assessed in these contexts. Drawing from previous research across the broader literature allowed us to expand our considerations of what to include in the ToI-CAPE model. This included: goal-setting with stakeholders, decisions about outcomes and how to assess outcomes, identification of all of the system components, and emphases on equity from start to finish.

Overall, this effort introduces the ToI-CAPE model focused on equitable CS education and the impacts of interventions that encompass each component of CAPE. This new model is important for researchers and evaluators who perform or want to understand how to structure and perform equity-focused research (including evaluation research) across complex systems that range from a school’s or school region’s capacity to offer computing related education to the students’ experiences engaging in that research.

The introduction of a new model to enable equity-focused research in CS education is timely, given the pressing need to reach all students with computing education in growing primary and secondary classrooms. CAPE’s equity focus has provided a formal way to center equity, and the addition of the ToI helps understand (in)equity across interventions and systems. Although the ToI-CAPE model is new, its grounding in extant theories and frameworks, displays the usefulness of an equity-focused tool in conducting research and evaluation of the impacts of CS education initiatives. We invite others to use the ToI-CAPE model and build upon this work to help ensure the focus of successful interventions includes equitable outcomes for all learners.