Shaping the Discourse around Quality EdTech in India: Including Contextualized and Evidence-Based Solutions in the Ecosystem

Abstract

This paper examines the extent to which an initiative in India, namely EdTech Tulna, has been able to move towards decolonization of EdTech by shaping the discourse around the adoption and use of good quality and contextual educational technology solutions for Indian learners. Set up as a collaboration among researchers, practitioners, teachers and governments, EdTech Tulna aims to encourage the selection of EdTech solutions that are appropriate for the community they are designed for, rather than adopting solutions that market themselves or those that have been successful in Western countries. The paper adopts the lens of justice-oriented design and first critically examines the design of the EdTech Tulna index. Then, it examines the success and hurdles of the collaborative efforts towards the implementation of contextualized and evidence-based solutions in the ecosystem. By analyzing stakeholder interviews and meeting notes, this paper addresses two questions. First, how does Tulna assist in identifying quality contextual solutions that are likely to enhance the learning of children in India? Second, how do state government officials and practitioners collaborate with researchers to use research-based standards for selecting such solutions? The discussions outline the progress and draw a broad contour of the road ahead.

1. Introduction

Globally, the availability and usage of educational technology (EdTech) solutions in the school education space have surged in the last two decades [1]. This trend is seen in developing countries like India too, with the influx of EdTech solutions spanning a range in terms of age group, pedagogical design, technologies, cost and instructional setting [2]. In India, the dependence on EdTech increased after the school closures following the outbreak of COVID-19, with more parents from different socio-economic backgrounds investing in smartphones and digital devices than ever before [3]. In an era where consecutive yearly reports and research [4] have been documenting the fall in learning levels among Indian children [5], a new reliance has been developing on technology to bridge the learning gap [6]. While studies have documented a link between positive learning outcomes and the use of various technology-supported learning models in India (for example, [7,8,9]), there has been considerable evidence of no significant outcomes and negative outcomes as well [1]. At the same time, techno-optimism and techno-solutionism, which have increased in the post-pandemic era, are not without critique [10]. A significant challenge that has emerged, and is commonly faced by teachers, parents and state officials, is choosing the appropriate technology [11].

A large proportion of the EdTech solutions are paid and hence the access gap remains, potentially leading to the formation of a new form of education inequity. To address this gap, state governments in India provide not only hardware but also appropriate learning software [12]. However, the common approach for the selection and implementation of EdTech solutions in government schools can be critiqued as a manifestation of digital neocolonialism [13]. State governments procure ‘globally tested’ EdTech learning software, which is then mandated to be used in the local schools. Learners may become mere recipients of learning materials created for a different context, typically for learners in English-speaking Western countries, thereby erasing their identity [14]. Left unchecked, content can colonize education, and pedagogy that is not well-designed for the intended learners can significantly hinder learning [15,16].

As an approach to addressing the above challenges, we discuss a collaborative effort in India called EdTech Tulna [17], and examine the extent to which it has been able to move towards decolonizing EdTech in India. As part of the EdTech Tulna initiative (hereafter referred to as Tulna, which means to measure or judge in Hindi), researchers, non-governmental organizations, teachers and state governments in India are working together to come to a common understanding of what good quality EdTech means in their context, keeping the learning of children as the key consideration [18]. The Tulna initiative comprises subject-specific indexes aligned to the respective curricula. These indexes define quality standards relevant to the local context in India and evaluate EdTech products against these standards before use. In addition, Tulna indexes include explicit criteria to ensure that teachers are supported when they use the EdTech solutions. Tulna examines if the solutions under consideration are appropriate for the community for which they have been designed, thereby questioning the ‘core-to-periphery’ implementation in the EdTech domain [19]. Another goal of Tulna is to foster a collaborative decision-making process for the adoption of context-appropriate EdTech solutions. Tulna aims to empower end users such as teachers and governments with the knowledge and resources for making evidence-driven decisions.

In this paper, we adopt the justice-oriented design framework [20] to critically examine two research questions. The first research question is RQ1: How does Tulna assist in identifying quality contextual solutions that are likely to enhance the learning of children? This is addressed in Section 4, where we detail the Tulna design process and analyze selected criteria from the index from the justice-oriented design lens. The second research question is RQ2: How can Tulna be used to involve multiple relevant stakeholders in making decisions regarding the selection and use of contextual EdTech solutions? This is discussed in Section 5, where we report the findings of a qualitative study using thematic analysis to examine Tulna’s role in the EdTech procurement process of a state government.

At this stage, we briefly discuss our positionality as researchers and authors of this paper. We place ourselves on a range in the ‘insider-outsider’ continuum [21], as we have some similarities with other stakeholders in the ecosystem. In the process of designing the index, as insiders, we evaluated products, published reports and worked on building capacity among new evaluators and trainees. However, while supporting the state officials during EdTech procurement, we found ourselves as outsiders to the process of implementation. Our involvement was primarily focused on training the evaluators chosen by the state to interpret the index, whereas the state officials held power over the final decision-making. Our positionality and multiple identities have implications in the description of the findings and their interpretation.

2. School Education in India and the Role of EdTech

2.1. Background

The school education system in India is a concurrent subject; that is, both the central government and the state governments are responsible for enacting and implementing education policies. The Right of Children to Free and Compulsory Education Act, commonly known as the RTE Act of 2009, made elementary education, that is, Grades 1–8, compulsory for all children in India [22]. Due to concerted efforts by the central and state governments, India has achieved near-universal enrolment in school education, and students across socio-religious groups have seen an increase in school enrolment and attendance [5]. Public schools have become increasingly accessible across the states and there has been an increase in the presence of technology in schools [18]. Between 2007–2008 and 2017–2018, the percentage of children aged 6–14 years attending English medium schools increased from 12 to 23 at an all-India level [23]. The demand for EdTech for learning the English language has also risen in recent years [24]. While some scholars have argued that the spread of English is a linguistic imperialism that can erode countries’ culture and linguistic ecology [25], others have argued that the English language is an agent of decolonization that has allowed the urban poor to access opportunities present in the global economy [26]. Nonetheless, English EdTech has gained considerable popularity in the Indian context. A survey conducted in six states in India found that ‘English (84%) and Mathematics (76%) were the most studied subjects using EdTech tools’ [24].

2.2. Growth of EdTech in India

There has been a gradual emphasis in India on integrating educational technology in classrooms and at home. The National Educational Policy [27] focuses on catalyzing the use of technology for improving pedagogy and content at all levels of education, and the National Educational Technology Forum facilitates decision-making about the use of EdTech in educational institutions across the country. In parallel, non-governmental organizations focusing on educational policy and strategy have channelled their efforts to catalyze the demand and supply of quality EdTech solutions for various users [28]. Yet another dimension to this push for EdTech in India is from the private companies and startups that have mushroomed over the past few years in response to the rising demand. Relatively recently, researchers from a leading research institute in India have been actively working on shaping the decision-making process to ensure systematic and research-based design, evaluation and selection of EdTech solutions [9].

Earlier, the EdTech programs in government schools in India primarily focused on input parameters such as deploying hardware, installing software and conducting teacher training, and there was minimal focus on the quality of the software and its impact on learning [29]. There are only a few mechanisms in place (e.g., [30]) to check the suitability, usability and content accuracy of the quality of software. In many cases, this mode of the selection process for hardware in ICT procurement tends to prioritize the lowest-cost option that meets technical and financial requirements [29] without the active participation of important stakeholders, such as principals and teachers. Recently, there has been a shift towards adopting a more participatory approach in the implementation of EdTech which involves multiple stakeholders such as district officials, principals, teachers, EdTech providers, civil society organizations and academics [12,18].

2.3. The EdTech Tulna Initiative

The EdTech Tulna initiative [17] was seeded in March 2020 as a partnership between the Educational Technology interdisciplinary program at the Indian Institute of Technology (IIT) Bombay, a premier research institute in India, and the Central Square Foundation, a non-governmental organization (NGO) focusing on educational technology, policy and strategy. Over time, other stakeholders, such as state governments along with their governance consulting groups and teachers, came under the umbrella of this initiative [31]. The key goal of this initiative is to make quality a focus on both the supply and demand side of the EdTech ecosystem in India. It aims to reduce information asymmetry by building a shared understanding of what ‘good’ EdTech looks like. The members of Tulna engage in a range of activities such as establishing quality standards, designing reliable and valid evaluation instruments for evaluating EdTech products and generating evaluation reports for public view. Another major effort of Tulna is to support decision-makers in a variety of settings, such as in the procurement of quality EdTech products by state governments for large-scale deployment in schools or the selection of specific EdTech products by teachers for their local needs. Tulna members also extend training and capacity building related to identifying good quality EdTech products and engaging with international agencies for knowledge sharing.

3. Conceptual Framework

We adopt the justice-oriented learning design framework [20] to critically evaluate how the design of the Tulna index and implementation process can deconstruct established power structures and reshape notions of expertise. The justice-oriented framework underscores the idea that addressing injustice does not have a universal solution. It introduces three major approaches: Justice-as-content, Justice-as-pedagogy, and Justice-as-process [20]. In the Justice-as-content approach, the emphasis is on decolonizing the content of education, or what is taught, and ensuring it is relevant and contextual based on the target audience and intended purpose. It aims to shift from traditional, potentially biased perspectives and embrace a more inclusive and diverse educational narrative. Justice-as-pedagogy involves encouraging learners to critically engage with, reflect on and challenge the content they are being taught. Justice-as-process revolves around decolonizing the actual educational processes and ensuring diversity or plurality of stakeholders and ideas in the decision-making process both in the design and implementation of EdTech. By doing so, the aim is to break down existing power structures, fostering a collaborative and inclusive approach to shaping educational technology.

In our analysis, we use the Justice-as-content and Justice-as-process aspects. Utilizing the Justice-as-content approach of the framework, we closely examine some of the criteria under content quality and pedagogical alignment within the Tulna index to analyze how the criteria actively contribute to the essential task of decolonizing EdTech (Section 4). Focusing on the implementation or adoption within a state government (Section 5), we bring in the Justice-as-process approach to the analysis. This enables us to examine how a variety of stakeholders and a range of ideas have collectively contributed to the decision-making process. This in-depth examination sheds light on the collaborative and inclusive strategy employed throughout implementation, emphasizing the significance of integrating diverse perspectives in decolonizing the EdTech implementation and adoption process.

4. Justice-as-Content: The Contextual Relevance of the EdTech Tulna Index

In this section, we address RQ1, examining how Tulna facilitates the identification of quality contextual solutions that are likely to enhance the learning of children. This section begins with an overview of Tulna’s design process, leading to an analysis of select criteria from the index. This analysis is framed within the framework of ‘Justice-as-content’, providing an examination of the contextual relevance for learners.

4.1. Tulna Index Design Process

The Tulna index design is rooted in research from the learning sciences and has been adapted for Indian learners through input from teachers and user studies in classrooms. Figure 1 schematically illustrates the basis of the Tulna index. Overall, 150+ research articles have been reviewed to iteratively formulate the criteria in the Tulna index.

Theoretical frameworks related to the use of technology in education, such as meaningful learning with ICT [32] and Technological Pedagogical Content Knowledge (TPACK) [33], as well as broader educational design principles such as constructive alignment [34], have informed the Tulna index design. Well-established pedagogical strategies such as formative assessment and constructive feedback [35], scaffolding [36] and the importance of situatedness in learning [37] have strongly influenced criteria in the pedagogical alignment dimension. The Technology and Design dimension includes criteria such as principles for user interface design [38] and universal design for learning [39]. In addition, domain-specific literature has been reviewed in detail to reflect the nuanced differences in learning different subjects (see, for example, [40], in which we describe the criteria in the index for English language learning Edtech products for Indian learners).

Two key policy frameworks by the Indian government have contributed significantly to the Tulna index: the National Curriculum Framework 2005 [41] and the more recent National Education Policy [27]. Both of these policy documents provide detailed guidelines and analysis on what should be taught to children in India and how it should be taught. Sufficient prominence is given to educational technologies while keeping the teacher in a central role. Diversity and plurality of various types are recognized and valued while emphasizing the role of the mother tongue and children’s socio-cultural backgrounds.

During the initial design phase of the Tulna index, we conducted a needs analysis study with the objective of gathering requirements from diverse stakeholder groups. This study utilized a semi-structured interview approach, conducted for approximately one hour each, via telephone or video conferencing. The study consisted of nine participants representing various stakeholder categories, including consultants who advised the government on EdTech adoption, school principals and parents of children in classes 6–10. Thematic persona analysis of the interview data revealed three distinct personas: Enterprise User, Local Community User, and Private User, each with unique needs and priorities regarding EdTech adoption. The findings highlighted stakeholders’ demand for a reliable evaluation index and product reviews. They also emphasized the importance of contextualization in EdTech design, urging the inclusion of criteria that assess how well products integrate with the local context. Moreover, insights gained from stakeholder interviews played a crucial role in informing the design of the public portal, focusing on features such as product categorization, sorting, filtering and access to detailed evaluation reports. Additionally, as part of the Tulna index refining process, user studies were also conducted with students and teachers in government schools across two states to further elucidate the contextual parameters influencing the quality of personalized EdTech learning solutions [42].

4.2. Evaluation Parameters in Tulna Index

The Tulna index is classified into three dimensions—Content Quality, Pedagogical alignment and Technology and Design. The Content Quality dimension focuses on maintaining accurate, relevant and inclusive content. Evaluation criteria within this dimension include content accuracy, correctness and clarity in assessment, language comprehensibility, alignment to national standards, curriculum alignment, inclusivity in learner representation and bilingual use. The Pedagogical Alignment dimension emphasizes the use of effective teaching strategies informed by educational research and policies and aligned to learning objectives. Evaluation criteria within this dimension include the constructivist approach, addressing alternate conceptions, content in context, learner scaffolding, cognitive engagement, motivational features, logical chunking and connectedness, learning objective alignment, pedagogy-assessment method alignment, cognitive levels, feedback quality, opportunities for collaboration, adaptivity and teacher support. In the Technology and Design dimension, the benchmarks revolve around seamlessly integrating technological features and user-friendly interface design to enhance the overall learning experience. Evaluation criteria within this dimension include interface design, learner navigation and pace, universal design, analytics for learners’ progress, tools to support problem-solving and meaningful interactivity. A description of the evaluation parameters is available on the Tulna webpage [17].

Each criterion serves as a guide to evaluate EdTech solutions on a three-point scale: “potential to improve”, “valuable” and “exemplary”, along with detailed descriptors. The available EdTech solutions in the ecosystem are evaluated using this set of criteria to determine their quality.

4.3. Analyzing Tulna Criteria Based on Justice-as-Content Framework

The following section presents an analysis of some criteria from the index that aim to ensure the content’s contextual relevance for learners, aligning with the concept of ‘Justice-as-content’ as discussed in Adam’s [20] research.

4.3.1. Content Quality Dimension

The dimension of Content Quality checks several features of EdTech solutions related to the content; for instance, the accuracy of the content, the vocabulary and accent, coverage of age-appropriate skills and representation of sections of society. We draw on three criteria under this dimension and outline the details of the criteria that specifically focus on how the EdTech solutions are suitable for learners in India.

• Language comprehensibility: Use an easily understandable vocabulary and accent, keeping the intended learners in mind.
  This criterion checks whether the accent and the vocabulary used in the EdTech product are likely to be comprehended by the target learners. In our context, it checks for an Indian accent, even if the content is presented in English. A foreign-accented voice adds to the cognitive load of the learners [43]. A product is considered to be exemplary when the learners are likely to follow the accent without additional effort and the vocabulary is age-appropriate for the learners.
• Bilingual use: Use English technical terms as well as vernacular terms to present mathematical terms so that the learners become well-acquainted with the language of Mathematics.
  A dominant language of EdTech solutions in India is English. However, having all solutions in English is not an ideal case for the learners, many of whom do not speak English as a first or even a second language. The Tulna index encourages EdTech product designers to make their products available in multiple native languages, or at least to support the learners by providing scaffolds in the relevant native language. The presence of multiple languages in EdTech solutions would not only help learners stay engaged and obtain the support that they need but would also avoid their loss of identity from being exposed to content in a non-native language.
• Inclusivity in the representation of the learners: Address the diversity of target learners in terms of gender, race, socio-economic background, religion and appearance while creating content.
  EdTech solutions often are designed to include characters of fair skin and certain body types, which misrepresents the heterogeneity in Indian society. This criterion encourages the inclusion of individuals from different sub-sections of society in terms of body types, age, gender and ability, as well as clothes and accessories reflecting religion that an Indian learner is likely to observe around them. Products that are built for different country contexts and overrepresent fair-skinned people or support stereotypes are penalized under this criterion.

4.3.2. Pedagogical Alignment Dimension

The criteria under the Pedagogical Alignment dimension focus on how the content is presented to the learners and the pedagogical practices followed by the EdTech products. The following criteria show how the content is evaluated for contextual solutions that are rooted in practices that are common to the cultural context of India.

• Content in context: Pay close attention to the learner’s context (who is learning) and location (where is the learning taking place) while designing pedagogy.
  The Tulna index checks whether the product design is rooted in the local and cultural context of the learners. This can be represented in terms of the choice of clothes, food, festivals or setting, to name a few. Products that are directly adopted from Western contexts or that are not designed for Indian learners are unlikely to accurately represent the needs of the intended learners and are penalized.
• Teacher support: Design supports for the teacher so that they know how to use the product meaningfully and can customize it to an extent in response to learners’ needs on the ground.
  Proficiency in the use of digital devices cannot be expected from all teachers, many of whom have been exposed to technology only in their adulthood. Thus, presenting an EdTech product without any support and guidance is unlikely to be successful in the classroom. This criterion checks whether the product design considers teachers as central agents of teaching. It focuses on the support provided to teachers on using the product effectively by integrating it with classroom teaching, and whether the teachers’ agency is valued.
• Opportunities for collaboration: Facilitate collaboration and scaffold learning via peer-to-peer interaction and feedback.
  Western-centric epistemological and pedagogical foundations in EdTech tend to prioritize individual learning paths and goals over emphasizing collective learning paths [44]. However, learning in small groups is more beneficial than individual learning [45] and Indian culture is deeply embedded in collectivism rather than individualism. Under this criterion, the products are penalized if they do not encourage group-based learning, especially the ones that are designed for classroom interventions.

5. Justice-as-Process: The Adoption Process of EdTech Tulna Index by an Indian State Government

This section delves into RQ2, investigating how Tulna facilitates the engagement of multiple stakeholders in decision-making concerning the selection and implementation of contextual EdTech solutions. We analyze a qualitative study examining Tulna’s role in the procurement of EdTech, specifically its integration within a state government. By employing a thematic analysis through the lens of ‘Justice-as-process’, we explore how the involvement of different stakeholders and their varied perspectives collectively influence the decision-making process.

5.1. Methods

5.1.1. Participants

The participants involved in this study encompass a diverse group with distinct roles and expertise. The researchers, who are the authors of this paper, primarily focused on developing standards and evaluation instruments, designing evaluation processes and conducting research on evaluating the quality of EdTech products. While they possess prior experience in technology in education, their knowledge of classroom teaching is limited. The government evaluators (also referred to as practitioner evaluators) are government school teachers (N = 37) who bring extensive classroom teaching experience and some familiarity with technology. However, their exposure to evaluating EdTech products is limited. Purposive sampling was used to conduct interviews of the members of the NGO (N = 2) and members from the governance consulting group (N = 4). The members of the NGO have expertise in implementing EdTech solutions to enhance learning outcomes on a larger scale. The governance consulting group members are responsible for assisting the state government in making informed decisions during the adoption process. Lastly, government officials, holding senior bureaucratic positions, serve as key decision-makers in the state.

5.1.2. Procedure

This study delves into the adoption process of an evaluation instrument by state governments in India, focusing on designing spaces for deliberation in implementing Tulna. Figure 2 outlines the various stages of this adoption process and the stakeholders involved at each stage. Stages 1–3 entail weekly meetings between researchers, monthly meetings with the NGO partner, and regular engagement with government officials and the governance group. Stage 4, a closed-door event, involves confidential government decision-making. Note that the stages are merely logical groupings for ease of understanding of an otherwise fluid and continuous interaction that happened between the specified stakeholders between January 2021 to September 2022. Utilizing multiple vignettes, the community’s functioning during this process is analyzed.

5.1.3. Data Collection and Analysis

Data for this study comprise multiple sources, including internal meeting notes, participant observations, artefacts generated by the community, survey questionnaire responses from government teachers after they completed the evaluation process, semi-structured interviews with various stakeholders and post-evaluation reflections. Of these, interview data are the primary focus, supplemented by additional sources for corroboration or counter-analysis.

The goal of the survey questionnaire was to understand teachers’ experiences in using EdTech products and evaluate them with the Tulna index. It contained questions on a Likert scale as well as open-ended questions. In addition to participants’ prior experience with EdTech products, the survey sought teachers’ feedback about the effectiveness of the training structure, ease of understanding the index and challenges faced during evaluation. The survey also sought to understand changes in participants’ perceptions of EdTech quality and their willingness to use the Tulna index in the future.

Stakeholder interviews were conducted virtually in English or Hindi, recorded and transcribed. Sample interview questions for the evaluators included: ‘Describe your experience using the Tulna index’, ‘What questions arose during the evaluation process? How did you find solutions for those?’ Questions for the governance consulting group members include: ‘Did the initial discussions consider software quality or hardware?’, ‘When and how did the conversation about quality start?’ and ‘What role did Tulna play in the adoption process?’ The responses were coded to include instances where stakeholders mentioned their experience of using the Tulna index, collaborating with other stakeholders in the process and the significance of language in this process. The coding process involved the first author primarily, with verification from other authors, resolving disagreements through discussion and triangulation with multiple data sources.

5.2. Findings

5.2.1. Vignette 1: Multi-Voiced Approach to a Fair and Just Evaluation

This vignette analyzes to what extent the multi-voicedness and cross-sectoral collaborative approach has been able to facilitate justice in the process by aiding decentralized decision-making to procure EdTech products at the state level. Overall, Tulna acted as an anchor that aided decentralized decision-making by providing the metrics and toolkits for evaluating the quality of EdTech solutions and developing a common language related to the quality of EdTech solutions. The evaluation panel consisted of government school teachers and district-level officials who were subject-matter experts. The final scores were consolidated and the decisions were made by the state-level officials.

As a collective, all the members of the community focused on helping decision-makers select the best EdTech product among the available ones in the ecosystem. Although the state officials made the final decision regarding the approval of the final index and the choice of the EdTech product by aggregating scores across the selection procedure, the selection procedure had multiple steps. In each of the steps, multiple stakeholders—the researchers, governance consulting group members and teachers—came together to support the process with their wide range of expertise.

There was considerable emphasis on involving teachers and bureaucrats in the state in the technical evaluation process.

Members from the governance consulting group (GCG) were situated in the context of understanding current trends in technology use for classroom education. They had extensive knowledge of solutions available in the market but had limited access to education research. The role of Tulna team members was to establish a common language among the stakeholders to help them understand how to measure the quality of EdTech solutions. This was done through successive meetings and by explaining the detailed index to the stakeholders. The language in the index bridged the gap between the researchers, practitioners, teachers and state officials.

Mr. A (GCG): “Quality of PAL [Personalized Adaptive Learning] was a critical thing in our mind. Of course, our understanding of how this is measured was very, very limited because we only knew certain private players who were making noise in the market at the time… we understood that there were some good players in the market who delivered PAL but we actually had no idea of how exactly this was measured and how quality of a PAL product was measured.”

For the government teachers who had extensive knowledge of pedagogical practices related to the classroom, the Tulna index gave them a way to channel their experience towards the selection and use of EdTech products.

Teacher S: “Personally it gave me a systematic way of judging something. Creating parameters, taking into all the aspects or factors of judging a thing, and doing it in a good way, you get an overall picture of the thing.”

5.2.2. Vignette 2: Empowering the Teachers with Decolonized Content and Training

This vignette presents further evidence of the decolonized evaluation process but also highlights how the teachers were empowered to inform the process with the help of decolonized content. Public school teachers were invited by the state government bureaucrats to serve as state evaluators based on their experience and familiarity with the use of EdTech in their classrooms. For these teachers, the onboarding process comprised a series of training workshops. The online training, conducted over three days by researchers, aimed to improve participants’ proficiency in using and interpreting the Tulna index. Specifically focusing on three critical dimensions of the index—Content Quality, Pedagogical Alignment and Technology and Design—the sessions provided comprehensive explanations of the criteria within each dimension, complemented by examples. The training sessions were mostly kept bilingual, acknowledging the linguistic backgrounds of the participants. They were structured to be concise yet interactive, fostering active engagement among participants. This format encouraged attendees to actively participate and prompted them to seek clarification on evaluation criteria whenever necessary. Following the online training, teachers engaged in independent evaluations, using the Tulna index to assess EdTech solutions.

Ms. B (GCG): “The role that Tulna played was to build the capacity of multiple stakeholders, all who are part of this decision-making process, at various stages. It builds the capacity and understanding of what good can look like around software and their confidence to make high stakes decisions…build capacity of multiple stakeholders.”

During these workshops, the expert evaluators leveraged the expertise of the governance consulting group to customize the workshops to address the needs of the state and the teachers. The governance consulting group members were trusted by government officials, and they supported this training process through various means, such as translating or explaining the scientific terms in the regional language, making sense of the evaluation training and interpreting the criteria in the evaluation instrument. Engaging in this process helped in the enculturation of the governance consulting group members and teachers to the accurate use of the evaluation instrument. These workshops also helped the research team become encultured in classroom practices and challenges faced by the practitioners, thus enabling iterative refinement of the training process.

Hindi was the native language of the state where the adoption process was being held. Hence, the training was bilingual (English and Hindi) to help teachers understand the evaluation process and criteria easily. While the teachers understood English, they were more comfortable in their regional language. However, the evaluation index itself was in English. This was mainly driven by the researchers’ positionality, all of whom hailed from different parts of India and spoke English as a common language, and the major language of operation in the research institute was English.

Ms. AK (NGO): “…one point of difficulty was… understanding the language of the framework itself…”

After the training, the teachers evaluated the competing products for state adoption using the index provided by the researchers. At this stage, they operated independently and were supported by the research team only in terms of interpretation of the index. The researchers themselves did not see the EdTech products to avoid biasing the scores of the evaluators. The evaluators were provided with the detailed evaluation instrument on the first day of the evaluation. They then engaged in understanding and assimilating the evaluation criteria before using them to evaluate the EdTech products given to them. They followed the rules and evaluation norms set by the government bureaucrats. Debates and discussions were encouraged before the formal evaluations started and were discouraged during the evaluation process to avoid biasing each other’s evaluation scores. The emphasis was on independent evaluations. There was also a strong emphasis on making all voices heard during the evaluation process and engaging in a process that was fair, transparent, confidential and rigorous to the extent possible given the practical constraints.

Ms. R (NGO): “The idea was the evaluation team would do the heavy lifting of all of these evaluations, analysis and all of that. And the high-powered committee would take the final call. Which is basically a very rigorous process. It allows stakeholders from teachers to senior bureaucrats in the states to get involved and it does it in a way that the louder voice, the more powerful voice is not of state or not amplified because of their position. Rather, everyone gets an equal voice in the process at some stage, right, and gets aggregated as you move forward.”

Thus, there was an emphasis on including different stakeholders, undoing power structures in the process of decentralized decision-making and enabling teachers to have agency and training to exercise that agency during the evaluation process. We believe this is critical because then the teachers are directly participating and influencing the selection of EdTech products for their classroom at the government level.

We acknowledge that some shortcomings still exist in this process. As mentioned before, the researchers’ primary language of communication and work is English and the index, too, was designed in English (later translated into Hindi). This may have reinforced power structures, positioning stakeholders as having different proficiency in English at different levels of the power hierarchy. The bilingual training was designed to mitigate this effect and aid better comprehension and discussion, but its usefulness may have been limited. Further, the researchers provided training, governance consulting group members supported the implementation process and the evaluators independently evaluated the competing products. However, the final decision to select one EdTech product for the entire state was taken by the state officials. While we hope that the elaborate evaluations informed this decision-making process, the lack of publicly available documentation of the final decision-making process makes it hard to comment upon.

Nonetheless, since it would be the teachers who would be the end users of the EdTech in the classroom, their greater involvement helps amplify their voices. The entire process, followed by in-depth feedback from the participants and the stakeholders, helped the researchers to reflect on the justice embedded in the process and articulate steps for future collaborations. Further, as a spillover effect, and indicating the potential for long-term capacity building, the teachers reported that the training process and evaluation process increased their exposure to critically evaluating and using EdTech in their future pedagogical practices.

Teacher S: “Yes Now I am evaluating my own teaching and try to find out the points where I need improvement.”

Teacher P: “Earlier I used to focus only on content delivery but after training, I came to know about various pedagogical aspects which imparts an important role on learners’ ability.”

6. Discussion

To address RQ1, we analyzed select criteria from the Tulna index, focusing on the Content Quality and Pedagogical Alignment dimensions. The Tulna index has incorporated criteria, based on learning theories, national policies and a stakeholder analysis study, that enable the identification of culturally relevant, linguistically accessible and pedagogically aligned EdTech solutions for diverse learners in India. These criteria include language comprehensibility, bilingual use, inclusivity in learner representation, content in context, teacher support and opportunities for collaboration. This resonates with prior research stressing the importance of contextualized and culturally relevant educational content [43,46,47]. To address RQ2, we showed that the Tulna index promotes a shared understanding of quality in assessing EdTech solutions. Empowering teachers through training facilitates their active participation in decision-making. Prior research [48] highlights the importance of stakeholder involvement in enhancing the effectiveness and adoption of educational interventions.

One of the central limitations of this paper is that the authors represent only the researcher group in the community. The voices of other members were brought to the forefront with the help of the vignettes, but they were not active contributors to the interpretation of data and writing of this paper. The paper, at present, is written by the researchers themselves. The paper-writing process could also become part of the participation and reification process around which the community grows.

Second, the index is designed using international literature from peer-reviewed journals in English, and evidence from the local context, wherever available. In future, more research based on the local context and local languages is required, which can further inform the design of the index.

Third, considering the plurality in Indian languages as discussed earlier, the initial index was designed in English to be able to communicate with a wider group of stakeholders and state governments. The timeline of this study and the interviews of the stakeholders are based on the indexes that were designed in English, leading the government school teachers to face certain hurdles. More recently, there has been an effort to translate the index into local languages. In response to the requests from the teachers in the study, the evaluation index was recently translated into one of the dominant Indian languages (Hindi). However, this attempt has been very recent and is not captured in the dataset or analysis in this paper. Such initiatives, although desirable to reduce the cognitive burden of the teachers and enable equitable access to the research-based index for everyone, are challenging to execute in India, which has 23 official languages and 780 spoken languages. In the future, this is likely to reduce the barriers to decolonizing the process even further, but as for this paper, the language of the index remains a limitation.

7. Conclusions

In this paper, we outline the progress made toward encouraging the design, use and adoption of good-quality and contextual educational technology solutions in India through active collaboration among stakeholders. The discussion shows the usage and adoption of contextual solutions that are likely to enhance children’s learning. The discussion also highlights the challenges that the researchers had to navigate to ensure a collaborative approach while working with state officials. The overall discussion suggests that while there has been some progress made in the direction of decolonizing EdTech for Indian learners, there is more collaborative effort required in the future. Researchers, through their research, can define standards for contextual solutions. EdTech product designers need to consider that while designing their products, the states have to incentivize the designers by procuring only good quality and contextual solutions. Moving forward, besides using the global literature to gather evidence on good quality solutions, it would be of paramount importance to generate local evidence of what works, and under which conditions it works, in India.