Educational Technology Procurement at Canadian Colleges and Universities: An Environmental Scan

Abstract

There has been an increase in the use of education technology (EdTech) within post-secondary institutions, which has resulted in an unprecedented overflow of EdTech in the market. Institutions then make decisions on which EdTech to procure. This procurement process occurs on a continuum, where on one extreme, an institution takes a decentralized (bottom–up) approach where individuals within an institution independently decide on EdTech procurement, or a centralized (top–down) approach where the institution decides on criteria and standards that the EdTech must meet. This study administered a questionnaire and conducted structured interviews to explore how important standards are, and to identify the associated challenges with implementing centralized procurement. It was distributed to individuals involved in EdTech procurement at universities and colleges across Canada. The results showed that standards related to Privacy and Security, Accessibility, and Care of Data Practices play a larger role in EdTech procurement within most institutions. The use of standards is increasing as institutions become more centralized; however, they are not yet relied on in a structured way. This study suggests ways to move towards a procurement process that incorporates standards and addresses many of the identified challenges with procuring EdTech, thus, improving the efficiency and efficacy of EdTech procurement.

1. Introduction

In recent years, there has been a rise in the use of education technologies (EdTech) to improve student achievement [1,2].EdTech hub defines EdTech as “technologies—including hardware, software, and digital content—that are designed or adapted for educational purposes” [3]. This rise in the use of EdTech stresses the need for research focusing on how EdTech is procured within institutions [4]. Although there is abundant research studying EdTech decision making at the K-12 level [5], relatively little is known, however, about how universities and colleges procure EdTech. This is especially true for studies assessing EdTech procurement in post-secondary institutions in the Canadian context. As such, this study will focus on the EdTech procurement process within Canadian post-secondary institutions.

The unprecedented rise in the use of EdTech within post-secondary institutions has resulted in an overflowing list of EdTech that is available on the market. Various stakeholders within post-secondary institutions have to then decide which EdTech to procure [5,6]. This process is crucial, as it can affect the quality of the education students receive, can determine what they are taught, and how this information is provided to them [4,7].

Such procurement processes can occur on a continuum ranging from a decentralized (bottom–up) to a centralized (top–down) approach [5,8,9]. The decentralized approach is when EdTech procurement is independently decided at an individual or departmental level [5,8,9]. While the decentralized approach to EdTech procurement allows for quicker decisions and provides educators with more autonomy to make their choices, as this approach has fewer layers of bureaucracy, institutions are more at risk for security breaches, inconsistencies across departments, purchases that do not integrate well with the current technologies used, and duplicate purchases [5]. In contrast, a centralized approach is when the institution decides on learning priorities that the EdTech must meet [5,8,9]. The centralization of EdTech procurement allows for more clarity on what products are procured, the needs they meet, and how the procurement process is carried out [5,9]. An institution taking a centralized approach may require the EdTech they procure to meet specific standards, including those related to interoperability, accessibility, privacy, and security. Although centralized procurement is more time consuming, it increases the efficiency and effectiveness of the procurement process.

Within a non-profit or public institution, EdTech procurement is often decentralized, which may lead to technology proliferation problems at the cost of students [9,10]. When EdTech is procured in a centralized manner, institutions can better assess their spending, as it is easier to keep track of the EdTech being procured [5]. This can have long-term benefits and can reduce the costs associated with procurement. Alternatively, decentralized procurement is thought to improve adoption rates, as the end-users are involved in what is being purchased. However, this approach may lead to duplicate purchases or purchases that do not integrate with the current technology infrastructure of the school.

Findings from a qualitative study that interviewed university directors of learning technology or equivalents identified the need for the centralization and standardization of EdTech procurement, which is currently inadequate [11]. However, a shift to a centralized approach may be challenging, as the current governance processes for EdTech procurement have been described as lengthy and inefficient [11]. In a way, this makes sense when we view the university as a complex organization resembling “organized anarchies”, in which their associated characteristic properties may dictate how (i) choices such as EdTech procurement may often be made based on ill-defined and inconsistent goals, (ii) it may be challenging to procure EdTech that suits the differing interests of faculty, students, departments, and other stakeholders, and (iii) the decision makers for EdTech procurement may keep changing [12].

To address the gap in our understanding of the EdTech procurement process within the Canadian post-secondary context, the current study provides an environmental scan of the various standards relevant to the adoption of education technologies in post-secondary institutions across Canada. Using a mixed-method approach, we reached out to those who procure educational technologies in Canadian colleges and universities, asking them to participate in a questionnaire followed by an optional structured interview. The participants were asked questions related to the current standards adopted by the institutions, the role of standards and centralization within the institutions’ procurement process, and their perceptions of other aspects of educational technology adoption that might provide the potential for the development of future standards.

2. Methods

Our research utilized a mixed-methods approach. Prospective participants were initially contacted through e-mail outreach that explained the purpose of the work, the incentives offered, and then provided a link to our questionnaire (See Appendix A). The final question on the questionnaire invited interested participants to volunteer (with further incentives) to participate in a structured interview (See Appendix B). The study was approved by the University of Toronto Research Ethics Board.

2.1. Participants and Recruitment

Our goal was to reach and entice as many participants as possible who participate in the procurement of education technologies in Canadian colleges and universities. Recruitment proved to be a very significant challenge. Each representative from 10 Canadian post-secondary institutions across six provinces responded to our questionnaire. An in-depth structured interview was conducted for two participants who completed the questionnaire. Each participant received a CAD 25 Amazon gift card for completing the questionnaire, as well as entering their name in a draw to win one of two gift cards for CAD 100. To compensate them for participating in the interviews, the participants received a CAD 100 gift card.

2.2. Materials and Procedure

Our primary data vehicle is the questionnaire provided in Appendix A. It includes an initial section to obtain some general information about the respondents and their institutions, and a second section focused more specifically on how the procurement process occurs within the respondent’s institution. There are then sections that gain further information related to specific standards, some of which are commonly used and some of which are possibilities for the future. Thus, we ask about standards related to (1) Security and Privacy, (2) Accessibility, (3) Learning Tool Interoperability, (4) Equity, Diversity, and Inclusion, (5) Business Continuity and Business Recovery, (6) Care of Data Practices, and (7) Analytics Flowing to the institutional LMS. We then include an open option for the respondents to highlight any standards-relevant issues we might have missed. Finally, they were invited to take part in our structured interview to allow for a richer conversation around standards.

The questions for the structured interview are presented in Appendix B. The purpose of structured interviews is to keep things roughly focused while still allowing the respondent a lot of freedom to provide additional richness that might be missed by a questionnaire. Thus, the questions are intended to provoke deeper thought without overly constraining the responses. Each interview was audio recorded, thereby allowing the primary researcher to qualitatively code and report on the responses.

3. Results

3.1. Questionnaire Findings

The findings from the questionnaire will be presented across sections that mirror those of the questionnaire itself.

3.1.1. Section A

Section A of the questionnaire asks specific questions about the respondents’ role and home institutions. All 10 respondents were from different institutions across six of the ten provinces. The institutions ranged from small- to large-sized colleges depending on enrollment size and number of faculty. The first few questions of this section are presented in

Table 1. Basic information about the respondents and their institutions.

Respondent	Institution	Province	Enrollment	Number of Faculty	Home Department	Position within Unit
1	McGill University	Quebec	40,000	2200	Teaching and Learning Services	Director
2	Cambrian College	Ontario	5000	500	Teaching & Learning Innovation Hub	Senior Educational Technology Specialist
3	University of Manitoba	Manitoba	35,000	5400	Teaching & Learning Centre	Director
4	Fleming College	Ontario	6300	213 FT, 420 Contract	Academic Operations	Associate Vice President
5	Saskatchewan Polytechnic	Saskatchewan	15,000	1000	Learning and Teaching	Associate vice-president
6	University of Lethbridge	Alberta	8000	430	Teaching Centre	Associate Director
7	University of New Brunswick	New Brunswick	10,000	600	Centre for Enhanced Teaching and Learning	Director
8	Toronto Metropolitan University	Ontario	50,000	1000	Economics	Professor
9	University of Waterloo	Ontario	1352	1352	Information Systems and Technology Department	Director, Instructional Technologies and Media Services
10	University of Saskatchewan	Saskatchewan	1950	1950	Information and Communications Technology	Sr. Director—Enterprise Solutions

.

The respondents were then asked three questions about the extent to which they viewed their institution, in general, as highly unified (all entities work together) or siloed (composed of relatively independent entities related to subunits of their university). The specific questions and associated responses are given in

Table 2. Siloed versus Unified Decision Structure.

Question	Mostly Siloed	Somewhere in Between	Mostly Unified
A8. “…where would you put your institution on the unified versus siloed continuum?”	2	6	2
A10. “…where do you think your institution will be on this continuum 5 years from now?”	0	7	3
A12. “…where do you think is the optimal point on this continuum?”	0	4	6

.

These numbers suggest that, while 20% of the universities in our sample considered themselves “mostly siloed”, this was not generally viewed as the optimal situation (Table 2, A10). In general, a more unified (or at least mixed) situation is preferable and seems to be the direction in which things are moving.

For each of the questions above, the respondents were also allowed to include open-ended comments. An examination of these comments revealed that some participants thought an approach that is a mix between siloed and unified was optimal, as there are many benefits to collaborations and the coordination of processes. It was said that this could help to make the student experience more cohesive.

While related to the previous questions, the next three shifted the focus away from the structure of the institution and onto the manner in which procurement targets are identified. Are they identified by individual faculty members who then advocate centrally for the technologies they find (i.e., bottom–up), or are they identified by centralized units in the institution (e.g., Centres of Teaching and Learning) and then brought to the faculty? Our findings are depicted in

Table 3. Bottom–up vs top–down decision making.

Question	Bottom–Up	Somewhere in Between	Top–Down
A14. “…where would you put your institution on the bottom-up versus top-down approach to educational technology procurement?	0	8	2
A16. “…where do you think your institution will be on this continuum 5 years from now?	0	7	3
A18. “…where do you think is the optimal point on this continuum?	0	7	3

.

None of the institutions indicated relying solely on a bottom–up process, suggesting that more centralized components of the university do become part of the procurement process at some point, even if it might be a faculty member that brings an original awareness of some technology. In fact, this sort of hybrid seems to be the dominant approach, although clearly some are taking a much more top–down approach as well.

For each of the questions above, the respondents were also allowed to include open-ended comments. Participants who commented on the question asking where their institution was on the bottom–up versus top–down continuum stated that their institution took an approach that was in the middle of the bottom–up and top–down continuum. Many participants who stated that their institution took an approach that was in the middle of the bottom–up and top–down continuum commented that some of the technology that was procured was brought by the teaching centre or IT department. Other times, a faculty member may suggest an EdTech and the institution would then assess this product according to the institution’s criteria, existing EdTech used by the school, and interest levels for the proposed EdTech. Of the three participants who identified that their institution took a top–down approach, one commented that, although their procurement process was centralized (top–down), instructors were still able to suggest tools they found. The institution would then follow up on all suggestions, but they may not be approved if they are too expensive, do not pass security and privacy concerns, or if the university has a tool that fulfils the same function. One participant stated that they thought their institution would shift from taking a bottom–up approach to taking a mixed approach in the next five years. They commented that there were some efforts towards starting to improve feedback opportunities for faculty and students in centrally managed classrooms, but these were in early stages.

3.1.2. Section B

The next section of the questionnaire was intended to elucidate the procurement process as it is currently used in Canadian colleges and universities. The questions used in this section were adapted from Morrison et al., 2019 [1]. We first provided the respondents with a list of influences and experiences and asked them which influences and experiences informed their approach to identifying, selecting, and procuring educational technologies. The list is provided in

Table 4. Influences and experiences relevant to the identification, selection, and procurement of Educational Technologies.

Possibility	Average Out of 7
Pilot tryouts of products	5.44
Recommendations from end-users (educators and administrators)	5.00
Your recommendations based on your own online research	4.78
A set of identified features that the EdTech must meet	4.44
Choosing from a list of “approved” (or recognized) vendors/brands	4.22
Products with the lowest cost	4.00
Non-rigorous evaluation evidence (e.g., from vendors’ in-house studies)	3.44
“Bundled” products (both software and hardware together)	3.33
Recommendations from sales representatives or other consultants	3.11
Conferences that include Educational Technology Vendors	2.89
Recommendations or ratings on an informational website (please specify which)	2.56
Rigorous evaluation evidence (from published studies, literature reviews, etc.)	2.33
One “other” was provided by one respondent—“Trusted Software Partners—Infotech, Gartner, Educause, etc.” … the score is from that respondent	3.00

along with their average score based on a seven-point Likert scale ranging from 1 (not at all) to 7 (extensively). The items were reordered from highest to lowest score to highlight the influences and experiences that were most strongly endorsed.

We then gave the respondents a list of roles played by individuals within their organization and asked them to rate the degree to which each individual was involved in the procurement process. Each role was given a value from 1 (not at all) to 7 (extensively). The results, listed in order from most to least involved, are presented in

Table 5. Roles participating in the procurement process.

Role	Average Out of 7
Yourself (The Respondent)	5.11
Educators	4.89
Administrators	4.78
Chief Information Officer	4.33
Chief Academic Officer	4.00
Chief Purchasing Officer	3.89
Technology Director	3.89
Chief Financial Officer	2.11
Students	2.00
Four “Other” responses were provided with the number in brackets corresponding to the number of respondents. The score shown is the mean score provided by those (that) individual(s)
The unit more broadly (1)	7.00
Accessibility and Privacy Officer/Legal Council (1)	7.00
Centre of Teaching and Learning (2)	5.50

along with the mean ratings.

This data suggest that, although students are the end-users of the technologies that colleges and universities procure, they do not seem to have a “seat at the procurement table”.

3.1.3. Section C

Section C focused on specific areas in which specific standards might play roles in procurement.

The first question we asked was for them to estimate the percentage of educational technologies used on their campus that had undergone a formal vetting process prior to procurement. Their responses are provided in Figure 1.

It is interesting to note that, although two thirds of Canadian colleges and universities formally vet their technologies, one third does not yet conduct that at least for all technologies in use.

In the sections of the questionnaire that follow, we asked a common set of questions, but did so separately for each of a number of specific areas where specific standards might play a role. To ease the presentation of results, we combined the answers across these areas into single tables related to each question. For example, for each of the areas listed, we asked “which standards do you focus on when assessing this issue?”.

Table 6. Responses to “Which standards do you focus on when assessing these issues? (open ended)” across a number of areas relevant to standards.

Issue	Type of Standard	Frequency
Security and Privacy	Provincial regulations	1
	FIPPA	2
	FIAA	1
	HECVAT	1
	SOC2	1
	AODA	1
	IT responsible for this	2
	Learning outcomes	1
	Personal Information Protection and Electronic Documents Act (PIPEDA)	1
	Security, Compliance and Monitoring	1
Accessibility	WCAG	6
	WCA	1
	Universal design principles	1
	Learning Outcomes	1
Learning Tool Interoperability	LTI compliance	3
	API	1
	D2 L compliance	2
	LMS	1
	Not familiar with this	1
Equity, Diversity, and Inclusion	Director of EDI responsible for this	1
	Canadian Human Rights	1
	UDL	1
	Accessibility	1
	Inclusion	1
	Ease of use	1
	None	4
Business Continuity and Disaster Recovery	Data security protocols	1
	Server reliability	1
	Help and assistance	1
	Regular back up, able to recover data	2
	IT is responsible for this	2
	None	3
Care of data practices	Main concern is data storage in Canada and America	1
	PIPEDA,	1
	Information Security Management	1
	Institutional Data Handling Policies	1
	IT is responsible for this	1
	Security	1
	Confidentiality and privacy	1
	Backups	1
	Breach language in contracts	1
	Require information on overall system	1
	Not sure	1
Analytics Flowing from the Application to the LMS	Responsible for ensuring privacy and security are respected	1
	LTI	2
	API	1
	Data encryption	2
	Privacy and security	1
	Proponent can outline common approaches for real-time or batch integration and any known or common limitations associated with each (e.g., request throttling or parts of solution not accessible through API/batch feed, etc.)	1
	None	1

presents their responses to this question for all the issues we asked about.

We then asked “Do you look for specific certificates to verify the vendor has met this standard and, if so, which certificates?”. The responses are provided in

Table 7. Responses to “Do you look for specific certificates to verify the vendor has met this standard and, if so, which certificates? (open ended)” across a number of areas relevant to standards.

Issue	Type of Certificate	Frequency
Security and Privacy	Security checks are conducted	1
	Privacy addendum	1
	NIST Cybersecurity Framework	1
	CIS Controls	1
	ISO 27001	1
	A self-assessment for all security, compliance and monitoring is conducted	1
	IT department responsible for this	1
	Other individuals are responsible for this	2
Accessibility	WCAG	1
	Security Compliance	1
	Verification of standards of interest (for centrally vetted tools)	1
	None	2
Learning Tool Interoperability	API (for centrally vetted tools)	1
	LTI (for centrally vetted tools)	1
	Accessible documentation	1
	None	3
Equity, Diversity and Inclusion	AODA	1
	None	4
Business Continuity and Disaster Recovery	Ask for a copy of BC and DR plans (for centrally vetted tools)	1
	None	4
Care of data practices	Yes (no further information provided)	3
	None	1
Analytics Flowing from the Application to the LMS	API (for centrally vetted tools)	1
	LTI (for centrally vetted tools)	1
	No	3

.

Responses show that various standards and certificates are used to assess these factors, however, many institutions also indicated “none” in some of these areas.

Following up from the previous question, we then asked “If a vendor claims to meet the standard but does not have a certificate to verify that, which of the following is true?”. Once again, the responses broken down by issue are presented below, in

Table 8. Responses to “If a vendor claims to meet the standard but does not have a certificate to verify that, which of the following is true?” across a number of areas relevant to standards.

	It Would Be a Deal Breaker, End of Discussion	Consideration Would Continue, but the Vendor Would Be Required to Be Certified before Any Deal Was Formalized	You Would Perform an Internal Assessment to Ensure the Standard Is Met	Consideration Would Continue without Further Stipulation … We Trust the Vendors When They Say a Standard Has Been Met
Security and Privacy	2	0	5	2
Accessibility	0	0	6	1
Learning Tool Interoperability	1	0	4	2
Equity, Diversity, and Inclusion	0	0	4	2
Business Continuity and Disaster Recovery	0	0	5	3
Care of Data Practices	3	0	2	2
Analytics Flowing from the Application to the LMS	0	0	3	3

.

These data suggest that, of all the examined areas, Privacy and Security and the related Care of Data Practices were the two the institutions would most likely like to see certified by some external body such as the Canadian Standards Association (CSA). Even in those cases though, and for all other cases, it seemed that formal certification was seldom a deal breaker and institutions were generally willing to investigate the issue on their own, or trusted that the vendor had addressed that issue when they said they had.

The final issue we addressed in Section C was the importance placed on meeting a given standard fully. Specifically, the respondents were asked “Using a 1 to 7 scale with 7 meaning “absolutely critical”, how important is it to your institution that this standard be met fully?” The responses are presented in

Table 9. Responses to “Using a 1 to 7 scale with 7 meaning “absolutely critical”, how important is it to your institution that this standard be met fully?” across a number of areas relevant to standards. Items are listed in descending order according to importance.

Area	Mean
Security and Privacy	6.33
Accessibility	5.63
Care of Data Practices	5.57
Learning Tool Interoperability	5.14
Equity, Diversity, and Inclusion	5.14
Business Continuity and Disaster Recovery	4.86
Analytics Flowing from the Application to the LMS	4.80

and Figure 2.

Once again, we see that Security and Privacy, Accessibility, and Care of Date Practices seemed to be the areas where standards are currently playing the largest role in EdTech procurement, with Equity, Diversity, and Inclusion; Business Continuity and Disaster Recovery; and Analytics Flowing from the Application to the LMS playing lesser roles.

We then asked if there were any other issues that educational institutions should consider when vetting educational technologies. Additional issues that EdTech are assessed for include available features that support online teaching and learning; Software as a Service; use for the EdTech across faculties; and Accounts, Access, and Role Management, Recovery and Data Protection, Incident Management Performance, Support and Training, Release, and Environment Management. The participants then responded to the same follow-up questions asked for the seven other issue areas we identified. The responses to these questions are reported in

Table 10. Responses to “Which standards do you focus on when assessing this issue?” across additional issues EdTech are vetted for.

Issue	Type of Standard
Available features that support online teaching and learning (green)	Student-centered, easy to use functionality
Software as a Service (red)	Security certification and data storage
Use for the EdTech across faculties (yellow)	N/A
Accounts, Access, and Role Management, Recovery and Data Protection, Incident Management Performance, Support and Training, Release, and Environment Management (brown)	N/A

,

Table 11. Responses to “Do you look for specific certificates to verify the vendor has met this standard and, if so, which certificates? (open ended)” across additional issues EdTech are vetted for.

Issue	Type of Certificate
Available features that support online teaching and learning (green)	N/A
Software as a Service (Red)	N/A
Use for the EdTech across faculties (yellow)	N/A
Accounts, Access, and Role Management, Recovery and Data Protection, Incident Management Performance, Support and Training, Release, and Environment Management (brown)	N/A

,

Table 12. Responses to “If a vendor claims to meet the standard but does not have a certificate to verify that, which of the following is true? across additional issues EdTech are vetted for.

Issue	Response
Available features that support online teaching and learning (green)	You would perform an internal assessment to ensure the standard is met
Software as a Service (Red)	You would perform an internal assessment to ensure the standard is met
Use for the EdTech across faculties (yellow)	N/A
Accounts, Access, and Role Management, Recovery and Data Protection, Incident Management Performance, Support and Training, Release, and Environment Management (brown)	You would perform an internal assessment to ensure the standard is met

and

Table 13. Responses to “Using a 1 to 7 scale with 7 meaning “absolutely critical”, how important is it to your institution that this standard be met fully?” across additional issues EdTech are vetted for.

Area	Score
Available features that support online teaching and learning (green)	4
Software as a Service (Red)	6
Use for the EdTech across faculties (yellow)	N/A
Accounts, Access, and Role Management, Recovery and Data Protection, Incident Management Performance, Support and Training, Release, and Environment Management (brown)	6

below.

These additional issues show that there are many factors related to standards that are considered across institutions. This also provides information on the various areas that standards created for EdTech procurement should assess for. These responses further show the degree of differences in factors and standards that are considered across institutions. Of the participants who identified other issues, none identified any relevant certificates used to verify this standard has been met.

At the very end of Section C, we also asked two open-ended questions with respect to vetting educational technologies. We first asked “What challenges exist that might make it difficult for your institution to vet technologies fully?”. The challenges listed included a lack of both financial resources, as well as limited staff, issues with vendor responsiveness and transparency when procuring EdTech, and provincial regulations that some vendors may not meet. The second open-ended question asked was “Is there any other issue that you think educational institutions should be considering when vetting educational technologies for use in the classroom?”. The participants stated that institutions should consider functionality and who is responsible for the approval of the EdTech, as well as user experience and usability.

3.1.4. Section D

We had one final question of relevance in Section D. Specifically, we asked “Did we miss something that you think is relevant to the procurement process in general, or the role that standards and certificates play in that process?” One person shared that individuals from various departments such as Procurement, IT, and Teaching and Learning are differently responsible for parts of this process; thus, creating a system that facilitates and streamlines this process is challenging.

Overall, the data from the questionnaire suggest that standards are having an impact, and this impact is increasing as institutions become more centralized. Standards seem to be guiding and informing the procurement process, though they are not yet relied on in a formal, structured way (i.e., we need to see certificates related to X, Y, and Z before we will consider your technology). There may be room for more standards, and for selling the value of a standards-based procurement process to further standardize the features of the technologies that students use in the classroom.

3.2. Interview Findings

The questions asked in this interview were organized according to three sections, and our results are reported according to these sections.

3.2.1. Demographics

The participants were first asked to provide demographic information and information regarding the institution they worked for before beginning the interview. They were asked to confirm the name of their institution, their department, role at the institution, and if the institution was siloed or unified.

3.2.2. EdTech Procurement Experience

This section asked about the participants’ experience with the EdTech procurement process at their institution. The participants were asked to think about a specific EdTech they were involved in procuring, and then were asked to state how their institution became aware of this EdTech, how it was procured, and their role in achieving this. One participant mentioned the EdTech TopHat, and stated that someone contacted their institution informing them of this tool. Another participant said the EdTech Respondus was procured at the start of the COVID-19 pandemic to monitor non-proctored exams. This institution became aware of Respondus when they were searching for an EdTech to monitor exams. Both participants stated that the institution met with the vendors to learn more about the products and determine if it fitted their needs prior to procuring the EdTech. The roles of the participants in the procurement process differed. One participant was concerned with the user end, whereas another participant was the director of the Centre for Teaching and Learning and was heavily involved in procurement processes.

The participants were asked to identify where the EdTech they mentioned and their institution more generally was on the bottom–up/top–down continuum. These responses are reported in

Table 14. Breakdown of where institutions are on the bottom-up vs top-down continuum currently and for the identified EdTech.

	Specific EdTech Bottom–Up vs. Top–Down	Overall Institution Bottom–Up vs. Top–Down
Participant 1	Top–Down (TopHat)	Mix of both bottom–up and top–down
Participant 2	Bottom–up (Respondus)	Mix of both bottom–up and top–down

.

3.2.3. Perception

When asked how they felt about the current EdTech procurement process, the participants responded quite differently. One participant said it could be greatly improved, as no clear process of what should be conducted existed. The second participant reported that they did not have an issue with the current EdTech procurement process and stated that there was little leeway on what could be procured due to policies, budgets, and provincial rules which must be followed.

The participants then commented on the processes they thought should be used to procure educational technologies, if there were no barriers. One participant said that EdTech procurement had become very department- and faculty-focused, because their needs are so varied. They commented that, in the past, EdTech was procured and used across the campus by all departments, however this is not a practical approach today. As such, it was suggested that departments might benefit from making some decisions with regard to the tools procured, within the policies and financial constraints of the institution.

The second participant said that all information regarding the EdTech being procured and the criteria it meets should be well communicated across the institution, rather than each individual or department involved in the procurement process assessing the EdTech only for the criteria of interest and not sharing this information with other departments. It was suggested that, to streamline the process, one department should be responsible for assessing if the EdTech meets all the needs of the institution, rather than being assessed by various departments. Since this participant identified a mismatch between the perceived optimal approach and the approach taken by the institution, they were asked to share their thoughts on the barriers that made it challenging for their institution to implement the optimal approach. One participant said that they thought the speed at which EdTech was introduced in schools resulted in a lack of processes to direct how EdTech should be procured. This participant also said that there are challenges with changing existing processes after the fact and, as a result, the current processes in place continue to be used.

3.2.4. Standards

The participants were asked what criteria were considered when procuring EdTech for the classroom. One participant said security, privacy, and accessibility regarding its usability in different countries for students accessing the platform internationally. Accessibility and the need for the platform to be used internationally were considered to be especially important. The second participant said applicability was considered. Both participants said cost was also considered, with this being the most important criteria for the second participant, as well as privacy and accessibility. The ways in which EdTech was assessed against these criterions also differed between the two institutions. One of the participants stated that IT ensured that the criterions were met, whereas the second participant stated that EdTech had to meet the policies and regulation in place to ensure the criterions were met.

3.2.5. Challenges

The participants identified various challenges to procuring EdTech. One participant said these challenges included ensuring there was enough funds to procure EdTech and the length of time of the procurement process. The second participant said that, sometimes, there were challenges when interacting with certain vendors, as many lacked adequate information when discussing their product with institutions. As a result, intuitions need to be very selective and ensure enough research is conducted before proceeding with a vendor. The participants were then asked what factors would influence the way EdTech was procured. One participant said need greatly influenced how EdTech was procured. The second participant said that the way EdTech was procured depended on how the institution was structured. At this institution, subject experts were able to assess EdTech according to the identified criterions and were then able to adopt the EdTech. Additional comments were provided to explain that this would be different in institutions with one department that focused on ensuring the EdTech met all requirements, rather than engaging with faculty and administrators.

Lastly, we asked the participants if, in their experience, there were situations where having a standard would enhance the procurement process, and possibly if there were any cases where removing a standard would enhance the procurement process. One participant was unable to comment on standards, as this was not their role in the EdTech procurement process. The second participant did not believe that removing any standards could enhance the procurement process and emphasized the necessity for standards, stating that rules are needed in order to properly procure EdTech.

4. Discussion

The results of this study show the need for a coherent process across the institution that outlines an optimal procurement process. As expected, there seems to be many differences in the way EdTech is procured across institutions. Some institutions vet EdTech centrally and are more concerned with standards being met, whereas other institutions may take a decentralized approach, permitting educators and administrators to make decisions as to which EdTech is used in the classroom. We were slightly surprised that a hybrid approach was the most common, as the bottom–up approach was very common in Canadian universities for many years. However, it now seems that top–down approaches are becoming common, but that hybrid approaches that combine central institutional elements along with individual faculty members seems most common and most preferred. Once again, a deep consideration of standards is more likely when central elements of the institution are involved in the procurement process, so these findings are heartening from the perspective of ensuring certain standards are in place.

This study shows that there is an understanding of the importance of standards in the procurement process, however, there seems to be challenges and inconsistencies with the extent that standards are used in this process. When the participants were asked to state how critical certain standards were to their institutions, standards related to Equity, Diversity, and Inclusion; Business Continuity and Disaster Recovery; and Analytics Flowing from the Application to the LMS ranked lower than the other standards. It is important to note that clear standards do not yet exist for these three areas. In some sense, then, these numbers suggest these areas are perceived as important, and if associated standards did exist, these numbers would likely be higher.

The findings of this study can be used to create a roadmap outlining the steps institutions can take to move towards a coherent centralized procurement process.

1.

Identify specific standards.

Individuals involved in EdTech procurement should work collaboratively to create a process for procurement that considers standards that are identified as important for their organization and will be used to vet all EdTech. This process and the steps that need to be taken by the different individuals involved in this process should be made known to all individuals involved. Clear communication between the individuals involved in this process ensures that people know what stage of procurement an EdTech being procured is at. This improves the efficiency of the procurement process, which was cited as a challenge.

2.

Involving educators and administrators in the procurement process.

All stakeholders, including educators and administrators, should be involved in all decisions that relate to the EdTech procurement process. Some educators and administrators prefer a bottom–up approach, as they are more involved in the process. Not having these individuals involved may lead to poor adoption rates, as they may be unaware of the function of the EdTech. To mitigate this, educators and administrators should be included by asking their opinions on various EdTech products that the institution wants to procure. If there is an agreed need for the EdTech, then the procurement process can continue in a centralized manner. This prevents educators, administrators, and departments from acting in siloes, and reduces the probability of them procuring EdTech that does not meet the institution’s standards, while still maintaining an element of the decentralized process.

3.

Involving students in the procurement process.

The literature on involving all stakeholders in the procurement process is not well developed. Our findings showed that, among stakeholders, students were the least involved in procurement processes. Student input could provide an essential missing perspective. Thus, the procurement process would benefit from involving students to understand their needs, as well as feedback on the EdTech being procured.

Although this roadmap provides general information on the possible actions institutions could take to improve the procurement process, institutions would also benefit from the creation of resources that specify how to best transition toward a top–down approach. A lack of reliance on standards when procuring and vetting EdTech may be due to the current lack of a widely accepted standard, and thus, may point to areas where new standards could play a key role. Thus, to increase the use of standards in this process and to improve communication within an institution, there needs to be guidelines which outline the various standards and certificates that relate to security and privacy; accessibility; Learning Tool Interoperability; equity, diversity, and inclusion; business continuity and business recovery; care of data practices; and analytics flowing to the institutional LMS. This document should also describe who is responsible for the various parts of the procurement process to ensure clarity and consistency across the institution. Having a clear guideline in place can reduce the confusion within institutions on the procurement process and help to improve the efficiency of EdTech procurement. Future research could conduct a cost–benefit analysis which would provide institutions with valuable insights for decision making.

5. Conclusions

This study presents findings from the questionnaire and interviews, which aimed to understand how institutions procure EdTech, the use of standards when procuring EdTech, as well as some challenges with the current procurement process.

The results from the questionnaire provide a reasonably clear sense of the role standards are currently playing in terms of the procurement of educational technologies at colleges and universities across Canada. The points we would like to highlight are the following:

• In certain areas, especially those related to Privacy and Security, Accessibility, and Care of Data Practices, standards seem to be playing a relatively large role in terms of vetting educational technologies. In other areas, the role of standards is reduced.
• Even where standards are deemed important, having those standards formally certified does not seem critical to most institutions at this time. Most institutions seem comfortable vetting the technologies themselves or, in some cases, simply trusting third-party vendors.
• There are several new areas where the development and promotion of new standards would seem to have value in the procurement of educational technologies. Perhaps the one of greatest interest would be standards around Equity, Diversity, and Inclusion, although standards related to Business Continuity and Disaster Recovery, and standards around the provision and digital sharing of specific learning analytics also have the potential to be valued.

The findings from the interviews suggest that the EdTech procurement process appears to be quite different between interview participants. Although the participants thought this process could be improved, there were differences in how they believed this could be achieved. One participant thought there should be more autonomy offered to departments and faculties, as they would have the best understanding of the EdTech they need. The second participant thought that communication within the organization is a necessary change, and suggested implementing procedures that clarify the procurement process to ensure everyone involved knows their role, as well as the roles of their colleagues. The need for and importance of standards to regulate this process were evident, again, further stressing the need for standards to improve the efficacy and efficiency of the procurement process within institutions.