Strengthening Sustainability in Agile Education: Using Client-Sponsored Projects to Cultivate Agile Talents

Abstract

The success of agile in software development (SD) has sparked the application of agile in non-SD domains such as business management to improve operational efficiency and innovation. Despite the rising industry demands for agile talents in the non-SD domains, agile education falls short of client-sponsored projects, calling into question the sustainability of agile education. This study makes up for the gap and illustrates an eight-month endeavor where scrum practices and values were imbued in a client-sponsored project. The analysis of qualitative and quantitative data gathered throughout the eight-month project illustrates a large disparity among students in their scrum application, reveals top challenges faced by students in their scrum application, and suggests the impact of the scrum application on the quality of student work. The findings of the study set a solid foundation based on which future agile education could be enhanced to strengthen the sustainability of agile education to meet industries’ rising demands for agile talents in non-SD domains.

1. Introduction

Higher education plays a key role in educating future decision makers for a sustainable future [1,2]. However, to be effective in the role requires higher education to be flexible and adaptive in its teaching pedagogy. As a matter of fact, the sustainability of higher education lies in its ability to respond to change, the ability that is considered “one of the main responsibilities of education” [3].

One of the recent changes that higher education has been trying to respond to is the rising demand for agile talents [4]. Since the announcement of the Manifesto for Agile Software Development, which epitomizes the customer-centered, iterative, and adaptive nature of a myriad of agile methods including scrum, Crystal, and eXtreme Programming [5,6,7], agile practices have gained wide recognition and become mainstream in the software industry [8,9]. By applying scrum and other methods, firms have significantly enhanced the software product quality, reduced the time to market, and boosted the bottom line [10,11,12,13].

The success of using agile in software development (SD) has inspired non-SD areas to apply agile to transform the ways they do things to achieve operational efficiency and agility [6,14,15]. The adoption of scrum and other agile methods beyond information technologies (IT) and software development has quickly debunked the misunderstanding that agile is only for IT and software development [6]. Global surveys of agile practitioners noted that there is “a company-wide integration of scrum values and practices” [13] which are taking hold in diverse industries” [11].

However, organizations have been plagued by the shortage of agile skills, as McKinsey predicts the demand for agile skills will continue to outstrip supply [16]. Most of the failed adoptions of agile in non-SD initiatives can be attributed to two main reasons: (1) the lack of understanding of agile practices and values, which results in misperceptions of agile as unstructured and undisciplined [17,18,19], and (2) difficulties faced by team members in adjusting their behaviors and mindsets to agile values such as customer centricity, close team collaboration, and timely adaptation to changing business needs [5,6].

The review of the current literature on agile education, however, reveals concentrated educational efforts in the Science, Technology, Engineering, and Mathematics (STEM) discipline [20,21,22,23,24,25,26]. As a matter of fact, over 90% of agile education publications focus on software development [22]. A close examination of agile education studies in the non-SD context indicates two main focuses of agile in education [27]: using agile in pedagogy design [28,29] and teaching agile content [30,31,32,33]. This research centers on the latter.

Diverse teaching methods were experimented. Some educators opted for short simulations/in-class exercises to introduce the agile project management concepts. For example, Sibona and colleagues [31] incorporated an origami active learning exercise in teaching agile principles. Schmitz [30] applied a role-play simulation to instruct students in agile methods. May and colleagues designed a ball game to familiarize students with the scrum approach in their system analysis and design course [34]. Valle and O’ Mara used a simulated in-class scrum exercise to train students in adaptative project management [35]. Lebens focused just on student experiences in stand-up meetings [33].

While short-duration simulations/in-class exercises are effective in giving students general knowledge of agile, a semester-long group project affords students an opportunity to apply agile to create a new product [20,36,37]; thus, it is considered the most effective approach to agile education [38]. Projects infused with the innovative adaption of agile methods to the non-SD context have been designed, including wiki projects [20], working design prototypes [36], data analytics reports [39], and a time schedule for an academic semester [37]. These group projects, however, lack the involvement of external clients. To remedy the limitation, educators play a part either as a client [37,39] or as a product owner [20] to simulate the real-world agile experience.

While offering students a useful understanding of agile concepts and practices [4,27], the current pedagogical approach to group projects confines agile projects within the academic context and therefore may not prepare students sufficiently enough to meet the market demand. Client-sponsored projects are considered an effective tool for promoting experiential learning, fostering engaging and meaningful learning experiences [40,41,42], and offering the real-world experience increasingly favored by employers [43,44]. In the context of agile education, client-sponsored projects offer additional unique learning opportunities. In particular, in client-sponsored projects, clients’ requirements are open-ended and change over time [42,45], which requires student teams to adapt their work to meet the client’s needs. Unlike the traditional academic projects where student teams are provided with the resources they need to complete their work, in client-sponsored projects, student teams have to seek out resources and client support by themselves, placing a higher demand for team collaboration [46].

Given the lack of studies on using client-sponsored projects to teach agile, the study makes up for the gap by illustrating how agile practices and values were imbued in a client-sponsored project. In particular, the study intends to address two research questions: (1) To what extent are business management students able to apply agile knowledge and skills in a client-sponsored project? (2) What is the impact of agile practices on the quality of student projects? To answer the questions, this study reports an eight-month endeavor of incorporating scrum in a client-sponsored project where 90 students were formed into 17 teams, working as free consultants to help their client identify a feasible IT solution. Qualitative and quantitative data pertaining to students’ application of the scrum practices and team performance over the eight-month period were collected and analyzed.

2. Materials and Methods

2.1. Scrum and Scrum Team Artifacts and Teamwork Process

Introduced in the United States by Ken Schwaber and Jeff Sutherland in 1995 [47], scrum is “a framework within which people can address complex adaptive problems, while productively and creatively delivering products of the highest possible value” [48]. To support the values and principles of the Agile Manifesto, scrum defines three team roles, three pillars guiding teamwork, and an iterative teamwork process.

Central to scrum are the three roles of a scrum team: (1) product owner (i.e., manages product backlogs), (2) scrum master (i.e., facilitates the development team to complete their tasks), and (3) development team (i.e., responsible for completing the tasks of the sprint backlogs) [31]. The team members work together to support the three pillars [48]: transparency (i.e., a common understanding of significant aspects of a scrum process such as artifacts), inspection (i.e., frequent inspections of scrum artifacts and progress toward a sprint goal to detect undesirable variances), and adaptation (i.e., timely adjustments to the process and the artifacts to minimize further deviation).

To maximize transparency, the scrum team designs three artifacts including product backlog (i.e., an ordered list of what is needed in the product), sprint backlog (i.e., a set of product backlog items selected for a sprint and work needed to be done to deliver the specified functionality), and increment (i.e., the sum of all the Product Backlog items completed during a sprint and the value of the increments of all previous sprints). The artifacts are shared among all parties involved in the scrum process to ensure that a common understanding is achieved.

To further support transparency and enable inspection and adaptation, Schwaber and Sutherland suggest an iterative teamwork process (rituals) which encompasses four formal events including sprint planning (e.g., prioritize product backlogs), daily scrum (i.e., 15-min time-boxed to check progress towards the sprint goal), sprint review (i.e., occurs at the end of a sprint to demonstrate the work done and revise the product backlog), and sprint retrospective (i.e., inspect how the last sprint went and create a plan for improvements in future sprints). Through the iterative process, the scrum team adds functionalities incrementally to the satisfaction of the product owner.

Success in scrum projects relies on a self-organizing and cross-functional scrum team to optimize flexibility, creativity, and productivity [48]. Furthermore, to bring to life the three scrum pillars, team members need to be committed to the goals of the scrum team, focused on the work of sprints, open about all the work and the challenges with performing with the work, and respectful of each other [48].

2.2. Client-Sponsored Project Background

The project, offered in an urban university in Canada, was mandatory for all undergraduate students who were in their final year of the Business Technology Management (BTM) program. Situated under a business school, the program was developed in response to the lack of business skills among information systems students and aimed to equip graduates with necessary technical and business skills to excel in today’s organizations (https://itactalent.ca/our-programs/employer-defined-education/btm/ (accessed on 30 June 2020)). The key learning objective was for students to apply knowledge and skills gained from all other courses in the program to identify a feasible IT solution for a client. To achieve the objective, students were required to form a team of five or six (Team members were self-selected, as small self-organizing teams are found effective in agile development [49,50,51] (Boehm, 2002; Hoda, et al., 2013; MaVoy and Sammon, 2005; Rising and Janoff, 2000)) and work as consultants (for free) for a client during two consecutive semesters to help the client improve process performance by recommending a feasible IT solution. At the beginning of the first semester, students were expected to have formed a group and secured an external client. The project bore great importance regarding the BTM program, as the quality of student projects would affect the program reputation and image.

In each semester, student teams needed to complete a set of deliverables which built upon each other and led up to the final product. As shown in

Table 1. Client-sponsored project structure and deliverables (In the first two weeks of semester I, the instructor gave each team time to practice on the three scrum artifacts and scrum processes. The last week of the semester was the exam period and thus was not included in the sprints. By the same token, the last three weeks of semester II were devoted to student presentations to their client of their final work and thus were not included in the sprints).

Semester	Deliverables Details	Sprint Schedule
Semester I	Process analysis: as-is process, process analysis, existing technologies supporting the process Process design: to-be process, gap analysis, a list of key performance indicators that would measure the performance of the to-be process change management: assessment of change and change management tactics	9 sprints in total: Sprint 1 started in week 3 and ended in week 11
Semester II	Request for Information (system requirements for the technology solution including functional, non-functional, and implementation requirements) Vendor solutions Cost–benefit analysis Solution recommendation Implementation plan for the solution + Revised interim report	9 sprints in total: Sprint 1 started in week 1 and ended in week 9
Weight	30% each semester	15% each semester

, semester I focused on process analysis and design, with key deliverables, compiled into the interim report, including process analysis (i.e., as-process description and diagram, process analysis, existing technologies supporting the process), process design (i.e., to-be process, gap analysis, a list of key performance indicators that would measure the performance of the to-be process), and change management (i.e., assessment of change and change management tactics).

Based on the semester I analysis, in semester II, student teams were required to (1) propose a to-be process, (2) gather system requirements for a technology solution that supports the to-be process, (3) identify and compare vendor solutions, (4) perform a cost– benefit analysis, (5) recommend the most feasible solution, and (6) propose an implementation plan for the solution. Deliverables for semester II included the request for information, feasibility analysis, and solution implementation plan. Each report was worth 30% of each student’s final grade.

2.3. Project Design: Incorporating the Scrum Teamwork Artifacts and Process

The decision to incorporate scrum in the project was motivated by four considerations. First, there existed great uncertainty in the details of the deliverables, which varied from client to client and may evolve over the course of the project. For example, a client initially focused on a sales process; after the sales process was mapped out and analyzed, the client realized the bottleneck of the process was the lack of the ability to tailor sales materials to each prospect, causing delays in the sales process and affecting sales performance. The insight gained from the process analysis shifted the focus of the project from the sales process to the process of customizing sales materials to different categories of prospects. Based on the discovery, the team eventually proposed a business intelligent system. The uncertainty inherent in each client project warranted an iterative and adaptive approach.

The second consideration had to do with the lack of previous client-sponsored project experience from both students and clients. For example, students would not be able to foresee the number of interviews that needed to be conducted from the outset, as the amount of effort required depended on the information gathered along the way. Third, previous studies have indicated the benefits of iterations in helping students reflect on and adjust the work produced [52]. Through an iterative process, students can make steady progress, which keeps them focused and motivated [53].

Last, the scrum approach was endorsed in the client-sponsored project course because it was the most widely applied in the non-software development context, suggesting (1) its relevance and applicability to the business context and (2) rising industry demands for students with agile knowledge and skills. As students took the course in the last year of the BTM program, having practical experience in applying scrum would equip them with the skills and experience demanded by industries. Below, it is described how the scrum approach was incorporated into the project.

2.3.1. Scrum Artifacts

The project had three scrum artifacts: product backlog, sprint backlog, and increment. The product backlog contained an ordered list of project deliverables from the client and from the instructor. In each sprint, each team needed to compile a sprint backlog detailing work to be completed. For increments, at the end of each sprint, teams were encouraged to integrate the work completed in the sprint with the work completed in the previous sprint(s). This requirement, however, was optional, as the sprint had a short duration, during which students might not have had the time to integrate the content.

2.3.2. Scrum Roles

Both the external clients and the instructor shared common responsibilities as a product owner. For example, both of them provided project requirements (product backlog) that needed to be fulfilled by student teams. Regular team meetings took place with the instructor and each team’s respective client; both owners would go over team progress and offer inputs to backlog prioritization as well as feedback on students’ work completed.

Despite similarities in their responsibilities, however, the external clients and the instructor differed in the focus of their involvement. In particular, a client heavily stressed practical values from the student project, including detailed documentations of existing processes, system requirements, a comprehensive list of vendor products, a thorough cost–benefit analysis of top vendors, and a risk mitigation strategy for system implementation. By comparison, the instructor, serving the role of a gatekeeper of academic integrity and quality standards, focused on helping students complete the project with the ultimate goal of fulfilling the expected learning objectives.

Another scrum role is Team. Each team was composed of five to six students, including a scrum master. Team members were self-selected and had autonomy over how they organized their work and communicated with each other. As for the role of scrum master, each team chose a leader to coordinate team efforts (e.g., arranging meetings, following up on tasks, setting up communication channels) and facilitate communications with its client and the instructor.

2.3.3. Scrum Processes

The project incorporated all of the scrum processes except for daily scrum, which was difficult to implement due to the academic setting. For sprint planning, each team was required to identify the tasks that needed to be performed in the upcoming week, specify how the tasks were connected with the previous tasks completed, assign tasks among themselves, and identify deliverables. A sprint review took place at the end of a sprint to review the work done and update the product backlog. For the sprint retrospective, all teams were required to discuss among themselves where they performed well and where they should improve and to describe the action plan for the next sprint.

The scrum processes were supported by the project management platform Wrike, which was online and mobile-accessible and encompassed features including task scheduling, task tracking, file storage and annotation, chat messaging, and virtual meeting. Each student was given a Wrike account linked to his/her team account. All the tasks planned or completed as well as communications (e.g., messages) were transparent to all team members as well as the instructor and could be easily searched and visually represented in different formats (e.g., Gantt chart, Kanban).

2.3.4. Alignment with the Agile Values

It should be noted that the project design adhered to the four agile values [7] (

Table 2. Inject agile values in the client-sponsored project.

Agile Value	Corresponding Project Design
Individuals and interactions over process and tools	Students were provided an agile platform to plan, allocate, monitor, and self-evaluate their performance.
Working software over comprehensive documentation	The project, with a strong emphasis on practical values to external clients, advocated for the quality of analysis and design (i.e., the identification on business processes, analyses of issues with the processes, and process improvement), of the proposed system requirements, as well as of the feasibility of a proposed solution.
Customer collaboration over contract negotiation	Students acted as a team of consultants and worked closely with their client throughout the semesters. They held regular meetings through which findings were discussed and sprint backlogs were prioritized.
Responding to change over following a plan	Upon receiving feedback from the client as well as the instructor, students adjusted their product/sprint backlog and enhanced the work performed.

). In particular, the project provided student groups with an easy-to-use project management platform (Wrike) to facilitate group communications and collaboration, the project deliverables were of value to practitioners, each student team worked closely with its client, and student teams adjusted their sprint backlogs and the product backlog in response to clients’ changing needs.

2.4. Research Design and Data Collection

Ninety students formed seventeen teams, each of which was sponsored by a real client. The seventeen clients came from diverse industries (e.g., consulting, retail, food and restaurant, manufacturing, logistics, advertising), and each of them had a unique process (e.g., inventory management, sales management, requirement management, customer inquiry prioritization and coordination) for a student team to improve. With the exception of two groups (which had three and four members, respectively) (two students dropped out of the course for various reasons (e.g., health, missing prerequisites)), the rest of the groups consisted of either five or six group members.

A mixed method approach was employed to glean qualitative and quantitative data over the course of the eight months (

Table 3. Data collection methods.

Data Collected	Relation to the Scrum Practice	Timeline of the Data	Measure(s)
Weekly sprint updates	Sprint planning Sprint review Sprint retrospective	Two semesters long and covered the span of 18 sprints	Qualitative: Sprint updates Quantitative: Marks (out of 100 points) given by the instructor
Interim report	Interim team performance	Submitted at the end of Semester I	Quantitative: Marks (out of 100 points) given by a Graduate Assistant (GA)
Final report	Team performance	Submitted at the end of Semester II	Quantitative: Marks (out of 100 points) given by the instructor
Focus groups	Explored challenges faced by student teams	Sprint five; five groups volunteered; GA performed the discussion	Qualitative: analysis by the instructor
Online survey	Challenges faced in the application of the scrum practices	Sprint 8; 70 students responded	Quantitative: ranking of identified challenges

). Qualitative data include 306 weekly sprint updates from all 17 teams (17 × 18 =306) and data collected from the focus group interview on challenges in applying agile in team projects; quantitative data consist of marks on each team’s weekly sprint updates (17 × 18 = 306), the mark for each team’s interim report and final report (17 for the interim report and 17 for the final report), as well as the online survey (70 responses out of 90 students).

Participating in the survey and focus group discussions was completely voluntary. With the student emails provided to the RA by the instructor, the RA used his research account email to recruit research participants and answer student inquiries. In the recruitment letter, the RA explained the voluntary nature of the research and how the identities of research participants would be protected. In particular, students’ identifiable information would be removed before the commencement of the data analysis, and the RA would be solely in charge of data collection and interview transcription.

As indicated above, the diverse data capture student teams’ application of the agile method, the challenges they faced in applying agile, and team performance. Most importantly, the longitudinal data enable the generation of unique insights into the two questions. Using multiple types of data (qualitative and quantitative) through multiple methods of data collection (e.g., student teams’ updates, reports, objective evaluations by the instructor, focus group interview, and online survey) reduces individual biases and provides a rich, divergent and/or complementary view of a phenomenon [54], thus enhancing the validity of the research [55,56,57]. Given the paucity of the research on using client-sponsored projects to teach agile, this design is well suited for the study.

2.4.1. Weekly Sprint Updates

To evaluate the extent to which the scrum practices were applied in each team’s capstone project, each team was required to use the project management tool Wrike to complete weekly sprint updates in which they were expected to document planned tasks (sprint planning), update their product backlog (sprint review), and reflect on the last sprint and identify areas of improvement in future sprints (sprint retrospective).

For each main task completed, the team commented on the quality of work (e.g., “asking our client specific metrics on their process”) and also specified how they would improve in future sprints under the Takeaways subheading (e.g., “make sure we get the metrics we need to complete the preliminary report deliverables”). The overall assessment indicates that the team was working closely with both the instructor and the client, both of whom were informed of the team’s progress and provided feedback which was later incorporated into the future plan. The update was concluded with individual work allocation with clear deliverables.

The instructor, with access to each team’s Wrike account, reviewed and marked each team’s weekly updates based on the extent to which the scrum activities were performed based on three criteria: clarity (e.g., whether the task was defined clearly), completeness (e.g., whether tasks were allocated and a retrospective was submitted), and level of details (e.g., whether enough details were provided for the reader to understand planned tasks/areas of improvement). Teams with sprint updates marked 80 or above are deemed to have performed all three scrum processes (planning, review, and retrospective): a perfect mark was given to updates that are clear and thorough in all three aspects, while updates marked between 80 and 99 lacked clarity and thoroughness in some (if not all) aspects.

2.4.2. Interim and Final Report

Team performance was assessed by both the interim and final report. The interim report built the foundation for the final report, where an IT solution was proposed, vendor solutions were evaluated, and a final recommendation was made. The reports, evaluated based on not only the completeness of the content but also the depth and breadth of analyses, required team members to closely work with each other to ensure consistency in presentation, style, and references. Good team efforts often result in quality reports that read well with logical and convincing arguments and a smooth flow throughout all sections.

2.4.3. Focus Group Discussion

To understand challenges faced by teams in applying the scrum practices, in sprint five, the Graduate Assistant (GA) sent out a request for voluntary focus group discussions. Five groups (whose identities were kept anonymous from the instructor) volunteered and were then interviewed by the GA. The questions asked included “How did you find applying the scrum practices in the project?” and “What were challenges your group faced in completing the weekly updates?” The GA noted the responses and identified the list of 15 challenges.

The instructor performed an independent review of the interview transcript and identified 14 challenges. The comparison of the two sets of ratings revealed 13 common challenges. Cohen’s Kappa was 0.81, indicative of the reliability of the identified list. The GA and the instructor then went through the lists together. The review resulted in the removal of five challenges that were unrelated to the scrum practices (e.g., confusion over medium choice (e.g., face to face, online) for team meetings) and the consolidation of two similar challenges. In the end, eight challenges were finalized and included in the online survey.

2.4.4. Online Survey

To examine common challenges faced by the cohort, the instructor launched an online survey in sprint 8. Students were asked to rank the identified challenges in the anonymous and voluntary survey. In the end, seventy responses were received, resulting in a 77.8% response rate. The survey results shed light on the top challenges that the student teams struggled with.

2.5. Data Analysis

Excel was used to detect changes in the quality of the weekly sprint updates and discern patterns in the quality change over the course of the eight months. The analytical tool was also applied to analyze the relationship between the quality of the weekly sprint updates and the group project scores. The weekly sprint updates offered rich complementary information on potential reasons causing some teams to lapse in their application of the agile method. To analyze the qualitative data, the tool NVivo was utilized to identify and categorize common challenges faced by the teams.

3. Results

3.1. Application of the Scrum Practices

As explained above, the extent to which teams followed scrum processes was evaluated based on the marks given to the weekly sprint updates. A mark of 80 demarcates teams that followed the scrum practices from those that did not. An examination of all sprint updates reveals that the quality of sprint updates improved over time, as evidenced in (1) the increase in the average of sprint updates from 62.9 (which is the lowest) in sprint 1 to above 91 in the last three sprints and (2) the increase in the number of quality sprint updates from 5 in sprint 1 to 14 in sprint 18. These changes are indicative that the students gradually followed the three scrum processes.

However, there exist fluctuations in the marks of sprint updates over the course of the two semesters (

Table 4. Teams’ application of sprint updates over the course of 18 sprints.

Mark Categories of Weekly Updates	Sprints
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18
100	4	7	6	5	9	12	8	13	11	4	7	5	10	8	7	10	10	10
80–99	1	1	1	1	3	0	4	0	1	1	1	3	3	3	4	3	4	4
60–79	9	9	4	7	4	4	5	4	3	7	4	8	4	5	5	4	3	2
<60	0	0	4	1	0	1	0	0	0	0	5	0	0	0	1	0	0	1
0	3	0	2	3	1	0	0	0	2	3	0	1	0	1	0	0	0	0
Average mark	62.9	79.9	66.2	64.4	84.4	90.4	86.8	91.5	81.4	62.8	75.2	76.5	90.6	80.3	85.0	91.1	92.9	91.2

). For example, the number of teams with a perfect update score, starting at 4 in the first sprint, peaked at 13 in sprint 8, dropped to 4 in sprint 10, and eventually stabilized at 10 in the last three sprints. The same fluctuational pattern was observed in the number of sprints with marks between 80 and 99, which started at 1 in the first sprint, dropped to 0 in sprint 6, sharply rose to 4 in sprint 7, steeply dropped to 0 in sprint 8, and finally stabilized at 4 in the last two sprints.

The fluctuations in the frequency of scrum activities demand more in-depth investigations into each team’s sprint updates. A close examination of the 17 student teams’ quality sprint updates suggests four clusters, each representing a consistency level in the application of the scrum practices throughout the 18 sprints: “Consistent,” “Adapting,” “Challenged,” and “Regressed,” which are detailed below and summarized in

Table 5. Overview of sprint updates and team performance.

Group	Weekly Sprint Updates (Percentage of Sprints with Marks between 80 and 100)			Sprint Application	Team Performance
	Term I A	Term II B	Change from A to B (%)	Cluster	Interim Report	Average (Cluster)	Final Report	Average (Cluster)	Mark Change (%)
2	55.56	37.50	−32.5	Regressed	67.0	66.9	70.0	73.33333	4.5%
9	66.67	62.50	−6.2		60.7		73.0		20.3%
11	33.33	12.50	−62.5		73.0		77.0		5.5%
17	11.11	12.50	12.5	Challenged	83.0	74.9	79.0	78.00	(0.05)
7	11.11	25.00	125.0		71.0		81.0		14.1%
3	22.22	37.50	68.8		70.8		74.0		4.5%
16	44.44	75.00	68.8	Adapting	59.6	78.2	73.0	84.14286	22.5%
5	44.44	75.00	68.8		90.0		94.0		4.4%
13	44.44	87.50	96.9		87.0		95.0		9.2%
14	55.56	62.50	12.5		70.0		73.0		4.3%
8	55.56	75.00	35.0		90.8		91.0		0.2%
10	66.67	100.00	50.0		78.3		82.0		4.7%
15	77.78	87.50	12.5		72.0		81.0		12.5%
12	88.89	100.00	12.5	Consistent	78.0	83.4	90.0	92.5	15.4%
1	88.89	100.00	12.5		80.0		89.0		11.3%
4	100.00	100.00	0.0		83.0		93.0		12.0%
6	100.00	100.00	0.0		92.5		98.0		5.9%
Average	16.67	31.25	87.5		76.9		83.1

.

“Consistent” cluster: To be classified as “consistent,” a team must have 80% or more of their sprint updates marked 80 or above in both semesters. Four teams (1, 4, 6, and 12) fall into the category. Among them, two teams (4, 6) led the class in their quality updates in semester I and kept their perfect performance in semester II, implying strong collective abilities and skills in applying the scrum practices. Teams 1 and 12 were able to increase the percentage of quality updates from 88.9% in semester I to 100% in semester II, suggesting a significant effort to consistently and closely follow the scrum practices in the second semester.

“Adapting” cluster: The cluster includes teams that exhibited some level of inconsistent agile application (e.g., sprint updates fell below 80%). Seven teams (5, 8, 10, 13, 14, 15, and 16) belong to the cluster. Among them, teams 14 and 15 registered a 12.5% increase in scrum application from semester I to semester II, while team 13 displayed close to a 100% improvement. However, some of these teams (5, 8, 14, and 16) missed at least one update.

“Challenged” cluster: Teams grouped under the “Challenged” cluster delivered a consistently low level of sprint updates. There are three teams (3, 7, and 17) in the cluster. Despite the fact that all teams were able to improve the percentage of quality updates in semester II, the highest percentage of their quality sprint updates was 37.5% (team 3). The consistently low ratio of quality updates implies that the teams were challenged in their application of the scrum practices in their project and unable to consistently follow the practices in both semesters.

“Regressed” cluster: The cluster includes teams whose agile application dropped from semester I to semester II. Three teams (2, 9, and 11) fall into this category. While team 9 showed a slight decrease from 66.7% to 62.5% in the percentage of quality sprint updates, team 2 registered a more than 60% drop from 55.6% to 37.5%, followed by team 11, whose percentage of quality sprint updates dropped from 33.3% to a meager 12.5%. The downward trend suggests little to no application of the scrum practices in the second semester.

3.2. Team Performance

Team performance was evaluated based on the marks given to the interim report and the final report. Evaluating team performance at the two different times renders a clear picture of changes in team performance over time and enables the examination of the potential influence of the scrum practices. Overall, most of the teams demonstrated improvement in performance from semester I to semester II. In semester I, there were three teams that received marks 90 or above, four that received marks between 80 and 89, eight that received marks between 70 and 79, and two that received marks below 70, resulting in a class average of 76.9 in semester I. In semester II, most of the teams improved their performance (with the exception of team 17), resulting In a class average of 83.1. However, the overall picture does not answer the question “How are a team’s scrum practices related to team performance?”, which is explored in the following section.

3.3. Scrum Application and Team Performance

To discern the potential impact of the scrum application on the quality of student work (team performance in the report marks and the report mark change percentage), each cluster is captured and analyzed below.

“Consistent” cluster: The teams in this cluster demonstrated leading performance in the entire cohort, with an average of 83.4 in semester I and of 92.5 in semester II, an increase of 11%. The lowest mark rose from 78 in semester I to 89 in semester II.

“Adapting” cluster: The seven teams in this cluster exhibited the overall second-to-best performance, with an average of 78.2 in semester I and 84.1 in semester II, registering a surge of 7.5%. The lowest mark increased from 59.6 in semester I to 73 in semester II.

“Challenged” cluster: The three teams in the cluster displayed a mixed picture of team performance. The average showed a slight increase (4%) to 78 (semester II) from 74.9 (semester I). The lowest mark improved from 70.8 to 74.

“Regressed” cluster: The performance of the three teams in the cluster sat at the bottom of the cohort. Despite a 9.1% increase in the average from 66.9 to 73, the lowest marks of the cluster, 60.7 (semester I) and 70 (semester II), remained the lowest in the entire cohort.

3.4. Focus Group Discussion and Online Survey on Agile Challenges

The focus group discussion offers insights into teams’ experience with scrum application. All five teams expressed a general consensus on the value of scrum in improving team member accountability, team collaboration, and the transparency of project progress. One team spoke very highly of the scrum practices:

Collectively, we felt that making short-term schedules on a weekly basis suit our project best. Our team paid close attention to the details in project content outline and this assisted us in defining the activities necessary to keep the project on track.

For the other teams, however, the scrum practices were not viewed so positively. In particular, these teams pointed out the challenges of performing weekly sprints and confusion over the level of details of their sprint updates. Some also expressed the frustration of changing team members’ attitudes and behaviors. One team member indicated:

I suggested that our team should impose strict deadlines and followed the scrum processes closely. However, there was resistance from team members.

Based on the focus group discussion, the RA and the instructor identified eight challenges that the student teams faced in their agile application, which cover all three scrum processes: planning (i.e., estimate time taken for a task, specify individual work allocation, clearly define a task, break down a complex task into subtasks, produce work on the weekly basis), review (e.g., identify what needs to be done, identify deliverable task), and retrospective (i.e., evaluate each other’s work).

3.5. Survey Results

Seventy students shared their rankings of the eight challenges. As shown in Figure 1, the top three challenges are “Produce work on the weekly basis,” followed by “Estimate time taken for a task” and “Specify individual work allocation,” all of which are related to scrum planning.

4. Discussions

This study makes up for the lack of studies on applying client-sponsored projects to teach agile to non-SD students by illustrating how the scrum practices and values were imbued in a client-sponsored project and how teams performed over an eight-month period. The research is also unique in its use of longitudinal qualitative and quantitative data to gain a deep understanding of teams’ application of the scrum practices and team performance.

Pertaining to the first research question, the cohort took foundational courses introducing the scrum methodology and agile project management and were given two weeks to refresh their understanding on the scrum practices. Despite the preparation, the average sprint update mark was 62.9 in the first sprint and fluctuated until sprint 4 (Table 4).

This finding suggests that the introductory lessons on agile may not prepare students well enough to apply scrum in a real-world project. In addition, the result indicates that not only were students underprepared to apply agile, but they were also challenged to consistently adhere to the prescribed scrum practices.

The focus group interview and the online survey shed light on the challenges faced by the teams. As revealed in Figure 1, while the challenges encountered by the students covered all three scrum processes, the top three challenges (“Produce work on the weekly basis,” “Estimate time taken for a task,” and “Specify individual work allocation”) were related to scrum planning. It is possible that the skills of agile application may not be easily acquired through the traditional projects that take place in the academic setting without external client involvement. These challenges suggest the value of client-sponsored projects in training and enhancing students’ agile skills. In addition, this finding indicates the need to differentiate student teams’ scrum application in each of the three processes and discern its impact on team performance.

Despite the dismal start, the majority of the teams found ways to improve their scrum practices, as evidenced in the increased percentage of quality sprint updates from semester I to semester II (Table 5). Aside from diligent efforts from each team, two factors may have contributed to the improvement: (1) the constant engagement from the clients, which motivated and pressured student teams to deliver on a continuous basis, and (2) the detailed feedback given by the instructor to each team’s weekly sprint update might have guided the team’s agile efforts.

The excerpts from two teams’ sprint updates support the assumption (Appendix A). Team 16, in its sprint 5 update, indicated the involvement of both the client and the instructor in their project and acknowledged the importance of both parties in project success. Team 4, in its sprint 2 update, showed that the team provided the client contact persons (Lise and Tracey), the project deliverables, and the timeline, which was deemed crucial to keeping the team on track.

Regarding the second research question, the data analysis on the four clusters of scrum application suggests a positive relationship between the scrum practices and quality team performance. Team clusters that demonstrated a consistent application of the scrum practices outperformed other clusters that did not. The team cluster regressed in their scrum application and fared the worst in class. The correlation between the percentage of agile application in semester I and the marks of teams’ interim reports was 0.23 (non-significant), and the correlation between agile application in semester II and the marks of teams’ final reports was 0.62 (<0.01). The result suggests a strong and positive impact of scrum application on team performance and highlights the importance of consistent agile application in team performance.

This finding is substantiated by a close examination of weekly sprint updates and offers some insights into why the three teams regressed in their agile practices. One sample update from team 2 demonstrates the consistent lack of clear work allocation and sprint review (Appendix B); the sample update from team 17 missed key information needed and was carelessly written (Appendix C); the update from team 9 pointed out the issue of team members falling back to the old habits (Appendix D). All these are indicative of the lack of commitment to the scrum approach and suggestive of difficulties in adjusting members’ behaviors and attitudes to agile.

While previous research recognizes the lack of collaboration and open communication as common issues in student teams [9,42,58], this study offers rich evidence for the significant challenge of teaching agile: applying agile practices requires changes in behaviors and mindsets [5,6]. If not all members of student teams possessed commitment and appreciation for the agile values and practices and were able to break off from the old habit (e.g., delaying work to the last minute), team performance would suffer. Future research could explore creative approaches to improving agile commitment and appreciation from all members in agile teams.

While offering insightful findings on teaching agile using client-sponsored projects, the study is not without its limitations. First, the results of the study shall be carefully reviewed with a consideration of the context. For example, the fact that the average of sprint updates dipped sharply in sprint 10 (Table 4) could be caused by the Christmas and New Year holiday break, a factor beyond the control of the students, which resulted in four weeks of non-activities and attributed to a slow start in semester II. Second, the study investigates agile application in a small number of teams (17). Future research could conduct a large-scale study to establish the significant causal impact of agile application on team performance.

However, as an early study examining agile education in a client-sponsored non-SD project, the findings bear significant implications for future educational efforts on agile and sustainability in education. First, educators could utilize the project design where scrum was successfully integrated into a client-sponsored non-SD project. In particular, the study explicates how the scrum framework [47] was tailored to the project, including the role and responsibilities of external clients, the customization of the scrum schedule, the adoption of the Wrike platform for team project management and collaboration, as well as the evaluation framework of scrum application by student teams.

Second, the study suggests that introductory agile courses may not prepare students well enough for applying the scrum practices in client-sponsored projects. In particular, students may be ill-equipped with agile planning skills. Therefore, it is advisable that educators who consider using client-sponsored projects assess the agile knowledge and skills of all students before the commencement of the project and identify areas where training shall be targeted.

Third, while it is understandable that student teams apply agile practices differently [36], the finding that only a small percentage of teams were able to consistently and closely follow the scrum practices calls into question the sustainability of agile education. To address the issue, a two-pronged approach is proposed. The first calls for a more active role of educators, who could explore innovative approaches to spark student commitment, including a digital standup, variable sprint length, and rotating scrum master [20,59]. Innovative tools (e.g., team collaboration personality) and techniques such as inclusion and agile bootcamps could stimulate team collaboration and open communication [26,60,61]. Second, to better equip students with agile skills and values, educators could supplement introductory courses with client-sponsored projects where students could gain practical knowledge and skills, which will enable them to become effective decision makers in the future.

5. Conclusions

The last two decades have witnessed increasing recognition of the value of agile to improving operational efficiency and sparking innovation in non-SD domains such as business management. While the rising agile adoption in industries results in intensified demand for agile talents, agile education for business management students lacks client- sponsored projects through which in-depth agile skills and experience could be gained. This study makes up for the gap and reports an eight-month endeavor where scrum practices and values were imbued in a client-sponsored project. Through the analysis of qualitative and quantitative data gathered throughout the eight-month period, this study illustrates a large disparity among students in their scrum application, reveals the top challenges faced by students in their scrum application, and explores the impact of scrum application on the quality of student work. The findings of the study set a solid foundation based on which future agile education could be enhanced to strengthen the sustainability of agile education to meet industries’ rising demand for agile talents in non-SD domains.