1. Background
In recent years, data-driven modeling, action, and strategies have become popular, and the education community has witnessed increased interest in data-driven decision-making (DDDM) [
1]. “Driven by data” requires more than just the existence of good data. It requires a meaningful process for developing data frameworks, questions, conducting exploration and analysis, and ultimately, using the data to help arrive at strategies and solutions. It is critical to leverage data-driven decision-making and spurn innovation in higher education by having an optimal strategy that prioritizes data standardization, integration, transparency, quality, and reliability and makes the most of their resources. Data-driven decision-making values and prioritizes decisions supported by high-quality, verifiable data that has been effectively processed and analyzed. Institutions can use a large amount of data being generated from various sources to improve the students’ learning experience, enhance research initiatives, support effective community outreach, and develop campus infrastructure to bring in sustainability [
2].
Early intervention with at-risk students through coordinated student support services can enhance student retention and student success rates and ultimately student satisfaction and enhanced learning. However, faculty productivity levels are just as important as student success rates. Enrollment numbers can often affect faculty productivity levels, which is why a reliable and predictive reporting process is vital. The unique structure and culture of higher education workplaces and the ways in which faculty work is assigned, assumed, and rewarded also contribute to workload mismatch and faculty dissatisfaction [
3]. Departments include faculty members of different career stages, appointment types, and concentration areas. Faculty workloads differ depending on discipline, institution, and the various contributions faculty members can make in domains of scholarship [
4]. Although frameworks for work activities are fixed (e.g., teaching load), other modifiers are discretionary and unregulated. Faculty typically choose whether or not to serve on committees, take on new students as advisees, prepare academic programs for external review, and apply for and pursue externally funded research, to name only a few work activities. The organization of faculty workload and distribution allows for significant unintended ambiguity and or inequities. Research has shown that educational practitioners make decisions based on intuition [
5]. Problems of equity are rarely resolved quickly and without considerable price in faculty morale. Lack of accountability and transparency, ambiguity built into the system, a continuum of preferred/rewarded roles, and lack of department consensus on equity across domains of scholarship can help explain why so many national faculty surveys report high levels of workload dissatisfaction [
6].
It is important to use data to determine the causes of a problem before taking improvement actions. We aim to improve: transparency in what faculty are doing, accountability, clarity in roles and expectations, and flexibility to acknowledge and reward diversity in performance and participation. Transparency increases the sense of accountability and trust between members and leaders, facilitates perceptions of procedural and distributive justice, and leads to greater organizational commitment [
7]. The immediate need is to establish robust data systems, combined with a well-coordinated, multi-stakeholder collaboration across administrative and academic units to drive evidence-based modeling of academic management. Data-driven decision-making will enable administrators to determine successes and challenges, identify areas of improvement, and help evaluate the effectiveness of programs [
8]. Aligning meaningful goals for evidence-based academic management brings with it unique opportunities and challenges. These include assembling high-quality raw data, conducting analysis that ensures results are relevant, and using relevant data to inform academic decisions. It is important to maximize data utilization and interpretation to evaluate past challenges and predict future ones; use both quantitative and qualitative data; understand data limitations; turn to multiple sources of data; and integrate, review, and validate data. Work allocation systems with comprehensive, objective criteria can help resolve inequity. Prior literature has highlighted the use of data to better understand our own operations and pave the way for developing new, innovative approaches for greater institutional efficiency. Transparency increases the sense of accountability and trust between team members and leaders and is conducive to greater organizational commitment [
9,
10].
The objective of our study is to describe the design, development, and implementation of a human-centered, internet-enabled dashboard, with data-driven, evidence-based models of academic development in the context of CUNY Graduate School of Public Health and Health Policy utilizing the SMAART model. To the best of our knowledge, it is the first study outlining the value of implementing collaborative, coordinated, data-driven decision-making approaches across various administrative and academic units towards enhancement of a transparent, equitable, sustainable environment of academic success.
2. SMAART Model
The SMAART model (Sustainable, Multisector, Accessible, Affordable, Reimbursable, and Tailored), a Population Health Informatics (PopHI) framework, is conceptualized using the principles of Data, Information, and Knowledge (DIK); human-centered approach; cognitive fit theory; information processing theory; and humanistic, behavioral, and learning theories to develop a data-driven, evidence-based model of academic management (
Figure 1). The DIK pathway makes data more meaningful [
11,
12]. The principles of a human-centered approach involve: (a) active involvement and understanding of users, (b) understanding task requirements, (c) appropriate allocation of function between user and system, (d) iteration of design solutions, and (e) multidisciplinary design teams [
11,
12]. Understanding users is an important aspect for creating useable health information applications. Task classification has shown to create useful applications [
13]. Cognitive fit theory (CFT) explains how graphical displays affect the decision processes [
14,
15]. CFT depends upon fit between information presentation and tasks used by decision maker. Cognitive fit identifies an appropriate representation for a given task performed by users [
15]. Information processing theory facilitates presentation of information as a meaningful unit [
16]. When the information presentation matches the task, it produces faster and more accurate results. These benefits translate into system and task-related performance factors. Learning, behavioral, and humanistic theories suggest that information needs to be highly interconnected, relevant to the learner, in multiple content formats and with feedback given based on responses [
17].
In the SMAART Model, Sustainability highlights the use of real data and evidence to ensure that developed solutions are Sustainable, Multisectoral indicates integration of data from disparate sources to influence outcomes, Accessible indicates making information and services accessible to all, Affordable indicates creating solutions that are affordable to all, Reimbursement reflects upon return on investment with cost-effective solutions, and Tailored indicates meeting the needs of the individuals across diverse settings.
3. Methods
The alignment of academic and student affairs within the CUNY Graduate School of Public Health and Health Policy was institutionalized in July 2018 with the goal of a collaborative leadership model to ensure its optimal functioning. This alignment brought with it several challenges and opportunities.
One challenge was to define roles and responsibilities as there were several functions that overlapped across different student and academic affair units.
Second, it was important to listen to the experiences and recommendations of the administrative staff of various units who were responsible for performing student functions in various capacities. This required continuous, ongoing coordination and collaboration across multiple individuals from different units with different perspectives incorporating the principles of diversity, inclusiveness, and equity. These activities involved meticulous and continuous oversight and involved several rounds of deliberations and meetings with various stakeholders.
Third, data across faculty workload including teaching, research buy out, instructional class size, service, and new course development all was recorded across different formats and platforms, resulting sometimes in both data being incomplete or inaccurate. Each data source included different data fields in different data formats.
Fourth, similar data-related challenges were observed across student data related to various student support services including academic probation, student advisement, International Student Office, experiential learning, and career services. The data being recorded in various formats limited its integration, hindering optimal decision-making to provide tailored and student services.
We implemented combined principles of human-centered design, cognitive theory, information processing theory and learning, behavioral, and humanistic theories to conduct user, task, and requirement analysis.
Users included staff of the various student and academic support services (
Figure 3). Understanding user characteristics, needs, and preferences are critical for developing human-centered platforms. In addition, user tasks and operations need to be clearly defined and collectively evaluated to meet the goals of data exploration, analysis, and knowledge development. We gathered information on how students set up appointments with the various student support services, data elements recorded on existing intake forms, and methods of tracking these appointments.
To develop data-driven, evidence-based model of academic management by designing, developing, and implementing SMAART SPH Outcomes Dashboard, the following steps were performed.
The following steps were performed towards developing a model of data-driven, evidence-based academic management.
The first step towards development of an evidence-based model of academic management involved formation of several working groups that would bring together representation of various academic and student affairs and other administrative services as needed. The working groups included Academic and Student Affairs Advisement, Academic Council, curriculum implementation and innovation, and online education (
Table 1). Each working group was led by the Senior Associate Dean of Academic and Student Affairs with representation from different academic and administrative units (
Table 1). The Academic and Student Affairs Advisement working group was aimed to discuss barriers and challenges related to student advisement across various academic and administrative units. Student advisement topics discussed were related to the choice of electives, course enrollment, financial aid and scholarship, field practicum placement, and software and hardware issues. The aim of the academic council working groups was to understand faculty needs, opportunities, and challenges related to implementation of new curricula and those involving approval of new academic programs. Issues related to faculty workload were also discussed. The curriculum implementation and innovation working group discussed issues related to the implementation of the new MPH curriculum and course offerings and examined these from the perspective of registration, advisement, and field practicum. Further, the focus was to discuss the tracking of online course development. Similarly, the goal of the online working group was to finalize instructors for developing the new online courses and related contracts and needed coordination with the Office of Legal Affairs. Tracking of the progress of the online course development and quality assurance to review the quality of content was finalized. This process helped to develop a repository of existing data sources and identify gaps in the availability of data elements critical towards evidence-based decision-making.
Activities planned during several of these working groups were aimed to ensure effective communication, collaboration, and coordination between various academic and administrative units.
- 2.
Student data
To provide sustainable, multisectoral, accessible, affordable, reimbursable, and tailored solutions for evidence-based academic management, several disparate data sources were recorded. These included student enrollment data, graduation data, academic probation data, and international student data. The goal was to integrate key data fields across all these different data sources in a format that is readily analyzable, useable, and can provide accessible and meaningful information to address the needs of the diverse stakeholders. Each data set had a unique set of variables that were essential and could generate meaningful insights (
Figure 4). The unique ID across these various datasets was EMP ID and email address.
- 3.
Faculty workload data
Faculty members’ main responsibilities are teaching and research; however, they also manage outreach, service activities, and many times administrative work. Extensive faculty work also takes place outside of the classroom, such as advising, mentoring, writing grants, running labs, and guiding students through independent study. Data plays a pivotal role in contributing to the development of a transparent and an equitable model of academic development. However, several challenges related to the faculty workload data included:
What are the key components of faculty workload data (teaching, research, etc.?).
Lack of reporting and integration of teaching, research and administrative workload data in a timely manner.
Clarification for classroom staffing such as rules for adjunct and college assistant allocations based on class size.
Clearly defined compensation/credit buy-out for designated leadership roles.
Framework for online course development and related compensation/course credit.
Additional challenges related to the faculty workload data included (i) disparate data sources, (ii) heterogeneity in data formats, (iii) validation of data, and (iv) integration of key data components of faculty workload such as teaching, research, and other administrative services. For implementation of new online courses, new academic programs, and new MPH curriculum, along with teaching of the existing curriculum, there existed a need for a clearly defined, fair, equitable faculty workload policy. These efforts required communication with all stakeholders including department chairs and school-wide faculty, budget, and finance units so that an optimal and an agreeable faculty workload policy and policy for classroom staffing could be implemented.
Prior faculty workload data was gathered, and information on various data variables including classroom size, additional teaching assistance, and research effort was all gathered to generate individual faculty workload profiles. Each faculty within the school submitted their faculty workload, and their approval was sought regarding the data validity. In addition, continuous ongoing deliberations with the department chairs and their department faculty and school leadership was done, and supporting data were presented to formulate new policy. Several iterations of the policy were done at the department and then at the school-wide level before a formal policy was institutionalized and ready to be implemented in the fall 2019. Several challenges were encountered in the management and tracking of faculty workload data and included data inconsistency, data redundancy, data completeness, and data reporting (
Figure 5). We focused on workload inequity within departments because of the critical role departments play in faculty retention, satisfaction, recognition, and professional growth [
18]. Senior faculty and department chairs can reduce workload inequity with routine practices and conditions that can shape a more positive work allocation experience.
- 4.
Student services data
Various student support services recorded data across different platforms such as Microsoft Excel, Google Docs, FlexBooker, Calendly, and LibApps, thus limiting the integration of student service data in a meaningful manner. Some of the common data variables recorded across several student services such as academic advisement, career services, International Student Office, library, Office of Experiential Learning, and writing and quantitative services included first name, last name, email, degree, program, and open-ended information as notes (
Table 2). Additional data fields were identified that would help better quantify the student needs and allow us to provide essential services and track its usage.
5. Discussion and Conclusions
Using the principles of Data, Information, and Knowledge, combined with principles of a human-centered approach and available data sources, we designed, developed, and implemented an internet-enabled interactive SMAART SPH Outcomes Dashboard to enhance a model of evidence-based academic management.
The SMAART SPH Outcomes Dashboard addresses the need to establish a cohesive, coordinated, and multi-stakeholder collaboration and aims to contribute towards a transparent, equitable, data-driven, and evidence-based model of academic management. The results of our approach describe the importance of reviewing, compiling, and validating previous faculty workload as well as implementing a new approved faculty workload policy, policy for compensation of designated leadership roles, and classroom staffing with rules of adjunct and college assistant allocations. As such, departments that make data visible (e.g., number of advisees, committees, class sizes) and replace informal processes with readily available systematized policies are likely to have faculty with higher levels of satisfaction.
Faculty are often evaluated on perceptions of their teaching effectiveness, their service activities, and their research productivity [
20]. However, faculty must properly allocate and manage their time to meet their institutional standards in each area. Administrators and department chairs have the means to facilitate or hinder faculty research productivity through teaching and service assignments [
20].
Leadership is critical in determining the results of change-management programs. During periods of change management, leaders must demonstrate skills and behaviors that deepen trust and commitment, align strategies, focus actions, expand capabilities, and clarify progress. The SMAART model facilitates collaborative, cohesive, and well-coordinated decision-making. It guides relevant stakeholder engagement, recording of meaningful data, and generating meaningful information to communicate relevant findings back to the stakeholders for input and facilitate a transparent, equitable, evidence-based model of academic management.
In a data-driven culture, decisions are based on factual, verifiable information with an underlying data-driven identification of trends and projections for future needs. The availability of ‘big data’ for large-scale analysis and the development of advanced technologies and practices have established new modes of quantitative knowledge production and decision-making [
21].
Silos across an institution are far too common, but there is a good chance that multiple departments may be facing similar challenges which can be revealed by data available to all. Data-driven breakdown of silos can facilitate inter-departmental collaboration towards the common goals of student success and knowledge creation. Similarly, a data-driven culture requires storing consistent, reliable, and validated data in a systematic way and specifically with savings in time and cost. In addition, the use of historical data can guide predictions on what is to come in the future. The datafication of teaching in higher education remains a key focus for future studies [
21]. The flow in data use in education has provoked widespread concern with the development of teachers’ (and students’) data literacy. Teachers need the data skills to be able to use new data systems and to be able to comprehend and utilize the data they produce for purposes of professional improvement, assessment of students, and better institutional management. Hence, critical data literacy should be a requirement of university managers and administrators.
One of the limitations of the proposed study is to describe the broader, long-term impact of the SMAART Outcomes Dashboard to facilitate equitable, data-driven, evidence-based decision-making.
Future work is planned to evaluate the practical application and value as well as effectiveness of the SMAART model in facilitating the implementation of faculty workload policy, classroom size, and additional teaching assistance in a uniform, equitable, transparent manner across multiple departments and increasing faculty satisfaction. In addition, the impact of the SMAART model will be measured to assess timely, at-risk intervention of students going into academic probation and how the various student services coordinate, collaborate, and communicate with each other, enhancing not only student outcomes but also overall institutional effectiveness.
The aim of this paper is to describe the approach that higher education institutions could possibly follow to implement evidence-based, data-driven thinking in a sustained manner. Decisions in higher education institutions have an impact on various stakeholders. It is critical to make the best-informed decision possible. The evidence-based practice facilitates evidence from multiple sources in a conscientious, explicit, and judicious manner [
22].