Next Article in Journal
Prediction of Students’ Use and Acceptance of Clickers by Learning Approaches: A Cross-Sectional Observational Study
Previous Article in Journal
Narrative Inquiry on the Teaching of STEM to Blind High School Students
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Academic Advising Systems: A Systematic Literature Review of Empirical Evidence

1
Department of Computer Science and Engineering, University of Applied Sciences of Thessaly, Larissa 41110, Greece
2
School of Science & Technology, Hellenic Open University, Patras 26335, Greece
3
Department of Business Administration, University of Applied Sciences of Thessaly, Larissa 41110, Greece
4
Innopolis University, Kazan 420500, Russia
*
Author to whom correspondence should be addressed.
Educ. Sci. 2017, 7(4), 90; https://doi.org/10.3390/educsci7040090
Submission received: 12 September 2017 / Revised: 8 December 2017 / Accepted: 13 December 2017 / Published: 19 December 2017

Abstract

:
This paper aims to provide the reader with a comprehensive background for understanding current knowledge on Academic Advising Systems (AAS) and its impact on learning. It constitutes an overview of empirical evidence behind key objectives of the potential adoption of AAS in generic educational strategic planning. The researchers examined the literature on experimental case studies conducted in the domain during the past ten years (2008–2017). Search terms identified 98 mature pieces of research work, but inclusion criteria limited the key studies to 43. The authors analyzed the research questions, methodology, and findings of these published papers and categorized them accordingly. The results have highlighted three distinct major directions of the AAS empirical research. This paper discusses the emerged added value of AAS research and highlights the significance of further implications. Finally, the authors set their thoughts on possible uncharted key questions to investigate both from pedagogical and technical considerations.

1. Introduction

With the advent of flexible curriculum systems in many Higher Educational Institutions (HEI) and an ever wider variety range of courses and programs being offered, there is a present need to ensure that students make the best use of available information to make more informed decisions regarding their academic plan [1]. Besides required courses, which are compulsory for each student to be taken, HEI also offer elective courses, but generally students lack information about the objectives and the content of the course and often fail to take the appropriate ones for their academic plan. Moreover, because the number and variety (in backgrounds, in knowledge, in goals) of students is also expanding rapidly, it is increasingly important to tailor learning processes to students, since the same learning pathway is unlikely to best serve all students [2].
HEI usually employ guidance counselors, people who are tasked with helping students making their choices. Three principle models of advising include developmental advising, prescriptive advising, and intrusive advising, each of which is informed by the goals of the advisor-student interaction [3]. In all approaches, this support is based on the establishment of educational collaboration between students and their academic advisors. Advisors assist students by guiding them through the university’s educational requirements, helping them schedule the most suitable modules, introducing them to pertinent resources, promoting leadership and campus involvement, assisting in career development, helping them with the timely completion of their studies, and helping them find ways to make their educational experience personally relevant [4]. However, in practice the counselors are often overloaded with too many students and not enough time, and some students are not satisfied with the quality of academic advising that the counselors provide [5]. Good advising yields a good outcome in terms of understanding, planning, and applying strategies for academic success, while bad advising will be frustrating and may have a damaging effect on students’ progress [6].
In order to support the educational process and to relief the HEI actors, a software system is needed that will handle the advisory process in an efficient and effective way. An AAS can serve as a strategic partner responsible for the process of supporting, motivating the student’s study plan, and assisting in the achievement of their educational goal [7]. However, unlike most existing recommendation systems, AAS require dealing with a large decision space, which grows combinatorially with the number of courses, programs, and the students’ different backgrounds, knowledge, and goals but is also subjected to many constraints (e.g., course prerequisites, maximum credit hour load, course priorities, etc.). In [8], the authors describe the conceptual framework of a web-based AAS. In their study, more than 60% of users, consisting of 361 students and 155 faculty members, agreed that the tasks of academic-related matter such as course registration, course selection, academic progress, course information, course scheduling, plan of study, and academic calendar should be included in the AAS. They also discuss the importance of making AAS that are more than data repositories and including more intelligence so the systems are able to provide reliable advice when helping the advisor and the student alike.

2. Motivation and Rationale of the Study

The motivation for this review is derived from the fact that empirical evidence is required for theoretical frameworks to gain acceptance in the scientific community [9]. A search in relevant literature did not reveal any reviews of empirical evidence of the added value of research in the specific domain. Consequently, there was a need to supply the audience with an accredited overview. This paper aims to fill that gap.
The value of any single study is derived from how it fits with and expands previous work, as well as from the study’s intrinsic properties. Thus, putting together all the unbiased, credible results from previous research would be a step towards understanding the whole picture and constructing a map of our knowledge of the domain. In a sense, the rationale of our study was to manage the overwhelming amount of publications through the critical exploration, evaluation, and synthesis of the previous empirical results that merit reflection.
This paper’s goal is to carry out a systematic review of empirical evidence in order to contribute towards
  • a complete documentation of the applied research approaches so far;
  • a feasibility study that captures the strengths and weaknesses of research in the domain, and
  • the identification of possible threats, and thus to motivate the research community to redefine or refine related questions or hypotheses for further research (opportunities).

3. The Research Questions

The following research questions need to be addressed and are distinguished into primary (generalized: set to fulfill the goals of the review) and secondary (sub-objectives/specific: refine the primary—explanatory):
  • RQ1 (Primary)—Research Objectives: What are the basic research directions of AAS up to now (in terms of measurable metrics) and which approaches do researchers follow to achieve these goals?
    • RQ1.1 (Secondary)—What are the most significant results from past AAS research that constitute empirical evidence with regard to their impact on the learning process?
    • RQ1.2 (Secondary)—Interpretation of the results: What do these results indicate regarding the added value of this technology?
  • RQ2 (Primary)—Future challenges: Which other emerging research approaches should be explored in the AAS research area?

4. Research Methodology

Our process for finding and including or excluding papers is summarized in Figure 1. To increase our coverage, we searched for relevant papers by conducting both a systematic search of available research paper databases and by “snowballing”, starting with a set of core papers believed to be in-scope, and expanding our set of consideration based on papers referenced by these papers.
Two main methods were initially employed for this search. The first was to use our University’s academic search engine, which covers several academic databases such as Xplore, ScienceDirect, SpringerLink, and Wiley. The second was to consult Google Scholar (http://scholar.google.com/), which covers more publications but is less structured and provides also more low quality material. Search terms used included “academic advising systems, academic planning” in combination with “universities, higher educational institutions”, and all words were used in different combinations. The search process spanned from March 2017 to August 2017. The parameters of the search were bound within the last ten years (2008–2017), in which emergence and adoption of AAS has grown based on the number of relevant research papers.
Due to the orientation of this work towards the practical implementation and exploitation of AAS, at the end of the data collection stage, the authors explicitly determined the article inclusion/exclusion criteria (Table 1).
This first round of analysis led to the identification of only 19 unique papers. Due to such a limited number of publications, the literature base was expanded by applying a snowballing technique [10] to the initial set of papers. The reference list of each of the papers was screened for relevant contributions and 43 papers were selected for detailed analysis at the end of the process. Saturation was used as the stop criterion: we stopped the search when new papers no longer provided new challenges. The selected papers were later analyzed by classifying their content and contribution in relation to the AAS.
Next, the authors proceeded on with an article classification according to the adopted research strategy (category), recommendation technique (method), research objectives (goals), and results. Finally, we used non-statistical methods to evaluate and interpret findings of the collected studies, and to conduct the synthesis of this review.

5. Results

In this section, the authors present their findings based on the analysis of the published case studies.
According to the followed research strategy, most of the published case studies are exploratory or experimental studies. Some of them are evaluation studies, while others are empirical studies or surveys. Furthermore, the research topics differ from study to study, but most of them focus on science, technology, engineering, and mathematics (STEM).
One important parameter is the research method adopted by authors to propose personalized academic recommendations. In the field of AAS, the most popular filtering algorithms are knowledge-based, followed by Computational intelligence-based, Collaborative filtering-based, Hybrid, and content-based. Table 2 displays the classification of the key studies according to the recommendation method they adopt.
The article classification according to the research objectives (goals) is illustrated in Table 3. As seen in this table, the majority of studies investigate issues related to Selecting Courses followed by Long-Term Academic planning and Choosing Programs/Majors.

6. Key Studies Analysis

In this section, the authors present the findings of the review process and answer the initial set research questions RQ1 and RQ1.1. The rest of the research questions (mostly the results of the case studies and their comparative evaluation, as well as current and future trends, possible gaps, and new research directions) are discussed in the next section.
RQ1: What are the basic research directions of AAS up to now (in terms of measurable metrics), and which approaches do researchers follow to achieve these goals?
Firstly, we used technology roadmapping to explore the relationships between technological resources, objectives, and the changing environment [49]. Figure 2, Figure 3 and Figure 4 show the frequency of the categories listed on Table 2 and Table 3 over the last ten years.
Looking at Figure 2, Figure 3 and Figure 4, we can observe some trends. In Figure 2, it seems that publications related to AAS peaked in 2010–2012, but is now beginning to rise again. The same could be said for “Knowledge-based” research method in Figure 3 and for “Selecting courses” research objective in Figure 4. However, as the counts for the remainder of the categories captured in these charts are very low overall, other trends may not be significant.

6.1. Choosing Programs/Majors

Major selection is a very important step in the academic student life. Authors in [7,36], implemented a case based reasoning (CBR) system that recommends to the candidate the most suitable major, after comparing the historical cases by the student case. Paper [42] presents two novel nearest neighbor-like classification algorithms for program recommendation, which provide a program planning service to academic advisors and students of post-secondary institutions, while paper [32] proposes an auto-decision system, which helps distance education students choose their majors. Furthermore, the basic idea in paper [29] is to design a model for testing and measuring the student capabilities like intelligence, understanding, comprehension, and mathematical concepts, plus his/her past academic record and his/her intelligence level, and applying the results to a rule-based decision support system to determine the compatibility of those capabilities with the available faculties/majors.
A HEI is always interested in predicting the number of students into programs, because it is important for the school to track the interest of students in program studies so as to ensure that adequate classroom seats are available, appropriate instructors are available, and an appropriate schedule is produced for the students. Paper [38] presents an approach using hybrid recommendation based on data mining techniques and knowledge discovery rules, while papers [40,46] propose a hybrid model of neural network and decision tree classification for tackling college admissions prediction problems for each available program.
Paper [21] suggests an expert system implemented and tested using Oracle Policy Automation (OPA) software for scholarship recommendation and eligibility checking.

6.2. Selecting Courses

As seen from Table 3, selecting courses is a primary research objective. One crucial issue in this category that authors attempt to address is how to propose careful recommendations for different courses according to students’ goals and preferences. More specifically, the authors seek to prepare course lists that satisfy several student and university constraints, some of which depend on individual student cases [2,5,6,22,24,27,28,50]. In some case studies, authors choose to apply computational, intelligence-based algorithms to reach a degree of automatic advising by combining genetic algorithms with decision trees for developing the short-term curricular schedule, as well as by combining perception marks with the registered courses [41] or by assisting in data mining and intelligent adaptive fuzzy logic for implementing an elective course suggestion system [44]. In paper [37], the authors present recommendations of student courses using ant colony optimization and concluded that their solution is promising, since it overcomes most of the disadvantages of classical approaches based on collaborative filtering in terms of performance. AAS proposed in [24] employ recommendations based on decision support tools, while other authors [33,34], use expert systems combined with semantic infostructures [22,28,47], or database management systems [6]. The course recommendation problem gets more complex when considering a large number of courses that have different weights, values, and priorities addressed by the Course-Petri net, a specialization of Petri net that is used on paper [26] as the foundation for development of an advising system. Student diversity is another aspect that complicates the problem [13,14]. For example, authors in [13] introduce an XML user-based collaborative system called Automatic Academic Advisor, which advises a student to take courses that were taken successfully by students with the same interests and academic performance. In another case study, K-means algorithm has been used to determine the similarity of the students [14].
Another orientation is anticipating students’ future course selection as a means of short/mid-term demand forecasting [12,31]. Paper [1] proposes a two-stage user-based collaborative filtering process using an artificial immune system for the prediction of student grades, along with a filter for professor ratings in the course recommendation for college students.

6.3. Long-Term Academic Planning

Long-term academic planning is a challenging and time-consuming task involved in AAS helping a student prepare a complete study plan towards graduation. More importantly, because the number and variety (in backgrounds, in knowledge, in goals) of students is expanding rapidly, it is more and more important to tailor course sequences to students, since the same learning pathway is unlikely to best serve all students [16,17,23,25]. For example, the system described in [15] implicitly implements, via the decision tree, many academic rules and allows a systematic and exhaustive browse of the different student plan instances, and it permits a methodological assessment and measurement of the appropriateness of a given student academic plan. Work in [18] follows mathematical optimization to generate a set of optimal or near optimal alternative plans that are of a similar quality and yet structurally different. Human intelligence is then employed to analyze these alternatives and to either approve one of them or refine the problem settings for generating further solutions.
To reduce the time-to-graduation, it is therefore of paramount importance for the student to elect courses in a foresighted way by taking into account the possible subsequent course sequences (including which courses are mandatory and which ones are not, and the course prerequisites) and when the various courses are offered. The problem of planning the elective courses is modelled in [20] using integer programming and three different solution approaches are suggested, including a Branch-and-Price framework using partial Dantzig–Wolfe decomposition.
RQ1.1: What are the most significant results from past AAS research that constitute empirical evidence with regard to their impact on the learning process?
According to the research objectives explored by the authors, Table 4a displays a categorization of the algorithmic-oriented findings from the collected studies. Table 4b displays a categorization of the pedagogy-oriented findings.

7. Discussion and Future Research

We examined the frequency of AAS publications and categories derived from Table 2 and Table 3 in Figure 2, Figure 3 and Figure 4. AAS publications have peaked in 2011 and declined in 2013 but now are beginning to rise again. “Knowledged-based” approaches are on the rise, peaking in 2010–2012. “Selecting courses” are also on the upwising, rising in 2015. Approaches for “Choosing Programs/Majors” show a small decline, while the remainder of the categories does not show obvious trends. From the former analysis, it becomes apparent that, recently, the educational research community has started applying sophisticated algorithmic methods on AAS. As seen in Table 4, the landscape of the AAS research combines diverse and often conflicting aspects and results; however, the authors have highlighted three distinct major axes of the AAS empirical research including:
  • Pedagogy-oriented issues (e.g., student modeling, prediction of performance, assessment and feedback, reflection and awareness): Several studies focus on pedagogically meaningful analysis of students’ data in order to shed light on the whole picture. Academic advising, as a teaching and learning process, requires a pedagogy that incorporates the preparation, facilitation, documentation, and assessment of advising interactions.
  • Learning analytics (e.g., content analysis, discourse analytics, prediction and information visualization): A number of studies combine institutional data, statistical analysis, and predictive modeling to create intelligence upon which learners, instructors, or administrators can change academic behavior.
  • Educational Data Mining (e.g., data mining, machine learning, and statistics): Several studies focus on techniques, tools, and research designed to automatically extract meaning from large repositories of data generated by or related to people’s learning activities in educational institutions.
These three axes are not completely autonomous, since significant overlaps may occur. However, this statement could only constitute a limitation that does not reduce the added value of the findings.
RQ1.2: What do these results indicate regarding the added value of this technology?
One of the most important goals of the systematic review was to reveal the added value of the field explored. From the above findings, it follows that analysis of factors that influence academic decisions, student academic profile, and preferences in order to “recommend” the appropriate learning resources has always been a request. In many cases, a typical software solution to AAS includes a rule-based expert system. However, the dynamic nature of program requirements might turn maintenance of the system into a crippling task. Implementing a user-based collaborative AAS is an appropriate choice, although a major problem limiting the usefulness of such a system that should be addressed is the sparsity problem, which refers to a situation in which data are insufficient to identify similarities in students’ interests. AAS research results indicate that hybrid and content-based AAS are also well suited to the academic advising domain, because these approaches allow the system to adapt to the changes.
Researchers set academic advising within limits in which previously it was almost impossible to infer recommendations automatically, due to the high levels of complexity that such a process demands. In today’s advanced learning contexts, the AAS research community determines simple and/or sophisticated approaches for recommending learning resources and explores their capabilities by tracking actual changes and the progress of the learning process. The goal is to identify the most significant factors in order to develop better systems. These systems will allow HEI and students to evaluate and adjust their learning strategies and improve their performance in terms of learning outcomes.
Moreover, the learning analytics dimension and the opportunity of applying educational data mining approaches are also explored with encouraging results. Consequently, the research community could gain insight into the learning mechanisms that previously were a “black box”.
RQ2: Which other emerging research approaches should be explored in the AAS research area?
Complementary, the literature overview has revealed a number of unexplored issues in this rapidly growing domain, including (but not limited to) the following:
  • Many modern educational models (for example Accelerated Study in Associate Programs (ASAP) [51] and Guided Pathways (GP) [52]) share an emphasis on acceleration, programs that offer fewer choices and more support, greater transparency of paths to completion for students, and more mandatory and intrusive advising from day one through completion [53]. In their book “Redesigning America’s Community Colleges”, Bailey, Jaggars, and Jenkins described the idea of Guided pathways, a framework around which to structure program maps, meta-majors, e-advising, and early alert systems [54]. AAS are critical to enabling the kind of monitoring and support demanded by these models and must be understood as tools that are part of the broader reform. HEI need to carefully consider and plan how to change advising structures and daily practices so that existing advisors can leverage the potential of emerging AAS research trends to improve student outcomes.
  • Technology acceptance is also a well addressed issue in educational research. Regarding AAS acceptance, authors in [55] proposed a model that considers mainly two parameters: usability and efficiency. However, more parameters should be explored in order to create a reliable AAS acceptance model, e.g., effectiveness, maintainability, and portability. Researchers from the AAS domain could also examine respective models that are suitable for the purposes of AAS.
  • The review process yielded very few articles related to scholarship recommendation and eligibility checking, exploring life, and career goals. Assisting students in the clarification of their life/career goals means helping students explore and define plans for the realization of these goals and evaluating the progress of their efforts. It would be interesting to take advantage of the plethora of results from AAS research by introducing innovative educational recommender systems in these areas.
  • One primary way of assisting student’s career development is by helping them understand their own intrinsic interests and abilities through self-exploration and career exploration [56]. In this context, existing literature highlights a need to examine AAS in a more holistic manner, one that considers the connected nature of student’s interests, skills, and personality type. For example, one of the most frequently used classification systems guiding personality type exploration that can be utilized by AAS researchers is Holland’s [57] theory of vocational personality types and work environments.

8. Conclusions

Previous works on AAS research provided significant insight into the conceptual basis of this rapidly growing domain. However, these studies did not conduct an analysis of actual research results. The current paper presents a systematic review of empirical evidence of AAS research. We searched the literature and gathered representative, mature, and highly-cited articles of real case studies with actual data from AAS domain. The analysis of selected case studies and their results shed light on the approaches followed by the respective research communities and revealed the potential of this emerging field of educational research. Along with the arising opportunities, we discovered a number of gaps that require the researchers’ attention. Table 5 illustrates our findings regarding the strengths, weaknesses, opportunities, and threats (SWOT) of AAS research.
Every HEI needs to have effective academic advising mechanisms in order to increase student development, which in turn can benefit enrollment, retention, and graduation rates. Many HEI offer AAS designed to help students and their academic advisors recommend learning resources and review degree requirements and the student’s progress towards the intended degree. However, existing advising support software tools can augment the student-advisor relationship, but cannot and should not replace in-person advising [6,8,16,18,58]. This is the point at which learning science, psychology, pedagogy, and computer science intersect. The issue of understanding the deeper learning processes and recommending them remains in the middle of this cross-path.
This work can be beneficial for several types of readers. For researchers interested in AAS, this paper helps one to build upon existing work, avoiding the proverbial ‘reinvention of the wheel’, helping to understand trends, and guiding efforts towards new directions. For practitioners, this paper helps demonstrate the ways in which AAS approaches can be integrated into existing system development paradigms, offering ideas on how academic advising can be adopted in practice, including pointers to work containing further details. We believe that this active research area will continue contributing with valuable pieces of work towards the development of more efficient and effective advising services to both learners and HEI.

Author Contributions

Three authors contributed equally to the study.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Chang, P.; Lin, C.; Chen, M. A hybrid course recommendation system by integrating collaborative filtering and artificial immune systems. Algorithms 2016, 9, 47. [Google Scholar] [CrossRef]
  2. Xu, J.; Xing, T.; Member, S.; Van Der Schaar, M. Personalized course sequence recommendations. IEEE Trans. Signal Process. 2016, 64, 5340–5352. [Google Scholar] [CrossRef]
  3. Noaman, A.Y.; Ahmed, F.F. A new framework for e academic advising. Procedia Comput. Sci. 2015, 65, 358–367. [Google Scholar] [CrossRef]
  4. Pizzolato, J.E. Complex partnerships: Self-authorship and provocative academic-advising practices. NACADA J. 2006, 26, 32–45. [Google Scholar] [CrossRef]
  5. Unelsrød, H.F. Design and Evaluation of a Recommender System for Course Selection. Master’s Thesis, Institutt for Datateknikk og Informasjonsvitenskap, Trondheim, Norway, June 2011. [Google Scholar]
  6. Daramola, O.; Emebo, O.; Afolabi, I.; Ayo, C. Implementation of an intelligent course advisory expert system. Int. J. Adv. Res. Artif. Intell. 2014, 3, 6–12. [Google Scholar] [CrossRef]
  7. Mostafa, L.; Oately, G.; Khalifa, N.; Rabie, W. A case based reasoning system for academic advising in egyptian educational institutions. In Proceedings of the 2nd International Conference on Research in Science, Engineering and Technology (ICRSET’2014), Dubai, UAE, 21–22 March 2014. [Google Scholar]
  8. Ishak, I.B.; Lehat, M.L.B. A conceptual framework of web-based academic advisory information system. In Proceedings of the 2012 IEEE Symposium on Humanities, Science and Engineering Research, Kuala Lumpur, Malaysia, 24–27 June 2012; pp. 957–961. [Google Scholar]
  9. Follette, W.C.; Houts, A.C. Models of scientific progress and the role of theory in taxonomy development: A case study of the DSM. J. Consult. Clin. Psychol. 1996, 64, 1120–1132. [Google Scholar] [CrossRef] [PubMed]
  10. Goodman, L. The annals of mathematical statistics. JSTOR 1961, 32, 1. Available online: http://www.jstor.org/stable/i312806 (accessed on 14 December 2017).
  11. Lin, F.; Leung, S.; Wen, D.; Zhang, F.; Rory, K. e-Advisor: A Multi-Agent System for Academic Advising. Available online: http://io.acad.athabascau.ca/~oscar/pub/ABSHL2007.pdf (accessed on 14 December 2017).
  12. Poeppelmann, D. A refined case-based reasoning approach to academic capacity planning. In Proceedings of the 6th Conference on Professional Knowledge Management: From Knowledge to Action, Bonn, Germany, 21–23 February 2011; Available online: https://www.semanticscholar.org/paper/A-Refined-Case-Based-Reasoning-Approach-to-Academi-P%C3%B6ppelmann/4c8e926dc484c684e826248ece160e4a0475c624?tab=abstract (accessed on 14 December 2017).
  13. Dhabi, A.; Advisor, A. Automatic Academic Advisor. In Proceedings of the 2012 8th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom), Pittsburgh, PA, USA, 14–17 October 2012; pp. 262–268. [Google Scholar]
  14. Ganeshan, K.; Li, X. An intelligent student advising system using collaborative filtering. In Proceedings of the 2015 IEEE Frontiers in Education Conference (FIE), El Paso, TX, USA, 21–24 October 2015. [Google Scholar]
  15. Werghi, N.; Kamoun, F.K. A decision-tree-based system for student academic advising and planning in information systems programmes. Int. J. Bus. Inf. Syst. 2009, 5, 1–18. [Google Scholar] [CrossRef]
  16. Roushan, T.; Chaki, D.; Hasdak, O.; Rasel, A.A.; Rahman, M.A.; Arif, H. University Course Advising: Overcoming the challenges using decision support system. In Proceedings of the 2013 16th International Conference on Computer and Information Technology (ICCIT), Khulna, Bangladesh, 8–10 March 2014; pp. 8–10. [Google Scholar]
  17. Ai-Nory, M.T. Simple decision support tool for University Academic Advising. In Proceedings of the 2012 International Symposium on Information Technology in Medicine and Education (ITME), Hokkaido, Japan, 3–5 August 2012; Volume 1, pp. 53–57. [Google Scholar]
  18. Mohamed, A. A decision support model for long-term course planning. Decis. Support Syst. 2015, 74, 33–45. [Google Scholar] [CrossRef]
  19. Koutrika, G.; Bercovitz, B.; Garcia-Molina, H. FlexRecs: Expressing and combining flexible recommendations. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data, Providence, RI, USA, 29 June–2 July 2009. [Google Scholar]
  20. Kristiansen, S.; Sørensen, M.; Stidsen, T.R. Elective course planning. Eur. J. Oper. Res. 2011, 215, 713–720. [Google Scholar] [CrossRef]
  21. Engin, G.; Aksoyer, B.; Avdagic, M.; Bozanlı, D.; Hanay, U.; Maden, D.; Ertek, G. Rule-based expert systems for supporting University Students. Procedia Comput. Sci. 2014, 31, 22–31. [Google Scholar] [CrossRef] [Green Version]
  22. Henderson, L.K.; Goodridge, W. AdviseMe: An intelligent web-based application for academic advising. Int. J. Adv. Comput. Sci. Appl. 2015, 6, 233–243. [Google Scholar]
  23. Mohamed, A. Interactive decision support for academic advising. Qual. Assur. Educ. 2016, 14, 251–267. [Google Scholar] [CrossRef]
  24. Feghali, T.; Zbib, I.; Hallah, S. A web-based decision support tool for academic advising. Educ. Technol. Soc. 2011, 14, 82–94. [Google Scholar]
  25. Prof, A.; Shakeel, M. Student advising & planning software. Int. J. New Trends Educ. Implic. 2012, 3, 158–175. [Google Scholar]
  26. Hashemi, R.R.; Blondin, J. SASSY: A petri net based student-driven advising support system. In Proceedings of the ITNG2010—7th International Conference on Information Technology: New Generations, Las Vegas, NV, USA, 12–14 April 2010; pp. 150–155. [Google Scholar]
  27. Laghari, M.S.; Khuwaja, G.A. Electrical engineering department advising for course planning. In Proceedings of the IEEE Global Engineering Education Conference (EDUCON), Marrakech, Morocco, 17–20 April 2012. [Google Scholar]
  28. Al Ahmar, M.A. A prototype student advising expert system supported with an object-oriented database. Int. J. Adv. Comput. Sci. Appl. 2011, 100–105. [Google Scholar] [CrossRef]
  29. Aslam, M.Z.; Khan, A.R. A proposed decision support system/expert system for guiding fresh students in selecting a faculty in Gomal University. Ind. Eng. Lett. 2011, 1, 33–41. [Google Scholar]
  30. Naini, V.R.; Sadasivam, R.S.; Tanik, M.M. A web-based interactive student advising system using Java frameworks. In Proceedings of the 2008 IEEE Southeastcon, Huntsville, AL, USA, 3–6 April 2008; pp. 172–177. [Google Scholar]
  31. Albalooshi, F.; Shatnawi, S. Online academic advising support. In Technological Developments in Networking, Education and Automation; Springer: Dordrecht, The Netherlands, 2010; pp. 1–5. [Google Scholar]
  32. Zhou, Q.; Yu, F. Knowledge-based major choosing decision making for remote students. In Proceedings of the 2008 International Conference on Computer Science and Software Engineering (CSSE 2008), Wuhan, China, 12–14 December 2008; Volume 5, pp. 474–478. [Google Scholar]
  33. Al-Ghamdi, A.; Al-Ghuribi, S.; Fadel, A.; AL-Ruhaili, F.A.A.T. An expert system for advising postgraduate students. IJCSIT Int. J. Comput. Sci. Inf. Technol. 2012, 3, 4529–4532. [Google Scholar]
  34. Nambiar, A.N.; Dutta, A.K. Expert system for student advising using JESS. In Proceedings of the 2010 International Conference on Educational and Information Technology (ICEIT 2010), Chongqing, China, 17–19 September 2010; Volume 1, pp. 312–315. [Google Scholar]
  35. Nguyen, T.B.; Hoang, T.A.D.; Tran, H.; Nguyen, D.N.; Nguyen, H.S. An integrated approach for an academic advising system in adaptive credit-based learning environment. VNU J. Sci. Nat. Sci. Technol. 2008, 24, 110–121. [Google Scholar]
  36. Deorah, S.; Sridharan, S.; Goel, S. SAES- expert system for advising academic major. In Proceedings of the 2010 IEEE 2nd International Advance Computing Conference (IACC), Patiala, India, 19–20 February 2010; pp. 331–336. [Google Scholar]
  37. Sobecki, J.; Tomczak, J.M. Student courses recommendation using ant colony optimization. In Intelligent Information and Database Systems; Nguyen, N.T., Le, M.T., Swiatek, J., Eds.; Springer: Berlin, Germany, 2010; Volume 5991, pp. 124–133. Available online: https://pdfs.semanticscholar.org/5d8e/d2f7f865f94ce0f84815d7105d1eb45b4b4e.pdf (accessed on 14 December 2017).
  38. Ragab, A.H.M.; Mashat, A.F.S.; Khedra, A.M. HRSPCA: Hybrid recommender system for predicting college admission. In Proceedings of the International Conference on Intelligent System Design and Application (ISDA), Kochi, India, 27–29 November 2012; pp. 107–113. [Google Scholar]
  39. Lee, Y.; Cho, J. An Intelligent Course Recommendation System. SmartCR 2011, 1, 69–84. [Google Scholar] [CrossRef]
  40. Fong, S.; Si, Y.; Biuk-Aghai, R.P. Applying a hybrid model of neural network and decision tree classifier for Predicting University Admission. In Proceedings of the 7th International Conference on Information, Communications and Signal Processing (ICICS 2009), Macau, China, 8–10 December 2009. [Google Scholar]
  41. Abdulwahhab, R.S.; Salem, H.; Makhmari, A. An educational web application for academic advising. In Proceedings of the 2015 IEEE 8th GCC Conference and Exhibition (GCCCE), Muscat, Oman, 1–4 February 2015; pp. 1–4. [Google Scholar]
  42. Meller, T.; Wang, E.; Lin, F.; Yang, C. New classification algorithms for developing online program recommendation systems. In Proceedings of the International Conference on Mobile, Hybrid, and On-Line Learning (ELML’09), Cancun, Mexico, 1–7 February 2009. [Google Scholar]
  43. Williams, P. Fuzzy academic advising system for on probation students in colleges of applied sciences—Sultanate of Oman. In Proceedings of the International Conference on Computing, Electrical and Electronic Engineering (ICCEEE), Khartoum, Sudan, 26–28 August 2013; pp. 372–377. [Google Scholar]
  44. Adak, M.F.; Yumusak, N.; Campus, E. An elective course suggestion system developed in computer engineering department using fuzzy logic. In Proceedings of the 2016 International Conference on Industrial Informatics and Computer Systems (CIICS), Sharjah, UAE, 13–15 March 2016. [Google Scholar]
  45. Goodarzi, M.H.; Rafe, V. Educational advisor system implemented by web-based fuzzy expert systems. Asian J. Inf. Technol. 2012, 11, 77–82. [Google Scholar] [CrossRef]
  46. Fong, S.; Biuk-aghai, R.P. An Automated University admission recommender system for secondary school students. In Proceedings of the 6th International Conference on Information Technology and Applications, Sydney, Australia, 28–29 October 2017; pp. 978–981. [Google Scholar]
  47. Huang, C.Y.; Chen, R.C.; Chen, L.S. Course-recommendation system based on ontology. In Proceedings of the 2013 International Conference on Machine Learning and Cybernetics (ICMLC), Tianjin, China, 14–17 July 2013; Volume 3, pp. 14–17. [Google Scholar]
  48. Koutrika, G.; Bercovitz, B.; Ikeda, R.; Kaliszan, F.; Liou, H.; Garcia-Molina, H. Flexible recommendations for course planning. In Proceedings of the IEEE 25th International Conference on Data Engineering (CDE’09), Shanghai, China, 29 March–2 April 2009. [Google Scholar]
  49. Phaal, R.; Farrukh, C.; Probert, D. Technology roadmapping—A planning framework for evolution and revolution. Technol. Forecast. Soc. Chang. 2004, 71, 5–26. [Google Scholar] [CrossRef]
  50. Deline, G.; Lin, F.; Wen, D.; Gasevic, D. Ontology-driven development of intelligent educational systems. In Proceedings of the IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PacRim 2007), Victoria, BC, Canada, 22–24 August 2007; pp. 34–37. [Google Scholar]
  51. Kolenovic, Z.; Linderman, D.; Karp, M.M. Improving student outcomes via comprehensive supports: Three-year outcomes from CUNY’s Accelerated Study in Associate Programs (ASAP). Community Coll. Rev. 2013, 41, 271–291. [Google Scholar] [CrossRef]
  52. Jenkins, D.; Cho, S.-W. Get with the program ... and Finish it: Building guided pathways to accelerate student completion. New Dir. Community Coll. 2013, 2013, 27–35. [Google Scholar] [CrossRef]
  53. Tennat, G. SIX SIGMA: SPC and TQM in Manufacturing and Services; Gower Publishing Ltd.: Aldershot, UK, 2001; p. 6. [Google Scholar]
  54. Jenkins, D.; Bailey, T.; Jaggars, S. Redesigning America’s Community Colleges: A Clearer Path to Student Success; Community College Research Center: New York, NY, USA, 2015. [Google Scholar]
  55. Ding, X. Web-Based Academic Advising System; Florida Atlantic University: Boca Raton, FL, USA, 2002; p. 95. [Google Scholar]
  56. Turner, S.; Conkel, J.; Starkey, M.; Landgraf, R.; Lapan, R.; Siewert, J.; Reich, A.; Trotter, M.; Neumaier, E.; Huang, J.P. Gender differences in Holland vocational personality types: Implications for school counselors. Prof. Sch. Couns. 2008, 11, 317–326. [Google Scholar] [CrossRef]
  57. Holland, J.L. Making Vocational Choices: A Theory of Vocational Personalities and Work Environments; Psychological Assessment Resources: Lutz, FL, USA, 1997. [Google Scholar]
  58. Mattei, N.; Dodson, T.; Guerin, J.T.; Goldsmith, J.; Mazur, J.M. Lessons learned from development of a software tool to support academic advising. In Proceedings of the 2014 Zone 1 Conference of the American Society for Engineering Education (ASEE Zone 1), Bridgeport, CT, USA, 3–5 April 2014; Volume 38238. [Google Scholar]
Figure 1. Survey methodology.
Figure 1. Survey methodology.
Education 07 00090 g001
Figure 2. Classification of research papers by year of publication.
Figure 2. Classification of research papers by year of publication.
Education 07 00090 g002
Figure 3. Roadmaps for Academic Advising Systems (AAS) research method evolution.
Figure 3. Roadmaps for Academic Advising Systems (AAS) research method evolution.
Education 07 00090 g003
Figure 4. Roadmaps for AAS research objective evolution.
Figure 4. Roadmaps for AAS research objective evolution.
Education 07 00090 g004
Table 1. Inclusion/exclusion criteria.
Table 1. Inclusion/exclusion criteria.
IncludeExclude
  • Articles published in Journals with Impact Factor
  • Full-length articles published in International Conference/Workshop Proceedings
  • Approaches applied to formal education (e.g., universities, colleges, distance education, etc.)
  • Date from 2008 to 2017
  • Articles that do not present significant empirical evidence (e.g., theoretical and conceptual articles, essays, tool demonstration, etc.)
  • Book chapters
  • Approaches applied to non-formal education (e.g., Virtual Learning Environments, MOOC, etc.)
Table 2. Classification of case studies according to the research method.
Table 2. Classification of case studies according to the research method.
Research MethodAuthors & Year (Paper Ref.)
Content-basedMostafa et al., 2014 [7], Lin et al., 2015 [11], Poeppelmann, 2011 [12]
Collaborative filtering-basedChang et al., 2016 [1], Unelsrød, 2011 [5], Dhabi & Advisor, 2012 [13], Li, 2015 [14]
Knowledge-basedXu et al., 2016 [2], Werghi, Naoufel, & Kamoun, 2009 [15], Roushan et al., 2014 [16], Ai-nory, 2012 [17], Mohamed, 2015 [18], Koutrika, Bercovitz, & Garcia-Molina, 2009 [19], Kristiansen, Sørensen, & Stidsen, 2011 [20], Engin et al., 2014 [21], Henderson & Goodridge, 2015 [22], Mohamed, 2016 [23], Feghali, Zbib, & Hallah, 2011 [24], Prof & Shakeel, 2012 [25], Hashemi & Blondin, 2010 [26], Laghari & Khuwaja, 2012 [27], Ahmar, 2011 [28], Aslam & Khan, 2011 [29], Naini, Sadasivam, & Tanik, 2008 [30], Albalooshi & Shatnawi, 2010 [31], Zhou & Yu, 2008 [32], (Al-ghamdi et al., 2012 [33], Nambiar & Dutta, 2010 [34], Nguyen, Hoang, Tran, Nguyen, & Nguyen, 2008 [35]
HybridDeorah, Sridharan, & Goel, 2010 [36], Daramola, Emebo, Afolabi, & Ayo, 2014 [6], Sobecki & Tomczak, 2010 [37], Ragab, Mashat, & Khedra, 2012 [38], Lee & Cho, 2011 [39], Fong, Si, & Biuk-aghai, 2009 [40]
Computational intelligence-basedAbdulwahhab, Salem, & Makhmari, 2015 [41], Meller, T., Wang, E., Lin, F., & Yang, 2009 [42], Williams, 2013 [43], Adak, Yumusak, & Campus, 2016 [44], Goodarzi & Rafe, 2012 [45], Fong & Biuk-aghai, 2009 [46]
Table 3. Classification of case studies according to the research objectives.
Table 3. Classification of case studies according to the research objectives.
Research ObjectiveAuthors & Year (Paper Ref.)
Choosing Programs/MajorsMostafa et al., 2014 [7], Meller, T., Wang, E., Lin, F., & Yang, 2009 [42], Deorah et al., 2010 [36], Aslam & Khan, 2011 [29], Zhou & Yu, 2008 [32], Ragab et al., 2012 [38], Fong et al., 2009 [40], Fong & Biuk-aghai, 2009 [46], Engin et al., 2014 [21]
Selecting CoursesDhabi & Advisor, 2012 [13], Deline, G., Lin et al., 2015 [11], Li, 2015 [14], Daramola et al., 2014 [6], Sobecki & Tomczak, 2010 [37], Adak et al., 2016 [44], Unelsrød, 2011 [5], Chang et al., 2016 [1], Koutrika, Bercovitz, & Garcia-Molina, 2009 [19], Xu et al., 2016 [2], Huang, Chung-Yi, Chen, & Chen, 2013 [47], Koutrika, Bercovitz, Ikeda, et al., 2009 [48], Engin et al., 2014 [21], Albalooshi & Shatnawi, 2010 [31], Feghali et al., 2011 [24], Laghari & Khuwaja, 2012 [27], Ahmar, 2011 [28], Naini et al., 2008 [30], Albalooshi & Shatnawi, 2010 [31], Nguyen et al., 2008 [35], Poeppelmann, 2011 [12], Al-ghamdi et al., 2012 [33], Nambiar & Dutta, 2010 [34], Abdulwahhab et al., 2015 [41], Hashemi & Blondin, 2010 [26], Henderson & Goodridge, 2015 [22]
Long-Term Academic planningWerghi et al., 2009 [15], Roushan et al., 2014 [16], Williams, 2013 [43], Kristiansen et al., 2011 [20], Albalooshi & Shatnawi, 2010 [31], Mohamed, 2016 [23], Feghali et al., 2011 [24], Prof & Shakeel, 2012 [25], Ai-nory, 2012 [17], Mohamed, 2015 [18]
Table 4. (a) Cassification of the results of AAS case studies (algorithmic). (b) Classification of the results of AAS case studies (pedagogical).
Table 4. (a) Cassification of the results of AAS case studies (algorithmic). (b) Classification of the results of AAS case studies (pedagogical).
(a)
ObjectivesResults
Content-based
  • Cosine Similarity function can be used to effectively measure the similarity between the new student case and the other old cases by checking the nearest features and values [7]
  • Ontology-driven Software Development is appropriate for the development of intelligent academic advising system with an ontology-based architecture and can leverage familiar and proven software engineering tools and techniques [50]
  • A case-based reasoning combined with rule-based reasoning AAS is able to acquire new knowledge as usage of the system increases, while its rules can also be modified with minimal effort. This is unlike when an ontology is used for knowledge representation, which, although quite effective, require an advance investment in quality ontology development before efficient course advising can be obtained [6]
Collaborative filtering-based
  • The Authors in [1] use a demographic property of the student population and their department to segregate the data and to address the sparsity problem that limits the usefulness of collaborative filtering
  • AAS can predict a student’s academic performance and interest for a course based on a collection of profiles of students who have similar interests and a similar academic performance in prior courses. In paper [13], students in a Computer Science major are categorized in biclusters based on their similarity on course features: comprehension skills, memorization skills, programming skills, math skills, inferential thinking skills, problem solving skills, application of strategies skills
  • K-means algorithm can be used to determine the similarity of the students. It was identified that when K = 7, the clusters generated from the K means algorithm are more informative and effective [14]
  • Using an existing and relatively large dataset can be helpful in testing out the collaborative filtering signals, as well as in choosing the appropriate correlation algorithm [5]
  • Using friend relationships to weight the correlation used in the collaborative filtering process did not turn out to improve the accuracy of the recommendations [5]
Knowledge-based
  • Contrary to other contributions, which were intensively based on rule-based approaches, paper [15] proposes a Decision Support System that formulates the student planning and advising tasks as a search problem
  • In paper [20], the problem of planning the elective courses is modelled using integer programming and three different solution approaches are suggested, including a Branch-and-Price framework using partial Dantzig–Wolfe decomposition. The suggested algorithms achieved better results than the currently applied meta-heuristic
  • By implementing a prescriptive advising model and a developmental advising model, system testing revealed that 93% of academic advising test cases show an agreement between the system advising in course selection and human advising [28]
  • Authors in [29] collected all the required criterions, abilities, and capabilities for each faculty/major and converted the knowledge into facts and rules in CLIPS syntax, which is suitable for forward reasoning and can be used easily
Hybrid
  • Authors in [37] concluded that Ant Colony Optimization-based method is promising, since it enables students to overcome of the disadvantages of classical approaches and gives higher values of performance measures.
  • Authors in [42] conclude that the proposed nearest neighbour-like algorithms outperform the two well-known classification algorithms Naïve Bayes and J48 algorithm in terms of student classification success rate when there is uncertainty present in the data
  • Classification algorithms can be used to identify what programs a student may fit into; however, to involve multiple schools would require undesirable delays in a web application [42]
  • Paper [36] acquires implicit and latent interests of students in addition to the explicitly stated ones, with the help of questionnaire responses that also determine the degree of dilemma that the candidate faces and the time taken by him/her to respond to each question
Computational intelligence-based
  • The major advantage of fuzzy AAS is that knowledge gradually turns into wisdom and can be used as a decision making tool in critical situations that replace the conventional FAQ [45]
  • In paper [46] the experiments showed that the hybrid decision tree and neural network approach improved accuracy in classification task
(b)
ObjectivesResults
Choosing Programs/Majors/HEI
  • Authors in [46] claim that scores are an important factor for qualifying students to universities, especially for Mathematics and English courses. In their experiments, they found that when students’ score of Mathematics and English are higher than 80%, almost all of them will be admitted to the universities.
  • Many students choose a university faculty/major because it has a good social reputation or their friends have chosen it [29]. In the same paper, using abilities tests, intelligence tests, and their past academic record, authors can measure some student capabilities and abilities and determine which faculty/major is suitable for them
Selecting Courses
  • Authors in [1] claim that more feedback information from students is needed for effective course selection and introduces a quality-control mechanism based on filtering courses with poor instructor ratings
  • AAS can be used to overcome ineffective student academic advising in Distance Education aimed at predicting a student’s academic performance and interest for a course, based on a collection of profiles of students who have similar interests and academic performance on prior courses [13].
  • Authors in [5,14] suggested that students’ major is a possible factor that is related to the students’ performance. Experiment claim that male students aged between 24 and 27 in the software major, mostly of European, Maori, and Asian backgrounds, as well as middle aged (around 45) students, are more likely to be high performing students. On the other hand, young male students with network majors are more likely to be low performing students
  • Complex cases for AAS have to do with students that have changed from one program to another (many of them more than once) and have failed and dropped courses that are spread among different departments. Cases in which a student has failed multiply in different departments and are more intricate to handle even for the human course adviser [6]
  • Authors in [44] observed that the success of students in their previous required courses and the student’s skills are found to be determinants of student’s success in elective courses
  • Authors in [2] claim that an AAS framework has important implications on how the curriculum planner should design the curriculum and allocate teaching resources
  • AAS approaches for rolling prediction of future course enrolments should aim at minimizing maintenance effort, especially in terms of adaptability to curricula changes and graduation requirements [12]
Scheduling Courses/Academic planning
  • Authors in [43] propose a multi-criteria fuzzy system model in academic advising area to advise probation students to register an appropriate number of credit hours so as to minimize their risk of losing their enrolment due to an incorrect decision. The factors investigated are: (1) the Cumulative Grade Point Average, (2) the Number of Covered Credit Hours, (3) Number of times that the student is on Probation, and (4) the maximum grade that student has received.
  • The path associated with the students’ academic plan can be used to derive a metric that measures the similarity of the students’ course history. This will be quite useful for mining student profiles and analyzing and predicting student performance [15]
  • In small institutions, an issue that needs to be considered in an AAS is the number of students in each class to guarantee a minimum number of enrolled students [17]
  • Solving the Elective Course Planning Problem optimal can be used to reduce students’ switch to another high school, which is highly undesirable [20]
  • During academic planning, an “explanation component” is important to explain to students the rationale behind courses assignments to various semesters, or changes to study plans after modifying any input setting [23]
  • More than 90% of respondents rated the online AAS increased their awareness of the curriculum [24]
  • Multiple course selections from different faculties and departments enable students to maximize their opportunities in registering courses of their own interest and completing their degree requirements in the best possible way [25]
Long-Term Course Planning
  • Students with similar background will achieve similar expected Grade Point Average (GPA) by following the same course sequence recommendation policy [2]
  • By analyzing the differences between the generated academic plans, students should be able to better understand their choices and thus make appropriate decisions [18]
Table 5. Strengths, weaknesses, opportunities, threats (SWOT) of AAS research.
Table 5. Strengths, weaknesses, opportunities, threats (SWOT) of AAS research.
StrengthsWeaknesses
  • Large volumes of available educational data increased accuracy of experimental results.
  • Use of pre-existing powerful and valid algorithmic methods.
  • Interpretable multiple recommendations to support learners/teachers.
  • More precise user models for guiding adaptation and personalization of systems.
  • Reveal critical moments and approaches of learning.
  • Lack of holistic AAS approach encompassing all multi-facet factors like student competencies, interests, and personality type.
  • Misinterpretation of results due to human judgment factors, wrong input data, etc.
  • Heterogeneous data sources: not yet a unified data descriptive vocabulary—data representation issues.
  • Mostly quantitative research results. Qualitative methods have not yet provided significant results.
  • Information overload—complex systems.
  • Uncertainty: In many cases, only skilled advisors could trust the recommendations and interpret the results correctly.
OpportunitiesThreats
  • Modeling programs of studies and learning pathways for data standardization, data interoperability, and compatibility among different tools and applications support multiple course selections from different faculties and departments generalized platform development.
  • Intellectual and affective learning opportunities based on sophisticated decisions.
  • Self-reflection/self-awareness in intelligent, autonomous and massive systems.
  • Feed machine readable results from the AAS procedures to other data-driven systems for diving decision making.
  • Acceptance Model: e.g., perceived usefulness, goal expectancy, perceived playfulness, trust, etc.
  • Over-analysis: the depth of analysis becomes profound and the results lack generality.
  • Possibility of misclassification of outcomes.
  • Trust: contradictory findings during implementations.

Share and Cite

MDPI and ACS Style

Iatrellis, O.; Kameas, A.; Fitsilis, P. Academic Advising Systems: A Systematic Literature Review of Empirical Evidence. Educ. Sci. 2017, 7, 90. https://doi.org/10.3390/educsci7040090

AMA Style

Iatrellis O, Kameas A, Fitsilis P. Academic Advising Systems: A Systematic Literature Review of Empirical Evidence. Education Sciences. 2017; 7(4):90. https://doi.org/10.3390/educsci7040090

Chicago/Turabian Style

Iatrellis, Omiros, Achilles Kameas, and Panos Fitsilis. 2017. "Academic Advising Systems: A Systematic Literature Review of Empirical Evidence" Education Sciences 7, no. 4: 90. https://doi.org/10.3390/educsci7040090

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop