State-of-the-Art Survey on Deep Learning-Based Recommender Systems for E-Learning

Abstract

Recommender systems (RSs) are increasingly recognized as intelligent software for predicting users’ opinions on specific items. Various RSs have been developed in different domains, such as e-commerce, e-government, e-resource services, e-business, e-library, e-tourism, and e-learning, to make excellent user recommendations. In e-learning technology, RSs are designed to support and improve the learning practices of a student or an organization. This survey aims to examine the different works of literature on RSs that corroborate e-learning and classify and provide statistics of the reviewed articles based on their recommendation goals, recommendation techniques used, the target user, and the application platforms. The survey makes a prominent contribution to the e-learning RSs field by providing an overview of current research and traditional and nontraditional recommendation techniques to provide different recommendations for future e-learning. One of the most significant findings to emerge from this survey is that a substantial number of works followed either deep learning or context-aware recommendation techniques, which are considered more efficient than any traditional methods. Finally, we provided comprehensive observations from the quantitative assessment of publications, which can guide and support researchers in understanding the current development for potential future trends and the direction of deep learning-based RSs in e-learning.

1. Introduction

We form opinions about things we do not care for, like, or dislike daily. It happens more often in our daily life; for instance, we may decide to go to school in the morning, play soccer in the afternoon, and watch an action movie at night. Similarly, when trying to purchase an item from a store, one can decide to purchase a snack, an item from the dairy section, a book, or a beverage. However, making the right decision is one of the challenges people face in their daily activities. Thus, there is a need for an intelligent system to help predict user preferences for new items. Recommender systems (RSs) emerged to deal with this problem to help users find what is genuinely relevant to their needs [1]. Such systems have intelligently changed how we find articles, information, and even how we see others. RSs’ main function is to predict user interest by relating the user’s history, information, profile, and queries used, searched, created, and expressed [2]. Recently, learning has drastically shifted from a traditional classroom to an e-learning environment [3]. In technology-enhanced learning, RSs are used to find and suggest suitable learning objects to the learner, and a learning object (e.g., a problem) has several categories that indicate topics or fields [4]. RSs are generally essential in education and any activity involving accessing and sharing resources among people or communities. It uses different users’ and items’ characteristics, such as their interests, educational backgrounds, levels of expertise, geographical locations, and so on, to propose sequences or novel resources that might interest users [5]. However, when providing recommendations for e-learning contexts, the system should consider the users’ interests and learning goals [6].

The past seven years have seen increasingly rapid advances in RSs. One of the primary concerns of the RSs research community within that period was the concept of choosing the candidate recommendation technique that can be used to compute users’ preferences on new items efficiently. Many of today’s RSs for learning are built based on the three primary traditional recommendation techniques, the same as RSs in other domains, such as e-commerce, namely collaborative filtering, content-based filtering, and hybrid techniques [7]. However, along with this growth of RSs, there is increasing concern over several significant drawbacks related to recommendation techniques. Over-specialization, cold-start, and data sparsity problems are the major drawbacks of traditional methods [8]. Nontraditional methods such as deep learning, context-aware, and multi-criteria recommendations extending traditional techniques have recently emerged. The existence of many alternatives in choosing the technique to use helped tremendously to improve the accuracy of recommendations that can be offered to users. Presently, RSs go beyond using only traditional recommendation techniques by considering the pedagogical requirements of the user instead of only the preferences. In this survey, we present, among other things, a synthesis of prior work on RSs, and an overview of both traditional and other techniques of building RSs for learning that are less problematic and more efficient than traditional recommendation techniques.

Furthermore, another critical aspect of building RSs is generally related to various user tasks that the system can support in a particular application domain. In technology-enhanced learning, RSs help learners achieve their desired learning goal by recommending a sequence of items, finding novel items, providing annotation in context, finding learning peers, recommending learning pathways, etc. [7]. This survey checked and classified research papers on RSs for learning that were published between 2015 and 2021 based on what they are recommending to the user (supporting the user tasks), the platforms (e.g., mobile, chatbot, or web-based), the recommendation techniques used, and categories of users (students or teachers).

Therefore, this study makes contributions to research on RSs for learning by demonstrating the following:

• Overview of traditional and nontraditional recommendation techniques;
• Synthesis of prior work on RSs for learning based on:

  a

  Recommendation techniques;

  b

  Recommendation goals (supporting user tasks);

  c

  Platforms (mobile or web-based);

  d

  Kind of users (For students, teachers, or both students and teachers).

• Proposing potential future research direction on RSs for learning and other related application domains.

At the end of this survey, our results should be able to answer the following seven essential questions:

• To what extent is research in deep learning-based RSs in e-learning?
• What is the frequency of usage of each recommendation technique? In other words, which techniques are used more often to implement RSs for learning?
• Which user tasks (recommendation goal) do people pay more attention to?
• Are RSs for learning capable of making unexpected and fortunate recommendations?
• What is the ratio between chatbot, mobile, and web-based RSs for learning?
• Technology-enhanced learning concerns teaching and learning, which received considerable attention from researchers on RSs?
• With new recommendation techniques (deep learning, context-aware, and multi-criteria rating techniques), has research in this domain shifted toward those new techniques?

The overall structure of this paper takes the form of five sections, including this introductory section. Section 2 contains a brief review of RSs for learning, related works, and evaluation methods; Section 3 explains the classification methodologies and an overview of recommendation techniques; Section 4 contains the findings of our work. Additionally, a conclusion and some potential future research directions are presented in Section 5.

2. Recommender Systems for E-Learning

E-learning is a way of learning, in which a learner can study online from anywhere, which can be self-paced with available learning resources. E-learning is one of the fastest learning methods in recent years due to the rapid growth in technology development [9]. Learners can now search for desired educational information, products, and services via computers and mobile devices [10]. The number of educational resources continues to grow, making it increasingly difficult for traditional search engines to meet requirements related to online searches for information about educational products and services during the learning process [11].

Problems of information overload create difficulties for people to discover items or make the right decision on a particular item that can satisfy their needs. The same issues arose when learners tried to find suitable learning materials. RSs are intelligent systems that can solve problems of information overload by using various techniques to find and recommend valuable items to users. They are subclasses of information filtering systems aimed at predicting preferences or ratings that a learner would provide to learning items and creating lists of relevant items for recommendations, ranked according to their various degrees of relevancy to the items required. RSs for learning cover almost all fields of technology-enhanced learning, such as mobile learning, formal learning, informal learning, and traditional and modern ways of learning. Various definitions have been suggested for RSs in general. In learning, RSs for learning have similarly retained almost all their definitions. The only distinction is that the term “items” used in this context strictly refers to a learner (or sometimes to both learners and students). RSs have become one of the main tools for personalized content filtering in the educational domain [11]. As indicated by H. Drachsler et al. in [12] and G. McCalla in [13], RSs for learning aim to support learners by providing valuable learning materials. In addition to what RSs do in other application domains, such as e-commerce, to recommend products to customers, RSs for learning also support various user tasks such as recommending learning peers, lessons and lectures, self-assessment materials, etc.

2.1. Related Works

Research in RSs for e-learning is continuously growing, with an effort by authors to summarize and map different aspects of this field.

Drachsler et al. in [12] presented a state-of-the-art review on technology-enhanced learning RSs. This study considered 82 papers published between 2000 to 2014 and provided a comprehensive overview of the area. The authors introduced parameters for evaluating technology-enhanced learning RSs and analyzed different recommendation techniques and sources of information. Khanal et al. [14] presented a systematic review of machine learning-based RSs for e-learning. The authors developed a taxonomy that accounts for components required to develop effective RSs. The study focused on four traditional recommendation techniques: collaborative filtering, content-based, knowledge-based, and hybrid approaches. The authors’ analysis was based on machine learning algorithms and the method of evaluating the RSs. They also addressed challenges regarding input and output characterization. They summarized the overall findings with an observation that machine learning techniques, algorithms, datasets, evaluation, valuation, and output are necessary components in Rs for e-learning.

Zhang et al. [15] presented a survey on RSs for e-learning. The authors reviewed and analyzed the research on e-learning RSs, identified the traditional recommendation techniques used in e-learning, and identified new research directions. They also proposed a framework with three major components: a user interface, a database server, and a recommendation engine. This paper highlighted how recommendation techniques could support learners in universities and life-long learners to gain skills to stay competitive. It aimed to provide guidance for researchers and practitioners in developing e-learning RSs. Urdaneta-Ponte et al. [1] conducted a review on RSs for education. The authors surveyed 98 articles to analyze the work undertaken in RSs that support educational practices to acquire information related to the type of education and areas dealt with, the developmental approach used, and the elements recommended.

Rivera et al. [16] presented a systematic mapping to investigate the use of RSs in education. The authors extensively reviewed 44 research papers to extract and classify relevant data to obtain valuable insights about the uses, approaches, and challenges addressed by RSs. Salazar et al. [17] conducted a systematic review of affective RSs in the learning environment. The authors presented a macro-analysis, identifying the primary authors and research trends. They also summarized different aspects of RSs, such as the techniques used in affectivity analysis, the source of data collection, and the state of the art of influence of emotions in the educational field.

2.2. Methods of Evaluating Recommender Systems for Learning

Frameworks have been suggested for evaluating the system, the usefulness of recommendations given to users, and evaluating whether users are delighted with the services the system offers. Mojisola et al. [18] have surveyed various methods for evaluating RSs for learning. The main aim of evaluating RSs is broadly categorized into three parts. First is measuring the performance of the systems, which entails the performance of algorithms used or the system’s general performance from a technical point of view. The second is measuring the impact on learning to evaluate if the learning performance of the user has improved after using the system for a considerable period. Third is user-centric evaluation, which measures users’ contentment and satisfaction with the system. The first evaluation category can be achieved offline using a dataset containing interactions between users and similar systems to evaluate the effectiveness of the recommendation algorithm or the entire system. The impact of the system on learning can be measured scientifically from the users’ perspectives to determine the level of learning improvements and how the system influences their studies. Finally, real-life testing allows users to use the system over a long period to measure their satisfaction. One way to achieve this is through interviewing users either by face-to-face interaction or via the distribution of questionnaires.

3. Classification Method

As highlighted previously, the motivation of this study is to find and understand the research directions of prior works on RSs for learning by examining published articles, then provide interested researchers and practitioners’ insight on the state of the art and what needs to be done for potential future research. Therefore, a methodology must be set to systematically extract and analyze the research papers.

Classification to achieve the desired goal’s methods are followed based on the recommendation techniques, the user tasks supported by the system (recommendation goal), the system platforms, and the people using the system. Therefore, all the reviewed papers are classified based on seven recommendation techniques, six recommendation goals, three platforms (web-based, mobile-based, or chatbot), and the type of users, which can be teacher, student, or both students and teachers. This section presents a brief review of classification categories, especially the recommendation technique that is said to be the central nervous system of RSs, which controls how predictions should be made.

3.1. Classification Based on Recommendation Techniques

The fundamental recommendation question is to check if a user $u\in U$ will be interested in item $i\in I$, for $U$ and $I$ as the domain of users and items, respectively. The most common ways to answer such questions are:

• To find out the set of items that $u$ liked previously and then find the similarity between them and $i$;
• To find out people who like $i$ and try to compute their similarities with $u$.

In each of the above two cases, the similarity values are used to measure the degrees to which $u$ is interested in $i$.

Many researchers and industries have used different approaches to build robust RSs that intelligently recommend learning or teaching materials to users. The most used techniques are collaborative filtering, content-based filtering, knowledge-based, and hybrid-based recommendation techniques. Other RSs use these techniques to make recommendations, such as context-aware, multi-criteria RSs, deep learning, and machine learning techniques. This section sheds more light onto various RS techniques.

3.1.1. Collaborative Filtering

Collaborative filtering is the most widely used and popular technique for building RSs that make predictions and suggestions based on ratings of other system users [19]. The belief regarding this technique is that ratings of other users can be used to provide reasonable predictions of the current user. It is based on the nature of humans to “use what is popular among my peers, or wisdom of the crowd” to recommend items to users [20]. The collaborative filtering technique presumes that if users agree on ratings of some selected items, they will have similar opinions on other new items. Its utility function $f$ uses users’ or items’ similarities to predict the rating of unknown item $i$ by user $u$ as $f:\left(u, i\right)\to r$ for which $r$ can be an actual number within some interval (from 1 to 10), a binary rating (yes/no, like/dislike, etc.), or determined implicitly by the system. Several algorithms fall under collaborative filtering, such as user-based, item-based, stereotype/demographic, etc. Taking a user-based collaborative filtering algorithm, according to [9], the rating of new $i$ by the $u$ denoted by $r_{u, i}$ can be obtained either by taking the average of all ratings of $i$ given by some users in $U$ as in Equation (1) or by finding the similarity between all users who rated $i$ and the user $u$, then calculate the sum of the products of their similarities and the ratings they gave to $i$, and finally, by multiplying the normalizing factor as in Equation (2), where $\aleph$ is the normalizing factor which most of the time is considered to be the inverse of $\sum _{v\in U}\left|sim\left(u,v\right)\right|$, or by jointly using Equations (3) and (4).

$$r_{u,i}=\frac{1}{K}{\sum }_{v\in U}{r}_{v,i}$$

(1)

$$r_{u,i}=\aleph {\sum }_{v\in U}sim\left(u,v\right)\times r_{v,i}$$

(2)

$$r_{u,i}={\overline{r}}_u+\aleph {\sum }_{v\in U}sim\left(u,v\right)\times (r_{v,i}-{\overline{r}}_v)$$

(3)

$${\overline{r}}_u=\left(1/\left|I_u\right|\right){\sum }_{i\in I_u}{r}_{u,i}$$

(4)

where $I_u=\left\{i\in I|r_{u,i}\ne \varnothing \right\}$.

On the other hand, as observed by [8], some algorithms used probabilistic methods to predict $r_{u,i}$ based on the Bayesian probability that $u$ rated $i$, given that they previously rated some items $i^{*}\in I$. Equation (5) is the relation used for this approach.

$$r_{u,i}=E\left(r_{u,i}\right)={\sum }_{k=0}^{n}i\times \mathrm{Pr}(r_{u,i}=k|r_{u,i^{*}},i^{*}\in I_u)$$

(5)

$\mathrm{Pr}\left(r_{u,i}=R|r_{u,i^{*}}, i^{*}\in I_u\right)$ can be estimated using either a Bayesian network or cluster models as proposed by [21]; the variable $R$ is a non-negative integer $\le n$.

The question is how to find the similarities between users or items that can be used to perform the necessary computations. There are many similarity metrics, such as the Pearson correlation coefficient, Euclidean distance, cosine measure, Spearman correlation, Tanimoto coefficient, log likelihood, etc. The Pearson correlation coefficient is one of the many similarity metrics that has been used for a long time, but one needs to understand it very well before using it because it fails to give an accurate result when applied to a sparse or minimal dataset. It is not a flawed metric, and at the same time, it is not always good.

The formulas for calculating users’ similarities $sim \left(u_i,u_j\right)$ using Pearson correlation and cosine measures are in Equations (6)–(9), respectively.

$$sim\left(u,v\right)=\frac{{\sum }_{i\in I_{u,v}}(r_{u,i}-\overline{r_u})\left(r_{v,i}-\overline{r_v}\right)}{\sqrt{{\sum }_{i\in I_{u,v}}{(r_{u,i}-\overline{r_u})}^2{\sum }_{i\in I_{u,v}}{\left(r_{v,i}-\overline{r_v}\right)}^2}}$$

(6)

$$\cos (\overrightarrow{u},\overrightarrow{v})=\frac{{\sum }_{i\in I_{u,v}}{r}_{u,i}{r}_{v,i}}{\sqrt{{\sum }_{i\in I_{u,v}}{r}_{u,i}{}^2}\sqrt{{\sum }_{i\in I_{u,v}}{r}_{v,i}{}^2}}$$

(7)

where $I_{u,v}=\left\{i \in I|r_{u,i}\ne \varnothing \mathrm{and} r_{v,i}\ne \varnothing \right\}$; $\overline{r_u}$ and $\overline{r_v}$ are taken over ratings given by $u$ and $v$ to a common item $i$, respectively. Furthermore, in Equation (9), the term $\overrightarrow{u}·\overrightarrow{v}$ is a dot product between vectors $\overrightarrow{u}$ and $\overrightarrow{v}$.

Similarly, similarities between items $i$ and $j$ can be found using Equation (8) below.

$$sim\left(i,j\right)=\frac{{\sum }_{u\in U_{i,j}}(r_{u,i}-\overline{r_u})\left(r_{u,j}-\overline{r_u}\right)}{\sqrt{{\sum }_{u\in U_i}{(r_{u,i}-\overline{r_u})}^2{\sum }_{u \in U_j}{\left(r_{u,j}-\overline{r_u}\right)}^2}}$$

(8)

where $U_{i,j}$ is the user who rated items $i$ and $j$ while, for the cosine measure, $sim (u, v)=\cos (\overrightarrow{u}, \overrightarrow{v})$, so that:

$$\cos (\overrightarrow{u},\overrightarrow{v})=\frac{\overrightarrow{u}·\overrightarrow{v}}{{\Vert \overrightarrow{u}\Vert }_2\times {\Vert \overrightarrow{v}\Vert }_2}=\frac{{\sum }_{i\in I_{u,v}}(r_{u,i}{r}_{v,i} )}{\sqrt{{\sum }_{i\in I_{u,v}}{r}_{u,i}{}^2}\sqrt{{\sum }_{i\in I_{u,v}}{r}_{v,i}{}^2}}$$

(9)

The term $\overrightarrow{u}·\overrightarrow{v}$ is a dot product between vectors $\overrightarrow{u}$ and $\overrightarrow{v}$.

3.1.2. Content-Based Technique

Unlike collaborative filtering, which uses similar rating patterns over users to make predictions, the content-based technique does not require the ratings of other users to make valuable recommendations. It is designed to use users’ profiles to recommend items due to their similarities with items the user was already interested in the past. The word similarity used here does not necessarily refer to the same similarity used to define collaborative filtering, which was based on rating correlations, but the similarity between attributes of the current item and those of items the user liked previously. Content-based filtering systems try to match features of items stored in the user’s profile. At the same time, the references are kept with the features of the newly presented items to make recommendations. Therefore, only items that the user previously liked will be recommended.

Contrary to collaborative-based systems that recommend items based on preferences from other users with similar tastes, content-based systems always make recommendations in a personalized manner. Whether the user will like the item or not relies on matching the features of the user’s profile and the item’s attributes. Case-based reasoning and attribute-based techniques are the most common content-based techniques [7].

3.1.3. Knowledge-Based Technique

The two techniques discussed above require a reasonable volume of data representing users’ previous purchasing history and rating experiences, making them unsuitable to the domain where items are highly customized. Such customized products include automobiles, financial services, real estate, tourism guide systems, and so on, rarely purchased in large quantities, as such ratings are not sufficiently available. The knowledge-based technique does not require any rating correlation or user profile to make predictions, making it convenient to recommend items not regularly purchased [22]. Knowledge-based RSs use item attributes, user specifications, and domain knowledge to generate recommendations. Instead of depending on users’ ratings, a knowledge-based system only needs information about users’ preferences, such as why they are looking for that item based on a lower price, high quality, or luxury. The system requires the information to make suitable recommendations by considering items that satisfy users’ requirements.

3.1.4. Hybrid Recommendation Technique

RSs can be built using collaborative, content-based filtering or knowledge-based techniques. Still, each of them has its peculiar weaknesses, as observed by Manouselis et al. in [7], where they highlighted some of the common pros and cons of each recommendation technique; for example, data sparsity, new user, and new item problems are some of the significant deficiencies of collaborative filtering, of which some of them can be controlled by content-based filtering. Therefore, combining two techniques with a hybrid recommendation technique can minimize such individual drawbacks. The term hybridization or, more formally, hybrid recommendation technique does not only mean it can be between different recommendation techniques (content-based, collaborative filtering, and knowledge-based) but also between algorithms under the same technique. Consider case-based reasoning and attribute-based algorithms; all of them are content-based techniques. The former suffers from a cold-start problem and new user/item problems, while the latter does not have such disadvantages. Conversely, domain dependencies are a severe problem of attribute-based algorithms that case-based reasoning does not have. The same complication can be revealed under collaborative filtering algorithms as both item-based, and user-based techniques suffer from cold-start problems. In contrast, stereotype collaborative filtering is believed to be free from cold-start problems [7]. Knowledge-based systems are unlike collaborative and content-based techniques that suffer from sparsity and cold-start problems at some level. It is a good alternative for hybrid RSs.

3.1.5. Context-Aware RS

The concepts of RSs we discussed in this paper focus on suggesting items relevant to the users’ needs that are not concerned about any other factor influencing the recommendation process. These types of RSs are traditional or two-dimensional (user, item) RSs, as their rating functions consider only two entities (users and items) to predict ratings. In a context-aware recommendation, other factors known as context are integrated into the rating function as necessary and sufficient conditions for making a meaningful recommendation [23]. The rating function discussed earlier can be modified to $f :u\times i\times C\to r$, where $C$ is the domain of the contextual information required for the recommendation. The word context has many meanings depending on the domain of discussion, such as medicine, cognitive science, psychology, computer science, and social science, to mention a few. No specific definition can satisfy the various research disciplines; therefore, the real meaning of context is domain-dependent. In the RS domain, context refers to any factor(s) or additional information that can be used to generate a more intelligent and effective recommendation [24]. Verbert et al. in [25] classified the term context used for building context-aware RSs into three different classes:

• User Context: i.e., the location, companions, social situation, or even the user profile;
• Computing Context: i.e., communication cost and bandwidth, the strength of the connectivity, and available resources, such as a computer, printer, workstation, etc.;
• Physical context: i.e., weather conditions, noise, traffic levels, etc.

Context-aware RSs are generally called multidimensional RSs [26] because, given n dimensions of contexts to $c_1, c_2,\dots , c_n$, the rating functions can then be defined as $f :u\times i\times c_1\times c_2\times \dots \times c_n\to r$.

There are three main approaches to modeling context-aware RSs: prefiltering, post-filtering, and contextual modeling paradigms, which differ only in the contexts in which the context is used before making the final recommendation.

The prefiltering paradigm applies the context to filter out all items that do not satisfy the context to have contextualized data and passes the result to the next level for the usual two-dimensional $user\times item$ recommendation. Conversely, a post-filtering approach performs the traditional two-dimensional $user\times item$ predictions before filtering the items that do not conform to the contextual information. On the other hand, contextual modeling reduces context filtering and two-dimensional $user\times item$ computations to just a brief explanation of the three paradigms of context-aware recommendation techniques. A more detailed description can be found in the work of Adomavicius and Tuzhilin in [26].

3.1.6. Multi-Criteria RS

Both traditional and context-aware RSs are implemented based on a single rating value r to represent the degree to which users are interested in items. Although single-rating RSs have been implemented in applications across several domains, research has shown that their functionality is limited because the likeness of an item by users may depend on several items’ attributes [27]. Interestingly, RS researchers in industries and academic institutions have already started shifting toward multi-criteria rating systems [28]. As an illustration, Zagat’s guide (https://www.zagat.com accessed on 5 July 2022), a restaurant guide that issues a restaurant rating criterion based on three categories (food, service, decor), online shops such as Rakuten (http://www.rakuten.com/ accessed on 5 July 2022), and retailers of consumer electronics (https://circuitcity.com accessed on 5 July 2022) are examples of systems that are developed based on multi-criteria RSs.

We are familiar with the form and how ratings take place in traditional recommendation approaches from the beginning of this section. We know how the rating functions work, how to compute similarities between users (Equations (6) and (9)) if necessary, as well how to use the similarity $sim\left(u,v\right)$ to calculate ratings (Equations (2) and (3)). Similarly, multi-criteria recommenders are designed to achieve the same goal of finding products/resources that might be useful and interesting to users. They vary only by acquiring more information about items from the users that is useful in making meaningful recommendations. In multi-criteria recommenders, the rating function $f\left(u,i\right)$ is of the form $f :u\times i\to r_0, r_1,r_2,\dots ,r_n$, where $r_0$ is the overall rating, similar to $r$ in traditional and context-aware RSs, and $r_k$ are the criteria ratings.

Table 1. Example of Four Criteria Rating Problem.

User/Item	Item1	Item2	Item3	Item4
User1	2.5,4,3,2,1	3.5,5,2,4,3	4,4,5,5,2	1,1,0,2,1
User2	?,?,?,?,?	1,0,0,0,4	2,3,2,2,1	0,0,0,0,0
User3	?,?,?,?,?	4,4,5,5,2	5,5,5,5,5	?,?,?,?,?
User4	4.5,4,5,5,4	1.5,2,1,1,2	5,5,5,5,5	0,0,0,0,0
User5	3,4,3,2,3	?,?,?,?,?	?,?,?,?,?	2.5,3,3,2,2

is a sample of ratings in multi-criteria systems using four criteria and an overall rating to predict user preferences on items [27]. The ratings with question marks (?,_?,?,?,?) are the unknown ratings that need to be predicted.

Adomavicius, in [28], outlined two methods for calculating similarities between users or items for rating predictions in multi-criteria RSs. One of the proposed methods uses the same similarity metrics discussed in Equations (6) and (9) to make predictions by making some minor adjustments to account for all $n+1$ different similarities, which are briefly demonstrated below:

We know that for each $u$ and $i$, the utility function $f\left(u,i\right)$ produces $n+1$ different ratings, where $n$ is the number of criteria used. $f\left(u,i\right)=r_0, r_1,r_2,\dots ,r_n$, which requires calculating $n+1$ similarities, one similarity for $r_0$ and others for each rating $r_k$. Let $si{m}_m\left(u,v\right)$ be the similarities between users $u$ and $v$, where m = 0, 1, 2, …, n, then $si{m}_0\left(u,v\right)$ is the similarity between $u$ and $v$ with respect to the overall rating $r_0$, and $si{m}_1\left(u,v\right)$ is the similarity between $u$ and $v$ with respect to rating $r_1$, up to $si{m}_n\left(u,v\right)$, which is the similarity between $u$ and $v$ with respect to the last rating $r_n$. The overall similarity is computed by taking the average of the n similarities or by considering the slightest (worst-case) similarity, as in Equations (10) and (11), and then finally by going back to our rating prediction algorithms in Equation (2) or Equations (3) and (4) on page 4. Note that for any item $i$, the utility function $f$ for $u$ and $v$ with respect to $i$ is $f\left(u,i\right)=r_0, r_1,r_2,\dots ,r_n$ and $f\left(v,i\right)=r_0^{*}, r_1^{*},r_2^{*},\dots ,r_n^{*}$, respectively.

$$si{m}_{average}\left(u,v\right)=\frac{1}{n+1}{\sum }_{m=0}^{n}si{m}_m\left(u,v\right)$$

(10)

$$si{m}_{Min}\left(u,v\right)=Min\underset{m=0,1,n}{\underbrace{si{m}_m\left(u,v\right)}}$$

(11)

This method applies only to algorithms that require similarities between users or items to make predictions, such as collaborative filtering.

However, other methods work by assuming a solid relationship between criteria ratings and the overall rating as used in [20,29,30,31,32,33]. For instance, in Table 1, one can observe that the overall rating $r_0$ has a well-defined relationship with the rating of their corresponding criteria $r_1,r_2,r_3$, and $r_4$. This relationship can be formally represented in Equation (12).

$$r_0=\phi \left(r_1,r_2,r_3,r_4\right)$$

(12)

Generally, for any number of criteria $n$, the relationship function can be extended to account for all n, as in Equation (13).

$$r_0=\phi \left(r_1,r_2,\dots ,r_n\right)$$

(13)

The function $\phi$ is called the aggregation function. If we can define how to find ratings for each criterion and define the function $\phi$, then $r_0$ will be computed easily, as outlined by Adomavicius et al. in [27], who demonstrated how to arrive at $r_0$ pictorially. The following steps summarize the necessary procedures needed to compute $r_0$ using the aggregation function approach:

• Break down the multi-criteria ratings into single ratings and use any traditional recommendation technique to find the values. I.e., instead of $f :u\times i\to r_0, r_1,r_2,\dots ,r_{n }$, use $f :u\times i\to r_k$, where $(k=1,2,\dots ,n$);
• Define the function $\phi$ that can take multi-criteria ratings to produce $r_0$, just as in Table 1, where the function was defined as the average of multiple ratings;
• Lastly, Equation (13) will be used to predict $r_0$.

3.1.7. Deep Learning-Based RS

Traditionally, RSs are based on clustering, nearest neighbor, and matrix factorization methods. Deep learning is a subset or type of machine learning method based on artificial neural networks that teach computers or machines by imitating how the human brain gains certain knowledge. Thus, deep learning is a neural network with more than three layers. In recent years, deep learning has seen tremendous growth in its popularity and usefulness, largely due to more powerful computers, larger datasets, and techniques to train deeper networks [34]. The influence of deep learning demonstrates its effectiveness when applied to RSs, as it has been dramatically changing recommendation architecture and providing more opportunities to improve the performance accuracy (e.g., recall, precision, etc.) of RSs [35]. Deep learning methods mainly used for RSs are deep belief networks (DBN), multilayer perception (MLP), auto-encoder (AE), convolutional neural networks (CNN), recurrent neural networks (RNN), restricted Boltzmann machine (RBM), neural autoregressive distribution estimation (NADE), and adversarial networks (AN) [35].

3.1.8. Other Techniques

Many researchers applied data mining techniques to improve the performance of RSs for learning. These techniques include neural networks, k-nearest neighbor, regression, association rule, decision trees, link analysis, support vector regression, and so on. Therefore, it is essential to classify RSs based on these data mining techniques.

3.2. Classification Based on Recommendation Goals

Building RSs for any application domain is associated with the user tasks that the system can support. Understanding the different goals for which RSs are being implemented can help researchers accurately evaluate the systems’ functions. Herlocker et al., in [36], have identified other user tasks that help to differentiate among various evaluation measures, where they categorize those user tasks and recommendation goals as follows:

• Recommend Sequence: For recommending a series of related items;
• Find Credible Recommenders: Making recommendations while at a testing stage;
• Annotation in Context: Suggest an additional recommendation to the user while using the existing one;
• Find Good Items: Recommending the user a ranked list of specific items;
• Just Browsing: Recommend items even when the user did not request them;
• Find All Good Items: Recommending all appropriate items.

Furthermore, according to [7], RSs for e-learning should achieve more goals than the recommendation goals identified by [36]. They, therefore, outlined three more recommendation tasks as enumerated below:

• Find Novel Items: Provide the learner with novel or new learning objects;
• Find Peers: Find and recommend other learners with similar interests as the learner;
• Find Good Pathways: Recommending different possible learning pathways.

3.3. Classification Based on Platform

Research is continually carried out in RSs for different applications, and the implementation method is essential. Implementation of RSs can be done in four ways depending on the goal the researcher aims to achieve. Below are ways RSs can be implemented:

• Online (also called web-based): implementation online can be accessed easily once the user has an electronic device connected to the internet;
• Offline: it is an implementation that is not hosted on the web;
• Chatbot: involves using an agent (a computer program that simulates and processes human conversations as if they were real people) to interact with users;
• Mobile-based: an implementation that can be accessed via a mobile device.

3.4. Classification Based on User Type

RSs in e-learning can be implemented for several audiences depending on the problem it aims to solve. The system’s user type is important as it tells who will appropriately utilize it. In technology-enhanced learning, the users of the systems can be (but are not limited to) the students, teachers, or both teachers and students.

4. Results and Discussion

This survey was achieved by examining published articles found in central databases such as IEEE, ACM, ScienceDirect, WoS, Google Scholar, and Scopus. These articles were analyzed not just based on the classification method highlighted in Section 3, but the demographic distribution, recommendation element, and publication type were also considered. This section presents the result of the analysis conducted.

4.1. Recommendation Goals

The survey evaluated each proposed system’s goals using six user task categories. Still, due to a close similarity between the ”Find Good Items” and ”Find All Good Items” categories, we treat any paper that falls under these two classes as being in the same category. Moreover, we consider incomplete systems (work-in-progress papers) as part of our target papers if the required information can be identified.

Table 2. Recommendation goals.

Tasks	Works on the User Task
Find Novel Items	[37,38,39,40,41,42,43]
Find Peers	[19,43,44,45,46,47,48]
Recommend Sequence	[49,50,51,52,53,54,55,56]
Annotation in Context	[57,58,59,60,61]
Find Good Pathways	[62,63,64,65,66,67,68,69]
Find Good Items/Find All Good Items	[44,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151]

presents the breakdown of the research papers investigated according to the recommendation goals. Furthermore, Figure 1 displays a simple statistical analysis of the number of research papers under each recommendation goal. Thus, it is apparent from Table 2 and the chart (Figure 1) that most of the work was conducted to provide a sequence or find all good learning objects.

Interestingly, all six recommendation goals have been covered by a reasonable sample of papers that support learners in accomplishing those specific tasks. The recommendation element was also covered in this survey. Most research papers reviewed recommended learning resources and courses for students or teachers. The study sequence, books, online courses, etc., were not left out of the recommendations. Figure 2 shows that most of the surveyed papers recommended learning resources, while projects, movies, learning design, journals/conferences, and difficulty ranking were the least recommended to learners.

4.2. Recommendation Techniques

Different recommendation techniques have been used in implementing various RSs.

Table 3. Recommendation techniques.

Year	Collaborative Filtering	Content-Based	Knowledge-Based	Rule-Based	Hybrid	Context-Aware	Deep Learning	Others
2021	[68,138,141,146,149]	[61]	[69]	[138,148]	[139,144,145]	[140]	[56,142]	[137,143,147,151]
2020	[48,126,127,129,131,133,150]		[67]	[130]	[114,123,128,135,136]	[55]	[54,125,132]	[124,134]
2019	[106,107,108,111,112,120,121]	[113]	[118]	[119]		[60,110]	[53,106,115,116,122]	[109,117,118]
2018	[43,64,86,103,104,105]	[42,96,101]	[98]		[52,65,97,99,102]	[43]	[42,66]	[100]
2017	[41,84,87,92,95]	[39]	[94]		[40,59,85,89,90,93]		[88]	[91]
2016	[19,73,75,76,77,79]	[79]	[80,81]		[51,63,82,83]			[74,78]
2015	[37,38,47,71,72]	[44]	[71]		[45,46,50]	[57,58]		[49,62,70]

presents all the papers based on the technique used and their corresponding years of publication to classify the research papers based on multiple recommendation techniques. To measure the annual frequencies of each RS technique, Figure 3 summarizes the number of paper publications under each category every year. Despite some of their weaknesses, it is apparent from this finding that the most used recommendation techniques are collaborative filtering and hybrid. Nevertheless, context-aware recommendations, a popular nontraditional technique that integrates usual recommendations with specific conditions under which recommendations should be made, have also been explored in building many RSs for learning. Previous sections of this paper have highlighted the significant advantages of using the multi-criteria recommendation technique, and its apparent benefits have been outlined in various works in the literature, such as in [27].

Based on our findings in terms of recommendation algorithms used, it was observed that recommendation approaches are becoming more evenly distributed as nontraditional techniques are still being explored for research in RSs to make learning more impactful. In this era of Big Data, implementing new techniques can significantly influence user satisfaction and provide a promising solution. In recent years, deep learning and other artificial intelligence techniques such as evolutionary computing, genetic algorithms, fuzzy logic, and Bayesian techniques are some of the “Other techniques” increasingly attracting considerable attention in RSs for learning.

4.3. Recommendation Based on Platform

As part of the survey from Figure 4, it was observed that only a few systems were implemented using chatbots and mobile. 1% of the implemented systems are mobile-based, which was the work of [57], and 2% of the implemented systems are chatbot-based RSs for learning which was the works of [100,143], while 59% were implemented online.

4.4. Recommendation Based on User Type

As pointed out in the introductory section, research has consistently shown that the scope of RSs for technology-enhanced learning does not only cover supporting learners with useful learning objects. It also supports teachers by providing teaching materials, recommending different motivational approaches to motivate their students, and assisting teachers in generating a lesson plan and other teaching activities. Unfortunately, issues related to building RSs for teachers have not been treated extensively compared to learner-based RSs. Among all the papers we surveyed, the dominating user type is students, as shown in Figure 5, while [43,47,96,97,118,136,143] 6% of all systems considered were built either specifically for teachers or for both teachers and students.

The sudden outburst of Covid-19 in 2020 has affected the educational pattern and shifted the method of learning to online, which many countries adopted. From the map in Figure 6 and

Table 4. Regional distribution of surveyed papers.

Country	Alpha Code	No. of Papers	Country	Alpha Code	No. of Papers	Country	Alpha Code	No. of Papers
China	CN	34	Pakistan	PK	2	Mexico	MX	1
India	IN	12	Ecuador	EC	2	Colombia	CO	1
USA	US	10	Germany	DE	2	Canada	CA	1
Morocco	MA	9	Bangladesh	BD	2	Israel	IL	1
Indonesia	ID	4	South Korea	KR	2	Czech	CZ	1
Brazil	BR	4	Serbia	RS	1	Australia	AU	1
Greece	GR	3	Saudi Arabia	SA	1	Thailand	TH	1
Jordan	JO	3	Malaysia	MY	1	Taiwan	TW	1
Vietnam	VN	2	Italy	IT	1	New Zealand	NZ	1
Tunisia	TN	2	Philippines	PH	1	France	FR	1
Spain	ES	2	Nigeria	NG	1	Lithuania	LT	1
Netherlands	NL	2	Japan	JP	1	UK	GB	1

, it can be observed that China, India, USA, and Morocco are the leading countries with the highest number of publications from our survey. The dark blue on the map shows the countries with the highest number of paper publications, and it fades as the number of publications reduces, e.g., Nigeria, Serbia, Czech, etc.

4.5. Classification Based on Publication Type

The current study was not explicitly designed to evaluate all research papers related to RSs. Instead, it was limited only to those systems that could be applied directly to technology-enhanced learning with students and teachers as the target users of the systems. Thus, 116 papers on RSs for learning were surveyed and classified according to the classification procedure. These papers are from conferences, journals, and book sections, with most of the surveyed papers being from conferences, as seen in Figure 7.

4.6. Deep Learning-Based RS in E-Learning Research

Deep learning-based RS has been implemented in e-learning mainly because of its flexibility, representation learning, sequence modeling, and nonlinear transformation. This section analyzes different works from the literature that have been proven to be effective and implemented using deep learning.

Ref. [42] proposed a content-based learning resource recommendation algorithm based on CNN to solve the automatic multimedia learning resource recommendation problem. The proposed CNN is used to predict the latent factors from the text information of multimedia resources. To train the CNN, the authors solved the input and output using a language model, a proposed latent factor model that was regularized by the L₁-norm, and the split Bregman iteration method. Their proposed approach uses text information directly to make content-based recommendations without tagging. The authors believe that the Bregman iteration method greatly improves CNN training efficiency.

Ref. [53] aimed to explore how to take advantage of the existing MOOCs to improve the learning quality of formal courses provided by accredited educational establishments. Therefore, the authors proposed MOOCs RSs for formal learning platforms. The platform was able to recommend effective MOOCs to learners in the formal curriculum. Their proposed system is based on Siamese long short-term memory (LSTM) networks to measure the semantic similarity between courses’ descriptions.

Ref. [54] proposed a deep learning-based course recommendation model that generates views from a different perspective and makes intelligent course recommendations for students. The proposed approach was implemented by collecting various data types to generate student and curriculum models. The data were then generated by studying the actual relationship between specific models. Finally, a deep learning technique was used to extract key features, select different recommended features, and generate different views to further recommend to students.

Ref. [56] proposed a deep learning-based RS that simultaneously integrates content (heterogeneous) information and sequence data in the candidate-generation process. The proposed model utilizes the sequence data of user history and the item’s heterogeneous information to improve the prediction accuracy of deep RSs. Their proposed approach modified the first layer of the deep learning-based recommendation model by concatenating a vector containing data regarding item order. Thus, the average preference vector and the sequential preference vector models can be aware of more information on the user, leading to a more accurate recommendation.

Ref. [66] presents a novel model of full-path learning recommendation in an online learning environment. This model relies on clustering and machine learning techniques based on a feature similarity metric on learners. The proposed approach was implemented by clustering a collection of learners and training a long short-term memory (LSTM) model to predict their learning paths and performance. A personalized full-path learning recommendation was then selected from the results of path prediction for a full practical path recommendation specifically for test learners.

Ref. [88] proposed a recommendation framework for e-learning based on deep learning to solve the limitation of effective methods in e-learning. The proposed model was based on a strong capability to learn from a training set and improvements over previous methods. The model was trained using the traditional k-nearest neighbor (KNN) method, which guarantees the model’s accuracy. Their system recommends new items whose similarity cannot be calculated. Additionally, their approach greatly reduces the heavy burden of a running system, which is helpful in the actual practice of RSs. The authors concluded that the framework offers a new recommendation method for more personalized learning in the future.

Ref. [106] proposed a deep learning-based RS approach called neural collaborative filtering (NCF) for predicting the grade a student will earn in a course that he/she plans to take in the next term. The deep learning-inspired approach provided added flexibility in learning the latent spaces compared to previous grade prediction methods based on matrix factorization (MF), where students and courses are represented in a latent “knowledge” space. The proposed approach incorporated instructor information in addition to student and course information. Finally, for proper analysis of the learned model parameters, the authors assumed that the embeddings obtained for students, courses, and instructors should be non-negative. This non-negative NCF model, referred to as the NCFnn model, added a rectified linear unit (ReLU) on the embedding layer of NCF.

Ref. [115] proposed a novel deep reinforcement learning framework (DRE) for adaptively recommending exercises to students with the optimization of three objectives: review and explore, smoothness of difficulty level, and engagement. The framework proposed two different exercise Q-networks (EQN) to generate recommendations for an agent. The first one is a straightforward yet effective EQNM with the Markov property, where the next recommendations to students only depend on their current exercising performance. In contrast, the second is a more sophisticated EQNR with a recurrent manner, where they tracked all students’ exercising history to make decisions. Both architectures are model-free; thus, the authors did not need to calculate the state transitions in the exercise space. The authors also proposed three novel reward functions to formally capture and quantify those three objectives so that the DRE could update and optimize its recommendation strategy by interactively receiving students’ performance feedback.

Ref. [116] introduced a novel goal-based RS based on adaptations of RNN for suggesting courses to help students prepare for target courses of interest, personalized to their estimated prior knowledge background and zone of proximal development. The authors validated several model variations against test sets representing the tasks of grade prediction, prerequisite prediction, and course selection prediction. These three prediction validations were chosen to suggest whether the approach warranted testing in the wild. The proposed approach model’s definition was based on a popular variant of RNNs called LSTM. LSTM was used because it helps RNNs to learn temporal dependencies by adding several gates, which can retrain and forget select information.

Ref. [122] proposed a time slice imputation for personalized goal-based RSs in higher education. This paper builds on previous research in [110] by improving the tractability of generating a recommendation from the architecture introduced in [110] and applying this improvement to the task of course preparation recommendation, as it was applied in the prior work, and extending its application to MOOC quiz preparation, where it has not yet been applied. The proposed improvement in tractability was achieved by treating the space of recommendations (e.g., courses or course pages) as weights in a neural network that stochastic gradient descent can learn on a per-learner basis to optimize for a future goal (e.g., grade on a course or quiz), while the prior work [110] approached this optimization by enumerating every possible recommendation and evaluating it as an input to the model.

Ref. [125] proposed an attention-based CNN to obtain a user’s profile, predict user ratings, and recommend the top n courses for MOOC learners. The proposed approach was implemented by first representing learner behaviors and learning histories into feature vectors. The attention mechanism was then used to train the neural network to improve relevance estimation according to the differences between the estimation scores and the actual scores given by users. Finally, the trained model recommended courses to learners. The authors presented a web-based course RS framework. Their proposed system can identify the learning habits of students and assist them in identifying preferred courses. However, it may still suffer the problem of recommending similar courses due to the huge number of courses with the development of MOOCs.

Ref. [132] proposed a personalized course recommendation system based on eye-tracking technology and deep learning. The proposed approach is a novel click-through rate (CTR) model for personalized online course recommendations with discriminative user, item, and cross features. The feature representation ability of the CTR model was improved, and the serious challenge of cold-start problems was alleviated. Furthermore, transfer learning was introduced to deal with the problem of insufficient data in model training. More specially, eye-tracking technology was applied to learn the users’ cognitive styles, which are visualized with a heat map and fixation point trajectory to achieve a reasonable interface for personalized course recommendations. Finally, the recommendation interface was sent to the learners according to the users’ cognitive styles.

Ref. [142] proposed an efficient e-learning recommendation (EELR) system for user preferences using a hybrid optimization algorithm (HOA). The EELR system constructs an HOA with a deep recurrent neural network (DRNN) and an improved whale optimization (IWO) algorithm. The proposed approach was implemented by first utilizing DRNN to order and analyze the e-learner types with their group, which was used to classify with deep multi-layers. After that, the learners’ conduct and inclinations were examined by completing the mining of the arrangements observed frequently by the IWO calculation. Finally, the proposal of e-learning depends on the appraisals compared to these arrangements observed frequently. This proposed system was to implement and validate numerous e-learning entries against client inclinations over some undefined time frame and proved to be more proficient and exact in contrast with the customary recommender framework. The authors concluded that this strategy could help learners to grasp the knowledge system and learning direction and improve their learning efficiency.

5. Conclusions and Future Work

RSs for learning has become an independent and vital research domain aimed at designing, implementing, testing, and evaluating systems to know how efficient recommendation algorithms are or investigate t whole systems’ performances. Other evaluation methods include assessing the impact of systems toward achieving desired learning goals and measuring users’ satisfaction and trust in systems. Different systems have been developed to run on various computer devices and environments. This research was undertaken to give an overview of various RS techniques and statistics of research papers on RSs for learning based on (1) the technique(s) used to design and implement the system; (2) the users’ tasks supported by the systems; (3) the platforms (chatbot, mobile or web-based); and (4) the kind of users of the system (students or teachers).

This survey has shown significant findings regarding the recommendation goals, techniques, and platforms used for its implementation. Firstly, it was observed that a great deal of research was implemented using traditional techniques, mostly collaborative filtering and hybrid approaches. Still, a substantial number of works also followed either deep learning, hybrid, or context-aware recommendation techniques, which are considered more efficient than traditional methods. Secondly, in terms of supporting user tasks, it was shown that most systems either recommended “sequence” or “find all good learning objects” to the learner. The third significant finding concerned the platforms for which the proposed systems were designed to run, showing that very few systems were designed to run on mobile platforms. This result has shown a great need to improve and change our research direction toward mobile-based applications, as much of the current literature has confirmed that smartphones and tablets reduce the frequency of using ordinary cell phones and, to a large extent, replace personal computers and other devices. One of the most striking new recommendation techniques that can significantly improve recommendation accuracy is the multi-criteria recommendation technique, which has been used and proven by researchers in industries and academic institutions to provide more accurate predictions than traditional techniques [152]. Many researchers have also established its efficiency in the RS domain [27]. Few recent research publications, such as [48,72,84,91,120], have applied the multi-criteria recommendation technique.

Nevertheless, another emerging area that goes in line with RSs for learning in terms of faster growth and development is the domain of social networking, which, if integrated with RSs for learning, can enable learners to connect, communicate, and share knowledge online. Fulfilling this can make social media and RSs major players in e-learning, as many learners can get in touch with each other in various ways. Unfortunately, even though it is not clearly stated within the aim of this survey, we tried to filter those systems developed to take advantage of social media to improve learning object recommendation. None of the papers were purposely designed as social network-based RSs for learning. The implication is that there is a need for research in this domain to focus on merging different social network sites with RSs for effective learning object recommendations.

Generally, some gaps were identified in this survey. Due to these gaps, we suggested that future work should focus on the following:

• Building RSs for learning to support various user tasks to provide serendipitous recommendations;
• Putting more effort into developing mobile-based and chatbot-based RSs for learning;
• Building RSs specifically for teachers, as only 6% of our reviewed papers were designed to be used by teachers;
• Nontraditional RSs, especially multi-criteria ratings RSs, which have direct applications in this domain, are also promising approaches for future work. Moreover, the primary concern still open to the RSs community is finding suitable optimization algorithms that can explore the relationships between multiple ratings to compute an overall rating [153]. Future research needs to explore selection techniques such as intelligent data pre-processing and segmentation to ascertain the best criteria for modeling multi-criteria RSs for learning.
• Integrating learning styles into learning systems can lead to an intelligent and adaptive learning system that can adjust the content to ensure faster and better performance in the learning process [154]. Several systems, such as [148], proposed assessing students’ learning styles, which can be easily extended to a context-aware recommendation;
• Most researchers followed a traditional approach; therefore, more efforts similar to introducing transfer learning, as suggested by [148], need to be put in place to tackle data sparsity problems in collaborative filtering and over-specialization problems in content-based filtering. These problems can be mitigated by successfully implementing serendipitous RSs for learning. Further work is required to build RSs for learning to solve these problems and avoid recommending apparent items;
• As mentioned in the last paragraph, one of the outstanding issues for RSs for learning is the modern-day changes that show how social media has become a significant player in almost all our daily activities. Consequently, the challenge lies in following this trend and developing social network-based RSs for learning; this will be an exciting area for future research.

While the paper overcame the limitations of most recent surveys by considering research papers from journals and conferences and by examining papers irrespective of the country, due to practical constraints and the language barrier, the reader should bear in mind that the study was based on papers written in English. Therefore, one possible extension of the current study is to consider non-English papers, master and doctoral dissertations, related works from textbooks, news articles, and so on to provide an extended classification model.

Furthermore, searching the reviewed papers was performed using keywords related to RSs and learning, such as personalized systems, using names of recommendation techniques, journals, and conference proceedings, in addition to the keywords mentioned on the first page of the paper. In addition to these keywords and phrases, we did not search papers using names of other data mining techniques or authors whose research interest is mainly on RSs for learning. Other possible means of searching could be names of subcategories of recommendation techniques such as case-based reasoning, attribute-based and demographic-based techniques, and other non-English words that describe the system in another language. Further studies that consider these new search techniques will need to be undertaken.