FutureCite: Predicting Research Articles’ Impact Using Machine Learning and Text and Graph Mining Techniques

Abstract

The growth in academic and scientific publications has increased very rapidly. Researchers must choose a representative and significant literature for their research, which has become challenging worldwide. Usually, the paper citation number indicates this paper’s potential influence and importance. However, this standard metric of citation numbers is not suitable to assess the popularity and significance of recently published papers. To address this challenge, this study presents an effective prediction method called FutureCite to predict the future citation level of research articles. FutureCite integrates machine learning with text and graph mining techniques, leveraging their abilities in classification, datasets in-depth analysis, and feature extraction. FutureCite aims to predict future citation levels of research articles applying a multilabel classification approach. FutureCite can extract significant semantic features and capture the interconnection relationships found in scientific articles during feature extraction using textual content, citation networks, and metadata as feature resources. This study’s objective is to contribute to the advancement of effective approaches impacting the citation counts in scientific publications by enhancing the precision of future citations. We conducted several experiments using a comprehensive publication dataset to evaluate our method and determine the impact of using a variety of machine learning algorithms. FutureCite demonstrated its robustness and efficiency and showed promising results based on different evaluation metrics. Using the FutureCite model has significant implications for improving the researchers’ ability to determine targeted literature for their research and better understand the potential impact of research publications.

1. Introduction

There has been a great emphasis on the data mining fields of computer science that can be applied in different fields. Specifically, text and graph mining have been extensively used to extract hidden interesting findings, insights, and features from vast amounts of data in various domains [1,2,3,4,5]. The exponential growth in academic publications in recent years has made it increasingly difficult for researchers to identify significant literature for their work. Citation numbers that were obtained from different databases such as Google Scholar, SCOPUS, ORCID, and Web of Science (WOS), are often used to measure a research paper’s popularity, quality, and significance [6]. However, citation numbers cannot be a reliable indicator of the recent paper’s importance. As a result, more attention has been given to the problem of predicting future citation levels for publications across different domains [7] Machine learning (ML) and deep learning (DL) have been utilized by incorporating text and graph features to solve such problems [8].

Previous studies have emphasized the importance of text-based features using text mining and text embedding techniques in different domains [9,10,11], including our problem of citation level prediction [12,13]. These text-based features have been obtained through feature extraction, a critical step in ML, as it involves transforming raw data into a set of features used to train the ML model. For instance, West and coauthors [14] designed a citation recommendation system based on scientific papers’ content and citation relationships. They employed text-based features such as titles, abstracts, and keywords to capture the scope and topics of publications and enhance the citation recommendation’s accuracy.

Although the above and other studies focused on employing text-based features to predict the future research article impact and citation counts, they should have considered crucial features derived from complex networks formed with citations, articles, and authors. Therefore, recent studies utilized graph-based features to predict the importance of the publications [8,15,16]. For example, CiteRivers [17] relies on visualization capabilities, including graph visualization analysis. Thus, they extended the capability of the previous methods, which were dependent only on visualization, and represented a new visual interactive analysis system for investigating and analyzing the citation patterns as well as contents of scientific documents and then spotting the trends they obtained. Another method [18] developed an enhanced recommendation system based on the Author Community Topic Time Model (ACTTM) and bilayer citation network. They created an author–article citation network model consisting of author citation layers and paper citation layers. After that, they proposed a topic modeling method for recommending authors/papers to researchers. The third study utilized graph-based attributes, formulating the problem as a supervised link prediction in directed, weighted, and temporal networks [19]. They developed a method to predict the links and their weights and then evaluated it by conducting two experiments that demonstrated the effectiveness of their approach. The last study to mention is a graph centrality base [20] that applied graph mining techniques to find papers’ importance by computing the centrality measures such as PageRank, Degree, and Betweenness Closeness. They demonstrated their finding that topologically based similarity using the Jaccard and Cosine similarity algorithms outperforms the same similarity algorithms applied to textual data. Moreover, a few recent studies have taken advantage of graph-based and text-based features to predict future research paper citations and proved their efficiency [21]. One of these studies, ExCiteSearch [22], is a framework developed to enable researchers to choose relevant and important research papers. ExCiteSearch implemented a novel research paper recommendation system that utilizes both abstract textual similarity and citation network information by conducting unsupervised clustering on sets of scientific papers.

Recent studies take advantage of graph embedding techniques to encode nodes or edges into low-dimensional space and generate feature representation through unsupervised learning methods, successfully enhancing the performance of different downstream tasks such as node classification and link prediction [23,24,25]. For instance, the Paper2Vec method utilized text and graph data to generate a scientific paper representation [26]. Unlike methods that rely on traditional graph mining for feature extraction, Paper2Vec employs a graph embedding technique to generate a paper representation that can be used for different tasks, including node classification and link prediction related to predicting the paper’s significance. Paper2Vec leveraged the power of unsupervised feature learning from graphs and text documents by utilizing a neural network to generate paper embedding. This method demonstrated the robustness of paper embeddings using three real-world academic datasets. The significance of the research paper embeddings generated by the Paper2Vec method has been validated via different experiments on three real-world academic datasets, indicating Paper2Vec’s capability to generate strong and rich representations.

Even though several approaches have been proposed to predict future citation levels using different feature models, there are still some limitations and more room to improve the prediction performance. Most existing methods that predict the importance of a research paper depend either on graph data or text data, and a few studies have integrated both text and graph sources to extract features. Using only one source of the features often results in incomplete representations of a paper’s potential impact. Additionally, current feature selection and extraction techniques are limited, often relying on metadata that does not fully capture a paper’s significance. Additionally, contextual content analysis is not utilized enough, causing less accurate predictions for recently published articles. Moreover, the problem of citation level prediction remains challenging due to the imbalance in citation distribution. This leads to biased predictions, and thus the need for effective feature selection and extraction techniques. In this regard, we contribute to the existing literature by presenting an effective method called FutureCite for predicting citation levels by integrating text and graph mining techniques using both text and graph information. A novel aspect of our approach is formulating the problem as a multilabel classification task, where we use a range of citation levels as current indicators of a paper’s importance. FutureCite is designed to provide researchers with a tool to identify promising research papers that are influential in their field and better understand the research impact. We utilized various feature extraction techniques and a dataset of thousands of publications from the data mining field to develop our method. This study contributes to the field of citation analysis by addressing the challenge of predicting future citation levels.

The overall structure of this paper is as follows. Section 1 introduces the motivation for predicting future citation problems, conducts a literature review of recent research that has been published contributing to this topic, and finally presents the fundamentals of some important graph and text mining concepts used in this work. Section 2 describes the datasets that have been utilized in this project. Section 3 presents the FutureCite model’s methodology phases and introduces the problem formulation. Section 4 discusses the results and performance of our proposed approach. Finally, we conclude our work, highlight some limitations, and outline potential future directions for research in this area.

2. Materials

To develop the FutureCite model, we utilized a popular scholar dataset [27], which has been collected from scientific research articles that have been published in different venues (i.e., conferences and journals) such as ICDE, KDD, TKDE, VLDB, CIKM, NIPS, ICML, ICDM, PKDD, SDM, WSDM, AAAI, IJCAI, DMKD, WWW, KAIS and TKDD. This dataset was extended by adding the references for all documents (i.e., research articles).

In this study, we utilized part of this scholarly dataset that focuses on data mining publications provided by the Delve system’s authors [27]. Thus, the dataset we used to develop the FutureCite model is the data mining publications dataset. Each research article in the dataset has several attributes: paper ID, venue name, authors’ names, publication year, paper title, index keys (i.e., keywords), the paper abstract, and references. We divided our input dataset into four types based on the feature categories we will extract.

• First, the research paper data we utilized to extract text and metadata features in our dataset consists of 11,941 papers. However, after applying different filters to the dataset (described in the cleaning and preprocessing phase), we obtained 6560 research papers. Thus, the size of our dataset is moderate.
• Second, the citation graph data, where all papers and their corresponding references are provided, formulate citation edge lists consisting of 133,482 papers and 335,531 relationships (i.e., citations). This citation graph is utilized to define the class labels using different thresholds for each label. These thresholds were selected based on the distribution of citation counts in the dataset, as will be explained later.
• The third type is the Author–Coauthor Graph, comprising 10,270 authors and 26,963 author–coauthor relationships.
• The fourth and last data type, the Author–Research graph, consists of 10,270 authors and 6560 publications and the relations between them.

The details of each type, with its descriptions, are discussed later in the feature extraction section.

3. Methods

3.1. Problem Formulation

This study describes the objective of predicting the research paper’s future citation level as supervised learning, specifically multilabel classification. Thus, given a dataset of data mining publications, we are aiming to predict each paper citation level based on predefined class labels. As mentioned in Section 2, all data samples (i.e., published research papers) in our dataset have their features and can be represented as vector X = {x₁, x₂, …, x_n} where n is the number of all data samples. In addition to the data samples, we also provided all data samples with their class labels Y = {y₁, y₂, …, y_n} since our problem is supervised learning. We prepared the class labels based on the citation numbers of the paper in our dataset. The process of creating the class labels is explained in Section 3.3.

We extracted several features from different perspectives for each data sample (research paper), explained later. The classification model goal is to find the hidden patterns and associations between research papers and their class labels based on the features we extracted and then predict the actual class labels.

3.2. FutureCite Model Workflow

Figure 1 provides the workflow to develop the FutureCite model, which involves five main steps. These steps are summarized as follows:

• Data preparation and preprocessing;
• Feature extraction and integration. These features are grouped based on three categories: graph-based, text-based, and metadata-based.
• Data normalization and sampling;
• Class label preparation;
• ML classification, where several classifiers are built for class prediction.

A detailed explanation of each step is provided in the following subsections.

3.3. Class Label Creation

Since our model is supervised learning, as explained earlier, we must provide the class labels along with the data samples. Therefore, each research paper in the dataset should have a class label. All class labels were created using the citation graph based on the in-degree measurement. Thus, we began by utilizing a specific part of the dataset, including the research papers and their references, to build the citation graph. A graph G = (V, E) consists of a set of vertices (i.e., nodes) V and a set of edges E. In our study, each node in the citation graph represents the research paper, and each edge between two nodes represents the citation relationship (i.e., paper 1 is a reference for paper 2, or paper 2 cites paper 1). Consequently, we obtained a directed unweighted graph consisting of 133,482 nodes (including papers and references) and 335,531 edges (links). Next, we applied a graph mining technique to calculate each node’s in-degree feature that reflects the citation number of each paper.

Table 1. Top-5 Research Paper In-degree Feature.

Paper ID	In-Degree
085B9585	394
812313D9	371
70128864	362
641D5808	337
59C818AC	327

shows an example of a top-5 paper in-degree.

Next, we applied a graph mining technique to calculate each node’s in-degree feature, a key metric that reflects the citation number of each research paper. After that, we obtained the in-degree distribution of all research papers, as shown in Figure 2. Also, we provide an illustrative example of a top-5 research paper in-degree measurement as shown in Table 1. As illustrated in Figure 2, we observed extremely low values for in-degree features across the dataset. We calculated the in-degree average to investigate this metric further, which equals 2.2. This average in-degree provides a baseline for acquiring the class labels in our data.

Despite the average in-degree being 2.2, we made a necessary adjustment to the threshold average to be 5. This procedure was crucial as many references (i.e., research papers) within our dataset have no citations (i.e., in-degree = 0). This choice is of benefit to account for the skew in our data caused by research articles with no citations and to allow for additional meaningful classification. Following this adjustment, we conducted an empirical analysis of the distribution of the in-degree values concerning the new threshold average. Based on this analysis, we determined four class labels for research papers in our dataset. These classes correspond to different ranges of citation counts, as detailed in

Table 2. Class Label Rules and Description.

Rule	Class Label	Label Description	Number of Research Papers in This Class
In-degree > 50	1	Highly cited	44
In-degree > 20	2	Well cited	180
In-degree > 5	3	Above/ average cited	1118
In-degree ≤ 5	4	Below average cited	5218

. Each paper in the dataset is assigned to one of these classes according to its in-degree value, which measures its citation count. To better understand the complete process for class label creation, we provide a pseudocode for this process, as illustrated in the algorithm representation shown in Figure 3.

3.4. Data Preprocessing and Feature Extraction

In the FutureCite study, since we performed the data preprocessing differently considering the feature category, we will explain each feature group’s data preprocessing and extraction steps together. As a result, the data preprocessing step was applied three times, removing certain data samples from each group.

The feature extraction techniques we employed, based on text and graph mining, are specifically designed to capture various aspects of the data mining publications datasets. Three categories of features were utilized: text-based, graph-based, and metadata-based. The features of each research paper cover various perspectives, including the quality and significance of the publication venue, the collaboration between authors, the individual authors’ expertise and influences, the content of the publications themselves, and other features. These features were then used to develop and train a classification model using multiple ML algorithms to predict paper citation levels based on predefined class labels, ranging from low to highly cited. Each feature category comprises multiple characteristics that have been created based on specific attributes of the dataset. Each category is explained in more detail in the following subsections and illustrated in Figure 1.

3.4.1. Metadata-Based Features

By metadata, we refer to the extra information that we can add to each research paper, which is, in our context, the publication venues and years. Venue-related features are generated based on the venue’s name, known as the impact factor, and the h-index of each publication venue. The h-index is a metric used to evaluate the conference or journal’s quality [28]. To ensure the quality of our publication’s datasets and the relevance of the research papers to the same domain, we implemented a preprocessing procedure on all research papers in our datasets, consisting of 11,941 papers. During this process, we filtered out any research paper unrelated to the data mining conferences or journals and those with a count of fewer than 50 in our datasets. In this step, it is important to have all publications datasets related to the same field to find hidden patterns and associations in the data. Consequently, the number of research papers we utilized was reduced, resulting in our obtaining 7069 research papers published in 23 venues. In addition, we filtered out all research papers that were published before 2001, resulting in 6560 research papers. The data mining venues within our limited datasets and the corresponding number of research papers published in each venue from our datasets are shown in Figure 4. Finally, the h-indices are collected from the Google Scholar website and incorporated as a new feature (venue-h-index) for each research paper. For instance, the Knowledge Discovery and Data Mining (KDD) conference has a good reputation and exhibited an h-index of 42.

Another example is the IEEE Transactions on Knowledge and Data Engineering journal, one of the most popular and strongest data mining conferences, with an h-index of 99. Although only one feature is related to the venue, it indicates the research paper’s significance considering the conference or journal reputation, which, in turn, affects the predictions of paper citation futures.

3.4.2. Text-Based Features

Text-related features have a crucial role in capturing the topics and contents of the research papers. These features include the research paper title and abstract. We started by combining all titles and abstracts for the 6560 documents (i.e., research papers) and then preprocessing them. The data preprocessing steps that we applied involved text tokenization, punctuation elimination, stop and common word removal, and text stemming. After that, a bag-of-words (BOW) feature vector was constructed using the Term Frequency–Inverse Document Frequency (TF–IDF) vectorizer technique [29], which is a popular natural language processing (NLP) technique that is a commonly used technique for determining the importance of a term in a document. TF–IDF reflects the relevance of a term to a document by measuring the frequency of the term in the document and inversely scaling it by the frequency of the term across all documents in the corpus. It is used to build a bag of word feature vectors that have been used extensively for several methods that rely on text-based features for prediction. We optimized several hyperparameters for TF–IDF vectorizers such as max_df, min_df, ngram_range, and max features to obtain the best and most effective bag-of-words FV. For example, if max_df = 0.4, it means, when building the vocabulary, “ignore terms that appear in more than 40% of the documents” (i.e., have a document frequency strictly higher than the given threshold). Likewise, if min_df = 0.01, it means “ignore terms that appear in less than 1% of the documents”.

As a result of this process, we obtained 802 features from the text-based data. However, we considered this number of features a high dimensionality of the feature space. Therefore, dimensionality reduction approaches are employed to mitigate the issue of having many correlated features. This means we need to reduce the number of features into fewer uncorrelated variables by taking advantage of the existing correlations between the input variables in the dataset. We utilized two approaches which are: principal component analysis (PCA) and singular value decomposition (SVD). However, after comparing the performance of those two approaches, we selected the PCA approach since it provides superior results to SVD. PCA effectively reduces the number of features to 420, which captures around 85% of the variance. Consequently, the resulting FV from the text-based category is 420.

3.4.3. Graph-Based Features

In the graph-based features, we focus on extracting the information relevant to the authors of each research paper. Two graphs, the author–coauthor graph, and the author–paper graph, are constructed to extract author-based features. Since we used all author and co-author names in both graphs, we began with the preprocessing step on all authors’ and co-authors’ names before constructing a graph. This preprocessing step comprises converting the names to uppercase, removing all dots (“.”), eliminating middle names, and matching some first names with corresponding initials.

The first graph (G1(V, E)) represents the author–coauthor relationship. This undirected, unweighted graph is built by converting all author–coauthor relationships into an edge list, resulting in 10,270 nodes V (authors) and 26,963 edges E (relation). We visualize the author–coauthor graph to represent the relationships between authors since data visualization (in our case, graph visualization) is an essential tool used frequently in different domains, allowing researchers to explore large and complex datasets more effectively and gain useful insights from the data [30,31]. However, to achieve the best visualization, we applied a filtering procedure by removing all authors that have no connection with other authors or the authors with connections below a specific threshold. This process helps to reduce less important authors and enhance the visualization for more impactful authors. Figure 4 visualizes the author–coauthor graph where the node size indicates the importance of the corresponding authors that appear the most in our dataset. As we can see, the largest node represents Jiawei Han, one of the pioneering scientists in data mining, who has current citation records of more than 250,000. Another insight we can obtain from the visualized graph in Figure 5 is the edge width that represents the connection strength between authors based on their author–coauthor relationship in several publications. Furthermore, the color indicates the community of authors.

After formulating the graph, graph mining techniques then employed three global ranking algorithms to extract three graph properties for each author, which are degree centrality (DC), betweenness centrality (BC), and PageRank [32]. These three features derived by graph mining prove their significance and efficiency in several ML problems across various domains [33,34,35,36,37] and predict future citation problems. The definition and equation for each measurement are as follows:

• Node Degree—C_D(v) [38]: represents the sum of all edges connected to a node v. In a directed graph, the degree can be divided into two categories: in-degree and out-degree.

$$C_D\left(v_i\right)=d_i={\sum }_j{A}_{ij}$$

(1)

• Page Rank [39], equivalent to the eigenvector centrality C_E of node v, measures the node’s importance based on its neighboring nodes’ importance.

$$C_E\left(v_i\right)={\sum }_{v_j\in N_i}{A}_{ij}{C}_E\left(v_j\right)$$

(2)

• Betweenness Centrality C_B [40] of a node v counts the shortest paths that pass through the node. It is calculated using the following equation:

$$C_B\left(v_i\right)={\sum }_{v_s\ne v_i\ne v_t\in V,s<t}{\displaystyle \frac{{\sigma }_{st}\left(v_j\right)}{{\sigma }_{st}}}$$

(3)

where σ_st is the total number of shortest paths between nodes v_s and v_t, and σ_st(v_i) is the number of shortest paths between nodes v_s and v_t that pass along node v_i.

After employing the equations above, we obtained three properties of each author, as shown in

Table 3. The Top-5 authors in our dataset from the author–coauthor graph.

Author Name	Degree	PageRank	Betweenness Centrality
PHLIP YU	165	0.002163	0.068579
CHRISTOS FALOUTSOS	182	0.002010	0.066565
JIAWEI HAN	224	0.002468	0.063534
HEIKKI MANNILA	57	0.000814	0.020424
QIANG YANG	88	0.001045	0.019324

, where we list several examples of the Top-5 prominent authors in our datasets. Furthermore, three features for each data sample (i.e., research paper) were calculated. Those features are the average degree, the average page rank, and the average betweenness centrality of all authors associated with each paper.

Additionally, we constructed a second graph known as the author–paper graph. We created an edge list that connects the 10,270 authors to the 6560 publications, resulting in 16,830 nodes. An edge is created if an author’s name appears in the publication authors list. The degree of each author node is computed as a feature for each author. Consequently, the average of all authors’ degrees is calculated as a fourth graph-based feature and assigned to each paper. The resulting FV from the graph-based category is 4, but it is important in citation-level prediction.

3.4.4. Features Integration

Upon completing the feature extraction process, we obtained one venue-related feature, 420 text-related features, and four graph-related features. The order of the documents remained consistent across all feature category extraction processes, facilitating the concatenation process to integrate all features into a single large comprehensive FV. After this, we applied a min-max normalization to all features based on each column, ensuring that all features have the same scale.

3.5. FutureCite Predictive Model

3.5.1. Sampling Techniques for Imbalanced Data

Figure 3 shows that the number of data samples in each class is different. We can see clearly that the numbers of data samples in classes 3 and 4 (i.e., classes with average and above average labels) are much larger than in classes 1 and 2. This issue is called imbalanced data and must be addressed since the ML classifiers face a problem in prediction based on imbalanced data. The imbalanced data affect the ML models’ ability to classify most test samples into the majority class when the minority class lacks information. We addressed this issue by applying random oversampling [41] in the training data to balance the data. This technique is implemented using an imblearn Python package version 0.6.1 [42].

3.5.2. Multilabel Classification Model

In the FutureCite model, after completing the feature extraction process and obtaining a feature vector (FV) for all research papers, we fed the FVs with their class labels into several supervised ML classifiers, which are the support vector machine (SVM) [43], Naïve Bayes (NB) [44], random forests (RF) [45], and eXtreme Gradient Boosting (XGBoost) classifiers [46].

The SVM classifier works by finding the hyperplane that maximally separates the classes in the feature space. SVM uses a kernel function to transform the input features into a higher dimensional space where the classes can be more easily separated. The optimal hyperplane is found by maximizing the margin, which is the distance between the hyperplane and the closest data points from each class. Naïve Bayes is a probabilistic algorithm that is based on Bayes’ theorem. It works by calculating the probability of each class given the input features and selecting the class with the highest probability as the output. Both SVM and NB perform well in many applications, particularly when dealing with high-dimensional data such as text classification, and this is the reason we picked them in our prediction. The third classifier we applied is RF, which demonstrates its effectiveness in prediction because it operates efficiently on extensive datasets and is less prone to the overfitting issue. The last classifier is XGBoost, implemented using an optimized distributed gradient boosting library, called XGBoost [46]. XGBoost utilized classification and regression trees (CART), which use continuous scores assigned to each leaf and then summed them up to provide the final prediction instead of having equal weight as the standard decision trees.

In our study, for each classifier we optimized several critical parameters using the training datasets to improve the classifier’s performance. For example, for the SVM classifier, we optimized the regularization parameter C, the kernel function where we set it to the radial basis function (RBF), and the class weight to be balanced. For the NB classifier, the parameters include fit-prior bool, specified to learn the class’s prior probabilities or not, and class prior, which is the prior probability of the classes. Finally, for the RF classifier, the parameters include the number of trees in the forest (i.e., n estimators), the maximum depth of the trees, and the function to measure the quality of a split (criterion).

To implement our FutureCite method, we utilized the following tools:

• Python 3.3 [47] is used for implementing all project phases, including preprocessing, feature extraction, training and validation, and classification. We utilized several supported packages, including scikit-learn for ML algorithms, NetworkX for graph mining [48], imblearn package [49] to handle imbalanced class labels, Matplotlib to plot different figures, and Pandas data frame to deal with the data preparation and preprocessing [50].
• Gephi 0.9.2. [51]: Gephi is used to visualize and analyze graph data, including the author–coauthor and citation graphs. The data are preprocessed and prepared to be in a suitable shape for this tool. Gephi allows us to explore the structure of the citation network and gain valuable insights into the relationships between different research papers, authors, and venues. These tools are essential for developing a comprehensive method for predicting the future citation level of research papers.

4. Results and Discussion

This section describes the evaluation protocols, the experiments conducted, and the results of our FutureCite prediction performance based on several evaluation metrics. We further highlighted several possible characteristics that could boost the FutureCite method prediction performance.

4.1. Evaluation Metrics and Protocols

Several performance metrics are used to evaluate the prediction performance of the multilabel classification models. However, since our datasets are highly imbalanced, typical accuracy is not accurate or meaningful. Therefore, we employed three evaluation metrics derived from the confusion matrix [52], which are precision (also called positive predictive value), recall (also called sensitivity or true positive rate (TPR)), and the F1 score (a combination of precision and recall metrics) [53]. The calculation of these evaluation metrics is based on true positive (TP), and false positive (FP), which represent correctly and incorrectly predicted samples, and the true negative (TN) and false negative (FN), which represent the samples that are correctly and incorrectly predicted, respectively. Equations (4)–(6) illustrate the precision, recall, and F1-score calculations.

Precision = TP/(TP + FP)

(4)

Recall = TPR = TP/(TP + FN)

(5)

F1-score = 2 × Precision × Recall/(Precision + Recall) = 2TP/(2TP + FP + FN)

(6)

For the evaluation setting and protocol, we applied a hold-out validation approach. We split the dataset into 80% for training and 20% for testing in a stratified manner where we ensure the same proportion of each class label in both training and test splits. However, we repeated this process five times for sufficient test samples to generate reliable evaluation metrics. This setup is similar to five-fold cross-validation. We initially shuffled the data and then divided it into 20% and 80% for training and testing, respectively. Subsequently, we selected another 20% for testing and allocated the remaining 80% for training, and so on. In each cycle, we tested each classifier to predict the 20% of hold-out samples (i.e., test set) and calculated the three evaluation metrics identified above. Finally, we averaged the results over the five-test split for each evaluation metric.

4.2. FutureCite Prediction Performance Evaluation

The results of the SVM, NB, RF, and XGBoost classifiers, based on precision, recall, and an F1-score evaluation metric, are presented in

Table 4. Classification Evaluation Score.

Classifier	Precision	Recall	F1-Score
SVM	77.9%	75.6%	76.73%
NB	74.2%	64.6%	69.08%
RF	81.5%	78.8%	80.13%
XGBoost	82.4%	80.3%	81.30%

The bold font indicates the best and the italic indicates the second best.

and reveal interesting findings.

From the results reported in Table 4, it is evident that the XGBoost classifier demonstrates better performance than all other classifiers: Naïve Bayes, SVM, and RF. It achieves a precision of 82.4%, a recall of 80.3%, and an F1-score of 81.3%. XGBoost performs the best since it is a gradient-boosting algorithm, an ensemble-learning technique that combines multiple weak learners’ predictions to create a strong predictive model. Another advantage of XGBoost is that it provides a parallel tree boosting that improves the prediction performance in terms of speed. The second-best performance was achieved by RF, which obtained results close to XGBoost. The reason behind the RF classifier’s efficiency can be attributed to its ensemble-learning approach, which proves its effectiveness in dealing with imbalanced datasets. Another reason that XGBoost and RF performed the best can relate to the extensive optimization of several hyperparameters of XGBoost and RF resulting in improved performance. The third-best performance was achieved by the SVM classifier in all evaluation metrics, obtaining results lower than the RF classifier by 3.6%, 3.2%, and 3.4% in terms of precision, recall, and F1 score, respectively. SVM demonstrates its robustness in dealing with document data. On the other hand, the Naïve Bayes obtained the lowest results, although it approves its efficiency in dealing with document data as reported in the literature. This might be because of the feature combinations we derived, consisting of text and graph data.

Additionally, we conducted an ablation study to examine our method’s performance using different features categories, which highlight the feature set’s importance. We started by investigating our method performance using only the text-based features since the number of these features is much larger than the number of other feature categories. After that, we added the other feature categories to the text-based features to compare the performance of using only text-derived features to using all integrated features (text-based, graph-based, and metadata-based). The results (shown in

Table 5. Prediction performances of the XGBoost and RF classifiers using different sets of FV in terms of three evaluation metrics.

Features Category	Classifier	Precision	Recall	F1-Score
Text-based features	RF	79.7%	77.2%	78.43 %
	XGBoost	81.3%	78.7%	80.00%
Integrated features: text-based, graph-based, and metadata-based	RF	81.5%	78.8%	80.13%
	XGBoost	82.4%	80.3%	81.30%

) demonstrated that integrating graph- and metadata-based features with text-based features enhanced the performance and increased all evaluation metrics: precision, recall, and F1-score using the RF classifier. Despite the number of graph-based features being small, combining them with text-based features improved the performance for both RF and XGBoost classifiers in terms of all three evaluation metrics, which demonstrates the importance of these graph-based and metadata-based features.

Furthermore, we conducted the last evaluation process to demonstrate the practicality of the FutureCite model’s predictive power by analyzing and validating some results using the literature review. We selected some published research papers from our test data and analyzed the prediction results. For example, we picked a research paper titled “Simrank: a measure of structural-context similarity” [54], which is predicted to be a high-quality paper (i.e., the predicted class label is 1) by our model using the three classifiers, SVM, NB, and RF. We verified this result by examining the research paper details, such as the authors, venues of the paper, and the current citation count. Our examination shows that the current citation for this paper is 2700 and was published in SIG KDD, one of the most important conferences in the data mining and machine learning fields. Furthermore, the authors are popular, have a strong reputation, and have scholarly profiles with impressive citation records. All these factors confirm that this research paper is indeed significant, as predicted by our model.

Overall, the results indicate that the classification model, based on citation graph mining and machine learning, effectively classifies research papers into different classes based on their citation counts. This model can be utilized to identify papers with a high impact and potential significance, as well as papers that fall below average. Such insights can be valuable and useful for researchers and decision-makers across various fields interested in identifying and analyzing research papers with different levels of impact.

5. Conclusions, Remarks, and Recommendations

In this paper, we developed the FutureCite Model, which predicts the future citation of research papers to reveal their importance and significance. We formulated the problem as a multi-label classification task, where each paper was predicted to belong to one of four classes that we derived: ”Highly cited”, ”Well cited”, ”Above/ average cited”, and ”Below average cited”. Four classifiers were used for prediction; two demonstrated their robustness and efficiency in such a problem. The results highlight the effectiveness of our approach. We conducted a results analysis and validation to identify highly cited papers and show evidence for those papers to be predicted as significant. The FutureCite model leverages a combination of graph-based and text-based features, which capture both the structure and content of the data-mining publications dataset, improving the model’s performance in predicting the citation level and demonstrating its effectiveness. The outcomes of our work have implications for diverse stakeholders such as researchers, funding agencies, and academic institutions, as they can make informed decisions based on our models’ predictive capability.

Although our method proves its efficiency and robustness, it also faces some limitations and challenges that must be addressed to ensure its effectiveness and reliability in practice. One of the major limitations of this study is the size of the used dataset, which may not accurately represent the full range of research papers. It also can lead to another challenge of data availability, where our method relies on access to a large dataset of published research papers and their citation information. This information may not be available or may require considerable effort and resources to collect. Another limitation is the prediction performance, which can be greatly improved using massive data, especially the deep learning models we omit due to our dataset size.

Despite the above limitations, the FutureCite model contributes to the research paper citation prediction field as a powerful and useful application that demonstrates its effectiveness using graph and test mining and machine learning techniques. The approach can be applied to various fields, provide insights into the impact and significance of research papers, and aid researchers in their research and literature.

Future work can involve various areas of exploration and improvement. One aspect is to utilize larger datasets, as working with massive amounts of data can allow us to employ deep learning techniques for feature generation and prediction. Deep learning models usually perform better when trained on extensive datasets. Another aspect of future work is refining our model, FutureCite, and improving its performance. This can be achieved by exploring and incorporating additional features into the model. One potential approach to integrating more essential features is to leverage graph-embedding techniques such as node2vec, graph neural networks (GNNs), and others which can automatically generate crucial features based on the underlying graph structure and topology. Furthermore, incorporating recent text-embedding techniques such as bidirectional encoder representations from transformers (BERT) embedding or other large language models (LLMs) techniques can enhance the model’s ability to capture semantic information and relationships. Combining text and graph features from these embeddings can comprehensively represent the research papers.