CSRLoan: Cold Start Loan Recommendation with Semantic-Enhanced Neural Matrix Factorization

Abstract

Recommending loan products to applicants would benefit many financial businesses and individuals. Nevertheless, many loan products suffer from the cold start problem; i.e., there are no available historical data for training the recommendation model. Considering the delayed feedback and the complex semantic properties of loans, methods for general cold start recommendation cannot be directly used. Moreover, existing loan recommendation methods ignore the default risk, which should be evaluated along with the approval rate. To solve these challenges, we propose CSRLoan for cold start loan recommendation. CSRLoan employs pretraining techniques to learn the embeddings of statements, which captures the intrinsic semantic information of different loans. For recommendation, we design a dual neural matrix factorization (NMF) model, which can not only capture the semantic information of both loan products and applicants but also generate the recommendation results and default risk at the same time. Extensive experiments have been conducted on real-world datasets to evaluate the effectiveness and efficiency of the proposed CSRLoan.

1. Introduction

With rapidly increasing online lending requests, it is very difficult to choose the correct loan products with human decisions. Loan recommendation, which recommends loan products to applicants, has been considered as a critical task for many microfinance applications, such as Kiva, Lendio, etc. Empowered by machine learning techniques, a branch of recommendation methods [1,2,3,4,5] has been proposed to match appropriate users with loans, but these methods highly require sufficient historical data for training. Nevertheless, both the applicants and the products are updated frequently in loan recommendations. There is insufficient time for collecting sufficient data to train a user-specific recommendation model, which leads to a serious cold start problem. Thus, cold start loan recommendations, which recommends the correct loan projects to applicants without using substantial historical data, are designed to solve this problem and can be widely used in many circumstances.

To remedy the cold start problem, online loan platforms ask the applicants to provide detailed/short statements about the purpose of money. We argue that the semantic information contained in the statements is useful for loan recommendations. Even so, it is also challenging to build the cold start loan recommendation system due to the following reasons. (1) The complexity of semantic information involves statements comprising non-structured text data and can be in different languages, which make it difficult to understand their semantics. (2) The difficulty of delayed feedback, according to the report of Kiva.org (https://www.kivaushub.org/kivaprocess, accessed on 8 October 2022), constitutes feedback durations of one specified loan ranging usually from 6 months to 3 years, and the pay back period is also varied. It is feasible to obtain the feedback once we recommend and approve the loan. (3) The challenge with respect to mixture evaluations matrices is that the recommendation of the loan project should not only consider the conversion rate but also the default risk.

Despite the fact that a great deal of studies have been conducted for cold start recommendations, none of them can handle all three challenges. To solve the first challenge, many context-based recommendation methods [6,7] are proposed. These methods convert the statements to several categories to capture the semantics of statements. We argue that semantic information can be better modeled by semantic representations, yet these methods fail. For the second challenge, although many cold start recommendation models, such as JIM [8] and CTLM [9], are proposed for general recommendations, the problem’s definitions in their works are quite different from ours. However, most of these works are designed for instant feedback applications and cannot be used for loan recommendations. Apart from these methods, previous credit scoring methods [10,11,12,13] are also related to our problem but these methods cannot handle cold start problems. Moreover, for the third challenge, there is no existing work considering the mixture evaluation metrics.

With the emergence of natural language processing techniques, the semantic of both applicant statements and the loan project characters can be represented in a unified hyperspace, which enables us to solve the challenges systematically. Unfortunately, most of existing loan recommendation techniques [3,14,15] follow collaborate filtering frameworks, which cannot be easily adapted to textual data. Even worse, recently proposed methods [16,17] assume that the interactions of applicants are available and employ graph neural networks to model it. These methods are not suitable in our task because the applicant’s interactions violate the cold start setting.

In this paper, we propose a novel semantic-enhanced neural matrix factorization for cold start recommendations for loan, abbreviated as CSRLoan. Instead of modeling the user-specified preference, CSRLoan directly used the provided semantic information to build the recommender system. Loans that have similar statements in semantics would be recommended to the same applicants. In this way, both the applicants and the loans can be viewed as several latent groups. When a new loan comes, the model will first decide the group it belongs to and then recommends it to the right applicant.

CSRLoan contains three key modules, i.e., statement encoding, dual NMF, and mixture learning target. To capture the semantic information of statements, CSRLoan employs a pre-training algorithm to learn the initial representations of used word embeddings and to conduct a transformer network to encode each statement into a representation vector. Taking the statement’s vector as the input, CSRLoan designs a unbalanced NMF model to convert the applicants and loan project into a semantic space and recommends the loan project nearby the applicant in that space. To optimize the model parameters, we propose a mixture loss that balances the influence of conversion rate and default risk.

The main contributions of this paper can be summarized as follows:

• To the best of our knowledge, this is one of the pioneer works that model the semantics of statements with pre-training techniques and utilize them for loan recommendations. The intrinsic characters of statements are suitable for solving the cold start recommendation problem.
• We propose CSRLoan, a dual neural matrix factorization model for cold start loan recommendation. It first learns the representations of statements. Then, the loan projects and applicants are embedded in a semantic space for better recommendation.
• We conduct extensive experiments on a real-world dataset. The results show the superiority of CSRLoan compared with all baselines.

2. Related Work

In this section, we summarize the related works of CSRLoan. These works can be classified into three categories: loan recommendation, cold start recommendation, and pre-training for semantic modeling.

2.1. Loan Recommendation

Despite the emergence of social loans, few research works [3,14,15,17,18] have been developed to solve the loan recommendation problem. These methods can be roughly divided into two categories, i.e., recommend loan projects to applicants [3,16,18] and recommended applicants to loan projects [17].

For the first category, Zhao et al. [18] proposed a collaborate filtering approach to recommend optimal loan projects to applicants. Lee et al. [3] focused on the fairness of loan recommendation and utilized the matrix factorization framework to select the correct loan projects. Recently, Liu et al. [16] proposed a graph convolution network to recommend loan projects while considering the return rate of applicants. However, all these methods utilize either the loan–applicant interactions or applicant–applicant interactions to build the recommendation model. In our problem, such historical data are not available, which limits their usage.

For the second category, only one work [17] aimed to find potential lenders to loan projects. Zhang et al. employed random walk techniques on historical loan-applicant graph to obtain the embeddings of applicants. Then, a similar measure was adapted to rank potential lenders. Nevertheless, this work is not relevant to our task. We focus on recommending loan projects to applicants without using much historical data.

Moreover, credit-scoring methods [10,11,12,13], which assign the credit level for applicants, are also related to our problem. The traditional machine learning methods such as decision tree [13] and softmax regression [11] methods are employed to build a classifier to generate the credit levels. However, these methods ignore the semantic information of statements, which cannot be used for our problem. Recently, a transformer-based method [10] is proposed for credit scoring. Benefitting from the representation power of the transformer, the proposed method achieves the state-of-the-art performance on this task. Nevertheless, without considering the property of loans, it is not suitable for loan recommendation tasks.

2.2. Cold Start Recommendation

The cold start problem has been studied in the recommender system community for many years. The main issue of this problem is that there is insufficient information for making recommendations. According to the literature [19], existing research studies can be categorized into two groups, i.e., the explicit solutions and implicit solutions. Next, we present the related works.

For explicit solutions [20,21,22], the recommender system directly interacts with users or experts to collect the information. The user is either asked to fill some questionnaire or rate some products. To select the appropriate questions, active learning-based methods [20,22,23] are proposed to collect adequate information without overwhelming users. Apart from the active learning approaches, using an interview is another method for obtaining user preferences. An item set is employed to test the preferences of users. In each round, one item is selected to display and collect the response. There can be three types of responses: like, dislike, and unknown [21]. Although explicit solutions are efficient, users are often reluctant to participate in the query process, which limits their usage.

Another group of cold start recommendation methods includes implicit methods [6,24,25,26,27,28]. These methods try to understand new user preferences via minimum (or only once) interactions. Existing user information such as demographics or social relationships is employed to build the cold start recommender system. Based on the usage of external information, existing solutions in this group can be divided into user-demographic-based approaches [24] and social-relationship-based approaches [25]. Classification methods [26,27] are usually adopted to identify the character of users based on their demographics. Then, users having similar characteristics are usually recommended equally. To integrate the social relationship, many graph-based techniques [29,30] are proposed to capture the user’s similarity. However, finding useful information is challenging in these methods.

In loan recommendations, explicit solutions are not appropriate due to the high volume and diversity of applicants. The proposed method CSRLoan falls in the implicit group. To better model the semantic information, CSRLoan employs pre-training techniques and captures the influence of both default risk and conversion rate.

2.3. Pre-Training for Semantic Modeling

Pre-training is one of the new fronts in the field of matural language processing and can be used as a powerful tool for semantic modeling. The core philosophy of pre-training techniques is transfer learning or domain adaptation. Before the implementation of down-stream tasks, the self-supervised learning procedure is adopted to initialize the representations of some shared information, e.g., word embeddings or model parameters. One of the pioneer work in this field is the GloVe model [31]. It can be thought of as a mix of the count-based matrix factorization [32] and the context-based skip-gram [33] model together. This method learns the semantic of words by reconstructing the context surrounding them. GPT [34] is another milestone of pre-training techniques. It consists of two distinctive stages, unsupervised pre-training and supervised finetuning. The unsupervised loss that borrows the idea from the language models is designed to learn the initial parameters, which can be generalized to many other tasks via supervised finetuning. Recently, BERT [35] is the state-of-the-art pre-training method that shares the same idea of GPT. The main difference is the bidirectional encoder constructed by transformer [36]. Although many pre-training techniques are proposed, none of them are specifically designed for loan statements. In this paper, we propose an activity loss along with the mask language model to generate representations for loan recommendation.

3. Preliminaries

In this section, we first summarize the notations and define the loan recommendation problem. Based on the notations, we overview the proposed method CSRLoan. The notations used in this paper as listed in

Table 1. Notations used in this paper.

Notations	Descriptions
$u,v,s$	Applicant user, loan project, and the statement
$\mathcal{U},\mathcal{V}$	User set and loan project set
$r=(u,v,s)$	A historical record indicates u apply v with s
$T_u$	Historical loan records of user u
$\mathcal{D}$	Historical data of all loan records
$\mathbf{M}$	Word embedding matrix
d	Embedding dimension
$\mathsf{\Theta }$	All parameters in CSRLoan

.

3.1. Problem Formulation

A loan project in lending platform can be a company project, such as Webank small loan, or a personal lender. In most cases, the available information of a loan project is limited. Generally, we define the loan project as a tuple of id and k attributes, i.e., $v=(id,\mathbf{a})$, where $\mathbf{a}=\{a_1,\dots ,a_k\}$ contains the display name, loan purchase number, etc.. Specifically, a historical loan record can be organized into a triplet $r=(u,v,s)$, which indicates applicant u applying loan project v with proposal statement s. For a specific user, its historical loans form a sequence, i.e., $T_u=[r_1,r_2,\dots ,r_K]$. Due to the character of loans, most applicants are fresh users and have no historical loans.

Given the historical dataset of N loan records $\mathcal{D}=\left\{r_p\right|p\in 1,2,\dots ,N\}$ and a loan application of new user $u^n$ with statement $s^n$, the cold start loan recommendation problem aims to recommend loan projects that satisfy the applicant’s needs. Formally, the cold start loan recommendation problem can be formally defined as follows.

Cold Start Loan Recommendation: The goal of cold start loan recommendation is to predict the most likely loan project $\widehat{v}$ that the request of new user $u^n$ will be satisfied while minimizing the default risk by utilizing all information in $\mathcal{D}$:

$$\widehat{v}=arg\underset{v}{max}\mathbb{P}(v\in \mathcal{V};u^n,{\mathbf{s}}^n,\mathcal{D},\mathsf{\Theta })$$

(1)

where $\mathbb{P}$ is a measurement of the recommendation performance, $\mathcal{V}$ is the set of loan projects, and $\mathsf{\Theta }$ denotes all parameters in the recommendation model.

3.2. Overview of CSRLoan

CSRLoan is a semantic-enhanced model. It utilizes historical loan records and models the semantic information of applicant statements to achieve cold start recommendations. The overall architecture is illustrated in Figure 1. To model the semantic information of statements, CSRLoan employs a transformer-based neural network to pre-train the statement’s embedding model. Taking the statement’s embeddings as the input, CSRLoan employs a dual neural matrix factorization model to capture the conversation and default risk jointly. Moreover, a mixture loss is defined to optimize the entire model.

4. Methodology

CSRLoan consists of three key modules, i.e., statement encoding, dual NMF, and mixture learning target. Next, we specify the details of these modules.

4.1. Statement Encoding

The statements of user applications play a key role in solving cold start recommendation problems. The semantics in these statements reflect the user demographic and could be used to find similar users. In this section, we present a statement-encoding module to convert these statement texts into semantic representations.

A statement can be represented as a sequence of words. Given a statement s of user u, i.e., $s=[w_1,w_2,\dots ,w_K]$, CSRLoan first embeds each word into a vector with an embedding matrix, $\mathbf{M}$:

$$\begin{array}{cc}\hfill [{\mathbf{e}}_1,{\mathbf{e}}_2,\dots ,{\mathbf{e}}_K]& =Embedding(\mathbf{M},s)\hfill \end{array}$$

(2)

where $\mathbf{M}\in {\mathbb{R}}^{M\times d}$ is the word embedding’s matrix. To model the sequential information of s, we employ the sinusoidal values proposed by Vaswani et al. [36] for encoding the position’s information. This approach is an absolute position encoding called sinusoidal position embeddings. We construct sequential position embeddings as follows:

$${\lambda }_{kj}=\{\begin{array}{cc}sin({\mathrm{10,000}}^{\frac{j}{k}}),\hfill & \hfill \mathrm{if}k\mathrm{is}\mathrm{even}\\ cos({\mathrm{10,000}}^{\frac{j-1}{k}}k),\hfill & \hfill \mathrm{if}k\mathrm{is}\mathrm{odd}\end{array}$$

(3)

where $k=[1,\dots ,K]$ and $j=[1,\dots ,d]$. For the sequential position encoding of ${\mathbf{e}}_k$, we denote it as ${\lambda }_k\in {\mathbb{R}}^d$. The input vector can be represented as follows.

$$\begin{array}{cc}\hfill {\mathbf{i}}_k& ={\mathbf{e}}_k+{\lambda }_k\hfill \end{array}$$

(4)

After the generation of input vectors, we feed them into a transformer network to obtain semantic embeddings:

$$\begin{array}{c}\hfill {{\mathbf{h}}^{\prime }}_i^{\ell +1}={\mathbf{O}}_h^{\ell }{\mathbf{concat}}_{k=1}^H\left(\sum _{j=1:K}{\mathbf{w}}_{ij}^{k,\ell }{\mathbf{V}}^{k,\ell }{\mathbf{h}}_j^{\ell }\right),\\ \hfill \mathbf{where},{\mathbf{w}}_{i,j}^{k,\ell }={\mathbf{softmax}}_j\left(\frac{{\mathbf{Q}}^{k,\ell }{\mathbf{h}}_i^{\ell }·{\mathbf{K}}^{k,\ell }{\mathbf{h}}_j^{\ell }}{\sqrt{d_k}}\right),\end{array}$$

(5)

where ${\mathbf{Q}}^{k,\ell },{\mathbf{K}}^{k,\ell },{\mathbf{V}}^{k,\ell }\in {\mathbb{R}}^{K\times d},{\mathbf{O}}_h^{\ell }\in {\mathbb{R}}^{d\times d},k=1$ to H all indicate the number of attention heads. For numerical stability, the outputs after taking exponents of the terms inside $softmax$ are limited to the range of $(-5,5)$. Then, the attention output ${{\mathbf{h}}^{\prime }}_i^{\ell +1}$ are fed into a feed-forward network (FFN), which contains the residual connections and batch normalization modules. For conciseness, we omit the equations of identical operations with transformer.

After the P layers of transformer, we obtain the hidden representations of all words, i.e., $[{\mathbf{h}}_1^P,\dots ,{\mathbf{h}}_K^P]$. To generate the final embedding of s, we employ the mean readout function on the node’s embedding. The embedding, ${\mathbf{e}}_s$, of trajectory s is computed as ${\mathbf{e}}_s={\sum }_{i=1}^K{\mathbf{h}}_i^P/K$. For conciseness, we represent the above-mentioned operations as follows:

$$\begin{array}{cc}\hfill {\mathbf{e}}_s& =Encode{r}_{trans}\left(s\right)\hfill \end{array}$$

(6)

Instead of initializing the model parameters randomly, we design a pre-training strategy to obtain the initial model’s parameters. Traditional pre-training utilizes an encoder-decoder framework. It masks 20% percentage of tokens and ties to predict the masked tokens in the decoded sequence. Similarly to the encoder, we can also build a decoder to reconstruct s:

$$\begin{array}{cc}\hfill [{\widehat{w}}_1,{\widehat{w}}_2,\dots ,{\widehat{w}}_K]& =Decode{r}_{trans}\left([{\mathbf{h}}_1^P,\dots ,{\mathbf{h}}_K^P]\right)\hfill \\ \hfill {\mathcal{L}}_{mask}& =\sum _{i\in MP}CrossEntropy(w_i,{\widehat{w}}_i)\hfill \end{array}$$

(7)

where $MP$ is the masked position. For loan applications, there is an “activity” label y along with the statement indicating whether the loan is defaulted or not. It consists of three states, i.e., default, active, and completed. Thus, we propose a new pre-training task, which predicts the activity label based on the representation.

$$\begin{array}{cc}\hfill {\mathbf{f}}_s& =MLP\left({\mathbf{e}}_s\right)\hfill \\ \hfill {\mathbf{p}}_s& =Softmax\left({\mathbf{f}}_s\right)\hfill \\ \hfill {\mathcal{L}}_{activity}& =CrossEntropy({\mathbf{p}}_s,y)\hfill \end{array}$$

(8)

The final loss of pre-training can be represented as follows.

$$\begin{array}{cc}\hfill {\mathcal{L}}_{pre-train}& =\sum _{i=1:N}{\mathcal{L}}_{mask}^i+{\mathcal{L}}_{activity}^i\hfill \end{array}$$

(9)

After the pre-training, we employ the encoder to obtain the representation of the statement.

4.2. Dual Neural Matrix Factorization

In this section, we introduce the dual neural matrix factorization module, which not only fits the historical loan records but also estimates the default risk of loan. The dual neural matrix factorization module consists of three parts, i.e., feature encoding, loan recommendation, and default risk estimation.

Feature encoding: In this part, we convert the features of applicants and loan projects into vectors. Note that the loan project $v=(id,\mathbf{a})$ and applicant $u=(uid,{\mathbf{a}}^{\prime })$. ${\mathbf{a}}^{\prime }$ represents the features of applicants, including the gender, age, and income grades. These features can be classified into two groups, i.e., value features ${\mathbf{a}}_f$ and category features ${\mathbf{a}}_c$. For value features, we first normalize them into $(0,1)$ and feed them into the MLP layer to obtain the input. For category features, we employ embedding techniques to convert them into vectors:

$$\begin{array}{cc}\hfill {\mathbf{f}}_f& =MLP(Normalize\left([a_1,a_2,\dots ,a_{k_f}]\right)\hfill \\ \hfill {\mathbf{f}}_c& =MLP\left(concat\left(Embedding\left([a_1,a_2,\dots ,a_{k_c}]\right)\right)\right)\hfill \end{array}$$

(10)

where $k_v$ and $k_c$ are the number of features. We assume that the dimensions of ${\mathbf{f}}_f$ and ${\mathbf{f}}_c$ are equal to d. For u and v, we use the same method to generate their value features and category features. Then, we concatenate them and feed them to a $MLP$ to obtain their features.

$$\begin{array}{cc}\hfill {\mathbf{f}}^u& =MLP(concat({\mathbf{f}}_f^u,{\mathbf{f}}_c^u))\hfill \\ \hfill {\mathbf{f}}^v& =MLP(concat({\mathbf{f}}_f^v,{\mathbf{f}}_c^v))\hfill \end{array}$$

(11)

For the applicant, we also involve the statement embedding to the features and obtain the final representation of the applicant.

$$\begin{array}{cc}\hfill {\mathbf{f}}^u& =MLP\left(concat(\mathbf{s},{\mathbf{f}}^u)\right)\hfill \end{array}$$

(12)

Loan recommendation. Taking the representations of u and v as inputs, CSRLoan employs a neural matrix factorization method to fit historical loan records. In this part, we first construct an applicant–loan matrix $\mathbf{R}$. Each element ${\mathbf{R}}_{i,j}$ in the matrix indicates the number of user $u_i$ that successfully completed loan project $v_j$. Then, we fit these numbers with the representation vectors.

$$\begin{array}{cc}\hfill o_{i,j}^r& =ML{P}_r\left(concat({\mathbf{f}}^u,{\mathbf{f}}^v)\right)\hfill \end{array}$$

(13)

Default Risk Estimation. Similarly to loan recommendations, CSRLoan employs another neural matrix factorization model to fit the historical default loans. We also construct an applicant-loan matrix $\mathbf{D}$. Each element ${\mathbf{D}}_{i,j}$ in the matrix indicates the number of user $u_i$ defaulting the loan project $v_j$.

$$\begin{array}{cc}\hfill o_{i,j}^d& =ML{P}_d\left(concat({\mathbf{f}}^u,{\mathbf{f}}^v)\right)\hfill \end{array}$$

(14)

4.3. Mixture Learning Target

As the recommended loan projects should not only be attractive for applicants but also have low default risk, we designed a mixture loss $\mathcal{L}$ to optimize the parameters in CSRLoan. Formally, $\mathcal{L}$ consists of two parts. The first one is the recommendation loss ${\mathcal{L}}_{rec}$, which guides the model to recommend attractive loan projects.

$$\begin{array}{cc}\hfill {\mathcal{L}}_{rec}& =\sum _{i,j\in {\mathbf{S}}_s}MSE(o_{i,j}^r,{\mathbf{R}}_{i,j})\hfill \end{array}$$

(15)

The second one is the default risk loss ${\mathcal{L}}_{risk}$, which guides the model to predict the default risk.

$$\begin{array}{cc}\hfill {\mathcal{L}}_{risk}& =\sum _{i,j\in {\mathbf{S}}_d}MSE(o_{i,j}^d,-{\mathbf{D}}_{i,j})\hfill \end{array}$$

(16)

Finally, the total loss can be formulated as follows.

$$\begin{array}{cc}\hfill \mathcal{L}& ={\mathcal{L}}_{rec}+{\mathcal{L}}_{risk}\hfill \end{array}$$

(17)

All the parameters in CSRLoan can be updated in an end-to-end manner. We update the parameters with backpropagation algorithms and employ an Adam optimizer for optimization. When the dual NMF model is well-trained, we use a combination of recommendation results and default risks to generate the recommended loan projects. Once a new application $<u,s>$ arrived, CSRLoan calculates the outputs ${\mathbf{o}}_u^r$ and ${\mathbf{o}}_u^d$ of all loan projects. Then, we add them together and select the loan projects with higher scores as the recommended loan projects:

$$\begin{array}{cc}\hfill {\mathbf{score}}_u& ={\mathbf{o}}_u^r+\alpha ·{\mathbf{o}}_u^d\hfill \end{array}$$

(18)

where $\alpha$ is a hyperparameter for balancing the contribution of loan recommendations and default predictions.

5. Experiment

In this section, we conduct extensive experiments to demonstrate the effectiveness of CSRLoan. Our experimental evaluation is designed to answer several research questions (RQs).

• RQ1: Does CSRLoan outperform other general recommendation methods and loan recommendation methods?
• RQ2: What is the capability of the proposed pre-training techniques and dual NMF?
• RQ3: What are the influences of different hyper-parameter settings?

Next, we introduce the experimental settings, experimental results, ablations studies, and hyper-parameter studies.

5.1. Experimental Settings

We first introduce the datasets, compared baselines, evaluation metrics, and parameter settings of our experiments. Then, we evaluate CSRLoan against other state-of-the-art algorithms.

5.1.1. Datasets

In our experiments, we use a open-source crowdfunding dataset, Kiva, to evaluate the performance of CSRLoan. Kiva is an online crowdfunding platform for extending financial services to poor and financially excluded people around the world. Kiva lenders provided over USD 1 billion in loans to over two million people. In order to set investment priorities, help inform lenders, and understand their target communities, knowing the level of poverty of each applicant is critical. This dataset includes the loan data in the year 2014 containing 1,419,607 loans and 2,349,174 registered lenders across more than 100 countries. We overview the statistics of the top 10 countries in Figure 2. As shown, Philippines has the highest loan records, and we chose the Phillippines to evaluate the performance of CSRLoan.

In our experiments, we filter the users with at least one approved loan to evaluate the performance of CSRLoan. After this procedure, we obtain about 850,000 users with over one million loans. For cold start setting, we split the dataset by users. As shown in

Table 2. Statistical information of the dataset.

Dataset	# of Samples	# of Users	# of Default Samples	Average Length of s
Train	634,144	501,813	50,900	13.7
Validation	53,703	48,083	3833	11.6
Test	388,405	302,401	6210	12.8

, we provided the statistics of the used dataset. The pre-training and finetuning are conducted on the training set. In the finetuning procedure, we chose the best model with the performance on validation set. Finally, the performances on the test set are reported in this paper.

For better understanding CSRLoan, we provided some examples of user applications to showcase the used features. As illustrated in

Table 3. Examples of the user applications.

ID	Amount	Categories	Statement	Country	Region	Ages	Income	Gender	Repayment	Activity Label
1	575	Transportation	to repair and maintain the auto rickshaw used in their business	Pakistan	Lahore	30	14,000	female	irregular	default
2	150	Transportation	to repair their old cycle-van and buy another one to rent out	India	Maynaguri	22	6000	female	bullet	completed
3	200	Arts	to purchase an embroidery machine and new materials	Pakistan	Lahore	25	8000	female	irregular	completed
4	250	Services	purchase leather for my business using ksh 20000	Kenya		23	6000	female	irregular	completed
5	200	Agriculture	to purchase a dairy cow and start a milk products business	India	Maynaguri	25	8000	male	bullet	overdue
6	400	Services	to buy more hair and skin care products	Pakistan	Ellahabad	30	8000	female	monthly	completed
7	475	Manufacturing	to purchase leather, plastic soles and heels in different sizes	Pakistan	Lahore	46	19,000	female	monthly	completed
8	625	Food	to buy a stall, gram flour, ketchup, and coal for selling ladoo	Pakistan	Lahore	35	24,000	male	irregular	default

, we can find that the numerical features include ages, income, and loan amount and the category features include country and gender repayment. We can also observe that the statement of a loan is a short description of the usage. Moreover, examples of loan projects are also provided in

Table 4. Examples of the loan projects.

ID	Loan Theme	Require Partner	Duration	Amount
638631	General	Yes	2 years	8000
640322	General	Yes	0.5 years	12,000
641006	Higher Education	Yes	3 years	40,000
641019	Higher Education	No	2 years	2000
641594	Subsistence Agriculture	Yes	2 years	10,000
642256	Extreme Poverty	Yes	1 years	20,000

. We can find that the features of loan projects also contain the two types of features.

5.1.2. Evaluation Metrics

We use two different metrics for performance evaluation, Hit Ration (HR@K) and Normalized Discounted Cumulative Gain (NDCG@K). HR@K measures whether the loan shows within the top K in the ranked list while the NDCG@K takes the position of the loan into account and penalizes the score if it is ranked lower in the list.

5.1.3. Compared Baselines

In this study, we compare CSRLoan with seven representative baselines to evaluate the performance. Because of the particular data, i.e., statement text, used in our problem, we only chose recommendation methods that can be easily extended to the text information as the baselines. Notice that many graph neural-network-based methods [16,17] are proposed recently, and we do not compare CSRLoan with them due to the cold start setting. We roughly divide compared baselines into three groups: general recommendation methods, loan recommendation methods, and ablations of CSRLoan.

General recommendation methods.

• MF-BPR [37]: A Bayesian personalized ranking optimized MF model with a pairwise ranking loss. It is tailored toward recommendations with implicit feedback data.
• CML [2]: A recently proposed algorithm that minimizes the distance between each user–loan interactions in Euclidean space.

Loan recommendation methods.

• MBR [14]: A motivation-based recommendation method to utilize the unstructured data. This is the state-of-the-art method for loan recommendations.
• FAR [3]: A fairness-aware recommendation method based on one-class collaborative-filtering techniques.

Ablations of CSRLoan.

• NMF: Traditional NMF that removes the pre-training and default risk estimation module.
• CSRLoan ${}_{pr{e}^{-}}$: Directly train CSRLoan with randomly initialized statement encoder.
• CSRLoan ${}_{ris{k}^{-}}$: A variant of CSRLoan which removes the default risk estimation module.

Note that some of these compared methods are not for loan recommendations, and we utilize historical data to train these models and use the same testing data to evaluate the performances.

5.1.4. Parameter Settings

For CSRLoan, we set the dimensions of all embedding vectors as 32. In the recommendation process, the trade-off hyper-parameter $\alpha$ is set as $0.5$. In CSRLoan, all MLPs are three-layer fully connected neural networks. For the transformer module, we use two-layer transformer to moderate the number of parameters. The attention head H is empirically set to 7. To train the parameters, we randomly mask 20% positions of the input statement and predict the masked words in the decoder. Moreover, all models are trained by Adam. The training batch size is set as 128. The maximum epoch is set as 50 for both pre-training and finetuning operations.

5.2. Experimental Results

The comparison experimental results of CSRLoan are shown in

Table 5. Performance comparison of different methods in terms of HR@5, NDCG@5, HR@10, and NDCG@10. The bold values indicate the best performances.

Type	Method	HR@5	NDCG@5	HR@10	NDCG@10
General	MF-BPR	0.02134	0.01841	0.05263	0.03991
	CML	0.04630	0.03877	0.1518	0.1247
Loan Rec	FAR	0.04261	0.02974	0.0993	0.04971
	MBR	0.05216	0.03234	0.1377	0.0860
Ablations	NMF	0.0841	0.0710	0.2432	0.1221
	CSRLoan ${}_{pr{e}^{-}}$	0.0922	0.07082	0.2310	0.1778
	CSRLoan ${}_{ris{k}^{-}}$	0.1021	0.08113	0.2530	0.1809
Our	CSRLoan	0.1150	0.0970	0.2233	0.1928

. For a better understanding, we also illustrate the performance changes in CSRLoan and other selected baselines in Figure 3 and Figure 4 with the increase in the recommended number of loan projects. Next, we analyze the results and answer the research questions: RQ1.

As shown in Table 5, the performance of CSRLoan beats all compared baselines on all evaluation metrics consistently. For example, compared with the most competitive method, i.e., NMF, CSRLoan achieves about 30% relative improvement on HR@5 (from 0.0841 to 0.1150). This result shows the superiority of the proposed modules and answers the RQ1. Among all the recommendation methods, MBR is the strongest baseline. Although it models the semantic information of unstructured data, it is also inferior to CSRLoan. The reason is that MBR only captures the category motivation information of statements and cannot borrow knowledge from the statement’s semantics. For different groups of baselines, the loan recommendation methods outperform the general ones, which shows the special characteristics of loan data.

As illustrated in Figure 3 and Figure 4, the performances of CSRLoan continuously outperform the compared baselines. When the return number is $N=5$, the performance gaps between different methods are relatively small. With the increase in N, the performances of all methods first increase and then become stable. From Figure 4, we can observe that CSRLoan significantly outperforms the most competitive baseline, i.e., MBR.

5.3. Ablation Studies

To answer the RQ2, we compared the performance of CSRLoan with its ablations.

Contribution of pre-training strategy. Comparing CSRLoan with CSRLoan ${}_{pr{e}^{-}}$, the average relative improvement is above 10%. This is because our model can capture the semantic information of statements with the pre-training procedure.

Contribution of default risk estimation. Comparing CSRLoan with CSRLoan ${}_{ris{k}^{-}}$, CSRLoan achieves better performances. The results indicate that the bad loan projects having high default risks are filtered with the prediction model filter, which results in higher performances than other methods.

5.4. Hyper-Parameter Studies

To answer RQ3, in this subsection, we evaluate the influences of the hyper-parameter’s settings. Specifically, we analyze the impacts of two key parameters of CSRLoan, i.e., the dimension d of memory representation and the trade-off factor $\alpha$ of the two losses in the joint objective.

Influence of embedding dimension d. For dimension d, we change d from 2 to 128 in CSRLoan. The performance of loan recommendation is shown in Figure 5. With the increase in d, we find that the performance first increases and then decreases. One potential reason is that the memory provides too little information when the d is too small, while it can include more irrelevant information when it is too large.

Influence of trade-off hyperparameter $\alpha$. For $\alpha$, we search $\alpha$ from $0.1$ to $0.9$. The results are shown in Figure 6. The changing pattern is quite similar to d, which indicates that both the pre-training strategy and default risk prediction modules are critical for the task.

6. Conclusions

In this paper, we presented a novel semantic-enhanced dual NMF-based model, namely CSRLoan, for the cold start loan recommendation problem. Different from existing loan recommendation methods, CSRLoan leverages the knowledge from applicant’s statements and default records to enhance the recommendation performance of new user applications. Specifically, we designed two key procedures in CSRLoan to achieve cold start recommendations. Firstly, a transformer module was utilized to model the semantic information of statements. Along with the transformer, we also proposed pre-training techniques to initialize the parameters of the statement encoder. Secondly, we proposed a dual NMF model to capture the information on both success loans and default risk. A mixture loss was designed to optimize the parameters of CSRLoan. In this manner, we can recommend loans for new applicants based on the provided statements and personal information.

Moreover, we conducted extensive experiments to show the superiority of CSRLoan and verified the effectiveness of the newly proposed techniques. Compared with the baselines, CSRLoan significantly outperforms existing loan recommendation methods under different evaluation metrics. According to the ablation study, we found that the proposed statements encoder and dual NMF model can enhance the recommendation performance substantially. In addition, the hyperparameters in CSRLoan were tested. The results indicate that CSRLoan is not sensitive to these hyperparameters. It can achieve good performance in a broad range of parameter settings.