Blin: A Multi-Task Sequence Recommendation Based on Bidirectional KL-Divergence and Linear Attention

Abstract

Sequence recommendation is a prominent research area within recommender systems, focused on predicting items that users may be interested in by modeling their historical interaction sequences. However, due to data sparsity, user interaction sequences in sequence recommendation are typically short. A common approach to address this issue is filling sequences with zero values, significantly reducing the effective utilization of input space. Furthermore, traditional sequence recommendation methods based on self-attention mechanisms exhibit quadratic complexity with respect to sequence length. These issues affect the performance of recommendation algorithms. To tackle these challenges, we propose a multi-task sequence recommendation model, Blin, which integrates bidirectional KL divergence and linear attention. Blin abandons the conventional zero-padding strategy, opting instead for random repeat padding to enhance sequence data. Additionally, bidirectional KL divergence loss is introduced as an auxiliary task to regularize the probability distributions obtained from different sequence representations. To improve the computational efficiency compared to traditional attention mechanisms, a linear attention mechanism is employed during sequence encoding, significantly reducing the computational complexity while preserving the learning capacity of traditional attention. Experimental results on multiple public datasets demonstrate the effectiveness of the proposed model.

1. Introduction

Finding content that interests users quickly and accurately amid vast amounts of information has always been a challenging problem. To address this issue, recommender systems have become indispensable tools for online users to filter their preferred information based on their historical behavior interactions. Sequential recommendation [1], which dynamically models user preferences over time, has proven effective and has garnered widespread research attention.

In recent years, with the rapid advancement of deep learning, a range of sequential recommendation methods have emerged, encompassing architectures such as recurrent neural networks (RNNs), convolutional neural networks (CNNs), graph neural networks (GNNs), and Transformers [2,3,4,5,6,7,8]. Among these, Transformer-based methods for sequential recommendation have gained significant attention due to their exceptional performance in sequence encoding [5,6,7,8]. However, current research still faces several challenges: (1) Sequence padding space utilization issues and data sparsity problems: most sequence recommendation methods handle the padding of original short sequences with zeros [9], which do not contribute meaningful information and, thus, result in inefficient space utilization. Additionally, in the field of sequence recommendation, data sparsity is a prevalent issue. User interaction sequences are typically short and sparse, making it challenging for models to accurately capture user preferences and behaviors due to the limited data available. This sparsity further exacerbates the inefficiency of traditional zero-padding methods, as the padding does not add valuable information and fails to address the underlying lack of data. (2) The computational complexity of attention mechanisms: when dealing with long sequences, the computational complexity of dot product operations in attention layers increases quadratically with sequence length. (3) The limitation of a single loss function: most models rely solely on a single loss function to update their model parameters, which poses certain limitations to learning user preference representations from interaction sequences.

In response to these issues, this paper proposes a multi-task sequential recommendation model named Blin, which integrates bidirectional KL divergence and linear attention. Blin departs from the traditional zero-padding of original short sequences by employing a method called random repeat padding (RandomPad for short) to enhance the original sequence data. This involves independently applying RandomPad twice to generate two padded sequences, which are jointly input into the model. Additionally, a linear attention mechanism is utilized for the padded sequences in the model to significantly reduce the computational complexity of dot product operations in the attention layers, while preserving the learning capability of traditional attention mechanisms. Finally, bidirectional KL divergence loss is added as an auxiliary task for regularization [10,11], aiming to align or make similar the probability distributions derived from the representations of different padded sequences. This assists the model in learning the desired target distributions.

The main contributions of this paper are as follows:

• Blin adopts the RandomPad method to replace traditional zero-padding, thereby alleviating data sparsity issues and improving the effective utilization of input space.
• Blin employs a linear attention mechanism that reduces the computational complexity of the attention dot product operations from quadratic O(n²) to near-linear O(n), while ensuring the accuracy of attention mechanisms for long sequences.
• Blin introduces bidirectional KL divergence loss as an auxiliary task for sequential recommendation, aiming to regularize the probability distributions derived from different representations of padded sequences. This loss, combined with the sequence recommendation loss, jointly updates the model parameters to achieve enhanced user preference representations.
• Experimental evaluations on multiple public datasets, compared with representative baseline methods, demonstrate that Blin achieves performance improvements over baseline methods to varying degrees.

2. Related Work

2.1. Sequence-Based Recommendation

Sequence recommendation tasks involve predicting users’ next behavioral preferences based on their historical interaction behaviors. Early research, including pioneering methods based on Markov chains [12], initially explored simple, low-order sequential dependencies. However, these approaches were constrained by assumptions, making them inadequate for handling complex situations involving high-order dependencies. With the rapid development of deep learning, a variety of sequence recommendation methods have emerged. Among these, methods based on self-attention mechanisms stand out for their excellent performance and have received extensive research attention. Kang et al. [5] were pioneers in integrating self-attention mechanisms into sequence recommendation, effectively capturing user preference representations and achieving an improved performance over previous methods. FDSA [13] introduces multiple attention blocks to depict potential features. Sun et al. [6] proposed BERT4Rec, which utilizes bidirectional attention mechanisms to model user interaction sequences and employs the Cloze objective to improve the efficiency of the training process. Li et al. [14] introduced TiSASRec, incorporating time interval information into attention mechanisms to capture the impacts of different time intervals on item prediction. Furthermore, an increasing number of studies are combining contrastive learning with attention mechanisms [15,16,17,18]. For instance, CL4Srec [15] utilizes a contrastive learning framework to obtain self-supervised signals from original user sequences and proposes three data augmentation methods to construct these signals. CoSeRec [16] builds upon CL4SRec by integrating item similarity information into contrastive learning objectives to maximize sequence consistency enhancement.

2.2. Linear Attention Mechanisms

Traditional attention mechanisms incur high computational costs, posing challenges in model design and practical applications. To address the high computational costs associated with Transformers, recent studies have adopted linear attention mechanisms as a solution. Specifically, linear attention replaces the Softmax function in traditional dot product attention with a designed kernel function. In this approach, attention can be computed more efficiently by first calculating ${\mathbf{K}}^{\mathrm{T}}\mathbf{V}$. While linear attention is highly efficient in terms of computational complexity, designing a linear attention module with the same learning capability as Softmax attention remains a challenge. This issue has led to a series of emerging studies. The Performer [19] approximates Softmax operations using orthogonal random features, while Li et al. [20] proposed using a first-order Taylor expansion to approximate the expanded form of the Softmax function. K et al. [21] utilized simple feature mapping represented by the elu function as a kernel function. Shen et al. [22] introduced Efficient Attention, applying Softmax separately to Q and K to ensure that each row of $\mathbf{Q}{\mathbf{K}}^{\mathrm{T}}$ summed to 1. Hydra Attention [23] selects cosine similarity as a kernel function, reducing the computational complexity to O(nd).

However, the methods mentioned above still face several challenges. Sequential recommendation models typically process input sequences into a fixed length for ease of model training. Most sequence recommendation models handle user sequences by padding with zeros [9], where these padded zeros do not contribute to the model training, leading to inefficient utilization of the input space and difficulty in learning long-term user behavior representations. Furthermore, models based on self-attention mechanisms suffer from high computational costs [20,21,22,23,24,25]. Although current research on linear attention mechanisms has significantly reduced computational complexity, these methods often sacrifice accuracy. In other words, existing research has not effectively preserved the learning capability of traditional attention mechanisms.

The Blin model proposed in this paper addresses these challenges by employing RandomPad to effectively increase the utilization of input space for model input sequences. Additionally, improvements are made to the linear attention mechanism by incorporating a DWC module [24,25], which enhances the ability to capture local features while maintaining effective attention. Moreover, Blin introduces bidirectional KL divergence loss as an auxiliary task to regularize the probability distributions derived from different representations of padded sequences, aiding the model in learning more aligned target distributions.

3. Methodology

In this section, we first introduce the problem definition of sequential recommendation. Then, we provide a detailed description of each component of the Blin model, including RandomPad, the linear attention mechanism, and the bidirectional KL divergence. The overall framework of the model is illustrated in Figure 1.

3.1. Problem Statement

In the task of sequence recommendation, let $\mathbf{U}$ denote the set of users and $\mathbf{V}$ denote the set of items. For each user $\mathrm{u}\in \mathbf{U}$, there exists a time-ordered user interaction sequence ${\mathbf{S}}_{\mathrm{u}}=[{\mathrm{s}}_1^{\mathrm{u}},{\mathrm{s}}_2^{\mathrm{u}},\dots ,{\mathrm{s}}_{\mathrm{t}}^{\mathrm{u}},\dots ,{\mathrm{s}}_{\left|{\mathrm{S}}^{\mathrm{u}}\right|}^{\mathrm{u}}]$, where $\left|{\mathrm{S}}^{\mathrm{u}}\right|$ denotes the number of interactions in sequence ${\mathbf{S}}_{\mathrm{u}}$ and ${\mathrm{s}}_{\mathrm{t}}^{\mathrm{u}}$ represents the item interacted by user $\mathrm{u}$ at time $\mathrm{t}$. The task of sequence recommendation is to predict a Top-N candidate list of items that a user may find interesting at the next time step, based on their historical interaction sequence ${\mathbf{S}}_{\mathrm{u}}$. given by:

$$\begin{array}{cc}\arg max& \mathrm{P}\left(v_{\left|{\mathbf{S}}^{\mathrm{u}}\right|+1}=v^{*}\left|{\mathbf{S}}_u\right.\right)\\ v^{*}\in \mathbf{V}& \end{array}$$

(1)

This equation can be understood as calculating the probability for all candidate items and selecting the highest one for recommendation.

3.2. RandomPad

Most common sequence models can only handle fixed-length sequences. Additionally, due to the issue of data sparsity, the user sequences in sequential recommendations are often short. Therefore, sequence padding is a frequently used technique in training sequence models [5]. Before model training, the maximum sequence length n that the model can handle is defined. For sequences with an initial length greater than n, the most recent n items are selected as the input sequence. If the initial sequence length is less than n, padding is required on the left side of the sequence.

The general operation of sequence padding can be described as follows:

$$SeqOper\left({\mathit{S}}_u\right)=SeqOper\left(\left[s_1,s_2,s_3,\dots ,s_{\left|{\mathit{S}}^u\right|}\right]\right)=\left\{\begin{array}{cc}T\left({\mathit{S}}_u\right),& \left|S^u\right|<n,\\ {\mathit{S}}_u,& \left|S^u\right|=n,\\ \left[s_{c+1},s_{c+2},s_{c+3},\dots ,s_n\right],& \left|S^u\right|>n,\end{array}\right.$$

(2)

where $\left|{\mathbf{S}}^{\mathrm{u}}\right|$ represents the initial sequence length, $\mathrm{SeqOper}\left({\mathbf{S}}_{\mathrm{u}}\right)$ indicates the preprocessing operation on sequence ${\mathbf{S}}_{\mathrm{u}}$ before training, and $\mathrm{T}\left({\mathbf{S}}_{\mathrm{u}}\right)$ represents the padding of the initial sequence ${\mathbf{S}}_{\mathrm{u}}$. The usual method for sequence padding is zero-padding, where $\mathrm{c}=|{\mathbf{S}}^{\mathrm{u}}|-n$ represents the number of items that need to be truncated from the initial sequence.

Traditional sequence padding typically uses zero-padding to extend the initial sequence length to the maximum sequence length n:

$$\mathrm{Z}\left({\mathbf{S}}_{\mathrm{u}}\right)=\mathrm{Z}\left(\left[{\mathrm{s}}_1,{\mathrm{s}}_2,{\mathrm{s}}_3,\dots ,{\mathrm{s}}_{\left|{\mathbf{S}}^{\mathrm{u}}\right|}\right]\right)=\left[{\mathrm{0,0},\dots ,\mathrm{s}}_{\mathrm{k}+1},{\mathrm{s}}_{\mathrm{k}+2},{\mathrm{s}}_{\mathrm{k}+3},\dots ,{\mathrm{s}}_n\right]$$

(3)

where $\mathrm{Z}\left({\mathbf{S}}_{\mathrm{u}}\right)$ indicates zero-padding for sequence ${\mathbf{S}}_{\mathrm{u}}$, and $\mathrm{k}=n-\left|{\mathbf{S}}^{\mathrm{u}}\right|$ represents the number of zeros that need to be added, with $|{\mathbf{S}}^{\mathrm{u}}|<n$.

Although traditional zero-padding is simple to implement, it can lead to a waste of input space. Blin uses RandomPad instead of traditional zero-padding. A comparison between traditional zero-padding and RandomPad is shown in Figure 2. The specific definition of the RandomPad padding method is as follows:

$$\mathrm{R}\mathrm{a}nd\mathrm{o}\mathrm{m}\mathrm{P}\mathrm{a}d\left({\mathsf{\alpha },\mathbf{S}}_{\mathrm{u}}\right)=\underset{\left(\mathsf{\alpha }+1\right){\mathbf{S}}_{\mathrm{u}}}{\underbrace{\left[\left.{\mathbf{S}}_{\mathrm{u}}\right|\left.0\right|\left.{\mathbf{S}}_{\mathrm{u}}\right|\left.0\right|\left.{\mathbf{S}}_{\mathrm{u}}\right|\dots \left.\left|0\right.\right|\left.{\mathbf{S}}_{\mathrm{u}}\right|\right]}}$$

(4)

where $\mathrm{RandomPad}\left({\alpha ,\mathbf{S}}_{\mathrm{u}}\right)$ represents the random repeat padding for ${\mathbf{S}}_{\mathrm{u}}$, $\left|{\mathbf{S}}^{\mathrm{u}}\right|<n$ and $\alpha \in [1,\max ]$ represents the randomly generated number of padding repetitions. $\max =⌊n/|{\mathbf{S}}^{\mathrm{u}}|⌋$ represents the maximum number of repetitions that can be applied to the initial sequence. When $\max =0$, it means that the initial sequence is a long sequence, and there is not enough space to accommodate the maximum sequence length $n$. In this case, the number of padding times $\alpha =0$. Any remaining space after RandomPad is filled with zeros. Adding zeros between repeated sequences prevents the tail of the initial user sequence from predicting the head.

3.3. Embedding Layer

First, we constructed an embedding table $\mathbf{M}\in {\mathrm{R}}^{\left|\mathrm{V}\right|*d}$ using the set of projects, where $\left|\mathrm{V}\right|$ denotes the number of projects in the set and d represents the embedding dimension. We converted the user training sequence ${\mathbf{S}}_{\mathrm{u}}=[{\mathrm{s}}_1^{\mathrm{u}},{\mathrm{s}}_2^{\mathrm{u}},\dots ,{\mathrm{s}}_{\mathrm{t}}^{\mathrm{u}},\dots ,{\mathrm{s}}_{\left|{\mathbf{S}}^{\mathrm{u}}\right|}^{\mathrm{u}}]$ into a fixed-length sequence ${\mathbf{S}}_{\mathrm{u}}=\left[{\mathrm{s}}_1,{\mathrm{s}}_2,\dots ,{\mathrm{s}}_n\right]$, where $n$ is the maximum length the model can handle. If the sequence length exceeded $n$, we considered only the most recent $n$ user interactions. If the sequence length was less than $n$, we used RandomPad to pad the left side of the sequence until its length reached $n$. Next, we retrieved the project embedding matrix $\mathbf{E}\in {\mathrm{R}}^{n*d}$ corresponding to the user interaction sequence ${\mathbf{S}}_{\mathrm{u}}$ by querying the project embedding table $\mathbf{M}$, where E_i = M_si. For positional embedding, we introduced a learnable positional embedding $\mathbf{P}\in {\mathrm{R}}^{n*d}$. By combining project embeddings with positional embeddings, we obtained the final sequence embedding:

$$\widehat{\mathbf{E}}=\left[\begin{array}{c}{\mathbf{M}}_{{\mathbf{s}}_{\mathbf{1}}}+{\mathbf{P}}_{\mathbf{1}}\\ \begin{array}{c}{\mathbf{M}}_{{\mathbf{s}}_{\mathbf{2}}}+{\mathbf{P}}_{\mathbf{2}}\\ \begin{array}{c}\dots \\ {\mathbf{M}}_{{\mathbf{s}}_{\mathbf{n}}}+{\mathbf{P}}_{\mathbf{n}}\end{array}\end{array}\end{array}\right]=\left[\begin{array}{c}{\mathbf{s}}_{\mathbf{1}}\\ \begin{array}{c}{\mathbf{s}}_{\mathbf{2}}\\ \begin{array}{c}\dots \\ {\mathbf{s}}_{\mathbf{n}}\end{array}\end{array}\end{array}\right]$$

(5)

3.4. Transformer Layer

Utilizing the sequence embeddings obtained from the previous section, we input them into the Transformer module. We stacked multiple Transformer layers to capture hierarchical dependencies within the input sequence, aiming to better learn the representations of user preferences. However, traditional dot product attention methods involve high computational costs in the attention matrix calculation process. To mitigate these costs while ensuring model effectiveness, Blin employs a linear attention mechanism by designing a unique kernel function to replace the Softmax function in the self-attention mechanism. This changes the computation order in the attention dot product operation, from first calculating $\mathbf{Q}{\mathbf{K}}^{\mathrm{T}}$ to first calculating ${\mathbf{K}}^{\mathrm{T}}\mathbf{V}$, thereby significantly reducing the computational cost. The computational complexity of traditional dot product attention is reduced from $\mathrm{O}(n^{2}d)$ to O(nd²). In scenarios where $n\gg d$, this complexity can be approximated as linear $\mathrm{O}(n)$.

3.4.1. Definition of Traditional Dot Product Attention Method

The key component of the Transformer is self-attention, which effectively captures dependencies between different positions in a sequence. Each element in the sequence is weighted based on the importance of other elements. Typically, this is computed using the dot product attention function [26]:

$$\mathbf{Q}=\mathbf{E}{\mathbf{W}}_{\mathrm{Q}},\mathbf{K}=\mathbf{E}{\mathbf{W}}_{\mathrm{K}},\mathbf{V}=\mathbf{E}{\mathbf{W}}_{\mathrm{V}}$$

$$\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(\mathbf{Q},\mathbf{K},\mathbf{V}\right)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\mathrm{T}}}{\sqrt{d}}}\right)\mathbf{V}$$

(6)

where $\mathrm{E}\in {\mathrm{R}}^{n\times d}$ represents the input sequence matrix, $\mathbf{Q}=\mathbf{E}{\mathbf{W}}^{\mathrm{Q}},\mathbf{K}=\mathbf{E}{\mathbf{W}}^{\mathrm{K}},\mathbf{V}=\mathbf{E}{\mathbf{W}}^{\mathrm{V}}$ respectively, denote queries, keys, and values. $\left\{{\mathbf{W}}^{\mathrm{Q}},{\mathbf{W}}^{\mathrm{K}},{\mathbf{W}}^{\mathrm{V}}\right\}\in {\mathbf{R}}^{d\times d}$ are three projection matrices, with $\sqrt{d}$ as a scaling factor. The softmax function is applied row-wise to $\mathbf{Q}{\mathbf{K}}^{\mathrm{T}}$.

3.4.2. Generalization of Kernel-Based Dot Product Attention

Assuming matrix $\mathbf{D}$ represents the output of Equation (6), where the ith row of $\mathbf{D}$ is denoted as ${\mathbf{D}}_i$, Equation (6) can be generalized for attention computation under any similarity function:

$${\mathbf{D}}_i={\left(\begin{array}{c}{\displaystyle \frac{\mathrm{S}i\mathrm{m}\left({\mathbf{Q}}_i,{\mathbf{K}}_m\right){\mathbf{V}}_i}{{\sum }_{j=1}^n\mathrm{S}i\mathrm{m}\left({\mathbf{Q}}_i,{\mathbf{K}}_j\right)}}\end{array}\right)}_{m=1}^n$$

(7)

where ${\mathbf{Q}}_i=\left({\mathrm{Q}}_{i1},{\mathrm{Q}}_{i2},\dots ,{\mathrm{Q}}_{id}\right)$ represents the ith row of matrix $\mathbf{Q}$, ${\mathrm{K}}_j={({\mathrm{K}}_{j1},{\mathrm{K}}_{j2},\dots ,{\mathrm{K}}_{jd})}^{\mathrm{T}}$ represents the jth column of matrix ${\mathbf{K}}^{\mathrm{T}}$, and ${\mathrm{V}}_i=\left({\mathrm{V}}_{i1},{\mathrm{V}}_{i2},\dots ,{\mathrm{V}}_{id}\right)$ represents the ith row of matrix $\mathbf{V}$, where $i\in (1,2\dots ,n)$. $\mathrm{f}={\left(\mathrm{f}(x)\right)}_{x=1}^n$ denotes the content of vector $\mathrm{f}$ as $\mathrm{f}=\left(\mathrm{f}\left(1\right),\mathrm{f}\left(2\right),\dots ,\mathrm{f}(n)\right)$. Equation (6) is equivalent to Equation (7) using Softmax attention, where the similarity measure is $\mathrm{Sim}\left(\mathbf{Q},\mathbf{K}\right)=\exp (\mathbf{Q}{\mathbf{K}}^{\mathrm{T}}/\sqrt{d})$. $\mathrm{Sim}\left(\mathbf{Q},\mathbf{K}\right)$ calculates the similarity between all queries and keys, and since $\mathbf{Q},\mathbf{K}\in {\mathrm{R}}^{n\times d}$, the product of $\mathbf{Q}$ and $\mathbf{K}$ leads to $\mathrm{O}(n^2)$ computational complexity. Recent studies have aimed to reduce computational costs by introducing carefully designed kernel functions as approximations of the original similarity function. Specifically, $\mathrm{Sim}\left({\mathbf{Q}}_i,{\mathbf{K}}_j\right)$ can be further generalized as $\mathrm{Sim}\left({\mathbf{Q}}_i,{\mathbf{K}}_j\right)=\mathsf{\varphi }({\mathbf{Q}}_i){\mathsf{\varphi }({\mathbf{K}}_j)}^{\mathrm{T}}$, transforming Equation (7) into:

$${\mathbf{D}}_i={\left(\begin{array}{c}{\displaystyle \frac{\mathsf{\varphi }\left({\mathbf{Q}}_i\right){\mathsf{\varphi }\left({\mathbf{K}}_m\right)}^{\mathrm{T}}{\mathbf{V}}_i}{{\sum }_{j=1}^n\mathsf{\varphi }\left({\mathbf{Q}}_i\right){\mathsf{\varphi }\left({\mathbf{K}}_j\right)}^{\mathrm{T}}}}\end{array}\right)}_{m=1}^n$$

(8)

Equation (8) continues to simplify:

$${\mathbf{D}}_i={\left(\begin{array}{c}{\displaystyle \frac{\mathsf{\varphi }\left({\mathbf{Q}}_i\right){\mathsf{\varphi }\left({\mathbf{K}}_m\right)}^{\mathrm{T}}{\mathbf{V}}_i}{\mathsf{\varphi }\left({\mathbf{Q}}_i\right){\sum }_{j=1}^n{\mathsf{\varphi }\left({\mathbf{K}}_j\right)}^{\mathrm{T}}}}\end{array}\right)}_{m=1}^n$$

(9)

It can be observed that, by extending the original similarity function $\mathrm{Sim}\left({\mathbf{Q}}_i,{\mathbf{K}}_j\right)$ to $\mathsf{\varphi }({\mathrm{Q}}_i){\mathsf{\varphi }({\mathrm{K}}_j)}^{\mathrm{T}}$, the computation of attention can prioritize the calculation of ${\mathrm{K}}^{\mathrm{T}}\mathrm{V}$. Since ${\mathbf{K}}^{\mathrm{T}}\in {\mathrm{R}}^{d\times n}$ and $\mathbf{V}\in {\mathrm{R}}^{n\times d}$, the complexity of the product between ${\mathbf{K}}^{\mathrm{T}}$ and $\mathbf{V}$ is $\mathrm{O}(d^2)$. In scenarios where $n\gg d$, the computational complexity of kernel-based dot product calculation can be approximated as $\mathrm{O}(n)$, significantly reducing the computational cost compared to traditional dot product attention.

3.4.3. Linear Attention Mechanism

From the above reasoning, it is evident that achieving linear computational complexity in attention mechanisms hinges on determining an appropriate kernel function, specifically using $\mathsf{\varphi }(\mathbf{Q}){\mathsf{\varphi }(\mathbf{K})}^{\mathrm{T}}$ to replace the traditional dot product attention operation $\mathrm{Sim}\left(\mathbf{Q},\mathbf{K}\right)$. Blin employs an effective method of attention to replace traditional dot product attention, specifically using the Softmax function as the kernel function ($\mathsf{\varphi }\left(\mathbf{Q}\right){\mathsf{\varphi }\left(\mathbf{K}\right)}^{\mathrm{T}}={\mathsf{\sigma }}_{\mathrm{row}}\left(\mathbf{Q}\right){\mathsf{\sigma }}_{\mathrm{col}}{\left(\mathbf{K}\right)}^{\mathrm{T}}$). This transforms the single Softmax operation in traditional dot product attention into two separate Softmax operations. Verified to be a straightforward and effective replacement for traditional dot product attention, the following equation describes the effective attention mechanism:

$$\mathrm{E}\left(\mathbf{Q},\mathbf{K},\mathbf{V}\right)={\mathsf{\sigma }}_{\mathrm{r}\mathrm{o}\mathrm{w}}\left(\mathbf{Q}\right)\left({\mathsf{\sigma }}_{\mathrm{c}\mathrm{o}\mathrm{l}}{\left(\mathbf{K}\right)}^{\mathrm{T}}\mathbf{V}\right)$$

(10)

where ${\mathsf{\sigma }}_{\mathrm{row}}(·)$ and ${\mathsf{\sigma }}_{\mathrm{row}}(·)$ apply the Softmax function along each row or column of the matrix, respectively. Effective attention essentially selects the Softmax function as the kernel function, transforming the single Softmax operation in traditional dot product attention into two separate Softmax operations. This transformation changes the execution order of dot product operations from $\mathbf{Q}{\mathbf{K}}^{\mathrm{T}}\mathbf{V}$ to $\mathbf{Q}\left({\mathbf{K}}^{\mathrm{T}}\mathbf{V}\right)$, thereby reducing computational costs. Although this transformation is not entirely equivalent, in practice, their effects are very similar. The main propertes of ${\mathsf{\sigma }}_{\mathrm{row}}\left(\mathbf{Q}{\mathbf{K}}^{\mathrm{T}}\right)$ are that the sum of each row is 1 and the elements in each row are non-negative. The matrix ${\mathsf{\sigma }}_{\mathrm{row}}\left(\mathbf{Q}\right){\mathsf{\sigma }}_{\mathrm{col}}{\left(\mathbf{K}\right)}^{\mathrm{T}}$ also possesses these properties, which has been confirmed in previous studies.

The Softmax function tends to globally attend to the entire sequence, but its global nature may lead to neglecting local information when capturing long-range dependencies. Additionally, the exponential nature of Softmax may cause attention weights to overly focus on a few items in the sequence, exaggerating their importance and neglecting others, resulting in information loss. To address the limitations of the Softmax function, this paper proposes a simple and effective improvement: integrating a Depthwise Convolution (DWC) module into the attention matrix. The DWC module helps the model to capture local structures and relationships more comprehensively within the sequence, enabling the attention mechanism to compute attention weights based not only on exponential functions, but also on richer local features. Figure 3 contrasts traditional dot product attention with the linear attention mechanism. The final output representation of the linear attention mechanism is:

$$\mathbf{D}={\mathsf{\sigma }}_{\mathrm{r}\mathrm{o}\mathrm{w}}\left(\mathbf{Q}\right)\left({\mathsf{\sigma }}_{\mathrm{c}\mathrm{o}\mathrm{l}}{\left(\mathbf{K}\right)}^{\mathrm{T}}\mathbf{V}\right)+\mathrm{D}\mathrm{W}\mathrm{C}\left(\mathbf{V}\right)$$

(11)

Here, $\mathbf{Q}=\mathbf{E}{\mathbf{W}}^{\mathrm{Q}},\mathbf{K}=\mathbf{E}{\mathbf{W}}^{\mathrm{K}},\mathbf{V}=\mathbf{E}{\mathbf{W}}^{\mathrm{V}}$ represent queries, keys, and values, respectively, where $\left\{{\mathbf{W}}^{\mathrm{Q}},{\mathbf{W}}^{\mathrm{K}},{\mathbf{W}}^{\mathrm{V}}\right\}\in {\mathrm{R}}^{d\times d}$ are three projection matrices. $\mathbf{D}$ denotes the output representation of the attention layer. Additionally, multiple attention functions can be used in parallel to enhance model expressiveness $\mathbf{D}\leftarrow \mathrm{MultiHead}(\mathbf{E})$.

Feedforward Neural Network: Since the linear attention layer primarily operates on linear projections, a feedforward neural network is introduced to impart non-linear characteristics to the model:

$${\mathbf{T}}_i=\mathrm{F}\mathrm{F}\mathrm{N}\left({\mathbf{D}}_i\right)=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left({\mathbf{D}}_i{\mathbf{W}}^{\left(1\right)}+{\mathbf{b}}^{\left(1\right)}\right){\mathbf{W}}^{\left(2\right)}+{\mathbf{b}}^{\left(2\right)}$$

(12)

Here, ${\mathbf{W}}^{\left(1\right)},{\mathbf{W}}^{\left(2\right)}\in {\mathrm{R}}^{d\times d}$ are weight matrices and ${\mathbf{b}}^{\left(1\right)},{\mathbf{b}}^{\left(2\right)}\in {\mathrm{R}}^d$ are learnable bias vectors. ${\mathbf{D}}_i$ and ${\mathbf{T}}_i$ represent the behavioral representations of the ith item in $\mathbf{D}$ and $\mathbf{T}$, respectively.

Finally, to accurately capture deep representations of user behavior, a strategy of stacking Transformer layers is employed, where the definition of the Lth layer is as follows:

$${\mathbf{D}}^{\left(\mathrm{L}\right)}=\mathrm{L}\mathrm{A}\left({\mathbf{T}}^{\left(\mathrm{L}-1\right)}\right)$$

(13)

$${\mathbf{T}}_i^{\left(\mathrm{L}\right)}=\mathrm{F}\mathrm{F}\mathrm{N}\left({\mathbf{D}}_i^{\left(\mathrm{L}\right)}\right)$$

(14)

Here, ${\mathbf{D}}^{\left(1\right)}=\mathbf{D}$, ${\mathbf{T}}^{\left(1\right)}=\mathbf{T}$, LA represents the linear attention layer. In addition, residual connections [27], dropout [28], and layer normalization techniques [29] can be applied to ensure stability and expedite convergence during training. Here, this paper succinctly summarizes the final output of the Transformer layer as ${\mathbf{T}}^{\left(\mathrm{L}\right)}$.

3.5. Bidirectional KL Divergence Loss

Using the item-embedding method from Section 3.3, two sequence embeddings ${\widehat{\mathbf{E}}}_1$ and ${\widehat{\mathbf{E}}}_2$, which have been padded using RandomPad, are formed. These embeddings, ${\widehat{\mathbf{E}}}_1$ and ${\widehat{\mathbf{E}}}_2$, are then input into the Transformer architecture based on the linear attention mechanism, as described in Section 3.4. This process ultimately yields two output representations ${{\mathbf{T}}_1}^{\left(\mathrm{L}\right)}$ and ${{\mathbf{T}}_2}^{\left(\mathrm{L}\right)}$. To understand the relationship between users and items in sequence recommendation, a similarity function (such as the dot product) can be used to measure the distance between user output representations and item representations. Specifically, for the user output representations ${{\mathbf{T}}_1}^{\left(\mathrm{L}\right)}$ and ${{\mathbf{T}}_2}^{\left(\mathrm{L}\right)}$, we compute corresponding similarity probability distributions $\mathrm{P}({{\mathbf{T}}_1}^{\left(\mathrm{L}\right)})$ and $\mathrm{P}({{\mathbf{T}}_2}^{\left(\mathrm{L}\right)})$ to predict the next item of interest for user $\mathrm{u}$. The specific formula is as follows:

$$\mathrm{P}\left({\mathbf{T}}^{\left(\mathrm{L}\right)};\mathsf{\alpha }\right)={\displaystyle \frac{\exp \left({\mathbf{T}}^{\left(\mathrm{L}\right)}{\mathrm{e}}_{\mathrm{t}+1}^{+}\right)}{\exp \left({\mathbf{T}}^{\left(\mathrm{L}\right)}{\mathrm{e}}_{\mathrm{t}+1}^{+}\right)+\sum _{{\mathrm{e}}_{\mathrm{t}+1}^{-}\in \mathbf{V}}\exp \left({\mathbf{T}}^{\left(\mathrm{L}\right)}{\mathrm{e}}_{\mathrm{t}+1}^{-}\right)}}$$

(15)

$${\mathcal{L}}_{\mathrm{s}\mathrm{e}\mathrm{q}}=-\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{P}\left({\mathbf{T}}^{\left(\mathrm{L}\right)};\mathsf{\alpha }\right)$$

(16)

Here, ${\mathrm{e}}_{\mathrm{t}+1}^{+}$ represents a positive sample, ${\mathrm{e}}_{\mathrm{t}+1}^{-}$ represents a randomly sampled negative sample from the item set $\mathbf{V}$, and $\mathsf{\alpha }$ represents all trainable parameters in the model. ${\mathcal{L}}_{\mathrm{seq}}$ represents the sequence loss function.

After applying Equation (15) to the user output representations ${{\mathbf{T}}_1}^{\left(\mathrm{L}\right)}$ and ${{\mathbf{T}}_2}^{\left(\mathrm{L}\right)}$, we obtain two different similarity probability distributions $\mathrm{P}({\mathbf{T}}_1)$ and $\mathrm{P}({\mathbf{T}}_2)$. Blin introduces bidirectional KL divergence loss to regularize these two different similarity distributions. Figure 4 illustrates a specific example, showing the process of an auxiliary learning task executed by Blin for a simple user interaction sequence ${\mathbf{S}}_{\mathrm{u}}=[{\mathrm{s}}_1,{\mathrm{s}}_2]$. The numbers ①–⑥ indicate the sequence of steps. Before introducing the bidirectional KL divergence loss, let us first define the KL divergence. For two probability distributions $\mathrm{P}\left(\mathrm{x}\right)$ and $\mathrm{Q}\left(\mathrm{x}\right)$, the KL divergence is defined as:

$${\mathrm{D}}_{\mathrm{K}\mathrm{L}}(\mathrm{P}|\left|\mathrm{Q}\right)={\sum }_{\mathrm{x}}\begin{array}{c}P\left(\mathrm{x}\right)\log \left({\displaystyle \frac{\mathrm{P}\left(\mathrm{x}\right)}{\mathrm{Q}\left(\mathrm{x}\right)}}\right)\end{array}$$

(17)

KL divergence measures the expected amount of additional information required to represent $\mathrm{P}\left(\mathrm{x}\right)$ using the probability distribution $\mathrm{Q}\left(\mathrm{x}\right)$ given the probability distribution $\mathrm{P}\left(\mathrm{x}\right)$, and is commonly used to measure the difference between two probability distributions. While bidirectional KL divergence can take into account the two-way difference between two distributions at the same time and is more comprehensive in terms of optimization objectives, bidirectional KL divergence loss is defined as follows:

$${\mathcal{L}}_{\mathrm{b}\mathrm{k}\mathrm{l}}={\displaystyle \frac{1}{2}}\left({\mathrm{D}}_{\mathrm{K}\mathrm{L}}\right(\mathrm{P}({{\mathbf{T}}_{\mathbf{1}}}^{\left(\mathrm{L}\right)};\mathsf{\alpha })\left|\right|{\mathrm{D}}_{\mathrm{K}\mathrm{L}}\left(\mathrm{P}\right({{\mathbf{T}}_{\mathbf{2}}}^{\left(\mathrm{L}\right)};\mathsf{\alpha }\left)\right)+{\mathrm{D}}_{\mathrm{K}\mathrm{L}}\left(\mathrm{P}\right({{\mathbf{T}}_{\mathbf{2}}}^{\left(\mathrm{L}\right)};\mathsf{\alpha }\left)\right|\left|{\mathrm{D}}_{\mathrm{K}\mathrm{L}}\right(\mathrm{P}({{\mathbf{T}}_{\mathbf{1}}}^{\left(\mathrm{L}\right)};\mathsf{\alpha })\left)\right)$$

(18)

Bidirectional KL divergence loss helps in measuring the difference between two probability distributions. By minimizing this metric, the difference between the two distributions can be reduced. In other words, using the bidirectional KL divergence loss combines the KL divergences between the two distributions and their mutual information, aiding the model in learning a target distribution that better aligns with expectations.

3.6. Model Training

As shown in Figure 1, the Blin model adopts a multitask training strategy, jointly optimizing the sequence recommendation prediction task and auxiliary learning tasks. The sequence loss and the final training objective are defined as follows:

$${\mathcal{L}}_{\mathrm{s}\mathrm{e}\mathrm{q}}=-{\displaystyle \frac{1}{2}}\left(\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{P}\left({{\mathbf{T}}_{\mathbf{1}}}^{\left(\mathrm{L}\right)};\mathsf{\alpha }\right)+\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{P}\left({{\mathbf{T}}_{\mathbf{2}}}^{\left(\mathrm{L}\right)};\mathsf{\alpha }\right)\right)$$

(19)

$$\mathcal{L}={\mathcal{L}}_{\mathrm{s}\mathrm{e}\mathrm{q}}+{\lambda \mathcal{L}}_{\mathrm{b}\mathrm{k}\mathrm{l}}$$

(20)

where ${\mathcal{L}}_{\mathrm{seq}}$ is the sequence loss, $\mathcal{L}$ is the total loss function of the model, and $\lambda$ is the hyperparameter controlling the intensity of the bidirectional KL divergence loss.

4. Experiments

4.1. Datasets

To validate the effectiveness of the model, experiments were conducted on four widely used public datasets, including three short-sequence datasets (Beauty, Sports, Yelp) and one long-sequence dataset (ML-1M). A brief introduction of these datasets is provided below:

• Amazon: The Amazon dataset is a large-scale dataset that records user reviews of products on the Amazon website, making it a classic dataset for recommendation systems. We selected Beauty and Sports as two different datasets from the Amazon dataset.
• Yelp: The Yelp dataset is a well-known open-source dataset obtained through a business platform.
• MovieLens-1M (ML-1M): ML-1M is a dense movie recommendation dataset widely used for evaluating recommendation algorithms.

For all the datasets, to ensure data quality, preprocessing was performed by removing users and items with fewer than five interactions. The interaction records of each user were aggregated and sorted based on timestamps to obtain each user’s interaction sequence. For each user, their two most recent interactions were used as the test set and validation set, respectively, while their remaining interactions were used as the training set.

Table 1. Dataset Statistics.

Dataset	Users	Items	Interactions	Sparsity
Beauty	22,363	12,101	198,502	99.73%
Sports	35,958	18,357	296,337	99.95%
Yelp	30,431	20,033	316,354	99.95%
ML-1M	6041	3417	999,611	95.16%

provides the statistics of the four training sets.

4.2. Evaluation Metrics

This study employs two widely used evaluation metrics in recommendation systems: Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG).

• HR@K: HR@K is a metric used to measure the recall rate. Specifically, HR@K evaluates the proportion of the top K positions in the recommendation list that successfully “hit” the user’s true preference items.
• NDCG@K: NDCG@K focuses more on the ranking performance of recommendations. In NDCG@K, not only should the recommended items match the user’s true interests, but the items ranked higher should also match the user’s higher interest levels.

4.3. Comparison Methods

We select the following models as baseline models for comparison:

• GRU4Rec [3]: An RNN-based method that introduces Gated Recurrent Units (GRUs) to explore the dependencies between items in a sequence.
• Caser [2]: A CNN-based method that uses horizontal and vertical convolution to model users’ dynamic preferences.
• SASRec [5]: The first method to introduce Transformers into sequential recommendation, modeling user behavior sequences based on a unidirectional self-attention mechanism.
• TiSASRec [14]: Builds on SASRec by incorporating absolute positions and the time intervals between items for sequence modeling.
• CL4SRec [15]: Proposes various data augmentation methods to construct contrastive learning tasks, adding a contrastive learning objective to the original SASRec objective.
• LinRec [30]: A Transformer-based sequential recommendation model that proposes L2-normalized linear attention mechanisms to reduce the computational cost of traditional attention.
• MStein [31]: Calculates the Wasserstein distance between augmented sequences as a self-supervised learning framework for mutual information in sequential recommendation.

4.4. Experimental Details

We implemented Blin and the baseline methods using the PyTorch framework. For all methods, the training batch size was fixed at 256, and the Adam optimizer with a learning rate of 0.001 was used. The embedding dimension size was set to 64, and the maximum sequence length was set to 100. For methods involving attention mechanisms, the number of attention heads and attention layers were both set to 2. The dropout rate for each dataset was set to 0.5. Other hyperparameters for each baseline were configured according to the original papers. For the hyperparameters in our proposed Blin method, $\lambda$ was set to 2.

4.5. Overall Performance

The performances of all the methods on the three datasets are shown in

Table 2. Overall performance, the best result for each row is indicated in bold.

Datasets	Metric	GRU4Rec	Caser	SASRec	TiSASRec	CL4SRec	Linrec	MStein	Blin
Beauty	HR@10	0.1601	0.1498	0.2775	0.2788	0.2931	0.2772	0.3184	0.3319
	HR@20	0.2172	0.2012	0.3572	0.3606	0.3648	0.3558	0.3876	0.4051
	NDCG@10	0.1013	0.0966	0.1752	0.1794	0.1906	0.1774	0.2122	0.2218
	NDCG@20	0.1152	0.1078	0.1961	0.1989	0.2102	0.1978	0.2294	0.2446
Sports	HR@10	0.1526	0.1453	0.2763	0.2782	0.2854	0.2766	0.2988	0.3112
	HR@20	0.2322	0.2221	0.3769	0.3794	0.3780	0.3762	0.3941	0.4102
	NDCG@10	0.0824	0.0813	0.1574	0.1581	0.1739	0.1597	0.1824	0.1911
	NDCG@20	0.1037	0.1015	0.1836	0.1846	0.1964	0.1859	0.2067	0.2158
Yelp	HR@10	0.2569	0.2546	0.4243	0.4244	0.4462	0.4239	0.4772	0.4894
	HR@20	0.4273	0.4252	0.5988	0.5902	0.6003	0.5978	0.6281	0.6403
	NDCG@10	0.1253	0.1234	0.2349	0.2448	0.2652	0.2366	0.2846	0.2944
	NDCG@20	0.1684	0.1666	0.2796	0.2803	0.3048	0.2842	0.3235	0.3369
ML-1M	HR@10	0.3322	0.3145	0.5588	0.5609	0.5811	0.5626	0.6068	0.5701
	HR@20	0.4301	0.4028	0.6747	0.6842	0.7024	0.6807	0.7252	0.6884
	NDCG@10	0.1823	0.1726	0.3022	0.3084	0.3232	0.3068	0.3371	0.3124
	NDCG@20	0.2269	0.2165	0.3846	0.3928	0.4142	0.3892	0.4308	0.3985

, with the best results in each row highlighted in bold. From a comparison of the experimental results, the following conclusions can be drawn:

Firstly, it can be observed that Transformer-based methods (SASRec and TiSASRec) consistently outperform traditional machine learning methods (GRU4Rec and Caser). This result highlights the superiority of self-attention mechanisms in capturing the global dependencies in sequence modeling. Additionally, methods based on contrastive learning (CL4SRec and MStein) consistently perform better than other methods. This is because data augmentation effectively mitigates the data sparsity problem, and contrastive learning methods capture more meaningful features in the sequence by maximizing mutual information, thereby improving the performance of sequential recommendations.

The proposed Blin method demonstrates a superior performance across all evaluation metrics and datasets when compared to all baseline methods on the short-sequence datasets (Beauty, Sports, and Yelp). Unlike the other baseline methods, Blin employs a RandomPad strategy for padding during the sequence-filling stage instead of traditional zero-padding. This approach improves the utilization of the input space compared to traditional methods. To address the increased computational cost due to padding enhancement, the linear attention mechanism is used to significantly reduce the computational complexity of traditional dot product operations. Additionally, the DWC module is incorporated to help the model capture richer local features. Ultimately, the bidirectional KL divergence loss regularizes the probability distributions obtained from different sequence representations, helping the model to learn a target distribution that better aligns with expectations.

It is noteworthy that Blin’s performance on the ML-1M dataset did not meet expectations. This may have been due to the limitations of Blin in handling long-sequence datasets. The RandomPad padding strategy loses its effectiveness when dealing with long sequences. In the ML-1M dataset, the average number of user interactions exceeds 160, and even with the maximum sequence length set to 200, most user interaction sequences still lack sufficient space for RandomPad padding. Additionally, RandomPad padding might interfere with the model’s ability to capture long-term user preferences, which also affects Blin’s performance. Nonetheless, Blin still outperforms other models in terms of its overall performance.

4.6. Ablation Study

To validate the effectiveness of different components of Blin, this paper constructs the following variants and conducts ablation experiments. All experimental variants are evaluated on the Beauty dataset, and the experimental results are shown in

Table 3. Ablation study on Beauty dataset.

Model	HR@10	NDCG@10
(A) Blin-ZC	0.3167	0.2088
(B) Blin-D	0.3278	0.2196
(C) Blin-DWC	0.3256	0.2178
(D) Blin-BKL	0.3198	0.2114
(E) Blin	0.3319	0.2218

.

• Blin-ZC: Removes the RandomPad padding method and replaces it with traditional zero-padding, using item cropping as the sequence data augmentation method.
• Blin-D: Replaces the linear attention mechanism used in this paper with a traditional attention mechanism.
• Blin-DWC: Removes the DWC module from the linear attention mechanism.
• Blin-BKL: Removes the auxiliary learning task and relies solely on sequence loss for model training.

By comparing (A) and (E), it is observed that removing the RandomPad padding method significantly reduces the model performance. This is because RandomPad utilizes idle input space to provide more samples for the model and pays more attention to long-tail items, thereby helping the model to learn better user representations. Comparing (B–C) with (E), it is evident that replacing the linear attention mechanism with a traditional dot product attention mechanism results in a decreased model performance. This is primarily because the DWC module can alleviate the information loss issue inherent in Softmax, helping the model to obtain richer local features and, thus, compute better attention weights. By comparing (D) and (E), it is found that removing the auxiliary learning task from the Blin model leads to a significant performance decrease. This indicates that bidirectional KL divergence loss contributes significantly to model performance; regularizing the probability distributions obtained from different sequence outputs via bidirectional KL divergence loss helps the model to learn target distributions that better match expectations.

4.7. Hyperparameter Study

In this section, the impact of the auxiliary learning task on the model performance is studied by sequentially setting parameter λ to {0.1, 0.5, 1, 2, 3}. HR@10 and NDCG@10 are used as evaluation metrics, and experiments are conducted on the Beauty and Sports datasets. The experimental results are shown in Figure 5.

By observing the changes in the curves of Figure 5, it is observed that the value of λ exhibits a trend of an increasing performance followed by a decrease. Specifically, the model performance peaks when λ is around 2. This result further confirms the effectiveness of the auxiliary learning task; through bidirectional KL divergence loss, the model indeed learns more valuable target distributions.

4.8. Computational Cost Analysis

To study the improvement in the model computational efficiency by the linear attention mechanism discussed in Section 3.4.3, we integrate the linear attention mechanism into the classic Transformer sequence models SASRec and BERT4Rec. GPU memory usage and training time are selected as computational cost indicators, with the training time measured by the time taken for the model to run 10 epochs. All experiments are conducted on the ML-1M dataset, and the final experimental results are shown in Figure 6.

From Figure 6, it is observed that models using the linear attention mechanism significantly reduce the GPU memory usage and training time compared to the initial models. This indicates that the linear attention mechanism used in the Blin model effectively mitigates the high computational costs associated with traditional dot product operations.

5. Conclusions and Outlook

5.1. Conclusions

The Blin model proposed in this paper significantly improves the issues of data sparsity, sequence padding space utilization, the computational complexity of attention mechanisms, and the limitations of a single loss function in sequence recommendation through innovative RandomPad and linear attention mechanisms. By employing the RandomPad padding method, Blin overcomes the inefficiency of traditional zero-padding strategies and addresses data sparsity. Combined with the linear attention mechanism, Blin reduces the computational complexity of handling long sequences from $\mathrm{O}(n^2)$ to nearly linear $\mathrm{O}(n)$, greatly enhancing computational efficiency while maintaining traditional attention learning capabilities. The introduction of bidirectional KL divergence loss as an auxiliary task enhances the regularization effect of probability distributions across different sequence representations, further improving the accuracy of the model predictions.

The innovation of the Blin model lies not only in its technical breakthroughs, but also in its effective response to longstanding issues in sequence recommendation systems. Blin’s success validates the feasibility and effectiveness of introducing multitask learning and linear attention mechanisms in sequence recommendation tasks. By integrating random repetitive padding and linear attention mechanisms, Blin offers new insights for addressing data sparsity and computational complexity issues.

5.2. Outlook

Despite the significant progress Blin has made in addressing data sparsity and computational complexity in sequence recommendation, there are areas that warrant further exploration. Firstly, when the initial user sequence length is greater than or equal to half of the maximum sequence length that the Blin model can handle, the padding times $\alpha$ will always be zero, rendering the RandomPad padding method ineffective. Considering the limitations of the RandomPad padding method in handling long sequences, future research should focus on improving Blin’s padding strategy for long sequence datasets. Secondly, with the increasing application of graph neural networks in sequence recommendation, integrating graph neural networks with the Blin model to leverage graph structure information for enhancing sequence modeling capabilities is a promising direction to explore. Finally, extensive research on auxiliary learning in sequence recommendation has demonstrated its effectiveness in improving model performance, highlighting the significant potential of optimizing recommendation systems through additional tasks. Future research can further explore and improve the design of auxiliary tasks in sequence recommendation to enhance the performance and applicability of sequence recommendation models.