RMT: Real-Time Multi-Level Transformer for Detecting Downgrades of User Experience in Live Streams

Abstract

Live-streaming platforms such as TikTok have been recently experiencing exponential growth, attracting millions of daily viewers. This surge in network traffic often results in increased latency, even on resource-rich nodes during peak times, leading to the downgrade of Quality of Experience (QoE) for users. This study aims to predict QoE downgrade events by leveraging cross-layer device data through real-time predictions and monitoring. We propose a Real-time Multi-level Transformer (RMT) model to predict the QoE of live streaming by integrating time-series data from multiple network layers. Unlike existing approaches, which primarily assess the immediate impact of network conditions on video quality, our method introduces a device-mask pretraining (DMP) technique that applies pretraining on cross-layer device data to capture the correlations among devices, thereby improving the accuracy of QoE predictions. To facilitate the training of RMT, we further built a Live Stream Quality of Experience (LSQE) dataset by collecting 5,000,000 records from over 300,000 users in a 7-day period. By analyzing the temporal evolution of network conditions in real-time, the RMT model provides more accurate predictions of user experience. The experimental results demonstrate that the proposed pretraining task significantly enhances the model’s prediction accuracy, and the overall method outperforms baseline approaches.

1. Introduction

The exponential expansion of live-streaming platforms like TikTok has underscored the indispensable role of robust network infrastructure in supporting this rapidly evolving ecosystem. However, the accompanying surge in network traffic often leads to increased latency, even on resource-rich nodes during peak times, resulting in Quality of Experience (QoE) downgrade events for users. Accurately detecting these QoE downgrade events is crucial for enhancing service quality and reducing operational costs.

Recent user experience downgrade event detection methods can be divided into the following two categories: QoS (Quality of Service)-based methods and QoE-based methods. QoS-based methods generally use software-defined networking (SDN) technologies [1,2,3,4] and Service Function Chains (SFC) [5] to assess QoE by utilizing network-centric metrics such as bandwidth, latency, jitter, and packet loss. For example, research on QoS has focused on optimizing SDN-IoT network management [6,7], satisfying multiple QoS constraints in SDN [8], and improving network QoS by extending OpenFlow and OF-Config protocols for autonomous configuration in OpenFlow switches [9], while these studies effectively measure QoS, they fall short in evaluating QoE [10]. QoE-based methods QoE is a user-focused metric that can be modeled in relation to QoS using a linear model [11]. However, a simple linear relationship does not accurately capture the metric between QoE and QoS. Telecom operators are actually concerned with the actual user experience, such as the video buffering event (QoE downgrade event), rather than a single numerical value. Therefore an increasing number of methods based on deep learning are being applied to QoE-based methods. For example, Lopez-Martin et al. use a deep learning model to extract information from network packets to predict QoE (good or bad quality) [12]. Kiani et al. proposed moving QoE models for real-time QoE measurement during DASH video streaming, demonstrating their accuracy and suitability for content service providers [13]. Hoßfeld et al. [14] propose an approach using Convolutional Neural Networks, measuring fine-grained video QoE in real-time from encrypted traffic. Canovas et al. [15] utilize Bayesian-regularized neural networks to estimate QoE, classify traffic patterns, and dynamically adjust video characteristics based on QoS parameters to optimize QoE during critical transmission moments. Huang et al. [16] propose a Deep Reinforcement Learning (DRL)-based approach for adaptive multimedia traffic control in SDN to optimize QoE. Lopez-Martin et al. [15] developed a multimedia traffic management system using Bayesian-regularized neural networks (BRNN) to estimate QoE.

In summary, current QoS and QoE evaluation methods utilize network status information, such as transmission delay, bandwidth, and packet loss rate at each port, which is then applied to linear or nonlinear models for prediction. However, network operators face certain challenges when implementing these methods on metropolitan area networks (MANs), limiting their effectiveness. As depicted in Figure 1, a MAN typically follows a three-tier architecture consisting of a Broadband Remote Access Server (BRAS) [17] or Virtual Broadband Remote Access Server (vBRAS), an Optical Line Terminal (OLT) [18,19] layer, and an Optical Network Unit (ONU) [20] layer. Current prediction models primarily evaluate the quality of individual ONUs by analyzing their performance alongside that of upstream BRAS and OLT layers. However, they fail to account for the impact of bypass devices, which significantly influence QoE. For instance, a single BRAS can connect to multiple OLTs, and vice versa, resulting in interactions between OLTs that share the same BRAS. This omission underscores the need for a more comprehensive approach to fully capture the network’s dynamic behavior.

To address these limitations, this study proposes a real-time multi-layer Transformer for detecting QoE downgrade events in live streams. The proposed method constructs a multi-layer Transformer model for an optical fiber network. First, to ensure consistency across different devices, we define a unified data representation for each layer’s devices. The multi-layer Transformer then collects the status information from the BRAS, OLT, and ONU layers, incorporating the bypass device status to predict the QoE for each user node. Experimental results demonstrate that our method achieves over a 10% improvement in prediction accuracy compared to existing approaches.

To train and evaluate our proposed method, we further created a Live Stream Quality of Experience (LSQE) dataset, comprising five million records from over 300,000 users over a period of 7 days. Experimental results on the LSQE dataset demonstrate that the proposed method achieves higher accuracy and F1 scores than existing approaches. The contributions of this study are as follows:

• We developed the Live Stream Quality of Experience (LSQE) dataset, which includes five million records from over 300,000 users in a 7-day period.
• We introduced a device-mask pretraining (DMP) method that applies a pretraining technique to cross-layer network device data, enabling a better understanding of cross-layer device correlations.
• We proposed a real-time QoE downgrade event prediction model based on a multi-layer Transformer. Experimental results show that the DMP task significantly improves the model’s prediction accuracy, and the proposed method outperforms baseline approaches.

The rest of this paper is organized as follows: Section 2 reviews the related work; Section 3 introduces the LSQE dataset; Section 4 presents the proposed Real-time Multi-level Transformer model; Section 5 details the experimental design and results; and Section 6 discusses the findings and concludes the paper.

2. Related Works

2.1. QoE/QoS Prediction for Live Video Quality Prediction

Predicting QoE and QoS allows network operators to accurately assess the quality of various services, including the Internet of Things (IoT) [6,7], streaming video [13], and Internet Protocol Television (IPTV) [11]. In recent years, research has increasingly shifted from focusing on network-level parameters to emphasizing subjective user perception. This shift highlights the evolution of QoS assessments from traditional network performance metrics to user-perceived quality. Although QoE and QoS are not mutually exclusive, QoE extends the concept of QoS by incorporating the user’s subjective perception of service performance.

QoS evaluates network quality using metrics such as bandwidth, packet loss, and delay jitter. QoE, being a superset of QoS, incorporates the user perspective and is influenced by factors across multiple levels such as the following: system level (including bandwidth variation, packet loss, delay, jitter, end-user device configuration, and browser), context level (such as user location and the purpose of streaming, whether for education or gaming), user level (expectations), and content level (bitrate, resolution, and video quality) [14].

Recent works on QoE prediction can be categorized into the following two main approaches: legacy machine learning methods [7,21] and neural network-based methods [15,16,22,23]. Legacy machine learning methods: Said et al. proposed a QoE evaluation scheme for IoT data across different optical network layers [7]. Ibarrolar et al. developed a customer satisfaction model to collect QoE labels and employed a regression model to estimate the impact of various network parameters [21]. Neural network methods: Lopez et al. introduced a model that combines Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and a Gaussian classifier to predict the QoE (good or bad experience) for live videos [12]. Canovas et al. applied Bayesian Regularized Neural Networks (BRNNs) to classify different types of traffic based on user-perceived quality [15]. Madanapalli et al. [12] utilized a single Long Short-Term Memory (LSTM) neural network along with a multi-layer perceptron (MLP) to predict QoE. Huang et al. proposed an adaptive multimedia traffic control mechanism using Deep Reinforcement Learning (DRL), which combines deep learning with reinforcement learning and learns through trial-and-error-based rewards [16].

2.2. Transformer

To clarify our proposed methods, we begin with a review of the multi-layer perceptron (MLP), often referred to as the feed-forward neural network (FNN). This is the simplest and most widely recognized form of a multi-layer neural network. Subsequent hidden layers within the MLP are responsible for extracting an abstract representation through nonlinear transformations.

$$y^l=h_{W^l,b^l}^l=f(W^l{x}^{l-1}+b^l),$$

(1)

where l denotes the l-th layer; f can be a sigmoid, tanh, or ReLU function [24]; and W and b represent the weights and bias of this layer, respectively. Although a single-layer perceptron network is theoretically capable of capturing any nonlinear function, it is challenging to train, and its performance on various tasks remains unsatisfactory. Consequently, numerous neural network models have been developed, such as Convolutional Neural Networks (CNNs) [25], Long Short-Term Memory Networks (LSTMs) [26], Multi-Layer Perceptrons (MLPs), and Recurrent Neural Networks (RNNs) [27]. Following the introduction of the Transformer model [28], the architecture of neural network models began to stabilize.

The Transformer is a neural network model based on the self-attention mechanism, proposed by Google for natural language processing. Soon after its introduction, components of this model were widely adopted in large language models such as BERT [29,30,31,32] and GPTs [33,34,35]. Models utilizing the self-attention layer have shown superior performance compared to CNNs, LSTMs, MLPs, and RNNs on both Natural Language Processing (NLP) [32,36] and Computer Vision (CV) tasks [37,38,39]. Given an input matrix $X\in {\mathbb{R}}^{N\times H}$, where N represents the length of the input sequence and H represents the dimension of each element in the sequence, the output of the self-attention layer can be expressed as follows:

$$\mathrm{Att}\left(X\right)=\mathrm{softmax}\left({\displaystyle \frac{X{W}_Q{W}_K^T{X}^T}{\sqrt{d}}}\right)X{W}_V,$$

(2)

where $W_Q$, $W_K$, $W_V\in {\mathbb{R}}^{H\times d}$ denote the trainable parameters for the query, key, and value matrices, respectively, and d denotes the dimension of the input nodes. Attention networks construct an attention map using the query and key matrices to find the relationships between different nodes in the network, and they output a weighted value matrix $\mathrm{Att}\in {\mathbb{R}}^{N\times d}$. It is worth noting that the multiplication of $W_Q$ and $W_K$ creates a $d\times d$ dimensional matrix, which can also be interpreted as a graph neural network [40,41].

The matrix X is processed through multiple attention blocks, each consisting of a multi-head self-attention layer, a dense layer, and two normalization layers. The self-attention layer with input matrix X is defined by Equation (2). Multiple self-attention layers are then concatenated into a single vector and reshaped via a trainable parameter $W^o$ as follows:

$$MultiHead\left(X\right)=\mathrm{Concat}(At{t}_1,\dots ,At{t}_h)W^o,$$

(3)

where $W^o\in {\mathbb{R}}^{hd\times H}$ is a trainable parameter, h is the number of self-attention heads, and $MultiHead$ represents the multi-head attention operation. To prevent gradient vanishing, the Transformer employs a residual connection by adding the input and output of the multi-head attention layer, followed by batch normalization (BN) to ensure a more stable gradient, as follows:

$$M_{att}=BN\left(MultiHead\left(X\right)\right)+X,$$

(4)

where $BN$ represents the batch normalization layer, and $M_{att}\in {\mathbb{R}}^{N\times H}$ denotes the normalized output of the multi-head attention layer. Similarly, the dense layer is also designed with a residual connection followed by batch normalization, as follows:

$$M_{dense}=BN\left(Dense\left(M_{att}\right)\right)+M_{att},$$

(5)

where $Dense$ represents the dense layer, and $M_{dense}\in {\mathbb{R}}^{N\times H}$ represents the output matrix.

3. Live Stream Quality of Experience Dataset

The Live Stream Quality of Experience Dataset is a comprehensive collection of data that captures key metrics related to the quality of experience for live streaming services. This dataset includes various parameters such as video resolution, frame rate, latency, packet loss, and user engagement. Its purpose is to provide insights into the factors that influence user satisfaction and to support improvements in live streaming service quality. To efficiently collect user experience data for live stream videos, we developed a network monitoring tool designed to track the network traffic of Optical Network Unit (ONU) devices.

We collected optical data from 300,000 users within a Metropolitan Area Network (MAN) over a seven-day period, resulting in 12 million data records totaling 500 GB. This dataset comprises data from the following three network layers: BRAS, OLT, and ONU. The original dataset is organized into 15 tables. Specifically, we collected 8.93 million records (287.5 GB) from the ONU layer, 780,000 records (95 GB) from the OLT layer, and 420,000 records (31.8 GB) from the BRAS layer.

The attributes of the BRAS, OLT, and ONU layer data collected from the Metropolitan Area Network (MAN) are presented in Table 1, Table 2 and Table 3, providing a comprehensive overview of network monitoring and QoE management as follows:

• Table 1: BRAS layer—the BRAS (Broadband Remote Access Server) layer attributes encompass device-specific information (IP, model number, port/slot), traffic data (inbound/outbound rate differences, CRC error counts), and bandwidth metrics (inbound/outbound rates, utilization percentages). Additionally, it includes received optical power details (with maximum, minimum, and average power levels) to monitor the optical signal and a “QoE downgrade event Flag” to identify instances of degraded service quality.
• Table 2: OLT layer—attributes of the OLT (Optical Line Terminal) layer are categorized into device info and transmission details. Device info includes port IP, location city, device ID, and board performance. Transmission metrics cover inbound/outbound traffic rates, port bandwidth, and bandwidth utilization. It also provides optical power levels for both received and transmitted signals, contributing to assessing network performance, with a “QoE downgrade Flag” to denote QoE downgrade events.
• Table 3: ONU layer—the ONU (Optical Network Unit) layer attributes are broken down into device info (device ID, local identifier, city code, CPU, and memory usage) and transmission metrics, similar to the OLT layer. The transmission data include received and transmitted optical power levels, port bandwidth, bandwidth utilization, and the maximum number of user devices supported. Like the other layers, the table features a “QoE downgrade Flag” to denote QoE downgrade events.

Table 1. Attributes of the BRAS layer device.

BRAS Layer
No.	Attributes	Description
1	BRAS IP	BRAS device data
2	BRAS Device IP
3	BRAS Model Number
4	BRAS Port/slot
5	Inbound Rate Difference (%)	BRAS traffic, error rate, utilization data
6	Outbound Rate Difference (%)
7	CRC Inbound Error Count
8	CRC Outbound Error Count
9	Inbound Rate (Gb/s)
10	Outbound Rate (Gb/s)
11	Bandwidth Utilization (%)
12	Received Optical Power (dBm)	BRAS received optical power
13	Maximum Received Optical Power (dBm)
14	Minimum Received Optical Power (dBm)
15	Average Received Optical Power (dBm)
16	QoE Downgrade Event Flag	Label

Table 1. Attributes of the BRAS layer device.

BRAS Layer
No.	Attributes	Description
1	BRAS IP	BRAS device data
2	BRAS Device IP
3	BRAS Model Number
4	BRAS Port/slot
5	Inbound Rate Difference (%)	BRAS traffic, error rate, utilization data
6	Outbound Rate Difference (%)
7	CRC Inbound Error Count
8	CRC Outbound Error Count
9	Inbound Rate (Gb/s)
10	Outbound Rate (Gb/s)
11	Bandwidth Utilization (%)
12	Received Optical Power (dBm)	BRAS received optical power
13	Maximum Received Optical Power (dBm)
14	Minimum Received Optical Power (dBm)
15	Average Received Optical Power (dBm)
16	QoE Downgrade Event Flag	Label

Table 2. Attributes of the OLT layer device.

OLT Layer
No.	Attributes	Description
1	Port IP	Device info
2	Location City
3	Device Id
4	OLT Single Board Performance
5	Port ID	OLT transmission info, including received and transmitted optical power
6	Inbound Rate (Gb/s)
7	Outbound Rate (Gb/s)
8	Port Bandwidth (Gb/s)
9	Bandwidth Utilization (%)
10	Received Optical Power (dBm)
11	Transmitted Optical Power (dBm)
12	QoE Downgrade Event Flag	Label

Table 2. Attributes of the OLT layer device.

OLT Layer
No.	Attributes	Description
1	Port IP	Device info
2	Location City
3	Device Id
4	OLT Single Board Performance
5	Port ID	OLT transmission info, including received and transmitted optical power
6	Inbound Rate (Gb/s)
7	Outbound Rate (Gb/s)
8	Port Bandwidth (Gb/s)
9	Bandwidth Utilization (%)
10	Received Optical Power (dBm)
11	Transmitted Optical Power (dBm)
12	QoE Downgrade Event Flag	Label

Table 3. Attributes of the ONU layer device.

ONU Layer
No.	Attributes	Description
1	Device Id	Device info
2	Local Identifier
3	City Code
4	CPU Usage (%)
5	Memory Usage (%)
6	Received Optical Power (dBm)	ONU transmission info, including input and output.
7	Transmitted Optical Power (dBm)
8	Port Bandwidth (Gb/s)
9	Bandwidth Utilization (%)
10	Max User Device Count
11	QoE Downgrade Event Flag	Label

Table 3. Attributes of the ONU layer device.

ONU Layer
No.	Attributes	Description
1	Device Id	Device info
2	Local Identifier
3	City Code
4	CPU Usage (%)
5	Memory Usage (%)
6	Received Optical Power (dBm)	ONU transmission info, including input and output.
7	Transmitted Optical Power (dBm)
8	Port Bandwidth (Gb/s)
9	Bandwidth Utilization (%)
10	Max User Device Count
11	QoE Downgrade Event Flag	Label

4. Multi-Level Transformer

In this section, we introduce the Real-time Multi-level Transformer (RMT) for predicting the Quality of Experience (QoE) in live video streaming. The RMT model integrates real-time data from multiple network layers, overcoming the limitations of previous methods that required synchronized data collection across network devices. Due to varying data collection frequencies among multi-level devices, traditional approaches often incurred delays while waiting for data from all layers. To address this, the RMT updates prediction results from each layer in real-time.

As illustrated in Figure 2, the proposed model consists of the following three layers: the BRAS, OLT, and ONU. Each layer predicts its corresponding QoE label and simultaneously provides cross-layer information to the other layers. During training, we first pretrain each model using a device-masked approach, followed by fine-tuning for the QoE prediction task. To mitigate prediction delays caused by varying data collection frequencies, we introduce a caching mechanism that leverages the output cache from higher network layers to provide real-time forecasts.

4.1. BRAS, OLT, and ONU Network

To address the inconsistent numbers of nodes and attributes across different layers, we structured the input data for each layer in a standardized format. Specifically, the input data for the BRAS, OLT, and ONU layers are represented as N nodes, each having H attributes. Consequently, the input data are organized into an $N\times H$ matrix, denoted as X. If the number of nodes is less than N, the matrices are padded with zero vectors. Similarly, nodes with fewer than H attributes are also padded with zeros to ensure uniformity.

The BRAS layer comprises n multi-head attention blocks, each containing an attention layer, a dense layer, and an Add&Norm layer. As noted in the related works section, given an input X, the attention block is formulated as follows:

$$AB\left(X\right)=BN\left(Dense\left(M_{att}\left(X\right)\right)\right)+M_{att}\left(X\right),$$

(6)

where $AB$ represents the attention block. The input and output dimensions of the attention block are $R^{N\times H}$, which allows the blocks to be stacked end to end. Therefore, we employ the attention block as the fundamental model for the BRAS, OLT, and ONU layers. To enhance the accuracy of QoE prediction, the OLT and ONU networks incorporate information from their upper-layer networks, represented as follows:

$${\tilde{L}}_{input}=\mathrm{Concat}(B_{\mathrm{out}},L_{input})W^L,$$

(7)

$${\tilde{U}}_{input}=\mathrm{Concat}(L_{out},U_{input})W^U,$$

(8)

where $L_{\mathit{input}}$ and $U_{\mathit{input}}$ represent the input device information for the OLT and ONU layers, respectively, and $L_{\mathit{out}}$ represents the output of the OLT layer.

4.2. Device-Masked Pretraining

Pretraining is a semi-supervised learning method that creates training tasks using unlabeled data without human intervention, allowing the model to learn intrinsic patterns from the data. Originally proposed to develop Large Language Models (LLMs), pretraining has been successfully applied in models such as BERT and GPT. This paper introduces a device-mask pretraining (DMP) method designed to learn correlations among devices across different optical network layers, thereby enhancing prediction accuracy in the context of quality prediction.

Device-masked pretraining enables the model to predict masked values based on unmasked values. As illustrated in Figure 3, given inputs X, L, and U, the loss function of the device-masked pretraining method for the BRAS, OLT, and ONU layers is defined as follows:

$${Loss}_{dmp}\left(\theta \right)={\displaystyle \frac{1}{2NH}}\sum _{i=1}^N\sum _{j=1}^H{\left(h_{\theta }^{i,j}\left(X\right)-y_i^j\right)}^2,$$

(9)

where $h_{\theta }^{i,j}\left(X\right)$ represents the predicted attribute of the $i_{\mathrm{th}}$ device’s $j_{\mathrm{th}}$ value, and $y_i^j$ represents its actual value. We mask not only all devices but also specific values within each device. This approach enables the model to learn both the correlations between devices and the patterns within each individual device.

4.3. Time-Delayed Fine-Tuning

After pretraining, we fine-tune the pretrained model for the QoE prediction task. As mentioned in the first section, there is a time misalignment issue among the different layers of network devices. To address this, we employ a time-delay method to synchronize the timing. During fine-tuning, we use previous values as inputs to train the BRAS, OLT, and ONU layers. For instance, if the ONU layer updates its data at time t, the inputs $X_{input}$ and $L_{input}$ will be the delayed data from the previous time point, represented as $X_{input}^{t-1}$ and $L_{input}^{t-1}$ as follows:

$${\widehat{Y}}_{bras}=BRAS\left(L_{input}^{t-1}\right),$$

(10)

$${\widehat{Y}}_{olt}^t=OLT(B_{\mathrm{out}}^{t-1},L_{input}^t),$$

(11)

$${\widehat{Y}}_{onu}^t=ONU(L_{output}^t,U_{input}^t),$$

(12)

where ${\widehat{Y}}_{olt}^t$ represents the binary classification prediction label of the OLT layer at time t. We use the binary cross-entropy loss function for fine-tuning the models, which is defined as:

$$\mathrm{Loss}=-\sum _{i=1}^n{y}_{i}log{\widehat{y}}_i.$$

(13)

During the testing phase, we do not repeatedly generate $X_{\mathrm{input}}$ and $L_{\mathrm{input}}$; instead, we reuse the output vectors from the corresponding layers to save computational resources.

5. Experiments

5.1. Experiments Settings and Details

Dataset: Experiments are conducted using the database mentioned in Section 3. In our experiments, we aligned the timestamps of the OLT, ONU, and BRAS information, integrating them into a dataset containing 42,000 entries with a total data volume of 31.8 GB to evaluate our method.

Training and testing partition: To simulate rich and scarce data conditions, we used standard k-fold and reverse k-fold cross-validation for evaluation. The standard k-fold cross-validation uses k-1 folds for training and the remaining fold for testing, whereas the reverse k-fold cross-validation uses one-fold validation for training and the remaining k-1 folds for testing.

Evaluation metric: Performance is evaluated using accuracy, precision, recall, and F1 Score. The definitions are as follows:

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}},$$

(14)

$$Precision={\displaystyle \frac{TP}{TP+FP}},$$

(15)

$$Recall={\displaystyle \frac{TP}{TP+FN}},$$

(16)

$$F1=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}},$$

(17)

where True Positive ($TP$) represents the number of positive samples that the model correctly predicts as positive. False Positive ($FP$) represents negative examples that are incorrectly predicted to be positive, also known as a “false alarm”. Similarly, True Negative ($TN$) represents negative samples that are correctly predicted, and False Negative ($FN$) represents positive samples that are incorrectly predicted.

Implementation details: We use the Transformer architecture to implement the proposed multi-level Transformer model. Models across different layers are pretrained and fine-tuned on a server with four Nvidia GeForce RTX 3090 GPUs. The server is rented from the AutoDL platform, and the specific manufacturer, city and country of the equipment cannot be identified. In the deployment phase, because data must be collected from the BRAS, OLT, and ONU layers, a single server’s general computing power is insufficient. Therefore, we deploy the models for each layer on separate machines and use PyTorch’s RPC framework to facilitate message communication between different servers.

Baseline methods: Although deep learning has been widely applied in fields such as computer vision (CV) and natural language processing (NLP), its application in Quality of Experience (QoE) assessment remains relatively limited. Currently, available research is based on simpler network models.

• BRNN [15]: Bayesian-regularized neural networks for predicting QoE status, using patterns derived from objective QoE measurements and network statistics collected by the SDN controller to classify multimedia traffic.
• DRL [16]: An adaptive multimedia traffic control mechanism based on Deep Reinforcement Learning (DRL), which integrates deep learning and reinforcement learning to learn from rewards through trial and error.
• CNN [12]: A hybrid model combining a Convolutional Neural Network (CNN), a recurrent neural network, and a Gaussian process classifier.
• LSTM [12]: An LSTM-based binary classifier that differentiates between live and on-demand streams in real-time with an accuracy exceeding 95%.

5.2. Experimental Results

5.2.1. Comparison with Other Methods

Table 4. The evaluation results of different methods on the ONU, OLT, and BRAS layers were obtained using 5-fold cross-validation.

Layer	Method	Acc	Pre	Rec	F1
ONU	CNN	0.715	0.641	0.767	0.699
	LSTM	0.762	0.687	0.822	0.749
	BRNN	0.748	0.664	0.819	0.734
	DRL	0.776	0.712	0.832	0.768
	MT(Ours)	0.806	0.725	0.872	0.792
OLT	CNN	0.665	0.611	0.727	0.659
	LSTM	0.692	0.627	0.762	0.684
	BRNN	0.708	0.634	0.768	0.694
	DRL	0.716	0.682	0.802	0.718
	MT(Ours)	0.766	0.695	0.842	0.732
BRAS	CNN	0.745	0.671	0.817	0.739
	LSTM	0.822	0.747	0.872	0.809
	BRNN	0.808	0.694	0.879	0.774
	DRL	0.836	0.762	0.872	0.798
	MT(Ours)	0.856	0.765	0.912	0.822

presents the evaluation results of different methods applied to the ONU, OLT, and BRAS layers. The results indicate that the proposed method outperforms all other methods across every optical network layer. Notably, the accuracy of all methods is relatively lower in the OLT layer, whereas performance at the BRAS layer is slightly better. This discrepancy can be attributed to the functional nature of the OLT layer. Despite its crucial role, the OLT layer primarily acts as a distribution device. Provided there are no intrinsic issues within the OLT layer, its parameters have minimal impact on the Quality of Experience (QoE). Consequently, this results in observed prediction deviations for most methods in the OLT layer.

5.2.2. Ablation Study

To evaluate our proposed device-masked pretraining (DMP) method, we conducted assessments using both pretrained and non-pretrained models. The results, presented in

Table 5. Evaluation results of proposed methods with or without device-masked pretraining on the ONU, OLT, and BRAS layers. (5-fold cross validation).

Layer	Method	Acc	Pre	Rec	F1
ONU	MT w/o DMP	0.788	0.712	0.855	0.778
	MT w/ DMP	0.806	0.725	0.872	0.792
OLT	MT w/o DMP	0.756	0.682	0.832	0.725
	MT w/ DMP	0.766	0.695	0.842	0.732
BRAS	MT w/o DMP	0.836	0.762	0.872	0.798
	MT w/ DMP	0.856	0.765	0.912	0.822

, demonstrate that the MT model with DMP outperforms models without DMP pretraining across various network layers. Notably, the performance improvement is more significant in the BRAS layer than in the other layers, indicating that the DMP method enhances model performance by enabling it to learn both explicit and implicit connections between network devices. The improvement in the BRAS layer is particularly pronounced, while the OLT layer exhibits the least improvement. This disparity can be attributed to the more complex topology of devices in the BRAS layer compared to those in the OLT layer. Additionally, our analysis suggests that the intricate relationships and dependencies in the BRAS layer provide more opportunities for the DMP method to capture valuable patterns, resulting in greater performance gains. In contrast, the simpler and more straightforward topology of the OLT layer limits the extent to which the DMP can improve prediction accuracy.

6. Discussion and Conclusions

In this paper, we define the standard data representation for metropolitan optical networks in each device layer and created a Live Stream Quality of Experience (LSQE) dataset, containing 5 million records from over 300,000 users over a 7-day period. The LSQE database is utilized to evaluate both previous works and our proposed method. We introduce a multi-layer Transformer model for real-time video quality prediction, pretrained with a device-masked pretraining task, and fine-tuned to predict the QoE of real-time video. The experimental results demonstrate that the proposed RTM model outperforms other methods, and the DMP approach effectively captures valuable patterns among devices, leading to significant performance gains. Future work will focus on refining the DMP method to further enhance its effectiveness across all network layers and exploring its application to other types of network infrastructures.

Future work will focus on refining the DMP method to further enhance its effectiveness across all network layers and exploring its application to other types of network infrastructures. In addition, considering the real-world network conditions, we realize that there is still room for potential improvement during the implementation process. On the one hand, although the current model implementation shows great promise, its complexity is relatively high, especially in the implementation stage of higher network layers where computational resource requirements are large. Subsequently, the algorithm complexity can be optimized to reduce resource consumption, so as to better adapt to actual networks with limited resources, such as small network environments. On the other hand, most of the experiments were carried out under controlled conditions, and insufficient consideration was given to the real-time changes in actual networks. Future studies should further investigate the adaptability of the model when facing real-time situations, such as dynamic network changes, attacks, and failures, to improve its stability and reliability in actual complex network environments.