Video Quality Modelling—Comparison of the Classical and Machine Learning Techniques

Featured Application

Featured Application: The presented solution can allow for automated video quality assessment and quick and effective modelling of the video quality experienced by the user.

Abstract

The classical objective methods of assessing video quality used so far, apart from their advantages, such as low costs, also have disadvantages. The need to eliminate these defects results in the search for better and better solutions. This article proposes a video quality assessment method based on machine learning using a linear regression model. A set of objective quality assessment metrics was used to train the model. The results obtained show that the prediction of video quality based on a machine learning model gives better results than the objective assessment based on individual metrics. The proposed model showed a strong correlation with the subjective user assessments but also a good fit of the regression function to the empirical data. It is an extension and improvement of the efficiency of the classical methods of objective quality assessment that have been used so far. The solution presented here will allow for a more accurate prediction of the video quality perceived by viewers based on an assessment carried out using a much cheaper, objective method.

1. Introduction

The demand for video content is growing rapidly, which is why most platforms embed and distribute videos on apps to increase views and engagement. According to a report published in January 2023 by Sandvine, an application and network analytics company, video represented almost 66% of the total online volume in the first half of 2022 [1]. Previous CISCO forecasts said that video traffic on the Internet would increase by more than 80% [2]. The Mobility Report from Ericsson (June 2024) shows that mobile network data traffic grew 25% between Q1 2023 and Q1 2024, and at the end of 2023, video traffic accounted for 73% of all mobile data traffic. This traffic growth is driven by both increasing smartphone subscriptions and increasing average data volume per subscription, fuelled primarily by increased viewing of video content [3]. While forecasts vary, they all indicate that Internet traffic is growing rapidly, with the majority (currently three-quarters to four-fifths) of traffic related to video content delivery. Such a high network load, as well as the different quality of video recordings resulting from the coding technique used, bitrate, and resolution, may result in insufficient video quality for the end user. Poor quality can cause negative user feelings and reduce interest in video transmission. Therefore, in order to provide the end user with appropriate video quality, it is important to be able to assess and control this quality.

The main goal of the work is to find a method for modelling video image quality based on measurable objective parameters, which will enable users to predict the quality assessed to a better extent than the existing methods.

The main contribution is building a machine learning model that, on the one hand, is able to take into account much larger amounts of data than classical quality modelling methods and thus achieve greater convergence with the subjective assessments of users and, on the other hand, can help automate the video quality assessment process.

The remainder of the paper is organised as follows: Section 2 provides a reference to the literature on video quality assessment issues and the methods used. The next part of the article presents a general outline of the research methodology used by the authors, along with a description of research materials and methods. In Section 4, the authors present the results of the research. Section 5 contains a discussion of the results obtained in relation to previous research and other publications. The final part of the work presents the most important conclusions from the research and indicates directions for further work.

2. Related Work

The quality of a video can be assessed using objective or subjective methods [4]. Due to the time-consuming and relatively lengthy nature of subjective methods, a lot of effort has been put into developing objective methods for assessing video quality. A good overview of this type of method can be found in Ref. [5], which presents a classification scheme of objective quality assessment methods. These methods can be divided differently depending on the adopted criteria. One of the possible classifications assumes a division into full-reference (FR), reduced-reference (RR), and no-reference (NR) methods. FR methods assume that, in addition to the evaluated material, reference video is available [6]. If only partial information about the source video is available, RR methods can be used [7]. The third case (NR method) is used when there is only access to a distorted signal, and then the quality of the video is assessed without knowing the source of the video material [8]. These metrics can be used naturally to evaluate a wide range of video degradation. However, apart from the above function of objective metrics, attempts are often made to use them to directly assess video quality. Most often, this is performed by mapping the values of individual objective metrics onto the subjective assessments of users obtained during parallel research with a group of testers. Often, objective quality indicators are used to build quality models and then automatically predict the average user rating, which is cheaper than obtaining this type of data from expensive subjective tests.

The most frequently used objective metrics include: Aligned Peak Signal-to-Noise Ratio (APSNR) [9]; Delta, Mean Absolute Difference (MSAD) and Mean Squared Error (MSE) [10]; New Quality Index (NQI) [11,12]; Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) [13]; Visual Information Fidelity (VIF) [14], Video Multi-Method Assessment Fusion (VMAF) [15], and Video Quality Metric (VQM) [16,17]. There are many discussions and considerations in the scientific literature on the possibility of using specific objective metrics as predictors of the quality perceived by users [18]. Generally, these indicators can be divided into two groups. The first one includes purely mathematical video quality measures, such as Delta, MSAD, MSE, and PSNR, in which the error is calculated mathematically as the difference between the original and processed pixels. The second group includes metrics such as SSIM, NQI, and VQM, where the overall quality assessment also takes into account perceptual quality [19]. Depending on the method of operation, the values of individual metrics correlate with the results of subjective research to varying degrees [20,21].

Despite significant progress in the development of objective techniques for measuring the quality of video transmission, the subjective assessment of video quality still plays an important role in video processing, especially when quality degrades at various stages of video signal acquisition, compression, transmission, and display. The final verifier of the quality of video transmission devices is their user, that is, a person. Taking into account the generally subjective evaluation of video quality, it can be said that the purpose of the assessment is to evaluate and differentiate the quality of the video signals that reach the viewer. The process of visual perception can be considered the viewer’s reaction to the stimulus they receive. The viewer’s reaction depends on the stimulus received as well as on the conditions in which the viewer finds themselves. It can therefore be said that the viewer’s reaction depends not only on external factors affecting them at a given moment but also on internal factors. From a physiological and psychological point of view, the viewer’s reaction can be considered in terms of a sensational and emotional reaction. A sensation reaction occurs as a result of a given stimulus exceeding certain sensitivity thresholds or thresholds of visual sensation categories. More complex and difficult to analyse is the emotional reaction, which is not the result of the characteristics of the received signal but of the habits and individuality of the viewer. It can be said that the sensory reaction is a reflection of the visual image created in the human mind, while the emotional reaction is a reflection of the person’s attitude towards this image. By ensuring appropriately stable evaluation conditions, it can be expected that the differences in impression reactions between individual viewers are smaller than the differences in their emotional reactions. Therefore, one of the main goals of objectifying video signal quality assessments is to limit the impact of emotional reactions on the final result as much as possible. This effect is achieved by introducing an appropriate number of assessment statistics, the appropriate selection and training of the assessment team, and the appropriate formulation of test tasks. Test tasks include the selection of test material, measurement principles, and the method for analysing the results obtained during measurements.

A comprehensive review of subjective video quality assessment methodologies and objective algorithms for assessing video quality is presented in Ref. [22]. The process is even more complicated when assessing the quality of both audio and video content. An example of a work presenting the results of research in which audiovisual materials were examined, i.e., with simultaneous presentation of image and sound and various degrees of degradation, is Ref. [23]. The authors emphasise here that the audio has a significant impact on the overall quality perceived by users, but the video component has a decisive influence. Perceptual quality assessment is time-consuming, expensive, and requires subjective assessment with high skills with the participation of a specially trained group of viewers [24]. There are examples in the literature where, to reduce research costs, simpler tools and laboratory research methods are sought that provide results comparable to classical methods [25].

In recent years, machine learning models have become more common and are used not only to reduce the costs of quality modelling, among others, by introducing automation of activities but also to improve their accuracy. Ref. [26] presents an ML-based model for assessing video quality based on an analysis of the encoding video settings of transmitted contents and the intrinsic characteristics of the image to objectively estimate the mean opinion score (MOS) in correlation with subjective results. The use of data mining techniques to combine a collection of parameters associated with the video transmitted improves the performance of traditional quality evaluation. Some work has been devoted to comparing the performance of machine learning models to predict user-rated quality, depending on the type and number of inputs used to train the model [27]. Studies have also used artificial neural networks (ANNs) to build tools that will predict video quality based on user ratings. An example of such work is Ref. [28], where the authors present the concept of a measurement tool (VQMT) that allows an objective assessment of a given video material in close correlation with the perception of the human visual system (HVS). In summary, new approaches based on machine learning models, using larger amounts of data, provide the opportunity not only to automate the quality assessment process but also to develop models that are more accurate than existing, classical quality modelling methods.

3. Materials and Methods

This article summarises the previous research by the authors [29] on the relationships between the results of subjective and objective video quality evaluation, including their substitutability, and is another step towards accelerating and automating the video quality assessment process.

3.1. Reference Video Material

The reference video (see Figure 1) included dynamic scenes, namely a fragment of the start of a horse race [30].

The video sequence was encoded using three codecs (H.264 [31], H.265 [32], and VP9 [33]), four image resolutions (360p, 480p, 720p, and 1080p), and 18 selected encoding rates (from 300 to 6000 kbps). The encoding speed range was selected based on previous research in such a way that the set of test video samples covered the entire quality scale of subjective ratings, i.e., from 1 to 5. There were a total of 216 video samples.

3.2. Research Methods Used

The quality assessment of the analysed samples included three steps:

• Step 1: subjective quality assessment using an appropriate group of testers;
• Step 2: objective quality assessment using a dedicated software tool;
• Step 3: quality assessment using a machine learning model.

The detailed research procedure (flow diagram) is presented in Figure 2.

Step 1 is based on the results of subjective measurements obtained in previous studies [29,34]. The Double Stimulus Impairment Scale Method (DSISM) [24] was used to subjectively assess video quality. In this method, the viewer compares two video sequences and evaluates the degree of deterioration of the second signal (the evaluated sequence) in relation to the first signal (the reference sequence). The assessment used a five-point MOS scale, where a value of 5 means imperceptible deterioration and 1 is very bothersome [35]. Subjective quality measurements were performed in a laboratory room that met the requirements of the recommendations of the International Telecommunication Union [24,35]. Test signals with different transmission conditions (bitrate, resolution) were randomly presented to viewers on a 60-inch TV screen. The audience consisted of students from Wroclaw University of Science and Technology, aged 20–21, with normal visual acuity and correct colour discrimination. The test signals encoded with the H.264 standard were evaluated by a team of 45 people, and H.265 by 35 people. Each observer provided their assessment of the deterioration of the quality of the second video compared to the first one in a special questionnaire after watching the original and encoded sequences. After entering the spreadsheet, all evaluations were subjected to statistical analysis according to the procedure described in the ITU-R BT.500 recommendation [24]. This analysis allowed for the rejection of estimates beyond the assumed 95% confidence interval. The statistical analysis procedure according to the ITU-T P500 recommendation is presented below. The MOS value was calculated separately for each encoding technique, screen resolution, and bitrate, denoted as (c), (r), and (i), respectively:

$${MOS}_{c,r,i}={\displaystyle \frac{1}{N}}{\sum }_{k=1}^N{O}_{c,r,i,k} ,$$

(1)

where O is the assessment value of the deterioration of the quality of the k-th observer, N is the size of the observer group, c is the coding technique (c = 1—H.264, 2—H.265), r is the image resolution (r = 1—1280 × 720, 2—1920 × 1080), and i is the bitrate (i = 1, 2,…18).

The standard deviation S_c,r,i was calculated according to the relationship as follows:

$$S_{c,r,i}=\sqrt{{\sum }_{k=1}^N{\displaystyle \frac{{MOS}_{c,r,i}-O_{c,r,i,k}}{\left(N-1\right)}}} .$$

(2)

According to BT.500, a 95% confidence interval is defined as:

$$\left[{MOS}_{c,r,i}-{\delta }_{c,r,i}, {MOS}_{c,r,i}+{\delta }_{c,r,i}\right],$$

(3)

where the confidence interval coefficient σ_c,r,i is calculated according to the relationship:

$${\delta }_{c,r,i}=1.96{\displaystyle \frac{S_{c,r,i}}{\sqrt{N}}} .$$

(4)

In Step 2, an objective assessment of the quality of individual video samples was carried out. The evaluation was performed using the full reference method, where each tested video sample was compared with the reference material (uncompressed) frame by frame, pixel by pixel. For each video sample tested, the average values of the following ten objective metrics were calculated: APSNR, Delta, MSAD, MSE, NQI, PSNR, SSIM, VIF, VMAF, and VQM. It was made using a software tool called the Video Quality Estimator (VQE), version 4.3 [36].

The primary goal of the third part of the research (Step 3) was to prepare a model that, based on input data that objectively describes the quality of a video, would predict the subjective quality rating. The prediction of subjective ratings can be performed on the basis of a single objective quality metric. However, the question is: what happens if we use a set of objective parameters from different metrics, i.e., information about the luminance or chrominance (or a combination of them) for individual pixels to predict the subjective quality assessment? It was assumed that since each parameter and metric focus on slightly different properties of the video or the way its quality deteriorates, the appropriate combination of parameter values should improve the prediction accuracy (reduce the error, i.e., the difference between the average subjective rating and the predicted subjective value). As previously mentioned, in this task we had 216 videos (samples) with different bitrates for four different resolutions and three different codecs. For each video, an average subjective rating and a set of objective evaluation parameters (32 parameter values for each sample) were assigned. Therefore, we had a task that could be solved using supervised machine learning—a method in which the training data set used to train the algorithm contained the solution to the problem, known as labels or classes. According to the flow diagram (see Figure 2), the set of prepared video samples was divided into two subsets, one of which was used to train the machine learning model and the other was intended to test the trained model.

3.3. Using a Machine Learning Model

Initially, the data were split into a training set and a test set. The training set contained 180 videos with resolutions of 360p, 480p, 720p, and 1080p for the codecs H.264, H.265, and VP9. The test set contained 36 videos with resolutions of 720p and 1080p for the codecs H.264 and H.265. Videos with resolutions of 720p and 1080p for the codecs H.264 and H.265 were divided between the training and test series in such a way that the distribution of ratings was similar. The scenario that includes the phase of training the model and then using it for quality prediction is shown in Figure 3.

First, a subset of the training data was used to train the model, and then the model was used to predict the subjective video quality rating. In both cases, i.e., during model training and prediction, separate subsets of objective quality metrics values were used as input to the model. Various learning algorithms were tested to build a machine learning model that would best predict video quality. As input to the model, a subset of objective quality assessment values obtained from individual quality metrics (10 metrics, 32 parameters in total) and a subset of user assessment (MOS) values obtained during subjective studies were used.

Tests were performed for the following models:

• Linear regression assumes a linear relationship between the input variables (features) and the output variable (result). The linear regression model is represented by:

$$y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\dots +{\beta }_p{x}_p+ϵ,$$

(5)

where:

$y$—dependent variable,

${\beta }_0$—intercept,

${\beta }_1,{\beta }_2,\dots {\beta }_p$—the regression coefficients,

$x_1,x_2,\dots x_p$—the independent variables (features),

$ϵ$—random error.

2.

Ridge regression is a variant of linear regression that adds L2 regularisation to prevent overfitting by penalising large regression coefficients. The ridge regression model minimises the loss function.

$$L\left(\beta \right)={\sum }_{i=1}^n{\left(y_i-{\beta }_0-{\sum }_{j=1}^p{\beta }_j{x}_{ij}\right)}^2+{\sum }_{j=1}^p{\beta }_j^2,$$

(6)

where:

λ—regularisation parameter.

3.

Lasso regression adds L1 regularisation, which can lead to some coefficients becoming zero, thus performing feature selection. The Lasso model minimises the loss function.

$$L\left(\beta \right)={\sum }_{i=1}^n{\left(y_i-{\beta }_0-{\sum }_{j=1}^p{\beta }_j{x}_{ij}\right)}^2+\lambda {\sum }_{j=1}^p\left|{\beta }_j\right|.$$

(7)

4.

Elastic Net combines L1 (Lasso) and L2 (Ridge) regularisation to gain the benefits of both methods. The Elastic Net model minimises the loss function.

$$L\left(\beta \right)={\sum }_{i=1}^n{\left(y_i-{\beta }_0-{\sum }_{j=1}^p{\beta }_j{x}_{ij}\right)}^2+{\lambda }_1{\sum }_{j=1}^p\left|{\beta }_j\right|+{\lambda }_2{\sum }_{j=1}^p{\beta }_j^2,$$

(8)

where:

${\lambda }_1$ and ${\lambda }_2$—regularisation parameters.

5.

Support Vector Regression (SVR) is a variant of Support Vector Machines (SVMs) that is used for regression problems. SVR tries to find a regression line that is maximally distant from the nearest data points. The loss function in SVR is:

$$L\left(\beta \right)={\displaystyle \frac{1}{2}}{‖\beta ‖}^2+C{\sum }_{i=1}^{n}max\left(0,\left|y_i-\left({\beta }_0+{\sum }_{j=1}^p{\beta }_j{x}_{ij}\right)\right|-ϵ\right) ,$$

(9)

where:

$C$—regularisation parameter,

$ϵ$—width of the insensitive band.

6.

Decision trees split the data into subsets by making decisions based on feature values. Each internal node represents a test on an attribute, each branch represents the outcome of the test, and each leaf represents a class or target value. The data are split into nodes by maximising a certain criterion, such as entropy for classification or mean squared error for regression.

7.

Random forests are an ensemble learning method that trains multiple decision trees on random subsets of data and features. Predictions for new samples are made by averaging (for regression) or majority voting (for classification) the results from individual trees.

$$\widehat{y}={\displaystyle \frac{1}{M}}{\sum }_{m=1}^M{h}_m\left(x\right),$$

(10)

where:

$M$—number of trees in the forest,

$h_m\left(x\right)$—prediction from the m-th tree for sample x.

8.

Gradient Boosting Machines (GBMs) iteratively build a sequence of models, each of which corrects the errors of the previous models. Each new model minimises the loss function by adding it to the previous models.

$$F_m\left(x\right)=F_{m-1}\left(x\right)+{\gamma }_m{h}_m\left(x\right),$$

(11)

where:

$F_m\left(x\right)$—prediction at iteration m,

${\gamma }_m$—scaling factor,

$h_m\left(x\right)$—new model learning on the residuals (errors) of the previous model.

9.

kNN is an algorithm that predicts the outcome based on the k nearest neighbours in the feature space. The prediction for the kNN regression is the average target value of the neighbours.

$$\widehat{y}={\displaystyle \frac{1}{k}}{\sum }_{i=1}^k{y}_i,$$

(12)

where:

$y_i$—target value of the i-th nearest neighbour.

4. Results

The models presented in Section 3 were evaluated to determine their effectiveness in predicting the subjective quality rating based on the given objective parameters (

Table 1. Performance comparison results of the tested machine learning models.

	RMSE 1	MSE 2	MAE 3	MaxAE 4
Linear Regression	0.27	0.07	0.19	0.83
Ridge Regression	0.52	0.27	0.39	1.37
Lasso Regression	0.55	0.30	0.42	1.53
Elastic Net	0.56	0.32	0.41	1.55
Support Vector Regression	0.34	0.12	0.23	0.96
Decision Tree Regressor	0.36	0.13	0.22	1.23
Random Forest Regressor	0.35	0.13	0.24	1.13
Gradient Boost Machines	0.36	0.13	0.23	1.26
k-Nearest Neighbours	0.40	0.16	0.25	1.39

¹ root mean squared error, ² mean squared error, ³ mean absolute error, ⁴ max. absolute error.

).

The best results were obtained for the linear regression (LR) model. The LR model obtained prediction results that were most similar to those obtained during subjective assessment of image quality, as indicated by the smallest error values. The beginning of this section presents the results of classic subjective and objective video quality evaluation results and the correlations between them, while the rest of the section discusses the results of quality prediction using a machine learning (linear regression) model. The results, presented in tables and graphs, respectively, are then discussed.

4.1. Subjective Quality Assessment

The results obtained from the video quality assessment are presented in Figure 4 and

Table 2. The mean value of the video quality assessment (MOS) for the H.264 and H265 codecs, the standard deviation (S), and the confidence interval coefficient (δ) for two resolutions.

Bitrate	H.264						H.265
	1280 × 720			1920 × 1080			1280 × 720			1920 × 1080
[kbps]	MOS	S	δ	MOS	S	δ	MOS	S	δ	MOS	S	δ
300	1.16	0.43	0.13	1.05	0.32	0.1	1.09	0.29	0.12	1.13	0.34	0.14
400	1.33	0.57	0.17	1.19	0.45	0.13	1.61	0.5	0.2	1.65	0.49	0.2
500	1.81	0.55	0.16	1.35	0.57	0.17	2.09	0.42	0.17	2.04	0.64	0.26
600	2.24	0.66	0.2	1.58	0.59	0.18	2.48	0.59	0.24	2.48	0.51	0.21
700	2.71	0.56	0.17	1.98	0.64	0.19	2.87	0.34	0.14	2.83	0.58	0.24
800	2.86	0.64	0.19	2.4	0.54	0.16	3.09	0.42	0.17	3.13	0.76	0.31
900	3.12	0.54	0.16	2.81	0.55	0.16	3.26	0.45	0.18	3.35	0.49	0.2
1000	3.26	0.54	0.16	3.16	0.57	0.17	3.43	0.51	0.21	3.48	0.59	0.24
1500	3.74	0.66	0.2	3.86	0.47	0.14	3.78	0.42	0.17	3.91	0.51	0.21
2000	4.07	0.7	0.21	4.26	0.49	0.15	4.09	0.67	0.27	4.22	0.52	0.21
2500	4.19	0.59	0.18	4.4	0.54	0.16	4.3	0.56	0.23	4.43	0.51	0.21
3000	4.28	0.55	0.16	4.49	0.51	0.15	4.43	0.59	0.24	4.57	0.51	0.21
3500	4.33	0.47	0.14	4.54	0.55	0.17	4.48	0.59	0.24	4.7	0.47	0.19
4000	4.4	0.49	0.15	4.63	0.49	0.15	4.57	0.51	0.21	4.78	0.42	0.17
4500	4.49	0.51	0.15	4.7	0.46	0.14	4.61	0.5	0.2	4.83	0.39	0.16
5000	4.56	0.5	0.15	4.79	0.41	0.12	4.7	0.47	0.19	4.87	0.34	0.14
5500	4.65	0.48	0.14	4.84	0.37	0.11	4.74	0.45	0.18	4.91	0.29	0.12
6000	4.72	0.45	0.14	4.91	0.29	0.09	4.78	0.42	0.17	4.91	0.29	0.12

. As seen in Figure 4, the video quality is directly proportional to the encoding bitrate used. For both codecs and both image resolutions, the greatest quality changes are observed at low bitrates, i.e., in the range of up to approximately two megabits per second. Above this threshold, the video quality in all cases analysed was above four on the MOS scale, indicating that most users rated this quality as good or very good.

It can also be noted that the general rule is that for a given encoding rate, the video quality for the H.265 codec is higher than for the H.264 codec. At the same time, it can be noted that higher image resolution results in higher quality perceived by users. It can also be seen that changes in encoding rates in the range of values below two megabits greatly affect the change in video quality, while for higher values of encoding rates, the changes in quality are much smaller. The detailed results of the subjective tests conducted by the authors on the examined video material are presented in Table 2, which shows the mean assessment of video quality (MOS), standard deviation (S), and confidence interval coefficient (δ).

4.2. Objective Quality Assessment

However, since the relationship between the encoding speed and the quality perceived by users may vary depending on the analysed video [37,38], content providers and streaming platforms should take this into account when preparing content, which should be tailored to the network bandwidth requirements and the type of user equipment while maintaining the highest possible quality. This makes the assessment of video quality an activity that must be repeated over and over again for different types of content. Since subjective assessment in such a case becomes a very demanding and expensive activity, it is necessary to use objective methods. The results obtained from the objective quality evaluation of the videos examined are presented in Figure 5. Each of the metrics shows that as the encoding speed increases, the quality of the presented video increases.

It should be emphasised here that the interpretation of the results expressed by individual metrics differs. Each of these metrics can be used to objectively assess the degree of deterioration of the tested video compared to the reference material, which is often uncompressed video. However, a high rating expressed by a given metric does not always correspond to the high quality of the assessed video material. Sometimes such a high value means a low degree of degradation of the test video compared to the reference material. Some of the presented measures show a direct proportional correlation with the degree of compression of the assessed material in relation to the reference material, while others show an inverse relationship. An important task is to find an appropriate criterion for comparing individual metrics. This criterion may be the correlation coefficient between the values of a given metric and the subjective evaluation of the quality expressed by users [39,40]. A high correlation between the values of objective measures and the results of subjective user assessments would justify the sense of building a quality model. Such a model, taking into account the values of individual metrics, would allow for predicting the quality perceived by users without the need to conduct costly and long-term subjective research. The determination of the correlation coefficient (r) should be performed using an appropriate correlation test. For data with a normal distribution, Pearson’s correlation coefficient (PCC) can be determined for this purpose, but when the data are not normally distributed, a nonparametric measure is most often used, i.e., Spearman’s rank-order correlation coefficient (SROCC) is calculated.

4.3. Objective vs. Subjective Quality Assessment

The normality of the data distributions for the individual metrics was verified using the Shapiro–Wilk test, which in the vast majority of cases resulted in the rejection of the hypothesis of the normality of these distributions at the 5% decision level (p < 0.05). This required the use of Spearman’s rank-order correlation test, which is a non-parametric equivalent of Pearson’s correlation. The authors conducted a correlation analysis. The results showed a strong relationship between the values of individual objective metrics and the quality perceived by users (see

Table 3. Correlations between the values of objective quality metrics and subjective user scores.

	APSNR	Delta	MSAD	MSE	NQI	PSNR	SSIM	VIF	VMAF	VQM
Spearman’s r 1	0.97	−0.64	−0.97	−0.97	0.96	0.97	0.98	0.97	0.98	0.99

¹ Spearman’s r = Spearman’s rank-order correlation coefficient (SROCC).

).

As Table 3 shows, almost all analysed metrics correlate with subjective assessment results at levels higher than 95%, as shown by Spearman’s r coefficients. The VQM metric had the highest correlation coefficient (99%). In contrast, the Delta metric showed the lowest correlation (64%). The authors proposed quality of perception (QoP) models based on each individual metric using linear regression using Formula (13):

QoP = a + b ∗ M_X,

(13)

where QoP is the quality perceived by the users, a and b are the equation coefficients (a is the intercept and b is the slope), and M_X denotes the value of the x metric. In this work, the concept of QoP is used as a narrower version of the concept of quality of experience (QoE) often used in the literature. The determined model parameters for individual metrics are presented in

Table 4. The parameters of the quality models based on individual objective metrics.

MX	APSNR	Delta	MSAD	MSE	NQI	PSNR	SSIM	VIF	VMAF	VQM
x	1	2	3	4	5	6	7	8	9	10
Intercept (a)	−5.58	4.49	6.05	4.39	0.10	−0.33	−13.48	−0.38	−0.38	5.38
Slope (b)	0.25	−4.58	−1.03	−0.03	10.07	1.02	18.53	6.92	0.05	−1.04
R-Square (COD) 1	0.92	0.60	0.90	0.74	0.88	0.92	0.89	0.92	0.91	0.91
RSS 2	4.32	21.28	5.44	13.50	6.37	4.32	5.71	4.27	4.98	4.98
Root-MSE (SD) 3	0.36	0.79	0.40	0.63	0.43	0.36	0.41	0.35	0.38	0.38

¹ coefficient of determination, ² residual sum of squares, ³ root mean square of the error, or residual standard deviation.

. However, it should be noted that the proposed linear regression models, based on individual metrics, are characterised by much lower and more varied values of the coefficient of determination (COD), also called the R-square. COD is a statistical measure to qualify linear regression. This coefficient indicates which percentage of the variation in the response variable, in this case, QoP, is explained by the fitted regression line. The higher the value of the coefficient of determination, the better the model will be.

In order to help interpret the regression model more intuitively, a fitted curves plot is presented in Figure 6 and Figure 7.

Figure 6 and Figure 7 show the relationships between the values of the individual objective metrics and the subjective evaluation values determined on the basis of the linear regression model. The 95% confidence bands show the limits of all possible linear regression lines for the data. In other words, we are 95% confident that we can say that the line of best fit is within the confidence intervals. The prediction band for the desired confidence level (1-α) forms the interval within which 100(1-α)% of all experimental points in a series of measurements are expected to fall. For alpha = 0.05 for the prediction band, we can say with 95% confidence that the expected data point will fall within this range. In other words, if we add one more measurement data point whose independent variable (here the value of a given metric) is within the range of the independent variable of the original data set, there is a 95% chance that the predicted value (here a subjective assessment) will appear in the prediction band. In Figure 6 and Figure 7, it can be seen that in the case of better correlated metrics, i.e., with a higher correlation coefficient and smaller RSS and root MSE (SD) values, the confidence bands and prediction confidence bands are correspondingly narrower. Analysing the values of the determination coefficient included in Table 4, it can be seen that the best results are provided by models based on the APSNR, PSNR, and VIF metrics—in each of these cases, the R-Square is 0.92, with a high correlation coefficient (r = 0.97). Good modelling results for these three metrics were also confirmed by the lowest RSS and MSE (SD) values. An interesting case here is the VQM metric, which shows a very high correlation value with the subjective assessment values but a lower coefficient of determination than the three above-mentioned metrics. The question arises about the possibility of creating a quality model combining the features of existing models and thus obtaining an even better mapping of objective parameters to the subjective perception of video quality by users. A machine learning model was used for this purpose. The set of values of all objective metrics analysed here and the corresponding values of subjective assessments were used to train it.

4.4. ML-Based Results

The authors proposed a linear regression model that shows the relationship between the subjective video quality predicted by the machine learning model, here called the machine learning score (MLS), and the video quality subjectively perceived by users (QoP) expressed on the five-point MOS scale. A plot of the fitted curves is shown in Figure 8. This chart compares video quality prediction results obtained from the machine learning model (MLS) with actual user-perceived quality (QoP) results.

The experimental result presented here was obtained using the following steps: Initially, the ML model was trained based on a set of quality parameters expressed using all objective metrics analysed for the set of test video samples. In the next step, the trained model, based on a set of objective parameters describing another set of video materials, made a subjective quality prediction (MLS). In parallel, the same set of video materials was rated by users who assessed their quality on the MOS scale in subjective tests. In the last step, a comparison of the ratings obtained from the ML model and subjective research was made, and a linear regression model was proposed to describe the relationship between both sets of ratings. Finally, a quality model based on the machine learning score (MLS) was proposed:

QoP = 1.12 ∗ MLS − 0.51,

(14)

where QoP (quality of perception) is the predicted quality perceived by the users, and MLS denotes the value of the machine learning score obtained from the ML model. The parameters determined for the ML model are presented in

Table 5. The parameters of the ML-based linear regression model.

Parameter	Machine Learning Score (MLS)
Spearman’s r	0.99
R-Square (COD)	0.97
RSS	1.82
Root-MSE (SD)	0.23

. As can be seen, the ML-based linear regression model predicts user perception of video quality (QoP) significantly better than the models previously presented based on single classical metrics.

Compared to Table 4, the residual sum of squares and the root mean square of the error are much smaller than the analogous values for each of the classic metrics. Finally, the coefficient of determination in this case is 5% higher than the highest coefficient of determination presented by the above-mentioned classic models. Based on the above research results, the proposed machine learning model can be used to predict the results of subjective image quality assessments depending on the codec used, image resolution, and compression level (see Figure 9).

Figure 9 shows a comparison of the video quality assessment results using two codecs and two spatial resolutions, carried out using two quality assessment methods. The first method involved a subjective assessment carried out by a group of users who rated them on a scale from 1 (worst quality) to 5 (best quality). In this case, the rating of each of the evaluated video samples was averaged (MOS). In the second case, the results of the prediction of perceived quality (MLS), based on a set of objective parameters carried out by a machine learning model, were presented.

5. Discussion

The analysis of the results obtained in the above-described research shows that the proposed machine learning model allows one to predict the results of the subjective assessment of video material based on its measurable objective features in a better way than the existing classic quality models. Classic models describing the relationship between the quality perceived by users and the quality of measurable features of the analysed video materials are usually based on a very limited number of measurable parameters of the evaluated video. One of the reasons for these limitations is the desire to limit the complexity of the final model, which should be simple enough to be suitable for implementation in practice. Otherwise, a model that is too extensive may, on the one hand, lose its generality of application and, on the other hand, not be easy to use. Most of the classic quality models presented in this work, based on single objective metrics, are characterised by a high correlation with the results of subjective research. As shown in Table 3, only the model based on the Delta metric showed a low correlation coefficient, while the remaining metrics correlated with subjective results at a level not lower than 0.96. The highest correlation coefficient (r = 0.99) was obtained for the VQM metric.

It should be noted that the results obtained here do not differ significantly from the values presented in the literature and, in many cases, even exceed them. For example, in Ref. [41], the highest values of the correlation coefficient for identical metrics (different databases were tested) were r_PSNR = 0.89, r_SSIM = 0.92, and r_VIF = 0.98, respectively. While the correlation coefficient presented in Ref. [41] for the VIF metric was slightly better than in our case (r_VIF = 0.97), the other two metrics of our study gave much better results (r_PSNR = 0.97, r_SSIM = 0.98). In Ref. [5], the authors also compared various objective video quality metrics, such as PSNR, SSIM, and VQM, and checked how they correlate with subjective user ratings. According to the authors of the work, objective quality measures show a high correspondence to subjective quality grades, i.e., close to or higher than 0.9. The values presented there are lower than those obtained in our research. The authors of Ref. [4] give a comprehensive survey of the evolution of video quality assessment methods, analysing their characteristics, advantages, and drawbacks. They discuss, among others, relationships between quality of service (QoS) and quality of experience (QoE) for different kinds of models, i.e., engineering approach models (based on modelling video artefacts such as edginess, etc.) and structural similarity-based models. The correlation of the results of the quality assessment carried out using these models with the quality perceived by users during subjective tests ranges from 0.77 to 0.97. In Ref. [42], the authors explore the relationship between image information and visual quality. They proposed a measure of visual information fidelity for the assessment of image quality and validated the performance of the algorithm with an extensive subjective study involving hundreds of images. Depending on the objective metric used, the authors obtained correlation coefficients (SROCC) ranging from 0.82 to 0.95.

In our research, we would like to draw attention to an additional parameter, which is the coefficient of determination (COD), showing how well the model (here a linear regression model based on a specific objective metric) predicts the result (here the video quality perceived by users). Analysing the contents of Table 3 and Table 4, it can be seen that although the values of selected objective metrics show a strong correlation with the results of subjective research, the coefficients of determination for models based on these metrics are lower. For example, the highest correlation coefficients were obtained for the metrics SSIM (r = 0.98) and VQM (r = 0.99), and the quality models built based on these metrics correctly predicted 89% (COD = 0.89) and 91% (COD = 0.91) of the results, respectively.

The results of the video quality prediction obtained thanks to the machine learning model implemented by the authors are characterised not only by a very high correlation coefficient with the subjective evaluation of the users (r = 0.99) but also by a high coefficient of determination at the level of 97% (COD = 0.97).

6. Conclusions

This article contains selected issues and problems related to video quality assessment. Attention was drawn to the need to automate the quality assessment process and use objective methods despite their limitations. Examples of applications of various objective quality assessment metrics described in the literature are shown. As the research results show, none of these metrics are without flaws. It is also difficult to authoritatively say which is better because it often depends on the specific needs and capabilities of the person performing the test. A reasonable evaluation criterion seems to be to determine the extent to which a given metric correlates with the results of the subjective quality assessment made by users using the video service and how well a quality model can be built based on such a metric. The research results described in the literature show that most quality models built on the basis of this type of metric provide results that correlate with the actual results of subjective research at the level of 82–95%. Our ML-based model has a correlation of 99% and a determination coefficient of 0.97. Thus, the main goal of the work, which was to improve the ability to predict the quality perceived by users based on the evaluation results using an objective method, was achieved.

Our research not only improves the accuracy of video quality prediction but also automates the evaluation process, which has important practical applications in streaming services and videoconferencing platforms, significantly improving the user experience and optimising network resources.

Future work to further improve the model will be carried out in terms of expanding the volume of data for training the machine model and searching for even better algorithms.