Video Quality Analysis: Steps towards Unifying Full and No Reference Cases

Round 1

Reviewer 1 Report

Dear authors,

I have finished the review of your paper.

I have the following recommendations. 

1. Please provide a summary of the models and parameters employed in the ML part. Be specific in info like number of layers, activation functions, kernels, and hyperparameters. 

2. Improve the quality of the figures. In some cases, they are too small and in many cases they were taken of the Word Processor (the red marks of the spell and grammar check are visible).

Author Response

Many thanks for the kind review. Your comments have been incorporated, which has strengthened the paper.

1. Intro section now outlines the rest of the paper, as well as sets the goals of the paper.

2. The methods have been described in more detail, and fig. 7 has been added, giving details of an example NN model used.

3. The results have been made clearer, and several of the figures have been converted to Latex. However, because of the narrow format of MDPI publications (only about 2/3rds of the page is used), and the great density of numbers needed for figures 4-6, I have kept them as image files, but which are still sharp upon magnification. I spreadsheet with the data is provided if the presentation of these figures can be improved.

4. The Results, Discussion, and Conclusions sections summarize what has been achieved in the paper, as well as what remains for future work.

Best.

Reviewer 2 Report

standards-1772880_peer review_comments

The paper investigates some variants of the popular VMAF video quality assessment algorithm for the FR case, using both support vector regression and feedforward neural networks. they extend it to the NR case, using some different features but similar learning, to develop a partially unified framework for VQA. When fully trained, FR algorithms such as VMAF perform very well on test datasets, reaching 90%+ match in PCC and SRCC; but for predicting performance in the wild, we train/test from scratch for each database. With an 80/20 train/test split, they still achieve about 90% performance on average in both PCC and SRCC, with up to 7-9% gains over VMAF, using an improved motion feature and better regression. Moreover, they even get decent performance (about 75%) if they ignore the reference, treating FR as NR, partly justifying their attempts at unification. In the true NR case, typically with amateur user-generated data, they avail of many more features, but reduce complexity vs. recent algorithms VIDEVAL, RAPIQUE, yet achieve performance within 3-5%. Moreover, they develop a method to analyze the saliency of features, and conclude that for both VIDEVAL and RAPIQUE, a small subset of their features provide the bulk of the performance. they also touch upon the current best NR methods: MDT-VSFA, and PVQ that break above 80% performance. In short, they find encouraging improvements in trainability in FR, while constraining training complexity against leading methods in NR, elucidating the saliency of features for feature selection.

The following are my concern about the manuscript:

Ø  The paper does not clearly explain the design methodology of the proposed method because it looks like a review paper but without the structure of the review article. I suggest the authors to clearly explain their design methodology as their claim contributions and objectives were not properly described

Ø  Please clearly identify the novelty of the work and differentiate it from the large body of research that exist with sufficient references

Ø  Define the abbreviation in the first time they appear

Ø  An introduction lacks the background study of the subject as well as problem statement of the study and the gap the study intended to fill and address

Ø  Some commentary on deficiencies of related work might be beneficial

Ø  You should include a block diagram or flow chart to depicts the procedure of the proposed model

Ø  Many variables in the equations were not defined such and some are not explained in the text

Ø  Most of the Figures such as Figure 9 were not clearly presented. Clear figures should be replaced

Ø  All the sections of the paper need to be revised. The methodology and the contribution was not clearly presented. Therefore, some questions remain about the experimental approach that should be expounded in the paper. The paper is not logically written and it is difficult to follow and understand in many part of the paper. The structure of the paper is rather confusing.

Ø  Extensive editing of English language and style required in this paper as there are many grammatical and syntax error in the paper

Ø  The references were not arranging in order. They should also conform to the standard MDPI format. I advise the authors to check the MDPI format

Author Response

Thank you for your review, which we have tried to respond to.

1. The Intro section has been expanded to include an overview of the paper, as well as its key contributions.

2. We believe all cited references are relevant to the undertaking of the paper.

3. We believe the research design is appropriate.

4. The exposition of the method, as well of the results, has been improved, fig. 7 has been added with further details, and some of the text and figures have been improved to indicate the contributions of this paper, as well as what remains open. Two figures have been cleaned up and converted to Latex, while figures 4-6 are quite dense, and appear reasonably presented as figures.

5. We believe the updated conclusions are supported by the results.

Round 2

Reviewer 1 Report

Dear authors,

I have finished the review of this revised version. You have addressed the issues of the first version of your paper. I have no more concerns about this work.