Automatic Grammatical Evolution-Based Optimization of Matrix Factorization Algorithm
Round 1
Reviewer 1 Report
In this article, an improvement method where Grammatical Evolution (GE) is used to automatically initialize and optimize the algorithm and some of its settings. This enables the algorithm to produce optimal results without requiring any prior or in-depth knowledge, thus making it possible for an average user to use the system without going through a lengthy initialization phase. I feel that the proposed method is interesting, however, please reconsider this manuscript after MAJOR revision. Some comments listed as below may help you:
1、The abstract part should be briefly describe the compared results to show the good performance of your proposed method.
2、At the part of parameters of the evolution, the authors should briefly show the reason why you chose this three selection percentage,10%, 20% and 30%.
3、Equation. (1) in Page 3 would easily puzzle the readers, add more background knowledge about MF algorithm.
4、Equation. (2) in Page 3 mentions “qTi pu"should it be changed to so that it could be more logical with the last equation?
5、The comparative results in Figs. 1-3 are encouraged to put together to show the effective of the proposed method.
6、Section 5 should be more concise. What’s more, the authors display many experiment results to evolve the update equations via different databases, I recommend adding a summary which may help readers understand your paper.
7、The selected significance levels when testing with different data sets are different in Section 5? It's logical but too verbose, maybe keep the same p-value to test the proposed method is more reasonable.Suggest organizing the results of different data after the same processing into tables for analysis.
8、Check the full paper to avoid errors caused by spelling or grammar problems, such as:
1>. In Page 3, line 88:“These factors are then used to calculate the missing values in the original matrix which, in turn, are used as predicted ratings.”，the word ‘predicted’ should be replaced by ‘predicting’.
2>. In Page 7, lines 227-228,“we use a modus operator to map the codon value to in range of the number of rules for that symbol”, should be changed as,“we use a modus operator to map the codon value to the range of the number of rules for that symbol.”
3>. In Page 8, lines 242-243, “This is due the changed position in the genotype string, same codons select different derivation rules thus resulting in a completely different phenotype.”，word ‘to’ is missing after word ‘due’.
9、The authors are encouraged to briefly analyze the limitations of the proposed method, and provide the plan about the future work.
Author Response
Please find our replies in red below. Also most of the changes in the revised manuscript are also highlited in red.
1、The abstract part should be briefly describe the compared results to show the good performance of your proposed method.
We have expanded the abstract accordingly.
2、At the part of parameters of the evolution, the authors should briefly show the reason why you chose this three selection percentage,10%, 20% and 30%.
The third percentage is the sum of the first two, which was not all that clear from the text. We corrected this issue by adding a remark to that effect.
The first two percentages are—as is a usual practice in evolutionary algorithms—an educated guess supported by some preliminary experiments. We made a short remark to that effect in the paper.
3、Equation. (1) in Page 3 would easily puzzle the readers, add more background knowledge about MF algorithm.
We have added additional explanation to 2.1. in order to better help understant what the equation represents.
4、Equation. (2) in Page 3 mentions “qTi pu"should it be changed to so that it could be more logical with the last equation?
Done.
5、The comparative results in Figs. 1-3 are encouraged to put together to show the effective of the proposed method.
Thank you for the suggestion. It took some effort since we had to re-run some of the experiments to obtain the data points, but all three images were merged, re-scaled and generally improved.
6、Section 5 should be more concise. What’s more, the authors display many experiment results to evolve the update equations via different databases, I recommend adding a summary which may help readers understand your paper.
We had a long discussion on how to make Section 5 shorter and more concise. We agreed, however, that any shortening of the text would be at the price of losing important information and discussion. But indeed the text is somehow lengthy and therefore hard to follow. To remedy this, we have followed the reviewer's advice and inserted a short outline of the presented results at the beginning of Section 5. In addition we also added a summary of results as a final subchapter (5.7).
7、The selected significance levels when testing with different data sets are different in Section 5? It's logical but too verbose, maybe keep the same p-value to test the proposed method is more reasonable.Suggest organizing the results of different data after the same processing into tables for analysis.
The significance level in all chapters was set to α=0.05. The p-values are the result of the statistical test and should be lower than the selected significance level. We added additional explanation to 2.5 to clarify this and added and additional table to 5.7 to better summarize the results.
8、Check the full paper to avoid errors caused by spelling or grammar problems, such as:
The paper was re-read, edited and corrected by a native speaker as per your suggestion.
9、The authors are encouraged to briefly analyze the limitations of the proposed method, and provide the plan about the future work.
Added to conclusion. Thank you for your suggestion on how to improve our article.
Reviewer 2 Report
This paper proposes Grammatical Evolution to automatically initialize and optimize the settings of Matrix Factorization algorithm used to develop a Recommendation System. The idea is interesting and has important practical implications. I am only concerned with the following missing aspects of the paper
1- A review of the previous works on using evolutionary computation methods to optimize the recommendation systems is not provided
2- A comparison of the obtained research results with some related evolutionary computation approaches or baseline approaches is missing
3- I think adding a stand alone section to describe how Grammatical Evolution is applied to optimize Matrix Factorization would improve the paper readability
Author Response
Please find our answers in red below. We have also highlited all the modifications in the revised article in red to help track all the changes. The only exception is chapter 3.3. which is completely new.
1- A review of the previous works on using evolutionary computation methods to optimize the recommendation systems is not provided
Thank you for your suggestion - we added several recent work from this field into the introduction and revised it accordingly.
2- A comparison of the obtained research results with some related evolutionary computation approaches or baseline approaches is missing
This was a bit trickier since a lot of similar work (i.e. evolutionary non-negative matrix factorization) focused on non-recommender datasets. But we found two cases where they used the same dataset (references 30 and 33) and included them in table 3 for comparison.
3- I think adding a stand alone section to describe how Grammatical Evolution is applied to optimize Matrix Factorization would improve the paper readability
Wrote an additional chapter (3.3) that shows how a chromosome results in a new latent factor calculation function. In addition added Algorithm 2 to section 3.4 that shows how these new functions are used.
Thank you for your comments on how we could improve our article!
Reviewer 3 Report
The authors proposed a new "Automatic Grammatical Evolution-based
Optimization of Matrix Factorization Algorithm".
The paper is well written and up to standard. The simulation results support the conclusion of the paper. However, in order to further enhance the quality of the paper. I have the following comments (*optional):
- In section 1, Introduction, the author could explain more about the reasons why "Grammatical Evolution" instead of other metaheuristic optimization algorithms (e.g. GA, PSO, etc.) is selected for this problem.
- It may be useful to include a table in section 5 to compare the performances of the proposed algorithms with other existing algorithms. The table could either compare the performances of the algorithms in qualitative or quantitative ways.
- The authors could highlight more future applications of the proposed algorithms in section 6.
Author Response
Please find our answers in red below. We have also highlited all the modifications in the revised article in red to help track all the changes. The only exception is chapter 3.3. which is completely new.
- In section 1, Introduction, the author could explain more about the reasons why "Grammatical Evolution" instead of other metaheuristic optimization algorithms (e.g. GA, PSO, etc.) is selected for this problem.
Done - introduction was expanded to explain that we wanted to evolve new function that can be used to calculate latent factors. The suggested algorithms (GA and PSO) focus on optimizing existing parameters. Genetic Programming can be used to synthesize new expressions but suffers from certain limitations in it's grammar due to requirement of all used function being inter-compatible. Grammatical Evolution solves this by allowing the use of more specific grammar, which is why we chose to use it.
- It may be useful to include a table in section 5 to compare the performances of the proposed algorithms with other existing algorithms. The table could either compare the performances of the algorithms in qualitative or quantitative ways.
Done - added an additional chapter (5.7) which serves as a summary of our results and also features a summary in table form. Thank you!
- The authors could highlight more future applications of the proposed algorithms in section 6.
Added an additional paragraph that both show the weak points of our approach and present future improvements and applications.
Thank you for your suggestions on how to improve our paper.
Reviewer 4 Report
This is a nice article making a contribution. Results are displayed clearly and they are tehccnislly sound. I only have some minor comments, reported below.
There is a long introduction where RSs are described. I understand this is needed to explain why is important to optimise the matrix factorisation algorithm. However, information on how heuristics are successfully improved by overcoming initialisation problems is missing (and this is more relevant to the paper, as it presents an improved use of an optimisation heuristic). You may want to find an appropriate place for a paragraph describing how successful, and often simple, procedures can boost performances such as the initialisation method by Poikolaninen et al (Cluster-based population initialization for differential evolution frameworks) and other simple frameworks.
Please perform a spelling check to remove typos and other English language details, see eg. emerging as one of automatic optimization approaches —> emerging as one of the automatic optimization approaches
Make sure that you clarify how those parameter settings for the optimisation algorithms were found to be optimal and explain how solutions generated during the search are kept feasible.
Author Response
Please find our answers in red below. We have also highlited all the modifications in the revised article in red to help track all the changes. The only exception is chapter 3.3. which is completely new.
There is a long introduction where RSs are described. I understand this is needed to explain why is important to optimise the matrix factorisation algorithm. However, information on how heuristics are successfully improved by overcoming initialisation problems is missing (and this is more relevant to the paper, as it presents an improved use of an optimisation heuristic). You may want to find an appropriate place for a paragraph describing how successful, and often simple, procedures can boost performances such as the initialisation method by Poikolaninen et al (Cluster-based population initialization for differential evolution frameworks) and other simple frameworks.
Done - added to chapter 3.2. The approach proposed in the suggested article is definitely interesting and we are already considering implementing it in our framework for future use. In the presented article the ramped half and half already produced good results (we were able to find a solution in each run) which is why we did not look to further improve the initialization stage.
Please perform a spelling check to remove typos and other English language details, see eg. emerging as one of automatic optimization approaches —> emerging as one of the automatic optimization approaches
The article was re-edited and proofread by a native speaker as per your suggestion.
Make sure that you clarify how those parameter settings for the optimisation algorithms were found to be optimal and explain how solutions generated during the search are kept feasible.
We have added an explanation that most of the GE settings were found to be optimal via prior experience and experiments and added references to those experiments. Also expanded 3.4. to show that we used pre-processing to determine if the individuals chromosome would be unfeasible and how we reacted in such a case.
Thank you for you suggestion on how to improve our paper!
Round 2
Reviewer 1 Report
The author has made corresponding modifications according to the previous suggestions. however, please accept the manuscript after MINOR revision. There are some advice to your figures:
<1>The curve color in fig1 to 3 is not displayed.
<2>The fig 2-3 are not very intuitive. It is suggested to change linearity to the same continuous curves, which are distinguished only by colors.
<3> The trends of Convergence of the RMSE value using equations (9) in fig 2-3 are slightly different from that before modification.
Author Response
Thank you for you comments. I have re-made the images in color and corrected the references in text. All images are now solid color lines.
The difference in fig 2-3 comes from the fact that all three curves are now scaled into a single image. The original (split) figures did not use the same scale which resulted in program 9's curve being a bit more noticeable.
<1>The curve color in fig1 to 3 is not displayed.
<2>The fig 2-3 are not very intuitive. It is suggested to change linearity to the same continuous curves, which are distinguished only by colors.
<3> The trends of Convergence of the RMSE value using equations (9) in fig 2-3 are slightly different from that before modification.
Reviewer 2 Report
The authors have addressed my concern adequaely
Author Response
Thank you for your comments!
