An Algorithm for Making Regime-Changing Markov Decisions
Round 1
Reviewer 1 Report
I would suggest to rephrase the abstract in the sense that you do not address or make comparisons with robust stochastic control. Instead, please consider to highlight the reformulation as a convex switching problem.
Author Response
Thank you very much for your help. All recommendations have been addressed carefully, which helped improving the work.Reviewer 2 Report
- Please explain carefully how the practical example of $6 is imbedded into the formal scheme.
- P4, (14) the set of unit vectors in $R^d$ is not finite. If you mean the coordinate vectors, why the transition kernel preserves them?
- P5, (16) $e=\frac{V(y){\hat s}}{||V(y){\hat s}||}$ is a unit vector. Then $\Gamma^T e$ is a unit vector for any orthogonal matrix $\Gamma$. However, it is not true for some stochastic matrices $\Gamma$ as defined.
- P7, (27)-(30) Please provide either exact references or proofs.
Author Response
Thank you very much for your help. All recommendations have been addressed carefully, which helped improving the work
- Please explain carefully how the practical example of $6 is imbedded into the formal scheme.
I have added a section 6 Algorithm implementation... . At the end of this section, there is an explanation how convex switching and indirect observations are connected, with a diagram showing stylized implementation followed in the illustration.
- P4, (14) the set of unit vectors in $R^d$ is not finite. If you mean the coordinate vectors, why the transition kernel preserves them?
Yes, orthonormal basis vectors, (14) changed accordingly
- P5, (16) $e=\frac{V(y){\hat s}}{||V(y){\hat s}||}$ is a unit vector. Then $\Gamma^T e$ is a unit vector for any orthogonal matrix $\Gamma$. However, it is not true for some stochastic matrices $\Gamma$ as defined.
I included a Remark with thank as footnote
- P7, (27)-(30) Please provide either exact references or proofs.
References are given
Reviewer 3 Report
The authors focus their study on the processes of optimal sequential decision-making which are optimized by using the Markov decision theory in cases of incomplete and uncertain information. The authors introduce a new approach that reformulates the original problem and they present numerical algorithms in order to solve it.
The manuscript is overall well written and easy to follow and the authors have well thought out their main contributions. The provided theoretical analysis is concrete, complete, and correct and the authors have provided all the intermediate derivations in order to enable the reader to easily follow it.
The provided numerical results are rich in order to show the pure operation and the performance of the proposed framework and the authors have clearly identified and quantified its main benefits.
The authors should consider the following suggestions provided by the reviewer in order to improve the scientific depth of their manuscript, as well as they should address the following comments in order to improve the quality of presentation of their manuscript.
Initially, in Section 1, the authors should discuss also game theoretic approached, such as Tsiropoulou, E.E., et al. "Uplink Power Control in QoS-aware Multi-Service CDMA Wireless Networks." J. Commun. 4.9 (2009): 654-668, and machine learning approaches, such as Huang, Xin-Lin, Xiaomin Ma, and Fei Hu. "Machine learning and intelligent communications." Mobile Networks and Applications 23.1 (2018): 68-70, that have been used in the literature in order to deal with the problem of decision-making under incomplete and uncertain information.
The authors should include also a table summarizing the main notation that has been used in the manuscript and provide the corresponding units of all the involved metrics.
Furthermore, the authors should include an additional subsection providing the theoretical analysis of the computational complexity of the proposed framework and clarifying the information and the control flow within a realistic system and corresponding implementation.
Based on the previous comment, the authors should provide some indicative numerical results quantifying the computational complexity of the proposed framework in terms of execution time in order to be implemented.
Finally, the overall manuscript should be checked for typos, syntax, and grammar errors in order to improve the quality of its presentation.
Author Response
Initially, in Section 1, the authors should discuss also game theoretic approached, such as Tsiropoulou, E.E., et al. "Uplink Power Control in QoS-aware Multi-Service CDMA Wireless Networks." J. Commun. 4.9 (2009): 654-668, and machine learning approaches, such as Huang, Xin-Lin, Xiaomin Ma, and Fei Hu. "Machine learning and intelligent communications." Mobile Networks and Applications 23.1 (2018): 68-70, that have been used in the literature in order to deal with the problem of decision-making under incomplete and uncertain information.
->Yes, thank you very much. Achieving optimal decisions through game is important and the references are included now
The authors should include also a table summarizing the main notation that has been used in the manuscript and provide the corresponding units of all the involved metrics.
->Table included
Furthermore, the authors should include an additional subsection providing the theoretical analysis of the computational complexity of the proposed framework and clarifying the information and the control flow within a realistic system and corresponding implementation.
-> Section 6 is added and diagram of control flow is presented
Based on the previous comment, the authors should provide some indicative numerical results quantifying the computational complexity of the proposed framework in terms of execution time in order to be implemented.
-> Reference to more advanced implementations are given whose performance analysis is published
Finally, the overall manuscript should be checked for typos, syntax, and grammar errors in order to improve the quality of its presentation.
-> Proofreading completed
Round 2
Reviewer 2 Report
- P.12-13 the classical results of Faustmann, the well-known results of Faustmann Pls give exact references
- P.8 sufficiently reach function family Do you mean a rich function family?
- P.2 are acting a in a game Del: a
