Web Application for Retrieval-Augmented Generation: Implementation and Testing

Round 1

Reviewer 1 Report

The authors want explore, with their work, the implementation of Retrieval-Augmented Generation (RAG) technology with open-source Large Language Models (LLMs). 

The papaer is enough complete, but a few point need to discuss:

How do you evaluate a rag model? Explain better.

Insert a paragraph to discuss how it can be applied in the field of medicine.

Minor editing of English language required

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

A very opportune study, as the field of AI, specially when approaching LLMs is becoming a de facto solution for businesses, demanding scientifically-supported analysis as done in your paper. Excellent work!

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

This manuscript entitled “Web Application for Retrieval-Augmented Generation: Implementation and Testing”, has been fully reviewed.. It has detailed information with extensive experimental evaluations and a comprehensive review of relevant metrics. Specifically, the incorporation of blockchain technology for assessment result management is innovative and ensures data integrity and reproducibility. I believe this article is well-structured, with a clear division into sections that logically progress from introduction to conclusion. The development of the PaSSER web application is a significant contribution, providing a practical tool for the community to experiment with and expand upon RAG research. While the use of multiple LLMs and a variety of performance metrics provides a thorough evaluation framework, there are some more questions waiting to be solved. Please find the reviewer’s questions and comments below.

1.        The integration of blockchain technology for managing and storing assessment results is innovative, but could you provide more details on the choice of blockchain technology (e.g., Ethereum, Hyperledger) and why it was chosen over traditional databases for this application? Additionally, how do you ensure the scalability and manage the costs associated with blockchain transactions, especially considering the potentially high volume of data generated during testing?

2.        The paper mentions the integration of Mistral:7b, Llama2:7b, and Orca2:7b models. Could you discuss the criteria for selecting these specific models for integration into PaSSER? Are there plans to include more LLMs, especially newer models that might be released, to ensure the application remains up-to-date with advancements in language model technology?

3.        The use of blockchain is highlighted as a key feature for storing assessment results securely. However, blockchain technology can sometimes face scalability issues, leading to increased costs and slower transaction times. How do the authors plan to address these potential scalability and performance issues, especially as the application grows in popularity and the volume of test data increases?

4.        While the paper discusses the use of various NLP metrics to evaluate LLM performance, it does not detail how the success of integrating blockchain technology is measured. Could the authors describe the metrics or indicators they use to assess the effectiveness and efficiency of blockchain integration within the PaSSER application? How do they evaluate whether blockchain technology is adding value to the application in terms of security, transparency, and data integrity?

The English level of the manuscript is generally high, with clear articulation of complex technical concepts and a structured presentation of the research work. However, there is room for minor improvements to enhance readability and ensure the manuscript is accessible to a broader audience, especially this paper has so many technical details.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

1. The problem is not mentioned that how the authors select the method in the abstract.

2. The authors more concentrate the paper organize in the abstract not the results, conclusion and future work.

3. Most the paragraph is not cited in the introduction like line 65 to line 75. The authors need to provide related references for each paragraph.

4. The authors mentioned PaSSER technology in one of the contributions but not provide the literature about it. The authors need to provide the literature of the technologies used in the contribution.

5. Figure 1 PaSSER app framework, need to more elaborate that it is developed or will be developed.

6. Figure 2 and figure 4 title is very long, need to be short and specific.

7. The authors need to provide the block diagram of implementation.

8. The code provides in the lines 352, 373 and 417, need to provide the pseudo code.

9. Equations 4-9, where the authors implemented these equations.

10. The authors provide more equation but not showing the use of these equations.

11. Lines 754 to 760 are not clear.

12. Figure 6-9 are not clear performance. The authors need to redraw which will be clear for reader.

13. The conclusion is very long, need to rewrite to make short and specific with the manuscript.

14. The references should be updated.

Moderate editing of English language required

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

the authors modified the paper as requested

Minor editing of English language required

Reviewer 3 Report

Thanks for the responses to my four questions. 

Proofreading is suggested.

Reviewer 4 Report

The authors response all my comments

 Moderate editing of English language required