Semi-Supervised Classification with A*: A Case Study on Electronic Invoicing
Round 1
Reviewer 1 Report
In this manuscript, Bernardo Panichi and Alessandro Lazzeri present the development of a semi-supervised framework coupled with A star graph search for improved classification. It's concluded that this algorithm showed a consistent increase in accuracy between 1% and 4% and a reduction of the discard rate of instances from 39% to 14 % on electronic invoices.
Overall, this is a well written and clearly detailed article, especially on the method. I appreciate the importance of machine learning techniques to support the accountant. The contribution of this paper also lies in being able to use both labelled and unlabeled data for learning in a semi-supervised approach. The novelty of using A star graph search for error correction of pseudo-labels is interesting. However, there may be room for improvement in this manuscript on study design and algorithm verification. A few suggestions, mainly concerning critical evaluation, are given below.
Major Concerns:
1. First, the description of methodological approach is very clear and well written, but there is some concerns about the limited experiments and analysis of the results.
a) I’m curious about why there is no pre-reserved independent dataset. The 14% discarded data may be suitable as an independent test dataset with manual verification to certify the validity of the labels proposed by this work.
b) It’s recommended to add an ablation experiment on A star graph search to prove its effectiveness on overall performance. I agree that the discard rate is significantly smaller than the baseline framework. However, how does classification accuracy change after the introduction uncertain labels?
c) What is the performance of human inferring the category of the invoice entry, when “placed side by side with the work of an accountant”? It may be better to provide a reference value, e.g accuracy of the artificial process, to obtain the conclusion that "at least comparable to a human result".
d) It’s recommended that data presented in a plot or chart shows data distribution clearly. It may be better to overlay the corresponding data points (as dot plots) with ox-and-whisker plots.
2. Second, there may be confusion when matching "invoice lines and the corresponding accounting accounts" with the following questions.
a) I agree with the authors that “machine learning techniques however, we often run into low quality data for the training”, therefore, the data cleaning and pre-processing are particularly important. Considering “many records are usually discarded during the preprocessing”, is the applied method optimal for generating labels for training? Because the algorithm has limitations on the invoice which” consists of many rows (sometimes even in the hundreds)”. 39% of the invoices have not been solved by the knapsack problem. Is there any other possible algorithm to reduce the discard rate, and further replace the subsequent pseudo-labeling and correction process (by A* search).
b) Considering the rigor of accounting legislation, we have reason to believe that there must be a solution to the matching problem in each invoice. Therefore, incomplete supervision may not be necessary to some extent. Of course, the time and space complexity of the calculation should also be considered with detailed analysis.
c) How does the algorithm handle the addition of Categorical data in data preprocessing?
3. Third, there are some concerns about the training stage, including both pre-training and the final training.
a) Considering “the accuracy of which will be limited to an extent equal to the number of input-output pairs that we have been able to reconstruct so far”. Is there a change in accuracy due to differences in the size of the training sample? Is it related to sample distribution and continuity of the starting dataset D?
b) The current 61% label rate may be too high for a semi-supervised framework. What is the appropriate label rate in this case of electronic invoicing? This and the previous questions can be considered simultaneously.
c) It is also recommended to reconsider the sample distribution during the tenfold cross validation, and the performance of different classifiers. For instance, why is there not much improvement in SVM performance with an increase in training samples (augmentation of the supervised set)?
4. I am minimally concerned with methods for validating pseudo labels via A star search.
a) I appreciate the importance of “eliminating the uncertainty on the pseudo-labels proposed by the pre-trained classifier”. Readers may be curious about the characteristics of the 14% discarded data that cannot be modified by A * search. Are they with a particularly large number of rows N and M, or is there something wrong with the number value of c and b? It’s recommended to provide some insights through analysis and discussion on the discarded data.
b) It’s recommended to train a final classifier with the augmented dataset without validating these pseudo-labels with A * search. Then, check for improvements in the A * search.
c) I am not sure what the meaning of the E[] in Equation 3 is. Is this a mathematical expectation? There may be errors in this equation, because the desired search depth should be less with higher accuracy.
d) Is there any other algorithm more suitable for further reducing the discard rate than A* search? Is there a better heuristic for this problem?
5. There are some suggestions on the future works.
a) The underperforming invoices may be subject to repeated training and validated.
b) Active learning can be considered in future work, which manually annotates the 14% discarded sample that has not been correctly classified.
Minor Concerns:
1. Line 21-22, "a lot of problems due to manual input of the invoices in the systems or to OCR limits have been exceeded" is difficult to read.
2. Line 69, “particularly interesting in two literatures"?
3. Line 74-78, it is too long to read and understand, please have a check.
4. Line 186, “Each entry of this state vector can take as its value one of the M account codes in S" is a bit difficult to read.
5. Line 227-228, "… the time required to A* to reach a goal node" maybe better in “…, the time required for A* to reach a goal node”.
6. Figure 6, are the vectors concatenated (+) or multiplied (*)?
7. Line 261, what does “problem 1” mean? This was also the case in Line 280, “referring to 3”.
8. Line 268, please have a check at “were excluded a priori”.
9. Line 299, “in the most important performance table of our work 4” seems to have room for improvement.
10. Line 312, should "utilizing invoice amounts" be "utilizing numerical values c and b"?
11. The references 11-12, 17-18 should be improved.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Dear authors, for me is a pleasure to review this manuscript, but I recommend you improve some point of views to enhance the quality and structure of thsi manuscript:
1º. The title is very long for search engines.
2º. The abstract section has to provide: the main objectives, methods, findings, and the new contribution of this paper to literature review in this topic.
3º. This study cannot tackle the main objectives and results with old references. Authors need to be aware that you are tacking a topic related to the new technologies and these are changing every day. For instance, to included references in the year: 1991, 1995, 1996, 1975, 1998, 2006, 2011... this is not realistic, we are in 2023 and readers need to know the last information and studies related to this topic and scope.
4º. I inserted some words of your title and Google Scholar showed me these results:
Liu, L., Wang, B., He, X., Wang, J., Zheng, Y., & Yan, Y. (2021, August). Establishing an electronic invoice management platform based on information system. In Journal of Physics: Conference Series (Vol. 2004, No. 1, p. 012013). IOP Publishing.
Cedillo, P., García, A., Cárdenas, J. D., & Bermeo, A. (2018, April). A systematic literature review of electronic invoicing, platforms and notification systems. In 2018 International Conference on eDemocracy & eGovernment (ICEDEG) (pp. 150-157). IEEE.
Poel, K., Marneffe, W., & Vanlaer, W. (2016). Assessing the electronic invoicing potential for private sector firms in Belgium.
Koch, B. (2017). E-invoicing/E-billing. Significant market transition lies ahead. Billentis.
5º. Authors need to included more updated authors in this paper because sometimes it seems to detect plagiarism. For example, authors wrote this paragraph: One of the tasks that benefits more of the digitization process is the bookkeeping of the invoices in the general book. In fact, a lot of problems due to manual input of the invoices in the systems or to OCR limits have been exceeded. However, there are still many issues that the accountants have to face. For example, the bookkeeper records each entries of the invoice according to a precise system of categories. This system is very flexible and, with some degree of freedom, each firm can customize the set of categories according to their needs. This implies that the same entry may generate different recordings in different firms. Similarly, the same invoice issuer may provide several services or products with different invoice reasons. While it is trivial (and time consuming) for a human to read the reason and infer the category of the entry, this task may be not so easy for an automatic system. Finally, the accounting legislation is continuously evolving to keep the pace with the economy, and this changes impact the logic of the process.
Who authors said and supported this sentence. Authors must support sentences with other studies. In the entire paper, I saw a lot of paragraphs like this.
6.º Introduction section seems the literature review section included Figures 1. It is not correct. The introduction section must stage the main objectives and research questions, the main gaps in this topic, why authors worked in this study, and a narrative which is related to this research and authors were interested on work in this paper. I did not see it in this section.
7º. Indeed, authors speak about the methodology section in the introduction section and included Figure 1. This information and Figure should be added in Methodology section.
8º. Literature review need to be structured according to keywords and subsections, and supported by updated authors. Real examples from companies must be implemented like images and descriptions.
9º. Problem description section, authors wrote other main objective: "the aim of this work is to provide an accountant with an artificial intelligence system capable of understanding the content of the various rows that make up an invoice and associate the appropriate account code with each of them" I did not see this goal in the introduction section, in fact, artificial intelligence was not included as a keyword. Furthermore, Authors presents a different goal in the abstract section: "The proposed method improves the performance of an invoice entry classifier in a semi-supervised framework by combining the classifier with the A* graph search algorithm", Obviously, this "study" mix a lot of concepts and information which it is really difficult to understand it. This manuscript need to be considerably improved to be published in all terms.
10º. Authors did not explain in details the methodology section, and authors did not support their methods with other recent studies. Moreover, authors wrote their own findings but these were not compared with other studies like I showed previously from Google Scholar database, Why? There is not a discussion and debate in results and conclusion and discussion sections. I did not understand it.
11º The conclusion section and discussion is an abstract of the literature review, In addition, authors wrote this sentence: Our study demonstrates the effectiveness of the proposed semi-supervised framework using A* graph search in transforming a fully supervised framework into a semi-supervised one, " I really did not see this information.
12º. Conclusion section is very short and authors add nothing new to the literature review in this topic, and 5.2 and 5.3 sections are very long. Overall, this manuscript tries to seem a report from a private company, and this does not provide the quality to be published in this journal.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
Dear author,
We extend our gratitude to you for submitting your manuscript for review. After undertaking a comprehensive analysis, we have discerned several areas of improvement that could be made to enhance the clarity, structure, and technical depth of the paper. We have provided a detailed list of comments and recommendations below:
The abstract is lacking in clarity in its present form. It would be advantageous to the reader if the core problem and the unique contribution of the proposed solution were succinctly highlighted.
We suggest revising the structure to more distinctly present the problem, method, and results.
The introduction appears to blend elements that might be better separated. We suggest differentiating between the general introduction and the related work sections for improved structure and readability.
Some technical aspects mentioned in the introduction are not clear. Consider providing more context or a brief explanation of technical terms or concepts to guide the reader better.
We recommend incorporating a comparative table to highlight the advantages, disadvantages, limitations, and research gaps of existing methods. This will provide a quick visual summary and clearly position your contribution in the broader research landscape.
The description of your methodological approach lacks technical clarity. To aid understanding, consider adding a step-by-step breakdown of the algorithm or a flow chart detailing the proposed approach.
It would also benefit the reader if the integration of the A* graph search algorithm with the semi-supervised framework was elucidated more comprehensively.
The presented results seem insufficient to justify your claims. Consider adding more detailed results, possibly with varied datasets or different experimental setups.
It might also be beneficial to compare and contrast your results with existing methods in a tabular or graphical format.
Please separate and distinctly demarcate the conclusion and future work sections. This provides better structure and clarity for readers to understand the contributions and future directions of the research.
I've noticed that many of the references cited are quite old. Given the topic's relevance and rapid advancements, I recommend incorporating more recent papers, especially those related to "A Semi-Supervised Framework Coupled with A* Graph Search for Improved Classification."
Throughout the paper, the quality of English needs significant improvement. Consider proofreading the paper for grammatical errors, sentence structure, and clarity or engaging a professional editing service.
while the topic is intriguing and holds promise, the paper requires a significant overhaul in terms of clarity, structure, and depth. Implementing the suggested changes will greatly enhance its value and appeal to the target audience.
We eagerly await the revised version of your manuscript.
Throughout the paper, the quality of English needs significant improvement. Consider proofreading the paper for grammatical errors, sentence structure, and clarity or engaging a professional editing service.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
Dear authors,
I have reviewed the manuscript again, and I consider that this has notably improved because authors have modified some mistakes according to reviewers' suggestions.
Good luck!
Reviewer 3 Report
The manuscript offers a novel approach to electronic invoicing classification, utilizing a semi-supervised method with A*. The methodology's presentation is clear, and its adaptation to the specific case is commendable. The experiments show promising results, validating the efficacy of the proposed method. However, the authors could strengthen the paper by comparing with contemporary semi-supervised techniques in terms of efficiency and accuracy. Furthermore, a deeper insight into the algorithm's scalability, especially in high-dimensional spaces, would be beneficial for potential applications. Overall, the paper makes a significant contribution awould fit well in our publication.
