A Novel Tsetlin Machine with Enhanced Generalization

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

1. Although Tsetlin Machine provides some explanatory advantages through logical expressions, it is pointed out in the literature that these generated rules may not be intuitive and easy to understand in some cases, especially in the application fields that need high explanatory power.

2. It is mentioned in the literature that the training process of Tsetlin Machine is inefficient when dealing with large-scale data sets. How to solve it?

3. There are some parameters that need to be selected manually in 3.Tsetlin Machine, such as the number of states of Tsetlin Automata and the parameters fed back by Type I/II. The selection of these parameters may have an important impact on the performance of the model, but how to choose the appropriate parameters?

4.Tsetlin Machine is mainly designed to process binary features, but it is difficult to process continuous features, and it usually needs discretization or conversion, which may lead to information loss or increased computational complexity.

5. It is mentioned in the literature that although Tsetlin Machine performs well in classification and regression tasks, its application in other tasks (such as clustering and sequence modeling) is still limited.

6. Terms such as "regularized tsetlin machine (REGTM)" appear many times in the article, and should be abbreviated when used for the first time, and then the abbreviated form should be used uniformly.

7. The content of the picture is not clear.

8. The way of quoting literature is scattered, and too many quotations in some sentences affect the fluency of the text.

Comments on the Quality of English Language

Extensive editing of English language required.

Author Response

Comment 1: Although Tsetlin Machine provides some explanatory advantages through logical expressions, it is pointed out in the literature that these generated rules may not be intuitive and easy to understand in some cases, especially in the application fields that need high explanatory power.

Response 1: Thank you for highlighting this concern. You are correct that while Tsetlin Machines (TMs) can offer advantages in terms of generating logical expressions, the interpretability of these rules can vary. In some cases, especially in domains requiring high explanatory power, the rules produced may not always be as intuitive as desired. The interpretabilitry of Tsetlin Machines has been widely studied in the literature. Tsetlin Machines implements clauses and Boolean logic that can simplify rules and give more control over how to make the model interpretable.

Comment 2: It is mentioned in the literature that the training process of Tsetlin Machine is inefficient when dealing with large-scale data sets. How to solve it?

Response 2: Thank you for pointing out the concern regarding the training inefficiency of Tsetlin Machines with large-scale datasets. Increasing the efficiency of Tsetlin Machine training process to deal with large-scale data sets is ongoing research. The implementation detailed for improving the efficiency is beyond the scope of our paper as we are more focused on exploring the implementation of regularizers. However, one way of increasing the efficiency was proposed using Sparse Tsetlin Machines [1].

[1]. Østby, Sebastian, Tobias M. Brambo, and Sondre Glimsdal. "The Sparse Tsetlin Machine: Sparse Representation with Active Literals." arXiv preprint arXiv:2405.02375 (2024)

Comment 3: There are some parameters that need to be selected manually in 3.Tsetlin Machine, such as the number of states of Tsetlin Automata and the parameters fed back by Type I/II. The selection of these parameters may have an important impact on the performance of the model, but how to choose the appropriate parameters?

Response 3: Thank you for the comment. Currently, these parameters are selected by running experiments with different values and selecting the parameter with the best accuracy. Even though selecting these parameters is important, our method does not focus on the selection of these parameters and currently we are not aware of any method that can select these parameters automatically. However, these parameters can be found grid search or some other form of optimization techniques, details of which can be found in [1] and [2].

[1] Granmo, Ole-Christoffer. "The Tsetlin Machine--A Game Theoretic Bandit Driven Approach to Optimal Pattern Recognition with Propositional Logic." arXiv preprint arXiv:1804.01508 (2018)

[2] Abeyrathna, K. Darshana, Ole-Christoffer Granmo, and Morten Goodwin. "Extending the tsetlin machine with integer-weighted clauses for increased interpretability." IEEE Access 9 (2021): 8233-8248

Comment 4: Tsetlin Machine is mainly designed to process binary features, but it is difficult to process continuous features, and it usually needs discretization or conversion, which may lead to information loss or increased computational complexity.

Response 4: You’re right that Tsetlin Machines (TMs) are primarily designed for binary features, which can present challenges when dealing with continuous features. There is research being done to design algorithms for binarization techniques in Tsetlin Machines, e.g. in [1].

[1] Mathisen, Erik, and Halvor S. Smørvik. Analysis of binarization techniques and Tsetlin machine architectures targeting image classification. MS thesis. University of Agder, 2020.

[1] Mathisen, Erik, and Halvor S. Smørvik. Analysis of binarization techniques and Tsetlin machine architectures targeting image classification. MS thesis. University of Agder, 2020.

Comment 5: It is mentioned in the literature that although Tsetlin Machine performs well in classification and regression tasks, its application in other tasks (such as clustering and sequence modeling) is still limited.

Response 5: Thank you for pointing out the application of Tsetlin Machines. We agree that currently the focus of Tsetlin Machines is mainly on classification and regression tasks. Research on application of Tsetlin Machine in other tasks is on-going. Our work on regularizers aims to be step towards application of Tsetlin Machines in other domains and tasks.

Comment 6: Terms such as "regularized tsetlin machine (REGTM)" appear many times in the article, and should be abbreviated when used for the first time, and then the abbreviated form should be used uniformly.

Response 6: Thank you for pointing it out. We will fix this error in the update version of the manuscript.

Comment 7: The content of the picture is not clear.

Response 7: Thank you for the comment. We will fix the content of the pictures.

Comment 8: The way of quoting literature is scattered, and too many quotations in some sentences affect the fluency of the text.

Response 8: Thank you for the comment. We will fix the appearance of too many quotations in sentences.

 

Reviewer 2 Report

Comments and Suggestions for Authors

The article proposes a variant of the Tsetlin Machine (TM) called the Regularized Tsetlin Machine (RegTM), which incorporates regularization techniques to enhance the generalization capabilities. The RegTM uses two types of regularizers: the Moving Average Regularizer (MAR) and the Weighted Regularizer (WER). Additionally, the paper uses the Sigmoid function as an alternative to the unit-step function for differentiability. The experiments are conducted on toy benchmark datasets like MNIST and CIFAR10. 

1. Limited experiments. The experiments are primarily conducted on two tiny-scale benchmark datasets. The generalizability to other datasets or real-world applications has not been thoroughly explored.

2. Lack of theoretical analysis: There is a lack of theoretical analysis to support why the proposed regularizers and the Sigmoid function would improve generalization beyond the empirical results presented.

3. The authors should discuss some related works about interpretable machines like KAN [1,2].

[1]Liu, Ziming, et al. "Kan: Kolmogorov-arnold networks." arXiv preprint arXiv:2404.19756 (2024).

[2]Li, Chenxin, et al. "U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation." arXiv preprint arXiv:2406.02918 (2024).

4. While regularization aims to reduce overfitting, the paper does not extensively discuss how the choice of regularizer parameters might lead to overfitting if not correctly tuned.

5. Can this method be deployed for fine-grained tasks like segmentation and detection, which may be more practical?

Comments on the Quality of English Language

The motivation in the introduction section about the RegTM can be elaborated.

Author Response

Comment 1: Limited experiments. The experiments are primarily conducted on two tiny-scale benchmark datasets. The generalizability to other datasets or real-world applications has not been thoroughly explored.

Response 1: Thank you for your comment. We agree, we conduct experiments on benchmark dataset that has been used in multiple research papers for testing the Tsetlin Machine algorithms and its variants. In the future, we will test our algorithm on different tasks, like few-shot learning and in other data domains like natural language processing.

Comment 2: Lack of theoretical analysis: There is a lack of theoretical analysis to support why the proposed regularizers and the Sigmoid function would improve generalization beyond the empirical results presented.

Response 2: Thank you for the comment. Regularizers improve model performance by improving generalization and mitigating overfitting. Our primary goal here was to assess the real-world performance of regularizers in Tsetlin Machine, and experimental analysis provided direct insights into its effectiveness on practical datasets. Additionally, while theoretical analysis would not provide the complete insight for understanding the efficacy of regulizers in Tsetline Machines. As a result, we prioritized empirical validation through experiments.

Comment 3: The authors should discuss some related works about interpretable machines like KAN [1,2].
[1]Liu, Ziming, et al. "Kan: Kolmogorov-arnold networks." arXiv preprint arXiv:2404.19756 (2024).
[2]Li, Chenxin, et al. "U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation." arXiv preprint arXiv:2406.02918 (2024).

Response 3: Thank you for suggesting interesting papers on interpretable AI. Kolmogorov-Arnold networks (KANs) are a type of neural network based on the idea that any multivariate function can be expressed in terms of simpler, univariate functions. Tsetlin Machines are based on propositional logic and uses the concept of Tsetlin Automata which was first proposed by Michael Lvovitch Tsetlin. Unlike neural networks, Tsetlin Machines do not have continuous functions but are discrete binary values. Tsetlin Machines are much less computationally expensive and can easily be implemented using simple hardware.

Comment 4: While regularization aims to reduce overfitting, the paper does not extensively discuss how the choice of regularizer parameters might lead to overfitting if not correctly tuned.

Response 4: Thank you for raising this important point. We agree that the choice of regularization parameters is crucial in balancing underfitting and overfitting. While our paper primarily focuses on the impact of regularization in reducing overfitting, we did not delve extensively into parameter tuning due to scope constraints. We do however conduct experiments that show the effect of the hyperparamters on the accuracy.

Comment 5: Can this method be deployed for fine-grained tasks like segmentation and detection, which may be more practical?

Response 5: Currently, Tsetlin Machines have extensively been used for image and text classification. The implementation of Tsetlin Machine in fine-grained tasks like segmentation and detection is widely unexplored. In the future, we aim to explore the application of Tsetlin Machines in image segmentation and detection and for few-shot learning. We believe that by implementing regulairzers, it would help implement Tsetlin Machines for these tasks and improve the effectiveness of Tsetlin Machines.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

These problems  have not been solved in the revised version. The  revisions should  be highlighted and added in the author's reply.

1.It is mentioned in the literature that the training process of Tsetlin Machine is inefficient when dealing with large-scale data sets. How to solve it?

2.There are some parameters that need to be selected manually in 3.Tsetlin Machine, such as the number of states of Tsetlin Automata and the parameters fed back by Type I/II. The selection of these parameters may have an important impact on the performance of the model, but how to choose the appropriate parameters?

3.Tsetlin Machine is mainly designed to process binary features, but it is difficult to process continuous features, and it usually needs discretization or conversion, which may lead to information loss or increased computational complexity.

4.s mentioned in the literature that although Tsetlin Machine performs well in classification and regression tasks, its application in other tasks (such as clustering and sequence modeling) is still limited.

Comments on the Quality of English Language

Moderate editing of English language required.

Author Response

Comment 1: It is mentioned in the literature that the training process of Tsetlin Machine is inefficient when dealing with large-scale data sets. How to solve it?
Response 1: Thank you for the comment. We have added an additional paragraph in Section Conclusion (pg: 11, paragraph 3, line 296-302)
Comment 2: There are some parameters that need to be selected manually in 3.Tsetlin Machine, such as the number of states of Tsetlin Automata and the parameters fed back by Type I/II. The selection of these parameters may have an important impact on the performance of the model, but how to choose the appropriate parameters?
Response 2: Thank you for the comment. We have added an additional paragraph in Section Conclusion (pg: 11, paragraph 4, line 303-310)
Comment 3: Tsetlin Machine is mainly designed to process binary features, but it is difficult to process continuous features, and it usually needs discretization or conversion, which may lead to information loss or increased computational complexity.
Response 3: Thank you for the comment. We have added an additional paragraph in Section Conclusion (pg: 11, paragraph 5, line 311-314)
Comment 4. As mentioned in the literature that although Tsetlin Machine performs well in classification and regression tasks, its application in other tasks (such as clustering and sequence modeling) is still limited.
Response 4: Thank you for the comment. We have added an additional paragraph in Section Conclusion (pg: 11, paragraph 6, line 315-320)

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have addressed my concerns.

Author Response

Thank you for reviewing our paper and for all the feedback.

Round 3

Reviewer 1 Report

Comments and Suggestions for Authors

There are significant differences between “electronics-3186930-peer-review-v2.pdf” and “electronics-3186930-peer-review-v3.pdf”. The essential parts should not be deleted in electronics-3186930-peer-review-v3.pdf.

Comments on the Quality of English Language

Minor editing of English language required.

Author Response

Comments 1: There are significant differences between “electronics-3186930-peer-review-v2.pdf” and “electronics-3186930-peer-review-v3.pdf”. The essential parts should not be deleted in electronics-3186930-peer-review-v3.pdf.

Response 1: We have uploaded the final version with the changes.