NeuChain+: A Sharding Permissioned Blockchain System with Ordering-Free Consensus

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The aim of the paper to improve the performance of blockchains. The authors created a blockchain prototype called NeuChain+ to reduce cross-shard transaction time. They investigated scalability and performance of the suggested system.

The introduction is appropriate, I suggest a more detailed description of NeuChain [25] to see what the improvement is of this paper.

 Achievements

·        Improved Cross-Shard Transaction Protocol: the authors proposed a new cross-shard transaction processing protocol, “cross-reserve”, which efficiently employs deterministic concurrency control having less computational cost.

·        Optimized Cross-Shard Latency Reduction: the authors proposed a method to optimize cross-shard commit latency by pipelining and parallelizing read sets transmission during transaction execution, enhancing overall efficiency.

·      New Sharding Execute-validate Architecture: The authors introduced a Sharding-EV architecture, enhancing scalability while maintaining high throughput by eliminating explicit ordering constraints, thus advancing the field.

 ·      Successful NeuChain+ prototype implementation: the authors successfully implemented the NeuChain+ sharding blockchain prototype, a refinement of their previous work, NeuChain. Experimental results underscore NeuChain+'s superior performance, demonstrating substantial throughput improvement compared to prior iterations and other state-of-the-art blockchain systems under the SmallBank workload.

References:

The number of references is 50 which provides a good overview of the related work. References are from the recent period.

There is one self-citation: [25], which is necessary, as contains the previous work of the authors, which was improved

[10] - I could not find the referenced facts, please review the reference

Findings

 Scalability seems to improve kTPS on a linear way, is this the case? What are the limitations of the number of nodes? In the introduction they stated that even Visa requires tens of kTPS only, so a 4 nodes configuration could serve Visa’s needs ?

 Storage consumption is still 400 MB/min – which is an improvement compared with 1600 Mb/min but still unacceptable in practice (576 GB / day)

Please provide an analysis for greater than 60s or discuss why that a short period is enough

The paper says: “storage optimization is not entirely linear” – Figure 9 looks linear

Structure of the paper

 The manuscript is clear and relevant for blockchain systems. Its structure is appropriate.

Areas to improve the paper:

To test their work, the authors utilized simulation data of “SmallBank”. No detailed description was provided on the transactions data, in my opnion that must be improved. The transactions follow a uniform distribution which does not happen in the real world.

The performance, Transaction Arrival Rates are discussed correctly and compared with competitors. No test hypothesis was given, the paper just logs the results.

Algorithms could be better described and detailed in order to reproduce (i.e. implement)  them.

Figures and tables are appropriate. They are easy to understand. The performance data on which the figures are based is not available.

Implementation of the evaluation is too short. I suggest detailing the implementation after the algorithm description part.

Conclusions should be extended with a discussion of whether the performance of the proposed algorithm is appropriate for real word usage.

Data Availability Statement: In this section, please provide details regarding where data supporting reported results should be found. Please consider making the data public or name the author who is responsible for the dataset.

In the conclusions please highlight the achievement and put the statistics in a section called results.

Comments on the Quality of English Language

I suggest revising the following. 

we implement a blockchain prototype -> we have implemented a blockchain prototype

experimented on - > experimented with

S1 commit T locally -> S1 commits T locally

Therefore it also uses 2PC -> Therefore, it also uses 2PC

transaction will be pended -> transaction will be pending

. Therefore it only needs a simple -> . Therefore, it only needs a simple

neuchain+ can be committed -> Neuchain+ can be committed

Client divides the complete transaction set into 2 -> The client divides the complete transaction set into 2

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

According to my assessment, the weaknesses of the article:  

-1) Section 3.3.1 states:  Notably, the read-write set of transactions is unknown before execution, we assume the shards involved in a transaction are known.

It is not clear how this sentence should be understood. Does this mean that in the NeuChain+ system, we always strictly predetermine the number of shards into which we decompose the unknown number of transactions? This assumption needs clarification and justification, as it is acceptable and does not affect the properties of the transaction processing protocol shown in Figure 4.

-2) In section 5.2, conflict detection is described under the description of Algorithm 3. Authors write:

However, if the transactions with RAW dependency but no simultaneous WAR dependency, it can be ensured that there are no cyclic dependencies among transactions. Therefore, such transactions can be committed by reordering [44] transactions.

 It is unclear how cyclic dependencies between transactions are detected and how to select transactions that can be committed by reordering.

-3) Is it possible that in a transaction graph with n shards and a large number of m transactions, the number of cycles in the transaction graph can increase exponentially with n and m? If so, then Conflict Detection Algorithm 3 may be inefficient.

-4) No information about the computational complexity of the developed algorithms 1, 2, 3.

-5) Chapter 6 presents the performance evaluation results and benchmarking of the developed NeuChain+ system against other solutions. Are these results averages of several measurements using a uniformly distributed workload of transactions?

-6) The graphs showing the values of the performance parameters of the compared blockchain systems should be supplemented with tables with specific values, e.g., averages, numbers of measurements, min, max, and standard deviations.

-7) The results indicate that the performance parameters of NeuChain and NeuChain+ are similar, e.g., the throughput of NueChain+ is approximately 1.7 times that of NeuChain, and its latency is at 0.179s, slightly higher than NeuChain.

It's not clear if this is the average throughput over several measurements. And if so, are these differences statistically significant? One would need to consider using some statistical test, such as a t-test or others, to assess the statistical significance of the differences in performance, i.e., throughput, latency, between NeuChain and NeuChain+, choosing a specific confidence level.

In conclusion, the article should not be published in its current version, as, in my opinion, it needs a significant revision.

Comments on the Quality of English Language

The linguistic side of the work is correct.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

Figure 8. Performance comparison of scaling the number of nodes.

I don't see values for Abort(Meepo). If it is 0, maybe highlight it in the description above.

Also, maybe a different color palette where Throughtput and Abort would be more similar for same chain.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

Strong Points:
============

1. Innovative Cross-Shard Transaction Protocol: The introduction of the cross-reserve protocol is a significant innovation. By eliminating the need for costly cross-shard coordination, it addresses a critical bottleneck in existing sharding implementations, potentially improving performance and scalability.

2. Ordering-Free Execute-Validate Architecture: Utilizing an ordering-free EV architecture is a noteworthy advancement. This approach simplifies the transaction processing pipeline and reduces overhead, which can lead to more efficient and faster transaction processing in permissioned blockchain systems.

3. Scalability and Performance: The experimental results are promising, showing that NeuChain+ achieves substantial throughput improvements (1.7-75.3×) under the SmallBank workload. This demonstrates the practical potential of the proposed system in real-world scenarios where high transaction volumes are common.

4. Comprehensive Evaluation: The paper provides a thorough evaluation of NeuChain+, comparing it against state-of-the-art blockchain systems. The detailed performance metrics and the substantial throughput improvements lend credibility to the claims of scalability and efficiency.

5. Practical Applications: The proposed system has broad applications in digital assets, supply chains, government services, and Web 3.0. Its ability to enhance throughput and scalability makes it highly relevant for these domains, where efficient and reliable transaction processing is crucial.

Weak Points

==========

1. Complexity of Implementation: The implementation details of NeuChain+ are complex, and the paper may not sufficiently simplify these concepts for a broader audience. A more detailed explanation of the architecture and protocol mechanisms could enhance understanding and reproducibility.

2. Limited Workload Scenarios: The experimental evaluation is based primarily on the SmallBank workload. While this is a common benchmark, additional workload scenarios should be considered to validate the system's performance across diverse and more complex applications.

3. Security Considerations: The paper lacks a detailed discussion on the security implications of the proposed cross-reserve protocol. Ensuring that the new protocol maintains the same level of security as traditional methods is crucial for its adoption in sensitive applications.

4. Generalizability of Results: The results, although impressive, are based on a specific set of experimental conditions. The paper should discuss the generalizability of these results and how NeuChain+ would perform under different network conditions, shard configurations, and transaction types.

5. Comparison with a Broader Set of Systems: While the paper compares NeuChain+ with state-of-the-art blockchain systems, a broader comparison with more diverse systems, including both permissioned and permissionless blockchains, could provide a more comprehensive view of its advantages and limitations.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have satisfactorily commented on inaccuracies in the original manuscript and filled in missing elements.

Comments on the Quality of English Language

The linguistic side of the work is correct. 

Reviewer 4 Report

Comments and Suggestions for Authors

Authors have thoroughly addressed my all comments.