*5.3. Comparative Evaluation*

As a key parameter in blockchain network, transaction *tx* committed time indicates how long a *tx* can be finalized in a new block on distributed ledger, and it is closely related to block confirmation time in consensus protocol. Given different blockchain benchmarks, we evaluate the end-to-end time latency of committing transactions along with other key performance metrics. For Ethereum, we used smart contract to record transactions on blockchain. For the Tendermint network, we used the built-in kvstore as an ABCI (Application BlockChain Interface) app to save transactions.

We conducted 50 Monte Carlo test runs, where a node sends a 1KB *tx* per second (TPS) to the blockchain network and waits until *tx* has been confirmed on the distributed ledger. Figure 7 shows the distributions of time delay for committing transactions given different blockchain networks. Each green bar indicates standard deviation with a mean represented by a red dot. The gray line shows the entire data range, and the black star is the median. Tendermint uses a BFT consensus protocol to achieve high efficiency; therefore, the mean of *tx* committed time is about 3 s given one voting round per second. Unlike Tendermint, Ethereum relies on probabilistic PoW consensus, which has variable block confirmation times. Thus, *tx* committed time in the Ethereum network varies with largest standard deviation. To guarantee synchronous epoch rounds for PoENF consensus, we set TΔ conservatively to 2 s based on the maximum time to ensure txs and blocks propagation in a P2P consensus network, including 20 validators. Hence, the range of latency in EconLedger is smaller than Ethereum, and *tx* committed time is almost stable (about 5.5 s).

**Figure 7.** Time latency for committing transactions. Comparative evaluations on different blockchain networks.

Table 4 provides a comprehensive performance of committing transactions on different blockchain networks regarding several key performance matrices. Given the above *tx* committed time, which uses the mean in Figure 7, the tx rate *tx*/*s* is evaluated by calculating how many tx can be processed per second in the blockchain network. The Ethereum block size is bounded by how many units of gas can be spent per block, which is known as the block gas limit [46]. Currently, the maximum block size is around 12,000,000 gas (accessed at 20 July 2020), and the base cost of any transaction is about 21,000; thus, each block in Ethereum can include about 571 transactions. In our private Ethereum network, we can obtain the *tx* rate as (571.4/4.6) ≈ 124 tx/s. Tendermint and EconLedger both use fixed 1MB block. Given 1 KB per transaction, a block in Tendermint can store a maximum of 1000 transactions; thus, the tx rate is about (1000/2.9) ≈ 344 tx/s. For EconLedger, each transaction is about 430 bytes such that a block can record the maximum of 2400 transactions, then it achieves higher tx rate at around (2400/5.5) ≈ 436 tx/s.


**Table 4.** Comparative evaluation of launching transactions on different blockchain platforms.

In order to evaluate resource consumption by running blockchain benchmarks, we used the "top" command to monitor system performance of machines. We considered CPU and memory usage on Desktop (Ethereum miner) and Rpi (Tendermint and EconLedger validator). Due to computation intensive PoW algorithm, the mining process of Ethereum almost occupies full CPU capacity and consumes about 1.2 GB memory. Therefore, such a huge computation cost prevents resource constrained edge devices mining in Ethereum network. Unlike Ethereum, Tendermint and EconLedger use lightweight consensus algorithms to achieve efficiency in CPU and memory usage such that they are both suitable for deploying validators on edge devices. EconLedger almost has the same amount of memory usage as Tendermint in terms of running time. However, EconLedger has the higher *tx* committed time than Tendermint does, and it only needs 40% of the computation resource that Tendermint does.
