Reply to Morillas-Jurado et al. Benford Law to Monitor COVID-19 Registration Data. Comment on “Farhadi, N.; Lahooti, H. Forensic Analysis of COVID-19 Data from 198 Countries Two Years after the Pandemic Outbreak. COVID 2022, 2, 472–484”

In our paper Forensic Analysis of COVID-19 Data from 198 Countries Two Years after the Pandemic Outbreak [1], published in COVID in 2022, we cited the work Applying Benford’s Law to Monitor Death Registration Data: A Management Tool for the COVID-19 Pandemic [2]—hereafter the paper. The authors of the paper commented on our review [3]. Most notably, we sincerely thank the authors, Francisco Gabriel Morillas-Jurado, María Caballer-Tarazona, and Vicent Caballer-Tarazona (hereafter the authors) for their feedback and efforts.

First, Morillas-Jurado et al. used pandemic data, particularly in an unclear range. In the original paper’s Abstract, the authors referred to “February to August 2020” (without exact dates) as the first epidemic wave in Spain. Later, in Section 3, Data and Source, the authors addressed the timeframe “March to June 2020.” This is problematic, as they did not offer the exact timeframe of their study. For us, it was impossible to replicate and regenerate their results.

Second, the selection of the number of “deaths per day recorded by the different ACs during the period” represents another problem in our eyes. Based on our assessment, we recognize that selecting “the number of deaths” as the single goodness of fit test may have impacted the statistical results of the authors. It is commonly accepted that the conformity to Benford’s Law (BL) improves as the range of the dataset increases. The numbers of daily deaths are insufficient for BL evaluation as they only extend over a few orders of magnitude. Confirmed death cases in Spain had an order of magnitude of 3 from 1 January 2020 to 30 June 2020. It is not clear why the authors used the number of deaths only. The body of knowledge suggests that the underpinning data must be large enough to assess compliance with the first digit law. The respective data are too small for an assessment based on BL. When the paper was published, the number of daily incidents had a larger order of magnitude for the same period, up to 4.

Third, the anomalies in the first wave in Spain were shortly mentioned in the paper. However, our review found that, for example, a negative number of −1918 was reported for the number of deaths on 25 May 2020. Later, this number was adjusted by the authorities. The authors argued that there were some errors in the transcription of government data. Some errors involved simple changes in the numbers due to transcription errors or changes in the cause of death for some deceased. However, in our eyes, this is proof of inconsistency in recording COVID-19 incidents that affected the quality of epidemic observations in Spain, resulting in poor conformity to BL. It shows that the number of daily deaths during the first epidemic wave was not a reliable variable. The paper provided a preliminary discussion about these anomalies and their impact on the BL assessment, which may have required a qualitative evaluation of regional data and policies. Inconsistent reporting procedures and policies may account for the lack of reliability.

Fourth, using the Chi-Square test as the sole measure of goodness of fit to assess the consistency of epidemic data with the law of leading digits raises the following concerns: the Chi-Square test is sensitive to sample size and is not recommended when making inferences about very large or very small data sets. Other statistical procedures for assessing the BL of discrete data are less sensitive to the sample size and are recommended to be applied.

Last, the use of the Monte Carlo simulation also seems problematic here. The underpinning assumptions of the simulation were derived from the small sample size with unexplained anomalies. In our eyes, using the Monte Carlo simulation to provide statistical evidence is NOT a remedy when assessing the Benfordness of small datasets with questionable abnormalities, as conducted in the paper.