Bounds on Performance for Recovery of Corrupted Labels in Supervised Learning: A Finite Query-Testing Approach

Abstract

Label corruption leads to a significant challenge in supervised learning, particularly in deep neural networks. This paper considers recovering a small corrupted subset of data samples which are typically caused by non-expert sources, such as automatic classifiers. Our aim is to recover the corrupted data samples by exploiting a finite query-testing system as an additional expert. The task involves identifying the corrupted data samples with minimal expert queries and finding them to their true label values. The proposed query-testing system uses a random selection of a subset of data samples and utilizes finite field operations to construct combined responses. In this paper, we demonstrate an information-theoretic lower bound on the minimum number of queries required for recovering corrupted labels. The lower bound can be represented as a function of joint entropy with an imbalanced rate of data samples and mislabeled probability. In addition, we find an upper bound on the error probability using maximum a posteriori decoding.

1. Introduction

Recently, deep neural networks using large datasets showed successful roles in a number of machine learning tasks, such as object recognition [1,2], prediction systems [3,4,5], and natural language processing [6,7,8]. This success is due to the availability of vast but well-labeled data. However, the problem with obtaining such labeled data is that it is time-consuming and expensive. Source labeling platforms, such as Amazon’s Mechanical Turk, a kind of non-expert system, are now widely used for reducing costs. Nevertheless, in some cases, unreliable labels are frequently generated [9,10,11,12]. In addition, labeling data is sometimes a complex and difficult task, even for experienced domain experts [13,14]. Even adversarial attacks lead to manipulated labeling results [15]. We call unreliable labels corrupted or noisy labels because they are assigned label values that are different from the true label values. The proportion of noisy labels in the current datasets was reported from 8.0% to 38.5% [16,17,18,19]. The training of deep neural networks with noisy labels is more sensitive to overfitting [20,21]. Improved methods have been developed to better learn from these corrupted labels, including regulation techniques, the structure of deep neural networks, removing unreliable data, and improving loss functions [22]. Our finite query-testing scheme is considered in the context of a supervised learning problem, which is essentially a structure that requires labeled data. The reliability of the labels affects the overall performance with respect to the training process. In particular, incorrect labels mainly lead to overfitting. In this work, we focus on the supervised learning problem to improve the accuracy of the labels rather than semi-supervised and unsupervised learning.

Suppose there are N (i.e., $N>1000$) data samples that are labeled, where a subset of samples is corrupted due to labeling by non-experts. The goal of this work is to find noisy data using a finite query-testing scheme in supervised learning. It is clear that label experts identifying all the data one by one is extremely costly and exhausting. We aim to find a subset of the given noisy data from non-experts while keeping the number of queries for identifications by experts as small as possible, which is approximately greater than $\frac{H\left(\mathbf{s}\right)}{logK}$, where $\mathbf{s}$ denotes the mislabeled vector, K is the number of classification, and $H\left(\right)$ is the entropy based 2. For a multiple classification problem in supervised learning, mislabeled data samples are corrected to the ground truth using the finite query-testing scheme. Each data sample is considered a finite value expressed as 0 and $K-1$ for K classifications. In this work, experts can specify K labels for N data samples using crowdsourcing approaches. Crowdsourcing, for example, asks experts: Do the two data samples belong to the same category? Or is it the same? Through such queries, experts identify whether multiple data samples are the same or different at once [23]. It then recovers the labeling based on the crowdsourced queries. This idea was used in entity resolution studies [24,25]. Since each labeling from experts takes time and money, our goal is to accurately identify data samples while minimizing the number of queries for crowdsourcing. Recently, query algorithms to solve this problem have been proposed and developed [24,25,26]. Experts can potentially process more than two data samples at a time. If possible, it is good to take the number of samples involved in the query to be as small as possible. Also, instead of answering ’same class’ type queries, experts use a few simple functions on small input datasets.

Queries for experts are classified into adaptive and non-adaptive schemes. Adaptive and nonadaptive schemes have been proposed for the crowdsourcing problem. An adaptive query like the equal class is an approach that can exactly recover all data samples. Nonadaptive cases require queries proportional to the square of the number of samples in multiple classes. This means that adaptive algorithms can achieve better performance than nonadaptive ones in the absence of additional information. In [25], a condition on how to reduce this gap is discussed. Despite the existence of the advantages of the adaptive algorithm as mentioned above, in such a case where it performs in crowdsourcing, a nonadaptive scheme that can be processed in parallel, which is advantageous for efficient processing time, is more realistic [25]. In practice, adaptive algorithms are additionally considered slow processing times, as crowdsourcing can result in long response times by experts. In this paper, we consider a nonadaptive scheme of our problem. That is, we perform the recovery of the corrupted labels after obtaining all query responses.

In [27], the authors show that a semi-supervised clustering model is equivalent to locally encodable source coding. Then, they find bounds on the minimum number of queries by performing a labeling recovery problem using XOR and AND queries for binary classification. In addition, in [28], the authors characterize information-theoretic bounds on the optimal number of queries to reliably recover labels given a combination of queries with constant d-degrees and noise parameters in binary classification. Both [27,28] are similar to our work but differ in the following ways. The main difference is that while in the two papers [27,28], they address the recovery problem for binary classification, we extend the problem to multiple classification. In addition to this, we use finite field operations to replace the XOR and AND operations used for binary classification. Furthermore, in our work, in finding a bound for the minimum number of queries, we obtain an upper bound using the MAP (maximum a poesteriori) decoding method. With this work, we not only compare the lower and upper bounds to each other but also show bounds on the performance of recovering corrupted data.

In this paper, we consider the problem of recovering corrupted labels in supervised learning. In deep neural networks for multiple classification, a small subset of data samples with true labels are inverted due to labeling errors. So a part of the entire dataset, corrupted by the labeling of non-experts, like an automatic classifier, is used for training. Our task is to identify the corrupted data samples with a minimum number of expert queries and recover them to their true labels. To this end, we aim to recover corrupted data samples with the assistance of an additional expert, called a finite query-testing system. In this paper, the query-testing system randomly selects a subset of data samples and responds to the results of their combinations via finite field operations. And we obtain a lower bound on the minimum number of queries to recover corrupted labels and an upper bound on the error probability with MAP decoding.

2. Related Work

Recovering unknown signals from arbitrary sets of combinations is called a constraint satisfaction problem. It has been an active research topic in computer science, statistics, and biology, and has inspired machine learning and cryptography [29]. In particular, our problem is in line with the random planted d-XOR satisfaction problem. It aims to find a small number of binary variables that satisfy a set of constraints that the XOR combination of a set of randomly and uniformly chosen variables should be equal to 0 or 1. For the pairwise queries, it has been mainly focused on the field of community detection [30]. It was shown that the information-theoretic bound for optimal sample complexity is achieved when measurements are compromised by binary symmetric noise. In [25], the authors considered the case where the number of communities could be greater than two, and proved an information-theoretic bound on the number of connections and an estimated approach went to this bound. The parity codes with greater than three degrees was considered in terms of locally encodable coding or the community recovery of hypergraphs [31]. In the case of even degrees, the authors in [30] considered the phase transition for the recovery of optimal complexity. In addition, in [31], the authors showed that the recovery of optimal complexity for parity codes with a constant degree scales as the order of degree grows, up to an order of log scales.

When the error rate in channel coding theory is not known, the code design is an important issue, which has been an intensive research field over the past half century. If data transmitted from a source through a communication channel with random loss arrive at a destination, they are modeled as a binary erasure channel with a random error rate. Unlike repeat codes that are used for TCP/IP schemes that cause long delivery delays due to a feedback channel from the destination, fountain codes called forward error correction codes have shown successful performance in delivering data reliably [32]. Suppose there is a set of input symbols, the fountain code generates rateless parity codes called output symbols. Fountain codes guarantee that input symbols can be recovered with high probability at a small overhead symbol for any set of output codes. Many efforts have been made to design practical fountain codes that achieve the reliable recovery of information bits with small overhead and low encoding and decoding complexity. Our construction of a query matrix is closely related to the fountain code design because choosing a subset for coded labels is like designing a codeword. However, the fountain codes do not take into account the ability of experts to respond to complex data labels. More specifically, it is a good idea to use the relationship between the label difficulty and the number of data samples to be combined because the higher the difficulty of the label, the more the expert must identify the labels one by one.

Semi-supervised learning is a technique that allows you to train a model without having to label some of the data. There are two main ways to train a model with noisy data. The first is to train a robust model that performs well under noisy labels. This can be done by using regularization or loss functions to avoid overfitting to noisy labels. The second is to detect and remove data that are likely to be a noisy label, and then train the model with the remaining data. The co-teaching method [33] helps detect noisy data by having two networks train together, with one network selecting data to use in training the other. In [34], O2U-NET repeated learning from underfitting to overfitting several times by changing the learning rate as a parameter. It detected noisy labels by using the average of losses for each datum and improved the reliability of the performance. In addition, in [35], the authors computed the average of the losses using an exponential moving average to determine the noisy labels more reliably than a simple average.

3. Finite Query-Testing Scheme

3.1. Problem Statement

In this subsection, we define a finite query-testing scheme for the reconstruction of corrupted labels in supervised learning. First, we consider a random vector $\mathbf{x}$ of size N, $\mathbf{x}={\left(x_1,x_2,\dots ,x_N\right)}^T$. The value of each element of $\mathbf{x}$ is to represent K classification. For $i\in \left\{1,2,\dots ,N\right\}$, $x_i=0$ means that the i-th data are the first classification, and when the value is $K-1$, $x_i=k-1$, it corresponds to the K-th classification. In the end, $\mathbf{x}$ is represented by a random vector of size N with K values. In this work, we consider labeling imbalanced data. Imbalanced data are a dataset in which the labeling distribution of K categories is such that the majority of data belong to a particular category, and the minority of data belong to the other categories. For example, classifying a disease from a patient’s diagnostic image or identifying a defect from observing the condition of a product are examples of imbalanced data. This is when the majority of the classification labeling is normal and a small percentage of the data is abnormal. For such imbalanced data, we assume that each element of the random vector has the following probability distribution. We assume a probabilistic model, where the first classification appears with the greatest probability and the others occur evenly:

$$Pr\left(x_i=\theta \right)=\left\{\begin{array}{cc}1-p& \mathrm{if}\theta =0,\\ \frac{p}{K-1}& \mathrm{if}\theta \ne 0,\end{array}\right.$$

(1)

where p is the imbalance rate of classification, $1-p$ is the probability of classifying as normal, and $\theta$ is a dummy variable. We assume that the imbalance ratio is less than 0.5, $p<0.5$. In other words, we restrict ourselves to classification problems, where the probability of being a certain classification (e.g., normal) is greater than or equal to the majority. We define data with K classes, and now we aim to describe how an expert oracle (labeler or worker) can use a finite field to combine labeling tasks and collect data.

Now, we consider a situation where the oracle has mislabeled the data contained in a given query. The main benefit of our query-testing system is that we can find a true label vector $\mathbf{x}$ with the correction of data labeling, even if the oracle responds with a mislabeled answer. The reason we consider this situation is that the oracle’s skill and expertise can lead to mislabeling. For example, in medical imaging, the same data can be labeled differently by experts with different levels of experience and proficiency. Let $\mathbf{c}$ be a corrupted label vector that contains the error corresponding to $\mathbf{x}$, where $\mathbf{c}={\left(c_1,c_2,\dots ,c_N\right)}^T$. We represent the transition from a true label vector $\mathbf{x}$ to a corrupted label vector $\mathbf{c}$ due to mislabeling using the following conditional probability with independently identical distribution for each element:

$$Pr\left(c_i|x_i\right)=\left\{\begin{array}{cc}1-\gamma & \mathrm{if}{c}_i=x_i,\\ \frac{\gamma }{K-1}& \mathrm{if}{c}_i\ne x_i,\end{array}\right.$$

(2)

where $\gamma$ is the mislabeling probability that the oracle classifies the given data differently in the K categories. In fact, this mislabeling probability is very small. In our work, it is possible to perfectly recover the original true vector $\mathbf{x}$ by considering this mislabeling probability. Let $\mathbf{s}$ be a sparse vector represented by addition on a finite field between the corrupted label vector $\mathbf{c}$ and the true vector $\mathbf{x}$, as $\mathbf{s}=\mathbf{c}\oplus \mathbf{x}$, where the symbol ⊕ is the addition on finite fields. Note that the vector $\mathbf{c}$ is labeled with small mislabeled probability, then, using the following relationship, $\mathbf{s}\oplus \mathbf{c}=\mathbf{x}$, we identify the original true vector $\mathbf{x}$. Herein, the vector $\mathbf{s}$ is sparse because the mislabeling probability $\gamma$ is very small. In other words, $\mathbf{s}$ is a sparse vector with mostly zero elements. Therefore, we see that finding a sparse vector is equivalent to recovering a true label vector.

In our query-testing scheme, an oracle responds to labeling information by querying the data. A query can be thought of as a random subset of data that is composed of one or more data. For example, a subset of a sparse vector consisting of data 1, data 5, and data 7 is requested to the oracle as a single query (e.g., a query containing three data on a single screen on the web). For each data (data 1, 5, and 7), the oracle performs labeling for each of the K categories. The oracle then combines the results of each of the three data to answer the query once. Here, the query and response are performed once, but the oracle performs the labeling on three data. We assume that there exist M queries in our query-testing scheme, $j\in \left\{1,2,\dots ,M\right\}$. Let us vectorize all the $y_j$s and call $\mathbf{y}$ the response vector, as $\mathbf{y}={\left(y_1,y_2,\dots ,y_M\right)}^T$. We define the j-th response $y_j$ to the query labeled by the oracle using a finite field with size K as follows:

$$y_j=\sum _{i\in {\mathcal{R}}_j}{s}_i,$$

(3)

where we let ${\mathcal{R}}_j$ be the data selection set to be included in the j-th query, which is the set of indices of the data to be included in the j-th query that tells us which of the N data of the sparse vector $\mathbf{s}$ to include to combine with a single query. For the previous example, it would be ${\mathcal{R}}_j=\left\{1,5,7\right\}$. And the addition and multiplication operations in (3) are performed on finite fields. Note that the value of the response $y_j$ does not imply K classification but rather addition on a finite field. And actual data agents do not collect the element $s_i$; they collect $y_j$. If the data to be labeled are sensitive and need to be protected, the oracle does not pass $s_i$ directly to the data agent. Instead, it passes $y_j$, which is the result of combinations in a finite field on the labeled values of the data in the query. This has a privacy benefit because it essentially does not store the original data $s_i$ on the data agent.

The next consideration is to understand how to generate ${\mathcal{R}}_j$. In fact, the data selection set for queries has a significant impact on the recovery performance of corrupted data. A well-crafted data selection set can achieve the condition that the original true data can be fully recovered with a minimum number of queries. Similarly, problems, such as the sensing matrix design in compressed sensing [36], parity check matrix design in error correction codes [37], and group matrix design in group testing problems [38,39], are analogous to the generation of a data selection set. According to [36], the role of matrix design holds paramount significance within the domain of compressed sensing, owing to its direct influence on the efficacy of signal recovery from sparse measurements. A well-crafted matrix design underpins the capability to achieve precise signal reconstruction utilizing a limited number of measurements, thereby optimizing the efficiency of data acquisition processes. Conversely, the selection of an inadequately structured matrix can considerably degrade the quality of signal reconstruction, compelling the requirement for an increased number of measurements, thereby negating the inherent advantages of compressed sensing in terms of efficiency and resource optimization. Let $Q_{ji}$ be whether or not the i-th sparse data participate in the j-th query. To define our data selection set, we assume the following probability distribution for each $Q_{ji}$ for whether the i-th data are included in the j-th query:

$$Pr\left(Q_{ji}=\theta \right)=\left\{\begin{array}{cc}1-\delta & \mathrm{if}\theta =0,\\ \delta & \mathrm{if}\theta =1,\end{array}\right.$$

(4)

where $\delta$ is the probability that the i-th data are included in the j-th query. A large $\delta$ means that a lot of data are included in the same query and the oracle is being asked to label it. Conversely, a small $\delta$ indicates that you are asking the oracle to label a query that consists of a small number of data.

For the corrupted labels, our query-testing system is represented by the following formula based on finite field operations:

$$\mathbf{y}=\mathbf{Q}\mathbf{s},$$

(5)

where $\mathbf{Q}\in {\left\{0,1\right\}}^{M\times N}$ is a query matrix with binary numbers of size $M\times N$, $\mathbf{s}\in {\mathbb{F}}_K^{N\times 1}$ is a sparse vector of finite size K and length N, and $\mathbf{y}\in {\mathbb{F}}_K^{M\times 1}$ is a response vector of finite size K and length M. And the addition and multiplication in (5) follow the operations defined in a finite field of size K. In this paper, we assume that K is either a prime or a power of a prime number. So far, we defined the problem for the query-testing system considered in this work. We formulate it as the inverse problem of finding $\mathbf{s}$ given $\mathbf{Q}$ and $\mathbf{y}$ as in (5).

3.2. Decoding

In this paper, to find a true label vector $\mathbf{x}$, we consider using the MAP (maximum a posteriori) decoding method. The MAP method is a process of selecting a possible candidate random vector that maximizes the posterior probability. In this work, we find $\mathbf{x}$ such that the conditional probability is maximized by the following argument:

$$\widehat{\mathbf{x}}=arg\underset{\mathbf{x}}{max}Pr\left(\left.\mathbf{x}\right|\mathbf{y},\mathbf{Q}\right).$$

(6)

We rewrite (6) as follows:

$$\begin{array}{cc}\hfill Pr\left(\left.\mathbf{x}\right|\mathbf{y},\mathbf{Q}\right)& =\frac{P\left(\mathbf{x},\mathbf{y},\mathbf{Q}\right)}{P\left(\mathbf{y},\mathbf{Q}\right)}\hfill \\ \hfill & \propto P\left(\mathbf{x},\mathbf{y},\mathbf{Q}\right)\hfill \\ \hfill & =\sum _{\mathbf{s}}P\left(\mathbf{x},\mathbf{y},\mathbf{Q},\mathbf{s}\right)\hfill \\ \hfill & =\sum _{\mathbf{s}}P\left(\mathbf{x}\right)P\left(\left.\mathbf{s}\right|\mathbf{x}\right)P\left(\mathbf{Q}\right)P\left(\left.\mathbf{y}\right|\mathbf{Q},\mathbf{s}\right)\hfill \\ \hfill & \propto \sum _{\mathbf{s}}P\left(\mathbf{x}\right)P\left(\left.\mathbf{s}\right|\mathbf{x}\right){\mathbb{1}}_{\mathbf{y}=\mathbf{Qs}}.\hfill \end{array}$$

(7)

The fourth line of (7) is due to an independent condition. In addition, the conditional probability $P\left(\left.\mathbf{y}\right|\mathbf{Q},\mathbf{s}\right)$ is an indicator function ${\mathbb{1}}_{\mathbf{y}=\mathbf{Qs}}$ that satisfies the following condition, $\mathbf{y}=\mathbf{Qs}$:

$$P\left(\left.\mathbf{y}\right|\mathbf{Q},\mathbf{s}\right)=\left\{\begin{array}{cc}1& \mathrm{if}\mathbf{y}=\mathbf{Qs},\\ 0& \mathrm{if}\mathbf{y}\ne \mathbf{Qs}.\end{array}\right.$$

(8)

For $\widehat{\mathbf{x}}$ obtained from the MAP decoding in (6), we define an error event if it differs from realization $\mathbf{x}$. In this paper, the error probability $P_E$ is for the error event when the result obtained from decoding differs from the original true value as $P_E=Pr\left\{\widehat{\mathbf{x}}\ne \mathbf{x}\right\}$.

3.3. Labeling Error

Our query-testing system is characterized by the ability to correct errors in the traditional labeling process. In traditional labeling, data are selected and labeled once per query, so the true value of the labeling is inverted according to the oracle’s skill and expertise, and used as a dataset for supervised learning. The data are then used as a dataset for supervised learning without any further correction or modification of the data labeling. In the previous section, we used a probabilistic model to represent the errors that oracles make when labeling data. The next step is to define the probability distribution of the response vector of the query-testing system due to these labeling errors. Using (1) and (2), we lead to the following probability model for the element of the sparse vector:

$$P\left(s_i=\theta \right)=\sum _{\beta \in {\mathbb{F}}_K}P\left(x_i=\beta \right)P\left(c_i=\theta |x_i=\beta \right)=\left\{\begin{array}{cc}1-q& \mathrm{if}\theta =0,\\ \frac{q}{K-1}& \mathrm{if}\theta \ne 0,\end{array}\right.$$

(9)

where $1-q=(1-p)(1-\gamma )+\frac{p\gamma }{K-1}$.

Next, based on the query matrix, the oracle selects a subset of the data to be included in a single query. This is performed by generating a subset of the data using the query matrix by (4) defined earlier. If there is only one instance of data in the subset, $|{\mathcal{R}}_j|=1$, then the corresponding response result $y_j$ is equal to the probability distribution in (9). Let us go a step further and see what happens to the response result $y_j$ after the subset contains two data entries. For example, in our query-testing system, the probability of a response result involving the first and second data is $P(s_1+s_2)$ for $s_1\ne 0,s_2\ne 0$ as $|{\mathcal{R}}_j|=2$. In the same way, the probability of the response result involving the second and third data is $P(s_2+s_3)$ for $s_2\ne 0,s_3\ne 0$. At this point, we can see that the probability distributions of the two response results above are identical, $P(s_1+s_2)=P(s_2+s_3)$. The probability of one response result from the combination of the three data is also $P(s_1+s_2+s_3)$ for $s_1\ne 0,s_2\ne 0,s_3\ne 0$, as $|{\mathcal{R}}_j|=3$, and, eventually, the probabilities of all the possible response results from selecting three data from N data are the same.

As a more general case, consider a sparse vector $\mathbf{s}$ with the first d nonzero elements, ${\left|\right|\mathbf{s}\left|\right|}_0=d$, and let $P_d$ be the probability that $P_d:=Pr\left({\sum }_{i=1}^d{Q}_{ji}{s}_i=0\right)$, where $s_i\ne 0$ due to operations of a finite field. Given the probability distribution of $Q_{ji}$ in (4), we can write the probability $P_d$ in a recursive form as follows:

$$\begin{array}{cc}\hfill P_d& =Pr\left(\sum _{i=1}^{d-1}{Q}_{ji}{s}_i=0\right)Pr\left(Q_{jd}{s}_d=0\right)+\sum _{\theta \in {\mathbb{F}}_K\setminus \left\{0\right\}}Pr\left\{\sum _{i=1}^{d-1}{Q}_{ji}{s}_i=\theta \right\}Pr\left\{Q_{id}{s}_d=-\theta \right\}\hfill \\ \hfill & =P_{d-1}\left(1-\delta \right)+\left(1-P_{d-1}\right)\frac{\delta }{K-1}.\hfill \end{array}$$

(10)

Define $H_d:=P_d-K^{-1}$. The recursive expression is obtained as follows:

$$H_d=H_{d-1}\left(1-\frac{\delta }{1-K^{-1}}\right).$$

(11)

Solving (11), we find the probability $P_d$

$$P_d=K^{-1}+\left(1-K^{-1}\right){\left(1-\frac{\delta }{1-K^{-1}}\right)}^d.$$

(12)

So far we defined the problem of the query-testing system we are considering. Next, we investigate from an information-theoretic point of view how many queries are needed to perfectly recover the true data $\mathbf{x}$ as a function of the percentage of imbalanced data. We also use the MAP decoding method to obtain an upper bound on the error probability that recovery will fail.

4. Bounds on Performance for Recovery of Corrupted Labels

4.1. Lower Bound

Now consider the minimum number of queries to require the recovery of a corrupted label vector in the finite query-testing scheme. To handle this, Fano’s inequality theorem [40] is used, which was introduced by information theory. In this section, we exploit Fano’s inequality theorem to derive a lower bound on the error probability for the recovery of the corrupted labels in our finite query-testing scheme.

Theorem 1. 

For any recovery scheme, a true label vector defined in (1), and transition probability in (2), a lower bound on the error probability $P_E$ such that

$$P_E\ge \frac{H\left(\mathbf{s}\right)-MlogK-1}{NlogK},$$

(13)

where $H\left(·\right)$ is the entropy function and the base of log is 2.

Proof 

(Proof of Theorem 1). Let $\widehat{\mathbf{x}}$ be the estimated true label vector of $\mathbf{x}$ by any recovery scheme. Using Markov chain, we have $\mathbf{x}\to \mathbf{s}\to \left(\mathbf{Q},\mathbf{y}\right)\to \widehat{\mathbf{x}}$. From this processing, we consider the two following inequalities:

$$H\left(\mathbf{x}\left|\mathbf{s}\right.\right)\le H\left(\mathbf{x}\left|\widehat{\mathbf{x}}\right.\right),$$

(14)

and

$$H\left(\mathbf{s}\left|\mathbf{Q},\mathbf{y}\right.\right)\le H\left(\mathbf{x}\left|\widehat{\mathbf{x}}\right.\right).$$

(15)

Using Fano’s inequality, the following bound is obtained:

$$H\left(\mathbf{x}\left|\widehat{\mathbf{x}}\right.\right)\le 1+P_{E}log\left(K^N-1\right).$$

(16)

From (14) and (16), one lower bound on the error probability is

$$P_E\ge \frac{H\left(\mathbf{x}|\mathbf{s}\right)-1}{NlogK},$$

(17)

where once we know the corrupted labels of both entropies are the same, $\mathbf{s}=\mathbf{c}\oplus \mathbf{x}$, there is no randomness. Therefore, (17) is ignored with respect to finding the bound on the error probability. In addition, we have other bounds of $P_E$ using (15) and (16) as follows:

$$P_E\ge \frac{H\left(\mathbf{s}\left|\mathbf{Q},\mathbf{y}\right.\right)-1}{NlogK}.$$

(18)

The conditional entropy $H\left(\mathbf{s}\left|\mathbf{Q},\mathbf{y}\right.\right)$ of (18) can be rewritten as

$$\begin{array}{ccc}\hfill H\left(\mathbf{s}\left|\mathbf{Q},\mathbf{y}\right.\right)& =& H\left(\mathbf{s}\right)-I\left(\mathbf{s};\mathbf{Q},\mathbf{y}\right)\hfill \\ \hfill & =& H\left(\mathbf{s}\right)-\left(I\left(\mathbf{s};\mathbf{Q}\right)+I\left(\mathbf{s};\mathbf{y}\left|\mathbf{Q}\right.\right)\right)\hfill \\ \hfill & \stackrel{\left(a\right)}{=}& H\left(\mathbf{s}\right)-\left(H\left(\mathbf{y}\left|\mathbf{Q}\right.\right)-H\left(\mathbf{y}\left|\mathbf{Q},\mathbf{s}\right.\right)\right)\hfill \\ \hfill & \stackrel{\left(b\right)}{\ge }& H\left(\mathbf{s}\right)-MlogK,\hfill \end{array}$$

(19)

where $I\left(·\right)$ denotes mutual information, and (a) is due to the independence of both terms $\mathbf{s}$ and $\mathbf{Q}$. The last line of (19) is that if we know $\mathbf{Q},\mathbf{s}$, the randomness of $\mathbf{y}$ vanishes, so the conditional entropy $H\left(\mathbf{y}\left|\mathbf{Q},\mathbf{s}\right.\right)=0$. And the following inequality is obtained:

$$H\left(\mathbf{y}\left|\mathbf{Q}\right.\right)\le H\left(\mathbf{y}\right)=MlogK.$$

(20)

Therefore, the lower bound on $P_E$ is

$$P_E\ge \frac{H\left(\mathbf{s}\right)-MlogK-1}{NlogK}.$$

(21)

This is complete for the proof of Theorem 1. □

Considering two bounds (18) and (21), we obtain the lower bound as if $H\left(\mathbf{x}\right)-H\left(\mathbf{s}|\mathbf{x}\right)\ge 0$, then (16) holds; otherwise, (21) is the lower bound on $P_E$. The two parameters p and $\gamma$ from (1) and (2) decide the lower bound as shown by entropy $H\left(\mathbf{s}\right)$ in Theorem 1. At the other extreme, we can see that the closer the two parameters p and $\gamma$ are to zero, the more the error rate converges to zero.

4.2. Upper Bound

In this subsection, we consider an upper bound on the error probability for the MAP decoding in the finite query-testing scheme. For the proof of the upper bound, we divide into two parts the definition of the error event and formulate the probability of decoding the error. Note that $\mathbf{x}$ is the true labels, $\mathbf{s}$ is the corrupted labels, and the response results of their combinations are collected as $\mathbf{y}=\mathbf{Qs}$. We rewrite the a posteriori probability

$$P\left(\mathbf{x}\left|\mathbf{Q},\mathbf{y}\right.\right)\propto \sum _{\mathbf{z}}P\left(\mathbf{x}\right)P\left(\mathbf{z}|\mathbf{x}\right){\mathbb{1}}_{\mathbf{y}=\mathbf{Qz}},$$

(22)

where $\mathbf{z}\in {\mathbb{F}}_K^N$ is a dummy vector that satisfies the condition on $\mathbf{y}=\mathbf{Qz}$. In this work, the decoder knows $\mathbf{Q}$ and $\mathbf{y}$. Using MAP decoding, we estimate the true labels from (22),

$$\widehat{\mathbf{x}}=arg\underset{\mathbf{x}}{max}\sum _{\mathbf{z}}P\left(\mathbf{x}\right)P\left(\mathbf{z}|\mathbf{x}\right){\mathbb{1}}_{\mathbf{y}=\mathbf{Qz}}.$$

(23)

An error event occurs if there exists any feasible vector $\mathbf{x}\ne \overline{\mathbf{x}}$ such that

$$\sum _{\mathbf{z}}P\left(\mathbf{x},\mathbf{z}\right){\mathbb{1}}_{\mathbf{Qs}=\mathbf{Qz}}\le \sum _{\mathbf{z}}P\left(\overline{\mathbf{x}},\mathbf{z}\right){\mathbb{1}}_{\mathbf{Qs}=\mathbf{Qz}}.$$

(24)

So far, we define the error event and now derive an upper bound on the error probability. With a given $\mathbf{x}$ and $\mathbf{s}$, let $P\left(\mathcal{E}|\mathbf{x},\mathbf{s}\right)$ be the conditional error probability. We have an average error probability as follows:

$$P_E=\sum _{\mathbf{x}}\sum _{\mathbf{s}}P\left(\mathbf{x},\mathbf{s}\right)P\left(\mathcal{E}\left|\mathbf{x},\mathbf{s}\right.\right).$$

(25)

We use the two typical sets in [40]. For any $ϵ>0$, let ${\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N$ be the typical set for $\mathbf{x}$ with respect to $P\left(\mathbf{x}\right)$ and ${\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N$ be the conditional typical set for $\mathbf{s}$ distributed with $P\left(\mathbf{s}\right|\mathbf{x})$. For any sufficiently large number N, both typical sets are respectively defined as

$${\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N=\left\{\mathbf{x}\in {\mathbb{F}}_K^N:\left|-\frac{1}{N}logP\left(\mathbf{x}\right)-H\left(\mathbf{x}\right)\right|\le ϵ\right\},$$

(26)

and

$${\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N=\left\{\mathbf{s}\in {\mathbb{F}}_K^N:\left|-\frac{1}{N}logP\left(\mathbf{s}|\mathbf{x}\right)-H\left(\mathbf{s}|\mathbf{x}\right)\right|\le ϵ\right\}.$$

(27)

Note that $H(\mathbf{x},\mathbf{s})=H\left(\mathbf{x}\right)+H\left(\mathbf{s}\right|\mathbf{x})$ if $\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N$ and $\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N$. We denote the joint typical set as ${\mathcal{T}}_{\left[\mathbf{x},\mathbf{s}\right]2ϵ}^N$ and the set of pairs $(\mathbf{x},\mathbf{s})\in {\mathbb{F}}_K^N\times {\mathbb{F}}_K^N$ such that

$$\left|-\frac{1}{N}logP\left(\mathbf{x},\mathbf{s}\right)-H\left(\mathbf{x},\mathbf{s}\right)\right|\le 2ϵ.$$

(28)

As a result, we have two bounds by using the Shannon–McMillan–Breiman theorem [40], $P\left(\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\right)\ge 1-ϵ$ and $P\left((\mathbf{x},\mathbf{s})\in {\mathcal{T}}_{\left[\mathbf{x},\mathbf{s}\right]2ϵ}^N\right)\ge 1-2ϵ$. And the cardinality of the joint typical set is $\left|{\mathcal{T}}_{\left[\mathbf{x},\mathbf{s}\right]2ϵ}^N\right|\le 2^{N\left(H\right(\mathbf{x},\mathbf{s})+2ϵ)}$. To handle the error probability, we decompose into two parts, and find bounds on the probability of decoding error as

$$\begin{array}{cc}\hfill P_E& =\sum _{\left(\mathbf{x},\mathbf{s}\right)\in {\mathcal{T}}_{\left[\mathbf{x},\mathbf{s}\right]2ϵ}^N}P\left(\mathbf{x},\mathbf{s}\right)P\left(\mathcal{E}\left|\mathbf{x},\mathbf{s}\right.\right)+\sum _{\left(\mathbf{x},\mathbf{s}\right)\notin {\mathcal{T}}_{\left[\mathbf{x},\mathbf{s}\right]2ϵ}^N}P\left(\mathbf{x},\mathbf{s}\right)P\left(\mathcal{E}\left|\mathbf{x},\mathbf{s}\right.\right)\hfill \\ \hfill & \stackrel{\left(a\right)}{\le }\sum _{\left(\mathbf{x},\mathbf{s}\right)\in {\mathcal{T}}_{\left[\mathbf{x},\mathbf{s}\right]2ϵ}^N}P\left(\mathbf{x},\mathbf{s}\right)P\left(\mathcal{E}\left|\mathbf{x},\mathbf{s}\right.\right)+\sum _{\left(\mathbf{x},\mathbf{s}\right)\notin {\mathcal{T}}_{\left[\mathbf{x},\mathbf{s}\right]2ϵ}^N}P\left(\mathbf{x},\mathbf{s}\right)\hfill \\ \hfill & \stackrel{\left(b\right)}{\le }\sum _{\left(\mathbf{x},\mathbf{s}\right)\in {\mathcal{T}}_{\left[\mathbf{x},\mathbf{s}\right]2ϵ}^N}P\left(\mathbf{x},\mathbf{s}\right)P\left(\mathcal{E}\left|\mathbf{x},\mathbf{s}\right.\right)+2ϵ,\hfill \end{array}$$

(29)

where (a) is due to $P\left(\mathcal{E}\left|\mathbf{x},\mathbf{s}\right.\right)\le 1$ and (b) comes from the definition of the joint typical set. For all the average query matrices, we consider overall query matrices in (29),

$$P_E\le \sum _{\mathbf{Q}}P\left(\mathbf{Q}\right)\sum _{\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N}\sum _{\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N}P(\mathbf{x},\mathbf{s})P\left(\mathcal{E}\right|\mathbf{x},\mathbf{s},\mathbf{Q})+2ϵ.$$

(30)

where the conditional error probability of (30) is an indicator function as

$$P\left(\mathcal{E}\right|\mathbf{x},\mathbf{s},\mathbf{Q})=\left\{\begin{array}{cc}1& \mathrm{if}\left(29\right)\mathrm{holds},\\ 0& \mathrm{otherwise}.\end{array}\right.$$

(31)

Using a part of Gallager’s error exponents [41], the conditional error probability is bounded by

$$P\left(\mathcal{E}\right|\mathbf{x},\mathbf{s},\mathbf{Q})\le \sum _{\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}}\frac{{\sum }_{{\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N}P(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}){\mathbb{1}}_{\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}}}{{\sum }_{{\mathbf{z}}_{\mathbf{2}}\in {\mathbb{F}}_K^N}P(\mathbf{x},{\mathbf{z}}_{\mathbf{2}}){\mathbb{1}}_{\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{2}}}}.$$

Then, (30) can be rewritten by

$$P_E\le \sum _{\begin{array}{c}\mathbf{Q}\\ \mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\end{array}}P\left(\mathbf{Q}\right)\sum _{\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N}P(\mathbf{x},\mathbf{s})\sum _{\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}}\frac{{\sum }_{{\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N}P(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}){\mathbb{1}}_{\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}}}{{\sum }_{{\mathbf{z}}_{\mathbf{2}}\in {\mathbb{F}}_K^N}P(\mathbf{x},{\mathbf{z}}_{\mathbf{2}}){\mathbb{1}}_{\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{2}}}}+2ϵ.$$

(32)

Considering the right-hand side of (30) for a sufficiently large N, we have the following bound:

$$\frac{{\sum }_{\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N}P(\mathbf{x},\mathbf{s}){\mathbb{1}}_{\mathbf{y}=\mathbf{Qs}}}{{\sum }_{{\mathbf{z}}_{\mathbf{2}}\in {\mathbb{F}}_K^N}P(\mathbf{x},{\mathbf{z}}_{\mathbf{2}}){\mathbb{1}}_{\mathbf{y}={\mathbf{Qz}}_{\mathbf{2}}}}\le 1.$$

(33)

Using the bound (33), (32) is further bounded as

$$\begin{array}{cc}\hfill P_E& \le \sum _{\begin{array}{c}\mathbf{Q}\\ \mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\end{array}}P\left(\mathbf{Q}\right)\sum _{\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N}\sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N\end{array}}P(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}){\mathbb{1}}_{\mathbf{Qs}}={\mathbf{Qz}}_{\mathbf{1}}+2ϵ\hfill \\ \hfill & \le \sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}\sum _{\begin{array}{c}\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N\end{array}}P(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}})\sum _{\mathbf{Q}}P\left(\mathbf{Q}\right){\mathbb{1}}_{\mathbf{Qs}}={\mathbf{Qz}}_{\mathbf{1}}+2ϵ,\hfill \end{array}$$

(34)

where ${\sum }_{\mathbf{Q}}P\left(\mathbf{Q}\right){\mathbb{1}}_{\mathbf{Qs}}={\mathbf{Qz}}_{\mathbf{1}}=P(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}})$. We can rewrite (34) as

$$P_E\le \sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}\sum _{\begin{array}{c}\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N\end{array}}P(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}})P(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}})+2ϵ.$$

(35)

We denote $\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=d$. For example, $d=0$, then $P(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}})=1$. We decompose (35) into two parts when $d=0$ or not:

$$P_E\le P_{E1}+P_{E2}+2ϵ,$$

(36)

where

$$P_{E1}:=\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}\sum _{\begin{array}{c}{\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N\end{array}}P(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}),$$

and

$$P_{E2}:=\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}\sum _{\begin{array}{c}\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N\setminus \mathbf{s}\end{array}}P(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}})P(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}).$$

Theorem 2. 

For our MAP decoding, a sufficiently large number N, vanishing the error probability $P_E$, the upper bound on the number of queries M is greater than

$$\begin{array}{c}\hfill \frac{M}{N}>\frac{H(\mathbf{x},\mathbf{s})+2ϵ}{log{\left(K^{-1}+\left(1-K^{-1}\right){\left(1-\frac{\delta }{1-K^{-1}}\right)}^{\lceil \alpha N\rceil }\right)}^{-1}},\end{array}$$

(37)

where $0<\alpha <0.5$.

4.3. Proof of Theorem 2

For $P_{E1}$, we have the following under $\mathbf{x}\ne \overline{\mathbf{x}}$:

$$P_{E1}=\sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}P\left(\overline{\mathbf{x}}\right)\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N\end{array}}P\left({\mathbf{z}}_{\mathbf{1}}\right|\overline{\mathbf{x}}),$$

(38)

where (38) can be divided into two parts as $P_{E1}=P_{E11}+P_{E12}$ by the typical set ${\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N$. Due to the assumption $\mathbf{x}\ne \overline{\mathbf{x}}$, we have

$$\begin{array}{cc}\hfill P_{E11}& =\sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\end{array}}P\left(\overline{\mathbf{x}}\right)\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\setminus \overline{\mathbf{x}}\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N\end{array}}P\left({\mathbf{z}}_{\mathbf{1}}\right|\overline{\mathbf{x}})\hfill \\ \hfill & \le \sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\end{array}}P\left(\overline{\mathbf{x}}\right)\sum _{\begin{array}{c}{\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N\setminus {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\end{array}}P\left({\mathbf{z}}_{\mathbf{1}}\right|\overline{\mathbf{x}})\hfill \\ \hfill & \le ϵ,\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill P_{E12}& =\sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\end{array}}P\left(\overline{\mathbf{x}}\right)\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N\end{array}}P\left({\mathbf{z}}_{\mathbf{1}}\right|\overline{\mathbf{x}})\hfill \\ \hfill & \le \sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\end{array}}P\left(\overline{\mathbf{x}}\right)\hfill \\ \hfill & \le ϵ.\hfill \end{array}$$

Then, we consider the second part $P_{E2}$ and enumerate the same probability with the Hamming weight, $\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=d$:

$$P_{E2}=\sum _{d=1}^N\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}\sum _{\begin{array}{c}\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d\end{array}}P\left(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}\right)P\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=d\right.\right).$$

(39)

Note that $P\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=d\right.\right)$ comes from (12):

$$\begin{array}{cc}\hfill P{\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=d\right.\right)}^M& ={\left(K^{-1}+\left(1-K^{-1}\right){\left(1-\frac{\delta }{1-K^{-1}}\right)}^d\right)}^M\hfill \\ \hfill & \le {\left(K^{-1}+\left(1-K^{-1}\right)\left(1-\frac{\delta }{1-K^{-1}}\right)\right)}^M\hfill \\ \hfill & ={\left(1-\delta \right)}^M,\hfill \end{array}$$

(40)

where the second inequality is due to $P\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=d\right.\right)$ being a monotonically decreasing function as d increases.

Next, for $0<\alpha <0.5$, (39) can be decomposed into two probabilities as follows $P_{E2}=P_{E21}+P_{E22}$:

$$P_{E21}=\sum _{d=1}^{\lfloor \alpha N\rfloor }\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}\sum _{\begin{array}{c}\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d\end{array}}P\left(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}\right)P\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=d\right.\right),$$

(41)

and

$$P_{E22}=\sum _{d=\lceil \alpha N\rceil }^N\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}\sum _{\begin{array}{c}\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d\end{array}}P\left(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}\right)P\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=d\right.\right).$$

(42)

First, we consider the probability $P_{E21}$,

$$\begin{array}{cc}\hfill P_{E21}& =\sum _{d=1}^{\lfloor \alpha N\rfloor }\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}\sum _{\begin{array}{c}\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d\end{array}}P\left(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}\right)P\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=d\right.\right)\hfill \\ \hfill & \le \sum _{d=1}^{\lfloor \alpha N\rfloor }\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}\sum _{\begin{array}{c}\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d\end{array}}P\left(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}\right)P\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=1\right.\right)\hfill \\ \hfill & ={\left(1-\delta \right)}^M\sum _{d=1}^{\lfloor \alpha N\rfloor }\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}\sum _{\begin{array}{c}\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d\end{array}}P\left(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}\right).\hfill \end{array}$$

(43)

In (43), we change the order of summation and take the result in (40). And we further bound

$$\begin{array}{cc}\hfill P_{E21}& \le {\left(1-\delta \right)}^M\sum _{d=1}^{\lfloor \alpha N\rfloor }\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}\sum _{\begin{array}{c}\mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\end{array}}P\left(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}\right)\sum _{{\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d}\mathbb{1}\hfill \\ \hfill & \le {\left(1-\delta \right)}^M\sum _{d=1}^{\lfloor \alpha N\rfloor }\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\end{array}}\sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}P\left(\overline{\mathbf{x}}\right)\sum _{{\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d}\mathbb{1}\hfill \\ \hfill & \le {\left(1-\delta \right)}^M\sum _{d=1}^{\lfloor \alpha N\rfloor }\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\end{array}}\sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}P\left(\overline{\mathbf{x}}\right)\left|\left\{{\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d\right\}\right|,\hfill \end{array}$$

(44)

where $\mathbb{1}$ denotes the indicator function. For the cardinality of the conditional set in the last line of (44), we find its bound as,

$$\begin{array}{cc}\hfill \left|\left\{{\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d\right\}\right|& =\left(\genfrac{}{}{0pt}{}{N}{d}\right){(K-1)}^d\hfill \\ \hfill & \le 2^{NH(d/N)}{(K-1)}^d.\hfill \end{array}$$

Because the entropy $H(d/N)$ is a monotonically increasing function up to $d/N<1/2$ and the size of the typical set comes from [40], we obtain the exponent terms as follows:

$$\begin{array}{cc}\hfill P_{E21}& \le {\left(1-\delta \right)}^M\sum _{d=1}^{\lfloor \alpha N\rfloor }\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\end{array}}\sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}P\left(\overline{\mathbf{x}}\right)\left|\left\{{\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d\right\}\right|\hfill \\ \hfill & \le {\left(1-\delta \right)}^M\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\end{array}}\sum _{d=1}^{\lfloor \alpha N\rfloor }{2}^{NH(d/N){(K-1)}^d}\hfill \\ \hfill & \le {\left(1-\delta \right)}^M{2}^{N\left(H\right(\mathbf{x},\mathbf{s})+2ϵ)}\alpha N{2}^{NH\left(\alpha \right)}{\left(K-1\right)}^{\alpha N}.\hfill \end{array}$$

(45)

The exponent term of the last line of (45) is

$$-N\left(-\frac{M}{N}log(1-\delta )-H(\mathbf{x},\mathbf{s})-2ϵ-H\left(\alpha \right)-\frac{log\left(\alpha N\right)}{N}-\alpha log(K-1)\right).$$

(46)

In order to vanish the error probability $P_{E21}$ such that the inner term of (46) is greater than 0, taking $N\to \infty$, the following exponent has the bound of $M/N$:

$$\begin{array}{c}\hfill \frac{M}{N}>\frac{H(\mathbf{x},\mathbf{s})+H\left(\alpha \right)+\alpha log(K-1)+log\left(\alpha N\right)/N+2ϵ}{log{(1-\delta )}^{-1}}.\end{array}$$

(47)

Next, for $P_{E22}$, we find the bound using the same way for $P_{E21}$:

$$\begin{array}{cc}\hfill P_{E22}& =\sum _{d=\lceil \alpha N\rceil }^N\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\end{array}}\sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d\end{array}}P\left(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}\right)P\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=d\right.\right)\hfill \\ \hfill & \le \sum _{d=\lceil \alpha N\rceil }^N\sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\end{array}}\sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\\ {\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N:{\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}\parallel }_0=d\end{array}}P\left(\overline{\mathbf{x}},{\mathbf{z}}_{\mathbf{1}}\right)P\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=\lceil \alpha N\rceil \right.\right)\hfill \\ \hfill & \le \sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\end{array}}\left(\sum _{\begin{array}{c}\overline{\mathbf{x}}\in {\mathbb{F}}_K^N\setminus \mathbf{x}\end{array}}P\left(\overline{\mathbf{x}}\right)\right)\left(\sum _{\begin{array}{c}{\mathbf{z}}_{\mathbf{1}}\in {\mathbb{F}}_K^N\end{array}}P\left({\mathbf{z}}_{\mathbf{1}}\right)\right)P\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=\lceil \alpha N\rceil \right.\right)\hfill \\ \hfill & \le \sum _{\begin{array}{c}\mathbf{x}\in {\mathcal{T}}_{\left[\mathbf{x}\right]ϵ}^N\\ \mathbf{s}\in {\mathcal{T}}_{\left[\mathbf{s}|\mathbf{x}\right]ϵ}^N\end{array}}P\left(\mathbf{Qs}={\mathbf{Qz}}_{\mathbf{1}}\left|\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=\lceil \alpha N\rceil \right.\right)\hfill \\ \hfill & \le 2^{N\left(H\right(\mathbf{x},\mathbf{s})+2ϵ)}{\left(K^{-1}+\left(1-K^{-1}\right){\left(1-\frac{\delta }{1-K^{-1}}\right)}^{\lceil \alpha N\rceil }\right)}^M,\hfill \end{array}$$

(48)

where $\parallel \mathbf{s}-{\mathbf{z}}_{\mathbf{1}}{\parallel }_0=d$ is ignored and the bound of the typical set is used. Finally, we find the exponent of the right-hand side of (48). For $N\to \infty$, vanishing the error probability, the exponent of (48) is bounded as

$$\begin{array}{c}\hfill \frac{M}{N}>\frac{H(\mathbf{x},\mathbf{s})+2ϵ}{log{\left(K^{-1}+\left(1-K^{-1}\right){\left(1-\frac{\delta }{1-K^{-1}}\right)}^{\lceil \alpha N\rceil }\right)}^{-1}}.\end{array}$$

(49)

4.4. Discussion

In this subsection, we discuss our findings from several perspectives. The first of these is that the upper bound on the error probability obtained by the MAP decoding is as shown in (37), where $P_{E1}$ is negligible if $ϵ$ is trivially small. On the other hand, $P_{E2}$ is expressed as the sum of two probabilities, $P_{E2}=P_{E21}+P_{E22}$. Let us see which of these is the more significant factor. From (40), the denominator of the right-hand side of (49) satisfies the following inequality:

$$log{(1-\delta )}^{-1}\le log{\left(K^{-1}+\left(1-K^{-1}\right){\left(1-\frac{\delta }{1-K^{-1}}\right)}^{\lceil \alpha N\rceil }\right)}^{-1}$$

So we conclude an upper bound on the error probability as in Theorem 2.

Theorems 1 and 2 state the lower and upper bounds on the error probability for recovering corrupted data, respectively. Combining the two conditions, we see that the upper bound should be larger than the lower bound. In other words, the range for $\alpha$ mentioned in Theorem 2 is determined from two conditions. Using the vanishing of the error probability as $N\to \infty$, the ratio of the number of queries in $M/N$ for Theorem 1 to hold is simply bounded as follows:

$$\frac{M}{N}>\frac{H\left(\mathbf{s}\right)}{logK}$$

(50)

From (37) and (50), we see that the upper bound requires a large number of queries to recover corrupted data. This leads to the conclusion that the upper bound is expressed as a function of the joint entropy $H(\mathbf{x},\mathbf{s})$. As mentioned earlier, the joint entropy is represented as the conditional entropy $H\left(\mathbf{s}\right|\mathbf{x})$ and the entropy $H\left(\mathbf{x}\right)$ that both the transition and generation of the true label vector are a form of compression. In addition, consider the case of $K=2$ as a binary classification, where our lower bound is a necessary condition for the number of queries that matches the result of Theorem 5 in [27].

5. Conclusions

We considered the problem of recovering corrupted labels in supervised learning. However, a subset of the data samples was replaced with incorrect labels due to the non-expert’s lack of skill and experience. In this paper, we proposed the finite query-testing scheme. Then, we aimed to recover corrupted data samples with additional expert labeling tasks. Our testing system worked on identifying corrupted data samples with minimal expert queries and recovering them to their true label values. In this work, we considered a probabilistic model for generating queries based on combinations on finite fields. Our finite query-testing scheme utilized queries that combined random subsets of data samples. Our goal was to recover corrupted data samples with a minimum number of queries. To this end, we obtained lower bounds using Fano’s inequality. In addition, we obtained upper bounds on the probability of error using the MAP decoding method. We showed the number of queries required for recovery, not only for binary classification but also for corrupted data in multiple classification problems. In this work, we were not able to verify the applicability of our scheme in an unsupervised learning environment, but we suggest further research to extend it to semi-supervised and unsupervised in future studies, where our proposed finite query-testing scheme may work.