Nonparametric Additive Regression for High-Dimensional Group Testing Data

Abstract

Group testing has been verified as a cost-effective and time-efficient approach, where the individual samples are pooled with a predefined group size for subsequent testing. Recent research has explored the integration of covariate information to improve the modeling of the group testing data. While existing works for high-dimensional data primarily focus on parametric models, this study considers a more flexible generalized nonparametric additive model. Nonlinear components are approximated using B-splines and model estimation under the sparsity assumption is derived employing group lasso. Theoretical results demonstrate that our method selects the true model with a high probability and provides consistent estimates. Numerical studies are conducted to illustrate the good performance of our proposed method, using both simulated and real data.

1. Introduction

Group testing is a proven cost-effective and time-efficient approach. In group testing, individual samples are pooled with a predefined group size and subsequently tested. Only those pools yielding positive results require additional tests, including individual tests. Dorfman [1] introduced group testing to screen syphilis among soldiers. Afterward, it has been widely applied to detect infected samples in a large population, including a variety of infectious diseases such as chlamydia, gonorrhea [2,3,4], HIV, the hepatitis B virus [5], West Nile virus [6], SARS-CoV-2 [7], and COVID-19 [8]. Group testing is also used in many other fields such as genetics [9], blood safety [10], drug discovery [11], and hyopneumoniae [12]. In particular, the efficiency of this approach is remarkable, especially when the prevalence of the disease is low. Gregory et al. [13] noted that the State Hygienic Laboratory (SHL) in Iowa saved approximately $3.1 million from 2009 to 2014 in screening female subjects for chlamydia. Verougstraete et al. [4] evaluated the feasibility of CT and NG molecular testing in a large number of female sex workers and showed that pool testing compared with single-site samples reduced the cost by $35\%$.

Given the rapid advancement of digital technologies, it is possible to access extensive covariates, encompassing personal details such as age and occupation, electronic medical records, genetic data, and environmental variables. In recent literature on group testing, covariate information has been explored for estimation and case identification [14,15,16]. Vansteelandt et al. [17] proposed estimating individual infection probabilities by utilizing the test results obtained from initial pools. Xie [18] developed a general regression methodology that relates the group testing responses to individual covariate information. Black et al. [19] improved the Halving group testing procedure by taking into account population heterogeneity. It is evident that covariant information improves the estimate of individual risk probability, further reduces the test number, and lowers the cost [20].

Most existing research on group testing by incorporating covariate information has been developed under a parametric regression model, such as the logistic model. Zhang et al. [2] addressed the goal of case identification by incorporating retesting results. McMahan et al. [21] developed a Bayesian framework for group testing with covariate information. Gregory et al. [13] used the adaptive elastic net for group testing with high-dimensional covariates. Lin et al. [3] considered variable selection for multiple-infection group testing data. Alternatively, semiparametric or nonparametric regression has also received attention, but it mainly focuses on low-dimensional group testing. Yuan et al. [22] considered semiparametric isotonic regression models for the simultaneous estimation of the conditional probability curve and covariate effects, in which an expectation–maximization (EM) algorithm is used to overcome the computational challenge. Delaigle et al. considered nonparametric regression methods for group testing data [23,24,25]. Liu et al. [26] developed a Bayesian generalized additive regression approach to model the individual-level probability of disease. However, rare work has been conducted on high-dimensional covariates within the semiparametric or nonparametric regression framework for group testing.

In a standard group testing procedure, test results are collected at every stage, and high-dimensional covariate information is obtained during the initial enrollment of specimens. To maximize the utility of this extensive data, we propose an integrated modeling framework that employs test results from every phase of the group testing algorithm and combines them with high-dimensional covariate information. We employ a generalized nonparametric additive model framework to analyze the heterogeneous high-dimensional group testing data. The unknown nonlinear components are approximated with B-splines [27] and estimated by the group lasso method [28]. Four commonly used group testing algorithms are considered: master pool, Dorfman, Halving, and array testing [1,29,30,31]. To enhance computational efficiency, we utilize the EM algorithm in conjunction with the stochastic gradient descent algorithm. Theoretical results demonstrate that our method selects the true model with high probability and provides consistent estimates for components.

The remaining text is organized as follows. In Section 2, we establish our estimation method under the generalized nonparametric additive model through the B-spline approximation, along with the corresponding algorithm. Subsequently, the theoretical properties of the proposed method are investigated and proved. In Section 3, we derive the conditional expectation utilized in the EM algorithm to enhance computational efficiency. In Section 4 and Section 5, we conduct extensive simulations and make some real applications. The numerical result shows the good performance of our method. We conclude our methods in Section 6. The technical details of the proofs are presented in Appendix A.

2. Main Results

2.1. Notation

Throughout this work, we define ${\mathbb{R}}^d$ as the set of d-dimensional real number vectors. We denote ${\lambda }_{min}\left(\mathit{A}\right)$ and ${\lambda }_{max}\left(\mathit{A}\right)$ as the minimum and maximum eigenvalues of matrix $\mathit{A}$, respectively. Furthermore, we employ $\left[m\right]$ to define the set $\{1,\cdots ,m\}$. The notation ${\parallel ·\parallel }_{\infty }$ represents the sup norm. Let ${ϵ}_n=o\left(1\right)$ indicate that ${ϵ}_n\to 0.$ We use $E(·)$ to denote the expectation concerning the underlying probability space. The symbol $\stackrel{d}{\to }$ denotes the convergence in distribution. We denote $a_n\succ b_n$ to signify the relationship $\frac{b_n}{a_n}=o\left(1\right).$ Let ${\parallel f\parallel }^2=E\left(f^2\right)$ and ${\parallel f\parallel }_n^2=E_n\left(f^2\right)$ with $E_n$ represent the empirical measures. The superscript “T” indicates the transpose of a matrix or a vector.

2.2. Model

Suppose that n individuals are recruited to participate in testing. We denote by ${\tilde{Y}}_i$ the true status of the i-th individual, indicating if it is infected or not (${\tilde{Y}}_i=1$ or 0) for $i\in \left[n\right]$. Then ${\tilde{Y}}_i$ follows a Bernoulli distribution with a risk probability of $p_i$. Simultaneously, we gather and express individual information as ${\mathit{X}}_i={(X_{i1},\cdots ,X_{i{q}_n})}^T\in {\mathbb{R}}^{q_n}$ for the $i^{th}$ individual, where ${\mathit{X}}_i$ has a dimension of $q_n$. Assuming that ${\tilde{Y}}_i$ follows the generalized additive model (GAM) given ${\mathit{X}}_i$,

$$\mathrm{P}({\tilde{Y}}_i=1|{\mathit{X}}_i)=g(\mu +\sum _{j=1}^{q_n}{f}_j\left(X_{ij}\right)),$$

(1)

where the link function is logit, $g\left(t\right)={\displaystyle \frac{e^t}{1+e^t}}$, $\mu$ represents the intercept, and $f_j$ ($j\in \left[q_n\right]$) is an unknown smooth nonlinear function [27,32,33]. The corresponding true values are denoted as ${\mu }^{*}$ and $f_j^{*}$, respectively. The dimension of covariates $\mathit{X}$ is notably large and increases alongside n. Typically, there is an assumption of model sparsity, which implies that only a subset of the covariates $\mathit{X}$ correlates with the response variable. Let the true model be represented as ${\mathsf{\Omega }}^{*}=\{j\in \left[q_n\right]:f_j^{*}(·)\ne 0\}$, where its size is denoted by $s_0$. Without loss of generality, we assume that the unknown additive component $f_j^{*}$ is centered, meaning that $E\left[f_j^{*}\left(X_j\right)\right]=0$.

In group testing, the true status of an individual is not directly observable; instead, tests are performed at the group level. Moreover, for groups flagged as high-risk due to positive results, additional tests, including individual tests, may be conducted, as individuals within these groups are more likely to be infected [34]. A typical group testing algorithm usually contains two steps. First, partition the specimens into J groups, and perform tests for each group. Second, conduct additional tests for the groups that tested positive in the first step. We denote the J groups as ${\left\{{\mathcal{P}}_{j,1}\right\}}_{j=1}^J$ with group sizes ${\left\{k_j\right\}}_{j=1}^J$, satisfying ${\cup }_{j=1}^J{\mathcal{P}}_{j,1}=\{1,\dots ,n\}$. We define ${\mathcal{P}}_j=({\mathcal{P}}_{j,1},\cdots ,{\mathcal{P}}_{j,T_j})$, where ${\mathcal{P}}_{j,1}$ denotes the initial $j^{th}$ group, and ${\mathcal{P}}_{j,l}$ denotes the subsequent subgroups or individuals within this group for $l=2,\cdots ,T_j$, with $T_j$ representing the total number of tests associated with the $j^{th}$ group. In Figure 1, we present a typical example of employing the Halving algorithm with a group size of 6, where the number is $T_j=6$. Correspondingly, let ${\mathit{Z}}_j$ encompass all the test results associated with ${\mathcal{P}}_{j,1}$, including subgroup and individual tests, denoted by ${\mathit{Z}}_j=(Z_{j,1},\cdots ,Z_{j,T_j})$.

Considering measurement error, we denote sensitivity and specificity by $S_e$ and $S_p$, respectively. Sensitivity is the probability that a test correctly identifies a positive sample as positive, while specificity is the probability that a test correctly identifies a negative sample as negative. We assume these values are known and do not depend on the group size. We denote ${\tilde{\mathit{Z}}}_j=({\tilde{Z}}_{j,1},\cdots ,{\tilde{Z}}_{j,T_j})$ as the true status corresponding to ${\mathit{Z}}_j$, where ${\tilde{Z}}_{j,l}={max}_{i\in {\mathcal{P}}_{j,l}}{\tilde{Y}}_i$. Given the true status ${\tilde{Z}}_{j,l}$, the outcome $Z_{j,l}$ follows a Bernoulli distribution with probability ${{\mathrm{S}}_{\mathrm{e}}}^{{\tilde{Z}}_{j,l}}{(1-{\mathrm{S}}_{\mathrm{p}})}^{1-{\tilde{Z}}_{j,l}}$ for $l\in \left[T_j\right]$ and $j\in \left[J\right].$ They are conditionally independent given their true status. Given ${\mathit{X}}_1,\cdots ,{\mathit{X}}_n$, the individual statuses ${\tilde{Y}}_1,\cdots ,{\tilde{Y}}_n$ are conditionally independent. Similarly, the group statuses $\{{\tilde{Z}}_{j,l},l\in \left[T_j\right]\}$ are conditionally independent, given $\{{\tilde{Y}}_i,i\in {\mathcal{P}}_{j,1}\}.$ These assumptions are commonly made in the literature [13,16]. Let $\tilde{\mathit{Y}}=({\tilde{Y}}_1,\cdots ,{\tilde{Y}}_n)$ and $\mathbb{X}={({\mathit{X}}_1,\cdots ,{\mathit{X}}_n)}^T$. Then, the log-likelihood function of $\tilde{\mathit{Y}}$ is expressed as follows:

$$\begin{array}{c}\hfill ln\mathrm{P}\left(\tilde{\mathit{Y}}\right|\mathbb{X})=\sum _{i=1}^n[{\tilde{Y}}_{i}ln\left(p_i\right)+(1-{\tilde{Y}}_i)ln(1-p_i)],\end{array}$$

(2)

where $p_i=\mathrm{P}({\tilde{Y}}_i=1|{\mathit{X}}_i)=g(\mu +{\sum }_{j=1}^{q_n}{f}_j\left(X_{ij}\right)).$ We note that the individual true status $\tilde{\mathit{Y}}$ is unknown, whereas the group-level test results $\mathit{Z}=\{{\mathit{Z}}_j,j\in \left[J\right]\}$ are observed. The corresponding log-likelihood function is as follows:

$$\begin{array}{c}\hfill \mathrm{P}\left(\mathit{Z}\right|\mathbb{X})={\displaystyle \sum _{\tilde{\mathit{Y}}\in {\{0,1\}}^n}}P\left(\mathit{Z}\right|\tilde{\mathit{Y}})P\left(\tilde{\mathit{Y}}\right|\mathbb{X}),\end{array}$$

(3)

where the conditional probability $P\left(\mathit{Z}\right|\tilde{\mathit{Y}})$ is formulated as follows:

$$\begin{array}{ccc}\hfill P\left(\mathit{Z}\right|\tilde{\mathit{Y}})& =\prod _{j=1}^{J}P\left({\mathit{Z}}_j\right|\tilde{\mathit{Y}})\hfill & \hfill =\prod _{j=1}^J\prod _{l=1}^{T_j}{\left\{S_e^{{\tilde{Z}}_{j,l}}{(1-S_p)}^{1-{\tilde{Z}}_{j,l}}\right\}}^{Z_{j,l}}{\left\{{(1-S_e)}^{{\tilde{Z}}_{j,l}}{S}_p^{1-{\tilde{Z}}_{j,l}}\right\}}^{1-Z_{j,l}}.\end{array}$$

With the above likelihood function, our goal is to estimate the true model ${\mathsf{\Omega }}^{*}$ and obtain an accurate estimate of the risk probability through the model (1). As model (1) is a nonparametric additive model, it prompts the use of the B-spline method, known for its effectiveness in modeling such nonparametric frameworks.

Let $S_n$ denote the space of polynomial splines of degree $l\ge 1$, and let ${\varphi }_{jk}(·),$ $k\in \left[m_n\right]$ denote the standardized B-spline basis functions satisfying $\parallel {\varphi }_{jk}{\parallel }_{\infty }\le 1$. Then for any $f_{nj}\in S_n,$ there exist coefficients ${\beta }_{jk},k\in \left[m_n\right]$, such that

$$\begin{array}{c}\hfill f_{nj}\left(x\right)=\sum _{k=1}^{m_n}{\beta }_{jk}{\varphi }_{jk}\left(x\right).\end{array}$$

Using $f_{nj}\left(x\right)$ allows us to approximate the nonlinear function $f_j\left(x\right)$. Let ${\mathit{\beta }}_j=({\beta }_{j1},\cdots ,$${\beta }_{j{m}_n}{)}^T$ and ${\mathbf{\Phi }}_{ij}={({\varphi }_{j1}\left(X_{ij}\right),\cdots ,{\varphi }_{j{m}_n}\left(X_{ij}\right))}^T$. Then $\mu +{\sum }_{j=1}^{q_n}{f}_j\left(X_{ij}\right)$ could be approximated by ${\mathbf{\Phi }}_i^T\mathit{\beta }$, where ${\mathbf{\Phi }}_i={(1,{\mathbf{\Phi }}_{i1}^T,\cdots ,{\mathbf{\Phi }}_{i{q}_n}^T)}^T$ and $\mathit{\beta }={({\beta }_0,{\mathit{\beta }}_1^T,\cdots ,{\mathit{\beta }}_{q_n}^T)}^T$ includes the intercept ${\beta }_0$. Therefore, the risk probability $p_i=g(\mu +{\sum }_{j=1}^{q_n}{f}_j\left(X_{ij}\right))$ is approximated by $p_{i\mathit{\beta }}=g\left({\mathbf{\Phi }}_i^T\beta \right)$ for $i\in \left[n\right]$. Consequently, the corresponding likelihood function is

$$P_{\mathit{\beta }}\left(\mathit{Z}\right|\mathbb{X})={\displaystyle \sum _{\tilde{\mathit{Y}}\in {\{0,1\}}^n}}P\left(\mathit{Z}\right|\tilde{\mathit{Y}})P_{\mathit{\beta }}\left(\tilde{\mathit{Y}}\right|\mathbb{X}),$$

with $P_{\mathit{\beta }}$ represents the probability measure with respect to the parameter $\mathit{\beta }$ and

$$\begin{array}{c}\hfill ln{P}_{\mathit{\beta }}\left(\tilde{\mathit{Y}}\right|\mathbb{X})=\sum _{i=1}^n[{\tilde{Y}}_{i}ln({\displaystyle \frac{p_{i\mathit{\beta }}}{1-p_{i\mathit{\beta }}}})+ln(1-p_{i\mathit{\beta }})].\end{array}$$

(4)

Given the high dimensionality of $\mathbb{X}$, we propose maximizing $P_{\mathit{\beta }}\left(\mathit{Z}\right|\mathbb{X})$ using the group lasso penalty to select the important components and obtain the estimator:

$$\begin{array}{c}\hfill \widehat{\mathit{\beta }}=arg\underset{\mathit{\beta }}{max}\{ln{P}_{\mathit{\beta }}\left(\mathit{Z}\right|\mathbb{X})-n\lambda \sum _{j=1}^{q_n}\parallel {\mathit{\beta }}_j{\parallel }_2\},\end{array}$$

(5)

where $\lambda$ is the tuning parameter, and the intercept ${\beta }_0$ is not penalized. The model ${\mathsf{\Omega }}^{*}$ is estimated by $\widehat{\mathsf{\Omega }}=\{j\in \left[q_n\right]:\parallel {\widehat{\mathit{\beta }}}_j{\parallel }_2\ne 0\}$, and the nonlinear components are estimated by ${\widehat{f}}_{nj}\left(x\right)={\sum }_{k=1}^{m_n}{\widehat{\beta }}_{jk}{\varphi }_{jk}\left(x\right),j\in \left[q_n\right].$

2.3. The Estimation Procedure with EM Algorithm

The individual status $\tilde{\mathit{Y}}$ is latent and not directly observable, posing a challenge in maximizing (5) to obtain the estimator. To address this issue, we employ the EM algorithm, which iterates with two steps: the E-step and the M-step. For the ${(t+1)}^{th}$ iteration, these two steps are carried out as follows: in the E-step, we calculate the conditional expectation of the complete log-likelihood function of $\tilde{\mathit{Y}}$, given the observed data $\mathit{Z}$ and the $t^{th}$ iteration ${\mathit{\beta }}^{\left(t\right)}$, denoted as follows: $\mathrm{E}[lnf\left(\tilde{Y}\right|\mathit{\beta })|\mathit{Z},{\mathit{\beta }}^{\left(t\right)}]$. In the M-step, we maximize the regularized function $S^{\left(t\right)}\left(\mathit{\beta }\right)=\mathrm{E}[lnf\left(\tilde{Y}\right|\mathit{\beta })|\mathit{Z},{\mathit{\beta }}^{\left(t\right)}]-n\lambda {\sum }_{s=1}^{q_n}{\parallel {\mathit{\beta }}_s\parallel }_2$, and update the parameter with ${\mathit{\beta }}^{(t+1)}$. Our method is summarized in Algorithm 1.

Algorithm 1 Regularized GAM for group testing.

Input: The observed outcomes $\mathit{Z}$, covariates $\mathbb{X}$, and the initial value ${\mathit{\beta }}^{\left(0\right)}$. Compute: Step 1 (E-step). Given the parameter at the $t^{th}$ iteration, ${\mathit{\beta }}^{\left(t\right)}$, we calculate the conditional expectation $\mathrm{E}[lnf\left(\tilde{Y}\right|\mathit{\beta })|\mathit{Z},{\mathit{\beta }}^{\left(t\right)}]$. Step 2 (M-step). We update the iterative parameter ${\mathit{\beta }}^{(t+1)}$ by maximizing the objective function $S^{\left(t\right)}\left(\mathit{\beta }\right)=\mathrm{E}\left\{{\sum }_{i=1}^n[{\tilde{Y}}_{i}ln({\displaystyle \frac{p_{i\mathit{\beta }}}{1-p_{i\mathit{\beta }}}})+ln(1-p_{i\mathit{\beta }})]|\mathit{Z},{\mathit{\beta }}^{\left(t\right)}\right\}-n\lambda {\sum }_{s=1}^{q_n}{\parallel {\mathit{\beta }}_s\parallel }_2$. Step 3. We repeat the E-step and M-step until the parameters converge. Output: The estimate $\widehat{\mathit{\beta }}$, ${\widehat{f}}_{nj},j\in \left[q_n\right]$ and $\widehat{\mathsf{\Omega }}$.

We should note that the conditional expectation in the E-step exhibits variability across different group testing methods. For a comprehensive understanding, specific details are presented in Section 3 to facilitate the implementation of the EM algorithm.

2.4. The Consistency of the Estimate

In this section, we establish the theoretical results of our method. We commence by presenting essential conditions that are required to establish our main results. These conditions are commonly employed in B-spline methodologies. Within Conditions D and E, a set, $U_m$, is introduced and defined as follows: the results $\{{\mathit{Z}}_1,\cdots ,{\mathit{Z}}_J\}$ are categorized into M classes, with $U_m$ representing the index set to which ${\mathit{Z}}_j$ belongs to the m-th category for $m\in \left[M\right]$.

$A.$

The nonparametric components ${\left\{f_j^{*}\right\}}_{j=1}^{q_n}$ belong to the function class $\mathcal{F}$, where the $r^{th}$ derivative $f^{\left(r\right)}$ exists and is Lipschitz of order $\alpha$,

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathcal{F}=\{f(·):|f^{\left(r\right)}\left(s\right)-f^{\left(r\right)}\left(t\right)|\le K|s-t{|}^{\alpha },\forall s,t\in [a,b]\}\end{array}\end{array}$$

for some positive constant K, where r is a non-negative integer, $[a,b]$ is the support of the covariate, and $\alpha \in (0,1]$, such that $d=r+\alpha \ge 0.5.$

$B.$

$\mathrm{E}{f}_j^{*}=0$, and the marginal density function $g_j$ of $X_j$ adheres to $0<K_1\le g_j\left(X_j\right)\le K_2<\infty$ on $[a,b]$ for $1\le j\le p$, where $K_1$ and $K_2$ are constants.

$C.$

${min}_{j\in {\mathsf{\Omega }}^{*}}\{\parallel f_j^{*}{\parallel }_2\}\succ q_n{n}^{-\frac{2d-1}{2(2d+1)}},$ with $q_n=o\left(n^{\frac{3}{4}-\frac{1}{2(2d+1)}}\right).$

$D.$

$|U_m|/J\to {\gamma }_m\in (0,1)$ as $J\to \infty$.

$E.$

$I_m\left({\mathit{\beta }}^{*}\right)=-\frac{1}{|U_m|}{\sum }_{j\in U_m}{\displaystyle \frac{{\partial }^{2}log{P}_{\mathit{\beta }}\left({\mathit{Z}}_j\right|\mathbb{X})}{\partial \mathit{\beta }\partial {\mathit{\beta }}^T}}{|}_{\mathit{\beta }={\mathit{\beta }}^{*}}$ is positive-definite and possesses a minimum eigenvalue larger than ${\tau }_m$ for any $m\in \left[M\right]$.

Conditions A–C are commonly employed for nonparametric additive models, with similar assumptions made by Huang et al. [35]. When the dimension $q_n$ is of the order $O\left(1\right)$, Condition C is less stringent than Condition (A1) as described in Huang et al. [35]. Conditions D and E are presumed to pertain to the high-dimensional group testing data, guaranteeing an adequate dataset size for all categories of testing results, as employed in work by Gregory et al. [13].

Let the coefficients ${\mathit{\beta }}^{*}$ represent the population-level parameter used to approximate the nonparametric function, given by

$${\mathit{\beta }}^{*}=arg\underset{\mathit{\beta }}{min}{E}_Z{P}_{\mathit{\beta }}\left(\mathit{Z}\right|\mathbb{X}).$$

Under mild conditions, our penalized estimate $\widehat{\mathit{\beta }}$ converges to ${\mathit{\beta }}^{*}$, as demonstrated in the following theorem:

Theorem 1. 

Given Conditions A–E, $m_n=O\left(n^{\frac{1}{2d+1}}\right)$ and $\lambda =o(q_n^{-1/2}{n}^{\gamma /2}/\sqrt{n})$, we have

$${\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}^{*}\parallel }_2=O_p\left(q_n^{-1/2}{n}^{-(1-\gamma )/2}\right),$$

where $\gamma =\frac{1}{2d+1}$.

This theorem reveals that the difference between ${\mathit{\beta }}^{*}$ and its estimate converges to zero in probability, achieving a convergence rate of $O_p\left(q_n^{-1/2}{n}^{-(1-\gamma )/2}\right)$. According to Huang et al. [35], we set $m_n=O\left(n^{\frac{1}{2d+1}}\right)$ to achieve the optimal convergence rate. In what follows, we derive the convergence rate between $f_j^{*}$ and its estimate ${\widehat{f}}_{nj}={\mathbf{\Phi }}_j^T\widehat{\mathit{\beta }}.$ Consequently, we validate the model selection property of our method.

Theorem 2. 

Given the conditions in Theorem 1, we have

(1) 

${\parallel {\widehat{f}}_{nj}-f_j^{*}\parallel }_2=O_p(q_n{n}^{-\frac{2d-1}{2(2d+1)}}).$

(2) 

$P({\mathsf{\Omega }}^{*}\subset \widehat{\mathsf{\Omega }})\to 1$ as $n\to \infty .$

The above theorem demonstrates the convergence of the estimator ${\widehat{f}}_{nj}$ to $f_j^{*}$, which quantifies the importance of the $j^{th}$ component. According to Huang et al. [35], we set $m_n=O\left(n^{\frac{1}{2d+1}}\right)$ to achieve the optimal convergence rate. We also establish the sure screening property of the set $\widehat{\mathsf{\Omega }}$, encompassing the true model ${\mathsf{\Omega }}^{*}$ with a probability approaching 1.

3. The Conditional Expectation

In this section, we present the derivations of the conditional expectations for ${\tilde{Y}}_i$ based on the observed status of groups, a crucial component for the EM algorithm. These expectations are denoted by $w_i^{\left(t\right)}=\mathrm{E}\left[{\tilde{Y}}_i\right|{\mathit{Z}}_1,\cdots ,{\mathit{Z}}_J,{\mu }^{\left(t\right)},{\mathit{\beta }}^{\left(t\right)}]$. Given that ${\tilde{Y}}_i$ follows a Bernoulli distribution and is independent, we express $w_i^{\left(t\right)}$ as $w_i^{\left(t\right)}=P({\tilde{Y}}_i=1|{\mathit{Z}}_1,\dots ,{\mathit{Z}}_J,{\mu }^{\left(t\right)},{\mathit{\beta }}^{\left(t\right)})$. This conditional expectation is derived based on the group testing algorithm we implemented. Detailed derivations are provided for three group testing methods: the master pool, Dorfman’s algorithm, and Halving. However, the derivation for the array method is exceedingly complex and has been omitted in this context for brevity.

3.1. Master Pool

With the master pool method, specimens are divided into non-overlapping groups, with each individual exclusively assigned to one of these groups. Testing is performed on each group, without requiring additional tests. Then we have the following:

$${\omega }_i^{\left(t\right)}=\left\{\begin{array}{cc}P({\tilde{Y}}_i=1|Z_{j,1}=0)=(1-S_e)·p_{iB}^{\left(t\right)}/(1-P(Z_{j,1}=1))\hfill & \mathrm{if}{Z}_{j,1}=0,\hfill \\ P({\tilde{Y}}_i=1|Z_{j,1}=1)=S_e·p_{iB}^{\left(t\right)}/P(Z_{j,1}=1)\hfill & \mathrm{if}{Z}_{j,1}=1,\hfill \end{array}\right.$$

where $P(Z_{j,1}=1)=S_e+(1-S_e-S_p){\displaystyle \prod _{i\in {\mathcal{P}}_j}}(1-p_{iB}^{\left(t\right)})$.

3.2. Dorfman

In Dorfman’s algorithm, specimens are partitioned into non-overlapping groups, similar to the master pool method, and group tests are conducted. Subsequently, for those groups testing positive, individual tests are conducted. When $Z_{j,1}=0$, the conditional expectation mirrors that of master pool testing, specifically ${\omega }_i^{\left(t\right)}=P({\tilde{Y}}_i=1|Z_{j,1}=0)=(1-S_e)·p_{iB}^{\left(t\right)}/(1-P(Z_{j,1}=1)).$ When the $j^{th}$ group tests positive, $Z_{j,1}=1$, individual tests are performed subsequently. Let ${\mathit{Z}}_{j,s:t}=(Z_{j,s},\cdots ,Z_{j,t})$ and ${\mathit{Z}}_j$ is short for ${\mathit{Z}}_{1:T_j}$. Then we have the following:

$${\omega }_i^{\left(t\right)}=P({\tilde{Y}}_i=1|Z_{j,1}=1,{\mathit{Z}}_{j,2:T_j})=\frac{P({\tilde{Y}}_i=1,Z_{j,1}=1,{\mathit{Z}}_{j,2:T_j})}{P(Z_{j,1}=1,{\mathit{Z}}_{j,2:T_j})}.$$

The denominator is expressed as follows:

$$\begin{array}{cc}\hfill & P(Z_{j,1}=1,{\mathit{Z}}_{j,2:T_j})\hfill \\ \hfill & =\sum _{{\tilde{\mathit{Z}}}_j}P(Z_{j,1}=1,{\mathit{Z}}_{j,2:T_j}|{\tilde{\mathit{Z}}}_j)P\left({\tilde{\mathit{Z}}}_j\right)\hfill \\ \hfill & =\sum _{{\tilde{\mathit{Z}}}_j}{S}_e^{{\tilde{Z}}_{j,1}}{(1-S_p)}^{1-{\tilde{Z}}_{j,1}}P\left({\mathit{Z}}_{j,2:T_j}\right|{\tilde{\mathit{Z}}}_{j,2:T_j})P\left({\tilde{\mathit{Z}}}_j\right),\hfill \end{array}$$

where ${\tilde{\mathit{Z}}}_j=({\tilde{Z}}_{j,1},\cdots ,{\tilde{Z}}_{j,T_j})$ and ${\tilde{\mathit{Z}}}_{j,2:T_j}=({\tilde{Z}}_{j,2},\cdots ,{\tilde{Z}}_{j,T_j}).$ Without loss of generalization, we assume that ${\tilde{Y}}_i$ represents the true status of the first individual of the group, with similar calculations applied to other cases. Subsequently, the numerator is as follows:

$$\begin{array}{cc}\hfill & P({\tilde{Y}}_i=1,Z_{j,1}=1,{\mathit{Z}}_{j,2:T_j})\hfill \\ \hfill & =P(Z_{j,1}=1,Z_{j,2},{\mathit{Z}}_{j,3:T_j}|{\tilde{Y}}_i=1)P({\tilde{Y}}_i=1)\hfill \\ \hfill & =S_e{S}_e^{Z_{j,2}}{(1-S_e)}^{1-Z_{j,2}}P\left({\mathit{Z}}_{j,3:T_j}\right)p_{iB}^{\left(t\right)}\hfill \end{array}$$

3.3. Halving

With the Halving algorithm, the testing procedure is performed in three stages. The first stage is identical to the master pool method. In the second stage, groups that test positive are evenly split into two subgroups. In the third stage, these subgroups are tested, and positive ones undergo further individual tests. When $Z_{j,1}=0$, the conditional expectation aligns with master pool testing, with ${\omega }_i^{\left(t\right)}=P({\tilde{Y}}_i=1|Z_{j,1}=0)=(1-S_e)·p_{iB}^{\left(t\right)}/(1-P(Z_{j,1}=1)).$ Assuming the $j^{th}$ group tests positive with $Z_{j,1}=1$, sequential tests are further carried out. These tests can unfold in any of the following scenarios:

(1)

${\mathit{Z}}_j=(1,0,0)$ signifies that the group containing the $i^{th}$ specimen initially tested positive in the first stage, followed by a negative result in a subsequent subgroup test; ultimately, the $i^{th}$ specimen is concluded with a negative outcome. Then, we have the following:

$${\omega }_i^{\left(t\right)}=P({\tilde{Y}}_i=1|{\mathit{Z}}_{j,1:3}=(1,0,0))=\frac{P({\tilde{Y}}_i=1,{\mathit{Z}}_{j,1:3}=(1,0,0))}{P({\mathit{Z}}_{j,1:3}=(1,0,0))},$$

where the denominator is as follows:

$$\begin{array}{cc}\hfill & P({\mathit{Z}}_{j,1:3}=(1,0,0))\hfill \\ \hfill & =\sum _{{\tilde{\mathit{Z}}}_j}P({\mathit{Z}}_{j,1:3}=(1,0,0)|{\tilde{\mathit{Z}}}_{j,1:3})P\left({\tilde{\mathit{Z}}}_{j,1:3}\right)\hfill \\ \hfill & =\sum _{{\tilde{\mathit{Z}}}_j}{S}_e^{{\tilde{Z}}_{j,1}}{(1-S_p)}^{1-{\tilde{Z}}_{j,1}}{(1-S_e)}^{{\tilde{Z}}_{j,2}+{\tilde{Z}}_{j,3}}{S}_p^{2-{\tilde{Z}}_{j,2}-{\tilde{Z}}_{j,3}}P\left({\tilde{\mathit{Z}}}_{j,1:3}\right).\hfill \end{array}$$

For simplicity, we assume that ${\tilde{Y}}_i$ is the first individual specimen ${\mathcal{P}}_{j,2}$. The derivation is similar to other cases. Then, the numerator is as follows:

$$\begin{array}{cc}\hfill & P({\tilde{Y}}_i=1,{\mathit{Z}}_{j,1:3}=(1,0,0))\hfill \\ \hfill & =P({\mathit{Z}}_{j,1:3}=(1,0,0)|{\tilde{Y}}_i=1)P({\tilde{Y}}_i=1)\hfill \\ \hfill & =S_e(1-S_e)P(Z_{j,3}=0)p_{iB}^{\left(t\right)}.\hfill \end{array}$$

(2)

For ${\mathit{Z}}_j=(1,1,0)$, we have the following:

$${\omega }_i^{\left(t\right)}=P({\tilde{Y}}_i=1|{\mathit{Z}}_{j,1:3}=(1,1,0),{\mathit{Z}}_{j,4:T_j})=\frac{P({\tilde{Y}}_i=1,{\mathit{Z}}_{j,1:3}=(1,1,0),{\mathit{Z}}_{j,4:T_j})}{P({\mathit{Z}}_{j,1:3}=(1,1,0),{\mathit{Z}}_{j,4:T_j})},$$

where the denominator is as follows:

$$\begin{array}{cc}\hfill & P({\mathit{Z}}_{j,1:3}=(1,1,0),{\mathit{Z}}_{j,4:T_j})\hfill \\ \hfill & =\sum _{{\tilde{\mathit{Z}}}_j}P({\mathit{Z}}_{j,1:3}=(1,1,0),{\mathit{Z}}_{j,4:T_j}|{\tilde{\mathit{Z}}}_j)P\left({\tilde{\mathit{Z}}}_j\right)\hfill \\ \hfill & =\sum _{{\tilde{\mathit{Z}}}_j}{S}_e^{{\tilde{Z}}_{j,1}+{\tilde{Z}}_{j,2}}{(1-S_p)}^{2-{\tilde{Z}}_{j,1}-{\tilde{Z}}_{j,2}}{(1-S_e)}^{{\tilde{Z}}_{j,3}}{S}_p^{1-{\tilde{Z}}_{j,3}}P\left({\mathit{Z}}_{j,4:T_j}\right|{\tilde{\mathit{Z}}}_{j,4:T_j})P\left({\tilde{\mathit{Z}}}_j\right).\hfill \end{array}$$

For the numerator, we consider the two situations where the $i^{th}$ specimen is located. If ${\tilde{Y}}_i$ is the first individual in the first subgroup, correspondingly, ${\tilde{Z}}_{j,4}=1$, then the numerator is as follows:

$$\begin{array}{cc}\hfill & P({\tilde{Y}}_i=1,{\mathit{Z}}_{j,1:3}=(1,1,0),{\mathit{Z}}_{j,4:T_j})\hfill \\ \hfill & =P({\mathit{Z}}_{j,1:3}=(1,1,0),{\mathit{Z}}_{j,4:T_j}|{\tilde{Y}}_i=1)P({\tilde{Y}}_i=1)\hfill \\ \hfill & =S_e^{2+Z_{j,4}}{(1-S_e)}^{1-Z_{j,4}}P(Z_{j,3}=0)P\left({\mathit{Z}}_{j,5:T_j}\right)p_{iB}^{\left(t\right)}.\hfill \end{array}$$

Otherwise, if ${\tilde{Y}}_i$ corresponds to the first individual in the second subgroup, then the numerator is as follows:

$$\begin{array}{cc}\hfill & P({\tilde{Y}}_i=1,{\mathit{Z}}_{j,1:3}=(1,1,0),{\mathit{Z}}_{j,4:T_j})\hfill \\ \hfill & =P({\mathit{Z}}_{j,1:3}=(1,1,0),{\mathit{Z}}_{j,4:T_j}|{\tilde{Y}}_i=1)P({\tilde{Y}}_i=1)\hfill \\ \hfill & =S_e(1-S_e)P(Z_{j,2}=1)P\left({\mathit{Z}}_{j,4:T_j}\right)p_{iB}^{\left(t\right)}\hfill \end{array}$$

(3)

For ${\mathit{Z}}_j=(1,0,1)$, we have the following:

$${\omega }_i^{\left(t\right)}=P({\tilde{Y}}_i=1|{\mathit{Z}}_{j,1:3}=(1,0,1),{\mathit{Z}}_{j,4:T_j})=\frac{P({\tilde{Y}}_i=1,{\mathit{Z}}_{j,1:3}=(1,0,1),{\mathit{Z}}_{j,4:T_j})}{P({\mathit{Z}}_{j,1:3}=(1,0,1),{\mathit{Z}}_{j,4:T_j})},$$

where the denominator is as follows:

$$\begin{array}{cc}\hfill & P({\mathit{Z}}_{j,1:3}=(1,0,1),{\mathit{Z}}_{j,4:T_j})\hfill \\ \hfill & =\sum _{{\tilde{\mathit{Z}}}_j}P({\mathit{Z}}_{j,1:3}=(1,0,1),{\mathit{Z}}_{j,4:T_j}|{\tilde{\mathit{Z}}}_j)P\left({\tilde{\mathit{Z}}}_j\right)\hfill \\ \hfill & =\sum _{{\tilde{\mathit{Z}}}_j}{S}_e^{{\tilde{Z}}_{j,1}+{\tilde{Z}}_{j,3}}{(1-S_p)}^{2-{\tilde{Z}}_{j,1}-{\tilde{Z}}_{j,3}}{(1-S_e)}^{{\tilde{Z}}_{j,2}}{S}_p^{1-{\tilde{Z}}_{j,2}}P\left({\mathit{Z}}_{j,4:T_j}\right|{\tilde{\mathit{Z}}}_{j,4:T_j})P\left({\tilde{\mathit{Z}}}_j\right).\hfill \end{array}$$

Similarly, we derive the numerator according to the location of the $i^{th}$ specimen. If it is assigned in the first subgroup with ${\tilde{Z}}_{j,4}=1,$ then the numerator is as follows:

$$\begin{array}{cc}\hfill & P({\tilde{Y}}_i=1,{\mathit{Z}}_{j,1:3}=(1,0,1),{\mathit{Z}}_{j,4:T_j})\hfill \\ \hfill & =P({\mathit{Z}}_{j,1:3}=(1,0,1),{\mathit{Z}}_{j,4:T_j}|{\tilde{Y}}_i=1)P({\tilde{Y}}_i=1)\hfill \\ \hfill & =S_e(1-S_e)P({\mathit{Z}}_{P_{j,3}}=1)P\left({\mathit{Z}}_{j,4:T_j}\right)p_{iB}^{\left(t\right)}.\hfill \end{array}$$

If it is assigned in the second subgroup, then the numerator is as follows:

$$\begin{array}{cc}\hfill & P({\tilde{Y}}_i=1,{\mathit{Z}}_{j,1:3}=(1,1,0),{\mathit{Z}}_{j,4:T_j})\hfill \\ \hfill & =P({\mathit{Z}}_{j,1:3}=(1,1,0),{\mathit{Z}}_{j,4:T_j}|{\tilde{Y}}_i=1)P({\tilde{Y}}_i=1)\hfill \\ \hfill & =S_e^{2+Z_{j,4}}{(1-S_e)}^{1-Z_{j,4}}P(Z_{j,2}=0)P\left({\mathit{Z}}_{j,5:T_j}\right)p_{iB}^{\left(t\right)}.\hfill \end{array}$$

(4)

For ${\mathit{Z}}_j=(1,1,1)$, we have the following:

$${\omega }_i^{\left(t\right)}=P({\tilde{Y}}_i=1|{\mathit{Z}}_{j,1:3}=(1,1,1),{\mathit{Z}}_{j,4:T_j})=\frac{P({\tilde{Y}}_i=1,{\mathit{Z}}_{j,1:3}=(1,1,1),{\mathit{Z}}_{j,4:T_j})}{P({\mathit{Z}}_{j,1:3}=(1,1,1),{\mathit{Z}}_{j,4:T_j})},$$

where the denominator is as follows:

$$\begin{array}{cc}\hfill & P({\mathit{Z}}_{j,1:3}=(1,1,1),{\mathit{Z}}_{j,4:T_j})\hfill \\ \hfill & =\sum _{{\tilde{\mathit{Z}}}_j}P({\mathit{Z}}_{j,1:3}=(1,1,1),{\mathit{Z}}_{j,4:T_j}|{\tilde{\mathit{Z}}}_j)P\left({\tilde{\mathit{Z}}}_j\right)\hfill \\ \hfill & =\sum _{{\tilde{\mathit{Z}}}_j}\left[\prod _{k=1}^3{S}_e^{{\tilde{Z}}_{j,k}}{(1-S_p)}^{1-{\tilde{Z}}_{j,k}}\right]P\left({\mathit{Z}}_{j,4:T_j}\right|{\tilde{\mathit{Z}}}_{j,4:T_j})P\left({\tilde{\mathit{Z}}}_j\right).\hfill \end{array}$$

In this case, the location of the $i^{th}$ specimen in the subgroups will not affect the conditional expectation. Therefore, assuming it is the first individual in the first subgroup. Given ${\tilde{Y}}_i=1$, then we have ${\tilde{Z}}_{j,1}={\tilde{Z}}_{j,2}={\tilde{Z}}_{j,4}=1$, and, therefore, the numerator is derived as follows:

$$\begin{array}{cc}\hfill & P({\tilde{Y}}_i=1,{\mathit{Z}}_{j,1:3}=(1,1,1),{\mathit{Z}}_{j,4:T_j})\hfill \\ \hfill & =P({\mathit{Z}}_{j,1:3}=(1,1,1),{\mathit{Z}}_{j,4:T_j}|{\tilde{Y}}_i=1,)P({\tilde{Y}}_i=1)\hfill \\ \hfill & =S_e^{2+Z_{j,4}}{(1-S_e)}^{1-Z_{j,4}}P(Z_{j,2}=1)P\left({\mathit{Z}}_{j,5:T_j}\right)p_{iB}^{\left(t\right)}.\hfill \end{array}$$

4. Simulation

In this section, we conduct numerical studies to demonstrate the performance of Algorithm 1 and assess the effectiveness of modeling group testing data within the framework of the generalized additive model. We adapt the methodology outlined by Liu et al. [26] to generate group testing data with high-dimensional covariates. We employ the following additive models:

Example 1. 

($n=1000$ and 500, $q_n=50$, $s_0=4$). The model is $\mathrm{P}({\tilde{Y}}_i=1|{\mathit{X}}_i)=g(\mu +f_1\left(X_{i1}\right)+f_2\left(X_{i2}\right)+f_3\left(X_{i3}\right)+f_4\left(X_{i4}\right))$, where $\mu =-4$, $f_1\left(x\right)=2x$, $f_2\left(x\right)={(2x-\frac{1}{4})}^2-1.4$, $f_3\left(x\right)=3sin\left(x\right)+cos\left(x^2\right)-\frac{5}{6}$, $f_4\left(x\right)=3e^{(-{(x+1.5)}^2/0.8)}-\frac{1}{4}$ and $f_j=0,j=5,\cdots ,q_n.$ The covariates are generated from the uniform distribution $U(-1,1)$. In this case, the average prevalence rate is about $0.112$. Figure 2 presents the true components and their corresponding estimators in a simulation run.

Example 2. 

($n=500$, $q_n=50$ and 100, $s_0=4$, $\rho =0.2$). In contrast to Example 1, Example 2 incorporates a correlation structure within the data while utilizing the same functions for data generation. The covariates ${({\mathit{X}}_1,\cdots ,{\mathit{X}}_n)}^T$ are generated as

$${\mathit{X}}_j=\frac{{\mathit{W}}_j+t\mathit{U}}{1+t^2},j\in \left[q_n\right],$$

where ${\mathit{W}}_1,\cdots ,{\mathit{W}}_{q_n}$ and $\mathit{U}$ are i.i.d. samples from Uniform $(-1,1)$, and $t=\sqrt{\frac{\rho }{1-\rho }}$. In this case, the disease prevalence rate is about $0.064$.

Example 3. 

($n=1000$, $q_n=50$ and 100, $s_0=3$, $\rho =0.2$). The model is defined as $\mathrm{P}({\tilde{Y}}_i=1|{\mathit{X}}_i)=g(\mu +f_1\left(X_{i1}\right)+f_2\left(X_{i2}\right)+f_3\left(X_{i3}\right))$, with $\mu =-3.5$, $f_1\left(x\right)=2x$, $f_2\left(x\right)={(2x-\frac{1}{4})}^2-1.1$, $f_3\left(x\right)=-3sin\left(\frac{\pi }{4}x\right)+co{s}^2\left(\frac{\pi }{4}x\right)-0.9$ and $f_j=0,j=4,\cdots ,q_n.$ The covariates are simulated as in Example 2, and the average disease prevalence rate can be approximated as $0.063$.

Example 4. 

($n=500$, $q_n=50$ and 100, $s_0=3$, $\rho =0.2$). Following Example 3, we employ the same function for data generation and conduct simulation experiments with varied values of $q_n$ while reducing the sample size to 500. The individual average prevalence rate approximates around $0.10$.

We implement our method using four group testing algorithms: master pool testing (master pool), Dorfman, Halving, and array. For the master pool, individuals are grouped into non-overlapping pools with a size of 4, conducting tests without any subsequent individual tests. For Dorfman and Halving testing, the group size is fixed at 4. In the case of array testing, individuals are arranged into $4\times 4$ arrays. The sensitivity and specificity are set at $0.98$.

Let $\left|\right|{\widehat{f}}_{nj}{\left|\right|}_n^2=\frac{1}{n}{\sum }_{j=1}^n{\widehat{f}}_{nj}^2\left(X_{ij}\right)={\widehat{\mathit{\beta }}}_j^T\left(E_n{\mathbf{\Phi }}_j{\mathbf{\Phi }}_j^T\right){\widehat{\mathit{\beta }}}_j$ denote the effect of a component. The estimators $\widehat{\mathit{\beta }}$ are obtained using Algorithm 1. Since we utilize stochastic gradient descent to optimize the regularized objective function, it is possible that the coefficients of non-significant components may not precisely equal zero. As illustrated in Figure 2, a variable is deemed significant and selected if $\parallel {\widehat{f}}_{nj}{\parallel }_n$ deviates significantly from zero. In contrast, for other non-significant variables, such as $x_5$ and $x_6$ depicted in Figure 2, their functional estimates closely approximate zero.

To assess the performance of Algorithm 1 in variable selection and model estimation, we conduct experiments across the four examples, repeating the procedure 100 times for each example. The initial parameters for each experiment follow a standard normal distribution. We calculate the number of true positives (TPs) and false positives (FPs) to evaluate variable selection attributes. Additionally, we employ the mean prediction error (PE) to measure the overall prediction error of the model, defined as $\mathrm{PE}=E_X[{(\widehat{f}\left(X\right)-f\left(X\right))}^2]$.

Table 1. Overall performance of our method under four group testing algorithms.

Model	Setting	Procedure	PE	TP	FP
Example 1 ($q_n=50$, $s_0=4$)	$n=1000$	Master Pool	0.82 (0.12)	3.92 (0.27)	3.09 (0.29)
		Halving	0.83 (0.11)	3.96 (0.20)	2.98 (0.14)
		Dorfman	0.76 (0.01)	4.00 (0.00)	1.98 (0.15)
		Array	0.80 (0.01)	3.99 (0.10)	2.20 (0.45)
	$n=500$	Master Pool	1.57 (0.51)	3.83 (0.38)	3.22 (1.45)
		Halving	1.45 (0.55)	3.89 (0.31)	3.18 (1.42)
		Dorfman	0.95 (0.01)	3.90 (0.30)	2.94 (0.34)
		Array	0.71 (0.01)	4.00 (0.00)	1.01 (0.17)
Example 2 ($n=500$, $s_0=4$)	$q_n=50$	Master Pool	1.67 (0.32)	3.74 (0.44)	3.94 (1.78)
		Halving	1.32 (1.01)	3.71 (0.48)	3.75 (1.67)
		Dorfman	1.19 (0.01)	3.88 (0.36)	1.94 (0.23)
		Array	1.10 (0.01)	3.92 (0.31)	1.93 (0.26)
	$q_n=100$	Master Pool	1.72 (0.43)	3.65 (0.58)	3.21 (1.71)
		Halving	1.52 (0.28)	3.57 (0.59)	2.72 (1.54)
		Dorfman	1.45 (0.01)	3.64 (0.57)	2.02 (1.41)
		Array	1.29 (0.01)	3.85 (0.48)	2.48 (1.72)
Example 3 ($n=1000$, $s_0=3$)	$q_n=50$	Master Pool	1.61 (0.08)	2.89 (0.45)	3.97 (0.33)
		Halving	1.64 (0.24)	2.85 (0.48)	3.51 (0.88)
		Dorfman	0.60 (0.01)	2.91 (0.35)	1.73 (0.44)
		Array	0.54 (0.01)	3.00 (0.00)	1.87 (0.34)
	$q_n=100$	Master Pool	0.92 (0.01)	2.89 (0.40)	0.00 (0.00)
		Halving	1.23 (0.15)	2.74 (0.50)	1.91 (0.29)
		Dorfman	0.85 (0.01)	2.84 (0.44)	0.97 (0.17)
		Array	0.95 (0.08)	2.92 (0.37)	2.01 (0.18)
Example 4 ($n=500$, $s_0=3$)	$q_n=50$	Master Pool	1.47 (0.35)	2.78 (0.48)	3.13 (1.01)
		Halving	1.48 (0.21)	2.69 (0.53)	3.16 (1.36)
		Dorfman	0.81 (0.02)	2.93 (0.36)	0.44 (0.50)
		Array	0.86 (0.01)	2.91 (0.38)	0.46 (1.11)
	$q_n=100$	Master Pool	1.52 (0.16)	2.69 (0.53)	3.17 (1.54)
		Halving	1.46 (0.28)	2.73 (0.51)	3.55 (1.55)
		Dorfman	1.31 (0.02)	2.88 (0.40)	2.96 (1.97)
		Array	1.10 (0.01)	2.90 (0.39)	1.98 (0.83)

reports the average and corresponding standard deviations of TP, FP, and PE. The results demonstrate that Algorithm 1 effectively identifies significant variables and exhibits high robustness. In Table 1, we set a threshold of $\tau =0.05$ to obtain the estimate $\widehat{M}$, employing stochastic gradient descent to optimize the objective. Notably, only a few cases are observed where variables that are not significant are mistakenly identified.

To further validate the performance of Algorithm 1, we utilize two metrics, namely PE_fj to signify the prediction error of each component, and PE_non to represent the average prediction error of non-significant variables. In

Table 2. The estimates of the functions using Example 1 and Example 2.

Model	Setting	Procedure	PEf1	PEf2	PEf3	PEf4	PEnon
Example 1 ($q_n=50$)	$n=1000$	Master Pool	0.04 (0.03)	0.17 (0.10)	0.18 (0.09)	0.15 (0.05)	0.02 (0.01)
		Halving	0.05 (0.04)	0.16 (0.11)	0.17 (0.08)	0.14 (0.05)	0.01 (0.01)
		Dorfman	0.03 (0.01)	0.06 (0.02)	0.08 (0.01)	0.11 (0.01)	0.02 (0.01)
		Array	0.04 (0.01)	0.06 (0.03)	0.09 (0.02)	0.12 (0.02)	0.02 (0.01)
	$n=500$	Master Pool	0.05 (0.06)	0.40 (0.14)	0.41 (0.17)	0.12 (0.04)	0.01 (0.01)
		Halving	0.04 (0.03)	0.23 (0.09)	0.31 (0.08)	0.10 (0.04)	0.02 (0.01)
		Dorfman	0.03 (0.01)	0.17 (0.03)	0.17 (0.02)	0.07 (0.01)	0.01 (0.01)
		Array	0.01 (0.01)	0.16 (0.02)	0.24 (0.01)	0.05 (0.02)	0.01 (0.01)
Example 2 ($n=500$)	$q_n=50$	Master Pool	0.32 (0.06)	0.22 (0.06)	0.44 (0.17)	0.14 (0.03)	0.02 (0.01)
		Halving	0.14 (0.14)	0.28 (0.30)	0.27 (0.22)	0.14 (0.08)	0.01 (0.01)
		Dorfman	0.19 (0.01)	0.21 (0.04)	0.39 (0.01)	0.04 (0.01)	0.02 (0.01)
		Array	0.24 (0.01)	0.20 (0.04)	0.51 (0.01)	0.06 (0.01)	0.01 (0.01)
	$q_n=100$	Master Pool	0.31 (0.17)	0.14 (0.11)	0.20 (0.19)	0.09 (0.04)	0.01 (0.01)
		Halving	0.36 (0.20)	0.13 (0.12)	0.18 (0.08)	0.09 (0.03)	0.02 (0.01)
		Dorfman	0.05 (0.01)	0.12 (0.03)	0.08 (0.01)	0.06 (0.01)	0.01 (0.01)
		Array	0.05 (0.01)	0.15 (0.07)	0.09 (0.02)	0.07 (0.01)	0.01 (0.01)

and

Table 3. The specific results of Example 3 and Example 4.

Model	Setting	Procedure	PEf1	PEf2	PEf3	PEnon
Example 3 ($n=1000$)	$q_n=50$	Master Pool	0.09 (0.02)	0.19 (0.26)	0.07 (0.24)	0.04 (0.02)
		Halving	0.06 (0.03)	0.17 (0.26)	0.06 (0.25)	0.05 (0.01)
		Dorfman	0.01 (0.02)	0.10 (0.01)	0.02 (0.02)	0.02 (0.01)
		Array	0.01 (0.01)	0.07 (0.01)	0.01 (0.01)	0.01 (0.01)
	$q_n=100$	Master Pool	0.04 (0.02)	0.25 (0.09)	0.03 (0.01)	0.01 (0.01)
		Halving	0.02 (0.01)	0.13 (0.03)	0.06 (0.02)	0.01 (0.01)
		Dorfman	0.03 (0.01)	0.24 (0.02)	0.04 (0.01)	0.02 (0.01)
		Array	0.03 (0.01)	0.12 (0.01)	0.03 (0.01)	0.01 (0.01)
Example 4 ($n=500$)	$q_n=50$	Master Pool	0.23 (0.14)	0.41 (0.21)	0.06 (0.07)	0.01 (0.01)
		Halving	0.19 (0.07)	0.51 (0.17)	0.06 (0.02)	0.02 (0.01)
		Dorfman	0.19 (0.01)	0.13 (0.01)	0.02 (0.01)	0.01 (0.01)
		Array	0.14 (0.01)	0.12 (0.01)	0.01 (0.01)	0.02 (0.01)
	$q_n=100$	Master Pool	0.21 (0.02)	0.10 (0.01)	0.27 (0.05)	0.02 (0.01)
		Halving	0.22 (0.05)	0.08 (0.02)	0.32 (0.10)	0.01 (0.01)
		Dorfman	0.06 (0.01)	0.15 (0.01)	0.13 (0.02)	0.01 (0.01)
		Array	0.09 (0.01)	0.16 (0.01)	0.26 (0.01)	0.01 (0.01)

, we present the outcomes of four models using four group testing methods. Notably, the prediction error (PE_fj) associated with each significant variable is relatively small, which serves as an indication that the B-spline successfully captures the unknown nonlinear functions using Algorithm 1. These findings ascertain Algorithm 1’s outstanding capability in variable selection, enabling it to precisely identify important variables while minimizing the probability of mistakenly selecting non-significant variables.

Among the four different group testing methods, the Dorfman and array methods demonstrate superior performance, compared to the master pool and Halving methods. This characteristic verifies the fact that Dorfman employs sufficient individual repeated testing and array utilizes more group-level tests. In contrast, Halving adopts a sub-group retest, and subsequently, the individual test, while the master pool only uses the initial pools and does not involve any retesting.

5. Application

In our study, we perform regression analysis using the data set from the 1999–2004 National Health and Nutrition Examination Survey (NHANES). NHANES is a continuous cross-sectional survey conducted in the United States, which utilizes a probabilistic sampling method to select participants. The survey collects demographic information, health history, and behavioral data through interviews with participants and their families. The dataset contains 5515 observations ($n=5515$) and classifies individuals as diabetics or non-diabetic patients. Following Yu et al. [36], we select a subset of covariates for regression analysis and subsequently divide the dataset into a training set ($n=4412$) and a test set ($n=1103$). The subset contains 14 variables that are potentially associated with the risk of diabetes: education ($X_1$), household income ($X_2$), physical activity ($X_3$), hypertension ($X_4$), family history ($X_5$), sex ($X_6$), race and ethnicity ($X_7$), alcohol use ($X_8$), BMI ($X_9$), height ($X_{10}$), waist circumference ($X_{11}$), weight ($X_{12}$), age ($X_{13}$), and smoking ($X_{14}$). To exam the performance of our method for high-dimensional data, we add non-significant variables that follow the standard normal distribution, increasing the dimension to $q_n=500$.

To perform the group testing algorithm, we choose Dorfman’s algorithm with a group size of $k=2,5,8$, setting the sensitivity to $S_e=0.95$ and the specificity to $S_p=0.98$. Based on all available risk factors collected from each individual, we use the following model to describe an individual’s diabetes status:

$$\mathrm{P}({\tilde{Y}}_i=1|{\mathit{X}}_i)=g({\beta }_0+{\mathit{X}}_{i1}^T{\mathit{\beta }}_1+\cdots +{\mathit{X}}_{i7}^T{\mathit{\beta }}_7+\sum _{j=8}^{q_n}{f}_j\left(X_{ij}\right)),$$

(6)

where the categorical variables $X_1$ to $X_7$ are transformed using one-hot encoding, represented by ${\mathit{X}}_{ij}$ for $j=1,\cdots ,7$. Additionally, nonlinear functions are applied to variables $X_8$ through $X_{14}$ to capture their relationship with the diabetes status more accurately. The unknown function, $f_j(·)$, in (6) allows variables to have a non-linear influence on the state of diabetes. We assume that the link function $g(·)$ is a logistic function. For comparison, we also report the results using the individual data ($k=1$) and a generalized linear regression model (GLM) based on the original data. To demonstrate the performance of our algorithm, we report the following statistics: accuracy (ACC), positive predictive value ($PPV=\frac{TP}{TP+FP}$), negative predictive value ($NPV=\frac{TN}{TN+FN}$), and recall ($Recall=\frac{TP}{TP+FN}$), where TP, FP, TN, and FN represent the numbers of true positives, false positives, true negatives, and false negatives, respectively. The results are presented in

Table 4. The performance of the models; GLM is $q_n$ = 14 in a low dimension, while in our method, $q_n$ = 500.

Metrics	Our Method				GLM
	k = 1	k = 2	k = 5	k = 8
ACC	0.8296	0.8250	0.8268	0.8277	0.8150
PPV	0.5800	0.5703	0.5761	0.5766	0.5472
NPV	0.9027	0.8993	0.8977	0.9006	0.8952
Recall	0.6360	0.6228	0.6140	0.6272	0.6096

.

Our algorithm exhibits an accuracy of approximately $83\%$, while the GLM model demonstrates an accuracy of $81.50\%$ when correctly classifying individuals as diabetics or non-diabetics. Our algorithm also has better performance than directly using GLM on the other three measures. It is expected that individual tests ($k=1$) achieve the highest precision. However, our group testing algorithms with $k>1$ perform competitively while saving a considerable number of tests compared with individual tests. For continuous variables from $X_8$ to $X_{500}$, our algorithm selects the variables $X_{10}$, $X_{11}$, and $X_{13}$. The other variables have little or no impact on the response. The estimates for the functions $f_{10}$, $f_{11}$, and $f_{13}$ are shown in Figure 3, along with the estimates obtained using GLM. It is noteworthy that our method demonstrates a slightly higher accuracy compared to the GLM model, indicating its potential advantages in capturing more complex relationships and improving predictive capabilities.

6. Concluding Remarks

In this study, we propose a generalized nonparametric additive model to characterize high-dimensional group testing data. We mainly use the B-spline to approximate the unknown nonparametric function. Theoretical results demonstrate that our method selects the true model with high probability and provides consistent estimates for components. Our work extends the existing works from a generalized linear model to a more flexible generalized nonparametric model for high-dimensional data.

Consideration of the measurement error in a test kit is essential for practical applications. The unobservable nature of an individual’s true status, treated as a latent variable, prompts us to use the EM algorithm to maximize the regularized log-likelihood function for high-dimensional data. The E-step of the EM algorithm varies according to the structures of different group testing procedures and needs to be calculated separately to speed up the algorithm.

Another critical issue for group testing is case identification, whose efficiency could be improved with the help of covariates. In future work, we will study the improvement of our method for case identification, and implement an informative retest procedure to accurately identify the infections and further reduce costs.