A Dunnett-Type Test and Its Sample Size Calculation for Comparing K ROC Curves with a Control

Abstract

Diagnostic biomarkers are key components of diagnostics. In this paper, we consider diagnostic biomarkers taking continuous values that are associated with a dichotomous disease status, called malignant or benign. The performance of such a biomarker is evaluated by the area under the curve (AUC) of its receiver operating characteristic curve. We assume that, together with the disease status, one control and multiple experimental biomarkers are collected from each subject to test if any of the experimental biomarkers have a larger AUC than the control. In this case, each experimental biomarker will be compared with the control so that a multiple testing issue is involved in the comparisons. In this paper, we propose a simple non-parametric statistical testing procedure to compare $K(\ge$2) experimental biomarkers with a control, adjusting for the multiplicity and its sample size calculation method. Our sample size formula requires the specification of the AUC values (or the standardized effect size of each biomarker between the benign and malignant groups) together with the correlation coefficients between the biomarkers, the prevalence of the malignant group in the study population, the type I error rate, and the power. Through simulations, we show that the statistical test controls the overall type I error rate accurately and the proposed sample size closely maintains the specified statistical power.

1. Introduction

Different types of biomarkers are measured from the tumor, blood, or urine using molecular, biochemical, physiological, anatomical, or imaging methods. The observed biomarkers are used for various purposes during the diagnosis and treatment of diseases. For example, diagnostic biomarkers are used to diagnose diseases, predictive biomarkers are used to predict the response to specific treatments, and prognostic biomarkers are used to measure the aggressiveness of a disease for patients with no treatment or a non-targeted treatment. When the outcome is dichotomous and the biomarker is continuous, the performance of a biomarker is evaluated by the area under the curve (AUC) of its receiver operating characteristic (ROC) curve, which plots the sensitivity against one minus the specificity calculated for all the possible cutoff values of the biomarker. In this paper, we consider diagnostic biomarkers taking continuous values that are associated with a dichotomous disease status, called benignity or malignancy, but the proposed methods can be used for any type of biomarker with such properties.

Various parameters have been used to measure the performance of biomarkers and compare among different biomarkers. For instance, sensitivity at a fixed specificity level determined by the cutoff value of each biomarker may be compared between biomarkers [1,2]. Some investigators proposed testing methods to compare the whole or partial AUC of ROC curves of biomarkers [3,4,5,6,7].

Emir et al. [8] extended the method of Delong et al. [6] to comparison of ROC curves from longitudinally collected biomarkers, and Jung et al. [9] proposed an ROC-type method to compare the performance of biomarkers that are associated with a time-to-event endpoint such as progression-free survival.

Sample size calculation is an important step in the development stage of medical projects. Hwang and Su [10] proposed a sample size formula for the test statistic of Delong et al. [6]. Their variance formula requires the specification of so many parameter values. Furthermore, these parameters are not conceptually interpretable, so it is almost impossible to accurately specify these parameters in sample size calculations. This can prevent us from using their sample size formula in designing real biomarker projects. Jung [11] proposed a simple non-parametric test statistic that is asymptotically identical to that of Delong et al. [6] and its sample size formula. Compared to the formula of Hwang and Su [10], his formula requires specification of much smaller number of design parameters and the design parameters are intuitively interpretable. These sample size methods are limited to the comparison of two biomarkers.

In this paper, we consider the cases where a control biomarker (usually an existing one) and $K(\ge$2) experimental biomarkers are collected from each subject together with a dichotomous disease status. We want to discover the experimental biomarkers that are more closely associated with the disease outcome than the control biomarker. To this end, we compare the AUC of the ROC curve of each experimental biomarker with that of the control using the two-sample test statistic [11]. With K experimental biomarkers to be compared with the control, we will conduct K statistical tests, so the discovery of new biomarkers will involve a multiple testing issue. We propose a multiple testing procedure using a Dunnett-type approach [12,13]. We derive a sample size formula for the multiple testing procedure too. While the statistical test is completely non-parametric, its power depends on the distributions of the biomarkers. Furthermore, the biomarkers to be compared are correlated because they are collected from each patient. This results in additional problems in deriving the statistical testing method and its sample size formula.

Since the AUC of an ROC curve is a rank statistic, it is invariant to the monotone transformations of biomarker values. Hence, if we find a transformation of biomarker distributions to symmetric distributions, such as normal distributions, with a location-shift model and unit variances, then, for a sample size calculation, we need to specify only the location parameter between the benign and malignant groups for each biomarker (which corresponds to the AUC) and the correlation coefficient between the paired biomarkers after the transformation. A log-transformation followed by standardization often carries out this goal. Through simulations, we show that our multiple testing procedure accurately controls the overall type I error rate, called the family-wise error rate (FWER), and its sample size formula closely maintains a specified statistical power.

2. A Dunnett-Type Test to Compare K Biomarkers with a Control

Suppose that biomarker 0 denotes a control (or existing) biomarker and biomarkers $1,\dots ,K$ denote K experimental biomarkers that are collected from each subject. Let $X_k$ and $Y_k$ denote the random variables for observations of biomarker k(=0, 1, $\dots ,K)$ from malignant and benign cases, respectively. In a data set, we have copies of $X_k$, $(x_{k1},\dots ,x_{k,m})$, from m malignant cases, and copies of $Y_k$, $(y_{k1},\dots ,y_{k,n})$, from n benign cases. For the total sample size N(=$m+n)$, $m/N\to \gamma \in (0,1)$ denotes the prevalence of malignancy in the target population. As can be seen from our sample size formula and the results of numerical studies in Section 5, our proposed test will have a maximum power if benign and malignant cases are balanced, i.e., $\gamma =1/2$. Let $F_k\left(t\right)=P(X_k\le t)$ and $G_k\left(t\right)=P(Y_k\le t)$ denote the cumulative density function (CDF) of biomarker k for malignant cases and benign cases, respectively, and ${\widehat{F}}_k\left(t\right)=m^{-1}{\sum }_{i=1}^{m}I(x_{ki}\le t)$ and ${\widehat{G}}_k\left(t\right)=n^{-1}{\sum }_{j=1}^{n}I(y_{kj}\le t)$ be their empirical estimators.

Without loss of generality, we assume that the raw biomarker values are non-negative and a large biomarker value is associated with a higher chance of malignancy. As such, for a cutoff values t, ${\overline{F}}_k\left(t\right)=1-F_k\left(t\right)$ is called the sensitivity and $G_k\left(t\right)$ is called the specificity of biomarker k. For ${\overline{G}}_k\left(t\right)=1-G_k\left(t\right)$, the ROC curve of biomarker k is the plot of $({\overline{G}}_k\left(t\right),{\overline{F}}_k\left(t\right))$ with respect to all possible cutoff value t.

AUC of the ROC curve for biomarker k is ${\theta }_k=P(X_k\ge Y_k)={\int }_0^{\infty }{G}_k\left(t\right)d{F}_k\left(t\right)$, which is estimated by

$${\widehat{\theta }}_k={\int }_0^{\infty }{\widehat{G}}_k\left(t\right)d{\widehat{F}}_k\left(t\right)={\displaystyle \frac{1}{mn}}\sum _{i=1}^m\sum _{j=1}^{n}I(x_{ki}\ge y_{kj})$$

For experimental biomarker $k(=1,\dots ,K)$, we want to test the null hypothesis $H_k:{\theta }_k={\theta }_0$ against an alternative hypothesis ${\overline{H}}_k:{\theta }_k>{\theta }_0$ using ${\widehat{\theta }}_k-{\widehat{\theta }}_0$. It was shown that [11], for $k=0,1,\dots ,K$, we have

$${\widehat{\theta }}_k-{\theta }_k=m^{-1}\sum _{i=1}^m{ϵ}_{ki}+n^{-1}\sum _{j=1}^n{e}_{kj}+o_p\left(N^{-1/2}\right)$$

(1)

where ${ϵ}_{ki}={\int }_0^{\infty }{G}_k\left(t\right)d\{I(x_{ki}\le t)-F_k\left(t\right)\}=G_k\left(x_{ki}\right)-{\theta }_k$ and $e_{kj}={\int }_0^{\infty }\{I(y_{kj}\le t)-G_k\left(t\right)\}d{F}_k\left(t\right)={\overline{F}}_k\left(y_{kj}\right)-{\theta }_k$. Hence, for $k=1,\dots ,K$,

$$\sqrt{N}({\widehat{\theta }}_k-{\widehat{\theta }}_0)=\sqrt{N}\{m^{-1}\sum _{i=1}^m({ϵ}_{ki}-{ϵ}_{0i})+n^{-1}\sum _{j=1}^n(e_{kj}-e_{0j})\}+{\Delta }_k\sqrt{N}+o_p\left(1\right)$$

(2)

where ${\Delta }_k={\theta }_k-{\theta }_0$.

Note that $({ϵ}_{0i},{ϵ}_{1i},\dots ,{ϵ}_{Ki})$ and $(e_{0i},e_{1i},\dots ,e_{Ki})$ are $K+1$-dimensional independent random vectors with mean 0 because malignant and benign groups are independent. Furthermore, under $H_k$, we have ${\Delta }_k=0$, so that, by the central limit theorem (CLT), $\sqrt{N}({\widehat{\theta }}_k-{\widehat{\theta }}_0)/{\widehat{\sigma }}_k$ converges to $N(0,1)$, where

$${\widehat{\sigma }}_k^2=N\{m^{-2}\sum _{i=1}^m{({\widehat{ϵ}}_{ki}-{\widehat{ϵ}}_{0i})}^2+n^{-2}\sum _{j=1}^n{({\widehat{e}}_{kj}-{\widehat{e}}_{0j})}^2\}$$

and ${\widehat{ϵ}}_{ki}$ and ${\widehat{e}}_{kj}$ are obtained from ${ϵ}_{ki}$ and $e_{kj}$, respectively, by replacing $F_k$ and $G_k$ with their consistent estimators ${\widehat{F}}_k$ and ${\widehat{G}}_k$. That is,

$${\widehat{ϵ}}_{ki}={\widehat{G}}_k\left(x_{ki}\right)-{\widehat{\theta }}_k={\displaystyle \frac{1}{n}}\sum _{j=1}^{n}I(y_{kj}\le x_{ki})-{\widehat{\theta }}_k$$

and

$${\widehat{e}}_{kj}=\widehat{\overline{F}}\left(y_{kj}\right)-{\widehat{\theta }}_k={\displaystyle \frac{1}{m}}\sum _{i=1}^{m}I(x_{ki}\ge y_{kj})-{\widehat{\theta }}_k$$

The univariate (or unadjusted) test rejects $H_k$ if $Z_k=\sqrt{N}({\widehat{\theta }}_k-{\widehat{\theta }}_0)/{\widehat{\sigma }}_k>z_{1-\alpha }$, where $z_{1-\alpha }$ denotes the $100(1-\alpha )$ percentile of $N(0,1)$ distribution. Appendix A of Jung [11] shows that the variance estimator ${\widehat{\sigma }}_k^2$ is asymptotically identical to that of DeLong et al. [6]. In this sense, our univariate test statistic is asymptotically identical to that of DeLong et al. [6].

When there are multiple experimental biomarkers (i.e., $K\ge 2$), the biomarker discovery will have increased false positivity by the multiple univariate tests. For accurate control of false positivity, we need a multiple testing adjustment. We propose to control the FWER, i.e., the probability to accept an experimental biomarker when all K values of them are no better than the control. For the overall null hypothesis $H_0={\cap }_{k=1}^K{H}_k$ and a common critical value $c=c_{\alpha }$, controlling the FWER at $\alpha$ is obtained by solving

$$\alpha =P\left\{{\cup }_{k=1}^K(Z_k>c)|H_0\right\}$$

(3)

with respect to c. Note that the critical region for our testing has a square shape. Delong et al. [6] proposed a one-way ANOVA-type test with an ellipsoidal-shape critical value, but our Dunnett-type test will be more appropriate when comparing multiple experimental biomarkers with a common control.

In order to calculate the FWER provided in Equation (3), we need the joint distribution of $(Z_1,\dots ,Z_K)$ under $H_0$. Applying the multivariate CLT to Equation (2), under $H_0$, $(Z_1,\dots ,Z_K)$ is asymptotically normal with mean 0 and variance $\Sigma$ that can be consistently estimated by

$$\widehat{\Sigma }={\left({\widehat{\rho }}_{k{k}^{\prime }}\right)}_{k,k^{\prime }=1,\dots ,K}$$

where ${\widehat{\rho }}_{k{k}^{\prime }}={\widehat{\sigma }}_{k{k}^{\prime }}/\sqrt{{\widehat{\sigma }}_k^2{\widehat{\sigma }}_{k^{\prime }}^2}$ and

$${\widehat{\sigma }}_{k{k}^{\prime }}=N\{m^{-2}\sum _{i=1}^m({\widehat{ϵ}}_{ki}-{\widehat{ϵ}}_{0i})({\widehat{ϵ}}_{k^{\prime }i}-{\widehat{ϵ}}_{0i})+n^{-2}\sum _{j=1}^n({\widehat{e}}_{kj}-{\widehat{e}}_{0j})({\widehat{e}}_{k^{\prime }j}-{\widehat{e}}_{0j})\}$$

For a given critical value c, the FWER is obtained by calculating the probability on the right-hand side of Equation (3) based on the multivariate normal distribution using simulations or a numerical integration method. Finally, the critical value $c=c_{\alpha }$ controlling the FWER at $\alpha$ is obtained by solving Equation (3) with respect to c and accepting (or discovering) biomarker $k(=1,\dots ,K)$ if $Z_k>c_{\alpha }$.

When $K=2$, the double integration for the probability calculation can be simplified to a single integration. If $(Z_1,Z_2)$ is a bivariate normal random vector with mean 0, variance 1, and correlation coefficient ${\rho }_{12}$, then the conditional distribution of $Z_1$ given $Z_2=z$ is normal with mean ${\rho }_{12}z$ and variance $(1-{\rho }_{12}^2)$. Hence, from Equation (3), we have

$$\mathrm{FWER}=P\left\{(Z_1>c)\cup (Z_2>c)\right\}=1-P\left\{(Z_1\le c)\cap (Z_2\le c)\right\}$$

so the critical value $c=c_{\alpha }$ controlling the FWER at $\alpha$ is obtained by solving the equation

$$\alpha =1-{\int }_{-\infty }^c\varphi \left(z\right)\Phi \left({\displaystyle \frac{c-{\widehat{\rho }}_{12}z}{\sqrt{1-{\widehat{\rho }}_{12}^2}}}\right)dz$$

(4)

with respect to c, where $\varphi (·)$ and $\Phi (·)$ are the probability density function (PDF) and CDF of $N(0,1)$ distribution, respectively.

For biomarker $k(=1,\dots ,K)$ with test statistic value $Z_k=z_k$, we may calculate the FWER-adjusted p-value by

$$p_k=P(Z_1>z_k,\mathrm{or}\dots ,\mathrm{or}{Z}_K>z_k|H_0)=P\left\{{\cup }_{k^{\prime }=1}^K(Z_{k^{\prime }}>z_k)|H_0\right\}$$

3. Sample Size Calculation for the Dunnett-Type Test

Now, we derive a sample size formula for a specified alternative hypothesis $H_a={\cup }_{k=1}^K{\overline{H}}_k$. For ${\Delta }_k>0$, we consider a specific alternative hypothesis

$$H_a:{\theta }_k-{\theta }_0={\Delta }_k \mathrm{for} k=1,\dots ,K$$

From Equation (2), we have $E({\widehat{\theta }}_k-{\widehat{\theta }}_0)={\Delta }_k$ for $k=1,\dots ,K$ under $H_a$.

We have to calculate the variances and covariances of $\sqrt{N}({\widehat{\theta }}_0,{\widehat{\theta }}_1,\dots ,{\widehat{\theta }}_K)$. Define ${\sigma }_k^2\left(ϵ\right)=\mathrm{var}\left({ϵ}_{ki}\right)$ and ${\sigma }_k^2\left(e\right)=\mathrm{var}\left(e_{ki}\right)$, i.e.,

$${\sigma }_k^2\left(ϵ\right)=\mathrm{var}\left\{G_k\left(X_k\right)\right\}={\int }_0^{\infty }{G}_k{\left(x\right)}^{2}d{F}_k\left(x\right)-{\theta }_k^2$$

and

$${\sigma }_k^2\left(e\right)=\mathrm{var}\left\{{\overline{F}}_k\left(Y_k\right)\right\}={\int }_0^{\infty }{\overline{F}}_k{\left(y\right)}^{2}d{G}_k\left(y\right)-{\theta }_k^2$$

For $0\le k\ne k^{\prime }\le K$, let $f_{k{k}^{\prime }}(x_k,x_{k^{\prime }})$ and $g_{k{k}^{\prime }}(y_k,y_{k^{\prime }})$ denote the PDFs of $(X_k,X_{k^{\prime }})$ and $(Y_k,Y_{k^{\prime }})$, respectively. We also define ${\sigma }_{k{k}^{\prime }}\left(ϵ\right)=\mathrm{cov}({ϵ}_{ki},{ϵ}_{k^{\prime }i})$ and ${\sigma }_{k{k}^{\prime }}\left(e\right)=\mathrm{cov}(e_{ki},e_{k^{\prime }i})$, i.e.,

$${\sigma }_{k{k}^{\prime }}\left(ϵ\right)=\mathrm{cov}\{G_k\left(X_k\right),G_{k^{\prime }}\left(X_{k^{\prime }}\right)\}={\int }_0^{\infty }{\int }_0^{\infty }{G}_k\left(x_k\right)G_{k^{\prime }}\left(x_{k^{\prime }}\right)f_{k{k}^{\prime }}(x_k,x_{k^{\prime }})d{x}_{k}d{x}_{k^{\prime }}-{\theta }_k{\theta }_{k^{\prime }}$$

and

$${\sigma }_{k{k}^{\prime }}\left(e\right)=\mathrm{cov}\{{\overline{F}}_k\left(Y_k\right),{\overline{F}}_{k^{\prime }}\left(Y_{k^{\prime }}\right)\}={\int }_0^{\infty }{\int }_0^{\infty }{\overline{F}}_k\left(y_k\right){\overline{F}}_{k^{\prime }}\left(y_{k^{\prime }}\right)g_{k{k}^{\prime }}(y_1,y_2)d{y}_{k}d{y}_{k^{\prime }}-{\theta }_k{\theta }_{k^{\prime }}$$

Let $s_k^2=\mathrm{var}\left({\widehat{\theta }}_k\sqrt{N}\right)$ and $s_{k{k}^{\prime }}=\mathrm{cov}({\widehat{\theta }}_k\sqrt{N},{\widehat{\theta }}_{k^{\prime }}\sqrt{N})$ for $0\le k,k^{\prime }\le K$. Then, from Equation (1), we have

$$s_k^2={\gamma }^{-1}{\sigma }_k^2\left(ϵ\right)+{\overline{\gamma }}^{-1}{\sigma }_k^2\left(e\right)$$

and

$$s_{k{k}^{\prime }}={\gamma }^{-1}{\sigma }_{k{k}^{\prime }}\left(ϵ\right)+{\overline{\gamma }}^{-1}{\sigma }_{k{k}^{\prime }}\left(e\right)$$

where $\overline{\gamma }=1-\gamma$.

Hence, we have $V=\mathrm{var}\{\sqrt{N}({\widehat{\theta }}_1-{\widehat{\theta }}_0),\dots ,\sqrt{N}({\widehat{\theta }}_K-{\widehat{\theta }}_0)\}={\left({\sigma }_{k{k}^{\prime }}\right)}_{k,k^{\prime }=1,\dots ,K}$ with

$${\sigma }_{kk}={\sigma }_k^2=\mathrm{var}\left\{\sqrt{N}({\widehat{\theta }}_k-{\widehat{\theta }}_0)\right\}=s_k^2-2s_{k0}+s_0^2$$

and, for $k\ne k^{\prime }(=1,\dots ,K)$,

$${\sigma }_{k{k}^{\prime }}=\mathrm{cov}\{\sqrt{N}({\widehat{\theta }}_k-{\widehat{\theta }}_0),\sqrt{N}({\widehat{\theta }}_{k^{\prime }}-{\widehat{\theta }}_0)\}=s_{k{k}^{\prime }}-s_{k0}-s_{k^{\prime }0}+s_0^2$$

By the multivariate CLT applied to (2) under $H_a$, $\sqrt{N}({\widehat{\theta }}_1-{\widehat{\theta }}_0,\dots ,{\widehat{\theta }}_K-{\widehat{\theta }}_0)$ is asymptotically normal with mean $\sqrt{N}({\Delta }_1,\dots ,{\Delta }_K)$ and variance V. So, under $H_a$, we have

$$Z_k=U_k+{\displaystyle \frac{{\Delta }_k\sqrt{N}}{{\sigma }_k}}$$

where

$$U_k={\displaystyle \frac{\sqrt{N}({\widehat{\theta }}_k-{\widehat{\theta }}_0-{\Delta }_k)}{{\sigma }_k}}$$

Note that $(U_1,\dots ,U_K)$ is normal with mean 0 and variance–covariance of $\Sigma ={\left({\rho }_{k{k}^{\prime }}\right)}_{k,k^{\prime }=1,\dots ,K}$ with ${\rho }_{kk}=1$ and ${\rho }_{k{k}^{\prime }}={\sigma }_{k{k}^{\prime }}/\left({\sigma }_k{\sigma }_{k^{\prime }}\right)$ for $1\le k\ne k^{\prime }\le K$.

Hence, for given N, the power $1-\beta$ of the Dunnett-type test controlling the FWER at $\alpha$ is provided as

$$1-\beta =P\left\{{\cup }_{k=1}^K(Z_k>c_{\alpha })\right|H_a\}$$

$$=P\left\{{\cup }_{k=1}^K\left(U_k>c_{\alpha }-{\displaystyle \frac{{\Delta }_k\sqrt{N}}{{\sigma }_k}}\right)\right\}$$

(5)

where $c=c_{\alpha }$ is obtained by solving

$$\alpha =P\left\{{\cup }_{k=1}^K(U_k>c)\right\}$$

(6)

For given power $1-\beta$, the required sample size N is obtained by solving Equation (5) with respect to N. The probabilities on the right-hand sides of Equations (5) and (6) can be solved using simulations or a multivariate numerical integration.

When $K=2$, the conditional distribution of $U_1$ given $U_2=u$ is normal with mean ${\rho }_{12}u$ and variance $(1-{\rho }_{12}^2)$, so Equations (5) and (6) are simplified to

$$1-\beta =1-{\int }_{-\infty }^{{\overline{c}}_1}\varphi \left(u\right)\Phi \left({\displaystyle \frac{{\overline{c}}_2-{\rho }_{12}u}{\sqrt{1-{\rho }_{12}^2}}}\right)du$$

(7)

and

$$\alpha =1-{\int }_{-\infty }^{c_{\alpha }}\varphi \left(u\right)\Phi \left({\displaystyle \frac{c_{\alpha }-{\rho }_{12}u}{\sqrt{1-{\rho }_{12}^2}}}\right)du$$

(8)

where ${\overline{c}}_k=c_{\alpha }-{\sigma }_k^{-1}{\Delta }_k\sqrt{N}$ for $k=1,2$.

4. Transformation of Biomarker Data to Normal Distributions

The Dunnett-type test statistic is completely non-parametric, but the asymptotic distribution and power of the test statistic depend on the distributions of biomarkers for benign and malignant groups through the values of ${\theta }_k$ under $H_a$ and the variance–covariance of $({\widehat{\theta }}_1-{\widehat{\theta }}_0,\dots ,{\widehat{\theta }}_K-{\widehat{\theta }}_0)$, as shown in the previous section. So, we have to assume a parametric model for sample size calculation.

Biomarkers usually take positive values. If we know the distributions of original biomarker values, then we may calculate a sample size based on them. However, we rarely have good information about the original distributions. Furthermore, for most multivariate positive distributions, their means, variances, and correlation coefficients are mutually associated, so it is difficult to specify the parameter values for a sample size calculation.

Since AUC is a rank statistic, it is invariant to monotone transformations of $X_k$ and $Y_k$. We can find a transformation, such as logarithm, to change the distribution of a positive random variable to a (approximately) normal distribution, for which the parameters are not associated with each other and can be easily specified. Such a transformation tends to stabilize the variance of random variables too.

We assume that, after a log-transformation, each biomarker marginally has normal distributions with a common variance (less strictly, a location-shift model) between malignant and benign groups. Then, we standardize the data by subtracting the mean of benign group and dividing by the common standard deviation. Then, for the resulting data of biomarker k, $X_k-{\delta }_k$ and $Y_k$ are independent $N(0,1)$ with a location parameter ${\delta }_k(>0)$. Note that ${\delta }_k$ is the standardized effect size of log-transformed biomarker values between malignant and benign groups. It is known that ${\theta }_k$ and ${\delta }_k$ have a 1-to-1 relationship, i.e., ${\theta }_k={\int }_{-\infty }^{\infty }\varphi (z-{\delta }_k)\Phi \left(z\right)dz$ [11].

With such a transformation, it is easy to conceptualize the design parameters for sample size calculations. Let ${\varphi }_r(x,y)$ denote the PDF of bivariate normal distribution with 0 means, unit variances, and correlation coefficient r. Then, we have $F_k(t+{\delta }_k)=G_k\left(t\right)=\Phi \left(t\right)$, $f_{k{k}^{\prime }}(x_k,x_{k^{\prime }})={\varphi }_{r_{k{k}^{\prime }}}(x_k-{\delta }_k,x_{k^{\prime }}-{\delta }_{k^{\prime }})$, and $g_{k{k}^{\prime }}(y_k,y_{k^{\prime }})={\varphi }_{r_{k{k}^{\prime }}}(y_k,y_{k^{\prime }})$, where $r_{k{k}^{\prime }}=\mathrm{corr}(X_k,X_{k^{\prime }})=\mathrm{corr}(Y_k,Y_{k^{\prime }})$. Furthermore, after such a transformation by Appendix B of Jung [11], we have ${\sigma }_k^2\left(ϵ\right)={\sigma }_k^2\left(e\right)$ and ${\sigma }_{k{k}^{\prime }}\left(ϵ\right)={\sigma }_{k{k}^{\prime }}\left(e\right)$, so we have $s_k^2={\left(\gamma \overline{\gamma }\right)}^{-1}{\sigma }_k^2\left(ϵ\right)$ and $s_{k{k}^{\prime }}={\left(\gamma \overline{\gamma }\right)}^{-1}{\sigma }_{k{k}^{\prime }}\left(ϵ\right)$, and ${\sigma }_k^2$ and ${\sigma }_{k{k}^{\prime }}$ are simplified to

$${\sigma }_k^2={\left(\gamma \overline{\gamma }\right)}^{-1}\{{\sigma }_k^2\left(ϵ\right)-2{\sigma }_{k0}\left(ϵ\right)+{\sigma }_0^2\left(ϵ\right)\}$$

(9)

and

$${\sigma }_{k{k}^{\prime }}={\left(\gamma \overline{\gamma }\right)}^{-1}\{{\sigma }_{k{k}^{\prime }}\left(ϵ\right)-{\sigma }_{k0}\left(ϵ\right)-{\sigma }_{k^{\prime }0}\left(ϵ\right)+{\sigma }_0^2\left(ϵ\right)\}$$

(10)

where

$${\sigma }_k^2\left(ϵ\right)={\int }_{-\infty }^{\infty }{\Phi }^2\left(x\right)\varphi (x-{\delta }_k)dx-{\theta }_k^2$$

and

$${\sigma }_{k{k}^{\prime }}\left(ϵ\right)={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }\Phi \left(x_k\right)\Phi \left(x_{k^{\prime }}\right){\varphi }_{{\rho }_{k{k}^{\prime }}}(x_k-{\delta }_k,x_{k^{\prime }}-{\delta }_{k^{\prime }})d{x}_{k}d{x}_{k^{\prime }}-{\theta }_k{\theta }_{k^{\prime }}$$

Note that the ranges of integrations are changed from $(0,\infty )$ for the original data to $(-\infty ,\infty )$ for the transformed data.

From Equations (9) and (10), we find that the sample size depends on the prevalence of malignancy $\gamma$ through $\gamma \overline{\gamma }$, so the required sample size is minimized when $\gamma =\overline{\gamma }=1/2$ and is unchanged between two $\gamma$ values with identical $|\gamma -1/2|$ value. Furthermore, since ${\rho }_{k{k}^{\prime }}={\sigma }_{k{k}^{\prime }}/\left({\sigma }_k{\sigma }_{k^{\prime }}\right)$, the correlation matrix of $\sqrt{N}({\widehat{\theta }}_1-{\widehat{\theta }}_0,\dots ,{\widehat{\theta }}_K-{\widehat{\theta }}_0)$, $\Sigma ={\left({\rho }_{k,k^{\prime }}\right)}_{k,k^{\prime }=1,\dots ,K}$, is free of $\gamma$.

Assuming that, for biomarker k, $X_k$ and $Y_k$ marginally follow normal distributions after a log-transformation, a sample size calculation is conducted as follows.

• Specify

  (a)

  $\mathrm{FWER}=\alpha$ and $\mathrm{power}=1-\beta$;

  (b)

  Prevalence of malignancy $=\gamma$;

  (c)

  For biomarker $k(=0,1,\dots ,K)$, standardized effect size ${\delta }_k$ between benign and malignant groups or AUC ${\theta }_k={\int }_{-\infty }^{\infty }\varphi (z-{\delta }_k)\Phi \left(z\right)dz$ under $H_a$; and

  (d)

  $r_{k{k}^{\prime }}=\mathrm{corr}(X_k,X_{k^{\prime }})=\mathrm{corr}(Y_k,Y_{k^{\prime }})$.

• Calculate ${\rho }_{k{k}^{\prime }}=\mathrm{corr}({\widehat{\theta }}_k-{\widehat{\theta }}_0,{\widehat{\theta }}_{k^{\prime }}-{\widehat{\theta }}_0)$ using Equations (9) and (10) based on specified ${\delta }_k$ (or ${\theta }_k$) and $r_{k{k}^{\prime }}$.
• Calculate $c_{\alpha }$ by solving Equation (6).
• Calculate the required sample size N by solving Equation (5).

5. Numerical Studies

Suppose that there are two experimental biomarkers and a control biomarker following the trivariate normal distribution model with a location-shift as discussed in the previous section. We consider prevalence of malignancy $\gamma =0.1$, 0.3, or 0.5, ${\theta }_0=0.6$, 0.7, or 0.8, AUCs for biomarkers ${\theta }_1={\theta }_2={\theta }_0$ under $H_0$, and ${\theta }_1={\theta }_2={\theta }_0+\Delta$ with $\Delta =0.1$ or 0.15 under $H_a$. Using the one-to-one relationship between AUC ${\theta }_k$ and location-shift parameter ${\delta }_k$ between $X_k$ and $Y_k$, we identify the location parameters $({\delta }_0,{\delta }_1,{\delta }_2)$ corresponding to the specified $({\theta }_0,{\theta }_1,{\theta }_2)$ values. We assume that correlation coefficients between any pair of biomarkers are identically $r=0.2$ or 0.4. Under each design setting of $({\theta }_0,\Delta ,r,\gamma )$, we calculate the critical value $c_{\alpha }$ for 1-sided FWER $\alpha =0.05$ from Equation (8) and calculate sample size N for power $1-\beta =0.8$, 0.85, or 0.9.

We conduct simulations to check if the calculated sample size is accurately powered or not. At first, we generate $B=5000$ samples of size N under the design setting of the sample size calculation. That is, for $m=\gamma N$ and $n=(1-\gamma )N$, $\{(x_{0i},x_{1i},x_{2i}),i=1,\dots ,m\}$ and $\{(y_{0j},y_{1j},y_{2j}),j=1,\dots ,n\}$ are independent trivariate normal random vectors with mean vectors $({\delta }_0,{\delta }_1,{\delta }_1)$ and $(0,0,0)$, respectively, but with variances 1 and compound symmetry structure with correlation coefficient r. Here, ${\delta }_0$ and ${\delta }_1$ are location parameters corresponding to AUCs ${\theta }_0$ and ${\theta }_1={\theta }_2={\theta }_0+\Delta$, respectively. The empirical power $1-\widehat{\beta }$ is estimated by the proportion of samples rejecting $H_0$ among the $B=5000$ simulation samples. Also, to check if the proposed Dunnett-type test accurately controls the FWER, we generate $B=5000$ samples of size N under $H_0$ (i.e., with $\Delta =0$), and the empirical FWER $\widehat{\alpha }$ is estimated by the proportion of samples rejecting $H_0$.

(

Table 1. Required sample size N (and empirical FWER level $\widehat{\alpha }$ and power $1-\widehat{\beta }$ from simulations) for 1-sided $\alpha =0.05$ test of $H_0:{\theta }_1={\theta }_2={\theta }_0$ vs. $H_a:({\theta }_1>{\theta }_0)\cup ({\theta }_2>{\theta }_0)$ given $r=\mathrm{corr}(X_k,X_{k^{\prime }})=\mathrm{corr}(Y_k,Y_{k^{\prime }})$ for $k\ne k^{\prime }$, AUCs $({\theta }_0,{\theta }_1,{\theta }_2)$ under $H_a$, and prevalence of malignancy $\gamma$.

$1-\mathit{\beta }$	r	${\mathit{\theta }}_0$	${\mathit{\theta }}_1={\mathit{\theta }}_2$	$\mathit{\gamma }=0.1$	$\mathit{\gamma }=0.3$	$\mathit{\gamma }=0.5$
0.8	0.2	$0.6$	$0.7$	$703(0.057,0.806)$	$302(0.049,0.801)$	$253(0.052,0.804)$
			$0.75$	$299(0.063,0.813)$	$128(0.061,0.814)$	$108(0.056,0.812)$
		$0.7$	$0.8$	$565(0.059,0.823)$	$242(0.053,0.814)$	$204(0.048,0.814)$
			$0.85$	$234(0.065,0.841)$	$100(0.054,0.820)$	$84(0.058,0.829)$
		$0.8$	$0.9$	$370(0.063,0.847)$	$159(0.057,0.837)$	$134(0.053,0.848)$
			$0.95$	$153(0.070,0.917)$	$66(0.058,0.883)$	$55(0.051,0.879)$
	0.4	$0.6$	$0.7$	$540(0.052,0.813)$	$232(0.052,0.815)$	$195(0.051,0.797)$
			$0.75$	$231(0.058,0.823)$	$99(0.055,0.802)$	$84(0.049,0.807)$
		$0.7$	$0.8$	$440(0.055,0.830)$	$189(0.046,0.804)$	$159(0.055,0.816)$
			$0.85$	$186(0.065,0.846)$	$80(0.058,0.834)$	$67(0.061,0.821)$
		$0.8$	$0.9$	$300(0.060,0.863)$	$129(0.053,0.832)$	$108(0.052,0.845)$
			$0.95$	$135(0.070,0.929)$	$58(0.064,0.900)$	$49(0.049,0.898)$
0.85	0.2	$0.6$	$0.7$	$816(0.050,0.864)$	$350(0.059,0.856)$	$294(0.053,0.849)$
			$0.75$	$347(0.066,0.863)$	$149(0.051,0.858)$	$125(0.055,0.846)$
		$0.7$	$0.8$	$655(0.059,0.871)$	$281(0.053,0.866)$	$236(0.046,0.851)$
			$0.85$	$271(0.059,0.890)$	$117(0.057,0.862)$	$98(0.060,0.871)$
		$0.8$	$0.9$	$430(0.062,0.893)$	$184(0.060,0.881)$	$155(0.050,0.877)$
			$0.95$	$178(0.069,0.938)$	$77(0.055,0.933)$	$64(0.054,0.922)$
	0.4	$0.6$	$0.7$	$626(0.047,0.864)$	$269(0.056,0.858)$	$226(0.051,0.856)$
			$0.75$	$269(0.063,0.864)$	$115(0.055,0.863)$	$97(0.049,0.855)$
		$0.7$	$0.8$	$510(0.056,0.877)$	$219(0.056,0.854)$	$184(0.049,0.864)$
			$0.85$	$216(0.066,0.887)$	$93(0.057,0.877)$	$78(0.056,0.877)$
		$0.8$	$0.9$	$349(0.059,0.895)$	$150(0.054,0.886)$	$126(0.056,0.874)$
			$0.95$	$157(0.064,0.946)$	$67(0.051,0.939)$	$57(0.056,0.929)$
0.9	0.2	$0.6$	$0.7$	$969(0.055,0.912)$	$416(0.050,0.901)$	$349(0.049,0.903)$
			$0.75$	$412(0.058,0.905)$	$177(0.056,0.903)$	$149(0.053,0.909)$
		$0.7$	$0.8$	$779(0.055,0.906)$	$334(0.050,0.906)$	$281(0.049,0.905)$
			$0.85$	$322(0.060,0.922)$	$138(0.058,0.913)$	$116(0.058,0.909)$
		$0.8$	$0.9$	$511(0.052,0.932)$	$219(0.053,0.920)$	$184(0.048,0.919)$
			$0.95$	$212(0.065,0.971)$	$91(0.060,0.957)$	$76(0.051,0.959)$
	0.4	$0.6$	$0.7$	$744(0.056,0.904)$	$319(0.054,0.904)$	$268(0.054,0.895)$
			$0.75$	$319(0.061,0.905)$	$137(0.057,0.903)$	$115(0.055,0.908)$
		$0.7$	$0.8$	$607(0.057,0.911)$	$260(0.052,0.911)$	$219(0.048,0.913)$
			$0.85$	$257(0.054,0.924)$	$110(0.054,0.922)$	$93(0.057,0.918)$
		$0.8$	$0.9$	$415(0.062,0.937)$	$178(0.053,0.917)$	$150(0.049,0.927)$
			$0.95$	$186(0.066,0.974)$	$80(0.056,0.968)$	$67(0.048,0.965)$

may be placed around here.)

The results of sample size calculations and simulations are summarized in Table 1. As expected, sample size increases in $1-\beta$ and decreases in $|\gamma -1/2|$, $\rho$, ${\theta }_0(>1/2)$, and $\Delta ={\theta }_k-{\theta }_0$ for $k=1,2$. From the simulation results, we find that our test statistic controls the FWER accurately and the calculated sample sizes are quite accurate in the sense that the empirical powers are close to the corresponding nominal power $1-\beta$ overall. For some design settings, however, the proposed test is slightly anti-conservative and the calculated sample size is slightly overpowered, especially if the calculated sample size is small and the allocation between cases and controls is very unbalanced (i.e., $\gamma =0.1$). The calculated sample sizes are never underpowered, at least for the design settings considered in this numerical study. Similar results were observed from numerical studies on two-sample version of this test [11]. In designing a study, slightly overpowered sample sizes are less of an issue than underpowered ones. If we want a more accurately powered sample size, we may calculate one using simulations. In this case, the sample size calculated from our formula can be used for validation of the sample size estimated from simulations.

Example: Suppose that there exists a biomarker A and we want to investigate whether we can improve its performance by adding a new biomarker B or C. So, biomarker 0 is A, 1 is A + B, and 2 is $A+C$. After a log-transformation and standardization with the mean of benign group and the common standard deviation as in Section 4, we assume that $(X_0,X_1,X_2)$ and $(Y_0,Y_1,Y_2)$ are independent trivariate normal random vectors with unit variances commonly, but with mean vectors $(0,0,0)$ and $({\delta }_0,{\delta }_1,{\delta }_2)$, respectively. More specifically, if $({\tilde{X}}_0,{\tilde{X}}_1,{\tilde{X}}_2)$ and $({\tilde{Y}}_0,{\tilde{Y}}_1,{\tilde{Y}}_2)$ denote the random vectors for the log-transformed biomarker data for malignant and benign groups, respectively, for $k=0,1,2$, let ${\mu }_k$ denote the mean of ${\tilde{Y}}_k$ and $s_k^2$ the common variance of ${\tilde{X}}_k$ and ${\tilde{Y}}_k$. Then, the standardized biomarker values will be $X_k=({\tilde{X}}_k-{\mu }_k)/s_k$ and $Y_k=({\tilde{Y}}_k-{\mu }_k)/s_k$. In the data analysis, ${\mu }_k$ and $s_k^2$ will be replaced by their sample estimates. Assuming that B and C add a similar amount of information to A, let $\mathrm{corr}(X_0,X_1)=\mathrm{corr}(X_0,X_2)=\mathrm{corr}(Y_0,Y_1)=\mathrm{corr}(Y_0,Y_2)=0.3$ and $\mathrm{corr}(X_1,X_2)=\mathrm{corr}(Y_1,Y_2)=0.3\sqrt{2}$. If we assume ${\theta }_0=0.7$ and ${\theta }_1={\theta }_2=0.8$, the corresponding location-shift parameters are $({\delta }_0,{\delta }_1,{\delta }_2)=(0.742,1.190,1.190)$. By assuming a prevalence of malignancy $\gamma =0.5$, we have ${\rho }_{12}=\mathrm{corr}({\widehat{\theta }}_1-{\widehat{\theta }}_0,{\widehat{\theta }}_2-{\widehat{\theta }}_0)=0.65$. Using $\alpha =0.05$, the required sample size for 80% of power is $N=184$, or $m=\gamma N=92$ malignant cases and $n=(1-\gamma )N=92$ benign cases. From 5000 simulation samples, we observe an empirical FWER of $\widehat{\alpha }=0.056$ under $H_0$ and power of $1-\widehat{\beta }=0.888$ under $H_a$. We find that this sample size is somewhat overpowered.

6. Discussion

A Dunnett-type testing method is proposed to compare the performance of a control biomarker with $K(\ge$2) new biomarkers using ROC curves. The test statistic is completely non-parametric and robust. Also proposed is a sample size calculation method for the Dunnett-type test. For a sample size calculation, we have to specify the distribution of the biomarkers. The biomarkers to be compared are collected from each subject, so they are usually correlated. Furthermore, the raw data of the biomarkers are usually positive, so their parameters, such as means, variances, and correlation coefficients, are complicatedly associated. As a result, it is difficult to specify the parameter values in a sample size calculation.

In our numerical studies, we assume equal correlation coefficients among the biomarkers for simplicity. But, the true correlation structure may be more complicated among real biomarkers. Our test statistic automatically accounts for any type of the true correlation structure by using the so-called GEE-type approach [14]. In sample size calculation, we have to specify the true correlation coefficients based on our best estimation. If we have pilot data, we may use the estimated correlation coefficients as the true ones in the sample size calculation. In this sense, we may consider a two-stage design of biomarker projects to estimate the correlation coefficients among the biomarkers from the first-stage data and calculate the sample size for the whole project combining the stage 1 and 2 data sets. In this case, we do not need a type I error adjustment for the final testing since we do not conduct any statistical testing at the first stage.

Since the AUC of ROC curves is a rank statistic, the AUCs and the proposed test statistic are invariant to a monotone transformation of biomarker data. Utilizing this nice property, Jung [11] proposed to calculate sample sizes based on the log-transformed data for the easy specification of the parameter values. We have extended his two-sample testing method to $(K+1)$-sample testing by controlling the FWER for multiple testing adjustment. Through simulations, we find that the Dunnett-type test controls the FWER accurately and the calculated sample sizes maintain the specified power closely overall. One-sided critical regions are used in the presentation of our method, but the extension to a two-sided test is straightforward.

To simplify the discussion, we have assumed that the transformed data follow a normal distribution, but it can be any distribution with a location-shift model between the benign and malignant groups. Whatever the transformed distribution, the correct distribution functions should be used in the sample size calculation. Note that there exists a transformation for any distribution transforming to a normal distribution, although it may not be a log-transformation. However, a log-transformation performs this goal relatively well in the real analysis of positive data.

The Dunnett test is popularly used in clinical trial or retrospective clinical studies to compare multiple experimental treatment groups with a common control group with a multiple testing adjustment. Usually, the groups to be compared are independent. For example, a Dunnett-type test and its sample size calculation method were proposed to compare $K(\ge$2) experimental arms with a common control arm using a time-to-event endpoint [15]. In this paper, however, the $K+1$ biomarkers are to be compared since all the $K+1$ biomarkers are collected from each subject.

Contrary to the case of the Dunnett-type test, we may want to compare the $K\ge 3$ groups without a common control. In this case, we usually use a one-way ANOVA test. Jung and Hui [16] proposed a sample size calculation method for the ANOVA-type log-rank tests to compare $K(\ge$3) treatment groups using a time-to-event endpoint.

Our Fortran program, developed for sample size calculation and simulations, is available from the author upon request.