MODE: Minimax Optimal Deterministic Experiments for Causal Inference in the Presence of Covariates

Abstract

Data-driven decision-making has become crucial across various domains. Randomization and re-randomization are standard techniques employed in controlled experiments to estimate causal effects in the presence of numerous pre-treatment covariates. This paper quantifies the worst-case mean squared error of the difference-in-means estimator as a generalized discrepancy of covariates between treatment and control groups. We demonstrate that existing randomized or re-randomized experiments utilizing Monte Carlo methods are sub-optimal in minimizing this generalized discrepancy. To address this limitation, we introduce a novel optimal deterministic experiment based on quasi-Monte Carlo techniques, which effectively minimizes the generalized discrepancy in a model-independent manner. We provide a theoretical proof indicating that the difference-in-means estimator derived from the proposed experiment converges more rapidly than those obtained from completely randomized or re-randomized experiments using Mahalanobis distance. Simulation results illustrate that the proposed experiment significantly reduces covariate imbalances and estimation uncertainties when compared to existing randomized and deterministic approaches. In summary, the proposed experiment serves as a reliable and effective framework for controlled experimentation in causal inference.

1. Introduction

In recent years, data-driven decision-making has been routinely employed in various contexts, including evaluating the efficacy of medical treatments, vaccinations, training programs, marketing campaigns, and other interventions. The crux of effective decision-making lies in the ability to manage uncertainty and make informed choices based on available information. The concept of entropy, which quantifies uncertainty in a probability distribution, plays a crucial role in understanding how information is generated and utilized in these contexts.

Suppose an experimenter has collected data on n units, along with baseline covariate information. The question arises: how should treatments be allocated based on these pre-treatment covariates to provide precise causal estimates of treatment effects on potential outcomes? Here, the balance of information—essentially the reduction of uncertainty—across treatment groups becomes vital for drawing valid conclusions.

Randomized experiments, including completely randomized experiments and re-randomized experiments [1], are commonly employed as control methods. However, ref. [2] argued that a deterministic experiment can strictly dominate all randomized experiments in minimizing estimation error when continuous covariates are present. The ability of a deterministic design to minimize covariate imbalance reflects its potential to reduce uncertainty in treatment effect estimates, thereby leading to more reliable causal inferences. This deterministic experiment seeks to balance the covariate distribution across treatments, resulting in fairer experiments relative to random assignments. Consequently, deterministic experiments often appear more appealing in practice.

Ref. [2] developed a Bayesian optimal deterministic experiment (BODE) by minimizing the expected mean squared error (MSE) risk of the estimated treatment effect across assumed stochastic processes for the potential outcomes. However, identifying a BODE requires specifying approximate prior distributions for the potential outcomes, which introduces additional uncertainty in the modeling process.

In this paper, we introduce a novel type of optimal deterministic experiment, termed the minimax optimal deterministic experiment (MODE). The design criterion to be minimized is the maximum MSE risk associated with treatment assignment over a class of mean and variance functions of potential outcomes. This model-independent approach inherently reduces uncertainty by avoiding reliance on specific parametric assumptions. The bias component of the maximum risk is proportional to the generalized discrepancy, which quantifies covariate imbalance by measuring the difference between the empirical distributions of covariates in treatment and control groups. This generalized discrepancy serves as a critical measure of information loss, providing insight into the uncertainty surrounding treatment assignments.

Quasi-Monte Carlo methods are employed to generate the MODE. From a covariate balance perspective, the covariates in a randomized experiment can be viewed as a Monte Carlo sample from the covariate population. As a result, existing randomized experiments are sub-optimal, and the proposed MODE exhibits reduced covariate imbalance and lower estimation MSE. Typically, the MODE decreases the MSE of the difference-in-means estimator based on the completely randomized experiments (CRE) from $O_p\left(n^{-1}\right)$ to $o\left(n^{-1}\right)$ when potential outcomes are fixed. Thus, for most online control experiments prioritizing estimation accuracy and the effective management of uncertainty, the proposed MODE is preferred. Additionally, we provide re-randomization relaxation of the MODE for situations where randomization is necessary due to practical constraints.

The remainder of this paper is organized as follows. In Section 1.1, we review related experimental designs for causal inference. Our main contributions are summarized in Section 1.2. In Section 2, we formulate the problem of finding a MODE. Section 3 derives the large sample properties of the MODE. Section 4 theoretically compares the MODE with existing randomized experiments. Section 5 compares the finite sample performance of the MODE with that of existing experiments across different potential outcomes models. Finally, we conclude this paper and discuss future works in Section 6. For clarity, all proofs are provided in Appendix B.

1.1. Related Works

Completely randomized experiments are considered the gold standard for estimating causal effects, as randomization balances all potential confounding variables—both observed and unobserved—on average, thereby providing reliable estimates of treatment effects. However, as highlighted by [3], in a particular randomized experiment, covariates between treatment groups may become significantly unbalanced, leading to increased estimation error and greater uncertainty about the treatment effects. As pointed out by [1], with 10 independent covariates, the probability of at least one covariate showing a significant difference at the 5% significance level between treatment and control groups is about 40%. Similarly, ref. [4] emphasized that achieving covariate balance (e.g., through matching) enhances robustness to varying parametric assumptions.

Numerous methods exist in the design stage of an experiment to address covariate imbalance, including blocked randomization [5] and re-randomization [1,6]. Blocking is a classical approach that is effective for balancing a limited number of discrete covariates. In contrast, re-randomization, as proposed by [1], offers a comprehensive framework for reducing covariate imbalance, thereby enhancing the reliability of causal estimates. In this method, the randomized experiment is repeated until the covariates between treatment groups meet a pre-specified balance criterion. Thus, re-randomization generalizes blocking to accommodate multiple covariates with various values. For an in-depth review of the design and analysis of randomized experiments, please refer to [7].

However, from the perspective of statistical decision-making, randomized experiments are dominated by deterministic experiments [2]. Recent works by [8,9] have introduced optimal deterministic experiments based on assumed parametric potential outcome models. The BODE proposed by [2] depends on the chosen prior for potential outcomes, which adds another layer of uncertainty. All of these optimal designs are sensitive to the specified potential outcome models. In contrast, the proposed MODE is model-independent, thereby mitigating the uncertainty associated with model specification.

The generalized discrepancy utilized in the MODE is analogous to the optimization objective of kernel optimal matching [10], which is employed for covariate balancing in the analysis stage of an experiment. However, the generalized discrepancy is applied during the design stage, while the optimization variables in kernel optimal matching are the weights for the estimator. Unlike post hoc methods, the MODE is implemented entirely in the design phase, ensuring it remains unaffected by outcome data. More convincing reasons for why as much as possible should be done in the design phase of an experiment can be found in [11].

1.2. Main Contributions

Firstly, we introduce a novel type of model-independent deterministic experiment, termed the MODE, which minimizes uncertainty in treatment effect estimation. Secondly, we provide a theoretical proof demonstrating that the proposed MODE reduces covariate imbalance and the estimation MSE of the completely randomized experiment or the re-randomization experiment using Mahalanobis distance, decreasing it from $O_p\left(n^{-1}\right)$ to $o\left(n^{-1}\right)$ when potential outcomes are fixed. The third contribution of this paper is establishing the relationship between the maximum risk of causal estimators and the generalized discrepancy, emphasizing how reducing this discrepancy enhances the overall information quality in causal inference.

2. Problem Setups

Let $\mathcal{D}={\left\{(T_i,X_i,Y_i)\right\}}_{i=1}^n$ denote the observable data for a population of n units, where $T_i$ is a binary treatment for the i-th unit, i.e., the i-th unit is treated if and only if $T_i=1$. $X_i$ and $Y_i$ represent the p-dimensional pre-treatment covariates and the real-valued outcome for the i-th units, respectively. Let $\mathcal{X}={\left\{X_i\right\}}_{i=1}^n$ represent all the p-dimensional covariates. We define $n_1={\sum }_{i=1}^n{T}_i$ and $n_0={\sum }_{i=1}^n(1-T_i)$ as the numbers of treated and untreated units, respectively. The set ${\mathbb{T}}_{n,n_1}=\{(T_1,\cdots ,T_n)\in {\{0,1\}}^n,{\sum }_{j=1}^n{T}_j=n_1\}$ includes all possible treatment assignments with $n_1$ treated units.

Under the finite population potential outcomes framework and the stable unit-treatment-value assumption [12], we have $Y_i=T_i{Y}_i\left(1\right)+(1-T_i)Y_i\left(0\right)$, where $Y_i\left(1\right)$ and $Y_i\left(0\right)$ are the potential outcomes for the i-th unit under the treatment and control, respectively. This paper considers the randomness arising from both treatment assignment and potential outcomes. For $t\in \{0,1\}$ and $i,j\in \left[n\right]\triangleq \{1,2\cdots ,n\}$, we assume that the conditional mean and covariance of potential outcomes given all covariates are as follows:

$$\begin{array}{cc}\hfill E(Y_i\left(t\right)\mid \mathcal{X})& =f_t\left(X_i\right);\hfill \\ \hfill \mathrm{cov}(Y_i\left(t\right),Y_j\left(t\right)\mid \mathcal{X})& ={\sigma }_t^2\left(X_i\right)\mathbb{I}\{i=j\};\hfill \\ \hfill \mathrm{cov}(Y_i\left(0\right),Y_j\left(1\right)\mid \mathcal{X})& =0,\hfill \end{array}$$

(1)

where $f_t(·)$ and ${\sigma }_t^2(·)$ are unknown mean and variance functions, respectively, and $\mathbb{I}\{·\}$ is the indicator function. The conditions in (1) impose restrictions on the correlations between potential outcomes and covariates, and they hold if $(X_i,Y_i\left(0\right),Y_i\left(1\right)),i\in \left[n\right]$ are mutually independent [10].

Typically, the target of interest is to effectively estimate the average treatment effect

$$\tau =n^{-1}\sum _{i=1}^{n}E(Y_i\left(1\right)-Y_i\left(0\right)\mid \mathcal{X}),$$

(2)

using the difference-in-means estimator

$$\widehat{\tau }=n_1^{-1}\sum _{j=1}^n{T}_j{Y}_j-n_0^{-1}\sum _{j=1}^n(1-T_j)Y_j.$$

(3)

In this paper, we aim to design a deterministic treatment assignment $\mathcal{T}\in {\mathbb{T}}_{n,n_1}$ with $1\le n_1\le n-1$ to improve the estimation precision of the difference-in-means estimator. For simplicity, let $\delta =(f_1\left(x\right),f_0\left(x\right),{\sigma }_1^2\left(x\right),{\sigma }_0^2\left(x\right))$ represent the vector of mean and variance functions. The conditional MSE of $\widehat{\tau }$ given $\mathcal{T}$ can be decomposed as follows:

$$\mathrm{MSE}(\widehat{\tau }\mid \mathcal{T},\mathcal{X},\delta )=E\{{(\widehat{\tau }-\tau )}^2\mid \mathcal{T},\mathcal{X},\delta \}=V(\mathcal{T},\mathcal{X},\delta )+B(\mathcal{T},\mathcal{X},\delta )$$

where the conditional variance and squared bias are given by the following:

$$\begin{array}{cc}\hfill V(\mathcal{T},\mathcal{X},\delta )& =n_1^{-2}\sum _{j=1}^n{T}_j{\sigma }_1^2\left(X_j\right)+n_0^{-2}\sum _{j=1}^n(1-T_j){\sigma }_0^2\left(X_j\right),\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill B(\mathcal{T},\mathcal{X},\delta )& ={\left(n_1^{-1}\sum _{j=1}^n{T}_j{f}_1\left(X_j\right)-n_0^{-1}\sum _{j=1}^n(1-T_j)f_0\left(X_j\right)-\tau \right)}^2.\hfill \end{array}$$

(5)

From the above expression, the MSE risk depends on the treatment assignment $\mathcal{T}$, the set of covariates $\mathcal{X}$, and the vector of unknown functions $\delta$. Therefore, to minimize the MSE, we can carefully design $\mathcal{T}$ by fully utilizing the information contained in $\mathcal{X}$ and $\delta$. Different assumptions regarding $\delta$ lead to distinct optimal treatment assignments [8,9]. However, these assumptions are unverifiable since the potential outcomes are unobservable. In this paper, we adopt a minimax framework to identify the most robust treatment assignment with respect to a class of unknown mean and variance functions. We do not impose specific functional forms on these unknown functions; instead, we restrict them to belong to a function class $\Delta$, as rigorously defined in (7). The maximum risk of a treatment is quantified by the worst-case MSE for that treatment over all possible true mean and variance functions in $\Delta$, i.e.,

$$R\left(\mathcal{T}\right)=\underset{\delta \in \Delta }{max}\mathrm{MSE}(\widehat{\tau }\mid \mathcal{T},\mathcal{X},\delta ).$$

(6)

The most robust treatment assignment with respective to $\Delta$ is the one for which the maximum risk achieves the minimum over all possible treatments in ${\mathbb{T}}_n\triangleq {\cup }_{n_1=1}^{n-1}{\mathbb{T}}_{n,n_1}$. More precisely, we define the MODE as follows.

Definition 1. 

A treatment ${\mathcal{T}}^{*}$ is called a MODE if $R\left({\mathcal{T}}^{*}\right)={min}_{\mathcal{T}\in {\mathbb{T}}_n}R\left(\mathcal{T}\right).$

Therefore, the task of obtaining a MODE can be viewed as a game between the experimenter, who chooses a treatment assignment $\mathcal{T}$ from ${\mathbb{T}}_n$, and nature, which chooses the worst-case potential outcomes structures from $\Delta$. Additionally, the MODE is model-independent, as the maximum risk defined in (6) accounts for the uncertainties of all possible potential outcomes models. If there are multiple treatments with the minimum maximum risk, the MODE in Definition 1 is not unique, but they are equivalent in terms of minimizing the maximum risk. Therefore, we can choose any one of them to conduct the experiment.

3. Minimax Optimal Deterministic Experiments

In this section, we first derive two equivalent expressions for the maximum risk in (6). We then consider the MODE, as outlined in Definition 1, under both fixed and divergent sample sizes. Additionally, we provide an algorithm for generating the MODE.

3.1. Expressions for the Maximum Risk

For any $\mathcal{T}\in {\mathbb{T}}_{n,n_1}$ with $1\le n_1\le n-1$, let ${\mathcal{X}}_1=\{X_j\mid T_j=1,j\in \left[n\right]\}$ and ${\mathcal{X}}_0=\{X_j\mid T_j=0,j\in \left[n\right]\}$ denote the covariate information of the treated and untreated units, respectively. To ensure that the maximum risk $R\left(\mathcal{T}\right)$ in (6) is well defined, we assume that

$$\Delta =\{(f_1,f_0,{\sigma }_1^2,{\sigma }_0^2)\mid f_t\in \Omega ,{(f_t,f_t)}_{\Omega }\le {\gamma }^2,{\sigma }_t^2\left(x\right)\le {\sigma }^2,t\in \{0,1\},$$

(7)

where ${\gamma }^2$ and ${\sigma }^2$ are pre-specified parameters, and $\Omega$ is a reproducing kernel Hilbert space, equipped with the inner product ${(·,·)}_{\Omega }:\Omega \times \Omega \to \mathbb{R}$ and the reproducing kernel $K(·,·):\mathcal{X}\times \mathcal{X}\to \mathbb{R}$. A crucial feature of this kernel is its reproducibility, meaning that ${(K(x,·),f(·))}_{\Omega }=f\left(x\right)$ for any $x\in \mathcal{X}$ and $f\in \Omega$; see [13,14].

After embedding the mean functions into the reproducing kernel Hilbert space $\Omega$, we provide an analytical expression for the maximum risk by bounding the bias term $B(\mathcal{T},\mathcal{X},\delta )$ using a generalized Koksma–Hlawka inequality.

Theorem 1. 

For any $\mathcal{T}\in {\mathbb{T}}_{n,n_1}$ with $1\le n_1\le n-1$, the maximum risk $R\left(\mathcal{T}\right)={max}_{\delta \in \Delta }\mathrm{MSE}(\widehat{\tau }\mid \mathcal{T},\mathcal{X},\delta )$ is expressed as follows:

$$R\left(\mathcal{T}\right)={\sigma }^2(n_1^{-1}+n_0^{-1})+{\gamma }^2{(D_K({\mathcal{X}}_1,\mathcal{X})+D_K({\mathcal{X}}_0,\mathcal{X}))}^2,$$

where ${\sigma }^2$ and ${\gamma }^2$ are constants specified in Δ, and $D_K{({\mathcal{X}}_t,\mathcal{X})}^2$ is the squared generalized discrepancy between ${\mathcal{X}}_t$ and $\mathcal{X}$, defined by

$$D_K{({\mathcal{X}}_t,\mathcal{X})}^2=n^{-2}\sum _{x,y\in \mathcal{X}}K(x,y)-2{\left(n{n}_t\right)}^{-1}\sum _{x\in \mathcal{X},y\in {\mathcal{X}}_t}K(x,y)+n_t^{-2}\sum _{x,y\in {\mathcal{X}}_t}K(x,y).$$

(8)

Theorem 1 shows that the variance component of the maximum risk is minimized by any balanced experiment in ${\mathbb{T}}_{n,n/2}$, i.e., $n_1=n_0=n/2$. The bias component of the maximum risk is determined by the sum of the generalized discrepancies of the treated and untreated covariates. The generalized discrepancy defined in (8) can alternatively be expressed as the squared distance between the empirical distribution functions of the point sets ${\mathcal{X}}_t$ and $\mathcal{X}$, denoted as $F_{n_t}\left(x\right)$ and $F_n\left(x\right)$, respectively. Specifically, we have

$$D_K{({\mathcal{X}}_t,\mathcal{X})}^2=d_{\Omega }{(F_{n_t}\left(x\right),F_n\left(x\right))}^2,$$

where $d_{\Omega }(·,·)$ is the distance metric induced by the inner product ${(·,·)}_{\Omega }$. A point set $\mathcal{P}\subset \mathcal{X}$ with a small discrepancy $D_K(\mathcal{P},\mathcal{X})$ indicates that the empirical distributions of $\mathcal{P}$ and $\mathcal{X}$ are close in terms of the distance metric $d_{\Omega }(·,·)$.

Different choices of the reproducing kernel Hilbert spaces correspond to different generalized discrepancies, resulting in varying expressions for maximum risks. We provide two widely used reproducing kernel Hilbert spaces, as described in [15,16].

Example 1. 

Let $\Omega =X_2\left({[0,1]}^p\right)$, which is the set of all real-valued functions on ${[0,1]}^p$ that have square-integrable mixed first derivatives [15]. In this context, $D_K({\mathcal{X}}_t,\mathcal{X})$ can represent various types of discrepancies, such as the generalized $L_2$-discrepancy and the centered $L_2$-discrepancy, etc. For instance, the kernel function for the centered $L_2$-discrepancy is $K(x,y)={\prod }_{j=1}^p(1+0.5|x_j-0.5|+0.5|y_j-0.5|-0.5|x_j-y_j\left|\right)$.

Example 2. 

Let $\Omega =\mathcal{N}\left({\mathbb{R}}^p\right)$. For the explicit form of this space, refer to [16]. It has been established that the space $\mathcal{N}\left({\mathbb{R}}^p\right)$ contains the Sobolev space ${\mathcal{W}}_{(p+1)/2,2}$ when p is even; otherwise, $\mathcal{N}\left({\mathbb{R}}^p\right)={\mathcal{W}}_{(p+1)/2,2}$, where ${\mathcal{W}}_{s,2}$ is the set of functions whose s-th order derivatives are square-integrable. In this context, $D_K({\mathcal{X}}_t,\mathcal{X})$ is the energy distance between ${\mathcal{X}}_t$ and $\mathcal{X}$, with the kernel function defined as $K(x,y)=-{\left\{{\sum }_{j=1}^p{(x_j-y_i)}^2\right\}}^{1/2}$.

According to the property of the generalized discrepancy defined in (8), we provide another expression for the maximum risk.

Corollary 1. 

The maximum risk expression in Theorem 1 is equivalent to the following:

$$R\left(\mathcal{T}\right)={\sigma }^2(n_1^{-1}+n_0^{-1})+{\gamma }^2{D}_K{({\mathcal{X}}_1,{\mathcal{X}}_0)}^2.$$

Corollary 1 indicates that the bias component of the maximum risk can be represented as the squared generalized discrepancy between the treated covariates ${\mathcal{X}}_1$ and the untreated covariates ${\mathcal{X}}_0$. Consequently, the closer the empirical distributions of ${\mathcal{X}}_1$ and ${\mathcal{X}}_0$ are, the smaller the maximum risk becomes. The generalized discrepancy $D_K({\mathcal{X}}_1,{\mathcal{X}}_0)$ defined in (8) can be equivalently expressed as a maximum mean discrepancy [17], given by the following:

$$D_K({\mathcal{X}}_1,{\mathcal{X}}_0)=\underset{\varphi \in \Omega ,{(\varphi ,\varphi )}_{\Omega }\le 1}{max}|n_1^{-1}\sum _{x\in {\mathcal{X}}_1}\varphi \left(x\right)-n_0^{-1}\sum _{x\in {\mathcal{X}}_0}\varphi \left(x\right)|.$$

Thus, $D_K({\mathcal{X}}_1,{\mathcal{X}}_0)$ serves as a measure of covariate imbalance that incorporates a range of mean differences rather than focusing solely on a specific mean difference, as done with the Mahalanobis distance defined in (14). A zero generalized discrepancy implies that the covariate distributions in the treatment and control groups are identical. This distributional consistency cannot be achieved through mean matching alone. Additionally, using the equivalent expression for maximum risk in Corollary 1 enhances computational efficiency, as evaluating the maximum risk of a treatment requires calculating the value of the discrepancy function only once.

3.2. MODE and Asymptotic MODE

For any fixed sample size n and discrepancy function $D_K(·,·)$ in (10), we identify the MODE in Definition 1 by solving the following:

$$({\mathcal{X}}_1^{*},{\mathcal{X}}_0^{*})=arg\underset{({\mathcal{X}}_1,{\mathcal{X}}_0)\in {\mathbb{X}}_n}{min}R\left(\mathcal{T}\right),$$

(9)

where ${\mathbb{X}}_n=\{({\mathcal{X}}_1,{\mathcal{X}}_0)\mid {\mathcal{X}}_1\subset \mathcal{X},1\le |{\mathcal{X}}_1|\le n-1,{\mathcal{X}}_0=\mathcal{X}\setminus {\mathcal{X}}_1\}$. The treatment ${\mathcal{T}}_n^{*}$ corresponding to the above $({\mathcal{X}}_1^{*},{\mathcal{X}}_0^{*})$ is a MODE. When n is small, we can obtain the MODE ${\mathcal{T}}_n^{*}$ by enumerating all elements in ${\mathbb{X}}_n$. However, this solution depends on the pre-specified parameters ${\gamma }^2$ and ${\sigma }^2$, and it becomes almost infeasible for large n values since the cardinality of ${\mathbb{X}}_n$ grows exponentially to $2^n-2$. Consequently, we explore the concept of the asymptotic MODE as n approaches infinity. To make progress, we require that the considered discrepancy function belongs to the following set:

$${\mathcal{D}}_{\mathcal{X}}=\{D_K(·,\mathcal{X})\mid \underset{\mathcal{P}\subset \mathcal{X},\mid \mathcal{P}\mid =\alpha n,\alpha \in (0,1)}{min}{D}_K(\mathcal{P},\mathcal{X})=o\left(n^{-1/2}\right)\}.$$

(10)

Notably, a Monte Carlo point set of size $\alpha n$ with $\alpha \in (0,1)$ will typically have a discrepancy of order $O_p\left(n^{-1/2}\right)$. In contrast, a quasi-Monte Carlo point set, also referred to as representative points and defined as the minima of a discrepancy, possesses a discrepancy of order $o\left(n^{-1/2}\right)$. This low-discrepancy property supports the theoretical assertion that quasi-Monte Carlo points outperform Monte Carlo points and is fulfilled by commonly used discrepancies. For example, ref. [16] demonstrated that the n-point support points converge at a rate of $O\left(n^{-1/2}{(logn)}^{-(1-v)/\left(2p\right)}\right)$ for any $v\in (0,1)$, measured by the energy distance presented in Example 2. Additionally, Theorem 1 of [18] shows that if the empirical distribution function of the covariates $F_n\left(x\right)$ is joint independent, the n-point data-driven subsamples converge at a rate of $O\left(n^{-1+v}\right)$ for any $v\in (0,1/2)$, measured by the generalized $L_2$-discrepancy presented in Example 1. Thus, all discrepancies in Examples (1) and (2) belong to ${\mathcal{D}}_{\mathcal{X}}$.

For any given discrepancy function in ${\mathcal{D}}_{\mathcal{X}}$, we identify an asymptotic MODE as follows.

Theorem 2. 

For any discrepancy $D_K(·,\mathcal{X})$ in (10), the balanced treatment assignment ${\mathcal{T}}_n^{*}$ with ${\mathcal{X}}_1^{*}=arg{min}_{{\mathcal{X}}_1\subset \mathcal{X},\mid {\mathcal{X}}_1\mid =n/2}{D}_K({\mathcal{X}}_1,\mathcal{X}\setminus {\mathcal{X}}_1)$ and ${\mathcal{X}}_0^{*}=\mathcal{X}\setminus {\mathcal{X}}_1^{*}$, is an asymptotic MODE, with its maximum risk given by $R\left({\mathcal{T}}_n^{*}\right)={min}_{\mathcal{T}\in {\mathbb{T}}_n}R\left(\mathcal{T}\right)=4{\sigma }^2{n}^{-1}+o\left(n^{-1}\right)$, as $n\to \infty$.

Theorem 2 ensures that the variance component of the maximum risk of the asymptotic MODE is minimized, while its bias component becomes negligible compared to its variance component by minimizing the covariate imbalance, defined by the discrepancy between the treated and untreated covariates. Consequently, experimental units with non-homogeneous covariates perform comparably to those with homogeneous covariates when utilizing the asymptotic MODE.

3.3. Algorithm for Generating Asymptotic MODE

Based on Theorem 2, we can implement the asymptotic MODE in the following steps. Firstly, we search for $n/2$ points from $\mathcal{X}$ that have the lowest discrepancy relative to the entire covariate dataset $\mathcal{X}$. This subset is denoted as ${\mathcal{X}}_1^{*}$, and the remaining $n/2$ covariates are denoted as ${\mathcal{X}}_0^{*}$, i.e., $\mathcal{X}\setminus {\mathcal{X}}_1^{*}$. Then, we can assign the treatment to the units with ${\mathcal{X}}_1^{*}$ (or ${\mathcal{X}}_0^{*}$) and the control to the units with ${\mathcal{X}}_0^{*}$ (or ${\mathcal{X}}_1^{*}$). The above processes are summarized in Algorithm 1.

Algorithm 1: MODE: minimax optimal deterministic experiment
	Input:  $\mathcal{X}={\left\{X_i\right\}}_{i=1}^n$: the covariates of n experimental units; $K(·,·)$: a valid kernel function defined on $\mathcal{X}\times \mathcal{X}$; M: the maximum allowable sampling times.
	Output: The MODE with treatment units ${\mathcal{X}}_1^{*}$ and control units ${\mathcal{X}}_0^{*}$.
1	if $K(x,y)=-{\left\{{\sum }_{j=1}^p{(x_j-y_j)}^2\right\}}^{1/2}$ then
2			${\mathcal{X}}_1^{*}=Twin(\mathcal{X},n/2),$ where $Twin(\mathcal{X},n/2)$ is the $n/2$ points obtained by the twinning method [19].
3	else if $K(x,y)=\mathbb{K}(T_{\mathcal{X}}\left(x\right),T_{\mathcal{X}}\left(y\right))$ then
4			${\mathcal{X}}_1^{*}=\mathrm{DDS}(\mathcal{X},n/2)$, where $\mathrm{DDS}(\mathcal{X},n/2)$ is the $n/2$ points obtained by the data-driven subsampling method [18].
5	else
6			for m=1:M do
7				${\mathcal{X}}_{1m}=SRS(\mathcal{X},n/2),D_m=D_K({\mathcal{X}}_{1m},\mathcal{X}\setminus {\mathcal{X}}_{1m})$.
8			end
9			$m^{*}=arg{min}_{m=1:M}{D}_m,{\mathcal{X}}_1^{*}={\mathcal{X}}_{1m^{*}}$.
10	end
11	${\mathcal{X}}_0^{*}=\mathcal{X}\setminus {\mathcal{X}}_1^{*}$.

This algorithm systematically selects treatment and control groups to minimize covariate imbalance, thereby enhancing the robustness of treatment assignment. Any valid discrepancy on $\mathcal{X}\times \mathcal{X}$ can be used as the input for Algorithm 1. The optimality of the MODE is guaranteed by Theorems 2 and 3, provided the selected discrepancy belongs to ${\mathcal{D}}_K$.

We tailor different optimization methods to various discrepancy functions. For continuous covariates, we recommend using the energy distance [16] or the empirical F-discrepancy [18]. When employing the energy distance as the discrepancy, characterized by the kernel $K(x,y)=-{\left\{{\sum }_{j=1}^p{(x_j-y_j)}^2\right\}}^{1/2}$, we recommend utilizing the twinning method proposed by [19] to identify ${\mathcal{X}}_1^{*}$. For the empirical F-discrepancy using the kernel $K(x,y)=\mathbb{K}(T_{\mathcal{X}}\left(x\right),T_{\mathcal{X}}\left(y\right))$, where $T_{\mathcal{X}}\left(x\right)={(F_1\left(x_1\right),\cdots ,F_p\left(x_p\right))}^T$, $F_j\left(x_j\right)$ is the empirical distribution function of the j-th component of $\mathcal{X}$, and $\mathbb{K}(·,·)$ is any valid kernel on ${[0,1]}^p\times {[0,1]}^p$, we recommend the data-driven subsampling method proposed by [18] to find ${\mathcal{X}}_1^{*}$. For the details of the twinning and the data-driven subsampling methods in Algorithm 1, please refer to [19] and [18], respectively. For discrete covariates, the Lee-discrepancy [20] is preferred. For other valid discrepancies, a Monte Carlo approximation method is applied. This involves randomly sampling M subsets from $\mathcal{X}$ and selecting the one, ${\mathcal{X}}_1^{*}$, with the minimum discrepancy value. Ref. [2] demonstrated that after $M=500$ times sampling, the probability that the output ${\mathcal{X}}_1^{*}$ is better then 99% of all possible candidates is already larger than 99%, and significantly larger values of M are also feasible in practice.

Example 3. 

This example demonstrates the usefulness of the MODE procedure using the red wine quality dataset from the UCI databases library. We evaluate the impact of treatments, such as a new storage method, on red wine quality. Two key physicochemical properties, "pH” and “alcohol,” which may influence the final outcome, are treated as covariates. We assume that all these covariates of the red wines have been collected. To minimize the influence of covariates on the estimation of treatment effects, it is essential to ensure that the covariates of the treatment and control groups are as similar as possible.

For clarity, we focus on the first 100 samples in the dataset, treating them as experimental units, and scale their covariates into the range ${[0,1]}^2$. Given that the covariates are continuous, we adopt the energy distance with the kernel function $K(x,y)=-{\left\{{\sum }_{j=1}^p{(x_j-y_j)}^2\right\}}^{1/2}$ in Algorithm 1. Figure 1 illustrates the covariates of equally sized treatment and control groups divided by the CRE and MODE methods. Intuitively, the covariates of the treatment and control groups created by MODE appear more balanced than those by CRE, particularly near the $(1,1)$ point. Quantitatively, the Mahalanobis distance between the covariates of the treatment and control groups under MODE is $0.05$, significantly smaller than the $3.41$ observed with CRE. This demonstrates that MODE provides a more effective approach for achieving covariate balance, which is crucial for reducing bias in treatment effect estimation.

4. Theoretical Comparison with Randomized Experiments

In this section, we theoretically compare the proposed MODE with completely randomized experiments and re-randomized experiments. When the potential outcomes are fixed, the MODE reduces the MSE of the CRE or the re-randomization using Mahalanobis distance, from $O_p\left(n^{-1}\right)$ to $o\left(n^{-1}\right)$. Thus, the proposed MODE serves as a super effective covariate balance technique.

4.1. Comparison with Completely Randomized Experiments

For simplicity, we denote the difference-in-means estimators based on the CRE and the MODE as ${\widehat{\tau }}_{cre}$ and ${\widehat{\tau }}_{mode}$, respectively. Under the CRE, where $\mathcal{T}$ is randomly sampled form ${\mathbb{T}}_{n,n_1}$ with $1\le n_1\le n-1$, ${\widehat{\tau }}_{cre}$ serves as an unbiased estimator for $\tau$, i.e., $E({\widehat{\tau }}_{cre}\mid \mathcal{X},\delta )=\tau$, with a conditional variance given by the following:

$$\mathrm{var}({\widehat{\tau }}_{cre}\mid \mathcal{X},\delta )={\left({\displaystyle \genfrac{}{}{0pt}{}{n}{n_1}}\right)}^{-1}\sum _{\mathcal{T}\in {\mathbb{T}}_{n,n_1}}\mathrm{MSE}({\widehat{\tau }}_{cre}\mid \mathcal{T},\mathcal{X},\delta ).$$

(11)

The expression in (11) indicates that this randomization variance under the CRE equals the average of the conditional MSEs across treatments in ${\mathbb{T}}_{n,n_1}$. In contrast, the MODE in (1) minimizes the maximum MSE over a class of mean and variance functions.

We define $S^2\left(f_t\right)={(n-1)}^{-1}{\sum }_{i=1}^n{(f_t\left(X_i\right)-{\overline{f}}_t)}^2$ for $t\in \{0,1\}$, and $S^2(f_1,f_0)={(n-1)}^{-1}{\sum }_{i=1}^n{({\tau }_i-\tau )}^2$, where ${\overline{f}}_t=n^{-1}{\sum }_{i=1}^n{f}_t\left(X_i\right)$ for $t\in \{0,1\}$, and ${\tau }_i=f_1\left(X_i\right)-f_0\left(X_i\right)$ for $i\in \left[n\right]$. The above randomization variance can be expressed more precisely as follows (see Appendix A for details):

$$\begin{array}{cc}\hfill & \mathrm{var}({\widehat{\tau }}_{cre}\mid \mathcal{X},\delta )=E_V(\mathcal{X},\delta )+V_E(\mathcal{X},\delta )\hfill \\ \hfill & E_V(\mathcal{X},\delta )=n_1^{-1}{S}^2\left(f_1\right)+n_0^{-1}{S}^2\left(f_0\right)-n^{-1}{S}^2(f_1,f_0),\hfill \\ \hfill & V_E(\mathcal{X},\delta )={\left(n{n}_1\right)}^{-1}\sum _{i=1}^n{\sigma }_1^2\left(X_i\right)+{\left(n{n}_0\right)}^{-1}\sum _{i=1}^n{\sigma }_0^2\left(X_i\right).\hfill \end{array}$$

(12)

The term $E_V(\mathcal{X},\delta )$ aligns with the variance expression when the potential outcomes are fixed [21], while the term $V_E(\mathcal{X},\delta )$ quantifies the randomness stemming from the potential outcomes.

Following the percent reduction in variance proposed by [1], we define the percent reduction in MSE as the percentage by which the MODE reduces the randomization MSE of the difference-in-means estimator:

$$\mathrm{PR}({\widehat{\tau }}_{cre},{\widehat{\tau }}_{mode}\mid \mathcal{X},\delta )={\displaystyle \frac{100(\mathrm{var}({\widehat{\tau }}_{cre}\mid \mathcal{X},\delta )-\mathrm{MSE}({\widehat{\tau }}_{mode}\mid {\mathcal{T}}_n^{*},\mathcal{X},\delta ))}{\mathrm{var}({\widehat{\tau }}_{cre}\mid \mathcal{X},\delta )}}$$

(13)

where ${\mathcal{T}}_n^{*}$ is the asymptotic MODE provided in Theorem 2. It is important to note that this percent reduction in MSE depends on the covariates $\mathcal{X}$ and the unknown mean and variance functions in $\delta$. Under certain mild conditions, we provide a lower bound for this percent reduction in MSE.

Assumption 1.

 (i) $0<\underset{n\to \infty }{lim\; inf}{n}_1/n\le \underset{n\to \infty }{lim\; sup}{n}_1/n<1$; (ii) $n^{-1}{\sum }_{i=1}^n{f}_t{\left(X_i\right)}^2=O_p\left(1\right)$, $n^{-1}{\sum }_{i=1}^n{\sigma }_t^2\left(X_i\right)=O_p\left(1\right)$ for $t\in \{0,1\}$.

The first condition is necessary for the existence of the difference-in-means estimator. The second condition ensures that certain moments of the covariates are bounded in probability, which is easily satisfied if the mean and variance functions are bounded.

Theorem 3. 

Under the mild conditions specified in Assumption 1, the percent reduction in MSE defined in (13) satisfies the following:

$$\underset{n\to \infty }{lim}\mathrm{PR}({\widehat{\tau }}_{cre},{\widehat{\tau }}_{mode}\mid \mathcal{X},\delta )\ge {\displaystyle \frac{E_V(\mathcal{X},\delta )}{E_V(\mathcal{X},\delta )+V_E(\mathcal{X},\delta )}}>0.$$

Theorem 3 establishes that the proposed MODE effectively reduces the MSE of the difference-in-means estimator compared to the CRE. This reduction becomes more pronounced when the term $V_E(\mathcal{X},\delta )$ defined in (12) is small, or when the term $E_V(\mathcal{X},\delta )$ defined in (12) is large. An intuitive explanation of Theorem 3 is that while the CRE balances covariates on average, the actual distributions of covariates in the treatment and control groups may remain unbalanced in a specific experiment, resulting in larger estimation variance. In contrast, the MODE carefully arranges the treated and untreated units to ensure that their covariate distributions align as closely as possible, thereby achieving a smaller MSE.

It is evident that under Assumption 1, $\mathrm{var}({\widehat{\tau }}_{cre}\mid \mathcal{X},\delta )=O_p\left(n^{-1}\right)$. If the potential outcomes are fixed, i.e., ${\sigma }_1^2\left(x\right)\equiv 0$ and ${\sigma }^2\left(x\right)\equiv 0$, Theorem 2 implies that $\mathrm{MSE}({\widehat{\tau }}_{mode}\mid {\mathcal{T}}_n^{*},\mathcal{X},\delta )=o\left(n^{-1}\right)$. Thus, the MODE demonstrates superiority over the CRE.

Corollary 2. 

Under the mild conditions outlined in Assumption 1, if ${\sigma }_1^2\left(x\right)\equiv 0$ and ${\sigma }^2\left(x\right)\equiv 0$, then the MODE reduces the MSE of the CRE from $O_p\left(n^{-1}\right)$ to $o\left(n^{-1}\right)$, i.e., ${lim}_{n\to \infty }\mathrm{PR}({\widehat{\tau }}_{cre},{\widehat{\tau }}_{mode}\mid \mathcal{X},{\delta }_0)=100\%$ with ${\delta }_0=(f_1\left(x\right),f_2\left(x\right),0,0)$.

Corollary 2 indicates that the convergence rate of the MSE is improved by employing the MODE compared to the CRE when the potential outcomes are fixed. This further establishes the superiority of the proposed MODE.

4.2. Comparison with Re-Randomized Experiments

In this subsection, we demonstrate that the proposed MODE outperforms the re-randomized experiment using Mahalanobis distance [1], referred to as ReM, in terms of minimizing the MSE. Furthermore, the proposed MODE can also be identified as a specific case of a re-randomized experiment.

The re-randomization technique enhances the performance of a randomized experiment by excluding assignments that yield unbalanced covariate distributions prior to the experiment’s initiation. Specifically, a re-randomization criterion is defined as a binary function $\varphi$ such that a treatment assignment $\mathcal{T}$ belongs to the restricted randomization set if and only if $\varphi \left(\mathcal{T}\right)=1$. The restricted set for ReM is defined as follows:

$${\varphi }_m\left(\mathcal{T}\right)=\mathbb{I}\{\mathcal{T}\in {\mathbb{T}}_{n,n_1},M({\mathcal{X}}_1,{\mathcal{X}}_0)\le a\},$$

where a is a pre-specified constant, and $M({\mathcal{X}}_1,{\mathcal{X}}_0)$ is the Mahalanobis distance between the treatment and control groups defined by the following:

$$M({\mathcal{X}}_1,{\mathcal{X}}_0)={\displaystyle \frac{n_1{n}_0}{n}}{({\overline{X}}_1-{\overline{X}}_0)}^T\mathrm{cov}{\left(X\right)}^{-1}({\overline{X}}_1-{\overline{X}}_0),$$

(14)

where ${\overline{X}}_1=n_1^{-1}{\sum }_{i=1}^n{T}_i{X}_i$, ${\overline{X}}_0=n_1^{-1}{\sum }_{i=1}^n(1-T_i)X_i$, and $\mathrm{cov}\left(X\right)$ is the sample covariance matrix. The ReM is considered balanced when $n_1=n/2$.

Consider the scenario where the potential outcomes are fixed and the causal effect is additive, i.e., ${\sigma }_1^2\left(x\right)\equiv 0,{\sigma }_0^2\left(x\right)\equiv 0$ and $Y_i\left(1\right)=Y_i\left(0\right)+\tau$ for $i\in \left[n\right]$. In this case, the balanced ReM reduces the variance of the CRE to $\{1-(1-v_a)R^2\}\mathrm{var}({\widehat{\tau }}_{cre}\mid \mathcal{X},\delta )$ (Theorem 3.2 of [1]), where $v_a=P({\chi }_{p+2}^2\le a)/P({\chi }_p^2\le a)$, ${\chi }_p^2$ is the Chi-square distribution with p degrees of freedom, and $R^2$ represents the squared multiple correlation between ${\left\{Y_i\left(0\right)\right\}}_{i=1}^n$ and ${\left\{X_i\right\}}_{i=1}^n$. Thus, the balanced ReM enjoys the same convergence rate with the CRE. We summarize this conclusion as the following corollary.

Corollary 3. 

If ${\sigma }_1^2\left(x\right)\equiv 0,{\sigma }^2\left(x\right)\equiv 0$, $Y_i\left(1\right)=Y_i\left(0\right)+\tau$ for $i\in \left[n\right]$, and $n^{-1}{\sum }_{i=1}^n{f}_t{\left(X_i\right)}^2=O_p\left(1\right)$ for $t\in \{0,1\}$, then the variance of the balanced ReM is of order $O_p\left(n^{-1}\right)$.

Theorem 2 implies that $\mathrm{MSE}({\widehat{\tau }}_{mode}\mid {\mathcal{T}}_n^{*},\mathcal{X},\delta )=o\left(n^{-1}\right)$ if ${\sigma }_1^2\left(x\right)\equiv 0$ and ${\sigma }^2\left(x\right)\equiv 0$. Thus, as a result of Corollary 3, the proposed MODE improves the convergence rate of the estimated treatment effect compared to the balanced ReM from $O_p\left(n^{-1}\right)$ to $o\left(n^{-1}\right)$. This further reinforces the superiority of using the proposed MODE.

Next, we establish a connection between the MODE and re-randomization. In our setting, the restricted set for the MODE, ${\mathcal{T}}_n^{*}$, can be defined as follows:

$${\varphi }_{mode}\left(\mathcal{T}\right)=\mathbb{I}\{\mathcal{T}\in {\mathbb{T}}_{n,n/2},D_K({\mathcal{X}}_1,{\mathcal{X}}_0)=\underset{({\mathcal{X}}_1,{\mathcal{X}}_0)\in {\mathbb{X}}_n}{min}{D}_K({\mathcal{X}}_1,{\mathcal{X}}_0)\},$$

where ${\mathbb{X}}_n=\{({\mathcal{X}}_1,{\mathcal{X}}_0)\mid {\mathcal{X}}_1\subset \mathcal{X},1\le |{\mathcal{X}}_1|\le n-1,{\mathcal{X}}_0=\mathcal{X}\setminus {\mathcal{X}}_1\}$. The unknown minimum value of the discrepancy in this re-randomization criterion can be estimated via Monte Carlo sampling. Specifically, we can randomly sample from ${\mathbb{T}}_{n,n/2}$ a feasible number of times and take the minimum discrepancy among those samples as its estimate.

To leverage the strengths of randomized-based inference, we observe that the restricted set ${\varphi }_{mode}\left(\mathcal{T}\right)$ typically comprises either singletons or sets of small cardinality. Therefore, we relax the re-randomization criterion ${\varphi }_{mode}\left(\mathcal{T}\right)$ to the following:

$${\varphi }_{mode,\alpha }\left(\mathcal{T}\right)=\mathbb{I}\{\mathcal{T}\in {\mathbb{T}}_{n,n/2},D_K({\mathcal{X}}_1,{\mathcal{X}}_0)\le {\widehat{q}}_{D_K,\alpha }\},$$

(15)

where ${\widehat{q}}_{D_K,\alpha }$ is the empirical $\alpha$-quantile of the distribution of all discrepancy values over ${\mathbb{X}}_n$. It is evident that the above re-randomization, ${\varphi }_{mode,\alpha }\left(\mathcal{T}\right)$, degenerates into the MODE as $\alpha \to 0$ and into the CRE as $\alpha \to 1$. Thus, the re-randomized relaxation in (15) with $\alpha \in (0,1)$ effectively combines the randomness of the CRE with the efficiency of the MODE, making it useful when the experimenter is constrained to use randomization due to practical considerations.

5. Simulations

In this section, we compare the performance of several experimental approaches, including the balanced CRE, the balanced ReM utilizing the critical value $a={\chi }_{p,0.2}^2$ [1], the BODE with default priors as outlined by [2], and the proposed MODE employing the energy distance with the kernel function $K(x,y)=-{\left\{{\sum }_{j=1}^p{(x_j-y_j)}^2\right\}}^{1/2}$. Each experiment is repeated 1000 times across all settings. The simulation results indicate that the proposed MODE demonstrates reduced covariate imbalance and lower estimation uncertainty compared to existing methods.

5.1. Covariate Imbalance

We examine two sample size and dimensionality scenarios: (i) $n=1000$ and $p=5$; (ii) $n=2000$ and $p=10$. In each case, the covariates are generated as $X_{ij}\stackrel{i.i.d.}{\sim }U[-3,3]$ and $X_{ij}\stackrel{i.i.d.}{\sim }N(0,3)$ for $i\in \left[n\right],j\in \left[p\right]$. We then apply the four experimental designs to the covariate set $\mathcal{X}={\left\{{(X_{i1},\cdots ,X_{ij})}^T\right\}}_{i=1}^n$. The Mahalanobis distance $M({\mathcal{X}}_1,{\mathcal{X}}_0)$, as defined in (14), and the energy distance $D_K({\mathcal{X}}_1,{\mathcal{X}}_0)$, which represents the discrepancy function in (8) with the kernel function $K(x,y)=-{\left\{{\sum }_{j=1}^p{(x_j-y_j)}^2\right\}}^{1/2}$, are employed as two measures of covariate imbalance.

Figure 2 illustrates the covariate imbalances across the various experimental methods. The CRE method balances covariates on average; however, many CRE instances exhibit substantial covariate imbalance. In contrast, the ReM method maintains both the Mahalanobis distance and energy distance of covariates between treatment and control groups within a smaller range. The BODE further enhances covariate balance through Monte Carlo optimization. Notably, the proposed MODE demonstrates the smallest covariate imbalance across different settings. These findings suggest that the experimental units in the treatment and control groups based on MODE are more homogeneous, resulting in more comparable experimental outcomes.

5.2. Estimation Precision

To evaluate the precision of difference-in-means estimators derived from various experiments, we assume that the responses $Y_1,\cdots ,Y_n$ are generated from the following potential outcomes model:

$$\left\{\begin{array}{cc}\hfill & Y_i=T_i{Y}_i\left(1\right)+(1-T_i)Y_i\left(0\right),i\in \left[n\right];\hfill \\ \hfill & Y_i\left(1\right)=f_1\left(X_i\right)+{\sigma }_1\left(X_i\right)u_i,i\in \left[n\right];\hfill \\ \hfill & Y_i\left(0\right)=f_0\left(X_i\right)+{\sigma }_0\left(X_i\right)v_i,i\in \left[n\right];\hfill \\ \hfill & \{X_{ij},i\in \left[n\right],j\in \left[p\right]\}\stackrel{i.i.d.}{\sim }U[-3,3];\hfill \\ \hfill & \{u_i,i\in \left[n\right]\}\stackrel{i.i.d.}{\sim }N(0,1),\{v_i,i\in \left[n\right]\}\stackrel{i.i.d.}{\sim }N(0,1);\hfill \end{array}\right.$$

where $T_i=1$ if and only if the i-th unit is treated in the corresponding experiment. We consider four different specifications for mean and variance functions:

C1.

$f_1\left(x\right)=e^{\tilde{x}},f_0\left(x\right)=1+\tilde{x},{\sigma }_1^2\left(x\right)\equiv 0,{\sigma }_0^2\left(x\right)\equiv 0$;

C2.

$f_1\left(x\right)=e^{\tilde{x}},f_0\left(x\right)=1+\tilde{x},{\sigma }_1^2\left(x\right)\equiv 1,{\sigma }_0^2\left(x\right)\equiv 1$;

C3.

$f_1\left(x\right)=e^{\tilde{x}}+e^{\tilde{x}/2},f_0\left(x\right)=e^{\tilde{x}}-e^{\tilde{x}/2},{\sigma }_1^2\left(x\right)\equiv 0,{\sigma }_0^2\left(x\right)\equiv 0$;

C4.

$f_1\left(x\right)=e^{\tilde{x}}+e^{\tilde{x}/2},f_0\left(x\right)=e^{\tilde{x}}-e^{\tilde{x}/2},{\sigma }_1^2\left(x\right)\equiv 1,{\sigma }_0^2\left(x\right)\equiv 1$,

where $\tilde{x}={\sum }_{j=1}^p{x}_j/\sqrt{p}$. In C1 and C3, we assume that the potential outcomes are fixed.

Table 1. Empirical MSEs ($\times {10}^4$) for difference-in-means estimators based on 1000 replications across various experiments. The columns labeled PR indicate the percentage reduction in MSE of the MODE compared to the CRE. The rows labeled C1–C4 represent the four specifications of the potential outcomes model.

	CRE	ReM	BODE	MODE	PR	CRE	ReM	BODE	MODE	PR
	$p=5,n=250$					$p=5,n=1000$
C1	620	254	124	57	91%	129	62	36	8	94%
C2	623	347	211	142	77%	148	77	55	30	80%
C3	1212	652	389	149	88%	272	165	101	18	93%
C4	1496	771	451	235	84%	301	171	119	38	87%
	$p=10,n=500$					$p=10,n=2000$
C1	508	311	273	70	86%	118	68	70	11	91%
C2	539	374	312	112	79%	133	88	76	22	83%
C3	1374	1087	904	226	84%	321	236	213	34	90%
C4	1466	1117	916	265	82%	306	253	217	44	86%

presents the empirical MSEs of different-in-means estimators across the various experiments. As the sample sizes increase, the empirical MSEs decrease, as expected; however, the percentage reductions in MSE of the MODE compared to the CRE become more pronounced, indicating that the MSEs associated with the MODE converge more rapidly. Under each configuration of dimension and sample size, the ReM reduces the randomization variance of the CRE by constraining the Mahalanobis distances of the CREs. The BODE generally yields a smaller MSE than the ReM. Notably, the proposed MODE achieves the smallest MSE across various scenarios. Furthermore, the percentage reduction in MSE of the MODE over the CRE is particularly significant when the potential outcomes are fixed (C1 and C3), consistent with the optimality outlined in Corollary 2.

We also compare the differences between the true distribution of potential outcomes and the empirical distributions of the observed outcomes derived from various experiments.

Table 2. Average Kolmogorov–Smirnov values ($\times {10}^2$) for potential and observed outcomes from the treatment group, based on 1000 replications of various experiments. The rows labeled C1–C4 correspond to the four specifications of the potential outcomes model.

	CRE	ReM	BODE	MODE	CRE	ReM	BODE	MODE
	$p=5,n=250$				$p=5,n=1000$
C1	5.07	4.47	4.02	3.30	2.57	2.39	2.03	1.56
C2	5.29	5.00	4.68	4.41	2.64	2.49	2.40	2.38
C3	5.45	4.62	3.92	3.14	2.61	2.35	2.12	1.61
C4	5.21	4.67	4.46	4.30	2.70	2.47	2.34	2.21
	$p=10,n=500$				$p=10,n=2000$
C1	3.74	3.44	3.19	2.93	1.92	1.70	1.58	1.34
C2	3.80	3.65	3.39	3.47	1.93	1.72	1.73	1.69
C3	3.68	3.38	3.21	2.88	2.00	1.67	1.55	1.37
C4	3.68	3.62	3.38	3.22	1.88	1.83	1.73	1.61

presents the Kolmogorov–Smirnov values ${sup}_{y\in \mathbb{R}}|F\left(y\right)-F_n\left(y\right)|$ for the different experiments, where $F\left(y\right)$ and $F_n\left(y\right)$ represent the empirical distribution functions of the potential and observed outcomes from the treatment group, respectively.

As anticipated, all Kolmogorov–Smirnov values decrease, indicating that the observed outcomes converge to the true potential outcomes in distribution as the sample sizes increase. Across all settings, the MODE yields the lowest Kolmogorov–Smirnov values, suggesting that the observed outcomes are closer to the true potential outcomes in distribution. This property of matching distributions, which benefits from quasi-Monte Carlo techniques, also enhances the estimation of treatment effects beyond the average treatment effect.

6. Conclusions and Discussion

This paper introduces a novel measure of covariate imbalance, termed the generalized discrepancy, and proposes a MODE designed to minimize this discrepancy. Both theoretical analysis and simulations demonstrate that the proposed MODE outperforms existing methods, such as CRE and ReM, in terms of minimizing MSE. Thus, the MODE serves as a super-effective framework for controlled experimentation in the presence of covariates.

Importantly, the covariate imbalance, defined as the generalized discrepancy, quantifies the differences in covariate distributions between treatment and control groups. This measure can also be utilized in designing experiments aimed at improving the accuracy of estimating various causal effects, such as quantile treatment effects and average treatment effects on the treated group. Such applications enable researchers to evaluate the impacts of interventions more comprehensively.

For most control experiments where estimation accuracy is the primary objective, such as online A/B tests, the proposed MODE is the preferred method. Since MODE is deterministic, randomization inference based on the MODE is limited [22]; however, asymptotic inference remains feasible. In situations where exact inference is the primary goal or randomization is necessary due to practical constraints—such as policy requirements, ethical considerations, or the presence of potential unobserved confounding variables—we recommend using re-randomization based on the MODE described in (15). This approach randomizes a set of treatments that are near the minima of the maximum risk, thus enhancing covariate balance.

The MODE can be directly extended to multi-level experiments by sequentially minimizing generalized discrepancies across treatment groups. Thus, it offers a potential alternative to the OSAT tool [23] for sample-to-batch allocation in genomics experiments. The MODE proposed in this paper assumes that all covariates are collected in advance of conducting the experiment, including scenarios such as online A/B tests, classrooms with student data, or companies with employee records. Another potential future research direction involves adaptive allocation, particularly in biomedical experiments with human subjects, where treatments must be assigned either individually or in batches as participants are enrolled.