Improved Branch-and-Bound Antenna Selection Algorithm for Massive MIMO

Abstract

The rapid proliferation of wireless devices and the escalating demand for ultra-reliable, high-capacity communication networks have propelled massive multiple-input multiple-output systems as a cornerstone technology for next-generation wireless standards. Massive multiple-input multiple-output systems deploy hundreds of antennas at both the transmitter and the receiver, leading to high computational complexity in many antenna selection algorithms. Existing approaches often achieve reduced complexity at the expense of partial performance compromise. To address this challenge, this paper proposes an Improved Branch-and-Bound Antenna Selection algorithm that reduces complexity while maintaining the required performance. The algorithm iteratively eliminates the antenna contributing least to channel capacity from the candidate set. Through the mechanism of reverse-stacking nodes, the conventional stack-based search process is modified. Most critically, by employing dynamic stack management and effective pruning conditions, substantial pruning operations can be implemented during subsequent search procedures, significantly accelerating the identification of the optimal antenna subset. Simulation results demonstrate that the improved algorithm reduces computational complexity from an order of 10³ to 10² while maintaining equivalent channel capacity. Furthermore, through a single execution, the algorithm can obtain optimal antenna subsets with varying sizes within specified ranges, effectively overcoming the limitation of the traditional Branch-and-Bound algorithm that requires repeated executions for different subset dimensions.

1. Introduction

With the rapid evolution of fifth-generation (5G) and the emergence of sixth-generation (6G) mobile communication technologies, users now expect greater network stability, better signal quality, and higher data speeds. Massive multiple-input multiple-output (multiple-input multiple-output, MIMO) systems have become a key technology for boosting wireless channel capacity and spectral efficiency [1,2,3,4,5,6,7], but their real-world implementation comes with significant challenges. Traditional MIMO systems require a dedicated radio frequency (RF) chain for each antenna, with each chain consisting of multiple components like digital-to-analog converters and power amplifiers. As the number of antennas grows, the system becomes increasingly complex due to three main factors: higher hardware costs, an explosion in RF chain requirements, and difficulties in acquiring accurate channel state information (CSI) [8,9,10,11]. Antenna selection techniques help address these challenges by dynamically choosing which antennas to activate, reducing the number of required RF chains without sacrificing performance. Larsson et al. [5] highlighted that an effective antenna selection strategy should incorporate low-power components and leverage channel reciprocity to simplify CSI estimation. Recent advancements have led to a variety of algorithmic approaches, each optimized for different applications and offering unique advantages.

Convex optimization has been widely employed in antenna selection algorithms. Gao et al. [12] reformulated antenna selection as a convex optimization problem by relaxing binary constraints to maximize downlink capacity. While this approach improves performance in high-interference scenarios, it suffers from high computational complexity and a strong reliance on precise CSI. In contrast, Li et al. [13] proposed a convex optimization algorithm that prioritizes maximizing total received power, integrating signal-to-leakage-and-noise ratio (SLNR) precoding to optimize the bit error rate (BER). However, this method overlooks channel correlation, limiting its effectiveness in environments with severe user interference. Subsequently, Mendonça et al. [14] introduced greedy algorithms based on a matching pursuit (MP) and its quantized variant (MPGBP), which iteratively select antenna subsets to maximize channel correlation while minimizing the mean square error (MSE) and transmission power. Their symbol-level algorithm jointly optimizes antenna selection and precoding, eliminating matrix inversion operations to reduce computational overheads. The MPGBP further alleviates amplifier linearity requirements through quantization. However, greedy algorithms inherently risk converging to local optima, leading to suboptimal solutions. Additionally, symbol-level strategies degrade in performance under rapidly varying channels, while quantization may introduce residual errors. As a result, these algorithms perform well in static channel conditions but require adaptive enhancements for dynamic environments, making them particularly suitable for latency-tolerant applications.

Chen et al. [15] developed an intelligent antenna selection algorithm that integrates a Monte Carlo tree search (MCTS) with self-supervised learning. This method leverages MCTS’s heuristic search to balance exploration and exploitation, while a linear regression model extracts channel norm and correlation features from the CSI, generating prior probabilities to accelerate search convergence. Their work introduces two key innovations. First, it pioneers the integration of the MCTS and self-supervised learning for antenna selection, enabling closed-loop optimization through online training sample generation. Second, it proposes a computational efficiency framework that combines incremental channel capacity calculation with ternary correlation approximation. Despite its advantages, this approach has three key limitations. First, the linear regression model has limited representational capacity, potentially struggling to extract discriminative channel features in complex propagation conditions. Second, the data migration and parallelization mechanisms introduce significant hardware implementation overheads. Finally, its scalability remains unverified for ultra-large-scale antenna arrays (more than 64 transmit or receive antennas), particularly concerning computational resource allocation and real-time decision latency.

In recent years, machine learning and deep learning have been increasingly applied to antenna selection algorithm design. Bouchibane et al. [16] proposed a deep learning-based antenna selection algorithm leveraging Convolutional Neural Networks (CNNs). This approach formulates antenna selection as a classification problem, using CNNs to extract features from the channel matrix and directly predict the optimal antenna subset. The CNN model is trained offline on a large dataset of channel samples and exhaustive search results, enabling it to learn the mapping between the CSI and optimal subsets. This method marks the first application of CNNs in large-scale antenna selection, significantly reducing real-time computational overheads through an “offline training–online inference” paradigm. Despite its potential, this approach faces several challenges. First, label generation relies on an exhaustive offline search, making data generation and model training increasingly complex as the number of antennas grows. Additionally, the CNN model considers only channel amplitude, neglecting phase information, which may lead to the loss of important channel features. Finally, its generalization ability remains limited, with performance in non-uniform arrays or correlated channels requiring further validation.

A Branch and Bound (BAB) search, a method capable of effectively reducing the search space, was first proposed by Narendra and Fukunaga in 1977 [17]. In 2015, Gao et al. [18] first proposed the application of the BAB search for antenna selection, which selects the antenna subset that maximizes the minimum singular value (MSV) of the channel submatrix, thereby maximizing the post-processing signal-to-noise ratio (SNR) and reducing the BER. In 2018, Gao et al. [19] reapplied the BAB algorithm for antenna selection, with the notable distinction that their methodology adopted a channel capacity maximization criterion. Commencing from an empty set, the researchers iteratively incorporated the antenna providing the maximum incremental channel capacity gain into the optimal subset. The existing literature demonstrates that in the application of Branch and Bound (BAB) algorithms to antenna selection, researchers have systematically reduced the computational complexity by implementing three pivotal mechanisms: (1) The establishment of monotonic criteria to ensure progressive optimization, (2) the development of pruning strategies to eliminate non-viable search branches, and (3) the dynamic updating of lower bounds throughout the search process to maintain solution quality. Gaikwad et al. [20] similarly demonstrated the superiority of the BAB algorithm in complexity reduction for antenna selection, though it should be noted that the computational complexity of the BAB-based antenna selection algorithm remains dependent on the initial ordering strategy, exhibiting exponential growth with an increasing number of antennas in worst-case scenarios.

Building on the research on antenna selection algorithms, we observe that different selection strategies significantly impact the performance of large-scale MIMO systems. While each of the aforementioned algorithms optimizes specific performance metrics, they all have inherent limitations and fail to comprehensively enhance the MIMO system performance. Therefore, a more robust and versatile antenna selection algorithm is needed to achieve optimization without compromising other key performance aspects. This paper proposes the Improved Branch-and-Bound (I-BAB) algorithm for antenna selection. By introducing a monotonicity criterion and constructing an optimized search tree, the algorithm iteratively eliminates antennas that contribute the least to performance gain from the candidate set of receiving antennas, ultimately identifying the optimal antenna subset. Simulation results demonstrate that the I-BAB algorithm significantly reduces computational complexity while maintaining channel capacity performance comparable to the traditional Branch-and-Bound (BAB) algorithm. Notably, I-BAB’s unique ability to generate optimal subsets for all possible antenna subset sizes within a user-defined range eliminates the traditional requirement for predetermined subset sizes, greatly enhancing operational flexibility.

In future research on antenna selection algorithms, innovative approaches incorporating cutting-edge technologies such as dynamic optimization, energy complementarity, and machine learning could be explored for algorithm advancement [21]. Building upon the mechanism of dynamic polarization state regulation through polarization controller angle modulation in fiber cavities demonstrated in [21], future investigations may try to develop similar polarization-adaptive control strategies to address nonlinear channel environments. Notably, while employing high-gain antennas to enhance channel capacity during antenna selection inevitably increases power consumption, the stable generation of high-energy noise pulses achieved in [21] provides crucial energy management references for the design of high-power consumption antenna systems.

Wan et al. proposed a machine learning model based on a Support Vector Machine (SVM) [22]. The SVM achieves classification by identifying an optimal hyperplane to separate samples from different classes while maximizing the inter-class margin. For nonlinear problems, the SVM employs kernel functions (e.g., a polynomial kernel) to map data into a high-dimensional space for linear separation, enabling high-accuracy classification. This model demonstrates strong noise resistance through filtering and dimensionality reduction operations to effectively process noise. It provides a valuable reference framework for future antenna selection algorithms employing machine learning techniques. However, the dimensionality reduction operations incur high computational resource demands, and the model requires a substantial number of labeled samples for training.

The remainder of the paper is organized as follows. Section 2 presents the most representative antenna selection algorithms and introduces their search mechanisms and operational processes in detail. Section 3 describes our enhanced Branch-and-Bound (I-BAB) algorithm, providing its system model, mathematical framework, and new search strategy. In Section 4, a comprehensive numerical simulation comparing I-BAB with existing algorithms is conducted. Finally, Section 5 concludes the paper.

2. Representative Methods in Antenna Selection

Consider a MIMO system where the base station (BS) is equipped with N_t transmit antennas and the user terminal has N_r receive antennas. To optimize reception, a subset of M_r (M_r < N_r) antennas is selected from the N_r available antennas. The channel model is defined as:

Y = Hx + n,

(1)

where $\mathit{Y}\in {ℂ}^{N_r\times 1}$ is the received signal vector; $\mathbf{H}\in {ℂ}^{N_r\times N_t}$ is the channel matrix; $\mathit{x}\in {ℂ}^{N_t\times 1}$ is the transmitted signal vector; n is the additive complex Gaussian noise vector following $\mathit{n}\in \mathrm{C}\mathrm{N} (0, {\mathbf{I}}_{N_r})$.

Antenna selection algorithms aim to choose M_r antennas from the available N_r antennas at the receive side to optimize system performance. In ideal independent and identically distributed (i.i.d.) Rayleigh fading channels, all antennas contribute approximately equally to system performance, resulting in limited antenna selection gains [23]. However, in practical propagation environments, significant asymmetry in channel gains arises due to factors such as large-scale fading, spatial correlation, and variations in antenna directivity [24,25]. These channel variations have motivated the development of various antenna selection algorithms, which can be classified based on their underlying search methodologies.

2.1. Exhaustive Search Algorithm

The core principle of the exhaustive search algorithm [26] lies in its global exploration of all potential antenna combinations, subsequently selecting the optimal subset based on specific performance metrics, such as channel capacity, signal-to-noise ratio (SNR), or Frobenius norm. The standard search procedure follows these steps:

• Enumerate all possible antenna combinations: Calculate all feasible antenna subsets using the combinatorial formula defined as:

  $$\mathrm{C}(N_r, M_r) =\left({\displaystyle \genfrac{}{}{0pt}{}{N_r}{M_r}}\right)={\displaystyle \frac{N_r!}{M_r!\left(N_r-M_r\right)!}},$$

  (2)
• Produce the sub-channel matrixes: Select the corresponding columns from the full channel matrix H to form the submatrix ${\mathbf{H}}_s$, ${\mathbf{H}}_s\in {ℂ}^{M_r\times { N}_t}$.
• Calculate system performance indicators: For each submatrix, calculate the corresponding performance indicators.
• Select the optimal antenna combination: Find the antenna combination S_opt that makes the performance optimal.

An exhaustive search algorithm can guarantee the identification of the optimal receive antenna combination, thereby achieving peak system performance. However, its critical limitation lies in the exponential growth of subset cardinality as N_r and M_r increase, which rapidly escalates the computational complexity. This inherent drawback renders it practically unfeasible for implementation in large-scale MIMO systems. Consequently, it necessitates the investigation of suboptimal algorithms characterized by reduced computational overheads while maintaining near-optimal performance, aiming to establish an optimal trade-off between algorithmic efficacy and computational tractability.

2.2. Greedy Selection Algorithm

The greedy algorithm is a heuristic algorithm commonly employed for solving combinatorial optimization problems [27]. In antenna selection problems, the fundamental principle of the greedy algorithm involves iteratively selecting the currently optimal antenna to maximize specific system performance metrics. The typical implementation steps of the greedy algorithm are as follows:

• Initialization: Set the selected antenna subset S = ∅.
• Iterative selection: At each step, iterate over all remaining unselected antennas k ∈ {1, 2, …, N_r}∖S, evaluate the performance gain after incorporating antenna k; select the antenna k* that maximizes the contribution to a specific system performance metric (e.g., channel capacity), then update S = S ∪ {k*}. Repeat Step 2.
• Termination: When |S| = M_r, cease selection and output the current antenna subset S as S_opt.

Compared with the exhaustive search algorithm, the greedy algorithm significantly reduces computational complexity and demonstrates strong applicability in large-scale MIMO systems, enabling rapid generation of approximate optimal solutions under scenarios with limited computational resources or time-constrained conditions. However, its stepwise decision-making mechanism that iteratively selects the locally optimal choice may lead to sub-optimal global solutions. Furthermore, the algorithm’s optimality guarantee relies on idealized channel condition assumptions. When confronted with practical channel impairments such as spatial correlation, hardware imperfections, and dynamic interference, the greedy approach may fail to achieve optimal performance due to its myopic optimization strategy and sensitivity to channel model inaccuracies.

2.3. Branch and Bound Algorithm

The Branch and Bound algorithm is a systematic search methodology designed for solving combinatorial optimization problems [17,18,19,20]. Its core principle lies in constructing a solution space tree for the problem, iteratively decomposing the problem into sub-problems (branching), and employing upper or lower bound computations to prune non-optimal branches (bounding). This dual mechanism effectively narrows the search space, thereby circumventing exhaustive enumeration of all possible solutions. The algorithm ultimately identifies the optimal antenna subset through this process. The search procedure is implemented as follows (with channel capacity maximization as the selection criterion):

• Initialization: Build the root node of the solution space tree, set the lower capacity bound of the current optimal subset B = −∞, and the initial antenna subset S = ∅.
• Branching: Generate child nodes from the current node, each corresponding to adding a new receiving antenna k_i to the current antenna combination, S_i = S_i ∪ {k_i}.
• Bounding: For each child node S_i, its corresponding channel capacity C_i is calculated.
• Pruning: If the C_i of a child node is less than the B of the current optimal subset, pruning the child node avoids further searching.
• Update optimal result: If the number of antenna subsets corresponding to a child node |S_i| = M_r and its channel capacity C_i > B, update S_opt = S_i, B = C_i.
• Iteration: Repeat steps 2 to 4 until the search space is fully explored or the termination condition is met.

The Branch and Bound algorithm guarantees the identification of a globally optimal receiver antenna combination, thereby optimizing system performance. By employing pruning operations, this algorithm significantly reduces the number of antenna combinations requiring evaluation, demonstrating superior computational efficiency compared to exhaustive search approaches. However, the implementation of the Branch and Bound algorithm involves relative complexity, necessitating precise design of branching and bounding strategies along with effective methods for calculating upper and lower bounds. Although the pruning mechanism reduces computational complexity, the algorithm may still incur substantial computational costs when handling large-scale antenna configurations.

2.4. Random Search Algorithm

This algorithm is an optimization method based on randomized strategies [28]. By randomly sampling in the solution space, the performance index of sampling points is evaluated; we then find the approximate optimal solution. In the antenna selection problem, the random search algorithm selects a subset of receiving antennas at random, evaluates its impact on system performance (such as channel capacity), and finally selects the optimal antenna subset. The search process is as follows:

• Initialization: Set the maximum number of iterations N_iter and the current optimal channel capacity C_opt = 0.
• Iterative process:

  (1)

  Randomly select antenna subset: Randomly select M_r antennas among N_r antenna receiving antennas to form antenna subset S.

  (2)

  Construct sub-matrix: According to the selected antenna subset S, extract the corresponding sub-matrix ${\mathbf{H}}_s$ from the channel matrix H.

  (3)

  Calculate channel capacity: Calculate channel capacity C_s corresponding to the current antenna subset.

  (4)

  Update best solution: if C_s > C_opt, update C_opt = C_s and S_opt = S.

• Termination conditions: When the maximum number of iterations N_iter is reached, the search process is terminated; then, the optimal antenna subset S_opt and corresponding channel capacity C_opt are the output.

The random search algorithm exhibits straightforward implementation that circumvents the need for intricate mathematical derivations, demonstrating applicability to antenna selection problems across diverse scales while maintaining inherent parallelization compatibility. However, due to its stochastic sampling mechanism, the algorithm may require extensive iterations to attain near-optimal solutions. The algorithmic performance proves critically dependent on the sampling quality, potentially resulting in solution quality instability. Furthermore, when applied to large-scale antenna configurations, the method may incur substantial computational resource demands due to its exhaustive search characteristics.

Existing antenna selection algorithms face a trade-off between performance and computational complexity. While an exhaustive search guarantees global optimality, its combinatorial complexity grows exponentially with the number of antennas, rendering it impractical for a large-scale MIMO system [26]. Greedy algorithms reduce the computational load by iteratively selecting locally optimal antennas, yet they are prone to sub-optimal solutions with approximately 15% capacity loss [27]. The branch-and-bound (BAB) method approximates theoretical upper bounds through pruning strategies; the algorithm complexity is relatively low, but the implementation is complicated. Random search algorithms require substantial iterations to approach optimal solutions, while exhibiting unstable solution quality. Recent advancements in artificial intelligence (AI) have introduced novel approaches to antenna selection, such as deep learning-based classification methods [29,30,31,32,33,34]. However, these techniques remain constrained by their dependence on extensive labeled datasets and limited model generalizability.

3. Efficient Antenna Selection via Enhanced Branch-and-Bound Optimization

This paper considers a large-scale MIMO system equipped with N_t transmit antennas and N_r receive antennas, which can be conceptualized as a communication system comprising a base station with N_r receive antennas and N_t users, as illustrated in Figure 1. The study focuses on receive-side antenna selection. It is assumed that the channel undergoes independent and identically distributed (i.i.d.) small-scale Rayleigh flat fading, with large-scale fading effects such as path loss being eliminated. Channel State Information (CSI) is exclusively available at the receiver. The transmitted signal components follow independent Gaussian distributions with equal power allocation. The received signal is defined as:

$$\mathbf{y}=\sqrt{\rho }\mathbf{H}\mathit{x}+\mathit{\mu },$$

(3)

where $\mathbf{H}\in {ℂ}^{N_r\times N_t}$ is a complex Gaussian random variable whose mean is 0 and variance is 1; $\mathit{x}\in {ℂ}^{N_t\times 1}$ is a transmitted signal vector; μ is an additive complex Gaussian noise vector following $\mathit{\mu }\in \mathrm{C}\mathrm{N}(0, {\mathbf{I}}_{N_r})$; ρ is a signal to noise ratio (SNR). In this paper, we consider N_t < N_r; the channel capacity in the MIMO system is defined as [35]:

$$C={\log }_2\det ({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}^{\mathrm{H}}\mathbf{H}),$$

(4)

where (·)^H denotes the Hermitian transpose. We investigate the antenna selection problem where d antennas are selected from an original array of N_r receive antennas (d < N_r). Let ${\mathbf{H}}_s\in {ℂ}^{d\times N_t}$ denote the resultant channel submatrix obtained by selecting the corresponding d rows from the original channel matrix H; then, the channel capacity of the antenna subset obtained after antenna selection is as follows:

$$C^{*}={\log }_2\det ({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_s^{\mathrm{H}}{\mathbf{H}}_s).$$

(5)

Therefore, antenna selection can be translated into the following optimization problems:

$${\mathbf{H}}_s^{\mathit{opt}}=\arg \underset{{\mathbf{H}}_s\in {\mathrm{A}}_s}{\max } {\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_s^H{\mathbf{H}}_s\right),$$

(6)

${\mathrm{A}}_s$ is the set of matrices corresponding to all possible antenna subsets.

3.1. Upper Bound of Channel Capacity of MIMO System

Given a channel matrix H with independent and identically distributed (i.i.d.) complex Gaussian entries, we initially assume each transmit antenna is received by a dedicated set of N_r receive antennas without interference from other signal components. As established in references [35,36,37], the upper bound on the capacity of fully digital MIMO systems can be formulated as:

$$C_{ub}={\sum }_{l=1}^{N_t} {\log }_2\left(1+{\displaystyle \frac{\rho }{N_t}}{\gamma }_l\right),$$

(7)

where the random variables ${{\{\gamma }_l\}}_l=1, 2, \cdots , N_t$, represent the chi-square distributed with 2$N_r$ degrees of freedom, corresponding to the transmit antennas $l=1, 2, \cdots , N_t$. Conversely, if we assume that each receiving antenna has its own N_t transmitting antennas, the upper bound on the capacity of a full-digital MIMO system can be given by Equation (8):

$${\tilde{C}}_{\mathit{ub}}={\sum }_{l=1}^{N_r} {\log }_2\left(1+{\displaystyle \frac{\rho }{N_t}}{\phi }_l\right),$$

(8)

where the random variables ${{\{\phi }_l\}}_l=1, 2, \cdots , N_r$, denote the chi-square distributed with 2$N_t$ degrees of freedom [35], corresponding to the receive antennas $l=1, 2, \cdots , N_r$. Based on the aforementioned equations, the upper bound of the system capacity after implementing antenna selection at the receiver end can be expressed as:

$$C_S={\sum }_{l=1}^d {\log }_2\left(1+{\displaystyle \frac{\rho }{N_t}}{\phi }_{\left(l\right)}\right),$$

(9)

where the random variables $\{{{\phi }_{\left(l\right)}\}}_l=1, 2, \cdots , N_r$, denote the ordered chi-square distributed with 2N_t degrees of freedom, i.e., ${\phi }_{\left(1\right)} {>\phi }_{\left(2\right)}>\dots > {\phi }_{\left(N_r\right)}>0$. Therefore, it can be concluded that the antenna selection method exclusively selects the top-d largest antennas. Furthermore, as demonstrated in [35], d < N_t constitutes a necessary condition for Equation (9) to establish an upper bound.

3.2. Calculation of Antenna Gain

Let ${\mathbf{H}}_L$ denote the sub-matrix formed by selecting L rows from matrix H through an L-step selection process, under which the channel capacity satisfies $C_L={\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_L^{\mathrm{H}}{\mathbf{H}}_L\right)$, $L=0, 1, 2, \cdots , d-1$, $C_0=0$. We select the k-th row of matrix H from the remaining candidate antennas and incorporate it into ${\mathbf{H}}_L$ in the (L + 1)th step; the k-th row of H is denoted by ${\mathbf{h}}_k^{\mathrm{H}}$, then the newly formed antenna subset can be expressed as ${\mathbf{H}}_{L+1}=\left[{\displaystyle \genfrac{}{}{0pt}{}{{\mathbf{H}}_L}{{\mathbf{h}}_k^{\mathbf{H}}}}\right]$. $C_{L+1}$ can be written as:

$$\begin{array}{ll}{C}_{L+1}& ={\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_{L+1}^{\mathrm{H}}{\mathbf{H}}_{L+1}\right)\\ & ={\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_L^{\mathrm{H}}{\mathbf{H}}_L+{\displaystyle \frac{\rho }{N_t}}{\mathbf{h}}_k{\mathbf{h}}_k^{\mathrm{H}}\right)\\ & =C_n+{\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_L^{\mathrm{H}}{\mathbf{H}}_L\right)}^{-1}{\mathbf{h}}_k{\mathbf{h}}_k^{\mathrm{H}}\right)\\ & \stackrel{\left(1\right)}{=} C_L+\underset{{\Delta }_{k, L}}{\underbrace{{\log }_2\left(1+{\displaystyle \frac{\rho }{N_t}}{\mathbf{h}}_k^{\mathrm{H}}{\mathbf{G}}_L{\mathbf{h}}_k\right)}},\end{array}$$

(10)

where step “(1)” establishes the Sylvester determinant identity, ${\mathbf{G}}_L={\left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_L^{\mathrm{H}}{\mathbf{H}}_L\right)}^{-1}$; it is evident that ${\mathbf{G}}_L$ is a positive definite. Therefore, when ${\mathbf{h}}_k\ne 0$, ${\Delta }_{k, L}>0$. Therefore, $C_L$ is an increasing function, thereby further substantiating the theory that channel capacity increases with the number of antennas.

The aforementioned greedy selection algorithm operates by evaluating the contribution of each candidate antenna to the channel capacity, specifically through calculating the increment ${\Delta }_{k, L}$ at each iteration. It selects the antenna with the maximum ${\Delta }_{k, L}$ value to incorporate into the antenna subset, thereby maximizing the channel capacity. Consequently, in antenna selection applications, the greedy algorithm is often employed as a method to obtain sub-optimal solutions. We now proceed to derive the specific mathematical expression for the increment ${\Delta }_{k, L}$ within this algorithmic framework. The Sherman–Morrison formula was applied to reformulate the ${\mathbf{G}}_L$ [38]:

$$\begin{array}{ll}{\mathbf{G}}_{L+1}& ={\left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_{L+1}^{\mathrm{H}}{\mathbf{H}}_{L+1}\right)}^{-1}\\ & ={\left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_L^{\mathrm{H}}{\mathbf{H}}_L+{\displaystyle \frac{\rho }{N_t}}{\mathbf{h}}_{K_L}{\mathbf{h}}_{K_L}^{\mathrm{H}}\right)}^{-1}\\ & ={\mathbf{G}}_L-{\mathbf{g}}_{L+1}{\mathbf{g}}_{L+1}^{\mathrm{H}}, L=0, 1, \cdots , d-1,\end{array}$$

(11)

where $K_L$ represents the index of the antenna selected at the Lth step in the given algorithm. ${\mathbf{g}}_{L+1}={\displaystyle \frac{{\mathbf{G}}_{L+1}{\mathbf{h}}_{K_L}}{\sqrt{{\left(\frac{\rho }{N_t}\right)}^{-1}+{\mathbf{h}}_{K_L}^{\mathrm{H}}}{\mathbf{G}}_L{\mathbf{h}}_{K_L}}}$. Define ${\alpha }_{k, L+1}={\mathbf{h}}_k^{\mathrm{H}}{\mathbf{G}}_{L+1}{\mathbf{h}}_k$, which can be further transformed as:

$$\begin{array}{ll}{\alpha }_{k, L+1}& ={\mathbf{h}}_k^{\mathrm{H}}{\mathbf{G}}_{L+1}{\mathbf{h}}_k\\ & ={\mathbf{h}}_k^{\mathrm{H}}\left({\mathbf{G}}_L-{\mathbf{g}}_{L+1}{\mathbf{g}}_{L+1}^{\mathrm{H}}\right){\mathbf{h}}_k\\ & ={\alpha }_{k, L}-{\left|{\delta }_{K_L, k, L+1}\right|}^2,\end{array}$$

(12)

where ${\delta }_{K_L, k, L+1}={\mathbf{h}}_k^{\mathrm{H}}{\mathbf{g}}_{L+1}$, thus, ${\Delta }_{k, L+1}$ can be expressed as:

$${\Delta }_{k, L+1}={\log }_2\left(1+{\displaystyle \frac{\rho }{N_t}}{\alpha }_{k, L+1}\right).$$

(13)

Based on the derivations presented above, the antenna index selected at the L-th step can be determined by the following expression:

$$K_{L+1}=\arg {\max }_{k\in {{\rm Y}}_{L+1}} {\alpha }_{k, L+1}$$

(14)

where ${{\rm Y}}_{L+1}$ denotes the candidate antenna set at step L + 1, ${{\rm Y}}_{L+1}={{\rm Y}}_L\setminus K_L$. The symbol “\” represents the set difference operation, which removes antenna $K_L$ from set ${{\rm Y}}_L$.

3.3. Branch-and-Bound Antenna Selection

The BAB algorithm is an efficient method for search space reduction. Compared to exhaustive search methods, its key mechanism for significantly reducing computational complexity lies in establishing well-designed pruning criteria. By eliminating child nodes that fail to satisfy search constraints through these pruning conditions, the algorithm effectively enhances search efficiency.

When utilizing the BAB algorithm to maximize channel capacity through receive-side antenna selection, the optimal subset $S_{\mathit{opt}}$ is initialized as an empty set, lower bound $B=-\infty$, while the initial candidate antenna set $K=\{1, 2, \cdots , N_r\}$. Each row in the current candidate antenna set is evaluated for its channel gain magnitude ${\Delta }_k$. These gains are sorted in descending order, following which the row corresponding to the maximum gain value in the channel matrix is selected, analogous to the greedy algorithm. This selected row, combined with N_t transmit antennas, forms a sub-channel matrix H_s. The corresponding channel capacity C_s of this sub-matrix is then calculated. If the value of C_s is less than B, prune the corresponding antenna element. If the value of C_s outperforms B, add this row to subset $S_{\mathit{opt}}$ and update the candidate set to $K=\{1, 2, \cdots , N_r\} \backslash S_{\mathit{opt}}$. This iterative process continues until the size of subset $S_{\mathit{opt}}$ equals the threshold d, at which point the search terminates and B is updated with the current C_s value. The complete BAB algorithm workflow is formally summarized in Algorithm 1.

It is evident that when applying the BAB search algorithm to antenna selection, the process incrementally incorporates antennas with higher gain into the optimal antenna subset starting from an empty set. Specifically, in our constructed search tree, the tree depth corresponds to the predefined size of the optimal antenna subset. The search initiates from the root node and proceeds upward layer by layer. At each hierarchical level, the optimal node is selected and added to subset $S_{\mathit{opt}}$ until the search reaches the leaf nodes.

However, certain limitations in the BAB search procedure remain non-negligible: (1) Since the channel capacity value corresponding to the first generated child node cannot be known a priori, initializing the lower bound $B$ to negative infinity is theoretically justified. However, the update of $B$ only occurs upon reaching leaf nodes, which substantially weakens the pruning constraints imposed by $B$ throughout the search process. This may result in unnecessary exploration of non-optimal nodes. (2) The parameter d requires a predefined configuration. Moreover, the algorithm must be executed repeatedly for different d values to obtain their corresponding optimal antenna subsets.

Algorithm 1: Antenna selection based on BAB search method

$1: \mathrm{Input}: N_r$$, N_t$$, d$.

$2: \mathrm{Initialization}: \mathrm{Set} \mathrm{the} \mathrm{selected} \mathrm{antenna} \mathrm{subset} S=\varnothing$$, S_{\mathit{opt}}=\varnothing$$; \mathrm{candidate} \mathrm{set} K=\left\{1, 2, \dots , N_r\right\}$$; B=-\infty$.

$3: \mathbf{for} i$$=1 \mathrm{to} \left|K\right|$ do

$\mathrm{Calculate} {\Delta }_{k, i}$$, k\in K$

end for

$\mathrm{Sort} {\Delta }_{k, i}$$\mathrm{in} \mathrm{descending} \mathrm{order} \mathrm{to} \mathrm{get} \mathrm{an} \mathrm{ordered} \mathrm{index} \mathrm{vector} \mathit{J}$.

$\mathbf{for} i=1$$\mathrm{to} \left|\mathit{J}\right|$

$j={ \left[\mathit{J}\right]}_{i }$

$\mathrm{Create} \mathrm{child} \mathrm{node} S=S\cup j$.

$C_s={\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_s^{\mathrm{H}}{\mathbf{H}}_s\right)$$; {\mathbf{H}}_s$$\mathrm{represents} \mathrm{the} \mathrm{channel} \mathrm{submatrix} \mathrm{associated} \mathrm{with} \mathrm{child} \mathrm{node} S$.

$\mathbf{if} C_s< \mathrm{B}$, then

Prune the child node to avoid further searching and break the loop.

else

$\mathbf{if} C_s>$ B, then

$\mathbf{if} \left|S\right|=d$, then

$\mathrm{Update} S_{\mathit{opt}}=S$$, C_{\mathit{opt}}=C_s$$, B=C_s$, break.

else

$K= \{1, 2, \dots , N_r\}\setminus S$, go back to step 3.

end if

end if

end if

end for

$4: \mathrm{Output} S_{\mathit{opt}}$$\mathrm{and} C_{\mathit{opt}}$.

Therefore, we raise two critical questions regarding the existing limitations of the BAB algorithm: (1) Given that the BAB algorithm still searches for certain unnecessary nodes, can we develop a mechanism to prune more redundant nodes effectively? (2) When parameter d extends from a single value to multiple values within a range, is it feasible to obtain optimal subsets corresponding to different d values through a single algorithm execution?

3.4. Improved Branch-and-Bound Algorithm

In response to the aforementioned two issues, we propose an Improved BAB algorithm (I-BAB). Although the antenna channel capacity increases with the quantity of antennas, practical constraints including hardware costs and computational space limitations necessitate the selection of an optimal subset of antennas to satisfy actual communication requirements. As proposed in [21], the essence of antenna selection lies in constructing an appropriate search tree and executing a tree traversal to identify suitable tree nodes. Through the derivation presented in Section 3.2, we establish a monotonic criterion: if antenna subset $H_1\subseteq H_2$, the channel capacity $C_1\le C_2$. The monotonicity remains valid under non-ideal practical conditions, with the rigorous proof provided in Appendix A.

The I-BAB algorithm improves the search process by initiating from the complete set of candidate antennas and iteratively generating child nodes through the progressive elimination of antennas exhibiting minimal contributions to channel capacity. As demonstrated in Section 3.2, we can analogously derive the channel capacity decrement value ∆ resulting from the removal of a specific antenna. Let ${\mathbf{H}}_l$ denote the matrix obtained by removing l rows from the channel matrix H through a l-step selection process, where the channel capacity $C_l={\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{{\mathbf{H}}_l}^{\mathrm{H}}{\mathbf{H}}_l\right)$, $l=1, 2, \cdots , N_r-d$, $C_0={\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}^{\mathrm{H}}\mathbf{H}\right)$. Assuming at the (l + 1)th selection step, we select and remove the $k_{l+1}$th row from ${\mathbf{H}}_l$; ${\mathbf{h}}_{k_{l+1}}$ denotes $k_{l+1}$th row in matrix ${\mathbf{H}}_l$, resulting in a new antenna subset ${\mathbf{H}}_{l+1}={\mathbf{H}}_l\backslash \left\{{\mathbf{h}}_{k_{l+1}}\right\}$. Consequently, the channel capacity $C_{l+1}$ is expressed as:

$$\begin{array}{ll}{C}_{l+1}& ={\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_{l+1}^{\mathrm{H}}{\mathbf{H}}_{l+1}\right)\\ & ={\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{{\mathbf{H}}_l}^{\mathrm{H}}{\mathbf{H}}_l-{\displaystyle \frac{\rho }{N_t}}{\mathbf{h}}_{k_{l+1}}{\mathbf{h}}_{k_{l+1}}^{\mathrm{H}}\right)\\ & =C_l-{\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{{\mathbf{H}}_l}^{\mathrm{H}}{\mathbf{H}}_l\right)}^{-1}{\mathbf{h}}_{k_{l+1}}{\mathbf{h}}_{k_{l+1}}^{\mathrm{H}}\right)\\ & \stackrel{\left(2\right)}{=} C_l-\underset{{\Delta }_{k_{l+1}, l}}{\underbrace{{\log }_2\left(1+{\displaystyle \frac{\rho }{N_t}}{\mathbf{h}}_{k_{l+1}}^{\mathrm{H}}{\mathbf{G}}_l{\mathbf{h}}_{k_{l+1}}\right)}},\end{array}$$

(15)

where step “(2)” establishes the Sylvester determinant identity, ${\mathbf{G}}_l={\left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_l^{\mathrm{H}}{\mathbf{H}}_l\right)}^{-1}$. Similarly, referring to Section 3.2, the loss reduction ${\Delta }_{k_{l+1}, l}$ by eliminating the $k_{l+1}$th antenna can be calculated as:

$${\Delta }_{k_{l+1}, l+1}={\log }_2\left(1+{\displaystyle \frac{\rho }{N_t}}{\alpha }_{k_{l+1}, l+1}\right),$$

(16)

where ${\alpha }_{k_{l+1}, l+1}={\mathbf{h}}_{k_{l+1}}^{\mathrm{H}}{\mathbf{G}}_{l+1}{\mathbf{h}}_{k_{l+1}}={\alpha }_{k_{l+1}, l}-{\left|{\mathbf{h}}_{k_{l+1}}^{\mathrm{H}}{\mathbf{g}}_{l+1}\right|}^2$. The antenna index selected at the $k_{l+1}$th step can be found by the following expression:

$$K_{l+1}=\arg {\min }_{k\in {{\rm Y}}_{l+1}} {\alpha }_{k_{l+1}, l+1},$$

(17)

where ${{\rm Y}}_{l+1}$ denotes the candidate antenna set at step l + 1,${{\rm Y}}_{l+1}={{\rm Y}}_l\setminus K_l$; $K_l$ serves as the index for the antenna selected for elimination at step l.

Certainly, the aforementioned derivations were conducted under the condition that the specific value of d, representing the optimal antenna subset size, is predetermined. The I-BAB algorithm proposed in this paper differs from the conventional BAB algorithm in that the size of the optimal antenna subset can be defined as a range rather than a specific value. Specifically, we assign MAX as the maximum threshold and MIN as the minimum threshold for the optimal antenna subset size |$S_{\mathit{opt}}$|. Therefore, when identifying the antenna index selected in step l + 1 of the algorithm, we observe non-uniqueness in the index determination, such that: $K_{l+1}={\arg }_{k_{l+1}\in {{\rm Y}}_{l+1}} {\alpha }_{k_{l+1}, l+1}$. Let S denote the set of N_r receiving antennas, which corresponds to all rows of the channel matrix H, S = {1, 2, ⋯, N_r}, F = S, then push F into the stack as the root node. Let d denote the cardinality of the optimal antenna subset currently under search (MAX ≥ d ≥ MIN), which undergoes a gradual decrement from MAX to MIN. The implementation steps of the proposed I-BAB algorithm are as follows:

• Pruning Operation: If the channel capacity of the current node $C_{\mathit{current}}$ is less than $C_{\mathit{opt}, d}$ (where $C_{\mathit{opt}, d}$ represents the historical optimal value corresponding to d), the generation of child nodes for the current node is terminated. This constitutes the critical mechanism of the pruning operation.
• Generation of Child Nodes: The number of child nodes m for the current node can be calculated using the following formula:

  $$m=\text{current-node}.t-(n-d-\text{current-node}.l-1).$$

  (18)
  In Equation (18), t represents the size of the current candidate antenna subset; n denotes the cardinality of the complete receiving antenna set, i.e., n = N_r; l indicates the level of the current node in the search tree, initialized as l = 0. Specifically, for each current node, all possible subsets are generated by removing individual antennas one at a time, followed by calculating the corresponding capacity degradation ${\Delta }_i$ for each subset ($i=1, 2, \cdots , N_r$); after sorting ${\Delta }_i$ in ascending order, the top-m ${\Delta }_j$ ($j=1, 2, \cdots , m$) with minimal descent values are selected. Subsequently, the corresponding subset $S_{\mathit{child}, j}$ associated with each chosen ${\Delta }_j$ is designated as child nodes.
• Reverse Stack Pushing: In conventional stack-based algorithms, child nodes are pushed into the stack in the order of $S_{\mathit{child}, 1}, S_{\mathit{child}, 2}, \cdots {, S}_{\mathit{child}, m}$. However, the I-BAB algorithm implements a critical modification where child nodes are instead inserted in the reverse order $S_{\mathit{child}, m}, S_{\mathit{child}, m-1}, \cdots {, S}_{\mathit{child}, 1}$. This pivotal procedural adjustment facilitates substantial pruning operations in subsequent computations, thereby effectively reducing algorithmic complexity.
• Update Optimal Solution: When the current subset size $\left|S_{\mathit{current}}\right|=d$, update $S_{\mathit{opt}, d}=S_{\mathit{current}}$, $C_{\mathit{opt}, d}=C_{\mathit{current}}$; d is updated to the subsequent value within the specified constraint range. Then repeat the above steps.
• Termination Criterion: The search terminates upon stack emptiness.
• Output: Sequentially output the corresponding $S_{\mathit{opt}, d}$ and $C_{\mathit{opt}, d}$ for each d (MAX ≥ d ≥ MIN).

The detailed algorithmic workflow of I-BAB is comprehensively outlined in Algorithm 2.

Algorithm 2: I-BAB algorithm for multi-cardinality antenna selection

1: $\mathbf{Initialize}: \mathrm{Set} S=\{1,2,\cdots , N_r\}$$, F=S$$, l=0$$, n=\left|S\right|$$, C_{\mathit{current}}={\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}^{\mathrm{H}}\mathbf{H}\right)$$, C_{\mathit{opt}, d}=-\infty$$, S_{\mathit{opt}, d}=S$$, \forall d\in [\mathit{MIN}, \mathit{MAX}]$.

$\mathrm{Create} \mathrm{root} \mathrm{node} S$$, F$$, C_{\mathit{current}}$$, t=n$$, l$. Push root node into stack.

2: while stack is not empty:

current-node = Pop the top node from stack.

$\mathbf{for} i=\mathit{MAX} \mathrm{to} \mathit{MIN}$ do

$\mathbf{if} {\text{current-node}.C}_{\mathit{current}}>C_{\mathit{opt}, i}$, then

$d=i$, break

end if

end for

$\mathbf{if} d\ge \mathit{MIN} \mathrm{and} \le \mathit{MAX}$, then

$\mathbf{if} \left|\text{current-node}.S\right|=d$, then

$C_{\mathit{opt}, d}={\text{current-node}.C}_{\mathit{current}}$$, S_{\mathit{opt}, d}=\text{current-node}.S$.

$\mathbf{else}, m=\text{current-node}.t-(n-d-\text{current-node}.l-1)$

if $m>0,$ then

For$\mathrm{each} k\in \text{current-node} .F \mathrm{calculate} {\Delta }_{k_{l+1}, l+1}={\log }_2\left(1+{\displaystyle \frac{\rho }{N_t}}{\alpha }_{k_{l+1}, l+1}\right)$$, \mathrm{sorting} {\Delta }_{k_{l+1}, l+1}$$\mathrm{in} \mathrm{ascending} \mathrm{order}, \mathrm{the} \text{top-}m {\Delta }_j (j=1, 2, \cdots , m ) \mathrm{with} \mathrm{minimal} \mathrm{values} \mathrm{are} \mathrm{selected}, \text{i.e.}, {\Delta }_1<{\Delta }_2<\cdots <{\Delta }_m , \mathrm{then} \mathrm{we} \mathrm{can} \mathrm{get} \mathrm{an} \mathrm{ordered} \mathrm{index} \mathrm{vector} \mathit{J}$.

$\mathbf{for} i=m \mathrm{to} 1$, do

$\mathrm{Create} \mathrm{child} \mathrm{nodes} {\text{child-node}}_i$$<S_{\mathit{child}, i}$$, C_{\mathit{current}, i}$$, F_i$$, l_i$$, t_i$>,

$\mathrm{where} S_{\mathit{child}, i}=\text{current-node} .S\backslash {\left[\mathit{J}\right]}_i$,

$C_{\mathit{current}, i}={\log }_2\det \left({\mathbf{I}}_{N_t}+{\displaystyle \frac{\rho }{N_t}}{\mathbf{H}}_{\mathit{child}, i}^{\mathrm{H}}{\mathbf{H}}_{\mathit{child}, i}\right)$$, {\mathbf{H}}_{\mathit{child}, i} {\mathbb{C}}^{\left|S_{\mathit{child}, i}\right|\times N_t}$ denotes the channel submatrix constituted by the receive antennas corresponding to $S_{\mathit{child}, i}$ and transmit antennas),

$F_i=\text{current-node} .F\backslash \{{\left[\mathit{J}\right]}_m, \cdots , {\left[\mathit{J}\right]}_i\},$

$l_i=\text{current-node} .l+1,$

$t_i=\text{current-node} .t-i$.

Push the child node into stack.

end for

end if

end if

end if

end while

3: $\mathbf{for} i=\mathit{MAX} \mathrm{to} \mathit{MIN}$, do

$\mathrm{Output}<S_{\mathit{opt}, i}$$, C_{\mathit{opt}, i}$>.

end for

Regarding the two limitations identified in the BAB algorithm in Section 3.3, the proposed I-BAB algorithm demonstrates significant improvements. For question 1, we first employ the Last-In-First-Out (Last-In-First-Out, LIFO) property of a stack to implement Depth-First Search (DFS). The critical operation involves a preprocessing step where child nodes are reverse sorted based on their reduction value ${\Delta }_j$ ($j=1, 2, \cdots , m$) before being pushed onto the stack. Why does reverse push operation enhance pruning efficiency? Without implementing reverse stacking, child nodes are sequentially pushed into the stack as $S_{\mathit{child}, 1}, S_{\mathit{child}, 2}, \cdots {, S}_{\mathit{child}, m}$. According to the Last-In-First-Out (LIFO) principle of stack structures, this configuration prioritizes access to child node $S_{\mathit{child}, m}$ (the child node inducing the maximum reduction) while deferring access to $S_{\mathit{child}, 1}$ (the child node yielding the minimum reduction) until the final traversal stages. In other words, $S_{\mathit{child}, 1}$ retains the antenna contributing most significantly to channel capacity. After reversing the stack sequence, child nodes are pushed into the stack sequentially following $S_{\mathit{child}, m}, S_{\mathit{child}, m-1}, \cdots {, S}_{\mathit{child}, 1}$, with $S_{\mathit{child}, 1}$ being prioritized for the traversal. Since the stack structure inherently implements a depth-first search (DFS), the optimal subset corresponding to current parameter d and its maximum channel capacity $C_{\mathit{opt}, d}$ can be identified during the exploration of $S_{\mathit{child}, 1}$’s entire descendant nodes. Subsequent nodes popped from the stack and their offspring will inherently exhibit channel capacity values less than or equal to $C_{\mathit{opt}, d}$, thereby activating the pruning mechanism to eliminate redundant computations. The reverse stacking mechanism inverts the node visitation order, enabling earlier evaluation of critical branches and thereby accelerating unnecessary path elimination during pruning processes.

Second, we introduced Equation (18) to determine the number of child nodes. This approach eliminates the need to generate all possible child nodes while ensuring that the number of generated child nodes is sufficient to identify the optimal subset. Equation (18) guarantees the preservation of optimal solutions during child node generation, primarily due to the following factors: (1) Due to the monotonicity property: if antenna subset $H_1\subseteq H_2$, the channel capacity $C_1\le C_2$; the removal of the antenna contributing least to capacity will not compromise the optimality potential of the remaining subset. (2) By sorting branches in ascending order of capacity reduction magnitude, we prioritize preserving those whose removal minimally impacts the system capacity, thus maintaining the integrity of high-contribution antenna combinations. (3) Integrated with the monotonicity criterion, pruning is selectively applied only to branches incapable of surpassing known optimal values, thereby eliminating suboptimal candidates while preventing premature exclusion of potentially superior solutions. Furthermore, among the m child nodes, we apply the pruning criteria to further prune the search space, meaning not all m child nodes will be systematically explored. Since the reduction magnitude of nodes increases progressively during later search stages, the necessity of searching subsequent nodes diminishes accordingly. The introduction of Equation (18) ensures that nodes encountered later in the search sequence will exhibit progressively fewer child nodes, potentially even zero child nodes. When m = 0, this condition indirectly achieves pruning. These two mechanisms collectively ensure that no unnecessary nodes are searched during the process.

For question 2, in the I-BAB algorithm, nodes are successively pushed onto the stack in descending order of their channel capacity contribution reduction. Upon identifying the optimal subset corresponding to the current d-value, we update d to the next value within the specified range. The unexplored nodes retained in the stack can then be employed to determine the corresponding optimal subset under the updated d-value. This iterative process continues until the search for d = MIN is completed. The distinctions and interrelationships between the BAB algorithm and the I-BAB algorithm are systematically summarized in

Table 1. Distinctions between the BAB and I-BAB algorithms.

Comparison Dimension	BAB	I-BAB
Search Direction	Incremental approach: Starts from an empty set and iteratively add antennas.	Decremental approach: Starts from the full antenna set and iteratively removes the least contributing antennas.
Stack Management	Natural order stacking with DFS search.	Reverse-pushing: Child nodes are sorted by contribution loss (ascending) and pushed in reverse order, prioritizing exploration of branches with maximum contributions.
Antenna Subset Size	Requires a predefined fixed d. Multiple executions are needed for different values.	Supports range-based definitions: Generates optimal subsets for all d ∈ [MIN, MAX] through a single execution.
Pruning Mechanism	Relies on an initial lower bound (B = −∞) with low pruning efficiency.	Dynamic pruning: Combines reverse stacking and range constraints to significantly reduce redundant node access, enhancing pruning efficiency.

and

Table 2. Interrelationships between the BAB and the I-BAB algorithms.

BAB	Improvement	I-BAB
Generates all possible child nodes	Introduction of child node generation Formula (Equation (18))	“Full generation” → “Selective generation”
Natural-order stacking may lead to inefficient deep searches in suboptimal branches.	Child nodes are sorted by Δi (ascending) and reverse- pushed into the stack.	“Blind search” → “Guided search”.
Requires separate executions for each d.	A single execution covers d ∈ [MIN, MAX].	“Single-objective” → “Multi-objective”

, respectively. This comparative analysis enables a more intuitive and rigorous comprehension of superiority in the I-BAB algorithm.

Finally, it is crucial to address the fact that while the I-BAB algorithm progressively eliminates antennas based on their contribution values to system capacity, this elimination sequence inherently relies on the assumption of a strong correlation between the greedy backward path and the globally optimal forward selection. Therefore, a rigorous validation of this assumption is imperative to ensure the rationality and correctness of the reverse elimination mechanism in the I-BAB algorithm. The mathematical proof is provided in Appendix B.

4. Results and Discussion

In this section, we evaluate the performance of the I-BAB algorithm based on two key metrics: channel capacity and computational complexity. For this experiment, we assume that channel matrix H is perfectly estimated at the receiver and conduct extensive simulations to approximate the upper bound of system capacity. Even when N_r increases to 256, 10,000 trials are sufficient to achieve accurate results. In each experiment, we randomly generate an independent and identically distributed (i.i.d.) channel matrix H and compute the capacity upper bound using Equation (9). We fix N_t = 4 and gradually increase N_r from 8 to 256 while keeping the number of selected antennas d constant at 4. To comprehensively evaluate the performance, we compare the I-BAB algorithm against both the conventional BAB algorithm and the greedy selection algorithm in our channel capacity analysis. As a benchmark, we simulate the channel capacity of a small-scale MIMO system with N_t antenna elements at both the transmitter and receiver, emphasizing the advantages of large-scale MIMO systems in enhancing channel capacity. Additionally, we use the channel capacity derived from Equation (9) as a reference for comparison.

Due to the reliance of CNN-based training models on exhaustive search algorithms for antenna classification and labeling, their application remains challenging in MIMO systems with N_r > 16. Therefore, they are not included as reference algorithms in our comparative analysis. Similarly, the self-supervised MCTS algorithm struggles to scale effectively to large-scale MIMO systems with N_r > 64. Its computational efficiency heavily depends on GPU acceleration, yet the acceleration performance varies significantly across different GPU configurations. Specifically, due to discrepancies in GPU specifications between our experimental setup and that in [15], we have excluded direct comparisons between the I-BAB algorithm and the self-supervised MCTS approach.

Figure 2 shows the Cumulative Distribution Function (CDF) curves of channel capacity after antenna selection with the I-BAB algorithm. The horizontal coordinate in Figure 2 represents the channel capacity and the vertical coordinate represents the cumulative distribution function. Here, d is set to an exact value rather than a range, making it easier to compare with other antenna selection methods. Overall, MIMO systems with antenna selection algorithms demonstrate significantly higher system capacity than small-scale N_t × N_t MIMO systems. As the number of N_r increases from 8 to 256, the system channel capacity improves accordingly. For instance, at CDF = 1 (representing the maximum achievable capacity), the I-BAB algorithm achieves a peak channel capacity of 23.5 bps/Hz at N_r = 8, increasing to 25.7 bps/Hz at N_r = 256, reflecting a gain of 2.2 bps/Hz. It can also be noted that both the BAB and I-BAB algorithms exhibit progressively tighter convergence toward the theoretical upper bound of the full antenna array configuration. A detailed analysis reveals that within the antenna configuration range of N_r = 8 to 256, the I-BAB algorithm achieves a channel capacity remarkably close to that of the BAB algorithm, while the greedy algorithm also performs comparably to the BAB. For instance, when the CDF = 0.1 with N_r = 8, the channel capacity obtained through the BAB algorithm reaches approximately 17.23 bps/Hz, while a greedy search yields 17.17 bps/Hz and I-BAB achieves 17.09 bps/Hz. This represents a difference of 0.14 bps/Hz compared to the BAB algorithm and 0.08 bps/Hz relative to the greedy search. Notably, at CDF = 1, all three algorithms demonstrate an equivalent maximum channel capacity of 23.5 bps/Hz.

Subsequently, we configured N_t = 4 with N_r ranging from 128 to 256 and performed simulations of the channel capacity CDF under different algorithms by setting d to 8 and 16, respectively. The resultant CDF curves are illustrated in Figure 3 and Figure 4. As illustrated in Figure 2, the channel capacity of various algorithms demonstrates an increasing trend with N_r increasing from 8 to 256 while maintaining constant N_t and d values. This observation confirms that the channel capacity escalates with the N_r growth when d is configured as 8 and 16, respectively. To specifically compare the channel capacity discrepancies induced by different antenna selection algorithms under distinct d configurations, the simulation results for N_r = 8 and N_r = 32 are intentionally omitted in Figure 3 and Figure 4, thereby focusing analytical attention on the comparative performance evaluation between different d scenarios.

From Figure 3 and Figure 4, we can see that three antenna selection algorithms exhibit minimal disparities in channel capacity under identical operational conditions. Notably, when d = 8, the maximum channel capacities achievable by all three algorithms at CDF = 1 exceed those observed at d = 4, irrespective of whether N_r = 128 or N_r = 256. Furthermore, at d = 16 with CDF = 1, all antenna selection strategies demonstrate higher maximum channel capacities compared to both d = 4 and d = 8 configurations. Specifically, when d = 8 and N_r = 128, the peak channel capacity reaches 28.64 bps/Hz, exceeding the d = 4 case by 3.39 bps/Hz. With N_r = 128 fixed, increasing d to 16 yields a maximum capacity of 31.49 bps/Hz, demonstrating a 2.85 bps/Hz improvement over the d = 8 configuration. These results corroborate the theory that channel capacity scales with the antenna quantity in multiple-input multiple-output systems, while confirming the broad applicability of the I-BAB algorithm across varying d values. However, the convergence behavior between the three antenna algorithms and the simulated bound exhibits degradation with an increasing d, a phenomenon that merits further investigation in subsequent research to optimize algorithmic performance.

The computational complexity of the I-BAB algorithm was evaluated by measuring the number of nodes accessed during the search process. We set N_t = d = 4 and gradually increased N_r from 8 to 256. As benchmarks, we also evaluate the exhaustive search algorithm and the greedy selection algorithm, representing the worst-case and best-case scenarios, respectively. The results are shown in Figure 5. Since the exhaustive search algorithm lacks pruning operations, the number of nodes it explores grows exponentially with an increasing N_r. In contrast, the greedy selection algorithm computes only d nodes, resulting in significantly lower computational complexity. As illustrated in Figure 5, the horizontal coordinate represents the number of receiving antennas N_r and the vertical coordinate represents the number of nodes visited during the search; the number of nodes visited by the I-BAB algorithm is approximately one-tenth of that required by the BAB algorithm. This highlights the efficiency of the I-BAB-based antenna selection method in reducing the search complexity compared to the traditional BAB algorithm. However, its computational cost remains notably higher than that of the greedy search algorithm. Furthermore, when N_r increases from 8 to 256, the computational complexity of both the BAB and I-BAB algorithms grows at a moderate rate, increasing by nearly two orders of magnitude. In contrast, the exhaustive search algorithm exhibits a much steeper rise in complexity, reaching approximately five orders of magnitude. This confirms that in large-scale antenna configurations, the I-BAB algorithm offers a significant advantage in computational efficiency.

Next, we analyze the algorithmic complexity from the perspective of the actual runtime. As previously mentioned, the number of selected antennas in the proposed I-BAB algorithm is no longer confined to a specific value but can be multiple values within a range. Through a single execution, the I-BAB algorithm can obtain the optimal subset corresponding to each value in this range, thereby eliminating the limitation of conventional BAB algorithms that require multiple executions for different values of d. We set N_r = 16. For the BAB algorithm, we employed MATLAB R2023a to obtain the runtime at d = 2, 4, and 8, respectively. For the I-BAB algorithm, we configured d = {2, 4, 8} and similarly acquired corresponding runtime measurements using MATLAB. The execution time of both algorithms is systematically presented in

Table 3. Execution time of I-BAB and BAB when N_r = 16.

Name of Algorithm	Value of d	Execution Time in Seconds	Total Time
BAB	d = 2	0.001967s	0.172804 s
	d = 4	0.004900s
	d = 8	0.165937s
I-BAB	d = {2, 4, 8}	0.030619s	0.030619 s

. As can be observed from Table 3, the BAB algorithm requires three executions for three distinct d values. When d = 2, the execution time is measured at ${1.967\times 10}^{-3}$ s, whereas for d = 8, it increases to approximately $1.659\times {10}^{-1}$ s. This clearly demonstrates that the execution time of the BAB algorithm exhibits a significant upward trend with an increasing d; it is evident that the execution time of the BAB algorithm demonstrates a significant growth trend as the d-value increases. Meanwhile, we calculated that the total computation time required for three runs of the BAB algorithm is approximately $1.73\times {10}^{-1}$ s. In contrast, the I-BAB algorithm only requires a single execution to obtain optimal antenna subsets corresponding to three different d values, with a significantly reduced computation time of merely $3.0619\times {10}^{-2}$ s. Compared with the BAB algorithm, the marked reduction in runtime of I-BAB not only demonstrates its lower computational complexity but also highlights a crucial advantage: the enhanced operational flexibility in practical implementations.

Finally, we investigate the performance of the I-BAB algorithm under varying SNR conditions. As shown in Figure 6, the horizontal coordinate represents the signal-to-noise ratio, and the vertical coordinate represents the channel capacity; the channel capacity exhibits a monotonic increase with the SNR. In the low-SNR regime (SNR < 5 dB), the capacity grows linearly, while transitioning to the high-SNR regime (SNR > 5 dB) results in decelerated growth, demonstrating logarithmic characteristics. This SNR-dependent behavior aligns with fundamental information-theoretic principles governing additive white Gaussian noise channels. It can also be observed that across different SNR levels, massive MIMO systems employing antenna selection algorithms still exhibit significantly higher system capacity compared to Nt × Nt small-scale MIMO configurations.

Furthermore, we compare the channel capacity performance of three antenna selection algorithms under varying SNR levels. As illustrated in Figure 6, when the SNR increases from 0 to 50 dB, the channel capacity curves of the I-BAB algorithm remain closely aligned with those of BAB and greedy search algorithms. When the $\mathrm{SNR} \le 15\mathrm{dB}$, the three curves exhibit near-overlapping characteristics, indicating negligible discrepancies between I-BAB and BAB algorithms. Remarkably, during the SNR increase from 0 to 10 dB, the I-BAB algorithm achieves a marginally higher channel capacity than the BAB. For instance, at SNR = 5 dB, I-BAB yields 15.72 bps/Hz versus the BAB’s 15.52 bps/Hz, showing a slight increase of 0.2 bps/Hz. However, when the SNR exceeds 15 dB, the IBAB’s capacity begins to underperform the BAB. At SNR = 25 dB, the I-BAB delivers 24.09 bps/Hz compared to the BAB’s 24.41 bps/Hz, with a differential of 0.32 bps/Hz. Specifically, under a SNR of 40 dB, the I-BAB algorithm exhibits a 0.51 bps/Hz deficit in channel capacity compared to the BAB algorithm. This performance gap widens to 0.58 bps/Hz when the SNR increases to 50 dB. Empirical observations demonstrate that for SNR levels exceeding 15 dB, the discrepancy in channel capacity between I-BAB and BAB implementations progressively expands with ascending SNR values, albeit at a slow rate of increment. The performance gap remains modest, but the relative inferiority of the I-BAB suggests potential areas for algorithmic refinement. Therefore, enhancing the performance of the I-BAB under SNR > 15 dB conditions warrants further investigation.

All results mentioned above are derived from simulations under ideal channel conditions. To validate the practical performance of the I-BAB algorithm, we constructed a 4 × 8 MIMO testbed using USRP N310 (Figure 7), which constitutes a MIMO system with N_r = 8 and N_t = 4. The SNR range was configured from 5 to 25 dB to emulate both indoor office environments (with strong multipath propagation) and outdoor open scenarios (dominated by line-of-sight transmission). Channel data acquisition was implemented through Orthogonal Frequency Division Multiplexing (OFDM) pilot signals with a typical 20 MHz bandwidth. Experimental measurements are systematically documented in

Table 4. Performance comparison in real-world environments when N_r = 8, N_t = 4, ρ = 15 dB.

Name of Algorithm	Total Capacity (Mbps)	Throughput (Mbps)	Computational Latency (ms)
BAB	470	249.1	245.32
I-BAB	467.4	247.72	31.75

and

Table 5. Performance comparison in real-world environments under different SNRs.

SNR (dB)	Throughput of BAB (Mbps)	Throughput of I-BAB (Mbps)
5	164.47	166.66
15	470	467.4
25	258.77	255.39

.

Channel capacity represents the theoretical upper bound derived from idealized simulations, whereas the practical throughput denotes the empirically measured achievable rate under real-world channel conditions. As indicated by simulation data in Figure 2, the corresponding channel capacities under this configuration are 23.5 bps/Hz for the BAB and 23.37 bps/Hz for the I-BAB. In Table 4, the aggregate capacity is calculated as the bandwidth multiplied by channel capacity. Under ideal conditions, the actual throughput should closely approach this theoretical maximum. However, the observed throughput in Table 4 demonstrates a performance gap from the aggregate capacity, primarily attributable to practical limitations in the USRP N310-based 4 × 8 MIMO system, including the inherent protocol overhead, BER impacts, and hardware efficiency constraints. Notably, the measured throughput of BAB and I-BAB algorithms reaches 249.1 Mbps and 247.72 Mbps, respectively, exhibiting a marginal difference of 1.38 Mbps. This comparative analysis reveals that both the BAB anI-BAB maintain comparable throughput performances under identical hardware constraints, which aligns with the simulation results obtained in ideal channel conditions. Simultaneously, the proposed I-BAB algorithm achieves a latency of 31.75 ms, merely 13% of the BAB algorithm’s latency, further demonstrating its marked effectiveness in reducing computational complexity. Due to practical equipment cost constraints, the application of the I-BAB algorithm for performance validation in large-scale systems with higher antenna counts was not implemented; this will be addressed through subsequent research endeavors where we will create necessary conditions to conduct further explorations.

Table 5 presents the actual throughput of the BAB algorithm and I-BAB algorithm under different SNR conditions. As demonstrated in the table, the throughput of the I-BAB marginally exceeds that of the BAB at SNR = 5 dB. When the SNR reaches or exceeds 15 dB, the BAB’s throughput becomes consistently lower than the I-BAB’s. These results align with the conclusions derived from the simulation data in Figure 6. In the high SNR region, although the throughput gap between the I-BAB and BAB remains marginal with a relatively low growth rate of disparity, the problem of further enhancing the throughput performance of the I-BAB algorithm under high SNR conditions still warrants further investigation in future research.

5. Conclusions

This paper proposes an improved BAB search algorithm (I-BAB) under the assumption of independent and identically distributed (i.i.d.) Rayleigh flat-fading channels. Extensive simulations are conducted to evaluate the channel capacity and computational complexity performance of antenna selection using the I-BAB algorithm. Experimental results demonstrate that as the number of antennas at the receiver increases, the channel capacity achieved by the I-BAB algorithm remains comparable to that of the conventional BAB algorithm, while exhibiting significantly higher search efficiency than both the BAB and exhaustive search methods. This validates the theory that the I-BAB algorithm can substantially reduce computational complexity without sacrificing channel capacity, thereby achieving comprehensive performance enhancement for massive MIMO systems.