Maximizing Channel Capacity of 3D MIMO System via Antenna Downtilt Angle Adaptation Using a Q-Learning Algorithm

Abstract

3D MIMO introduces the vertical dimension of the antenna downtilt angle to make the direction of signal transmission more accurate to improve system capacity. In this paper, we verify the effect of antenna downtilt angle on channel capacity through simulations of four fixed antenna downtilt angles, 90, 96, 99, and 102 degrees under the conditions that the distance between mobile station (MS) and base station (BS) is 250 m, and the heights of antenna in BS and MS are 25 m and 1.5 m, respectively. The simulation results show that the antenna downtilt angle of 96 degrees has a larger channel capacity than the others. In addition, we proposed an adaptive optimization method by applying the Q-learning algorithm to adaptively optimize the antenna downtilt angles to maximize system capacity. The performance of the proposed method is to investigate the Q-learning algorithm with three different discount rates at 0.9, 0.5, and 0.1, and four different propagation distances on 20 × 1 and 60 × 4 MIMO. We demonstrate that there is only a 1% difference between the adaptively optimized antenna downtilt angle and the ideal optimal antenna downtilt angle when the discount rate of Q-learning algorithm is 0.9, and its channel capacity performance can reach more than 99.72% of the ideal optimal one.

1. Introduction

In mobile communication, after following 4G LTE (Long Term Evolution, LTE) and LTE-Advanced, 5G (the fifth generation mobile communication) announced standards and commercial operation in 2020. One of the key technologies is 3D MIMO (Multiple Input Multiple Output) technology [1,2]. Among the currently major technologies of 5G, 3D MIMO can be applied downward to the current 4G network to improve system performance. 3D MIMO uses a technology similar to Phased Array Radar (PAR) to greatly improve spectral efficiency through more accurate beamforming and multiplexing [3]. 3D MIMO is better than 4G in improving system capacity, coverage, anti-interference and transmission rate. It is an extremely important key technology for 5G technology currently developed around the world [4].

At the same time, new antenna technologies and network architectures have propelled the expansion of MIMO technology to higher dimensions. The 3D MIMO technology of Array Antenna came into being. Compared with the traditional MIMO technology, 3D MIMO increases the vertical dimension of the beam. By combining the horizontal and vertical dimension beams, the signal transmission in space is closer to reality. The three-dimensional spatial characteristics of the antenna can be adjusted by the antenna downtilt angle to realize the 3D Beamforming, which can effectively improve the signal strength and reduce the interference.

Beamforming technology is a critical technology in 3D MIMO. The advantages of beamforming technology are mainly reacting in the following aspects: expanding coverage, improving cell edge transmission quality, and anti-interference. The horizontal azimuth angle in traditional beamforming technology cannot accurately transmit communication for different users, and the communication quality is poor [5]. Because of conventional beamforming technology problems, 3D beamforming technology applies. 3D beamforming technology combines beams in horizontal and vertical dimensions, which can accurately beamform different users, improve Spectrum Efficiency and Channel Capacity, and effectively improve communication quality [5].

The massive parameters have to be optimized in 3D beamforming technology to achieve the maximum performance of 3D MIMO systems. Therefore, machine learning (ML) catches a lot of our attention. Machine learning is increasingly adopted to develop advanced methods that can adaptively extract patterns based on given environmental measurements and performance indicators. These patterns can optimize the parameter settings for different interesting problems [6,7,8].

Because of the importance of beamforming in the vertical dimension in 3D MIMO systems, this paper will investigate the effect of antenna downtilt on channel capacity in 3D MIMO systems; and apply it with an enhancement machine learning. The Q-learning algorithm [9,10] enables antenna downtilt to perform an adaptive operation method so that the 3D MIMO system can automatically adjust the optimal antenna downtilt angle based on the position of the MS to maximize the channel capacity of the communication system.

This work firstly inspects the pros and cons of fixing a specific elevation angle on channel capacity. Based on the literature [11], four fixed tips, 90, 96, 99, and 102 degrees, are employed for this paper’s study, which should determined in advance. A specific angle has a significant improvement in beamforming and channel performance. However, this method fails to adapt the system to adjust the antenna downtilt angle to the optimal one while MS is moving.

Secondly, we investigate the proposed method that adopts a reinforcement learning method named the Q-learning algorithm to maximize the channel capacity employing optimizing the antenna downtilt angle of a single MS at different distances. Based on the Q-learning algorithm, the reward value is given according to the size of channel capacity to varying angles of the antenna inclination. The Q-Table is generated by learning the iterative operations. Each reward value of the Q-Table corresponds to a specific state, that is, the channel capacity at a particular antenna downtilt angle. Eventually, according to the data in the Q-Table, an optimal antenna downtilt angle can be selected to maximize the channel capacity. Two different antenna configurations of 20 × 1 and 60 × 4 are considered to verify the performance of the proposed method under different transmission distances. We use MATLAB software to simulate and verify the performance of proposed method in this paper.

The remainder of the present work is organized as follows: Section 2 explains the system model and Q-learning algorithm. Section 3 presents the method of using a Q-learning algorithm to optimize antenna downtilt angles for maximum channel capacity adaptively. Section 4 presents the simulation results to investigate the performance of the proposed method. Finally, conclusions are given.

2. System Model and Q-Learning Algorithm

2.1. MIMO System Architecture

Mobile communication standards such as 3GPP SCM, ITU, and WINNER follow the geometry-based random channel model modeling method [12,13,14,15,16,17]. For a geometry-based model (GBM), the propagation path is described by statistical parameters, including delay, amplitude, angle of arrival (AoA), angle of departure (AoD), and flakes. It is highly accurate and close to the physical environment [18,19,20,21]. In the existing 2D channel models of 3GPP SCM and WINNER, the propagation path is only described by the azimuth angle of the horizontal dimension. These 2D models all have a fixed antenna downtilt angle value of π/2. However, considering the 3D channel model as shown in Figure 1, the literature [4,11] pointed out that the vertical component in the emission vector occupies a large part of the energy, so the propagation path only considering the azimuth in the horizontal direction does not conform to the actual situation. In addition, when the antenna downtilt angle is fixed, the elevation degrees of freedom representing the channel are not utilized. Dynamic adjustment of the antenna downtilt angle at the base station (BS) transmitter can provide multiple possibilities for transmitting 3D beamforming signals to the mobile station (MS), significantly improving system performance. Therefore, extending the existing 2D channel model to a 3D channel needs to consider the elevation angle of the propagation path and introduce the parameter into the channel model instead of being assumed to be fixed [22,23,24,25].

According to the antenna configuration in [5], the effective 3D channel matrix ${[H]}_{su}$ between the sth antenna of the base station (BS) transmitter and the uth antenna of the mobile station (MS) receiver can be written as

$${\left[H\right]}_{su}=\frac{1}{N}{\displaystyle \sum _{n=1}^N{\alpha }_n\sqrt{g_t({\varphi }_n,{\theta }_n,{\theta }_{tilt})}{e}^{(ik(s-1)d_t\sin {\varphi }_n\sin {\theta }_n)}\sqrt{g_t({\phi }_n,{\vartheta }_n)}}{e}^{(ik(u-1)d_r\sin {\phi }_n\sin {\vartheta }_n)}$$

(1)

where ${\varphi }_n$ and ${\theta }_n$ are the azimuth and elevation angles of the nth transmit path, respectively, ${\phi }_n$ and ${\vartheta }_n$, their relationship as shown in Figure 1, are the azimuth and elevation angles of the nth receive path, respectively, ${\theta }_{tilt}$ is the antenna downtilt angle, ${\alpha }_n$ represents the random amplitude of the nth path, $\sqrt{g_t({\varphi }_n,{\theta }_n,{\theta }_{tilt})}$ and $\sqrt{g_t({\phi }_n,{\vartheta }_n)}$ represent the global azimuth map of the transmitter antenna and the receiver antenna, respectively, $d_t$ and $d_r$ represent the distance between the transmitter antenna and the receiver antenna, respectively, and k represents the wave number. Finally, the array response matrices of the transmitter and receiver antennas are expressed by

$${\left[a_t\left(\varphi ,\theta \right)\right]}_s={{\displaystyle e}}^{ik(s-1)d_t\sin \varphi \sin \theta }$$

(2)

and

$${\left[a_t\left(\varphi ,\theta \right)\right]}_u={{\displaystyle e}}^{ik(u-1)d_r\sin \phi \sin \vartheta }$$

(3)

respectively [11], where s and u represent the sth transmitter antenna and the uth receiver antenna, respectively.

In order to make the concept of antenna downtilt angle specific in practical application scenarios, the International Telecommunication Union (ITU) uses a narrow-band beam to approximate the global azimuth of each antenna expressed by [13]

$$\sqrt{{10}^{0.1\times \{17-\min [(A_V(\theta ,{\theta }_{tilt})-A_H(\varphi )), 20]\}}}$$

(4)

in which

$$A_H(\varphi )=-\min [12{(\frac{\varphi }{{\varphi }_{3dB}})}^2, 20] dB$$

(5)

$$A_V(\theta ,{\theta }_{tilt})=-\min [12{(\frac{\theta -{\theta }_{tilt}}{{\theta }_{3dB}})}^2, 20] dB$$

(6)

In Equation (5), ${\varphi }_{3dB}$ is the horizontal half beam width; in Equation (6), ${\theta }_{3dB}$ is the vertical half beam width. At the receiver, the antenna pattern expression $\sqrt{g_t({\phi }_n,{\vartheta }_n)}$ has no antenna gain, so its gain is set to 0 dB. According to the above, the expression of the antenna pattern in the 3D channel model includes the parameter ${\theta }_{tilt}$, so the value of ${\theta }_{tilt}$ will affect the channel performance. The goal of this study is to dynamically adjust the antenna downtilt angle to maximize the channel capacity.

2.2. Maximum Channel Entropy

The channel model released by commercial standards [13] is more accurate and clearly defines a more rigorous 3D channel model. Since it is a commercial standard, the theoretical analysis of this 3D channel model is rather complex. Firstly, multiple propagation paths lead to many random variables (RVs) in the model. Secondly, in addition to numerous random variables in the model, there are also multiple nonlinearly correlated random variables, AoDs and AoAs parameters. To facilitate a theoretical analysis, reconstructing an equivalent channel model is necessary and further derives a more concise channel model architecture. Therefore, Ref. [26] uses the principle of maximum entropy for Equation (1) to bridge the gap between commercial standards and theory and derives an equivalent channel model of Equation (1), which can avoid the problem of a large number of random variables, and enable the model to be used in a natural environment.

Ref. [4] proves that the model generated by the maximum entropy principle can conform to the actual environment. In addition, inspired by the Bayesian parameter estimation analysis method in the maximum entropy principle to analyze the modeling problem, the author of Ref. [26] uses this architecture as a theoretical basis to create the channel model of the 2D MIMO system conforming to the actual environment characteristics.

The channel model proposed in the commercial standard is the Geometry-based Stochastic Model (GBSM). This model assumes that there is only one reflection (Single Bounce Scattering) between the transmitter and receiver [11]. In addition, the positions of the scatterers, the base station (BS), and the mobile station (MS) are assumed to be fixed. Therefore, the modeler can calculate (1) AoDs, (2) AoAs, (3) the azimuth of the nth transmitting path at the base station (${\varphi }_n$), (4) the elevation angle (${\theta }_n$) of the nth transmitting path at the base station, (5) the azimuth angle (${\phi }_n$) of the nth receiving path at the mobile station, (6) the elevation angle (${\vartheta }_n$) of the nth receiving path at the mobile station, etc. In Ref. [11], it is assumed that the channel model parameters, except for the amplitude, including AoD and AoA, are all fixed values in the modeling stage. Then, the only random variable in the channel model is the size of the vibration (α), which is a normal distribution. Hence, a channel model that meets the virtual environment can be simulated.

It is proved in [11] that when the number of propagation paths, the AoD, AoA, and the transmitting and receiving power of the propagation paths are all known, the maximum entropy channel model can be expressed as follows:

$$H=\Psi P_{R_X}^{1/2}(\Omega \circ G)P_{T_X}^{1/2}{\Phi }^H$$

(7)

where P_Tx and P_Rx represent the transmitting and receiving power, respectively, $\Omega$ is the mask matrix of the capture path gain, $\Psi$ and $\Phi$ are the antenna array responses, and “°” is the Hadamard Product. In addition, another assumption for maximizing entropy is to make the elements of the unknown random matrix G have zero mean, and the variance is one independent and identically distributed (i.i.d) Gaussian variable [11].

In order that the expressions in the channel model of Equation (1) can be expressed in the structural manner of Equation (7), let A and B be the determinants of $N_{BS}\times N$ and $N_{MS}\times N$ given by Equations (8) and (10) matrix, respectively:

$$A=\frac{1}{N}[a{}_t({\varphi }_1, {\theta }_1), a{}_t({\varphi }_2, {\theta }_2) \cdots a{}_t({\varphi }_N, {\theta }_N)]\circ \left[\begin{array}{ccc}\sqrt{g({\varphi }_1,{\theta }_1,{\theta }_{til{t}_1})}& \cdots & \sqrt{g({\varphi }_N,{\theta }_N,{\theta }_{til{t}_1})}\\ ⋮& \ddots & ⋮\\ \sqrt{g({\varphi }_1,{\theta }_1,{\theta }_{til{t}_{N_{BS}}})}& \cdots & \sqrt{g({\varphi }_N,{\theta }_N,{\theta }_{til{t}_{N_{BS}}})}\end{array}\right]$$

(8)

where

$$a_t\left({\varphi }_n,{\theta }_n\right)={\left[{\left[a_t\left({\varphi }_n,{\theta }_n\right)\right]}_{s=1},\cdots ,{\left[a_t\left({\varphi }_n,{\theta }_n\right)\right]}_{s=N_{BS}}\right]}^{\mathit{H}},$$

(9)

$$B=\left[a_r\left({\phi }_1,{\vartheta }_1\right),\cdots ,a_r\left({\phi }_N,{\vartheta }_N\right)\right],$$

(10)

and

$$a_r\left({\phi }_n,{\vartheta }_n\right)={\left[{\left[a_r\left({\phi }_n,{\vartheta }_n\right)\right]}_{u=1},\cdots ,{\left[a_r\left({\phi }_n,{\vartheta }_n\right)\right]}_{u=N_{MS}}\right]}^T$$

(11)

In addition, array response and antenna azimuth diagram expressions have been included in A and B. P_Tx and P_Rx in Equation (7) are identity matrices because the transmitting and receiving powers are combined into the antenna azimuth map. In addition, because of the single bounce scattering model, the $\Omega$ Mask Matrix is a diagonal matrix. Let $\Psi$ and $\Phi$ be represented by B and A, respectively; then, the solution of the maximum entropy problem of the 3D MIMO channel model of $N_{MS}\times N_{BS}$ single bounce scattering in Equation (1) can be expressed as

$$H=\frac{1}{\sqrt{N}}Bdiag\left(\alpha \right)A^H$$

(12)

where $\alpha$ is a vector of rank N, and each element in the vector is an i.i.d. Gaussian matrix variable with zero mean and one variance.

2.3. Calculation of 3D MIMO Channel Capacity

Equation (1) is the element in Equation (12), so Equation (1) can be represented by Equation (13):

$${\left[H\right]}_{su}=h_{i,j}$$

(13)

Then, Equation (12) can be expressed as

$$H=\left[\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \cdots & h_{1,N}\\ h_{2,1}& h_{2,2}& \cdots & h_{2,N}\\ \cdots & \cdots & \ddots & \cdots \\ h_{M,1}& h_{M,2}& \cdots & h_{M,N}\end{array}\right]$$

(14)

The parameters in the channel model will be defined as follows.

2.3.1. Elevation Angle between Transmitting and Receiving Path (AoDs, AoAs)

According to Ref. [11], since it is close to the commercial communication standard, we use the angle distribution and antenna azimuth map specified in the commercial standard for analysis. The elevation angles of the transmitting and receiving paths are drawn according to the Laplace distribution density of the elevation angle mentioned in the [11], and the expression of the elevation angle is as in Equation (15):

$$f_{\theta }\left(\theta \right)=e^{\left(-\frac{\sqrt{2}\left|\theta -{\theta }_0\right|}{\sigma }\right)}$$

(15)

where ${\theta }_0$ is the mean value of the elevation angles of departure (EAoDs) or the elevation angles of arrival (EAoAs), and $\mathsf{\sigma }$ is the angular spread in the elevation.

Next, the azimuth angles of departure (AAoDs) of the transmitting path and the azimuth angles of arrival (AAoAs) of the receiving path can be expressed by the VMF (Von Mises-Fisher distribution) distribution, as in Equation (16):

$$f_{\varphi }\left(\varphi \right)=\frac{e^{\left(k\cos \left(\varphi -\mu \right)\right)}}{2\pi I_n\left(k\right)} , n=0$$

(16)

where $I_n\left(k\right)$ is the modified Bessel function, n is the order, and μ is the mean of the azimuth angles of departure (AAoDs) or the azimuth angles of arrival (AAoAs). k is the density of scattering. The larger the k is, the more the azimuth of the path is concentrated on the mean value of the azimuth, and the smaller it is, the more scattered. Since the channel is assumed to be a single bounce scattering, the angle is generated only once and is assumed to be a fixed angle in the channel.

2.3.2. Antenna Downtilt Angle and Elevation Angle of Line of Sight

In Equation (4), ${\theta }_0$ in the channel model is set as the Elevation Angle of Line of Sight ${\theta }_{LoS}$ [11] between the base station (BS) and the mobile station (MS). Let $\Delta h$ represent the height difference between the mobile station (MS) and the base station (BS), and x and y, respectively, represent the distance difference between the base station (BS) and the mobile station (MS) in x–y coordinates, then the line-of-sight axis elevation angle ${\theta }_{LoS}$ can be defined as

$${\theta }_{LoS}=\frac{\pi }{2}-ta{n}^{-1}\frac{\Delta h}{\sqrt{x^2+y^2}}$$

(17)

Then, the antenna downtilt angle ${\theta }_{tilt}$ can be defined as shown in Figure 2.

2.3.3. Channel Capacity

The MIMO channel is often changed randomly, so the H channel matrix is a random matrix, which means that the MIMO channel capacity is also random and time-varying. It is assumed that the channel matrix is a stationary random process; in other words, the MIMO channel capacity can be calculated based on the time average of the signal transmission on the channel. To sum up the above, the formula of MIMO channel capacity is as Equation (18):

$$\mathrm{C}={\displaystyle \sum }_{i=1}^{r}lo{g}_2\left(1+\frac{E_x}{N_{Tx}{N}_0}{\gamma }_i^{opt}{\lambda }_i\right)$$

(18)

where ${\lambda }_i$ is the eigenvalue obtained by performing singular value decomposition of the channel matrix H, so the channel matrix H can be rewritten as

$$H=U\mathsf{\Lambda }{V}^H$$

(19)

where U and V are unitary matrices ($U{U}^H=1$, $V{V}^H=1$), and $\Lambda$ is a real diagonal matrix, which only has non-zero values on the diagonal ${\lambda }_0\ge {\lambda }_1\ge \cdots \ge {\lambda }_{n_{min}-1}$, where $n_{min}=\min \left(n_t,n_r\right)$. In addition, ${{\displaystyle \sum }}_{i=1}^r{\gamma }_i^{opt}=N_{Tx}$.

2.4. The Effect of Antenna Downtilt Angle on Channel Capacity

The 3D channel model proposed in [11] is constructed on the condition that the channel parameters are known, and is derived from the system-level 2D random channel model that appears in standards such as 3GPP, ITU, and WINNER. Ref. [11] constructs the channel matrix of 3D MIMO under the condition of known AoDs and AoAs, which can accurately represent the channel model. The simulation results in [11] show that choosing an accurate antenna downtilt angle at the base station (BS, transmitter) can greatly improve the performance, so the antenna downtilt angle has a critical impact on the channel capacity.

2.4.1. Distributions of EAoDs and EAoAs

In Equation (15), the elevation angle distribution of the path is Laplace distribution, and the spread degree of the path will increase as s increases. The minor s is, the more the spread degree of the path will be concentrated on the mean value of the elevation angle. Figure 3 is the cumulative density function (CDF) diagram when the mean elevation angle is zero (${\theta }_0=0$) degrees, and $\sigma$ = 5°, 10°, 15°, and 20°. Figure 3 shows that, when σ is larger, the degree of dispersion of its angles is greater. Conversely, it is more concentrated in the region where the average angle equals to zero (${\theta }_0=0$).

2.4.2. Distributions of AAoDs and AAoAs

In Equation (16), the von Mises distribution (VM) is used, which is a particular case of the von Mises–Fisher distribution (VMF) [27,28,29]. The original von Mises–Fisher distribution is shown in Equation (20) [29]:

$$f_p\left(x;\mu ,k\right)=C_p\left(k\right)e^{\left(k\cos \left(x-\mu \right)\right)}$$

(20)

in which

$$C_p\left(k\right)=\frac{k^{p/2-1}}{{\left(2\pi \right)}^{p/2}{I}_{p/2-1}\left(k\right)}$$

(21)

When p = 2, it can be simplified to von Mises distribution (VM), such as Equation (22)

$$C_p\left(k\right)=\frac{k^{p/2-1}}{{\left(2\pi \right)}^{p/2}{I}_{p/2-1}\left(k\right)}, if p=2;$$

(22)

that is,

$$C_2\left(k\right)=\frac{k^{2/2-1}}{{\left(2\pi \right)}^{2/2}{I}_{2/2-1}\left(k\right)}=\frac{1}{2\pi I_0\left(k\right)}$$

(23)

Equation (23) is substituted into Equation (20) to obtain Equation (16).

In Equation (16), it is assumed that the mean value of the azimuth angle between the transmitting path and the receiving path is zero degrees ($\mu =$ 0°). Thus, when the value of k is more significant, the azimuth angle of the path will be more concentrated towards the mean value of the azimuth angle ($\mu =$ 0°). Therefore, for the overall performance, the larger the value of k, the more the beamforming ability can be significantly improved, and the overall channel capacity improved. The Probability Density Function of Path Azimuth is shown in Figure 4.

2.4.3. Simulation Parameters with Different Antenna Downtilt Angles

The simulation in this section will be carried out in accordance with the actual environment, and the parameters are set as shown in

Table 1. Simulation parameters.

Parameter	Value
Base Station (BS) Height (m)	25
Mobile Station (MS) Height (m)	1.5
Distance (m)	250
Carrier Frequency	2.65 GHz
Transmission Power	20 dBm
Bandwidth	20 MHz
Noise Power	−174 dBm/Hz
Number of Base Station (BS) Transmitting Antenna	20
Number of Mobile Station (MS) Receiving Antenna	1
Number of Paths	40
Path Attenuation	$PL=128.1+37.6\times 10\log \left(d_{km}\right)+6$
Vertical Half Beam Width $({\theta }_{3dB}$)	15°
Horizontal Half Beam Width $({\varnothing }_{3dB}$)	70°
Emission Angle Spread $({\sigma }_{BS}$)	7°
Incidence Angle Spread $({\sigma }_{MS}$)	10°
Transmitting Path Divergence Degree ($k_{BS}$)	5
Receiving Path Divergence Degree ($k_{MS}$)	5
Line-of-Sight Axis Elevation $({\theta }_{LoS}$)	95.37° (Distance: 250 m)
Antenna Downtilt Angle $({\theta }_{tilt}$)	90° 96° 99° 102°

.

The simulation result in Figure 5 shows that the channel capacity is the largest when ${\theta }_{tilt}=$ 96°, the smallest when ${\theta }_{tilt}=$ 102°, and when ${\theta }_{tilt}=$ 90° reduced to 2D MIMO. From the perspective of the average channel capacity of the system, when the CDF value is 0.5, the difference between 96° and 102° is about 0.471 bits per Hz (0.471 bps/Hz). Furthermore, it can seem that, when the antenna downtilt angle (${\theta }_{tilt}$) is close to the line-of-sight axis elevation angle (${\theta }_{LoS}$), the channel capacity is the largest. In other words, a slight angular difference in the antenna downtilt angle will have a significant impact on the channel capacity. The above simulation results are obtained under the condition of fixed four antenna downtilt angles. In the wireless communication system, how to adaptively select an optimal antenna downtilt angle is a topic worthy of in-depth discussion.

2.5. Q-Learning Algorithm

Q-learning is a reinforcement learning technique in machine learning. Q-learning aims to learn a policy that tells the agent under what circumstances to act. It does not require an environment model and can handle random transitions and rewards without modification. For any finite decision process, Q-learning finds an optimal policy that maximizes the expected value of the total premium after all successive steps, starting from the current state. For any given Markov decision process, Q-learning can determine to provide an optimal action selection policy by considering infinite exploration time and a partially random policy. Q-learning is a function of reward used to support reinforcement learning, which can represent the value of an action taken in a certain state.

Q-learning is an off-policy reinforcement learning algorithm. Q-learning uses the value function of the action with the highest value in the next state, not the value function that the learning agent actually uses in the next state, which is the meaning of off-policy. The operation mode behind Q-learning depends on the value iteration. The calculation method is expressed as

$$Q\left(s_t,a_t\right)\leftarrow Q\left(s_t,a_t\right)+\alpha \left[R\left(s_t,a_t\right)+\gamma \underset{a}{max}Q\left(s_{t+1},a\right)-Q\left(s_t,a_t\right)\right]$$

(24)

where $Q\left(s_t,a_t\right)$ represents the value of value function (Q value) obtained after taking action $a_t$ in the state $s_t$; $R\left(s_t,a_t\right)$ is the reward or value obtained after taking action $a_t$ in the current state $s_t$; $\alpha$ is the learning rate, whose value is $0\le \alpha \le 1$; this parameter is used to determine the convergence rate, the larger $\alpha$ is, the faster the convergence speed is. In a fully deterministic environment, a learning rate of $\alpha$ = 1 is optimal. The discount rate $\gamma$ determines the importance of future rewards, usually a constant less than 1, used to control the expected reward value in the future. If γ = 0, it means that the agent is short-sighted and only based on the current strategy. The smaller the discount rate, the more it appears that the agent focuses on the current and recent learning. On the contrary, the larger the discount rate, the more foresight the agent has and the more attention paid to the follow-up learning outcomes and the value of the rewards for future learning. $\underset{a}{max}Q\left(s_{t+1},a\right)$ means to select the value of value function (Q value) of action $a$ with the highest value function in all possible actions from state $s_t$ to state $s_{t+1}$, used for adjustment purposes, and not the value of value function of the action actually taken by the next state. The value of the value function of the action actually taken in the next state should be $Q\left(s_{t+1},a_{t+1}\right)$. Figure 6 shows the Q-learning algorithm.

3. Adaptive Optimization of Antenna Downtilt Angle Based on the Q-Learning Algorithm

According to the results shown in Section 2.4, antenna downtilt angle affects prominently the performance of channel capacity of 3D MIMO systems. The fixed ideal optimal antenna downtilt angle is not useful in a real environment. Therefore, the 3D MIMO system needs to be able to adaptively find an optimal antenna downtilt angle, and the fixed parameter method cannot make the system find an optimal antenna downtilt angle. This section will explain how to use the Q-Learning algorithm helps the base station (BS) antenna to automatically search for the best antenna downtilt angle, so that the channel capacity can be improved through the training of reinforcement learning, so that the channel capacity can achieve the best improvement.

3.1. Simulation Procedure with the Q-Learning Algorithm

In Equation (24), $\alpha$ is the learning rate or step size, which determines the degree to which the newly acquired information covers the old information. A factor of $\alpha =0$ makes the agent learn nothing (fully exploiting previous knowledge), while a factor of $\alpha =1$ makes the agent only consider the latest information (ignoring previous knowledge to explore future possibilities). In a fully deterministic environment, the learning rate of $\alpha =1$ is optimal; then, Equation (24) will be reduced to Equation (25) as follows:

$$Q\left(s_t,a_t\right)\leftarrow R\left(s_t,a_t\right)+\gamma \underset{a}{max}Q\left(s_{t+1},a\right)$$

(25)

The discount rate $\mathsf{\gamma }$ determines the importance of future rewards. When γ approaches zero, the agent will only consider the current reward, and when γ approaches 1, it will make the agent strive for long-term high rewards. If $\mathsf{\gamma }=1$, the learning result after iteration will diverge. This study will discuss the optimization degree of the antenna downtilt angle and the value of the channel capacity when γ = 0.9, 0.5, and 0.1 are considered.

Q-learning is about updating numbers in a two-dimensional array (action space × state space), similar to dynamic programming, and it does not have the ability to estimate unseen states. Hence, it is necessary to understand all the states and give the reward value, and then carry out the establishment of the future state to evaluate the Q table. Therefore, first of all, it required to determine the establishment of R (reward matrix). In this study, it is assumed that the antenna moves 0.5 degrees each time. When the antenna starts machine learning, it goes to the first random angle ${\theta }_n$ when the state n (channel capacity c_n). Its next action is either to move forward 0.5 degrees, or back 0.5 degrees, and randomly decide to move forward or backward, but it cannot jump to any other angle. Therefore, if its next action is to move forward by 0.5 degrees, reaching the next state n + 1 is the angle ${\theta }_n+0.5$ (the channel capacity is c_n+1), at this time, the channel capacity is given a reward value R; then c_n + R = c_n+1, and then R = c_n+1 − c_n. If it goes back 0.5 degrees, the next state n − 1 is the angle ${\theta }_n-0.5$ (the channel capacity is c_n−1); at this time, the channel capacity is given a reward value R; then, c_n + R = c_n−1, and then R = c_n−1 − c_n. When the next state of the antenna is the current state and the angle does not move, the reward R is 0, and, at the same time, the reward R of any other angle state that the antenna cannot reach is also set to 0.

From the initial Q matrix to zero matrix, which means that the environment is unknown, learning and training are carried out through iterative operations, so that the expected value of the total reward is maximized after all consecutive steps, the best Q table is established, and finally the angle of the best Q value is found, which is the optimal antenna downtilt angle. That is, when a channel capacity value is found whose channel capacity value is both greater than the previous angle and the next angle, it means that it is the maximum channel capacity, and the angle at which it is located is the optimal antenna downtilt angle.

The detailed operation procedure will be described as follows:

1.

In order to understand all states first and then set the reward value, the search angle is between 90°–125°, and the interval is 0.5 degrees, so a total of 71 angles need to be searched.

1.

Calculate the channel capacity at each angle.

1.

Set the parameters in Equation (24).

α: 1 (Control the convergence rate, known as the learning rate, in the hope of finding the best antenna inclination in a short time, so α is set to 1).

γ: 0.9 (The larger the discount rate is, the more the agent pays attention to the results of training over a long time, so it is set to 0.9).

After setting the above parameters, Equation (26) can be obtained:

$$Q\left(s_t,a_t\right)\leftarrow R\left(s_t,a_t\right)+\gamma \underset{a}{max}Q\left(s_{t+1},a\right) , \gamma =0.9$$

(26)

4.

The initial environment is unknown and the initial Q matrix is a 71 × 71 zero matrix because it has not been iterated,

5.

Build R (reward matrix), c_n: channel capacity at the nth angle.

$$\mathbf{R}=\begin{array}{cc}& \begin{array}{ccccccc} a_1 & \hspace{1em}{a}_2\hspace{1em}& a_3 & \hspace{1em} a_4 & \hspace{1em} a_5\hspace{1em}& \hspace{1em}\cdots \hspace{1em}& \hspace{1em}{a}_{71}\hspace{1em}\end{array}\\ \begin{array}{c}{S}_1\\ S_2\\ S_3\\ S_4\\ S_5\\ ⋮\\ S_{71}\end{array}& \begin{array}{c}\left[\begin{array}{ccccccc}0& c_2-c_1& 0& 0& 0& 0& 0\\ c_1-c_2& 0& c_3-c_2& 0& 0& 0& 0\\ 0& c_2-c_3& 0& c_4-c_3& 0& 0& 0\\ 0& 0& c_3-c_4& 0& c_5-c_4& 0& 0\\ 0& 0& 0& c_4-c_5& 0& \ddots & 0\\ 0& 0& 0& 0& \ddots & 0& c_{71}-c_{70}\\ 0& 0& 0& 0& 0& c_{70}-c_{71}& 0\end{array}\right]\end{array}\hfill \end{array}$$

(27)

The 1st to 71st columns on the left side of the R matrix are the 1st to 71st angles, called the state column $s_t$. Furthermore, on the upper side, the 1st to 71st columns marked are the actions to be taken $a_t$ (the action is from the left state column and the nth angle to the (n + 1)th angle or the (n − 1)th angle) because the movement of the angle can only be randomly moved from the nth angle to the (n + 1)th angle or the (n − 1)th angle. Thus, the reward matrix is only part of the adjacent two angle compensation values, as the above table is the circled part (if n at this time is the second point of view, there is a square circle for parts for the second Angle to pay the first point of view, there is an oval circle for parts for the second Angle to a third Angle compensation, and so on, to generate the compensation matrix R matrix.

6.

$\underset{a}{\gamma max}Q\left(s_{t+1},a\right)$ of all possible actions $a$ in Angle (state, $s_t$) moving to the next Angle (state, $s_{t+1}$): select the action ($\underset{a}{max}$) that can obtain the maximum Q value after performing, and multiply the maximum Q value by γ.

7.

Iterative process: randomly select the angle (state, $s_t$) to perform the action ($a_t$), obtain the reward value (R($s_t$, $a_t$)), and then move from the angle (state, $s_t$) to the next angle (state, $s_{t+1}$). Among all possible actions $a$, the maximum Q value is multiplied by γ to iterate, and the Q table can be obtained after the iteration.

$$\mathbf{Q}=\begin{array}{cc}& \begin{array}{ccccccc} a_1 & a_2\hspace{1em}& a_3 & a_4\hspace{1em}& a_5\hspace{1em}& \cdots \hspace{1em}& a_{71}\hspace{1em}\end{array}\\ \begin{array}{c}{S}_1\\ S_2\\ S_3\\ S_4\\ S_5\\ ⋮\\ S_{71}\end{array}& \begin{array}{c}\left[\begin{array}{ccccccc}0& Q_{1,2}& 0& 0& 0& 0& 0\\ Q_{2,1}& 0& Q_{2,3}& 0& 0& 0& 0\\ 0& Q_{3,2}& 0& Q_{3,4}& 0& 0& 0\\ 0& 0& Q_{4,3}& 0& Q_{4,5}& 0& 0\\ 0& 0& 0& Q_{5,4}& 0& \ddots & 0\\ 0& 0& 0& 0& \ddots & 0& Q_{70,71}\\ 0& 0& 0& 0& 0& Q_{71,70}& 0\end{array}\right]\end{array}\hfill \end{array}$$

(28)

8.

The angle (state) corresponding to the optimal Q value in the search Q matrix is the optimal antenna downtilt angle.

3.2. Ideal Optimal Channel Capacity Based on Ideal Condition ${\theta }_{tilt}={\theta }_{LoS}$

In this section, we use MATLAB software (Portola Valley, CA, USA) developed by the MathWorks to execute the operation procedure of a Q-learning algorithm to find out the ideal optimal channel capacity based on the ideal condition assumption of ${\theta }_{tilt}={\theta }_{LoS}.$

For the purpose of inspecting the proposed method normally, we consider two different antenna configures of 20 × 1 and 60 × 4. The simulation parameters for them are listed in

Table 2. Simulation parameters for 20 × 1.

Parameter	Value
Base Station (BS) Height (m)	25
Mobile Station (MS) Height (m)	1.5
Distance (m)	100, 150, 200, 250
Carrier Frequency	2.65 GHz
Transmission Power	20 dBm
Bandwidth	20 MHz
Noise Power	−174 dBm/Hz
Number of Base Station (BS) Transmitting Antenna	20
Number of Mobile Station (MS) Receiving Antenna	1
Number of Paths	40
Path Attenuation	$PL=128.1+37.6\times 10\log \left(d_{km}\right)+6$
Vertical Half Beam Width $({\theta }_{3dB}$)	15°
Horizontal Half Beam Width $({\varnothing }_{3dB}$)	70°
Emission Angle Spread $({\sigma }_{BS}$)	7°
Incidence Angle Spread $({\sigma }_{MS}$)	10°
Transmitting Path Divergence Degree $(k_{BS}$)	5
Receiving Path Divergence Degree ($k_{MS}$)	5
Line-of-Sight Axis Elevation (${\theta }_{LoS}$)	103.2246° (Distance: 100 m) 98.9040° (Distance: 150 m) 96.7015° (Distance: 200 m) 95.3700° (Distance: 250 m)
Antenna Downtilt Angle $({\theta }_{tilt}$)	${\theta }_{LoS}$ for ideal condition 90°–125° for adaptive adjustment

and

Table 3. Simulation parameters for $60\times 4.$

Parameter	Value
Base Station (BS) Height (m)	25
Mobile Station (MS) Height (m)	1.5
Distance (m)	100, 150, 200, 250
Carrier Frequency	2.65 GHz
Transmission Power	20 dBm
Bandwidth	20 MHz
Noise Power	−174 dBm/Hz
Number of Base Station (BS) Transmitting Antenna	60
Number of Mobile Station (MS) Receiving Antenna	4
Number of Paths	40
Path Attenuation	$PL=128.1+37.6\times 10\log \left(d_{km}\right)+6$
Vertical Half Beam width (${\theta }_{3dB}$)	15°
Horizontal Half Beam Width (${\varnothing }_{3dB}$)	70°
Emission Angle Spread (${\sigma }_{BS}$)	7°
Incidence Angle Spread (${\sigma }_{MS}$)	10°
Transmitting Path Divergence Degree ($k_{BS}$)	5
Receiving Path Divergence Degree ($k_{MS}$)	5
Line-of-Sight Axis Elevation (${\theta }_{LoS}$)	103.2246° (Distance: 100 m) 98.9040° (Distance: 150 m) 96.7015° (Distance: 200 m) 95.3700° (Distance: 250 m)
Antenna Downtilt Angle (${\theta }_{tilt}$)	${\theta }_{LoS}$ for ideal condition 90°–125° for adaptive adjustment

, respectively.

3.2.1. Channel Capacity of 20 × 1 Antenna Configuration at ${\mathsf{\theta }}_{\mathrm{tilt}}={\mathsf{\theta }}_{\mathrm{LoS}}$

Figure 7 and

Table 4. For 20 × 1, ${\theta }_{tilt}={\theta }_{LoS},$ simulation results at D = 100 m, 150 m, 200 m, 250 m.

Distance (m)	Receiver SNR	Channel Capacity
100	39.6 dB	13.09 bps/Hz
150	33 dB	10.89 bps/Hz
200	28.3 dB	9.331 bps/Hz
250	24.6 dB	8.124 bps/Hz

show that D = 100 m (${\theta }_{tilt}={\theta }_{LoS}=$ 103.22°), the maximum channel capacity is 13.09 bps/Hz, and the SNR at the receiver is 39.6 dB; D = 150 m (${\theta }_{tilt}={\theta }_{LoS}=$ 99.904°), the maximum channel capacity is 10.89 bps/Hz, and the receiver SNR is 33 dB; D = 200 m (${\theta }_{tilt}={\theta }_{LoS}=$ 96.7015°), the maximum channel capacity is 9.331 bps/Hz, and the receiver SNR is 28.3 dB; D = 250 m (${\theta }_{tilt}={\theta }_{LoS}=$ 95.37°), the maximum channel capacity is 8.124 bps/Hz, and the receiver SNR is 24.6 dB. Depending on the distance, the longer the transmission distance, the lower the channel capacity due to the path attenuation.

3.2.2. Channel Capacity of 60 × 4 Antenna Configuration at ${\mathsf{\theta }}_{\mathrm{tilt}}={\mathsf{\theta }}_{\mathrm{LoS}}$

Figure 8 and

Table 5. For 60 × 4, ${\theta }_{tilt}={\theta }_{LoS}$, simulation results at D = 100 m, 150 m, 200 m, 250 m.

Distance (m)	Receiver SNR	Channel Capacity
100	157.6 dB	51.80 bps/Hz
150	131 dB	43.03 bps/Hz
200	112.3 dB	36.78 bps/Hz
250	7.8 dB	31.95 bps/Hz

show that D = 100 m (${\theta }_{tilt}={\theta }_{LoS}=$ 103.22°), the maximum channel capacity is 51.80 bps/Hz, and the SNR at the receiver is 157.6 dB; D = 150 m (${\theta }_{tilt}={\theta }_{LoS}=$ 99.904°), the maximum channel capacity is 43.03 bps/Hz, and the receiver SNR is 131 dB; D = 200 m (${\theta }_{tilt}={\theta }_{LoS}=$ 96.7015°), the maximum channel capacity is 36.78 bps/Hz, and the receiver SNR is 112.3 dB; D =250 m (${\theta }_{tilt}={\theta }_{LoS}=$ 95.37°), the maximum channel capacity is31.95 bps/Hz, and the receiver SNR is 7.8 dB. Due to the path attenuation, the longer the transmission distance, the lower the channel capacity.

4. Results and Discussion

The performance of the proposed method used in real circumstances without an ideal condition assumption will be shown and discussed in this section. In the following simulation, learning rate = 1, and the effect of different discount rate of Q-learning algorithm was mentioned.

The simulation results for two different antenna configurations of 20 × 1 and 60 × 4 by using a Q-learning algorithm to adaptively adjust the antenna downtilt angle to optimize the channel capacity in real circumstances are as follows.

4.1. Optimized Channel Capacity of 20 × 1 Antenna Configuration Based on $Adaptive{\theta }_{tilt}$

4.1.1. Optimized Channel Capacity and Optimized ${\theta }_{tilt}$ at D = 100 m

The simulation results from Figure 9 shows that, when the discount rate is equal to 0.9, the antenna downtilt angle found after optimization is closest to the ideal antenna downtilt angle, and the channel capacity is also the largest. From the comparison in

Table 6. Optimized values for antenna downtilt and channel capacity with different discount rates at D = 100 m.

$\mathit{\gamma }$	Optimization Angle	Error Rate	Optimized Capacity	Capacity Efficiency
0.9	103°	0.22%	13.08 bps/Hz	99.92%
0.5	115.5°	11.89%	11.51 bps/Hz	87.93%
0.1	119.5°	16.77%	10.45 bps/Hz	79.83%

, when the discount rate is equal to 0.9, the error between the optimized antenna downtilt angle and the optimal ideal antenna downtilt angle is 0.22%, which is the smallest error in all errors. Moreover, the optimized channel capacity can reach 99.92% of the optimal ideal channel capacity.

4.1.2. Optimized Channel Capacity and Optimized ${\theta }_{tilt}$ at D = 150 m

Figure 10 shows that, when the discount rate is equal to 0.9, the antenna downtilt angle found after optimization is closest to the ideal antenna downtilt angle, and the channel capacity is also the largest. From the comparison in

Table 7. Optimized values for antenna downtilt and channel capacity with different discount rates at D = 150 m.

$\mathit{\gamma }$	Optimization Angle	Error Rate	Optimized Capacity	Capacity Efficiency
0.9	99°	0.10%	10.89 bps/Hz	100%
0.5	104.5°	5.66%	10.53 bps/Hz	96.69%
0.1	116.5°	17.79%	7.86bps/Hz	72.18%

, when the discount rate is equal to 0.9, the error between the optimized antenna downtilt angle and the optimal ideal antenna downtilt angle is 0.1%, which is the smallest error in all errors. In addition, the optimized channel capacity can reach 100% of the optimal ideal channel capacity.

4.1.3. Optimized Channel Capacity and Optimized ${\theta }_{tilt}$ at D = 200 m

Figure 11 shows that, when the discount rate is equal to 0.9, the optimized antenna downtilt angle is closest to the optimal ideal antenna downtilt angle, and the channel capacity is also the largest. From the comparison in

Table 8. Optimized values for antenna downtilt and channel capacity with different discount rates at D = 200 m.

$\mathit{\gamma }$	Optimization Angle	Error Rate	Optimized Capacity	Capacity Efficiency
0.9	97°	0.31%	9.313 bps/Hz	99.81%
0.5	103°	6.51%	8.883bps/Hz	95.2%
0.1	92°	4.86%	9.064 bps/Hz	97.14%

, when the discount rate is equal to 0.9, the error between the optimized antenna downtilt angle and the optimal ideal antenna downtilt angle is 0.31%, which is the smallest error of all errors. Moreover, the optimized channel capacity can reach 99.81% of the optimal ideal channel capacity.

4.1.4. Optimized Channel Capacity and Optimized ${\theta }_{tilt}$ at D = 250 m

Figure 12 shows that, when the discount rate is equal to 0.9, the optimized antenna downtilt angle is closest to the optimal ideal antenna downtilt angle, and the channel capacity is also the largest. From the comparison in

Table 9. Optimized values for antenna downtilt and channel capacity with different discount rates at D = 250 m.

$\mathit{\gamma }$	Optimization Angle	Error Rate	Optimized Capacity	Capacity Efficiency
0.9	95°	0.39%	8.109 bps/Hz	99.82%
0.5	91°	4.58%	7.885 bps/Hz	97.06%
0.1	107°	4.86%	6.702 bps/Hz	82.50%

, when the discount rate is equal to 0.9, the error between the optimized antenna downtilt angle and the optimal ideal antenna downtilt angle is 0.39%, which is the smallest error of all errors. Moreover, the optimized channel capacity can reach 99.82% of the optimal ideal channel capacity.

4.2. Optimized Channel Capacity of 60 × 4 Antenna Configuration Based on Adaptive ${\theta }_{tilt}$

4.2.1. Optimized Channel Capacity and Optimized ${\theta }_{tilt}$ at D = 100 m

The simulation results from Figure 13 show that, when the discount rate is equal to 0.9, the antenna downtilt angle found after optimization is closest to the optimal ideal antenna downtilt angle, and the channel capacity is also the largest. From the comparison in

Table 10. Optimized values for antenna downtilt and channel capacity with different discount rates at D = 100 m.

$\mathit{\gamma }$	Optimization Angle	Error Rate	Optimized Capacity	Capacity Efficiency
0.9	103°	0.22%	51.75 bps/Hz	99.90%
0.5	115.5°	11.89%	45.25 bps/Hz	87.36%
0.1	119°	15.23%	41.25 bps/Hz	79.63%

, when the discount rate is equal to 0.9, the error between the optimized antenna downtilt angle and the optimal ideal antenna downtilt angle is 0.22%, which is the smallest error of all errors, and its optimized channel capacity can reach 99.90% of the optimal ideal channel capacity.

4.2.2. Optimized Channel Capacity and Optimized ${\theta }_{tilt}$ at D = 150 m

The simulation results from Figure 14 show that, when the discount rate is equal to 0.9, the antenna downtilt angle found after optimization is closest to the optimal ideal antenna downtilt angle, and the channel capacity is also the largest. From the comparison in

Table 11. Optimized values for antenna downtilt and channel capacity with different discount rates at D = 150 m.

$\mathit{\gamma }$	Optimization Angle	Error Rate	Optimized Capacity	Capacity Efficiency
0.9	99°	0.90%	42.95 bps/Hz	99.81%
0.5	104.5°	4.60%	41.51 bps/Hz	96.47%
0.1	119°	19.11%	27.23 bps/Hz	63.28%

, when the discount rate is equal to 0.9, the error between the optimized antenna downtilt angle and the optimal ideal antenna downtilt angle is 0.90%, which is the smallest error of all errors, and its optimized channel capacity can reach 99.81% of the optimal ideal channel capacity.

4.2.3. Optimized Channel Capacity and Optimized ${\theta }_{tilt}$ at D = 200 m

The simulation results from Figure 15 shows that, when the discount rate is equal to 0.9, the antenna downtilt angle found after optimization is closest to the optimal ideal antenna downtilt angle, and the channel capacity is also the largest. From the comparison in

Table 12. Optimized values for antenna downtilt and channel capacity with different discount rates at D = 200 m.

$\mathit{\gamma }$	Optimization Angle	Error Rate	Optimized Capacity	Capacity Efficiency
0.9	97°	0.31%	36.71 bps/Hz	99.81%
0.5	105°	8.58%	33.69 bps/Hz	91.60%
0.1	112°	15.82%	26.84 bps/Hz	72.97%

, when the discount rate is equal to 0.9, the error between the optimized antenna downtilt angle and the optimal ideal antenna downtilt angle is 0.31%, which is the smallest error, and the optimized channel capacity can reach 99.81% of the optimal ideal channel capacity.

4.2.4. Optimized Channel Capacity and Optimized ${\theta }_{tilt}$ at D = 250 m

The simulation results from Figure 16 shows that, when the discount rate is equal to 0.9, the antenna downtilt angle found after optimization is closest to the optimal ideal antenna downtilt angle, and the channel capacity is also the largest. From the comparison in

Table 13. Optimized values for antenna downtilt and channel capacity with different discount rates at D = 250 m.

$\mathit{\gamma }$	Optimization Angle	Error Rate	Optimized Capacity	Capacity Efficiency
0.9	96°	0.66%	31.87 bps/Hz	99.75%
0.5	93.5°	1.96%	31.73 bps/Hz	99.37%
0.1	111.5°	16.91%	21.11 bps/Hz	66.07%

, when the discount rate is equal to 0.9, the error between the optimized antenna downtilt angle and the optimal ideal antenna downtilt angle is 0.66%, which is the smallest error, and the optimized channel capacity can reach 99.75% of the optimal ideal channel capacity when the discount rate is 0.9.

5. Conclusions

This article verified the effect of antenna downtilt angle on channel capacity that a little deviation from an optimum angle will decrease the channel capacity significantly. In addition, this paper proposed an adaptive optimization method by applying the Q-learning algorithm to adaptively optimize the antenna downtilt angles to maximize system capacity for real circumstances. Simulation results show that the proposed method could adaptively adjust the antenna downtilt angle to obtain the channel capacity of more than 99.72% of the ideal optimal one on ideal condition assumption when the discount rate of Q-learning algorithm is 0.9.