Multi-Camera Networks for Coverage Control of Drones

Abstract

Multiple unmanned multirotor (MUM) systems are becoming a reality. They have a wide range of applications such as for surveillance, search and rescue, monitoring operations in hazardous environments and providing communication coverage services. Currently, an important issue in MUM is coverage control. In this paper, an existing coverage control algorithm has been extended to incorporate a new sensor model, which is downward facing and allows pan-tilt-zoom (PTZ). Two new constraints, namely view angle and collision avoidance, have also been included. Mobile network coverage among the MUMs is studied. Finally, the proposed scheme is tested in computer simulations.

1. Introduction

Unmanned aerial vehicle (UAV) technology is currently a growing area. It offers many potential civil applications, inspiring scientists to undertake the development of new algorithms to automate UAV systems. Employing multiple unmanned multirotors (MUMs) [1,2,3,4,5,6,7,8] is rapidly becoming possible due to the development of computer hardware and communication technology. The use of UAVs is advantageous when compared to a single one. For example, when a task is very difficult, a single UAV may take a long time or may not be able to accomplish it effectively. The research challenge is then to develop the appropriate cooperation logic so that the UAVs work together to complete missions effectively and efficiently.

Coverage control is attracting research interest in MUMs [9]. The basic principle is to drive MUMs to optimal coverage for a given environment. The early works in coverage control address the visibility problem [10] or the Watchmen Tour Problem (WTP) [11], which determines the optimal number of guards and their routes, respectively, to observe a given area. These approaches form the primary research in this area, but they are not suitable for real applications. It is necessary to real world constraints in the development of coverage algorithms for MUMs. Cortés et al. [12,13] have proposed some approaches for coverage control, which are based on the Voronoi algorithm for MUMs. Here, the Voronoi algorithm is a decentralized iterative scheme to partition a 2D plane into several cells. Thereafter, the coverage problem received greater attention and many methods [4,14,15,16] have been proposed. Here, we give several examples. Schwager et al. [14,17] presented a distributed algorithm for dealing with the coverage, which controls MUMs to implement their coverage by configuring position and pan and tilt parameters. Piciarelli et al. [15] considered a fixed camera network and proposed a camera PTZ reconfiguration for coverage control with considering a relevance parameter, where an ellipse sensing field of view (FOV) is assumed. Parapari et al. [4] presented a distributed collision avoidance control for MUMs, but no PTZ configuration. Wang and Guo [16] handled the coverage problem by deriving a distributed control from a potential function, but no PTZ configuration. It should be noted that the result of Schwager et al. [17] is significant since the sensor used is the downward facing camera which is a practical configuration. The difficulty in this approach is that the sensing field of view is an arbitrary convex polygon of four sides. Most results do not consider the case of MUMs with downward facing cameras and arbitrary trapezoidal FOV flying in the sky. However, the camera focal parameter was not considered for configuration in that paper. In order to minimize mobility and hovering MUMs over the monitoring area, a simultaneous configuration of PTZ parameters is an important topic in coverage control. In addition, sometime a sensor would face an obstacle (e.g., a tree). Without the consideration of this situation in the coverage control, the sensor view may be occluded by dynamic or static obstacles. Thus, the view angle parameter is also important and should be considered. In a later work, Arslan et al. [18] presented a circular image sensor and developed an algorithm of configuring PTZ parameters. Unfortunately, the result of Arslan et al. [18] is based on a circular FOV, but the actual FOV is a trapezoidal one. In addition, Arslan et al. [18] use a fixed camera network to configure PTZ parameters without tuning the cameras’ positions.

In this paper, we develop a distributed coverage control approach for MUMs with downward facing cameras. The proposed approach is based on the result of Schwager et al. [17]. A modified camera model is adopted in this paper. Based on this model, we extend Schwager et al.’s method to include more control variables in coverage control. The following features are improved compared with Schwager et al.’s method: (1) camera sensor model close to actual one; (2) configured rotation and PTZ parameters simultaneously; (3) imposed view angle constraints; (4) controller that can avoid collision among UAVs; (5) network convergence with more control variables and guaranteed collision avoidance.

Originally, a primary version of this paper was published in ICUAS 2018 [19]. We extend [19] to more contents: (1) the stability analysis is given for the view angle control; (2) we extend the coverage algorithm by incorporating with collision avoidance and the stability analysis is given; (3) more examples are tested in the simulation.

The paper is structured as follows: in Section 2, problem formulation and our research objectives are briefly described. In Section 3, the solution of a distributed coverage control with consideration of the unmanned aerial vehicle (UAV) position, rotation and PTZ parameters, and view angle constraint, is given. The simulation study is given in Section 4. Finally, the conclusion is given in Section 5.

2. Problem Statements

In this section, we describe the problem of coverage control of MUMs with downward facing cameras. We first define the environment and the camera model and then give the coverage control objective.

2.1. Environment

Let Q be a bounded environment $Q\subset R^2$. The degree of interest of the area in its interior is represented by a density function $\phi \left(q\right)$ where $q\in Q$.

2.2. Camera Sensor Model

We consider n UAVs in an environment Q. Each UAV with a downward facing camera is located in position:

$$\begin{array}{c}\hfill p_i={[x_i,y_i,z_i]}^T.\end{array}$$

Figure 1 shows the camera model, where the center of the camera is at the center of the lens; f is the focal length of the lens; $L_l$ is the length of the camera image; $L_w$ is the width of the camera image; each camera has a rectangular field of view (FOV), which is the intersection of the cone of the camera lens with the environment Q. There are two coordinate systems in the coverage problem: one is the camera coordinates (CC) of UAV i, and the other one is the global coordinates (GC). The CC is at the center of the camera lens, with the z-axis pointing downward facing to the ground through the lens, while the GC is fixed on the ground, with the z-axis points upward normal to the ground. In CC, the camera view is like a pyramid object whose base is a rectangle shape and whose sides are triangles uniting at a common apex, the center of the camera lens. The sides of the pyramid have the outward normal vectors $e_1,e_2,e_3,e_4$. Here, we derive the first outward normal vector $e_1$. The side view of the camera with pyramid FOV is shown in Figure 2. From the figure, it is known that two triangles $\Delta v_{1}o{v}_2$ and $\Delta v_{3}o{v}_4$ are similar since the vector $e_1$ and the line $o{v}_3$ are perpendicular. Thus, the projection of the normal vector $e_1$ on the y-axis is $o{v}_2$ and its value is f, while the rejection of the normal vector $e_1$ from the y-axis is $v_1{v}_2$ and its value is $-\frac{L_l}{2}$. Therefore, the normal vector $e_1$ in the CC system is $[0f-\frac{L_l}{2}]$. We are interested in the unit normal vector and thus it follows that:

$$\begin{array}{cc}\hfill e_1& ={[0\frac{2f}{\sqrt{L_l^2+4f^2}}-\frac{L_l}{\sqrt{L_l^2+4f^2}}]}^T\end{array}$$

(1)

In a similar way, we can obtain the other remained three outward normal vectors:

$$\begin{array}{ccc}\hfill e_2& =& {[\frac{2f}{\sqrt{L_w^2+4f^2}}0-\frac{L_w}{\sqrt{L_w^2+4f^2}}]}^T\hfill \end{array}$$

(2)

$$\begin{array}{ccc}\hfill e_3& =& {[0-\frac{2f}{\sqrt{L_l^2+4f^2}}-\frac{L_l}{\sqrt{L_l^2+4f^2}}]}^T\hfill \end{array}$$

(3)

$$\begin{array}{ccc}\hfill e_4& =& {[\frac{2f}{\sqrt{L_w^2+4f^2}}0-\frac{L_w}{\sqrt{L_w^2+4f^2}}]}^T\hfill \end{array}$$

(4)

For the GC system, we want to transform it to get a CC system, which can be represented in 3D rotation matrix form as R. This matrix can be implemented by using two elementary transforms, each matrix realizes a rotation. This is achieved by rotating the z-axis by $\pi /2$ whose transformation is given by:

$$\begin{array}{c}\hfill R_i\left(\frac{\pi }{2}\right)=\left[\begin{array}{ccc}cos(\pi /2)& sin(\pi /2)& 0\\ -sin(\pi /2)& cos(\pi /2)& 0\\ 0& 0& 1\end{array}\right],\end{array}$$

(5)

and then turning the x-axis over by $\pi$ whose transformation is given by:

$$\begin{array}{c}\hfill R_i\left(\pi \right)=\left[\begin{array}{ccc}1& 0& 0\\ 0& cos\left(\pi \right)& sin\left(\pi \right)\\ 0& -sin\left(\pi \right)& cos\left(\pi \right)\end{array}\right],\end{array}$$

(6)

We can express this transformation by:

$$\begin{array}{c}\hfill R_i^{\frac{\pi }{2}\to \pi }=R_i\left(\pi \right)R_i\left(\frac{\pi }{2}\right)=\left[\begin{array}{ccc}0& 1& 0\\ 1& 0& 0\\ 0& 0& -1\end{array}\right],\end{array}$$

(7)

Finally, for a 3D point v in GC, we can transform the vector $(v-p_i)$ into the CC system by the transformation $R_i^{\frac{\pi }{2}\to \pi }$, that is $R_i^{\frac{\pi }{2}\to \pi }(v-p_i)$. For a $q\in Q$, the vector $I_{32}q-p$ (where $I_{32}=\left[\begin{array}{cc}1& 0\\ 0& 1\\ 0& 0\end{array}\right]$) expressed in CC is given by $R_i^{\frac{\pi }{2}\to \pi }(I_{32}q-p)$.

Now, we are concerned about the pyramid FOV, as shown in Figure 1. This pyramid has four sides and they intersect with the ground to get four edges of the camera FOV. Each edge is defined as $l_k,k=1,2,3,4$. If the point q is on the edge $l_k$ of the camera FOV, then:

$$\begin{array}{c}\hfill e_k^T{R}_i^{\frac{\pi }{2}\to \pi }(I_{32}q-p_i)=0,k=1,2,3,4\end{array}$$

since the outward normal vector $e_i$ and the vector $R_i^{\frac{\pi }{2}\to \pi }(I_{32}q-p_i)$ are perpendicular. Thus, for a given $q\in Q$, the camera FOV is represented by:

$$\begin{array}{c}\hfill B_i=\{q:\left[\begin{array}{c}{e}_{1i}^T{R}_i^{\frac{\pi }{2}\to \pi }(I_{32}q-p_i)\le 0\\ e_{2i}^T{R}_i^{\frac{\pi }{2}\to \pi }(I_{32}q-p_i)\le 0\\ e_{3i}^T{R}_i^{\frac{\pi }{2}\to \pi }(I_{32}q-p_i)\le 0\\ e_{4i}^T{R}_i^{\frac{\pi }{2}\to \pi }(I_{32}q-p_i)\le 0\end{array}\right]\}.\end{array}$$

(8)

The observed area A from the camera, which is a rectangle form is given by:

$$\begin{array}{c}\hfill A=z^2\frac{L_l{L}_w}{f^2}\end{array}$$

which is obtained directly from Figure 1. The number of pixels are given by $N_l$ and $N_w$. The area/pixel is given by:

$$\begin{array}{c}\hfill \frac{areaA}{pixel}=\frac{L_l{L}_w{z}^2}{N_l{N}_w{f}^2}.\end{array}$$

(9)

Thus, we have a function:

$$\begin{array}{c}\hfill g(p_i,q)=\left\{\begin{array}{c}\frac{L_{li}{L}_{wi}{z}_i^2}{N_{li}{N}_{wi}{f}_i^2},\mathrm{for}q\in B_i\hfill \\ \infty ,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(10)

Remark 1.

It should be noted that the area/pixel is different from the result of Schwager et al. [17], where Schwager et al. used a magnification factor to represent the area observed from the camera, while our expression is more practical by using area and pixel resolution. The proposed sensor model also differs from that of Schwager et al.’s work. The center of the UAV (or camera) is defined at the center of the camera lens in our model, while the center of the camera in Schwager et al.’s work was defined at the center of the camera focal. If the zoom parameter f is configured based on Schwager et al.’s work, this will result in the changes of the UAV altitude (it is redundant to adjust the zoom parameter in this situation). Whereas, the configuration based on our sensor model will not affect the UAV altitude. In addition, Schwager et al.’s work does not consider the size of the camera image, which affects the FOV.

2.3. Wireless Camera Networks

In this paper, the coverage algorithm is applicable to the wireless camera networks formed of cameras with PTZ. This implies that each UAV is communicated with other UAVs. The main assumptions of this network are described below.

• Each camera can obtain its position and velocity by its GPS built in UAV.
• Each camera can obtain its FOV information by its camera image.
• The camera network is connected; therefore, each camera has a sensing and communication capability or multi-hop transmission capacity to transmit related information to neighbor cameras.
  It should be noted that in the present wireless camera network, (a) we need to transmit relevant FOV information to adjacent cameras to find FOV overlap, it is not necessary to transmit all FOVs of cameras in the environment. Therefore, the information broadcast can be set like two or three hops before it stops rebroadcasting (UDP protocol can be used). (b) It needs to communicate with all of the cameras to obtain only the position information of other UAVs.

2.4. Optimization Coverage Problem

For the multi-UAV coverage control, we have to design a cost function that can be used for evaluating the coverage performance. It is a natural way to use the information, as discussed in the previous section. Defining the cost function (see [17]) as our control objective is given by:

$$\begin{array}{c}\hfill \mathcal{J}={\int }_Q{(\sum _{i=1}^{n}g{(p_i,q)}^{-1}+{{\rm Y}}^{-1})}^{-1}dq\end{array}$$

(11)

where ${\rm Y}>0$ is a positive constant. This constant is used to avoid the integration part as zero due to $g{(p_i,q)}^{-1}$. It may be interpreted as baseline information of the environment. The optimization coverage problem of multi-UAVs is to minimize the objective function (11).

3. Distributed Coverage with Configuration of Position, Yaw and PTZ

Our goal is to develop a distributed control for the coverage problem, where each UAV exchanges information with its neighboring UAVs. Thus, the area can be represented by:

$$\begin{array}{c}\hfill J_{Nq}={(\sum _{i\in N_q}g{(p_i,q)}^{-1}+{{\rm Y}}^{-1})}^{-1}\end{array}$$

(12)

where $N_q$ is the camera set described by:

$$\begin{array}{c}\hfill N_q=\{i:q\in B_i,i=1,2,...n\}.\end{array}$$

The objective function becomes:

$$\begin{array}{c}\hfill \mathcal{J}={\int }_Q{J}_{Nq}\phi \left(q\right)dq\end{array}$$

(13)

When the UAV heading yaw and pan-tilt-zoom (PTZ) cameras are considered in UAV camera networks, many problems arise. One of the problems is how to achieve the coverage by adjusting position, yaw (rotation) and PTZ parameters in a MUM system. Since the camera FOV affects coverage, any strategy for coverage control must be associated with cameras PTZ parameters. In this section, we develop the control law to consider such a general case, the configuration of position, yaw (rotation) and PTZ cameras. Consider the state of the UAV i with gimbal camera to be given by:

$$\begin{array}{c}\hfill {\chi }_i={[x_i,y_i,z_i,{\psi }_i^r,{\psi }_i^p,{\psi }_i^t,f_i]}^T,\end{array}$$

where ${\psi }_i^r,{\psi }_i^p,$ and ${\psi }_i^t$ are the rotation (or yaw), pan and tilt angles, respectively, and $f_i$ is the focal length of the camera i. Usually, the rotation (yaw) is controlled by UAV, while the PTZ parameters are controlled by a gimbal camera system. We assume that both the UAV and camera are considered as one particle (camera lens representing this point) and their positions are the same one represented by this point. As explained in Section 2, the center of the UAV position is the camera lens. Consider the state equation of the camera i, that is:

$$\begin{array}{ccc}\hfill {\dot{{\chi }_i}}_i& =& u_i\hfill \end{array}$$

(14)

where:

$$\begin{array}{c}\hfill u_i={[u_{xi},u_{yi},u_{zi},u_{{\psi }_i^r},u_{{\psi }_i^p},u_{{\psi }_i^t},u_{f_i}]}^T.\end{array}$$

Let $p_i={[x_i,y_i,z_i]}^T$ be the position of the camera i at the center of the lens (it is also the position of UAV). It should be noticed that for each UAV their dynamic is independent. For neighboring UAVs, the wireless communication network is assumed to be connected during the mission.

As shown in Section 2, a rectangular FOV on the GC coordinates is converted into one which is based on the CC system, by rotating a matrix $R_i^{\frac{\pi }{2}\to \pi }$. Sometimes we have to configure the yaw angle such that the coverage is achieved optimally. In this situation, a rotation matrix for rotating the yaw (rotation) angle is introduced as described below:

$$\begin{array}{c}\hfill Rotationaboutz-axis\left(yawangle\right)\\ \hfill R_{yaw}\left({\psi }_i^r\right)=\left[\begin{array}{ccc}\cos {\psi }_i^r& \sin {\psi }_i^r& 0\\ -\sin {\psi }_i^r& \cos {\psi }_i^r& 0\\ 0& 0& 1\end{array}\right].\end{array}$$

(15)

Similarly, we can tune pan and tilt angles to control the camera to get a better FOV on the ground. The rotation matrices corresponding to the pan and tilt transformation are given by:

$$\begin{array}{c}\hfill Rotationaboutx-axis\left(panangle\right)\\ \hfill R_{pan}\left({\psi }_i^p\right)=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos {\psi }_i^p& \sin {\psi }_i^p\\ 0& -\sin {\psi }_i^r& \cos {\psi }_i^r\end{array}\right],\end{array}$$

(16)

and:

$$\begin{array}{c}\hfill Rotationabouty-axis\left(tiltangle\right)\\ \hfill R_{tilt}\left({\psi }_i^t\right)=\left[\begin{array}{ccc}\cos {\psi }_i^t& 0& -\sin {\psi }_i^t\\ 0& 1& 0\\ \sin {\psi }_i^t& 0& \cos {\psi }_i^t\end{array}\right],\end{array}$$

(17)

respectively. Finally, for a 3D point v in a GC system, we can transform the vector $(v-p_i)$ into the CC system by rotating $R_i$, which is given by:

$$\begin{array}{c}\hfill R_i=R_{tilt}\left({\psi }_i^t\right)R_{pan}\left({\psi }_i^p\right)R_{yaw}\left({\psi }_i^r\right)R_i^{\frac{\pi }{2}\to \pi }\end{array}$$

(18)

For a point q inside FOV, we can express the vector $(I_{32}q-g_i)$ in the CC system by $R_i(I_{32}q-p_i)$. Next, we are concerned about the FOV cone, as shown in Figure 1. By the transformation, the camera FOV shown in (8) in the CC system is changed to:

$$\begin{array}{c}\hfill B_i=\{q:\left[\begin{array}{c}{e}_{1i}^T{R}_i(I_{32}q-p_i)\\ e_{2i}^T{R}_i(I_{32}q-p_i)\\ e_{3i}^T{R}_i(I_{32}q-p_i)\\ e_{4i}^T{R}_i(I_{32}q-p_i)\end{array}\right]\le 0\}\end{array}$$

(19)

It should be noted that when tuning yaw or PTZ parameters, FOV is not a rectangular shape, as shown in Section 2, but it is a trapezium or trapezoid in this transformation $R_i$. Thus, the area of such FOV shape can be viewed as a mapping defined by $C_{fov}$ from the image area $L_l{L}_w$ to an area of a trapezium FOV on the ground, that is expressed as $C_{fov}:L_l{L}_w\to FOV$. Therefore, the $\frac{area}{pixel}$ in (9) should be changed to:

$$\begin{array}{c}\hfill \frac{area}{pixel}=\frac{C_{fov}}{N_l{N}_w{f}^2}{\Vert I_{32}q-p_i\Vert }^2,\end{array}$$

(20)

where $\Vert I_{32}q-p_i\Vert$ is the the distance between the camera and the point q inside FOV. Now, due to the use of yaw and PTZ cameras, the function $g(p_i,q)$ is changed to:

$$\begin{array}{c}\hfill g(p_i,q)=\left\{\begin{array}{c}\frac{C_{fov}\left(L_{li}{L}_{wi}\right)}{N_{li}{N}_{wi}{f}_i^2}{\Vert I_{32}q-p_i\Vert }^2,\mathrm{for}q\in B_i\hfill \\ \infty ,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(21)

Consider the state equation of UAV i with the PTZ camera, that is:

$$\begin{array}{ccc}\hfill {\dot{{\chi }_i}}_i& =& u_i\hfill \end{array}$$

(22)

$$\begin{array}{ccc}\hfill u_i& =& -\rho \frac{\partial \mathcal{J}}{\partial {\chi }_i},{\chi }_i=[p_i,{\psi }_i^r,{\psi }_i^p,{\psi }_i^t,f_i]\hfill \end{array}$$

(23)

where $\rho >0$ is a factor, and the gradient $\frac{\partial \mathcal{J}}{\partial {\chi }_i}$ (see Appendix A) is given by:

$$\begin{array}{c}\hfill \frac{\partial \mathcal{J}}{\partial p_i}=\sum _{k=1}^4\underset{Q\cap l_{ki}}{\int }(J_{Nq}-J_{Nq\setminus \left\{i\right\}})\phi \left(q\right)\frac{R_i^T{e}_{ki}}{\Vert I_{23}{R}_i^T{e}_{ki}\Vert }dq\\ \hfill -\underset{Q\cap B_i}{\int }\frac{2N_{li}{N}_{wi}{f}_i^2{J}_{Nq}^2(I_{32}q-p_i)}{L_{li}{L}_{wi}{\Vert I_{32}q-p_i\Vert }^4}\phi \left(q\right)dq,\end{array}$$

(24)

$$\begin{array}{c}\hfill \frac{\partial \mathcal{J}}{\partial {\psi }_i^w}=\sum _{k=1}^4\underset{Q\cap l_{ki}}{\int }(J_{Nq}-J_{Nq\setminus \left\{i\right\}})\frac{\phi \left(q\right)e_{ki}^T\frac{\partial R_i}{\partial {\psi }_i^w}(p_i-I_{32}q)}{\Vert I_{23}{R}_i^T{e}_{ki}\Vert }dq\end{array}$$

$$\begin{array}{c}\hfill w\in \{r,p,t\}\end{array}$$

(25)

where $\frac{\partial R_i}{\partial {\psi }_i^w}$ can be obtained by differentiating $R_i$ with respect to ${\psi }_i^w$ directly, and:

$$\begin{array}{cc}\hfill \frac{\partial \mathcal{J}}{\partial f_i}=& \sum _{k=1}^4\underset{Q\cap l_{ki}}{\int }(J_{Nq}-J_{Nq\setminus \left\{i\right\}})\frac{\phi \left(q\right)\frac{\partial e_{ki}^T}{\partial f_i}{R}_i(p_i-I_{32}q)}{\Vert I_{23}{R}_i^T{e}_{ki}\Vert }dq\\ & -\underset{Q\cap B_i}{\int }\frac{2J_{Nq}^2{f}_i{N}_l{N}_w}{L_l{L}_w\left|\right|I_{32}q-p_i{\left|\right|}^2}\phi \left(q\right)dq.\end{array}$$

(26)

It is observed from (26) that if we fix the parameters $x,y,z$, the updating parameter f will not affect the position of UAV. This implies that we can hover the UAV at a certain position and tune PTZ parameters to get the optimal coverage. The factor $\rho$ is used to affect the convergence to the coverage. Large values of $\rho$ can speed up the convergence. However, it cannot be increased to a very large value due to the control limits.

The following theorem is given for configuring the parameters $x_i,y_i,z_i,{\psi }_i^r,{\psi }_i^p,{\psi }_i^t,f_i$ simultaneously.

Theorem 1.

In a network of n UAVs governed by dynamics (22) with downward facing cameras, the gradient of the cost function $\mathcal{J}$ with respect to the state variables ${\chi }_i={[x_i,y_i,z_i,{\psi }_i^r,{\psi }_i^p,{\psi }_i^t,f_i]}^T$ is given by (24)–(26).

The proposed coverage control is a decentralized form. Based on the gradient descent, the configuration of the control variables is an iterative process, as shown in Figure 3. The iteration is terminated when the desired coverage rate is reached, where the coverage rate is $\frac{multipleUAVscoverageoftheareaofinterest}{areaofinterest}$.

Remark 2.

In [17], the configuration of the focal parameter f is not considered. In the proposed algorithm, we can configure the position $(x,y,z)$, yaw and PTZ parameters. This improves the results of [17].

3.1. Incorporating View Angle

In a practical situation, sometimes a sensor (e.g., solid-state LiDAR) faces an obstacle (e.g., a tree), which may be between the sensor and the target and it can not capture the object image even if the target is within the view angle coverage of the sensor; this is known as an occlusion. If we consider the view angle as a target feature, this implies that the coverage must include this feature. Without the consideration of this feature in the coverage control, the target may be occluded by dynamic or static objects. For example, we hope the camera i to rotate its angle to 30 degrees so that we can see the observed region clearly. This is a reference following control problems.

Let us consider this situation in coverage control. Assume that we require the UAV to follow a desired view angle ${\psi }_i^{wD}$, $w\in \{r,p,t\}$. The controller design should be to achieve this objective, i.e., ${\psi }_i^w$ approaches the desired ${\psi }_i^{wD}$ as soon as possible. Let $\widetilde{{\psi }_i^w}={\psi }_i^{wD}-{\psi }_i^w$. We propose the following controller:

$$\begin{array}{c}\hfill u_{{\psi }_i^w}={\dot{{\psi }_i}}^{wD}+k_{pi}^w\widetilde{{\psi }_i^w}\end{array}$$

(27)

where $k_{pi}^w>0$ is the constant control gain, which can be determined by users.

Assumption A1.

The desired angle ${\psi }_i^{wD},w\in \{r,p,t\}$ is smooth and bounded, and its derivative ${\dot{{\psi }_i}}^{wD}$ is also bounded.

Rearranging the state equations of the camera i, we get:

$$\begin{array}{cc}\hfill {\dot{\eta }}_i& =u_{\eta i}\end{array}$$

(28)

$$\begin{array}{cc}\hfill {\dot{\psi }}_i^w& =u_{{\psi }_i^w},w\in \{r,p,t\}.\end{array}$$

(29)

where ${\eta }_i = {[x_i,y_i,z_i,f_i]}^T,u_{\eta i} = {[u_{xi},u_{yi},u_{zi},u_{fi}]}^T,{\psi }_i^w = {[{\psi }_i^r,{\psi }_i^p,{\psi }_i^t]}^T$ and $u_{{\psi }_i^w} = {[u_{{\psi }_i^r},u_{{\psi }_i^p},u_{{\psi }_i^t}]}^T$.

We have the following theorem to ensure the stability.

Theorem 2.

In a network of n UAVs governed by dynamics (22) with downward facing cameras, the control laws $u_{xi},u_{yi},u_{zi},u_{fi}$ are designed by (23), while the control laws $u_{{\psi }_i^w},w\in \{r,p,t\}$ are designed by (27). If Assumption 1 holds, then:

(1) the proposed control laws lead to a bounded coverage of $\mathcal{J}$;

(2) ${lim}_{t\to \infty }\widetilde{{\psi }_i^w}=0$.

Proof. 

See Appendix A. □

Remark 3.

For a camera network, it is possible that some UAVs use the position and PTZ control with/without considering view angle. In this situation, those UAVs without considering view angle are referred to as zero view angle control, i.e., the desired view angle is set to ${\psi }_i^{wD}=0$. Thus, we have a similar result as in Theorem 2.

Remark 4.

The stability in Theorem 2 ensures that (1) the coverage is bounded when applying the proposed coverage control with view angle; (2) it is converged to the desired view angle.

3.2. Incorporating Collision Avoidance

Collision avoidance is a practical problem in multi-UAV networks. So far, the control design does not consider collisions with other UAVs when implementing the coverage objective. In what follows, we incorporate a potential function to design a coverage control with autonomously avoiding collisions.

Define a distance as $d_{ij}=\left|\right|p_i-p_j\left|\right|,i,j=1,2,...n,i\ne j$. Let ${\Gamma }_i$ be the set of indices of cameras of the ith UAV for which $d_{ij}$ is within certain regions; that is, ${\Gamma }_i=\{j:r<d_{ij}\le R,j=1,2,...n,j\ne i\}$. The view of $r<d_{ij}\le R$ is shown in Figure 4, where r and R ($R>r>0$) are the radii of the avoidance and detection regions. The potential function [20] based on n UAVs is given by:

$$\begin{array}{c}\hfill V_{P_i}=\sum _{j=1,j\ne i}^n{v}_{P_{ij}}\end{array}$$

(30)

where:

$$\begin{array}{c}\hfill v_{P_{ij}}={(\min \left\{0,\frac{d_{ij}^2-R^2}{d_{ij}^2-r^2}\right\})}^2\end{array}$$

(31)

This is a repulsive potential function. The partial derivative of $v_{P_{ij}}$ with respect to $p_i={[x_i,y_i,z_i]}^T$ is given by:

$$\begin{array}{c}\hfill \frac{\partial v_{P_{ij}}^T}{\partial p_i}=\left\{\begin{array}{cc}0,& if{d}_{ij}\ge R\\ 4\frac{(R^2-r^2)(d_{ij}^2-R^2)}{{(d_{ij}^2-r^2)}^3}{(p_i-p_j)}^T,& ifr<d_{ij}<R\\ 0,& if{d}_{ij}<r\end{array}\right.\end{array}$$

(32)

Rearrange the state equations of the UAV i as:

$$\begin{array}{cc}\hfill {\dot{p}}_i& =u_{pi}\end{array}$$

(33)

$$\begin{array}{cc}\hfill {\dot{\beta }}_i& =u_{{\beta }_i},\end{array}$$

(34)

where ${\beta }_i={[{\psi }_i^r,{\psi }_i^p,{\psi }_i^t,f_i]}^T$.

Consider the following performance criterion:

$$\begin{array}{c}\hfill V=\mathcal{J}+\frac{1}{2}\sum _{i=1}^n{V}_{P_i}\end{array}$$

(35)

The following theorem is given to establish a stability result.

Theorem 3.

In a network of n UAVs governed by dynamics (22) with downward facing cameras, the control laws are given by:

$$\begin{array}{ccc}\hfill u_{pi}& =& -\rho (\frac{\partial \mathcal{J}}{\partial p_i}+\sum _{j=1,j\ne i}^n\frac{\partial v_{P_{ij}}}{\partial p_j})\hfill \end{array}$$

(36)

$$\begin{array}{ccc}\hfill u_{{\beta }_i}& =& -\rho \frac{\partial \mathcal{J}}{\partial {\beta }_i},\hfill \end{array}$$

(37)

with (32), (A9), (A17) and (A18). If the initial configuration such that $d_{ij}\left(0\right)>r$, then:

(1) The proposed control laws lead to a bounded coverage of $\mathcal{J}$;

(2) The collision among UAVs is avoided.

Proof. 

See Appendix A. □

Remark 5.

If we incorporate the view angle control, in this situation, we have a similar conclusion as in Theorem 3.

4. Simulation Studies

Simulation tests of the proposed coverage controls are given in this section. The HP computer with intel (R) Core(TM) i7-4770 CPU was used. The operation system was window 10. MATLAB software was used for simulations. In all case studies, the environment Q was the same as in the rectangle form $[-200,200]\times [-200,200]\subset R^2$. All UAVs were identical with the camera model, as shown in Figure 1. The parameters of the camera image sensor were $L_l=3$ mm, $L_w=5$ mm, $N_l=640$ pixels, $N_v=480$ pixels and ${\rm Y}$ was chosen as 1000.

4.1. Case 1

We considered 2 UAVs for coverage control. The sensor model in Section 2.2 was used in a rectangle form. The configuration of the camera network included UAV positions without rotation and PTZ tuning. The proposed algorithm (23) was used to control UAVs such that the coverage cost function was minimized. We have considered a density function $\varphi \left(q\right)=e^{-{\left(\frac{x-20}{20}\right)}^2-{\left(\frac{y-40}{40}\right)}^2}$ which is an area of interest (convex type). Two UAVs try to cover the area of interest by tuning the UAV position (x,y,z) parameters. Simulation results are illustrated in three parts: initial stage, after the initial stage and final stage. Initially, two UAVs started as seen in Figure 5. Figure 6 shows a representation of the coverage improvement after the initial stage. The final stage is shown in Figure 7. It was observed that the camera network coverage increased significantly through multiple rounds and the area was almost covered in the final stage.

We consider configuring more parameters such as the UAV position (x,y,z), rotation, pan and tilt angles, where the proposed controls (23) were used. However, one UAV is required to configure the view angle (rotation), where the proposed control (27) was used. A density function $\varphi \left(q\right)=e^{-{\left(\frac{x}{20}\right)}^2-{\left(\frac{y-20}{40}\right)}^2}+e^{-{\left(\frac{x-50}{50}\right)}^2-{\left(\frac{y-50}{20}\right)}^2}$ is a region of interest, which is a non-convex type, as shown in Figure 8a. Three UAVs try to cover the area of interest as soon as possible. Simulation results are illustrated in three parts: initial stage, after the initial stage and final stage. The first UAV, which is marked in blue color, configures the (x,y,z), rotation, pan and tilt parameters, but the rotation angle is required to follow the desired rotation angle 90 degree, starting at the x-axis of the CC system, which is right-handed. The other two UAVs use the coverage control with the configuration of the camera position $(x,y,z)$, rotation, pan and tilt parameters. Initially, three UAVs start grouped, as seen in Figure 8 and the coverage is poor. Later, Figure 9 shows a representation of the coverage improvement after the initial stage and the $1/2$ coverage of the area of interest is achieved. It is observed from Figure 8 and Figure 9 that the view angle of the first UAV (marked in blue color) is controlled in 90 degree. The final stage is shown in Figure 10, where the rotation angle of the first UAV is still controlled in 90 degrees. It is observed from Figure 8, Figure 9 and Figure 10 that the camera network coverage is increased significantly through multiple rounds and 93% of the area of interest is almost covered in the final stage.

4.2. Case 3

We configured the camera focal length (zoom parameter), which was not controlled in Cases 1-2. The altitude (z) was fixed at 35 m high so that the camera’s FOV was observed as updating the camera’s zoom parameter. Thus, the configuration parameters include the camera’s position (x,y), rotation and PTZ. We used the coverage control laws, as proposed in (23). The density function $\varphi \left(q\right)$ was the same as in Case 2. Four UAVs were intended to be used in the coverage control laws to cover the area of interest automatically. Simulation results are illustrated in three parts: initial stage, after the initial stage and final stage. Initially, four UAVs grouped, as seen in Figure 11, where the initial focal length was 8 for all cameras. It is observed that at the initial stage, the multi-UAVs were poor in coverage. After that, the coverage control was used to drive all UAVs to improve the coverage by configuring the (x,y), rotation and PTZ. Figure 12 shows a representation of the coverage improvement after the initial stage and almost $1/2$ the coverage of the area of interest was achieved. It is observed that the the FOV of each UAV is changing such that the coverage performance is improved, especially in the size of the FOV of each UAV. This is because the zoom parameter was configured. The final stage is shown in Figure 13. It is observed that even though the altitude of all UAVs was fixed, the camera network coverage was achieved as about 97 percent by controlling four UAVs by configuring the zoom and other parameters.

The previous results did not consider collision avoidance when configuring the camera parameters. In a practical situation, it is quite dangerous without collision avoidance mechanisms in controlling UAVs. In this case, we show the configuration of the parameters (x,y,z), rotation, pan and tilt with the collision avoidance proposed in Section 3.2. The density function $\varphi \left(q\right)$ was the same as in Case 2. Four UAVs try to cover the area of interest area as soon as possible. It is required to have a coverage rate of 85 percent. Assume that the radii of the avoidance and detection were $r=$ 30 and $R=35$, respectively. Figure 14 shows a representation of the coverage without considering the collision avoidance, while Figure 15 shows a representation of the coverage with the collision avoidance function. The horizontal distance profile between the ith UAV and jth UAV ($i\ge j$) is shown in Figure 16. It is observed from Figure 16 that the minimum separation distance was about 25 m (<the radius of the avoidance) when, without considering the collision avoidance, it was greater than 30 m (>the radius of the avoidance) when considering the collision avoidance. This verifies that the proposed collision avoidance can work well. Nearly 85 percent of the camera network coverage rate was achieved. In this case, the coverage rate was less than in Cases 1–3, since we intend to demonstrate the collision avoidance and use a large radius of avoidance, i.e., 30.

The proposed decentralized control requires knowing the position and FOV information of the neighboring UAVs. As discussed in Section 2.3, a communication network should be built for exchanging information from each other. The proposed coverage control is an iterative process based on the gradient descent. Simulation results have shown that the desired coverage rate can be reached by multi-UAV configurations. Even if during the initial stage, UAVs are poor in coverage (see Cases 2 and 3), the desired coverage rate can still be achieved by configuration of the variables. Moreover, as a collision avoidance term is added into the control, it is not necessary to worry about avoiding issues when conducting a coverage control, which was observed in Case 4. The drawback of the proposed algorithm is that the FOV of the ith UAV may be outside of the area of interest, which is observed from Figure 13.

5. Discussion and Conclusions

The development of coverage strategies is the main topic in multi-UAV systems. Technical considerations in camera networks consisting of multi-UAVs are still a challenge. Little work has been done in the design of the coverage control that involves the configuration of multiple parameters and view angle and collision avoidance.

This paper has presented a solution for coverage control of the multi-UAV system with downward facing cameras. We have extended the result of [19] to more contents. For example, we consider more parameters for tuning and enhancing control freedom. The view angle is discussed when designing the coverage issue. In addition, the proposed coverage control involves the collision avoidance issue. This is quite important when multiple UAVs are configuring the variables to achieve the coverage objective. The theoretical analysis is also given. The simulation tests have shown that the multi-UAVs with cameras can cover the whole interesting area automatically and verified that the proposed coverage control scheme is successful. The drawback of the proposed algorithm is that the FOV of the ith UAV may be outside of the area of interest. The current version also cannot handle invisible and obscured areas. This is a 3D coverage issue.

In future research we will consider the deformation of reality that may arise in the image due to the differentiation of the tilt angle or rotation. Another objective is to implement the proposed method on a real multi-UAV system.