*4.1. Examples of the OF Estimation in Approaching to the Earth*

In our experiments, we used standard HERO 5 camera (GoPro Inc., San Mateo, CA, USA) and did not change its parameters during the flight. Since the aim of our research is the analysis of possible applications of video navigation in UAV, the camera is at the nose part of the light aeroplane. This aeroplane performed the series of approaches to the runway with accurate recording of the images and telemetric data. During the descent, the resolution of the camera sharpens, and it captures more and more tiny details which prevent good estimation of the image movement. Thereby, the estimation of the OF, which behaves more or less regularly at relatively high altitude, becomes absolutely chaotic when it approaches the earth surface. It is possible to observe this effect in the sequence of frames where the image motion velocities calculation done L–K estimates (see Figure 3), where the image is in almost natural scale with averaging 2 × 2. It is necessary to coordinate the camera resolution with the current altitude of flight to avoid this effect. Another effect is the increasing image velocity and therefore the additional blurring of the image, which needs the coordinated increasing of the frame rate. Of course, all such features usually do not present in standard cameras and their usage in navigation demands the developments of special observation tools. However, some issues may solve by onboard image processing. It means the artificial change of the resolution by averaging. The effect of the averaging is in Figures 3 and 4.

**Figure 3.** Optical flow estimated at landing with averaging 2 × 2 at the altitudes 195 m (**right**), 50 m (**centre**), and 20 m (**left**).

**Figure 4.** Optical flow estimated at landing with averaging 8 × 8 at the altitudes 195 m (**right**), 50 m (**centre**), and 20 m (**left**).

One can see the estimation of the OF becomes more and more regular with increasing the averaging scale, especially at low altitudes though it is impossible to use the high averaging scale for all altitudes of flight. At low altitudes, the blurring due to the image shift really needs the increasing of the frame rate. At high altitudes, where the averaging leads to the decreasing of the resolution, one cannot get a good quality of the OF estimation.

#### *4.2. Switching of Scaling by Means of the Altitude Estimation*

Another difficulty with the use of the OF is the change of scale and thereby the image shift rate caused by the change of UAV velocity and the flight altitude. Just as an example one can see the estimate of the OF based on Lucas-Kanade [37] algorithm obtained during the real flight at landing from 300 m to the runway (see Figures 5 and 6). As a result, the estimate of altitude, based on the OF estimated by the image of natural scale works more or less satisfactory up to the height 150 m and gives absolutely wrong values after (see Figure 7). Therefore, one needs to manipulate the zoom of the camera (change of focal length) in order to coordinate it with the characteristics of the OF. Meanwhile, one can use another approach, and to change the OES resolution by pixels averaging. The averaging of pixels made the OF calculation somewhat regular even at low altitudes (see Figure 8) with averaging by 16 × 16 at the altitude ≈30 m. One can compare it with Figure 6 It provides the reliable work of the algorithm up to the height of approximately 5–10 m (see Figures 9 and 10).

**Figure 5.** Optical flow estimated by Lucas-Kanade algorithm at the beginning of glissade, height ≈ 200 m, level of averaging 2 × 2. One can observe a rather regular nature of the optical flow which permits to estimate the flight parameters of the unmanned aerial vehicle with more or less high accuracy.

**Figure 6.** Optical flow estimated by Lucas-Kanade algorithm at the end of glissade, height ≈ 30 m, level of averaging 2 × 2. One can observe the very chaotic nature of the optical flow which prevents the estimation of the flight parameters.

**Figure 7.** Estimated altitude with the aid of images registered by optoelectronic system without averaging in comparison with the real one given by inertial navigation system with satellite measurements. One can see that estimation on the basis of the Lucas-Kanade algorithm without averaging does not work in the entire range of the altitudes less than 300 m.

In recent work [46], we realised the idea of the scaling control using the current estimation of the altitude. Generally, it is difficult to evaluate the effect of this approach since the estimate of altitude without the use of other sensors is based just on the OF estimation which depends on the altitude itself.

**Figure 8.** Optical flow estimated by Lucas-Kanade algorithm at the end of glissade, height ≈ 30 m, level of averaging 16 × 16. By comparing with Figure 6 one can observe much more regular behaviour of the optical flow which gives a better estimation of the flight parameters.

**Figure 9.** Estimated altitude in comparison with the real one obtained with a various level of resolution with averaging from 1 × 1 to 8 × 8. One can see that only the averaging 8 × 8 gives the acceptable accuracy of the altitude estimation in the range from 300 m to 50 m. Scale 1, 2, 4, 8 estimated altitude obtained by optical flow and Kalman filtering with different level of averaging, namely from 1 to 8. Base is the program motion. INS is the altitude from the inertial navigation system.

**Figure 10.** Estimated altitude in comparison with the real one obtained with a various level of resolution with averaging from 16 × 16 to 30 × 30. One can see that the averaging greater than 16 × 16 gives the acceptable accuracy of the altitude estimation in the range from 50 m to 5 m. Scale 16, 24, 30 estimated altitude obtained by optical flow and Kalman filtering with different level of averaging, namely from 16 to 30. Base is the program motion. INS is the altitude from the inertial navigation system.

The series of experiments based on the real data shows that it is possible to adapt the level of the averaging in order to expand the range of altitude estimation by the OF.

#### *4.3. Test of the Algorithm Based on the Scale Switching via Current Altitude Estimation*

As follows from the above consideration one needs to change the resolution of OES in coordination with the height of flight, and of course, the observed image is to be taken into account. Generally it is a nontrivial problem which needs consideration shortly, however now one can suggest the empirical algorithm for the averaging scale changing, based on the data which we have at hands. This empirical algorithm works using current height estimation as described in Table 1 [46].


**Table 1.** Change of the averaging level as a function of the height estimate.

One can see that the switching of scale improves the altitude estimation and permits to extend the algorithm operating range (see Figure 11).

Therefore, the possible solution of the OF usage is the changing of the averaging level during the descent. However, what kind of measurements could serve as a sensor for such switching of averaging? In the previous work [46], we examined the switching utilising the altitude estimation. However, at low altitudes all measurements and estimates become unreliable, so we tried to compare the OF velocities computed by L–K algorithm with the OF velocities calculated with the exact formulas and current estimation of the altitude. The parameters of the scaling change algorithm are in Table 1 [46]. The corresponding result of the altitude estimation is in Figure 11. One can observe satisfactory work of the OF in the altitude estimation. However, it is not a clean experiment since the estimation is

done using the knowledge of the current altitude which is necessary to transform the OF data into real velocities.

**Figure 11.** Estimated altitude in comparison with the real one obtained with the aid of averaging algorithm parameters described in Table 1. Estimated is the altitude obtained by optical flow and Kalman filtering. Base is the program motion. INS is the altitude from the inertial navigation system. SNS is the altitude from the satellite navigation system.

#### **5. Scale Switching by the Comparison of Calculated and Estimated OF**

#### *5.1. Comparison of Calculated and Estimated OF*

Here we use another sensor for the scaling switching that is the difference between the estimated and the calculated OF velocities.

The results of experiments related to the comparison of exact and L–K estimated velocities are in Figures 12 and 13. One can observe that the OF becomes useless when close to the earth, though the coordinated increasing of scaling level enlarges the range of reliable measurements.

**Remark 3.** *In all these pictures and below velocity is measured in <sup>F</sup>* <sup>∗</sup> <sup>10</sup>−3*units/s*, *where <sup>F</sup> is the lens focal length in meters.*

In Figures 12 and 13, the exact OF value calculated using (1) with known values of the current flight parameters obtained from INS and corrected with the aid of Kalman filtering. The L–K parameters estimated on from the current video sequence registered by the onboard camera.

**Figure 12.** Optical flow velocities via Lucas-Kanade (bold) in comparison with exact values (dots) calculated for different level of scaling: **left**– 4 × 4, **right**–8 × 8.

**Figure 13.** Optical flow velocities via Lucas-Kanade (bold) in comparison with exact values (dots) calculated for different level of scaling: **left**–16 × 16, **right**–30 × 30.

#### *5.2. OF Estimation as a Sensor for Scaling Switch*

As specified in the Introduction, we used the series of video sequences captured during the series of descents, where we estimated the OF velocities with the aid of L–K algorithm and compared it with theoretical values corresponding to the current aeroplane altitude. It is evident that the difference increases in approaching to the earth surface, meanwhile the OF permits to evaluate velocity at 25–30 m of height, though the difference becomes unacceptable below. It means that the noise in the velocities estimation via OF exposed to considerable perturbations which corrected by filtering utilising the dynamical model. The usage of Kalman filter based on the UAV dynamical model, observations of the OF, and the control accelerations permit to estimate the altitude in the range from 30–5 m. The picture in Figure 14 shows how to coordinate the scaling with the chosen threshold, which is equal to 0.2 unit/s, starting from averaging 4 × 4 at the beginning of glissade and increasing up to 30 ×30 when the aeroplane is close to the earth, approximately at the height of 5 m. Red vertical lines show the switch of scaling based on the difference of estimated and calculated OF velocities. Figure 15 shows the altitude estimate based only on the OF and filtering. Of course, it is just an experiment demonstrating the ability of the OF in such a complicated situation. In reality, it must combine with other altimeters, but if they fail to work accurately due to some reasons, the OF could serve as a reserved one.

**Figure 14.** The difference (error) of the optical flow velocities via Lucas-Kanade and exact values, calculated at different altitudes from 300 m to 0 m. Vertical red lines show moments of switching of the averaging levels, that are: 4 × 4, 8 × 8, 16 × 16, and 30 × 30 from right to left.

**Figure 15.** The altitude estimation via optical flow with switching averaging scale. The estimation of the altitude is made on the basis of standard Kalman filtering by fusion of data from the control system and the optical flow measurements. Of course, at low altitudes, less than 25–30 m, the OF measurements are corrupted by very high noise, but the usage of dynamical model and data from control system permit estimate reliably the altitude up to 5 m. Estimated is the altitude obtained by OF and Kalman filtering. Base is the program motion. INS is the altitude from the inertial navigation system. SNS is the altitude from the satellite navigation system.

Finally, the comparison of the altitude descent curves can be carried out. The program trajectory is the reference to the other ones. The results of the altitude tracking error are in Table 2. The rightmost column represents overall statistics for the new scale switching algorithm. The scale switch occurs when OF velocity error raises above level 0.2 in Figure 14. Conducted trial shows that the altitude estimate through the video sequence is quantitatively comparable with two other ones.


**Table 2.** Sample statistics of the altitude tracking error to the program trajectory.

**Remark 4.** *In these experiments, the estimation with the aid of L–K algorithm done by standard software and needs the frames from the video sequence and the current scale size in pixels for calculation of the OF velocities. The OF exact value was calculated with the aid of* (2) *with values of flight parameters obtained from INS corrected by Kalman filter* (5)*,*(6)*. One can observe that the difference of exact and estimated OF velocities gives threshold points which are different in comparison with Table 1.*

#### **6. Conclusions**

In summary, the article presents the investigation of the OF usage as a sensor at the UAV landing. In general, it needs the adaptation of shooting parameters to the current altitude and velocity of flight that leads to the necessity of changing the camera characteristics such as virtual resolution and the frame rate. However, one can resolve the problem by using image processing such as change the resolution by controllable scaling. By using the difference of the OF velocity estimated by L–K algorithm and calculated via exact formulas and filtered by Kalman estimate as a sensor of the scale switching, it becomes possible to achieve the reliable altitude estimation up to 5 m. It shows how to provide the data fusion of the OF and filtering in the complicated problem of the UAV landing.

**Author Contributions:** Conceptualization, A.M., B.M.; methodology, K.S.; software, K.S., A.P.; validation A.P.

**Funding:** This research was partially funded by Russain Academy of Sciences and Foundation for Basic Research.

**Acknowledgments:** The authors would like to thank Russian Foundation for Basic Research for partial support of the research by grants 16-31-60049, and 16-08-01076.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**

The following abbreviations are used in this manuscript:


#### **References**


© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
