Product Tests in Virtual Reality: Lessons Learned during Collision Avoidance Development for Drones

Abstract

Virtual reality (VR) and real-world simulations have become an important tool for product development, product design, and product tests. Product tests in VR have many advantages, such as reproducibility and shortened development time. In this paper, we investigate the virtual testing of a collision avoidance system for drones in terms of economic benefits. Our results show that virtual tests had both positive and negative effects on the development, with the positive aspects clearly predominating. In summary, the tests in VR shorten the development time and reduce risks and therefore costs. Furthermore, they offer possibilities not available in real-world tests. Nevertheless, real-world tests are still important.

1. Introduction

Virtual reality (VR) and real-world simulations have become an important tool for product development, product design, and product tests. Virtual reality technology, used at various stages of product development, can enable the identification of potential ergonomic problems and, consequently, the definition of measures to solve or at least mitigate them [1,2]. Virtual reality can mimic the characteristics of real, existing, or fictional environments—at least in the relevant aspects being considered. However, it can also transcend the limitations of physical reality [3]. Virtual reality tools use computer modeling and simulation technologies that have been widely used for more than half a century in education, health, entertainment, culture, sports, engineering, the armed forces, and other sectors, but have only recently become a practical tool for the manufacturing industry [4]. While these considerations have so far focused on the human-in-the-loop scenario, i.e., mainly on ergonomics and user experience [5], the focus is now broadening to include hardware-in-the-loop.

Although simulation in virtual worlds has high initial investment costs—after all, a virtual environment must first be created, whereas the real world already exists—we demonstrate in this pilot study that the use of VR technology is already paying off in single projects. This hypothesis is the focus and was investigated in an exemplary project.

In this paper, we describe the virtual testing of a collision avoidance system in a VR environment. The context is a research project for developing a novel collision avoidance algorithm for small flying vehicles such as drones. Testing the system poses a high risk for the prototype drone since it is meant to work as a last resort before the vehicle collides with obstacles. The project therefore focused on VR testing at an early stage. This paper describes the experiences and lessons learned in this regard.

1.1. Testing in VR

Calculating and simulating physical processes is one of the first applications of scientific computing and has its origins before the development of the first computers [6,7]. With VR and computer-aided design, computer-aided engineering, computer-aided manufacturing, and computer-aided …(CAx), this development has evolved into a holistic approach.

Virtual reality is an important tool for all kinds of product tests. VR environments consist of entire rooms, as in a Cave Automated Virtual Environment (CAVE), first described by Cruz-Neira et al. [8], or can be experienced wearing a head-mounted display [9]. The latter are easy to handle and inexpensive [10]. In addition, pure simulation environments—without visualization—are occasionally referred to as VR environments, since they represent a (partial) aspect of reality.

Ottosson describes the use of virtual reality in the product development process [11]. Applying virtual reality technology accelerates a product’s life cycle [12]; furthermore, VR prototypes are less costly to produce, more flexible in modifications, and offer more possibilities for demonstrations [13]. VR technology has therefore established itself in product development [14,15].

Especially for products that are designed virtually and that are also used virtually/electronically, a test in VR is a natural choice. Park et al. [16] demonstrated design evaluations of digital consumer products using virtual reality-based functional behavior simulation. As the interaction with the virtual interface does not invalidate the usability evaluation itself [17], VR tests can be used widely and have exerted a huge impact on early-stage interactive product design processes [18].

However, it is not only for virtual products that evaluation in VR offers advantages. Several studies have confirmed that VR has considerable potential for applications in the area of ergonomics [19]. In the context of urban design and building information modeling, Bauer et al. show a CAVE environment for testing public buildings such as train stations [20]. They argue that evaluating, for example, the pedestrian guidance system before the building is finished helps to find errors early on and avoid costly alterations. In city development, the use of VR technology helps to communicate new mobility concepts and test design alternatives. Schrom-Feiertag et al. present a set of tools for efficient participatory planning with virtual and augmented reality [21].

VR technology has also become popular in industrial contexts and in production [22]. In their survey about industry use of virtual reality in product design and manufacturing, Berg and Vance conclude that VR is mature, stable, and usable [23]. They write that VR is actively being used in a number of industries to support decision making and enable innovation. For the time being, the entire product lifecycle is mapped in VR [24]: from virtual engineering [25], through process simulation [26], including partial aspects such as energy consumption [27], to the end user in the form of training courses [28].

While the terminology is usually user-centered (e.g., see [29] and Figure 1), we extend the term VR by automated, simulated product tests even without a human in the loop of product design, i.e., the existence of a virtual environment is not dependent on a human observer. In our use case, it is mainly drones that explore and fly through the virtual worlds. This approach is not new per se; however, it is still rarely applied [30]. Especially in the context of the development, construction, and design of drones, VR tests have so far been reduced to the user interface [31,32,33].

In summary, the main advantages of product testing in VR are manifold [34]:

• absolute control: A virtual environment provides absolute control. In extreme cases, the simulation can be deterministic and all random / external influences, i.e., the noise of the real world, such as weather aspects, materials with (natural) variations, etc., can be eliminated.
• reproducibility: The elimination of any randomness leads to a high degree of reproducibility, which is unattainable for real tests.
• shortened development time: In many test scenarios, development times can be drastically reduced. Especially in destructive tests, where the prototype is destroyed (e.g., crash tests), the time for building the prototypes is omitted.
• higher optimization potential: By not having to build real prototypes, a limiting scaling factor of product testing is eliminated. Many more product tests can be performed in VR, and by consequence many more product parameters can be optimized.
• hazard free tests: Virtual testing is risk-free—both for humans and for potentially expensive machines and equipment.
• unrealistic conditions: Although VR testing strives for the highest possible level of realism most of the time, unreal conditions are also an advantage that does not exist in reality: in VR, the environment can be abstracted and reduced to a minimal level. This reduction enables efficient debugging and testing of subsystems that would be difficult to test on their own in real life.

All of these points, especially the reduced development time and elimination of the hazard potential of developed prototypes, are the foundation of the hypothesis investigated.

1.2. Collision Avoidance

Collision detection is not the focus of this paper; however, collision detection provides the application context for product development. The aim of the corresponding project is the development of a bionic sensor system for reliable visual collision detection in unmanned flying objects. The model organism for this bionic approach constitutes the reliable optical collision detecting system of gregarious locusts. The neural basis for the bionic collision detector was first described by Judge and Rind [35]. The applications of a bionic collision detector are described by Yue and Rind [36], Fu et al. [37], and Cizek and Faigl [38]. The algorithm implemented in the hardware in the project is similar to that of Blanchard et al. [39], but instead of simulating artificial neurons connected via synapses, the gray values of the pixels are computed directly to estimate the collision risk.

Within the development of the new drone system, the activity of their right and left collision-detecting neurons was recorded in various critical flight situations. The results from this electrophysiological approach provided the basis for the development of a bionic algorithm for visual collision detection. This innovative concept is based on simple mathematical calculations that are performed at the level of pixels and efficiently extract the collision risk and a possible avoidance vector from the visual scene. In contrast to current systems, this bionic algorithm works without distance estimation and object recognition and is similar to the collision detection algorithm described for vehicles [40]. The new algorithm has been implemented on hardware in order to perform parallel image computations for real-time collision risk estimation and the computation of an evasion vector using Field-Programmable-Gate-Array (FPGA) technology installed in test drones. The sensor technology has been optimized on a virtual flight controller by using virtual reality technology before being tested with real drones.

2. Methods

The development of a new collision avoidance system for drones needs to be tested thoroughly. Within the project’s use-case, the tests are rather risky as the avoidance system should work as a last resort before the drone crashes into an obstacle, i.e., the test scenarios consist mostly of near-collision situations testing the behavior of the new algorithm in a new drone prototype. The failure of these tests could lead to the total loss of the drone. Since the drone is also a new development and only exists as a few individually manufactured prototypes, the manufacturing costs per drone are many times higher than the market prices of even professional drones. The total loss of the drones must therefore be avoided by all means. A virtual test environment is a safe place to run many tests without any harm for the drone. In this paper, we want to investigate the research question:

What are the benefits of testing a drone in a CAVE virtual environment?

• and

What are the financial incentives to use VR technology?

Product Test Environment

Based on an existing virtual environment, we developed a test system for the drone. In the original configuration, it is possible for a human to navigate through the environment by various input devices, such as game controllers or optical human motion tracking devices.

For the product tests of the drones, the CAVE virtual environment has been adjusted and extended in various ways (see Figure 2 (left)).

• simulation-based input device: In the initial configuration of the CAVE, control was assumed by a human. This task must now be taken over indirectly by the drone. In detail, the output signals from the drone controller, which is otherwise connected to the drone’s motors, are sent to a physics simulation server, which uses a physically correct real-time simulation to calculate the position of the drone in 3D. These data from the flight simulator are then the input data for control within the virtual world. For the drone to explore the virtual world, a corresponding network protocol has been added that handles the current position and orientation in real time.
• real standstill: The advantages of a CAVE—for example, in comparison to head-mounted displays—are its own body perception and the possibility to move (with the limitations of the CAVE) together with other persons in order to explore virtual objects. In our test setup, the drone is placed at the center of the CAVE without the possibility to move. It explores its surroundings using its installed cameras. The new drone uses two cameras to enlarge its field of view. As the physical position of the drone remains unchanged during the tests, the CAVE tracking system has been deactivated.
• stereoscopic rendering: A major advantage of the new collision detection algorithm is its simplicity. It does not require 3D reconstruction or computationally intensive object detection. The fields of view of the two cameras of the drone are therefore without overlap; they merely enlarge the field of view of the drone. The usual stereoscopic rendering in the CAVE is disabled since the collision avoidance algorithm is designed for a monoscopic view. By not using 3D stereo via shutter technology, the full brightness is available for the monoscopic test setup.
• high refresh rate: The frame rate of the new collision detection algorithm is not based on human perception, but on the processing speed of the algorithm implemented in FPGA hardware. This is 120 Hz and the projection systems of the CAVE use the same frequency in the tests. This adjustment is necessary to prevent flickering effects.
• color space, contrast, and brightness: During the planning phase, particular attention was paid to the compatibility of the visualization in the CAVE with the camera system of the drone. Since the drone can only record gray images, the color representation is rather irrelevant; contrast and brightness, however, are suboptimal, especially compared to real outdoor tests.
  The CAVE uses Digital Light Processing (DLP) projectors with a maximum brightness of 7500 ANSI lumen. Each wall screen is 3.3 m by 2.5 m large. The quadratic floor has an edge length of 3.3 m (see Figure 2 (right)). Consequently, each screen produces no more than 900 lux. Different lighting conditions are listed in

  Table 1. The comparison with different lighting conditions [41] allows us to interpret the CAVE’s conditions and gives an impression of its maximum brightness.

  Illuminance	Example
  0.0001 lux	moonless night sky
  0.25 lux	clear night sky with full moon
  400 lux	ambient sunrise on a clear day
  1000–2000 lux	typical overcast day at midday
  >100,000 lux	bright sunlight

  for comparison.
  This initial situation is regrettably suboptimal. Unfortunately, the solution to this problem is not feasible, as it would involve expensive hardware replacement; the CAVE is only rented and the provider refuses to undertake this investment.

3. Results and Discussion

The evaluation of the new algorithm was carried out in 14 test sessions, each lasting half a working day. In total, eight test sessions were conducted in VR, whereas six test sessions were held outdoors.

In virtual reality, the drone was able to control the virtual representation by sending position changes to the visualization. Different lighting conditions were tested within the possibilities of the projection system. The camera system of the drone was modified to work in low light conditions and with changing brightness levels. To further investigate the collision avoidance algorithm, the realistic environment was exchanged for an abstract version for some test runs. In all sessions, there were severe problems, with the collision avoidance algorithm leading to a virtual crash. In a real-world test, this would have resulted in a damaged drone.

Despite the extensive planning in advance and the numerous adjustments, circumstances arose during the product tests that had both positive and negative influences.

−

The lighting conditions, which cannot be changed in terms of brightness and contrast, only allow a limited test scenario. In particular, tests of bright sunny days with high-contrast light–shadow configurations cannot be tested in VR—at least not in the currently used setup.

±

Testing in VR guarantees absolute control over the virtual environment and the test configuration. This absolute control has both positive and negative effects. On the one hand, control and reproducibility simplify debugging and automatic testing; in particular, random elements would complicate this process considerably in many cases. On the other hand, everything must also be controlled, i.e., only what is programmed in advance happens in VR. The design space of possibilities may contain variations, but will hardly open up new dimensions that have not been designed before (see Figure 3), e.g., if no flock of birds has been programmed, they will not appear; if no pedestrians or other road users have been animated, the roads will remain empty. This is why real tests are still being carried out in the drone project to test with real-world complexity.

+

Despite many error avoidance strategies (test-driven development, pair programming, etc.), one error remained undetected until the first test: the calculated escape vector to avoid a collision was scaled incorrectly. As a consequence, the algorithm did not react to any obstacles. Even when the drone navigated directly towards a building, the algorithm showed no reaction. In a real test, this error would have inescapably (literally) led to a crash and thus probably to the total loss of a prototype. However, this first, complete functional test, which is very critical due to the test scenario, first took place in the CAVE environment for good reason.

+

Preventing prototypes from real crashes in this VR environment leads to economic benefits: Irrespective of the costs of a prototype due to increased material and personnel expenses, more efficient test phases are the result. In this context, we would like to point out the potential delays that would have occurred in real tests. A damaged drone in the tests must be repaired at an unknown cost (in terms of personnel costs and material costs) before the next test starts. After all, a virtual drone does not need to be repaired after a crash.

+

A side effect of testing in VR only becomes clear in comparison to real outdoor tests: unlike real tests, no VR tests had to be postponed due to fog, rain, or other bad weather conditions. The project schedule is easier to adhere to in VR. However, this cannot be generalized for every VR product test project.

+

Even though VR tests usually aim for the highest possible degree of realism, unreal conditions are also an advantage that does not exist in reality. In VR, the surroundings can be made abstract in order to simplify debugging.

In short, the collision avoidance system of the drone can be tested in its entirety. The processing of the camera images is done in the same way as it is done in the real world. Furthermore, the test setup is an authentic representation of the real world, without the danger of crashing the drone into an obstacle.

The financial aspects of VR-driven product testing for drones are listed in

Table 2. The project budget shows the costs for 14 tests. Please note that no test was performed twice, i.e., this is not a comparison of a VR group with a control group in the statistical sense. Nevertheless, the costs show that it is worthwhile to perform all tests in VR environments that can be performed in VR and that can replace real tests.

	Testing in VR	Testing Outdoors
Completed test sessions	8	6
Total costs	EUR 19,167	EUR 40,377
	These costs already include EUR 2000 rent and EUR 1300 for the provision of the VR models.	This item includes all costs involved, including travel expenses, travel time, etc.
Costs per test session	EUR 2395	EUR 6729

. It quantifies the costs using the project costs budget.

Within the research project, four prototypes of the new drone were built having the new algorithm hardware embedded. The accumulated costs of these four developments in personnel time and material costs amounted to EUR 32,908, i.e., EUR 8227 per drone. Since, at minimum, one total loss could be avoided through the use of VR-based testing, at least the cost of one drone can be attributed to the savings of using VR.

4. Conclusions

There is a clear case for the increased use of VR testing based on experience gained in VR-based product testing. In virtual realities, a wide variety of scenarios can be reproducibly run through—with automatic testing even 24/7. This automatic testing under almost realistic conditions is otherwise not possible. Furthermore, the level of realism can be reduced to a minimum. Such a minimalist scene with only a plane and a box is shown in Figure 4. This reduction enables the efficient debugging and testing of subsystems that would be difficult to test in reality (see Figure 3).

During the test sessions, the development team was able to identify some disadvantages with the collision avoidance algorithm, including a severe problem that would most likely have caused the total loss of a valuable prototype. This avoided crash also ensures a positive economic balance: VR testing is more cost-effective than real tests, even if not taking into account any project delays that may occur due to the repeated construction of new prototypes.

Concerning the future of virtual test environments, we dare to predict that this technology will continue to grow in importance; as the use of autonomous systems—whether based on AI methods or not—increases, more and more tests will become necessary. Whether autonomous driving or AI-driven drones, millions of test trials can only be completed in a VR environment efficiently. How else, if not by virtually reconstructed scenarios and through appropriate test procedures, can it be ensured that accidents that occur once are not repeated?