Comparative Study of Physics Engines for Robot Simulation with Mechanical Interaction

Abstract

Simulation with a reasonable physical model is important to develop control algorithms for robots quickly, accurately, and safely without damaging the associated physical systems in various environments. However, it is difficult to choose the suitable tool for simulating a specific project. To help users in selecting the best tool when simulating a given project, we compare the performance of the four widely used physics engines, namely, ODE, Bullet, Vortex, and MoJoco, for various simple and complex industrial scenarios. We first summarize the technical algorithms implemented in each physics engine. We also designed four simulation scenarios ranging from simple scenarios for which analytic solution exists to complex industrial scenarios to compare the performance of each physics engine. We then present the simulation results in the default settings of all the physics engines, and analyze the behavior and contact force of the simulated objects.

1. Introduction

With the recent development of robotic technology, there is a growing movement to use robots for the efficiency and safety of tasks in real industrial fields [1]. To perform various and complex industrial scenarios using robots, a simple hand-tuned controller cannot achieve a high level of performance, which leads robots to perform tasks using model-based controllers. Furthermore, recently, data-driven approaches have been widely used for controlling robots, especially reinforcement learning [2], to enable robots to react to an unexpected environment. However, the development of control algorithms requires preprocessing such as validating the controller for model-based approaches, and obtaining a large number of data for meaningful generalization to various environments for data-driven approaches, which is time-consuming and dangerous when using a real robot.

Simulation with a reasonable physical model can overcome these challenges by allowing for users to develop control algorithms for robots in a variety of environments without the possibility of damaging the associated physical systems. For this, a large number of physics engines (e.g., ODE [3], Bullet [4], Vortex [5], MuJoCo [6]) that provide an approximate simulation of certain physical systems have been developed, but simulation results often show unstable or inaccurate behavior when performing complex multipoint contact simulations. To address these problems, a fast and physically accurate contact simulation framework should be developed, but currently developed physics engines still have many limitations. In a situation with such limitations, it is necessary to select a physics engine that is close to the real one for performing realistic simulations, but none of the physics engines has been consistently better than the others when considering various scenarios. There is also no well-organized document on the properties of each physics engine, and it is not intuitive how the properties of each physics engine affect the simulation results. Therefore, it is difficult to select a suitable physics engine for a specific project to be simulated.

In this paper, we compare the performance of four widely used physics engines for various scenarios to help users in deciding which physics engine to use by referring to the performance of each analyzed physics engine in this paper when simulating a given project. More precisely, we performed a variety of scenarios ranging from simple scenarios for which an analytic solution exists to complex scenarios used in the real industrial field (e.g., peg-in-hole assembly [7], bolt assembly [8]), and various object-to-object contact scenarios ranging from simple contact between primitive-shape objects to complex multipoint contacts between mesh objects. We also analyze how accurately physics engines simulate real-world physics by observing the contact force, which is important for the behavior of the object. These results can help many researchers in choosing a physics engine for a specific application, and in determining new research directions through the identified limitations.

Numerous studies have been proposed to compare the properties or performance of robotic simulators or physics engines. However, most studies have only tested simulators for simple scenarios for which analytic solutions exist, such as sliding an object on a slope or dropping an object to bounce using simple primitive-shape objects such as a sphere, box, and cylinder [9,10]. While there are studies that performed simulations for complex scenarios and analyzed the characteristics of each physics engine [11,12], these studies showed that the simulated objects were simple with a primitive shape, and the simulated scenarios did not cover widely performed tasks in real industries (e.g., assembly). In numerous studies, robot simulators that include modeling and visualization based on several physics engines to create an application for a physical robot (e.g., Gazebo, CoppeliaSim) were compared [13,14,15,16], and not physics engines, which affects the accuracy of the simulation. On the other hand, we analyzed the performance of several physics engines for various scenarios ranging from simple scenarios for which analytic solution exists to complex scenarios performed in the real industrial field, including primitive-shape and mesh objects. In addition, while other studies focused on the speed of simulation to analyze its performance, we observed how accurately the physics engines simulated real-world physics.

The rest of the paper is organized as follows. Section 2 presents the technical review of each physics engine. Section 3 presents information about the implemented scenarios to compare each physics engine. Section 4 presents the simulation results and discusses them. Summary and some comments on future works are then given in Section 5.

2. Physics Engine Review

To simulate the real world on the basis of a reasonable physical model, numerous physics engines have been developed. Widely used simulators by researchers are Gazebo, MuJoCo, and CoppeliaSim (previously V-REP), as described in [16]. Therefore, we selected MuJoCo, which is a simulator and physics engine, and ODE and Bullet, which are common physics engines of Gazebo and CoppeliaSim for comparison. Additionally, we included Vortex in the comparison group since it was analyzed to have a high level of accuracy, at least in simple surface/edge contact [17]. To perform simulations with these four physics engines, we used CoppeliaSim, which includes all of ODE, Bullet, and Vortex as physics engines, and MuJoCo. The contact simulation we wanted to test in this paper typically consists of the four following procedures. The first part is collision detection, which determines whether or not the objects intersect with each other. The second part is contact clustering, which reduces a large number of contact points into a manageable number of clusters to increase simulation speed. The third part is contact solving, which calculates the contact force acting on the objects. The last part is time integration, which updates the state of objects using the kinematics and dynamics of the objects, and the calculated contact force. Therefore, we summarize the technical algorithms used in each process of contact simulation, implemented in each physics engine by referring to several sources, and it could be helpful to users when predicting the simulation results of a physics engine. The description of each compared physics engine is as follows and summarized in

Table 1. Comparison of physics engines.

Physics Engine	Object	Collision Detection	Contact Clustering	Contact Solver	Integration	Coordinate
ODE	Primitive and mesh	Built-in	Ignores some contact points	LCP + Dantzig/PGS	Semi-implicit Euler	Maximal
Bullet	Primitive and mesh	Built-in	Maximizes contact area	LCP + Dantzig/PGS /NNCG/SI	Semi-implicit Euler	Generalized
Vortex	Primitive and mesh	Built-in	Details are not disclosed	LCP + direct/iterative method	Semi-implicit Euler	Generalized
MuJoCo	Primitive and convex mesh	libccd	Keeps only a few contact points in front	Convex optimization	Semi-implicit Euler/fourth-order Runge–Kutta	Generalized

.

2.1. ODE

Open Dynamics Engine (ODE) is an open-source physics engine that is integrated with several robot simulators, including Gazebo and CoppeliaSim. It has a built-in collision detection system that supports primitive-shape and mesh objects. When calculating the contact force, including friction, ODE uses a polyhedral approximation that relaxes Coulomb’s friction cone constraint as a pyramid shape, which transforms the contact problem into a linear complementarity problem (LCP [18]). To obtain the solution of LCP, ODE provides the Dantzig solver [19], which is a type of direct solver that finds an exact solution, and the projected Gauss–Seidel (PGS [20]) solver, which is a type of iterative solver that finds an approximated solution iteratively and quickly. The default setting in CoppeliaSim is PGS. When calculating the contact forces, ODE ignores some contact points to improve the simulation speed, although its details are not fully disclosed. Here, the default maximal number of contact points to generate in CoppeliaSim was 64.The integrator implemented in ODE that determined the numerical accuracy of the simulation was a semi-implicit Euler integrator [21]. Additionally, ODE supports multibody dynamics in maximal coordinate representation, but it does not support generalized coordinate representation, which is more computationally efficient and accurate for heavily constrained systems such as robots with a large number of joints.

2.2. Bullet

Bullet is an open-source physics engine, commonly used for computer games, computer graphics, and robotics. The Bullet collision detection algorithm proceeds similarly to most physics engines that support primitive-shape and mesh objects. When calculating the contact force, Bullet uses polyhedral approximation, similar to ODE, and provides Dantzig, PGS, nonsmooth nonlinear conjugate gradient (NNCG [22]), and the sequential impulse (SI [23]) solver. The default setting in CoppeliaSim is SI. Similar to ODE, Bullet also reduces the contact points to reduce the computational burden when calculating the contact force; the difference is that the rules for this are known, which is that the contact point with the deepest penetration is always maintained, and up to 4 contact points are determined to maximize the area covered by the contact points. The integrator in Bullet is a semi-implicit Euler integrator. Additionally, unlike ODE, Bullet uses the generalized coordinate representation.

2.3. Vortex

Vortex is a closed-source/limited free physics engine. Its collision detection algorithm consists of a series of overlap tests, beginning with fast, efficient, and approximate overlap tests on bounding volumes of the colliding objects to reduce computational complexity, and supports primitive-shape and mesh objects, similar to most physics engines. When calculating the contact force, Vortex also uses polyhedral approximation, and offers both a direct and an iterative solver to calculate the contact force. Here, unless simulation speed is more important than accuracy, such as in granular materials, the direct solver is the default setting. Similar to other physics engines, Vortex also reduces the contact points to increase the simulation speed, but the details are not fully disclosed. Vortex uses a semi-implicit Euler integrator and the generalized coordinate representation similar to Bullet.

2.4. MuJoCo

Multi-Joint dynamics with Contact (MuJoCo) is a commercial physics engine that requires a license. The collision detection algorithm in MuJoCo uses a general-purpose convex collider implemented by libccd [24], which is an open-source collision detection library, so it supports primitive-shape and convex mesh objects, but not nonconvex mesh objects. Unlike other physics engines, MuJoCo does not approximate the friction cone, but converts the contact problem into a convex optimization problem by relaxing the nonpenetration constraint between colliding objects [6,25]. To solve this convex optimization problem, MuJoCo provides the conjugate gradient method, Newton’s method, and PGS, which used to be the default. Similar to other physics engines, MuJoCo also reduces the contact points to reduce the computational burden by keeping only the first 16 contact points. In MuJoCo, two integrators are currently available, which are the semi-implicit Euler method and the fourth-order Runge–Kutta method. Additionally, it uses the generalized coordinate representation.

3. Robotic Scenarios

To compare the physics engines from various perspectives, we simulated four simulation scenarios. Two of them were simple scenarios with criteria that could be compared with an analytic solution, while the other two were complex scenarios required in the real industrial field. Details of each scenario are as follows.

3.1. Simple Scenario

We first constructed a simulation environment with primitive-shape objects to compare accuracy on the simple scenario for which an analytic solution exists. In this scenario, as shown in Figure 1, a total of 16 cubes with a length of 3 cm were placed on a box of 0.3 × 0.3 × 0.03 m³ at equal intervals, and a force was applied to the box to observe the behavior and the friction force of the cube. Here, we set the density of all the objects to be 2700 kg/m³, which is the density of aluminum. The coefficient of friction between the box and the cube was set to 0.3 to allow for the cube to move, and the coefficient of friction for floor was set to 0.

3.2. Multipoint Contact Scenario

Collisions between primitive-shape objects are generally single-contact (e.g., sphere on the box) or multipoint contact with only a small number of contact points (e.g., only the vertex of the box is set as the contact point candidate). However, when contact between objects with a complex geometry occurs, a large number of contact points are created, which causes the simulation to slow down. Therefore, as explained in Section 2, in most physics engines, the own contact clustering algorithm is used to reduce the number of contact points to reduce the computational load. To observe how the behavior of the object in simulation changes with this approximation, we simulated a multipoint contact scenario between a mesh cylinder as shown in Figure 2a and a primitive-shape box. In this scenario, as shown in Figure 3, a fixed box of 1 × 1 × 0.03 m³ was pressed down with a cylinder of 5 cm diameter and 5 cm height to maintain the contact, and we applied a lateral force to the cylinder to observe how the cylinder behaved during contact. Here, similar to the first scenario, we set the density of all the objects to be 2700 kg/m³. In this scenario, the friction force should act in the opposite direction of the moving direction when the cylinder slips. We designed this scenario to test the performance of each physics engine by observing the direction of the friction force.

3.3. Peg-in-Hole Scenario

It is necessary to compare the performance for complex industrial scenarios to test the scenarios required in the real industrial field in simulation. For this, we constructed a scenario in which a peg was inserted into a hole with 0.5 mm tolerance using a robot manipulator with an attached peg. Here, we used a 7-DOF robot manipulator (FRANKA EMIKA Panda), and the peg and hole were produced with mesh, as shown in Figure 2a,b to observe the behavior of complex assembly tasks. In this scenario, as shown in Figure 4, we designed the end-effector impedance control [26] to insert a peg into the hole by moving the peg downward. Additionally, we generated lateral force during insertion to produce a multipoint contact situation. Here, we observed the contact force of the peg during the insertion of the peg.

3.4. Bolting Scenario

The peg-in-hole task is widely used in the real industrial field, but the contact geometry is relatively simple because the peg has a convex shape. Therefore, we performed a bolt assembly simulation to test an industrial scenario with a more complex contact geometry. The contact geometry of the bolt and nut is very complex, as shown in Figure 2c,d, because it has a nonconvex shape, and there are many vertices on the surface where the contact surface changes rapidly. In this scenario, similar to the peg-in-hole scenario, we designed the end-effector impedance control to move the nut downward and rotate the nut after contact with the bolt to fasten the nut to the bolt. The current robot simulators have not been able to properly simulate these complex contact situations, but it is possible to fasten the nut in real experiments. Nevertheless, the reason for testing this scenario is to give information on the limitations of the current robot simulator and tasks that cannot be simulated.

4. Simulation and Results

In this paper, the default settings of all the physics engines were used for all scenarios, and only material properties (e.g., mass, moment of inertia, size) related to the object and friction coefficient were changed. However, in the case of Bullet, the default collision margin was set to be large, so we simulated the scenario with a small collision margin to test a contact scenario with a small tolerance.

4.1. Simple Scenario

In this scenario, we measured the average acceleration during the sliding while maintaining contact with the lower box for all of the upper boxes, multiplied this by mass to obtain the average friction force, and then calculated the error with an analytic solution. Then, we calculated the average of errors for all the upper boxes and used this as a performance index for each physics engine.

This performance index was tested for all the physics engines by varying the time step, and the results can be seen on the top of Figure 5. In addition, we obtained the penetration depths occurring at all the vertices of all the upper boxes and measured the maximal penetration depths as shown at the bottom of Figure 5 to observe how well the contact force prevented the penetration.

From the results, for a simple scenario, ODE and Bullet showed good performance on the basis of the contact force, and Bullet showed the best performance when penetration was also considered. In the case of MuJoCo, a relatively large contact force error was observed, and a large penetration occurred, since it ignored the nonpenetration constraint, as stated in Section 2.4. However, it improved the simulation speed compared to other physics engines not only in the simple scenario, but also in all other scenarios that we had implemented.

4.2. Multipoint Contact Scenario

In this scenario, we used the direction of the friction force acting on the cylinder as a performance index since most physics engines include approximation to the direction of the friction force (e.g., polyhedral approximation of friction cone). In particular, when multipoint contact occurred, the direction of the friction force could be significantly changed because the contact point was approximately reduced to a small number of contact points to reduce the computational load as explained in Section 2. Therefore, we applied force only in the x direction while pressing down to maintain the contact between the cylinder and box, and measured how much the y direction component of the friction force occurred compared to the x direction component by calculating an angle. Here, a large angle value means that a large error in friction force occurred, since the y-direction friction force should not be generated. In addition, we measured the penetration depth as in the simple scenario.

As shown in Figure 6, similar to the result of [17] referenced when including the Vortex as the physics engine to be compared, the Vortex showed the best performance in terms of contact force when multipoint contact occurred on a simple mesh surface. Here, in the case of ODE and Bullet, a failure occurred in which the cylinder fell as shown in Figure 7 at a specific time step, and the time step without data in Figure 6 was a failure case. Like in the simple scenario, Bullet had little penetration, and MuJoCo had large penetration. In addition, because of ignoring the nonpenetration constraint (i.e., soft contact) in MuJoCo, the cylinder continuously moved up and down.

4.3. Peg-in-Hole Scenario

In this scenario, we try to observe the contact force when multipoint contacts between mesh objects occur for scenarios that need to be implemented in the real industrial field. For this, we simulate the peg-in-hole scenario, and we measure the contact force of each physics engine. Here, we did not simply insert the peg vertically into the hole, but we insert the peg while applying force in the x direction after inserting the peg to a certain depth to generate a multipoint contact situation. Here, we set a tolerance of the size in which peg can be inserted into the hole since all the physics engines show behavior in which peg cannot be inserted in the case of peg-in-hole task with tight tolerance.

As shown in Figure 8, all the physics engines create an unrealistic contact force because it is discontinuous. More precisely, Bullet shows a relatively smooth and accurate contact force compared to other physics engines, while ODE generates a lot of discontinuous contact force, and Vortex shows that the z-direction contact force increases as the peg do not enter properly which is incorrect behavior. Unlike other physics engines, MuJoCo does not support nonconvex meshes, so it can be seen that the peg cannot enter into the hole as shown in Figure 9. For MuJoCo, to simulate a peg-in-hole task, the hole should be constructed with a combination of convex shape objects [27].

4.4. Bolting Scenario

As mentioned earlier in Section 3.4, all currently developed robot simulators do not properly simulate contact-intensive and tight-tolerance assembly tasks such as bolting scenarios.

This aspect can be seen in Figure 10, where all the physics engines failed to properly simulate bolting scenarios and showed unrealistic behavior. Here, the bolting scenario that was performed in the simulation is a scenario in which we confirmed that it is possible to experimentally fasten a nut with similar settings, as shown in Figure 11.

4.5. Discussion

From our observation, ODE and Bullet showed good performance for a simple scenario with primitive-shape objects, and Bullet and Vortex showed good performance for multipoint contact scenarios including mesh objects depending on the situation (Vortex for multipoint contact on simple surface, Bullet for the peg-in-hole scenario). The reasons for these results are that, in the case of ODE and Bullet, the contact simulation algorithms are basically similar, so they showed similar performance for simple scenarios, but in the case of multipoint contact scenarios, the method of reducing the number of contact points and the default option of contact solver were different, resulting in a difference in performance. In addition, since ODE uses maximal coordinates, in the case of scenarios involving multilink robot, simulation must be performed while satisfying the constraints between links compared to generalized coordinates that operate as one object, which can cause errors and reduce simulation accuracy. In the case of Vortex, it is difficult to know the exact reason, because most of the algorithms are not open to the public, but considering that the performance was relatively good for multipoint contact scenarios between complex objects, it seems that the algorithm had been tuned or improved, so that multipoint contact simulation of complex objects could be performed well. MuJoCo showed relatively poor performance in terms of contact force and penetration compared to other physics engines because it calculates the contact force by relaxing the nonpenetration constraints. However, this can greatly improve the simulation speed when multipoint contacts occur, so it is thought to be useful when it is necessary to quickly perform various scenarios such as reinforcement learning for tasks where contact force is not crucial rather than when simulating accurate behavior between complex objects.

These analytical results can be helpful reference materials for selecting a suitable physics engine for a specific project to be simulated. Furthermore, by identifying the limitations of the currently developed simulators, it would be possible to suggest points to be improved and future research directions in the field of simulation research.

5. Conclusions

In this paper, we summarized the technical algorithms and compared the performance of several physics engines (ODE, Bullet, Vortex, and MuJoCo). For this, we performed a variety of scenarios ranging from simple scenarios for which an analytic solution exists to complex industrial scenarios, and various object-to-object contact scenarios ranging from simple contact between primitive-shape objects to complex multipoint contacts between mesh objects. We identified the characteristics and limitations of existing physics engines. This process can give us useful information when we want to select physics engines depending on task and explore future research directions for simulation. Some possible future research topics include: (1) comparisons of more physics engines; (2) the extension of test scenarios to more various scenarios to provide more informative comparison results.