2.1. Ship Plans & 3D Model Creation in Rhino3D
The methodology is described in relation to only one of the four ships indicated above, the Russian Azov. For a better comprehension of the work in this project, the historic ship lines plan of Azov [
18] was used for the creation of the 3D ship models in Rhino3D [
19]. The historical research and documentation of the aforementioned blueprints was carried out by the NAVS project partners Institute of Mediterranean Studies, FORTH, and the Eugenides Foundation.
The geometry in Rhino3D is based on the NURBS [
20,
21] mathematical model, which aims to produce mathematically correct representations of curves and freeform surfaces [
22]. The development of ship sections, as seen with red NURBS curves in
Figure 2, is a basic method that is commonly used in ship design. These curves are smoothed out, in order to create a hull that is as smooth as possible.
In the next step, a subset of the faired curves was chosen to form a curve network that leads to a hull-surface. The surface was mirrored, and with the proper adjustments, the hull converted into a closed polysurface, which is the equivalent of a solid in Rhino3D. The hull, along with the keel, is shown in
Figure 2 (green), with the selected sections (red) aligned in 2D and 3D space.
Proper cuts in the solid hull were made, as shown in
Figure 3, to construct the deck flooring and the openings for the masts and cannons.
Figure 3 also depicts the deck components of the Azov, as well as the longitudinal wooden strips that run outside the hull and follow the geometry of the outer surface.
The full-rigged 3D model of Azov equipped with sails and masts is depicted In
Figure 4. The sailing ropes are also shown in the same figure. Finally, with all the gunport windows open, the cannons can be seen (starboard side).
2.2. Polygon Mesh Optimization
2.2.1. Conversion of a Rhino3D Model to a Polygon Mesh
As mentioned in the previous chapter, Rhino3D uses the NURBS mathematical model for the surfaces produced (
Figure 5).
This method cannot be used by the other software utilized in this project, so it was necessary to convert all the surfaces from NURBS surfaces into another surface type, called a polygon mesh. Polygon meshes are collections of vertices, edges, and faces that define an object and can be used by any 3D-modeling and design software. There are many common file formats that can contain meshes, and for this project the .FBX file format [
23] was chosen as it can contain many different objects separately, while also preserving the hierarchy of those objects.
In order to export a NURBS surface into a .FBX file containing all the polygonal meshes, a conversion from NURBS surface to polygonal mesh was deemed necessary. This conversion had to be as lean as possible, meaning as few polygons as possible while maintaining all the surface’s information (geometry and curvature). Furthermore, in order to make future steps easier, with regard to unwrapping and texturing of the models (on which more details are provided later), it was desirable to make all or most of the polygons quads, meaning that each polygon was defined by four edges and four vertices.
Rhino3D provides two ways for doing this conversion:
A quad-based topology means that a mesh is comprised mainly by quads. This is a generally preferable practice for 3D artists, for various reasons. A quad-based model is the most useful form for 3D modeling, as it ensures a clean topology and provides edge flow that can easily be adjusted. This is highly important when further editing is needed, for example when attempting to reduce polygon count, or when unwrapping is mandatory for texturing the mesh; as in this case. Secondly, subdividing a clean quad-based topology is a controlled and low-risk task, while creating edge loops in a primarily triangle-based mesh can be unexpected and cause shading issues that will be rendered into the game engines.
Whilst game engines such as Unity3d triangulate the 3D model during import, the triangulation of a quad is considered more predictable than that of an Ngon (a polygon with more than four edges), which may result in unwanted shadings and normal vector calculations.
Quad remeshing was chosen because it produced excellent quality polygon meshes that maintained all the original surface’s information while being solely comprised of quads. The following figure (
Figure 6) clearly shows, in contrast to the previous figure, that the mesh holes are completely defined by the quads around them, while the NURBS surface has no discernible way of defining the holes.
The quad mesh does have some issues, especially in the bow area, but all those are refined during the retopology that takes place in the immediate step following this conversion. The quad remeshing tool could not be used on all objects of the models, so retopology was completed on those objects.
After the quad-remeshing process was complete, all the meshes created were exported into a .FBX file which was then used for retopology, unwrapping, and texturing of the models.
2.2.2. Retopology
Retopology is “the act of recreating an existing surface with more optimal geometry” and it is used to modify a high-resolution 3D model to its optimal form, to be utilized for animation or texturing [
25]. The reason retopology was used in “The Ships of Navarino” was twofold. Firstly, since retopology is one of the most popular methodologies that 3D designers use to reduce polygon count, it was utilized for optimization purposes. Secondly, a low polygon model, with clean topology that provides edge flow, significantly reduces the UV mapping complexity, a process necessary for later texturing (
Figure 7 and
Figure 8).
There are various options for retopologization:
Manually, in a 3D-modeling software such as Blender. Given the built-in features of Blender, such as surface snapping and the shrink-wrap modifier, a designer can attempt to rebuild the high poly model polygon-by-polygon.
Using specialized retopology Blender add-ons that attempt to reduce complexity and accelerate the retopologizing process.
Using software outside Blender that offers more retopology tools.
Using some Blender retopology add-ons that allowed face snapping was the best option for this project.
After remeshing, retopologizing, and refining the meshes, the most optimal form of the ship models was achieved. The final models counted significantly less polygons and were lighter and easier to manipulate.
Converting a high poly to a low poly benefits not only the modeling process but also the quality of the final model, because details of the high poly can be maintained without causing performance penalties (
Figure 9).
By converting the high poly model into normal, occlusion, and other maps, details that demand a large number of polygons can be rendered into them and used as textures afterwards.
2.2.3. Texturing
3D Texturing refers to the process of creating detailed textures (maps) to be applied (mapped) onto the surface of a 3D model. In order for the texture to be applied correctly onto the surface, the model has to be prepared with a process called UV unwrapping.
2.2.4. UV Unwrapping
A UV map is the representation of the surface of a 3D model onto a 2D map that is later used for texturing the specific surface. The process of creating a UV map is called UV unwrapping. For 3D texturing it is necessary to unwrap all of an object’s surfaces to the UV space. To further explain, the process works as follows: the unwrapped surface is projected onto a specific place in the UV coordinates, which enables the surface to display the texture information on the 3D model. An example of a UV-unwrapped texture is shown in
Figure 10.
The way the UV maps are organized is crucial for the texturing process. For example, if two or more surfaces are overlapping in the UV map, these surfaces will keep rendering the same textured coordinates. In addition, the UV maps have to be distortion-free and maintain their proportion ratio, otherwise the texture appears stretched and disproportionate. UV unwrapping is also critical with respect to the texture quality. In fact, the larger the item on the UV space, the greater the density, meaning that there is more space available for painting details.
Another thing to take into account when unwrapping is the seams. A seam is the edge where the mesh surface is “cut” in order for the 3D model to be mapped onto the UV map. Seams can create visual discontinuation of the texture if they are misplaced, so when possible, they should be hidden in places not easily visible, or where the material changes.
All known 3D-modeling software include tools for UV unwrapping the 3D model, as this procedure is mandatory and texturing depends on that. “The Ships of Navarino”, were unwrapped into Blender and their UV maps organized hierarchically by size, in a way that most space was held by the surfaces that were most likely to be seen closely by the user.
2.2.5. Applying Textures to a Polygon Mesh
When a 3D model is successfully unwrapped there are two options:
3D texturing software, such as Substance 3D Painter, provide the necessary tools for painting high-quality and detailed maps (base color, normal, roughness, metallic, ambient occlusion, etc.), facilitating the creation of PBR materials. PBR stands for physical based rendering but it can also be known as physical based shading. It is a pipeline that simulates materials in a plausible way, aiming for realistic results.
Advanced masking and procedural texturing tools allow us to achieve textures that are much harder to create with a 2D program such as Gimp or Photoshop. Substance 3D Painter allows the creation of realistic materials using various sources and provides procedural tools to enhance the base materials with whatever is necessary for the most accurate resemblance. Finally, it bakes the textures to different maps according to the target engine, in this case the metallic channel of Unity3d’s standard shader.
Reference images are some of the most important resources for the design of the textures and, when talking about a realistic representation of a historical event, are strictly tied with the quality of the result. For “The Ships of Navarino”, there was access to a small archive of drawings and images resembling the warships, and to historical specialists that acted as counselors. On this basis the various materials of the ships were created. Given the fact that this project demanded careful spending on materials and texture sizes, a flexible organization of UV maps was used so that one texture would include more than one material and every material could be used in more than one mesh, and potentially in more than one ship.
To further elaborate on this, every texture was a blend of more than one material, depending on each ship’s demand for different materials (
Figure 11). Creating textures in this way also proved to be very useful in a later stage, when correcting a mesh’s material was an easy procedure when could occur just by moving the UV tiles to different coordinates. However, due to their unique nature, specific meshes such as a ship’s wheel (
Figure 12), anchor, or figureheads, were treated differently, as they were solely hand-painted and in fact, in more detail, so that the materials could be used exclusively on them. Additionally, corrections on these objects were a more complex procedure as they could only be completed with Substance Painter.
Following a pipeline that highlighted reusability, when possible, a significant reduction in draw calls was achieved, because inside Unity3d their number maintained a low reassuring performance whilst maintaining accurate representation of materiality on the ships (
Figure 13).
2.3. Developing and Optimizing a VR Application in Unity3d
When developing a VR application, it is very important that optimization is taken very seriously. There are many steps to optimizing any application made in Unity3d, especially for a VR project. The goal of the project was to keep as much of the visual fidelity as possible while also obtaining acceptable performance, to avoid the implications of an inadequate or inconsistent performance (nausea, motion sickness). In this chapter, the optimization steps that were implemented are thoroughly discussed.
2.3.1. Geometry Optimization to Reduce CPU Time
The first step is geometry and texture size optimization, both of which were already completed in the previous step. Since meshes require CPU time to be drawn, the fewer triangles they contain the better [
26]. Textures use both the CPU and GPU, for loading and displaying, respectively. Lowering their size is important to avoid long loading times between scenes and potential VRAM limitations on the graphics card, which could lead to stuttering. Additionally, separating the different items into separate meshes is considered beneficial for the third step of the optimization process, occlusion culling.
2.3.2. Choice of Lighting Method
The second step is using baked lighting instead of real-time lighting. Real-time lighting requires every triangle to be drawn multiple times in order to display the mesh, the texture that is applied on the mesh, and the shadows cast by the mesh or nearby meshes. Although this can produce more realistic lighting and shadows, it is very intensive on hardware and may produce lower than necessary performance, because VR is already intensive on the GPUs. On the other hand, baked lighting is precomputed and has a very small performance overhead, while also providing acceptable levels of lighting and shadow quality [
27].
In Unity3d, lighting is separated into direct and indirect (
Figure 14). Every ray cast from the light source that hits a mesh is considered direct lighting, and every surface that is lit from the bounces of those rays is considered indirect lighting [
27].
Baked lighting was chosen in order to save as much performance as possible, given the fact that each frame is drawn twice for any VR application (once for each eye). For this method a Lightmapper was needed, which is a piece of software that uses light sources in a scene to precompute direct and indirect lighting and shadows cast from all sources, and then compresses the data into textures called lightmaps. These lightmaps were then overlaid on the textures applied to each mesh to give the illusion of lighting and shadows. Unity3d does offer a Lightmapper that is built into the engine, but it was decided to use a Unity Asset Store package named “Bakery GPU Lightmapper” because of its superior speed and end result. Bakery’s standard shader is equivalent to Unity3d’s standard shader with a few extra features. For example, Unity3d’s built-in Lightmapper does not include lightmap specular highlights while Bakey GPU Lightmapper does. This makes scenes using Bakery a lot more vibrant and realistic [
28].
2.3.3. Occlusion Culling
The third step is occlusion culling. This is a process which prevents Unity3d from rendering objects that are outside of the frustum of the camera or hidden by other objects. This is completed as default in every frame, since the main camera performs culling calculations to determine which objects should not be drawn and can be supplemented by precomputed (“baked”) occlusion for all static objects for better performance. This process saves both CPU and GPU time, because there is less geometry to be drawn per frame. The performance impact of this process is dependent on scene size, object count, and camera count [
29] Only two cameras were used in the VR rig (one for each eye), aiming to balance the scene layout and object count so as to have as few objects as possible while occluding those necessary.
2.3.4. Achieving Visual Fidelity
A decent level of visual fidelity had to be maintained while keeping all optimization steps in mind. There are many factors that affect visual fidelity, but the two most important ones are shader workflow and post-processing.
Shader Workflow
Unity3d offers two shader workflows by default, the metallic workflow and the specular workflow [
30]. The textures and texture maps that were created in the previous steps were designed with the metallic workflow in mind, because it was considered that it would offer a better representation of the actual material being in mind during the design. The specular workflow gave all materials a more “plastic” quality than actually desired, meaning some objects became too glossy when they should have been rough (e.g., the wood deck floor in
Figure 15.
Post-Processing
There is a variety of post-processing filters available for use in Unity3d. Only anti-aliasing was used to reduce the number of jagged edges, bloom to enhance the bright spots of the scene (mostly light reflections), ambient occlusion to emphasize the difference between directly and indirectly lit surfaces of the models, and color grading to give a better look to the models. Of note, is the slight orange-yellow tint added and the enhancement of the shadows, especially between the lifeboats (
Figure 16).
2.3.5. Adding Functionality to the Application
Unity3d uses the C# scripting language, so all the scripts are written in C# based on object-oriented and component-based architecture. The scripts were organized into three main categories: scene management, virtual reality implementation, and user interface.
Scene Management
Scene management is relevant to every three-dimensional object, audio, language management, and every function created to support this application besides VR scripts and UI scripts. The majority of the objects are static, due to the limited interaction with the user (museum limitation due to time constraints), so their management is restrained to updates of the player’s position (ship, deck, cabin, masts).
There are some non-static objects, mainly the information points (tool-tips). Each information point indicates a position of interest, containing additional data (two pictures, 1. only info point, 2. opened info). All information points are buttons that display historical documents and pictures towards the player’s field of vision. The scripts that were developed for the information points are more complicated when compared to the rest of the scripts in this category. The aim was to develop scripts as generic as possible for the displaying images (multiple or none) in different sizes for each information point along with text (long or short). This was because information points had to be placed in various places on all ships (images for different aspects of information points: 1, only text; 2, text and one image; 3, text with two images side by side; 4, multiple text with multiple images).
As mentioned above, scripts were developed for audio and language management. We used background music and sound effects that change with the user’s position and enhance the VR experience. Furthermore, scripts that allow the use of multiple display languages were created.
Virtual Reality Implementation
Virtual reality implementation scripts/functionalities pertain to the user’s capability of moving in the three-dimensional world and the “bridging” between the Oculus device and the Unity3d engine. For the VR implementation it was decided to use the Unity asset “VR Interaction Framework” [
31], with some adjustments to meet specific needs.
User Interface (UI)
The user interface (UI) script category concerns the interaction between the user and the 3D environment. The greater number of scripts that were developed for the UI involves buttons that are embedded in the 3D environment and buttons that are ingrained in the user (hands/controllers). Simple code sequences for the buttons were designed for two reasons: firstly, these button-scripts may be executed continuously, and complex code could affect the application’s performance; secondly, the use of the buttons’ functionality is limited to changing the player’s position or to displaying objects (UI), so no complicated code chains are needed.
The VR Interaction Framework that was used contained models for virtual hands to give a better understanding of the user’s position. Instructions were added, that can be toggled on or off, to each of the user’s virtual hands in order to help the user identify the functions of each button and also to reduce the time that they would need to become accustomed to the VR controls. Controller models were also added to the virtual hands, to make it even easier for the users to orient their fingers on the controllers.
After setting up the scenes (one for each ship, plus one home scene), a map was created, on which there was one button for each ship. Each button has a different image and is easily discernible. Clicking one of the ship buttons brings up an information panel, which contains basic information about each ship, including the captain’s name, crew number, ship type etc. The user can then click the “explore” button to transition to any selected ship. This map has been included in every scene and can be accessed at any time, provided the user is close enough to use the buttons. Each user starts on the home scene, so when the user takes the headset off it automatically transitions back to the home scene. Additionally, two buttons have been included in the map so that the user can change between Greek and English. The localization system is set up in a way that it is easy to add more languages, should that be necessary during the exhibit’s lifespan.
Another way of transitioning between scenes was also created. Since the ships are large in size, it would be very inefficient for the user to be forced to return to the map in order to change scenes. Therefore, a portable UI panel was created. This is a simplified version of the map, as it only contains the images of the ships. The users can toggle this portable UI on or off with a button and transition between scenes from wherever they desire. This portable UI also allows the user to change freely between languages.
2.4. Changes Based on User Feedback
2.4.1. Fixes before Release
As with all applications, there were a few issues that occurred during the testing period. In order to fix all the bugs with the code and all visual issues faster, we consulted with people from the 3D application field (mostly friends and family).
The first issue was visual fidelity. At the start of development, much focus was placed on optimization, without giving enough weight to visual fidelity. However, with the techniques described in the previous chapter, such as baked lighting, this visual issue was fixed with no performance penalty.
The second issue was the way scene transitions were handled. At first, the only way was through the static map. The best suggestion was to create a portable UI that was simpler and smaller than the map (in order to avoid clipping with the surrounding geometry as much as possible), which was implemented as described in the previous chapter.
There were several issues with text displaying in the application outside of the Unity3d environment. These had to do with font quality and z-fighting, both of which were fixed.
Bakery has a mode for Lightmap baking called “SH” (short for spherical harmonics). This mode produces much more realistic results than the one used, but it did not function properly on the standalone application, so “full lighting” was determined to be the best choice. The reason behind the “SH” mode not functioning properly is unclear.
There was an issue with Oculus VR Interaction DLL, which rendered all of the VR Interaction Framework’s functionality unusable. This was fixed with an update to that specific DLL. Additionally, a few issues with the Oculus Dash support in the application existed at first, but those were also fixed with an update to the same DLL.
2.4.2. Problems Identified after Release
The only major bug found after release was a bug referring to the portable UI described in the previous chapter. If the user was fast enough to click the “explore” button while the fade between scenes was still happening, the application would crash. Fortunately, this bug was quickly patched before the exhibit was opened to the public.
There were issues with the user’s movement in a few places on the ships, which were caused by some misplaced colliders. This was quickly fixed, otherwise it made some areas of the ship inaccessible.
There were some minor changes on the flag textures of the ships, because the initial flags did not match the actual flags used by the navies in the 1800s.