Next Article in Journal
To Collaborate or Not: The Autonomous Vehicles Introduction Strategy of the Traditional Ride-Hailing Platform
Previous Article in Journal
The Impact of System Models on Systems Thinking in the Context of Climate Change in Elementary School
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Systems
Volume 13
Issue 4
10.3390/systems13040221
systems-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Evaluating the Benefits and Drawbacks of Visualizing Systems Modeling Language (SysML) Diagrams in the 3D Virtual Reality Environment

by
Mostafa Lutfi
* and
Ricardo Valerdi
Department of Systems and Industrial Engineering, The University of Arizona, Tucson, AZ 85721, USA
*
Author to whom correspondence should be addressed.
Systems 2025, 13(4), 221; https://doi.org/10.3390/systems13040221
Submission received: 17 January 2025 / Revised: 16 March 2025 / Accepted: 21 March 2025 / Published: 23 March 2025
(This article belongs to the Special Issue Digital Transformation of Engineering 4.0 and beyond with Model-Based Systems Engineering)
Browse Figures
Versions Notes

Abstract

:
Model-Based Systems Engineering (MBSE) prioritizes system design through models rather than documents, and it is implemented with the Systems Modeling Language (SysML), which is the state-of-the-art language in academia and industry. Virtual Reality (VR), an immersive visualization technology, can simulate reality in virtual environments with varying degrees of fidelity. In recent years, the technology industry has invested substantially in the development of head-mounted displays (HMDs) and related virtual reality (VR) technologies. Various research has suggested that VR-based immersive design reviews enhance system issue/fault identification, collaboration, focus, and presence compared to non-immersive approaches. Additionally, several research efforts have demonstrated that the VR environment provides higher understanding and knowledge retention levels than traditional approaches. In recent years, multiple attempts have been made to visualize conventional 2D SysML diagrams in a virtual reality environment. To the best of the author’s knowledge, no empirical evaluation has been performed to analyze the benefits and drawbacks of visualizing SysML diagrams in a VR environment. Hence, the authors aimed to evaluate four key benefit types and drawbacks through experiments with human subjects. The authors chose four benefit types—Systems Understanding, Information Sharing, Modeling and Training Experience, and Digital Twin based on the MBSE value and benefits review performed by researchers and benefits claimed by the evaluations for similar visual formalism languages. Experiments were conducted to compare the understanding, interaction, and knowledge retention for 3D VR and conventional 2D SysML diagrams. The authors chose a ground-based telescope system as the system of interest (SOI) for system modeling. The authors utilized a standalone wireless HMD unit for a virtual reality experience, which enabled experiments to be conducted irrespective of location. Students and experts from multiple disciplines, including systems engineering, participated in the experiment and provided their opinions on the VR SysML implementation. The knowledge test, perceived evaluation results, and post-completion surveys were analyzed to determine whether the 3D VR SysML implementation improved these benefits and identified potential drawbacks. The authors utilized a few VR scenario efficacy measures, namely the Simulation Sickness Questionnaire (SSQ) and System Usability Scale (SUS), to avoid evaluation design-related anomalies.

1. Introduction

Model-Based Systems Engineering (MBSE) is a formalized methodology that employs digital models for the design and analysis of systems and is utilized throughout the entire lifecycle of a system, from initial conception to eventual decommissioning. Contemporary systems engineers operate in an era characterized by the escalating complexity of modern systems [1,2]. To manage the complexities inherent in various Systems of Interest(SOI), researchers have proposed MBSE as an effective solution [3,4,5,6]. In the traditional document-centric systems engineering approach, any change in a requirement specification will mandate changes in multiple documents, where the specification parameter is utilized or mentioned. In a similar scenario, the responsible systems engineer will update the requirement specification once within the MBSE tool and propagate the change throughout the system model with less effort than in the document-centric approach. In addition, MBSE standardizes how system engineering artifacts should be defined for a system of interest, such as requirements, operational concepts, use cases, architecture, and behaviors. This standardization of systems engineering practice reduces the cost and effort required to develop a project or system by eliminating inconsistencies and waste early in the design process.
In recent years, several notable Model-Based Systems Engineering (MBSE) methodologies have been developed, including the Object-Oriented Systems Engineering Method (OOSEM), Object-Process Methodology (OPM), Model Driven Development Methodology (MDDM), State Analysis, Systems Modeling Toolbox (SYSMOD), IBM Harmony, and Arcadia-Capella [7,8,9]. Among these, OOSEM employs the Systems Modeling Language (SysML), which offers superior tool support compared to other methodologies. Consequently, SysML has become the most widely used MBSE language in both academic and industrial contexts [10,11,12]. According to Friedenthal, Moore, and Steiner, “SysML is a general-purpose graphical modeling language that supports the analysis, specification, design, verification, and validation for complex systems” [13]. SysML consists of nine diagram types, namely Package Diagram, Requirements Diagram, Block Definition Diagram (BDD), Internal Block Diagram (IBD), Use Case Diagram, Activity Diagram, State Machine Diagram, Sequence Diagram, and Parametric Diagram (Figure 1) [14].
In recent years, federal agencies, including the Department of Defense (DoD), NASA, defense contractors, National Laboratories, and other private organizations, have initiated a push for a Digital Engineering (DE) strategy that focuses on a combination of systems engineering practices (MBSE) with Digital Twin (DT), computational technology, analytics, data sciences, and industrial automation approaches [15,16]. A DT is a virtual depiction of an object or system that spans its lifecycle. It is frequently updated with real-time data and uses simulation, visualization, input from IoT sensors on physical equipment, and machine learning to optimize decision-making [17,18]. Digital models consist of Physical Twins and Virtual Twins and their interactions through data exchange. The physical space is a physical representation of the objects, virtual assets are graphical representations of the physical objects, and a database stores the interactions and feedback between the two spaces. The virtual twin can be visualized using various methods, including VR, Augmented Reality (AR), Mixed Reality (MR), dashboard technologies, and 2D simulations [19,20,21]. VR is a fast-developing technology of the 21st century, especially in the fields of education, gaming, and healthcare [22,23,24,25]. According to Coates et al., VR is an electronic simulation of environments with a head-mounted display and wired outfit, which allows the end user to interact in realistic three-dimensional situations [26]. A typical VR system comprises VR software, Head Mount Display (HMD), tracking sensors, feedback devices, onboard/external computers, and users [27,28]. Advancements in display technology, including higher resolution displays and faster refresh rates, have contributed to improved visual clarity and reduced latency for HMDs [29]. Different engineering fields have also leveraged VR technologies to accomplish various tasks, such as VR-based Civil Engineering training, education training on Manufacturing Sustainability for Industry 4.0, and aerospace training [30,31,32,33].
Due to technological advancements and increased design complexity, systems are becoming more difficult to represent using static modeling techniques. Hence, the corresponding SysML models are also more complex and larger due to the system’s complexity and size. Similarly, 2D SysML models are becoming increasingly difficult to comprehend. Moreover, the increasing adoption of DT technologies in systems engineering projects presents opportunities for integrating MBSE and DT methodologies, thereby enabling systems engineers to test, visualize, and operate large-scale system models [34,35,36]. Physical and virtual models of a DT can be integrated with a SysML system model to establish a communication hub between the DT infrastructure and systems engineering artifacts [37]. Furthermore, Model-Based Systems Engineering (MBSE) initiatives can facilitate the early integration of Digital Twin (DT) implementations within a program lifecycle [38,39]. Therefore, communicating SysML models with Virtual Reality tools enhances the DT experience and system understanding. As most VR environment tools are also equipped to develop AR/MR models, VR-SysML interoperability can be expanded to enable the integration of SysML models with AR/MR-based DT infrastructure [20,40].
Recent research findings have demonstrated that VR-enabled immersive design reviews help participants detect more design issues and faults [41,42]. Furthermore, compared to non-immersive approaches, these immersive methods augment collaborative engagement, focus, and sense of presence among participants [43,44,45]. Moreover, various researchers have found VR environments beneficial for visualizing complex structures and providing higher levels of understanding and knowledge retention than conventional two-dimensional approaches in the Chemical and Medical domains [46,47,48,49]. As SysML models can be highly complex if the system under consideration is a large interconnected system of subsystems, VR-based SysML diagrams can potentially improve the understanding of the system.
Several researchers have studied the effect of utilizing a 3D VR environment to visualize Visual formalism languages other than SysML. For example, multiple studies have confirmed that a 3D VR environment significantly enhances Business Process Models and Notation (BPMN) knowledge retention and collaborative processes [50,51,52]. Likewise, visualizing UML diagrams in a 3D VR environment enhances comprehension speed and provides a more engaging modeling experience than a conventional 2D environment [53,54]. As SysML is also a visual formalism language, the authors can elicit that the visualization of SysML diagrams in a 3D VR environment has the potential to enhance model understanding for complex systems and enable an engaging, collaborative environment among users and stakeholders compared to SysML diagrams in the 2D environment. However, there is a need to experimentally evaluate the suggested benefits of visualizing SysML diagrams in a 3D VR environment.
The authors chose the ground-based telescope system as the SOI due to the following factors: telescope systems heavily rely on digital data products, which makes them ideal for DE implementation; telescope systems need to generate virtual scenarios to analyze different aspects of scientific findings; and few MBSE reference models are available for ground-based telescope systems.
The following benefits are chosen for the VR-SysML evaluation based on the MBSE value and benefits review performed by Henderson and Salado and benefits claimed by the evaluations for similar visual formalism languages (e.g., UML, BPMN) [50,51,55,56]. Table 1 summarizes the selected benefit types.
This research paper evaluates the following question, “Does 3D VR SysML diagram visualization improve the benefit types mentioned in Table 1 over conventional 2D SysML visualization?”.

2. Related Work

Various researchers worldwide have implemented multiple research methodologies to explore the idea of implementing SysML diagrams in a 3D VR environment [57,58]. However, to the best of the author’s knowledge, few evaluations involving human subjects have been performed to determine whether 3D VR SysML visualization/modeling provides additional benefits over conventional 2D SysML diagrams. Oberhauser evaluated the VR-SysML concept using design science methods and principles [59]. In that research paper, a case study was used to compare 2D SysML diagrams with VR-SysML diagrams, where SysML diagrams supported additional capabilities and visualizations were reviewed. The review process involved no human subjects or expert opinions and relied on the author’s expertise in comparing the diagrams. The outcome of that evaluation showed that a large space in VR facilitated the depiction and visual navigation of large models, with improved analysis of the connection between the elements, diagrams, and models. Romero et al. used a model-centric design of a GO-TO telescope from Celestron to demonstrate how SysML in an immersive virtual environment supports the reviewing activities [60]. However, they only reviewed the design aspects of SysML in a VR environment by comparing them with 2D diagrams and CAD models. They suggested that SysML in a VR environment might encourage stakeholder communication, but no evaluation was performed to validate this claim. Although VR-SysML evaluations involving human subjects are scarce, there have been multiple evaluations of UML (SysML reused part of their diagrams from UML) 3D visualization involving human participants. Rodrigues et al. performed an experimental study to evaluate their VisArch3D approach, a UML-based software architecture visualization in 3D [61]. Eighteen participants were involved in that study, and the outcome of the study suggested that a 3D environment improved the understanding of UML models in systems with many elements. Yigitbas et al. developed a gamified UML learning environment named “GaMoVR” [62]. Then, they evaluated the effectiveness of the 3D game to help learn the UML class diagram modeling and compared it with a 2D desktop-based game. The study included 16 participants and used the MEEGA360 questionnaire. The study’s outcome indicated that GaMoVR significantly improved users’ motivation, enjoyment, and engagement, alongside perceived improvement in the overall learning outcome.

3. Methodology

The methodology section comprises four subsections. The first two sections delve into the development of 2D and 3D SysML System Models utilizing the Cameo Systems Modeler and Unity [63,64]. The third section outlines the evaluation setup procedures. The final section addresses considerations about the efficacy of virtual reality scenarios.

3.1. SysML Models in 2D Environment

In this research paper, the authors utilize the following SysML diagram types to depict the system hierarchy, requirements, structure, and behavior—Package Diagram, Requirement Diagram, Block Definition Diagram (BDD), Internal Block Diagram (IBD), Use Case Diagram, Activity Diagram, State Machine Diagram, and Parametric Diagram. Requirement Diagrams illustrate various requirements—such as system-level and interface requirements—and their traceability to other requirements, the model elements that fulfill them, and the test cases that verify them [65]. BDDs depict a system’s architecture by showing the hierarchical relationships among subsystem components and classifiers [66]. IBDs present parts, properties, ports, interfaces, and connectors within a specific block or system context, revealing the system’s internal structure and communication pathways [66]. Use Case Diagrams are used to visualize the relationships between a system’s principal functions and its human and non-human actors [66]. Activity Diagrams represent scenarios and behaviors at the system, subsystem, and component levels by illustrating the sequence of data and control flows between actions [66,67]. State Machine Diagrams convey information about dynamic system behaviors, including transitions between system states, the lifecycle of a block through various states, event triggers, and transition elements [66]. Finally, Parametric Diagrams are a distinct type of IBD that integrate structural and behavioral model elements with engineering, simulation, or mathematical models, enabling detailed analysis and verification of system properties [66,68].
The 2D system model uses the Cameo Systems Modeler and encompasses various Vera C. Rubin Observatory Subsystems. The Ground-Based Telescope System Model developed for this research is a large model consisting of multiple instances of the SysML diagram types. Hence, each diagram-type example is included here to demonstrate how SysML diagrams are used to model different system artifacts for the SOI.
Figure 2 shows the SysML package diagram of the system model, which demonstrates its hierarchy. In the hierarchy, the Requirements package comprises requirements diagrams detailing the various system requirements and their relationships. The Structure package encompasses the system architecture and communication interfaces, defining the relationship between the system’s composition and interactions. The Architecture package comprises a Telescope Mount System, Telescope Structural Features, a Supporting Telescope, and a Data Management System. However, only the Telescope Mount system was modeled in both SysML 2D and SysML 3D VR environments, as each system consists of many model elements. The Telescope Mount System consists of 562 subsystems and components and is spread over five levels of decomposition, denoted as L0 to L4. The Behavior package includes use cases, operational scenarios, and system states that capture the system’s dynamic behavior and functional interactions. Lastly, the parametric package incorporates in-model parameter calculations, data exchange mechanisms, and integration components that interface with external simulation and analysis tools, facilitating a detailed analysis and validation of the system performance.
This study uses SysML requirement diagrams to represent the requirement specifications and their relationships with lower-level requirements. Relationship types between requirements and lower-level requirements and/or other related elements are “containment”, “derive”, “refine”, “satisfy”, “verify”, and “trace”. Figure 3 shows a SysML requirement diagram with a containment relationship between the parent and child requirements for a telescope data collection system.
SysML BDDs depict the system architecture and relationships between the components of the different subsystems of the ground-based telescope. Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11 demonstrate how the authors decomposed different subsystems to a lower level using the BDD. Users can also specify the multiplicity to define the number of components necessary for a given subsystem. These figures also show the use of navigators to return to the previous level of decomposition.
Figure 8 depicts a lower-level block “Filter” with its compartment expanded to show the value property “FilterSize”. Figure 12 traces this value property to a pumping system requirement, “Filter Diameter”, with a constrained value (0.75 inches).
SysML IBDs represent communication and data flow between different subsystems. Specifically, the IBDs represent the interconnections, interfaces, and ports between the parts of a block. For example, Figure 13 depicts the telescope control system data flows. Signals are sent from one port to another through connectors to depict the data transfer between system components.
The authors use SysML Use Case diagrams to illustrate the system’s use cases and operations concept (OpsCon). Figure 14 depicts the data analysis of the Operational Concept (OpsCon), incorporating use case elements, human actors, and their interrelationships. However, non-human actors are typically represented as blocks in the diagram. The key relationship types available for a use case diagram are communication path, extend, include, and generalization.
The authors utilize SysML Activity diagrams to visualize the behavior of different subsystems, operational scenarios, and workflows for the proposed ground-based telescope system. In Figure 15, an activity diagram shows an operational scenario for disabling the telescope motions, where the dotted lines represent command flows between the activities. Solid lines can be used to represent data flows between the actions. In the diagram, actions are depicted as rectangular elements, each representing a specific operation being executed.
The authors deploy SysML state machine diagrams to model different operational states and transitions between states triggered by events, signals, and values. Figure 16 shows the telescope mount system’s operational states and their transition paths triggered by the signals. In this state machine diagram, when an errrorIsTrue signal is triggered, the system transitions to the FaultState from the DisabledState or EnabledState. If the responsible systems engineer decides that a maintenance activity is required to fix the error, a maintenance signal will be triggered, and a transition will occur from FaultState to MaintenanceState.
SysML parametric diagrams facilitate the combination of system models with engineering analysis models (e.g., performance, reliability, cost analysis, and simulation). However, a parametric diagram is more efficient than an activity diagram for handling high volumes of data and calculations when modeling large and intricate systems. Figure 17 shows a parametric diagram for calculating the mirror Fried Parameter allocation, where the constraint block contains the corresponding equation.

3.2. SysML Models in 3D VR Environment

A portion of the SysML system model in the Cameo Systems Modeler was then modeled in the VR environment using Unity 3D [64]. Nodes and edges were used to model the majority of the SysML diagrams. Wireless Meta Quest VR headsets (Figure 18) and controllers were used to develop, test, and later evaluate those 3D VR SysML diagrams [69]. The model consists of multiple scenes that visualize the different SysML diagram types as 3D viewpoints.
The 3D VR SysML model consists of multiple UI elements to help users navigate it. Figure 19 shows a UI component within the scene that shows the instructions. These instructions can be hidden or shown by triggering specific buttons on the headset controllers.
Package diagrams in SysML models provide an overview of the model’s structure. In the 3D VR SysML system model, the package diagram utilizes ‘UI Canvas’ and ‘Buttons’ to depict the four pillars of SysML and subsequent lower-level folders (Figure 2 and Figure 20). These buttons can be selected through the Rays being cast from the controllers within a scene. Hence, the user can navigate through the packages using the UI element scene and find the corresponding SysML diagrams.
SysML diagrams are drawn using the fundamentals of graph theory, that is, nodes and edges [70,71]. Here, cube or rectangular nodes represent the diagram’s elements (Block, State, Requirement, etc.), and the edges connect the elements. The diagrams can be imported from the 2D SysML model using the methodology developed by the authors previously [72]. However, for this evaluation research paper, a combination of automatic import and manual modeling was used to create different types of scenarios. These VR scenarios aim to enhance the 2D SysML experiences due to the natural benefit of 3D VR space. Figure 21 and Figure 22 show how selecting a block in a BDD can reveal the lower-level diagrams associated with the selected subsystem. Users can select a block by hovering over the ray using the controller within the scene. As soon as the subsystem block is selected, its color will change to green, and the lower-level diagram (if any) will be shown. In the 2D SysML model (Figure 5 and Figure 6), the user must navigate to the lower-level containment or use a hyperlinked shortcut to navigate to the lower- or higher-level diagrams. In contrast, in the 3D VR environment, multiple diagrams can be shown within the same viewpoint, enabling a higher sense of traceability between subsystems and components.
The next scenario demonstrates how the SysML BDD diagram communicates seamlessly with the virtual twin components to improve the understanding of the system components and enable an integrated design review. As shown in Figure 23 and Figure 24, when the harmonic drive block is selected, it shows the corresponding virtual twin (green) in the actuator. In addition, because the harmonic drive is an inner element, the outer components automatically become transparent to facilitate the ease of view. This feature is also applicable to large complex structures. Figure 25 and Figure 26 demonstrate a scenario in which a user can teleport to the inside of a large, complex object (Telescope Structures) utilizing the benefits of the 3D virtual environment. When the user presses the corresponding trigger buttons after selecting a block, the additional menu options in the UI enable the teleport function if the selected subsystem/component requires teleportation.
The succeeding scenario demonstrates how behavior diagrams can be made real-time interactive in the 3D SysML Model compared to the 2D SysML model. Figure 27 shows a UI element depicting a control room screen where telescope personnel interact with the control system software. A SysML state machine diagram accompanies the control screen to help understand the states and triggers based on the action performed on the screen. In addition, the green state indicates the system’s current state, which is FaultState (Figure 16 shows the same SysML diagram in 2D). However, as soon as the telescope personnel select yes to schedule maintenance, the state machine diagram of the MaintenanceState will show green, and a message will confirm the state on the control screen (Figure 28). Hence, this type of behavior scenario in the 3D SysML VR model facilitates faster training for the SysML diagrams and complex scenarios depicted by the corresponding behavior diagrams.
Another scenario developed in this research paper demonstrates how the 3D SysML VR model can improve the visualization of related multi-diagram elements compared to the 2D SysML model diagrams. Figure 29 shows the same BDD of the pumping system as that shown in Figure 8. The value property FilterSize can be observed in the compartments of the filter block in both diagrams. However, in the 3D SysML VR model, users can see the traced requirement for the oil pump filter diameter by selecting the value property (Figure 30). On the contrary, in the 2D SysML model, diagrams 8 and 12, or custom views showing both types of elements (which increases the modeling effort), are required to achieve the same level of understanding. Hence, the 3D SysML VR model reduces model navigation and provides an improved traceability overview.

3.3. Evaluation Set-Up

The evaluation consisted of two parts. In the first part, two groups of 14 (2D SysML vs. VR 3D SysML), totaling 28, participated in a knowledge test, which consisted of 15 questions. The 15 questions were asked based on the SysML 3D VR Scenarios and the corresponding 2D SysML diagrams. These questions are related to Model Understanding, Information Sharing, and Language Training (which high-level system contains an Earthquake Detection Digitizer? and how many different types of elements are shown in this diagram? If the error is resolved, which state will be in effect? etc). Meta Quest wireless VR headset and controllers were loaded with the 3D SysML VR Model scenarios, and a laptop was loaded with the COTS 2D SysML software namely Cameo Systems Modeler 2024x alongside individual drawing viewers (AutoCAD, SolidWorks) [63,69,74,75]. This setup facilitated the portable participation of users and the parity of model reviews (2D vs. 3D). The table below lists the questions used in this test. The test duration was 20 min.
In the second part, 30 participants were exposed to both the SysML 2D System Model and the SysML 3D VR System Model. Then, the participants completed a survey asking them about the benefit types and whether these benefits were improved/worsened/equal/indeterminable for the 3D VR model compared to the conventional 2D model. The survey included additional benefit evaluation questions for the participants who had at least limited experience with DT regarding whether the 3D SysML VR model improved their DT integration experience.
Hence, the two-part evaluation approach reduced the participants’ subjective bias (if any) in evaluating the 2D and 3D SysML diagrams. The first part of the knowledge test evaluated how well the participants understood the assigned types of SysML diagrams (2D/3D). Then, they experienced the alternate diagram model to complete the evaluation for part two. Part one did not include two participants because they were very experienced in SysML diagrams and could easily answer the questions from either of the tests. This evaluation was intended for external stakeholders, cross-discipline engineers, and system engineers without experience with SysML. The participants include students and professionals from various disciplines, including systems engineering.

3.4. VR Scenario Efficacy Considerations

The authors developed 3D SysML VR model scenarios considering user-friendliness. The user can explore scenarios while sitting on a chair and walking casually within a small room. The meta quest headset’s initial mapping capability enables obstacle alerts within the room’s perimeter. To make the turning experience effortless, the “snap turn” function is incorporated into the controller functions so that the user can turn with a click of the joystick in the desired direction. The scenes are free of eye-straining colors and have comfortable lighting. All instructions for the different UI elements are available with the click of a button on the controllers. In addition, most VR scenarios are staged within a telescope control room setup, reducing the infinite feeling and making the users feel at home.
Research findings have suggested that simulation sickness may hinder the overall efficacy of VR scenarios. Kennedy et al. developed a simulation sickness questionnaire encompassing 21 symptoms that can arise from exposure to virtual environments [76]. They defined three symptom clusters and corresponding weights through a factor analysis of approximately 1200 simulator sessions. Hence, the symptoms are categorized into three distinct areas: nausea, oculomotor disturbances, and disorientation. The questionnaire assesses the virtual reality environment’s simulation sickness by enabling users to rate their discomfort level (0–3). The overall score is computed by summing each category’s weighted scores and multiplying the aggregate by 3.74. The specific weights assigned to nausea, oculomotor disturbance, and disorientation calculation are 9.54, 7.58, and 13.92, respectively. Although the scenarios used for this research paper utilized Unity’s template scenes (lighting and frame rates are appropriate for the average users) to reduce simulation sickness, SSQ scores were recorded pre- and post-evaluation to eliminate the potential impact of simulation sickness on the evaluation results. Table 2 lists the symptoms assessed in the SSQ, categorized by Nausea, Oculomotor, and Disorientation.
The author’s next efficacy measure is System Usability (SU). SU pertains to how users can interact with a system efficiently and effectively to achieve their objectives. It evaluates the system’s ease of use, learnability, and operational efficiency, while also considering contextual factors and user requirements during interaction. Brooke developed an easy, reliable, and widely used Systems Usability Scale (SUS) [77]. The authors modified the existing SUS specifically for a VR system, consisting of 10 statements on a 5-point Likert scale. The final score of the usability study is calculated by summing the scores of all items and then multiplying the resulting sum by 2.5. Table 3 lists the SUS questions used in this VR scenario evaluation.

4. Results

As SysML 2D and SysML 3D VR knowledge tests were performed using two different groups, a t-test was performed to determine whether the two groups differed from each other in the knowledge test [78]. As shown in Figure 31, all the test results fall approximately along the reference line, and the Shapiro test confirmed that the two p-values are greater than the significance level of 0.05 (0.06, 0.25) [79]. Hence, the authors can confidently assert that the distribution of the test results was not significantly different from the normal distribution.
As 2D SysML and 3D VR SysML knowledge tests were performed using two different groups, a t-test was performed to determine whether the two groups differed from each other in the knowledge test [78]. The t-test results (Figure 32) showed that the mean test scores of the SysML 3D VR group were increased by 2.43 as compared to the SysML 2D group (SysML 2D: 8.86 ± 1.51, SysML VR:11.29 ± 1.27, 95% CI: −1.35 to −3.51, t = 4.60, df = 26, p ≤ 0.0001). Based on the test results, the authors can conclude that the SysML model knowledge test performance improved in a 3D VR environment compared to a conventional 2D environment.
After the knowledge test, a perceptive evaluation was completed, where the participants (N = 30, except for DT Integration, where N = 12) experienced both the 2D and 3D VR SysML models and then completed a survey. The survey asked the participants whether the SysML 3D VR System Model improved or worsened the benefit types compared with the conventional SysML 2D system model. Overall, the participants’ perceptions were in line with the knowledge test results and showed that most of the participants agreed with the statement that the 3D VR model improved the four benefits defined for this evaluation (Figure 33). “Modeling and Training” and “DT Integration” experiences (70%; 75%) received higher approvals among the participants than the “Systems Understanding” and “Communication or Information sharing” (63.3%; 60%). Figure 34 shows that increased benefit approvals were higher among the other discipline participants than among the systems engineering domain personnel. Figure 35 summarizes the overall approval and disapproval of the participants for each benefit type (SysML 3D VR vs. SysML 2D VR). Participants were also asked to highlight the drawbacks of visualizing SysML diagrams in a VR environment. Table 4 presents the major drawbacks identified by the participants based on their experiences with SysML 3D VR scenarios developed within the scope of this research paper.
Before jumping onto the 3D VR SysML model scenarios, the participants played demo scenarios available within the Meta Quest Controllers to familiarize themselves with the VR environment in general. The participants completed a pre-evaluation SSQ based on their prior experience with simulation or gaming software. The scores for nausea(N), oculomotor(O), and disorientation(D) in the pre-evaluation SSQ were 9.86, 13.90, and 14.38, respectively. The overall SSQ score (TS) was 14.59. These results suggested that the participants had experienced an acceptable degree of simulation sickness symptoms prior to this evaluation. After the evaluation, the participants were asked to complete a post-evaluation SSQ. The results of the post-evaluation SSQ were as follows: N—12.72, O—14.65, D—16.24, and TS—16.58. For a VR-related evaluation, these score increases were minimal and did not require any additional modifications to the test setup and scenarios. The authors found that the pre-evaluation SSQ scores were not significantly different from the post-evaluation SSQ scores, and the VR scenarios had no additional effect on the participants. The overall SUS score for the participants was 76.2 out of 100, which was greater than the minimum required score of 68 [80]. Hence, the authors could conclude that the VR scenarios used in the evaluation had above-average usability.

5. Discussion

Based on the results discussed above, the authors can assert that both the knowledge test and participants’ perception of the defined benefit types are in line and show a positive improvement for the 3D VR SysML model visualization compared to the conventional 2D SysML model visualization. The knowledge test results show improvement in the 3D VR SysML group because of the natural benefits of the VR scenarios (Figure 32). The 3D VR environment provides more enjoyable and engaging experiences than conventional constrained 2D environments. The unlimited space in the VR environment enables easier access to toggle and find relationships between higher- and lower-level subsystems and their components. In contrast, in the 2D SysML model, participants must navigate from diagram to diagram, generating additional cognitive load to perform the same task. Navigating through the 3D VR SysML model is analogous to gamified training, which utilizes game dynamics to motivate participants to complete training [31,62,81]. For example, instead of working on two separate applications to enable the Maintenance State (Cameo Systems Modeler for state machine diagram reference and control system interface to enable the commands), the VR model participants can utilize a scenario to perform the same task with improved motivation as they notice a real-time change in the state machine diagram within the same field of view (Figure 27 and Figure 28). VR scenarios enable virtual twin dissection accompanied by SysML BDDs and virtual tours to understand the larger subsystems and components, which cannot be matched with the 2D limited screen space. Recent advancements in the field of VR headsets have paved an easier path for the adoption of gamified 3D VR SysML models.
Even though the evaluation has shown positive results for the research question, a few issues are evident in the results. Figure 34 shows that the SE participants were less bullish on the 3D VR SysML model than the non-SE participants. This is because SE personnel are more familiar with systems engineering fundamentals than are non-SE personnel. This phenomenon is non-existent for Communication or Information Sharing, where both groups agree to an equal perception of improvement for the 3D VR SysML model. Among the four benefit types, “Modeling and Training Experience” and “DT Integration Experience” showed more positive results for the 3D VR SysML model (Figure 33). The authors elucidate that these two benefit types depend highly on interactivity, and the VR environment provides an engaging, interactive experience. If a SysML BDD supports a virtual twin system, the relationship between the components can be easily understood.
The evaluation also identifies the major drawbacks of the 3D VR SysML model, as shown in Table 4. The lack of adoption of VR technology is a key issue for the gamified SysML modeling approach in a VR environment. The cumbersome nature of the headsets is a problem. However, the technology is evolving rapidly, and VR headsets are becoming lighter and more capable [82]. In addition, mixed-reality sunglasses are entering the market, which may facilitate higher adoption among potential users. This research utilized a lightweight wireless headset that could be worn for longer periods; however, the participants marked “longer usability” as a key issue. A user may need to work for 8 long hours with breaks in a conventional office setting. According to the participants, the headsets must be lighter and comfortable enough to be worn all day without showing any signs of simulation sickness. Another key issue arises that is related to the modeling fundamentals of MBSE. Participants suggested that the key advantage of MBSE over document-centric SE is the reduction of redundancy by maintaining a single source of truth and reusing the same elements in multiple diagrams, if possible. This research paper utilizes a limited model, and redundancy is avoided chiefly. Modelers must be cautious while modeling in a 3D virtual environment so that even though the scenarios may change, the number of objects and prefabs remains as minimal as possible. As this is a new type of viewpoint compared to the conventional SysML diagrams, the participants suggested that during the early phase of the development, a conventional 2D model should always back up the VR model. Finally, the lack of keyboard support is another flaw of the 3D VR SysML model. This problem applies to any VR system, as typing with a controller on a virtual keyboard is sometimes difficult and requires practice. However, this research only evaluates the visualization and light interaction with the SysML diagrams in a VR environment. The authors plan to improve the 3D VR SysML model by enabling text input and updating the diagrams in future iterations.
Although in this research paper the authors have studied only the visualization of the SysML model in a VR environment, how VR can be integrated into the MBSE lifecycle needs to be addressed. As the VR environment facilitates the simulation of real-world objects with ample space and scalability options, it can facilitate the early design review of huge and complex structures (as shown in Figure 25 and Figure 26). The architecture and automobile industries already utilize VR technologies to develop, review, and update their product designs [83,84,85,86]. If interactable SysML diagrams accompany these VR implementations, users can update the design parameters, verify system requirements, and perform the trade-study analysis while staying in the same environment, which will, in turn, reduce the overall cost and effort. Digital twins can be integrated with MBSE tool suites through a digital thread, potentially becoming a fundamental component of model-based systems engineering (MBSE) [35,58]. MBSE tools can serve as the central hub of the integrated system, where they achieve two-way communication between the physical and virtual twin, enable remote digital twin, especially for monitoring operations and maintenance activities, and design digital twin artifacts in the VR environment through the ontological interplay of SysML diagrams. In their previous work, the authors demonstrated how two-way communication can be achieved between VR and MBSE tools [72]. As MBSE is a key aspect of creating unified data for the system lifecycle, two-way communication between MBSE and VR tools enables the maintenance of the VR-MBSE approach as a single source of truth. For example, simulation results within the VR environment can be imported into a SysML parametric diagram, which verifies a requirement, and input parameters can be transferred from the SysML model to the VR simulation scenario. The authors demonstrated how a scenario could accompany a state machine diagram to improve the understanding of the system (see Figure 27 and Figure 28). A similar implementation was demonstrated by Barosan et al., where a state machine diagram accompanied the visual representation of the fuel filling station states in the VR environment [87]. The state machine diagram’s state color changed based on the different transitions in the fuel filling station. These examples suggest that the behavior and properties of a system or subsystem can be visualized within a VR environment and accompanied by a behavior diagram (activity, state machine, use case) to facilitate early verification and validation. Hence, VR has the potential to be integrated into the preliminary design, verification, validation, and operations and maintenance stages of the MBSE-driven system lifecycle.

6. Conclusions

In this research paper, the authors evaluate four MBSE benefit types—system Understanding, Communication or Information Sharing, Modeling and Training Experience, and Digital Twin Integration Experience—for the 3D VR SysML system model compared to the conventional 2D SysML system model. Both the knowledge test and perceived evaluations confirm that 3D VR visualization of the SysML diagrams improves the benefit types than the 2D SysML visualization. The higher intractability, enthusiasm, and infinite space facilitate these positive results for the 3D VR environment and make it an ideal candidate for the gamification of SysML diagrams for training new systems engineers and for communication among various stakeholders. SysML diagrams accompanying the virtual twin component improve the overall understanding of the system and enhance the DT experience. However, 3D VR SysML modeling has some major issues (lack of adoption, cumbersome design, long-term usability, redundancy, less perceived reliability, and lack of text input) that must be rectified to be fully mass-employed, especially in industries.
There is scope for the multifaceted expansion of this research. SysML v2 will soon replace SysML v1 [88]. Unlike SysML v1, SysML v2 is no longer mostly derived from UML and supports textual notation and enhanced API support to facilitate interoperability and modeling. Hence, this 3D SysML model can be modified to visualize SysML v2 graphical diagrams. In addition, the evaluation can be expanded to mixed reality/augmented reality-based visualization of SysML diagrams and determine how it improves different MBSE benefit types. The authors completed the evaluation with a limited number of participants (thirty), which should be increased to a larger number to determine the scalability of the results.

Author Contributions

Conceptualization, M.L. and R.V.; methodology, M.L. and R.V.; software, M.L.; validation, M.L. and R.V.; formal analysis, M.L. and R.V.; investigation, M.L.; resources, M.L. and R.V.; writing—original draft preparation, M.L.; writing—review and editing, M.L. and R.V.; visualization, M.L.; supervision, R.V. All authors have read and agreed to the published version of the manuscript.

Funding

This research paper received no external funding.

Data Availability Statement

SSQ and SUS evaluation results were uploaded alongside the research paper. The remaining evaluation results are presented through the statistical diagrams used within the paper.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Calvano, C.N.; John, P. Systems engineering in an age of complexity. Syst. Eng. 2004, 7, 25–34. [Google Scholar] [CrossRef]
  2. Siddique, I. Systems engineering in complex systems: Challenges and strategies for success. Eur. J. Adv. Eng. Technol. 2022, 9, 61–66. [Google Scholar] [CrossRef]
  3. Morkevicius, A.; Aleksandraviciene, A.; Mazeika, D.; Bisikirskiene, L.; Strolia, Z. MBSE Grid: A Simplified SysML-Based Approach for Modeling Complex Systems. In Proceedings of the INCOSE International Symposium, Adelaide, Australia, 15–20 July 2017; Volume 27, pp. 136–150. [Google Scholar] [CrossRef]
  4. Duprez, J. An MBSE modeling approach to efficiently address complex systems and scalability. In Proceedings of the INCOSE International Symposium, Washington, DC, USA, 7–11 July 2018; Volume 28, pp. 940–954. [Google Scholar] [CrossRef]
  5. Zeigler, B.P.; Mittal, S.; Traore, M.K. MBSE with/out Simulation: State of the Art and Way Forward. Systems 2018, 6, 40. [Google Scholar] [CrossRef]
  6. Ciampa, P.D.; Nagel, B. Accelerating the Development of Complex Systems in Aeronautics via MBSE and MDAO: A Roadmap to Agility. In Proceedings of the AIAA Aviation 2021 Forum, Virtual Event, 2–6 August 2021; p. 3056. [Google Scholar] [CrossRef]
  7. Estefan, J.A. Survey of model-based systems engineering (MBSE) methodologies. Incose MBSE Focus Group 2007, 25, 1–12. [Google Scholar]
  8. Roques, P. MBSE with the ARCADIA Method and the Capella Tool. In Proceedings of the 8th European Congress on Embedded Real Time Software and Systems (ERTS 2016), Toulouse, France, 27–29 January 2016. [Google Scholar]
  9. Weilkiens, T.; Scheithauer, A.; Di Maio, M.; Klusmann, N. Evaluating and comparing MBSE methodologies for practitioners. In Proceedings of the 2016 IEEE International Symposium on Systems Engineering (ISSE), Edinburgh, UK, 3–5 October 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 1–8. [Google Scholar] [CrossRef]
  10. Lykins, H.; Friedenthal, S.; Meilich, A. 4.4. 4 Adapting Uml for an object-oriented systems engineering method (Oosem). In Proceedings of the INCOSE International Symposium, Minneapolis, MN, USA, 16–20 July 2000; Volume 10, pp. 490–497. [Google Scholar]
  11. De Saqui-Sannes, P.; Vingerhoeds, R.A.; Garion, C.; Thirioux, X. A taxonomy of MBSE approaches by languages, tools and methods. IEEE Access 2022, 10, 120936–120950. [Google Scholar] [CrossRef]
  12. Santos, T.L.; Soares, M.S. A survey on what users think about SysML. Syst. Eng. 2023, 26, 379–392. [Google Scholar] [CrossRef]
  13. Friedenthal, S.; Moore, A.; Steiner, R. A Practical Guide to SysML: The Systems Modeling Language; Morgan Kaufmann: Burlington, MA, USA, 2014. [Google Scholar]
  14. Friedenthal, S.; Wolfrom, J.A. Modeling with SysML. In Proceedings of the INCOSE International Symposium, Chicago, IL, USA, 12–15 July 2010; Volume 20, pp. 1847–1995. [Google Scholar]
  15. Zimmerman, P.; Gilbert, T.; Salvatore, F. Digital engineering transformation across the Department of Defense. J. Def. Model. Simul. 2019, 16, 325–338. [Google Scholar] [CrossRef]
  16. Hill, T.R.; Nicoli, P.; Pierce, G.J.; Woodham, K.; Gati, F. Digital Transformation of the NASA Engineering Domain. In Proceedings of the 2024 IEEE Aerospace Conference, Big Sky, MT, USA, 2–9 March 2024; IEEE: Piscataway, NJ, USA, 2024; pp. 1–15. [Google Scholar]
  17. VanDerHorn, E.; Mahadevan, S. Digital Twin: Generalization, characterization and implementation. Decis. Support Syst. 2021, 145, 113524. [Google Scholar] [CrossRef]
  18. Tao, F.; Zhang, H.; Liu, A.; Nee, A.Y.C. Digital twin in industry: State-of-the-art. IEEE Trans. Ind. Inform. 2018, 15, 2405–2415. [Google Scholar] [CrossRef]
  19. Liu, Y.K.; Ong, S.K.; Nee, A.Y.C. State-of-the-art survey on digital twin implementations. Adv. Manuf. 2022, 10, 1–23. [Google Scholar] [CrossRef]
  20. Tu, X.; Autiosalo, J.; Jadid, A.; Tammi, K.; Klinker, G. A mixed reality interface for a digital twin based crane. Appl. Sci. 2021, 11, 9480. [Google Scholar] [CrossRef]
  21. Kaarlela, T.; Pieska, S.; Pitkaaho, T. Digital twin and virtual reality for safety training. In Proceedings of the 2020 11th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), Mariehamn, Finland, 23–25 September 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 000115–000120. [Google Scholar] [CrossRef]
  22. Haluck, R.S.; Krummel, T.M. Computers and virtual reality for surgical education in the 21st century. Arch. Surg. 2000, 135, 786–792. [Google Scholar] [CrossRef] [PubMed]
  23. Rivas, Y.C.; Valdivieso, P.A.V.; Rodriguez, M.A.Y. Virtual reality and 21st century education. Int. Res. J. Manag. IT Soc. Sci. 2020, 7, 37–44. [Google Scholar] [CrossRef]
  24. Ma, M.; Jain, L.C.; Anderson, P. Future trends of virtual, augmented reality, and games for health. In Virtual, Augmented Reality and Serious Games for Healthcare 1; Springer: Berlin/Heidelberg, Germany, 2014; pp. 1–6. [Google Scholar] [CrossRef]
  25. Papanastasiou, G.; Drigas, A.; Skianis, C.; Lytras, M.; Papanastasiou, E. Virtual and augmented reality effects on K-12, higher and tertiary education students’ twenty-first century skills. Virtual Real. 2019, 23, 425–436. [Google Scholar] [CrossRef]
  26. Steuer, J.; Biocca, F.; Levy, M.R. Defining virtual reality: Dimensions determining telepresence. Commun. Age Virtual Real. 1995, 33, 1. [Google Scholar] [CrossRef]
  27. Biocca, F. Virtual reality technology: A tutorial. J. Commun. 1992, 42, 23–72. [Google Scholar] [CrossRef]
  28. Bamodu, O.; Ye, X.M. Virtual reality and virtual reality system components. Adv. Mater. Res. 2013, 765, 1169–1172. [Google Scholar] [CrossRef]
  29. Ren, Z.; Fu, X.; Dong, K.; Lai, Y.; Zhang, J. Advanced study of optical imaging systems for virtual reality head-mounted displays. Photonics 2023, 10, 555. [Google Scholar] [CrossRef]
  30. Salah, B.; Abidi, M.H.; Mian, S.H.; Krid, M.; Alkhalefah, H.; Abdo, A. Virtual Reality-Based Engineering Education to Enhance Manufacturing Sustainability in Industry 4.0. Sustainability 2019, 11, 1477. [Google Scholar] [CrossRef]
  31. Dinis, F.M.; Guimaraes, A.S.; Carvalho, B.R.; Martins, J.P.P. Virtual and augmented reality game-based applications to civil engineering education. In Proceedings of the 2017 IEEE Global Engineering Education Conference (EDUCON), Athens, Greece, 25–28 April 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 1683–1688. [Google Scholar] [CrossRef]
  32. Hilfert, T.; König, M. Low-cost virtual reality environment for engineering and construction. Vis. Eng. 2016, 4, 2. [Google Scholar] [CrossRef]
  33. Stone, R.J.; Panfilov, P.B.; Shukshunov, V.E. Evolution of aerospace simulation: From immersive Virtual Reality to serious games. In Proceedings of the 5th International Conference on Recent Advances in Space Technologies—RAST2011, Istanbul, Turkey, 9–11 June 2011; pp. 655–662. [Google Scholar] [CrossRef]
  34. Brusa, E. Digital twin: Toward the integration between system design and rams assessment through the model-based systems engineering. IEEE Syst. J. 2020, 15, 3549–3560. [Google Scholar] [CrossRef]
  35. Madni, A.M.; Madni, C.C.; Lucero, S.D. Leveraging digital twin technology in model-based systems engineering. Systems 2019, 7, 7. [Google Scholar] [CrossRef]
  36. Ferko, E.; Berardinelli, L.; Bucaioni, A.; Behnam, M.; Wimmer, M. Towards interoperable digital twins: Integrating SysML into AAS with higher-order transformations. In Proceedings of the 2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C), Hyderabad, India, 4–8 June 2024; IEEE: Piscataway, NJ, USA, 2024; pp. 342–349. [Google Scholar]
  37. Pierce, G.J.; Heeren, J.D.; Hill, T.R. Orion SysML model, digital twin, and lessons learned for artemis I. In Proceedings of the INCOSE International Symposium, Honolulu, HI, USA, 15–20 July 2023; Volume 33, pp. 290–304. [Google Scholar]
  38. Madni, A.M.; Nance, M.; Richey, M.; Hubbard, W.; Hanneman, L. Toward an experiential design language: Augmenting model-based systems engineering with technical storytelling in virtual worlds. Procedia Comput. Sci. 2014, 28, 848–856. [Google Scholar] [CrossRef]
  39. Hause, M. The digital twin throughout the se lifecycle. In Proceedings of the INCOSE International Symposium, Orlando, FL, USA, 20–25 July 2019; Volume 29, pp. 203–217. [Google Scholar] [CrossRef]
  40. Koca, B.A.; Çubukçu, B.; Yüzgeç, U. Augmented Reality Application for Preschool Children with Unity 3D Platform. In Proceedings of the 2019 3rd International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), Ankara, Turkey, 11–13 October 2019; pp. 1–4. [Google Scholar] [CrossRef]
  41. Riofrio-Morales, M.; Garcia, M.V. Training virtual reality-based system for detection and simulation of motors failures. In Journal of Physics: Conference Series; IOP Publishing: Bristol, UK, 2021; Volume 1983, p. 012099. [Google Scholar] [CrossRef]
  42. Wolfartsberger, J. Analyzing the potential of Virtual Reality for engineering design review. Autom. Constr. 2019, 104, 27–37. [Google Scholar] [CrossRef]
  43. Tea, S.; Panuwatwanich, K.; Ruthankoon, R.; Kaewmoracharoen, M. Multiuser immersive virtual reality application for real-time remote collaboration to enhance design review process in the social distancing era. J. Eng. Des. Technol. 2022, 20, 281–298. [Google Scholar] [CrossRef]
  44. Webb, M.; Tracey, M.; Harwin, W.; Tokatli, O.; Hwang, F.; Johnson, R.; Barrett, N.; Jones, C. Haptic-enabled collaborative learning in virtual reality for schools. Educ. Inf. Technol. 2022, 27, 937–960. [Google Scholar] [CrossRef]
  45. Petersen, G.B.; Stenberdt, V.; Mayer, R.E.; Makransky, G. Collaborative generative learning activities in immersive virtual reality increase learning. Comput. Educ. 2023, 207, 104931. [Google Scholar] [CrossRef]
  46. Brown, C.E.; Alrmuny, D.; Williams, M.K.; Whaley, B.; Hyslop, R.M. Visualizing molecular structures and shapes: A comparison of virtual reality, computer simulation, and traditional modeling. Chem. Teach. Int. 2021, 3, 69–80. [Google Scholar] [CrossRef]
  47. Ferrell, J.B.; Campbell, J.P.; McCarthy, D.R.; McKay, K.T.; Hensinger, M.; Srinivasan, R.; Zhao, X.; Wurthmann, A.; Li, J.; Schneebeli, S.T. Chemical exploration with virtual reality in organic teaching laboratories. J. Chem. Educ. 2019, 96, 1961–1966. [Google Scholar] [CrossRef]
  48. Shao, X.; Yuan, Q.; Qian, D.; Ye, Z.; Chen, G.; Le Zhuang, K.; Jiang, X.; Jin, Y.; Qiang, D. Virtual reality technology for teaching neurosurgery of skull base tumor. BMC Med. Educ. 2020, 20, 3. [Google Scholar]
  49. Venkatesan, M.; Mohan, H.; Ryan, J.R.; Schürch, C.M.; Nolan, G.P.; Frakes, D.H.; Coskun, A.F. Virtual and augmented reality for biomedical applications. Cell Rep. Med. 2021, 2, 100348. [Google Scholar] [CrossRef] [PubMed]
  50. Leyer, M.; Brown, R.; Aysolmaz, B.; Vanderfeesten, I.; Turetken, O. 3D Virtual World BPM Training Systems: Process Gateway Experimental Results. In Advanced Information Systems Engineering, Proceedings of the 31st International Conference, CAiSE 2019, Rome, Italy, 3–7 June 2019; Lecture Notes in Computer Science; Giorgini, P., Weber, B., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 415–429. [Google Scholar] [CrossRef]
  51. Pöhler, L.; Schuir, J.; Lübbers, S.; Teuteberg, F. Enabling Collaborative Business Process Elicitation in Virtual Environments. In Business Modeling and Software Design; Lecture Notes in Business Information Processing; Shishkov, B., Ed.; Springer International Publishing: Cham, Switzerland, 2020; pp. 375–385. [Google Scholar] [CrossRef]
  52. Oberhauser, R.; Pogolski, C.; Matic, A. VR-BPMN: Visualizing BPMN models in virtual reality. In Business Modeling and Software Design, Proceedings of the 8th International Symposium, BMSD 2018, Vienna, Austria, 2-4 July 2018; Proceedings 8; Springer International Publishing: Cham, Switzerland, 2018; pp. 83–97. [Google Scholar] [CrossRef]
  53. Reuter, R.; Hauser, F.; Muckelbauer, D.; Stark, T.; Antoni, E.; Mottok, J.; Wolff, C. Using Augmented Reality in Software Engineering Education? First insights to a comparative study of 2D and AR UML modeling. In Proceedings of the 52nd Hawaii International Conference on System Sciences, Grand Wailea, HI, USA, 8–11 January 2019. [Google Scholar]
  54. Oberhauser, R. VR-UML: The unified modeling language in virtual reality—An immersive modeling experience. In Proceedings of the International Symposium on Business Modeling and Software Design, Sofia, Bulgaria, 5–7 July 2021; Springer International Publishing: Cham, Switzerland, 2021; pp. 40–58. [Google Scholar] [CrossRef]
  55. Henderson, K.; Salado, A. Value and benefits of model-based systems engineering (MBSE): Evidence from the literature. Syst. Eng. 2021, 24, 51–66. [Google Scholar] [CrossRef]
  56. Henderson, K.; McDermott, T.; Van Aken, E.; Salado, A. Towards Developing Metrics to Evaluate Digital Engineering. Syst. Eng. 2023, 26, 3–31. [Google Scholar] [CrossRef]
  57. Lutfi, M.; Valerdi, R. Virtual reality in model based systems engineering: A review paper. In Proceedings of the International Conference on Human-Computer Interaction, 2020; Springer International Publishing: Cham, Switzerland, 2020; pp. 197–205. [Google Scholar] [CrossRef]
  58. Madni, A.M.; Purohit, S.; Madni, C.C. Exploiting Digital Twins in MBSE to Enhance System Modeling and Life Cycle Coverage. In Handbook of Model-Based Systems Engineering; Springer International Publishing: Cham, Switzerland, 2023; pp. 527–548. [Google Scholar] [CrossRef]
  59. Oberhauser, R. VR-SysML: SysML model visualization and immersion in virtual reality. In Proceedings of the International Conference of Modern Systems Engineering Solutions, Nice, France, 24–28 July 2022; pp. 61–66. [Google Scholar]
  60. Romero, V.; Pinquié, R.; Noël, F. An immersive virtual environment for reviewing model-centric designs. Proc. Des. Soc. 2021, 1, 447–456. [Google Scholar] [CrossRef]
  61. Rodrigues, C.S.C.; Werner, C.M.L.; Landau, L. VisAr3D: An innovative 3D visualization of UML models. In Proceedings of the 38th International Conference on Software Engineering Companion, Austin, TX, USA, 14–22 May 2016; pp. 451–460. [Google Scholar] [CrossRef]
  62. Yigitbas, E.; Schmidt, M.; Bucchiarone, A.; Bassanelli, S.; Engels, G. Gamification-and Virtual Reality-Based Learning Environment for UML Class Diagram Modeling. In Proceedings of the 2024 IEEE Global Engineering Education Conference (EDUCON), Kos Island, Greece, 8–11 May 2024; IEEE: Piscataway, NJ, USA, 2024; pp. 1–10. [Google Scholar] [CrossRef]
  63. Casse, O. SysML in Action with Cameo Systems Modeler; Elsevier: Amsterdam, The Netherlands, 2017. [Google Scholar]
  64. Isar, C. A glance into virtual reality development using unity. Inform. Econ. 2018, 22, 14–22. [Google Scholar] [CrossRef]
  65. dos Santos Soares, M.; Vrancken, J.L.M. Model-Driven User Requirements Specification using SysML. J. Softw. 2008, 3, 57–68. [Google Scholar] [CrossRef]
  66. Hause, M. The SysML modelling language. In Proceedings of the Fifteenth European Systems Engineering Conference, Edinburgh, UK, 18–20 September 2006; Volume 9, pp. 1–12. [Google Scholar]
  67. Jarraya, Y.; Debbabi, M.; Bentahar, J. On the meaning of SysML activity diagrams. In Proceedings of the 2009 16th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems, San Francisco, CA, USA, 14–16 April 2009; IEEE: Piscataway, NJ, USA, 2009; pp. 95–105. [Google Scholar] [CrossRef]
  68. Johnson, T.; Kerzhner, A.; Paredis, C.J.J.; Burkhart, R. Integrating models and simulations of continuous dynamics into SysML. J. Comput. Inf. Sci. Eng. 2012, 12, 011002. [Google Scholar] [CrossRef]
  69. Raymer, E.; MacDermott, Á.; Akinbi, A. Virtual reality forensics: Forensic analysis of Meta Quest 2. Forensic Sci. Int. Digit. Investig. 2023, 47, 301658. [Google Scholar] [CrossRef]
  70. Riaz, F.; Ali, K.M. Applications of graph theory in computer science. In Proceedings of the 2011 third international conference on Computational Intelligence, Communication Systems and Networks, Bali, Indonesia, 26–28 July 2011; IEEE: Piscataway, NJ, USA, 2011; pp. 142–145. [Google Scholar] [CrossRef]
  71. Moody, D. The “physics” of notations: Toward a scientific basis for constructing visual notations in software engineering. IEEE Trans. Softw. Eng. 2009, 35, 756–779. [Google Scholar] [CrossRef]
  72. Lutfi, M.; Valerdi, R. Integration of SysML and virtual reality environment: A ground based telescope system example. Systems 2023, 11, 189. [Google Scholar] [CrossRef]
  73. Canto. Available online: https://rubin.canto.com/v/ConstructionGallery/album/HDSNU?viewIndex=2&display=curatedView&from=curatedView&referenceTo=&filter=%7B%22fileTypes%22:%22ZIP%22%7D&column=other&id=i342ihfh8170r4s0ig1j79th10 (accessed on 24 December 2024).
  74. Autodesk Viewer | Free Online File Viewer. Available online: https://viewer.autodesk.com/ (accessed on 28 December 2024).
  75. Review 2D & 3D Designs with eDrawings. Available online: https://www.edrawingsviewer.com/review-2d-3d-designs-edrawings (accessed on 28 December 2024).
  76. Kennedy, R.S.; Lane, N.E.; Berbaum, K.S.; Lilienthal, M.G. Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 1993, 3, 203–220. [Google Scholar] [CrossRef]
  77. Brooke, J. SUS-A quick and dirty usability scale. Usability Eval. Ind. 1996, 189, 4–7. [Google Scholar]
  78. Kim, T.K. T test as a parametric statistic. Korean J. Anesthesiol. 2015, 68, 540–546. [Google Scholar] [CrossRef]
  79. Shapiro, S.S.; Wilk, M.B. An analysis of variance test for normality (complete samples). Biometrika 1965, 52, 591–611. [Google Scholar] [CrossRef]
  80. Bevan, N.; Carter, J.; Earthy, J.; Geis, T.; Harker, S. New ISO standards for usability, usability reports and usability measures. In Human-Computer Interaction. Theory, Design, Development and Practice, Proceedings of the 18th International Conference, HCI International 2016, Toronto, ON, Canada, 17–22 July 2016; Proceedings, Part I 18; Springer International Publishing: Cham, Switzerland, 2016; pp. 268–278. [Google Scholar] [CrossRef]
  81. Villagrasa, S.; Fonseca, D.; Durán, J. Teaching case: Applying gamification techniques and virtual reality for learning building engineering 3D arts. In Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality, Salamanca, Spain, 1–3 October 2014; pp. 171–177. [Google Scholar] [CrossRef]
  82. Hennig-Thurau, T.; Herting, A.M.; Jütte, D. Adoption of Virtual Reality Headsets: The Role of Metaverse Trials for Consumers’ Usage and Purchase Intentions. J. Interact. Mark. 2024. [Google Scholar] [CrossRef]
  83. Wu, W.; Sivaparthipan, C.B.; Sanz-Prieto, I. Application of automobile modelling optimization design based on virtual reality technology. J. Interconnect. Netw. 2022, 22 (Suppl. S2), 2143007. [Google Scholar] [CrossRef]
  84. Hassanien, A.E.; Gupta, D.; Khanna, A.; Slowik, A. (Eds.) Virtual and Augmented Reality for Automobile Industry: Innovation Vision and Applications; Springer: Berlin/Heidelberg, Germany, 2022. [Google Scholar]
  85. Taherysayah, F.; Malathouni, C.; Liang, H.-N.; Westermann, C. Virtual reality and electroencephalography in architectural design: A systematic review of empirical studies. J. Build. Eng. 2024, 85, 108611. [Google Scholar] [CrossRef]
  86. Yu, R.; Gu, N.; Lee, G.; Khan, A. A systematic review of architectural design collaboration in immersive virtual environments. Designs 2022, 6, 93. [Google Scholar] [CrossRef]
  87. Barosan, I.; Van Der Heijden, J. Integration of SysML models in a 3D environment for Virtual Testing and Validation. In Proceedings of the Federated Africa and Middle East Conference on Software Engineering, Cairo-Kampala, Egypt, 7–8 June 2022; pp. 39–45. [Google Scholar] [CrossRef]
  88. Bajaj, M.; Friedenthal, S.; Seidewitz, E. Systems modeling language (SysML v2) support for digital engineering. Insight 2022, 25, 19–24. [Google Scholar] [CrossRef]
Systems 13 00221 g001
Figure 1. SysML diagram types [14].
Figure 1. SysML diagram types [14].
Systems 13 00221 g001
Systems 13 00221 g002
Figure 2. Package diagram of the ground-based telescope system model.
Figure 2. Package diagram of the ground-based telescope system model.
Systems 13 00221 g002
Systems 13 00221 g003
Figure 3. The requirement diagram shows telescope data collection system requirements.
Figure 3. The requirement diagram shows telescope data collection system requirements.
Systems 13 00221 g003
Systems 13 00221 g004
Figure 4. BDD shows the telescope mount system and lower-level subsystems (L0).
Figure 4. BDD shows the telescope mount system and lower-level subsystems (L0).
Systems 13 00221 g004
Systems 13 00221 g005
Figure 5. BDD shows the telescope control system and lower-level subsystems (L1).
Figure 5. BDD shows the telescope control system and lower-level subsystems (L1).
Systems 13 00221 g005
Systems 13 00221 g006
Figure 6. BDD shows the earthquake detection system and lower-level components (L2).
Figure 6. BDD shows the earthquake detection system and lower-level components (L2).
Systems 13 00221 g006
Systems 13 00221 g007
Figure 7. BDD shows the fluid supply system and lower-level components (L2).
Figure 7. BDD shows the fluid supply system and lower-level components (L2).
Systems 13 00221 g007
Systems 13 00221 g008
Figure 8. BDD shows the pumping system and lower-level components (L3).
Figure 8. BDD shows the pumping system and lower-level components (L3).
Systems 13 00221 g008
Systems 13 00221 g009
Figure 9. BDD shows the Azimuth structure and lower-level components (L2). * means one-to-many multiplicity.
Figure 9. BDD shows the Azimuth structure and lower-level components (L2). * means one-to-many multiplicity.
Systems 13 00221 g009
Systems 13 00221 g010
Figure 10. BDD shows the hexapod system and lower-level components (L3).
Figure 10. BDD shows the hexapod system and lower-level components (L3).
Systems 13 00221 g010
Systems 13 00221 g011
Figure 11. BDD showing actuator test rig and lower-level components (L4).
Figure 11. BDD showing actuator test rig and lower-level components (L4).
Systems 13 00221 g011
Systems 13 00221 g012
Figure 12. Example of the traceability of a requirement to a “Value Property”.
Figure 12. Example of the traceability of a requirement to a “Value Property”.
Systems 13 00221 g012
Systems 13 00221 g013
Figure 13. IBD visualizing telescope control system data flows.
Figure 13. IBD visualizing telescope control system data flows.
Systems 13 00221 g013
Systems 13 00221 g014
Figure 14. Use case diagram showing data analysis OpsCon.
Figure 14. Use case diagram showing data analysis OpsCon.
Systems 13 00221 g014
Systems 13 00221 g015
Figure 15. Activity diagram of the “Disable Motion” operational scenario.
Figure 15. Activity diagram of the “Disable Motion” operational scenario.
Systems 13 00221 g015
Systems 13 00221 g016
Figure 16. State machine diagram depicting operational states of the telescope mount system.
Figure 16. State machine diagram depicting operational states of the telescope mount system.
Systems 13 00221 g016
Systems 13 00221 g017
Figure 17. Parametric diagram showing mirror fried parameter allocation calculation (Note, M depicts “Mirror”).
Figure 17. Parametric diagram showing mirror fried parameter allocation calculation (Note, M depicts “Mirror”).
Systems 13 00221 g017
Systems 13 00221 g018
Figure 18. Wireless VR headset (Meta Quest 2).
Figure 18. Wireless VR headset (Meta Quest 2).
Systems 13 00221 g018
Systems 13 00221 g019
Figure 19. User Instructions for selecting a model element within a SysML diagram.
Figure 19. User Instructions for selecting a model element within a SysML diagram.
Systems 13 00221 g019
Systems 13 00221 g020
Figure 20. SysML package diagrams depicted through interactive ‘Buttons’.
Figure 20. SysML package diagrams depicted through interactive ‘Buttons’.
Systems 13 00221 g020
Systems 13 00221 g021
Figure 21. Before the selection of the “Earthquake Detection System” block.
Figure 21. Before the selection of the “Earthquake Detection System” block.
Systems 13 00221 g021
Systems 13 00221 g022
Figure 22. After selecting the “Earthquake Detection System” block, a lower-level diagram becomes visible in the scene.
Figure 22. After selecting the “Earthquake Detection System” block, a lower-level diagram becomes visible in the scene.
Systems 13 00221 g022
Systems 13 00221 g023
Figure 23. Actuator test rig components are shown as BDD (top) and virtual twin (bottom).
Figure 23. Actuator test rig components are shown as BDD (top) and virtual twin (bottom).
Systems 13 00221 g023
Systems 13 00221 g024
Figure 24. The harmonic drive selected in the BDD (top, green) and the corresponding virtual twin-component shown automatically (bottom, green, and enlarged section).
Figure 24. The harmonic drive selected in the BDD (top, green) and the corresponding virtual twin-component shown automatically (bottom, green, and enlarged section).
Systems 13 00221 g024
Systems 13 00221 g025
Figure 25. Telescope Structures-BDD and 3D Views (From left, facility, dome, and telescope, (RubinObs/NSF/AURA)) [73].
Figure 25. Telescope Structures-BDD and 3D Views (From left, facility, dome, and telescope, (RubinObs/NSF/AURA)) [73].
Systems 13 00221 g025
Systems 13 00221 g026
Figure 26. Teleportation to the inside of the dome (RubinObs/NSF/AURA) [73].
Figure 26. Teleportation to the inside of the dome (RubinObs/NSF/AURA) [73].
Systems 13 00221 g026
Systems 13 00221 g027
Figure 27. FaultState showing the error for defective parts and available actions.
Figure 27. FaultState showing the error for defective parts and available actions.
Systems 13 00221 g027
Systems 13 00221 g028
Figure 28. The state changed to MaintenanceState (green).
Figure 28. The state changed to MaintenanceState (green).
Systems 13 00221 g028
Systems 13 00221 g029
Figure 29. Pumping system BDD in 3D SysML VR model (related elements are hidden).
Figure 29. Pumping system BDD in 3D SysML VR model (related elements are hidden).
Systems 13 00221 g029
Systems 13 00221 g030
Figure 30. The traced pumping system requirement is shown in the model after the user clicks the value property for additional information.
Figure 30. The traced pumping system requirement is shown in the model after the user clicks the value property for additional information.
Systems 13 00221 g030
Systems 13 00221 g031
Figure 31. QQ plot of the knowledge test data showing normality.
Figure 31. QQ plot of the knowledge test data showing normality.
Systems 13 00221 g031
Systems 13 00221 g032
Figure 32. Box Plot showing t-test results comparing SysML 2D and SysML VR groups.
Figure 32. Box Plot showing t-test results comparing SysML 2D and SysML VR groups.
Systems 13 00221 g032
Systems 13 00221 g033
Figure 33. Plot showing % of participants who agreed that VR 3D SysML improves the benefits.
Figure 33. Plot showing % of participants who agreed that VR 3D SysML improves the benefits.
Systems 13 00221 g033
Systems 13 00221 g034
Figure 34. Plot showing % of participants who agreed that 3D VR SysML improves the benefits (System Understanding (left), Communication or Information Sharing (middle), and Modeling and Training Experience (right). Note, DT integration experience participants were only twelve, mostly systems engineering personnel.).
Figure 34. Plot showing % of participants who agreed that 3D VR SysML improves the benefits (System Understanding (left), Communication or Information Sharing (middle), and Modeling and Training Experience (right). Note, DT integration experience participants were only twelve, mostly systems engineering personnel.).
Systems 13 00221 g034
Systems 13 00221 g035
Figure 35. Summarized survey results (system understanding (top left), communication or information sharing (top right), modeling and training experience (bottom left), DT integration experience (bottom right)).
Figure 35. Summarized survey results (system understanding (top left), communication or information sharing (top right), modeling and training experience (bottom left), DT integration experience (bottom right)).
Systems 13 00221 g035
Table 1. Benefit types for the proposed VR-SysML evaluation.
Table 1. Benefit types for the proposed VR-SysML evaluation.
Benefit TypesExplanation
System UnderstandingLevel of understanding between stakeholders,
Understanding of system requirements, design and behavior, Insight of the problem.
Communication or Information SharingCommunication with stakeholders/designers’ developers/various engineering disciplines, collaboration within the development team, Information capture, and knowledge sharing.
Modeling and Training ExperienceInteraction experience with the modeling environment, and level of motivation for training.
Digital Twin Integration ExperienceIf SysML diagrams accompany the virtual twins, how well it improves the DT experience for the user.
Table 2. Simulation sickness questionnaire scoring rubric [76].
Table 2. Simulation sickness questionnaire scoring rubric [76].
Weight
SSQ symptomNauseaOculomotorDisorientation
General discomfortoo
Fatigue o
Headache o
Eyestrain o
Difficulty focusing oo
Increased salivationo
Sweatingo
Nauseao o
Difficulty concentratingoo
Fullness of head o
Blurred vision oo
Dizzy (eyes open) o
Dizzy (eyes closed) o
Vertigo o
Stomach awarenesso
Burpingo
Total[1][2][3]
Scoring[1] × 9.54[2] × 7.58[3] × 13.92
Total score ([1] + [2] + [3]) × 3.74
Table 3. System usability scale rating statements [77].
Table 3. System usability scale rating statements [77].
#Statements
1I think that I would like to use these VR scenarios.
2I found these VR scenarios to be unnecessarily complex.
3I thought these VR scenarios were easy to use.
4I think that I would need the help of a support person to use the VR headset.
5I found the various functions in these scenarios were well integrated.
6I thought there was too much inconsistency in these VR scenarios.
7I would imagine that most people would learn to use these VR scenarios very quickly.
8I found these VR scenarios very cumbersome to use.
9I felt very confident using these VR scenarios.
10I needed to learn a lot of things before I could get going with these scenarios.
Table 4. Drawbacks of SysML 3D VR visualization.
Table 4. Drawbacks of SysML 3D VR visualization.
#DrawbacksFrequency
1Lack of Technology Adoption18
2Cumbersome Instruments14
3Long term Usability12
4Concerns about Modeling Redundancy9
5Reliability of the VR Scenarios8
6Lack of Keyboard Support 6
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Lutfi, M.; Valerdi, R. Evaluating the Benefits and Drawbacks of Visualizing Systems Modeling Language (SysML) Diagrams in the 3D Virtual Reality Environment. Systems 2025, 13, 221. https://doi.org/10.3390/systems13040221

AMA Style

Lutfi M, Valerdi R. Evaluating the Benefits and Drawbacks of Visualizing Systems Modeling Language (SysML) Diagrams in the 3D Virtual Reality Environment. Systems. 2025; 13(4):221. https://doi.org/10.3390/systems13040221

Chicago/Turabian Style

Lutfi, Mostafa, and Ricardo Valerdi. 2025. "Evaluating the Benefits and Drawbacks of Visualizing Systems Modeling Language (SysML) Diagrams in the 3D Virtual Reality Environment" Systems 13, no. 4: 221. https://doi.org/10.3390/systems13040221

APA Style

Lutfi, M., & Valerdi, R. (2025). Evaluating the Benefits and Drawbacks of Visualizing Systems Modeling Language (SysML) Diagrams in the 3D Virtual Reality Environment. Systems, 13(4), 221. https://doi.org/10.3390/systems13040221

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop