Design and Evaluation of an Asynchronous VR Exploration System for Architectural Design Discussion Content

Abstract

Design discussion is crucial in the architectural design process. To enhance the spatial understanding of 3D space and discussion effectiveness, recently, some systems have been proposed to support design discussion interactively in an immersive virtual environment. The entire design discussion can be archived and potentially become course materials for future learners. In this paper, we propose an asynchronous VR exploration system that aims to help learners explore content effectively and efficiently anywhere and at any time. To improve effectiveness and efficiency, we also propose a summarization-to-detail approach with the application space by which students can observe the visualization of spatial summarization of actions and participants’ dwell time or the temporal distribution of dialogues and then locate the important or interesting region or dialogue for further exploration. To further explore the discussion content, students can call the preview to see the time-lapse animation of the object operation to understand the change in models or playback to view the discussion details. We conducted an exploratory user study with 10 participants to evaluate user experience, user impression, and effectiveness of learning the design discussion course content using our asynchronous VR design discussion content exploration system. The results indicate that the interactive VR exploration system presented can help learners study the design discussion content effectively. Participants also provided some positive feedback and confirmed the usefulness and value of the system presented. Our applications and lessons learned have implications for future asynchronous VR exploration systems, not only for architectural design discussion content, but also for other applications, such as industrial visual inspections and educational visualizations of design discussions.

1. Introduction

1.1. Research Background

Recently, affordable, accessible, and powerful hardware has emerged in consumer VR, enabling new developments in applications. In recent years, we have seen widespread applications in gaming, training, social activities, and education. VR is becoming an important medium in design fields such as product design, architecture design, landscape design, and urban planning [1,2,3]. In architectural design, the design process evolves from early conceptual design to later detailed design. For each design stage in the design studio, review or discussion is vital for students to learn by doing [4,5]. Instructors and students meet each other face-to-face; the designer or instructor presents the design work; and the instructor then coordinates the discussion, asks questions, and provides suggestions. Traditionally, the participants have discussions based on the physical model and 2D drawings prepared by the designer and communicate via voice and sketching on paper or whiteboard. However, firstly, the physical model can usually only be viewed from the outside and cannot be modified during the discussion. Second, the 3D spatial understanding from 2D drawings or 2D sketching on paper may require considerable mental effort. Lastly, it is difficult for participants to go back to a place previously visited or to a previous state of discussion.

To overcome these problems and improve the effectiveness of design discussion, some research has proposed using VR as a representation medium for design reviews [5]. More recently, VR has been fully used to offer a design discussion platform in which participants can navigate and view the virtual model, discuss through voice and 2D/3D sketching, and modify the object during the discussion [6]. It has been reported that the 3D spatial understanding of the architecture model and sketching during the discussion requires little mental effort; the support for instant model modification greatly enhances the discussion’s effectiveness, since you can see what is suggested during the discussion [6]. Furthermore, in such systems, since discussion logs are archived, rollback functionality has been proposed to guide participants back to a place previously visited for some discussions in that state or to a previous discussion state to initiate a new direction of discussion [6]. The archived discussion log is outstanding course material because students who were not in the design discussion class will have the opportunity to learn architecture design by asynchronously viewing and interacting with the discussion content in their own time. Furthermore, students who were in the discussion class can also learn more by interactively reviewing the design discussion content in a post-hoc fashion.

1.2. Problems and Proposed Approaches

Since the design discussion course normally takes one or two hours, a challenging problem is how to present the course material in VR such that students can learn the architecture design effectively and efficiently. At the beginning of this research, we conducted a requirement interview with a group of architecture students and teachers who had previously experienced the traditional architecture design discussion. We found that learners are more concerned about what has happened in the course than about the sequence of discussions, and that realizing what has happened to the architecture models makes understanding the content more effective and efficient. Based on this observation, we proposed a learning strategy called “summarization-to-detail”. Learners are first offered an overview of the discussion by showing the spatial distribution of object changes and participants’ dwelling time, as well as the temporal distribution of dialogues; from the summerization visualization, learners can easily filter out a time interval for further exploration by selecting a region or a dialogue of interest.

In our requirement interview, we also observed that learners prefer to see the overview of object modifications before viewing the detailed discussion related to the object modifications. Therefore, the second problem is how to provide an overview of the object modifications occurring within the filtered time interval. Since each object modification could take time, and there might be several modifications occurring within the time interval, time-lapse animation of the objects’ change can be a reasonable choice. Moreover, the object modifications could happen in different locations and the animation is normally shown in a few seconds, so it is not always easy for learners to follow and see the time-lapsed animation of object changes in the 3D Virtual Environment (VE). For this problem, our solution is to present the time-lapse animation in the Worlds in Miniature (WIM) [7] of the VE so that the learner can inspect the animations without changing view and position.

To further meet what we learned from the requirement interview—“the overview of object modification is preferred to be seen before viewing the detailed discussion”—with the “summarization-to-detail” strategy, the learner’s exploration normally starts with a preview and then proceeds with playback, or directly enters into playback, and of course can go back and forth between preview and playback. The preview offers a highlight of objects that were modified within the time interval, followed by a time-lapse animation of the changes of the objects. Highlights and animation are shown in the WIM of the VE, as well as in the 3D VE. Learners can switch their view from WIM to the 3D environment for a closer look via a zoom-in-like operation that teleports the learner to an appropriate position and with the same view direction as that for the WIM. The playback is the normal playback of the discussion content in the virtual environment. Learners can follow the teacher’s position and view, or their own view, and in this manner can experience the discussion in the same way as students who were actually present in the design discussion in real-time.

To evaluate the effectiveness of the proposed system, we conducted an exploratory user study with 10 participants. Eight of them were architecture majors. Participants were asked to explore the discussion content on the design of VR architecture using our system, and we compared and contrasted the effectiveness and learning experience of using the presented system. Eight architecture majors were also asked to draw their own designs based on the design logic they learned from exploring the design discussion under both conditions. The user study revealed that the VR exploration system presented is capable of facilitating a more effective learning experience of the design discussion content. The participants also gave us several examples of positive feedback and confirmed the usefulness and value of our new system.

2. Related Work

2.1. Architectural Design Review and Discussion

The most popular medium for reviewing architectural design is drawing or sketching. During the review or discussion course, participants express their ideas by drawing on paper or sketching on a white board. The ambiguity and amorphousness of sketching is very helpful for architects in improving the ideas process [8]. Sketching also plays an important role in idea exchange and communication between participants [9]. However, 2D sketching requires significant mental effort in 3D spatial understanding and its association with the 3D architecture model.

After computers became a possible choice for the design process, many studies have been concerned with whether or not digital media can provide better representation for discussion, making 3D scene visualization and navigation possible. Recently, virtual reality (VR) has become a popular digital medium. VR can be used to improve the architectural education environment [3], not only due to the fact that it offers a much better sense of space, but it also makes participants feel more realistic with the presence of other users. John Messner [10] has presented ongoing research to improve construction education through the use of VR and 4D CAD modeling (3D design plus time) for construction processes and projects. The students were also very engaged with this type of interactive learning experience. Schnabel et al. [11] compared the understanding of spatial volume within VR with representation using conventional media. They demonstrated that implementing design in VR has significant advantages to the design process by enhancing the perception and understanding of 3D volumes. Bruder et al. created the Arch-Explore user interface, which allows users to explore the 3D models of architectural buildings through natural real walking [12]. They integrated multiple travel techniques to allow the exploration of larger virtual spaces than the real-world tracked space, such as redirected walking for natural walking and virtual portals for large-scale teleportation.

Other researchers have designed and developed an integrated system for architecture design discussion in VR. Hsu et al. [6] constructed a multi-user system that integrated 3D modeling and procedural modeling with a virtual reality platform in order to support interactive architectural design discussion. The system offers a minimap for users to see each other’s position, and they can transfer themselves to wherever they want by selecting the position on the minimap. This system also supports instant editing, 3D sketching, and on-surface sketching. Additionally, a rollback mechanism is designed, where users can roll back to a place previously visited for further discussions and then resume the discussion or go back to a previous state of discussion to initiate a new discussion direction.

2.2. Spatial–Temporal Data Summarization Visualization

Spatial–temporal data includes spatial location, time, and thematic attributes, which are very different and make the data understanding and visualization a difficult task. Cockburn et al. have provided an interesting review of interface techniques that enable users to work with and view different contextual and focused views of data that can be leveraged for a variety of information visualization applications, potentially including architectural design and post-experience visualization [13]. The interface and interaction techniques they explored spanned Overview and Detail, Zooming, and Focus and Contextual techniques. To visualize the paths of the participants, Kraak [14] first revealed the visited trajectories as a space–time cube. Coulton et al. [15] proposed a method to combine spatial and temporal information using the human geographers’ technique of paths to provide 3D visualizations of the participants’ movements. To summarize human actions in spatial and temporal space, Zeng et al. [16] proposed a system that translated activities into a series of line segments from source to terminal city in 3D space, stored in a spatial index that allows quick retrieval of relevant data. Landesberger et al. [17] presented a spatial–temporal bidirectional linkage method, allowing users to select objects in space to see how they develop over time, or to select object groups according to their temporal behavior and locate them in space. Guo et al. [18] introduced EventThread, which categorizes events into time-specific clusters and visualizes the clusters interactively. Shapiro [19] showed how space–time visualization is powerful to support specific types of learning environment designs, such as in museum studies and in social studies education.

Worlds in Miniature (WIM) are miniature virtual replicas of virtual space within the virtual world that have been used as a navigational and interaction tool in a variety of applications that span the virtuality continuum [20,21,22]. Recently, Danyluk et al. examined the WIM design space in XR applications and defined design dimensions that span Size and Scope, Abstraction, Geometry, Reference Frames, Multiples, Virtuality types [23]. They mapped new application examples of WIM that are the best exemplars of these categories, which inspired our WIM design for asynchronous information visualization for architectural design visualization.

2.3. Summarization-to-Detail Exploration and Visualization

It is quite common for human beings to learn or explore information from overview to detail. For example, in information visualization, it is common to use a summarization-to-detail approach. The very first visual information seek mantra, “overview first, zoom, and filter, then details on demand”, proposed by Shneiderman [24], emphasizes the interaction between the overview and the details. Overview provides a summarized view and allows the user to find regions where further exploration in more detail may lead to productive finding. Zhang et al. [25] proposed methods that automatically categorize numerical attribute values by exploiting the hidden domain knowledge, and then proposed an interesting measure for graph summaries to point users to the potentially most insightful summaries. Nam et al. [26] proposed a framework for the exploration of high-dimensional data that provides both overview and detail and adopted a sightseeing paradigm for data and space navigation that allows a better understanding of high-dimensional relationships. Chen et al. [27] visualized the traffic data of highways and allowed semantic zooming, which continuously changes the appearance of the graph to allow visualization of the overview to detail. Sarikaya et al. [28] identified key factors in the design of summary visualization. The analysis of design factors provided a more principled understanding of design practices for summary visualization. Finally, a recent review and analysis of Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis of the use of immersive technologies in Architectural Engineering and Construction (AEC) has revealed that, while there is plenty of use of XR technologies to facilitate design, there is little work on asynchronous design discussion visualization and interaction for users of these applications (i.e., educators, students, observers, and clients of the architectural design decision process) [29].

3. System Design and Description

3.1. Overview

As we mentioned in Section 1.2, the architecture design discussion may begin with an overall presentation about the design solution, and each segment of the following discussion usually focuses on a particular area of the model. Additionally, the requirement interview with a focus group of architecture students, teachers, and designers we conducted suggests that learners are more concerned with what has happened in the course rather than in the sequence of events. Therefore, realizing what has happened to the architecture models makes understanding the content more effective and efficient. Furthermore, the requirements interview also pointed out that it is preferred to see the overview of object modifications before listening to the relevant discussion.

Based on the interview observations, we proposed a learning strategy, called “summarization-to-detail”, in which the learner first sees the overview of actions that occurred during the discussion and then filters out a time interval of interest for further exploration; see Figure 1. The summary of actions consists of the spatial summarization of the modifications of the object and the dwell time of the participants, as well as the temporal distribution of the dialogues, and with an optional Theme River. Spatial summarization is visualized as heat maps on a 2D minimap of the architecture model, and the temporal distribution of dialogue is simply visualized as blocks along the timeline. A time interval will automatically be filtered when the learner selects a region on the minimap or a dialogue block. As shown in Figure 1, with the filtered time interval, the learner can explore the content using preview then playback or directly by playback. The learner can go back and forth between preview and playback or between summarization and detail. In the preview, objects that were modified within the time interval will be highlighted, followed by a time-lapse animation of the object modifications. Playback is the replay of the actions that occurred within the time interval. The learner can experience the playback with his or her own view, or with a view that is closest to the teacher’s.

The summarization visualization on the minimap, dialogues, timeline, Theme River, and buttons for preview and playback are organized on the main menu; see Figure 2. The menu is displayed on a translucent 2D plane as a heads-up display initialized to hover in a near-field space at a reachable distance to the participant. The minimap with heat maps is on the top left. The heat maps can render the density of actions that took place on objects in the environment, or the proportion of time participants’ spent in the architectural space during the design discussion. In addition to the heat maps, on the minimap we also display the modified objects as red dots and represent participants and their positions with icons. The blocks with numbers inside to the right of the minimap are floor numbers for the architecture model. Below the minimap there is a tree structure representing branches resulting from the discussion. Each branch represents a version of the model resulting from the discussion. Note that, by default, all branches are selected, and the learner can select one of the branches. The summarization visualization depends on the branches selected. Below the timeline there are blue blocks that represent dialogues. Two buttons are used to move the activated dialogue to the left or right. The three buttons below the dialogue blocks are for playback, pause, and preview, from left to right.

With a filtered time interval, once the preview button is activated, the main menu disappears, and a WIM is shown to the right of the learner, surrounded by the virtual architecture model. Highlight and time-lapse animations are displayed in WIM, as well as in the 3D virtual environment. The WIM will rotate according to the change in the learner’s view point so that the display of the highlight and animation always faces the learner. The WIM is rendered with a translucent background so that the learner can overlook the whole model and observe the highlight and animation functionality. If the learner wants to have a closer look at a highlight or animation, we provide a zoom-in-like operation that teleports the learner to an appropriate position in the 3D environment, with the same viewing direction as that for the WIM.

3.2. Summarization Visualization

We archived data in VR-based design discussion courses that included the movement of each participant, content of the discussion (e.g., voice, sketching), model modification log, bookmarks, and rollback information. Using this archived information, we interactively visualized the design discussion data in our VR system for learners using a combination of interaction metaphors that will be explained below.

3.2.1. Minimap and Spatial Summarization Visualization

Minimap is a widely used user interface. It provides a top-down view of the architecture model. For a multi-story architecture model, each floor has a minimap. Each minimap is a rendered image of the models on the floor on which a region partition is applied based on the model design and the designer’s opinions. On the minimap, we augmented with heap map for actions regarding objects, including modification, creation, and sketching, or heat map for the participants’ dwelling time, with red dots representing the modified objects and icons representing the participants; see Figure 2. With the minimap and its augmented information, the learner can have a clearer understanding of model space, where the modified objects are, and the density distributions of an object’s modification and participants’ dwelling time.

The heat map can be generated as follows. The map plane is a square in Unity and is first aligned with the minimap. Suppose that the heat map has a resolution of $n\times m$, with $nm$ vertices. The density value at each vertex is the accumulated weighted values contributed by the actions that occurred at the vertex or on the edges or faces adjacent to that vertex.

3.2.2. Timeline and Temporal Summarization Visualization

The targets of the design discussion are the architecture model and the model space. Therefore, ideally, content exploration can best be driven by the region of interest in the model space. However, when a participant comments on the design, the comment could be regarding a general design principle or be about the design of some objects that are not in same region as the participant. We finally decided to drive the content exploration based on the time interval of interest, which is quite similar to the video players that most people are familiar with. Therefore, when a region on the minimap or a dialogue is selected, the corresponding time interval will be filtered, and the content exploration can be triggered for that time interval.

In addition to the dialogue blocks along the timeline, we utilize Theme River to summarize the number of model operations, including transformation, sketching, and object creation, along the timeline, as shown in Figure 3. Theme River is a popular visualization that shows changes in events or variables over time. However, in our initial pilot study, which aimed to assess the system functionality and identify any potential problems, we found that it is not easy for architecture students to grasp the information presented by Theme River. Therefore, in the current system, Theme River functionality is an option. If, from Theme River, the learner finds a certain period of time interesting, he or she then needs to set the time interval manually for further detailed exploration. To this end, Theme River is placed above and aligned with the timeline, as illustrated in Figure 3.

3.2.3. Branches

The VR architectural design discussion system provides rollback functionality that allows the teacher to return to a previous state of discussion and initiate a new direction of discussion [6]. As a result, the log achieved may have more than one version represented as a tree structure, in which a branch is formed by traversing from the root to a leaf; see the bottom left of the main menu in Figure 2. By default, all branches are selected. Once a branch is selected on the main menu, all summarization information will be updated accordingly.

3.3. Filtering of Time Interval

The time interval for a region is derived as the time distribution range for dialogues and operations of objects related to the region. Object operations have been associated with the region by their locations. However, it is not always easy to find which regions are related to a dialogue or which dialogues are related to a region, since the target objects of a discussion dialogue may not be in the same region as the speaker. One way to do this is to interpret the content of the dialogue to identify which are the target objects automatically or manually. Alternatively, for a dialogue, we can find the regions that received the most attention from the participants during the time period of the dialogue. This is based on a phenomenon we observed in the design discussion where the teacher usually makes sure that most participants are focused on the target objects before he or she starts the voice communication. In the current version, we segmented the voice communication for a whole course into dialogues and manually identified the target objects for each dialogue. Note that, due to the branching of the design discussion, the derived time interval for a region may need to be divided into k subintervals if k branches are involved for exploration.

The time interval for a dialogue is the range of time span for the dialogue itself and the operations of the target objects of the dialogue.

3.4. Preview

The preview functionality aims to provide an effective and efficient way for learners to learn where and how objects are operated within the filtered time interval. To this end, two problems need to be carefully dealt with. First, since an object operation may take a while, we depict the modification process with a time-lapse animation that lasts for a few seconds. Note that the time-lapse animation of a model modification was generated by rendering the whole model change over a pre-specified time interval. Second, in the filtered time interval, different object operations are often spread over a large area. Hence, it is difficult for the learner to follow and see the time-lapse animations due to the possibility of motion sickness. Worlds in Miniature (WIM) is chosen to provide an overview of the animations.

For a filtered time interval, once the preview button on the main menu is triggered, the WIM, rendered with transparency, will immediately be displayed in front of the learner’s personal space. Since the animation of the object operations may cover a large area, WIM needs to rotate and scale automatically so that the part that covers the object highlight and the time-lapse animation can always face the learner at the proper size and is displayed with proper scale. Moreover, WIM is also surrounded by the 3D virtual architecture environment; see Figure 4. In the cases needed, we also provide an interface for the learner to translate, rotate, or scale the WIM manually.

Although WIM is advantageous for examining the object changes over the whole model, the learner may want to have a closer look at the highlight and animation. We provide a zoom-in-like functionality that, once triggered, will teleport the learner to an appropriate location in the 3D virtual environment and with the same viewing perspective as that for viewing WIM; see Figure 5. Additionally, viewing in the 3D model space would provide better immersion than WIM does. Switching between these two perspectives (WIM space or virtual model space) allows the learner to achieve a thorough viewing experience in the preview.

3.5. Playback

Playback is a replay of the design discussion. The architecture model, model modifications, sketching, voice dialogues, and movements of the participants will be displayed in the virtual environment. The learner observes and experiences the design discussion in the same way as those who actually joined the discussion; see Figure 6a,b. Note that in Figure 6b we see a snapshot of the discussion scenario in which there is an icon placed on top of the avatar who is speaking, and the sketched objects as well as objects that have been modified are highlighted with different colors.

One problem with playing back is how the learner can move effectively to observe the playback. In the VR architecture design discussion course, participants might move frequently and quickly, and the target objects of the discussion might be a little away from the participants. These observations imply that it is difficult for the learner to move effectively in order to see the playback, even with the help of the minimap. Note that, in playback mode, the learner needs to pay almost full attention to the discussion, so “how the learner moves” should be almost effortless. Since the teacher plays a leading role in the discussion, we provide a “follow teacher” functionality that transfers the learner to a position at the side of the teacher (i.e., rear right side of the teacher) and with a viewing direction similar to the teacher’s.

3.6. Gesture-Based Interaction and User Interface

Current implementation offers two options for the interface: HTC controller or gesture recognition supported by Leap Motion. In the interest of space, the interaction metaphor with gesture-based controls is elaborated below. In the proposed system, two categories of gestures are defined: traveling and system control. For traveling, there are gestures for forward movement, for ascending movement, and for descending movement, as shown in Figure 7. To control the system, there are gestures for pointing and manipulating WIM; see Figure 8a,b. Since these gestures differ significantly, it is unlikely that users will falsely trigger an unexpected functionality. However, it also necessary to train the user in this interaction metaphor for movement and control of the WIM. The “follow teacher” can be triggered on the main menu and on a menu that can be summoned only when the main menu is gone by using the pointing gesture; see Figure 8c. It is worth mentioning that, during the playback, we highlight the objects that have been modified and mark the time of the last modification on such objects to reduce the learner’s mental effort in memorizing what has happened.

The proposed system has three types of menu interface: the main menu described in Section 3.1 (Figure 2), the menu for supporting functionalities, and the menu to interact with WIM. For the main menu, we first considered designing it as a head-mounted menu, but we found that the menu often moved slightly. The accuracy and stability issues of the Leap Motion gesture tracking system further complicated the problem. The main menu is summoned by turning over the left-hand side, will appear statically in front of the user, and will disappear automatically when the preview or playback button is triggered, or when it does not receive the user’s attention for a few seconds. In this way, we made the main menu work like a head-mounted menu.

An auxiliary menu provides supporting functionality: “follow teacher” and “turn on/off the highlight of modified objects in the virtual model scene”. The auxiliary menu can be summoned by a pointing gesture only when the main menu is closed. As shown in Figure 8c, the menu contains two buttons and appears in front of the users’ right index finger and will disappear when the user lays down the right hand or when the main menu is turned on.

The third is the menu to manipulate the WIM. The menu will appear below the WIM when the time-lapse animation is over or paused. In this menu, there are translation, rotation, and scaling options to choose to manipulate WIM using the WIM gesture described in the previous subsection.

3.7. An Example Scenario

In this section, we will describe an example scenario to explore the content of the design discussion using our system. At first, when the system is activated, the learner is placed in the virtual model space from which he or she can navigate and look around to examine the model space. Before or while navigating the virtual model space, the learner can activate the functionality that highlights all the sketched objects and the objects that have been modified during the discussion by using the auxiliary menu (Figure 8c). With this action, the student can become familiar with the model space and conceptualize some ideas about the targets of the design discussion.

To begin exploring the content, the learner summons the main menu by turning over his or her left hand. The main menu will be shown and fixed in front of the user and within the learner’s personal space. The learner looks at the summarization visualization shown on the main menu, trying to grasp the distribution of actions over the model space and dialogue blocks along the timeline. The learner may decide to select a region with the densest object modifications or a dialogue about the course theme for exploration. Once the learner makes the selection, the filtered time interval will be shown above the timeline, and then he or she can activate the preview or playback button. Once the preview button is activated, the main menu disappears and the WIM is shown on the right side of the learner, surrounded by the virtual environment. In the WIM and virtual environment, the objects modified within the time interval will be highlighted, followed by the time-lapsed animation of the object modification. During animation in WIM, the learner can switch to the virtual environment for a closer look by activating the zoom-in button. During or after the preview, the learner then decides to look at the detailed discussion within the filtered time interval, summons the main menu again, and triggers the playback button. After exploring the content for the current time interval, the learner selects another region or dialogue for another exploration. This process repeats until the learner is satisfied with exploring the design discussion in a post-hoc fashion.

4. User Evaluation

We conducted an exploratory user study with 10 participants to assess user experience, user impression, and effectiveness of learning the design discussion course content using our asynchronous VR design discussion content exploration system. The research question we explored was “To what extent is the asynchronous architectural design discussion visualization system effective from a usability and user experience perspective?”

4.1. Participants

The age range of the participants was 20 to 25 years. Half of them were women, the other were men. Eight out of ten participants were architecture majors who had experience with the traditional architectural design discussion course and little VR knowledge. The other two participants were computer science majors who had experience using VR devices, but had not experienced architecture design or design discussion.

4.2. Course Material

The course material used in this study was gathered from a VR architectural design discussion course that took place in March 2019. The course lasted about an hour, and the participants included a teacher and two students. The model for the design discussion is a four-story building with a large platform on the left wing; see Figure 9. During the course, the participants discussed and revised the design of the platform to make it more aesthetic and functional.

The teacher began with a series of introductions on the structure of the building. This process lasted about 15 min. After that, the teacher made some comments on the design of the building and explained some design logic and its goals, followed by some suggestions for reviewing the design. He then drew diagrams using 2D/3D sketching functionality in the architecture design discussion system to explain the modification he suggested to apply to the platform. Students were then helped to perform the necessary operations for the modifications. Finally, at the end of the course, the teacher gathered the students to discuss and evaluate the results.

There are two design discussion branches in this course. The teacher decided to roll back to a previous state of the discussion about 30 min after the class began, and, from that point, initiated a new direction for discussion.

4.3. Study Design

Through user study, we want to investigate whether users can grasp and understand the content of the VR design discussion course thoroughly and efficiently. In addition to questionnaires, participants were asked to draw their own design based on the design logic they learned from the course. The designs were graded by the teacher who led the discussion.

4.4. Procedure

There are three phases in the user study. First, in the training phase, participants received a brief introduction to the system and then practiced with the system with enhanced training information, such as text and verbal guidance for each functionality by a facilitator. After practicing, the participants would be tested on five tasks to see if they understood fully how to use the system. Each task requires the coordinated use of several different functionalities.

After the training phase, the participants began exploring the course content using the system in the learning phase, which took approximately 20 min. Subsequently, participants completed post-experiment questionnaires, including the simulator sickness questionnaire (SSQ) [30], NASA TLX [31], presence [32], co-presence [33], system usability satisfaction [34], and the system evaluation questionnaire designed by the experimenter. Participants were also asked to draw their own design that met the design logic learned in learning phase.

4.5. Quantitative Results and Discussion

4.5.1. Simulator Sickness Questionnaire (SSQ)

The SSQ questionnaire was completed by the participants before and after the learning phase for comparison. We computed the average difference in the SSQ scores of the pre-learning phase score subtracted from the post-learning phase score for each item in the questionnaire. A Wilcoxon signed rank test conducted between pre- and post-SSQ scores did not reveal a significant difference. The most notable result comes from the symptom in question 2, which represents the fatigue symptom. Approximately 80% of the participants felt more tired after the user study. Due to the fact that our system requires a significant amount of mental effort, the users needs to stay focused for some time. The symptoms in Questions 1, 3, 4, and 5 refer to “general discomfort, headache, eye strain” and “difficulty focusing”, respectively. Approximately 30–40% of the participants felt that these symptoms increased after the user study. One possible reason for this result might be unfamiliarity with VR equipment. Most of the participants were novice users of VR, and may not have been able to adapt to the VR environment as quickly as we expected. The symptoms in Questions 8, 15, and 16 refer to “nausea, stomach awareness” and “burping”, respectively. According to the results, no users experienced these symptoms before or after using the VR system, which was highly encouraging. Figure 10 shows the results of the SSQ questionnaire.

4.5.2. NASA TLX Questionnaire

The NASA Task Load Index (NASA-TLX) is a widely used, subjective, multidimensional assessment tool that rates perceived workload in a task using a system. The average workload scores of all participants are shown in Figure 11.

As shown in Figure 11, the value for the temporal demand is approximately 30, while the values for other dimensions range from 100 to 160. Since experiencing the course content is a critical aspect in learning, in the learning phase we did not impose any kind of time limit and hence induced no time pressure on the participants. On the other hand, the frustration level received the highest demand in the study. This may be due to the poor performance of gesture recognition supported by the Leap Motion system for the menu interaction in front of the participant, and this should be improved in the future. Since the purpose of the system is to offer the participants an effective learning experience in VR, instead of the frustration level, the mental demand is expected to be the one with the highest value, as the participants would focus on the course content for quite a long time during the learning phase. However, we found that mental demand was somewhat lower than effort or frustration.

4.5.3. Presence and Co-Presence Questionnaire

Our VR system could potential enhance both presence via the object, environment, and architectural details to some extent, and co-presence via the shared presence of static avatars that represented students, teachers, or designers in the architectural design discussion content interactive visualization experience. The 17 items in the survey were rated on a Likert scale of 1 (strongly disagree) to 7 (strongly agree).

In the responses to items 8, 9, and 10, “Somehow I felt that the virtual world would surround me”, “I felt present in the virtual space”, “I did not pay attention to the real world”, participants’ mean presence scores ranged from five to six (slightly agree to moderately agree), indicating that participants commonly had the feeling of being present in the virtual environment and being totally immersed. The rather low score for item 5, “I did not feel present in the virtual space”, also agrees with the above observation. The rather low score received for item 11, “The virtual world seemed more realistic than the real world”, is reasonable, since the architecture model we reviewed at this design stage is not a final model, and it is a plain model without appropriate light and material setting. The low score (slightly disagree to neither agree nor disagree) for item 1, “I was aware of the real world while navigating in the virtual world (i.e., sounds, room temperature, other people, etc.)”, may be due to the fact that there were approximately 10 people inside the room during the user study, although each participant is in an isolated area surrounded by curtains. The response to item 4, “The experience in the virtual environment seemed consistent with my real-world experience”, also received a low score. It is hard to interpret this result, as there seems to be no corresponding real-world experience in this case. Figure 12 shows the results of the Presence and Co-presence Questionnaire.

According to the feedback, we can see that users are quite focused on learning in the VR environment during the user study; they are able to describe what happened in the design discussion and learn from it. During learning, it is acceptable for users to focus on the content and not be affected by what happens in the real world. This matched our goal of making the whole learning experience efficient; therefore, the user can always find interesting content easily, instead of spending their time on a not-so-attractive part of the design discussion content. However, users either agree or disagree that the system provides a convincing rendering effect to make the environment appear realistic.

4.5.4. System Usability Satisfaction Questionnaire

We employed the IBM system usability questionnaire, which consists of 20 questions, to measure the system effectiveness, efficiency, and satisfaction rated on a scale from 1 to 5. We found that most users agree that our system was easy to use, useful, and satisfactory for helping users during the learning phase of the interactive design discussion content exploration task. The response to the items “The information provided for the system is easy to understand” (M = 3.75, SD = 0.8), “The information is effective in helping me complete the task and scenarios” (M = 3.6, SD = 1.1), and “The organization of information on the system screen is clear” (M = 3.7, SD = 1.2) received relatively high scores from the participants. We also found that the participants approved the quality of information that our system provided and considered them helpful. The rather low point for item “The system gives error messages that clearly tell me how to fixed problems” (M = 2.85, SD = 1.5) may be due to the fact that the system does nothing when the menu selection failed due to poor gesture recognition. Similarly, for the item “The interface of this system is pleasant” (M = 3.0, SD = 1.8), the relatively low score implies that beautification of the interface is necessary in the future.

4.5.5. System Evaluation Questionnaire

This researcher-designed questionnaire’s purpose was to assess the features of our VR system from both a quantitative and a qualitative perspective. The questionnaire is divided into two parts: system overall evaluation and system interactive functionality.

4.5.6. System Overall Evaluation

In the questionnaire, there were seven questions that evaluated what the participants felt about the system, including “The system was interactive, enjoyable, satisfying, user friendly, usable, not frustrating to use, and useful” and rated it on a scale of 1 (strongly disagree) to 5 (strongly agree). Results are shown as items 1 to 7 in Figure 13. Based on feedback, most of the participants considered the system interactive (item 1) and usable (item 5), which means that most of them were able to learn and gained familiarity with how to interact with the system during the user study and found the system to work normally without any interruptions due to errors. Items 2, 3, and 4 received mean scores ranging from 3.6 to 3.8, indicating that participants tended to agree that the system is enjoyable, satisfying, and user-friendly. Finally, item 6, “The system was not frustrating to use”, received a rather low mean score of 3.1, implying that participants may have encountered some glitches such as gesture recognition problems. In order to compare the proposed system with the video-based learning system, we included items 8 to 11 to assess what the participants felt about the difference between the two systems, including “The VR exploration system provides a better understanding of 3D building space, gives a better learning experience, provides more effective learning due to the summarization-to-detail approach than the video-based system”, and item 11, “By using VR exploration system, you have completely realized the content of this VR architecture design discussion course”.

4.5.7. System Interactive Functionality

In order to evaluate the functional design of the system, the questionnaire assessed the user’s feelings about 11 functionalities provided in the system on a scale of 1 (worst) to 10 (best). The 11 functions evaluated were (1) color maps on minimap, (2) temporal distribution of dialogues in the timeline, (3) Theme River, (4) summarization visualization for understanding what happened in the discussion course, (5) preview, (6) Worlds in Miniature visualization and interaction, (7) playback, (8) summarization-to-detail approach for facilitating the understanding of the design discussion content, (9) the menu’s readability, (10) the menu’s usability, and (11) the menu’s convenience in interaction. The results are shown in Figure 14. Overall, all features received a mean score of more than 7 points. Features 2, 6, and 10 even received a mean score of 8. It seems that users consider the dialogue list to be helpful for them to realize the distribution of dialogues in the course. They were also fond of the Worlds in Miniature functionality, which offers a global perspective of the object modifications that occurred during the discussion.

4.5.8. Learning Effectiveness Assessment

Besides the evaluation of the system’s feature and user experience, it was also necessary to see if the participants really understood the content of the discussion course. Eight architecture majors were also asked to draw their own designs based on the design logic they perceived from the course content. The hand-drawn designs were graded by a teacher who led the design discussion for the course content; see Figure 15 for the scores of learning effectiveness for each participant that took the test. The design was evaluated on three different aspects: consistency, completeness, and originality. In the course material for the user study, the teacher mentioned revision advice for the building, such as staggered construction, protruding roof, walls with holes, and more thin frames. Participants needed to add these features to their design to receive a high consistency score. The more such features were in the design, the higher the grade that would be given with respect to completeness. However, participants were also expected to make their design unique, which would contribute to the originality score.

As shown in Figure 15, the eight participants were able to obtain scores higher than 75%, with an average of 81%. It appears that participants were able to grasp and understand the design logic behind the design discussion and were able to create their own designs afterwards. From this, we can conclude that the proposed system was successful in storytelling and extracting the focal discussion in the interactive immersive VR design discussion content exploration afforded by our system.

4.6. Discussion of Qualitative and Quantitative Results

In the system evaluation questionnaire, we also asked participants to express their opinions about what they liked or did not like the most about the features provided in the system and the overall learning experience. Most of the participants considered the system useful and helpful for them to understand and learn the design discussion course that we discussed her. Participants enjoyed the strengths of the system, such as the summarization visualization, timeline filtering, experiencing the design logic in 3D space, minimap visualization of important events, high levels of immersion in the discussion conversation, teacher following, ease of grasping the course content, minimap, and Worlds-in-Miniature interactions, as well as being able to perceive what objects changed over time, playback functionality, and the overall discussion exploration model that was employed for the summarization-to-detail approach.

We also received negative feedback regarding the system’s functionality, such as the lack of ease of control of the gesture-based interaction and the lack of aesthetics of the design of the menu system. Since none of the participants had any experience using gesture-based input devices before, they needed some time to become familiar with the gesture-based interaction metaphor in our system. The effective operational characteristics of the Leap Motion-based gesture tracking device were also limited; sometimes users lost tracking of their hands if their hands left the tracker workspace, and gesture recognition was susceptible to tracker latency and jitter. Therefore, our system received a high score on “frustration” on the NASA TLX Questionnaire. As for the design of menus, our system rendered the menu as a pop-up display that may have been too large to allow interaction from the perspective of some users. Some of the participants said that they had difficulty interacting with the menu as it required large movements of their hands. Additionally, it was also suggested that the aesthetics of the menus be enhanced.

From a subjective quantitative perspective, we were excited to find that the overall metrics of presence, system usability satisfaction, overall system functionality evaluation, and most importantly, the learning performance of the architectural design using our system were high. However, in future iterations of our research, we hope to take advantage of advances in gesture tracking technology to improve the usability and user experience of gesture-based interaction in our architecture design discussion visualization system.

5. Conclusions and Future Works

We have presented a novel asynchronous VR interactive exploration system for learners to grasp and understand the design logic or concept from the architecture design discussion content achieved in VR. We conducted an exploratory user study evaluating the effectiveness and efficiency of the proposed system from a usability and user experience perspective. We found that participants could be immersed in the design discussion scenario or atmosphere and obtaied a robust and useful spatial understanding about architecture design, decision processes, and contextual information in the design process. Additionally, with the proposed summarization-to-detail approach, participants were able to follow and explore the design discussion process effectively and efficiently. The results of our study revealed that the asynchronous VR exploration system could be potentially effective for architecture design education. From a broader impacts perspective, our application and lessons learned have implications not only on the design and development of future asynchronous VR exploration systems for architectural design discussion content, but also other applications, such as industrial visual inspections and educational visualizations of design discussion content in general.

In future work, we aim to improve the interaction interface and to enhance the main menu and Worlds-in-Miniature (WIM) interaction metaphors within our system. In our system, gesture-based interaction is used to interface with menus and as a command and control tool for navigation and WIM manipulation. The current implementation is powered by Leap Motion and suffers from accuracy and stability problems. A better gesture-based interface will greatly enhance the learning experience. Currently, WIM is a scaled-down model for the entire architecture model. Therefore, occlusion might become a problem when a time-lapse animation of object changes appears in only a small portion of the WIM. We will investigate a WIM for a partial architecture model in a future version of the system.