Co-Created Virtual Reality (VR) Modules in Landscape Architecture Education: A Mixed Methods Study Investigating the Pedagogical Effectiveness of VR

Round 1

Reviewer 1 Report (New Reviewer)

Comments and Suggestions for Authors

The article is well-written and investigates a unique option for co-created VR modules in landscape architecture education. The methodology is well-formatted. The analysis and discussion align pretty well with the abstract. A few points can be included to improve the article. One, what kind of coding system did the investigation use to analyze the survey data? It seems the collected data is categorized under a coding scheme. It is better to mention explicitly the coding scheme. How are the virtual artefacts allowing the users to make active design decisions? The authors may try to align the collected data with the concept of the constructivism theory of education which they mentioned in the text.  

Author Response

All the comments from the respected reviewers have been addressed in the updated manuscript.

 

The article is well-written and investigates a unique option for co-created VR modules in landscape architecture education. The methodology is well-formatted. The analysis and discussion align pretty well with the abstract. A few points can be included to improve the article.

One, what kind of coding system did the investigation use to analyze the survey data? It seems the collected data is categorized under a coding scheme. It is better to mention explicitly the coding scheme.

• In-vivo coding was used to label focus group transcripts. The codes were organized in a code-line, and thematic analysis was conducted to identify recurring topics. This approach facilitated a comprehensive analysis of the collected data and helped us understand the participants' perspectives more thoroughly. This is now discussed in a more elaborate manner in sub-chapter 3.3.

How are the virtual artefacts allowing the users to make active design decisions? The authors may try to align the collected data with the concept of the constructivism theory of education which they mentioned in the text. 

• We thank the reviewer for this insightful comment. While using VR for active design decisions/processes was beyond the scope of this research, we added a couple of lines in the conclusion section on how future research should investigate this area.

Reviewer 2 Report (New Reviewer)

Comments and Suggestions for Authors

This research investigates the pedagogical benefits and challenges of XR implementation in landscape architecture curricula and offers landscape architecture educators actionable insights and frameworks about utilizing the new learning tools. The research also offers a well working model how to combine traditional material and tools and gradually move to 3D and XR. The research is well documented with a comprehensive theoretical framework. Although my list of comments is long, the revision is mostly technical and therefore I have considered it to be a minor revision.

§  It is uncommon to use referencing in the abstract. I would suggest removing ref #1 here.

§  Figure 1: Should the mixed reality line be shorter and not to extend all the way to the other end of VR?

§  There are no references on lines 85-89 and there should be as this is not common knowledge.

§  I would suggest separating the introduction clearly from the theoretical framework by having the theoretical framework as chapter #2. In multiple places the sub-chapters are too short i.e. the sub-headings break the text too much making it difficult for the reader to form a coherent story in their mind (e.g. 1.4 and 2.1, however, please check the whole article).

§  When introducing LAAB, it would be informative to mention where LAAB operates (country/countries) and overall explain a bit more about LAAB for non-American readers. Also, your reference management software has not recorded LAAB correctly in the references section.

§  As the XR-Ed framework was created by Yang et al., it would be appropriate to acknowledge it as theirs (names and not just the reference number), similarly as in 1.3.1.1 Carl Steinitz’s framework. Same with Mishra/Koehler TPCK. Alternatively, if Steinitz’ framework is more fundamental work than the other two, you could consider renaming sub-headings 1.3.3 and 1.3.4 with something more general than the actual framework names.

§  Font in figure 2 left hand side (four objects) is too small to read.

§   It would be preferable to present the results of the focus group also in Results-chapter and then focus purely on the discussion in the Discussion-chapter.

§  I do not see any value in presenting the speculative figure 13

§  The complex question structures (double negatives) in the questionnaire is well discussed in the limitations. Some of the questions also have two separate issues to evaluate, maybe this could be mentioned as well (…material was interesting (one issue), but it did not help… (another issue)).

§  No new information should be introduced in conclusions i.e. references used here should be already introduced either in the theoretical framework or in the discussion. Therefore, referencing is not necessarily needed at all with conclusions.

§  In-text citation is in places a mixture of the name-year and number systems. Please check e.g. lines 176 Lee et al. (2022) and 181 Misius (2021).

§  Reference list needs attention, particularly the use of capitals (first letters of words) should be consistent throughout.

Author Response

All the comments from the respected reviewers have been addressed in the updated manuscript, as stated in the cover letter.

 

 

Author Response File: Author Response.pdf

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper shows empirical result on VR experience and how it interacts with work and how it can achieve realistic scaling and representation with potential for interactive feedback. The author shows aspects which underscore the potential of VR as a powerful tool in landscape architecture education, offering unique and effective learning methods that traditional  methods might not provide.

Comments on the Quality of English Language

Some English editing and clarification would help enhance the level of scholarship

For example,

p 9 - your quantitative aspect of the study involved administering an online survey to 16 undergraduate students and  7 on focus group - the sample size too small to make an impact? 

p 12 mentioned landscape architecture educational framework, pls list references and samples for this framework referred to.  You had the opportunity to talk about that on page 3 on your theoretical framework but you chose to talk about the constructivist principles instead. 

p 18 - in a new way - pls explain what is the new way? 

Reviewer 2 Report

Comments and Suggestions for Authors

The editors of the journal " Education Sciences" have requested a review of the following paper: " Co-Created Virtual Reality (VR) Modules in Landscape Architecture Education: A Mixed-Methods Study Investigating Pedagogical Effectiveness of VR."

The results presented and described in this evaluation do not claim 100% completeness and are to be understood as a selection of most important aspects and examples.

ABSTRACT:
In the beginning of the abstract, a claim is made that the use of XR requires more structured guidance. However, how this assertion specifically relates to architecture or landscape education is not clearly articulated, and the paper's target audience remains unclear. It is not specified whether the paper aims to provide guidelines for the use of VR in these specific fields or if it intends to address XR applications more broadly. The abstract lacks clarity on what readers can expect from the paper, leaving the overall purpose and focus ambiguous.

A claim is made suggesting that constraining VR to a presenter role restricts its potential, even though it incorporates interaction. The paper lacks a precise definition of interaction. What constitutes proper interaction? Simply engaging visual media with sound falls short of qualifying as interaction for experimental education in immersive reality.

Furthermore, starting with the claim that "despite the interactive capabilities of the VR, positioning it primarily as a presenter limits its utility," it initially suggests a fully interactive design process in VR. However, the experimental design process is entirely manual and not within a VR environment.

 

MANUSCRIPT:

In line 49, there is an open question that suggests the need for more longitudinal studies to comprehend the impacts and potential challenges of integrating XR into higher education. It is unclear whether this paper aligns with any other ongoing longitudinal studies.

In line 70, a comparison is drawn between AR and MR, while in Figure 1, AR is depicted as a subset of MR. The comparison appears to be inaccurate, as Figure 1 illustrates that AR and VR are distinct subsets of MR, while VR and MR seem to be the same but AR is not… In general, the authors seem to use the terminology of AR, MR, XR, VR, … quite exchangeable. In the context of this paper and the scope of the journal it may be good to have a clearer strategy.

In line 97, the authors highlight the utility of VR for courses like architecture and landscape design. The presented project involves four interaction steps between tutors and students, and three of them are not related to engaging in a virtual environment—specifically, sketching, CAD drawing, and digital modeling (even though the reviewer understands its meaning for the later described comparison and evaluation). The sole instance where students create a digital exhibition constitutes the final step. Therefore, the reference to the Covid situation may not be entirely apt, as students are not participating in interactive experimental classes; rather, they are showcasing final results in a virtual environment. This could be accomplished through a web-based system, cloud presentation, or any other digital medium, without incurring additional hardware and costs.

The paragraph (lines 107 to 111) presents either a hypothesis or a statement without evidence or reference. Simultaneously, this paragraph forms the basis for a self-fulfilling prophecy, assuming, "Need to integrate technological intervention into the educational framework. This indicates that more research on effective integration of XR technology in formal higher education is imminent." Please provide proper substantiation for the statement and clarify the link between this claim and the earlier assertion about the effectiveness of XR technology in complex tasks.

Line 112, Theoretical framework” - the numbering should be 1.3, and in line 117, it should be 1.3.1. Numbering errors persist through this chapter.

In line 116 of the theoretical framework, the author mentions the existence of various theoretical criteria, presenting some of them. However, the rationale for selecting these specific criteria is not provided. It remains unclear why readers should engage with and continue reading this particular set of materials.

Lines 131 and 132 - it is questionable whether knowledge is truly generated here or if it is pure information.

Line 136-137 - Repetition of already presented content. Address the repetition of content in mentioned lines, as it may affect the overall coherence of the manuscript.

Lines 144, 166, 180, 191, 195, 200, 204 - Correct the numbering inconsistencies in the manuscript.

In line 146, six terms are introduced to define the dimensions of XR-ED, yet there is no accompanying clarification or definition for these terms. The extent to which these terms align with physical laws, assembly realism, or the incorporation of realistic features in various experiences remains unclear. Moreover, their impact on learning, as well as their potential to either enhance or restrict creativity, lacks explicit exploration.

In Chapter “Challenges in Implementation”, challenges are mentioned, presumably contributing to the need for guidance, yet the paper does not thoroughly address them.

Lines 184/185 - The presented conclusion is questionable. Clarify and provide examples or evidence to support statements to avoid potential misunderstandings or contradictions within the text.

Lines 192 to 194 - In this paragraph, it is stated that XR can lead to disadvantages in learning, especially with complex content. This statement contradicts the earlier claim that this technology is suitable for complex tasks.

The paragraph from line 201 to line 203 is very general, not considering factors such as discipline dependence, technology affinity, the number of students in the course, and so on.

Chapter “Why Integrate XR in Landscape Architecture Education?” lacks exploration of other related VR and co-created VR projects in design, especially in architectural education. The references are general and insufficiently detailed. Half of the chapter consists of repetitions, and the remainder does not sufficiently support the claim for the need for guidance. There is a notable absence of previous examples of XR or VR implementation in architectural and landscape education. The VR references focus on general educational benefits. Additionally, paragraphs 207 and following, or 210 and following, repeat previously presented content. (see above)

The assertion in line 215 is unclear and needs better explanation or examples.

It is unclear why landscape architecture (line 217) is described as multidisciplinary, given that all examples listed have a clear disciplinary connection to architecture or landscape architecture. If relevant, explain the multidisciplinary characterization of landscape architecture in the context of this paper. The presented connection between landscape architecture and studio practice is unclear in the context of this paper, despite being supported by references.

Lines 222-231 - The reviewer is unclear about the significance of lines 222 to 231 for this paper. The focus suddenly shifts to professionals and clients instead of students and learners, leading ultimately from line 231 to the core statement, "This research aims to dive deep into the development process of XR development for landscape architecture, asking about the correct sequence of incorporating the technology with the conventional educational framework," which is not related to the preceding paragraph. Address the sudden shift in focus, ensuring coherence with the overall narrative and objectives of the paper.

Additionally, there is a question about whether the defined goal, "to dive deep into the development process of XR development," is accurate.

At the end of the introduction chapters, despite acknowledging the existence of XR frameworks and theoretical aspects, the reasons behind their selection are not elucidated. The paper notes the attention XR has gained in education in the last 20 years but fails to provide concrete examples, especially in architectural and landscape design education. While weaknesses are mentioned, they remain overly generalized.

Line 242 - The study involves 16 students, which is a very small number to achieve reliable results and draw meaningful conclusions. Justify the small sample size of 16 students in the study and discuss its implications for the reliability of results and conclusions drawn.

So far, landscape architecture has been the focus, especially in the title of the paper (reference to the previously mentioned multidisciplinarity). From line 244, it becomes clear that the learning goal is the understanding of a construction detail of the already realized bench from the Highline Park in New York.  Clarify the discrepancy between the initially discussed topic of landscape architecture and the later focus on a specific construction detail from the Highline Park in New York. Following line 249, complex construction details are mentioned, which seems questionable given the task of re-designing a park bench.

Avoid unnecessary repetition of content between Lines 260 vs 294 to enhance the overall flow of the manuscript.

Between line 260 and 294, traditional teaching methods such as freehand sketching and 2D computer drawings are described. These paragraphs make several claims without any evidence, such as in line 274, stating that hand drawing "was crucial for developing students' conception thinking and design skills." Also, the concluding sentence about 2D CAD drawing, "it bridges the gap between conceptual sketches and practical, executable design plans," seems questionable as it is unclear why switching from hand to computer drawings would enhance quality or enable execution.

The representations in Figure 3 are too small to evaluate.

Line 298 - Clearly articulate the innovation in the XR environment mentioned in Line 298 and provide supporting evidence for the claim.

In Chapter 2.2.3, at line 333, the paper states that the Quest 2 is chosen for its stand-alone feature, but at line 338, it mentions the connection to a PC. The benefit of being stand-alone while connected to a PC for processing is unclear. Additionally, at line 342, the combination of the stand-alone feature and computer connection is briefly mentioned but not explained, contradicting the claimed VR device selection criteria. Overall, it is unclear why all the software and hardware information is provided, as it is not of utmost relevance for later evaluation or results.

In the context of the quantitative and qualitative data collection and later evaluation (Chapter 2.3.1 and 2.3.2), the adequacy of the number of subjects is questioned again, whether it is large enough and sufficient for reliable evaluation and corresponding conclusions.

From Chapter 3 onwards, the results are presented. It seems that the authors have insufficient experience in creating questionnaires and their evaluation. As the descriptions make clear, the authors themselves found that the way questions are formulated has clear effects on the results. The reviewer questions the choice of questions, for example, the confusion arising from double-negatives. (eg – How to answer? – Yes, I agree that I did not understand. vs. No, I agree that I did not understand. Vs. No, I disagree that I did not understand.)

Especially in the upper half of Figure 7, the context seems to be little to not meaningful or even counterproductive because conclusions cannot be drawn due to the described confusion and the small number of participants.

The representations and font sizes of Figures 7, 8, 9, and 10 are difficult to read. Please ensure that figures are legible and directly correspond with the text. Some questions repeat, such as question 1 and question 6 in Figure 10.

Usually, the Discussion chapter (line 468 critical examination of research questions and results) serves for a critical evaluation. From the reviewer's perspective, the entire Discussion chapter is influenced by the authors' expectations. Address concerns about the author's self-fulfilling prophecy and provide clearer reasoning and evidence for statements (also in lines 481, and 489).

For example: Given Figure 10, the authors' conclusion (lines 471 to 474) is not understandable, as the presented results show an overwhelming preference for 3D modeling and CAD drawing.

In the discussion at line 481, the first claim in the abstract is reiterated. The paper suggests repositioning VR in education from presentation only to more interactive elements, but this claim is not thoroughly explored. Existing literature already showcases projects that have successfully implemented interactive VR in both lecture presentations and group design. Line 481 also states that maybe there is potential for integration into other learning phases and processes - in the context of this study, this seems to be an assumption by the authors.

Line 489 - "the integration of VR... could lead to a more comprehensive understanding and application of landscape architecture principles." On what basis is this statement made - on which datasets or results?

In Figure 13, questions 1 and 6 are almost identical, explaining the results of 25% and 31%. The text does not explicitly address the content of Figure 13, but with the same number of respondents, namely 16, the result is 4/16, 1/16, 4/16, 3/16, 6/16, and 5/16.

In lines 517 to 523, it is described that participants have gained a deeper understanding of details due to interactivity and immersion. The question of who and how the adequacy of details regarding feasibility and aesthetics was evaluated remains open. The question also arises about the consequences for learners if incorrect results are implemented without a quality control system (a kind of democratic process in implementing the results by students in the VR environment was described earlier).

 

The following examples, from the reviewer's perspective, include incorrect or unverifiable results and conclusions.

Line 564 - "utilizing XR to comprehend and integrate construction details furnishes a more hands-on experience, and users reported a significant advantage over traditional paper-based learning."

Line 592 - "all the users noted that XR technology is currently in its infancy, and the future potential is endless."

The bullet-point lists from line 606 to 681 are not very meaningful, more or less enumerations, and could be created in this form by Chat GPT.

Also, for Chapter 5 “Conclusions”, reevaluate and provide unbiased conclusions, avoiding statements that may appear as confirmation bias. Clarify statements like "the research confirmed the significant and hand of learning in landscape architecture construction through virtual reality..."