Piecewise: A Non-Isomorphic 3D Manipulation Technique That Factors Upper-Limb Ergonomics

Abstract

Virtual reality (VR) is gaining popularity as an educational, training, and healthcare tool due to its decreasing cost. Because of the high user variability in terms of ergonomics, 3D manipulation techniques (3DMTs) for 3D user interfaces (3DUIs) must be adjustable for comfort and usability, hence avoiding interactions that only function for the typical user. Given the role of the upper limb (i.e., arm, forearm, and hands) in interacting with virtual objects, research has led to the development of 3DMTs for facilitating isomorphic (i.e., an equal translation of controller movement) and non-isomorphic (i.e., adjusted controller visuals in VR) interactions. Although advances in 3DMTs have been proven to facilitate VR interactions, user variability has not been addressed in terms of ergonomics. This work introduces Piecewise, an upper-limb-customized non-isomorphic 3DMT for 3DUIs that accounts for user variability by incorporating upper-limb ergonomics and comfort range of motion. Our research investigates the effects of upper-limb ergonomics on time completion, skipped objects, percentage of reach, upper-body lean, engagement, and presence levels in comparison to common 3DMTs, such as normal (physical reach), object translation, and reach-bounded non-linear input amplification (RBNLIA). A 20-person within-subjects study revealed that upper-limb ergonomics influence the execution and perception of tasks in virtual reality. The proposed Piecewise approach ranked second behind the RBNLIA method, although all 3DMTs were evaluated as usable, engaging, and favorable in general. The implications of our research are significant because upper-limb ergonomics can affect VR performance for a broader range of users as the technology becomes widely available and adopted for accessibility and inclusive design, providing opportunities to provide additional customizations that can affect the VR user experience.

1. Introduction

Virtual reality (VR) is currently contributing to the development of psychomotor skills in education [1], training [2], and healthcare [3], among others, by allowing for immersion and exposure to scenarios that would otherwise be difficult or impossible to replicate in real life. However, despite the fact that recent breakthroughs in VR technology are mainly utilized in entertainment, one-size-fits-all solutions continue to be the norm, with advancements in comfort aimed at enhancing the user experience for more immersion and presence [4]. However, a lack of adequate ergonomics in virtual reality can have a negative effect on the user’s immersion and presence [5], as well as causing motion sickness [6]. Several contributors to the poor impact on the user experience include an inadequate reach, height, and interpupillary distance, which are primarily the result of one-size-fits-all solutions.

Our research focuses on the upper limb since effective user interactions are essential for conducting real-world activities or tasks in virtual reality [7]. Upper-limb interactions are essential in virtual reality by allowing for reaching and gripping, which require a combination of movements resulting from the 27 degrees of freedom (DOFs) from the shoulder to the wrist, [8]. The 27 DOFs allow for the flexion and extension, abduction and adduction, and pronation and supination of the upper limb. Moving the upper limb within its range of motion should not result in tiredness and pain, which cause fatigue and/or discomfort. However, repetitive or prolonged physical activity, such as keeping arms in midair, may create weariness and elevate exertion levels, which may develop into musculoskeletal problems, such as the gorilla arm [4]. Three-dimensional manipulation techniques can minimize cognitive stress [1] and hazardous body movements linked with musculoskeletal illnesses [4] when combined with good ergonomics.

Research on 3DMTs, such as isomorphic (a mathematical term meaning “sameness”) and non-isomorphic (i.e., “not the same”) techniques for reaching, selecting, and grabbing objects in VR [9], is providing solutions for enhancing the VR user experience, such as Erg-O, a non-isomorphic 3DMT that remaps the position of VR objects [10]. Isomorphic and non-isomorphic 3DMTs are essential in VR applications, such as entertainment, education, healthcare, and training, to ensure that users can properly interact with the virtual environment. Object translation, commonly referred to as “distance grab,” is a typical isomorphic 3DMT in which virtual objects are translated to the user’s hand by stretching, expanding their reach. This 3DMT is included in various VR software development kits (SDKs) (e.g., Oculus, Steam VR, and the virtual reality toolkit (VRTK)) as it allows for interaction with objects out of physical reach. Although object translation improves reach, it can negatively impact immersion, presence, and embodiment as interactions no longer require arm movements matching those in VR. This scenario has sparked interest in the development of novel 3DMTs [11,12,13]. Rietzler et al. (2018) [5] examined the effects of virtual hand offsets on immersion and presence relative to the real hand location by examining the felt weight of virtual items in four tasks that allowed participants to interact with virtual objects. When portraying the weight of virtual objects, the study found that the offsets favor immersion and presence marginally.

Li, Cho, and Wartell (2018) compared the effects of offsets on four reaching conditions, focusing on efficiency, accuracy, and timely completion [14]. The four reaching conditions were: (i) no offset (physical reach), (ii) a fixed offset where the virtual hand would be translated one meter from the real hand position, (iii) a dynamic non-isomorphic linear offset that increased the virtual reach by adjusting the magnitude of the actual hand within different distances from the HMD, and (iv) a non-isomorphic quadratic gain reach approach (similar to the GoGo non-linear mapping technique [15]). The study enrolled 24 participants who used the Oculus Rift DK2 (discontinued), the Leap Motion, and the Razer Hydra (a discontinued computer desktop motion-sensing VR controller that uses magnetic tracking). The test subjects were instructed to grab a color-changing sphere at four different depth clusters. The results show that participants could only reach spheres in all four cluster depths when using non-isomorphic offsets, and the dynamic offset condition (iii) outperformed all other reach techniques. It is important to note that the Leap Motion remains a relevant 3DUI for dexterity used in human–computer interactions for graphical user interfaces, exergames, and educational applications [16]. Although the Oculus Rift DK2 and Razer Hydra are discontinued, the findings from Li, Cho, and Wartell remain fundamental for understanding the role of offsets in 3DMTs.

Another 3DMT that facilitates virtual object manipulation employs multi-object re-targeting with freehand movement, based on isomorphic visual-to-physical mapping, and an optimized computation for rendering the interactable objects with the least visual difference [10]. Re-targeting produces ease-of-use manipulations by remapping the targets to appear closer to the user, partitioning the physical and visual space with tetrahedrons. The study presented three layouts and three re-targeting strategies to 12 participants, who completed nine trials in total. The three layouts included one employing a a normal mode or physical reach mode, and two variations of the proposed Erg-O 3DMT for spacial consistent and ergonomics.

Murillo, Subramanian, and Plasencia [10] used re-targeting to facilitate VE manipulation by remapping the targets closer to the user. This approach allows for multi-object re-targeting with freehand movement, isomorphic visual-to-physical mapping, and an optimized computation for rendering interactable objects with the least visual discrepancy. Participants in the study were presented with three interaction layouts and three 3DMTs. The layouts consisted of (i) 15 spheres arranged in a 5 × 3 grid, (ii) 24 spheres distributed evenly within the participant’s reach, and (iii) 24 spheres placed beyond the participant’s reach. The 3DMTs included a normal mode (physical reach) and two variations of Erg-O, focusing on spatial consistency by leveraging visual dominance to keep VR elements close to the user rather than focusing on ergonomics to maintain objects within comfortable reach. Participants completed surveys regarding comfort and physical exertion. The study’s findings indicate that spatially consistent remapping can reduce task execution times, whereas ergonomic configurations lead to the least amount of overstretching.

Wentzel, d’Eon, and Vogel [17] compared low and high non-linear input amplification techniques to create virtual offsets for Erg-O to better understand their role in comfort, physical motion path length, virtual motion path length, ease of reach, overstretching, body ownership, and sense of control. Erg-O was replicated, and ergonomics were calculated using the HTC Vive Pro HMD in conjunction with compatible HTC Vive trackers strapped to the participants’ shoulders, elbows, and waist, as well as the Vive controllers and a Microsoft Kinect for body tracking. The study recruited 18 participants (9 of whom had moderate VR experience) and presented them with three different offset layouts: (i) no amplification, (ii) low amplification, and (iii) high amplification, in addition to three variations of Erg-O, including ergonomics, reach limits, and world fixation. The results demonstrate an increased comfort without reducing the task performance or body ownership when participants utilized the amplification layouts. Furthermore, physical path lengths decreased whereas virtual path lengths increased due to the offset created in the VE. No differences in time to completion were found, and comfort was determined by comparing the arm movement with comfortable ergonomic shoulder and elbow flexion and extension.

A follow-up study saw the recruitment of 18 participants (11 of whom had prior VR experience) who now experienced offset amplifications ranging from 0 percent to 45 percent at 5 percent increase intervals [17]. For this study, the HTC Vive trackers were removed in favor of body tracking using the Microsoft Kinect. At the same time, the VR headset was changed to the Oculus Rift S for a simplified setup, not requiring external tracking cameras for the headset. According to the second study, the offset’s slope could not be less than one, and offsets that increase a user’s arm reach by up to 30 percent are acceptable as long as the user maintains a sense of body ownership. Wentzel, d’Eon, and Vogel concluded that the participants’ performance and time completion results were comparable to the Erg-O method. Nevertheless, their offset method is more easily reproducible in numerous VR and VE applications.

The evolution of 3DMTs toward providing manipulation techniques with a greater reach has led to the development of isomorphic and non-isomorphic solutions and their impacts on the overall user experience. According to the related works, offsets are crucial in extending the user’s arm for interactions beyond their physical reach. For example, adding reach offsets can improve efficiency, accuracy, immersion, and task completion. Other approaches consider the upper limb’s maximum reach to re-map targets closer to the user, but still out of physical reach, with positive effects on comfort, completion time, and physical effort. Finally, other methods employ linear amplifications to compensate for the user’s reach in virtual reality, influencing comfort, embodiment, and task performance. While offsets and maximum reach provide manipulation techniques to improve the user experience, the related works do not integrate ergonomic ranges of motion as part of the 3DMT.

Ergonomic ranges of motion measure the impact of reach on the interactions [17]. In order to prevent overexertion or musculoskeletal disorders such as the gorilla arm [4], it is essential to keep movements within the ergonomic comfort range of movement. As the use of virtual reality (VR) continues to grow in popularity, one-size-fits-all ergonomics fail to account for user diversity. For instance, the COVID-19 pandemic increased VR adoption as a means of overcoming the limitations of traditional video calls in order to develop experiential learning and hands-on experiences [18], elderly care [19], and social connectedness [20], among others. By combining manipulation techniques for 3DUI principles and ergonomics, we intend to address one-size-fits-all reach in VR by factoring the upper-limb range of motion within a comfortable range.

In this paper, we present the development and evaluation of Piecewise, a non-isomorphic 3DMT for reaching and grabbing objects in VR that considers the ergonomics of the upper limb. We investigated the effects of accounting for upper-limb ergonomics on time to completion, task completion, percentage of reach, upper-body lean, engagement, and presence when compared to typical 3DMT methods, such as object translation (distance grab), reach-bounded non-linear input amplification (RBNLIA) [17], and the normal mode that relies on physically reaching for the objects in VR.

We hypothesize that Piecewise, a 3DMT in which the user is given an extended reach interaction customized to their upper-limb ergonomics to reposition the virtual hand in VR in comparison to normal (physical reach), object translation, and RBNLIA, will: (i) enable the execution of pick and place tasks in less time than other 3DMTs, (ii) reduce upper-body lean in comparison to the other 3DMTs, (iii) reduce the number of objects skipped in comparison to the other 3DMTs, (iv) maintain the percentage of reach within the comfort range of motion relative to other 3DMTs, and (v) provide a greater presence and immersion relative to other 3DMTs.

2. Materials and Methods

Our study consisted of a virtual scenario developed with the Unity game engine (2019.2.15f1) [21] and OpenVR [22]. Unity and OpenVR were chosen because, at the outset of the development process, they offered the tools required to support multiple VR headsets. Support for multiple VR headsets was required due to the restrictions imposed by the COVID-19 pandemic, which made collecting data online and remotely necessary. The study evaluated the effects on the engagement, presence, time, and task completion of four 3DMTs, including our proposed Piecewise method, which considers upper-limb ergonomics in comparison to object translation (distance grab), RBNLIA [17], and the normal mode, which rely on physically reaching for the objects in VR. In addition to performing the task in VR, the VR Questionnaire Toolkit [23] was utilized to house and display the Game Engagement Questionnaire (GEQ) [24] and the Presence Questionnaire (PQ) [25] in VR to minimize distractions and maintain the user’s immersion and presence in VR.

2.1. Ergonomics

As depicted in Figure 1, human body movement includes flexion and extension in the sagittal or longitudinal plane, and abduction and adduction in the frontal or coronal plane. For comfort, these motions should remain within a specified range. According to Openshaw and Taylor [8], the optimal positioning for different types of tasks varies based on the activity, necessitating the need for ergonomic designs focusing on reducing fatigue and increasing comfort, which can influence productivity, reduce stress, and affect the quality of life.

The range of motion for comfort and safety is divided into four zones: (i) Zone 0, coded green for minimal stress on muscles and joints; (ii) Zone 1, coded yellow for minimal stress on muscles and joints; (iii) Zone 2, coded red for a greater strain on muscles and joints; and (iv) Zone 3 beyond the red zone for an extreme strain on muscles and joints that should be avoided. These zones indicate the optimal ranges for comfort, which is crucial for avoiding potential musculoskeletal disorders caused by repetitive movements, overexertion, and poor postures, such as the gorilla arm [4].

Piecewise, our proposed non-isomorphic 3DMT, is intended for reaching and grabbing objects in virtual reality. Piecewise takes upper-limb ergonomics into account through a calibration procedure that offsets the reach based on the range of motion to ensure that objects can be reached while remaining within the ergonomic comfort range (i.e., the green zone, Figure 1). The virtual scene requires the user to reach out and grab objects on a table’s left and right sides from various distances. In order to compare our proposed Piecewise 3DMT, a management subsystem enables switching between three additional implemented manipulation methods, including object translation, RBNLIA, and a normal mode. A high-level overview of Piecewise is presented in Figure 2. An output data subsystem uses metrics that are recorded for calibration and performance metrics capture, such as task and time completion.

2.2. Calibration

Upper-limb calibration was used to capture user ergonomics, including their height when seated (referenced from the virtual floor calibration), the arms’ length, and the virtual shoulders’ location. A four-pose calibration consisting of: (i) arms up, (ii) arms forward, (iii) arms down, and (iv) arms outward allowed us to define both arms’ length and virtual shoulder locations. The calibration poses are shown in Figure 3.

2.3. Three-Dimensional Manipulation Techniques

In addition to Piecewise, the following 3DMTs were added for our study: (i) the normal mode (physical reach), the object translation mode (also known as distance grab), and the RBNLIA mode [17]. The subsections that follow go over each 3DMT that was used.

2.3.1. Normal Mode

The normal mode is a universal, unaltered interaction method that relies on physical reach, as depicted in Figure 4a. This mode, which remains the most popular among VR installations and is easily accessible on a variety of SDKs, including OpenVR, SteamVR, and OpenXR, and it strongly depends on each user’s upper-limb length and available VR area.

2.3.2. Piecewise Mode

The Piecewise mode, which factors in upper-limb calibration, extends the user’s reach interaction by adding a scalar multiplier to the controller’s location to reposition the virtual hand in VR. Piecewise was named after its multi-behavioral function, which is achieved by relocating the virtual hand in the VE using the forward vector based on the controller’s local orientation (see Figure 4b). The visual modification is activated when the user’s arms reach percentage hits 60% to maintain the range of motion within the comfort zone where a normal reach is performed. To reposition the controller, the scalar multiplier is doubled from a slope of 1 to a slope of 2 when the user’s reach surpasses 60%. When the user’s arm is fully extended, the maximum reach is adjusted by the scalar multiplier to be 140 percent. For instance, if the user’s arms reach is 100 cm, the Piecewise mode’s maximum reach is 140 cm. Nevertheless, the user will not experience an increase in reach until they cross 60% of their arm’s reach; in this case, 60 cm.

2.3.3. Object Translation/Distance Grab Mode

The object translation mode (also known as distance grab) [26] is also readily available as a built-in interaction option in various VR SDKs, such as Oculus, SteamVR, OpenXR, and OpenVR. In the object translation mode, users can interact with far-off items by employing an extended invisible hand collider increase that secures objects to the virtual hand (Figure 4c). The distance grab is achieved by a constant range increasing the controller’s local forward vector (see Figure 4). The maximum reach range in the object translation mode is 140 percent of the user’s reach, with the arm’s length being utilized to determine the 40% reach increase [17]. The additional range increase when interacting in the VE keeps a reach difference of 40% in centimeters (cm) from the user’s actual maximum reach. The maximum reach possible with the object translation mode, for instance, is 140 cm if the user’s arms reach is 100 cm. However, the user will always have an additional reach range of 40 cm added to their virtual hand position.

2.3.4. Reach-Bounded Non-Linear Input Amplification Mode

The RBNLIA mode leverages upper-limb ergonomics to expand the user’s reach for virtual interactions (Figure 4d). RBNLIA is comparable to the Piecewise mode that we have proposed, with the primary difference being how the offset is determined. For example, RBNLIA exponentially increases the user’s range so that, if the user’s arms reach nears 100%, the exponential rise slows. The user interaction modification remains active throughout the mode and grants the user an additional 40 percent of reach. The virtual hand’s size will expand exponentially from 0% to 60% based on the percentage of reach. As depicted in Figure 4d, an RBNLIA or slowing of the virtual hand will occur between 60 and 100 percent of the user’s reach for fine motor interactions.

2.4. Study Design

A within-subjects study was conducted in which participants were exposed to all 3DMTs. A balanced Latin square was used to ensure that all participants access all conditions to minimize biases toward any interaction mode. Additionally, each condition was built into a separate executable file given to the participants who ran them as indicated by the study facilitator. The decision to provide the participants with the executables was made to minimize problems with larger files and errors when selecting the appropriate 3DMT. Due to the COVID-19 pandemic, the study was conducted online remotely, requiring participants to have access to a compatible desktop HMD and sufficient space for running the software in seated mode while having room to fully extend their arms forward, outward and upward as shown in Figure 3. The study software was compressed into an executable build and e-mailed to participants, along with additional documentation for running the study executable file, filling out the consent form, completing the surveys in VR after each manipulation technique, and finally uploading their data after the study was completed.

After the calibration (Figure 5a), the participants were immediately guided through the scene with a tutorial presenting the controller layout and interactions, with demonstrations given by a virtual avatar (Figure 5b). After completing the tutorial at the start of the study, the participants then proceeded to complete the pick-and-place task within the VE, employing the normal, Piecewise, RBNLIA, and object translation modes (Figure 5c). After completing each 3DUI mode, participants answered the questionnaires in VR, beginning with in-game GEQ and subsequently PQ (Figure 5d). The in-game GEQ survey captures each participant’s experience with each 3DMT. The PQ questionnaire assesses the level of presence felt by the participants. However, the auditory, haptic, resolution, and interface quality subscales were omitted from the analysis because they were irrelevant to the current study. Figure 5 depicts a high-level summary of the study stages.

2.5. Participants

The study recruited 20 individuals, 11 females and 9 males. Five percent of the participants were between the ages of 18 and 24, sixty-six percent were between the ages of 25 and 34, ten percent were between the ages of 35 and 54, and twenty percent were at least 55 years old. Seventeen individuals utilized the Meta Quest 1 HMD, whereas just three utilized the Meta Quest 2 HMD. A total of 75% of participants indicated no prior exposure to virtual reality, 15% reported using VR 1-5 times per month, 5% reported using VR 10–20 times per month, and 5% reported using VR 20+ times per month.

2.6. VR Scene

On a virtual table, the user is presented with a total of 12 virtual objects (cylinders) at varied distances within the ergonomic motion range reach, including Zone 0, Zone 1, and Zone 2 (see Figure 6). The objects on the left side of the virtual table are presented first, starting with Left object 1 (L1) and increasing to L6, followed by the objects on the right side, starting with Right object 1 (R1) and increasing to R6. The height of the virtual table was adjusted to 71 cm, which is the usual height for desks. In compliance with ergonomic requirements for the table height and comfortable range of motion [27], the table also contains the assistance, reply, and skip buttons, which were 38 cm from the user. The objects were equipped with interaction and tracking scripts to allow for interactions and timestamped metric monitoring. In order for the next object to appear, participants had to place the object in the Target area (T) in the bottom-center of the virtual table.

Data Collection and Analysis

The data collected from the user study included the positions of the HMD and controllers, the position of the virtual hand, the start and end times to interact with an object and each mode, and the responses to the surveys asked in VR. Additionally, questions about their preferences toward any 3DMT and how exhausted they felt after completing the study were presented using a Google Form. Using the collected data, participant demographics and familiarity with VR were summarized, and the Statistical Package for the Social Sciences (SPSS) was used to analyze the collected data (i.e., time completion of each interaction mode, time completion of each object, percentage of arms reach, upper-body lean, and skipped objects). Using SPSS, the data were evaluated using the Shapiro–Wilk, Friedman, and Sign tests.

3. Results and Discussion

Our study’s findings are organized into subsections based on time completion, skipped objects, percentage of reach, upper-body lean, engagement, and presence.

3.1. Time Completion

Time completion was utilized to determine how quickly a participant could interact with an object and how much time was spent in each interaction mode. The results indicate that the object translation mode had the shortest average time spent in the mode at 31 s (M = 2.756 s/object, SD = 2.422), followed by the normal mode at 33.52 s (M = 3.176 s/object, SD = 2.331), and then the two customized non-isomorphic interactions: RBNLIA mode at 37.28 s (M = 3.148 s/object, SD = 2.195), and Piecewise mode at 43.56 s (M = 3.728 s/object, SD = 3.797). A Shapiro–Wilk test found that the data set on time completion had not been normalized (see Tablereftab:TC swt), and a Friedman test revealed a statistically significant difference, $X^2$(3) = 137.25, p ≤ 0.001 (see

Table 1. Time completion Shapiro–Wilk results.

Shapiro–Wilk	N ${}^1$	OT ${}^2$	P ${}^3$	F ${}^4$
p-value	0.001	0.001	0.001	0.001

¹ Normal, ² Object translation, ³ Piecewise, ⁴ RBNLIA.

).

A Bonferroni-adjusted non-parametric Sign test revealed a significant difference between all modes except for object translation and normal (see

Table 2. Time completion Sign test results for the total time per mode.

Sign Test	P and N	OT and N	RBNLIA and N	OT and P	RBNLIA and P	RBNLIA and OT
p-value	0.001	0.239	0.001	0.001	0.001	0.001

). The results disprove the hypothesis that customized non-isomorphic interactions will enable participants to complete tasks more quickly. The time spent in a mode was related to the number of objects that were skipped. In comparison to the Piecewise and RBNLIA modes, the normal and object translation modes had a larger average number of objects skipped (see Figure 7).

3.2. Skipped Objects

Figure 7 presents the number of skipped objects per 3DMT, from the 48 objects (12 per mode): 29 objects were skipped in the normal mode, 15 objects were skipped in the object translation mode, 6 objects were skipped in the Piecewise mode, and 4 objects were skipped in the RBNLIA mode. The L6 object (the farthest object on the left) had the most skips (12), followed by the R6 object (the farthest object on the right), which was skipped 9 times (for the table layout distribution, see Figure 6). Curiously, some participants skipped over L1, L2, and R1 objects that were closest to them. Considering the motion data and participant input, we believe that this may have been caused by tracking errors and inadvertently pressing the skip button, an issue that will require future research.

The majority of skipped objects were encountered in the normal mode, as the farthest ones required participants to physically reach them, often leaning to extend their reach. The second most frequently skipped objects were met in the object translation mode, where collected motion data and participant input suggested that the virtual hand location made it difficult to aim at the objects. Unlike the normal and object translation modes, the participants skipped six objects when using Piecewise, and skipped four objects when using BRNLIA. Based on the number of skipped objects, Piecewise had a higher task completion than the normal and object translation modes.

3.3. Percentage of Reach

A Shapiro–Wilk analysis revealed that the data set for the percentage of reach did not have a normal distribution (see

Table 3. Percentage of reach Shapiro–Wilk significance results.

Shapiro–Wilk	N	OT	P	RBNLIA
p-value	0.001	0.001	0.001	0.001

), and a Friedman test indicated that there were no statistically significant differences ($X^2$(3) = 5.04, p = 0.168). The average reach percentage per object, based on mode, showed that the reach percentages were all within the 90% range, but somewhat lower for objects positioned on the right side of the virtual table (see

Table 4. Average percentage of reach of each object based on interaction mode.

Object	Normal	Distance	Piecewise	RBNLIA
L1	95.508%	96.176%	95.335%	93.625%
L2	95.703%	97.109%	97.132%	96.552%
L3	96.130%	95.398%	97.655%	98.246%
L4	96.519%	97.223%	98.541%	98.263%
L5	97.003%	96.752%	98.421%	97.698%
L6	96.824%	97.495%	98.394%	97.852%
R1	94.267%	92.687%	91.686%	90.999%
R2	96.890%	95.541%	93.585%	94.979%
R3	98.065%	97.333%	94.938%	96.933%
R4	97.804%	97.483%	94.947%	97.928%
R5	98.345%	98.112%	94.919%	98.802%
R6	99.035%	98.042%	94.977%	98.808%

). Participants exhibited higher reach percentages when using the Piecewise and RBNLIA modes for objects on the left side of the virtual table, and when using the normal, object translation, and RBNLIA modes for objects on the right side.

The percentage of reach results is comparable across all DMT options (see Table 4). We believe that this is the case because the participants behaved as they would in real life, where extending the arm to reach objects is a more natural interaction than maintaining a reach percentage and leaning forward to correct for distance.

3.4. Upper-Body Lean

A Shapiro–Wilk analysis indicated that the lean data set was not normalized (see

Table 5. Upper-body lean Shapiro–Wilk significance results.

Shapiro–Wilk	N	OT	P	RBNLIA
p-value	0.003	0.001	0.001	0.001

), and a Friedman test revealed a statistically significant difference, $X^2$(3) = 289.742, p ≤ 0.001. An additional analysis utilizing a non-parametric Sign test revealed statistically significant differences across all interaction types (see

Table 6. Upper-body lean Sign test significance.

Sign Test	P and N	OT and N	RBNLIA and N	OT and P	RBNLIA and P	RBNLIA and OT
p-value	0.001	0.001	0.001	0.001	0.004	0.001

).

Figure 8 provides an overview of the average lean per object based on the interaction mode. The results indicated that the normal mode had the most upper-body lean (M = 32.30 cm, SD = 17.998), followed by the object translation mode (M = 25.04 cm, SD = 17.948), RBNLIA mode (M = 16.10 cm, SD = 15.920), and Piecewise mode (M = 11.37 cm, SD = 15.236). The amount of leaning by participants using the normal mode was a result of extending their reach in the absence of any virtual visual manipulation or offsets. Interestingly, a similar behavior was present when using object translation, where leaning occured as an effort to increase reach and accuracy. Unlike the normal and object translation 3DMTs, Piecewise and RBNLIA both shifted the participant’s virtual hand position in the virtual environment, which resulted in less leaning.

3.5. Engagement

A Shapiro–Wilk test revealed that the data set was not normally distributed (see

Table 7. In-game GEQ Shapiro–Wilk test.

Shapiro-Wilk	Chal ${}^1$	Com ${}^2$	Fl ${}^3$	Neg ${}^4$	Pos ${}^5$	Sen ${}^6$	Ten ${}^7$
p-value	0.004	0.001	0.001	0.001	0.001	0.001	0.001

¹ Challenge, ² Competence, ³ Flow, ⁴ Negative affect, ⁵ Positive affect, ⁶ Sensory, ⁷ Tension.

), and a Friedman test revealed significance in six of the seven GEQ categories: competence ($X^2$(3) = 12.876, p=.005), sensory ($X^2$(3) = 94.320, p ≤ 0.001), flow ($X^2$(3) = 7.826, p = 0.050), tension ($X^2$(3) = 70.541, p ≤ 0.001), challenge ($X^2$(3) = 62.935, p ≤ 0.001), negative affect ($X^2$(3) = 127.286, p ≤ 0.001), and positive affect ($X^2$(3) = 40.271, p ≤ 0.001).

A non-parametric Sign test with a Bonferroni adjustment indicated statistically significant differences between the majority of modes in each GEQ category (see

Table 8. GEQ Sign test significance: P-Piecewise, N-Normal, OT-Object Translation, RBNLIA.

Sign Test	P and N	OT and N	FRBNLIA and N	OT and P	RBNLIA and P	RBNLIA and OT
Competence	0.076	0.002	0.001	0.001	0.0036	0.001
Sensory	0.001	0.019	0.001	0.001	1	0.001
Tension	0.001	0.001	0.001	0.001	0.156	1
Challenge	0.378	0.001	0.412	0.001	1	0.001
Negative	0.001	0.001	0.001	0.001	0.001	0.001
Positive	0.338	0.036	0.001	0.076	0.001	0.001

). The normal mode received the lowest scores for challenge, negative affect, and tension, indicating that this method of interaction was the least engaging. The Piecewise mode received the highest scores in the GEQ categories for flow, sensory, and immersion, while the RBNLIA mode received the lowest scores for negative affect and the highest scores for positive affect, and the object translation mode received the lowest scores for challenge and tension and the highest scores for competence.

The GEQ results (see Figure 9) indicate that non-isomorphic interactions were more engaging than isomorphic ones. The GEQ ratings corresponded to the degree of reach that was made available in each interaction style. The normal mode did not provide participants with enhanced reaching aids; as a result, the success rate of task completion was impaired, resulting in high scores in the categories of difficulty, negative affect, and tension. The Piecewise and RBNLIA modes both had a virtual offset and produced better engagement results. The participants’ reach range increased the most in the object translation mode, and the findings reflect this.

3.6. Presence

The Shapiro–Wilk test revealed that the data set was not normalized (see

Table 9. PQ Shapiro–Wilk test.

Shapiro-Wilk	CF	SF	DF	RF	INV/C	NAT
p-value	0.001	0.001	0.001	0.034	0.002	0.001

), and the Friedman test revealed a statistically significant relationship between the interaction modes in each of the PQ categories: control factors (CFs) ($X^2$(3) = 94.242, p ≤ 0.001), sensory factors (SFs) ($X^2$(3) = 36.000, p ≤ 0.001), distraction factors (DFs) ($X^2$(3) = 38.571, p ≤ 0.001), realism factors (RFs) ($X^2$(3) = 32.424, p ≤ 0.001), involvement/control (INV/C) ($X^2$(3) = 65.217, p ≤ 0.001), and natural (NAT) ($X^2$(3) = 33.288, p ≤ 0.001).

An additional analysis using a non-parametric Sign test with Bonferroni adjustment indicated significant differences between the majority of interaction modes within each PQ category (see

Table 10. PQ Sign test significance.

Sign Test	P and N	OT and N	RBNLIA and N	OT and P	RBNLIA and P	RBNLIA and OT
CF	0.001	0.118	0.001	0.001	0.001	0.441
SF	0.378	0.097	0.001	1.000	0.097	0.001
DF	0.097	0.014	0.001	0.441	0.001	0.020
RF	0.14	0.118	0.001	0.441	1.000	0.001
INV/C	1.000	0.001	0.001	0.014	0.001	0.441
NAT	0.001	0.014	0.076	0.014	0.001	0.009

).

Figure 10 shows the PQ results, where the RBNLIA mode received the highest scores for all PQ categories, including CFs, SFs, DFs, RFs, INV/C, and NAT. The Piecewise mode yielded the second-most positive outcomes, after the object translation mode, while the normal mode yielded the least positive results. These results indicate that non-isomorphic 3DMTs that factor upper-limb VR had a higher presence level than those that do not. In spite of the visual manipulation of the virtual hand in the VE, participants believed that the RBNLIA mode produced the strongest level of presence. We believe that these results are the product of RBNLIA slowing the virtual hand’s position utilizing a non-linear increase for finer motor control and interactions for all items within a 90% range.

3.7. Preferred 3DMT and Perceived Fatigue

After completing the GEQ and PQ surveys, the participants were asked to identify their preferred 3DMT. The object translation mode was preferred by 40% of participants, followed by RBNLIA with 32%, Piecewise with 12%, the normal mode with 8%, and, finally, no preference with 8%.

In addition, because the GEQ and PQ questionnaires were delivered in virtual reality, the participants were asked to report their level of fatigue upon completion of the study.The results indicate that 35% of participants reported “Feeling Great!”, 20% reported “feeling good”, 15% reported “feeling okay”, 20% reported “feeling tired”, and 10% reported feeling “Very Tired”.

3.8. Results Summary and Discussion by 3DMT

3.8.1. Normal Mode

Participants missed a total of 29 objects in the normal mode, which was viewed as the most difficult 3DMT, increasing the challenge and anxiety, thus causing the highest tension and having the most negative impact on participants, affecting the overall engagement and presence (see Figure 9). Due to the physical reach nature of the normal mode, the participants found that it felt more natural than the object translation and Piecewise modes (see Figure 10). Additionally, 8% of the participants preferred the normal mode in terms of engagement. The normal mode produced the highest reach percentage per object and the greatest upper-body lean as a result of no offsets or upper-limb customizations.

3.8.2. Object Translation Mode

Forty percent of participants indicated object translation as their preferred interaction option. The participants spent the least amount of time in this mode, completed each object in the shortest amount of time, assessed it to be the least difficult, and experienced the least amount of tension (see Figure 9). In spite of having the quickest completion time, the object translation mode was deemed to be the least challenging, with 15 objects skipped. The time completion results indicated that the participants interacted with objects the quickest in this interaction mode; however, despite having the highest additional reach throughout the entirety of the interactions in the VE, there were nine and eleven more skips than in the Piecewise and RBNLIA modes, respectively. This non-isomorphic customized mode not only supplied the same maximum reach as the Piecewise and RBNLIA modes, but also included an increment to produce a greater interaction range from the participant’s virtual hand. The large number of objects skipped in the object translation mode may have been influenced by the participant’s inability to comprehend the extent of their interaction range. This may also be the result of the absence of a visual input, such as a laser beam or other visual help, to identify the complete reach range.

3.8.3. RBNLIA Mode

Participants using the RBNLIA mode skipped the fewest amount of objects: four. Despite having the second-longest completion time per object, 32% of participants self-reported RBNLIA mode as their preferred form of engagement. The design of the RBNLIA mode, which slows the offset rise, enables greater control for fine-motor interactions. The RBNLIA mode scores from the in-game GEQ suggest that participants felt the most positive effect and the least negative effect (see Figure 9), while the PQ results reveal that people felt the most present in the VE when interacting in this mode (see Figure 10).

3.8.4. Piecewise

The Piecewise 3DMT provided participants with up to 60% of their reach, followed by a virtual hand offset from 60% to 100% of the participant’s reach. The Piecewise mode was self-reported as the most favored mode of interaction by 12% of participants, despite having the longest average completion time per object. We believe that the higher completion time is due to participants learning the technique and adjusting the reach based on their arm flexion and extension.

4. Conclusions

As VR continues to be adopted in non-entertainment scenarios such as education, training, and healthcare, users face usability difficulties due to a high user variability that hinders engagement, presence, and task execution. However, there have been advancements in providing VR customization in areas other than upper-limb ergonomics, such as setting the interpupillary distance, height calibration, adding tunneling vignettes for reducing motion sickness, and adjusting locomotion styles for ease of navigation. Therefore, it is essential to comprehend how upper-limb ergonomics influence presence, engagement, and task execution when integrated into a 3D manipulation technique.

This paper presented the development of a customized upper-limb 3DMT called Piecewise and compared it to existing isomorphic interactions, including normal and object translation (distance grab) modes, as well as RBNLIA, a non-isomorphic 3DMT for picking and placing objects set at various distances in front of the participants using both hands. Our research determined that the two 3DMTs that accounted for upper-limb ergonomics resulted in a greater presence and engagement, indicating that non-isomorphic techniques were more engaging than their isomorphic counterparts when immersing participants in the virtual environment. Despite having a greater number of omitted objects, isomorphic 3DMTs required less time to complete tasks than non-isomorphic modes according to our research. As a consequence of the number of skipped objects, non-isomorphic interactions permitted greater task completion with fewer skipped objects.

Following the analysis of the percentage of reach, both isomorphic and non-isomorphic 3DMTs produced comparable results. However, it is important to note that the participants had greater reach when using the Piecewise and RBNLIA modes for objects on the left side of the virtual table, and greater reach when using the normal, object translation, and RBNLIA modes for objects on the right side of the virtual table. Although the 3DMTs had no effect on the percentage of reach, the results of the upper-body lean revealed that the normal and object translation 3DMTs had a higher percentage than Piecewise and RBNLIA.

Our analysis of presence, engagement, time to completion, percentage of reach, and upper-body lean, as well as participant preferences, suggests that incorporating upper-limb ergonomics into a 3DMT has an effect on the performance of the task. Specifically, when compared to RBNLIA, our proposed Piecewise 3DMT ranked second in terms of preference, had the longest time to completion, the least amount of upper-body lean, the second highest rating in presence, and the most positive scores for flow, sensory, and imaginative immersion, while ranking behind RBNLIA’s negative and positive affect. In terms of difficulty, tension, and proficiency, normal and object translation modes outperformed both non-isomorphic 3DMTs.

This paper contributes to 3DMTs by presenting the effects of factoring upper-limb ergonomics toward ensuring arm motion within the comfort zone when reaching and grasping objects in VR compared to a non-isomorphic (i.e., RBNLIA) and two isomorphic 3DMT models (i.e., normal and object translation). Our findings align with previous research on 3DMTs, further corroborating the importance of motion within ergonomics comfort. Furthermore, our results show how maintaining upper-limb movement with an ergonomic comfort range of motion can impact body posture and presence. Such findings are essential for advancing 3DMTs that can be used for accessible and inclusive immersive experiences. As VR becomes more widely adopted, eliminating one-size-fits-all barriers to immersive technologies is pertinent. Customizations tailored to each user will aid in performing duties and the experience of a greater presence and engagement.

Future Work

Future research will investigate the optimal combination of isomorphic and non-isomorphic 3DMTs for maximizing immersion, presence, and task success. Due to the cross-applications of 3DMT, additional use cases will be investigated in fields such as medicine, engineering, and inclusive design in order to study the effects on psychomotor skills development. Additional qualitative data will be collected from participants to better comprehend their perceptions of the provided amplification at various upper-limb flexion and extension ranges. Lastly, a larger study will be conducted to increase statistical power.