Effects of Orientation and Appearance of a Synchronously Moving Object on Hand Movements

Abstract

Various devices have been developed to enable humans to control remote objects using active hand movements. However, it is still unclear how the visual characteristics of a synchronously moving object influences hand movements. This study investigates the effects of visual appearance and orientation of a hand-controlled object on hand movements using a novel visuomotor task. The motion of a visual image on a monitor reflected the participants’ right hand movements in the forwards-backwards direction, but not in the lateral direction (i.e., the lateral position of the image was fixed). Participants performed continuous goal-directed back and forth movements of the visual image for one minute. The image’s appearance (hand and arrow) and orientation (forward (FW), leftward (LW), and rightward (RW)) were manipulated. Unconscious lateral deviations (i.e., drift movements) of the participant’s hand during the task were evaluated. Regardless of appearance, the leftward and rightward image induced leftward and rightward drift movements, compared to the forward image. However, the modulation sizes were similar using arrow images, but not using hand images. Specifically, anatomically plausible hand images elicited greater drift movements than anatomically implausible images. This suggests that both orientation and appearance of a hand-controlled object influences hand movements according to stimulus-response compatibility and body-representation changes.

1. Introduction

Various devices have been developed to enable humans to control remote objects (e.g., a computer cursor or robotic hand) using active hand movements. In this situation, hand movements influence the motion of an object. However, it is also possible that the visual characteristics of objects affect hand movements, given stimulus response (SR) compatibility [1,2]. SR compatibility is defined as the degree to which the perception of a visual stimulus is compatible with a required action. For example, a previous study reported that when participants are required to respond to a left or right visual stimulus on a monitor by pressing a left or right key, left responses to left stimuli were faster than left responses to right stimuli, whereas right responses were faster to right stimuli than left stimuli [3]. In addition, using the lateralized readiness potential (i.e., side-specific response preparation), measured using electroencephalography, it has been reported that the perceptual processing of a left- or right-pointing arrow stimulus (i.e., implicit directional cue) could automatically and unconsciously elicit the corresponding response activation [4]. Based on the phenomenon of SR compatibility, it is predicted that when individuals move a remote object such as an arrow cursor toward a specific location using voluntary hand movements, an object in an orientation (i.e., implicit directional cue) that is incompatible with the hand movement direction (e.g., the right-pointing arrow cursor during a leftward hand movement) may influence goal-directed hand movements because the automatic and unconscious response activation elicited by the object’s orientation interrupts the goal action.

Although many previous studies have examined how a cursor moving in a different direction from the visually occluded hand (i.e., artificially perturbed visual feedback) affects motor learning and performance [5], very few studies have evaluated the effects of implicit directional cues of cursor orientation on the performance of hand movements in a situation in which motion directions of the hand and cursor are identical (i.e., no perturbed visual feedback). A small number of studies of human–computer interactions have examined the directional effects of an arrow cursor on cursor-positioning movements [6,7,8]. However, because these studies have produced inconsistent results, SR compatibility between the object’s orientation and object motion (i.e., hand movement) are still unclear. In a cursor-positioning task using a mouse, Phillips, Meehan, and Triggs [6] reported that an arrow cursor presented in an orientation that was compatible with the direction of cursor motion (e.g., rightward cursor motion using the right-pointing arrow) led to decreased performance in cursor-positioning movements. In contrast, Po, Fisher, and Booth [8] reported that arrow cursors oriented toward the lower-right corner of a monitor induced poor performance regardless of the direction of cursor motion. Because these studies focused on the effective use of visual cursors on graphical user interfaces, another approach would be needed to clarify the orientation effects of the moving object on hand actions.

In addition to SR compatibility, it is possible that the visual appearance of the synchronously moving object modulates hand movements. Specifically, a number of previous studies have found that a virtual hand image changes the proprioceptive hand representation (e.g., hand position or configuration) of an unseen actual hand (i.e., visual capture; [9,10]) and consequently modulates the performance of subsequent hand movements [11,12,13]. According to these previous findings, if the orientation of the virtual hand image, which can be moved by active hand movement, conflicts with that of the unseen participant’s hand, the actual hand would move toward the hand image orientation to resolve the visuo-proprioceptive conflict.

The current study developed a novel method using a horizontal planar manipulandum. The participants’ hand position in the forward-backward direction reflected the position of a visual image on a monitor, whereas the hand position in the lateral direction did not (i.e., the lateral position of the image was fixed). Participants were instructed to continuously move the visual image between two targets backward and forward using active hand movements for one minute. The appearance (arrow or hand) and orientation (forward (FW), leftward (LW), or rightward(RW)) of the visual image were manipulated. It was hypothesized that if the direction of the required hand actions (i.e., back and forth movements) was incompatible with that of the visual image (i.e., rightward or leftward image), the hand position in the lateral direction would drift toward the directional cue of the image during the task. This phenomenon is referred to as a drift movement hereafter.

2. Materials and Methods

2.1. Participants

Thirty-two physically and mentally healthy participants (8 men and 24 women; mean age ± SD; 25.7 ± 9.0 years) participated in this study. All participants were unaware of the purpose of the experiments, and all were strongly right-handed according to the Edinburgh Handedness Inventory, except for two participants [14]. None of the 32 participants had a history of visual, proprioceptive or neuromuscular disease according to a self-report questionnaire. This study was approved by the institutional review board of Kyorin University School of Medicine. All participants provided written informed consent in accordance with the Helsinki Declaration. Personal data were protected properly according to the ethical guidelines of medical and health research involving human subjects in Japan.

2.2. Apparatus

Participants sat in a chair in front of an apparatus for measuring the participant’s right hand position. The apparatus was a horizontal planar manipulandum with a two-joint aluminum mechanical arm (MP-201P, Uchida Electronics Co., Tokyo, Japan), in which two potentiometers were embedded (Figure 1a,b) [13,15,16]. A support plate (29 × 11 cm) was attached at one end of the mechanical arm, and its accurate 2D position could be determined based on output voltage signals from the potentiometers, which were digitally recorded at 100 Hz with a 14-bit A/D converter (USB-6009, National Instruments Co., Austin, TX, USA). A screw hole was located on the support plate at the distal end of the mechanical arm. Seated participants placed their middle fingertip of the right hand directly above the screw hole of the support plate, and their hand and arm were then fixed in place using a Velcro strap to prevent undesirable hand movements on the plate. Therefore, the tip of the participant’s middle finger always corresponded to the distal end of the mechanical arm during hand movement. Low-friction joints of the mechanical arm allowed the participants to move the support plate (i.e., right hand) smoothly in the 2D workspace relatively without resistance. A 46-inch monitor (LB-T461, SHARP, Osaka, Japan) supported by a custom-built aluminum frame (not shown in Figure 1b) was positioned 60 cm above the horizontal planar workspace with the screen facing down. A half-silvered mirror (45 × 60 cm) was placed horizontally 30 cm above the workspace so that visual images on the monitor were displayed on the workspace (Figure 1b). This setup prevented a direct view of the participant’s hand. Positional data of the right middle fingertip were acquired using two laboratory computers: one for analyzing data offline and the other for controlling visual images online. The delivery of visual images and experimental timing were controlled using Presentation version 16.0 (Neurobehavioral Systems Inc., Berkeley, CA, USA). The inherent delay was approximately 90 ms, which is below the threshold for detecting visual feedback delay [17].

2.3. Procedure

All participants performed six experimental sessions under different experimental conditions, wherein a visual image was manipulated. The condition was pseudo-randomly selected from a two-by-three factorial design: one factor was appearance (hand and arrow) and the other factor was orientation (rightward, forward, and leftward) (Figure 1c). Each session consisted of one practice trial and one test trial. The practice trial began with the presentation of a white dot (controllable object; 1.0 cm in diameter; not shown in Figure 1) and two vertically aligned green circles (2.0 cm in diameter) (targets; 6 cm in distance). Participants were able to move the white dot in any direction using active right hand movements. The white dot was always displayed on the left side of the right hand so that the distance between the dot and middle fingertip was 23 cm. After holding the white dot in the bottom green circle for 3 s, the targets changed color from green to red (i.e., start signal). The participant was then instructed to move the white dot back and forth between two targets continuously using back and forth hand movements. To keep movement time as constant as possible, periodic pure tones (500 Hz in frequency) were delivered once every 1.5 s during the practice trial (one cycle: 3.0 s). Participants were instructed to move the white dot to the target at the time of the tone delivery as accurately as possible. After 15 s, the color of the targets changed from red to green, signaling the completion of the practice trial, and the participant returned the white dot to the bottom target.

The test trial began immediately after the completion of the practice trial. After holding the white dot within the bottom target for 3 s, the white dot was changed to a visual image selected from a combination of appearances and orientations according to the experimental conditions (Figure 1c). The middle fingertip of the hand or the arrowhead corresponded to the bottom target, regardless of orientation. Immediately, the target color changed from green to red (start signal). Unlike the practice trial, the lateral movement of the visual image was fixed during the test trial so that only the participant’s hand movements in the front-back direction reflected visual image motion. As a result, the hand image or the arrow image moved straight between targets. Participants were asked to continuously move the hand image (i.e., fingertip) or arrow image (i.e., arrowhead) back and forth between the targets as accurately as possible (Figure 1d, left). In addition, they were instructed that the visual image moved only in the forward-backward direction during the test trial. Similar to the practice trial, periodic tones were also delivered. The targets were always displayed on the visual image (i.e., the visual image did not conceal the targets). After 60 s, the targets changed from red to green (completion signal) and participants returned the visual image to the bottom target (Figure 1d, right). Participants rested for 3 min between the experimental sessions. The experiment was begun at a convenient time (from 10 a.m. to 4 p.m.) for participants, but commencement was avoided within 1.5 h after eating. The experiment took approximately 30 min to complete.

2.4. Dependent Variables and Statistical Analysis

Two measurements were extracted from hand position data for 60 s in the test trial to capture time courses and the final results of hand deviation in the lateral direction from the start position (i.e., drift movement). For the first measurement, hand position data from 0 to 50 s were segmented in phases of 10-s durations (phase 1: 0–10 s; phase 2: 10–20 s; phase 3: 20–30 s; phase 4: 30–40 s; phase 5: 40–50 s). The positional data were then averaged within each phase. The latter measurement was drift movement from the start to the end position (Figure 1d, right).

Positive values were defined as leftward drift movements, and negative values were defined as rightward drift movements. All statistical analyses were performed using R statistical software, version 3.2.3 (R Foundation for Statistical Computing, Vienna, Austria), and the level of probability considered to indicate statistical significance was p < 0.05.

3. Results

The upper panels in Figure 2 indicate the time course of drift movements for the arrow (a) and hand images (b). For each image, a two-way analysis of variance (ANOVA) was used to examine drift movements, with orientation and phase as within-participant factors. For both the arrow and hand image, the results demonstrated the main effects of orientation (arrow: F (2, 62) = 13.0, p < 0.001, η²_p = 0.30; hand: F (2, 62) = 27.5, p < 0.001, η²_p = 0.47) and phase (arrow: F (4, 124) = 74.9, p < 0.001, η²_p = 0.71; hand: F (4, 124) = 69.0, p < 0.001, η²_p = 0.69) as well as an interaction between the two (arrow: F (8, 248) = 11.2, p < 0.001, η²_p = 0.26; hand: F (8, 248) = 25.7, p < 0.001, η²_p = 0.45). Post-hoc analyses showed that the participant’s right hand gradually deviated to the left from phase 1 to 5 (i.e., drift movement), regardless of orientation (ps < 0.05; Fisher’s least significant difference test; see asterisks). However, the orientation strongly influenced the amount of drift movement; the leftward and rightward (i.e., incompatible conditions; see squares and triangles) images induced significantly larger and smaller drift movements, respectively, than the forward images (i.e., compatible condition; see circles) after phase 2 (ps < 0.01; see daggers). In addition, the orientation-dependent modulation was greater in the hand image (effect size of orientation: η²_p = 0.47) than in the arrow image (η²_p = 0.30).

Figure 2c shows the final drift movements of the hand (gray bars) and arrow images (white bars) against three orientations. A two-way ANOVA was used to examine drift movements, using appearance and orientation as within-participant factors. As a result, both main effects were significant (appearance: F (1, 31) = 6.1, p < 0.05, η²_p = 0.16; orientation: F (2, 62) = 29.1, p < 0.001, η²_p = 0.48). In addition, the interaction between the two was also significant (F (2, 62) = 7.2, p < 0.01, η²_p = 0.19). For the hand image, post-hoc analysis confirmed significant differences in all pairs between three orientations (forward vs. leftward: p < 0.001; forward vs. rightward: p < 0.05; leftward vs. rightward: p < 0.001; Bonferroni test). A similar pattern was observed for the arrow image, although the differences in two of the pairs did not reach statistical significance (forward vs. leftward: p = 0.09; forward vs. rightward: p = 0.06; leftward vs. rightward: p < 0.001). These results indicate that, compared with the forward image, the leftward and rightward images induced larger and less drift movements, respectively, regardless of visual appearance. In addition, the differences between the hand and arrow images within each orientation were all significant (rightward: t (31) = 3.2, p < 0.01; forward: t (31) = 2.3, p < 0.05; leftward: t (31) = 2.3, p < 0.05; paired t-test). That is, drift movements were greater for the arrow image than the hand image under the forward (hand vs. arrow: 3.7 ± 3.5 vs. 5.1 ± 4.6 cm; mean ± SD) and rightward orientations (1.0 ± 3.5 vs. 2.9 ± 4.6 cm), whereas the opposite result was observed for the leftward orientation (8.5 ± 3.9 vs. 7.0 ± 4.0 cm). Although mean drift movements showed positive values across all conditions (i.e., leftward drifts), some participants exhibited rightward drift movements (i.e., negative values). Specifically, for the rightward orientation, the number of participants exhibiting rightward drift was significantly larger for the hand image (47%; (15/32)) than for the arrow image (22%; (7/32)) (χ² (1) = 4.4, p < 0.05; chi-squared test). No significant difference was found for the forward (16% vs. 9%; hand vs. arrow) and leftward orientations (3% vs. 3%) (ps > 0.4).

4. Discussion

Using a novel visuomotor task, the current study examined the effects that orientation and visual appearance of a synchronously moving object have on hand movements. Overall, leftward drift movements began in the early phase of the task and continued until the completion of the task. In accordance with SR compatibility, leftward and rightward images (i.e., incompatible conditions) induced leftward and rightward drift movements relative to forward images (i.e., compatible condition). The size of the orientation effect was similar for both the leftward and rightward orientations when using the arrow image, but it was larger for the leftward orientation than for the rightward orientation when using the hand image. Specifically, anatomically plausible hand images (i.e., leftward) induced larger drift movements than anatomically implausible images (i.e., rightward). These findings suggest that drift movements in the current study were closely related to both SR compatibility and body (hand) representation.

Even with the forward arrow and hand images, whose directional cues were compatible with hand movements, participants’ right hands gradually drifted toward the visual image during the task (i.e., leftward drift movements). This result is not surprising, given that the visual images including targets were displayed on the left side (closer to the body’s midline) of the participant’s right hand. Thus, in the absence of visual feedback regarding the actual hand position, it makes sense that the participants’ hands shifted toward the visual image or body’s midline. In fact, it has been shown that hand position drifts when subjects make repetitive movements in the absence of visual feedback due to misalignment between visual and proprioceptive information [18,19,20,21].

As expected, relative to the forward images, the leftward and rightward images elicited leftward (i.e., greater) and rightward (i.e., less) drift movements, regardless of appearance. This result supports the hypothesis that back and forth hand movements are modulated by implicit directional cues of the visual image according to SR compatibility; leftward images activated leftward motor responses, and vice versa. However, several previous studies have investigated the effects of cursor orientation on mouse positioning movements and reported that arrow cursors compatible with the direction of cursor motion were paradoxically detrimental to positioning performance [6,7]. The substantial methodological differences between these previous studies and the current experiment should be considered. In these prior studies, the kinematics and accuracy of discrete and fast positioning movements were estimated, and the significance of the results was discussed in relation to mid-flight movement corrections. In contrast, the current study assessed automatic and unconscious lateral deviations during continuous back-and-forth hand movements.

Regarding final drift movements (Figure 2c), the orientation effect was greater for the hand image than for the arrow image. This finding suggests the possibility that the orientation effects of the hand image resulted from not only SR compatibility, but also changes in hand representation. While the configuration of a participant’s unseen right hand was basically similar to that of the forward hand image during the task (Figure 1e), it differed significantly from the leftward and rightward hand images (i.e., visuo-proprioceptive conflict). According to a previous computational movement neuroscience study [22], when a person’s hand moves toward a specific target, the current hand configuration and position (hand representation) is computed on the basis of visual and proprioceptive information to plan motor commands. Therefore, it is assumed that visual images of the leftward and rightward hands influenced participants’ proprioceptive hand representation in terms of hand configuration, leading to rotation of the actual hand depending on the visual image to resolve the visuo-proprioceptive conflict (i.e., medial and lateral hand rotation for the leftward and rightward hand images, respectively). Consequently, medial and lateral rotations of the actual hand during the task would cause much greater (i.e., leftward) and less (i.e., rightward) drift movements, compared to the forward image condition.

This assumption might be partially supported by the drift movement difference between the leftward and rightward hand images. That is, modulation of the leftward hand image (difference: 4.8 cm, p < 0.001) relative to the forward image was greater than that of the rightward hand image (2.7 cm, p < 0.05). This asymmetrical result appeared to be similar to the anatomical properties of actual right-hand motion; it is easier to position the right hand in the orientation of the leftward hand image, compared with the right-hand image. In contrast, such a drift difference was not observed when using the arrow image; modulation leftward (2.2 cm, p = 0.09) and rightward (1.9 cm, p = 0.06) relative to the forward image was almost the same. A similar phenomenon has also been reported in some previous studies involving the mental rotation of body parts. For example, it was reported that reaction times in left–right judgments of hand images were shorter when the hand image was presented at anatomically plausible orientations compared with anatomically implausible orientations [23,24], suggesting that the mental rotation of a body part stimulus engages motor imagery. Although further investigations (e.g., using the non-dominant left hand) are required, it is inferred that the difference in drift movement between the leftward and rightward hand images in the current study reflects changes in hand representation relating to the anatomical plausibility of the hand image.

Today, virtual reality (VR) and robotic teleoperations are applied to various fields of engineering, medicine, rehabilitation, and education. The present results could have important implications for such practical situations, namely, on the unnatural relationships between an operator’s hand movement and the orientation of a controlled object, as the avatar’s hand or robot hand might depress motor performance. Moreover, such detrimental effects might be much greater if the controlled object resembles a human body part (e.g., hand), given the significant difference between the arrow and hand images in this study. The current findings show that influences of orientation and appearance of the hand-controlled object should be taken into consideration when VR or robotic teleoperations are applied.