Effects of Approach–Avoidance Swiping Interactions on the Valence Estimation Using Tablet AAT

Abstract

Bodily activity may influence subjects’ cognitive processing against embodied cognition. Approaching positive objects and avoiding negative ones facilitate the cognitive processing of emotional information by enhancing valence estimation. The effect may be termed the “Approaching positive and Avoiding negative Compatibility Effect (AACE)”. Implicit approach–avoidance behavior towards stimuli can be measured using the Approach–Avoidance Task (AAT). We recently expanded a touchscreen tablet AAT which seems a more flexible tool for measuring approach–avoidance effects on the valence estimation. In addition, the impact of emotional information on physical behavior might vary depending on the level of arousal. Therefore, we here integrated affective arousal with the AACE to investigate the change of valence estimations of emotional pictures with different (high/low) arousal levels before and after swiping them (toward/away) directly by hand on a touchscreen tablet. Eighty participants evaluated the valence of 40 emotional pictures from the International Affective Picture System (IAPS) twice, first after watching them and second after swiping them, either toward or away from their bodies. As hypothesized, the results are consistent with the AACE, that is, swiping positive pictures toward the body or swiping negative ones away on the touchscreen tablet directly by hand led to a positive change in their valence estimation. Additionally, the change of the valence estimation was significantly enlarged when approaching emotional pictures with higher affective arousal. However, this higher arousal effect was not found when swiping pictures away. We argue that the effect of affective arousal and valence on approach–avoidance behavior seems to be separated. The approaching movement (toward) was more susceptible to the higher arousal of the stimuli, while the avoidance movement (away) was more sensitive to the valence. Furthermore, the touchscreen tablet AAT seems efficient and can reliably measure known approach–avoidance behavior toward cognitive processing testing both in the laboratory and in the field.

1. Introduction

According to the theories of embodied cognition, perception and action in and with our environment influence our cognition and emotional functioning [1,2,3,4]. Bodily activity may influence affective responses [4]. For instance, positive stimuli (e.g., cartoon figures) are typically evaluated more positively if subjects exhibit “smiling” facial expressions, but negative stimuli are more negatively evaluated if subjects exhibit “frowning” expressions [5]. In general, embodiment studies on cognitive processing suggest that when bodily activity (e.g., postures or movements) is congruent with the mental representation of affective information, the processing of said information may be facilitated [6,7]. It has been shown that affective systems support behavioral tendencies wherein approaching positive stimuli (decreasing the distance between the stimuli and oneself), and avoiding negative stimuli (increasing such distance), are actions implemented more fluently than the inverse action patterns [8]. In line with this reasoning, approaching positive objects and avoiding negative ones would facilitate the cognitive processing of emotional information by positively changing the valence estimation [3,8,9,10,11] and increasing response speed [12,13,14]. The effect may be termed the “Approaching positive and Avoiding negative Compatibility Effect (AACE)”. The AACE is related to valence appraisals towards affective stimuli [9]. In the original Approach–Avoidance Task (AAT), Solarz [14] presented participants with positive and negative word cards and asked them to pull some cards toward themselves and to push other cards away from themselves. The findings indicated that participants’ cognitive processing of the affective information could be promoted in the consistent condition (approaching positive or avoiding negative), such as enhancing the valence estimation and shortening the reaction time. The tasks were manipulated through a joystick, which differs greatly from people’s naturalistic behavior. In recent years, other hardware devices such as tablets and computer mice have also been used to make the tests more consistent with real approach–avoidance behavior. They accomplish this by replacing the physical distance change in Solarz’s design with virtual or suggested distance change. In the joystick AAT, for example, distance change is suggested by a zooming effect, while participants pull or push joysticks to approach and avoid stimuli [15,16].

The mobile AAT provides a more natural and effective assessment of approach–avoidance behavior, in terms of pushing and pulling pictures, than a joystick [17,18]. Ikeda [19] examined the approach–avoidance response to three expression types using a tablet device. The avoidance response to the anger expression was confirmed, but the approach response to the happy expression was not confirmed. These results indicate that the use of tablet devices to examine approach–avoidance responses is effective. Against this background, the tablet AAT, which is more flexible, seems a particularly suitable tool to examine approach–avoidance behavior in the valence estimation.

Compared with traditional AAT, touchscreen-based AAT allows direct touch with their hands, which may lead to different effects on the experimental measurement of approach–avoidance behavior. As suggested in the near-hand effects reported by Brucker et al. [20], touching directly valence-laden stimuli on a touchscreen interface (e.g., swiping emotional pictures) could potentially deploy higher visuospatial attention to their content (e.g., positive or negative valence). Supporting this idea, experimental findings have shown that this attentional benefit may influence affective experiences when processing valence-laden stimuli near the hands. For instance, Cervera Torres et al. [3] claimed that manipulating the stimuli directly by hand in the touchscreen AAT paradigm modulates their cognitive processing. However, these studies typically include arm movements which mean, contrasting the naturalistic interaction with a touchscreen in daily life, a greater body gesture. Up to this point, it cannot be clearly stated whether minimum body movements (such as finger movements) may influence subjects’ valence processing against embodied cognition.

On the other hand, studies have shown arousal plays an unneglectable role in physical responses and the cognitive processing of emotional information [21]. It reflects a subjective representation of a physiological response to the stimuli [22]. Mather and Sutherland [23] held that arousal enhanced the perception and memory of obvious stimuli, and the perception of the stimuli facilitated a rapid physical response; thus, different levels of emotional arousal might cast different effects on approach–avoidance behavior. Citron et al. [24] experimented wherein the participants expressed, using keystrokes, approaching or avoiding emotional lexical items. The findings indicated that arousal seems to affect participants’ approach–avoidance tendencies when such tendencies are made explicit by the task. The study by Zsido et al. [25] indicated arousal could significantly modulate the valence effect on a cognitive task. By referring to the studies from Zsido‘s lab, Cao and Liu [26] used a motor priming paradigm mixed with a Go/NoGo task and manipulated the valence (negative, neutral, and positive) and arousal (medium and high) of target stimuli. The results identified the valence estimation is more sensitive to the negative valence stimuli with higher arousal than that with medium arousal level. Based on this reasoning, we can imagine that the stimuli with different levels of arousal may also have different effects on the affective cognition of the subjects’ approach–avoidance behaviors.

According to these findings, we introduced a new mobile AAT that can be loaded as a webpage on a tablet. Participants can naturally approach the valence-laden stimuli by swiping toward themselves or avoid stimuli by swiping away with their fingers without arm movements. They evaluated the valence of 40 emotional pictures from the International Affective Picture System (IAPS) twice, first after watching them and second after swiping them, either toward or away from their bodies. Accordingly, the following hypotheses were derived:

H1:

The touchscreen tablet AAT is effective in measuring approach–avoidance behavior in the appraisal of emotion.

H2:

Minimum body movements (such as finger movements) influence subjects’ valence processing against embodied cognition. This means that the direct swiping movements with the pictures is relevant to influence the subjects’ valence estimation towards the pictures, an AACE is expected. The compatibility state (i.e., swiping positive pictures closer and negative pictures away) results in a relatively more positive change in the valence estimations.

H3:

The valence estimation is more sensitive to the negative valence stimuli with higher affective arousal. The stimuli with different levels of arousal have different effects on the affective cognition of the subjects’ approach–avoidance behaviors.

2. Martials and Methods

2.1. Participants

To estimate the number of participants in each group, a sample size calculation was performed using G*Power 3.1 for repeated measured ANOVA, using a rejection criterion of 0.05 and 0.95 [1-beta] power, and medium effect (f = 0.25); the result showed a minimum of 27 participants were needed to each group (54 participants in all). To obtain robust results, 80 right-handed subjects (Mage = 21.80, SD = 0.83; 47.5% female) were recruited to participate in the experiment. All the subjects were second-year or third-year students from a university in Nanjing, Jiangsu province. They passed the college entrance examination and showed normal intelligence. Their vision or corrected vision were normal, and no physical condition measurably affected their right hands. Written informed consent was obtained from all participants prior to the start of the experiment. Each student was compensated with course credit while finishing the experiment. The experiment was carried out in a quiet laboratory with soft lighting and an appropriate temperature. Before the experiment, the researchers explained the experimental process and related matters in detail to the participants to ensure that they completely understood the experimental procedure. Then, the participants were randomly divided into two groups; one group was asked to swipe pictures toward their bodies while the other swiped away with their dominant right hands on a touchscreen tablet.

2.2. Materials and Apparatus

Twenty positive pictures (e.g., victory/field) and twenty negative pictures (e.g., fire/garbage) with high/low arousal from the International Affective Picture System (IAPS [27]) were used as stimuli. The IAPS ratings on arousal and valence scale are scored such that nine represents a high rating on the dimension (high arousal, high pleasure), and one represents a low rating on each dimension (low arousal, low pleasure). An analysis (shown in

Table 1. Means and standard deviations of valence and arousal scores as a function of valence and arousal.

	Positive Valence		Negative Valence
	Valence Scores	Arousal Scores	Valence Scores	Arousal Scores
High arousal	6.00 (0.23)	5.88 (0.14)	2.59 (0.55)	5.86 (0.09)
Low arousal	5.88 (0.19)	4.28 (0.18)	2.84 (0.18)	4.19 (0.98)

) of the variance of the pictures’ IAPS ratings confirmed significant differences between the two arousal categories (t_positive(18) = −22.57, p_positive < 0.001; t_negative(18) = −39.49, p_negative < 0.001). The stimulus was presented on a 10.5-inch standard touchscreen tablet which was uniform in brightness and contrast. To provide a task that can measure approach–avoidance behavior dynamically, we introduced a new, mobile AAT which can easily be used in the field. The mobile AAT can be loaded as a webpage and runs on a tablet. Stimuli are presented on the tablet screen and participants can naturally approach stimuli by swiping toward themselves and avoid stimuli by swiping away.

2.3. Procedure

The participants were in a natural sitting position with their eyes approximately 25 cm away from the screen. The touchscreen tablet was set widthwise on a table with the long side facing the participants’ position. The study continued to use the experimental paradigm adopted by Cervera Torres et al. (2019) [3] which was carried out in two sessions. First, the participants evaluated the valence of the 40 pictures with a nine-point Likert scale after watching. The second session was 48 h later. After each participant logged in to the test program, the pictures were randomly presented either on the near or far side of the screen from the participants’ perspective, while a target square was presented on the opposite side of the picture indicating the movement endpoint. The participants were randomly divided into two groups. One group swiped the pictures shown on the far side of the tablet toward the near side (located within the white square). The other group performed the opposite, swiping the pictures shown on the near side of the tablet toward the far side. (Figure 1). After the operation, the picture appeared at the center of the screen, with a nine-point Likert scale displayed under it. The participants evaluated the valence of the picture based on their immediate feeling. No limit was imposed on the duration of the estimation; the participants could rest at any time. A blank screen lasting 500 ms was displayed before the next picture was displayed.

2.4. Empirical Analyses

The data analyses were performed using SPSS 21.0. The Shapiro–Wilk test was employed to examine the normality of the data. The value of this test was 0.98, p = 0.18, indicating that the sample of this study had normal distribution. Sphericity assumptions were also met for ANOVAs. We analyzed valence change scores (i.e., valence estimation post-interaction minus baseline estimation) in relation to the between factor swiping movements (toward vs. away) and the within factor valence category (positive vs. negative), arousal level (high vs. low). Main effects and interactions are reported separately, paired with relevant follow-up ANOVAs or t-tests to further investigate significant interactions. Effect sizes are also presented as partial eta squared (ηp2).

3. Results

3.1. Results on AACE

The analyses did not show a significant main effect of swiping movement, F (1,18) = 0.53, p = 0.48. The valence category factor, on the contrary, showed significant effects, F (1,18) = 11.10, p = 0.004. The positive pictures (M = 0.34, SD = 0.73) significantly changed the valence estimation than that of negative pictures (M = −0.05, SD = 0.70). Unsurprisingly, there was a highly significant interaction between swiping movements and valence category, F (1,18) = 18.12, p < 0.001, indicating that the effect of picture valence in the change of the valence estimation depends on swiping movements.

A subsequent simple main effect test indicates a significant main effect of swiping on negative pictures, F (1,36) = 8.21, p = 0.007, ηp2 = 0.19. Swiping away the negative pictures more positively changed their valence estimation (M = 0.25, SD = 0.70) than that of swiping towards (M = −0.35, SD = 0.58). However, the difference between swiping positive pictures toward or away movements did not reach significant, t (38) = 0.49, p = 0.63, d = −0.16. Swiping the positive pictures towards the participants’ bodies (M = 0.55, SD = 0.70) increased the valence estimation more than swiping the negative ones toward (M = −0.35, SD = 0.58). These results are shown in Figure 2.

3.2. Results on the Effect of Arousal

The results of the three-way ANOVA demonstrated significant interaction effects between arousal, valence category, and swiping movements, F (1,18) = 9.26, p = 0.007. A simple main effect test was further performed to determine the conditions that affect valence estimation in approach–avoidance behavior.

A non-significant simple interaction effect of valence and swiping was discovered at the low arousal level, F (1,36) = 1.23, p = 0.27, ηp2 = 0.03. However, a significant simple interaction effect was revealed at the high arousal level, F (1,36) = 27.01, p < 0.001, ηp2 = 0.43. The change of valence estimations significantly enlarged at high arousal levels when swiping emotional pictures towards the body, F (1,36) = 37.78, p < 0.001, ηp2 = 0.51 (Figure 3). The change of valence estimations was significant when approaching stimuli with higher affective arousal. However, there was no significant difference between high and low arousal when swiping the pictures away from their bodies.

4. Discussion

The current research investigated the effects of swiping movement on the valence estimation of emotional pictures with different arousal levels. To do so, an AAT was performed on a touchscreen tablet to evaluate the change of valence estimation before and after swiping. Participants touched the valence-laden pictures and moved them with their right hand either toward or away from their bodies on the tablet. We expected that the tablet AAT is effective in measuring approach–avoidance behavior in the appraisal of emotion. Moreover, referring to previous studies, an effect of higher arousal on AACE was also expected. The results of the current study were mostly in line with our expectations. As predicted, swiping the positive pictures toward or negative ones away from the body on a touchscreen tablet led to a positive change in valence estimation. The results were consistent with the AACE which seems to confirm the effectiveness of the tablet paradigm. However, the results differed at the positive pictures from the findings of Cervera Torres et al. (2019) [3]. We did not find any significant effect on swiping movement toward positive pictures. This may be caused by different levels of bodily engagement. Our study only involved finger movements, which means lower bodily engagement than the arm movements in Cervera Torres’ research. The embodied interaction can be as little as moving hands while sitting or full-body engagement, gestures, and locomotion [28]. In another study focusing on the level of embodiment, Johnson-Glenberg et al. (2016) [29] found that a higher degree of sensorimotor engagement could lead to higher knowledge retention over time. We argue that the degree of embodiment is one of the most important features that can enhance cognitive information processing, such as valence estimation. Still, there are some other interesting findings of this study:

There is a significant three-way interaction effect on valence estimation among valence, arousal, and swiping movements. The results indicated higher arousal mainly takes effect on approaching movement. Specifically, when approaching higher affective arousal stimuli, the positive pictures were evaluated more positively, while the negative ones were evaluated more negatively in contrast. The relationship between bodily activity and the processing of information is thought to be bidirectional and is especially remarkable in the case of emotion [7]. Based on this reasoning, the current result is consistent with the study of Tang et al. (2019) [30], which reported that higher affective arousal plays an important role in individual behavior regulation and execution. However, this higher arousal effect was not found when swiping pictures away. We argue that the effect of arousal and valence seems to be separated. The significant difference in the approaching movement (toward) was mainly shown in the higher arousal index, while the avoidance movement (away) was more susceptible to valence or more sensitive to the valence of stimuli. The higher arousal effect may also be related to a direct touch of the stimuli. Future research might follow up on this in other AAT implementations, for example, virtual reality studies or computer gaming.

Finally, mention should be made of other recently proposed mobile tasks, each different from our version of mobile AAT. For example, Kakoschke and colleagues [31] implemented an AAT on a smartphone. Participants had to lean forward and reach out to distal stimuli and then drag these stimuli towards themselves (i.e., approach), and to grasp proximal stimuli and then drag these stimuli away from themselves (i.e., avoidance) on a touchscreen monitor. Meule et al. [32] developed another version of mobile AAT for the assessment of approach–avoidance tendencies towards palatable food, which is based on arm and hand movement on a touchscreen, thereby mimicking real-life grasping or warding movements. These tasks were flexible and naturalistic; however, leaning smartphone, arm movements, grasping or warding movements all changed the subject’s body posture. According to the theory of embodied cognition, the individual’s body posture and movement state will have a major impact on their emotional state and perception [2,6,33]. To directly address the role of arousal against the background of approach–avoidance behaviors on valence estimations, we limited the body movements of the subjects to minimum movements, allowing the subjects to change the visual distance to the stimuli by their fingers, which is more consistent with the natural approach–avoidance movement when browsing network information.

Future research should include a greater emphasis on the approach–avoidance effects based on force. For example, the force on the touchscreen may vary from attack (forward) or defense (backward) when playing roleplay games on the tablet under different emotion types. The measurement of force could help us further understand the motivational aspects of approach–avoidance behavior [34]. Meanwhile, studies have shown that the approach–avoidance response can be reversed in people with high-trait angry personalities [35]. Thus, we can imagine that the player’s personality may also be an important factor in testing the potential approach–avoidance effect. In addition, considering most users usually access digital information through touchscreen devices (such as smartphones and tablets) in a head-lowered posture, we can reasonably guess that such head-lowering may influence emotional cognition by swiping movements in a state of motion. Whether the findings reported by this study can be directly extended to such a state remains uncertain, and further investigation is required.

5. Conclusions

In conclusion, the main purpose of this study was to test the tablet AAT effectiveness to detect established approach–avoidance effects and the effect of higher affective arousal towards cognitive processing. We tested the touchscreen tablet version of the AAT by swiping away or toward the valence-laden pictures from the IAPS. The findings of the present study showed the sensitivity of the touchscreen tablet AAT to measure an approach–avoidance behavior to valence-laden pictures. Moreover, our experimental design allows for addressing outcome measures that are more closely connected to real-life application scenarios than in lab research on approach–avoidance effects, thereby providing an initial step in extending the existing literature on approach–avoidance effects towards a more applied perspective.

The overall result in our studies supports a viewpoint from embodied cognition and in extension AACE fields, postulating that the interaction with an interface may affect the estimation of represented information. According to the reported studies, it appears that swiping valence-laden pictures directly with hands on a touchscreen tablet may influence the subject’s affectivity, as reflected in valence estimations towards the pictures. This influence may be enlarged in the approaching movement with a higher affective arousal. There seems to be a separation between the effect of valence and arousal.

Follow-up research should also aim to address other limitations of the current study. Firstly, as a limitation, we should note that the sample consisted of young college students only. Meanwhile, previous studies have shown that older adults operate with a model of embodied cognition that is differently weighted than younger adults [36]. However, we cannot be sure that the observed effects are indeed specific to approach–avoidance behavior. Secondly, we manipulated task load using a tablet, but future research could manipulate load in other ways, for instance, smartphones or virtual reality.

These limitations and problems notwithstanding, we confirmed that neither valance nor arousal is sufficient to account for the effects of emotionally charged stimuli on cognitive performance; both must be taken into account. Furthermore, we conducted a tablet AAT and conclude that the paradigm could be used in future behavioral-related cognitive processing testing.