**1. Introduction**

The rise of tracking technologies has started to foster international collaborations that tackle the design of technologies for emotional awareness and regulation to support wellbeing and affective health. In fact, mental health research is trying to catch up with the affordances that ubiquitous technologies, wearable devices, and tracking systems offer in general, albeit not without challenges [1–3]. These can be addressed through interdisciplinary research bridging the gap between the fields of Human-Computer Interaction (HCI), Biosensing research, and Clinical psychology [4–6]. Projects such as AffecTech [6,7], have explored the development of digital platforms that position bodily affective awareness and engagements centrally, drawing on *somaesthetic design* [8]. Somaesthetic, or soma design for short, is grounded in somatic experiences, letting designers examine and improve on connections between sensation, feeling, emotion, subjective understanding and values. The soma design framework offers a coherent theoretical basis starting from the constitution and morphology of our human body and perception [8–10]. In particular, soma design emphasizes our ability to change and improve our aesthetic appreciation skills and perception. Other interaction design works within affecting computing exemplify ways to draw upon the body and its embodied metaphors [11,12]. This research builds on the growing HCI interest in affective technologies, whose ethical underpinnings could benefit from more consideration [13,14], addressing issues such as the pluralism of bodies, data privacy and ownership. The body has for a long time inspired emotion research across disciplines [15–18], as relevant connections exist between the body, emotion and movement, its interpretation, enacting and processing. Moreover, research studies point at the links between emotion and physical activity, for example, dance, exercise, movement, or paying attention to our body senses while immersed in nature [19–21]. Engagement with and through the body might therefore be a fruitful path to explore. There is room for an affective computing that does not look at the body as "an instrument or object for the mind, passively receiving sign and signals, but not actively being part of producing them"—as phrased by Höök when referring to dominant paradigms in commercial sports applications [22]. However, how to best address bodily movement and engagemen<sup>t</sup> beyond measuring cues and signals is unclear. Most studies in affective computing revolve around affect recognition from emotion detection and bodily data classification [23]. We take somaesthetic design—a design stance that draws upon the felt body and takes inspiration from experiencing it—and then combine it with the innovative integration of biosensors and actuators. The disruptive somaesthetic view, moves away from the idea of monitoring the body for the sake of bad habit reduction in pursuit of a healthy and long life [24]. Soma design, rather, lets us ge<sup>t</sup> attuned to our bodies and use sensations as a valuable resource instead of something to be improved to meet performance standards.

In this context, we present novel research on embodied interaction design couplings, that is, sensing-actuation combinations of aesthetically evocative body input-output modalities that render biodata shareable, body-centered, highly tangible or even able to be experienced collectively. The biosignals we address have become standard for physiological data tracking research, and are present in the low-cost BITalino biosensing platform [25]. Choosing them for the overview presented and further exploration is motivated by these two aspects, that is, standard and low-cost. The contribution of this paper is an approach to designing sensing-actuation orchestrations, that is, the ways in which body input-output systems and meanings are put in place, coupled, coordinated, customized, sequenced and exposed so that the underlying mechanisms can be better understood, challenged or extended—in other words, sketching, in hardware and software, tangible experiences that allow designers to design and improve overall orchestrated experiences addressing affective health. To make this design approach viable, we combine:


The individual components that take part in an interaction that integrates different body inputs and outputs must implement ways to communicate information, process and represent it, trigger events turning actions on/off and enabling interaction decisions. An orchestration of the protocols and interfaces involved could be beneficial for the design exploration or even for the introduction of use case scenarios, for example, closer to the actual psychotherapeutic practice [2,3,26]. In the discussion, we describe how these elements are shaping the future direction of our research, for example, extending interaction configuration tools with novel sensing-actuation couplings to better explore the design space of affective health technologies and their ethical underpinnings. Using technology for sensing and actuating upon our body, we can ge<sup>t</sup> access to bodily states from our physiology to then act in such a way that we help to alter or reassess our psychophysiological states. This construction process may be developed to extend our knowledge and expectations regarding the internal mechanics of our own body and serves as a bridge to design better informed affective health technologies. Moreover, not only does this approach aim to help having a better self-understanding but paves the way to put the body and its felt experience at the core of the design of such technologies.

The paper is organized as follows. In Section 2, we provide a brief overview of self-monitoring and affective technologies. Section 3 showcases a set of biosignals that we have had access to throughout our research and the information we can extract from their features in order to open a window to our internal psychophysiological processes. The features commonly available for each biosignal are listed to provide guidelines on what level of information is to be extracted. The actuation on the subject's body, addressed in Section 4, can be executed through a variety of mechanisms. We list actuation mechanisms that are available for interaction design using mainly consumer electronics. In Section 5, we present the design research approach that we have adopted, describing what the first-person perspective is and introducing soma design in this context. This design process has been applied to several explorations. The outcomes of our design explorations, coupling biosensing to actuation, are discussed (see Section 6). In this section, together with Section 7, we proceed by addressing coupling concepts and discussing the orchestration process. With the idea of orchestration, we highlight the role of technology-coordinated sequences and the possibilities brought by machine learning and advanced signal processing. We end by commenting on the ethical underpinnings of affective technology and somaesthetic design.

#### **2. Body-Centric Affective Technologies**

With the emergence of everyday personal sensing such as the sensing embedded in our permanently reachable phones, smart watches and fitness bracelets, HCI and ubiquitous computing scholars have highlighted the value of these technologies for innovative research. *Affective Computing* refers to computing that relates to, arises from, or deliberately influences emotions [27]. Technologies that we have seen permeate the *everyday* space with quantification, exercise tracking, and physical wellbeing, have also—perhaps in line with a more traditional affective computing view—made researchers dream of extended healthcare, diagnosis and monitoring applied as well to mental wellbeing [4,28–30]. As exemplified by Bardram and Matic [1], mental health research is catching up. In recent years, research on mobile and wearable technologies that track behavioral, psychological, and contextual signals has gained momentum in the field, albeit not without pending design challenges [31]. Following a research path toward ubiquitous technologies deployed in mental wellbeing domains may help to bring attention to such aspects as personalization, achieving forms of rapport or engagemen<sup>t</sup> not seen in traditional healthcare. The promise of affective computing is vast. In our view, we could argue that just as self-awareness plays a major role in the motivation of change in rehabilitation therapy, for example, in cardiac rehabilitation [32], psychotherapy could benefit from self-monitoring technologies revealing bodily dynamics. Awareness, for instance, may contribute both to a (re)assessment of emotions and behavioral change that are solid grounds of cognitive behavioral psychotherapy [33,34].

Emotion plays an integral role in design work, and design researchers are not exempt from its ups and downs [35–37]. As affective computing reaches maturity, alternative methods have emerged and reshaped traditional approaches to affect. In an effort to attend to emotions, rather than primarily recognizing them, researchers investigating what is known as the affect through interaction [22] prioritize making emotion available for reflection. In such line of thought, seeking emotion aside from context would not make sense. In this "*affect-through-interaction*" view, the role that emotion has had for a long time in artistic and design endeavors is acknowledged. This is exemplified by the analysis of Boehner et al. [38], later picked up by Howell et al. [39] to defy the role of personal sensing in design, in particular the role of biosensing. That is, by no means, to say that the progress that personal sensing has witnessed under the advent of affective computing should be diminished. Rather, dialogue with artificial intelligence research and attention to more cognitivist-oriented outcomes can strengthen the affect-interaction paradigm. From our standpoint, when designing technology-mediated experiences, we see the affect as a sociocultural, embodied, and interpretative construct. Hence, embarking on the challenge of creating use cases for novel technology that touch upon emotions, we start experiencing the body first (see Section 5). The examples and reflections laid down in this paper, the description of technologies we choose to design with, our AffecTech coupling results, and those we used as inspiration, convey directions in which we believe personal sensing, its mapping to actuation, and designing with the body are successfully integrated. Under the overarching lens of first-person design that provides strong foundations, paying respect to ethics, and "resisting the urge" [35] to engage users, we rediscover (and invite others to do so) technologies that are called upon to extend possibilities within affective interaction.

#### *State of the Art*

In the design space of affective interaction and physiological data, existing research has utilized visual and haptic technologies for affective feedback. *Affective Health* [40], for example, mapped skin conductance data measured from an electrodermal activity sensor (EDA) into a colorful spiral on a mobile phone screen. After using the mobile app for a month, users interpreted the skin conductance data as a tool to manage stress levels, track emotions, monitor personality, and even to change their behaviors. Khut [41] has been a pioneer in the area of designing heart rate based visual and sonic artworks for relaxation, both through a mobile app and large scale projections. HCI researchers have started to utilize alternative materials such as thermochromic ones to visually represent biosensing data. Howell et al. designed Ripple [42], a thermochromic-based shirt that changes colors responding to skin conductance. By using the garmen<sup>t</sup> over a two-day period, wearers were able to reflect on their emotions but they rarely questioned if the display was actually representing their feelings. In Reference [43], Umair et al. mapped skin conductance to haptic changes in addition to using visual thermochromic materials, that is, vibrations, heating, and squeezing effects. The feedback about the body properties measured is worn, felt or placed in contact with the body. The findings of these studies highlight that the material-driven qualities of such visual and haptic body interactions shape people's interpretation of how they identify, attribute, and regulate emotions in everyday life. Haptics have also been used with biosensors to regulate affect, which requires users to adapt their ongoing feelings. *EmotionCheck* [44] and *Doppel* [45] use vibrations simulating a slower heart rate sensation for the users and helped them decrease their anxiety. Recently, Miri et al. [46] used a vibration-based personalized slow-paced breathing pacer on the belly which delivered vibrations in a biphasic pattern for inhalation and exhalation and helped users in reducing anxiety during a stressor. With a research approach that explicitly sets out to design with the body—not as an object to be measured but "understanding the

body as a site of creative thinking and imagination" [47]—, works at the intersection of biosensing, interaction design and affective technologies offer an opportunity to study how to support the design of interactions that make us connect with our bodies [9,48,49].

#### **3. Sensing the Body**

Biosignals are time representations of changes in energy produced in the body. These changes correspond to energy variations of different nature, such as electrical, chemical, mechanical, and thermal (as presented in Table 1). With the turn of the 21st century and the advent of the digital era, the advances in the field of electronic components that spurred the development of computing, instrumentation, and algorithms left their impact on medical and biosignal devices. Biosensing and electrophysiology technologies were greatly improved, ready for the study of body functions and health monitoring in the context of clinical research. As technologies grew, the miniaturization and reduction of costs contributed to the growth of biosensing monitoring technologies beyond clinical settings as well. Physiology signals and sources of tracking information are more available than ever, ranging from electromyography (EMG), electrocardiography (ECG), electroencephalography (EEG), electrodermal activity (EDA) to electrooculography (EOG) or eye movement tracking.

**Table 1.** Parameters and type of energy measured through body sensing. Adapted from Reference [50].


A direct consequence of such rapid expansion is the creation of the sports & health monitoring markets that fill up the mobile app stores and provide remarkable revenues in the ubiquitous computing paradigm that we live in. The democratization of the study of biosignals, however, comes with interesting possibilities such as a better understanding of the self and a richer, unprecedented way to interact with technologies that accompany us. This yields an opportunity to define alternative ways to live an affectively healthy life.

As the maturity of open access physiology databases [51] backs up the improvement of processing algorithms, low-cost hardware platforms help populate the open source space [52] where users embrace biosensing, share ideas and drive the future of biosignals applied in different areas. Furthermore, the biosignals that were once limited to hospitals and clinics, or in specialized research labs, addressed in classical texts of physiology, are nowadays accessible in virtually any context by means of wearable technologies. In the review of Heikenfeld et al. [53], an interesting account of the transition from lab tracking to wearables during the 20th century is offered along an in-depth overview of body sensing mechanisms not only restricted to electrophysiology. The field of affective computing has consistently found in biosignals a relevant source of information [54]. Besides, the fact that biosensing platforms have jumped off the clinic has contributed to embracing them alongside other technologies like movement tracking, traditionally linked to behavioral and psychophysiology labs.

We present a selection of studied biosignals (see Figure 1) that can be incorporated into the creation of new technologies for affective health. We focus on a subset of biosignals present in the BITalino revolution do-it-yourself (DIY) low-cost biosensing platform [25,55,56] that backed and inspired some of our research in affective technologies. These, although not an exhaustive list, are to some extent physiological signals that have become standard for physiology tracking research—slowly crossing disciplines and making their way into affective health tracking, interaction design, and other domains of interest. Moreover, with objectives that range from out-of-the-lab psychophysiology tracking [57–59] to new perspectives in interaction design [43,49,60] our work has often addressed biosignals through other available biosignal research platforms beyond BITalino, such as biosignalsplux [61], Empatica

E4 [62], Arduino accessories like the Grove GSR [63], or even commercial wearables such as the Samsung Gear S2 [64] among others.

**Figure 1.** Visual representation of different biosignals: (**a**) Electromyography (EMG), (**b**) Electrodermal activity (EDA) and (**c**) Respiration signals. biosignals and icons obtained at PLUX S.A.

In this section we present a collection of these biosignals and offer a systematic but brief description on (1) How it works, summarizing the basic physiological principles that provide the biosignals energy observables; (2) What can be extracted from the collected biosignal; (3) Where the biosignal can typically be collected in the human body; (4) When, or how often, the signal should be sampled describing the concerns on the timing of the acquisition and in particular the typical sampling frequency of each biosignal; and (5) Limitations of the biosignal acquisition and processing with the challenges of noise or signal artifacts. All of them are examples of signals that we have addressed in our research. This non-exhaustive selection offers a good starting point for researchers interested to integrate biosignals in their design of technologies for wellbeing and mental health.

#### *3.1. Surface Electromyography (sEMG)*

**How it works:** The recording of the electrical activity produced by skeletal muscles receives the name of electromyography (EMG). Human muscles are made up of groups of muscle units that, when stimulated electrically by a neural signal, produce a contraction. The recording of the electrical activity of the muscles (voltage along time), traditionally relying on intrusive needle electrodes (intramuscular), is easily accessible nowadays by means of surface electrodes that capture the potentials of the fibers they lay upon. The result of this measurement is a complex surface electromyography signal (sEMG) that reveals data about movement and biomechanics of the contracted muscles (see Figure 1a).

**What:** Electromyography signals inform about the contraction of specific muscles and parts of the body. The EMG signal consists in the time representation of rapid voltage oscillations. Its amplitude range is approximately 5 mV. In terms of signal analysis, the EMG allows the assessment of several aspects such as muscle contraction duration, the specific timing at which movements or contractions are activated, the presence of muscular tension or fatigue, and the extent to which different fibers (area) are contracted. The analysis is conducted through noise filtering, together with feature extraction that yields contraction onset detection, the estimation of signal envelopes, and the computation of average frequencies. This lets subjects deepen their understanding of movement strategies, very relevant for embodied art and sports performance, improve muscle coordination, or even reveal existing movement patterns that they are unaware of.

**Features:** Onset instants; Max amplitude; Instant of maximum amplitude; Activation energy; Envelope.

**Where:** Having become the standard in EMG monitoring, bipolar surface electrodes consist of three electrodes. Two of them (+/ −) must be placed close to each other, on the skin that lies on top of the muscle under study, along the fibers' direction, while the third one is placed in a bony area where no muscular activity is present. This allows the measurement of electrical potential differences with respect to a common reference, yielding a unique signal that represents the muscular activity of the area.

**When/Frequency:** Given the fast muscle-neural activation nature of EMG signals and the presence of different active muscles contributing to the same signal, muscle activity must be acquired at sampling rates no lower than 200 Hz frequencies. Working at 500 Hz is desirable, while a sampling rate of 1000 Hz guarantees the tracking of all the relevant events at a muscular level.

**Limitations:** Surface EMGs are intrinsically limited to the access to superficial muscles. This is compromised by the depth of the subcutaneous tissue at the site of the recording which depends on the weight of the subject, and cannot unequivocally discriminate between the discharges of adjacent muscles. Proper grounding (reference electrode attached to a bony inactive muscular region) is paramount to obtain reliable measurements. Motion artifacts and muscular crosstalk compromise the assessment of the muscle activity under study. In this context, interference from cardiovascular activity is not uncommon, particularly in areas such as chest and abdomen. The presence of power supplies and mains (powerline) in the vicinity poses the risk of 50 Hz–60 Hz interference.

#### *3.2. Electrodermal Activity (EDA)*

**How it works:** Electrodermal activity (EDA), also known as galvanic skin response (GSR), measures the electrical properties of the skin, linked to the activation of the autonomic nervous system (or more precisely the sympathetic nervous system). By applying a weak current upon two electrodes attached to the skin, it is possible to measure the variations of voltage that are present between the measuring points (see Figure 1b). When placed at specific locations on the skin, the measured electrical signals are affected by the sweat secreted by the glands that are found in the dermis.

**What:** Electrodermal activity signals inform about the activity of the sympathetic nervous system. Given its electrolyte composition, the sweat secreted by sweat glands has an impact on the electrical properties of the skin. This phenomenon, visibly monitored in voltage signals by means of electrical conductance (or impedance/resistance, conversely), facilitates the assessment of arousal effects. Arousal is the physiological response that stimuli such as emotional or cognitive stressors trigger. The measurement of electrodermal activity is usually decomposed in two major behaviors present and superposed in any skin response signal, that is, the skin conductance (tonic) level, with slowly varying dynamics, and the skin conductance (phasic) responses, that exhibit relatively faster dynamics. In terms of signal analysis, this decomposition is accompanied by the assessment of characteristics such as the rate of detected EDA events, detection of onsets, and the characteristic rise and recovery times.

**Features:** Onset instant; Skin Conductance Response (SCR) rise time; SCR 50% recovery time; Event rate; Skin Conductance Level (SCL).

**Where:** EDA measurements use two electrodes to monitor changes in electric potential between two locations on the skin. Electrodes must be placed a few centimeters apart for differences to be relevant. The nature of the measurement technique and the phenomenon itself, makes hand palm a suitable electrode location, for which either palm placement or finger phalanges, most subject to skin sweating, are optimal for the monitoring of electrodermal activity. Additionally, foot sole placement, also affected by sweating glands, is not uncommon in EDA measurements given that particular use cases or settings require access to hands for carrying out certain activities. For the alternative

placements of the EDA sensors, such as forehead or wrist, the presence (or lack) of sweating glands remains a decisive factor in obtaining reliable measurements.

**When/Frequency:** Electrodermal activity is considered to be a slow physiological signal. Thus, sampling rate frequencies as low as 10 Hz allow a full representation of the skin conductance variations. Electrodermal activity peaks usually occur after few seconds from the exposure to a given stimulus (1–5 s).

**Limitations:** Electrodermal activity measurements use changes in electrical properties of the skin produced by sweating. Since sweating is not only triggered by arousal but also the human thermoregulation system, ambient heat and physical activity monitoring are aspects that limit the capabilities of EDA studies. In common practice, electrodermal sensors are usually prepared to obtain salient data from the most comprehensive userbase, providing relevant (measurable) changes regardless of the wide variety of sweating responses from subject to subject. However, it is not uncommon to find examples of subjects with either too high or too low skin conductance responses that complicate the measurements. Moreover, settings that involve an intense physical activity pose concerns on the electrode attachment and motion interference. The presence of power supplies and mains (power line) in the vicinity of the acquisition systems pose the risk of 50 Hz–60 Hz interference. With regard to feasibility, since traditional electrodermal activity studies rely on hands or feet electrode placement that compromises certain actions, attention needs to be given to the use case and activities that take place while monitoring, on a case by case basis.

#### *3.3. Breathing Activity*

**How it works:** Respiration (or breathing) sensors monitor the inhalation-exhalation cycles of breathing, that is, the process to facilitate the gas exchange that takes place in the lungs. In every breathing cycle, the air is moved into and out of the lungs. A breathing sensor uses either piezoelectric effects on bendable wearable bands or accessories (one of the most predominantly used technologies), respiratory inductance plethysmography on wired respiration bands around the thorax, microphonics on the nose/mouth airflow, plethysmographs (measuring air inflow) or radiofrequency, image and ultrasonic approaches. A review on breathing monitoring mechanisms is found in Reference [65]. For piezoelectric breathing sensors, thoracic or abdominal displacements (strain) produced in breathing cycles bend a contact surface that converts ressistive changes to continuous electrical signals (see Figure 1c).

**What:** A breathing signal informs about the respiration dynamics, that is, the dynamics of the process mediating gas exchange in the lungs, as well as supporting sound and speech production. The monitoring of the fundamental function of breathing brings in the assessment of breathing cycles and rates which in turn allows the study of apnoea-related problems (involving breathing interruptions), oxygen intake, metabolism of physical activity, and the effect of cognitive or emotional stressors in breathing. In terms of analysis, breathing cycles are studied using breathing rates, the maximum relative amplitude of the cycle, inhale-exhale volume estimation, inhale-exhale duration, and inspiration depth, that allow the characterization of several breathing patterns.

**Features:** Respiration rate; Inspiration duration; expiration duration; Inspiration-expiration ratio; Inspiration depth.

**Where:** A piezoelectric breathing sensor is usually located on the thoracic cavity or the belly, using a wearable elastic band. With adjustable strap and fastening mechanisms, the sensor can be placed slightly on one side where bending is most relevant, optimizing the use of the sensor range. These kinds of sensors, allow both the study of thoracic and abdominal breathing. With the development of conductive fabric, breathing sensors are making its way into the smart garmen<sup>t</sup> market in the form of T-shirts and underwear bands.

**When/Frequency:** Breathing is a relatively slow biosignal, with breathing rates often below 20 inhale/exhales per minute. A sampling rate frequency as low as 50 Hz is sufficient to capture the dynamics of respiration.

**Limitations:** While piezoelectric breathing sensors are prominent given the low cost and form factor advantages of wearable sensor platforms, deviations in placement have an effect in the relative range of the response signal. Movement artifacts, most relevant when physical activity is present, are a common source of problems. Respiration sensing techniques like the respiratory inductance plethysmography, compensate the highly localized piezoelectric approach with a sensor capturing the general displacement of the whole thoracic cavity, yielding a signal less prone to movement artifacts. The monitoring of breathing cycles is usually accurate, although the exploration of effects to be used as voluntary inputs in interactions, such as holding the breath, are not easily captured.
