Designing for Embodied Being-in-the-World: A Critical Analysis of the Concept of Embodiment in the Design of Hybrids

Abstract

This paper critically explores what it means to Design for Embodied Being-in-the-world (D4EB). It aims to uncover what this perspective means for designing hybrids, the new interactive physical-digital artefacts developed in wearable, tangible and ubiquitous computing and augmented reality. D4EB is contrasted with the principle of embodied representation, applied for example in designing tangible interfaces between users and digital information. In contrast, D4EB starts from our phenomenological ‘being-in-the-world’. Hybrids are conceived as participating in socially situated, sensorimotor couplings that govern the way the lived body operates in the lifeworld. D4EB rejects conceptual dualisms between the (representational) mind and the (physical) body and between (inner) mind and (outside) world. To illustrate its core principles, three design cases are presented. The cases are part of ongoing design-research that formed the basis for the framework. D4EB is further discussed in relation to personal identity, the role of external representations and the role of the designer. D4EB promises to open up a theoretically informed, largely unexplored design space, which can help designers utilize the full power of hybrid technologies. Hybrids may be designed to support people in their embodied being by sustaining, enriching and generating new ways of attuning to the lifeworld.

1. Introduction

Where does the blind man’s self begin? At the tip of the stick? At the handle of the stick? Or at some point halfway up the stick? These questions are nonsense, because the stick is a pathway … [T]o draw a delimiting line across this pathway is to cut off a part of the systemic circuit, which determines the blind man’s locomotion.

Gregory Bateson, Steps to an Ecology of Mind, [1] (p. 324).

Recent years have seen increased interest in the design of what I call physical-digital hybrids, or hybrids, for short. A hybrid is used here as an umbrella term for the interactive artefacts developed in fields such as Tangible Interaction [2], Ubiquitous Computing [3], Wearable computing [4], Augmented reality [5] and Smart or Intelligent Products [6]. In these young and exciting fields, mixing digital and physical forms has produced new, unconventional ‘things’ that we cannot always readily make sense of. Hybrids, by their very existence, seem to challenge the traditional distinction between digital information, on the one hand, and ‘physical stuff’, on the other. Hybrids are seen as blending digital objects with physical shapes [7]. Digital interaction is said to be woven in seamlessly with the action affordances of physical object forms [8]. Furthermore, hybrids are said to mediate in, and integrate with, social situations [9] or even to become true extensions of the body [10]. Hybrids therefore also seem to challenge traditional distinctions between ‘inner’ and ‘outer’, that is, between the human user and her social and physical environment. Hybrid design invites critical reflection on some of the explanatory concepts we have been using so far for describing human–technology interaction. There is a need for appropriate theory that does justice to hybrids, as hybrids, in order to help designers in getting the most out of this peculiar new design material [9,11,12,13]. In the remainder of the paper the single term ‘hybrid’ is favored over compounds like ‘physical-digital’, ‘tangible computing’ or ‘intelligent product’. However, note that even ‘hybrid’ implies a dualism, as something grown out of two different kinds of materials or species. For the moment, ‘hybrid’ will be retained for lack of a better alternative.

In previous years, we have conducted theoretically informed design research to get a grip on hybrids; analysing what hybrids ‘are’, what they ‘do’ in human practices, and perhaps more importantly for design, what they can be and could do. In particular, we explored the prospects of an embodied perspective on hybrids [12,14,15]. Embodied theories developed as an explicit reaction to lingering Cartesian dualisms in the philosophy of mind and cognitive science [16,17]. The promise is therefore that embodiment could very well offer a useful theoretical lens in the attempt to overcome similar dualisms in hybrid design [18,19,20]. This paper builds further on this premise, towards a comprehensive embodied hybrid design framework. A complementary reason for linking embodied theory to hybrid design is that reflecting on the latter may offer practice-based insights to inform the further development of embodied theory itself. In this paper, however, we focus primarily on the consequences of an embodied perspective for design.

There is a history of embodied theorizing in interaction design [9,13,14,21,22,23]. For present purposes, I will only mention a selection of the literature, pruned to what is most relevant for the development of the framework. Already in 1986, Winograd and Flores [24] formulated an alternative view on human–computer interaction grounded in the phenomenology of Martin Heidegger [25] (drawing strongly on Hubert Dreyfus [26,27]). A separate movement towards ‘embodied’ thinking came with Ed Hutchins’ distributed cognition ([28], see also work by Kirsh [29]), who showed how cognitive processes are represented in a distributed system of people and the various tools and artefacts used in human cultural practices. This view came to be quite popular in interaction design [30]. In line with this, Weiser [3] envisioned hybrid spaces with embedded digital technology blending in with the familiarity of the background, which he coined ubiquitous computing. At the same time, rooted in phenomenological sociology and anthropology, Lucy Suchman [31] and others [32,33] built up a career of investigating how people as embodied beings use concrete things in spatial settings as part of situated sense making. These early lines of research culminated in Paul Dourish’s groundbreaking work: where the action is, the foundations of embodied interaction, describing, from a phenomenological point of view, the embodied character of interacting with hybrid devices and spaces [9]. Alongside these developments, Toni Robertson [21] and Dag Svanaes [23] have been particularly important in bringing the phenomenology of Maurice Merleau-Ponty [34] to interaction design. Using an approach of research through design, the Eindhoven Designing Quality in Interaction group has been active for years investigating the value of action–perception coupling principles (in particular, Gibson’s perception theory [35]) and phenomenology for design [6,36,37,38]. Finally, a line of design-based research conducted by Lian Loke, Thecla Schiphorst and others developed into a strong movement-based vision building on body practices such as dance, somaesthetics and Feldenkrais [13,39,40].

As a result, embodiment has become a popular term, used, for example, in conference papers on hybrids [41,42]. However, there remain different interpretations of what is meant by the term. As I will show, it matters which interpretation one favours. In particular, as argued in [7], several design perspectives offer a representational, information processing account of embodiment. Embodied representation may yield powerful, user-friendly designs in various application domains (see, for example, [43,44]). The question explored in this paper, however, is about how to design explicitly for our ‘embodied being’ in the phenomenological sense, i.e., how hybrids may be designed to become part of, and help to sustain, the lived body in its lifeworld. While embodied representation is relatively well known in interaction design, designers may have less developed intuitions as to what it would mean to design for ‘embodied being’. To make way for this latter perspective, we will (temporarily) move away from the concepts of representation and information processing. In a stepwise fashion, a framework will be developed focused on embodied being-in-the-world, and it will explored why it may be worthwhile pursuing this particular perspective.

1.1. Looking at Embodiment: A Hybrid Example

To get a feel for the different ways in which ‘embodiment’ may be understood, consider this wearable ‘activity tracker’ (Figure 1). Imagine a runner in the woods, tracker on her wrist, occasionally glancing at the display to inspect her heart rate.

One interpretation of ‘embodiment’ could be simply to say that the device is worn on the body. A second interpretation would be to emphasize that the device measures bodily signals. A third interpretation would highlight that the device is moving with the body, and, since the body is located and oriented in a certain way, this creates a local context for the display. If designed right, the device, together with the context, and the knowledge of the user, could become a distributed system of information relevant to the task at hand.

For the phenomenologist Merleau-Ponty, however, embodiment refers to something else altogether. Following his reasoning, we could argue that, in some sense, using the device actually takes us out of our ‘embodied being’, at least initially. Consider how the runner, immersed in the activity, is ‘coping’, in Hubert Dreyfus’ terms, in habitual, improvised ways, with the situation [47]. She responds unreflectively and skilfully to affordances popping up in her sensory field, as the running track dynamically unfolds before her, in tight correlation with her own actions. Seen from this perspective, ‘the body’ is not an object in physical space. Instead, we are now concerned with what Merleau-Ponty calls the lived body:

“I move external objects with the aid of my body, which takes hold of them in one place and shifts them to another. But my body itself I move directly, I do not find it at one point of objective space and transfer it to another, … it is already with me” [34] (pp. 107–108). “What prevents [my body] ever being an object […] is that it is that by which there are objects … The body therefore is not one more among external objects, with the peculiarity of always being there”.

(ibid, p. 96)

The activity tracker, meanwhile, takes objective measurements, represented as symbols on a display, to be cognitively interpreted by a ‘mind’. When the runner inspects the screen, she is, if only for the briefest moment, taken out of her embodied being-in-the-world, and asked to stand over and against the world, as a mind. In addition, the information processed about this world concerns the body as a physical object, rather than the runner’s lived body.

For Merleau-Ponty, taking an embodied perspective means to reject any Cartesian reduction of human being into a ‘physical body’ on the one hand and a ‘mind’ on the other. It is the unified lived body, not the physical body or the ‘mind’, which anchors our most basic being-in-the-world [34] (p. 166). The question pursued in this paper is: what would it entail to design explicitly for this being-in-the-world?

Merleau-Ponty, however, also described how artefacts, such as the blind man’s cane, or the typist’s keyboard, may over time become incorporated in our habits [34]. Thus, we may imagine that the activity tracker likewise becomes incorporated into the runner’s practices. We will return to this issue in the discussion. For now, it is important to emphasize that this fact in and of itself does not provide guidance on how to design these artefacts. Does it matter how a thing is designed for the way it becomes incorporated, if at all? The activity tracker has (to the best of my knowledge) not been designed explicitly to become part of ‘being-in-the-world’. Would it have mattered, if it had? Would such a design project have yielded qualitatively different outcomes, perhaps less geared at delivering objective measurements of heart rate, and perhaps more aimed at sustaining the embodied activity of the runner in her world? What could such a hybrid look like?

1.2. Objectives and Paper Overview

Using three design cases to ground the analysis, the main question pursued in this paper is to explore what happens if we design hybrids explicitly to support a person’s most basic embodied being-in-the-world. The paper analyses how an embodied theoretical lens gives a different perspective on what hybrids may do for people than if we would hold on to more traditional, Cartesian conceptualizations. An embodied perspective, it is argued, may help designers to conceive of the hybrid as a unified entity in its own right, which, in turn, may help to make full use of the particular qualities of hybrid materials. It should be clear that no claims are made about whether such an embodied perspective is to be preferred over other design approaches: each design perspective may have its own merits. The present purpose is to show that this is a worthwhile approach, which opens up a new, largely unexplored design space with significant potential for utilizing hybrid technology in a meaningful way. The underlying vision however is human-centred in the sense that a form of hybrid design is envisioned, which takes into account the human being as a whole, situated in its own unique lifeworld.

The remainder of this paper is structured as follows. After the methods in Section 2, the cognitivist legacy in Human–Computer Interaction (HCI) is explained in Section 3, to provide context for the Cartesian dualisms that embodied theories respond to. Next, several views on embodiment in design are reviewed (Section 4, Section 5, Section 6 and Section 7). Representational interpretations, in particular the idea of artefacts figuring in distributed information processing (Section 5), will be contrasted with design approaches focusing on socially situated practices (Section 6), as well as with the perspective of enactive, sensorimotor couplings (Section 7). The concept of lifeworld is elaborated in Section 8. Situated practices, sensorimotor couplings and the lifeworld form the basic ingredients for the framework introduced in Section 9: Designing for Embodied Being in the World (D4EB). In the discussion (Section 10), D4EB is related to identity, the potential role of representation, and a reflection on the role of the designer. The paper ends with general conclusions (Section 11).

2. Method

The D4EB framework is the result of ongoing design-research involving a series of case studies in which hybrids were designed for several actual use contexts with as a guiding theme to design for embodied being-in-the-world. Three of these design cases will be used here to introduce and illustrate core elements of the framework. All projects followed a participatory [48], Research-through-Design methodology [49]. Bottom-up insights from the design explorations and participatory observations within the context-of-use were contrasted with embodied theory in an iterative fashion over periods of six months (in case 2) to several years (in cases 1 and 3). The process involved intermediate reflections on the design process and outcomes with input from design students, end-users, commercial designers, non-profit organizations and technology companies (i.e., the main stakeholders in each case study). The outcome of these reflections formed the core insights that led to the final analysis as presented here. Through ‘confronting’ theory with practice [50], the framework was iteratively adjusted and refined. Within each design case, theory was neither the main driver nor used as a final measure for making decisions: rather, theory played the role of one ‘stakeholder’ amongst others. (In the present paper, of course, theory has the centre stage). For more details about the cases, see: for Case 1: [51], for Case 2: [52] and for Case 3: [15,53].

3. The Cognitivist Legacy and the Persistence of Representational Thinking in Design

Historically, human–computer interaction (HCI) developed in parallel with cognitivism, a neo-Cartesian theory that portrays the mind as essentially software that runs on the hardware that is the brain [54]. For cognitivism, it is the software that really matters. The hardware implements the software functionalities, a philosophical position called functionalism [55]. The ‘peripherals’, i.e., the physical body with its sensors and motor capacities, form the interface by which the mind interacts with the environment. Using the computer as a metaphor, the mind is essentially understood as an inner ‘information process’ dealing with representations of the outside. René Descartes saw the mind as an altogether transcendent, non-physical essence, while cognitivism insists the mental is physically realized [56]. That is, cognitivism does not hold the mental to be non-physical, and yet the theory is dualistic in the sense that the representational world is qualitatively distinct from its physical implementation. This can be understood by seeing that different types of hardware could in principle implement the same software: the mapping between mental content and physical representation is in some sense arbitrary. A related dualism exists in the distinction between the ‘inner’ (mind) and the ‘outer’ (world). Cognitivism typically finds the mind to be ‘inside’, realized in the brain, representing the ‘outside’ world of physical things and other people. The cognitivist dualisms are depicted in Figure 2.

Cognitivism embraces the idea of interaction with the world as a largely sequential process, starting with perception of the environment, followed by internal processing resulting in understanding (revising the internal representation), which then leads to a plan for action, to be executed by the motor system [55]. Cognitivism has been fiercely debated in cognitive science and philosophy of mind, with a growing number of dissatisfied theorists searching for alternatives [16,18,19,57,58].

In traditional HCI [59], the human mind and the digital software of the computer are conceptually speaking on a par. Both consist of physically realized inner representations of the outside world, connecting to world by an interface. When using a computer, information flows back and forth between the user’s mind and the computer’s digital world via a channel provided by the human body and the system interface (see Figure 3).

Since the original work of Newell and Card [59], recent work in HCI has acknowledged the many complexities involved in the way the users interact with and make sense of technology, as, for example, regarding physical context, social setting, attention and affect [60]. Still, the basic idea of HCI as essentially an exchange of representations is persistent. Agre blames this on the error of using the computer’s internal representational processes as a model for all human practices:

“Computers are representational artefacts, and the people who design them often start by constructing representations of the activities that are found in the sites where they will be used … In this sense, computing has been constituted as a kind of imperialism; it aims to reinvent virtually every other site of practice in its own image”.

[61] (p. 131)

In the following sections, we will first review representational perspectives on embodied interaction and then build up our framework of designing for embodied being, moving beyond the representational lens altogether. In the discussion, we take up the topic of representation again (as an aspect of a situated practice), and we consider how it may be reintegrated with the framework.

4. Beyond Cognitivism

Embodied theories developed as a rejection of the cognitivist picture of the mind, in order to overcome its Cartesian dualisms [17]. Embodied theory portrays human sense-making, and the behaviour that goes with it, as a coordination achieved through a self-organizing network of brain, environment, and, as a central actor, the active body [16,57]. This network includes bodily constraints, homeostatic processes and dynamic relations between the active body and the physical and social environment, or lifeworld [16,62]. In Western culture, a large part of this environment consists of designed artefacts and spaces. Increasingly, it figures interactive technology. Seen this way, the question for designers of interactive technologies is how particular design choices impact the dynamics of this embodied and situated self-organizing sense-making network. It is these theories that may support designers in making these choices that we turn to now.

5. Designing Distributed Representations

Don Norman famously showed how people often rely on ‘knowledge-in-the-world’ instead of ‘knowledge-in-the-head’ [63]. Knowledge-in-the-world can be seen as distributed information processing [28,30]. The basic idea here is that cognition, as information processing, gets distributed over the brain and the environment. Hutchins, observing navigation practices on a large navy ship, discovered that externally available resources, either tools designed intentionally for a task, or ad hoc recruited props, may take care of part of the ‘thinking’ for us [28]. Using objects as external representations, we may ‘offload’ part of the cognitive burden onto the environment [29]. Philosopher Andy Clark calls such artefacts ‘cognitive scaffolds’, in reference to Vygotsky’s theory of scaffolded learning [16]. Clark goes on to explain how the physical constraints of our own body may also function as cognitive scaffolds. For example, our bodies have a front and back relative to how we move and the position of our eyes. As a result, we do not need elaborate mental modelling (as many robots would) to keep track of our bodily position in space [16]. We may also actively reorganize the world, or our bodily orientation, with as a result an external representation that makes it easier to solve the task at hand [29]. Clark argues convincingly that the inner mind can become so dependent on cognitive outsourcing to external artefacts that we should grant them the status of being part of the mind: mind as an extended distributed information system [64].

Distributed representation retains the overall idea of cognition as information processing, albeit extending beyond the brain into the world. What is needed for a person to respond appropriately to a situation is the right sort of information process, and the distributed system sorts out what is the least cognitively demanding way to implement it [65]. Designers may create technology that helps to spread the ‘cognitive load’ between inner thinking and external representations provided by technology [30]. We see this explicitly applied in, for example, hybrid surface technology supporting collaborative teamwork [66].

While it helps to overcome a dualism between ‘inner mind’ and ‘outer world’ (the vertical split in Figure 2), distributed representation retains a conceptual split between ‘the representation’ and ‘its physical implementation’ (the horizontal split in Figure 2). To illustrate how this split works out in design, consider, for example, the application of distributed representation in tangible interaction. In its original formulation [2], tangible interaction means to use physical objects as ‘external representations’ of digital content. As Ishii explains: “The key idea … is to give physical forms to digital information. The physical forms serve as both representations and controls for their digital counterparts” [2]. In terms of distributed representation, it means that the interface is not only to channel messages from the human mind to the digital mind, but that part of the information processing and representation takes place in the outside world between computer and user in the form of bodily manipulations of tangible objects and configurations of objects. This means that, in the design of the hybrid, stronger emphasis than before is put on utilizing the opportunities of manipulating physical form through bodily action. At the same time, ‘the physical’ is used essentially to represent the digital, and it is in the end the digital content that matters to the question of what the product ‘is’ and does for the user. The split between the content and the physical realization of a distributed representation remains (Figure 4). Again, this is not to say that distributed representations are ‘non-physical’, rather, their content is to be conceptually distinguished from their physical form.

By contrast, consider this sceptical review of Clark’s extended mind and the idea of external representation, offered by the phenomenologist Hubert Dreyfus who was one of the fiercest critics of representational cognitive science:

“Clark … attempt[s] to free us from the Cartesian idea that the mind is essentially inner by pointing out that in thinking we sometimes make use of external artefacts like pencil, paper, and computers. Unfortunately, this argument for the extended mind preserves the Cartesian assumption that our basic way of relating to the world is by using representations … be they in the mind or in notebooks in the world[.] … A]ll Clark … give[s] us as a supposedly radical new Heideggerian approach to the human way of being in the world is the observation that … thinking bridges the distinction between inner and outer representations”.

[67] (p. 254)

Reasoning with Dreyfus, I argue that designing hybrids as systems of distributed representation does not yet touch upon the phenomenological concept of the embodiment of our being-in-the-world. In what follows, I will take several steps towards this alternative interpretation of embodied design. As a first step, it may be fruitful to point out that external representations can only help us make sense of the world courtesy of their being situated within a certain context of practice [32,68]. This is why we turn first to the theory of social situatedness, showing the way artefacts are used in human sense-making is established in action, within a practice, in which the artefact has a use [31,69].

6. Designing for Social Situatedness

With origins in anthropology and social science, various researchers investigated the way tools become incorporated in socially situated practices [31,33,58]. This line of research stresses the value of concrete circumstances of the social situation. The fact that people’s use of objects is often ‘publicly available’ and ‘collaboratively organized’ are essential:

“… cognitive phenomena have an essential relationship to a publicly available, collaboratively organised world of artefacts and actions, and … the significance of artefacts and actions, and the methods by which their significance is conveyed, have an essential relationship to their particular, concrete circumstances”.

[31] (p. 50)

Where distributed cognition sees people and objects as essentially ‘nodes’ in a distributed representational system, social situatedness explains how objects play their part in what is first and foremost a social activity:

“… the real cornerstone of knowledge is people … [A] distinction [needs to be made] … between the idea that knowledge can be represented and stored and the view that it has to be contextualized and made relevant to the settings in which it has to be applied. Meaning is not inherent to information; information is made meaningful”.

[9] (p. 185)

In other words, Clark’s ‘cognitive scaffolds’ can only exist in the context of a social setting. The meaning of objects is established in the context of social interrelations, roles, norms and culture ([70], especially p. 277). In distributed representation, physical objects are media for ‘storing knowledge in the world’. Social situatedness instead emphasizes first and foremost how artefacts (for example, cardboard ‘flight-strips’, used by air-traffic controllers [33]), function as active components coordinating a social activity. ‘What it takes to be a representation is to be used as a representation in the course of some activity … in systems of practice’ [9] (p. 208). It is important to emphasize the reciprocal relationship at work here. Not only do artefacts acquire meaning courtesy of being situated in social practices; the same artefacts play an active role in the formation and sustaining of said practices. I will illustrate the principle of socially situated practices in hybrid design with the first design case: Floor-It.

Case 1: Designing for socially situated practices.

Floor-It [51] (Figure 5) is a hybrid space, which can be seen as a form of augmented reality, or as a full body-scale interactive surface. It consists of a large, interactive floor projection of photographs, which are brought in by the participants of a so-called ‘creative session’. Each participant is surrounded by his own photographs, in a personal circle that moves along with one’s body in space. Participants may enlarge, reorganize and flip orientation of the pictures in support of the creative conversation, using their feet. The design goal was to support shared sense-making in creative teams.

In twenty, twenty-minute brainstorm sessions, sets of three people with different disciplinary backgrounds were asked to collaboratively create a new communication concept to promote a nature park to elementary school children [51]. The participants first individually brainstormed ideas and then took digital photos of their sketches and sticky-note texts to the interactive space. We compared ten sessions on the floor with ten sessions where pictures were randomly projected as a matrix on a wall.

Qualitative analysis of the video [71] revealed that the pictures on the hybrid floor supported nonverbal interaction, including bodily orientation and gesturing, by which participants would keep the conversation going. For example, pointing to a picture in ones’ own circle would function to first draw attention to oneself and then in a flowing movement, pointing to a related picture of another person, would invite that person to speak next (see Figure 5, right). In other words, participants coordinate their bodily interactions as part of participatory sense-making, as described by De Jaegher and Di Paolo [72]. Participatory sense-making is not so much about exchanging information as it is about working to establish and maintain a dynamic social coupling through ongoing interactions. Rather than being explicitly focused on the pictures as containers of information, participants used the pictures as ‘embodied’ scaffolds through which they maintained contact with each other. We could describe the role of the pictures as what philosopher Heidegger would call ‘zuhanden’ (ready-to-hand): they took the role of implicit, transparent tools used in extension to a person’s gestures and bodily posture in order to maintain the conversation. In the version on the wall, by contrast, the participants would intuitively stand next to each other, facing the wall, and they were focused on the set of pictures, which became the main object of attention. While the instruction did not mention it, participants readily started to group, select, discard and name the items on the wall. In the terminology of Heidegger, we could say that the pictures on the wall were ‘vorhanden’ (present-at-hand): they were objects, not transparent bodily extensions, which participants would explicitly focus on and reason about. The interactive wall, in contrast to the floor, functioned as a point of access to digital information, and the implicit aim of the collaborative activity became to organize this information: creating the best possible configuration of pictures represented the ‘result’ of the creative session.

7. Designing for Sensorimotor Couplings

Floor-It illustrates how external representations do not necessarily have a fixed, predefined meaning, but take on meaning as part of the way they scaffold the dynamic of a socially situated practice. Floor-It also shows that such practices are sustained by ongoing bodily interactions of the individual participants. De Jaegher and di Paolo explicitly build their concept of participatory sense-making on ‘sensorimotor couplings’. Theories of sensorimotor coupling generally focus on how individual agents make sense of the world by skilfully interacting with it [18,19,35,73].

The core idea is that sensorimotor couplings describe the way by which the living body continuously self-organizes into coordinating pattern in response to perturbations. Coordination is established through the formation of couplings between perception (the activity of our senses) and action (the activity of our musculoskeletal, i.e., ‘motor’-system). These couplings lead to behavioural patterns (habits, routines, skills) that fit the given situation [73]. In phenomenological terms, sensorimotor couplings produce an attunement, or ‘grip’ on the world, and this grip is continuously re-established in response to changes in the relation between the lived body and the world [34]. The development of a sensorimotor coupling can be seen as the development of a ‘skill’: a successful way of doing things that is stable enough to pop up when needed [47]. Andy Clark gives a nice example of a baseball outfielder who needs to catch a ball [16]. Instead of calculating first the goal position and running speed to catch the ball, an outfielder simply starts running, meanwhile making sure that the ball maintains a straight horizontal line in his visual field. This is the coupling between action and perception. By continual adjustments of running speed in order to maintain that straight line, the outfielder is guaranteed to be right at the spot where and when to catch the ball. James Gibson’s affordances [35] can be seen as the way the world shows up as affording some action, with a certain sensorimotor coupling in place (see also the section on affordances in Case 2 below). Crucially, artefacts can become part of sensorimotor couplings, as when a craftsman skilfully uses a tool [47], a musician her instrument, or the blind man his cane [34].

Case 2: Designing for sensorimotor couplings.

The concept study F.E.E.L. (Feelings and Experiences for Embodied Learning) illustrates the idea of sensorimotor coupling [52] (See Figure 6). F.E.E.L. was designed to persuade household members to reduce shower time. Instead of simply telling users how much water they are using and reminding them to save water (a cognitive, information approach) or literally prevent them to do so (e.g., shutting off the supply), we decided to intervene in the sensorimotor couplings that underlie the gradual build up towards that familiar moment one just feels one is finished and quits voluntarily. Could we ‘tweak’ the underlying couplings such that people would experience this moment earlier? The concept proposes an interactive floor that physically stimulates the feet in a certain rhythmic ‘massage’. The temporal structure of the sequence is always the same for each shower session, progressing from begin, to the middle, to the end. The system first measures average shower time for each user. After several days, the user will be accustomed to the rhythm and thereby will implicitly associate the temporal choreography with the normal shower routine. After several days, the pattern will start gradually reduce its length over a period of weeks, while retaining the same overall choreography. The idea is that, along with this change, the user will adjust his embodied routine with the effect that the moment one feels one is done arrives sooner, being synced to the moment stimulating pattern. The user will save water, while not being forced, and not having to decide consciously: he will just ‘feel like it’.

Persuasive design typically proceeds to communicate a message to the user—for example, by telling the user how much water has been used. Often the message is implicit, encoded into a more subtle form of ‘nudging’ [74], e.g., by opening of the shower door, or creating a chilly wind in the cubicle, etc. The risk with such strategies could be that a person after awhile becomes annoyed, or bored or simply neglects the cue. Sensorimotor coupling theory would explain this by saying the ‘nudge’ remains an ‘external’ trigger: it has not become part of ones’ embodied attunement to the world. F.E.E.L. illustrates the alternative approach that would help precisely transform this attunement. A sensorimotor coupling would be developed that self-organizes into the embodied experience of ‘the feeling of wanting to quit’.

On Affordances

The term ‘affordance’, coined by James Gibson as part of his theory of direct perception [35], has spurred heated debates in psychology as well as in design [75]. While the debate is not of central concern to this paper, I offer a few thoughts here as it relates to the analysis so far. Don Norman, who contributed greatly to popularizing the term in interaction design [63], has been accused of introducing an incorrect interpretation of the term ‘affordance’ to the field [75,76]. If we frame the critique in terms of the present discussion, we could say that Norman and his followers interpret ‘affordances’ as ‘distributed representations’. Seen as a distributed representation, the affordance remains at the core a kind of ‘message’: the designed form encodes information, which ‘tells’ the user how the product should be used (e.g., whether a knob is for turning or for pushing, and so on). Norman’s interpretation is useful for design (he later renamed it ‘perceived affordance’ [77]), but it is also incompatible with Gibson’s original. Gibson’s intention was precisely to radically do away with representational talk altogether, insisting that affordances are directly perceived, unmediated by any form of encoding [35]. Gibson’s affordances are the natural outcomes of sensorimotor couplings. As aspects of couplings, they are not the designer-defined cues-for-use, encoded in a product shape. Instead, affordances develop from moment-to-moment as part of the skilled ways of perceiving (how the world ‘shows up’) in situations. The question is if and how we can nevertheless design technology specifically in support of the formation of such skilled couplings.

8. The Lifeworld

Sensorimotor couplings and the social setting naturally come together in what we call the lifeworld, a term already used in previous sections. What is this lifeworld? Agre and Horswill describe the lifeworld as:

“an environment described in terms of the customary ways of structuring the activities that take place in it [including] the conventional uses of tools and materials … [Originating in] phenomenological sociology … it refers to the familiar world of everyday life, and specifically that world as described in the terms that make a difference for a given way of life. Cats and people, for example, can be understood as inhabiting the same physical environment but different lifeworlds. Kitchen cupboards, windowsills, and the spaces underneath chairs have different significances for cats and people, as do balls of yarn, upholstery, television sets, and other cats. Similarly, a kitchen affords a different kind of lifeworld to a chef than to a mechanic … A lifeworld, then, is not just a physical environment, but the patterned ways in which a physical environment is functionally meaningful within some activity”.

[62] (p. 111)

In other words, the things, spatial setting and the people and social setting that are part of our skilled practices form our lifeworld: the world within which the body lives. This is not a purely objective physical reality, waiting for a mind to be inspected and cognitively understood: the lifeworld is always already familiar to us. This world is not observed by us but lived; we are in it. [25]. It is the world of pre-reflective experience, in which we already find ourselves before we do any explicit thinking. Phenomenologically speaking, artefacts like hammers and canes mediate the way the world shows up for us [78]. This also means we to some extent create the world that we perceive, through our actions. As Varela put it, the lifeworld is ‘enacted’: “[M]eaning is in the engagements in which an organism builds its world” [57] (p. 358). Margaret Mead similarly argued how the organism, through its actions, co-determines its world:

“The sort of environment that can exist for the organism, then, is one that the organism in some sense determines. If … there will be an increase in the responses of the organism to its environment, … the organism will have a correspondingly larger environment …”.

[79] (p. 245), quoted in [80]

The lifeworld is not purely physical space, and, likewise, it is neither purely a ‘mental’ space: the lifeworld is not the representational world of ‘information’, it is the world in which our lived body operates. In addition, while the lifeworld is socially situated, it is not purely social: in the lifeworld, one may bump into things and hurt oneself. The lifeworld is both physical and social.

Our sensorimotor skills, the concrete environment of artefacts and spaces, and the social practices that we participate in, are all ingredients to the lifeworld. I illustrate the lifeworld with a final design example of a supportive hybrid technology adding to a person’s lifeworld.

Case 3: Supporting the lifeworld

MYDAYLIGHT is an interactive system of wirelessly connected led-light units, designed to support high functioning, semi-independent living persons on the autistic spectrum in managing daily affairs in the home [53] (See Figure 7). Using the system, one can plan activities to do the next day. The system will then provide subtle visual hints at designated times in the form of coloured light. These attention cues are distributed in space in a way that is determined by the person using the system, and the location of the lamp units can be changed during use. In this way, the personal structure in a person’s own home is ‘augmented’ with attention-grabbing elements highlighting the aspects of the room that are relevant for the activity one intended to do. The lights help a person to focus on the task at hand and function as subtle reminders. They specifically do not instruct or explain what to do, or how because this is something that clients are perfectly capable of remembering. One may also first in an improvised manner decide to do something and then ‘tag’ this action by grabbing a new light unit and activating it. Such tags can later on be turned into repeated reminders that return each day. The system supports a person’s own particular ways of managing daily activities and helps to develop personally effective routines.

The system aims to help creating the right sorts of sensorimotor couplings—holding attention where it should be (not being distracted, but also not getting ‘stuck’ in one activity without moving to the next one) and these activities are tied into the physical location and available socio-material structure. This is structure that would typically sustain desired activities but which may trigger undesired responses (e.g., anxiety, stress, distraction, confusion) for persons on the spectrum.

The system also aims to transform couplings with others, e.g., caretakers, family: negotiating how to use the system and where to place the lamps together with care-professionals or family members could empower the person on the spectrum to express his unique ways of dealing with daily affairs, while also aligning these with the interests of others in ways that are ultimately beneficial for all.

One crucial feature in the design is that it aims to help gradually transform the lifeworld. The set of lights create a supporting structure for gradually developing new routines in dealing with the world. At the same time, it invites reorganizing ones’ daily environment around the lights [29] such that the world has an increasingly better fit to ones’ routines. Mediated by the hybrid system, the lifeworld and the routines ideally co-evolve into a desired ‘grip’.

9. Designing for Embodied Being-in-the-World

We are now ready to bring the main ingredients together into a theoretically informed vision of designing for embodied ‘being-in-the-world’ (D4EB). The D4EB framework is illustrated in Figure 8. In D4EB, hybrids are situated ‘in between’ the ‘inner’ person and ‘outer’ environment, and ‘in between’ the representational world of information and purely physical world, or, rather, D4EB rejects these dualisms altogether, replacing them with a picture of human being as characterized by a lived body in a lifeworld. D4EB promotes designing hybrids to take part in both our socially situated practices and, in our sensorimotor couplings, or, taken together, it envisions hybrids to support socio-sensorimotor couplings. Socio-sensorimotor coupling is what characterizes the activity of the lived body in its lifeworld.

D4EB means not making the conceptual distinction between digital functionality and physical form of the hybrid. The form and interactive behaviours of the hybrid artefact operate as a unified whole to support and transform the ways in which the lived body is active, seeking attunement, or grip, in relation to the lifeworld [47]. D4EB also makes no a priori distinction between ‘inner self’ and ‘outer world’. The lived body that characterizes my being is always already a being-in-the-world. D4EB is not about how technology can support me in having information about the world: it is about scaffolding the way I can be in the world.

We may imagine the hybrid either to function, relatively speaking, more as an extension of the lived body, similar to the blind man’s cane, or, again relatively speaking, more as an integrated aspect of the lifeworld, but, as a being-in-the-world, there is no true distinction. Furthermore, we may envision hybrids to create completely new kinds of socio-sensorimotor couplings, which will augment our bodily skills as well as enriching the lifeworld.

10. Discussion

D4EB is a design-theoretical framework that opens up a space for designing hybrids that will enable people to interact with the world in new ways. Its purpose is to help designers to go beyond the conventional dualisms between the digital and the physical and between the ‘inner’ person and the ‘outer world’, by designing instead for embodied being-in-the-world. At present, it is a rough sketch, and further research is needed to fill in much of the details. As a design-research framework, it should itself be open to change, in response to new bottom-up insights derived from projects in which concrete hybrid artefacts are created for real users in real use-contexts, neither should the case studies presented here be seen as final examples of D4EB. The framework and the case studies most of all provide an ‘embodied lens’ that points towards what could be designed if we would exploit more fully the rich potential of hybrids. Designers are invited to use and reflect on the framework and contribute to its further refinement.

In the remaining discussion, I briefly reflect on three issues: D4EB and personal identity, D4EB and representation and, related, learning from reflecting, and D4EB and the role of the designer.

10.1. Empowering a Person’s Being

D4EB portrays hybrid technologies as not just objects ‘out there’, as things apart from our self, but rather as elements within the socio-sensorimotor loops that form our ‘being’. In his book, Situated Cognition [58], Bill Clancey explains how dealing with artefacts as part of the general process of sense-making creates ”a form of feedback between how the world is perceived and how the person conceives his or her identity” (p. 27). Clancey sees a clear relation between how a person learns to make sense of the world and how he learns to make sense of himself: a person’s understanding of ‘What is happening?’ is really ‘What is happening to me now?’ (ibid).

This self-referential aspect is reflected in FLOOR-IT, where we observed how people position themselves in relation to others, using the hybrid floor to do so, as part of the process of collaboratively trying to develop an idea. In concrete terms: Me showing you this picture here, communicates not just the idea, but it also expresses who I am and where I stand in relation to you. Even more strongly, we see this relation in MYDAYLIGHT, where the hybrid system is designed to empower a person who has by nature an ‘atypical’ way of acting in and perceiving the world. MYDAYLIGHT helps such a person to learn to organize the lifeworld in a way that respects the highly diverse and personal ways in which this may be done. However, a person who is discovering ‘what works for him’ is at the same time making sense of ‘who he is’. This self-reflective process may be enhanced when the configuration of the technology becomes part of the negotiations with caregivers and family, i.e., part of a social setting.

Heidegger already discussed how the tools, or equipment of a carpenter not only help do the job, but also establish the carpenter as a carpenter within the social community [25]. Riemer and Johnston [81] point out that much of today’s information technology precisely lacks this quality of equipment:

“Use of equipment is at the heart of our human way of being, which is to be engaged in practices. Practices and equipment are constitutive of self … [I]ndividuals express their (professional) identities through the equipment they use. Replacing this equipment with information technology might in the worst case equate to tearing apart one’s (professional) lifeworld, one’s existence, which was built on the basis of what one does and therefore how (in what way) one ‘is’”.

[81] (p. 10)

All of this means that D4EB contains risk, and raises ethical concerns. Instead of ‘just’ providing people with useful information, or ‘just’ a practical use, hybrids potentially transform, or to put it even stronger, they may become part of who we are: they take part in our being. If done right, this would make design all the more meaningful. D4EB aims to design hybrids that function as equipment to enable people to be (come) most fully who they are. Learning to skilfully use hybrids as part of ones’ being-in-the-world would then align with being an autonomous person in a community of practice. One starting point for design would be to demand that using a hybrid should give rise to the experience of something ‘I do (with the tool being part of who I am)’, rather the experience of something ‘the tool does for me’ or ‘to me’. However, the complexities involved in ensuring users remain in control over this process require substantive further research [15].

10.2. Representational Artefacts: Coming Full Circle

The analysis deliberately turned away from designing objects as external representations. We explored what happened if hybrids would be designed to function not (primarily) as systems of external representation, but instead would tap into the underlying socio-sensorimotor loop characterizing the lived body in its lifeworld. Still, as part of socio-sensorimotor coupling, objects may certainly come to function as representations. This means the idea of external representation can be accommodated within D4EB. Representations in D4EB may form a secondary interactive loop, piggybacking on more basic socio-sensorimotor couplings. This, however, means to revisit the question of what it is exactly that the designer gets to fix ‘on beforehand’, and what meanings will ‘emerge’ in improvised ways from the socio-sensorimotor loop itself (see also below: the role of the designer). In contrast to conventional, designed representations with a predefined meaning, such emergent representational objects may not always bear clear-cut, consistent meanings, as their role as representations is enacted in a particular local setting. For example, consider that it is difficult, if not impossible, to pinpoint with certainty whether, say, the light of a MYDAYLIGHT lamp represents: “You should do the dishes now”, or rather “Do now whatever you wanted to”, or merely “Look here” (… and act on whatever the context affords), or even just: “Hey, you!” Perhaps the only thing we can assess is whether the light produced a particular desired effect in experience or behaviour, regardless of what it represented.

Eventually, a person may start to use more explicit representations (e.g., language) to further ‘fix’ the emerging meanings. This is in fact the process described by Schön as reflective practice [82], whereby a person first engages with the world and then at some point explicitly frames a particularly significant pattern in his experience using a representational scheme. Such frames henceforth bring their constraining influence to the further development of new skilled couplings. In other words, the reflective framing scaffolds the development of new skills [82]. While this is fully compatible with socio-sensorimotor coupling (representational framing building on top of couplings), it is here that the D4EB framework may lose significance to inform design, and other frameworks, such as those of distributed representation [30], or embodied metaphors [83], may become more relevant.

10.3. The Role of the Designer

D4EB invites us to critically rethink the role of the designer with respect to the way hybrids can come to support our lifeworld. From a traditional perspective, the user has to adapt to the rules for interaction defined by the designer. It is also apparent that users are creative in (re)appropriating such artefacts in practices, often changing the meaning and use as originally intended [69]. Designing distributed representations, if done well, already suggests that the tool will be more easily appropriated into embodied practices. In tangible interaction, for example, “body-friendly” interfaces to digital information are created, embedded in the world. However, what the tangible interaction refers to, the content of the representation, is still typically defined by the designer. Thus, while the user’s understanding of how the interface should be used may emerge “in interaction” [8], the meaning of the digital functionality of the product, as a whole, may not. D4EB would instead suggest that hybrid artefacts, as part of our ‘being’, should not so much contain meaning but instead make it possible for people to generate meaning. Hybrids do not contain prefab ‘information that is useful for the user’; they are the sorts of tools with which people create such information in action. They are integrated elements of the activity of a lived body in its lifeworld, which makes sense.

D4EB is not so much about designing embodied interfaces (to existing digital contents). It considers ‘human–world’ interaction more broadly and asks how hybrids could ‘mediate’, ‘scaffold’, ‘support’, ‘intervene’, and ‘catalyse’ the way a person can ‘be’ in the world. How that may happen is partly covered by theories of social situatedness, which show how people use artefacts in the social setting to develop and sustain shared meanings [9]. This would suggest that it is mainly the users themselves who determine what artefacts are to them and what they use them for. D4EB, however, asks in more detail how exactly the appearance and interactive behaviour of the artefact impacts the way we take them up in our embodied couplings. The theory of sensorimotor coupling shows that the details of an object form or behaviour influences the way this object concretely affords action. In addition, incorporating this object in my embodied actions transforms the way I perceive the world (as when the perceptual world of the blind man gets transformed when he uses his cane). Here, design detail matters. To get the details right in actual design projects with real users and contexts, skilled designers are needed. Hybrid designers need to know the fine-grained details of how physical form and behaviours of the hybrid influence embodied couplings in the lifeworld. At the same time, however, these fine-grained details are always highly situated and dependent on people’s skills. Methodologically, this dilemma would at least invite a strongly participatory approach, in which users and their lifeworlds are brought together with skilled interaction designers. It would also invite a strongly iterative approach with many intermediate prototypes, each of which can be experienced in context and reflected upon to get first-hand experiences of ‘being with the hybrid in my lifeworld’, in order to inform the next design iteration.

11. Conclusions

One way to think about D4EB could be to ask how hybrids could come to operate similarly to the way traditional tools, like the carpenter’s hammer, or the blind man’s cane, as part of embodied couplings to the world. However, this would not by itself address the specific value of hybrid technologies. What added qualities arise if, say, the blind man’s cane becomes interactive, with sensors and actuators? D4EB tries to answer this question without immediately thereby reframing the cane as ‘just’ a physical carrier for an add-on digital application. Reasoning from D4EB, for example, one may critically examine the recently developed iCane (cf. https://www.i-cane.nl) as perhaps lacking a deep integration of the ‘add-on’ digital functionalities with the embodied operation of the cane as such. D4EB investigates what the value of hybrid technology could be while staying firmly in touch with our embodiment. D4EB hybrids are not merely something ‘in between’ digital computing and physical products: hybrids, going beyond hammers or traditional computer applications, have the potential to provide for a tremendous flexibility in continuously changing the way they behave and respond, while remaining situated in the world in which we are embedded ourselves as well. Hybrids may just be the ideal sorts of technologies to accommodate and enrich our embodied ‘being in the world’. They are potentially capable of continuously adjusting, transforming and augmenting our embodied being. They may enlarge our lifeworld itself in unexpected ways, generating new forms of sensitivity to the world that we at present cannot even imagine.