1. Introduction
A common tacit principle and a long-standing vision of the future has been that the development of technology and automation should make our lives easier and relieve us, for example, of dull and repetitive tasks. Moreover, it is commonly believed that the increased leisure time provided by technological innovations create better circumstances for developing ourselves in terms of cognition, creativity and culture. However, attitudes to the entry of technology into our lives have not always been this optimistic. At the outset of the computerization of Western societies, the father of cybernetics, Norbert Wiener ([
1], p. 220), saw that while automation “…can be used for the benefit of humanity…it may also be used to destroy humanity, and if not used intelligently it can go very far to that direction.” The immediate concern for Wiener was that the social dangers of the technology revolution would be overlooked for the sake of economical profits and worshipping of a “bronze idol” of new machines, but at the same time he was confident that there was a general awareness of our obligations to make sure machines are used for the benefit of man,
i.e., “…for increasing his leisure and enriching his spiritual life” ([
1], p. 221).
Can we still today afford to be lulled into the presumption that the on-going paradigm of technologization will just lead to a “better life”? Surely, it seems that history has mostly corroborated Wiener’s hopes rather than his worst fears. On the other hand, the time might be ripe for re-evaluating his initial warning as a consequence of the pervasive nature in which new computerized devices and interfaces are being introduced into our everyday life.
In Marshall McLuhan’s [
2] media theory, technology was conceived as an extension of the body, allowing us to reach beyond our natural bodily limits. McLuhan’s insight was that a technological medium affects the society in which it plays a role not only by its content, but by the characteristics of the medium itself. For instance, a light bulb does not have content in the way that the Internet offers us visual and textual information, yet inventing the light bulb has had enormous social effects; that is, enabling people to do things at night. For McLuhan, a light bulb is an example of a technological medium without any content in a manner similar to paper, streets or bicycles. We expect most digital interfaces, such as location-aware mobile technologies, to have graphical and textual content in the sense that McLuhan pointed out. However, we assume that wireless technologies also include content that is not easy to analyze, since it concerns our tactile-kinesthetic sense and bodily movements.
After McLuhan, researchers have examined how digital interfaces, as extensions of human behavior, affect our embodied living (e.g., [
3,
4]). Digital technologies are profoundly changing the way we perform some of our most basic everyday activities such as shopping, learning, working, courting, discussing and healing, but the actual world has not become obsolete or less important. As interfaces have become increasingly smaller and inexpensive, it is inevitable that there will be more and more informational devices surrounding us, infiltrating our daily life or filtering out some phenomena. In McLuhan’s terms, it might be appropriate to ask what kind of extensions these devices might develop to our bodies and to our ultimately embodied essence of living as human beings. This article will first examine some of the challenges of technologization that designers of human-technology interaction, either implicitly or explicitly, should be aware of, focusing on the relationship between the smart, automated technologies embedded in life and people as embodied, social, active, moving, sensing, feeling—as well as cognizant—human beings.
1.1. Dangers of Disengaging Ourselves from the Tangible World
If human-technology interaction design starts with an implicit idea that “technology should save us from doing this and relieve us from doing that”, are we not then placing the technological medium essentially
between a person and his interactional environment? Thus, the big question is: to what degree does the new interface-medium of smart devices disengage us from tangible experiencing of the actual world and the sensorimotor demands it poses? Keeping in mind the very nature of organism-environment interaction to which our species has developed and adapted in the phylogenetic continuum, how can we assess what type of our everyday actions (even the trivial ones) are dispensable and what kind of activity remains vital for living as human beings? The more we distance ourselves from the doing and moving, the less we have of the tactile and bodily “friction of life” through which we fundamentally experience and conceive our being-in-the-world (e.g., [
5]).
As the physical necessities in our interaction with the environment decrease, the emphasis in life encounters might move towards more abstract and information-based activities, due to the information-oriented (rather than action-oriented) ontology of human-technology interfaces. In this context, actions are often seen as means to “input” information to a device so that it is able to “output” or perform something for us. The emphasis of such interaction is on exchange of information. We cannot overlook the fact that when we increasingly value information as the essence of work and a culture—at the expense of bodily and other physical aspects—we are also in danger of ultimately weakening the very abilities for performing cognitive tasks.
The recognition that human cognition is tied to embodied action (e.g., [
6,
7,
8]), has become fundamental to recent trends in interaction design studies (e.g., [
9]). Even the abstract concepts of language and thinking arguably depend on the dynamic, sensorimotor patterns of a living, experiencing body [
10,
11]. In addition, seeing cognition as based on a history of enaction between an organism and an environment points to the possible causality that embodied interaction has on the continuum of genetic development of a cognitive faculty [
6]. This line of thought accords with the recent hypothesis about the fragile genetic basis of human intellectual fitness and its assumed correlation to the level of demand for sensorimotor and spatial reasoning skills that life requires of an individual [
12,
13]. Thus it may well be that the biggest threat that automation can pose to humanity is the decay of human fitness not only in a physical but also in a cognitive sense, if we choose not to appreciate the corporeal, interactional coupling of man and his environment. Contrary to the strong Artificial Intelligence (AI) hypothesis (see [
14]), which sees humans as equivalents to machines, we humans are born to act out our knowledge—not to process it as abstract information.
1.2. Towards the Designs that Bodily Engage Us with the World
An important challenge we want to posit here for future interaction design is to create technology that does not distance us from our embodied nature of living. In a perfect world, the mediating interfaces should be designed to conform to this nature, in order to become such an extension of a body that fuses with the sensorimotor schemas of natural interactions. The basic idea of exploiting natural interaction patterns in design is not novel, but we argue that the traditional use-centered and device-centered approach to interaction design severely limits the possibilities for applying this general principle in designs. In the traditional approach, a person is seen as a user whose actions are directed at a device in order to exploit its functions. This kind of interaction model runs the risk of being chauvinistic in its premise of having a kind of pre-assigned relationship between user and a certain device (or devices). It is often assumed that the user’s intentionality is focused on a device or its interface, and that the user’s activity occurs within a fixed “sphere” around the device.
As we see it, the starting point of interaction design should not be exclusively fixed to user-device relations, or to any specific contextual circumstances of use. Instead, the primary starting point should be set to the activities of a person in their daily environments. From this stance, devices are accounted for in terms of how they influence what we do or interface to our routine activities. In other words, devices may be assessed in terms of how they make us move and make us feel involved, how they support our activities and even produce novel yet natural ways to engage us with the physical and social environment. As noted earlier, introduction of a new technology inevitably changes the way we do things and how we on the whole perceive in terms of action affordances. Interaction design thus needs ways for acknowledging how the designs tap into the whole of the everyday continuum of movements and how to bring this aspect of human-technology choreographies into the scope of design methodology.
Relevantly to the above discussion, promising ideas of enactive user interfaces have recently emerged in the use-oriented field of human-computer interaction. As a key example, Froese and his colleagues [
15] have proposed a novel approach for conceptualizing user interface. The basic idea of this enactive approach is that user interface should be
experientially transparent and should not be something that is positioned between the user and his operational environment. For instance, instead of using a hand for giving commands to the device
via gestural input, the device itself (such as the enactive torch presented in Froese’s article) can experientially become an extension of our hand, seamlessly augmenting the enactive sense-making of the user. The enactive approach to user interfaces has in many ways parallel objectives with the present study, but being use-oriented in its nature, it still fails to provide the broader framework of movement-oriented interaction design that goes beyond the scope of user interfaces. In this article, we begin to outline a
choreography-based approach to interaction design, which is intended to provide this broader framework currently missing.
The enactive use-oriented and our choreographic approaches are neither contradictory nor exclusive to each other. Both of them are basically seeking ways to avoid a kind of human-technology interaction that overlooks the bodily engagement involved in it. From their differing perspectives, they both aim at such implementations of automation and technological designs that support the crucial role of the moving body in our thinking and sense-making processes. Therefore, in an ideal situation, both approaches are utilizable in tandem.
2. Primacy of Movement
As Sheets-Johnstone ([
16], p. 132) argues “…movement is the generative source of our primal sense of aliveness and our primal capacity for sense-making…”. To understand more profoundly embodied, kinesthetically oriented interaction with digital devices, we need a more coherent theoretical foundation that can develop meaningful understanding of complex kinaesthetic and affective aspects as they appear through bodily movements.
Traditionally, motor activity and kinesthesia have been considered a less “high” cortical function than mental activity such as linguistic or mathematical reasoning. In Jean Piaget’s [
17] developmental psychology, motor activities are treated as a preliminary phase on the way to higher cognitive operations. However, recent discussion in phenomenological philosophy (e.g., [
16,
18,
19]) and philosophical cognitive sciences (e.g., [
20,
21]) argues that perception cannot be understood without reference to action. Movement has primacy to perception (e.g. [
22,
16]). As Noë [
21] points out, perception is not something that happens to us, it is something we do. People do not perceive the world statically, but by actively exploring the environment. The world makes itself available to us though movement and interaction. When people merely touch an object, they understand little of what is perceived unless they move their hands and explore its contours and texture [
23]. When we lift an object, this reveals something about its weight, rubbing our fingers across it tells us about its texture and shape, and squeezing it says something about its compressibility ([
20], p. 50).
Sheets-Johnstone ([
16], p. 55) has criticized the mechanization of the body, an approach where the
qualia of movement is seen as a mere sensorimotor feedback. For Sheets-Johnstone, “thinking in movement” involves neither linguistic related embodied metaphors nor mechanistic input-output modeling but is tied to an on-going qualitatively experienced dynamic in which movement possibilities arise and dissolve. By
kinesthesia, she means the sense of movement,
i.e., a bodily felt sense of the direction of our movement, its speed, its range, its tension and so on [
22]. In this way, kinesthesia provides information about changes of locations and motility as social interaction.
Body movement alone can generate a lot of perceptual information. We expect inherently that certain movements are our own and some movements belong to other moving entities. If somebody pushes me across the room or if I take a leap across the room, the qualities of movement are similar, but I consider that only in the latter case is it
me who is the agent of the movement. Infants are sensitive to the difference between something starting to move on its own and something being pushed or otherwise made to move [
24]. “Self-motion” is the start of an independent trajectory where no other object or trajectory is involved. Very young children appear to understand the difference between inanimate and animate movement (e.g., [
16,
20]). Inanimate and animate movements are a key clue to our identification of various things in our environment both generally and specifically. In terms of animate movement, we frequently identify people’s sex and age from their movements how they walk, stand or sit.
Many neuropsychological studies support the claim that basic processes underlying embodied action are activated even in the absence of physical movement. When viewing static visual objects or patterns such as fossils, people tacitly recognise the presence of movement. Studies reveal that people infer dynamic information about movement when perceiving static shapes, such as when reading handwriting (e.g., [
25]). When research participants view a picture of a man jumping off a wall and are asked to remember the man’s position, their memory of his position is systematically biased forward along the trajectory of his jump [
26].
Ideomotor action refers to the phenomenon that an actual physical movement is related to imagining that movement. This implies that, for instance, just thinking about an action can develop ability to perform that action. Merely thinking about a kind of person can induce ideomotor mimicry of the person’s behavior [
27]. Particular areas of the cortex are activated not only when people imagine themselves making different movements, but also when people speak the name of a tool [
28].
Daniel Stern [
29] uses the term
vitality affect to describe the kinetic qualities, which allow an individual to discern the affective aspects of movements and their dynamic shifts. The notion of vitality affect is connected to psychoanalysis and empirical development psychology, and has a central place in Stern’s relatively complex and differentiated model of the child’s early development [
30]. Vitality affects are connected to vital life processes such as breathing, becoming hungry, falling asleep, waking up,
etc., as referred to by vitality [
30]. The point is the vitality affects refers to internal vital—heartbeat, pulse and breathing and constant sensations of the body. However, vital affects refer neither to a kind of biological-existential basic mode nor to proprioception or intero sensations, since vital processes are placed in the relation between the infant and the adult. When Stern exemplifies the vitality affects, it is most often with a string of words—they are that which is “surging”, “fading away”, “fleeting”, “exploding”, “effortful”, “accelerating”, “decelerating”, “climaxing”, “bursting” and “drawing out”. Those words can also characterize the body’s movement qualities in relation with other people or with handling different things. This intimate bodily connection is most clearly expressed in the phenomenon called “affect attunement” [
29,
30]. Affect attunement is, in principle, present in all kinds of interactions and is generated through the adult’s early handling of the child’s body, for instance through several calming, rhythmic bodily interactions. Through attuning to dynamic shifts in kinetic qualities, we are able to share the lived body’s inner experiences. This is also known as interpersonal communion, which, however, is not restricted to person-to-person interactions, but is present in the embodied processes of interacting with various artifacts and natural objects as well.
3. Outlining the Choreographic Approach to Interaction Design
3.1. Choreography as a Concept for Design
When movement seems to take a central role in our cognitive processes, how should we study and theories embodied movements and motion in interactive design as content rather than a pure medium? We assume that in HCI (human-computer interaction) embodied interaction is often formulated in terms of gestures or other discrete user actions. Thus, the traditional use-oriented approach recognizes movements, but only a fraction of movements are taken as relevant to interaction. In other words, only those movements that are directly related to the use of a given application, are of interest. Moreover, the ontology of interaction rarely emphasize temporality or procedures, but rather, actions are described in structural terms (see an example in [
31]). We suggest that the dynamic flow of movements should be analyzed forming meaningful interactions and relations between different animate or inanimate agents. By using here the term choreography, we want to capture not just agents’ movements and gestures but the constellations of movements in which different agents are involved.
Choreography is usually related to dancing and bodily movement patterns, performed by professional dancers at theatres but in recent years choreography has been admitted as a theoretical concept in different disciplines (e.g., [
32]). In terms of interaction design, choreography refers to acknowledging how design choices affect movements and actions (including the ideomotoric ones) while also taking into account the pre-existing choreographies of the given situations. In fact, most movements we make are
pre-choreographed by the physical, cultural, social, political and technical environment in which we are embedded. The architectural solutions of buildings as well as the whole infrastructure pre-choreograph our bodily movements, providing or suppressing opportunities for social interaction with other people. Pre-choreographies attributed to HCI designs can be seen in terms of action affordances the design implies for the user. The term affordance has its roots in J. J. Gibson’s [
33] ecological psychology, but it has become widely popular in interaction design practices [
34,
35].
By choreography, we mean all bodily movements and other activities in which movements appear to form meaningful interactions and relations between different animate or inanimate agents. It includes both a plan for the action, the action itself and all the agents it draws together. In terms of interaction design choreography refers to acknowledging of how design choices affect movements and actions while also taking into account the pre-existing choreographies of the given situations. The approach does not distinguish between artifacts of different technological nature; table, tablet-computer and walls of a room can be conceptualized through choreography. However, our sense of movement varies, depending on how we move our bodies in handling these objects and how their materiality responds to our movements. Thus, kinesthesia has a central role in what kind of interactions with digital devices we consider meaningful and immersive.
We assume that choreography is a fruitful theoretical concept in understanding interaction design because of its scalability, which allows examination from micro movements, such as touch on an iPad to macro level movements, such as the system of manufacturing and transporting these devices. In the following section, through these levels of examining human-technology choreographies, we aim at better defining the potentials of the choreography approach to interaction design.
3.2. Three Levels for Outlining Movements
The choreographic approach to analyzing interaction design can capture different levels of movements, which we call here
micro,
local and
macro movements. By micro movements we mean, for instance, touching an iPad or just imagining touching,
i.e., performing an ideomotor action. By local-level movements, we mean how, for instance, the actions we perform an iPad relate to the interaction with the immanent physical or virtual environment and how these actions are connected to our flow of other activities or social relations we create and maintain by using this device. Macro movements refer to a large-scale system of movements that expands beyond the present situation and its agents. Macro-level choreographies might, for instance, tap into manufacturing and transporting these iPads in global trade. On the other hand, they also relate to the ways that the collective usage of iPads integrates in our everyday ecosystems and creates new choreographic habits. All these different levels of movements are connected, and all of them offer different perspectives for contributing to interaction design (see summary in
Table 1).
In everyday life, we are mostly focused on local movements: We do ordinary things by moving in home environments, go shopping to a supermarket by car or exercise in a gym. Even if we may feel our movements as self-motion, our everyday moving is pre-choreographed by physical and social environments. As we wander in the in its corridors, stairs and rooms of a building, this physical space creates for us a pre-choreography within spatial limitations and possibilities. We may make our individual movement choices, but environments include choreographic affordances, which direct our gestures, movements and interactions. Instead of regarding buildings and infrastructure as static elements we could consider them inherently as moving-oriented spaces and shapes. Movement can be felt to be fully integrated into buildings, streets and interfaces by exploring how these things appear to us when we handle or move with, in or on them.
In terms of micro and local movements, we frequently overlook a prime element in these forms of bodily motions: the simple ineffable pleasure of, and being in, action. Charles Rosen [
36] argues that one cannot even become a professional pianist if one does not deeply enjoy the physical movements of one’s fingers on the keys. Pianists do not devote their lives to their instrument simply because they love music: that would not be enough to justify a dreary existence of stuffy airplanes, uncomfortable hotel rooms, and the hours spent trying to get the local piano technician to adjust the soft pedal. There has to be an intellectualized and physical need to have contact with the keyboard—simply the mechanics and complexity of playing—which may be connected with a love of music but which is by no means totally coincident with it (see also [
37]).
By analyzing micro-level movements and the related experiences involved in the application use, designers are able to develop such interface elements that engage with the dynamicity of our acting-sensing bodies and enactive minds. By shifting to the local-level perspective, the focus of this bodily engagement shifts to the intentional, environment-oriented and social aspects of interaction, and how the micro-level (actual and imagined) movements connect to the choreographic continuum of the user’s actions. Finally, the shift to the macro-level perspective brings up questions of the choreographic sustainability of our everyday milieus and large-scale socio-cultural effects of the design choices.
Macro level movements are not necessary as experiential and physical as local and micro movements, but still there seems to be a strong connection to how affect attunement to an interface becomes collective and affects the ways these interfaces are distributed globally. Macro level movements are characterized as movements that cross our own physical limits forming connections and dynamics beyond our embodied capabilities. For instance, in handling ordinary items, such as a vacuum cleaner and a cell phone, we are, in fact, one agent of a complex choreography of the global manufacturing and trade, which produces different items for us. For instance, thinking about thecoffee-making process at home, we may notice that the coffee beans were produced in Columbia and our coffee-maker made in China, thus this everyday action produces a kind of global choreography amongst different agents and their role as producers or consumers in the global market.
Table 1.
A summary of the proposed levels of examination.
Table 1.
A summary of the proposed levels of examination.
| Micro | Local | Macro |
---|
Movement focus | Actual and imaginary movements and the related experiential qualities Subtleties and habituations of muscular activity | Intentionality of movement (i.e., how actions are directed/related to the environment) Interactions with the perceived environment Social interactions | Interconnections to other milieus Experiential coherence between milieus and environments |
Spatial focus | Kinesphere,
i.e., the reachable space around the body Inner Kinesphere, i.e., the space inside the body | Environment,
i.e., physical and social space in which moving bodies interact with things and other bodies Virtual space (immanent) | Geographical space Virtual space (extended/distributed) |
Agency | Personal | Personal and interpersonal | Collective |
Value for design | Bodily engaging user interface elements | Usable, engaging environments Revealing connections between user interfaces and user’s continuum of movements | Usable, engaging “digital ecosystems” Sustainable design |
3.3. Case: Remote Control Device
As we stated above, we assume that these three different levels of movements are never separated in using devices in everyday life. Thus, next, we would like to take a more detailed look at the kind of dynamics that using a simple movement-based device may mobilize culturally, socially and economically over our own embodied initiatives.
The television remote control device (RCD) is a good example of the simple movement-based devices we have used more or less automatically in watching TV from a couch. The development of the television remote control started from a simple, well-defined practical problem: how to change channels and adjust the volume without leaving the viewing location. The first solution was to bring the control panel to the viewing place using cables (the 1959’s “Lazy Bone”). Later the device was developed to work wirelessly. In many respects, the basic concept has remained the same over the last sixty years, which indicates that it has served the original need well. The interesting issue then becomes the different ways in which the remote control has influenced our media culture. Firstly, before the advent of RCDs, commercials were placed between the programs. Unfortunately for advertisers the RCD made changing the channel after a program so effortless and quick that the visibility of commercials was drastically reduced. Therefore, the commercials had to be placed in breaks during the program itself—so called “commercial breaks”—in order to increase the probability that they would be seen. This, in turn, changed the structure of program. Secondly, it was noted that RCD users tended to switch channel before the final credits of a program, so in response a split-screen technique was created: the credits role on one half of the screen during the last scene. Thirdly, it is fairly obvious that the huge growth in the number of channels available would hardly have happened without the RCD; the culture of zapping through channels was enabled by the RCD. The zapping culture, in turn, largely determines the production of all televised material: At whatever moment you switch to a given channel, there have to be elements that entice you to stop zapping and stay on that particular channel [
38,
39].
The RCD has thus become the enabler of zapping culture, which in turn has revolutionized much of our media consumption and production. In other words, the RCD helped to control the television set, but as a side effect, dramatically influenced the content creation of television programs. It can be argued that the fragmented world of media and the increasing restlessness of our technology-saturated everyday life are in part caused by the trend that the television remote control launched.
Figure 1.
Illustration of the remote control device (RCD) case in terms of the levels of examination.
Figure 1.
Illustration of the remote control device (RCD) case in terms of the levels of examination.
We are not arguing that the world would be a better place without RCD in our living rooms, but the change in our media consumption culture described above was hardly in the mind of the pioneers of the RCD. They perhaps had an implicit idea of micro and local level choreography, but ended up by revolutionizing the macro-level behavior in an unplanned manner. In the proposed choreography approach, the human activity, rather than execution of functions of an application, is central. Choreographic examination helps us to conceive how the physically more active traipse between couch and television has been replaced with imaginary movements in virtual spaces (
i.e., movements across channels—the imagined “terrain” of content streams). Eventually, this change in local choreographies caused the described network effects in our media culture. The aim of the
Figure 1 is to highlight the fact that even when designing individual user-interface elements, the designer needs to be aware of his role as the designer of an information society. Thus all the levels of choreographic examination need to be present in all phases of design.
4. Conclusions
In this paper, we have outlined a choreography-based approach to interaction design. Our aim has been to offer a methodological approach to analyze, not just single gestures or discrete events, but more complicated movement dynamics that enmesh human-computer interaction. In the traditional approach to interaction design, a person is considered to be a user whose actions are directed to a device in order to exploit its functions so that the user’s activity occurs within a fixed “sphere” around the device. At first sight, these two approaches may appear to conflict. We argue, however, that this is not necessarily the case. The two approaches may start from opposite ends but ideally meet in the implementation of the technology. The setting resembles building a bridge from both ends at the same time. With skillful planning, these two constructions meet in the middle and finally form a seamless bridge. This requires close cooperation between the builders at each end, just as does the successful implementation of interactive technology; there has to be the expertise of technical opportunities has to be available side by side with the understanding of the human being as an embodied, active creature.
Our concern has been that interaction design development, which has emphasized abstract and information-based activities, is drifting far away from the physical necessities in our interaction with the environment. In terms of movement-based interfaces, actions and movements are often seen merely as means rather than contents. We assume that the understanding of movement has remained rather mechanistic, emphasizing mere haptic and sensorimotor feedback from interfaces with the result that digital interfaces do not support sensitive or intelligent tactile-kinesthetic input and output in user experiences. In addition, designers might also have ignored what kind of dynamics the use of a simple movement-based device may mobilize culturally, socially and economically over our own embodied initiatives.
Our stance is that the choreographic approach can reveal new potential in developing human-technology interfaces that support creative movement practices and stimulate imagination within movement experiences. We argue that without the awareness of the cognitive and experiential processes of the moving body we cannot really develop “smart” interfaces that feed our movement capabilities and function as enactive extensions to our body. Human-technology interfaces should motivate us to move and interact with our environments in new, engaging ways rather than forcing us to conform to interface-centric boundaries. Movement choices are always partly intuitive reactions, so users are not necessarily aware of the manipulations to which scenarios draw them. We see choreographies mainly as a utility for putting the design focus on the activity instead of on shapes and structures of objects, and for outlining the ways the intended use of technologies collide with the flow of our daily activities. In our sketch of the choreographic approach, the starting point of design is not the technology or even an assumed need for it, but the embodiment of human-technology choreographies, consisting of an experiential continuum of movements.