Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
4.3
2.5
Journals
MTI
Special Issues
Multimodal Learning
share announcement

Multimodal Learning

A special issue of Multimodal Technologies and Interaction (ISSN 2414-4088).

Deadline for manuscript submissions: closed (21 May 2018) | Viewed by 51321

Special Issue Editors

Prof. Dr. Carey Jewitt

E-Mail Website
Guest Editor
Institute of Education, University College London, London WC1E 6BT, UK
Interests: multimodality; interdisciplinary methods and methods innovation; digital interaction and communication
Special Issues, Collections and Topics in MDPI journals
Prof. Dr. Sara Price

E-Mail Website
Guest Editor
Institute of Education, University College London, London WC1E 6BT, UK
Interests: emerging technologies; embodiment; multisensory interaction; learning; cognition
Special Issues, Collections and Topics in MDPI journals
Dr. Kate Cowan

E-Mail Website
Guest Editor
UCL Knowledge Lab, Institute of Education, University College London, London, UK
Interests: multimodal methodologies; digitally-mediated teaching and learning; early childhood education; play

Special Issue Information

Dear Colleagues,

This special issue aims to explore the opportunities and challenges of multimodal approaches to design and interaction in the context of technology-mediated learning. Multimodal learning is at the intersection of social and technological developments in the interdisciplinary area of education. Learning practices, artefacts, and environments have always been multimodal: touch-based exhibits in museums, the visual design of learning resources, the spatial design of classroom environments, and the embodied play of young children, to name a few examples. While multimodal interaction is not “new”, digital technologies have served to foreground, emphasize, and reconfigure the multimodal character of learning in significant ways—including the range of representational and communicational forms drawn into learning, and their relationships and roles in learning. The theorization of mind and body, learner agency, spatiality, and theoretical turns across disciplinary boundaries to the visual, the “turn to the body”, and most recently “the sensory turn” have also influenced understanding of learning as multimodal. This is reflected in the development of multimodal interaction technologies (e.g. sensors, natural user interfaces, and whole-body interfaces) that increasingly exploit bodily modes and generate sensory experiences to support learning. Together, this raises questions of what learning is, where and how it happens, and how to foster and design for learning. In addition, technology provides educational researchers with access to new forms and scales of data with which to investigate multimodal interaction via a range of technological tools that capture real-time multimodal interaction (e.g., video recording, sensor-based, and GPS devices).

We encourage authors to submit original research articles, case studies, reviews, theoretical and critical perspectives, and viewpoint articles on multimodal technologies and learning, including but not limited to:

  • Theoretical framings and the design of multimodal learning artefacts and environments
  • Critical engagement with the pedagogic and learning considerations and consequences of multimodal technologies
  • How new advances in interactive technologies influence how we understand and constitute multimodal learning
  • Exploration of methods or methodological approaches appropriate for designing and researching multimodal digital interactions for learning
  • Case studies of the design and research of multimodal technology-mediated learning
  • Empirical studies on the application of multimodal learning environments in the wild
  • Emerging trends and potentials in the research and design of multimodal technologies for learning (e.g., the use of automatic capture and sensing techniques of sensory forms of interaction)

Of particular interest are articles that critically explore multimodal technologies and learning, in both formal and informal learning contexts, on the topics of multi-sensory and multimodal learning, playful learning, designing for inclusion and/or different abilities, and interdisciplinary perspectives on teaching and learning.

Prof. Carey Jewitt
Prof. Sara Price
Dr. Kate Cowan
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Multimodal Technologies and Interaction is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 1600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Published Papers (10 papers)

Download All Papers
Order results
Result details
Show export options expand_more Show export options expand_less
Select all
Export citation of selected articles as:

Research

23 pages, 5653 KiB  
Article
“There Was No Green Tick”: Discovering the Functions of a Widget in a Joint Problem-Solving Activity and the Consequences for the Participants’ Discovering Process
by Patrick Sunnen, Béatrice Arend and Valérie Maquil
Multimodal Technol. Interact. 2018, 2(4), 76; https://doi.org/10.3390/mti2040076 - 26 Oct 2018
Viewed by 3146
Abstract
In recent years, tangible user interfaces (TUI) have gained in popularity in educational contexts, among others to implement problem-solving and discovery learning science activities. In the context of an interdisciplinary and cross-institutional collaboration, we conducted a multimodal EMCA-based video user study involving a [...] Read more.
In recent years, tangible user interfaces (TUI) have gained in popularity in educational contexts, among others to implement problem-solving and discovery learning science activities. In the context of an interdisciplinary and cross-institutional collaboration, we conducted a multimodal EMCA-based video user study involving a TUI-mediated bicycle mechanics simulation. This article focusses on the discovering work of a group of three students with regard to a particular tangible object (a red button), designed to support participants engagement with the underlying physics aspects and its consequences with regard to their engagement with the targeted mechanics aspects. Full article
(This article belongs to the Special Issue Multimodal Learning)
►▼ Show Figures

Figure 1

13 pages, 3584 KiB  
Article
Multimodal Technologies in LEGO House: A Social Semiotic Perspective
by Kate Cowan
Multimodal Technol. Interact. 2018, 2(4), 70; https://doi.org/10.3390/mti2040070 - 10 Oct 2018
Cited by 3 | Viewed by 4862
Abstract
Children’s playworlds are a complex interweaving of modes, with the border areas between the digital and non-digital often becoming increasingly blurred. Growing in popularity and prevalence, multimodal technologies blending digital and non-digital elements present novel opportunities for designers of toys and play-spaces as [...] Read more.
Children’s playworlds are a complex interweaving of modes, with the border areas between the digital and non-digital often becoming increasingly blurred. Growing in popularity and prevalence, multimodal technologies blending digital and non-digital elements present novel opportunities for designers of toys and play-spaces as well as being of interest to researchers of young children’s contemporary play and learning. Opened in Denmark in September 2017, LEGO House defines itself as the ‘Home of the Brick’, a public attraction aiming to support play, creativity and learning through multiple interactive LEGO experiences spanning digital and non-digital forms. Offering a rich context for considering multimodal perspectives on contemporary play, this article reports on a range of multimodal technologies featured in LEGO House, including digital cameras, scanners, and interactive tables used in combination with traditional LEGO bricks. Three LEGO House experiences are considered from a multimodal social semiotic perspective, focusing on the affordances of multimodal technologies for play, and the process of transduction across modes, in order to explore the liminal border-areas where digital and non-digital play are increasingly mixed. This article proposes that LEGO House presents an innovative ‘third space’ that creates opportunities for playful interaction with multimodal technologies. LEGO House can be seen as part of a growing recognition of the power of play, both in its own right and in relation to learning, acknowledging that meaning-making happens in informal times and places that are not positioned as direct acts of teaching. Furthermore, it is suggested that multimodal technologies embedded into the play-space expand opportunities for learning in new ways, whilst highlighting that movement between digital and non-digital forms always entails both gains and losses: A matter which needs to be explored. Highlighting the opportunities for meaning-making in informal, play-based settings such as LEGO House therefore has the potential to recognise and give value to playful meaning-making with multimodal technologies which may otherwise be taken for granted or go unnoticed. In this way, experiences such as those found in LEGO House can contribute towards conceptualisations of learning which support children to develop the playfully creative skills and knowledge required for the digital age. Full article
(This article belongs to the Special Issue Multimodal Learning)
►▼ Show Figures

Figure 1

27 pages, 6386 KiB  
Article
The Communicative Effectiveness of Education Videos: Towards an Empirically-Motivated Multimodal Account
by John A. Bateman and Florian Schmidt-Borcherding
Multimodal Technol. Interact. 2018, 2(3), 59; https://doi.org/10.3390/mti2030059 - 12 Sep 2018
Cited by 16 | Viewed by 4625
Abstract
Educational content of many kinds and from many disciplines are increasingly presented in the form of short videos made broadly accessible via platforms such as YouTube. We argue that understanding how such communicative forms function effectively (or not) demands a more thorough theoretical [...] Read more.
Educational content of many kinds and from many disciplines are increasingly presented in the form of short videos made broadly accessible via platforms such as YouTube. We argue that understanding how such communicative forms function effectively (or not) demands a more thorough theoretical foundation in the principles of multimodal communication that is also capable of engaging with, and driving, empirical studies. We introduce the basic concepts adopted and discuss an empirical study showing how functional measures derived from the theory of multimodality we employ and results from a recipient-based study that we conducted align. We situate these results with respect to the state of the art in cognitive research in multimodal learning and argue that the more complex multimodal interactions and artifacts become, the more a fine-grained view of multimodal communication of the kind we propose will be essential for engaging with such media, both theoretically and empirically. Full article
(This article belongs to the Special Issue Multimodal Learning)
►▼ Show Figures

Figure 1

17 pages, 1789 KiB  
Article
Tangible Representational Properties: Implications for Meaning Making
by Taciana Pontual Falcão
Multimodal Technol. Interact. 2018, 2(3), 54; https://doi.org/10.3390/mti2030054 - 05 Sep 2018
Cited by 1 | Viewed by 3439
Abstract
Tangible technologies are considered promising tools for learning, by enabling multimodal interaction through physical action and manipulation of physical and digital elements, thus facilitating representational concrete–abstract links. A key concept in a tangible system is that its physical components are objects of interest, [...] Read more.
Tangible technologies are considered promising tools for learning, by enabling multimodal interaction through physical action and manipulation of physical and digital elements, thus facilitating representational concrete–abstract links. A key concept in a tangible system is that its physical components are objects of interest, with associated meanings relevant to the context. Tangible technologies are said to provide ‘natural’ mappings that employ spatial analogies and adhere to cultural standards, capitalising on people’s familiarity with the physical world. Students with intellectual disabilities particularly benefit from interaction with tangibles, given their difficulties with perception and abstraction. However, symbolic information does not always have an obvious physical equivalent, and meanings do not reside in the representations used in the artefacts themselves, but in the ways they are manipulated and interpreted. In educational contexts, meaning attached to artefacts by designers is not necessarily transparent to students, nor interpreted by them as the designer predicted. Using artefacts and understanding their significance is of utmost importance for the construction of knowledge within the learning process; hence the need to study the use of the artefacts in contexts of practice and how they are transformed by the students. This article discusses how children with intellectual disabilities conceptually interpreted the elements of four tangible artefacts, and which characteristics of these tangibles were key for productive, multimodal interaction, thus potentially guiding designers and educators. Analysis shows the importance of designing physical-digital semantic mappings that capitalise on conceptual metaphors related to children’s familiar contexts, rather than using more abstract representations. Such metaphorical connections, preferably building on physical properties, contribute to children’s comprehension and facilitate their exploration of the systems. Full article
(This article belongs to the Special Issue Multimodal Learning)
►▼ Show Figures

Figure 1

19 pages, 4020 KiB  
Article
Maker Literacies and Maker Citizenship in the MakEY (Makerspaces in the Early Years) Project
by Jackie Marsh, Hans Christian Arnseth and Kristiina Kumpulainen
Multimodal Technol. Interact. 2018, 2(3), 50; https://doi.org/10.3390/mti2030050 - 28 Aug 2018
Cited by 43 | Viewed by 7245
Abstract
In this paper, the potential relationship between creative citizenship and what may be termed ‘maker literacies’ is examined in the light of emergent findings from an international project on the use of makerspaces in early childhood, “MakEY” (see http://makeyproject.eu). The paper outlines the [...] Read more.
In this paper, the potential relationship between creative citizenship and what may be termed ‘maker literacies’ is examined in the light of emergent findings from an international project on the use of makerspaces in early childhood, “MakEY” (see http://makeyproject.eu). The paper outlines the concept of creative citizenship and considers the notion of maker literacies before moving on to examine how maker literacies might be developed in early-years curricula in ways that foster civic engagement. Three vignettes are offered of makerspaces in early-years settings and a museum in Finland, Norway, and the UK. The activities outlined in the vignettes might be conceived of as ‘maker citizenship’, a concept which draws together understandings of making, digital literacies, and citizenship. The paper considers the implications of this analysis for future research and practice. Full article
(This article belongs to the Special Issue Multimodal Learning)
►▼ Show Figures

Figure 1

17 pages, 784 KiB  
Article
To Boldly Go: Feedback as Digital, Multimodal Dialogue
by James Lamb
Multimodal Technol. Interact. 2018, 2(3), 49; https://doi.org/10.3390/mti2030049 - 27 Aug 2018
Cited by 10 | Viewed by 7383
Abstract
This article is concerned with digital, multimodal feedback that supports learning and assessment within education. Drawing on the research literature alongside a case study from a postgraduate program in digital education, I argue that approaching feedback as an ongoing dialogue presented in richly [...] Read more.
This article is concerned with digital, multimodal feedback that supports learning and assessment within education. Drawing on the research literature alongside a case study from a postgraduate program in digital education, I argue that approaching feedback as an ongoing dialogue presented in richly multimodal and digital form can support opportunities for learning that are imaginative, critical, and in-tune with our increasingly digitally-mediated society. Using the examples of a reflective blogging exercise and an assignment built in the Second Life virtual world, I demonstrate how the tutor’s emphasis on providing feedback in multimodal form, alongside more conventional print-based approaches, inspired and emboldened students towards the creation of apt and sophisticated coursework. At the same time, the crafting of multimodal feedback carries resource implications and can sit uncomfortably with some deep-rooted assumptions around language-based representations of academic knowledge. This article should be seen in the context of a growing pedagogic and institutional interest in feedback around assessment, alongside the emergence of new ways of communicating and consuming academic content in richly multimodal ways. In this setting, multimodality, technology, and interaction refers to the digitally-mediated dialogue that takes place between the student and tutor around assessment. Full article
(This article belongs to the Special Issue Multimodal Learning)
►▼ Show Figures

Figure 1

19 pages, 1025 KiB  
Article
Debugging in Programming as a Multimodal Practice in Early Childhood Education Settings
by Mia Heikkilä and Linda Mannila
Multimodal Technol. Interact. 2018, 2(3), 42; https://doi.org/10.3390/mti2030042 - 13 Jul 2018
Cited by 33 | Viewed by 5664
Abstract
The aim of this article is to broadly elaborate on how programming can be understood as a new teaching scope in preschools, focusing specifically on debugging as one of the phases involved in learning to program. The research question How can debugging as [...] Read more.
The aim of this article is to broadly elaborate on how programming can be understood as a new teaching scope in preschools, focusing specifically on debugging as one of the phases involved in learning to program. The research question How can debugging as part of teaching and learning programming be understood as multimodal learning? has guided the analysis and the presentation of the data. In this study, and its analysis process, we have combined a multimodal understanding of teaching and learning practices with understandings of programming and how it is practiced. Consequently, the multidisciplinary approach in this study, combining theories from social sciences with theories and concepts from computer science, is central throughout the article. This is therefore also a creative, explorative process as there are no clear norms to follow when conducting multidisciplinary analyses. The data consist of video recordings of teaching sessions with children and a teacher engaged in programming activities. The video material was recorded in a preschool setting during the school year 2017–2018 and consists of 25 sessions of programming activities with children, who were four or five years old. The results show how debugging in early childhood education is a multimodal activity socially established by use of speech, pointing and gaze. Our findings also indicate that artefacts are central to learning debugging, and a term ‘instructional artefacts’ is therefore added. Finally, the material shows how basic programming concepts and principles can be explored with young children. Full article
(This article belongs to the Special Issue Multimodal Learning)
►▼ Show Figures

Figure 1

15 pages, 2832 KiB  
Article
Opportunities and Challenges of Bodily Interaction for Geometry Learning to Inform Technology Design
by Sara Price and Sam Duffy
Multimodal Technol. Interact. 2018, 2(3), 41; https://doi.org/10.3390/mti2030041 - 09 Jul 2018
Cited by 9 | Viewed by 4057
Abstract
An increasing body of work provides evidence of the importance of bodily experience for cognition and the learning of mathematics. Sensor-based technologies have potential for guiding sensori-motor engagement with challenging mathematical ideas in new ways. Yet, designing environments that promote an appropriate sensori-motoric [...] Read more.
An increasing body of work provides evidence of the importance of bodily experience for cognition and the learning of mathematics. Sensor-based technologies have potential for guiding sensori-motor engagement with challenging mathematical ideas in new ways. Yet, designing environments that promote an appropriate sensori-motoric interaction that effectively supports salient foundations of mathematical concepts is challenging and requires understanding of opportunities and challenges that bodily interaction offers. This study aimed to better understand how young children can, and do, use their bodies to explore geometrical concepts of angle and shape, and what contribution the different sensori-motor experiences make to the comprehension of mathematical ideas. Twenty-nine students aged 6–10 years participated in an exploratory study, with paired and group activities designed to elicit intuitive bodily enactment of angles and shape. Our analysis, focusing on moment-by-moment bodily interactions, attended to gesture, action, facial expression, body posture and talk, illustrated the ‘realms of possibilities’ of bodily interaction, and highlighted challenges around ‘felt’ experience and egocentric vs. allocentric perception of the body during collaborative bodily enactment. These findings inform digital designs for sensory interaction to foreground salient geometric features and effectively support relevant forms of enactment to enhance the learning experience, supporting challenging aspects of interaction and exploiting the opportunities of the body. Full article
(This article belongs to the Special Issue Multimodal Learning)
►▼ Show Figures

Figure 1

20 pages, 3453 KiB  
Article
Exploring Emergent Features of Student Interaction within an Embodied Science Learning Simulation
by Jina Kang, Robb Lindgren and James Planey
Multimodal Technol. Interact. 2018, 2(3), 39; https://doi.org/10.3390/mti2030039 - 02 Jul 2018
Cited by 11 | Viewed by 4054
Abstract
Theories of embodied cognition argue that human processes of thinking and reasoning are deeply connected with the actions and perceptions of the body. Recent research suggests that these theories can be successfully applied to the design of learning environments, and new technologies enable [...] Read more.
Theories of embodied cognition argue that human processes of thinking and reasoning are deeply connected with the actions and perceptions of the body. Recent research suggests that these theories can be successfully applied to the design of learning environments, and new technologies enable multimodal platforms that respond to students’ natural physical activity such as their gestures. This study examines how students engaged with an embodied mixed-reality science learning simulation using advanced gesture recognition techniques to support full-body interaction. The simulation environment acts as a communication platform for students to articulate their understanding of non-linear growth within different science contexts. In particular, this study investigates the different multimodal interaction metrics that were generated as students attempted to make sense of cross-cutting science concepts through using a personalized gesture scheme. Starting with video recordings of students’ full-body gestures, we examined the relationship between these embodied expressions and their subsequent success reasoning about non-linear growth. We report the patterns that we identified, and explicate our findings by detailing a few insightful cases of student interactions. Implications for the design of multimodal interaction technologies and the metrics that were used to investigate different types of students’ interactions while learning are discussed. Full article
(This article belongs to the Special Issue Multimodal Learning)
►▼ Show Figures

Figure 1

20 pages, 10585 KiB  
Article
An Exploratory Study of the Uses of a Multisensory Map—With Visually Impaired Children
by Emeline Brulé, Gilles Bailly, Anke Brock, Annie Gentès and Christophe Jouffrais
Multimodal Technol. Interact. 2018, 2(3), 36; https://doi.org/10.3390/mti2030036 - 24 Jun 2018
Cited by 12 | Viewed by 5351
Abstract
This paper reports an empirical study of a multisensory map used by visually impaired primary school pupils, to study human habitats and differences between urban, suburban and rural areas using a local example. Using multimodal analysis, we propose to examine how the use [...] Read more.
This paper reports an empirical study of a multisensory map used by visually impaired primary school pupils, to study human habitats and differences between urban, suburban and rural areas using a local example. Using multimodal analysis, we propose to examine how the use of smell and taste shape pupils’ engagement and the development of a non-visual knowledge of geography. Our research questions include: How do pupils try to make sense of this unusual material, in conjunction with the tactile, audio and tangible material used in this lesson? How does the special education teacher support the development of these interpretations? Multisensory material has the potential to support experiential and embodied learning: were these promises achieved? Our findings show how this multisensory map reconfigures spatial occupation and interaction dynamics, and that it has the potential to make the classroom more pervasive to pupils’ social, spatial and emotional lives. In doing so, it provides opportunities for the teacher to develop citizenship education. The paper provides concrete examples of uses of smell and taste in learning activities to support engagement, and has implications for pedagogical design beyond special education. Full article
(This article belongs to the Special Issue Multimodal Learning)
►▼ Show Figures

Figure 1

Show export options expand_more Show export options expand_less
Select all
Export citation of selected articles as:
 
Displaying articles 1-10
Back to TopTop