‘You Really Have to Get in There and Actually Figure It Out’: Engaging Pre-Service Teachers in Children’s Literature Through Transmodality

Abstract

Transmodality—the process of transforming a text or section of a text into another mode or modes—enables readers to engage deeply and imaginatively with literature through interpretation and response. It is a valuable pedagogical approach in initial teacher education, where pre-service teachers are developing dispositions towards reading and cultivating knowledge of literature. In this article, two case studies are presented of undergraduate and post-graduate courses that aimed to engage pre-service teachers with children’s literature by asking them to respond to texts through embodied and multimodal modes. The work is underpinned by theories that highlight the role of semiotic modes in reading and writing, with a focus on the gestural, spatial, and auditory modes. The first case study examines the ways in which gesture and space worked to create multimodal ensembles that communicate and make meaning. The second case study considers pre-service teachers engaged in transferring meaning across linguistic and aural modes as they read a classic literary text and composed a soundscape. In both cases, we consider how mode-switching developed and demonstrated pre-service teachers’ aesthetic, cognitive, and affective engagement as part of their embodied experience with literary texts. This research has implications for the way teachers and teacher educators can inspire engagement with children’s literature through embodied and multimodal ways in English curriculum contexts and initial English teacher education.

1. Introduction

Transmodality—the process of transforming a text or section of a text into another mode or modes—enables readers to engage deeply and imaginatively with literature through interpretation and response (Mills & Brown, 2022; McCormick, 2011; Enriquez & Wager, 2018; Nash, 2018). It is a valuable pedagogical approach in initial teacher education where pre-service teachers (PSTs) develop dispositions towards reading and acquiring knowledge of literature across the modes (Horst et al., 2023). It is essential for pre-service teachers to cultivate knowledge of literature and a love of reading, as it gives them the capacity to inspire and guide children and young people to read (Cremin et al., 2014). While the study of literature is not often prioritised in Initial Teacher Education (ITE) (Simpson, 2021), we contend that it is important for curriculum design to enable pre-service teachers to engage deeply with children’s literature (Farrar, 2021). Deeper engagement comes through meaningful interactions that are enhanced by knowing about literature, language, and the way literary texts are composed. It comes from opportunities to interpret, imagine, and experience the worlds of literary texts. These opportunities may be possible through the processes of reading and transforming literature into visual, spatial, or audio modes—processes that are known as transduction and/or transmediation. These processes can enable pre-service teachers to engage deeply with children’s literature.

In this article, we propose that transmodality can engage pre-service teachers with children’s literature in ways that value their own reader responses as they develop knowledge of text and cultivate pedagogical understandings (Skaar et al., 2018). Creating spaces in initial teacher education for PSTs to interact meaningfully with literature and share their reader responses in creative and thoughtful ways can inspire them to foster a love of reading in the children and young people they will teach (Cremin et al., 2009). Transforming literary texts from one semiotic mode into another semiotic mode creates these spaces and enables the interpretative and imaginative work needed to promote meaningful interactions with literature (Mills, 2011; Mills & Brown, 2022; McCormick, 2011). We contend that by immersing themselves in translating the written text into auditory, gestural, and spatial forms, pre-service teachers can engage with literature both personally and analytically, gaining a deeper understanding of what it means to “read”. While there are many ways of engaging pre-service teachers with children’s literature, in our undergraduate English education courses we aim to engage pre-service teachers by asking them to respond to texts through embodied and multimodal modes. In accordance with this aim, our English education courses provide opportunities to transform texts from linguistic to embodied and/or aural modes, which enables pre-service teachers to be immersed in literary texts. Our work is underpinned by theories that highlight the role of semiotic modes in reading and writing, with a focus on the affordances of the body and sound and aural meaning making (Kress, 2010; Van Leeuwen, 2011; Newfield, 2014; Jewitt, 2014; Rowsell, 2013; Mills, 2011). These theories have guided our curriculum design and informed our intentions to generate opportunities for PSTs to transfer meaning from one mode to another and to reflect on the process of doing so.

In recognition that transforming literary texts from one mode to another mode engages pre-service teachers with children’s literature in ways that value their personal responses as readers, we adopt the concept of reading as a transactional experience, which highlights the interaction between reader and text (Rosenblatt, 1938). This concept highlights that ‘…when individuals interpret texts, they bring their own knowledge, assumptions, and repertoires to make meaning’ (Colton et al., 2023, p. 219). Thus, the reader actively creates the literary work ‘under the guidance of the text’ (Rosenblatt, 1938). The meaning-making process is described by Rosenblatt as a ‘kind of shuffling back and forth as one or another synthesizing elements—a context, a persona, a level of meaning—suggests itself to him (sic)’ (Rosenblatt, 1978/1994, p. 10). Making meaning with the text through reading can be seen as a ‘dynamic happening’ that occurs between the reader and the text (Iser, 1978). Transforming texts from one mode to another enables pre-service teachers to interpret and synthesise elements as they make meaning with literary texts in dynamic and interactive ways. This can develop their enjoyment and knowledge of literature and enable them to teach literature with confidence and in knowledgeable ways.

Curriculum design in Initial Teacher Education (ITE) needs to facilitate deep engagement with children’s literature for pre-service teachers because they will be the next generation of teachers who inspire children and young people to read literary texts. Reading literature in particular provides children with material that engages them cognitively and emotionally and can thus enhance their understanding and enjoyment of reading (Ommundsen et al., 2022). Literature allows young readers to develop their thinking as they read between and beyond the lines and make sense (Sipe, 2012). Imagination, curiosity, experiences, and feelings are engaged when readers “fill in the empty spaces” (Farrar et al., 2021). However, there is evidence that literature is not part of many pre-service teachers’ lives (Skaar et al., 2018), and there is limited knowledge of children’s literature among PSTs beyond dominant authors (Farrar & Simpson, 2024; Cremin et al., 2009). Farrar (2021) has argued that there is a need to broaden awareness in PSTs of authors and texts and of the benefits of literature and to address the limited awareness of the pedagogical benefits of children’s literature for cognitive and linguistic development and the advantages for developing an understanding of self and the world. Pre-service teachers can become ‘reading teachers’ as well as teachers of reading (Cremin et al., 2009) and be able to draw on deep knowledge of texts and understanding of the nature of reading as they develop identities as readers (not just teachers of reading) (Cremin et al., 2014).

In our English and literacy initial teacher education courses, we aim to engage pre-service teachers with children’s literature by encouraging them to interact with texts through embodied and multimodal approaches. This aim is underpinned by the theory that meaning is actively constructed as readers make textual transactions across a range of modes such as visual, linguistic, aural, and spatial (Farrar et al., 2021). While we recognise that the affordances of multimodal literacies (i.e., textual, gestural, visual, spatial, audio, and digital) are many—especially for students who may be disadvantaged or disengaged by print-only (Enriquez & Wager, 2018), we are also motivated by the opportunities that multimodal reader response affords for generative and interpretive thinking. We propose that interacting with texts through embodied and multimodal approaches enables cognitive, affective, and aesthetic engagement, which can involve readers in ‘complex and meaningful transactions’ (as above p. 22). Interpreting, creating, and analysing multimodal texts expands pre-service teachers’ capacity to engage children with the forms of literature that are an integral part of children’s lives (Brosseuk & Downes, 2024).

1.1. Transmodality, Transduction and Transmediation

Our work is grounded in theories that emphasise the role of semiotic modes in reading and writing, with particular attention to the affordances of the body, as well as sound and aural meaning-making. In adopting a multimodal perspective in our practice, we acknowledge communication, representation, and interaction to be more than written language (Jewitt, 2014), which to date, has been the dominant and highly valued mode in secondary English and tertiary education. We also acknowledge that, as Nikolajeva states: “Literature uses language to communicate, and language consists of conventional semiotic signs, based on an agreement between the bearers of a particular language and culture” (Nikolajeva, 2010, p. 147). This means that when readers respond to literature by translating and transferring meaning into new texts, they are engaging semiotically—or in other words, they are interpreting signs to produce meaning in the context of culture.

Our work is informed by Kress’s (2010) social semiotic theories of multimodality and the key concepts of transmodality, which are known as transduction and/or transmediation. Kress defines modes as sets of resources that work to communicate and make meaning within a specific social context (Kress, 2010). For something to be considered a mode, there needs to be a shared cultural set of resources and a shared cultural sense of how these resources can be organised to realise meaning (Kress, 2010). In terms of the body, there is a gestural mode that is made up of resources such as gaze, facial expressions, posture, movement, and gesture (Kalantzis et al., 2016; Bull & Anstey, 2018); the ways in which these resources are selected and configured vary across social, cultural, and historical contexts (Kress, 2010). As bodies in the classroom, and in other social contexts, exist in space and in relation to other human and non-human elements (such as other bodies and furniture), it is also important to consider the spatial mode and how resources such as levels, angles, and proximity are utilised and interpreted with respect to how bodies are configured in space and in relation to other bodies (Kalantzis et al., 2016; Bull & Anstey, 2018). In terms of sound, there is the audio mode that is made up of resources such as speech, music, and other sounds that may all be qualified in terms of pitch, duration, tone, and texture (Kalantzis et al., 2016; Bull & Anstey, 2018). As resources for meaning making, sounds are tied to the social, historical, and cultural contexts in which they are produced, combined, and heard. As music and sound unfold over a duration of time and can be overlapping and layered in composition, there is a temporal and spatial element to the aural mode (Ahern, 2020). In addition, sound can be understood as an embodied and sensory experience in the processes of both production and reception.

In the case studies that follow, pre-service teachers are engaged in transforming texts from one mode to another mode or modes. These processes of transmodality in this context are understood as processes of transduction and transmediation. Transduction is explicated by Kress (2003) to describe the process of moving meaning-material from one mode to another or the remaking of meaning across modes. Through transduction, the same topic or idea in one mode is represented through a different mode or modes at a different time (Newfield, 2014). Transmediation is a term initially used by Suhor (1984) to describe the process of translating content from one sign system to another. The term was defined by Siegel (1995, p. 383) as a ‘process of rotating the content and expression planes of two different sign systems such that the content plane of the original sign is projected onto the expression plane of the new sign system’. Since then, the concept of transmediation has continued to be taken up by scholars interested in exploring multimodality and multiliteracies (Semali, 2002; Peña & James, 2023), while transduction continues to be used by scholars in the field of social semiotics.

The salience of media in the term transmediation points to the role of different media products, media types, and media traits in the process where the transfer of meaning across modes always entails some degree of transformation (Ellström, 2017). Despite some nuances in definition or use, both transduction and transmediation are terms that emerged from the field of semiotics and are similarly relevant to the process of translating meaning across modes. In a semiotic view, sign systems or codes are manipulated by humans in ways that are meaningful to them in order to make sense of the human experience (Semali, 2002), and multimodal literacies involve fluency across different sign systems, for instance, by inferring and translating meaning from one sign system into another (Jewitt & Kress, 2008; Rowsell, 2013). While our respective use of the terms ‘transduction’ or ‘transmediation’ is underpinned by our scholarship backgrounds and experiences, each of our case studies are similarly posed here to explore the role of transforming texts in relation to reader engagement.

Previous studies of children and young people shifting meanings across modes have explored transmediation as a process of meaning making and knowledge generation (Mills, 2011; Mills & Brown, 2022; McCormick, 2011). In her study of young children who were engaged in oral retelling of a well-known literary text, drawing storyboard images of key events in the narrative, and producing digital microdocumentaries based on the same text, Mills (2011) found that the process of transmediation enabled the children to discover the possibilities of different semiotic modes for meaning making. She argues that children engage with the world through multiple semiotic systems (not only writing), and these ways of making meaning can be encouraged to enhance the way they interact with texts. McCormick’s (2011) study of upper elementary students’ transmediation of poems into other modes such as visual art or dance highlighted how the process fostered analytical conversations around literature. Children were engaged in drawing portraits in order to interpret the metaphors in a poem and then to consider an idea using the mode of writing and dance. Through these processes, pre-service teachers expanded their awareness of nonliteral meaning and of how audiences might interpret a message.

Previous studies have explored how transforming linguistic texts into visual, embodied, and/or audio texts enhances opportunities for interpretation and response. In their study of ways that the performing arts provide students with opportunities to engage in reader response, Enriquez and Wager (2018) narrate the translation of poetry into song (music), a novel into dance, and ‘Treasure Island’ into a story drama. They concluded that the transmediation process highlighted students’ interactions with literary texts and produced new meanings and interpretations to share with peers. Nash (2018) also drew on Rosenblatt’s transaction theory to explain how the translation of a challenging poem into visual images enabled readers in a secondary English classroom to make meaning through personal and intertextual connections while developing a stronger critical sense of how different modes work. More recently, in their study of transmediation with virtual reality (VR) technologies, Mills and Brown (2022) investigated how children transferred narrative content from the written word to drawing (on paper) and to virtual painting (with a VR headset). They found that VR headsets could be used to translate knowledge across semiotic modes and that this was a process of generative meaning-making. While these studies provide evidence of the value of transduction and transmediation for enabling children to engage with literature, there is a lack of study into the application of these approaches in relation to pre-service teachers. In an effort to redress this gap, the research that underpins this article brings these processes into the ITE space.

1.2. Engagement

Prior studies that have investigated pre-service teachers’ engagement with children’s literature have focused on their disinterest in reading (Skaar et al., 2018) and limited knowledge of literature (Farrar, 2021; Farrar & Simpson, 2024). Simpson (2021) has argued that the limited hours dedicated to teaching about literature in ITE programs impact how confident and prepared pre-service teachers are for teaching with literature, let alone engaging their students (Simpson, 2021). In this study, the emphasis is on how the pre-service teachers in our courses were encouraged and enabled to engage deeply with literary texts in relation to their own reading practices and with reference to pedagogical approaches. In so doing, this study supports Simpson’s argument for more time to be given in teacher education programs for pre-service teachers to enjoy and engage with literature while developing content and pedagogical knowledge of literature.

This study is underscored by a multidimensional model of engagement that includes cognitive, affective, and aesthetic engagement as intra-actional elements. While studies of reading engagement in classrooms often attend to the behavioural and social as well as the cognitive and affective (McGeown & Smith, 2024), we have chosen not to focus on the behavioural, as the study took place in a university environment, or on the social, because our data did not provide adequate evidence for this. In our reading about student engagement, the behavioural refers to the ways that children’s behaviour (how they act in the classroom) is managed by teachers, which we decided was not relevant to the higher education context. We considered data with references to social engagement, but they were not sufficient to draw conclusions, as the data sources were not designed to capture this aspect. While there was evidence of social engagement as pre-service teachers created tableaux together and collaborated on producing soundscapes, the data did not show how mode-switching impacted this type of engagement. However, we have included aesthetic engagement because of its relevance to Rosenblatt’s transactional theory of reading. The concept of reader response, as elaborated by Rosenblatt and others, enabled us to consider how PSTs were responding to the texts in sensory and aesthetic ways. An elaboration of cognitive, affective, and aesthetic engagement, as we interpret it, is provided below.

Cognitive engagement features in many other studies of student engagement in reading and/or text production (Lee et al., 2021). Cognitive engagement involves the comprehension of ideas and the use of metacognition to think about the text (Guthrie et al., 2012). Instances of strategic thinking, metacognition, reflection, and planning reflect cognitive engagement (Zinsser, 1976; Harris et al., 2009). Cognitive engagement can be reflected in the way PSTs make connections between the text and their existing knowledge, and when they reflect on their comprehension of the text (Miyamoto et al., 2019). Affective engagement is also mentioned frequently in relation to reading (Lee et al., 2021). This dimension involves the feelings and emotions that are generated during the experience of reading and interpreting the text (Cook et al., 2020). When readers are immersed in the literary text, make connections between the text and their own personal experience and use their imagination, they can be regarded as affectively engaged (Kuzmičová & Cremin, 2021; McGeown & Wilkinson, 2021). Affective engagement reflects the interest and enjoyment in the activity (see Ives et al., 2022) and the inner bodily emotional involvement and ‘felt’ responses to texts.

Our analytical model includes aesthetic engagement with reference to Rosenblatt’s concept of aesthetic reading. This concept centres on the reader’s senses and imagination as they experience the text. Aesthetic experience is seen as a perceptual dimension that emerges from practical and embodied activity (Toro & Trasmundi, 2024). It is about perception and thus involves the activation of the senses, especially vision and listening, as well as readers’ bodies as sense perceptors (Trasmundi & Kukkonen, 2024) Aesthetics involves the skill of art making in a particular form and the cultural understanding to interpret and make meaning from an art form (from Vygotsky cited in Dawson & Lee, 2018, p. 23). While aesthetics can be understood as working in and through the arts, we also consider that, as Vygotsky argues, aesthetic experiences enable students to engage cognitively and emotionally (affect) (cited in Dawson & Lee, 2018, p. 22). Reading as an aesthetic experience has been described as a transitory process involving perceptions in the moment of interaction with the text (Iser, 1978). However, it appears that these perceptions can be captured especially through imaginative and creative responses that allow readers to engage with the visual and auditory modes.

2. Materials and Methods

In this article, we refer to two case studies where engagement in transferring meaning across semiotic modes with literary texts is analysed. The first case study was conducted by Sarah in the ‘English for Secondary Teaching 1’ course with 18 pre-service teachers. This is a course for post-graduate students working towards becoming secondary English teachers. The data from the first case study presented and analysed here are taken from a 30 min segment within one 3 h long workshop on ‘Working with Prose Texts’. The second case study was conducted by Jill in the ‘Literature and Digital Media’ course, with 47 of the 105 students agreeing to participate. An information sheet was given to students, and they were asked to return the signed consent form if they agreed to participate. All students were invited to participate, and 47 returned the consent form indicating their agreement as per ethics approval processes and requirements. ‘Literature and Digital Media’ is a course for undergraduate students preparing for practice in Early Years and Primary sites. The data from the second case study presented and analysed here are taken from a 45 min activity within a 2 h workshop on ‘Soundscapes’ and a blog post following the workshop.

In each case, a literary text for children or young people was selected by the course coordinators (authors of this article) for pre-service teachers to read and transform. These texts were familiar to the course coordinators as examples of quality Australian literature that would be suitable for use in primary or secondary classrooms as literary texts. The literary text in case study one is the young adult novel ‘The Incredible Here and Now’ by Felicity Castagna (2013), which is set in the western suburbs of Sydney and, through a series of vignettes, focuses on one summer in the life of 15-year-old protagonist Michael, with themes of grief, family, and love. The extract used in this workshop was selected because of its emphasis on the relationship between Michael and his older brother Dom. The way that authors construct and represent relationships between characters in literary texts is a key aspect analysed in English classrooms in schools. While there is no dialogue in the chapter, descriptions of bodily movements work to establish the characters and their interaction. It was this aspect of the literary text—the descriptions—that the workshop particularly aimed to explore. The transformation in case study one was a tableau or ‘freeze frame’. The PSTs read a short chapter from the novel that depicts the brothers Michael and Dom spending time together; then they worked to recreate the characters and relationship using bodies in a spatial arrangement, similar to a live photograph.

In case study two, the literary text is the novella ‘Storm Boy’ by Colin Thiele (1964), which is set in the remote coastal area of The Coorong in South Australia. The novella is about a young boy, called Storm Boy, who lives alone with his father in a humble shack in the sandhills. Storm Boy saves a young pelican who has been attacked by hunters in The Coorong wetlands, and they develop a strong bond as they explore the wild beaches, sandhills, and wetlands of The Coroong. The novella was selected by the course coordinator (author 1) because it is a classic quality literary text often used in primary school classrooms in Australia. The book communicates meaning primarily through the linguistic mode and is highly descriptive. This meant that it offered opportunities for students to explore written language and engage in transmediation processes, thus making it a suitable choice for this activity. In case study two, the transformation was a soundscape. In the course, the PSTs read sections of the novella that described The Coorong, focussing on the description of the storms that frequent the region at different times of the year. They worked in pairs or groups to create a soundscape, transmediating a section of the novella using sounds to recreate the meaning of the words in aural form.

It is important to note that in both cases, the pre-service teachers had been previously oriented to concepts that informed the ways in which they were working. In Sarah’s class, the concepts of embodiment, mode, multimodality, text, communication, and specific resources of gesture and space such as gaze, facial expressions, posture, levels, and proximity, were continually referred to and applied to frame their work, and the pre-service English teachers were encouraged to consider how these might inform their own practice as educators. In this course, Sarah also used a range of drama- and body-based strategies (Dawson & Lee, 2018) that directly related to these overriding ideas and gave bodies an active and central role in the learning experience. The selected text in Sarah’s workshop offered a pedagogical invitation to use and focus on bodies and the gestural mode. The concepts and meta-language that the pre-service teachers had already learnt enabled them to reflect on the processes of transduction they were enacting. In Jill’s course, the concept of transmediation was studied as a way of generating meaning with texts through interpretive and creative response; the course aimed to engage the pre-service teachers in a close reading of the text and to enable them to explore the affordances of word and sound as different semiotic modes. The pre-service teachers in Jill’s course had learnt about transmediation in lectures and readings and were applying theories of multimodality, semiotics, and text to consider pedagogical approaches with literature and digital media in the primary English curriculum. The activity to create a soundscape based on sections of a literary text encouraged them to think about the process of transmediation as meaning-making, and to consider how this approach may enhance young people’s engagement with literature in future practice.

This study adopts a multimodal approach to data collection and analysis of the artefacts (Jewitt et al., 2016) that were produced by pre-service teachers as they engaged with children’s literature within the social and cultural context of the initial teacher education courses and workshops (Jewitt et al., 2016). For Sarah, the artefacts were the tableaux produced in response to a chapter from ‘The Incredible Here and Now’. Sarah used photographs and video to capture bodies in action as her PSTs engaged and responded to the text through tableaux. As the PSTs viewed each other’s tableaux, they shared their observations and interpretations through a reflective in-class discussion; this was also captured on video and later transcribed. For Jill, the artefacts were the soundscapes that PSTs produced as they translated a short section of the novella. Resources for sound making included ipads with Garage Band, digital sounds freely available through public websites, musical instruments, human bodies/voices, and natural objects outside of the room (the classrooms are located adjacent to a waterway and park setting). The pre-service teachers also completed a blog post about the activity where they reflected on the process of translating meaning from the linguistic to the aural mode. The instruction for the blog post was to describe the process of transmediating a section of ‘Storm Boy’ into a soundscape.

Using digital tools (Flewitt et al., 2014) to collect the data provided a record of the artefacts and allowed us to revisit them repeatedly to carry out detailed and systematic analyses (Jewitt et al., 2016) of the ways in which different modes were working to create meaning. In each case, we moved beyond verbal language in our analysis. For Sarah, this involved watching the videos on mute so that she could pay close attention to the gestural resources of the body. For closer analysis, she also produced a visual transcript by taking sequential screenshots of the video every 5 s. These still images were then examined alongside the photographs she captured of each tableau. Sarah analysed these visual transcripts and photographs by identifying specific gestural and spatial resources and interpreting how they were working to communicate the PSTs’ understanding of the literary text. The audio transcript of the reflective in-class discussion also provided insight into the PSTs’ interpretation of the multimodal elements as they viewed the artefacts live.

In case study 2, the focus was on the transfer from the linguistic mode into the audio mode. The 47 blogs were analyzed to identify how meaning was being transferred and translated across the modes. This analysis focused on what the participants said they did, what they said about how they composed the soundscapes, and what decisions were being made as they composed the audio text (see

Table 1. Method of analysing the blog data.

Focus of Investigation	Further Details
What the pre-service teachers said they did	This category included instances of selecting, reading, and interpreting excerpts from the ‘Storm Boy’ text and identifying words and phrases within the selected excerpt, as well as translating those words into aural signs
What the pre-service teacher said about how they composed the soundscapes	This category included instances of selecting and combining aural signs into a cohesive aural text
What decisions were being made as they composed the soundscape	This category included instances of decisions about volume, pitch, texture, tone, and rhythm, as well as decisions about way that the soundscape was designed

).

Soundscapes produced by participants were selected based on how they demonstrated what had been found in the blog data. These soundscapes were analyzed as exemplars of semiosis through the sign systems of volume, pitch, texture, tone, and rhythm. As sounds exist in temporal and spatial contexts, it was important to consider the way that sounds were composed and interpreted across time and space to create meaning. Thus the use of the sign systems listed above was plotted horizontally across time and vertically across space.

Our analysis was informed by the understanding that the ways in which modal resources are selected and configured vary across different contexts, and the ways in which these are interpreted are linked to the socially and culturally understood significations of modes and resources (Kress, 2010). In our analyses, we paid attention to the ‘transmodal moment’ (Newfield, 2014) by examining the multimodal artefacts produced by our pre-service teacher in relation to the original literary text being read. In doing so, we looked for examples of how meaning in the linguistic mode of the novels was re-realised through a different mode in the respective artefact. This allowed us to consider the ways in which different modes and modal resources were being used by the pre-service teacher(s) to realise their different interests and perspectives in relation to the source texts (Newfield, 2014). In both cases, as well as analysing the ways in which meaning was transferred across modes, we also analysed what this revealed about the pre-service teachers’ engagement with the literary texts being studied. The key semiotic resources analysed in the case studies were expressed by bodies, in spaces, and with sounds. The questions guiding our analysis were:

• How did pre-service teachers engage with transferring meaning across modes?
• How did this mode-switching develop and demonstrate pre-service teachers’ engagement with literature?

3. Results

3.1. Case Study One

3.1.1. Transferring Meaning Across Linguistic and Gestural/Spatial Modes

In case study one, Sarah’s pre-service teachers engaged in translating meaning from the linguistic mode of the novel to the gestural/spatial modes of the tableau. They began by observing and analysing the gestural and spatial mode in the image on the front cover of a young adult novel ‘The Incredible Here and Now’ (Castagna, 2013), noticing and discussing features such as direct eye contact, close proximity, and the angling of the head. By selecting a text that featured a body prominently on the front cover, Sarah was able to draw the pre-service teachers’ attention to the role that specific bodily resources have in creating meaning, using metalanguage such as gaze to name these as part of our analysis. This was enhanced further through the high modality (Callow, 2023) of the photographic image and its naturalistic, lifelike presentation of a human. These initial ideas from the reading of the body on the front cover supported the PSTs’ further predictions of character from the back cover blurb. From this initial book orientation, they established some key interpretations about the relationship between the two brothers and the power dynamic between them.

Sarah then read aloud a short character-based chapter that shows the protagonist, Michael, and his older brother, Dom, hanging out at night in a chicken shop in the western suburbs of Sydney. As the teacher read, the PSTs highlighted words or phrases in the chapter that revealed something about the two brothers and their relationship with each other. As there is no dialogue in the chapter, they focused on how descriptions of bodily movements and interactions in the linguistic mode work to establish characters and relationships. Following the teacher’s read-aloud, the PSTs worked in small groups to create a tableau, considering how gestural and spatial resources (such as gestures, facial expressions, body language, and levels) might be used to represent power in the relationship between the two brothers in this chapter. In the process of creating their tableau, the PSTs referred to the text and shared the passages they had highlighted as they experimented with different modal resources to shift meaning from the linguistic to the gestural and spatial mode (see Figure 1).

3.1.2. Cognitive Engagement

The pre-service teachers read deeply and noticed specific details in the text as they worked to represent literal meanings in the text through gesture and space as indicated in the annotations in Figure 2. For example, most groups chose to portray Dom as he is described at the end of the chapter: ‘standing’, ‘side-on’ with ‘one hand on his hip’ and ‘the other holding up his chicken sandwich’ (Castagna, 2013). This is an example of transduction, as the meaning in the source text is carried over to the tableaux, and there is a shift from one mode to another (see Figure 2).

The pre-service teachers also read for inferential meanings in the text. For example, Dom’s power as the confident older brother is never explicitly stated in the chapter but is implied through Michael’s descriptions of admiration, such as ‘Dom is the chicken sandwich master’ (Castagna, 2013), and the comparisons he draws between Dom and himself: ‘He eats the whole thing in three mouthfuls…Me, I’ve got garlic sauce rolling down my arm before the food even gets to my mouth’ (Castagna, 2013). These inferences are represented gesturally through Dom’s open body language, straight back, and head held high (see annotations of gesture in Figure 3) whereas Michael is sat down with closed body language (see annotations of gesture in Figure 4).

From a social semiotic multimodal perspective, it is important to consider how some of the gestural resources being used in these tableaux have been used over time and how we bring our social and cultural experiences to bear as we create and view texts (Kress, 2010). The difference in levels, for example, does more than show that Dom is standing and Michael is sitting; it signals the authority and status of the older brother (see Figure 2, Figure 3 and Figure 4). Similarly, the fact that Dom is shown to be mid-movement and Michael stationary and slumped is a literal transduction of the explicit action of the source text but also creates the implied meaning that Dom is more dynamic and in control, and Michael is more awkward and self-conscious (see Figure 2, Figure 3 and Figure 4). Furthermore, gaze in the tableaux does more than show that Dom is looking at the car outside and Michael is looking at him; it demonstrates the different interests and motivations for the two characters: Michael towards Dom, Dom away from Michael and towards the car—this is further emphasised through the angling of bodies in the tableaux (see Figure 2, Figure 3 and Figure 4).

3.1.3. Aesthetic Engagement

In their work, the pre-service teacher initially had to make meaning from the original art form of the novel (through literal and inferential comprehension) and then create their own work of art in the form of a tableau. In doing so, they had to create a new text, drawing on their understanding of how the available resources for that art form might be selected and configured to make meaning. They then had to view each other’s tableaux, which involved them making meaning and interpreting this new visual and gestural art form (Dawson & Lee, 2018).

An example of aesthetic representation can be seen in one group’s symbolic interpretation of the chapter through tableau (see Figure 5). In analysing this tableau, the concepts of materiality and affordance from Kress’s (2010) social semiotic multimodal perspective are significant. Materiality refers to the modal resources themselves, and affordance refers to the meaning-making potentials of the resource and how these have been shaped by social and cultural use over time (Kress, 2010). In terms of materiality, we see in this tableau (see annotations of gesture in Figure 5) the body positioning of kneeling and the gestures of raising arms, shielding eyes, and kissing feet. These gestural resources all have cultural affordances of worship, and this is emphasised when they are configured together aesthetically in this way.

The meaning here is less literal and much more figurative. This group drew on the cultural affordances of those gestural resources to intentionally convey the meaning they took from the chapter: that Dom is worshipped by his younger brother. The reflective discussion of this tableau revealed interpretations just like this, with viewers commenting: ‘He’s a god’, ‘worshiping’ and ‘glory’, and explicitly identifying the gestural resources that created that meaning for them, such as ‘hands are up’, ‘praying’, and ‘open body language’ (see Figure 5).

As we view and interpret the tableau, we draw on other affordances to make new meanings. For example, the turn of the head, the gaze away, the straight and tall posture, and the hands on the hips might suggest arrogance or indifference to the admiration. The kneeling and cowing might suggest weakness or even fear (see Figure 5).

3.1.4. Affective Engagement

Creating the tableaux also helped the pre-service teachers engage affectively with the text as they used facial expressions, gestures, body language, and posture to understand and reveal character emotion. In the reflective discussion, the PSTs stated that the activity ‘puts them in the scene’, commenting that ‘when you’re asked to…make a representation of it, you really have to get in there and actually figure it out’ and that reflecting on tableaux is ‘a good way to get a connection with all the characters in the story’.

They also reflected that through the process of embodying and transforming the text through the gestural mode, they came to know it in a more affective way: they ‘felt’ the experiences of the characters and situations emotionally. For example, those embodying the character Michael commented on how they felt ‘so needy’ and ‘very submissive’, and those embodying the older brother Dom stated how they ‘took on some of his power, some of his feelings’ and ‘retain[ed] some sort of sense of power’ through being the ‘centre point’ and ‘focal point’ of the tableaux. Others recalled feeling ‘emotional’ in their embodied roles and noted, ‘when my position changes, my feelings changed’, and another reflected that the process of transduction through embodiment ‘really facilitates getting students to be empathetic towards the characters’.

3.2. Case Study Two

3.2.1. Transferring Meaning Across Linguistic and Audio Modes

In case study two, pre-service teachers engaged in translating meaning from the linguistic mode of the novella to the aural mode of the soundscape. Pre-service teachers engaged with transferring meaning across linguistic and aural modes by reading the text closely to identify specific words and phrases. They identified noun groups such as ‘rising sun’, ‘tallbirds cheering’, ‘pelicans running across the water’, and birds clapping. They identified adverbial phrases such as ‘happy splashing’, ‘gurgling and gargling’, ‘flapping wings furiously’, ‘diving into water’, and adjectives such as ‘shrieking’ and ‘howling’. Readers underlined and highlighted words in sentences and short passages, such as the adverbs and nouns in this section:

a dark storm came towering in heaving and boiling over Kangaroo Island…swept down towards them with lightening and black rain…flying cloud and gloom.

Words and phrases were identified in this way to inform the composition of the soundscapes. Adjectives and adverbs such as dark, rough, heavy, towering, loud, windy, and frightening were used as ‘the basis of my soundscape’. Paragraphs such as the one below were selected to ‘capture the essence of what the book is describing at that point in time’.

In July the winds lost their senses. Three great storms swept out of the South, the third one so terrible that it gathered up the sea in mountains, mashed it into foam, and hurled it against the shore. The waves came in like rolling railway embankments right up to the Sandhills where Hide-Away and Storm Boy lived. They lashed and tore at them as if they wanted to carry them away. The boobyalla bushes bent and broke.

Pre-service teachers engaged with transferring meaning across linguistic and aural modes by recreating particular sounds such as thunder, rain, wind, birds, waves, and rain.

To start with I choose to start my storm off with just the strong wind sounds as it says within the second sentence. Within the wind you are able to hear the seagulls and slowly, gently the iron sheets banging in the wind.

The transfer of meaning was augmented by using a range of aural codes and resources, such as volume, tone, intonation, dissonance, tempo, dynamic, texture, and timbre.

I started the sound of the wind quite quietly and made it so that it became louder as the storm ‘rolled in’. I then found multiple different clips of rain and added them in. To evoke the chaotic and powerful nature of the storm, I layered sounds of wind, rolling waves, thunder, and rain.

Figure 6 below represents the layered composition of a 42 s soundscape showing how the selected sounds were placed in a series across time and space. The shaded areas show the duration of the sound, and the darker shades represent louder volume and intensity.

Specific details were noticed in Thiele’s descriptive language and used to guide the transfer into aural mode. The ‘distant cry of birds’ informed the decision to layer the seagull sounds within the wind sound. The ‘eerie clang of metal’ informed the decision to bring in ‘slowly gently the iron sheets banging in the wind’. Specific details that described the way the waves, wind, people, and thunder sounded were noticed and used to generate specific types of sounds. Thiele’s text describes the sound of the waves as ‘ominous, roaring’ and ‘crashing’ and the people’s yells as ‘echoing’. The wind is ‘howling’, and the thunder ‘cracks’. These evocations of sound, or aural imagery, were used to select sounds and translate them using aural codes in interpretive compositions.

3.2.2. Cognitive Engagement

Cognitive engagement was reflected where pre-service teachers explained how they had inferred meaning as they interacted with the words of the text. A pre-service teacher said they had ‘unpacked’ the text (see Figure 7) to make sense of it as a series of sounds—‘loud’, dramatic’, chaotic’, ‘loud growling and rumbling’. The way they thought about the text was guided by the process of transmediation so that they focus on the sounds that might be inferred from the words. Thinking was also evident in this process of creating the soundscape as pre-service teachers were enabled to ‘delve deeper’ and engage imaginatively while transferring words to sounds—initially as a cognitive process and then in a materialised audio mode.

This activity made me think about how the sounds of rain, thunder, wind and trees moving can guide the audience into the clear conclusion that there is a storm without having to visually see it.

In their blogs, the pre-service teachers wrote about their reading as visualisation. The process of transferring meaning from the linguistic and aural modes was seen to enhance their visualisation or envisaging of the book. Visualisation is a key comprehension strategy that assists readers in making sense of written texts and being able to visualise the settings, character, and actions in a literary text helped readers to engage with ideas, feel an emotional response, and follow the story. They commented that the soundscape allowed them to ‘visualise and experience the events at a particular point in a story’ and to ‘envision the diegetic sounds of the whistling gusts of winds and thunderous waves to encapsulate the distressing themes and the chaos of the water. Participants noted that the soundscapes allowed them to imagine ‘they are immersed in the storm itself’ by ‘visualis(ing) the context and severity of the storm on a deeper level’.

Creating the soundscape enabled me to delve deeper into imaging these storms, which helped me better understand the chapter in Storm Boy.

3.2.3. Aesthetic Engagement

Pre-service teachers engaged with the aesthetic dimension when they responded to the text and the soundscape design in a contextual and wholistic way that showed appreciation for the ‘essence’ of the written passage. This was evident in their creative perception of the text, where they connected to the themes and atmospheres with a sense of the effect on themselves and a possible audience. Engaging with their senses, PSTs wrote about ‘envisioning … the sounds’ and about the ‘feelings we associate with storms’ as they composed their soundscapes. Sounds were associated with the participants’ imagined atmospheres, themes, and events in the book. Thus, ‘whistling gusts of winds and thunderous waves’ were identified ‘to encapsulate the distressing themes and the chaos of the water’. And ‘echoing, eerie sounds of the men crying for rescue in a slowed and reverbed approach’ were chosen ‘to create a dooming effect: depicting the terrifying event’.

The affordances of the media tools used to create the soundscapes were activated to represent, replicate, and enhance the aural elements of the text. This necessitated engaging with different ways (digital and analogue) of making, recording, and layering sounds, such as using Garage Band and a grand piano.

For example, in the novel whilst describing the storm Colin Thiele wrote, ‘The boobyalla bushes bent and broke.’ (Thiele, 1964, p. 55), and to represent this sound in our soundscape, we recorded bushes moving. Next, we used the deep notes on the grand piano shown in Figure 1 to replicate thunder to add more depth to the storm soundscape. Using the sound effects on the keyboard, we were able to make an eerie sound, which enhanced the effect of the storm.

The process of transmediating the words into sounds created an opportunity for pre-service teachers to engage aesthetically with the book—as they were encouraged to ‘think about literature in new ways’. One participant commented that: ‘Instead of just reading and analysing the text on paper, we had to consider how to convey the story’s themes, emotions, and atmosphere through sound’. They were involved in discussing the plot and characters while also considering ‘how music, sound effects, and our own voices could bring the story to life in an auditory format’. Moving across the modes of words and sounds encouraged response to the text in a wholistic way—themes, emotions, and atmospheres (as well as events)—and in imaginative ways—to ‘bring the story to life’ and ‘paint pictures in the listener’s minds’.

Another PST made links between the descriptive language and the central character, Storm Boy.

In chapter 1 of the novel the author is describing how Storm Boy got his name, discussing him being spotted calmly wandering down the beach during a storm that is described as ‘a dark storm’ that ‘towers in from the west during the day, heaving and boiling’ and ‘swept down towards them with lightning and black rain’.

Thus, there is evidence of close reading, with a focus on descriptive language and consideration of the setting in relation to character.

The selection of words and phrases from the book to inform the selection of sounds for the soundscape was followed by the process of composition, where sounds were manipulated and layered. Pre-service teachers wrote in their blogs about the process of combining and layering sounds as ‘an immersive experience, capturing the intensity and uncontrollable force described in the text’, and as a process where ‘each sound was combined’ and ‘coexisted to create a gloomy impact on how the audience may feel and perceive the event’. One PST commented that they had built the intensity of the story as they ‘layered the sounds together’.

These responses show how PSTs were engaging with the novella by understanding how the prose text works to create atmosphere and mood—such as engaging with the ‘intensity and uncontrollable force described’—and how their own aural text translates that mood: ‘a gloomy impact’. The combination and layering of sounds was an important technique to create cohesion, such as in the example in Figure 8:

3.2.4. Affective Engagement

Affective engagement was evident when PSTs reflected on the feelings and sensations that the words evoked for them, and how this was translated into their soundscape. Interpretations of the mood evoked by the text was reflected in language such as ‘ominous’ and ‘dooming effect’ and a ‘sense of threat or tension’. These interpretations show engagement with the way that the text is interpreted emotionally. Some PSTs commented on the affordance of the audio mode to affect the reader’s experience. In this comment, pre-service teacher reflects on sound elements as part of the reading experience—including here the use of voice in reading the text and the use of music.

This audio clip is used to enhance the mind of you, the reader. Allowing you to create meaning from the audio and sentence combine!

Problem solving how to create the sounds for the soundscape together generated thinking about the type and quality of sound that replicated the text. Engaging in this task with other readers generated creative thinking and ‘a lot of fun’. This blog frames the activity as a kind of enjoyable challenge and adventure where they ‘ventured outside’, ‘roamed’, and experimented with analogue sounds and ‘even using our voices’.

We used a program called Garage Band on the iPad. First, we attempted to use the instruments available on the app to replicate sounds such as wind, waves crashing and thunder. It was a bit of a struggle to get the right sound effect, so we put our thinking caps and decided to venture outside for some inspiration.

We roamed around outside, recording sounds like the mini river flowing, the trees rustling in the wind, and even using our voices to mimic the sounds we imagined would be present during a storm. It was a creative challenge, but we had a lot of fun experimenting with different sounds!

Collaborating in small groups or pairs to create the soundscape enabled them to read the text together, discuss their interpretations and compose a meaningful text in the audio mode. Engaging with other pre-service teachers extended their understanding of the story and enhanced their relationship with the narrative as they brought the text to life. This was part of the cognitive and aesthetic engagement with the literary text and with the creation of the new audio text.

(It) allowed us to think about literature in new ways. Instead of just reading and analysing the text on paper, we had to consider how to convey the story’s themes, emotions, and atmosphere through sound. This involved not just discussing the plot and characters, but also thinking about how music, sound effects, and our own voices could bring the story to life in an auditory format.

4. Discussion

This study has explored how undergraduate and post-graduate pre-service teachers engaged with children’s literature in two of their English/literacy education courses. The study drew on theories of social semiotics and multimodality to investigate the processes of meaning making within and across modes. These theories provided a framework through which we could analyse the reading of literary texts as a transaction, where the experiences and knowledge of the reader is brought into interaction with the semiotics of the text to make meaning. The concepts of cognitive, affective and aesthetic engagement were adopted to analyse what it meant to be engaged with children’s literature and to consider how the processes of transmodality enhanced the interaction.

A key finding was that the process of transmodality enabled the pre-service teachers to make inferences, think imaginatively and make meanings across modes. It was evident that they were able to engage closely with the words and phrases in the literary text to make inferences about characters and settings, which they then translated into newly created texts. The process of transfer and transformation encouraged imaginative thinking as pre-service teachers envisioned and materialised relationships, moods and environments. These cases show how readers can respond to texts through their bodies, senses and with the material resources of the gestural, spatial and audio modes (Kress, 2010). It also shows how the affordances of these resources enable new meanings to be made in different ways.

The process of transforming texts enabled these pre-service teachers to bring conscious awareness to the literary elements of the novels being studied and encouraged the development of a critical appreciation of the affordances of other modes through the creation of a new multimodal text in response to a literary text—for example by choosing particular aural textures, building or decreasing the volume, layering in new sounds or varying the speed and rhythm. Through the process of transduction or transmediation the PSTs were able to play with literal meanings as they transferred words into gestures or sounds and explored their interpretations together. Creating the tableaux brought a focus on character and relationships—for example through interpreting Michael’s character through the gestural mode and using levels to show a power differential between the brothers. Composing the soundscape brought a focus on setting and mood—for example by re-realizing words into sounds (wind, seagull, waves) and by making decisions about volume, pitch, tone and pace. Recreating relationships and moods in different modes forced appraisal of the affordances of the gestural/spatial and audio modes. Through these cognitive, affective and aesthetic engagements, PSTs were guided to learn how meanings are made by readers with the authors’ words, and how these meanings can be recreated with gesture/space or sound.

While cognitive, aesthetic and affective engagement were evident in both case studies, these facets were interwoven and interconnected. Pulling them apart for the purposes of this analysis revealed how cognitive, aesthetic and affective engagement overlapped and merged. It was evident that interpretation and inferring meaning was connected to the way the words of the text were being imagined and recreated. Embodying the characters and feeling the moods of the storms was at the same time affective and aesthetic engagement. Translating the words and paragraphs into soundscapes and tableaux enabled these pre-service teachers to think about the power of visualisation in reading literature as a way of both comprehending and enjoying text. The transmediation process supported readers to make connections between the language that describes sound (sound imagery) and the thematic moods being created in the context of the story. It enabled the PSTs to engage imaginatively and encouraged immersion into the world of the text. It is evident that the transmediation process brought ‘the words to life’ and ‘made the text come alive’.

As the PSTs reflected on their own textual transformations and on one another’s created texts, they went through yet another process of transduction (Kress, 2010) creating a chain of text- and meaning-making (Newfield, 2014). In case study one, this chain can be seen as meaning was shifted from the original novel in the linguistic mode, to the gestural and spatial tableaux and then to the spoken linguistic mode through the shared class discussions. In case study two, meaning was shifted from the linguistic novel to the audio mode of the soundscapes and then into the written linguistic mode of the blogs. This ongoing process of transduction (Kress, 2010; Newfield, 2014) generated new meanings that enhanced the PSTs’ understandings of the original literary text and provoked further discussion of the character and relationship and setting and mood respectively. Through this continual mode-shifting the PSTs came to know each original text on a deeper level, and the process of reading and responding process was complex and multi-layered (Newfield, 2014).

5. Conclusions

These case studies have much to offer in terms of pedagogical practice in initial teacher education and in school classrooms. Through these experiences pre-service teachers can consider the practical and theoretical benefits of working this way with their own students. This experience has implications for these pre-service teachers in terms of their practice as secondary or primary English teachers. They noted the benefits of working this way with their own students, commenting on how the activity necessitates close reading and attention to detail. Depending on which course they were part of, the pre-service teachers noted how embodiment puts you in the scene and facilitates empathy towards the characters, and how making soundscapes brings literature to life and helps readers to visualise the text. These experiences enabled them to enjoy and learn more about children’s literature and to understand more about the way that linguistic, visual and audio modes are composed and interpreted—experiences that will help them to inspire young readers to engage cognitively, affectively and aesthetically with literature.