Next Article in Journal
Representing Time-Dynamic Geospatial Objects on Virtual Globes Using CZML—Part I: Overview and Key Issues
Next Article in Special Issue
On the Use of Geographic Information in Humanities Research Infrastructure: A Case Study on Cultural Heritage
Previous Article in Journal
Improving the Separability of Deep Features with Discriminative Convolution Filters for RSI Classification
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Concept Paper

Once upon a Spacetime: Visual Storytelling in Cognitive and Geotemporal Information Spaces

Department for Knowledge and Communication Management, Danube University Krems, 3500 Krems, Austria
*
Authors to whom correspondence should be addressed.
ISPRS Int. J. Geo-Inf. 2018, 7(3), 96; https://doi.org/10.3390/ijgi7030096
Submission received: 24 January 2018 / Revised: 20 February 2018 / Accepted: 7 March 2018 / Published: 12 March 2018
(This article belongs to the Special Issue Storytelling with Geographic Data)

Abstract

:
Stories are an essential mode, not only of human communication—but also of thinking. This paper reflects on the internalization of stories from a cognitive perspective and outlines a visualization framework for supporting the analysis of narrative geotemporal data. We discuss the strengths and limitations of standard techniques for representing spatiotemporal data (coordinated views, animation or slideshow, layer superimposition, juxtaposition, and space-time cube representation) and think about their effects on mental representations of a story. Many current visualization systems offer multiple views and allow the user to investigate different aspects of a story. From a cognitive point of view, it is important to assist users in reconnecting these multiple perspectives into a coherent picture—e.g., by utilizing coherence techniques like seamless transitions. A case study involving visualizing biographical narratives illustrates how the design of advanced visualization systems can be cognitively and conceptually grounded to support the construction of an integrated internal representation.

1. Introduction

Stories surround us in many aspects of our daily lives—conversations, news reports, series, films, and books are only some of the most obvious examples. Cultures are flush with thousands of narratives, whether real, fictional, or hybrid; they organize knowledge about our collective past, document our contemporary experiences, emotions, hopes, fears, and visions, and sketch out possible ways into intended futures. Cultures inherit stories and invent new ones, which are collected and shared in everyday communities; they are studied and interpreted by arts, humanities, and history scholars. Stories are used to communicate and present complex contents effectively, and are taught in journalism, media training, and rhetoric seminars.
Geotemporal stories that more or less centrally tell a protagonist’s movement through a geographic space can be found in many different narrative genres. Prototypical examples are travel reports, quest narratives, or biographies. One historical case where movement is highly central in the story is Homer’s Odyssey, as the geographic positioning and the timespan of Odysseus’ stops during his journey are still a topic of discussion (Appendix A). Visualization (cp. Figure 1) can help readers to connect the plot to familiar places and to better understand the time course of the events.
It is not surprising that storytelling has also become a topic of interest in information visualization. The corresponding crossover genre of “narrative visualization” or “visual storytelling” has been explored and elaborated extensively during the last years [2,3,4,5,6,7,8,9,10].
These works developed and discussed a whole panoply of solutions, such as how to bring the framing methods of diagrammatic pictures and stories together—to the mutual benefit of both. Visualization designers reconsidered “author-driven” design elements and strategies (like sequential guidance, ordering, or messaging), which can enrich the usually “reader-driven” reception of visualizations to varying degrees [5]. Since this “narrative turn,” developers of visualization systems have been able to choose more consciously whether complex information should be designed as open installments to be freely explored by the users, or whether they should be guided through these worlds by predefined sequences and narrative paths [11]. However, these reflections were mostly restricted to narrative communication patterns and the visual design space, and did not address the cognitive foundations of storytelling.
What is it that makes storytelling such a powerful mode for processing information and communicating it to experts or general audiences? This paper approaches this question first from a cognitive science perspective, to better understand how narratives are processed and internalized (Section 2). In a second step, it turns to the question of how to support the visual analysis and comprehension of stories by different methods of spatiotemporal information visualization (Section 3). Building on these options, we introduce a case study and a visualization framework that puts emphasis on the cognitive integration and on the coupling of multiple perspectives (Section 4), and we outline options regarding how to further advance this visual storytelling environment (Section 5).

2. Cognitive-Scientific Foundation

The main reason for the cultural prominence and omnipresence of stories is that narration is not only a mode of presentation, but “a fundamental way of organizing human experience and a tool for constructing models of reality” [12] (p. 345) in everyday life. Bruner [13] proposed a “narrative mode of thought” (p. 97) that helps one to construct internal representations about events, human intentions, and actions in the world. The events and actions described in stories closely correspond to everyday sequences and experiences and, therefore, their representation and comprehension is more natural than of other types of information (e.g., descriptive) [14]. Questioning the causes of events and the intentions of actors and organizing incoming information within such chains of events is central to human thinking and sensemaking.
How can we define a story—or rather, a narrative—as it is termed in cognitive science? Wilkens et al. [15] (p. 324) define a narrative as “a chain of events related by cause and effect occurring in time and space and involving some agency.” Following this definition, a story consists of five central information elements: (1) multiple events, (2) a time frame, (3) a space, (4) involved actors or objects, and (5) causal relations between them (Appendix B).
On a cognitive level, story schemata are available concerning how a story is built, how it progresses, and what its constituents are [16], which reduce cognitive load and allow stories to be processed fast and efficiently. Guided by these schemata, the recipient of a story picks up narrative cues, relates the information to scripts and prior knowledge, and actively builds up an internal representation of the events and the involved actors, which results in the so-called “situation model” [17]. How this situation model is actually constructed and what its cognitive constituents are comprise a matter of complex discussions, which will be elaborated further down (see also Figure 3). The situation model is relational and multidimensional in nature and is continuously updated as the story unfolds (Event-indexing model, [18]): Each new event occurring within a story becomes cognitively assigned or connected to a time frame of the narrative, to a space (the scene or location where it happens), to one or multiple causes (the prior event(s) that influence(s) the event), the protagonist(s) actively or passively involved, and to the way the event relates to the protagonists’ goals or intentions. Figure 2 shows a conceptual draft of the event-indexing model, visualized as a time-oriented semantic graph (from left to right).
Depending on how much of these situative indices an event shares with prior events, it can be integrated into the situation model relatively easily [19]. During the construction of a situation model, story recipients try to establish coherence on a local level (no inconsistencies and contradictions between two following events) and on a global level (within the whole story [14]). The recipient aims to close existing coherence gaps or breaks (inconsistencies within one or multiple dimensions) within the story by not only drawing inferences between the pieces of information, but also with prior knowledge. A story can be better understood if it is coherently designed; that is, if it is told consistently, does not contradict itself or miss information within the chain of events, and if its global architecture is plausible with regard to the interplay of the five threads.
But, how are stories (or narrative data in general) internally represented? For multimodal information processing and learning in general, a bi-modal model has been developed by Schnotz [20]. Its basic assumption is a dual layer architecture: Based on Paivio’s dual-coding theory, Schnotz suggested that multimodal information is processed in parallel in (1) a verbal–propositional and (2) a visual–spatial system and leads to the construction and elaboration of multiple internal representations, which can be transferred to one another and are closely connected in an active process that generates a coherent knowledge structure (see Figure 3). Building on this model, we assume that stories—as information in general—are internalized and represented somewhere in these different modes: either verbal or visual, but mostly in a bi- or multimodal fashion. Visual–spatial and verbal–propositional information are drawn together to construct a coherent situation model of the story.
For the geotemporal visualization of stories, two story indices are of specific interest for us: space and time. Though the other indices (intentions, actors, causation, and events) are also highly relevant, they are not the focus of our paper. We will shortly elaborate on them in Section 5.1.

2.1. Space

Space is regarded a difficult dimension of stories, as its nonlinear nature does not match the linear sequence of events [19]. A map-based representation of a story can assist the construction of a spatially structured situation model, which integrates details about a story’s locations [21]. Still, the typical mental representations generated from maps and from stories differ. Tversky [22] postulated that the internal representation of a spatial environment is not a coherent image of the external representation. Rather, recipients only selectively internalize information cues that are relevant to them (and the current task), which may or may not be related to other pieces of information, and which are likely to contain different perspectives. These “cognitive collages” are not coherently organized and, therefore, do not allow perspective-taking, reorientation, and spatial inferences, compared to a fully integrated cognitive map. Rather, the information pieces are related by the spatial relations between them and to larger units or landmarks [23], to personal experiences, and to frames of reference [24]. In contrast to situation models, a cognitive map (or collage) is nonlinear (except maybe for the representation of a spatial path), noncausal, less coherent and less structured.
Therefore, an important question for storytelling with maps is how stories are internally constructed and represented on a cognitive level. Kosara [25] (para 1), emphasized that “pictures don’t tell stories, people do. An image, a visualization, data, etc., can only be the material the story is made from.” The recipient of a narrative visualization plus his or her prior knowledge, intentions, and interests decide what information is attended to and what is internalized. Whether he or she generates a sequential situation model, a cognitive map, or some partial or hybrid form cannot be prescribed by the geovisualization, even though some design cues can help recipients to generate a better integrated, more coherent internal representation.

2.2. Time

Time is one of the most important dimensions of a situation model, as temporal information helps the recipient to establish causal and motivational links between events [19]. Similarly, Kosara and MacKinlay [8] argued that the temporal structure of stories is a fundamental feature for storytelling with visualizations. Therefore, we may assume that sequential and chronological aspects dominate (or strongly structure) the mental representations of stories (or situation models), which goes together perfectly well with the time-oriented nature of speech or reading text.
Yet, if the comprehension of the chronological character of complex stories should also be supported visually, time must be visually encoded and represented, which requires design choices that are far from trivial. “Time presents specific challenges for the representation of data because time is a complex and highly abstract concept” [26] (p. 203). One of the most frequently used solutions is to map time to a spatial (i.e., linear or sequential) dimension of the pictorial space, which often results in timelines [27,28,29,30], or also in linearly juxtaposed representations of sequential art [31,32,33].
Visual cognition support for the comprehension of stories and narrative data often leads to the utilization of either timelines or maps (or a combination of both), which organize the pictorial space—and, thus, the users’ modes of thought—quite differently [34]. Maps represent data that are already spatial, while timelines “spatialize” the abstract concept of time, utilizing the pictorial space in a radically different way [35]. This presumably leads to the construction of two different internal representations, which require additional mental effort to synthesize. To help the recipients generate a situation model of the story that integrates geographic and temporal data more closely and supports cross-dimensional reasoning and perspective taking, a combined or hybrid visualization method is required. Therefore, in the following section, we discuss hybrid visualization techniques that integrate geographical and temporal data aspects—and thereby, facilitate visual storytelling in a spatiotemporally synoptic fashion.

3. Spatiotemporal Visualization Methods

How can we represent narrative data visually on the screens of visualization systems that aim to augment and amplify reasoning with these kinds of data? Numerous solutions to visualize spatiotemporal aspects of narrative data have already been studied and analyzed in the literature. To illustrate some of these techniques, we will zoom in on the example of individual movement data, which consists of “recording the location of a moving point object through time” [36] (p.183). Such data have also been called a “mobile trajectory,” “world line,” “life path,” “space–time path,” or “spatial history” [36]. The benefit of focusing on individual movement data is its elementary nature, which allows the construction of a great variety of story types or narrative forms by combinatory means. Even if individual biographies or space–time paths of objects and actors are only one basic type of spatiotemporal data, almost every other type of narrative utilizes them to draw them as brush strokes into a more complex and intertwined choreography.
With regard to individual movement data, cartographers have introduced multiple methods and means for representing movement, dynamism, and change [37,38], including static and animated maps, space–time cubes, and coordinated linked views (e.g., maps with timelines). In the following section, we take a closer look at a selection of five of these standard techniques, which have been widely discussed and are available in various visualization tools and geo-analytical packages (for a video demonstrating these visualization techniques in the PolyCube system, see https://youtu.be/PTsk-NROJhU). Figure 4 lists these common perspectives, each visualizing spatial and temporal data aspects in a hybrid fashion.
Each of these spatiotemporal representation techniques requires different affordances for the construction of a corresponding internal representation (see also [26]) and, thus, also for the construction of the story situation model. In the following section, we are going to discuss each of them from a cognitive perspective and what they mean for the integrated perception and understanding of a story’s spatial and temporal dimensions.
(a)
Multiple coordinated views combine a standard map with a time graph to visualize spatial and temporal data aspects in parallel [39]. This method utilizes separate representations for the locational and temporal distributions of story events and usually coordinates these views via linked interaction methods, e.g., allowing for linked brushing [40]. For multiple coordinated views, it is likely that users generate two separate internal representations (one for each view) with some links to one another. Depending on the visual work and interaction a user invests to bridge this split of attention [41], these links can be relatively densely knit.
(b)
Animation or slideshows (also “dynamic representations” [42], p. 6) map narrative time orientation to the time dimension of the visual representation [26]. As such, they can represent the movement of objects or actors as a continuous dynamic (i.e., as smooth animation) or as a discrete sequence of steps, which we refer to as a slideshow. These techniques can further be implemented as non-interactive or interactive representations [43] (p. 1588), allowing users to go back and forth in time. It is well known that animation can foster the perception of even subtle changes or display dynamics, but also that the user’s working memory is easily overwhelmed when too much information changes too fast [26]. If the visualization is more complex, a slideshow might be better suited, which reduces the temporal continuum to discrete intervals. Users then can interactively go back and forth from one story event to the next. Still, considering one’s visual view and comparing it to the next one is very demanding for the working memory and increases interaction costs by repeatedly going back and forth.
(c)
Layer superimposition techniques merge multiple temporal positions—or temporal layers—into one integrated representation, while using transparency to see all positions at once [32]. Time is mostly encoded with an additional retinal variable, like color, or with the annotation of temporal values or vectorial references, signifying a temporal sequence of positions in space. In Figure 1, the map on the left-hand side uses a numerical sequence and arrows to encode the time orientation of the narrative. By using such a technique, users can build up a spatiotemporally integrated internal representation of the story. Aside from the expected challenges posed by visual clutter and occlusion, a concern regarding this technique is whether time (e.g., encoded by a color scale) is visually salient enough to be as well integrated into the story situation model as geographic space.
(d)
Layer juxtaposition separates spatiotemporal data into multiple temporal layers, to arrange these layers in parallel—mostly along a spatial reading dimension. This results either in “small multiple” maps [43] or, more generally, in the hybrid genre of data comics [33,44]. In face of juxtaposed views, the user must sequentially read and compare multiple adjacent views to detect the visual changes and comprehend how the story unfolds over time. Though a lot of visual work is required to compare the different views, the user does not need to remember them like in a slideshow; thereby, the interaction costs are lower.
(e)
Space–time cube representations merge maps and timelines orthogonally within a cubic space, which allows one to map every space–time path as a three-dimensional trajectory [45,46,47,48]. Aside from providing such a direct integration of spatiotemporal coordinates, space–time cubes also come with the specific functionality of supporting the cognitive translation and navigation among all other spatiotemporal views [49] (see Section 4). From a cognitive perspective, space–time cube representations offer one perceptually integrated view in which the story can unfold. In contrast to a superimposition view, time is also mapped to space, making the temporal and geographic information of movement paths similarly salient. Therefore, the user can more easily build up a spatiotemporally integrated situation model of a story. However, in such a three-dimensional visualization, visual clutter—and increased interaction costs—are a constant challenge [50]. Still, evaluations confirm that space–time cube visualizations are easy to use and are especially suited for the exploration of spatiotemporal patterns [51,52].
In comparison, these well-established representation techniques find distinctly different solutions to represent spatial and temporal data aspects in an integrated or hybrid fashion. Table 1 summarizes their visual–analytical strengths and limitations and illustrates that every method comes with a specific profile, combining analytical benefits with particular costs.
Obviously, among these different options, there is no ideal view—but rather, multiple perspectives and methods exist to represent narrative data in a spatiotemporally hybrid way. For this reason, advanced visualization systems provide multiple spatiotemporal views as a “solomonic” design strategy—not to sacrifice the benefits of alternative perspectives, but to offer multiple ones. This allows one to “maximise insight, balance the strengths and weaknesses of individual views, and avoid misinterpretation” [53] (p. 9), and enables the user to select and switch between the most appropriate representations for the data and task at hand (ibidem, p. 10). For storytelling visualizations, multiple views offer different perspectives on a story, highlight different event indices, and let the user construct a more elaborated internal representation.
As time orientation plays a central role in the narrative data domain, we recommend to follow best practices of visualization system design and to deliberately implement multiple views for the complementary and flexible combinations of analytical perspectives. Yet, we also recommend deliberately addressing a neuralgic cognitive challenge, which emerges from using multiple views with their corresponding design strategies. Simply put, the utilization of multiple perspectives allows users to avoid analytical reductions, blind spots, and simplifications. However, multiple perspectives also challenge users’ coherent understanding: How do these different perspectives relate to each other? How do users not only perceptually integrate spatial and temporal perspectives (as all five abovementioned techniques assist them to do), but also cognitively integrate, couple, or synthesize these different spatiotemporal perspectives? This challenge is of special importance in narrative visualizations, as the coherence of the external representation is relevant for the construction of a coherent situation model. Coherence breaks have been shown to interrupt the construction of the internal representation and the recipient’s engagement with the story [14,54]. Similarly, for the design of visualization systems, coherence is an important factor to be considered, since it eases the construction, maintenance, and cognitive utilization of the internal representation [55]. Building on conceptual work, we report on the development of a web-based visualization environment, which emphasizes these exact development gaps that have remained largely unresolved by interface design so far.

4. Towards a Multi-Perspective Interface for Narrative Visualization

The PolyCube framework (PolyCube project: https://www.donau-uni.ac.at/en/polycube) has been developed to offer multiple temporal views on spatiotemporal data, while putting a focus on the overall ease of use, enhanced navigation, and on supporting the cognitive integration of insights from multiple views. Its central operating perspective is provided by a space–time cube representation and also offers access to perspectives of layer juxtaposition, layer superimposition, multiple coordinated views, and animation. One of its main design strategies is to translate or mediate these different perspectives by seamless layout transitions, which are illustrated further down (Figures 7 and 8). The visualization system is currently developed as a web-based visual analytics environment and has been explored with regard to two application scenarios, including a case study on cultural heritage collection data [56] and another on biography data [57].

4.1. Visualizing Biography Data and Historical Narratives

Digitization initiatives are transforming historiographical knowledge collections into semantically structured data and knowledge graphs [58,59]. This translation also involves large biographical lexica, which have been assembled to document the life stories of national cultural heroes—as well as many thousands of other figures that have influenced domains such as the arts, politics, or natural sciences—in a standardized manner. By the means of natural language processing techniques (Appendix C), a variety of entities and relations are extracted from thousands of life path narratives, including actors, organizations, social relations, and geographic places, which are all interwoven by the pathways of individual trajectories and temporally structured according to documented events and dates.
With regard to historical and cultural figures of Austria, the APIS project (https://www.oeaw.ac.at/acdh/projects/apis/) has developed a digital prosopographical information system out of the National Biographical Lexicon [60]. To explore the visual–analytical framework outlined so far, we extracted documented locations and corresponding time stamps for a dozen biographical narratives, which we injected into spatiotemporal visualization systems for further exploration. Figure 5 exemplarily shows the biographical trajectory of Joszef Szabo—an Austro-Hungarian opera singer and actor—visualized as a space–time path by the means of a commercial package (left-hand side, [61]) and by the prototypical PolyCube system, utilizing web technologies, such as d3.js, CSS3D and three.js (right-hand side).
We chose a space–time cube representation as a preferential entry point into narrative visualization because it immediately discloses a variety of spatiotemporal patterns (such as horizontal movements or vertical “stations”) to the highly skilled faculties of 3D gestalt perception (see Figure 5 and Figure 6). Furthermore, it allows on demands switching to all other spatiotemporal perspectives—while maintaining the orientation of analysts in a unique way (see Section 4.2).
With regard to the basic pattern language, three elements define individual choreographies: “The basic concepts in time geography are paths, stations, and prisms. Paths show movement behavior of objects through space and time […]. The stations indicate locations where people stay for longer moments […], and prisms representing space reachable within a given time budget” [62] (p. 57). Therefore, the meandering curves of historic actors disclose their spatiotemporal profile, consisting of their main biographical stations and connected by their movements and travels. As each historical character starts from different spatiotemporal coordinates—and behaves differently until his or her death—their space–time paths can be read and interpreted like a unique diagram. As soon as a minimum amount of familiarity (or visual literacy) with this visualization technique emerges, historians or humanists can utilize it for the purpose of close reading or viewing spatiotemporal story aspects—or zoom out for distant reading of multiple biographies or narratives in a mutual context, connection, or comparison (see Figure 6).
Space–time cube representations allow the direct construction of a model of a historic actor’s behavior and visual analysis of a multitude of movement patterns [63,64], but they also have some analytical limitations (see Table 1). Therefore, if visualization systems want to offer multiple alternative perspectives on spatiotemporal data—how can we maintain their orientation when switching from one perspective to another?

4.2. Supporting the Cognitive Integration of Multiple Perspectives

Bennett and Flach [65] define the extent to which an interface supports the users’ transitions between different perspectives or information activities as “visual momentum.” One of the most effective design strategies to increase visual momentum and enhance the “cognitive coupling” between different views is traceable transformations of arrangement principles, which work like movements of perspective-taking and changing in natural environments.
These seamless transitions work like a coherence technique [55] that connects one view with another by making explicit changes—but also constancies—between different perspectives. Various forms of morphing visually translate from one spatialization principle to another and, thus, help to build up a coherent representation of both views [30]. By changing layouts incrementally—as opposed to abrupt changes or hard cuts between views—the spatial rearrangement of the story-relevant elements into new constellations can be perceptually traced. These techniques correspond to the idea of preserving the mental map [66], the effort of keeping the number of changing elements to a suitable minimum, and the notion of making the new arrangement principles transparent.
With specific regard to the translation between spatiotemporal visualizations, Bach et al. [49] provide a conceptual key element: Space–time cube representations can transform into different spatiotemporal visualizations (including spatial or temporal perspectives only) by quasi-physical operations of flattening, cutting, traversing, or stretching (see Figure 7).
The PolyCube system supports these operations of alternative spatiotemporal perspective taking and cognitive coupling by seamless canvas transitions (for a video demonstration, see https://youtu.be/PTsk-NROJhU) [56,67]. Figure 8 illustrates two of these transitions, leading from space–time cube to layer juxtaposition, and from juxtaposition to a superimposition perspective. The expected cognitive effects on the macrocognitive integration and synthesis of insights, the orientation of the narrative analyst, and overall ease of use are currently being evaluated. In line with this endeavor, we see the development of coherence techniques (including hybrid views and seamless transitions) to be a research and design task of its own for complex interfaces, in order to better support cross-dimensional inferences and versatile model-based visual reasoning with narrative data [68].

5. Discussion

In this paper, we presented some basic assumptions about narrative cognitive processing and their implications for the design of narrative spatiotemporal visualizations. We outlined a novel design guideline and strategy to tackle the “representation challenge” of geovisualization: “To develop new forms of representation that support the understanding of geospatial phenomena and space-time processes. There is a need to take full advantage of technological advances that make it possible to […] generate complex multidimensional and dynamically linked views, merge representation with reality, and […] to develop methods that help users navigate within complex representations” [69] (p. 7). In the following section, we want to shortly discuss different related questions and challenges that mostly go beyond the core topic of this special issue, yet seem to be relevant for us, as they expand fields for further research.

5.1. Going beyond Space and Time

In this paper, we focused on the visualization of two event indices: space and time. However, what role do actors, events, intentions, and causation play in storytelling with geographic visualizations? On one hand, the displayed information content of the story here comes into play; in our case study of life stories, the protagonists pre-defined the main actors and we visualized their life events (as data points in the temporal and spatial information space). While the temporal chain of events already gives the recipient important cues on causes and intentions [19], the actual verbal description of these events will help the recipient to understand these further. On the other hand, we can also go beyond spatiotemporal data and visualize other aspects of a story: Persons or organizations (as collective actors), their relationships, as well as emotions and sentiments (Appendix D) are essential elements of a story, where established visualization techniques can assist recipients in gaining a broader picture of the story. Current reviews on text visualization [70] provide interesting resources for storytelling visualizations, going beyond space and time. In the following section, we will discuss different visualization techniques supporting distant reading on a story macro level (Section 5.2) and close reading on a story micro level (Section 5.3).

5.2. Distant Reading: Combining Geovisualization with Non-Geographic Visualization Techniques

What seems to be of utmost interest, from a more general visualization point of view, is the openness of the PolyCube approach to going beyond the geovisualization frontier and connecting with non-geographic information visualization methods [35]. This allows not only the maintenance of the visualization of geo-spatiotemporal movements as an analytical backbone for the investigation of narrative data, but also the ability to combine this perspective with other relevant imaging procedures. Figure 9 shows how biographical narratives can also be told and visualized against the background of non-geographic information spaces. Exemplarily, the left-hand side illustrates the path of an individual through the social–relational spacetime, generated by interaction, collaboration, or conflict [67], whereas the right-hand side shows how the same system can also visualize trajectories through a knowledge or topic space of a cultural domain [71]. In both cases, the spatiotemporal scaffold is maintained, while the geographic map on the data plane is substituted by a force-directed network graph or a hierarchically structured treemap. With regard to the event-indexing model (cp. Section 2), these methods also allow the development of visualization systems that support the comprehension and connection of other event indices than space or time, like actors and their interactions, as well as the development of sentiments and intentions, as long as they can be categorized and visually encoded into a non-geographic or structural topology.

5.3. Close Reading: Combining Visual Analysis with Textual Analysis

What also seems to be essential in the context of narrative visualizations is to offer optional access to textual representations of narrative data in parallel to graphic representations. This allows the analysis and “close-read” of a source text (or a visualization description or annotation), in comparison to diagrammatic representation, possibly including further supportive text visualization measures, such as coordinated highlighting or colored mark-up of textual entities (cp. [69]). Eccles et al. [3] showed how a system of coordinated multiple views can do this for space–time cube representations and, thus, provided one of many recent instances for the desideratum of a visualization system “that does not destroy the original text in the process” [72]. Another option for combining textual data with a graphic representation is to tell a story sequentially and incrementally on a textual basis, while zooming and panning to selections of a space–time path—as is already offered by tools like StoryMapJS (https://storymap.knightlab.com/) or ESRI storyteller (https://storymaps.arcgis.com/).

5.4. Visualizing Non-Linear Stories

What also becomes possible in the context of parallel views is the visual analysis of (non)linear story structures. Biographies are often told as chronological and linear stories in which the narrative order preserves and mirrors the story order of historical events [73]. In contrast, advanced storytelling frequently mixes things up and works with nonlinear narrative sequencing to create artful or surprising narrative arcs to convey a sense of mystery and tension and elevate the readers’ entertainment. As this narration technique also adds to overall sensemaking challenges, any combined depiction of story order and narrative order (e.g., by the use of dual timelines or the inclusion of textual data) allows the visualization of such nonlinearity of story development—and further adds to the users’ orientation and comprehension [74,75].

5.5. Going beyond the Situation Model—Narrative Effects and Drawbacks

As indicated in Section 2, the processing of narrative information is highly structured by cognitive schemata. Until now, we have only discussed the nature of the internal representation (situation model) and how it is constructed (event indexing). However, many further cognitive and motivational effects of narrative processing can be found in the literature, which are of interest for better understanding the effects of storytelling—positive and negative alike. On the positive side, existing story schemata reduce cognitive load and free cognitive resources, allowing us to process information more deeply and understand more complex information [76]. On an emotional–motivational level, identification with the story is a powerful mechanism, which can also result in narrative engagement, flow, and positive attitudes towards the topic [54]. On the negative side, incoming story information is checked only for plausibility and coherence, but not necessarily for truth [54], nor is it critically reflected. Therefore, it poses the risk of being misused for persuasive effects [77].

5.6. Evaluation Challenge

In their paper on cognitive and usability issues in geovisualization, Slocum et al. [42] emphasized that a usable interface should be built on a metaphor well-known by its users. Story schemata are similar to these metaphors in assisting the user (if activated) to process the information in a quick and efficient manner. But, how do we know that a story schema was activated? How can we know how well the user integrated geotemporal information in the story situation model?
A final, but nevertheless important, challenge is how to survey and evaluate situation models and whether they integrate spatial and temporal data. It is difficult to use the evaluation methods of narrative text comprehension research, as they often use relatively simple stories to single out the effects of interests (see also [78] on the relationship between psychological and cartographical research). When it comes to more complex visual stories, it is important to further develop these methods and combine them with qualitative methods (like the think-aloud technique) to develop a better understanding of the construction and enrichment of situation models from visual stories. Within the PolyCube project, we collected and reviewed a set of methodologies to assess internal representations resulting from the interaction with information visualizations [79]. Further research is needed to empirically evaluate these methodologies and identify the most suitable ones for the study of information visualization, in general, and of story situation models, in particular.

6. Conclusions

This special issue called for design guidelines and best practices for the development and deployment of expressive, perceptually salient, and cognitively supportive online geographic information stories. In this paper, we approached this question from a cognitive perspective and reviewed research on how stories are internally represented and cognitively constructed. The event-indexing model by Zwaan et al. [18] served as a reference for our discussion of geotemporal visualizations for storytelling: Each event of a story is indexed according to actors, their intentions, space, and time, and linked to prior events by causation. This model stems from research on narrative texts; although it was already transferred to other types of media (like films), we are not aware of publications that have transferred the event-indexing model to narrative visualizations. This model can serve as a reference for—and can provide guidance to—future empirical studies on this topic.
The event-indexing model suggests that the spatial and temporal information in a story can serve as a framework for the construction of the story situation model. We explored the design space in which (a) spatial and geographic information can be visually represented in an integrated fashion and (b) multiple perspectives can be coupled in a coherent manner. Standard techniques for visually representing spatiotemporal data (coordinated linked views, animation or slideshows, layer superimposition, juxtaposition, and space–time cube representations) all have different drawbacks and benefits from a cognitive perspective. To develop design guidelines, assisting visualization scholars and professionals in determining the representations that are best suited to supporting the smooth and synoptic construction of situation models (i.e., for which specific kinds of stories and users) is a topic for future research.
In absence of such guidelines and for more complex stories, we suggested offering multiple spatiotemporal representation techniques within one visualization system. We discussed techniques increasing the coherence between multiple views, e.g., by seamless transitions, and assisting users to cognitively integrate these visualizations and construct a coherent internal representation of the story from multiple analytical perspectives. As an outlook, we illustrated options to extend the geovisualization perspective and integrate non-geographic space-time cartographies to a synoptic visual-analytical environment of multiple coordinated cubes.
In a case study, we showed how this kind of visual analytics framework can support storytelling in history and biography contexts. As for further areas of application, we aim to explore how this framework also proves to be useful for visual storytelling in other (digital) humanities and social sciences domains, in order to shed light on a wide range of actor networks and their dynamics throughout the evolution of the human web [80].

Acknowledgments

The authors want to thank the team of the APIS project at the Austrian Center of Digital Humanities (ACDH) in Vienna, as well as Günther Schreder and Saminu Salisu for contributing to this work. This research was supported by a grant from the Austrian Science Fund (FWF), project number P28363-G24.

Author Contributions

The authors collaboratively conceived and designed the PolyCube project, which provides the background and the visualization framework of this article. This paper emerged through collaborative conceptualization and extensive discussion and negotiation processes of both authors. Eva Mayr conceived the cognitive foundation of the article and contributed the first drafts of the corresponding textual parts in Section 2, Section 3, Section 4 and Section 5. Florian Windhager conceived the visualization foundations and wrote the first drafts of the corresponding textual parts in Section 2, Section 3, Section 4 and Section 5, and designed the figures.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Even if scholarly attempts to match the semi-fictional locations of the Odyssey with real geographic places have been mocked since the early days of classical philology, the seminal ancient narrative provides a valuable case. With its nonlinear spatiotemporal storyline, it has not only challenged its recipients’ faculties of imagination since ancient times, but has also been said to be the archetypical narrative portraying the relentless striving of modern human subjectivity [81], thus influencing the self-conceptions and expectations of countless subjects, readers, and writers (from Virgil to James Joyce) for centuries to come.

Appendix B

Frequently, a certain amount of drama—caused by an inciting incident, a conflict or a desire—is also said to be an integral part of a story definition, which raises recipients’ interests, fosters engagement and, thus, drives and motivates the sequential progress of a narrative [82].

Appendix C

The use of natural language processing (NLP) for the automatic detection of events, persons, etc., out of biographical texts is an ongoing research endeavor [83]. Though NLP techniques progress fast, “it is a well-known fact that automatic text analyses do not yield perfect results” (p. 210).

Appendix D

Recent approaches additionally extract emotions and sentiments from (narrative) texts via natural language processing and visualize them [84,85].

References

  1. Shmoop Editorial Team. The Odyssey. Available online: https://www.shmoop.com/odyssey/ (accessed on 24 January 2018).
  2. Gershon, N.; Page, W. What storytelling can do for information visualization. Commun. ACM 2001, 44, 31–37. [Google Scholar] [CrossRef]
  3. Eccles, R.; Kapler, T.; Harper, R.; Wright, W. Stories in GeoTime. IEEE Vis. Anal. Sci. Technol. 2007, 7, 3–17. [Google Scholar]
  4. Wohlfart, M.; Hauser, H. Story telling aspects in volume visualization. In Proceedings of the 9th Joint Eurographics/IEEE VGTC conference on Visualization, Norrköping, Sweden, 23–25 May 2007; pp. 91–98. [Google Scholar]
  5. Segel, E.; Heer, J. Narrative visualization: Telling stories with data. IEEE Trans. Vis. Comput. Graph. 2010, 16, 1139–1148. [Google Scholar] [CrossRef] [PubMed]
  6. Hullman, J.; Diakopoulos, N. Visualization rhetoric: Framing effects in narrative visualization. IEEE Trans. Vis. Comput. Graph. 2011, 17, 2231–2240. [Google Scholar] [CrossRef] [PubMed]
  7. Caquard, S. Cartography I: Mapping narrative cartography. Prog. Hum. Geogr. Lond. 2013, 37, 135–144. [Google Scholar] [CrossRef]
  8. Kosara, R.; Mackinlay, J. Storytelling: The next step for visualization. Computer 2013, 46, 44–50. [Google Scholar] [CrossRef]
  9. Hullman, J.; Drucker, S.; Riche, N.H.; Lee, B.; Fisher, D.; Adar, E. A deeper understanding of sequence in narrative visualization. IEEE Trans. Vis. Comput. Graph. 2013, 19, 2406–2415. [Google Scholar] [CrossRef] [PubMed]
  10. Lee, B.; Riche, N.H.; Isenberg, P.; Carpendale, S. More than telling a story: Transforming data into visually shared stories. IEEE Comput. Graph. Appl. 2015, 35, 84–90. [Google Scholar] [CrossRef] [PubMed]
  11. Stefaner, M. Worlds, Not Stories. Well-Formed Data, 2014. Available online: http://well-formed-data.net/archives/1027/worlds-not-stories (accessed on 20 January 2018).
  12. Ryan, M.-L. Narrative. In Routledge Encyclopedia of Narrative Theory; Taylor & Francis Group: Oxfordshire, UK, 2010; pp. 344–348. [Google Scholar]
  13. Bruner, J. Narrative and paradigmatic modes of thought. In Learning and Teaching the Ways of Knowing; University of Chicago Press: Chicago, IL, USA, 1985; pp. 97–115. [Google Scholar]
  14. Graesser, A.C.; Olde, B.; Klettke, B. How does the mind construct and represent stories? In Narrative Impact: Social and Cognitive Foundations; Erlbaum: Mahwah, NJ, USA, 2002; pp. 231–263. [Google Scholar]
  15. Wilkens, T.; Hughes, A.; Wildemuth, B.M.; Marchionini, G. The role of narrative in understanding digital video: An exploratory analysis. Proc. Am. Soc. Inf. Sci. Technol. 2005, 40, 323–329. [Google Scholar] [CrossRef]
  16. Stein, N.L.; Kissel, V.I. Story schemata and causal structure. In Routledge Encyclopedia of Narrative Theory; Taylor & Francis Group: Oxfordshire, UK, 2010; pp. 567–568. [Google Scholar]
  17. Dijk, T.V.; Kintsch, W. Strategies of Discourse Comprehension; Academic Press: Cambridge, CA, USA, 1983. [Google Scholar]
  18. Zwaan, R.A.; Langston, M.C.; Graesser, A.C. The construction of situation models in narrative comprehension: An event-indexing model. Psychol. Sci. 1995, 6, 292–297. [Google Scholar] [CrossRef]
  19. Zwaan, R.A.; Radvansky, G.A. Situation models in language comprehension and memory. Psychol. Bull. 1998, 123, 162–185. [Google Scholar] [CrossRef] [PubMed]
  20. Schnotz, W. Integrated Model of Text and Picture Comprehension; Mayer, R.E., Ed.; The Cambridge Handbook of Multimedia Learning; Cambridge University Press: Cambridge, UK, 2014. [Google Scholar]
  21. Taylor, H.A.; Tversky, B. Spatial mental models derived from survey and route descriptions. J. Mem. Lang. 1992, 31, 261–292. [Google Scholar] [CrossRef]
  22. Tversky, B. Cognitive Maps, Cognitive Collages, and Spatial Mental Models; Frank, A.U., Campari, I., Eds.; Springer: Berlin/Heidelberg, Germany, 1993; Volume 716, pp. 14–24. [Google Scholar]
  23. Mark, D.M.; Freksa, C.; Hirtle, S.C.; Lloyd, R.; Tversky, B. Cognitive models of geographical space. Int. J. Geogr. Inf. Sci. 1999, 13, 747–774. [Google Scholar] [CrossRef]
  24. Tversky, B. Structures of mental spaces: How people think about space. Environ. Behav. 2003, 35, 66–80. [Google Scholar] [CrossRef]
  25. Kosara, R. Stories Don’t Tell Themselves. Available online: https://eagereyes.org/blog/2010/stories-dont-tell-themselves (accessed on 20 January 2018).
  26. Kriglstein, S.; Pohl, M.; Smuc, M. Pep up your time machine: Recommendations for the design of information visualizations of time-dependent data. In Handbook of Human Centric Visualization; Springer: Belin/Heidelberg, Germany, 2014; pp. 203–225. [Google Scholar]
  27. Rosenberg, D.; Grafton, A. Cartographies of Time: A History of the Timeline; Princeton Architectural Press: New York, NY, USA, 2012; ISBN 1-61689-058-4. [Google Scholar]
  28. Davis, S.B.; Bevan, E.; Kudikov, A. Just in time: Defining historical chronographics. In Electronic Visualisation in Arts and Culture; Springer: Belin/Heidelberg, Germany, 2013; pp. 243–257. [Google Scholar]
  29. Champagne, M. Diagrams of the past: How timelines can aid the growth of historical knowledge. Cogn. Semiot. 2016, 9, 11–44. [Google Scholar] [CrossRef]
  30. Brehmer, M.; Lee, B.; Bach, B.; Riche, N.H.; Munzner, T. Timelines revisited: A design space and considerations for expressive storytelling. IEEE Trans. Vis. Comput. Graph. 2017, 23, 2151–2164. [Google Scholar] [CrossRef] [PubMed]
  31. McCloud, S. Understanding Comics: The Invisible Art; HarperCollins Publishers: Northamp, MA, USA, 1993. [Google Scholar]
  32. Javed, W.; Elmqvist, N. Exploring the design space of composite visualization. In Proceedings of the 2012 IEEE Pacific on Visualization Symposium (PacificVis), Songdo, Korea, 28 February–2 March 2012; pp. 1–8. [Google Scholar]
  33. Zhao, Z.; Marr, R.; Elmqvist, N. Data Comics: Sequential Art for Data-Driven Storytelling; Technical Report; Human Computer Interaction Lab, University of Maryland: College Park, MD, USA, 2015. [Google Scholar]
  34. Tversky, B. Visualizing thought. Top. Cogn. Sci. 2011, 3, 499–535. [Google Scholar] [CrossRef] [PubMed]
  35. Skupin, A.; Fabrikant, S.I. Spatialization methods: A cartographic research agenda for non-geographic information visualization. Cartogr. Geogr. Inf. Sci. 2003, 30, 99–119. [Google Scholar] [CrossRef]
  36. Mountain, D. Visualizing, querying and summarizing individual spatio-temporal behaviour. In Exploring Geovisualization; Dykes, J., MacEachren, A.M., Kraak, M.J., Eds.; Elsevier: Amsterdam, The Netherlands, 2005; pp. 181–200. [Google Scholar]
  37. Andrienko, N.; Andrienko, G.; Gatalsky, P. Exploratory spatio-temporal visualization: An analytical review. J. Vis. Lang. Comput. 2003, 14, 503–541. [Google Scholar] [CrossRef]
  38. Andrienko, G.; Andrienko, N.; Dykes, J.; Fabrikant, S.I.; Wachowicz, M. Geovisualization of Dynamics, Movement and Change: Key Issues and Developing Approaches in Visualization Research. Inf. Vis. 2008, 7, 173–180. [Google Scholar] [CrossRef]
  39. Roberts, J.C. Exploratory visualization with multiple linked views. Explor. Geovis. 2005, 159–180. [Google Scholar] [CrossRef]
  40. Baldonado, M.Q.W.; Woodruff, A.; Kuchinsky, A. Guidelines for Using Multiple Views in Information Visualization. In Proceedings of the Working Conference on Advanced Visual Interfaces; AVI ’00, Palermo, Italy, 24–26 May 2000; ACM: New York, NY, USA, 2000; pp. 110–119. [Google Scholar]
  41. Sweller, J.; Ayres, P. The split-attention principle in multimedia learning. In The Cambridge Handbook of Multimedia Learning; Cambridge University Press: Cambridge, UK, 2006; pp. 135–146. [Google Scholar]
  42. Slocum, T.A.; Blok, C.; Jiang, B.; Koussoulakou, A.; Montello, D.R.; Fuhrmann, S.; Hedley, N.R. Cognitive and usability issues in geovisualization. Cartogr. Geogr. Inf. Sci. 2001, 28, 61–75. [Google Scholar] [CrossRef]
  43. Andrienko, G.; Andrienko, N.; Demsar, U.; Dransch, D.; Dykes, J.; Fabrikant, S.I.; Jern, M.; Kraak, M.-J.; Schumann, H.; Tominski, C. Space, time and visual analytics. Int. J. Geogr. Inf. Sci. 2010, 24, 1577–1600. [Google Scholar] [CrossRef]
  44. Moore, A. Maps as comics, comics as maps. In Proceedings of the 24th International Cartography Conference (ICC 2009), Santiago, Chile, 15–21 November 2009; pp. 15–21. [Google Scholar]
  45. Hägerstrand, T. What about people in regional science? Pap. Reg. Sci. Assoc. 1970, 24, 7–21. [Google Scholar] [CrossRef]
  46. Kraak, M.J. The space-time cube revisited from a geovisualization perspective. In Proceedings of the 21st International Cartographic Conference, Durban, South Africa, 10–16 August 2003; pp. 1988–1996. [Google Scholar]
  47. Kwan, M.P.; Lee, J. Geovisualization of human activity patterns using 3D GIS: A time-geographic approach. In Spatially Integrated Social Science: Examples in Best Practice; Oxford University Press: Oxford, UK, 2004; Volume 27. [Google Scholar]
  48. Kwan, M.-P.; Ding, G. Geo-narrative: Extending geographic information systems for narrative analysis in qualitative and mixed-method research. Prof. Geogr. 2008, 60, 443–465. [Google Scholar] [CrossRef]
  49. Bach, B.; Dragicevic, P.; Archambault, D.; Hurter, C.; Carpendale, S. A Descriptive Framework for Temporal Data Visualizations Based on Generalized Space-Time Cubes. Comput. Graph. Forum 2016, 36, 36–61. [Google Scholar] [CrossRef]
  50. Munzner, T. Visualization Analysis and Design; CRC Press: Boca Raton, FL, USA, 2014. [Google Scholar]
  51. Amini, F.; Henry Riche, N.; Lee, B.; Hurter, C.; Irani, P. Understanding data videos: Looking at narrative visualization through the cinematography lens. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea, 18–23 April 2015; pp. 1459–1468. [Google Scholar]
  52. Kjellin, A.; Pettersson, L.W.; Seipel, S.; Lind, M. Evaluating 2d and 3d visualizations of spatiotemporal information. ACM Trans. Appl. Percept. TAP 2010, 7, 19. [Google Scholar] [CrossRef]
  53. Kerracher, N.; Kennedy, J.; Chalmers, K. The design space of temporal graph visualisation. In Proceedings of the 18th Eurographics Conference on Visualization - Short Papers (EuroVis ’14), Swansea, Wales, UK, 9–13 June 2014; pp. 7–14. [Google Scholar]
  54. Busselle, R.; Bilandzic, H. Fictionality and Perceived Realism in Experiencing Stories: A Model of Narrative Comprehension and Engagement. Commun. Theory 2008, 18, 255–280. [Google Scholar] [CrossRef]
  55. Schreder, G.; Windhager, F.; Smuc, M.; Mayr, E. A Mental Models Perspective on Designing Information Visualizations for Political Communication. JeDEM EJ. EDemocr. Open Gov. 2016, 8, 80–99. [Google Scholar]
  56. Windhager, F.; Salisu, S.; Schreder, G.; Mayr, E. Orchestrating overviews. A synoptic approach to the visualization of cultural collections. Remaking Collect. Spec. Issue Open Libr. Humanit. 2018. submitted. [Google Scholar]
  57. Windhager, F.; Federico, P.; Salisu, S.; Schlögl, M.; Mayr, E. A synoptic visualization framework for the multi-perspective study of biography and prosopography data. In Proceedings of the 2nd IEEE VIS Workshop on Visualization for the Digital Humanities (VIS4DH’17), Phoenix, AZ, USA, 2 October 2017. [Google Scholar]
  58. Ter Braake, S.; Fokkens, A.S.; Sluijter, R.; Declerck, T. (Eds.) BD2015 Proceedings of the 1st Conference on Biographical Data in a Digital World (BD2015); Business Web and Media, Network Institute, Intelligent Information Systems: Amsterdam, The Netherland, 2015. [Google Scholar]
  59. Wandl-Vogt, E; Lejtovicz, K. (Eds.) Biographical Data in a Digital World 2017. Available online: http://doi.org/10.5281/zenodo.1041978 (accessed on 8 March 2018).
  60. Bernád, Á.Z.; Kaiser, M.; Mair, S.; Rind, A. Communities in Biographischen Netzwerken [Communities in biographical networks]. In Proceedings of the 10th Forum Media Technology, St. Pölten, Austria, 29–30 November 2017; pp. 83–87. [Google Scholar]
  61. Kapler, T.; Wright, W. GeoTime information visualization. Inf. Vis. 2005, 4, 136–146. [Google Scholar] [CrossRef]
  62. Kraak, M.-J.; Kveladze, I. Narrative of the annotated space–time cube–revisiting a historical event. J. Maps 2017, 13, 56–61. [Google Scholar] [CrossRef]
  63. Kristensson, P.O.; Dahlbäck, N.; Anundi, D.; Björnstad, M.; Gillberg, H.; Haraldsson, J.; Martensson, I.; Nordvall, M.; Staahl, J. An evaluation of space time cube representation of spatiotemporal patterns. IEEE Trans. Vis. Comput. Graph. 2009, 15, 696–702. [Google Scholar] [CrossRef] [PubMed]
  64. Dodge, S.; Weibel, R.; Lautenschütz, A.K. Towards a taxonomy of movement patterns. Inf. Vis. 2008, 7, 240–252. [Google Scholar] [CrossRef] [Green Version]
  65. Bennett, K.B.; Flach, J.M. Visual momentum redux. Int. J. Hum.-Comput. Stud. 2012, 70, 399–414. [Google Scholar] [CrossRef]
  66. Eades, P.; Lai, W.; Misue, K.; Sugiyama, K. Preserving the Mental Map of A Diagram; International Institute for Advanced Study of Social Information Science, Fujitsu Limited: Tokyo, Japan, 1991. [Google Scholar]
  67. Federico, P.; Aigner, W.; Miksch, S.; Windhager, F.; Smuc, M. Vertigo zoom: Combining relational and temporal perspectives on dynamic networks. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI2012), Capri Island, Italy, 22–25 May 2012; ACM: New York, NY, USA, 2012; pp. 437–440. [Google Scholar]
  68. Liu, Z.; Stasko, J.T. Mental Models, Visual Reasoning and Interaction in Information Visualization: A Top-down Perspective. IEEE Trans. Vis. Comput. Graph. 2010, 16, 999–1008. [Google Scholar] [CrossRef] [PubMed]
  69. MacEachren, A.M.; Kraak, M.-J. Research challenges in geovisualization. Cartogr. Geogr. Inf. Sci. 2001, 28, 3–12. [Google Scholar] [CrossRef]
  70. Jänicke, S.; Franzini, G.; Cheema, M.F.; Scheuermann, G. Visual text analysis in digital humanities. Comput. Graph. Forum 2017, 36, 226–250. [Google Scholar] [CrossRef]
  71. Smuc, M.; Windhager, F.; Sari, M.; Federico, P.; Amor-Amoros, A.; Miksch, S. Interweaving pathways of innovation. Visualizing the R&D dynamics of companies provided by patent data. In Proceedings of the XXXV Sunbelt Conference of the International Network for Social Network Analysis, Brighton, UK, 23–28 June 2015; p. 265. [Google Scholar]
  72. Bradley, A.J. Violence and the digital humanities text as Pharmakon. In Proceedings of the Digital Humanities, Hamburg, Germany, 16–20 July 2012. [Google Scholar]
  73. Genette, G. Narrative Discourse: An Essay in Method; Cornell University Press: Ithaca, NY, USA, 1983. [Google Scholar]
  74. Carter, S.; Cox, A.; Bostock, M. Dissecting a Trailer: The Parts of the Film That Make the Cut. Available online: http://www.nytimes.com/interactive/2013/02/19/movies/awardsseason/oscar-trailers.html (accessed on 8 March 2018).
  75. Kim, N.W.; Bach, B.; Im, H.; Schriber, S.; Gross, M.; Pfister, H. Visualizing Nonlinear Narratives with Story Curves. IEEE Trans. Vis. Comput. Graph. 2018, 24, 595–604. [Google Scholar] [CrossRef] [PubMed]
  76. Zumbach, J.; Mohraz, M. Cognitive load in hypermedia reading comprehension: Influence of text type and linearity. Comput. Hum. Behav. 2008, 24, 875–887. [Google Scholar] [CrossRef]
  77. Appel, M.; Richter, T. Persuasive effects of fictional narratives increase over time. Media Psychol. 2007, 10, 113–134. [Google Scholar]
  78. Roth, R.E.; Çöltekin, A.; Delazari, L.; Filho, H.F.; Griffin, A.; Hall, A.; Korpi, J.; Lokka, I.; Mendonça, A.; Ooms, K. User studies in cartography: Opportunities for empirical research on interactive maps and visualizations. Int. J. Cartogr. 2017, 28, 1–29. [Google Scholar] [CrossRef]
  79. Mayr, E.; Schreder, G.; Smuc, M.; Windhager, F. Looking at the representations in our mind: Measuring mental models of information visualizations. In Proceedings of the Beyond Time and Errors on Novel Evaluation Methods for Visualization (BELIV ’16), Baltimore, MD, USA, 24 October 2016; pp. 96–103. [Google Scholar]
  80. McNeill, J.R.; McNeill, W.H. The Human Web: A Bird’s-Eye View of World History; W.W. Norton & Company: New York, NY, USA, 2003; ISBN 978-0-393-05179-7. [Google Scholar]
  81. Adorno, T.W.; Horkheimer, M. Dialektik der Aufklärung; S. Fischer: Frankfurt, Germany, 1969. [Google Scholar]
  82. Polti, G. The Thirty-Six Dramatic Situations. Available online: www.pseudology.org/Literature/Polty_George_36_Dramatic_Situations2.pdf (accessed on 20 January 2018).
  83. Fokkens, A.; ter Braake, S.; Ockeloen, N.; Vossen, P.; Legêne, S.; Schreiber, G.; de Boer, V. BiographyNet: Extracting relations between people and events. In Europa baut auf Biographien: Aspekte, Bausteine, Normen und Standards für Eine Europäische Biographik; New Academic Press: Wien, Austria, 2017; pp. 193–224. [Google Scholar]
  84. Van Meersbergen, M.; Vossen, P.; van der Zwaan, J.; Fokkens, A.; van Hage, W.; Leemans, I.; Maks, I. Storyteller: Visual analytics of perspectives on rich text interpretations. In Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism, Copenhagen, Denmark, 7 September 2017; pp. 37–45. [Google Scholar]
  85. Kucher, K.; Paradis, C.; Kerren, A. The state of the art in sentiment visualization. Comput. Graph. Forum 2017, 37, 71–96. [Google Scholar] [CrossRef]
Figure 1. Homer’s Odyssey visualized on a map (adapted from [1]) and as a timeline.
Figure 1. Homer’s Odyssey visualized on a map (adapted from [1]) and as a timeline.
Ijgi 07 00096 g001
Figure 2. The comprehension of stories is said to result from the cognitive interconnection of narrated facts or events (e) according to five situative indices or data dimensions. Comprehension ensues if a recipient can situate events meaningfully in a developing situation or scenario that has been caused (c) by preceding events and interconnects specific actors (a) and their intentions (i) at a certain time (t), and in a certain place or space (s).
Figure 2. The comprehension of stories is said to result from the cognitive interconnection of narrated facts or events (e) according to five situative indices or data dimensions. Comprehension ensues if a recipient can situate events meaningfully in a developing situation or scenario that has been caused (c) by preceding events and interconnects specific actors (a) and their intentions (i) at a certain time (t), and in a certain place or space (s).
Ijgi 07 00096 g002
Figure 3. Model of multimodal cognition (Schnotz, 2014) adapted for the internalization of storytelling visualizations.
Figure 3. Model of multimodal cognition (Schnotz, 2014) adapted for the internalization of storytelling visualizations.
Ijgi 07 00096 g003
Figure 4. Hybrid views for visualizing spatial (brown) and temporal (orange) data dimensions in combination: (a) multiple coordinated views, (b) animation or slideshow, (c) color-coded layer superimposition, (d) layer juxtaposition and (e) space–time cube representation.
Figure 4. Hybrid views for visualizing spatial (brown) and temporal (orange) data dimensions in combination: (a) multiple coordinated views, (b) animation or slideshow, (c) color-coded layer superimposition, (d) layer juxtaposition and (e) space–time cube representation.
Ijgi 07 00096 g004
Figure 5. Space–time cube representation of the life-path of the Austro-Hungarian opera singer and director, Joszef Szabo (1816–1875) by the GeoTime suite (left) and by the web-based PolyCube system (right).
Figure 5. Space–time cube representation of the life-path of the Austro-Hungarian opera singer and director, Joszef Szabo (1816–1875) by the GeoTime suite (left) and by the web-based PolyCube system (right).
Ijgi 07 00096 g005
Figure 6. Comparatistic visualization of eight biographical narratives, as extracted from the APIS project.
Figure 6. Comparatistic visualization of eight biographical narratives, as extracted from the APIS project.
Ijgi 07 00096 g006
Figure 7. Space–time cube operations (in blue), seamlessly translating a space–time cube representation into four other perspectives [49,57] relevant for narrative visualization.
Figure 7. Space–time cube operations (in blue), seamlessly translating a space–time cube representation into four other perspectives [49,57] relevant for narrative visualization.
Ijgi 07 00096 g007
Figure 8. Animated canvas transitions of the PolyCube system, seamlessly translating a space-time cube representation into a juxtaposition perspective (top row) and from a juxtaposition into a layer superimposition perspective (bottom row).
Figure 8. Animated canvas transitions of the PolyCube system, seamlessly translating a space-time cube representation into a juxtaposition perspective (top row) and from a juxtaposition into a layer superimposition perspective (bottom row).
Ijgi 07 00096 g008
Figure 9. Illustration of the openness of the PolyCube framework to non-geographic visualization methods, like time-oriented network graphs (left) or cultural–categorial treemaps (right).
Figure 9. Illustration of the openness of the PolyCube framework to non-geographic visualization methods, like time-oriented network graphs (left) or cultural–categorial treemaps (right).
Ijgi 07 00096 g009
Table 1. Visual–analytical strengths and limitations of spatiotemporal representation techniques.
Table 1. Visual–analytical strengths and limitations of spatiotemporal representation techniques.
Coordinated ViewsAnimation/SlideshowLayer SuperimpositionLayer JuxtapositionSpace–Time Cube
Strengths
+
Plain 2D views
+
Connected by linking & brushing
+
Salient detection of change
+
Small screen occupancy
+
“Quasi-integration“ in working memory
+
Spatiotemporally integrated and aggregated vis
+
Small screen occupancy
+
Parallel visualization
+
High visual literacy to be expected (comics)
+
Spatiotemporally integrated view
+
Metashape
+
Attraction power
+
Cognitive & conceptual mediator
Limitations
-
Split attention effect
-
High visual work
-
No spatiotemporal integration
-
Memory overload
-
High interaction costs
-
Interpolation of animation can create data artifacts
-
Slideshow can reduced continuous temporal data to an ordinal variable
-
Visual clutter
-
Design too flat, generates little attraction
-
Time needs to be mapped to an additional visual cue (like color)
-
Large occupancy of display space
-
Temporal data is reduced to an ordinal variable
-
Split attention effect
-
High visual work
-
Visual clutter
-
High interaction costs
-
Medium occupancy of display space

Share and Cite

MDPI and ACS Style

Mayr, E.; Windhager, F. Once upon a Spacetime: Visual Storytelling in Cognitive and Geotemporal Information Spaces. ISPRS Int. J. Geo-Inf. 2018, 7, 96. https://doi.org/10.3390/ijgi7030096

AMA Style

Mayr E, Windhager F. Once upon a Spacetime: Visual Storytelling in Cognitive and Geotemporal Information Spaces. ISPRS International Journal of Geo-Information. 2018; 7(3):96. https://doi.org/10.3390/ijgi7030096

Chicago/Turabian Style

Mayr, Eva, and Florian Windhager. 2018. "Once upon a Spacetime: Visual Storytelling in Cognitive and Geotemporal Information Spaces" ISPRS International Journal of Geo-Information 7, no. 3: 96. https://doi.org/10.3390/ijgi7030096

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop