Harvesting Context and Mining Emotions Related to Olfactory Cultural Heritage

Abstract

This paper presents an Artificial Intelligence approach to mining context and emotions related to olfactory cultural heritage narratives, particularly to fairy tales. We provide an overview of the role of smell and emotions in literature, as well as highlight the importance of olfactory experience and emotions from psychology and linguistic perspectives. We introduce a methodology for extracting smells and emotions from text, as well as demonstrate the context-based visualizations related to smells and emotions implemented in a novel smell tracker tool. The evaluation is performed using a collection of fairy tales from Grimm and Andersen. We find out that fairy tales often connect smell with the emotional charge of situations. The experimental results show that we can detect smells and emotions in fairy tales with an F1 score of 91.62 and 79.2, respectively.

1. Introduction

Smell as an important component of human development is still a rather unexplored area. We know that the sense of smell affects our emotions, evokes memories, and is reflected in our subconscious. Smell and emotions are being researched by the experts from the psychology domain and human and animal studies, but we lack scientific standards, tools, and data for the effective identification, processing, and use of the influence of scents and the wider role of scent in the development of humanity. In this research, we apply machine learning techniques to examine and analyze cultural heritage texts with a special focus on olfaction and emotions. Specifically, using fairy tales as an example of cultural heritage, we identify odor-related contents and trace emotions linked to the identified units/objects. We apply semantic web technologies to detect objects associated with olfaction in textual sources and propose a new methodology for the extraction of emotions tied to smell. We present a new methodology for digital cultural heritage sources analysis. Our research indicates that, in historic texts, smell is often emotionally charged, which confirms existing research from several areas, including neuroscience, psychology, sociology, and literature studies. The main scientific contributions of this paper are the following:

• New emotion–olfaction detection text mining methods;
• A novel methodology to analyze contextual information related to mentions of smells in literary texts;
• An example of a novel approach to contextual emotions’ visualization;
• An overview of the role of smell and emotions in literature and linguistics;
• A neuroscience and psychological background for understanding the link between olfaction and emotions, and the importance of preserving cultural heritage related to olfaction based on this background.

The paper provides an interdisciplinary work with different views on the topic of emotional and olfactory analysis from technical, psychological, and cultural heritage perspectives. The innovative aspect of the paper lies in the combination of a smell and emotion extraction methodology in historical documents, represented by fairy tales. This has been done by providing a detailed discussion of the existing work and creating a tool to help domain experts in psychology and cultural heritage to explore the context, emotions, and smell experiences in fairy tales.

2. Related Work

In this section, we present related work in the area of Artificial Intelligence, semantic web, as well as an overview of relevant research and findings from computational linguistics, psychology, the neuroscience domain, and the digital cultural heritage area.

2.1. Olfaction, the History of Senses, and the Human Brain

Olfaction is the sense through which smells (or odors) are processed and experienced [1]. The processing, similar to other sensory domains such as vision, involves several levels. Most importantly, in the context of the current paper, the processing is tied to memory and emotional pathways. Evolutionary, it is considered the oldest of senses [2] and labeled as the “most primitive” [3]. Despite that, it remains essential for human survival [4,5], and fortunately, it can no longer be seen in the history of senses as a poor relative of the visual or auditory system. Rather, it is an avenue to research sensory processing. In addition, as historian Robert Jütte claims: “there can be no such thing as a natural history of the senses, only a social history of human sense perception” [6]. In her recent study on the relationship between smell and morality, Sara Silva elaborates on how medical and cultural historians, as well as anthropologists and psychologists, have emphasized the key role olfaction plays across time and place, not only through rituals, or as a means of diagnosing disease, but also as a warning mechanism regarding threats, dangerous environments, and people [7,8,9,10,11]. She explains how shared, familiar smells encourage a sense of identity and security, at both individual and group levels, whereas the opposite leads to smelly feelings of distrust, avoidance, and fear. Building on Classen’s thesis that smell is a cultural perception, Silva presents a historical overview of smell studies showing how in the eighteenth and nineteenth centuries, for instance, the tendency was to devalue smell, equating it with animals, savages, and degenerates. As historian Mark Jenner explains in his insightful “Follow Your Nose? Smell, Smelling, and Their Histories”, there was a negative association between “olfactory sensitivity” and “a lack of control over the emotions”, which seemed to suggest that “the sense of smell is inherently animalistic or instinctual and thus inclined to atrophy with civilization” [9]. From the Victorian period onward, particular smells and strong odors, such as perspiration, became associated with the lower classes and the proletariat in general, whereas more sophisticated scents, including soaps and light perfumes, were linked to the wealthy and aristocracy. In her discussion, Silva alludes to the emergence of what some scholars identify as a new olfactory standard, the “scent of cleanliness”, which led to a revolution in sanitary efforts during the nineteenth century [12]. Smell came to represent a sign of status and social class. Nowadays, in an increasingly globalized world, smell continues to be intimately connected to the politics of power and identity. Another important layer of smell lies in its connection to memory. Silva points out a particular expression coined by historian of medicine Jonathan Reinarz to describe authors who are able to tie the sense of smell to that of lasting memories. Reinarz calls these writers, including Charles Dickens and Émile Zola, “aromatic authors” [10]. On a similar note, psychologist Rachel Herz has also shown a relationship between memory and smell, arguing that “odors that evoke positive autobiographical memories have the potential to increase positive emotions, decrease negative mood states, disrupt cravings, and reduce physiological indices of stress” [13]. On the contrary, odors that elicit negative memories tend to lead to depressive dispositions.

Olfactory stimuli are complex, transient, and difficult to name (hence frequently described descriptively in connection to specific events: “smells like Christmas”) [14]. Yet, at the same time, odors can evoke memories with vividness and clarity not matched by other sensory inputs, including visual cues [15]. Odor-evoked memories can pertain to early age, earlier than those evoked by visual, auditory, or verbal cues [16]. The experience of odor-evoked autobiographical memories (OEAMs) is often so vivid that a significant number of people report feeling as if they were “brought back in time” [17]. A study by Arshamian and colleagues [15] showed increased activation in brain regions associated with visual vividness and emotions during odor-evoked recall of autobiographical memories (AM) compared to AMs evoked by verbal cues. Interestingly, in the same study, functional neuroimaging data showed that using a verbal description of AM-related odor increases the activation of olfactory and affective brain areas. Other studies showed a similar pattern when looking at the accuracy of recall of recently acquired information [17]: when new information was linked to scent, subjects were able to recall it faster and more accurately compared to information linked to visual cues. That remained true even if the scent was expressed in writing. Finally, odor-evoked memories are deeply saturated with emotions [15,18]. Evidence for the emotional potency of odor-evoked memories comes from studies using self-reported measures (e.g., [19]), as well as neuroimaging data [20]. Neuroanatomically, the olfactory system is connected to almost all parts of the limbic system—the part of the human brain associated with emotion regulation and experience and memory [21,22].

The Bayesian framework of the human brain assumes that our sensory perceptions—what we see, what we smell, or hear—are the outcome of an interesting process that builds on the incoming sensory input as well as the coded past inputs [23]. One way to see the effects of top-down processes at work is through sensory illusions. And while most scientific studies of illusions focus on illusions in the visual or auditory domain, a study by Herz and von Clef [24] provides an example of olfactory illusion, suggesting that top-down processes shape olfactory experiences and shift them away from the actual “raw” olfactory molecules. Further evidence of the role of past experience and learning on olfactory perceptions stems from studies showing that the human ability to discriminate odors is very low and improves with exposure [14].

2.2. Theories of Emotions—Psychological Perspective

“Emotions are the fire that fuels human behavior and the driving motivational forces in life” [25].

Despite the broad research contributions in affective science, defining emotion is still difficult. One possible definition of emotion is an “episodic, relatively short-term, biologically-based patterns of perception, experience, physiology, action, and communication that occurs in response to specific physical and social challenges and opportunities” [26]. Putting it simply, we can say that an emotion is a brief, active psychophysiological state due to appraisals about external events and objects. Emotions facilitate individuals’ responses to opportunities, challenges, and even threats [27]. Thus, one could say that emotions play a central role in the individual’s survival and social and environmental well-adjustment.

Several theories of emotion were developed in the field of psychology. For the purpose of this paper, we will rely on evolutionary theories, appraisal theories, and psychological constructionism theories. While evolutionary theories give emphasis to biologically evolved emotions, appraisal theories focus on the individual’s evaluation of events that lead to an emotional state, and psychological constructionism theories target the variation in emotional expression and experience between and within individuals [25].

Evolutionary theories of emotion are based on Darwin’s Theory of Evolution about the presence of facial emotional expressions across species, and its adaptive value to overcome environmental and social challenges [28]. Therefore, emotions are suggested to be evolved in response to adaptive issues, such as threats and the need for reproduction [29]. According to this view, individuals’ experience seven basic emotions—Surprise, Happiness, Sadness, Fear, Disgust, Contempt, and Anger. Basic emotions (basic emotions meet the following criteria: (1) they are universal expressions, (2) they have their own physiology print, (3) they are present in other primates, and (4) they result from environment automatic appraisals; the list of basic emotions is updated according to affective science development [25,30]) are innate, culturally universal, and automatically elicited in the presence of a relevant event (e.g., [29,31,32,33,34,35]). Every time that individuals are confronted with a relevant event, emotions provide individuals’ action tendencies—their affect program—to respond successfully toward challenges and opportunities [36,37,38].

Appraisal theories of emotion consider that emotions result from people’s evaluation of events for their well-being (e.g., [39,40,41]). Appraising events means that “individuals evaluate their surrounding circumstances in terms of how positive, novel, relevant to current goals, and congruent with norms their circumstances are, as well as whether the self or someone else was the initial cause of the circumstances” [25]. Therefore, contrary to evolutionary theories that describe emotions as discrete psychophysiological activations, in appraisal theories, emotions result from the subjective individuals’ event assessment; therefore, one could say that the same event will not elicit the same emotion across all individuals (e.g., [42,43]). Moreover, while different event appraisals evoke all components of emotion (psychophysiological activation, behavioral response, and cognitive response), some of them may be expressed or not due to social and cultural specificities. Thus, given an event, appraisal theories of emotion account for the unique experience of emotion rather than their uniformity across individuals [25].

Psychological constructionism theories of emotion intend to explain the heterogeneity of emotional experience and expression between and within individuals (e.g., [44,45,46]). According to this view, emotions are created psychological realities that rely on associative learning through life [25]. Thus, contrary to appraisal theories that defend an emotional triggering based on the appraisal of an event, the psychological constructionism theories suggest that the triggering of a specific emotion results from the socially learned association between the event and emotional response. Not all life experiences are categorized as emotionally charged; however, the emotional categorization of events is dependent on the individual’s knowledge about emotions [47] and their linguistic and cultural background [48]. Thus, not surprisingly, the expression of all components of emotion “depends upon the category that is used to construct an emotion in the situation in which is it occurring” [25]. Thus, “emotions are experienced when affective states are made meaningful as specific instances of the emotion categories that exist in a given culture” [49].

Communicating emotions through language is challenging. All nuanced emotions allied to cultural communication and language differences contribute to a rich field of research in emotion. The psychological constructionism views on emotion consider that emotions are experienced as a result of having affective states (body sensations) associated with emotion categories learned through the socialization process in each culture. The experience of emotion occurs when an event is categorized as emotionally meaningful.

The Conceptual Act Theory (CAT) addresses the role of language in the psychological constructionism of emotion (e.g., [50,51,52]). According to the CAT, emotion categories are abstract concepts based on language; thus, it is through emotion-related words that individuals make sense of their emotional experiences and perception of other people’s emotional experiences. Therefore, differences in language between individuals influence the emotional experience and perception of the same event (see [53]). The “emotion lexicon” used by individuals to describe their emotions are dependent on their culture, historical time (for a review, see [49]), and their own language development through the lifespan [54]. Despite the ambiguities that may arise in emotional categorization between individuals, we can say that the richer the emotional lexicon, the richer the emotional experiences.

2.3. Emotions and Smell

Smell has an emotional component, and it can serve as a cue to moods and motivations in social contexts. Indeed, a growing body of research shows that human beings communicate their moods and emotions through chemosignals [53]. The Search Engine Hypothesis [55] postulates that individuals automatically and unconsciously search for odor cues in the environment. This search includes smells that people are aware of and are not aware of and facilitates their navigation in different social situations and environments. Individuals can smell negative moods [56], fear, happiness, calmness, safety [57,58], and anxiety (e.g., [59]). Moreover, individuals can experience emotions when they smell them. A study that explored the relationship between smelling emotions (happiness, fear, and disgust) and facial expressions showed that individuals tend to match their facial expression according to the smelled emotion [60]. Environmental smell affects our emotions and social behaviors. The pleasant smell of flowers is associated with positive emotions and positive social interactions [61], calmness, and alertness (e.g., [62]). On the other hand, unpleasant smells are associated with an increase in stress (e.g., [63]) and avoidance behaviors [64]. Exposure to the Ethyl Mercaptan smell (e.g., sewage) affects judgments and performance [65], and exposure to industrial farming malodor is associated with depression and a poor quality of life (e.g., [63]).

From a social psychology perspective, the role of learning in shaping the experience of olfactory stimuli combined with a long history of linking discriminated groups with different, mostly negative, odors [66] carries the potential of using people’s subjective experiences of various scents as an indirect way to tab and measure stereotypes, prejudice, and negative affect toward minority groups. The unique capacity of odors to evoke strong emotional responses make odor-related stereotypes a specially powerful tool in propagating negative attitudes toward members of discriminated groups. However, while experimental studies can shed light on currently existing odor-based stereotypes, literary texts provide an excellent avenue to trace and study changes in the use of odor-related descriptions of people as tools to discredit and stereotype social groups.

2.4. Emotions in Literature

Emotions and feelings are essential to the function of literature. Through language, fictional worlds can be created that affectedly relate to our world [67]. Johansen [67] states that feelings are important to the function of literature, because through language, it is possible to create fictional worlds that in many respects are related to the life in our world. We are, as Jonathan Gottschall compellingly puts it, storytelling animals [68]. Indeed, narratives and literature are central to the construction of the self, embodied with memories, emotions, and specific cultural values. As Jesse Graham and Jonathan Haidt note, “people love stories. Cultures rely on stories to socialize their children, and narrative thinking has been called one of two basic forms of human cognition. Successful stories–the ones that get transmitted–are those that fit well with the human mind by eliciting strong emotions” [69]. This realization impacts deeply on the way we perceive (or want to be perceived) ourselves, and others, physically, morally, and emotionally. It is through shared stories that we build our social identities, judge right from wrong, ascribe value to experiences and things, and feel motivated to act according to common beliefs. Moreover, several studies have shown that reading fiction improves the development of the Theory of Mind (ToM) in comparison to reading non-fiction and identified a link between reading fiction and an enhanced capacity for empathetic responses [70,71,72].

One of the difficulties when thinking about emotion(s) has to do with its mercurial nature. As Sara Silva argues, “overall, the error seems to stem from a misguided preoccupation in limiting and circumscribing different dimensions into a sole reality instead of accounting for the multiplicity of representations” [73]. In her book, Emotions in History—Lost and Found (2011), Ute Frevert, a historian at the Berlin’s Max Planck Centre for the History of Emotions, calls attention to this mutability in meaning by elaborating on emotions’ constant cycle of loss and renewal: “Emotions and emotional styles fade away and get lost (like honour or acedia) but [the historical economy of emotions] also witnesses the emergence of new or newly framed emotions. Empathy, sympathy/compassion serve as great examples of emotions that are found and invented in the modern period” [74]. In 2014, Frevert and her team pursued this discussion and embarked on a study of the definition of emotions in English, German, and French encyclopedias since the seventeenth century in an attempt to show how these reference works acted as moral regulators of emotional expression.

2.5. Computational Linguistics: Extracting Emotions and Smells

With the progress in the domain of the applications of machine learning (ML), various approaches have been developed for text representation. Among them, Bag-Of-Words (BOW), where each sentence in the dataset is represented by a feature vector composed of Boolean attributes per word that occurs in the sentence. N-grams, on the other hand, describe sequences of units of length n: words, phonemes, characters, etc. While BOW considers words as independent entities and it does not take into consideration any semantic information from the text, N-grams can be used for catching syntactic patterns in text and may include important text features such as negations, e.g., “not happy”. Negation is an important feature for the analysis of emotion in text because it can totally change the expressed emotion of a sentence. For instance, the sentence “I’m not happy” should be classified into the sadness category and not into happiness. For these reasons, some research studies in sentiment analysis claimed that N-grams features improve performance beyond the BOW approach [75].

These feature representations have been used with traditional machine learning models, such as SVM or decision trees. Deep neural networks use special layers to represent the words in a dense vector representation called embedding layers, in contrast to the sparse BOW or NGRAM feature vector representation. This representation captures some semantic meaning of the words, where words that have the same meaning have a similar representation [76]. These word-embedding layers can be trained and used independently to generate feature vectors, which later could be used in different models, or it could be integrated as part of a larger neural network model.

Transformer-based architecture is a recent neural network architecture that has been the state-of-the-art method in several natural language processing tasks, ever since its introduction in [77], in 2017. One of the famous transformer-based models is BERT [78], which uses the encoder part of the transformer architecture. Several variations of BERT were later introduced, including RoBERTa [79], XLM-RoBERTa [80], and DistillBERT [81]. In our work, we utilize the aforementioned transformer-based models and apply them to the topic of detecting smell and emotions within the cultural heritage domain, bearing in mind that the specific combination of smell and emotions in the fairy tales domain was not previously covered in the related work.

Computational linguistics and literary studies have tried to capture the emotional landscape of literature through text mining methods, for instance, classifying novels on happy or sad endings through machine learning based on sentiment lexicons [82]. More specifically, Zehe et al. [82] presented an approach to classify novels as either having a happy ending or not, using features based on different sentiment lexica as input for a Support Vector Machine (SVM) classifier.

As emotions occur in the text at a specific moment in time and space, it becomes possible to link them to specific geographical locations. Interesting research work has been performed by Stanford researchers that mapped fear and happiness in historic London [83].

An interesting story-type clustering technique has been used by Reagan and colleagues [84]: story-type clustering showing a quadratic relationship between the story’s plot and the intensity of events. The visualization shows a time-series graph, with the x-axis representing a time in the story and the y-axis representing the events happening to the main characters that can be favorable (peaks on a graph) or unfavorable (troughs on a graph).

Zwaan et al. [85] present an appealing work on the applications of the Historic Embodied Emotion Model (HEEM) to emotions in historical texts. Research involving measuring self-reported affective responses to odors and odorous products by Ferdenzi et al. [86] resulted in the development of the cross-cultural Emotion and Odor Scales (EOSs). In this paper, we are using state-of-the-art text mining techniques to address the domain of smells (or odors) in connection with emotions in literature texts. The primary research goal of the paper is measuring self-reported affective responses to odors and odorous products.

In comparison to the existing related approaches to emotion mining, one of the goals of this research is to explore the specific topic in the olfactory area within specific cultural heritage resources, such as fairy tales. Similar to information extraction from text methods, we utilize state-of-the-art techniques to obtain emotions on the sentence level. However, for analysis and visualization scenarios, we also include context represented via semantic annotations.

Identification of the language used to refer to olfactory experiences is achieved by creating language resources and developing automated tools. Tekiroğlu et al. (2014) [87] have created and evaluated a sensorial lexicon for this target. Brate et al. (2020) [88] created a semi-supervised system for automatically detecting smell information. The need for an ontology that can enable us to be consistent in the annotation of olfactory information across studies was reported by Tonelli and Menini et al. (2021) [89] and Menini et al. (2022) [90]. On the basis of this ontology, the multilingual Odeuropa benchmark dataset was released [91]. The Odeuropa benchmark dataset is multilingual and consists of historical texts. We utilize this language resource to create transformer-based models [77] for smell-related sentence identification.

Semantic annotation based on Wikipedia concepts, often called simply wikification, has been the subject of much interest, starting with the work of Michalcea and Csomai [92], and may be seen as a special case of the entity linking problem, with Wikipedia concepts corresponding to entities. Wikification is generally based on two steps: (1) identify mentions of entities in the document to be annotated, and (2) for each mention, determine which (if any) of several candidate entities this mention refers to. This generally involves defining a similarity measure between a mention and a candidate entity, as well as some criterion that promotes the coherence of entities across multiple mentions (e.g., to ensure that the mention “Tesla” is not linked to the inventor if the rest of the document is about the car manufacturer). While older approaches generally used ad hoc measures of “semantic relatedness” and the like [93], more recent work often uses deep neural models to obtain vector representations of mentions (including their context) and entities [94,95]. Alternatively, some studies emphasize simple and fast-to-compute methods suitable for very large-scale wikification. For example, Shnayderman et al. [96] describe a system that identifies entities based on Wikipedia titles and redirection links, combined with a heuristic that improves precision by discarding the annotations they are least confident about.

Some recent work has also focused on wikification within a narrower problem domain, observing that typical general-purpose wikifiers may or may not perform well when applied to such a narrower domain. Examples include wikification of scientific abstracts [97], wikification for COVID-19 [98], and wikification of documents about software development [99]. Our work presented in this paper is involving wikification in the olfactory domain and may also be situated in this context.

In [100], the researchers perform an analysis on the topic of emotions in literature. In comparison to [100], we improve on the results by using different modeling approaches and testing against more models. We provide an extensive overview of the role of smell and emotions in examples of folktale, highlighting the importance of olfactory and emotional experiences from different perspectives. We present quantitative results along with qualitative scenarios for visualization purposes.

2.6. Review of Folktales

Because, in our research, specific attention is dedicated to the analysis of smells and emotions in fairy tales, in this section, we provide an overview of the related work in the area of folktales. Folktales’ enduring appeal is intrinsically linked to the power of storytelling and magic from time immemorial. This explains why their origin is one of the biggest mysteries in folktale studies. As scholars of the field widely acknowledge, its reconstruction is often frustrated not only by difficulties in defining the genre but also as a result of the rich interplay between oral and written traditions [101,102]. The German word märchen, usually translated into English as fairy tale or household tale, and to French as conte populaire, is the internationally established term associated with the genre [103]. A märchen is normally described as “a tale of some length involving a succession of motifs and episodes. It moves in an unreal world without definite locality or definite characters and is fuelled with the marvellous. In this never-never land, humble heroes kill adversaries, succeed to kingdoms and marry princesses” [104]. The motifs depicted in fairy tales are timeless and fairly universal, comprising dichotomies such as good/evil, right/wrong, punishment/reward, moral/immoral, trust/distrust, honesty/deceptiveness, male/female, amongst others. The potential of literature, particularly of traditional literature, to study behavior is nevertheless highly underestimated. Despite being often disregarded as simply fictional and even as a lesser form of literature, the extraordinary variability of tales, and of the values they communicate, makes them ideal case studies for cross-cultural comparisons on social dynamics, including cooperation, competition, decision making, etc. Indeed, albeit fictitious, they can function as important simulations of reality. Moreover, their impact in society is two-fold: on the one hand, they are capable of expressing emotions that people recognize and identify with (or not), while on the other hand, they have the potential to elicit emotional/moral responses. This knowledge is transmitted across generations and cultures, validating tales as a privileged means for the transmission of shared, collective values that drive our behavior. As Thompson recognizes, “like all other elements of culture, folktales are not a mere creature of chance. They exist in time and space, and they are affected by the nature of the land where they are current, by the linguistics and social contacts of its people, and by the lapse of the years and their accompanying historical changes” [104].

Stories commonly known today as fairy tales were first recorded during the nineteenth century in the excitement that followed Wilhelm and Jacob Grimm’s Children’s and Household Tales, first published in 1812, in a total of seven editions. It was in the early nineteenth century that folklorists such as the Grimm brothers started to worry about classification—a transversal concern to many areas in this period given that science was beginning to be professionalized. As tireless philologists, the brothers were certain that the stories they collected did not belong only to Germany but were remnants of an ancient Indo-European cultural tradition. Their conviction is clearly reflected in the index The Types of International Folktales, also known as ATU [105]. The catalogue is based on a numeric system, ranging from 1 to 2399 tale types across more than 200 cultures worldwide. The “Tales of Magic” category comprises types 300 to 749 and represents the largest and most widely shared group because it includes the canonical fairy tales we grew up with, such as Little Red Cap (ATU 333), Sleeping Beauty (ATU 410), Snow White (ATU 709), the Frog King (ATU 440), and many others.

3. Methodology

The proposed methodology for harvesting context and mining emotions from folk tales incorporates the following steps.

• Training smell models. Machine learning-based text classification models are developed for prediction if a sentence is about smell or not.
• Extracting smells. Smell-related sentences are extracted from the collection of historic texts. In addition, on the top of a smell detection model, the olfactory objects are extracted from text.
• Training emotions model. Deep learning techniques are used in the process of developing models for prediction of emotions.
• Extracting emotions. Predefined specific emotions are extracted from the collection of historic texts.
• Extracting context. We perform semantic annotation in the process of context identification.
• Visualizing smells, emotions, and context. For visualization purposes, in this research, we present a novel smell tracker tool that provides possibilities for the users to explore smells, emotions, and context extracted from digital cultural heritage texts (such as fairy tales).

Figure 1 contains an overview of the methodology architecture. The implementation section describes the specific details related to the methodology steps.

Figure 1. Methodology overview.

4. Implementation

This section describes the implementation details of the proposed methodology, including the data descriptions, and qualitative analysis of our dataset of fairy tales.

4.1. Implementation Details

For smell extraction, we utilized the English part of the Odeuropa benchmark [91,106] to create a machine learning-based text classification model that predicts whether a sentence is smell-related or not. Although the Odeuropa is annotated at token level, we converted the sentences that contain any smell event annotation to be smell-related and remaining sentences as not-smell-related. Out of the total 3141 sentences, 897 were marked as smell related. We randomly chose 650 (190 smell-related, 460 not-smell-related) sentences as held-out for evaluation. The rest served as training and development data for creating the machine learning model. The ratio between training and development sets is 0.85 and 0.15.

In addition, for extracting relevant olfactory objects related to the smell experience, we used a set of 430 predefined (by Odeuropa project partners [107]) olfactory objects. The examples of olfactory objects include “”fish”, “flower”, “rose”, etc.

For training our emotions models, we used the affect data distributed by Cecilia Ovesdotter Alm [108]. The final dataset consisted of 1207 sentences from tales, classified into the following emotion-related classes: Angry-Disgusted, Fearful, Happy, Sad, Surprised. Annotations have been produced by 2 annotators, each assigning a primary emotion and a mood from one of the classes, and only sentences with four identical classes (primary emotion and mood for both annotators) have been included. Table 1 shows the distribution of sentences by emotion.

Table 1. Emotion distribution in training data.

Emotion	Sentences
Angry-Disgusted	218
Fearful	166
Happy	445
Sad	264
Surprised	114
Total number	1207

To create a machine learning model that can classify sentences as smell-related or not, we fine-tuned BERT [78], RoBERTa [79], and macBERTh [109,110] models five times using five different random seeds (42, 43, 44, 45, 46), batch size of 64, maximum token length of 64, learning rate ($2\times {10}^{-5}$), epochs (30), and random splitting for obtaining a development set from the training set (0.15). All randomness that could originate from the software used was controlled and set to the same random seed. This means two runs of the same code with the same random seed generate exactly the same result.

The olfactory object extraction was built on top of the smell detection model. The goal of this step was to identify any olfactory object that might have linked to the smell event. From the sentences that were classified as containing smell, we identified the olfactory objects from the sentences using string matching, taking into consideration the different forms of keywords in the text.

The proposed methodology for training emotion detection models was based on deep learning techniques as well. For the experiment, we used an architecture that consists of a transformer, a dropout layer, and a linear layer, with Adam as optimizer. We used pretrained transformers, including BERT [78], DistilBERT [81], and XLM-RoBERTa [80], and fine-tuned them on the applicable dataset. Emotion classification has been performed using 80-10-10 split for train–evaluation–test, resulting in 965 examples for training, 121 for evaluation, and 121 for testing purposes. Training condition included stopping training after 15 iterations with no improvement of accuracy score on the evaluation data. The experiments were performed as multiclass classification settings.

For extracting context, we extracted entities with a relevant Wikipedia concept from the text. The JSI Wikifier tool [111] was used, which is a service developed in Jozef Stefan Institute, that annotates a given raw text with annotations, each representing a Wikipedia concept.

For each document in our dataset, we used Wikifier on the raw text provided and obtained a list of annotation objects; each contains information such as the annotation name representing the Wikipedia concept, the Wikipedia URL, page rank score, among others.

After all the annotation steps, the final data contain the following information:

• Tale title/name.
• Tale text.
• Sentences information:

  -

  Sentence text;

  -

  Sentence emotion.

• Semantic concepts.
• Olfactory objects.
• Dominant emotion: most frequent emotion.
• Emotion distribution: distribution of emotions in sentences.

This information was formatted and ingested into Elasticsearch database for further qualitative analysis.

4.2. Qualitative Analysis: Analyzing Fairy Tales Data Using the Smell Tracker Tool

For the purposes of a qualitative data analysis, we have obtained a collection of 367 fairy tales from Andersen tales and Brothers Grimm tales. These tales were processed using the pipeline mentioned in the methodology, and the results were integrated into the smell tracker analytical tool.

The smell tracker tool [112] provides an analysis of different cultural and literature corpora related to smell by means of interactive visualizations and a dashboard. In this paper, we will focus on the tales dashboard which contains the analysis of the annotated tales dataset with emotions and smell objects; Figure 2 shows a snapshot of the dashboard.

Figure 2. Snapshot of the smell track dashboard for tales dataset.

The dashboard can be split into three parts managed by a control panel, where the results can be filtered through keyword search, olfactory object, concept, tale name, and predominant emotion. Upon applying the filters, the records will be filtered to these matching the condition, and the results of all the analyses will be updated accordingly.

The first part contains the top-level analysis, which contains the total number of tales, and the dominant emotion distribution, which can be seen in Figure 2. These numbers change based on the selected filter, which would help in finding, for example, how many tales have the “garden” olfactory object, what are the tales’ emotion distribution (based on the top emotion) for tales that include the concept “Death”.

Furthermore, using the tales’ emotion distribution, the top tales for each emotion are shown along with the percentage of sentences with that particular emotion. In Figure 3, we see the tales with the largest ratio of fearful sentences.

Figure 3. List of most fearful tales based on sentences’ emotion distribution

The second part focuses on content analysis and extracted content, namely semantic concepts and olfactory objects. The content analysis is provided in the form of three tag clouds representing the semantic concepts, olfactory objects, and the significant terms. Significant terms are top terms that distinguish the selected tales’ content (after filtering) from the rest of the tales. Figure 4 contains the tag cloud of top keywords in happy-dominant tales vs, sad-dominant tales. In the figure, we see the difference in keywords distinguishing happy tales.

Figure 4. Top keywords extracted from sad (left) vs. happy (right) tales.

The tales’ timeline focuses on the tales’ emotional progression throughout the tale. It shows the evolution of events by means of emotions throughout the story and helps observe the different story lines, such as tales with happy endings, cautionary tales, etc. The timeline is calculated by splitting each tale into 10 chunks, and for each chunk, the number of sentences that fall under it and that have a particular emotion are calculated. The intensity score represents the share of sentences that have a particular emotion in that chunk from the total number of sentences in that chunk. By filtering for a specific tale, one can observe the story timeline for that particular tale; alternatively, by filtering for a specific concept, it shows the average emotional timeline for all tales that share this concept. Figure 5 shows the emotional storyline progression of the tale “The Frog King or Iron Heinrich”. A more detailed sentence-by-sentence emotions’ list is found in Appendix A.

Figure 5. Emotion timeline of the “The Frog King or Iron Heinrich” tale.

5. Evaluation

Transformer-based models are the state-of-the-art models in most NLP tasks. BERT-base models represent the base model of the transformer-based architecture. Moreover, both RoBERTa and XLM-RoBERTa use byte-level BPE as a tokenizer, making them more suitable for training on a cross-lingual setting. DistilBERT provides a lighter alternative to BERT that might be more suitable for production. Finally, macBERTh was trained on historical data, making it suitable given the same type of text used in the smell dataset.

For smell detection, the median F1-macro scores of the three different models, which are BERT, RoBERTa, and macBERTh, are 90.43, 92.11, and 92.72, respectively. The standard deviation for the results is comparable across the model types. Consequently, we utilized the macBERTh model that yields the best score to classify the sentences in our fairy tales corpus.

The generalizability of the fine-tuned model to the fairy tales corpus was measured by manually annotating all sentences that are predicted as smell-related and the same number of randomly chosen sentences that were predicted as not-smell-related in the fairy tales corpus. The model has identified 192 sentences as smell-related. Therefore, 384 sentences were annotated by two independent annotators. The inter-annotator agreement between these two annotators was 0.88 in terms of Krippendorf’s alpha. The annotators disagreed on 21 sentences. The performance of the best smell-sentence classifier is 91.62 F1-macro, which shows that the prediction quality did not drop significantly in the target domain.

For multiclass emotions learning, we trained a BERT-based model, DistilBERT-based model, and an XLM-RoBERTa model. The baseline is 36.8 percent.

In multiclass emotions learning, we observe that all models largely outperform the baseline, which is 36.8%. Moreover, BERT achieved slightly better results on F1 than XLM-RoBERTta and 2.4% than DistilBERT (see Table 2). Details on the class-by-class performance for the BERT-based model can be seen in the confusion matrix in Table 3.

Table 2. Results for emotion classification task with three deep learning models.

Model	Accuracy	Recall (Macro)	Precision (Macro)	F1 (Macro)
DistilBERT	80.2%	75.9%	78.5%	76.8%
BERT	81.8%	80.8%	79.9%	79.2%
XLM-RoBERTa	82.6%	78.4%	79.6%	78.6%

Table 3. Confusion matrix of the test data on emotion classification using the best performing model—BERT-based model.

Happy	40	1	2	0	2
Fearful	0	15	0	0	2
Sad	3	2	20	0	1
Angry-Disgusted	3	2	0	15	2
Surprised	0	1	0	1	9
—	Happy	Fearful	Sad	Angry-Disgusted	Surprised

6. Discussion

In the paper, we have proposed a novel methodology combining smell detection and emotion detection, along with its implementation and evaluation on fairy tales. Looking for context and emotions in digital cultural heritage sources brings a number of interesting challenges. First of all, because the combination of smells with context and emotions in historical sources and Artificial Intelligence (AI) applications is novel and unique, it is difficult to directly compare to other related work. A number of challenges associated with emotion detection for digital cultural heritage are related to the fact that the description and representation of emotional lexicons might change over time. While working with fairy tales, one can also observe that fairy tale adaptations to other languages might be different from the original texts. English adaptation might convey different emotions from the German original due to the cultural differences in expressing emotions [113].

In the proposed methodology, the smell-related sentences in texts are detected using a machine learning model trained on a benchmark dataset. The performance of the prediction on the target domain (sentences extracted from fairy tales) is comparable to the performance obtained in the source context (sentences extracted from historical books). Both the source and the target contexts are historical documents. We speculate that this similarity allows the model to perform well across these two domains. The smell-sentence classifier performance was 91.62 F1-macro. Although annotation of 192 random sentences that are not predicted as not-related-to-smell may not be sufficient, it is still a good indication of the completeness of the proposed model for smell-related sentence detection.

The machine learning model classified 30 sentences as smell-related, while both annotators label them as not-smell-related. The manual inspection of these sentences reveals that (i) short sentences such as “said Tailor Ölse” and “O Marjory!” and (ii) the ambiguity of some words such as the verb “smoke” cause the machine learning to yield wrong results. The reason for some of the errors was not clear. For instance, the sentences “He thought it strange that the old woman was snoring so loudly, so he decided to take a look.” and “And he talked about farming, but you couldn’t hear much of what he said, because of the coughing and gasping.” should not have been predicted as smell-related as they do not contain any smell-related information.

7. Conclusions

The transient character of olfactory stimuli and their experience, the role of learning through exposure in the recognition and accurate discrimination of scents, and the necessity to frequently rely on events to describe olfactory experiences (e.g., “the smell of Christmas”), combined with the constantly changing world and with it, the olfactory stimuli humans are subjected to, pose interesting challenges. How do we trace the changes in human olfactory experiences? Where can we find and how to best extract meaningful information about past olfactory realms and the emotional valence linked to odors long absent and forgotten? Harvesting context and mining odor-evoked emotions provide a unique opportunity to successfully tackle these issues. This paper describes an open-source tool that can afford researchers the means to address these and many other questions related to emotions and olfactory experiences.

In this paper, we presented a novel methodology combining state-of-the-art text mining techniques with the emotion mining of smells in the domain of cultural heritage narratives. We provided an overview of the role of smell and emotions in literature, as well as highlighted the importance of olfactory experience and emotions from psychology and linguistic perspectives. We applied AI techniques to analyze narratives, such as fairy tales, to identify and trace smell and to determine the emotions to which it was linked. We used semantic web technologies to detect the context associated with smelling and olfactory objects in textual sources. We suggested a new methodology for digital cultural heritage sources analysis and found that fairy tales often connect smell with the emotional charge of situations. We demonstrated a novel smell tracker tool that enables active user engagement in the process of cultural heritage analysis and provides interesting visualizations for cultural heritage (folklore) experts and for the general public.

In the evaluation phase, we conducted quantitative and qualitative evaluations, involving technical experts from the domain of Artificial Intelligence and machine learning for building state-of-the-art models for smell extraction and emotions extraction, as well as cultural heritage experts for qualitative recommendations in the field of fairy tale studies. The quantitative experimental results showed that we can detect smells and emotions with F1 scores of 92.7 and 79.2, respectively. In the appendix, we provided sentence-by-sentence predictions of the emotions of the “The Frog King or Iron Heinrich” tale recommended by fairy tale experts.

Future work will focus on applying the developed methodology for different literary genres, as well as the research for historical text analysis regarding the extracting smells and emotions detection problem.

Moreover, we focused our experiments on the English translations of fairy tales. In the future, we plan on testing the models against different translations and comparing the emotions exhibited in different variations and translations of the same tale.