Interdisciplinary IoT and Emotion Knowledge Graph-Based Recommendation System to Boost Mental Health

Abstract

Humans are feeling emotions every day, but they can still encounter difficulties understanding them. To better understand emotions, we integrated interdisciplinary knowledge about emotions from various domains such as neurosciences (e.g., neurobiology), physiology, and psychology (affective sciences, positive psychology, cognitive psychology, psychophysiology, neuropsychology, etc.). To organize the knowledge, we employ technologies such as Artificial Intelligence with Knowledge Graphs and Semantic Reasoning. Furthermore, Internet of Things (IoT) technologies can help to acquire physiological data knowledge. The goal of this paper is to aggregate the interdisciplinary knowledge and implement it within the Emotional Knowledge Graph (EmoKG). The Emotional Knowledge Graph is used within our naturopathy recommender system that suggests food to boost emotion (e.g., chocolate contains magnesium that is recommended when we feel depressed). The recommender system also answers a set of competency questions to easily retrieve emotional related-knowledge from EmoKG, such as what are the basic emotions and the more sophisticated ones, what are the neurotransmitters and hormones related to emotions, etc. To follow FAIR principles, EmoKG is mapped to existing knowledge bases found on the BioPortal biomedical ontology catalog such as SNOMEDCT, FMA, RXNORM, MedDRA, and also from emotion ontologies (when available online). We design the LOV4IoT-Emotion ontology catalog that encourages researchers from heterogeneous communities to apply FAIR principles by releasing online their (emotion) ontologies, datasets, rules, etc. The set of ontology codes shared online can be semi-automatically processed; if not available, the scientific publications describing the emotion ontologies are semi-automatically processed with Natural Language Processing (NLP) technologies. This research is also relevant for other use cases such as European projects (ACCRA for emotional robots to reduce the social isolation of aging people, StandICT for standardization, and AI4EU for Artificial Intelligence) and alliances for IoT such as AIOTI. The recommender system can be extended to address other advice such as aromatherapy and take into consideration medical devices to monitor patients’ vital signals related to emotions and mental health.

1. Introduction

What are emotions? Humans are feeling emotions every day, but they can still encounter difficulties understanding them. Emotions have been studied in various disciplines, from psychology to neuroscience. Collecting and integrating this interdisciplinary knowledge is challenging. In this paper, we integrated interdisciplinary knowledge about emotions from various domains (see Figure 1) such as neurosciences (e.g., neurobiology), physiology, and psychology (affective sciences, positive psychology, cognitive psychology, psychophysiology, neuropsychology, etc.). To organize the knowledge, we employ technologies such as Artificial Intelligence with knowledge graphs and semantic reasoning. Furthermore, Internet of Things (IoT) technologies can help to acquire physiological data knowledge. To facilitate the interoperability between sensors, software, and data, standards are also employed. In fact, the integration of heterogeneous technologies, and sensors requires an interoperable solution to describe sensors and data exchange. If the enterprises do not develop the full stack that is compatible from sensors to the final application for consumers as well as compatible with other solutions, they can fail. It demonstrates the need for interoperability between sensors, software, and the data processed. In this paper, we focused on data semantic interoperability to deduce meaningful information, which was applied to the IoT-based emotion domain.

Ontologies in affective science are disseminated on the web (or even unshared) in different formats and different structures. Ontologies have been developed in AI to facilitate knowledge sharing and reuse. “An ontology is an explicit specification of a conceptualization” according to Gruber et al [1]. Ontology methodologies such as 101 ontology methodology [2] and NeOn [3] encourage reusing existing ontologies. Knowledge Graph (shorturl.at/duz28, accessed on 17 September 2022) [4,5] is the new term, which was popularized by Google. Finding, reusing and adding explicit mapping are time-consuming challenging tasks.

The goal of this paper is to aggregate and make interoperable the interdisciplinary knowledge and implement it within the Emotional Knowledge Graph (EmoKG). To illustrate the use of our Emotion Knowledge Graph, we build the emotion naturopathy recommender system that suggests food to boost emotion (e.g., chocolate contains magnesium that is recommended when we feel depressed). The recommender system also answers a set of competency questions such as what are the basic emotions and the more sophisticated ones, what are the neurotransmitters and hormones related to emotions, etc. The recommender system can be extended to address other advice such as aromatherapy and take into consideration medical devices to monitor patients’ vital signals related to emotions and mental health.

Recommendation Systems (RS) [6,7] are embedded in the services that we are employing every day (e.g., videos with YouTube, products with Amazon, and movies with Netflix). RS system surveys (machine learning-based RS [8]) [9,10,11] do not focus on emotions to enhance our mental health. Nouh et al. [12] introduce the research need for Well-being RS. Garcia et al. [13] survey mental health services using IoT devices. There is still a research gap to design IoT-based emotional RS for mental health. There are several kinds of recommendation systems: (1) content-based (CB), which comes from information retrieval and information filtering; (2) collaborative filtering (CF), which predicts item utility for a specific user based on the items rated by other users in the past; (3) knowledge-based, which we focus on in this paper; and (4) hybrid approaches.

There is a need to design an emotional knowledge graph to be employed within a knowledge-based recommender system to boost mental health. To reuse past expertise and follow FAIR principles [14], the EmoKG emotional knowledge graph is mapped to existing knowledge bases found on the BioPortal biomedical ontology catalog such as SNOMEDCT, FMA, RXNORM, MedDRA, and also from emotion ontologies (when available online). We design the LOV4IoT-Emotion ontology catalog that encourages researchers to apply FAIR principles [15] by releasing online their (emotion) ontologies, datasets, rules, etc.

The set of emotion ontology codes shared online can be semi-automatically processed; if the ontology code is not available yet, the scientific publications describing the emotion ontologies are semi-automatically processed with Natural Language Processing (NLP) techniques to feed the emotion reasoning engine and build the naturopathy recommender system.

We address the following research questions (RQ):

• RQ1: Can we reuse the domain expertise, and what are the shortcomings of past ontology-based emotion IoT projects?
• RQ2: What are the sensors used in the emotion domain? Are there standardized sensor emotion dictionaries? Is there emotion ontology supported by standards?
• RQ3: What are the AI technologies (e.g., rule-based inference engine) to deduce meaningful information from emotion sensor data so developers can design faster IoT-based emotion services?
• RQ4: How to prove the veracity of the reasoning engine?

The key contributions of our research are:

• C1: Ontology-based emotion projects shared though the LOV4IoT-Emotion ontology catalog tool (

  Table 1. Related work synthesis: Ontology-based emotional projects and reasoning mechanisms employed. Legend: Ontology Availability (OA); when the code is available, the ontologies are classified on the top. Then, the ontology-based projects are classified by year of publications. Several years can appear to highlight that the ontology was maintained from the first publications to the latest publications that we are aware of.

  Authors	Year	Project	OA	Reasoning	Text Analysis
  BioPortal [20]	2021	Biomedical ontology catalog	✓	No	No
  	2009	but IoT ontologies for emotion not found
  LOV [21] Linked	2021	Ontology Catalog	✓	No	No
  Open Vocabularies	2015	designed by the Semantic Web Community
  Hastings, Larsen et al.	2018	Emotion Ontology (EMO)	✓	No	No
  [22,23]	2011	MFOEM: Mental health and disease ontologies
  Switzerland, Canada		Taxonomy of emotions
  Lin et al. [24,25]	2018	Emotion and Mood Ontology	✓	No	No
  	2017	representing visual cues of emotions
  Gil et al. [26] based on	2015	Emotion and Cognition ontology used to understand	✓	✓owl:Restriction	No
  Spain		emotions from Emotiv EEG neuroheadset for online learning
  Lopez et al. [27]	2008	Describing emotions	✓	No	✓
  Berthelon et al. [28,29]	2013	Emotion Ontology for Context Awareness	✓	✓Corese, SPARQL,	✓Camera, heart rate
  France		Taxonomy of 6 emotions		regression
  Sanchez-Rada et al.	2018	Onyx: Describing emotions on the web of data	✓	No SPIN mentioned	✓
  [30] UPM/Spain	2016
  Abaalkhail et al. [31]	2018	Survey on 20 ontologies for affective states	No	No	No
  		and their influences
  Tabassum et al. [32]	2018	EmotiOn ontology for emotion analysis	No	✓HermiT for consistency	✓
  		based on Plutchik’s wheel of emotions		and inference
  Chen et al. [33]	2015	ECG ontology	✓	✓Event–Condition–Action	✓ECG
  Arguedas et al. [34]	2015	Emotion and Mood Awareness for E-learning	No	✓Event–Condition–Action	✓
  		(wiki, chat, forum)		(ECA) rule system
  Tapia et al. [35]	2014	Semantic Human Emotion Ontology (SHEO) for text	No	✓SWRL rules	✓
  		and images DetectionEmotion: Facial complex emotions		for complex emotions
  Sykora et al. [36]	2013	Emotive ontology for social networks	No	No	✓
  		message analysis (Twitter)
  Ptaszynski et al. [37]	2012	Ontology for extracting emotion objects from texts (blogs)	No	No	✓
  Baldoni et al. [38]	2012	Emotion ontology for Italian art work annotated tags	No	✓Jena Reasoner	✓
  Honold et al. [39]	2012	Emotion ontology for nonverbal communication	No	✓Bayesian	No
  Eyharabide et al. [40]	2011	OLA: Ontology for Predicting Learners’	No	✓inference,	No
  		Affect to infer Emotions		Decision Tree (Weka)
  Francisco et al. [41]	2010	Recognizing emotions from voice and text	No	✓OWL DL, Pellet, Racer	✓
  	2006			owl:Restriction, inference
  Grassi et al. [42]	2011	Human Emotion Ontology (HEO) for voice,	No	No	✓
  	2009	text, gesture, and face
  Radulovic et al. [43]	2009	Smiley Ontology (SO): emoticons (smileys as texts or pictures)	No	No	✓
  		with annotation of emotion classes
  Yan et al. [44]	2008	Chinese emotion ontology based on HowNet for text analysis	No	✓Rules for annotation	✓
  Benta et al. [45]	2007	User’s affective states	No	✓Fuzzy Logic in OWL	No
  	2005	for Context-Aware Museum Guide		(logical inference)
  Garcia-Rojas et al. [46]	2006	Emotional face and body expression profiles	No	✓RacerPro and nRQL	No
  		for virtual human (MPEG-4 animations)		(natural Racer
  				Query Language)
  Obrenovic et al. [47]	2005	Ontology for description of emotional cues	No	No	✓
  	2003	from different modalities (text, speech)

  ); it addresses RQ1 in Section 3.3.
• C2: The emotion sensor dictionary is aligned with the ETSI SmartM2M SAREF for eHealth Aging Well (SAREFEHAW) standard ontology; it addresses RQ2 in Section 4.2.
• C3: Retrieval of emotional knowledge by the reasoner, used in emotion scenarios; it addresses RQ4 in Section 4.
• C4: Provenance that keeps track of the knowledge designed by domain experts which is explicitly encoded in the ontology catalog and rule datasets to prove the veracity of the reasoning engine; it addresses RQ4 in Section 3.6.
• C5: Standardization-compliancy: Lessons learned from semantic interoperability are disseminated within the ISO/IEC 21823-3 IoT semantic interoperability [16], and the Alliance for the Internet of Things Innovation (AIOTI) Standardization WG (https://aioti.eu/aioti-wg03-reports-on-iot-standards/, accessed on 17 September 2022), which includes the Semantic Interoperability Expert Group [17,18] where the rule-based inference engine is taken as a baseline [19]. SAREF designers are also members of AIOTI Standard WG. AIOTI has other subgroups such as as AIOTI Health WG and AIOTI Urban Living WG. Furthermore, to implement the emotional knowledge graph, we employ semantic web technologies (RDF, RDFS, OWL, SPARQL), which are W3C standards.

Structure of the paper: Related work on emotion ontologies and IoT-based emotional projects are described in Section 2. Interconnecting interdisciplinary emotional knowledge is described in Section 3, which includes the set of competency questions and the ontology catalog for emotions. The emotional recommender system is described in Section 4, which includes the sensor dictionary for emotion, its reasoner and emotion scenarios. Evaluation is included in Section 5. The paper concludes and envisions future work in Section 6.

2. Related Work: Ontology and IoT-Based Emotion Aware Recommender Systems

Existing surveys on emotion ontology-based projects are investigated in Section 2.1. We selected a set of emotion ontology-based projects since the ontology code is shared online in Section 2.2. Well-being recommender systems are introduced in Section 2.3. Shortcomings of the literature study are summarized in Section 2.4. More than 20 ontology-based emotion projects are sumamrized in Table 1, including those not sharing online the ontology code. Additional information about ontology-based emotion projects can be found in Appendix A.

2.1. Existing Surveys on Ontology-Based Emotion Aware Projects

Twenty ontology-based affective state projects which include affective state influences (Abaalkhail et al. [31]) are reviewed. In our paper, we synthesize the work in Table 1, and we focus on the availability on ontologies, reasoning employed within the project, and sensors employed.

2.2. Selected Ontology-Based Emotion Aware Projects Due to Ontology Availability

We collected the ontology-based emotion projects that share online the ontology code and emphasize them in this section.

Emotion Ontology (EMO) (Hastings et al. [22]) describes types of emotion for the affective science domain. EMO is aligned with Basic Formal Ontology (BFO) and the Ontology of Mental Disease (OMD). Mental health and disease ontologies (MFOEM) (Hastings et al. [48]) provides a taxonomy of emotions. Affective science, psychology, and the psychiatric domains are interconnected with the Mental Functioning Ontology (MF) (Larsen, Hastings et al. [23]), which describes concepts such as consciousness, perception, thinking, and believing. Within their state of the art [23]), only few ontologies are mentioned, such as Gene Ontology (GO) and Hastings’s ontology [48]. For this reason, we deeply investigated ontology-based emotion projects (see Table 1).

Patient-facing software tool, based on Visualized Emotion Ontology (VEO) and Emotion and Mood Ontology (Lin et al. [24,25]) describes and references 25 emotions and their visualizations to enhance patient interaction in clinical environments. The Visualized Emotion Ontology is based on the revised Ortony, Clore, and Collins’ models of emotions (OCC model). OntoKeeper is used to evaluate ontology quality.

EEG neuroheadset-based online learning used the Emotion and Cognition ontology to understand emotions (Gil et al. [26]).

Bodily expression-based emotion detection uses the Emotion Ontology (EmOCA) for Context Awareness (Berthelon et al. [28,29]). The ontology references six emotions and focuses on describing philia/phobia and their impact on emotion. Reasoning is achived with the CORESE inference engine, a SPARQL request, and the regression algorithm. This ontology-based motion detection is experimented with six men and four women between 20 and 36 years old.

Emotion annotation of corpora using Web of Data (Linked data) is achieved within the Eurosentiment EU FP7 project, which develops the Onyx ontology (Sanchez-Rada et al. [30]). The Onyx ontology is aligned with the Provenance Ontology, Lexicon Model for Ontologies (LEMON), EmotionML, NLP Interchange Format (NIF), OpenAnnotation, and WordNet-Affect. Differences between Onyx and Human Emotion Ontology (HEO) from Grassi et al. [42] are analyzed. Automatic reasoning, based on SPIN rules, enables: (1) composition of emotions such as Anticipation and Joy result in Optimism, and (2) applying categories based on the dimensions of an emotion.

Emotion-aware applications is based on an ontology describing emotions (Lopez et al. [27]), their detection, and expression systems. The ontology reuses DOLCE (http://www.loa.istc.cnr.it/dolce/overview.html, accessed on 17 September 2022) and FrameNet (https://framenet.icsi.berkeley.edu/fndrupal/, accessed on 17 September 2022) to model descriptions and situations.

Conclusion: Finding and reusing the right emotion ontology for our needs is challenging. There is a need for more tools to help choose the ontology fitting our needs and encouraging ontology best practices with FAIR pinciples.

2.3. Well-Being Recommendation Systems

Well-being recommendation systems are summarized in Table “Well-being and IoT-based emotion applications (positive and negative) related work synthesis” within our past publication [49].

Healthy food smart Recommender System of Hybrid Learning (SRHL) [12] personalizes well-being and prevents diseases. The hybrid RS (content-based and collaborative filtering) employs unsupervised machine learning algorithms and considers time, activity, location, monetary costs, ingredients, health, nutritional value, availability, and the effects of combining the ingredients.

A personalized well-being and health-care support system, called MiningMinds [50], is a rule-based system used in 40 contextual scenarios varying in location, activity, weather, and emotion. MiningMinds is evaluated with forty users (thirty males, ten females in the middle-aged group: 25–49 years, ten different nationalities) and ten domain experts.

The I-Wellness personalized therapy Recommender System [51] is a hybrid Case-Based Reasoning (CBR), integrated within wellness websites, which helps users search for personalized therapy treatment based on their health condition. The I-Wellness online system arranges flexible appointments for patients with a wellness center. I-Wellness comprises six modules: (1) user wellness information, (2) wellness recommendation, (3) package selection, (4) appointment scheduling, (5) point allocation, and (6) wellness monitoring. The prototype is not accessible online to be tested.

A healthy and active lifestyle personalized context-aware RS Android smartphone application, called Motivate [52], recommends twenty types of activities according to location, agenda, weather, profile (e.g., can cycle), and time. The Motivate application is evaluated from 15 November to 25 December 2010 (5 weeks) with six Android phone participants (five male, one female, range: 24–63 years) that are colleagues or friends and worked five days a week. Five participants have a healthy weight body mass index (BMI), and one is slightly overweight.

Conclusion: Those recommender systems use contextual information but do not exploit the emotions of the users. Recommender systems for enhancing people’s emotion and mental health are still lacking.

2.4. Limitations of the Ontology and IoT-Based Emotion Aware Literature Study

We summarize the following shortcomings of the ontology and IoT-based emotion aware state of the art analysis:

• Recommender systems use contextual information but do not exploit the emotions of the users. Recommender systems for enhancing people’s emotion and mental health are still lacking.
• There are emotion-aware recommendation systems for music but not for mental health nor using IoT (Abdul et al. [53]).
• There is no emotion knowledge graph considering neurotransmitters, hormones and relationships to physiological data (e.g., heart rate).
• There is no standard emotion ontology.
• Reviewing the state of the art is a time-consuming task. There is a need to share the literature analysis innovatively (e.g., an emotion knowledge repository supported by tools) to ease the work of other researchers. The ontology-based emotion projects are classified in Table 1.
• Few ontologies can be semi-automatically analyzed, since numerous ontologies are not accessible online. There is a need to disseminate ontology best practices such as FAIR principles [14] for better ontology analysis to semi-automatically extract emotional-based domain knowledge.
• There is a lack of research tools shared as open-source (e.g., web service, web application) that can be easily reused to analyze the strengths and weaknesses of the applications illustrating the research use cases.
• NLP techniques are applied on texts for sentiment analysis (e.g., Twitter, comments on blogs, etc.) rather than emotion ontologies or scientific publications describing ontologies.
• Explicit descriptions of food that boost emotion are missing within ontology-based emotion-aware systems.
• There is no naturopathy recommender system to boost emotion and mental health.

3. Interconnecting Interdisciplinary Emotional Knowledge

Interconnecting interdisciplinary emotional knowledge is described in Section 3.1. Competency questions are introduced in Section 3.2. An ontology-based IoT project catalog for emotion is presented in Section 3.3. Knowledge extraction from emotion ontologies is described in Section 3.4. Mapping to existing knowledge bases such as SNOMEDCT, FMA, RXNORM, MedDRA, LOINC, etc. and Emotion Ontologies is explained in Section 3.5. Provenance of the data is explained in Section 3.6. FAIR principles are introduced in Section 3.7. Finally, lessons learnt from automatic extraction or mapping are summarized in Section 3.8.

3.1. Interconnecting Interdisciplinary Emotional Knowledge

We investigated emotional-related knowledge from various interdisciplinary domains, as shown in Figure 1 and listed below. Note that we cited numerous books (in French); this is due to the availability of resources to investigate them. Moreover, we mainly focus on chapters related to emotion within those books.

• Affective Computing (Picard et al. [54] with a focus on emotion recognition by robots and wearable computers).

  –

  IoT for Emotions (in our past publication [55], see table on Well-being and IoT-based emotion applications (positive and negative)).

  –

  Internet of Robotic Things for Emotions (see our past publication on the Agile Co-Creation of Robots for Ageing (ACCRA) H2020 European project [49]).

• Artificial Intelligence with a focus on Knowledge Engineering (e.g., emotion ontologies) (Ontology Catalog for Emotion in Section 3.3).
• Biology

  –

  Human Physiology (Silverthorn et al. [56]).

  –

  Psychobiology (Breedlove et al. [57]).

  –

  Neurosciences (Purves et al. [58]—chapter on emotions and chapter on neurotransmitters), brain and behavior (Kolb et al. [59]). We focused on neurosciences to understand better neurotransmitters relevant for emotions.

  –

  Neurobiology of emotions (Belzung et al. [60,61]), brain chemistry with hormones (Loretta Graziano Breuning et al. [62] focuses on happy brain with serotonin, dopamine, oxytocin, and endorphin hormones).

  –

  Epigenetics [63]: Impact of emotions on DNA (Zammatteo et al. [64]).

  –

  Endocrinology to describe hormones. Endocrinology chapters can be found in physiology books (Silverthorn et al. [56]).

• Psychology:

  –

  Brain, Chemistry and Psychology relationships (Virol et al. [65]).

  –

  Affective Sciences (Hastings et al. [22] design an emotion ontology, Lisa Feldman Barrett et al. [66,67] focus also on the physiology of emotions).

  –

  Understanding human emotions from facial expressions (Ekman et al. [68]).

  –

  Cognitive Psychology (Lieury et al. [69]).

  –

  Psychophysiology (Morange-Majoux et al. [70]).

  –

  Neuropsychology (Roger Gil et al. [71]—chapter on neuropsychology of emotions).

  –

  Positive Psychology (Seligman et al. [72,73], Lecomte et al. [74], Palazzolo et al. [75]. We also have an interest in positive psychology at work (Arnaud et al. [76], Chief Happiness Officer from Motte et al. [77]).

  –

  Emotion Regulation (Mikolajczak et al. [78]).

  –

  Emotional Intelligence (Goleman et al. [79], Couzon et al. [80]).

3.2. Competency Questions

We defined a set of Competency Questions (CQ) that will be answered within prototypes described in Section 4.1:

• CQ: What are the basic emotions according to Eckman et al. [81]?
• CQ: What are the neurotransmitters relevant for emotions?
• CQ: What are the hormones relevant for emotions?
• CQ: What are the hormones related to stress?
• CQ: What are the hormones related to happiness?
• CQ: What are the physiological parameters/sensors relevant to deduce (basic) emotions?
• CQ: How to deduce meaningful information such as (basic) emotions from physiological data produced by sensors?
• CQ: What are the ontologies describing emotions?

3.3. Ontology-Based IoT Project Catalog for Emotion: LOV4IoT-Emotion

We design the LOV4IoT-Emotion, which is an ontology-based IoT emotion project knowledge base (summarized in Table 1) to release our state of the art analysis as an innovative and maintained tool. We are aware of Systematic Literature Review (SLR) guidelines such as [82]. Other ontology catalogs such as BioPortal [20] and Linked Open Vocabularies (LOV) [21] are not focused on IoT-based emotions applications. LOV4IoT-Emotion is more focused on the ontologies and scientific publications, sensors, and reasoning mechanisms employed as detailed in Table 1 which are not covered by other ontology catalogs. We continuously enrich the LOV4IoT ontology catalog [83] and address more and more keywords such as affective science, well-being, etc. LOV4IoT-Emotion provides a dump of ontology code, web services, and web-based ontology catalog, which were released for the Knowledge Extraction for the Web of Things Challenge. The IoT-based emotion ontologies analysis is also employed within the reasoning discovery explained hereafter. To extract knowledge, manual and semi-automatic analysis are performed [84,85] (see Section 3.4).

3.4. Knowledge Extraction from Emotion Ontologies

We semi-automatically analyzed the ontologies with Natural Language Processing techniques (NLP): (1) ontology code (RDF/XML) and (2) scientific publications describing the ontologies. When the ontologies can be processed, we selected a set of specific keywords (e.g., joy) to compare knowledge provided by ontologies.

From a set of publications on emotion ontology-based projects, we extract knowledge such as which sensors are employed, which reasoning mechanisms are used to interpret sensor measurements, is there an ontology available and reusable, etc. (as depicted in Figure 2).

Ontologies are compared with each other to extract a common pattern to later generate a unified and federated emotional knowledge graph. As an example, we automatically retrieve key phrases (see Figure 3) if they mention emotion-related keywords such as: “fear”, “terror”, “anxiety”, “stress”, “stressed”, “despair”, “crying”, “anger”, “angry”, “irritation”, “joy”, “happy”, “happiness”, “pleasure”, “excited”, “calm”, “tired”, “bored”, “sad”, “disgust”, “love”, “surprise”, “hate”, “jealous”, “emotion”, “apraisal”, “guilt”, “sad”, “valence”, “face”. Since there is no corpus for such tasks, we have semi-manually built the gold dataset, as depicted in Figure 3. Each column corresponds to an ontology, and each row represents an emotion-related term.

3.5. Mapping to Existing Knowledge Bases Such as SNOMED-CT, FMA, RXNORM, MedDRA, LOINC, MESH, GALEN, ChEBI, DBpedia, and Emotion Ontologies

We first describe key concepts related to hormones and neurotransmitters. We search on the Bioportal ontology catalog for key ontologies to be mapped with our Emotion Knowledge Graph. We found ontologies such as SNOMED-CT, Mapping Foundational Model of Anatomy (FMA), RXNORM, MedDRA, Logical Observation Identifier Names and Code (LOINC), Medical Subject Headings (MESH), GALEN, and Chemical Entities of Biological Interest Ontology (ChEBI). The mappings of hormones and neurotransmitters are summarized in

Table 2. Subset of mapping hormones and neurotransmitters to existing knowledge bases to demonstrate the difficulty to reuse only one knowledge base. Legend: * means that we use rdfs:seeAlso since the concepts are similar but might not be an equivalent class.

Name	SNOMEDCT	FMA	RXNORM	MedDRA	LOINC
Hormone	SNOMEDCT ✓	FMA ✓	RXNORM No	MedDRA No	LOINC No
Neurotransmitter	SNOMEDCT ✓	FMA ✓	RXNORM No	MedDRA No	LOINC No
Serotonin	SNOMEDCT ✓	FMA ✓	RXNORM ✓	MedDRA ✓ *	LOINC ✓
Dopamine	SNOMEDCT ✓	FMA ✓	RXNORM ✓	MedDRA ✓ *	LOINC ✓
Oxytocin	SNOMEDCT ✓	FMA ✓	RXNORM ✓	MedDRA ✓	LOINC ✓
Glutamate	SNOMEDCT ✓	FMA No	RXNORM ✓	MedDRA ✓	LOINC ✓
Cortisol	SNOMEDCT ✓	FMA No	RXNORM ✓ *	MedDRA ✓	LOINC ✓
Endorphin	SNOMEDCT ✓	FMA No	RXNORM ✓ *	MedDRA No	LOINC ✓ *
Insulin	SNOMEDCT ✓	FMA ✓	RXNORM ✓ *	MedDRA ✓	LOINC ✓
Glucocorticoid	SNOMEDCT ✓	FMA No	RXNORM No	MedDRA ✓ *	LOINC No *
Adrenaline			RXNORM No	MedDRA No	LOINC No
(Epinephrine)	SNOMEDCT ✓ *	FMA ✓ *
Noradrenaline	SNOMEDCT No	FMA No	RXNORM ✓ *	MedDRA ✓ *	LOINC No
(Norepinephrine)	SNOMEDCT ✓	FMA ✓	RXNORM ✓	MedDRA ✓	LOINC ✓
Prolactin	SNOMEDCT ✓	FMA ✓	RXNORM ✓	MedDRA ✓	LOINC ✓
Testosterone	SNOMEDCT ✓	FMA No	RXNORM ✓	MedDRA ✓	LOINC ✓
Oestrogen	SNOMEDCT No	FMA No	RXNORM No	MedDRA ✓	LOINC No
Aldosterone	SNOMEDCT ✓	FMA No	RXNORM ✓	MedDRA ✓	LOINC ✓

and

Table 3. Subset of mapping hormones and neurotransmitters to existing knowledge bases to demonstrate the difficulty to reuse only one knowledge base. Legend: * means that we use rdfs:seeAlso since the concepts are similar but might not be an equivalent class.

Name	DBpedia	MESH	GALEN	ChEBI
Hormone	DBpedia ✓	MESH No	GALEN ✓	ChEBI ✓
Neurotransmitter	DBpedia ✓	MESH ✓ *	GALEN ✓	ChEBI ✓
Serotonin	DBpedia ✓	MESH ✓	GALEN ✓	ChEBI ✓
Dopamine	DBpedia ✓	MESH ✓	GALEN ✓	ChEBI ✓
Oxytocin	DBpedia ✓	MESH ✓	GALEN No	ChEBI ✓
Glutamate	DBpedia ✓ *	MESH ✓ *	GALEN No	ChEBI ✓ *
Cortisol	DBpedia ✓	MESH ✓	GALEN ✓	ChEBI ✓
Endorphin	DBpedia ✓	MESH ✓ *	GALEN ✓	ChEBI ✓ *
Insulin	DBpedia ✓	MESH ✓	GALEN ✓	ChEBI ✓
Glucocorticoid	DBpedia ✓	MESH ✓	GALEN ✓	ChEBI ✓
Adrenaline	DBpedia ✓	MESH ✓ *	GALEN ✓	ChEBI ✓
Noradrenaline	DBpedia ✓ *	MESH ✓ *	GALEN ✓	ChEBI ✓
Prolactin	DBpedia ✓	MESH ✓	GALEN ✓	ChEBI ✓
Testosterone	DBpedia ✓	MESH ✓	GALEN ✓	ChEBI ✓
Oestrogen	DBpedia ✓ *	MESH No	GALEN ✓	ChEBI ✓
Aldosterone	DBpedia ✓	MESH ✓	GALEN ✓	ChEBI ✓

. We also use DBpedia due to its popularity and link emotion-related concepts to existing emotion ontologies when available online. Most of the emotion ontologies cannot be found on BioPortal; only the Hastings’s ontology [22] is referenced on BioPortal.

3.5.1. Mapping to SNOMED-CT

Systematized Nomenclature of Medicine for Clinical Terms (SNOMED-CT) (https://www.snomed.org/, accessed on 17 September 2022), designed by the College of American Pathologists, is the biggest biomedical ontology with more than 370,000 concepts. SNOMED-CT describes clinical terminology for patient electronic health records such as operations, diseases, devices, symptoms, treatments, and drugs. SNOMED-CT requires the payment of an annual fee to be used by health care organizations. SNOMED-CT is now part of UMLS.

We mapped Hormone, Neurotransmitter, Dopamine, Serotonin, Oxytocin, etc. classes to the ones from SNOMED-CT using

owl:equivalentClass property, since the URL names do not have an explicit name (they are numbers such as &SNOMEDCT;734617007). Adrenaline is mapped to SNOMED-CT via rdfs:seeAlso since the concept is “Epinephrine”.

Noradrenaline is mapped to SNOMED-CT via rdfs:seeAlso since the concept is “noradrenaline hydrochloride” and to “Norepinephrine”.

3.5.2. Mapping Foundational Model of Anatomy (FMA)

Foundational Model of Anatomy (FMA) (http://si.washington.edu/projects/fma, accessed on 17 September 2022) is an ontology-based knowledge base for biomedical informatics which defines the structural organization of the human body.

We mapped Hormone, Neurotransmitter, Dopamine, Serotonin, Oxytocin to the ones from FMA using owl:equivalentClass property, since the URL names do not have an explicit name (they are numbers such as &fma;fma12278). As an example, Glutamate was within FMA. Adrenaline is mapped to FMA via rdfs:seeAlso since the concept is “Epinephrine”. Noradrenaline is mapped to FMA

via rdfs:seeAlso since the concept is “Norepinephrine”.

3.5.3. Mapping to RXNORM

RxNorm (https://www.nlm.nih.gov/research/umls/rxnorm/index.html, accessed on 17 September 2022), part of UMLS, is maintained the by United States National Library of Medicine. It is a medicine terminology that describes clinical drugs. It is used to study drug-to-drug interactions, since a medical doctor cannot have a visibility of side effects on more than three medicines given to a patient. It is also used in personal health records applications.

We mapped Dopamine, Serotonin, Oxytocin, Glutamate to the concepts from RXNORM using the owl:equivalentClass property, since the URL names do not have an explicit name (they are numbers such as &RXNORM;7824). When the same terms are not found, we map the terms using the rdfs:seeAlso property: Cortisol is “cortisol succinate”, Endorphin is “beta-endorphin human”, Insulin is “insulin aspart, human/insulin degludec”, Noradrenaline is “Norepinephrine”.

3.5.4. Mapping to MedDRA

Medical Dictionary for Regulatory Activities (MedDRA) (https://www.meddra.org/, accessed on 17 September 2022) is an internationally used standardized medical terminology that describes medical conditions, medicines and medical devices. MedDRA assists regulators for sharing medical products information used by humans.

We mapped concepts such as Dopamine, Oxytocin, and Glutamate to the concepts from MedDRA using the owl:equivalentClass property, since the URL names do not have an explicit name (they are numbers such as &MEDDRA;10033329). Other concepts are mapped using the rdfs:seeAlso property, since the terms are not exactly the same such as: Serotonin is “Serotonin syndrome”, Dopamine is “Dopamine urine”, Glucocorticoid is “Glucocorticoid decreased”, and Noradrenaline is “Serum noradrenaline increased”.

3.5.5. Mapping to Logical Observation Identifier Names and Codes (LOINC)

Logical Observation Identifier Names and Codes (LOINC) is a common language for identifying health measurements, observations, and clinical documents, but it is not used for diagnosis and diseases.

We mapped Dopamine, Serotonin, and Glutamate to the ones from LOINC using owl:equivalentClass property, since the URL names do not have an explicit name (they are numbers such as &LOINC;MTHU013002). Endorphin is mapped to LOINC via rdfs:seeAlso since the concept is “Beta endorphin”. Glucocorticoid is mapped to LOINC via rdfs:seeAlso since the concept is “Synthetic glucocorticoid drug”. Noradrenaline is mapped to LOINC via rdfs:seeAlso since the concept is “Norepinephrine”.

3.5.6. Mapping to Medical Subject Headings (MESH)

Medical Subject Headings (MeSH) (https://www.nlm.nih.gov/mesh/meshhome.html accessed on 17 September 2022), part of the US NIH National Library of Medicine, is a thesaurus and vocabulary used for indexing, cataloging, and searching of biomedical and health-related information from MEDLINE/PubMed, the NLM Catalog, and other NLM databases.

We mapped concepts such as Dopamine, Serotonin, Oxytocin, Cortisol,

Prolactin, Insulin, Aldosterone, and Testosterone, etc. to the concepts from Medical Subject Headings (MESH) using the owl:equivalentClass property. The MESH ontology does not provide an explicit name within URLs (e.g., &mesh;D000450). Oestrogen is found within MESH as “estrogens”. Other concepts are mapped to MESH using the rdfs:seeAlso property, since we do not find an exact match of the term: Noradrenaline is “noradrenaline sulfate”, Adrenaline is adrenaline sulfate, Endorphin is “endorphin, humoral”, Glutamate is “Glutamic Acid”, Neurotransmitter is “Neurotransmitter Agents”. We did not find the concept Hormone with Mesh.

3.5.7. Mapping to GALEN

Generalised Architecture for Languages, Encyclopaedias, and Nomenclatures in medicine (GALEN) (https://opengalen.org/, accessed on 17 September 2022) was a European Union project (1992–1999) which provides an ontology that describes medical concepts for clinical systems. It became the OpenGALEN foundation, and 2020 marks OpenGALEN’s 30th anniversary. GALEN provides more than 25,000 concepts. GALEN can be used to describe medical procedures.

We mapped Hormone, Neurotransmitter, Serotonin Dopamine, cortisol, etc. to the ceoncepts from GALEN using owl:equivalentClass property. The GALEN ontology provides an explicit name (e.g., galen:Hormone). Oxytocin and glutamate were not found.

3.5.8. Mapping to Chemical Entities of Biological Interest Ontology (ChEBI)

Chemical Entities of Biological Interest (ChEBI) (https://www.ebi.ac.uk/ChEBI/, accessed on 17 September 2022) is a chemical ontology from the Open Biomedical Ontologies effort at the European Bioinformatics Institute.

We mapped concepts such as Neurotransmitter, Dopamine, Serotonin, Oxytocin, Cortisol, Adrenaline, Insulin, Prolactin, Noradrenaline, Aldosterone,

Testosterone, etc. to the concepts from ChEBI using the owl:equivalentClass property. The ChEBI ontology does not provide an explicit name within URLs (e.g., ChEBI:ChEBI_50114). Other concepts do not have the exact same term such as: Oestrogen is estrogen, Endorphin is “beta-endorphin”, and Glutamate is “glutamate(2)”.

3.5.9. Mapping to DBpedia

DBpedia is the semantic Wikipedia. It covers various domains but is less specific to the biomedical domain. DBpedia is not well known in the biomedical domain (not referenced within the Bioportal ontology catalog). However, it is intensively used as it is proven by the Linked Open Data cloud (https://lod-cloud.net/, accessed on 17 September 2022) or the Linked Open Vocabularies ontology catalog (https://lov.linkeddata.es/dataset/lov/vocabs/dbpedia-owl, accessed on 17 September 2022), which shows that DBpedia ontology has been used in at least 18 ontologies, uses 19 external ontologies, and it is used in at least 15 datasets.

Due to its popularity and numerous tools using DBpedia, we decided to map to DBpedia. We mainly used rdfs:seeAlso to map to terms such as Emotion Neurotransmitter. For instance, <rdfs:seeAlso rdf:resource="https://dbpedia.org/page/Emotion"/>, accessed on 17 September 2022). DBpedia uses the term Glutamic_acid for glutamate. Noradrenaline is mapped to Norepinephrine from DBpedia. Oestrogen is mapped to Estrogen from DBpedia.

3.5.10. Mapping to Emotion Ontologies

Our emotional knowledge graph adds explicit links to existing emotion ontologies:

• Emotion Ontology (EmOCA) for Context Awareness (Berthelon et al. [28] (http://ns.inria.fr/emoca/emoca.rdfs, accessed on 17 September 2022)) since it references six basic emotions: joy, fear, disgust, anger, sadness, surprise, and other concepts such as valence and arousal.
• Emotion Ontology (EMO) and mental health and disease ontologies (MFOEM) (Hasting et al. [22]) (https://raw.githubusercontent.com/jannahastings/emotion-ontology/master/ontology/MFOEM.owl, accessed on 17 September 2022).
• Visualized Emotion Ontology (VEO) (Lin et al. [24,25]) (https://bioportal.bioontology.org/ontologies/VEO accessed on 17 September 2022) for concepts such as Emotion, Fear, Surprise, Anger, Disgust, Pride, Interest, Pleased, Joy, Hope, Admiration, Disappointment, Distress, Hate, Shame, etc.
• Onyx Ontology (Lopez, Sanchez-Rada et al. [27,30]) (http://www.gsi.dit.upm.es/ontologies/onyx/1.5/ns accessed on 17 September 2022) for concepts such as Emotion, Appraisal, etc.
• Emotion and Cognition ontology (Gil et al. [26] (http://rhizomik.net/ontologies/2010/11/emotionsonto.owl accessed on 17 September 2022) for concepts such as Emotion, etc. We did not find a taxonomy of emotion in this ontology.

We explicitly added links such as <owl:equivalentClass rdf:resource = “obo;MFOEM_ 000053”/> or <rdfs:seeAlso rdf:resource = “obo;MFOEM_000196”/>

When the ontology code is not available, we enrich our emotional knowledge graph with more knowledge while reading the scientific papers. For instance, the taxonomy of mood states (see Figure A9) is based on Zenonos et al. [86] that references eight mood states that can be recognized: Excited, Happy, Calm, Tired, Bored, Sad, Stressed, Angry. We explicitly write the source of the knowledge within our knowledge graph to keep track of the provenance of data.

3.6. Keeping Track of Provenance Metadata

To keep track of the source of knowledge, we explicitly add this information within the rule RDF dataset and LOV4IoT RDF dataset as shown in Listing 1 by referring to the scientific publication in rdfs:comment and the link to the scientific paper with prov:hadPrimarySource from the W3C PROV-O ontology. For instance, prov:hadPrimarySource keeps track of the source by providing the URL of the scientific publication mentioning the rule or reasoning mechanism.

Listing 1. Adding provenance metadata using the W3C PROV Ontology.

Listing 1. Adding provenance metadata using the W3C PROV Ontology.

To describe the rule dataset itself, we use the Data Catalog Vocabulary (DCAT) as shown in Listing 2.

Listing 2. Describing the rule dataset using the Data Catalog Vocabulary (DCAT).

Listing 2. Describing the rule dataset using the Data Catalog Vocabulary (DCAT).

Table 4. Ontology Namespace. URLs accessed on 17 September 2022.

Namespace Prefix	Description Name	Namespace URL
dcat	Data Catalog Vocabulary	http://www.w3.org/ns/dcat#
prov	Provenance Ontology	http://www.w3.org/ns/prov#
dc	Dublin Core (DC)	http://purl.org/dc/elements/1.1/
dcterms	Dublin Core Metadata Terms	http://purl.org/dc/terms/
vann	Vocabulary for Annotating Vocabulary Descriptions	http://purl.org/vocab/vann/
vs	Vocab Status ontology	http://www.w3.org/2003/06/sw-vocab-status/ns#
rdf	Resource Description Framework	http://www.w3.org/1999/02/22-rdf-syntax-ns#
rdfs	Resource Description Framework Schema (RDFS)	http://www.w3.org/2000/01/rdf-schema#
owl	Ontology Web Language	http://www.w3.org/2002/07/owl#
m3	Machine-to-Machine Measurement Sensor Dictionary	http://sensormeasurement.appspot.com/m3#
saref-core	Smart Applications REFerence ontology	https://saref.etsi.org/core/

reminds of the namespaces used: the first column for the prefix name, the second column for the namespace description, and the third column for the ontology namespace URL.

3.7. FAIR Principles

Findable, Accesssible, Interoperable, Resuable (FAIR) principles [14] encourage researchers to share their reproducible experiments by publishing online the resources such as ontologies, datasets, rules, etc. The set of ontology code available online can be automatically processed; if the ontology code is not available, the scientific publications describing the emotion ontologies are semi-automatically processed with Natural Language Processing (NLP) techniques to feed the emotion reasoning engine to build the recommender system. As we mentioned earlier, the LOV4IoT-Emotion ontology catalog (Section 3.3) encourages FAIR principles.

3.8. Lessons Learnt from Automatic Extraction or Mapping

We encountered a set of errors while conducting automatic extraction or mapping:

• Dead ontology URL. The ontology URL was mentioned in the scientific paper but is not available anymore. For this reason, it is important to encourage better FAIR principles (e.g., findable resources).
• The ontology cannot be loaded. Errors such as Unable to complete the HTTP request are encountered. There are also issues when dealing with ontology code generated with various ontology editors, libraries, etc. in various formats such as RDF/XML, RDF/Turtle.
• The ontology can be loaded but no terms for automatic extraction can be found. It can happen when there are no label or comments within the ontology or when the ontology URL was automatically built (e.g., MEDDRA;10033329).
• To map the ontologies, we have to deal with synonyms as well.
• Picking the right ontology fitting our need is challenging; numerous ontologies can cover the same terms, but all terms that we need are not covered in only one ontology.

4. Emotional Recommender System: Knowledge Discovery and Reasoning for Emotion

The implementation of competency questions is described in Section 4.1. The sensor dictionary for emotion is defined in Section 4.2. The emotion reasoner is introduced in Section 4.3. Additional emotional use cases are summarized in Section 4.4.

4.1. Implementation: Emotional Knowledge Graph and Prototypes Answering Competency Questions

The set of implemented prototypes answers the competency questions introduced in Section 3.2:

• CQ: What are the basic emotions according to Ekman et al. [81] (see Figure A1). Ekman et al. defines six basic emotions: happiness/enjoyment, sadness, anger, fear, surprise, and disgust.
• CQ: What are the hormones relevant for emotions? (Figure A4). The source of knowledge is extracted for instance from the Brain, Chemistry and Psychology relationships’s book (Virol et al. [65]), Kolb et al. [59], among others).
• CQ: What are the hormones related to happiness? (Figure A6). For instance, serotonin dopamine, oxytocin, and endorphin are described within Breuning et al. [62], among others.
• CQ: What are the hormones related to stress? (Figure A5). The answer is based on Kolb et al. [59], among others.
• CQ: What are the neurotransmitters relevant for emotions? (Figure A3). The answer is based on Kolb et al. [59], Brain, Chemistry and Psychology relationships’s book (Virol et al. [65]), among others).
• CQ: What are the physiological parameters/sensors relevant to deduce (basic) emotions? (Figure A8 and see Section 4.2)
• CQ: What are the reasoning mechanisms to analyze data from physiological parameters/sensors to deduce (basic) emotions? (Figure A7 and Section 4.3)
• CQ: What are the ontologies describing emotions? (Figure A2 and Table 1).

4.2. Semantic Sensor Emotion Dictionary

We extended our sensor dictionary and applied it to the emotion domain. The sensor dictionary is compliant with the ETSI SmartM2M SAREF standard. The sensor dictionary classifies sensors, measurements, and units for the emotion domain. We also take into consideration synonyms. For each sensor, we keep the provenance of the source of knowledge using it (e.g., past projects referenced within the emotion IoT-based ontology catalog (see Section 3.3 and Table 1, and reasoning employed to infer additional knowledge from emotion sensor data (see the SLOR-Emotion reasoning discovery project in Section 4.3).

The sensor emotion dictionary is provided as a web service: (http://sensormeasurement.appspot.com/emotion/getAllEmotionSensors/?nameClass=EmotionM2MDevice&format=xml, accessed on 17 September 2022) (see Figure 4) or can be hidden within the Graphical User Interface (GUI) (see Section 4.3 and Figure A8).

Semantic Annotation: We design an SAREF-compliant sensor dictionary to map common terms. Emotion datasets describe sensor measurement; its value, its unit, and its timestamp are represented in JSON or XML and follow the SenML format (https://tools.ietf.org/html/rfc8428 accessed on 18 September 2022). Semantic annotation requires a rule repository to explicitly describe the context of the measurement. For instance, “t” or “temp” or “temperature” can represent a room temperature or a body temperature and will be annotated following the dictionary which is implemented as an ontology.

4.3. Knowledge Discovery and Reasoning for Emotion with Sensor-Based Linked Open Reasoning (S-LOR Emotion)

The sensor emotion dictionary described in Section 4.2 is used within the emotion rule discovery. We classified emotion sensors within a dictionary (Figure A7), where each sensor is automatically retrieved using SPARQL queries.

The reasoning engine uses data produced by sensors (e.g., heartbeat). The Semantic Annotator API component explicitly annotates the data (e.g., unit of the measurement, context such as body temperature or room temperature) and unifies the data when needed. The semantic annotation uses ontologies that can be found through our LOV4IoT-Emntion ontology catalogs (Section 3.5.10). The ontology chosen must be compliant with a set of rules to infer additional information. The Reasoning Engine API (inspired from [55,87] and supported by the AIOTI group on semantic interoperability [19]) deduces additional knowledge from the data (e.g., abnormal heartbeat) using rule-based reasoning. The IF THEN ELSE rules executed by the reasoning engine will add new data in the data storage. The enriched data can be later used within end-user applications (e.g., call the family or doctor when the heartbeat is abnormal since it might be an emergency, or deducing the fear emotion).

To implement the inference engine, we used the Jena framework; a rule code example is shown in Listing 3. The rule is compliant with our Machine-to-Machine-Measurement (M3) ontology which classifies sensor type, measurement type, units, etc. to ensure an interoperable knowledge-based reasoning. Our web services (

Table 5. Specification draft of the S-LOR Knowlege Reasoning API.

Web Service Description	Web Service URL and Example
Getting all rules for a specific sensor	http://linkedopenreasoning.appspot.com/slor/rule/{sensorType}
	Example:
	http://linkedopenreasoning.appspot.com/slor/rule/BodyThermometer (Last accessed on 18 September 2022)
	sensorType should be compliant with the classes referenced with M3 ontology
Update the rule dataset	We can add new sensors in the dataset and add new rules
	by adding a new instantiation within the dataset
	(see code example above and contribution to the
	semantic interoperability for IoT white paper 2019 [17])
	The GUI to add new sensors:
	http://linkedopenreasoning.appspot.com/?p=ruleRegistry (Last accessed on 18 September 2022)

) are employed to later code rules as depicted in Listing 3 for the scenarios according to the applications’ needs where the scenarios provide GUIs.

Listing 3. A rule example.

Listing 3. A rule example.

Various rules can be provided by the Sensor-based Linked Open Rules (S-LOR) tool (Figure A7) which references rules per domain and per sensor. A drop-down list is shown with a set of IoT sub-domains, such as emotion, that we are looking for. Once the domain is selected, the list of sensors relevant for this domain (e.g., skin conductance) is depicted. The developer clicks on the button “Get Project” to retrieve existing projects already using such sensors or the “Get rule” button to retrieve existing rules relevant for this sensor to deduce meaningful information from sensor data.

Demos provide tooltips to provide the veracity of the facts. The facts come, most of the time, from scientific publications referenced within the LOV4IoT catalog and the Sensor-based Linked Open Rules (S-LOR) tool. Such technical AI solutions can be later embedded within the robots to better understand users’ emotions from sensor devices.

A future work is to consider visual [88] and audio data [89] from more complex sensors such as a microphone and camera to detect emotions.

4.4. Emotional Use Cases

We reference in this section how the emotional knowledge graph and related tools are employed: (1) ACCRA H2020 European Project in Section 4.4.1, (2) AIOTI for Health and Urban Living in Section 4.4.2, (3) AI4EU H2020 European Project–Knowledge Extraction for the Web of Things (KE4WoT) Challenge in Section 4.4.3, and (4) other emotion use cases in Section 4.4.4 such as the naturopathy recommender system to boost mental health or the use cases answering the competency question from Section 3.2.

4.4.1. Emotional Robots to Reduce Social Isolation for Ageing People: ACCRA H2020 European Project in Collaboration with Japan

How can Internet of Robotic Things (IoRT) technology and co-creation methodologies help to design emotional-based robotic applications? To respond to this research question, the ACCRA (Agile Co-Creation of Robots for Ageing) project [49] designs advanced social companion robot-based applications to support active and healthy aging, so elderly people can stay independent at home and decrease their social isolation. Applications are designed following co-creation methodologies involving multidisciplinary stakeholders and researchers (e.g., elderly people, medical professionals, and computer scientists such as roboticists or IoT engineers). The ACCRA project provides three robots, Buddy, ASTRO, and RoboHon, used for daily life, mobility, and conversation, to understand and convey emotions in real time using IoT and AI technologies (e.g., knowledge-based reasoning).

4.4.2. AIOTI for Health and Urban Living

AIOTI Health WG and Urban Living WG (https://aiotieu.github.io/urbansociety/catalogue.html accessed on 18 September 2022) are collaborating to release a white paper in September 2022 for the AIOTI Signature Event 2022. AIOTI has been developed in collaboration with the Urban Society, Buildings, Innovation Ecosystems, Energy and Health communities a report that is covering topics of improving healthy urban lifestyle, value-based health care and mitigating the risk and social impact of isolation and loneliness in urban societies.

This report focuses on how IoT can be deployed to foster mental and physical health, thereby putting the weight on disease prevention instead of treatment. Furthermore, cultural and behavioral factors, including the degree of trust in IoT and other technical applications, digital data, governments and the industry itself, are explicitly included and covered by the report. We will present the results of the report and have an open discussion between contributors and experts on how IoT and technology can support to identify and mitigate diseases and increase prevention, thus supporting broader societal goals of benefiting people’s health and well-being.

4.4.3. AI4EU H2020 European Project-Knowledge Extraction for the Web of Things (KE4WoT) Challenge

To encourage research collaborations to automatically extract the knowledge from ontologies and scientific publication describing the ontology, we designed the Knowledge Extraction for the Web of Things (KE4WoT) Challenge supported by the AI4EU project and by the Web Conference Conference (WWW 2018) [90]. The KE4WoT challenge encourages processing the expertise published by domain experts and making the domain knowledge usable, interoperable, and integrated by machines. We released the set of ontologies in various domains (including emotion) as dumps, web services, and tutorials and made them available for the challenge.

Furthermore, the Linked Open Reasoning tool, introduced in Section 4.3, is referenced on the AI4EU platform (https://www.ai4europe.eu/research/ai-catalog/linked-open-reasoning accessed on 18 September 2022).

4.4.4. Other Emotional Use Cases

• Taxonomy of mood states (see Figure A9) (e.g., based on Zenonos et al. [86] references eight mood states: Excited, Happy, Calm, Tired, Bored, Sad, Stressed, Angry).
• IAMHAPPY: an IoT knowledge-based well-being recommendation system to encourage happiness [55].
• Naturopathy Recommender System extended for Emotion: We extended our naturopathy knowledge graph to cover better emotion-related knowledge and to enhance mental health, for instance:

  –

  Food that contains magnesium is recommended for depression (e.g., chocolate) (Figure A10).

  –

  Food that contains Vitamin D is recommended for depression (Figure A11).

• Heartbeat data to deduce fear emotion (Figure A12).
• Skin conductance data to deduce anxiety emotion (Figure A13).
• Luminosity data to deduce weather (cloud cover) (Figure A14).

5. Evaluation: Applying Semantic Web Best Practices to Enhance the Emotional Knowledge Graph

We employed our designed PerfecO methodology [91] that references a set of semantic web best practices and collection of tools providing web services to evaluate ontologies to enhance our emotional knowledge graph: (1) Syntactic Ontology Evaluation in Section 5.1, (2) ontology design evaluation in Section 5.2, (3) LOV4IoT-Emotion Ontology Catalog statistics in Section 5.3, and (4) the emotional knowledge graph evaluated with emotion scenarios in Section 5.4.

5.1. Syntactic Ontology Evaluation

The ontology can be loaded with Jena version 2.11, Protege 4.3 (Figure 5 and Figure 6), and GraphDB 9.10 (Figure 7). We evaluated our emotion ontology with the TripleChecker syntax validator in May 2022. Mistakes found such as “Literal value does not match datatype” (data format and decimal format) have been fixed.

We also used the W3C RDF Validator (https://www.w3.org/RDF/Validator/rdfval accessed on 18 September 2022) and OWL Manchester Validator (http://mowl-power.cs.man.ac.uk:8080/validator/validate accessed on 18 September 2022), and no errors have been found.

5.2. Ontology Design Evaluation

We evaluated our emotion ontology with OOPS! OntOlogy Pitfall Scanner! [92], which detects pitfalls. We have identified eleven pitfalls, which are classified as critical, important or minor:

• One critical pitfall: P31 Defining wrong equivalent classes.
• Four important pitfalls: P10 Missing disjointness, P34 Untyped class, P38 No owl ontology declaration, and P30 Equivalent classes not explicitly declared.
• Six minor pitfalls: P04 Creating unconnected ontology elements, P08 Missing annotation, P20 Missing ontology annotations, P22 Using different naming conventions in the ontology, P32 Several classes with the same label, P36 URI contains file extension.

We are currently fixing the pitfalls. To fix P30 Equivalent classes not explicitly declared, we use rdfs:seeAlso instead of owl:equivalentClass.

Regarding P30 Equivalent classes not explicitly declared, we disagree, Hastings et al. distinguish Joy (MFOEM_000034) from Pleasure (MFOEM_000035). For this reason, we kept both as separate concepts.

For the P34 Untyped class pitfall, we need to import the relevant ontologies: Hastings et al. FMA, VOAF, emoca, SNOMEDCT, MedDRA, RXNORM, VEO, etc.

We also added ontology metadata (Figure 8 and Listing 4) to our emotion ontology as recommended by the Linked Open Vocabularies ontology catalog [21].

Listing 4. Ontology Metadata Code Example of the Emotion Ontology.

Listing 4. Ontology Metadata Code Example of the Emotion Ontology.

To enhance clarity, we provide rdfs:label and rdfs:comment to explicitly describe concepts in natural language and even provide the source of the knowledge using the PROV-O ontology (as explained in Section 3.6).

To ensure deferencable URI, we tested the ontology with Vapour [93] to test the availability of the ontology on servers (e.g., deferencable URI). Deferencing Entity URI (without content negociation) passed (Figure 9). However, we can enhance availability by providing other sources such as HTML, JSON-LD, RDF/XML, and TURTLE. Moreover, http://sensormeasurement.appspot.com/ont/m3/emotion.owl (accessed on 17 September 2022) can be downloaded, but the server configuration must be improved. We are using Google Application Engine to host the application online, with web services running, and obtain the DNS name.

5.3. LOV4IoT-Emotion Ontology Catalog: Page View Statistics

The LOV4IoT-Emotion web page has been visited 5511 times (3987 as unique views) between January 2018, and March 2022 according to Google Analytics (Figure 10). Visitors return to this dataset which demonstrates its usefulness. To make the GUI more appealing, there is still a need for front-end developers. The results are encouraging to update the dataset with additional domains and ontologies.

AIOTI IoT Ontology Landscape: LOV4IoT is considered with the AIOTI (The Alliance for the Internet of Things Innovation) IoT ontology landscape survey form (https://ec.europa.eu/eusurvey/runner/OntologyLandscapeTemplate accessed on 17 September 2022) that references and classifies ontologies (https://aioti.eu/wp-content/uploads/2022/02/AIOTI-Ontology-Landscape-Report-R1-Published-1.0.1.pdf accessed on 17 September 2022), which is led by the Standard WG–Semantic Interoperability Expert Group. The group helps industrial practitioners and non-experts find which ontologies are relevant in a certain domain, where to find them, how to choose the most suitable, and ontology maintenance-related questions.

StandICT.eu 2023 Standardized Ontology Landscape: Another similar action to collect standardized ontologies is under development within the StandICT.eu 2023 project. A landscape on standardized ontologies should be released before the end of 2022.

5.4. Emotional Knowledge Graph Evaluated with Emotion Scenarios

A subset of emotional-related scenarios has been introduced in Section 4.4. More and more scenarios are integrated.

Rules are implemented and employed within the emotion scenarios (see

Table 6. Set of online demonstrators: ontology catalog, rule discovery, and full scenarios. accessed on 15 September 2022.

Tool Name	Tool URL
LOV4IoT-Emotion Ontology	http://lov4iot.appspot.com/?p=lov4iot-emotion
-based IoT Project Catalog
LOV4IoT-Emotion Ontology Web Service	http://lov4iot.appspot.com/?p=queryEmotionOntologiesWS
and Dumps (for Developers)
S-LOR Emotion Rule Discovery	http://linkedopenreasoning.appspot.com/?p=slor-emotion
M3-Emotion Full Scenarios	http://sensormeasurement.appspot.com/?p=emotion
M3 Emotion Ontology	http://sensormeasurement.appspot.com/ont/m3/emotion#

for demonstrator’s URLs such as sensor discovery, rule discovery, emotion ontology, etc.).

Technical Evaluation We summarize 16 rules that encourage ontology best practices in

Table 7. Ontology best practices: check list summary [91]. Legend: * means the level of complexity to achieve the rule. ** is secondary complexity. *** is the most complex one.

Rule Number	Description	Difficulty
Rule 1	Finding a good ontology name	*
Rule 2	Finding a good ontology namespace	**
Rule 3	Sharing your ontology online	**
Rule 4	Adding ontology metadata	**
Rule 5	Adding rdfs:label, rdfs:comment, dc:description for each concept and property	*
Rule 6	All classes start with an uppercase and properties start with a lowercase	*
Rule 7	Submitting your ontology to ontology catalogs	**
Rule 8	Reusing and linking ontologies	***
Rule 9	Deferenceable URI copy paste the namespace URL of your ontology in a web browser to obtain the code	**
Rule 10	Checking syntax validator	*
Rule 11	Adding ontology documentation	*
Rule 12	Adding ontology visualization	*
Rule 13	Improving ontology design	***
Rule 14	Improving dereferencing URI and content negotiation	***
Rule 15	Ontology can be loaded with ontology editors (e.g., Protege)	**
Rule 16	Registering your ontology on prefix catalogs	*

, so researchers will enhance the reusability and better interoperability of their emotion knowledge [91]. Each rule is associated with bad practices and best practice examples to ease the task of beginners in their learning journey (Step-by-step tutorial to improve the ontology quality, dissemination, reuse, etc. Semantic Web Best Practices: https://goo.gl/Rg4cGr, accessed on 18 September 2022) [94]. We also disseminated those best practices to ISO SC 42 Artificial Intelligence within ISO/IEC 5392 Knowledge Engineering Reference Architecture (KERA) and the Ontologies, Knowledge Engineering, and Representation (OKER) Report.

6. Conclusions and Future Work

Designing cross-domain emotion applications requires acquiring knowledge from different communities (e.g., affective sciences, affective computing, biology, neurosciences, psychology, physiology, psychophysiology, neuropsychology, etc.). We extended our sensor dictionary for the emotion domain. For each sensor, we discover related knowledge, extract rules, make rules compliant with our sensor dictionary, and integrate rules in a emotion-based scenario. Domain knowledge is also extracted from our LOV4IoT-Emotion IoT-based ontology catalog, referencing more than 49 projects (in March 2022). LOV4IoT reuses the domain expertise from past ontology-based emotion IoT projects. Integrating machine interpretable knowledge implemented within ontologies helps when domain experts are not available. We contributed to standards (e.g., we are editors of the ISO/IEC 21823-3 IoT semantic interoperability), Alliances for IoT such as AIOTI (Standard WG-IoT Semantic Interoperability sub-group, Health WG, and Urban WG), and (European) projects such as AI4EU, ACCRA and StandICT.eu 2023.

Mid-term challenges: There is not yet any emotion ontology that is standardized. The emotion knowledge graph could be disseminated more within standards such as W3C, ISO, NIST, IEEE, etc.

Long-term challenges: Deducing meaningful knowledge from IoT physiological data produced by sensors using more sophisticated AI technologies. Our SAREF-compliant semantic reasoning uses AI techniques such as knowledge and rule-based reasoning, but it could be enhanced addressing more sophisticated scenarios (e.g., dealing with ECG and EEG, which requires Machine Learning and Deep Learning). Encouraging more synergies among standards and communities (e.g., ETSI SmartM2M SAREF, W3C SOSA/SSN, W3C Web of Things, and iot.schema.org ) is partially addressed via the AIOTI Alliance and the StandICT project, and there is a need for more open-source tools, supports, etc.