**1. Introduction**

Food perception typically starts before the food is placed in the mouth. Before the food is consumed, consumers typically smell the food and see the food, often with the food package with brand name. Consequently, prior to tasting, consumers will already have expectations regarding the food's taste and flavor based on visual, smell, tactile, and sometimes even auditory cues. These expectations affect whether the consumer will eat the food. Numerous studies have focused especially on visual cues. They have demonstrated the effects of branding, names and sensory descriptors, and health and ingredient labels on food choice decisions. These "extrinsic" variables provide consumers with information regarding the product's quality, which is part of "brand equity" [1,2]. Brand equity combines perceived quality with factors such as brand loyalty, name awareness, and other associations that the consumer may have with the brand [3]. Obviously, the factors vary with the consumer's experience with the product and aid consumers in selecting prod-

**Citation:** de Wijk, R.A.; Ushiama, S.; Ummels, M.; Zimmerman, P.; Kaneko, D.; Vingerhoeds, M.H. Reading Food Experiences from the Face: Effects of Familiarity and Branding of Soy Sauce on Facial Expressions and Video-Based RPPG Heart Rate. *Foods* **2021**, *10*, 1345. https://doi.org/ 10.3390/foods10061345

Academic Editor: Damir Torrico

Received: 18 May 2021 Accepted: 9 June 2021 Published: 10 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

ucts that fit their attitudes [4,5]. Extrinsic variables may also exert direct effects on taste experiences [6,7].

Models such as the assimilation and contrast model [6,8] describe various ways of how expectations interact with actual experiences, depending on their overlap and discrepancies. In the assimilation model, the discrepancy is relatively small, and experiences are adjusted to the expectations. In contrast, when the discrepancy is relatively large, the differences between expectations and experiences are amplified. As a result, ratings for the experience will shift in the opposite direction of the expectations. Recently, examples of assimilation and contrast were provided by Głuchowski et al. [9], who investigated emotional and hedonic reactions to traditional and new innovative food dishes. The results showed that for traditional dishes, expected liking based on visual aspects was in line with experienced liking during tasting, whereas for new innovative dishes the experienced liking fell short compared to the expected liking. The results also demonstrate the importance of taking consumers' characteristics such as food neophobia and consumer innovativeness into account.

Most studies used subjective ratings and questionnaires, so-called explicit measures, to assess the effects of expectations on taste experiences, i.e., these studies rely on introspection while consumers may not even be aware of the way expectations affect their experiences (e.g., [10]). Additionally, virtually all studies measured the effect of expectation indirectly by measuring its effect on subsequent taste experiences, i.e., only very few studies try to measure the expectations themselves, except for Thomson, who uses a specially developed technique [11].

On the other hand, implicit measures of autonomic nervous system (ANS) responses, such as heart rate, and their behavioral correlates, such as facial expressions, offer a promising alternative. In fact, these measurements (1) do not rely on introspection and offer insight into the subconscious as well as conscious processes and (2) can be measured continuously during expectations and experiences (e.g., [12]). The continuous measurement of ANS responses and facial expressions facilitate insights in the different stages of the temporal development of responses and have been corner stones of theories of emotions, such as appraisal theories. Early responses reflect aspects of stimuli, such as the stimulus' novelty, relevance, or pleasantness. Later responses include cognitive reactions on the earlier responses.

Despite their potential, facial expressions and ANS responses have been used only in relative few food studies and with varying success: some studies failed to find effects on ANS responses or facial expressions between the tastes of foods, such as chocolates [13], beers [14], and energy drinks [15], whereas others found significant effects, even though they were small and difficult to interpret (e.g., [16] for juices, [17] for beers, and [18] for breakfast drinks). In studies where the foods were not only tasted but also visually inspected, the results typically showed clearer differences between foods. In a recent study, Bercik et al. [19] measured facial expressions as well as eye movements for three dishes with the same ingredients but served in three different ways. Results showed that the way the food was visually presented had significant emotional effects and influenced the way participants visually inspected the foods. Similarly, facial expressions were intensified when foods were tested in a natural consumption environment, such as one's own home, rather than in the unrealistic and uninspiring setting of a sensory laboratory [20], which suggests that this type of measure reflects the complete food experience determined by a combination of variables such as the food, the physical location, and the social environment, rather than by specific taste experiences.

Unfortunately, ANS measurements and some facial expression measurements typically cannot be used in realistic non-laboratory settings because they require sensors attached to face and body [9,13,14,16,18,21–24]. Recent technical developments have resulted in video-based automated analysis of facial expressions and even more recently in video-based recordings of heart rate using remote photoplethysmography or RPPG (e.g., [20,25]). If these new methods prove to be a good alternative for the gold-standard

methods such as electrocardiogram (ECG) and photoplethysmography (PPG), then these measurements would no longer require physical contact between the sensor and human volunteer. This would open a whole new range of applications such as testing in realistic consumption environments, and internet testing using a webcam, which is built-in in most notebooks nowadays.

This study explored the use of video-based facial expressions, RPPG heart rate, and Emoji scores of valence and arousal [26], for branded and unbranded foods. The results of this study complement results from the same study reported previously. Those results, based on invasive PPG heart rates, as well as skin conductance measures, showed that facial expressions, PPG heart rate, and skin conductance varied significantly with the level of familiarity of product for the consumer and with the type of branding and varied less with the brand of soy sauce. Visual analogue scale (VAS) liking and arousal scores also varied with the level of familiarity and with the brand of soy sauce. In contrast with the other measures, VAS scores did not vary with branding [27].

This study aims at (1) the comparison of valence and arousal from video-based facial expressions with Emoji scores during food tasting, and (2) the comparison of video-based RPPG heart rate and PPG heart rate during food viewing and tasting. Specifically, the sensitivity of the methods to detecting effects of branding, soy sauce, and degree of consumers' familiarity will be compared. For these purposes, samples of branded and unbranded soy sauces are presented to experienced and inexperienced consumers. During visual inspection and tasting, video-based facial expressions, RPPG and PPG heart rate, and Emoji scores of valence and arousal [26] were recorded.
