**4. Discussion**

The study investigated the effects of consumer' experience and branding on reactions to the sight and taste of branded and unbranded soy sauces. This study aimed at (1) the comparison of valence and arousal from video-based facial expressions and from Emoji scores during food tasting, and (2) the comparison of video based RPPG heart rate and PPG heart rate during food viewing and tasting.

With regard to the first aim, the study showed differences between the results from implicit (facial expressions and heart rate) and explicit (scores) responses. Explicit Emoji taste scores of arousal and valence showed significant differences between the brands of soy sauce but showed no effects of branding and consumer' experience. In general, facial expressions of arousal and valence primarily showed effects of branding and consumer' experiences. Moreover, these differences were not only found during tasting but also prior to tasting when the participants' looked at the unbranded and branded foods. This illustrates the importance of consumers' expectations based on so-called extrinsic food factors, such as the food's visible appearance, the packaging, and branding information. It also suggests that measures such as facial expressions and heart rate are better suited to investigate the role of extrinsic factors than the more traditional measures such as scores. Overall, this outcome corresponds with other results of the same study reported earlier [27], as well as results of a recent study that also showed that scores reflected differences between brands of soy sauce but were not affected by branding [29].

The PPG and facial expression results from the same study reported previously [27] failed to show systematic differences between soy sauces, whereas in the results reported here occasional differences between soy sauces were found during viewing as well as tasting. These differences are probably related to differences in the way the results of the earlier and this study were analyzed. In the previous study, changes in heart rate, and facial expressions, relative to the start of viewing and tasting were analyzed, whereas in this study absolute values were used. The fact that differences between soy sauces were found in the absolute values and not in the relative values suggest that these differences do not develop during tasting but have already developed previously during the viewing of the food and branding information, and the following anticipation phases, and are continued during tasting. This illustrates how measures such as facial expressions and heart rate provide a window into processes that start prior to tasting with expectations based on visual (as well as possibly olfactory and auditory) information and that affect experiences during subsequent tasting. These interactions between expectations and actual taste experiences are not (always) reflected in scores. The richness of the processing during visual inspection and subsequent tasting that is reflected in the heart rate and facial expressions may also be a reason why heart rates, and responses of the autonomic system in general, do not always show differences between products when they are based solely

on taste responses. For example, Gunaratne [41] presented participants with unbranded chocolate samples and failed to find effects of specific chocolate tastes on heart rate and skin temperature (see also [13,14], who showed similar results for chocolate and beer, respectively). Other studies did demonstrate significant product effects on ANS responses, but these effects were typically small and difficult to interpret (e.g., [18]). The higher sensitivity of heart rate and facial expressions compared to explicit scores for factors such as branding that are not directly related to the food's taste properties is in line with our previous study. In that study, participants evaluated foods repeatedly in their own home and in the sensory laboratory [20]. Factors such as time of the day, day of the week, and sample preparation and presentation procedure were kept constant, whereas the physical location was systematically varied. Again, the results showed that explicit scores reflected primarily the differences between the test foods, whereas the heart rate and facial expressions reflected differences between test locations as well. Combined, the results of these studies suggest that physiological measures and facial expressions may reflect experiences in general and not only experiences directly related to foods. These experiences may be related to a specific mood that is affected by the food that is consumed and the circumstances in which the food is consumed. A close link between the consumption context and how consumers feel has been postulated by others [42], and these feelings are thought to be underlying modulators of food perception, food liking, and overall enjoyment of human eating experiences [43]. Others, such as Meiselman [44], have also stressed the importance of assessing foods and drinks in accurate contexts in order to increase the external validity of the results of consumer tests. The present study demonstrates that physiological measures and facial expressions may be especially well suited for the assessment of general food experiences.

Subjective rating scores such as the Emoji scores used in this study are single point measurements where all the consumer' processes during viewing, anticipation, and tasting are collapsed over time into one single score. An obvious result is that many of the nuances pre- and during tasting are blunted and possibly lost. Facial expressions and heart rate offer a continuous measurement of the pre- and during tasting processes. The fact that facial expressions and PPG heart rate vary between viewing and tasting, and show variation over time, illustrates that these consumer' processes are highly dynamical. In order to understand consumer' processes, these dynamics need to be considered when selecting the appropriate measurement. This means that the measurements need to be continuous, like facial expressions and heart rate, and not static like scores.

The second aim of the study was the comparison of heart rate recorded with noninvasive video-based RPPG and with invasive PPG (which relates closely to ECG, the ground truth). A successful comparison would mean that physiological heart rate recordings are no longer limited to laboratory studies but could be also applied to studies in realistic consumer situations. This is important because consumer' reactions to foods are not only driven by food properties but also by the physical and social context in which the food is consumed. Another possible new application would be internet-based surveys where large numbers of consumers can be reached. Unfortunately, the comparison of RPPG and PPG heart rates indicates that at this moment the RPPG method cannot fully replace the PPG and other invasive methods. RPPG heart rates were systematically approximately 10–12 BPM slower than PPG heart rates, and this difference seemed to be larger for higher PPG heart rates associated with for example chewing activity. Similar results were reported previously by Kanemura et al. [33], who speculated that some of the differences could be related to the effects of chewing activity on skin color. In addition, PPG heart rates changed rapidly over time during viewing and tasting, whereas RPPG heart rates were relatively stable. Rapid changes in PPG heart rates have been associated with orienting responses to new and unexpected stimuli and attentional processes (e.g., [21,45]). The results of this study suggest that the current RPPG heart rate measurements cannot (yet) be used to study this detailed level of processes. Even though RPPG heart rates based on video images currently lack the high level of sensitivity provided by "gold-standard"

measurements, RPPG heart rates showed overall effects of consumers' experience, phase (viewing, anticipation, and tasting) and occasionally of brand of soy sauce, similar to PPG heart rates. Obviously, further development of the RPPG technique is needed for a closer match with PPG heart rates, and additional research is needed to explore the relation between both types of heart rate measurements further.

This research contributes—together with other studies—to the understanding of how experiences and expectations from packaging and branding interact with subsequent taste experiences. The results clearly demonstrate that taste reactions are directly related to reactions to packages and branding, and that both need to be considered for the development of new products. In traditional sensory research, taste experiences are typically tested with unbranded products, i.e., without any packaging and branding information. In subsequent consumer tests, only products that successfully pass the initial sensory taste test were tested together with branding and packaging information. This, as well as other research, indicate that this step-wise approach may not be the most efficient way to develop new successful products (see also [46]). This study also demonstrates the need to not only assess effects of branding/packaging on taste reactions but also to measure reactions to the branding/packaging itself. There reactions are especially difficult for consumers to articulate because they are often unaware of them. In those cases, measures such as facial expressions and physiological measurements may provide important insights, not only into the effects of branding/packaging but also into the interactions with specific consumer characteristics, such as the consumers' experience with the brand or product.

In conclusion, this study suggests that video-based measurements of facial expressions and RPPG heart rates offer opportunities for large-scale web-studies to investigate consumers' reactions to extrinsic food factors such as packages, labels, logos, and product information. In addition, facial expressions and RPPG heart rates may be well suited to study the effects of consumers' traits, such as level of experience with the foods, on their reactions to extrinsic food factors.

**Author Contributions:** Conceptualization: R.A.d.W., S.U., and D.K., Methodology: P.Z., R.A.d.W. Formal analysis: M.U., P.Z. Investigation, M.U., P.Z., and S.U. Writing original manuscript: R.A.d.W., D.K., and S.U., writing review + editing: M.H.V., P.Z., S.U., and M.U., supervision: M.H.V., project administration: M.H.V. All authors have read and agreed to the published version of the manuscript.

**Funding:** This study was partially funded by a grant from Kikkoman Europe R&D Laboratory B.V.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Social Sciences Ethics Committee (SEC). The Committee has concluded that the proposal deals with ethical issues in a satisfactory way and that it complies with the Netherlands Code of Conduct for Research integrity and the Netherlands Code of Ethics for Research in the Social and Behavioral Sciences involving Human Participants.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Conflicts of Interest:** This study was partially funded by a grant from Kikkoman Europe R&D Laboratory B.V. Shota Ushiama and Daisuke Kaneko are both employed by Kikkoman Europe R.&D Laboratory B.V. and by research organizations (WUR and TNO). A possible conflict of interest was prevented by following the WUR-integrity code (URL: https://www.wur.nl/en/About-WUR/ Integrity-and-privacy/Scientific-integrity.htm, (accessed on 5 April 2021)). Products from different suppliers were included in the study.
