Coffee Label Assessment Using Sensory and Biometric Analysis of Self-Isolating Panelists through Videoconference

Abstract

Label concepts, information, logos, figures, and colors of beverages are critical for consumer perception, preference, and purchase intention. This is especially relevant for new beverage products. During social isolation, many sensory laboratories were unable to provide services, making virtual sensory sessions relevant to studying different label concepts and design preferences among consumers. This study proposed a novel virtual sensory system to analyze coffee labels using videoconference, self-reported, and biometric analysis software from video recordings to obtain sensory and emotional responses from 69 participants (power analysis: 1 − β > 0.99) using six different label concepts: (i) fun, (ii) bold, (iii) natural, (iv) everyday, (v) classic, and (vi) premium. The results show that the label concept rated as having the highest perceived quality was premium, presenting significant differences (p < 0.05) compared to all of the other concepts. The least perceived quality score was attributed to the bold concept due to the confronting aroma lexicon (cheese dip), which is supported by previous studies. Furthermore, even though graphics, colors, and the product name can be considered positive attributes, they do not determine perceived quality or purchase intention, which was found for the bold, everyday, and classic concepts. The findings from this study were as expected and are consistent with those from similar publications related to labels, which shows that the proposed virtual method for sensory sessions and biometrics is reliable. Further technology has been proposed to use this system with multiple participants, which could help beverage companies perform virtual sensory analysis of new products’ labels.

1. Introduction

Packaging and the information presented on labels are the first points of contact between products and consumers, especially for new products and those unfamiliar to consumers. Therefore, packaging and labels play a significant role in determining preference, liking, and purchase intention [1,2]. Sensory analysis of packaging, labels, and label information is the easiest assessment that can be conducted in social isolation and when products are not as physically available for other sensory assessments. Packaging, labels, and label information can be presented to panelists and consumers through images as stimuli, which have been shown to render similar results statistically compared to those when the packaging is physically available [3] and compared to the sensory characteristics of other products, such as chocolate [4], yogurt [5], beef [6], and baby formula [3]. Furthermore, label designs are one of the most important factors defining the preference and purchase intention of consumers toward wines in general [7], consumers with low involvement [8], and millennials [9], especially when assessing labels with recognizable features, such as animals, novelty designs [7], and colors [10].

There are many other sensory assessments of labels of products for specific purposes, such as the investigation of the effects on the preference of product designation of origin (PDO) [11], the influence of the lexicon used on labels [12], and the sensory perception of labels of different products [1]. Many studies have also found that the preferred information on labels is related to health implications, such as for organic products, which are more related to quality perception [13]. In coffee labels and sensory perception, a study found that aroma descriptors on labels are more associated with positive perception and liking than taste descriptors [14]. The latter is different from other studies in which labels influence taste perception [15], preference, and purchase intention [4].

During the pandemic of 2020 (COVID-19), many sensory laboratories around the world had to cease operations during lockdown periods, and in 2021, many may not be able to resume normal sensory sessions or will be restricted due to social distancing. Hence, there is a requirement for the implementation of digital tools to perform sensory tests remotely. Many commercial software manufacturers started developing online tools for sensory analysis, such as Affectiva (Affectiva, Boston, MA, USA), Noldus (Face Reader, Noldus Information Technology, Wageningen, Netherlands), and MorphCast^® (MorphCast, Florence, Italy), among others. The Digital Agriculture, Food and Wine (DAFW) research group, belonging to the University of Melbourne, has been developing these tools since 2012, which has resulted in the BioSensory computer application (app) [16]. This app allows incorporating self-reported sensory assessments and biometrics, using videos from panelists recorded automatically for different questions that are customizable to fit different tests, such as in-app assessment of images, sound, and videos, in which hedonic, continuous line, check all that apply using either words or emojis, and ranking scales can be used. The technology and algorithms developed within the BioSensory app are based on artificial intelligence and machine learning to obtain biometrics from video analysis, such as heart rate changes, blood pressure, emotional response based on facial expressions, and posture changes. Hence, this system is versatile enough to be implemented using online communication through synchronous media with Zoom (Zoom Video Communications, Inc., San Jose, CA, USA), WebEx (Cisco Systems, Milpitas, CA, USA), Google Meetings (Google, LLC, Mountain View, CA, USA), or Hangouts (Google, LLC, Mountain View, CA, USA), and data gathering with, for example, Google Forms (Google, LLC, Mountain View, CA, USA), Microsoft Forms (Microsoft Corporation, Redmond, WA, USA), Compusense (Compusense Inc., Guelph, ON, Canada), RedJade (RedJade Sensory Solutions, LLC, Martinez, CA, USA), and Fizz (Biosystemes, Couternon, France), among others.

This research was based on the working hypothesis that by implementing online freeware resources and novel biometric methods that are comparable to more traditional techniques performed in sensory laboratories requiring the participants’ attendance, it is possible to perform sensory analysis of new product coffee labels in social isolation. Therefore, this study aimed to implement accessible digital tools, such as Zoom, Google Forms, and Affectiva, to remotely assess the sensory, emotional, and biometric responses of panelists to newly designed coffee labels using virtual sensory sessions.

2. Materials and Methods

2.1. Label Sample Description

Six labels for coffee pods with different intensity levels and caffeine content were designed based on the TNS NeedScope model™ (NeedScope International, Auckland, New Zealand) segments. These concepts had different market classifications: (i) fun, (ii) bold, (iii) everyday, (iv) natural, (v) classic, and (vi) premium (Figure 1a–f, respectively) [2,17,18]. The elements considered for the label design to meet each concept’s layers (archetypes, needs, and emotions) involved colors, the layout of the different components, fonts, patterns/textures, graphics, the brand, the logo, the intensity scale, and the product name. All labels were developed using Photoshop (Adobe Inc., San Jose, CA, USA) and Procreate (Savage Interactive, North Hobart, Tas, Australia).

2.2. Consumer Sensory Session

A virtual sensory session was conducted with 69 participants (Age: 21–53 years old; 78% female, 22% male) recruited from the staff and students from The University of Melbourne (UoM), Australia, and e-mails to personal contacts; participants were regular coffee consumers (at least once a week). According to the Power analysis conducted using SAS^® v. 9.4 (SAS Institute, Cary, NC, USA), the number of participants was statistically sufficient to find significant differences between samples (1 − β > 0.99). The sessions lasted 10–15 min per participant and were conducted via Zoom (Zoom Video Communications, Inc., San Jose, CA, USA) using Google Forms (Google, LLC, Mountain View, CA, USA) to display the questionnaire and labels. This setup allowed a host to monitor the session in the background and record videos of the participants while looking at the label (15 s) to obtain their biometrics through facial recognition to further assess their emotional subconscious responses [16,19]. Before the session, the participants were asked to read the plain language statement explaining the experiment and sign a consent form, both approved by the Human Ethics Advisory Group from the UoM (ID: 1953926.4). They were also instructed to take the session in a quiet place with uniform lighting and a neutral background to avoid any interruptions and potential bias. Besides the written instructions, a video was presented to each participant before the session explaining these instructions with subtitles for non-native English speakers. The testing setting of each participant was verified before the researcher started the sessions to initiate the test.

The labels were randomized once (prior to the questionnaire development) and presented in a fixed order for all participants; the order presented is shown in Figure 1. The questionnaire consisted of the assessment of acceptability based on different attributes, such as perceived strength, pleasantness (valence) [20], arousal [21,22], FaceScale (emotional response), perceived quality, and willingness to purchase (

Table 1. Attributes assessed for acceptability of the coffee pod label samples.

Attribute	Abbreviation	Scale	Anchors
Strength	Strength	9-point scale	1: Extremely mild–9: Extremely strong
Pleasantness (Valence)	Pleasantness	9-point hedonic scale	1: Unpleasant–9: Pleasant
Arousal	Arousal	9-point scale	1: Relaxed–9: Stimulated
FaceScale (Emotional response)	FaceScale	9-point hedonic scale
Perceived quality	PQuality	9-point scale	1: Extremely low–9: Extremely high
Willingness to purchase	WPurchase	9-point scale	1: Extremely unlikely–9: Extremely likely

), as well as check all that apply (CATA) questions for emojis and preferred areas of interest from label elements (

Table 2. Options provided for the check all that apply (CATA) questions for assessment of the coffee pod label samples.

CATA Emojis
CATA Area of Interest
Colors	Layout	Font	Pattern/Texture
Graphics	Brand	Logo	Intensity Scale
Product name

).

2.3. Video Analysis to Obtain Biometrics

All videos from participants recorded per sample were analyzed in batch using an automatic computer application developed by the Digital Agriculture Food and Wine (DAFW) group based on the Affectiva (Affectiva, Boston, MA, USA) software development kit (SDK). This application is able to assess participants’ facial expressions from videos using a histogram of the oriented gradient algorithms, which are automatically translated into emotions and related emojis (

Table 3. Attributes obtained from the participants’ video analysis for emotional responses.

Attribute	Type of Response	Attribute	Type of Response
Sadness	Emotion	Valence	Emotional dimension
Anger	Emotion		Emoji (Smiley)
Surprise	Emotion		Emoji (Laughing)
Fear	Emotion		Emoji (Disappointed)
Disgust	Emotion		Emoji (Rage)
Joy	Emotion		Emoji (Wink)
Engagement	Emotional dimension		Emoji (Scream)
Relaxed	Emotional dimension

) using support vector machine modeling algorithms in a batch analysis fashion [19,23].

2.4. Statistical Analysis

All quantitative data (Table 1 and Table 3) were analyzed through ANOVA to assess significant differences among samples (p < 0.05), with the least significant differences post hoc test (α = 0.05) using XLSTAT ver. 2020.3.1 (Addinsoft, New York, NY, USA). Frequency data were analyzed using Cochran’s Q test (p < 0.05) and McNemar with Bonferroni correction post hoc test for pairwise comparison using XLSTAT.

The XLSTAT software was also used to conduct multivariate data analysis as multiple factor analysis (MFA) with mixed data (quantitative and frequencies) based on correlations between variables and factors using all parameters assessed in the sensory session (Table 1, Table 2 and Table 3) and eliminating those whose correlation coefficient (r) did not contribute much to both factors (r < 0.35 in both F1 and F2). Furthermore, a customized Matlab^® R2020b (Mathworks Inc., Natick, MA, USA) code was used to develop a matrix using the quantitative data (Table 1 and Table 3) to assess only the significant correlations (p < 0.05) between the self-reported and biometric responses.

3. Results

3.1. Biometrics for Emotional Responses

Non-significant differences (p > 0.05) were found between the six coffee labels for any of the biometric emotional responses from participants; the means and standard error are shown in Supplementary Material Table S1. On the other hand,

Table 4. Mean ± standard error values from the acceptance test for the six different label samples (Figure 1).

Sample/Attribute	Strength	Pleasantness	Arousal	FaceScale	PQuality	WPurchase
Fun	4.69 d	6.23 b,c	4.89 b	5.86 b	5.66 d	4.84 b
	±0.22	±0.23	±0.25	±0.23	±0.23	±0.26
Bold	4.27 d	4.20 d	5.63 a	4.19 c	4.55 e	3.70 c
	±0.10	±0.20	±0.23	±0.21	±0.14	±0.24
Natural	6.41 b	7.31 a	3.75 c	7.09 a	7.20 b	6.80 a
	±0.25	±0.21	±0.19	±0.24	±0.24	±0.28
Everyday	5.53 c	6.00 c	4.39 b,c	5.95 b	5.88 c,d	5.50 b
	±0.19	±0.22	±0.20	±0.23	±0.22	±0.26
Classic	6.86 b	5.78 c	4.64 b	5.64 b	6.44 c	5.31 b
	±0.23	±0.14	±0.26	±0.17	±0.16	±0.20
Premium	8.48 a	6.77 a,b	6.27 a	6.94 a	7.95 a	6.61 a
	±0.20	±0.15	±0.22	±0.14	±0.17	±0.28

Different letters denote significant differences between samples (labels) according to the least significant difference (LSD) post hoc test (α = 0.05). NS: non-significant. Abbreviations: PQuality: perceived quality; WPurchase: willingness to purchase.

shows that all attributes from the acceptance test self-reported responses had significant differences (p < 0.05) between samples. The bold label was rated as the lowest in perceived strength (4.27), pleasantness (4.20), FaceScale (4.19), perceived quality (PQuality) (4.55), and willingness to purchase (WPurchase) (3.70), while the natural label was rated as the lowest in arousal (3.75). On the other hand, the premium label was rated the highest in strength (8.48), arousal (6.27), and PQuality (7.95). Furthermore, premium was non-significantly different to natural in pleasantness (6.77 and 7.31, respectively), FaceScale (6.94 and 7.09, respectively), and WPurchase (6.61 and 6.80, respectively).

3.2. Emotional Responses Based on Emojis

Table 5. McNemar test values from the emoji check all that apply test for the six different label samples (Figure 1).

Emojis/Samples	Fun	Bold	Natural	Everyday	Classic	Premium
	0.26 b,c	0.49 a,b	0.25 b,c	0.51 a	0.13 c	0.44 a,b
	0.17 a,b	0.29 a,b	0.29 a,b	0.12 b	0.41 a	0.13 b
NS	0.00	0.00	0.01	0.00	0.04	0.01
	0.12 b	0.09 b	0.23 a,b	0.12 b	0.38 a	0.42 a
NS	0.06	0.00	0.09	0.13	0.09	0.13
NS	0.03	0.00	0.01	0.00	0.07	0.03
	0.06 b	0.04 b	0.10 ab	0.01 b	0.28 a	0.00 b
NS	0.00	0.01	0.03	0.00	0.07	0.09
NS	0.23	0.13	0.07	0.06	0.20	0.06
NS	0.17	0.17	0.07	0.26	0.06	0.16
NS	0.09	0.01	0.07	0.01	0.07	0.04
	0.09 b	0.23 a,b	0.15 a,b	0.30 a	0.07 b	0.28 a,b
	0.39 a,b,c	0.52 a,b	0.39 a,b,c	0.61 a	0.20 c	0.32 a,b
	0.12 a,b	0.07 b	0.06 b	0.16 ab	0.03 b	0.32 a
NS	0.01	0.00	0.03	0.06	0.04	0.00
	0.32 a	0.17 a,b	0.25 a	0.16 a,b	0.23 a,b	0.04 b
	0.01	0.00 NS	0.04	0.00	0.15	0.00
	0.17 a,b,c	0.19 a,b	0.15 b,c	0.03 c	0.38 a	0.13 b,c

Different letters denote significant differences between samples (labels/columns) according to the McNemar (Bonferroni) post hoc test (p < 0.05). NS: non-significant.

shows that there were non-significant differences (p > 0.05) between samples in the selection of emojis, such as , , , , , , , , and . The premium label presented a high frequency of selections for positive emojis, such as , , , , and , and shared the emojis , , and with everyday and bold, presenting non-significant differences among these three labels. On the other hand, classic had the highest frequency of selections for doubtful/pensive emojis such as , , and . The neutral emoji was most selected for fun and natural labels.

3.3. Sensory Perception of Label Features

Table 6. McNemar test values from the area of interest (AOI) check all that apply test for the six different label samples (Figure 1).

AOI/Samples	Fun	Bold	Natural	Everyday	Classic	Premium
Colors	0.48 b,c	0.57 a,b	0.39 b,c	0.75 a	0.25 c	0.75 a
Layout	0.41 b,c	0.39 b,c	0.52 a,b,c	0.77 a	0.35 c	0.62 a,b
Font	0.48 a	0.29 a,b	0.25 a,b	0.45 a,b	0.22 b	0.42 a,b
Pattern/Texture	0.22 b	0.30 a,b	0.26 b	0.55 a	0.26 b	0.45 a,b
Graphics	0.15 b	0.35 b	0.36 a,b	0.60 a	0.17 b	0.26 b
* Brand	0.09 c	0.07 c	0.12 b,c	0.26 a,b	0.07 c	0.29 a
NS Logo	0.26	0.25	0.29	0.32	0.12	0.35
NS Intensity scale	0.36	0.26	0.36	0.44	0.33	0.51
NS Product name	0.28	0.19	0.19	0.2	0.29	0.26

Different letters denote significant differences between samples (labels/columns) according to the McNemar (Bonferroni) post hoc test (p < 0.05). NS: non-significant. * Brand was assessed using the critical difference Sheskin post hoc test due to a problem of transitivity using the McNemar test for this specific area of interest.

shows non-significant differences (p < 0.05) between samples for the selection of logo, intensity scale, and product name. In premium, the most selected areas of interest (AOI) were colors, layout, and pattern/texture, showing significant differences with fun and classic. The latter was the lowest in the frequency of selections for all AOI.

3.4. Multivariate Data Analysis

Figure 2a shows the MFA in which the factors represent a total of 74.41% of data variability (factor one: F1 = 45.13%; factor two: F2 = 29.27%). It can be observed that based on the correlation coefficients (r) between variables and factors, F1 was mainly represented by (r = 0.95), and (r = 0.94) on the positive side of the axis, and by pleasantness (r = −0.98), FaceScale (r = −0.95), and (r = −0.95) on the negative side. On the other hand, F2 was mainly characterized by (r = 0.87), and sadness (r = 0.86) on the positive side, and by valence (r = −0.74), and brand (r = −0.70) on the negative side of the axis. The bold label was associated with the selection of graphics as the best AOI and subconscious responses from biometrics, such as engagement, , anger, , , and sadness. The everyday label concept was associated with the selection of colors as the best AOI; biometric responses, such as joy and smile; self-reported responses, such as pleasantness; and a selection of emojis, such as , , and . Furthermore, the premium label was more associated with the self-reported responses, such as perceived strength; the selection of brand as the best AOI; PQuality; a selection of emojis, such as and ; and biometric responses, such as relaxed and valence. On the other hand, the classic label was associated with the self-reported responses for the selection of product names as the best AOI and most neutral to negative emojis, such as , , , , and . The natural and fun labels were located closer to the MFA center, with the former being located on the positive emotion side of the graph and the latter presenting negative associations with the bold label’s characteristics.

Figure 2b shows the matrix with significant correlations between the self-reported acceptability and biometric responses. It can be observed that, despite being low, there were negative and significant correlations (p < 0.05) between and pleasantness (r = −0.16), PQuality (r = −0.17), FaceScale (r = −0.12), and WPurchase (r = −0.17). Similar correlations were found between anger facial expression and the aforementioned self-reported responses. Furthermore, there was a negative low but significant correlation between surprise and FaceScale (r = −0.12), and positive correlations between self-reported arousal and engagement and smile (r = 0.11). On the other hand, PQuality was positively correlated with strength (r = 0.63), pleasantness (r = 0.71) and FaceScale (r = 0.79).

4. Discussion

4.1. Virtual Sensory Sessions

The virtual sensory sessions for individual participants ran smoothly, and data gathering and analysis were completed within three weeks, with only one person required for data collection and one for data analysis. The time of virtual sessions and data analysis can be reduced by having multiple participants in one session through Zoom, in which they can all be added to a single screen that is recorded. New video analysis algorithms developed by the DAFW group can crop each participant, thereby allowing them to be analyzed automatically through the batch code described in this study using Affectiva algorithms. An advantage of sensory tests during social isolation, using the available digital tools, is that potential panelists are in familiar environments, such as their homes, places dedicated to work, or where social interactions may occur when doing the test with family members or friends [24]. In places dedicated to work, consumers are more likely to test different products. To date, there are no published papers using or proposing virtual sensory sessions that include biometrics recording to be used in emergency situations, such as extensive lockdown during pandemics, in which participants may be presented with conditions that do not allow them to attend a sensory laboratory or to reach more participants in other cities or countries.

4.2. Emotional Responses from Biometrics and Emoji Selection

The most accepted label from this study with positive emotional responses and positive emojis was the premium concept, which is in accordance with other label studies for chocolates [4]. The opposite responses were obtained for the bold label with the cheese dip concept, which could have been associated with cheesy aroma in the coffee (Table 4 and Table 5). It has been shown that smell and positive aroma descriptors are determinants of consumer perception [15], which could explain this study’s results. Furthermore, the second-highest scores were for the natural concept, which incorporated grains of coffee and more earthy colors associated with a healthy option in the imagery, which is also in accordance with other studies involving healthy and organic concepts within labels [13,25,26].

4.3. Analysis of Areas of Interest within Labels

Following the results from the emotional and emoji responses, the premium label was highlighted with statistically higher areas of interest selection, such as colors, layout, and pattern/texture, with layout being the highest area of interest for the natural concept and color for the bold concept (Table 6). However, analysis of features was limited in this study; more specific assessments can be conducting when analyzing label features using eye-trackers [2,3,27,28,29] or more practically for virtual sensory sessions when using digital tools (from video analysis) for eye tracking to assess fixation number and duration that the panelists spend at each component [30,31]. The DAFW group has developed algorithms to analyze eye-tracking parameters and the emotional response based on facial expressions from the different features of labels [32]. Hence, further studies using the methodologies proposed here will enrich the number of parameters to be analyzed and find different patterns within the data or apply machine learning algorithms for artificial intelligence application to label analysis.

4.4. Multivariate Data Analysis

The multivariate data analysis (Figure 2a), including the emotional response, emoji selection, and analysis of label areas of interest, offered a more comprehensive analysis of the relationships and patterns among the different labels and concepts. The main addressable features with this analysis are the positive selection of graphics with the least preferred concept (bold), which shows that even if the graphics are preferred, the concepts within the label related to the specific product are more important for consumers. Similar assessment can be conducted for the concept of fun, which was closely related to the panelists’ color interests. Furthermore, it was also found that the name was mostly associated with positive and negative emojis, with the classic concept offering more interest but without correlated results for preference.

In the case of the correlations among the self-reported and subconscious responses (Figure 2b), it was found that consumers rate labels as higher quality when they perceive coffee as higher strength, when labels are more pleasant, and when they elicit more positive emotions; the latter is consistent with findings from Gunaratne et al. [2] for chocolate label assessment. Furthermore, the willingness to purchase is directly influenced by the perceived quality, as found in other research [33].

One of the limitations of this study with the methodology proposed was the requirement of testing one participant at a time due to video recording using the available meeting software capabilities. However, this was solved after the study was conducted by assessing a video of multiple participants in a single screen recording and automatically cropping participants for further analysis. This was achieved by coding video cropping using computer vision algorithms in Matlab^®. Furthermore, this study was based on online easy-to-access and free tools, such as Zoom (free sessions for up to 1 h), Google Forms, and Affectiva SDK. More information can be accessed using more specific self-reported software packages specific for sensory analysis, such as RedJade [34,35], Compusense [36,37], and Fizz [38], among others. However, these packages may be cost prohibitive for small and medium food and beverage companies.

Further research should be conducted using more specialized software to obtain more information that can be relevant to companies’ decision making regarding product and package development.

5. Conclusions

Social isolation during the pandemic of 2020 served as an incentive for many software companies to develop virtual or remote sensory tests to comply with social distancing and lockdowns. This study implemented new and emerging sensory tools to analyze coffee labels, thereby obtaining results consistent with previous studies for coffee and other beverage labels using normal sensory analysis with panelists attending physically sensory sessions. This research showed that free online software resources, such as remote meeting applications, video capture, and self-reported forms, are effective tools to carry out sensory analysis of labels outside the laboratory when social isolation and distancing are required, with results comparable to other more established methods. These techniques can also be applied to other food and beverage products by sending them to consumers via courier/mail to be tested using sensory techniques. The tools proposed in this study are free and open-source software tools, such as Google Forms, Affectiva, and Zoom, the latter with free usage for multiple 1 h sessions. These software packages are advantageous to conduct label and even food and beverage product sensory analysis compared to commercial software that can do similar sensory studies as described in this paper by incorporating facial expressions and emotional responses. However, they are cost-prohibitive, especially for small and medium food and beverage companies. Furthermore, this study proposed more efficient ways to conduct these sessions with multiple participants on single videos and incorporate eye-tracking software from videos to analyze the emotional response to specific features from different beverage products’ labels. The system proposed in this study could be of great benefit to food and beverage companies not only in the context of isolation conditions but by increasing the reach of the sensory trials to other countries, thereby incorporating a higher number of participants unrestricted by sensory laboratory space, different cultural backgrounds, or age and without international or language boundaries.