Facial Expression-Based Experimental Analysis of Human Reactions and Psychological Comfort on Glass Structures in Buildings

Abstract

For engineering applications, human comfort in the built environment depends on several objective aspects that can be mathematically controlled and limited to reference performance indicators. Typical examples include structural, energy and thermal issues, and others. Human reactions, however, are also sensitive to a multitude of aspects that can be associated with design concepts of the so-called “emotional architecture”, through which subjective feelings, nervous states and emotions of end-users are evoked by constructional details. The interactions of several objective and subjective parameters can make the “optimal” building design challenging, and this is especially the case for new technical concepts, constructional materials and techniques. In this paper, a remote experimental methodology is proposed to explore and quantify the prevailing human reactions and psychological comfort trends for building occupants, with a focus on end-users exposed to structural glass environments. Major advantages were taken from the use of virtual visual stimuli and facial expression automatic recognition analysis, and from the active support of 30 volunteers. As shown, while glass is often used in constructions, several intrinsic features (transparency, brittleness, etc.) are responsible for subjective feelings that can affect the overall psychological comfort of users. In this regard, the use of virtual built environments and facial expression analysis to quantify human reactions can represent an efficient system to support the building design process.

1. Introduction

In building design, the end-user’s comfort is a target for a multitude of applications. These include, for example, thermal comfort, indoor air quality, visual comfort, noise nuisance, ergonomics and vibrations. With around 90% of our lives spent in buildings, there is a strong link between comfort and the built environment [1,2]. For decades, various researchers have sought to understand how the characteristics of the built environment can impact the emotions, behaviors and physical well-being of end-users [2]. However, comfort itself depends on a great number of factors which can, if not addressed properly, lead to annoyance. Further, it is known that such analysis requires convergent teams from the humanities, arts, social sciences, technology, engineering and medicine. With a more explicit focus on engineering issues, comfort strictly relates to many topics of primary interest for building technology [3,4,5].

Structural issues and comfort can be, for example, considered in terms of vibration serviceability assessments. Usually, the evaluation of vibration issues in relation to possible annoyance risk is carried out in terms of acceleration peaks and recommended limit values. Similar methods can be used for the comfort analysis of floors under walking or standing conditions [6,7,8], and for tall buildings against wind [9,10,11]. As a matter of fact, human reactions and comfort levels strongly depend on engineering parameters, but also on the physiological perception of the frequency and amplitudes of vibrations. The operational context, including location (street, gym, office, home, etc.), time of day (morning or evening) and stimuli duration (seconds or hours) also have severe impacts on the degree of human tolerance [8].

The structural use of materials and design concepts in buildings is an additional factor that can implicitly result in possible discomfort for the occupants. The well-known psychological effect of architectural solutions can have both positive and negative effects, and can evoke subjective feelings that reduce to a minimum the mechanical efforts of structural designers [12]. This is the major challenge of architects when evoking emotions in the design of so-called “emotional buildings” [13,14,15,16,17,18]. In the context of the built environment, experimental measurements can be carried out to quantify emotions [19]. Smart sensors or virtual walks are proven to represent useful tools in support of the analysis of human reactions to visual stimuli [20,21].

Among other things, structural glass is known to represent an attractive material for construction [22,23]. Key aspects are its transparency and abilities to adapt to various configurations and replace/interact with more traditional constructional materials. Typical examples can be found in facades, roofs, walkways, bridges and balustrades, as in Figure 1a–c. Besides, glass is also recognized as one of the most vulnerable components in constructions, and thus to need special structural design efforts in order to ensure appropriate safety levels. Glass facades and windows, for example, are the first physical barrier for building occupants, both in presence of ordinary operational conditions but also during accidental events that could result in potential shards and injuries (Figure 1d,e). The high aesthetic impact of glass components and structures is thus often in contrast with a basic discomfort and lack of safe feelings for end-users.

Glass design can be highly demanding for pedestrian systems, due to the combination of dynamic mechanical parameters and complex human–structure interaction (HSI) phenomena that typically occur during a walk, but also due to subjective reactions [8].

In general terms, glass structures are often called “architectures of vertigo,” given that transparent load-bearing systems are increasingly conceived as “spaces of visceral thrills with deep socio-spatial implications” [23]. Glass floors, in particular, can be recognized as intense psycho-physiological stimuli for pedestrians, as they induce unavoidable sensory experiences that go beyond the conventional HSI design procedures of analysis.

2. Research Goals and Approach

2.1. Glazed Stimuli

In this paper, a virtual experimental approach is presented in support of building design. The goal of the developed method is to explore and quantify the prevailing reactions and psychological comfort levels for end-users exposed to different visual built environments. At the time of analysis, careful attention was paid to the selection of visual stimuli characterized by a primary role of structural glass components in buildings and constructions (i.e., Figure 2a,b). The methodology can be adapted to different topics and design fields.

While the use of glass in buildings has been increasing in the last few decades, with new challenges for structural designers [25], the impact on end-users is often severely affected by the material’s transparency and brittleness, and also by the lack of technical knowledge regarding its mechanical properties [28,29]. In this regard, it is expected that the building design process could take advantage of subjective measurements and their possible combination with mathematical models and engineering calculations. As schematized in Figure 2c, the basic approach takes benefit from the quantitative measures of emotions and subjective feelings of individuals exposed to pre-selected visual stimuli. Based on the analysis of facial micro-expressions, comfort levels and trends can be analyzed by sub-groups of subjects of stimuli.

2.2. Experimental Procedure

The overall methodology was elaborated and designed at the University of Trieste (Department of Engineering and Architecture, Italy) in Winter 2020. The experimental study followed the procedural steps in Figure 3, and was carried out remotely.

More precisely:

• STEP 0: the participants were first recruited. The group of volunteers (30 in total) included mostly students and researchers, both females and males (58% and 42% respectively), with an age range of 20–56 (28.8 years-old was the average, ±8.4; 26 years-old the median value). Some volunteers were students or workers living in Trieste, or residing in the Friuli Venezia Giulia Region. The age demographics and geographic distribution of participants are shown in Figure 4. Their educational experience and background skills were characterized by different fields of study or activity. Most importantly, no preliminary technical knowledge on the use and features of structural glass in construction was required at the time of the experiment. Similarly, no direct experience on glass structures was needed to join the virtual investigation.
• STEP 1: A preliminary computer assisted Web interviewing (CAWI) survey was shared online and used to assess the multitude of subjective reactions for participants under different stimuli (Section 3).
• STEP 2: The virtual experimental analysis was carried out (Section 4) to study human behaviors in glass-involving scenarios, based on facial expression analysis. To that end, the FaceReader^TM automatic facial expression recognition software (version 8, Noldus Information Technology bv, Wageningen, Netherlands) [31] was used in support of the quantitative analysis of experimental measurements. Two different visual stimuli were designed to assess the reactions of volunteers, namely, consisting of a set of static input items (Section 4.2) and a dynamic virtual reality (VR) video clip of pre-recorded walks in glass environments (Section 4.3). In both cases, based on STEP 1, special care was taken in the selection and arrangement of stimuli, so as to capture different reactions and emotions of participants.
• STEP 3: the post-processing analysis of experimental measurements from STEP 2 was partly based on the automatic software analysis, and further elaborated as discussed in Section 4.4 and Section 4.5. Detailed comparative results are presented in Section 5 and Section 6 for static and dynamic stimuli respectively. As shown, the analysis of experimental measures proves that the use of structural glass in buildings is still affected by scattered human reactions. Moreover, the results from the proposed methodology suggest that the design of glass structures could benefit from subjective parameters that should be taken into account in the overall design process.

2.3. Strengths and Limitations of the Study

The proposed experimental methodology took partial inspiration from past literature, which has proved that human emotions can be used as efficient guide for designers [32,33]. The study and measuring of human behaviors based on facial micro-expressions is in fact typical of many research and market fields, especially where understanding consumers feelings can help designers to establish an efficient emotional communication between consumers and products. It is in fact generally recognized that both verbal and non-verbal behaviors enable humans to communicate emotions [12,13]. Non-verbal communication includes physical behaviors that are commonly referred to as body language and gestures, including also facial expressions. Most importantly, facial expressions are essential for the quantitative measure of basic emotions, because they provide information about inner states of individuals.

For the present study, the use of a virtual experimental setup and its adaptation to the constructional field was suggested by several motivations. The first and most severe one was represented, at the time of the investigation, by severe movement limitations, remote teaching regulations and smart-working rules due to the COVID-19 pandemic emergency (i.e., obligations for “red zone”/maximum risk measures in the Italian territory). As such, the experimental analysis was carried out with the active support of volunteers, but respectfully of social distance rules and general temporary prohibitions. Furthermore, technical trends and considerations suggested the design of the experimental study. The sanitary emergency revealed in fact that the use of transparent materials for constructions is expected to increase in the next future [29,34,35], so as to facilitate the definition of new design concepts for the organization of private or public spaces in buildings. Differently, this increasing use of glass in structures can be associated with a further magnification of “architecture of vertigo” concepts [23] earlier described. Balanced technical solutions should be thus necessarily developed. Both the cited aspects were merged to support the design of the experimental strategy herein presented.

Besides, the investigation was carried out with a clear preliminary analysis of intrinsic limitations that should be taken into account for future extensions. The number of participants (30), for example, was selected on the base of individuals availability (i.e., to join remotely the study, with own devices and high-speed internet connection). A possible extension of group size, and demographic/geographic distribution, could facilitate a more generalized analysis of reactions for specific sub-groups of participants. Another limitation of the study was represented by the fully remote experimental procedure. Under COVID-19 restrictions, the volunteers were invited to take part from homes/different regions of Italy. As such, the measure of facial expressions was carried out with shared screens only, without additional smart sensors, wearable devices or VR glasses [36,37].

3. CAWI Survey

The preliminary STEP 1 took the form of a CAWI survey that was designed to include 40 questions. The survey was designed with both the simple multiple choice (SMC) approach and the open question (OQ) method. The goal was to expose the participants to well-defined built conditions characterized by the presence of structural glass at different levels (residential buildings, open spaces, public offices, etc.), and to capture their feelings as active occupants. In this context, the survey revealed some important aspects, and suggested the development of the virtual experimental investigation herein discussed. For the majority of participants, it was proved that:

• background technical knowledge regarding mechanical features of structural glass in buildings often lacks. This evokes discomfort and negative perceptions, especially for design scenarios characterized by contact of end-users with glass (partition walls, balustrades, roofs, etc.).
• The brittle behavior of glass represents the most influencing parameter, and is frequently associated to a common feeling of protection lack against accidental events (i.e., shards due to impact, etc.).
• The visual detection of minor damage or material degradation in glass structural components (delamination, etc.) evokes severely negative reactions, even in presence of safe mechanical performances.
• The use of structural glass in pedestrian systems generally evokes very positive feelings on the architectural side. When the same participants are asked to ideally walk on those systems, however, the presence of transparency and possible vibrations, spots or noise during walks results in discomfort and highly negative comments.

4. Static and Dynamic VR Experiments

4.1. Methods

The experimental investigation was based on a sequence of pre-selected input sources that were used to act as emotional visual stimuli for the involved participants. The so-called STEP 2 from the flowchart in Figure 3 was developed as schematized in Figure 5, and represented the core of remote analysis. STEP 2 as a whole consisted of three sub-steps for each participant. The process was repeated twice, first with a static stimulus (sequence of pictures described in Section 4.2) and later with a VR video clip of pre-recorded virtual walks (Section 4.3). The remote experiments were carried out with the support of personal computers for all the volunteers, a shared screen for the visual stimuli and a webcam for recording the facial expressions of participants.

Before watching the shared input source, the invited subjects were informed about the experimental procedure (STEP 2.1 in Figure 5). A special care was paid to verify the connection speed, and minimize possible delay in transmission and acquisition of shared data. Successively (STEP 2.2 in Figure 5), the visualization setup was also properly checked, so as to avoid disturbing effects for the quality level of signals, and thus for the post-processing analysis. More in detail, the attention was given to ensure an optimal and mostly uniform ambient illumination for the participants. The screen position was also assessed for each participant, in order to prevent severe facial distortions and to avoid introducing high head/eye inclinations through the experiments. Finally, the presence of participants with glasses or beard was separately noted, for a more refined post-processing analysis of measured data.

Once verified the basic operational conditions, the input source was shared separately for each volunteer (STEP 2.3 in Figure 5). No auditory stimuli were introduced. During the presentations, however, audio communication of volunteers with the project manager was allowed to provide technical feedback (when needed). During the presentations (STEP 2.3), the project manager webcam was used to record the facial expressions of participants, while looking at the shared screen. To this end, a preliminary analysis to capture the optimal webcam resolution and sampling rate was carried out. The final choice resulted in records with minimum 1280 × 720 (or 1920 × 1028) video resolution and around 4300 frames for each recorded signal (30–60 the range of frame rate; 12,432 kbits/s the average bitrate).

The analysis of recorded signals was carried out during STEP 3 of Figure 5. The interpretation of facial micro-expressions and emotions from the participants exposed to the shared input source was partly carried out with the use of the commercial software FaceReader^TM [31]. The automatic computational analysis was based on all the collected frames (≈4300 the average) for each one of the available records (30 volunteers × 2 stimuli), and required approximately 10 min of analysis/each. The overall elaboration of signals and data analysis was then carried out as in Section 4.4. For the current study, it is thus important to note that the automatic software analysis was used to provide row quantitative data only of nervous states. The detection of human reactions and comfort trends for individuals exposed to glass environments was manually derived as in Section 4.5.

4.2. Static Input Source

The invited participants were first subjected to a shared input presentation that consisted in a selection of 27 pictures for various glass structures and scenarios. These pictures were equally spaced at time intervals of 5 s, so that the total duration of the visual stimulus could not exceed a maximum of 120 s, to avoid annoyance. Figure 6 shows an example of input pictures. A special care was spent for their selection (to capture emotions in the invited participants), but also for the definition of the optimal sequence. To that end, the pictures were selected from magazines, scientific journals, webpages of construction companies or newspapers. For all the participants, the sequence and duration of the static presentation was kept fix. Regarding the sequence, the choice was to alternate items possibly associated to comfort or negative feelings.

4.3. Dynamic Input Source

The setup schematized in Figure 5 was adapted and applied to the group of participants. The source consisted in a pre-recorded VR clip with total duration of 120 s, see Figure 7. The second stage of the investigation was in fact carried out to capture any kind of possible variation in the amplitude and trend of emotions of participants, when exposed to a virtual walk in a glazing environment. Major benefit was taken from the availability of a VR context representative of a case-study building located in Paris, and characterized by a large amount of glass components in facades, roofs, floors, balustrades. The VR context was used to pre-record a set of walks that could be used as stimulus. Based on preliminary studies, the clip was designed and divided onto three walks (40 s/each):

• At the top terrace level (W1),
• At the ground level of the building, from the main entrance (W2),
• At the first story level (W3).

Similarly to Section 5, no auditory stimuli were considered for the dynamic setup, and the participants were asked to watch the shared screen and join the virtual walks.

4.4. Experimental Measurements

The measurement of relevant data for comfort analysis was carried out in the post-processing stage of records for each participant, when exposed to both static and dynamic VR glazed environments.

The FaceReader^TM software, in particular, was chosen because it allows to model accurately the face of a given individual, by taking advantage of 500 key points for the description of facial movements and micro-expressions. The software is based on Artificial Intelligence tools and can be used to detect subjective emotions and human reactions from input video records or even pictures. As a whole, the software output takes the form of a series of quantitative charts and data that can be used for the interpretation of basic emotional states. In addition to the reference “neutral” condition, the standard classification carried out by the software includes basic definitions for individual states (N = 6 in total) targeted as “happy”, “sad”, “scared”, “disgusted”, “angry” or “surprised”. From a practical point of view, the software offers an instantaneous evaluation of Action Units (AUs, ranging from 0 to 1) for all these measured emotions. Additional useful feedback can be obtained by the quantitative analysis of output parameters such as valence, arousal, gaze direction, head orientation, heart rate (based on remote skin conductance analysis) and few additional personal characteristics of participants (such as gender, average age, presence of glasses, beard).

For the present study, AU records were collected for each participant/virtual stimulus/basic emotion, as a function of the time of experiment (with 0 ≤ t ≤ 120 s). AU data were exported from the software analysis with a sampling rate of 1 s. The typical example of AU records is shown in Figure 8 for one of the participants subjected to the dynamic VR setup. The chart, more in detail, gives evidence of specific AU values over time of virtual stimulus (i.e., STEP 2.3 in Figure 5), but omits the first instants of the experimental arrangement, when some preliminary discussion was carried out with the participant to provide instructions and create a comfortable condition.

In the overall procedure, moreover, it has to be noted that the “neutral” state computed by the software was excluded from manual elaboration of measured data, as it is shown in Figure 8. In presence of VR input sources and visual stimuli that do not involve active movements of participants, it is in fact generally recognized that the expected reactions and emotions cannot be quantitatively and qualitatively compared to feelings evoked by real field experiments. Accordingly, combination of “neutral” data with other basic emotional signals for the present study would have alter most of the measured subjective reactions and comfort trends.

4.5. Manual Post-Processing Strategy for AU of Basic Emotions

The quantitative and qualitative analysis of human comfort was based on the detection and measure of the herein called “positive” (POS, in the following) or “negative” (NEG) feelings from all the participants, when visually exposed to specific glazed environments.

Given that several AU records were obtained (as for 30 participants, N = 6 basic emotions, 2 virtual scenarios), the first approach of manual data elaboration consisted in the individual analysis of collected AU records. This stage was developed to explore the subjective feelings of single volunteers when exposed to a specific building scenario (i.e., glass floor, roof, balustrade, etc.). In addition, AU calculations were used to correlate the subjective response of different participants when exposed to the same VR input, and thus to find a prevailing emotional reaction/comfort trend for groups of built environment conditions. This means that for each VR scenario, basic emotions were subdivided onto two major categories representative of the herein called POS or NEG feelings and comfort trends. In doing so, the approach schematized in Figure 9 was take into account. The automatic software analysis was first carried out at STEP 3.1.

According to AU charts as in Figure 8, the preliminary requisite of analysis consisted in the normalization of AU records of basic emotions in the time of experiment, so as to facilitate the objective comparison of multiple AU signals (STEP 3.2 in Figure 9). Starting from the exported original AU charts (AU_original), more in detail, the average AU for each emotion at the beginning of the individual experiment/participant (AU_initial,avg) was taken as a reference for calculating the variation of subsequent AU measurements in time, that is:

$${\mathrm{AU}}_{\mathrm{norm}}\left(t\right)={\mathrm{AU}}_{\mathrm{original}}\left(t\right)-{\mathrm{AU}}_{\mathrm{initial},\mathrm{avg}}$$

(1)

with 0 ≤ t ≤ 120 s and:

• AU_norm(t) the normalized record for each participant, emotion, VR scenario, at the time instant t;
• AU_original(t) the original signal record for each participant, emotion, VR scenario, at the time instant t;
• AU_initial,avg the mean value of each emotion, participant, VR scenario (preliminary stage).

Following Equation (1), the individual records of basic emotions, AU_norm(t), were successively grouped in basic pos*(t) and neg*(t) plots (STEP 3.3 in Figure 9). In the pos*(t) set, the AU data in time for feelings marked as “happy” were considered only:

$${\mathrm{pos}}^{*}\left(t\right)={\displaystyle \sum }_{i=1}^{N-1}{\mathrm{AU}}_{\mathrm{norm}}\left(t\right)=f\left(happy\right)$$

(2)

Similarly, the neg*(t) set included the input data from feelings marked as “sad”, “angry”, “scared”, “disgusted”:

$${\mathrm{neg}}^{*}\left(t\right)={\displaystyle \sum }_{i=1}^{N-1}{\mathrm{AU}}_{\mathrm{norm}}\left(t\right)=f\left(sad, angry,scared,disgusted\right)$$

(3)

Note in STEP 3.4 of Figure 9 that the “surprised” AU data (“SUR(t)”) were omitted from the so-calculated pos*(t) and neg*(t) signals, and analyzed separately. This because (depending on the individual subject, context and stimulus) “surprised” states can be typically associated with both positive or negative feelings for the participants. The final marking of emotional data was thus based on the analysis of prevailing AUs at a given time instant t. The SUR(t) input was in fact superimposed as:

$${\mathrm{POS}}^{*}\left(t\right)=\left\{\begin{array}{c}\mathrm{p}\mathrm{o}{\mathrm{s}}^{*}\left(t\right)+\mathrm{S}\mathrm{U}\mathrm{R}\left(t\right) if \mathrm{p}\mathrm{o}{\mathrm{s}}^{*}\left(t\right)>\mathrm{n}\mathrm{e}{\mathrm{g}}^{*}\left(t\right)\\ \mathrm{p}\mathrm{o}{\mathrm{s}}^{*}\left(t\right) if \mathrm{p}\mathrm{o}{\mathrm{s}}^{*}\left(t\right)<\mathrm{n}\mathrm{e}{\mathrm{g}}^{*}\left(t\right)\end{array}\right.$$

(4)

or

$${\mathrm{NEG}}^{*}\left(t\right)=\left\{\begin{array}{c}\mathrm{n}\mathrm{e}{\mathrm{g}}^{*}\left(t\right)+\mathrm{S}\mathrm{U}\mathrm{R}\left(t\right) if \mathrm{n}\mathrm{e}{\mathrm{g}}^{*}\left(t\right)>\mathrm{p}\mathrm{o}{\mathrm{s}}^{*}\left(t\right)\\ \mathrm{n}\mathrm{e}{\mathrm{g}}^{*}\left(t\right) if \mathrm{n}\mathrm{e}{\mathrm{g}}^{*}\left(t\right)<\mathrm{p}\mathrm{o}{\mathrm{s}}^{*}\left(t\right)\end{array}\right.$$

(5)

An example of calculation results is shown in Figure 10, in terms of POS*(t) and NEG*(t) measured states for one selected participant.

In conclusion, the final detection of comfort trends for a given visual stimulus was carried out as in STEP 3.5 of Figure 9. To that end, the percentage variation of elaborated signals from STEP 3.4 was measured over the time of analysis. At each increment t of the experiments, the emotional variation was calculated for both POS*(t) and NEG*(t) data, and defined as:

$$\Delta \%=100\frac{\mathrm{Y}\left(t_n\right)-\mathrm{Y}\left(t_{n-1}\right)}{\mathrm{Y}\left(t_{n-1}\right)}$$

(6)

with Y = POS*(t) or NEG*(t) respectively, as shown in the example of Figure 11.

The prevailing emotional state (and thus comfort trend) was established in terms of absolute maximum values of nervous feelings and max/min percentage variations in the time of analysis. Each time instant t was classified with a POS or NEG mark, uniquely detected from data elaboration. A “comfort weight” value C(t) = POS or NEG (based on the prevailing nervous state) was assigned to each participant/instant/scenario, to facilitate the comparative analysis of multiple signals and stimuli.

The global correlation of comfort trends with the VR source was in fact carried out based on the synchronized comparison of elaborated data for the 30 participants subjected to an identical stimulus (STEP 3.6). More precisely:

• For the static experiment (Section 5), the prevailing comfort trend was calculated over time intervals of 5 s (corresponding to the duration of a single picture presentation), in terms of average comfort level;
• For the dynamic experiment (Section 6), the attention was focused on specific frames of pre-recorded video clips, and quantified in terms of average comfort levels for selected context scenarios.

5. Discussion of Results from Static Experiment

5.1. Prevailing Reactions

Figure 12 shows the analysis of prevailing reactions in the group of participants, as obtained by the investigation of elaborated records in the time of analysis (Figure 12a), and by the comparison of group reactions for a given static stimulus (Figure 12b). POS and NEG values are reported in percentage values for the whole group of volunteers. As such, the comparative data can be used to address the prevailing emotional state (in terms of average or maximum/minimum reaction of participants), towards the imposed picture/context. Figure 12a, more in detail, shows a clear modification of human reactions for the assigned stimuli, with marked fluctuations of POS and NEG data. It can be easily perceived that the calculated POS signal prevails on the NEG one for some time intervals, while the opposite condition can be noticed for other instants. In some other cases, rather balanced POS and NEG values can be also observed.

Most importantly, the POS and NEG trends were elaborated to derive the prevailing emotion for each input item, as shown in Figure 12b. For each time interval of 5 s, the average POS and NEG reactions were first calculated and associated with the 27 input pictures. Further, the maximum and minimum reaction peaks were also analyzed, as emphasized in the chart. This operation allowed us to derive a more precise analysis of human responses and nervous states, as a function of the imposed stimulus.

From the collected data, in particular, it was possible to assert that:

• The static virtual experience generally highlighted a strong “neutral” reaction for most of the participants. The average value of such a state was calculated in around 60% of measured AU signals. This finding can be easily justified by the static nature of a visual stimulus, and can be considered as an intrinsic limit of the overall experimental approach.
• A more detailed analysis of POS and NEG results, as in Figure 12, gave evidence, for most of the 27 pictures, of clear emotional trends calculated from the software analysis of minor facial micro-expressions.

The analysis of average reactions in Figure 12b highlighted in fact that:

• For most of the 27 items, the “average” measure of reactions can provide a reliable measure of experimental data and comfort trends.
• Differently, extreme reactions (quantified as max/min values for the fluctuation of average data) still suggest a rather variable subjective response for some of involved individuals.
• Rather few pictures can be associated with a mostly uniform reaction from the whole group of participants (i.e., with minimum max/min variations compared to the average response). This is the case for items #15 (Figure 6) and #27 (Figure 13). Furthermore, in both cases, the NEG reaction prevails on the POS.

The detailed analysis of any kind of discomfort revealed that:

• Average NEG peaks (>60% of participants) were found to prevail for POS states for items #8, #12, #15 (Figure 6) and #18 (Figure 13).
• In general, the minimum absolute NEG response was calculated for 46% of the group of participants. This suggests a constant presence of discomfort for the proposed stimuli, for several volunteers, and a weak prevalence of POS states.
• For some of the input items (as for example #12 and #15 in Figure 6), the average NEG outcome was found in line with preliminary expectations (i.e., possible discomfort due to hazard or risk of falling).
• For picture #2 in Figure 6, a marked NEG trend was also measured, even without a clear safety risk perception for end-users. This is also the case for item #3 in Figure 13. Both the outcomes could thus suggest some discomfort due to the use of structural glass in new solutions for constructions.
• A moderate average NEG reaction (>55% of participants) was also measured for items #2, #3, #4, #5, #9, #25 and #27. NEG and POS reactions were obtained for the residual items (see also Figure 14).

The analysis of major comfortable states, finally, can be summarized as follows:

• Absolute POS peaks were measured for item #1 (Figure 6), but also items #10 and #16 (Figure 15).
• For items #1 and #16, more in detail, this result could be justified by the presence of private/residential spaces and comfortable context conditions that do not involve direct contact with end-users. Regarding item #16, the POS peak could be justified by the “safe” presence of individuals on the floor mockup.
• Relevant POS reactions (>52% of participants) were also observed for items #20 (Figure 6) and #23 (Figure 15). Again, it is interesting to notice that the presence of end-users with “safe” behavior in the context of presented static stimuli can have positive effect on the emotional state of participants. This is in contrast with the structural typology and built environment (i.e., structures with risk of falling).

5.2. Analysis of Reactions by Context of Pictures

Based on the experimental outcomes as in Figure 12, a special care was paid for the analysis of POS or NEG reactions for pictures grouped by context/building configuration. Two major categories of items were considered, namely represented by:

• Group-A: pedestrian systems or load-bearing elements characterized by risk of falling for the end-users (items #4, #8, #10, #13, #15, #17, #18, #20, #22, #23, #27).
• Group-B: elements characterized by the presence of damage and/or under hazard (items #6, #9, #12, #25).

In case of Group-A, a NEG reaction was typically found to prevail for the majority of pictures (8 of 11 items). For three pictures only (#10, #20, #23), the POS reaction minimally prevailed on the NEG measures. Comparative results are summarized in Figure 16a. Such an outcome suggests that pedestrian glass systems evoke the highest discomfort and emotional state for the selected stimuli, compared to other glazing solutions in buildings. The effect can be justified by the fact that, compared to other building conditions, the human interaction of pedestrians with the glass structure becomes predominant.

Further, the few pictures from Group-A with a prevailing POS reaction were found associated with comfortable presence of individuals (i.e., item #10), and thus suggesting an appropriate safety level for the participants of the present experiment. Anyway, it is necessary to explore further this kind of scenario and extend the virtual experimental stage with field experiments, so as to capture the human reactions of active pedestrians and combine their emotional feelings with the structure vibrations and other mechanical parameters of primary interest for dynamic characterization.

Regarding the experimental results for Group-B of stimuli, NEG reactions were found to prevail on POS for most of the pictures (3 of 4 items), see Figure 16b. For the whole set, the presence of visual damage in glass proved to involve discomfort for the participants, thereby resulting in a rather good correlation of experimental predictions with the expectations of planned stimuli. The exception was represented by item #6 (Figure 6), characterized by a balanced POS/NEG average response.

5.3. Analysis of Reactions by Sub-Groups of Participants

The analysis of human reactions was also focused on the emotional state of sub-groups of participants, as, for example, obtained by gender or demographic sub-groups. As far as the POS comfort state is take into account, results according with Figure 17 and Figure 18 can be observed. From Figure 17, more in detail, the total average POS response was found to prevail for the group of female volunteers, for most of items. This is the case for pictures #15 and #18 of Group-A and the whole Group-B. For some other items of Figure 17a, a more balanced reaction was indeed measured.

Another aspect worth to discuss could be related to the emotional response for demographic sub-groups of volunteers, as in Figure 18. The distribution of volunteers onto age groups as in Figure 4 is not uniform and sufficiently extended to draw general conclusions. However, it is possible to notice that the trend of POS reactions modifies with age. Especially for Group-B with damage/hazard in Figure 18b, POS data decrease with age increase. Further investigations are hence needed in this direction.

6. Discussion of Results from the Dynamic Experiment

6.1. Analysis of Reactions by Context of Walks

For the dynamic setup, the post-processing stage of measured AUs was qualitatively carried out as in Section 4. However, major attention was focused on the analysis of absolute POS and NEG reaction peaks over the time of the experiment. This choice was suggested by the dynamic nature of the stimulus, with fast sequence of frames/scenarios and a different expected response compared to the static setup in Section 5.

Typical results can be seen in Figure 19, where the sequence of three different VR walks (40 s/each) is also emphasized. More precisely, the Wn labels are used to denote:

• W1: a walk outside the building, at the level of the roof/tope terrace, characterized by risk of falling (glass balustrades), but also by the presence of glass facades;
• W2: a walk inside the building (ground level), with a glass floor and a glass roof on the top, glazing walls and internal partitions;
• W3: a final walk inside the building (first story level), characterized by the risk of falling (glass balustrades), but also the floor in the below ground level (W2).

Key labels for selected frames in Figure 19 are defined as:

• s1, s3, s4 = external balustrades, characterized by risk of falling for end-users (as in Figure 7c,d);
• s2 = external facades / walls, without contact with end-users (Figure 20a,b);
• s5, s7 = roofs, without contact with end-users (Figure 7f);
• s6 = floors with risk of falling (Figure 20c);
• s8, s9 = indoor balustrades, with risk of falling (Figure 20d);
• s10 = indoor walls, with possible contact of end-users (Figure 20e).

Worth of interest in Figure 19 is that most of the POS or NEG peaks can be associated with well-defined s1–s10 scenarios in which glazing structural components have a primary role in the building context.

More precisely, for the W1 walk:

• NEG reactions were measured in s1, s3 and s4 frames, when the pedestrian impacts the glass balustrades.
• The frame labelled as s2 is associated with glass facades and walls as in Figure 18a,b.
• During the W2 walk:
• POS reaction peaks were found, especially for s5 and s7 stimuli, to be characterized by the presence of the glass roof (Figure 7f), as well as the facade walls and the indoor partition walls for the building entrance (Figure 20c).
• The pedestrian glass floor (s6 and Figure 20d) still evoked negative perceptions for most of the participants.

Finally, for the W3 walk, the experimental measurements were still found in line with the earlier scenarios, given that:

• Major NEG peaks were related to the presence of glass balustrades with risk of falling for the end-users (Figure 20d).
• The glazing roof in s10, otherwise, reflected POS reactions peaks as previously observed for the W2 frames.

6.2. Analysis of Reactions under Static or Dynamic Stimuli

In conclusion, a comparative analysis was carried out in terms of subject/context of input stimuli for the group of volunteers, so as to capture any difference in the prevailing reactions due to a static or a dynamic setup. In Figure 21, results are shown in the form of average percentage values of measured POS and NEG reactions for selected categories of built environments and scenarios.

These are defined as:

• E1 = outdoor balustrades (with risk of falling);
• E2 = indoor balustrades (with risk of falling);
• E3 = outdoor facades/walls;
• E4 = indoor facades/walls;
• E5 = floors/walkways/pedestrian systems (with risk of falling);
• E6 = roofs.

For groups E1 and E2, the results are presented in Figure 21 for the dynamic input only, given that no direct correlation could be found with static items. Note, as also previously discussed, the remarkable NEG reaction of participants, both for outdoor or indoor balustrades. This reaction is typically observed when the end-user approaches the glass balustrades during virtual walks, and thus perceives a possible risk of falling.

Regarding the emotional states and reactions of participants towards glass facades and walls (E3 and E4 in Figure 21), the proposed experimental setup did not capture relevant modifications or prevailing nervous states. This effect can be quantified in mostly balanced POS/NEG reactions for both static and dynamic VR stimuli. Finally, it is possible to notice in Figure 21 that the configurations labelled as E5 (i.e., pedestrian systems) were mostly associated with prevailing NEG reactions. As shown, the discomfort and NEG reactions further increase with a dynamic stimulus, compared to the static one. This is not the case for glass roofs (E6) that do not involve direct contact with end-users and are characterized in Figure 21 by a slightly prevailing POS reaction, both under static and dynamic stimuli. In conclusion, it is worth noting that no items presenting a hazard or damage in the glass were taken into account for the comparative analysis of static and dynamic emotional states, due to the lack of appropriate comparative VR stimuli. Besides, data in Figure 21 still suggest a more concise emotional reaction of participants based on dynamic rather than static stimuli. As a further extension of the present analysis, it could be thus useful to take into account accidental scenarios and built environments in which glass walls are expected to act as physical barriers for the protection of occupants.

7. Conclusions

The optimization of human comfort in the built environment is a target for several design fields, and depends on a multitude of aspects. For engineering applications, some of these aspects can be mathematically controlled and limited to performance indicators, including structural, energy and thermal issues. Differently, human reactions are also sensitive to a various aspects that relate to subjective feelings, in the same way in which the so-called “emotional architecture” aims at evoking emotional responses from building users.

In this paper, an original experimental analysis was presented in support of the definition of new design strategies and tools. The goal was to assess the potential of virtual experimental methods and the possible combination of quantitative measures of subjective feelings with engineering targets, as well as architectural concepts and technology solutions for the constructional fields. Special attention was focused on glass structures and components in buildings, due to the material features, innovative use in constructions and emotional evocation for end-users. Major advantage was taken from the use of virtual experimental techniques and facial expression analyses able to capture the micro-reactions of the group of participants (30 volunteers).

By designing a visual stimulus composed of both static and dynamic virtual items for building configurations characterized by a prevailing role of structural glass elements at different context levels (roofs, floors, balustrades, etc.), the attention was focused on the quantification of “positive” or “negative” emotions and comfort feelings for the participants. As shown, the analysis of experimental measurements proved that the use of facial micro-expression recognition tools in support of comfort analyses can represent and efficient approach for optimal design.

Moreover, the proposed experimental methodology highlighted that the design of structural glass elements for buildings can be severely affected by subjective feelings and nervous states of end-users. This is especially the case for structural components that involve direct interaction and contact from end-users (such as floor or balustrades which present a risk of falling), but also the condition of structural components with visual damage. Differently, mostly positive feelings were experimentally measured for the presence of structural glass in walls, facades and roofs. In this regard, the experimental investigation could be extended to different contexts, and to groups of volunteers characterized by different age demographics, geographic distributions or background skills.