Exploring an Emoji-Based Evaluation and Intervention Method for Psychological Safety in Ongoing Co-Creations

Abstract

Psychological safety is pivotal for co-creation to build an open environment where innovative ideas can flourish. Traditionally, psychological safety has been evaluated from a stable and long-term perspective by implementing psychological scales. Consequently, existing interventions often focus on steadily enhancing psychological safety, which is less suitable for dynamic short-term co-creation settings. The purpose of this study is to introduce the use of emojis as a novel and intuitive interaction during co-creations and assess their effectiveness in evaluating and influencing psychological safety. We performed two experiments with 140 participants in total to test emojis as evaluations and interventions, respectively. The participants watched videos and annotated them with emojis based on their perceptions of emotions. This process allowed us to explore the relationship between perceived emotions and psychological safety. In the next phase, we embedded emojis directly into the videos to observe whether the participants’ emotional perceptions—and, consequently, their psychological safety—could be influenced by visual cues. Our findings demonstrate that positive emojis are positively correlated with psychological safety and negative emojis are negatively correlated with psychological safety. We also revealed that negative emojis significantly decreased psychological safety scores, whereas positive emojis did not lead to a corresponding increase.

1. Introduction

Organizations in today’s complex and interconnected world strive to foster internal and external collaboration to address challenges, mitigate risks, and cultivate innovative ideas. However, individuals may hesitate to share their insights or express authentic thoughts because of the perceived interpersonal risks inherent in social interactions. This hesitation impedes creativity and collaboration. Psychological safety, which is defined as the shared belief that taking interpersonal risks does not lead to negative consequences [1], plays a crucial role in creating an environment that encourages open dialogue and co-creation.

Psychological safety fosters learning behaviors and innovation within teams, particularly under complex work conditions [2], such as healthcare settings [3]. Open communication and error reporting—essential in such environments—are facilitated when team members feel safe to express themselves [3,4]. This improves problem solving and collaboration, particularly under high-pressure conditions [3]. The ability to voice concerns and ideas without fear of being reprimanded enhances individual and team performance [2]. From a personal perspective, psychological safety not only enhances learning behaviors but also contributes to job satisfaction and employee engagement. In organizations that effectively cultivate such environments, team members report higher motivation and willingness to collaborate toward shared goals [5].

In ongoing co-creative activities, such as innovation meetings, workshops, and brainstorming sessions, the immediate atmosphere is crucial for achieving successful outcomes. In these collaborative events, psychological safety operates as a dynamic and short-term concept, in contrast to traditional psychological safety studies that focus on long-term team relationships. This requires evaluating psychological safety in real time and applying interventions that have an immediate impact, diverging from traditional approaches, such as team-building exercises and formal training sessions.

Traditionally, psychological safety has been assessed using psychological scales [3,6]. However, these scales have been designed for stable, long-term teams, reflecting an approach that gradually builds trust over time. Consequently, existing interventions often focus on steadily enhancing psychological safety, which is a less suitable approach for dynamic short-term co-creation settings. Current research lacks exploration of methods for evaluating and enhancing psychological safety in real-time collaboration.

To address this gap, we introduced the use of emojis as a more intuitive and immediate way to assess and influence psychological safety during co-creation activities. In particular, we explored the following two key research questions:

• RQ1: Can emojis act as reliable indicators of psychological safety levels, and what relationships exist between specific emojis and perceptions of psychological safety?
• RQ2: Can the use of emojis serve as an effective intervention for enhancing psychological safety?

To address these research questions, we employed a novel experimental approach using videos depicting various psychological safety levels to simulate ongoing co-creative activities. This study aimed to measure the relationship between emoji use and psychological safety. Initially, the participants watched videos and annotated them with emojis based on their perceptions of the emotions. This process allowed us to explore the relationship between perceived emotions and psychological safety. We further trained a machine learning model to assess whether the use of specific emojis can predict psychological safety levels. In the next phase, we embedded emojis directly into the videos to observe whether the participants’ emotional perceptions—and, consequently, their psychological safety—could be influenced by visual cues. This dual-stage approach provides insight into the role of emojis as evaluative and interventional tools for enhancing psychological safety in ongoing collaborative settings.

The remainder of this paper is organized as follows: Section 2 presents the extant literature on psychological safety evaluations and interventions, as well as on emojis as interaction methods and their uniqueness. Section 3 presents Experiment 1, which employed emojis as an evaluation method for psychological safety. Section 4 presents Experiment 2, wherein emojis were tested as an intervention method. Finally, the conclusions are presented in Section 5.

2. Literature Review

2.1. Psychological Safety Evaluations and Interventions

Psychological safety refers to people’s perceptions of the consequences of taking interpersonal risks [7]. In particular, it is a state of reduced interpersonal risk and endorses the belief that taking interpersonal risks is safe in the workplace [2]. Although the definition may be slightly different in different research contexts, most have been extended based on the original definition [1].

2.1.1. Evaluations

In previous empirical research, psychological safety was measured predominantly through surveys [3,6]. In particular, the scale developed by Edmondson [1] and some of its versions are widely used because they evaluate psychological safety in a way that is conceptually consistent with its definition [8]. Consistently using a survey methodology that has been well validated in evaluating psychological safety helps the field develop quickly, as it formulates a standard that studies can measure against, and meta-analyses can be performed to obtain more insightful findings and research directions. However, relying solely on survey tools can be time-consuming for participants and has limitations due to self-report bias [9]; thus, scholars continue to develop complementary methods, such as observational measures [10]. Generally, psychological safety measurements can be divided into surveys and observation methods [3]. Thus far, most of the validated surveys have been adjusted from the original version by Edmondson [1], with seven questions assessing people’s attitudes toward making mistakes, bringing up problems, being different, taking risks, asking for help, feelings of safety, and self-value.

However, O’Donovan and McAuliffe [3] summarized observed behaviors and developed an observation method that complements the survey method. These behaviors are categorized into seven groups: voice behaviors (e.g., asking questions), defensive voice behaviors (e.g., showing aggression), silence behaviors (e.g., facial expression or body language indicating fear), supportive behaviors (e.g., sharing knowledge), unsupportive behaviors (e.g., reacting coldly), learning- or improvement-oriented behaviors (e.g., asking for help), and familiarity behaviors (e.g., non-work matters) [3]. The behaviors listed as observation components of psychological safety measures include comprehensive aspects of psychological safety; however, as tangible expressions of psychological safety, they overlap with the main manifestations of psychological safety discussed in the previous section.

Despite the maturity of the survey method and the development of observation methods in the field, this study encountered challenges. Apart from self-report bias, answering surveys can be time-consuming, which makes it difficult to increase the frequency of data collection; therefore, research on temporal changes has its limitations. In particular, we may measure psychological safety on a weekly or even daily basis, but answering questionnaires all the time can be irksome for respondents, and measuring changes in minutes or hours during a meeting is impossible. Moreover, observation measures require observers, either within or outside the group, which requires extra energy or labor. Ideally, the manifestation of psychological safety should become clearer and more evident, so that real-time automatic evaluation can be achieved. This study explores tools that can make psychological states easier to understand and that can be used to develop an ideal measurement.

2.1.2. Interventions

The purpose of understanding psychological safety is to improve it; thus, several studies have tried various implementation approaches. Some studies have considered psychological safety as a skill that can be mastered through training.

The most prevalent method involves psychological safety training. Some researchers have used simulations such as role-play as training methods. Pian-Smith et al. [11] provided participants with an opportunity to raise issues with experts employing simulation-based situations and techniques. They held educational sessions in which they provided the participants with tools to speak up. Their findings indicated an improvement in their speaking-up habits. Similarly, the “two challenge rule” and a role-playing exercise were among the topics included in a workshop hosted by Raemer et al. [12], in which participation in an experimental situation comprised dialogues with experts. However, the authors did not discover any statistically significant alterations in behaviors related to psychological safety. Thomas et al. [13] used simulations to address assertiveness, information sharing, speaking up, voice inquiries, and other team behaviors. The participants engaged in a simulated resuscitation, wherein they applied the skills they had learned. The intervention group exhibited more instances of questioning, exchanging information, and asserting themselves than the control group. The common idea of these interventions is to provide tools and behavioral models to participants, and they hypothesize that with training on using these tools, participants could become familiar with how to behave with psychological safety.

In addition to the simulation, a case study was conducted in a previous study on interventions for psychological safety. O’Connor et al. [14] presented recordings of attending physicians addressing difficult communication and assertiveness situations they encountered as interns, with a brief introduction to human factors and errors. While their intervention did not significantly impact the participants’ views on speaking up about stress or senior staff, the experimental group had noticeably more favorable sentiments toward doing so than the control group. To eliminate any implicit penalties for nurses speaking out, Sayre et al. [15] used recordings of senior staff conveying their expectations and encouragement of nurses to speak up. Following the films, the participants devised action plans and identified obstacles to speaking up. The survey results and individual lists significantly improved in the intervention group.

Considering the mixed results of different attempts at interventions, how to implement psychological safety remains a challenge in the field [2]. Notably, if we want to intervene during ongoing co-creation, an evaluation system that can capture temporal changes is required. Traditional evaluation and intervention methods no longer serve this purpose. Therefore, we introduced emojis as an additional form of interaction in co-creation to evaluate and intervene in psychological safety.

2.2. The Development of Emojis

The use of emojis has become prevalent in contemporary life since the rise of computer-mediated communication (CMC), especially in social media [16] and networking services such as Twitter and Instagram [17].

Most emojis express one emotion or more [18]; therefore, emotional lexicons must be developed to analyze these emotions. Novak et al. [19] separated emojis into positive, negative, and neutral categories by artificially annotating them and discovered that most of the emojis were positive. Kutsuzawa et al. [20] conducted an online survey to study the link between valence and arousal axes and the valence of emojis with respect to human emotional states. In total, 74 facial emojis were classified into six categories: strongly negative sentiment, moderately negative sentiment, neutral sentiment with a negative bias, neutral sentiment with a positive bias, moderately positive sentiment, and strongly positive sentiment [20]. Emojis have rich emotional meaning that correlates with psychological states and emotional responses. This is the foundation for developing psychological measurements using emojis and monitoring emotions in investigations, such as marketing. Emojis show great performance in developing new psychological measurement tools based on their affective nature and the long history of using emojis as complementary tools in investigations that are relevant to emotions.

Emoji assessment is a popular technique used by marketers to evaluate user sentiment [18]. This can be because emojis and emoticons are regarded as quick and intuitive methods of communicating emotions [21], which allow consumers to evaluate their experience with little cognitive load and time cost. Moreover, consumers’ capacity to describe and recognize stimuli using emojis is unaffected by gender, age, or frequency of use [22], so emojis can be used by a wide group of people with credibility. As a result of their convenience and reliability, using emojis has become a well-accepted way to reflect consumers’ emotions, and, in particular, to track the feelings consumers have toward particular goods, services, and brands [23]. Successful experiences in evaluating consumer attitudes and emotional states can be extended to other fields of study.

Emojis are suitable anchoring scales for evaluating emotional preferences of professional interest because of their affective character [24]. For example, to determine the five emojis that best reflect a bipolar continuum from strongly liked to strongly disliked, Phan et al. [24] first performed a content analysis. Their study proved that emoji anchors have psychometric properties identical to those of lexical anchors, bridging emojis, and psychological states. Furthermore, the differences between using emojis and Likert-type response categories were examined [25]. The Psychological Well-Being Scale, featuring three-, five-, and seven-point Likert-type and emoji response categories, was used to test for differences. The results indicated no significant differences across the exploratory and confirmatory factor analyses, as well as the reliability analyses [25].

Considering all the previous theories and practices of using emojis, we regard emojis as perfect social cues that can convey emotions in CMC and evaluate psychological states. The relationship between emojis and psychological safety was explored in this study to develop an emoji-based evaluation and intervention method for psychological safety in ongoing co-creation.

3. Experiment 1: Evaluating Psychological Safety with Emojis

There is a compelling need for more seamless and intuitive measurement techniques to rectify the discordance between traditional evaluation methods and the dynamic, real-time nature of psychological safety in collaborative settings, and increase the frequency of psychological safety evaluations. Traditional approaches involving the completion of psychological scales are valuable, but can be cumbersome and disruptive within the context of meetings and workshops. The purpose of this experiment was to find the relationships between emojis, emotions, and psychological safety so that we could train a model that can predict psychological safety based on emojis labelled by users. Ultimately, the overarching objective is to enable the prediction of psychological safety in an effortless and organic manner, which is conducive to real-time applications.

3.1. Materials and Methods

3.1.1. Materials

Videos. Psychological safety is divided into four levels. As team development is a key factor that may influence psychological safety and similar team behaviors can be expected from similar levels of psychological safety and team development, we combined team development theory with psychological safety to set different levels [26].

The group development progressed through four stages [26], each linked to psychological safety levels. In the dependency and inclusion stages, members rely heavily on their leader to avoid conflict and hold back ideas, reflecting low psychological safety. During counterdependence and fighting, disagreements arise as the team builds trust, marking a developing level at which members express differing views with some tension. In the trust and structure stage, roles and goals are aligned, fostering psychological safety through open discussions and knowledge sharing. Finally, the productivity and effectiveness stage enables members to focus on tasks with confidence and achieve high psychological safety through mutual trust and personal values. We prepared 16 videos, four for each level (Figure 1).

Emojis. According to current research [20], emojis can be classified into six clusters expressing strongly positive sentiment, moderately positive sentiment, neutral sentiment with a positive bias, neutral sentiment with a negative bias, moderately negative sentiment, and strongly negative sentiments.

We selected one emoji from each cluster as a representation, because the entire list of 74 emojis was too large for an online questionnaire. To decide which emojis to choose from, emotions that are closely linked to psychological safety should be considered. However, few studies have explored the relationship between emotions and psychological safety. The only research we found discussed how organizational care for employees’ well-being and work influences psychological safety, finding that emotions of pride, empathy, and support may positively influence psychological safety, and emotions of anxiety, stress, anger, unfairness, and vulnerability may negatively influence psychological safety [27].

Based on previous research, we decided to focus on one emotion in each cluster and selected the emojis that best corresponded to that emotion. From cluster 1 (strong negative sentiment) to cluster 6 (strong positive sentiment), the emojis were chosen with the purpose of representing angry 😠 (angry face), nervous 😟 (worried face), contempt 😕 (confused face), relaxed 😌 (relieved face), happy 😉 (winking face), and supportive 🥰 (smiling face with hearts) emotions, respectively.

Notably, participants’ interpretations of emojis differed based on their personal experiences. However, in our experiment, the precise meaning of each emoji was not the primary focus. Prior research by Kutsuzawa et al. [20] established the positive valence of emojis. Our objective was to explore the relationship between the positivity of these emojis and participants’ psychological safety scores.

3.1.2. Participants

The target group for our questionnaire was office workers aged 30–55 years, living in the Tokyo area. We invited 100 office workers who had been employed for several years, and so had the richest experience of different levels of psychological safety, because they needed to engage in teamwork and deal with interpersonal risks constantly. In addition, we wanted to control for geographical differences as much as possible in this first attempt. Therefore, we set the limitation that participants must live in the Tokyo area.

3.1.3. Methods

Participants were invited to complete an online questionnaire. First, a page with instructions and demonstrations was shown to all the participants to clarify the contents of the questionnaire and obtain their informed consent. Only participants who clicked on the “Agree” button at the end of the page were allowed to move on to the next step. We then used a simple screening question to check whether the participants read the questions and answers carefully. Following this, the participants who answered the screening question correctly were led to the main questionnaire. They answered basic personal questions regarding their age, sex, and personality. Personality was evaluated using the 10-item Big Five Scale [28]. Next, the participants were asked to watch four random videos presenting four different levels of psychological safety in random order. While watching each video, participants were instructed to label the emotions of the characters in the video every 30 s (Figure 2). When the participants finished watching one video, they answered the full psychological safety test scale with seven questions. We implemented a seven-item psychological safety scale based on the original measure developed by Edmondson [1], using a Japanese version adapted for this study. The items and their corresponding English versions are shown in

Table 1. Psychological safety measurement items.

Items Used in Our Experiment	Original Items Developed by Edmondson
このチーム内でミスをすると、多くの場合非難されてしまうだろう	If you make a mistake on this team, it is often held against you.
このチームメンバーの中に、異質な個性を理由に他者を拒絶する人がいるだろう	People on this team sometimes reject others for being different.
このチームメンバーと仕事をする際に、自分のスキルや能力が尊重され仕事に活かせていると感じられるだろう	Working with members of this team, my unique skills and talents are valued and utilized.
このチーム内では難しい問題・課題を互いに指摘し合うことができるだろう	Members of this team are able to bring up problems and tough issues.
このチーム内の全員が、他者を意図的に陥れるような行動をしないだろう	No one on this team would deliberately act in a way that undermines my efforts.
このチーム内でリスクの高い発言や行動をとっても安全だと感じられるだろう	It is safe to take a risk on this team.
このチーム内でほかのメンバーに助けを求めることが難しいだろう	It is difficult to ask other members of this team for help.

. Participants were asked to select from “strongly disagree” to “strongly agree”, and each item was scored from 1 to 7 accordingly. The psychological safety scores were calculated by summing all item scores, resulting in a total score ranging from 7 to 49. The participants then watched the remaining three videos, following the same procedures.

Finally, at the end of the questionnaire, participants were asked to match emojis and emotions in case they had very different understandings of the emojis. The order in which the levels were viewed was randomized for each participant, as was the specific video viewed at each level.

3.1.4. Data Analysis

This experiment followed a within-subjects design, and the data had a 2-level structure, namely personal attribute level and reaction level. All attribute-level variables were grouped by participant and modeled as predictors of dependent variables. In addition, we had six emojis as the main independent variable, which played the role of an explanatory mechanism for the final psychological safety scores.

We used hierarchical linear modeling (HLM), also known as multilevel modeling, random coefficient modelling, or mixed modeling, to handle data at multiple levels. The data we collected are nested within persons, which means we assume that personalities, sex, and age group may influence emoji selections and perceived emotions, and further influence psychological safety under certain circumstances. Thus, this research analysis emulated the work by Ozaki et al. [29] and modeled variables into within-person variables (emoji selections, psychological safety), which means the variables that may differ within a person according to situations, and between-person variables (personality, sex, age), which means that the variables remain stable in all given situations for an individual.

We structured a 2-level HLM, as illustrated in Figure 3. Specifically, the model includes a personal attribute level (between-person variables) and a reaction level (within-person variables). The data were grouped by individuals to control for individual differences. The regression analysis was conducted using Python8, leveraging the StatsModels library. The emojis were divided into two groups—negative and positive—to avoid multicollinearity. We used a proportion of emojis; therefore, when the participants tended to select more positive emojis, they tended to select fewer negative emojis. In other words, negative and positive emojis were highly correlated and encountered multicollinearity, which implies that the regression model could have been impacted. To explore the impact of each emoji on the psychological safety score, we performed an HLM regression separately for positive and negative emojis.

We implemented a machine learning method, XGboost [30], to explore the feasibility of using machine learning approaches to predict psychological safety based on emojis and personal attributes. The dataset was randomly split into training (80%) and testing (20%) subsets using the train_test_split method with a fixed random seed for reproducibility. The performance of the model was evaluated using mean squared error (MSE), root mean squared error (RMSE), and mean absolute error (MAE). Five-fold cross-validation was conducted to assess model generalizability. Additionally, feature importance scores were extracted as part of the model evaluation to examine the relative contributions of predictors.

3.2. Results

3.2.1. Positive Emojis and Psychological Safety Score

A hierarchical linear model (HLM) was employed to analyze the effects of individual-level variables (age, sex, and the Big Five personality traits) and positive emoji usage on psychological safety scores. Prior to conducting hierarchical linear modeling (HLM), we assessed the key assumptions. The data exhibited a nested structure, with participants nested within teams. To justify the use of HLM, we calculated the intraclass correlation coefficient (ICC) for the dependent variable, which was 0.409. This indicates that approximately 41% of the variance in the psychological safety scores resided at the group level, supporting the use of multilevel modeling. The residuals were visually inspected for normality using Q-Q plots, which showed that they followed an approximately normal distribution. The hierarchical structure of the data allows the model to account for between-group variance through random effects. Thus, traditional tests of homoscedasticity were not conducted.

The model included 400 observations nested within 100 individuals, each serving as their own group with a mean group size of 4.0. The analysis utilized the restricted maximum likelihood (REML) method, and the model converged successfully, with a log-likelihood of −1229.4200 and a scale parameter of 20.4333. The summary of the model is outlined in

Table 2. Summary of HLM for positive emojis.

Predictor	Coefficient	Stdandard Error	z	p > |z|
Intercept ***	27.308	3.8	7.187	0
Gender (T.2)	1.282	0.944	1.358	0.174
Extraversion	−0.093	0.167	−0.559	0.576
Agreeableness *	0.485	0.225	2.155	0.031
Conscientiousness	−0.027	0.195	−0.14	0.889
Neuroticism	−0.355	0.208	−1.704	0.088
Openness	−0.029	0.208	−0.14	0.888
Age *	0.141	0.069	2.043	0.041
emoji4 ***	8.423	1.09	7.729	0
emoji5 ***	10.62	2.664	3.987	0
emoji6 **	23.34	7.501	3.112	0.002

Note: Asterisks indicate statistical significance: p < 0.05 (*), p < 0.01 (**), p < 0.001 (***).

. After fitting the HLM model, multicollinearity among continuous predictors was assessed by calculating the Variance Inflation Factors (VIFs). All the VIF values were below 5, ranging from 1.06 to 1.73, indicating no serious multicollinearity issues.

The fixed effects model revealed several significant predictors. The intercept was significant (B = 27.308, p < 0.001), indicating a baseline psychological safety score of 27.308 when all the predictors were at their reference levels. Among the individual-level variables, agreeableness had a significant positive effect on the psychological safety scores (B = 0.485, p = 0.031), suggesting that higher agreeableness is associated with higher psychological safety scores. Age was also a significant predictor, with older individuals having higher psychological safety scores (B = 0.141, p = 0.041). This indicates that as individuals age, their psychological safety scores tend to increase.

Regarding emoji usage, all the positive emojis showed significant positive effects on psychological safety scores. The usage of emoji4 had a significant positive effect (B = 8.423, p < 0.001), indicating that it was associated with a substantial increase in psychological safety scores. Similarly, the usage of emoji5 was a significant predictor (B = 10.620, p < 0.001), suggesting that emojis have a strong positive association with psychological safety scores. The usage of emoji6 also showed a significant positive effect (B = 23.340, p = 0.002), indicating a notable increase in psychological safety scores. Specifically, the more positive the emoji, the higher the coefficient, reflecting a greater impact on psychological safety scores. For instance, emoji6, from strong positive sentiments [20], had the highest coefficient (B = 23.340), indicating its strong association with higher psychological safety scores. Coefficient (B) represents the change in psychological safety scores for a one-unit change in the predictor variable, holding all other variables constant. Thus, the higher coefficients for positive emojis suggest that emojis are associated with a larger increase in psychological safety scores.

3.2.2. Negative Emojis and Psychological Safety Score

The same method was used to analyze the negative emojis. The ICC for the dependent variable was calculated and found to be 0.476, indicating that approximately 48% of the variance in the psychological safety scores was attributable to group-level variance. This result supports the use of multilevel modeling for this analysis. The residuals were checked for normality using Q-Q. No significant violations were observed. The model includes 400 observations nested within 100 individuals, with each individual serving as its own. The analysis employed the REML method and successfully converged, resulting in a log-likelihood of −1210.1184 and a scale parameter of 17.5883.

Table 3. Summary of HLM for negative emojis.

Predictor	Coefficient	Stdandard Error	z	p > |z|
Intercept ***	34.211	3.891	8.793	0
Gender (T.2)	1.091	0.973	1.122	0.262
Extraversion	−0.004	0.171	−0.024	0.981
Agreeableness *	0.502	0.232	2.164	0.03
Conscientiousness	−0.057	0.201	−0.282	−0.282
Neuroticism	−0.312	0.214	−1.458	0.145
Openness	0.072	0.211	0.342	0.732
Age *	0.148	0.071	2.084	0.037
emoji1 ***	−36.398	5.415	−6.722	0
emoji2 ***	−15.609	2.4	−6.504	0
emoji3 ***	−6.005	1.042	−5.761	0

Note: Asterisks indicate statistical significance: p < 0.05 (*), p < 0.001 (***).

presents the results of the study. All the VIF values were below the threshold of 5, ranging from 1.03 to 1.72, indicating no concerns with multicollinearity among the predictors.

Similarly, among the individual-level variables, agreeableness had a significant positive effect on the psychological safety scores (B = 0.502, p = 0.030), indicating that higher agreeableness was correlated with higher psychological safety scores. Age was also a significant predictor, with older individuals exhibiting higher psychological safety scores (B = 0.148, p = 0.037), suggesting an increase in psychological safety scores with age.

Negative emoji use had several significant negative associations with psychological safety. The use of emoji1 had a pronounced negative effect (B = −36.398, p < 0.001), indicating a substantial decrease in psychological safety scores associated with this emoji. Similarly, the usages of emoji2 (B = −15.609, p < 0.001) and emoji3 (B = −6.005, p < 0.001) were significant predictors, suggesting that these emojis were also negatively associated with psychological safety scores.

These findings indicate that specific emojis are strongly associated with lower psychological safety scores. In particular, the more negative the emoji, the greater the negative coefficient, reflecting a more significant decrease in psychological safety scores. For example, emoji1 belongs to a strong negative sentiment cluster [20] and has the largest negative coefficient (B = −36.398), highlighting a larger change in psychological safety scores for a one-unit change in the predictor variable, holding all other variables constant.

3.2.3. XGboost Regression Model

The XGBoost model was evaluated on the training dataset for predicting the psychological safety scores (y) from all the Big Five factors: age, sex, six emojis, and slope. The performance metrics of the model indicated its effectiveness in this context. The MSE of the model is 25.996, which reflects the average squared difference between the predicted and actual values. A lower MSE value generally signifies a better fit of the model to the data. The RMSE of the model is 5.099. This metric, which is the square root of the MSE, provides an indication of the average magnitude of errors in the same units as the target variable, which in this case is the psychological safety score. Considering that the psychological safety scores ranged from 13 to 49, an RMSE of 5.099 suggests that, on average, the model’s predictions deviated from the actual scores by approximately five points. To further assess the model’s generalizability, five-fold cross-validation was conducted. The RMSE across folds ranged from 4.72 to 6.42, with a mean RMSE of 5.63 and a standard deviation of 0.63, indicating stable predictive performance across different data subsets.

Furthermore, the MAE of the model is 3.672, which represents the average absolute difference between the predicted and actual values. In consideration of the range of psychological safety scores, an MAE of 3.672 indicates that the model’s predictions are, on average, approximately 3.7 points away from the actual scores. This provides a tangible measure of prediction accuracy in practical terms.

The feature importance analysis revealed that emoji-related features generally contributed more to the model’s predictions, while both personal attributes and emoji features played meaningful roles. Among the personal attributes, neuroticism (0.103) and openness (0.103) showed relatively high importance, while extraversion (0.031) and gender (0.018) were lower in importance. For emoji-related features, emoji2 (0.156) and emoji6 (0.100) were among the most influential. Notably, our discussion primarily centers on the HLM results, which are more suitable for theory building and inferential interpretation in this study context.

3.3. Discussion

3.3.1. Personal Attributes

Our results demonstrated that agreeableness is the most influential personality trait in predicting psychological safety. This aligns with the existing literature suggesting that agreeable individuals, who are typically more cooperative, empathetic, and harmonious in social interactions, are more likely to foster and perceive a safe psychological environment [1,30]. These findings corroborate those of Newman et al. [6] and Frazier et al. [8], which also highlighted the role of agreeableness in promoting psychological safety within teams. This insight can be leveraged to design interfaces and collaborative tools that promote and reward agreeable behaviors. For instance, features that encourage positive feedback, empathetic communication, and cooperative task management can be integrated into collaborative platforms to enhance users’ overall sense of psychological safety. This can be particularly beneficial in virtual teams, online learning environments, and social media platforms, where fostering a supportive and safe atmosphere is crucial for effective interaction and engagement.

Age was also identified as a significant predictor of psychological safety, with older individuals reporting higher psychological safety scores. This finding contrasts with previous studies, such as that of Edmondson and Lei [7], which suggested that psychological safety perceptions might not vary significantly across different age groups. However, this supports the notion proposed by Li and Cropanzano [31] that older employees might feel more secure because of their accumulated experience and greater emotional regulation. This highlights the importance of age-related differences when designing psychological safety evaluations and intervention methods. Forms of age-specific customization, such as providing more intuitive and supportive guidance for younger users or offering opportunities for older users to share their experiences and mentors, can help create a psychologically safe environment for users of all ages.

3.3.2. The Perceptions of Others’ Emotions

In this study, we focused on the participants’ perceptions and interpretations of others’ emotions rather than the emotions themselves. Regardless of their true emotions, participants may perceive others’ emotions in various ways, which, in turn, leads to different levels of psychological safety. Our findings reveal that the more positive the emotions a person perceives in others, the greater the psychological safety they feel within the team. Specifically, when individuals perceive more positive emotions from fellow team members, they tend to feel a higher level of psychological safety. Conversely, when individuals perceive more negative emotions from team members, they tend to feel a lower level of psychological safety. Moreover, our findings indicate that the intensity of perceived emotions matters along with the frequency of their occurrence. The greater the negativity or positivity (higher arousal) conveyed by emojis, the stronger the perceived emotions among participants and the greater the impact on the final psychological safety scores.

The findings of this study emphasize the critical role of perceived emotions in influencing psychological safety within teams, aligning with and extending prior research in this area. Previous studies have highlighted the importance of emotional expression in shaping group dynamics and psychological safety. Edmondson [1] found that teams with members who openly and positively expressed emotions experienced greater psychological safety. However, our study shifted the focus from actual emotional expressions to perceptions of these emotions, revealing that the perceived emotional climate is equally, if not more, significant.

Our results indicate that individuals who perceive more positive emotions from team members tend to feel more psychologically safe. This is consistent with Newman et al. [6], who noted that positive emotional environments foster greater trust and a sense of security among team members. Perceived positivity likely reinforces an individual’s belief that the team is supportive and nonthreatening, thereby enhancing psychological safety. This finding suggests that, in team-based digital environments, designing systems that facilitate the expression and perception of positive emotions could be beneficial.

Conversely, our study found that perceptions of negative emotions were correlated with lower psychological safety. This supports the findings of Kahn [32], which demonstrated that a negative emotional climate can lead to fear and reluctance to engage, ultimately reducing psychological safety. The current study extends this understanding by showing that it is not just the presence of negative emotions, but the perception of these emotions by team members that can diminish psychological safety.

These insights have significant implications for the design of collaborative tools and interfaces that foster psychological safety. Tools that help team members express positive emotions and manage their perceptions of negative emotions can enhance psychological safety. For example, features that promote positive feedback, highlight team success, and provide support mechanisms for managing conflicts can create a more positive emotional climate. Positive emojis can be used to enhance the psychological safety of teams.

3.3.3. Emojis as Social Cues to Convey Emotions

In our study, emojis served as the primary social cues to convey emotions, thus making the perceived emotions more tangible. This tangibility is crucial for designing methods for evaluating and enhancing psychological safety based on emotional cues. Emotions are inherently fluid and dynamic, and often shift rapidly in response to varying stimuli. These emotional shifts are challenging to grasp fully in face-to-face interactions and become even more challenging to interpret and share in virtual environments [33].

The use of emojis as social cues has unique advantages. Emojis offer a standardized, albeit simplified, way of expressing and perceiving emotions through digital interactions. They help bridge the gap created by the lack of nonverbal cues in virtual communication, allowing users to express their feelings more explicitly and perceive these emotions more clearly. This is particularly important in team-based digital environments, where maintaining psychological safety is critical for effective collaboration.

The use of emojis as social cues opens new avenues for psychological safety interventions. For example, emoji usage patterns can serve as early indicators of changes in team dynamics and psychological safety. Interventions can be designed to promote positive emotional expressions and manage negative expressions, thereby fostering a supportive and safe virtual environment.

4. Experiment 2: Intervening in Psychological Safety with Emojis

4.1. Materials and Methods

4.1.1. Materials

We selected 4 of the 16 videos used in Experiment 1. Our goal was to reduce the ambiguity of implicit emotional signals in people’s behavior and encourage positive interpretations of emotional expressions. To achieve this, we chose videos with the largest standard deviations (SDs), indicating that participants had diverse opinions regarding these videos owing to their inherent ambiguity. Two videos were low-level, and two videos were from the developing and developed levels.

Selected videos were edited in two versions. In version A, more positive emojis were used as predictors of a higher level of psychological safety. We used the answers from Experiment 1 as a basis. To achieve this, we first assigned each emoji a value from one to six, with higher numbers representing more positive emotions. We then calculated the average positivity of the emojis chosen by the participants every 30 s during Experiment 1. Following this, we added two to the average positivity and determined the corresponding emojis. Conversely, in Version B, we used more negative emojis by subtracting two from the average positivity. Emojis were placed above the head of each character in the videos (Figure 4).

4.1.2. Participants

We invited 40 new participants to participate in this experiment, as in Experiment 1. The group consisted of 20 male and 20 female office workers aged 30–55 years residing in the Tokyo area. To avoid any potential bias from repeated exposure, the participants from Experiment 1 were excluded because watching the same videos multiple times or participating in similar experiments could have affected the results. To maintain consistent conditions, we replicated the procedures of Experiment 1, with each participant watching four videos. The order in which the participants watched these videos and the versions they saw were randomized.

4.1.3. Procedure

Participants were invited to join an online questionnaire system similar to that in Experiment 1 and underwent the same process, including instructions and consent, screening questions, personal information, watching videos, choosing emojis, and a psychological safety test scale. When participants watched the videos, they were explicitly informed that the emojis displayed in the videos were intended to reflect the true emotional states of the characters, and that these emotions were not to be interpreted by the participants themselves. The participants watched videos with emojis and chose the same emojis every 30 s, as shown in the videos.

4.1.4. Data Analysis

This experiment followed a between-subject design. The participants were asked to watch different versions of the same video, including the original version without emojis, version A with more positive emojis, and version B with more negative emojis.

As the purpose of the analysis was to compare the three different conditions, we first implemented an analysis of variance (ANOVA), because we compared the psychological safety scores across more than two groups—specifically, the original version and two modified versions (A and B) of the videos. Prior to the analysis, the normality of residuals was evaluated using a Q-Q plot, which did not reveal serious deviations from normality. Homogeneity of variances across groups was tested using Levene’s test and confirmed (p = 0.110).

To further clarify which pairs of video versions (Original, Version A, and Version B) showed significant differences in psychological safety scores, Tukey’s post hoc test was employed following a two-way ANOVA. Tukey’s test is particularly suitable in this context because it is designed to compare all possible pairs of group means, while controlling for the overall Type I error rate. This method is robust and is widely used when dealing with multiple comparisons because it adjusts the significance threshold to account for the number of comparisons made, thus reducing the likelihood of false positives.

4.2. Results

Table 4. Mean and SD of psychological safety for the different versions of the 4 videos.

	Original Version		Version A		Version B
	Mean	SD	Mean	SD	Mean	SD
Video 1	29.91	7.44	31.19	10.44	26.67	8.10
Video 2	35.73	7.14	36.56	6.78	32.65	6.93
Video 3	35.21	6.66	35.55	7.42	29.79	5.75
Video 4	31.06	7.30	33.44	6.87	30.71	5.81

presents the means and standard deviations (SDs) of psychological safety for the different versions of the four videos. Figure 5 shows the average psychological safety scores across different versions of the four videos.

As shown in the table and figure, Version A consistently had the highest scores among the three versions for all four videos, whereas Version B consistently had the lowest.

A two-way ANOVA was conducted to examine the effects of the video versions (Original, Version A, and Version B) and specific videos (videos 1, 2, 3, and 4) on the psychological safety scores (

Table 5. Results of the two-way ANOVA test.

	Sum_sq	df	F	PR (>F)
C(version)	753.34	2	8.06	0.00041
C(video)	1157.89	3	8.25	0.00003
Residual	11,409.64	224

). The analysis revealed a significant main effect of the video version, F (2, 244) = 8.06, p < 0.001, indicating that the different versions had a significant impact on psychological safety scores. Additionally, there was a significant main effect of the specific video, F (3, 244) = 8.25, p < 0.001, suggesting that psychological safety scores varied significantly depending on the video viewed. The residual sum of squares was 11,409.64 with 244 degrees of freedom, capturing the unexplained variability in psychological safety scores after accounting for the effects of the video versions and specific videos. These results suggest that both the versions of the videos and the specific videos themselves contributed significantly to variations in psychological safety scores.

Following the significant effects observed in the two-way ANOVA, Tukey’s Honest Significant Difference test was conducted to perform pairwise comparisons between the video versions (Original, Version A, and Version B) (

Table 6. Results of the Tukey’s Honest Significant Difference test.

Group 1	Group 2	Meandiff	p-adj	Lower	Upper	Reject
original	versiona	0.7209	0.791	−1.8832	3.3249	False
original	versionb	−3.171	0.0103	−5.7209	−0.6211	True
versiona	versionb	−3.8919	0.0022	−6.5871	−1.1966	True

). A comparison between the Original and Version A did not reveal a statistically significant difference in the psychological safety scores (mean difference = 0.7209, p > 0.05). The 95% confidence interval ranged from −1.8832 to 3.3249, indicating that the psychological safety scores for the two versions were comparable.

In contrast, a significant difference was found between the Original version and Version B (mean difference = −3.171, p < 0.01), with the 95% confidence interval extending from −5.7209 to −0.6211, suggesting that Version B was associated with significantly lower psychological safety scores than the original version. Additionally, the comparison between Version A and Version B also yielded a significant difference (mean difference = −3.8919, p < 0.01), with a 95% confidence interval from −6.5871 to −1.1966, further indicating that Version B resulted in significantly lower psychological safety scores compared to Version A.

4.3. Discussion

The results of this study reveal a compelling asymmetry in how visual cues influence psychological safety. While the inclusion of negative emojis significantly lowered the psychological safety scores, the addition of positive emojis did not produce a corresponding increase.

The negative version, which included more negative emojis, significantly lowered the psychological safety scores compared to both the original and the positive versions. This suggests that negative visual cues can have a powerful impact on viewer perceptions of psychological safety, likely by reinforcing negative emotions or creating a more hostile or critical atmosphere. This finding can be understood through the lens of negativity bias, a well-documented psychological phenomenon in which negative information has a greater impact on individuals’ perceptions and emotions than positive information [34]. Negative stimuli typically have a stronger impact on cognition and emotion than positive stimuli [35]. As a result, negative emojis may intensify viewers’ perceptions of a critical or unsupportive environment, thereby undermining their sense of psychological safety, which is critical for fostering open communication and well-being in group settings [1]. In contexts in which psychological safety is a concern, negative cues may be interpreted as signs of disapproval, criticism, or conflict, causing viewers to feel less safe and more cautious. This heightened sensitivity to negative social signals can undermine feelings of inclusion and support, which are the key components of psychological safety.

The significant decrease in psychological safety associated with the negative version underscores the potential risks of using negative visual cues in collaboration. Decreasing psychological safety can be easier than improving psychological safety using visual cues such as emojis. When implementing emojis in a co-creative space, the presence of negative symbols or cues that could inadvertently diminish the sense of security among participants should be avoided.

However, despite the intention to enhance psychological safety through the inclusion of more positive emojis, the results indicated that the positive version did not lead to a significant improvement in psychological safety scores compared with the original version. This finding suggests that simply adding positive cues in the form of emojis may not be sufficient to create a perceptible increase in viewers’ psychological safety. The effects of positive cues were weaker than those of negative emojis, because negative stimuli are often evaluated more strongly than equally intense positive stimuli [36]. This is also due to negativity bias in attention allocation [37].

In addition, the context in which these visual cues were presented may have influenced their effectiveness. The unexpected presence of negative symbols likely creates a stark contrast with viewers’ expectations, making them more impactful. However, the positive cues may not have been as surprising or may have been perceived as part of a normal, non-threatening environment, reducing their overall impact. The videos used in the experiment, which were designed as teamwork simulations, were generally mild and lacked strong conflicts or negative emotions, which could have differed from the viewers’ expectations in the negative version. These results suggest that while negative cues can significantly diminish psychological safety, positive cues may require more nuanced or contextually integrated approaches to achieve their intended effect.

Although the positive version did not result in a statistically significant increase in psychological safety scores, the higher average scores observed for this version suggest that positive cues may still possess the potential for enhancing psychological safety. This trend indicates that the positive emojis, while not sufficiently impactful in this context to reach significance, may still have influenced the safety perceptions in subtle ways. This lack of significance could be attributed to the context in which the cues were presented, suggesting that their impact might be more pronounced in different settings.

The potential for positive cues to improve psychological safety, as indicated by higher average scores, warrants further exploration. Therefore, future studies should examine these different settings. Videos were used to simulate teamwork in this study. Emoji intervention needs to be further tested in a real scenario.

5. Conclusions

In this study, emojis, as new social cues in this digital era, were tested as a medium for evaluating and intervening in psychological safety. Our two-step experiments explored the relationships between different emojis and psychological safety, as well as whether emojis could alter the perceptions of emotions and psychological safety.

Our findings demonstrate that positive emojis are positively correlated with psychological safety, with more positive emojis having a greater impact on psychological safety scores. Similarly, negative emojis were negatively correlated with psychological safety, with more negative emojis leading to lower psychological safety scores. The results of the XGBoost regression model with acceptable accuracy revealed the potential of using emojis to evaluate psychological safety.

We also revealed that negative emojis significantly decreased psychological safety scores, whereas positive emojis did not lead to a corresponding increase. These results underscore the powerful impact of the negativity bias and highlight the need for further exploration of how positive visual cues can be effectively leveraged to enhance psychological safety.

Building on these findings, this research contributes to the advancement of predictive models and interactive systems for psychological safety by introducing emojis as novel, real-time indicators of users’ emotional perceptions. Unlike traditional psychological safety assessments that rely on static, long-term evaluations, our approach enables dynamic, moment-to-moment tracking of psychological safety levels. The experimental findings demonstrating the correlation between emoji annotations and psychological safety, as well as the influence of embedded emojis on emotional perception, provide a foundation for AI-driven models that predict users’ psychological safety based on their real-time reactions. These insights can be integrated into interactive systems that adaptively respond to users’ psychological states, such as intelligent collaboration tools or co-creation platforms. By enabling systems to detect potential decreases in psychological safety and intervene accordingly, this advancement enhances the overall user experience. Users benefit from an environment that fosters openness, reduces emotional discomfort, and supports creativity, making co-creation processes more engaging and psychologically safe.

In this study, we have only glimpsed the potential for evolving psychological evaluation methods. One limitation of this study is the use of video simulations to represent emotional expressions and interpersonal interactions. While these simulations provided valuable insights, they may not fully capture the dynamics of real-time, live interactions. Future research should aim to validate these findings in real, collaborative environments, such as teamwork platforms or online collaboration applications, to assess the generalizability and applicability of the results. Moreover, due to the widespread use of emojis across cultures, future research should explore potential cross-cultural differences in their interpretation. Another area for future exploration could be the development of complementary interventions to enhance the effectiveness of positive stimuli. The systems we envision are yet to be fully realized. To achieve genuine real-time evaluation and intervention system development, a larger machine-learning dataset and further algorithm optimization will be essential.