No Pain, No Gain—Giving Real-Time Emotional Feedback in a Virtual Mirror Improves Collaboration in Virtual Teamwork

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

In this work, the author introduced real-time emotional feedback based on facial sentiment analysis techniques to videoconferencing for improving virtual team collaboration. The team performance was evaluated through the Mars simulation game with multi-objective optimization problems. Hypotheses were well tested against the null by analyzing the combination of dependent, independent, and control variables with ML algorithms. Limitations and future work are well discussed. I appreciate the amount of work and intellectual contribution and recommend publishing the paper once the following comments are addressed.

 

[Experimental]

1.1 Line 213 "Participant recruitment was facilitated through various channels, including mailing lists, classroom announcements, and direct contact, leveraging university networks such as the SDM master’s program at MIT", what information was shared with participants prior to the experiment?

 

1.2 Line 299 "To mitigate this, performance improvement was calculated as the difference in average ranks between the first and second halves of the task, indicating whether teams improved over time". Differences between two ranks are discrete values, and the absolute improvement value cannot be quantified (i.e. differences between Rank 1-2 and Rank 50-51 are not the same). Can the author justify using discrete values instead of continuous values for evaluation? For example, a continuous score based on a cost function involving normalized energy use and site utilization scores could be more suitable for downstream regression analysis, as shown in Figure 4.

 

1.3 Section 4.5.3 "Feature Interpretation with SHAP Values". I only see the discussion, but the actual SHAP experiment results are missing. Can the author plot the SHAP feature analysis using a beeswarm plot provided in the SHAP package for all the ML models? For example, see the tutorial at https://shap.readthedocs.io/en/latest/example_notebooks/api_examples/plots/beeswarm.html

 

[Minor]

2.1 In Figure 3, some of the contents are missing in the legend. What is the meaning of the number labeled next to each dot? And what does the color bar represent (i.e., explain Result ID)? Was the grey dots of simulation tradespace obtained through exhaustive search or a combination of all existing experiments?

 

2.2 Line 357 "...a unique mapping was created to match the IDs between the two output files". Can you state the time duration between two consecutive IDs?

 

2.3 Line 409 "The models were evaluated based on several key performance metrics, including Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and R2". RMSE can be directly calculated from MSE by taking the square root of MSE, so these two metrics are essentially the same in Table 5. For example, the MSE for Random Forest is 0.6894, and RMSE = sqrt(0.6894) = 0.8303. Therefore, there is no need to show both.

 

[Suggestion]

3.1 For better clarity, in Figure 3, it will be nice to show the direction of optimization with an arrow pointing towards the bottom right of the plot (with decreasing energy use and increasing site utilization). Another option is to plot the Pareto Front curve at the data boundary.

Author Response

"Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The article presents a novel investigation into the impact of real-time emotional feedback on virtual team performance. The study employs a robust experimental design with 28 teams collaborating in a simulated environment, providing a comprehensive dataset for analysis. This study presents a promising exploration of real-time emotional feedback in virtual teamwork, i.e., its innovative approach, comprehensive data analysis, and diverse participant pool. 

 

For improvement:

1. Thiss study includes a diverse participant pool, the relatively small sample size (84 participants) may limit the generalizability of the findings and implications (section 5.5). 

2. It relies heavily on facial and voice emotion recognition technologies, which may have limitations in accuracy, especially in real-world experimental conditions. More detail on the validation and reliability of these technologies can strengthen the results, e.g., tables 4-6.

3. It suggests that real-time feedback improved (team) performance, but it remains unclear which aspects of the feedback were most effective. A more detailed breakdown of feedback components and their individual impact can provide clearer guidance for practical applications.

4. It provides a snapshot of team performance during a single task. Longitudinal studies examining the lasting effects of emotional feedback on team dynamics and performance will offer deeper insights intoo the sustained benefits of such interventions. For emotional factors and emotion design principles, it is suggested to reference: doi.org/10.3389/frvir.2021.643331; doi.org/10.1016/j.ijadr.2023.06.002

5. Although it includes several control variables, additional factors such as participants' prior experience with virtual teamwork and familiarity with the simulation software can further influence the results.

 

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The article presents an study investigates the impact of real-time emotional feedback on the quality of team-work conducted over videoconferencing. The article is well-written. The methodology used is adequate. The experiments were conducted in different institutions. Statistical analyses performed revealed marginally significant differences in performance between the control and the group receiving real-time emotion feedback (treatment), alongside notable variations in emotional and conversational behaviors, suggesting an impact on social dynamics. These findings might be useful for researchers in the area of study. 

Minor comments: 

The authors could add a paragraph describing the organization of the article at the end of the introduction. Some paragraphs describing the organization of several sections could be added. 

Maybe some of the ML models utilized could be briefly explained, so that an audience without ML background can understand the differences among them. 

Author Response

Please see the attachment

Author Response File: Author Response.pdf