Supporting a Sustainable and Engaging Online Transition for Co-Design through Gamification
Round 1
Reviewer 1 Report
This is an interesting and relevant study about the use of gamification in online co-design. The authors target engagement and creativity in the co-design process and reach a set of 4 guidelines to design effective gamification for online co-design.
In the methodology it is not clear that participants were exposed to only one workshop. One participant is quoted in table 4 as saying: “Though the non-game version has the same structure, the way of the game version that presented is cooler”. This seems to suggest that she participated in two versions of workshop (a game version and a non-game version).
Regarding coding, it is not clear how this was implemented. Who conducted the coding? Was it independently conducted by more than one researcher or, at least, checked by other researchers?
Since results from participants from the 3 groups are not presented separately, it is not easy to discern results by age/professional group. However, it does seem that students were more comfortable with ideaGardener, probably because they are more familiar with similar games.
The discussion brings in results from the observations that were not presented in the results section. If such results were presented, then the first stated limitation for this study might be attenuated (“Due to the data collection methods, the validity of the findings strongly relies on the participants' honesty and sincerity.”)
State that “the creativity performances are improved” implies a comparison (are improved relatively to what?). This probably the greatest shortcoming in this article, since there is no baseline for establishing a comparison. The research design does include it and that may be just fine, but then comparisons are not strictly possible.
Even if the benefits in terms of sustainability may surpass any negative effects, I think these should, at least, be acknowledged. There are also costs in using online environments, e.g. energy consumption (both local and at the larger Internet infrastructure).
The article conclusion is essentially a summary. I think it should stress the results reached, including the guidelines provided in section 5.3.
109 references may be a lot for an article with about 18 pages, but they seem appropriate. The references list should be carefully proofread, because some apparently do not follow the journal guidelines (especially references to websites).
Some minor corrections are identified below.
Line 33: reference is quoted using author (date) instead of numeric referencing.
Line 117: affectionate is probably not the best term here.
Line 143: online formats is probably a better expression instead of online forms.
Line 155: something is missing at the end of the sentence (probably the word “changes”).
Liine 197: game-liked rules > probably game-like rules
Line 223-224: Hunter & Werbach (2012) and Zichermann & Cunningham (2011) [60,61] > Hunter & Werbach [60] and Zichermann & Cunningham [61]
Line 253: Due to > Since
Line 328-329: to investigate the observable behavioural and emotional engagement phenomenon > to investigate observable behavioural and emotional engagement
Lines 357-359: “Several participants emphasized the main difference between the online gamified workshops and more conventional ones is they are more enjoy the gamification one” > “Several participants emphasized they enjoyed online gamified workshops more, when comparing with previous experiences with conventional” is probably a better statement, assuming that participants were indeed comparing their online gamified workshop experience with previous conventional workshops.
Line 375: when someone starts to close the camera > when people start turning off their cameras
Table 7: mass up > mess up
Line 492: The data reveals both consistency and distinguishment > The data reveals both similarities and differences
Line 683: G. L > G. C
Author Response
Dear Reviewer,
We would like to thank you for the detailed review and the valuable comments provided. As requested, we have restructured paragraphs and added more text by expanding some sections and references. All suggestions are incorporated as shown in the revised manuscript.
Comment 1: In the methodology, it is not clear that participants were exposed to only one workshop. One participant is quoted in table 4 as saying: “Though the non-game version has the same structure, the way of the game version that presented is cooler”. This seems to suggest that she participated in two versions of workshop (a game version and a non-game version).
Response 1: For this study, each type of participant was exposed to only one workshop (as mentioned in section 3.2, lines 300-307). The students have indeed participated in multiple workshops during the course. However, they are not counted, influenced, or relevant for this study. To avoid ambiguity, we deleted all the descriptions of the "non-game version."
Comment 2: Regarding coding, it is not clear how this was implemented. Who conducted the coding? Was it independently conducted by more than one researcher or, at least, checked by other researchers?
Response 2: One author and an external expert conducted the coding. It follows an iterative procedure that allows triangulation, member checking, and peer examination to ensure credibility. We added such descriptions in section 3.4, lines 344-355.
Comment 3: Since results from participants from the 3 groups are not presented separately, it is not easy to discern results by age/professional group.
Response 3: We added the group information following each participant's quotations. (Table. 4,5,6,7,8,9)
Comment 4: The discussion brings in results from the observations that were not presented in the results section. If such results were presented, then the first stated limitation for this study might be attenuated (“Due to the data collection methods, the validity of the findings strongly relies on the participants' honesty and sincerity.”)
Response 4: The scope of the participants' webcam limits the video observation method. Therefore, instead of generating interpretable results, the observation is considered to support triangulation for the data gathered from interviews. We added such descriptions in section 5.4 lines 655-658, and section 3.3 lines 327-328.
Comment 5: State that “the creativity performances are improved” implies a comparison (are improved relatively to what?). This probably the greatest shortcoming in this article, since there is no baseline for establishing a comparison. The research design does include it and that may be just fine, but then comparisons are not strictly possible.
Response 5: We deleted the statement "creativity performances are improved." (Lines 563-564, 665, 677) We drew this conclusion from participant interviews, in which many of them stressed a feeling of being empowered with creativity. However, we agree that this is not rigorous evidence of a significant improvement.
Comment 6: Even if the benefits in terms of sustainability may surpass any negative effects, I think these should, at least, be acknowledged. There are also costs in using online environments, e.g. energy consumption (both local and at the larger Internet infrastructure).
Response 6: We acknowledged such negative effects in section 5.2, lines 568-569.
Comment 7: The article conclusion is essentially a summary. I think it should stress the results reached, including the guidelines provided in section 5.3.
Response 7: We significantly improved the conclusion section (lines 680-693). The main results and contributions are now summarized in the conclusion.
Comment 8: 109 references may be a lot for an article with about 18 pages, but they seem appropriate. The references list should be carefully proofread, because some apparently do not follow the journal guidelines (especially references to websites).
Response 8: We proofread the references list and fixed several mistakes.
Comment 9: Some minor corrections are identified.
Response 9: We conducted all the corrections that you have identified.
Reviewer 2 Report
This article presents an empirical study to investigate the benefits of gamification for online co-design. The authors approach the problem valuing the importance of the active participation of the actors in the design of solutions, to support social sustainability and innovation.
The introduction raises several edges to be studied, on the one hand to assess the usefulness of co-design, on the other hand the limitations that arise when having to do it online and finally demonstrate the usefulness of gamification mechanics in said online environment.
The topic seems completely interesting and pertinent to me, which is why its reading generates a lot of interest, and of course I recommend the work for publication, however I find that there are several previous considerations or minor changes to be made prior to its publication.
I found the historical approach to the concept of co-design very interesting, however, I would like you to position yourself towards the user-centered design approach, considering the ISO 9241-110 Principles of the Human-Centered Approach standard.
I would find it very useful to add a specific section of previous works or more recent related works, although it is true that they exist in the work, they are scattered in various sections of the document. I would also recommend reading Front A. A Participative End-User Modeling Approach for Business Process Requirements.
Although it is true that the article aims to measure the perceptions of gamified online co-design activities, I need a section that details, describes the ideaGardener tool, its features, functionalities, architecture, etc. It is interesting to know about the tool, but the authors do not specify the domain where the ideaGardener tool is available, nor was it possible to find it in a search engine. It would be very interesting to know your usability study for example.
When I reviewed the references, I found that not all of them respect the citation format, some underline the year and others do not, for example.
Not all tables respect the format required for the article, for example table 4,5, 6, 7, 8, 9 exceed the margins.
Research questions are answered, and the results are relevant and interesting. The conclusion section should be greatly improved.
Author Response
Dear Reviewers,
We would like to thank you for your interest, detailed review, and the valuable comments provided. As requested, we have restructured paragraphs and added more text by expanding some sections and references. All suggestions are incorporated as shown in the revised manuscript.
Comment 1: I found the historical approach to the concept of co-design very interesting, however, I would like you to position yourself towards the user-centered design approach, considering the ISO 9241-110 Principles of the Human-Centered Approach standard.
Response 1: We added such a positioning statement in section 1, lines 64,65.
Comment 2: I would find it very useful to add a specific section of previous works or more recent related works, although it is true that they exist in the work, they are scattered in various sections of the document. I would also recommend reading Front A. A Participative End-User Modeling Approach for Business Process Requirements.
Response 2: Instead of a section of 'related work,' we connected a storyline built on the emerging challenge (the pandemic), historical factors (co-design), and the promising new technique (gamification). The reasons to do so are 1. The true need for online co-design is almost completely new and emerges with the pandemic. There are very few relevant publications that investigate such topics to support a section of recent related work. 2. the related work on separately co-design or gamification seems not convincing enough. We believe that in order to bridge these two practices, the most 'readerfridenly' way is to scatter the related work content into a storytelling line.
Comment 3: Although it is true that the article aims to measure the perceptions of gamified online co-design activities, I need a section that details, describes the ideaGardener tool, its features, functionalities, architecture, etc. It is interesting to know about the tool, but the authors do not specify the domain where the ideaGardener tool is available, nor was it possible to find it in a search engine. It would be very interesting to know your usability study for example.
Response 3: We introduced the ideaGardener tool in terms of its objectives, design tasks, and game design elements. Such content can be found in section 3.1, lines 217-295. The tool is currently unavailable to the public but we are certain that it will be published very soon. A usability study is indeed has been carried out. However, the manuscript is still under consideration by a journal.
Comment 4: When I reviewed the references, I found that not all of them respect the citation format, some underline the year and others do not, for example.
Response 4: We proofread the references list and fixed several mistakes. We would like to point out in addition that according to the journal's 'instructions for Authors,' only the references of the journal article should bold the year.
Comment 5: Not all tables respect the format required for the article, for example table 4,5, 6, 7, 8, 9 exceed the margins.
Response 5: There are actually two kinds of table formats in the journal's template. Considering the length of participants' quotations, we applied the larger format of the table.
Comment 6: Research questions are answered, and the results are relevant and interesting. The conclusion section should be greatly improved.
Response 6: We significantly improved the conclusion section (lines 680-693). The main results and contributions are now summarized in the conclusion.
We have made all the changes (again thank you for your helpful suggestions), which allowed us to provide strong arguments. Therefore, we are confident that this revised version of the manuscript is greatly improved. In the meantime, we are prepared to conclude any further suggestions that the reviewers may have. We look forward to hearing from you soon.
Sincerely,
