Gamified Text Testing for Sustainable Fairness

Round 1

Reviewer 1 Report

The research is promising but a bit vague

1. Figure 2 and Figure 12 is the same. Please use another case to differentiate your discussion on rules and case ( other than just gender equality). 

2. No pilot data nor evidence on the design that being proposed. even though there is a case study discuss in the paper, its just one case. No info on how many success nor unsuccessful case yet.

3. only 2 gamification elements. How user will know their impact towards the design / rules? No leaderboard or badges or rewards?

 

Author Response

1. Figure 2 and Figure 12 is the same. Please use another case to differentiate your discussion on rules and case ( other than just gender equality).

Response: The identical figures have been edited in line with your valuable criticism.

2. No pilot data nor evidence on the design that being proposed. even though there is a case study discuss in the paper, its just one case. No info on how many success nor unsuccessful case yet.

Response: Our study is a draft in this form, and we plan to pilot this draft in a future study.

3. only 2 gamification elements. How user will know their impact towards the design / rules? No leaderboard or badges or rewards?

Response:  It may be considered that the game design we have developed is not very rich in terms of rewards at this stage. However, it is planned to develop reward mechanisms over time, as it is recognized and seen as an area of attraction by companies with the effect of the planned promotional activities.

 

Author Response File: Author Response.docx

Reviewer 2 Report

Comments

1. Resize figure 13 smaller

2. Add recent references

Author Response

1. Resize figure 13 smaller

Response: The size of the shape is regulated.

2. Add recent references

Response: The current literature has been strengthened in line with your valuable suggestions.

 

Author Response File: Author Response.docx

Reviewer 3 Report

The paper entitled “Gamified Text Testing for Sustainability Fairness” touches on the important topic of algorithmic biases and the authors suggest gamified strategies in order to support fairness in text testing. Despite that, the subject of the study is very timely and the combination of games or gamified systems with AI and inclusivity is very important, there are important aspects like background about gamification and combination of gamification in sustainability (more precisely inclusivity), frame gamification and justify the use of gamified strategies, structure the methodology and add implication that they need improvement.

Even though the introduction touches on gamification and SDT, Background (i.e., related work) does not mention any literature related to gamification and games, especially in the context of AI and inclusivity. The examples are very nicely written, but there is a wide range of topics covered and it is not also clear if this study is positioned among NLPs, and AI. I would suggest categorizing the related work among gamification and Text testing and within the latter category to mane subcategories per topic.

Currently, there are a lot of pages (relatively to the length of the paper) to describe metrics. I understand the need to describe them in order to make the understand the process, but the text is very fragmented, and it is hard to understand why these metrics are needed. My suggestion is:

-          Start with the objective

-          If some measures are common knowledge, then the authors could use the name and a reference maybe

-          A table with columns like the name of the variables uses, the equations, justification/explanation and its contribution to the methodology per category of metrics would be beneficial for both the reader and the description.

This flow will help the reader to understand the needs and it will better describe the suggested methodology. The above might not be possible for all the mentioned metrics of course, but a categorization will help (i.e., text testing, mutation testing, etc.) and it will also help not having so fragmented text.

There are a lot of figures but the connection between them is not clear either their description or the point in the methodology that they are used. The authors have clearly mentioned all the figures and they provide a description for them. However, I am struggling to understand the contribution of Figure 6, and Figure 7, and their description is not enough. For example, Figure 6 is common knowledge on the other hand the text lacks to explain how all these Figures and metrics are composed and synthesized in order to shape the general methodology followed.  In which parts of the methodology are these metrics used? Can they be illustrated in the figures? For example, Figure 10 describes nicely the methodology, and the paragraph above explains well the process.

Currently, gamification is defined: “Gamification broadly refers to technological, economic, cultural, and societal developments in which reality is becoming more gameful, and thus to a greater extent can afford the accruing of skills, motivational benefits, creativity, playfulness, engagement, and overall positive growth and happiness.” (Hamari,2019)

Based on that, phrases like: “the gamification that we developed…”, “a gamification was designed…” and similar need to be rephrased because they do not reflect the theory or literature nor can position this study among gamification of game studies.  

Given the above this study needs to define gamification, explain how the authors borrowed gamified strategies, and mainly why? Do they consider the dark side of gamification? What are the integrated motivational affordances and why? What is the aim of the design and what references justify it?

It is not also clear if this study will have AI or NLPs implications. Who are the users and why this methodology can help? Can help in inclusivity in general or in Fairness? Who can benefit from this study and what about the limitations also?

Minor comments

 There are abbreviations that are first mentioned and later fully described like ZDD

Be concise with figure or Figure

Table 1 format is distorted

References  

 

Hamari, J. (2019). Gamification. The Blackwell Encyclopedia of Sociology, 1-3.

Author Response

The paper entitled “Gamified Text Testing for Sustainability Fairness” touches on the important topic of algorithmic biases and the authors suggest gamified strategies in order to support fairness in text testing. Despite that, the subject of the study is very timely and the combination of games or gamified systems with AI and inclusivity is very important, there are important aspects like background about gamification and combination of gamification in sustainability (more precisely inclusivity), frame gamification and justify the use of gamified strategies, structure the methodology and add implication that they need improvement.

Even though the introduction touches on gamification and SDT,

Background (i.e., related work) does not mention any literature related to gamification and games, especially in the context of AI and inclusivity.

Response: In line with your valuable criticism, since gamification elements are used in our study to improve AI fairness, current research on gamification and artificial intelligence has been added to the literature.

The examples are very nicely written, but there is a wide range of topics covered and it is not also clear if this study is positioned among NLPs, and AI.

Response: In our study, NLP or other methods can be used to create rules, interpretations and tests. However, the aim of the study is not NLP. Our aim is to improve the fairness of artificial intelligence. For this purpose, gamification elements were utilized and various metrics were developed in the gamification design. In our design, rules, interpretations and tests can be developed using simple natural language processing methods as well as more complex and different structures such as machine learning. This gives participants the flexibility to develop tests in a wide variety of ways, while contributing to the main goal of the study: AI fairness.

I would suggest categorizing the related work among gamification and Text testing and within the latter category to mane subcategories per topic.

Response: In line with your valuable suggestions, the related studies are discussed in two categories: AI fairness and gamification and AI. However, since text testing is a topic that we have not yet come across in the literature, this topic could not be addressed in the literature. This is explained in the related works section of the paper.

Currently, there are a lot of pages (relatively to the length of the paper) to describe metrics. I understand the need to describe them in order to make the understand the process, but the text is very fragmented, and it is hard to understand why these metrics are needed. My suggestion is:

-          Start with the objective

Response: In line with your valuable suggestions, the goal of our study has been more clearly defined as follows:,

Based on the difficulties in automating fairness in texts, our study aims to create a new paradigm for fairness testing that will gather disparate proposals on fairness on a single platform, test them and develop the most effective method, and thus contribute to the general orientation on fairness. In order to ensure and sustain mass participation in solving the fairness problem, gamification elements were used to mobilize individuals' motivation.

 

 

-          If some measures are common knowledge, then the authors could use the name and a reference maybe

Response: In the Metrics section, lengthy descriptions of some commonly known methods have been removed and references to these methods have been provided instead.

-          A table with columns like the name of the variables uses, the equations, justification/explanation and its contribution to the methodology per category of metrics would be beneficial for both the reader and the description.

Response: In line with your valuable suggestions, an explanatory table about the functions of metrics has been added to the method section.

This flow will help the reader to understand the needs and it will better describe the suggested methodology. The above might not be possible for all the mentioned metrics of course, but a categorization will help (i.e., text testing, mutation testing, etc.) and it will also help not having so fragmented text.

There are a lot of figures but the connection between them is not clear either their description or the point in the methodology that they are used. The authors have clearly mentioned all the figures and they provide a description for them. However,

I am struggling to understand the contribution of Figure 6, and Figure 7, and their description is not enough.For example, Figure 6 is common knowledge on the other hand the text lacks to explain how all these Figures and metrics are composed and synthesized in order to shape the general methodology followed.

Response: In line with your valuable criticism, Figures 6 and 7 have been revised, the figures that were deemed unnecessary have been removed and the explanations of the included figures have been edited.

 In which parts of the methodology are these metrics used? Can they be illustrated in the figures? For example, Figure 10 describes nicely the methodology, and the paragraph above explains well the process.

Response: Metrics are used to determine users' gaming success. This is made more explicit in the paper. In addition, a table is created to show how metrics determine user game success.

Currently, gamification is defined: “Gamification broadly refers to technological, economic, cultural, and societal developments in which reality is becoming more gameful, and thus to a greater extent can afford the accruing of skills, motivational benefits, creativity, playfulness, engagement, and overall positive growth and happiness.” (Hamari,2019)

Based on that, phrases like: “the gamification that we developed…”, “a gamification was designed…” and similar need to be rephrased because they do not reflect the theory or literature nor can position this study among gamification of game studies. 

Response: In line with your valuable suggestions, Hamari's (2019) definition of gamification was added to the text and accordingly, statements such as "The gamification we developed...", "a gamification was designed..." were edited.

Given the above this study needs to define gamification, explain how the authors borrowed gamified strategies, and mainly why?

Response: In line with your valuable criticisms, the definition of gamification has been revised in our study and why gamification is used has been expressed more clearly.

Do they consider the dark side of gamification? What are the integrated motivational affordances and why?

Response: In line with valuable suggestions, at the end of the methodology section, the limitations of the bill and the motivational possibilities that could be integrated are presented.

 What is the aim of the design and what references justify it?

Response: In our study, based on the difficulties in automating fairness in texts, the idea of gathering disparate proposals on fairness on a single platform was taken as a starting point. In this framework, gamification design was utilized in order to develop proposals for AI fairness, to test the developed proposals and obtain the most effective method, to motivate civic participation in the studies developed for AI fairness and to make it sustainable. Because gamification is a frequently used method to increase motivation (Romano, Díaz, and Aedo 2022). It is important to gather the methods developed for artificial intelligence fairness on a single platform and to develop a user behavior in this direction in order for the developments to become sustainable with this participation. Gamification is known to be an effective tool for a planned change in user behavior (Deterding et al. 2011).

It is not also clear if this study will have AI or NLPs implications. Who are the users and why this methodology can help? Can help in inclusivity in general or in Fairness? Who can benefit from this study and what about the limitations also?

 

Response: The following clarifications have been added to the text based on valuable suggestions:

In general, everyone who cares about and works in the field of artificial intelligence fairness constitutes the target audience of our design. In particular, software developers who produce solutions on fairness constitute the participants of the fair-test game, while everyone who is affected by AI fairness constitutes the end-user. Our design is expected to become a ground where current practices on fairness can be applied and contribute to the sustainability of contributions on this issue by gathering them on a single platform.

Since there will not be enough participants when the system is first launched, the success will be low. In order to overcome this problem, it is planned to focus on promotional activities. In addition, it can be considered that the game design we have developed is not very rich in terms of rewarding at this stage. However, it is envisaged to develop reward mechanisms over time, as it becomes known by companies and starts to be seen as an area of attraction with the effect of promotional activities.

 

Minor comments

 There are abbreviations that are first mentioned and later fully described like ZDD

Be concise with figure or Figure

Table 1 format is distorted

References

Response: An explanation of the abbreviation ZDD has been added at the first occurrence of the phrase. Images have been organized as a single expression with the name "figure". Table 1 has become Table 2 due to the newly added table and its format has been organized. Finally, references have been reviewed and organized.  

Hamari, J. (2019). Gamification. The Blackwell Encyclopedia of Sociology, 1-3.

 

 

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

This article is about using gamification to increase motivation and involvement of user in order to support sustainable contribution on the AI fairness of the texts issues. The algorithm developed is quite promising and author did inform that they are still developing the reward mechanism as it is one of the important gamification elements for this suggested design. This design has not yet tested among pilot users yet, so as a conceptual paper, it is acceptable. Suggested design in this paper can help other researcher to explore further related to AI fairness, gamification and sustainability through involvement of users.