Learning Course: Application of Gamification in Teaching Construction and Building Materials Subjects

Round 1

Reviewer 1 Report

The paper is well-written, the narration is easy to follow, and it has the value of showing the expansion of gamification to yet another area of education.

Nonetheless, the title should definitely be rephrased, as the paper does not cover the "application of gamification in construction and building materials" but only in teaching such subjects.

The literature review is missing any mentions of similar examples of application of gamification based on student response systems such as Kahoot in other subject of education (and there are plenty of such examples that could be discussed).

My main dissapointment is in the "Outcomes" section. The author(s) do not provide any data that could help verify whether the application was actually succesful or not. The key questions that should be answered are: Was the average level of knowledge measured among the students who participated in the gamified course (or the share of students who attained specific score or just passed the exam) higher or lower compared to those who did not participate? How these results compare to the preceding years when there was no gamification involved?  Unfortunately, the reviewed paper does not explain that and the presented results are of secondary importance.

I therefore recommend the paper to be extended in the above-described aspects before it could be accepted for publication.

 

Author Response

The paper is well-written, the narration is easy to follow, and it has the value of showing the expansion of gamification to yet another area of education.

The authors are very grateful to the reviewer for the comments, which have helped them to address important details and to improve the quality of the paper. The manuscript has been revised and changes are highlighted in red throughout the text.

Nonetheless, the title should definitely be rephrased, as the paper does not cover the "application of gamification in construction and building materials" but only in teaching such subjects.

The title has been changed following the reviewer’s suggestion.

The literature review is missing any mentions of similar examples of application of gamification based on student response systems such as Kahoot in other subject of education (and there are plenty of such examples that could be discussed).

Thank you for the comment. The introduction has been revised (changes in red) and some examples have been introduced and commented.

My main dissapointment is in the "Outcomes" section. The author(s) do not provide any data that could help verify whether the application was actually succesful or not

Thank you very much for the comment. The “Outcomes” section has been revised in order to extend the results considering the reviewer's key questions. The following paragraph has been introduced explaining these questions:

“The students of this academic year had had a significantly lower university entrance grade, what had been progressively observed since the previous two years, and the worse results of some tests carried out in class were worrying. This conditioning factors suggested a probably significant decrease in the number of pass grades with respect to previous years. However, the number of students who passed the subject during the 2018-19 academic year was 69% of the students examined, being 9% higher than the previous academic year and only 3,4% lower than the value corresponding to the 2016-17 academic year. In this way, although more factors might be influencing, it can be said that the participation in the learning contest had a positive effect. Figure 2, with respect to the students who passed the subject, shows the influence resulting from participation in the contest, both in terms of those who achieved a passing grade and those who passed without it but improved their final grade. It can be said that the in-fluence of the contest in this sense was highly favorable. As shown in the graph, 66% of the students who passed (pass grade) did so thanks to their participation in the contest, and 25% of them improved their grades (with average mark slightly higher than 7). Only 9% of the students who passed the subject, obtaining an average grade of around 5, did not participate in the contest.”

The key questions that should be answered are: Was the average level of knowledge measured among the students who participated in the gamified course (or the share of students who attained specific score or just passed the exam) higher or lower compared to those who did not participate?

The text has been revised answering this question.

How these results compare to the preceding years when there was no gamification involved?  Unfortunately, the reviewed paper does not explain that and the presented results are of secondary importance.

Thank you very much for your comment. The “Outcomes” section has been rewritten in order to introducing additional data and improve it. The comparison with preceding years has been introduced and is exposed in the previous paragraph. In addition, some paragrahs that may relate to this comment have been introduced throughout the manuscript.

I therefore recommend the paper to be extended in the above-described aspects before it could be accepted for publication.

 

Reviewer 2 Report

This paper analyses a gamification activity, in the form of a learning contest, to improve students’ interest in studying  the some subject (materials and construction) and to help them learn and master it. Many studies highlight the effectiveness of transferring game-based elements and learning methodologies to the classroom for promoting assimilation of the complex concepts present in different areas of the student's curriculum. The experiment has been conducted in escuela de Ingenieros de Caminos of the UPM, during many academic years. The proposed initiative represents, in my opinion, an innovative method for university-level teaching.

I think it is necessary to show more examples of kahoot tests to understand how the lectures have converted the contents of the subject into games.

It would also be useful to understand how the student's satisfaction tests are structured.

I appreciated the enthusiasm with which the researchers tell their experience: perhaps, a little too self-referenced.

Author Response

This paper analyses a gamification activity, in the form of a learning contest, to improve students’ interest in studying  the some subject (materials and construction) and to help them learn and master it. Many studies highlight the effectiveness of transferring game-based elements and learning methodologies to the classroom for promoting assimilation of the complex concepts present in different areas of the student's curriculum. The experiment has been conducted in escuela de Ingenieros de Caminos of the UPM, during many academic years. The proposed initiative represents, in my opinion, an innovative method for university-level teaching.

The authors are very grateful to the reviewers for the comments, which have helped them to address important details and to improve the quality of the paper. The manuscript has been revised and changes are highlighted in red throughout the text.

I think it is necessary to show more examples of kahoot tests to understand how the lectures have converted the contents of the subject into games.

Thank you for the comment. The introduction has been revised (changes in red) and some examples have been introduced and commented. However, the paper is based on the experience with gamification through the learning contest. A paragraph has been introduced in the introduction section to clarify the scope of this work.

It would also be useful to understand how the student's satisfaction tests are structured.

The authors appreciate this comment. The students’ opinión has been collected through meetings in small groups, which we have called focus groups, and surveys. Sections 3 and 4 have been revised to describe and explain both methods and their result. Some additional comments on this issue have been introduced throughout the manuscript

I appreciated the enthusiasm with which the researchers tell their experience: perhaps, a little too self-referenced.

The authors appreciate the reviewers’ comment. in the referenced works there is a progression in the teaching and evaluation activities studied, and in the authors’ opinion the evolution of accumulated experience can be a quite interesting contribution to knowledge in this area.

Reviewer 3 Report

The paper proposes a topic of great topicality and importance. In the coming years, university teaching will necessarily have to innovate by finding new techniques and new teaching solutions. In this field, any attempt and/or experimentation is always very useful.
Working on the issue of motivation to learn and on the promotion of students' interest in studying can be the right way forward. It is useful to experiment with appropriate strategies to improve this process.

After reading the document, and with the aim of improving its quality, the following brief remarks are made to make the text clearer:

- A single year of experimentation seems too little to formulate scientifically valid conclusions. It would be useful to carry out a qualitative evaluation/assessment over a three-year period;

- The quantitative judgements and personal evaluations of the students are clear, while the qualitative judgements of the teachers are not adequately described. Bring more arguments on this aspect;

- If the level of satisfaction on the part of the students is considered in quantitative terms, it would be useful to estimate how well prepared the students are with regard to the following aspects: soft skills, notions, logic, ability to work in a team, etc. Consider whether these learning objectives have been achieved and if not, why!;

- The way in which the "competition" was carried out is not very clear (it would be useful to describe this activity better);

- Describe more fully how the Kahoot computer application works;

- In the conclusions, the limits and potential of this didactic experiment should be explained in more detail.

It may be useful to implement the bibliography on the themes of innovative teaching and gamification.

 

 

Author Response

The paper proposes a topic of great topicality and importance. In the coming years, university teaching will necessarily have to innovate by finding new techniques and new teaching solutions. In this field, any attempt and/or experimentation is always very useful.
Working on the issue of motivation to learn and on the promotion of students' interest in studying can be the right way forward. It is useful to experiment with appropriate strategies to improve this process.

The authors are very grateful to the reviewers for the comments, which have helped them to address important details and to improve the quality of the paper. The manuscript has been revised and changes are highlighted in red throughout the text.

After reading the document, and with the aim of improving its quality, the following brief remarks are made to make the text clearer:

- A single year of experimentation seems too little to formulate scientifically valid conclusions. It would be useful to carry out a qualitative evaluation/assessment over a three-year period.

The authors appreciate this comment and agree with the reviewer’s opinion. The idea was to apply the contest during the 2019-20 and 2020-21 academic years and compare results. However, this has been ultimately impossible due to the confinement that we have had at this time as a consequence to the pandemic. This point in explained in the discussion section:

“Based on these premises, the authors believe that the outcomes obtained in this activity are quite satisfactory. So, the idea was to continue using this learning technique during the 2019-2020 and 2020-21 academic years. However, as a consequence of the global health emergency in early 2020 triggered by the COVID-19 pandemic, the Universidad Politécnica de Madrid (UPM) was forced to modify its teaching methodologies. UPM scrambled to adapt and apply digital systems for remote learning. Due to this, the contest could not be held during the years 2020 and 2021.”

The outcomes section has been revised in order to extend the results in comparison with preceding years:

“The students of this academic year had had a significantly lower university entrance grade, what had been progressively observed since the previous two years, and the worse results of some tests carried out in class were worrying. This conditioning factors suggested a probably significant decrease in the number of pass grades with respect to previous years. However, the number of students who passed the subject during the 2018-19 academic year was 69% of the students examined, being 9% higher than the previous academic year and only 3,4% lower than the value corresponding to the 2016-17 academic year. In this way, although more factors might be influencing, it can be said that the participation in the learning contest had a positive effect. Figure 2, with respect to the students who passed the subject, shows the influence resulting from participation in the contest, both in terms of those who achieved a passing grade and those who passed without it but improved their final grade. It can be said that the in-fluence of the contest in this sense was highly favorable. As shown in the graph, 66% of the students who passed (pass grade) did so thanks to their participation in the contest, and 25% of them improved their grades (with average mark slightly higher than 7). Only 9% of the students who passed the subject, obtaining an average grade of around 5, did not participate in the contest.”

- The quantitative judgements and personal evaluations of the students are clear, while the qualitative judgements of the teachers are not adequately described. Bring more arguments on this aspect;

Thank you very much for the observation. The “Outcomes” section has been revised following the reviewers’ comment.

- If the level of satisfaction on the part of the students is considered in quantitative terms, it would be useful to estimate how well prepared the students are with regard to the following aspects: soft skills, notions, logic, ability to work in a team, etc. Consider whether these learning objectives have been achieved and if not, why!;

Thank you for the comment. The outcomes section has been modified according to the reviewer's questions.

Regarding the transversal competences, those that are worked with the learning contest are commented, although additional future work is necessary to be able to quantify the learning of these competences.

- The way in which the "competition" was carried out is not very clear (it would be useful to describe this activity better);

The text has been revised following the reviewer’s comment.

- Describe more fully how the Kahoot computer application works;

Thank you for the comment. The introduction has been revised (changes in red) and some examples have been introduced and commented. However, the paper is based on the experience with gamification through the learning contest. A paragraph has been introduced in the introduction section to clarify the scope of this work.

- In the conclusions, the limits and potential of this didactic experiment should be explained in more detail.

Thank you for the comment. According to the reviewer’s comment, the manuscript has been revised in order to explain the limits and potential of the teaching activity studied in more detail.

It may be useful to implement the bibliography on the themes of innovative teaching and gamification.

The introduction has been revised and some examples of gamification have been introduced.

Round 2

Reviewer 1 Report

Dear Authors,

>The title has been changed following the reviewer’s suggestion.

Thank you, but the current title still does not imply that gamification is applied to teaching those subjects. My suggestion:

Learning course: application of gamification in teaching construction and building materials subjects

>The introduction has been revised (changes in red) and some examples have been introduced and commented.

Thank you, but you took my remark too precisely and just mentioned these three works, whereas they deserve some more attention (please describe where it was applied, in teaching of what subjects, what was the measured outcome). And as this a central topic of this paper I'd suggest including 2-3 more references of this kind.
Besides, the ref. 15 seems malformed (surname of the first author is swapped with her first name).

>>My main dissapointment is in the "Outcomes" section. The author(s) do not provide any data that could help verify whether the application was actually succesful or not
>The following paragraph has been introduced explaining these questions:

Thank you. However, the new paragraph seems to address only my remark about the preceding years well, but not the key question remark.

Moreover, it certainly needs to be polished in terms of clarity.
Already its very beginning is far from clear "The students of this academic year had had a significantly lower university entrance grade, what had been progressively observed since the previous two years" - what was this supposed to mean? That they had lower university entrance grade than the students of both the two preceding years?
"the worse results of some tests carried out in class were worrying" - what "some tests"? From the construction and building materials subjects? And they were worrying for whom?

Later on: "although more factors might be influencing, it can be said that the participation in the learning contest had a positive effect". Well if you did not check out whether more factors were influencing, then you cannot say for sure "that the participation in the learning contest had a positive effect". What could be the other factors?

Besides, please use the American decimal point, not the European one (3,4%).

>>The key questions that should be answered are: Was the average level of knowledge measured among the students who participated in the gamified course (or the share of students who attained specific score or just passed the exam) higher or lower compared to those who did not participate?
>The text has been revised answering this question.

I'm sorry but you have not still answered the question. 
From what your wrote, it appears that you gave extra points to the students participating in the Kahoot game.
This resulted in them passing the exam or getting a better grade thanks to those extra points.
But it does not mean their level of attained learning objectives (knowledge or skills) was higher than those who did not take part in the game.
Have you compared these? That's the key question that your research should have answered.

 

Author Response

Reviwer # 1

Comments and Suggestions for Authors

Dear Authors,

>The title has been changed following the reviewer’s suggestion.

Thank you, but the current title still does not imply that gamification is applied to teaching those subjects. My suggestion:

Learning course: application of gamification in teaching construction and building materials subjects

Thank you for your recommendation. The authors think that this title is appropriate and has been changed following your suggestion.

>The introduction has been revised (changes in red) and some examples have been introduced and commented.

Thank you, but you took my remark too precisely and just mentioned these three works, whereas they deserve some more attention (please describe where it was applied, in teaching of what subjects, what was the measured outcome). And as this a central topic of this paper I'd suggest including 2-3 more references of this kind.

Thank you for your comment. The text has been revised following your recommendation. The following paragraph has been introduced in the section:

“In the literature there are numerous works on the use of gamification in university education, particularly through the computer application Kahoot [14-18], that have obtained significantly positive results. D.T.A. Lin, et al. [14] did an extensive study based on surveys to analyze the extent to which the use of Kahoot improved undergraduate students’ learning. This work was carried out with students of “English for the Media”, which covers theoretical and practical contents. Their results highlighted its effectiveness in encouraging and reinforcing learning in both dimensions. J.M. Campillo-Ferrer et al. [15] examined the effect that a gamification activity based on Kahoot carried out in “Teaching Social Sciences”, a second-year compulsory subject of the Primary Education degree at the University of Murcia, had on students’ level of motivation and learning. They found that this teaching technique improved students’ understanding of some concepts, increased their active participation in class, and motivated them towards learning. In the same field of knowledge, J.P. Hernández-Ramos et al. [16] studied the influence of using a teaching methodology gamified with Kahoot in higher education based on the students’ opinion of several educational degrees. In this case, students expressed a positive opinion about the usefulness of the teaching technique to help improve motivation for the study of the subject, particularly when used for self-study and self-evaluation. However, they were not convinced that the gamification methodology be more effective than other traditional methodologies. A.M. Diez-Pascual and M.P.G. Díaz [17] studied the use of the computer application Kahoot in a subject of the chemistry area. They found an improvement in the teaching–learning process of the students who used the tool in comparison with the control group that did not use it. The academic results showed an enhancement with increasing the frequency of use of Kahoot. M. Stoyanova et al. [18] aimed at analyzing the influence of gamification in mathematics subject with Kahoot on the students’ learning process and emotions. The results obtained showed that the students were happier, more motivated, and more focused on to study mathematics with the use of the teaching technique.”

Two more references have been introduced.

Besides, the ref. 15 seems malformed (surname of the first author is swapped with her first name).

This reference has been checked.

>>My main dissapointment is in the "Outcomes" section. The author(s) do not provide any data that could help verify whether the application was actually succesful or not
>The following paragraph has been introduced explaining these questions:

Thank you. However, the new paragraph seems to address only my remark about the preceding years well, but not the key question remark.

Moreover, it certainly needs to be polished in terms of clarity.
Already its very beginning is far from clear "The students of this academic year had had a significantly lower university entrance grade, what had been progressively observed since the previous two years" - what was this supposed to mean? That they had lower university entrance grade than the students of both the two preceding years?
"the worse results of some tests carried out in class were worrying" - what "some tests"? From the construction and building materials subjects? And they were worrying for whom?

Thank you very much for your comment. With the aim of clarifying the following paragraph has been introduced in section 1:

“Approximately 30% of the new students accessed to the degree in the second call, coming from other degrees in which their grade did not allow them to access. The grade of almost all of them was the minimum to access university studies”.

In addition, the following paragraph has been introduced in section 3:

“In the Construction and Building Materials subjects students who take continuous assessment take a weekly test throughout the academic year with the aim that students achieve a higher level of learning progressively. The mark obtained by the students who take the subject by continuous assessment has a component achieved by the active learning activities performed throughout the academic year (especially based in cooperative learning or one minute paper), to which in the academic year of study the learning contest is added. This component is called “mark class” and is equivalent to 30% of the final grade. This year the learning contest represents 10%, and the rest of activities 20%. The rest of the mark obtained by continuous assessment is composed of two partial exams. There is the possibility of passing the subject only by exams (the mark is the average of the two partial exams), however the reality is that the vast majority of students, around 98%, pass the subject by continuous assessment.”

In the first tests carried out, the students were not obtaining good grades in comparison with the results of previous years, and this caused the lecturers to be concerned that the final results were not good.

Later on: "although more factors might be influencing, it can be said that the participation in the learning contest had a positive effect". Well if you did not check out whether more factors were influencing, then you cannot say for sure "that the participation in the learning contest had a positive effect". What could be the other factors?

The authors greatly appreciate the observation. Indeed, the expression can be misleading, and might make the reader think that the positive effect is not sufficiently verified. This is not the case since the positive effect of the contest is widely validated by the results and the students’ opinions. There was an attempt to say that there are more teaching activities carried out along the academic year that can help to improve the learning results of the students. The expression has been changed to clarify this point:

“although the rest of teaching activities carried out along the academic year also might be influencing in the improvement of the learning results, it can be said that the participation in the learning contest had a positive effect”

Besides, please use the American decimal point, not the European one (3,4%).

This has been corrected.

>>The key questions that should be answered are: Was the average level of knowledge measured among the students who participated in the gamified course (or the share of students who attained specific score or just passed the exam) higher or lower compared to those who did not participate?
>The text has been revised answering this question.

I'm sorry but you have not still answered the question. 
From what your wrote, it appears that you gave extra points to the students participating in the Kahoot game.
This resulted in them passing the exam or getting a better grade thanks to those extra points.
But it does not mean their level of attained learning objectives (knowledge or skills) was higher than those who did not take part in the game.
Have you compared these? That's the key question that your research should have answered.

Thank you very much for the comment. The authors agree that this data may be important to assess if participation in the contest improves the students' learning objectives. However, we think that to complete the study it is necessary to take into account some more aspects as explained in the paper. The mark obtained by the students who take the subject by continuous assessment has a component achieved by the active learning activities performed throughout the academic year (especially based in cooperative learning or one minute paper), to which in the academic year of study the learning contest is added. This component is called “mark class” and is equivalent to 30% of the final grade. This year the learning contest represents 10%, and the rest of activities 20%. The rest of the mark obtained by continuous assessment is composed of two partial exams. There is the possibility of passing the subject only by exams (the mark is the average of the two partial exams), however the reality is that the vast majority of students, around 98%, pass the subject by continuous assessment. When considering the grade obtained regardless of the contest, the average grade obtained by the students who participated in the contest was 0.9 points higher than that of the students who did not participate in the contest. This last sentence is included in the text (in the Outcomes).

On the other hand, we think that the results and discussion set out in this work are interesting and useful to study the influence of the learning activity carried out. Firstly, the grade that students can obtain through the contest is not achieved for the sake of participating in it, but rather measures the student's knowledge achievement. The contest is about a part of the subject, not the entire content. In addition, the dynamics of the contest makes a fairly broad assessment of the learning of this part, which for example is not possible in an exam of the subject since the last covers more content in a more limited time. In this way, obtaining a good grade in this activity implies a good knowledge of this part. Due to the dynamics of the activity, although they have the guidance of the lecturers, the students must seek for information and prepare the topics, working in teams. This means an increase of workload, however this way of working using gamification, in which is based the contest, the students work in a more attractive way for them, improving the possibilities of acquiring the learning objectives. This fact is widely supported by the opinions of the students collected in the survey and in the “focus groups” carried out, as it is exposed in the paper. Finally, it should be noted that during the activity, transversal skills are worked for the professional future of the current civil engineer, mainly the ability to work in a team, autonomous learning, logic and critical reasoning. 

Reviewer 2 Report

The paper is clearer and focused, in my opinion. Well done

Author Response

The authors really appreciate the reviewer's help to improve  the quality of the manuscript

Reviewer 3 Report

My comments have been taken into account.
The additions to the paper seem relevant.
The question remains whether an experience (experiment) lasting only one year can have scientific validity.

Author Response

Reviewer #3

 

 

Comments and Suggestions for Authors

My comments have been taken into account.
The additions to the paper seem relevant.
The question remains whether an experience (experiment) lasting only one year can have scientific validity.

Thank you very much for your comments.

The authors think that the teaching activity based on the gamification methodology applied and the experience acquired in this work brings a different approach and may be interesting for other lecturers.

Round 3

Reviewer 1 Report

Dear Authors,

Thank you for all your corrections.

Regarding the last point, I'd like still to see a little change in the conclusion section:

r 492 : The results were positive, in terms of both the learning process and passing the subject.

mentioning that it was actually student's engagement which was improved which resulted in more of them passing the subject

Author Response

Dear Authors,

Thank you for all your corrections.

Regarding the last point, I'd like still to see a little change in the conclusion section:

r 492 : The results were positive, in terms of both the learning process and passing the subject.

mentioning that it was actually student's engagement which was improved which resulted in more of them passing the subject

The authors would like once again to take the opportunity to thank the reviewer for your detailed comments, which have helped us to improve the quality of the paper and to address important details.

The conclusion section has been revised in order to include the reviewer’s comment.