Effect of Game-Based Cognitive Training Programs on Cognitive Learning of Children with Intellectual Disabilities

Round 1

Reviewer 1 Report

The paper presents a study comparing "traditional" and AI-based training to enhance cognitive performance in people with disabilities. The results presented support the researchers' hypothesis. The paper is interesting and informative, but I feel the authors should work through their paper to fix some important issues. In particular:

 

1) Ethical aspects are not discussed (for example, will the control group receive the Neuro-World training after the end of the study?...). I think it'd be important to add a paragraph about this.

2) It's not clear if the neuro-world program has already been tested in previous studies about feasibility and efficacy in this population.

3) Inclusion and exclusion criteria should be clarified: use a clear schema. For example, what do the authors mean when they state that participants had to be naïf to AI training programs?

4) More information and details are needed about the traditional and new training program followed by the participants

5) It should be clarified ow the randomization was performed.

6) When reporting statistical data is better to be consistent in the use of notations. For example, in table X the * was used to mark a p<.05 while in table X+1 the * stands for p>.05. Be consistent in statistics presentation.

7) Did the authors collected feedback about the use of the Neuro-World from participants? Was it well accepted and easy to use? It should be important to know if participants in the two groups reported differences in experience with regard to satisfaction, emotional involvement, motivation, and so on.

8) Variables that were not expected to change could be excluded by the analysis (e.g., verbal understanding)

9) A paragraph discussing the limits of the study should be included

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

This paper addresses an important issue in the field of developmental disabilities, which is the effectiveness of cognitive rehabilitation through technology-based compared to conventional interventions. Even though an a-priori calculation of the sample size would help to understand the power of the results, the study enrolled a good number of children with developmental disabilities and adopted an interesting game-based cognitive training. Despite these aspects could be considered as strengths, in my personal opinion there is a number of (mainly methodological) major issues that undermine the validity of the results. My main concerns regard:

-the description of the sample – in the present form of the paper it is not clear which kind of developmental disabilities were presented by the enrolled children

-the procedure of group assignment and the blindness of the experimenters performing the pre- and post-training assessments. Since this study compares effectiveness of two different interventions, it would be helpful to consider the EQUATOR guidelines for clinical trials (CONSORT)

-the choice of outcome measures appropriate to the purpose of the study. I do not believe the WISC-IV and its subtests could be considered as outcome measure and, anyway, the authors should describe precisely which subtests constitute the outcome measures and why

- inadequate statistical analyses. Between-groups and within-group effects should be considered in the same model, as well as possible confounders/covariates

Here below I list a series of comments and suggestions that might help the authors to improve the paper.

Introduction

- First paragraph: Developmental disability is an heterogeneous group of behavioral and cognitive disorders due to  congenital alterations of brain structure or functions or acquired brain damage (e.g., cerebral palsy, TBI). Different types of developmental disorders present diverse neuropsychological profiles, with some abilities spared compared to others. I suggest that the authors better focus this paragraph according to the specific developmental disabilities presented by the recruited sample (see also the comment about Participants subsection).

-Page 2 line 64. This reference to mirror neurons comes a bit out of the blue, since the authors did not introduce this topic before. My suggestion is removing this sentence or adding a paragraph about how cognitive trainings might improve functionality of specific brain networks such as mirror neurons system.

-Page 2, lines 55-72. The literature cited here is not totally consistent with the main focus of the paper, which is cognitive rehabilitation of developmental disabilities. I suggest to add more information about previous studies that address this issue considering the specific diagnosis presented by the recruited sample (rehabilitation of acquired vs congenital developmental disorders is very different!).

- Page 2 line 81. The main aim of this work is comparing the effectiveness of the two trainings. Of course, a technology based program might be more interesting for children than a conventional cognitive training. Though, the authors did not collect data (or they did not present it in the manuscript) about children's motivation and interest for the game-based and conventional trainings. Thus, for sake of clarity I suggest to move this speculation to the discussion section.

- Still, it is not clear how the authors would verify the effectiveness, i.e., which specific cognitive functions would be improved by the trainings and how the authors would assess them. I suggest to better describe this aspect in the introduction and to explicit their hypotheses.

Materials and Methods

-This section in the present form is not well organized. I suggest splitting it into subsections (e.g., participants, procedure, trainings, outcome measures, analyses etc.)

- First paragraph. Although the authors report inclusion and exclusion criteria, the description of the sample is too generic. Which kind of developmental disabilities were recruited? Which was the mean IQ? I expect the authors would report a full description of clinical and socio-demographic information of the recruited sample. 

-page 2 line 97. A p < 0.05 actually means that significant differences were reliable for sex and age. Please clarify. Moreover, the authors should report the exact p- and t- and chi2 values, as well as an effect size measure (at least for significant comparisons).

-Table 1. In this form this table does not help readers to understand the characteristics of the sample. As stated above, I suggest to add clinical information about the diagnosis, the mean IQ, MRI report of brain damage/alterations if available, presence of other problems such as epilepsy, prematurity etc. For each numeric variable it would be useful adding a variability measure (i.e., standard deviation).

- Page 3 lines 102-103. Wisc-IV has been adapted and validated in each country where it was introduced. In this light, It is not clear the sentence “K-Wisc-IV aimed at theoretical foundation updates, increasing developmental suitability, improving psychometric properties, increasing convenience of tester, and enhancing clinical usefulness”. Please explain.  

-Page 3 lines 107-108. Why did the authors choose this age range? Which was the inclusion criterion for age? Please justify.

-Page 3 lines 110-114. Since in the first paragraph of the introduction the authors tell about impairments of specific functions, I would see outcome measures exactly mirroring these functions. I do not believe the WISC-IV is the correct instrument for providing such outcome measures.

-Table 2. Since it reports the standardized subtests included in the WISC-IV, this table is not necessary.

-Pages 3-4 lines 129-134. The Neuro-World items seem to address specific abilities, to which should correspond specific outcome measures. As highlighted in previous comments, I do not believe the WISC could provide such measures. if the authors sustain this view, they should provide a detailed explanation of which specific subtests of the WISC assess the functions addressed by the trainings, advancing specific hypotheses for each function.  

-Tables 3 and 4. I suggest to move the information contained in table 3 and in the caption of table 4 in the test, in a specific subsection describing the Neuro-world training.

-Page 5 lines 153-162. This paragraph is a bit confounding. I suggest the authors to maintain only essential information about level selection and AI analyses, avoiding description of activation procedure.

-Page 6. These paragraph are quite confusing as they report information about participants, the general procedure, the assessment etc. I suggest moving each information in the corresponding subsection.

-Page 6 lines 193-194. How did the authors calculate the sample size? Since there are several studies that have already investigated the effectiveness of technology-based cognitive programs, the authors should refer to this literature in order to calculate the correct sample size through a power-analysis (for example see Corti et al 2019 and Oldrati et al 2020 as reviews and meta-analyses) . Otherwise, they should state clearly that the sample size was not calculated a-priori.

-page 6 lines 194-204. How and when were participants assigned to the experimental or control groups? Which was the randomization criterion? Were the experts assessing the outcomes blind to the group assignment? Please clarify.

-Page 6 lines 205-212. The analysis are inappropriate for addressing the aim of the study. The authors should at least use a mixed-model ANOVA so as to consider both the within-group and the between-groups effects and their interaction. Moreover, other confounders such as age, specific diagnosis, general cognitive level etc. could undermine the analyses, and thus they should be controlled in some way (e.g., insert them as covariates, or considered them as stratification factors in group assignment).

Results

-The exact p and statistic (i.e., t value for Student's t-test, F for ANOVAs etc.) values should be reported for each result. Moreover, an effect size measure for each result is lacking and would help readers to understand the validity of these findings, also considering that the authors did not run an a-priori power analysis.

-Tables 5 6 7. These tables are redundant as they report information already described in the text.

Discussion

-First paragraph. I suggest avoiding to report numeric information (e.g. score increase/decrease).

-Page 9 lines 288-301. The possible associations between specific indexes of the WISC and the sub-items of Neuro-world should be previously introduced in the Methods section, so as to justify the chosen outcome measures.

-Page 9 lines 303-307. I think the authors should justify why they did not find improvements for PRI and VRI and/or why they excluded from this study the sub items (possibly) addressing these indexes. Also consider that previous research highlighted that PC-based cognitive programs are particularly effective in improving visual-spatial abilities, thus an improvement in PRI should be somewhat expected. I suggest the authors to better discuss these findings in light of previous literature.

-Page 9 line 308. I do not understand why speaking of this different, specific software in the discussion, since it was not previously introduced. Perhaps it would be useful move some parts of this paragraph in the introduction so as to justify why the authors chose Neuro-world and no other available trainings (e.g, Endeavor Rx, Lumosity etc.).

-Page 9 line 320. This paragraph should be moved to a specific subsection of Materials and methods.

- In general, a complete discussion of the results is lacking. I suggest the authors to stretch out this section by looking into other technology based or VR based programs for cognitive rehabilitation of children with developmental disabilities.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

Comments and Suggestions for Authors
It sounds interesting research in the effect of game-based cognitive training programs on cognitive learning of children with developmental disabilities. The research results have been validated by collecting children with developmental disabilities answers to the research. The contents are organized logically and presented properly. However, the experiment design and the result presented in this paper does not demonstrate innovation or originality. It hardly shows the novelty of this research. Anyway, the paper covers an interesting research topic and is easy to read. It seems that the children with developmental disabilities like the serious game and learn a lot from it. However, it has significant weaknesses:
I cannot find any new contribution. Serious game-based learning taught in many other learning courses. There are many papers about similar courses which are not cited. What is so special about this approach? It’s not clear what the authors wanted to evaluate, because there are no clear hypotheses / research questions / research goals. And it is lacking the research theory such as cognitive load theory and game-based learning theory to support this research cognitive assessment. It’s not clear if the evaluation is aligned with the learning goals of the cognitive training programs on cognitive learning of children with developmental disabilities.

My detailed comments are as follows:

1. The topic is novel, but the application proposed is not so novel.
2. It is noted that the research problem should be define, the goals and results of the study also should be clear to the reader.
3. The goals and results of the study are not clear to the reader.
4. In general, there is a lack of explanation of statistical methods such as ANOVA multiple comparison tests (Tukey's significant difference (Tukey HSD) test, Hochberg GT2 test, Gabriel test and Scheffé test) used in the study.
5. It lacks analytical methodologies to support author’s discoveries.
6. Furthermore, an explanation of why the authors did these various experiments should be provided.
7. Cognitive assessment and serious game-based learning needs to be presented by important reference for supporting it in this manuscript, but there are few important references was providing in this paper within 5 years.
8. There is nothing to explain about the research reliability and validity in this paper for your experiments.
9. Also, there are few explanations of the rationale for the study design.
10. The conclusions are overstated.
11. The author should accentuate his contributions in this manuscript.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

I feel the authors did a good job in revising their work and that the paper can be accpeted for publication

Author Response

Thank you for your comments. These comments have greatly helped improve the manuscript. Thank you again for your comments on approval for publication.

Reviewer 2 Report

The paper partially addresses my suggestions and comments and improvements have been made in introduction, discussion and in the general presentation of the interventions. Nonetheless, I still have concerns about the methodology and the statistical analyses, which, in the present form, prevent the authors to claim their conclusions. The major issue is that they did not change their statistical analyses as requested. In detail, now they use one-way ANOVAs model rather than t-tests but, since they are comparing two groups, the results are the same. The authors should at least use mixed-models ANOVAs so as to test for between- and within-group effects. Without these changes, I do not think that the results could be considered as reliable. A detailed description of my comments is reported below:

Introduction

• I appreciated the authors changed the first paragraph by adding more information about developmental disabilities. However, in this version of the manuscript it is clearer that the focus is on intellectual disabilities rather than developmental disabilities, which do not necessarily involve a general intellectual impairment. In this light, the authors may consider referring only to intellectual disabilities, in this section as well as in the title and in the whole paper. Moreover, since the diagnosis of intellectual disabilities is obviously secondary to assessment of the cognitive level, I do not understand why the authors refer to the diagnostic process in the abstract and in the introduction (page 2 line 55). My suggestion is to remove any references to diagnostic process for sake of clarity.
• Page 2, line 71. The authors cite cognitive therapy, which is a quite different intervention than a cognitive training and is out of the scope of the study. Please avoid referring to cognitive therapy in the text.
• Page 2, lines 95-96. In this form this sentence is not clear. Please add some information about the cited study to help readers understand why and how IQ improved.
• Now the introduction is consistent with the purpose of the study and with the results. However, there are still few references to previous studies about cognitive training of children with intellectual disabilities (there is a lot of literature on this topic!). My suggestion is removing references to cognitive training of elderly and post-stroke rehabilitation to focus on previous interventions with children with intellectual disabilities.
• The authors may consider adding their hypotheses at the end of the introduction, in order to help readers to better follow the results section.

Participants

• The authors now correctly report inclusion and exclusion criteria of the sample. For sake of clarity, I would make clearer that the children had a primary diagnosis of intellectual disability, and this was not secondary to acquired conditions (i.e., stroke, surgical resection of brain tumors etc.). Cognitive impairments and rehabilitative interventions are indeed quite different when dealing with congenital vs acquired disabilities.
• Here as well as in other part of the paper presentation of statistics is not consistent. In detail, when the result/comparison is not significant please use p > 0.05 or, better, report the exact value of p (see page 3 lines 135-136 as an example).
• Table 1. Demographic information is lacking in this table. Please report the number of male and female participants included in each group. I also suggest moving in this table the t/x2 and p values about the comparison between the two groups. Moreover, the IQ (called in the text evaluation score) is reported here as well as in the results section.
• It would be helpful to have other demographic information about the samples, particularly about familiar socio-economic status and parental education. These variables are indeed critically associated with intellectual functioning of children. If lacking this should be acknowledged in the limits section.

Measurement

• Somewhat related to the previous point, I have concerns that the FSIQ could be considered as the diagnostic factor and the main outcome measure. Rather than FSIQ, I suggest considering the performance in specific subtests of the WISC as outcome measures (as the authors reported in Table 2).

Interventions

• The authors now provide detailed information about the training. I only suggest the authors to specify which sub-items are included in each cognitive area.

Table 2

• The authors now report the matching between WISC subtests and Neuro-World subitems. This information could be useful to advance hypotheses in the introduction and to describe the specific outcome measures.

Procedure

• Page 6 line 257. It is not clear whether each subitem or each cognitive area was tested for 5 minutes (in the latter case the total amount of time would be 20 minutes).
• The authors added information about randomization, but they did not report whether the experimenters administering the pre- and post-training evaluations were blind to group assignment. This aspect should be acknowledged and eventually reported in the limits section.

Data Analysis/Results

• The authors now use one-way ANOVAS for both between- and within-group analyses. This procedure is incorrect, since there are only two groups so that there are no differences with the t-test analyses reported in the previous version of the manuscript. As I requested in the previous report, for each outcome measure the authors should adopt a mixed-model ANOVA so as to check for both between-groups and within-group analyses. Also consider that the comparison for the FSIQ at the pre-training evaluation is marginally significant, thus asking even for more caution in drawing conclusions about significant differences in this measure after the intervention.
• The authors still do not report any measures of effect size. Please report effect size for each statistic.
• Tables 3/4/5/6/7/8. Information reported in these tables could be easily resumed in one or two tables.
• It is not clear why the authors conducted and presented between-groups analyses only for the FSIQ and not for the indexes (PRI, VRI, WM, PS). As stated above, a mixed-model ANOVA would allow the authors to consider both between-groups and within-group effects in the same analysis.

Discussion

• Page 11 line 363. Why did the authors exclude perceptual reasoning trainings? Please explain and provide a rationale for this choice.
• Page 11 lines 373-374. “The degree of cognitive improvement can be higher than the current data”. This is a hypothesis of the authors that is not supported by current data. Please avoid this sentence or present it as hypothesis.
• What does this paper add to this research topic? This aspect is lacking in the discussion.
• Page 11 lines 375-376. Please add references to this statement.
• Page 11 lines 379-381. This sentence is unclear, please reformulate.
• Page 11 lines 382-383. The authors refer to motivational factors, but they do not assess these variables in the paper. This should better acknowledge as limit of the study.
• Limits section: There are more limits to be acknowledged (follow-up measures? Motivation? Familiar variables? Blindness? Etc.)

Author Response

Please see the attachment.

Author Response File: Author Response.docx