School Reform: New Future-Ready Quality Outcomes and Proposed Measures

Round 1

Reviewer 1 Report

The authors have embarked on a commendable endeavor by leveraging the power of natural language processing to assess ideation quality in university student essays. Unfortunately, upon thorough review of the submitted manuscript, I have identified several critical concerns that, in their current state, compromise the robustness and validity of the presented work. The foundation of the study rests on a series of hypotheses that appear to be misaligned with established principles in the field of learning analytics:

1. The first hypothesis postulates that ideation correlates with the number of ideas, represented as pairs of words. However, the complexity of an idea often extends beyond mere word pairs, making this assumption overly simplistic.

2. The second hypothesis suggests that ideation is gauged through word pairs derived from the title and context. This overlooks the possibility that ideation can manifest solely within the context, without any reference to the title.

3. The third hypothesis equates near-to-one cosine similarity values with divergent ideas. This is counterintuitive, as a cosine similarity value closer to one typically indicates semantic similarity, which is more indicative of convergent rather than divergent ideas.

4. The fourth hypothesis assigns a higher weight to convergent ideas compared to divergent ones. This contradicts the literature (lines 75-80), which underscores the importance of both convergent and divergent ideas in creative problem solving.

5. The fifth hypothesis aims to filter out "noise" by eliminating word pairs with a cosine similarity below 0.3, attributing these to pronouns or articles. However, in data science, such issues are typically addressed by removing stop words, rendering this hypothesis redundant.

Furthermore, the manuscript contains instances where definitions are provided without appropriate citations (e.g., digital literacy on line 566). Additionally, there are claims made without statistical backing, such as the comparison of an entire distribution against the extreme values of another (line 444) and the questionable representation of a small sample size in a box plot (Fig. 8).

Given these concerns, I believe that the manuscript requires substantial revisions to meet the rigorous standards of publication in our journal. I recommend that the authors revisit their hypotheses and validate their claims with appropriate statistical methods.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Please refer to attached file

Comments for author File: Comments.pdf

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

If Table 1 is divided over two or more pages, the column headings should be rewritten at the top of each new page.

The entries under Cosine Similarity in Table 2 should be written as regular double-sided inequalities:  0.60 < cosine <= 1,  0.45 < cosine <= 0.60, 0.30 < cosine < = 0.45.

Examples of how the vectors for words, and the cosine between a word pair is computed would be useful.

Put table 6 on one page - don't split it over two pages.  The same goes for table 7.

If someone filled a paper with the three words "leader", "governance", "accountability", it seems like they would receive a high ideation score without saying much in the process.

Words like "solutioning" and "operationalisation" should really be defined.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 4 Report

The paper has an exciting title and refers to general concerns related to higher education worldwide. Then, entering its content, you discover a new exciting approach, the one related to ideation and the generation of valuable ideas in the area of papers made by university students.

From the area of the abstract and the introduction, I was surprised and happy by this attempt to somehow quantify the process of generating ideas in the educational process and especially in encouraging students to generate ideas.

The documentation part is interesting, it appeals to several theories and approaches related to creativity, in general, solving problems creatively, the process of generating ideas and ways to analyze the quality in these types of processes. The approaches of Osborn from 1993, Isaken from 2011 and Mumford from 2018 are mentioned, so an appreciable period of research is covered with relevant extracts for each theory.

I cannot say that I have a very high competence related to the use of NLP-type technologies in the analysis of some texts. I understood that the advancement of technologies of this type and AI solutions are other elements that make text analyzes much more effective. But, precisely for this reason, I think that in the methodology part, the method of analysis can be detailed more, emphasizing the method in which the NLP-type solutions help in the analysis of the pairs of words analyzed and give them relevance.

The sample on which the research was carried out is not very significant, some results are predictable. The fact that the essays of doctoral students are better and include a more serious generation of ideas is to be expected, as they are in the position to produce doctoral theses that must bring serious new elements, while those from the master's degree are still in a process that does not have such a serious placement on the research side.

From the discussions and conclusions, it is not very clear how these researches can support a true reform in higher education (the title of the paper suggests this). That is why I think it is useful here in the discussions and conclusions to argue much more clearly what are the concrete contributions of this study on the elements of improvement, not to say reform, in the area of higher education.

I honestly believe that the approach is interesting, but there is still work to be done on the method of approach. And I'm really curious about the moment when a higher level of approach is reached, how it will be possible to qualitatively evaluate the generation of ideas in some papers (essays) on a given topic and their contribution to the creative solution of some problems (CPS - creative problem solving). This will have an impact on all educational levels, both on the didactic side and on the research side, both on the teaching side (teaching staff) and on the assimilation of skills (students).

From my point of view the English language is good, maybe some small improvements.

Author Response

Please see the attachment

Author Response File: Author Response.pdf