The Relationship between Learning Outcomes and Graduate Competences: The Chain-Mediating Roles of Project-Based Learning and Assessment Strategies

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for the opportunity to review this interesting manuscript.  I appreciate the importance of the topic, as well as the generally solid analyses and reporting of results herein.  Below are suggestions to improve the manuscript; the first two are denoted with asterisks to signify their centrality. 

* When I read the abstract and the start of the manuscript, I anticipated that AGC would be measured in an objective manner as a manifest variable (such as a portfolio, exam, or some other culminating product).  Instead, this latent variable reflects students' perceptions of how achieved competency (especially through the OBE process) could affect their workplace readiness/placement (Appendix 1, Section 2).  Importantly, to what extent can students realistically judge what competency and eventual workplace readiness would look like when addressing this construct?  In addition, were they well positioned to understand skills sought by employers and whether OBE is a desirable mechanism compared to other options to obtain those skills?  It is possible that overly optimistic assumptions were applied as to what students were able to provide on the survey.  Relatedly, the authors noted in the abstract that: "This study investigated how Outcomes-Based Education (OBE) influences the attainment of graduate competence (AGC) in China’s higher vocational education, using the theory of Constructive Alignment (CA) as its foundation." However, the OBE model tested is based on students' perceptions of various latent concepts depicted in Figure 1.  For a more comprehensive understanding of the OBE theoretical framework, it would seem prudent to incorporate instructors, program designers, program evaluators, and possibly administrators.  There should also be an accounting for "hard" competency outcomes (as well as certain course-level variables to examine whether courses deviated from program aims).  Clearly, this study cannot address all these wrinkles, but there should be more discussion of the strengths and weakness of choices made for the sake of a manageable survey and reasonable scope.     

* Students responded to questions pertaining to ILO, PjBL, and AS in an overall fashion, i.e., across many instructors, courses, and advising experiences in their program.  Based on survey design practice, "averaging" can introduce both increased cognitive load and lead to various types of bias, including recency bias and satisficing strategies.  Further, there were specialized terms in survey questions that students may not be entirely familiar with, such as "intended learning outcomes," "rubrics," "assessment criteria," and "assessment tasks."  Teachers and administrators may take these for granted, but will students interpret and apply them similarly when responding to this standardized instrument?  More to the point, were these terms defined for students when there was a "...brief introduction of OBE guidelines was included at the head of the questionnaire to enhance understand of OBE and assess its effect on attaining graduate competence" (Lines 291-293)? 

Abstract and throughout the rest of the text: Consistently use the acronym "AS" for Assessment instead of the three-letter version (see Lines 422 and 688 for unfortunate uses of the 3-letter acronym).  Notably, the authors used "AS" at the bottom of p. 26.

Line 42: I do not believe that TLA was defined prior.    

Figure 1: The authors could refer to a theoretical model rather than the more abstract OBE "framework."   

Lines 253-254: The authors write: "Informal interviews were conducted to validate the responses of the participating CBEC learners."  What were these interviews comprised of?  Did they serve as a qualitative pretest?  If so, what types of questions were posed and were any changes made to the questionnaire from insights gleaned from students?  Or were they conducted after the surveys to validate specific students' responses?  How many interviews were conducted? 

Line 254: A retrieval rate of 94% is impressive.  Were students incentivized to complete the survey, perhaps within their own institutions?

Section 3.1, Lines 256-257: There is mention that "the researchers will seek permission from the Human Research Ethics Committee..." This future tense is used in regard to consent throughout the entire paragraph.  I assume that the past tense was simply not updated in this manuscript version (and that the study was indeed reviewed and approved by the four deputy deans at the institutions and the USM prior to carrying out data collection and other aspects of the study).

Section 3.3: There are multiple mentions that specific statistical procedures "will be" carried out.  Obviously, the tense should be changed to the past.

Table 2 and accompanying text: Some readers may also wish to see reporting of McDonald's omega.  [Of course, the reliabilities were so stout with alpha, they probably won't differ much when using omega.]

Line 344: This heuristic applies to *standardized* factor loadings above 0.5.

Did the data meet the assumption of multivariate normality for maximum likelihood estimation (MLE)?  If not, were the authors concerned about potentially biased coefficients, hypothesis tests, and goodness of fit statistics?  Was the use of bootstrapping to compensate for biases stemming from non-normality (i.e., to yield bias-corrected standard errors and confidence intervals)?  If so, why not use bootstrapping for the CFA model in Table 2 as well as the mediation analysis in Table 6? 

Table 2: Since the authors rely on standardized coefficients in the text, unstandardized coefficients might be dropped for the sake of parsimony in this table.  At a minimum, standardized information might be introduced in the leftmost column rather than appearing so far to the right.

Goodness of fit statistics should also be presented for the CFA. 

Lines 415-417: The authors write: " The combined influence of ILOs and PjBL explained 25% of the variance in ASs, with coefficients of 0.290 and 0.295, respectively, suggesting a positive association with AGC."  The combined influence would be positive on AS instead of AGC.  

Lines 425-426: MLE in the face of multivariate non-normality does not imply robust results (but the bootstrapping would help).  Did the authors consider application of the Satorra-Bentler Adjustment toward robust MLE?  It would also yield some adjusted goodness of fit statistics.  

Table 6 is confusing to me.  What does "Distal" in the leftmost column signify?  Why not combine Tables 5 and 6?  [In Table 5, I would use "Not supported" instead of "Rejected" per the decision on H6.]

Lines 606-609: The authors write: "...this study utilized Bloom’s Taxonomy to craft clear and constructively aligned learning outcomes, clarifying what learners are expected to achieve by the end of a study program, the anticipated standard of achievement, and how they should demonstrate their learning." How was Bloom's Taxonomy used here?  I do not see much from Bloom with respect to specific ILO survey items. 

Comments on the Quality of English Language

The quality of language in the manuscript is quite good.

Author Response

Dear distiguished peer reviewer,

Thanks for valuable feedback! We will provide a point-by-point response to your valuable comments and upload it as a Word file. Please see the attachment. 

Warm regards, 

Ming Li & M.I.Rohayati

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

1. First of all, I would like to note how well does this manuscript correspond to the topics of the Sustainability magazine? From my point of view, the journal Education Sciences is closer to this manuscript. Authors should think about this. And if the authors decide that the journal Sustainability is closer, then this should be clearly highlighted, for example, in the annotation or by adding an appropriate figure

2. Secondly, the introduction practically does not justify the relevance in any way. It is very difficult to understand what is the problem? Why did the idea arise to use this particular approach in education? Is this specific to a particular university or college, or to a particular country or region? Authors should provide some case or data that substantiates the relevance and novelty.

3. It is not clear from the study how it relates specifically to e-commerce? Are the results applicable to other educational programs?

4. The discussion and conclusions are very vague. It is unclear what practical value this will provide for the educational process. What specific recommendations do the authors give for the use of these educational approaches?

Overall, the conceptual problem is the lack of a specific research objective. This also results in the absence of specific conclusions that should have been obtained.

Author Response

Dear Distinguished Peer Reviewer,

Thank you for your valuable feedback! I will provide a point-by-point response to your comments and upload it as a Word file. Please see the attachment.

Warm regards,

Ming Li & M.I. Rohayati

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

We can definitely agree with the authors that exploring new approaches to tertiary education is a globally debated topic, fuelled by a growing demand for graduates with new competencies. Unfortunately, there is a general consensus that in recent years there has been a 'skills gap' between labour market requirements and graduate preparedness. I consider the topic of the paper to be topical, with the potential to stimulate a broader academic discussion on possible solutions. The paper has a logical structure and the authors use correct methods. The content is balanced, relies on relevant sources and provides an interesting insight that can be useful in practice as well as for further academic discussion. However, I have some comments, that will contribute to improving the quality of the study. It is worth noting that in terms of OBEs' application within degree programmes, their highest level of demand is not the same across the existing range of degree programmes. The initial theoretical background gives the impression that the content of the education is less important and than the motivation of the students. The fundamental changes in the labour market mentioned by the authors due to digital transformation and automation also require a fundamental change in the structure of educational content as well as curricula, which must go hand in hand with the changes in approaches to education (the OBEs mentioned above).   It is also necessary to mention the changes in labour demand and preferred areas of tertiary education, especially in STEM education. An example from Europe is described e.g. by Nurkovic, R. (2020). STEM education in teaching geography in Bosnia and Herzegovina. Folia Geographica, 62(1), 127-141. There is also the question of the extent to which students' personal backgrounds and individual intellectual aptitudes influence the success of OBEs in achieving learning outcomes. For example, in the UK, Dr Mark Payne has addressed this issue (Payne, M. (2018). Slovak Roma village of origin and educational outcomes: A critical evaluation. Folia Geographica, 60(1), 31-49), drawing on Bronfenbrenner's (2005) 'Process-Person-Context-Time' (PPCT) to describe some of the experiences of Slovak Roma children in secondary school as they negotiate prevailing English-only language ideologies and complex curriculum challenges. Within the limitations of the study, the problem of assessing the achievement of set goals through uniform rubrics, which often leads to formalism, should also be mentioned.

The above comments do not reduce the quality of the study. In any case, this is a very topical issue that is currently being hotly debated, and many countries around the world are taking steps to implement changes in tertiary education. Therefore, I consider the empirical research of this study to be a valuable source of experience from China that will help in adjusting changes in educational policies so that secondary school graduates have adequate skills that are in demand in today's labour market. It is definitely a thought-provoking study, that has the potential to appeal to professionals once the above comments have been reflected. Once the comments have been included, the likelihood of academic responses will be increased by broadening the perspectives for further investigation to include new lenses on the necessary changes in tertiary education.

I recommend the study for publication after revision.

Author Response

Dear Distinguished Peer Reviewer,

Thank you for your valuable feedback! I will provide a point-by-point response to your comments and upload it as a Word file. Please see the attachment.

Warm regards,

Ming Li & M.I. Rohayati

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

Dear authors,

thank you for your great and useful work.

My positive feedback and my tips are in the app.

Sincerely,

the reviewer

Comments for author File: Comments.pdf

Author Response

Dear Distinguished Peer Reviewer,

Thank you for your valuable feedback! I will provide a point-by-point response to your comments and upload it as a Word file. Please see the attachment.

Warm regards,

Ming Li & M.I. Rohayati

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors' careful responses and revisions in version 2 of the manuscript addressed my suggestions and should provide clarification for the reader.    

One aspect from from my initial review should be worked further.  The authors tested univariate normality for each of the 24 items but did not test the *multivariate* normality assumption implicit with maximum likelihood estimation (across all joint distributions involving the 24 variables) or even bivariate normality (2 items at a time).  Approximately univariate normal distributions do not guarantee multivariate normality (but are an essential element toward it).  Multivariate kurtosis can be readily tested in Amos using the Mardia's coefficient for kurtosis and examining its critical ratio.  Non-normal kurtosis is especially damaging for CFA/SEM under MLE since kurtosis (in particular) can impact estimation of variances and covariances (see Byrne, Barbara B. 2016. Structural Equation Modeling with Amos: Basic Concepts, Applications, and Programming, 3rd edition. New York, NY: Routledge.) 

There is inconsistent use of "Appendix A" and "Appendix 1".

For ILO 2 in Appendix A, was the newly added phrase, "written using the verbs recommended in Bloom’s taxonomy," actually shown to students on the survey?  Or does it serve as a parenthetical remark to the manuscript reader only?

Comments on the Quality of English Language

Very good work here.

Author Response

Dear peer reviewer,

Thank you once again for your insightful comments, which have greatly helped improve our study. We have attached a point-by-point response to address your concern. Please see the attachment.

Warm regards,

Ming Li, M.I. Rohoyati

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Overall, the authors did a great job responding to my comments and provided detailed comments to my questions. The presented changes to the manuscript more clearly reflected the essence of the study. However, I am still not convinced that this study fits the scope of the Sustainability journal, despite the fact that the authors tried to allay my concerns.

Author Response

Dear peer reviewer,

Thank you once again for your insightful comments, which have greatly helped improve our study. We have attached a point-by-point response to address your concern. Please see the attachment.

Warm regards,

Ming Li, M.I. Rohoyati

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

Dear Colleagues,

I am delighted with the quantity and quality of the authors' work to improve the manuscript. From time to time, your actions and results have exceeded my expectations. I deeply apologize for the typo in my revision-1. Page 239 should be written instead of page 139 (comment 4).

 

Overall, it looks to me that the manuscript is ready for publication.

 

However, two doubts remained:

1. I agree with you that hypotheses, their appearance and development are placed in the section "Literary review".

However, the Sustainability-template recommends mentioning them in the Introduction section. This is about where you put the "research questions".

Considering that you have a large number of hypotheses, perhaps the situation can be left in this state. On the other hand, if every author breaks the rules, then soon we will start living without rules.

I will inform the editor about the situation.

 

2. There are several inaccuracies in your answer when you refer the reviewer to the answer in the corrected version of the manuscript:

- Response 5: (See paragraph 3 on page 18 of the updated text),

- Response 6: (See paragraph 3 and Table1 on pages 19-20 of the updated text),

- Response 8: (See paragraph 1 and Table 1 on page 25 of the updated text),

- Response 9: (See Discussion on pages 32-38 in the updated text),

- Response 11: (See paragraph 1 on page 41 in the updated text).

Please note that the corrected text has 30 pages.

 

I found the corrected fragment for answer 5 on page 8 in subsection 3.3.

Please provide the correct page and line numbers where the corrected fragments are listed (answers 6, 8, 9, 11). 

 

 

Best Regards

Author Response

Dear peer reviewer,

Thank you once again for your insightful comments, which have greatly helped improve our study. We have attached a point-by-point response to address your concern. Please see the attachment.

Warm regards,

Ming Li, M.I. Rohoyati

Author Response File: Author Response.pdf

Round 3

Reviewer 4 Report

Comments and Suggestions for Authors

Dear co-authors,

thank you for your work in improving the manuscript.

Although your letter did not result in replies to my comments, my experience helped me find corrected passages.

 

Everything is in okay.

Good luck.