Research Contribution of the Oldest Seat of Higher Learning in Pakistan: A Bibliometric Analysis of University of the Punjab
Round 1
Reviewer 1 Report
Comments on: Ahmad, Javed, Shahid and Khahro Research Contribution. of the oldest seat…
Overall Comments
This paper was a very readable bibliometric account of research outputs from the University of the Punjab. The account is framed by some discussion of the role of colonial education in Pakistan, and the geo-political changes which are taking place in the region. The paper is engaging, even if you do not have a specific interest in this specific institution, and the comparisons with other universities give the paper a bit of depth.
The bibliometric methods used here are fairly standard, and the paper is not methodologically innovative. However, it does add to a growing body of bibliometric work which looks at single institutions.
The submission needs to be reviewed by a competent sub-editor: there are a number of places where the English is not quite correct, but on the whole the submission is clear and well-argued. I disagree with some of the conclusions, which struck me as over-simplistic. These disagreements are noted in the detailed comments that follow.
Overall, I would recommend the paper might be published in Publications after some necessary
improvements are made. But Editor might like to consider whether a complete analysis from 1906
to present is the best way to present this data. Given that most of the relevant work (92%) was
published since 2000, it might be better to treat the historical data separately. Editor might also like
to consider whether the paper is mainly of interest to readers in Pakistan, and would be better placed
in a local journal than an international one.
Detailed Comments
These notes do not include places where the English needs to be tidied up.
Title:
It might be worth adding that the “Oldest Seat of Higher Learning in Pakistan” is the University of
the Punjab.
Abstract:
I am unsure how good Scopus is for bibliometric work in Pakistan. Authors seem to be aware of the
limitations - e.g. that Scopus does not record papers published in Urdu - but they don’t seem to have
corrected for this, or taken this limitation into account in their analysis. I am unclear whether
Universities in Pakistan have a mandate to publish in Urdu. I am also unclear whether the very large
number of Chinese collaborators reported towards the end of the paper is an indication that UoP has
a large output that appears in Chinese.
ln 44
Nalanda, Taxila and Vikramshila will be unfamiliar to most readers, and this brief reference does
scant justice to the importance of these centres. Taxila, for instance was operational hundreds of
years BC .Where are they, and why did they act as learning centres? What did they teach?
ln 74
Why is it “imperative” to know this? Is there a political agenda here?
ln 85
It’s not obvious how you would make this comparison.
ln 106
The humanitarian issues … I think this might be a reference to research in the Humanities?
If so, then the text implies that Authors think research in Humanities might not be comparable with
research in other areas. Is this intended?
ln 114
ICT needs to be spelled out the first time it is used.
ln 165
Is this a standard list of features, or just an arbitrary set?
Figure 1
This is a horrible chart! The columns overlap, so it is very difficult to see what is going on.
Personally, I would present this data as five separate charts, one for each university. I would also fix
the x-axis so that the dates are more intuitive, and the gaps are consistent.
Given that 92% of the papers appear after 2000, there is a very strong case for reporting the data in
two parts - publications pre-2000 and publications post-2000. I think this would show that the
relative status of the universities changes. UoP may be the oldest institution in Pakistan, but is was
not very productive in the 1906-1995 period. (Why not?).
ln187
When was HEC established? Does it have an explicit policy on research output?
How does the growth in research outputs compare with the growth in other countries? i.e. is this
growth a phenomenon that is particularly marked in Pakistan?
Figure 2
Research output from Baluchistan is clearly much lower than the other comparators. Does this mean
that it is NOT a useful comparison? Why does the Carnegie list think it is?
Figure 3
Why is this feature considered to be a relevant comparator?
Does HEC have a policy on open access journals?
Note that Open Access journals were not a feature of the earlier research scene, so the difference
only applies to about half of the data.
Figure 4
This too is a horrible chart!
What is the X-axis? Number of publications? (See my previous comment about treating the early
research separately.
Personally, I would rework this chart as five separate charts, one for each institution.
Also, I would re-order the subject areas so that similar subjects appear close together: e.g. Nursing,
Health Professions, Dentistry, Pharmacology and Medicine are clearly related, but the alphabetical
ordering artificially separates them.
Some of the figures are surprising. For example, Arts and Humanities account for a miniscule
proportion of the data set.
Figure 5:
Why is “preferred document type” considered to be an important factor?
Note that UB is clearly out of line with the other institutions. Why?
Figure 6 and 7 were really interesting, and I would have liked to see more discussion of these data.
For example, which areas generate these collaborations? Do the collaborations between UP and
other institutions show the same collaboration patterns as the comparator institutions, or is there
something special about UP in this regard? Does HEC encourage this type of collaboration?
Ln 238
Noble Prize awards should be Nobel Prize awards.
Figure 8
Another figure where it would make more sense to split the information into two charts.
Why is the number of authors per paper considered to be an important feature?
See my previous comment about treating the early research separately from the later research.
ln279
Is this a genuine result, or just an artefact? If all authors typically cite a paper that they write, then a
paper with six authors will typically generate more citations than a paper with only a single author.
An average of 12 citations means that each author is citing their own work only twice. In fact, if
single author papers are typically cited 6 times, then you might expect at least 36 citings for a sixauthor paper. This calculation makes 12 citations look somewhat feeble.
Note too that citation practice has change over the years.
ln 287
Open Access publications have only been around for a few years, so this analysis should not be
applied to the entire data set. In any case, the difference between 12 and 13 citations is clearly not
significant. Personally, I would drop this analysis.
However, it might be interesting to look in more detail at some of the outliers. How many papers
are cited more than a few hundred times, and which fields do they belong to?
In general, what is missing here is any analysis of the strengths of UP. It would be interesting to
know what fields the university is considered to excel in, and whether the publication data reflect
this status.
Figure 9
Personally, I would delete the small funders from this pie chart, or group them under an OTHER
heading. Is there a formal definition of a funding sponsor?
ln334
I find this recommendation to be a bit facile. (See my comments on the citation rates of multi-author
papers.) There are lots of reasons my working with colleagues is a good idea, but doing this with an
eye to increasing the citation rate of a paper is not one of the better ones.
ln339
This recommendation is more interesting, but it’s not actually supported by the data reported here.
I think authors need to do smaller scale evaluations of this idea.
ln 347
Prince Sultan University is in Riyadh, Saudi Arabia.
Why are they funding this work?
Author Response
Dear Reviewer
We truly acknowledge your feedback and have strived to address your all valuable comments.
Response to Comments of Reviewer One
|
Comment No |
Comment Description |
Authors’ Response |
|
1 |
Title should describe PU |
A sub title has been added to reflect University of the Punjab |
|
2 |
Ln 347
Why PSU is funding such a research |
The funding statement has been modified to no funding available. Thanks for pointing it out
Funding: This research is not funded by any funding agency.
|
|
3 |
Abstract:
I am unsure how good Scopus is for bibliometric work in Pakistan. Authors seem to be aware of the limitations -e.g. that Scopus does not record papers published in Urdu -but they don’t seem to have corrected for this, or taken this limitation into account in their analysis. I am unclear whether Universities in Pakistan have a mandate to publish in Urdu. I am also unclear whether the very large number of Chinese collaborators reported towards the end of the paper is an indication that UoP has a large output that appears in Chinese. |
Thanks for such a worthy and detailed review. This research focuses on scientific publications that are indexed in Scopus that is one of the largest databases. Authors are obliged to reviewer and have modified the statement as “one of the to the largest databases” as there are other major sources also. Secondly, Urdu is national language of Pakistan but very limited scientific literature is published that is usually in form of editorials, newspaper articles, locally published magazines and university newsletters but these contributions are based on pre-published literature usually translated into Urdu language. Universities in Pakistan don’t have mandate to publish in Urdu but all universities registered with HEC (Higher Education Commission -Pakistan) have mandate and are evaluated based on publications made in ISI and Scopus indexed journals. There are very rare publications in Chinese language and this boost with Chinese authors collaboration is due to Pakistani PhD students studying in China. |
|
4 |
ln 44 Nalanda, Taxila and Vikramshila will be unfamiliar to most readers, and this brief reference does scant justice to the importance of these centres. Taxila, for instance was operational hundreds of years BC . Where are they, and why did they act as learning centres? What did they teach? |
Yes, Its good suggestion. The sentence is deleted to avoid confusion. |
|
5 |
ln 74 Why is it “imperative” to know this? Is there a political agenda here? |
University of the Punjab is the oldest institute of higher education in Pakistan. Further, it is the largest university in the country and one of the largest universities in the region. So, it is significant to analyse the research performance of such a premier institute to know its research strengths and weaknesses to provide empirical evidence based on the findings of this research. |
|
6 |
ln 85 It’s not obvious how you would make this comparison |
The following criteria were used to select similar universities for comparison: · The selected university is in the “public” sector category. · The selected university is registered in the “general” category of subjects and fields. · The selected university is situated in other provinces/states of the country. · The selected university is governed by a similar provincial/state administration and has similar funding sources. · The university selected from outside the country must have been established in the same period by the same rulers of the country as of the University of the Punjab.
|
|
7 |
ln 106 The humanitarian issues … I think this might be a reference to research in the Humanities? If so, then the text implies that Authors think research in Humanities might not be comparable with research in other areas. Is this intended? |
The reference is provided to highlight the wide spread application of bibliometric method of research evaluation in different fields and subject areas. The authors are not intended to make comparison of Humanities with other research areas. |
|
8 |
ln 114 ICT needs to be spelled out the first time it is used |
Abbreviation added. |
|
9 |
ln 165 Is this a standard list of features, or just an arbitrary set? |
Yes, it is arbitrary set. |
|
10 |
Figure 1 This is a horrible chart! The columns overlap, so it is very difficult to see what is going on. Personally, I would present this data as five separate charts, one for each university. I would also fix the x-axis so that the dates are more intuitive, and the gaps are consistent. |
Figure 1 is revised. The data has been separated. The publications data of studied universities is categorized in two groups; before and after year 2000. |
|
11 |
Given that 92% of the papers appear after 2000, there is a very strong case for reporting the data in two parts - publications pre-2000 and publications post-2000. I think this would show that the relative status of the universities changes. UoP may be the oldest institution in Pakistan, but is was not very productive in the 1906-1995 period. (Why not?). |
Very valuable suggestion. The research output has been segregated into two parts as suggested. |
|
12 |
ln187 When was HEC established? Does it have an explicit policy on research output? How does the growth in research outputs compare with the growth in other countries? i.e. is this growth a phenomenon that is particularly marked in Pakistan? |
HEC was established in 2002 and yes, the HEC do have research policy which is considered in HEC university ranking. Yes, HEC marked phenomenal impact in higher education and research output of Pakistani Universities. |
|
13 |
Figure 2 Research output from Baluchistan is clearly much lower than the other comparators. Does this mean that it is NOT a useful comparison? Why does the Carnegie list think it is? |
BU is a state university and met the criteria made for the selection of the peer universities, therefore, it was part of comparison. |
|
14 |
Figure 3 Why is this feature considered to be a relevant comparator? Does HEC have a policy on open access journals? Note that Open Access journals were not a feature of the earlier research scene, so the difference only applies to about half of the data. |
Yes, HEC do have open access policy and the reviewer is right that this applies after HEC establishment. |
|
15 |
Figure 4 This too is a horrible chart! What is the X-axis? Number of publications? (See my previous comment about treating the early research separately |
Figure 4 is revised. |
|
16 |
Personally, I would rework this chart as five separate charts, one for each institution. Also, I would re-order the subject areas so that similar subjects appear close together: e.g. Nursing, Health Professions, Dentistry, Pharmacology and Medicine are clearly related, but the alphabetical ordering artificially separates them. Some of the figures are surprising. For example, Arts and Humanities account for a miniscule proportion of the data set. |
The Figure is revised as per suggestions. The subject categories have been grouped into relevant subject areas. |
|
17 |
Figure 5: Why is “preferred document type” considered to be an important factor? Note that UB is clearly out of line with the other institutions. Why? |
The authors intended to know the trend of type of documents that the researchers choose to publish their research in selected universities. BU output is as per the data. |
|
18 |
Figure 6 and 7 were really interesting, and I would have liked to see more discussion of these data. For example, which areas generate these collaborations? Do the collaborations between UP and other institutions show the same collaboration patterns as the comparator institutions, or is there something special about UP in this regard? Does HEC encourage this type of collaboration? |
Additional discussion of figure 6 and 7 have been added as suggested. Description about local and international collaborating institutions has been expanded. |
|
19 |
Ln 238 Noble Prize awards should be Nobel Prize awards. |
The spelling is corrected and verified. |
|
20 |
Figure 8 Another figure where it would make more sense to split the information into two charts. Why is the number of authors per paper considered to be an important feature? See my previous comment about treating the early research separately from the later research. |
The number of authors per publications were analysed to determine the authorship patterns that is an important indicator in bibliometrics studies. Further, it was done to see the relation between the level of collaboration and received citations. |
|
21 |
ln279 Is this a genuine result, or just an artefact? If all authors typically cite a paper that they write, then a paper with six authors will typically generate more citations than a paper with only a single author. An average of 12 citations means that each author is citing their own work only twice. In fact, if single author papers are typically cited 6 times, then you might expect at least 36 citings for a six-author paper. This calculation makes 12 citations look somewhat feeble. Note too that citation practice has change over the years. |
It is observed that higher author collaboration leads in higher citations from other scholars. The self-citation is not specifically observed neither it was in the scope this research. |
|
22 |
ln 287 Open Access publications have only been around for a few years, so this analysis should not be applied to the entire data set. In any case, the difference between 12 and 13 citations is clearly not significant. Personally, I would drop this analysis. However, it might be interesting to look in more detail at some of the outliers. How many papers are cited more than a few hundred times, and which fields do they belong to? In general, what is missing here is any analysis of the strengths of UP. It would be interesting to know what fields the university is considered to excel in, and whether the publication data reflect this status. |
It is indeed a good observation, however, due to paper scope and data limitation it is difficult to consider. The authors plan to conduct a separate study to address it. |
|
23 |
Figure 9 Personally, I would delete the small funders from this pie chart, or group them under another heading. Is there a formal definition of a funding sponsor? |
Figure 9 is modified as per worthy suggestion. |
|
24 |
ln334 I find this recommendation to be a bit facile. (See my comments on the citation rates of multi-author papers.) There are lots of reasons my working with colleagues is a good idea, but doing this with an eye to increasing the citation rate of a paper is not one of the better ones. |
The recommendation is rephrased in line with the suggestion. |
|
25 |
ln339 This recommendation is more interesting, but it’s not actually supported by the data reported here. I think authors need to do smaller scale evaluations of this idea. |
Yes, indeed good suggestion. The recommendation is generic so it is deleted. |
|
26 |
ln 347 Prince Sultan University is in Riyadh, Saudi Arabia. Why are they funding this work? |
Yes, the research is not funded by any agency, therefore the funding section is modified and PSU is added in the acknowledgments as it provides access to Scopus database used in the study. |
Reviewer 2 Report
The manuscript "Research Contribution of the Oldest Seat of Higher Learning in Pakistan: A Bibliometric Analysis" compares research output and citation impact of the University of the Punjab with four other universities in the region.
"The University of the Punjab leads the compared peer universities in the number of research publications and the citations received on these publications." Because it is the oldest of the compared universities? Citations accumuulate with time.
"Scopus is the largest indexing database of peer-reviewed literature." (lines 123/124 and abstract) Actually Dimensions and Microsoft Academic are larger.
The following paper should be cited when Scopus is described:
https://www.mitpressjournals.org/doi/full/10.1162/qss_a_00019
The following paper provides a nice overview of the major bibliographic databases:
https://arxiv.org/abs/2005.10732
Bar chart in Figure 1 is barely readable. Are the bars overlapping?
Percentages might be more helpful in Figures 3 and 5 than absolute values.
Legend in Figure 8 is confusing. Axis labels could help understanding the figure, too.
Impact is evaluated solely on raw citations without time or field normalization.
Does the National University of Sciences and Technology, Islamabad, fulfill the criteria to be a peer university of the University of the Punjab? If so, some normalized indicators from the Leiden Ranking could be used to compare them:
https://www.leidenranking.com/Ranking/University2020?universityId=18066&fieldId=1&periodId=10&fractionalCounting=1&performanceDimension=0&rankingIndicator=pp_top10&minNPubs=100
https://www.leidenranking.com/Ranking/University2020?universityId=9580&fieldId=1&periodId=10&fractionalCounting=1&performanceDimension=0&rankingIndicator=pp_top10&minNPubs=100
It is very hard to associate the funding bodies from the legend in Figure 9 with the pie slices. A table might be more informative.
Author Response
Dear Reviewer,
Response to Comments of Reviewer Two
|
Comment No |
Comment Description |
Authors’ Response |
|
1 |
The manuscript "Research Contribution of the Oldest Seat of Higher Learning in Pakistan: A Bibliometric Analysis" compares research output and citation impact of the University of the Punjab with four other universities in the region. |
The title has been modified as suggested. |
|
2 |
"The University of the Punjab leads the compared peer universities in the number of research publications and the citations received on these publications." Because it is the oldest of the compared universities? Citations accumulate with time. |
Yes, valid comment but the discussion in the paper is based on the data. |
|
3 |
"Scopus is the largest indexing database of peer-reviewed literature." (lines 123/124 and abstract) Actually Dimensions and Microsoft Academic are larger. |
Yes, the sentence is changed. |
|
4 |
The following paper should be cited when Scopus is described: |
Yes, good suggestion. The citation has been added. |
|
5 |
The following paper provides a nice overview of the major bibliographic databases: |
The citation has been added. |
|
6 |
Bar chart in Figure 1 is barely readable. Are the bars overlapping? |
Figure 1 is revised as suggested. |
|
7 |
Percentages might be more helpful in Figures 3 and 5 than absolute values. |
Thanks for suggestion. The figures 3 and 5 are revised. |
|
8 |
Legend in Figure 8 is confusing. Axis labels could help understanding the figure, too. |
Axis Labels have been added. |
|
9 |
Impact is evaluated solely on raw citations without time or field normalization. |
Indeed, good observation but it is done on generic basis due to data limitation and paper scope. |
|
10 |
Does the National University of Sciences and Technology, Islamabad, fulfill the criteria to be a peer university of the University of the Punjab? If so, some normalized indicators from the Leiden Ranking could be used to compare them: |
The following criteria were used to select similar universities for comparison: The selected university is in the “public” sector category. The selected university is registered in the “general” category of subjects and fields. The selected university is situated in other provinces/states of the country. The selected university is governed by a similar provincial/state administration and has similar funding sources. The university selected from outside the country must have been established in the same period by the same rulers of the country as of the University of the Punjab.
Based on the above criterion, National University of Science and Technology, Islamabad does not fall in the list of selected peer universities. |
|
11 |
It is very hard to associate the funding bodies from the legend in Figure 9 with the pie slices. A table might be more informative. |
Figure 9 is redesigned |
Round 2
Reviewer 1 Report
I confirm that authors have addressed the issues that I raised in my earlier report.
I hope that authors will at some time in the future look more closely at Baluchistan University. This institution seems to be quite different from the other comparators, and it would be interesting to know why.
Author Response
Respected Reviewer,
Thanks alot for your kind suggestion and constructive comments. It has helped us alot in enhancing our research.
Sure we will look into University of Baluchistan in future as a case study with comparable universities.
Reviewer 2 Report
Many figures still lack axis labels.
The numbers of the research output share should be written next to the pie piece in Figure 2.
As the manuscript still contains only raw citations rather than normalized values, this should be listed as a limitation. Note that Scopus provides a normalized impact value: the Field-Weighted Citation Impact.
Author Response
Respected Reviewer,
We truly acknowledge your constructive feedback and we have addressed all the comments.
1 : All of the figures have been added with axis labels
2 : We have added the Figure 2 comments. The following description is added
Figure 2 shows that major share of publication is made from KU, PU that is above 59% while the least share is done from BU in this research study contributing only 3.78%. PU has highest share of 30.90%, KU has second highest publication share with 28.55%, AU has publication share 23.19% and UP has 13.56%.
3 : The limitation of raw citation count has been added to paper in limitation section.
Thanks a lot
Round 3
Reviewer 2 Report
The manuscript has improved in this round of revision.
Limitation 5 should be extended by a comment that raw citation counts might be misleading due to differences in field and time whereas normalized citation scores often yield a better proxy for paper quality.
The authors might want to justify their analysis method by making a connection to contextualized bibliometrics, see for example DOI 10.3346/jkms.2018.33.e138 and https://inis.iaea.org/search/search.aspx?orig_q=RN:48044423.
"The United Kingdom, Germany, Italy, Netherlands, and Sweden were the top collaborating destinations respectively from the European region." The term "destinations" sounds wrong in this context. It implies travel.
Typo: "Collabrating" in Figure 7
Author Response
We would like to thank reviewer for commending our improvements made to research article.
Suggestion no 1: “raw citation counts might be misleading due to differences in field and time whereas normalized citation scores often yield a better proxy for paper quality”
Response: Thankyou indeed for the very valuable suggestion. The limitation 5 has been modified and extended in the paper as suggested stating the limitation of raw citations and highlighting the benefits of using normalized citation scores.
Suggestion no 2 : The authors might want to justify their analysis method by making a connection to contextualized bibliometrics, see for example DOI 10.3346/jkms.2018.33.e138 and https://inis.iaea.org/search/search.aspx?orig_q=RN:48044423.
Response: We are really thankful for a valuable suggestion and we have browsed the relevant articles suggested by reviewer. After looking at the suggested articles, we are of the opinion that it will deviate from scope of our study. However, we would like to commend the worthy reviewer for guiding us to future research based on contextual bibliometric study.
Suggestion no 3: "The United Kingdom, Germany, Italy, Netherlands, and Sweden were the top collaborating destinations respectively from the European region." The term "destinations" sounds wrong in this context. It implies travel.
Response: Thank you for the correction. The term “destinations” has been replaced with “countries”.
Suggestion no 4: Type "Collabrating" in Figure 7
Response: It is corrected now. Thanks for the keen observation and review.
