# **Modern Problems of Scientometric Assessment of Publication Activity**

Edited by Oleg V. Mikhailov Printed Edition of the Special Issue Published in *Publications*

www.mdpi.com/journal/publications

# **Modern Problems of Scientometric Assessment of Publication Activity**

# **Modern Problems of Scientometric Assessment of Publication Activity**

Editor **OlegV.Mikhailov**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editor* Oleg V. Mikhailov Kazan National Research Technological University Russia

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Publications* (ISSN 2304-6775) (available at: https://www.mdpi.com/journal/publications/special issues/scientometric assess).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-4831-9 (Hbk) ISBN 978-3-0365-4832-6 (PDF)**

© 2022 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

# **Contents**


# **About the Editor**

#### **Oleg V. Mikhailov**

Oleg V. Mikhailov is the author of over 1300 publications in nine languages (Russian, English, German, Japanese, Hungarian, Czech, Spanish, Tatar and Mongolian). He has written 19 books, including 14 monographs published in the main Academic Publishing Houses of the Russian Federation. Professor Mikhailov authored about 50 review papers (over 30 of them were published in international journals), over 270 papers in 30 leading Russian journals, over 350 papers in more than 60 international journals with a high impact factor (produced by Multidisciplinary Digital Publishing Institute (MDPI), Elsevier, Springer, Taylor & Francis, World Scientific Publishing) and indexed in Web of Science/Scopus. He is also responsible for 125 inventions.

Oleg V. Mikhailov is an Editorial Board Member of the International Journal of Molecular Science, published by MDPI in the section "Physical Chemistry & Chemical Physics". In 2019-2022, he was invited as a Guest Editor of eight Special Issues in International Journal of Molecular Science, Materials, Molecules, Publications, and Inorganics, published by MDPI and indexed in Web of Science/Scopus.

### **Preface to "Modern Problems of Scientometric Assessment of Publication Activity"**

At the end of the second decade of the 21st century, 60 years had passed since the creation by the American researcher Eugene Garfield of the first ever research institution dealing with the problem of the citation of scientific works—the Institute of Scientific Information (ISI)—which became the final result of his organizational work begun by him in 1955. At the same time, a citation database was created named "Science Citation Index" (SCI), which included information about the citation of scientific journals in the field of natural sciences; after that, a similar indexing of journals in the social and human sciences appeared, such as the "Social Sciences Citation Index" (SSCI) and the "Arts and Humanities Citation Index" (AHCI). Subsequently, on the basis of all this, an international citation database of scientific journals, now known as Web of Science (abbreviated as WoS), arose. Since 1964, the well-known parameter for assessing the citation of scientific journals has been in effect—the so-called impact factor (IF), which became the first bibliometric indicator officially recognized in the scientific community to assess the level of a journal's authority. Largely due to this, in the last decade of the 20th century an opinion began to actively form that the criterion for the success of any researcher is the presence of publications in journals with a high impact factor. Since its introduction, other indicators have appeared for scientific journals, which are in one way or another related to the citation of articles published in them, in particular SJR (SCImago Journal Rank) and SNIP (Source Normalized Impact per Paper). Currently, there is a very significant number of bibliometric indicators, not only for evaluating scientific journals, but also—in an even greater number—for evaluating the publication activity of both individual research scientists and the research teams that are composed of them. Thus, the idea of the need to use citations and related bibliometric indicators has deeply penetrated the mindset of many modern research scientists; to date, the scientometrics and sociology of science has accumulated a very significant body of literature on the problems of citation and its significance in science.

In this context, however, two polar points of view can be distinguished, according to the first of which it is possible and necessary to use citation indicators to evaluate the scientific activities of research scientists (albeit with certain reservations), while according to the second, it is impossible in principle. Supporters of the former believe that the quantitative indicators of the researcher's activity (the number of publications, the credibility of the scientific publication where they are published, data on their citation and derivatives from them) somehow correlate with the effectiveness and significance of the research conducted by the researcher, as well as their recognition in the wider scientific community. Supporters of the second point of view object to the very possibility of such an assessment, which essentially boils down to the fact that, despite the apparent "impartiality" of digital indicators, one cannot be completely sure of not only their objective reflection of reality, but even in understanding what exactly is hidden behind these indicators. Moreover, imposing the principles of evaluation with the help of quantitative indicators, in their opinion, can be detrimental to science, since it can lead to an artificial overestimation of these indicators, "chasing numbers", and in the long term, to a drop in the quality of scientific research itself. Both the first and the second points of view have the right to exist, and the question of the appropriateness of using bibliometric indicators for assessing the scientific activity of both individual researchers and research groups/teams remains debatable to this day. It was in connection with this that the idea of creating a special issue of Publications appeared, in which the above problem could be covered in relation to one aspect or another.

The Special Issue contains seven articles, including two review articles, three full-text research articles, and two brief reports. An introductory article by Guest Editor, Dr., Prof. O.V. Mikhailov, is also featured in which a general characterization of the problem to which this Special Issue is devoted, is given.

The issue opens with an article entitled -Conference Accreditation and Need of a Bibliometric Measure to Distinguish Predatory Conferences by Pooyan Makvandi (Italy), Anahita Nodehi (Italy) and Franklin R. Tay (USA), https://doi.org/10.3390/publications9020016, devoted to the problem of the scientometric evaluation of various scientific conferences. This problem, despite its relevance, is very poorly covered in modern literature, especially at a time when the number of conferences around the world is growing rapidly (including the so-called "predatory conferences" that profit from unsuspecting researchers without the main goal of promoting science or cooperation). In this article, a bibliometric measure is proposed that allows scholars to evaluate the quality of a conference before attending.

Next is the article -Russian University Teachers' Perceptions of Scientometrics by Andrei V. Grinev, Daria S. Bylieva and Victoria V. Lobatyuk (all – from Russia) ¨ https://doi.org/10.3390/publications9020022. This article is devoted to the question of the attitude of teachers of Russian universities to scientometrics and its indicators imposed on them by the administration of universities and the state since 2012. In addition to substantiating the relevance of the problem, the article contains a brief critical outline of the main scientometric parameters and their application in practice in Russia. According to the authors of the article, the teaching staff of Russian universities understand the specifics of scientometrics, treat it relatively positively, and in recent years have managed to adapt to the new requirements of the administration in terms of implementing scientometric tasks and standards.

In the article -Author-Suggested, Weighted Citation Index: A Novel Approach for Determining the Contribution of Individual Researchers by Tanmoy Konar (India) https://doi.org/10.3390/publications9030030, a new scientometric index is described, which, by design, should reflect the scientific contribution of any individual researcher to a particular scientific article. In this regard, the author of this article suggested that when submitting a scientific article, the corresponding author should provide a statement agreed by all authors, containing weighting factors in relation to each author of the article; the author who contributed most to the article will receive a higher weighting coefficient, the sum of which should be equal to one. This article, unfortunately, turned out to be the only one in this issue in which a new bibliometric indicator is proposed.

The problems of the formation and accumulation of intellectual capital in the modern world are discussed in a review article entitled -Intellectual Capital: A Review and Bibliometric Analysis by Wilder Quintero-Quintero, Ana Beatriz Blanco-Ariza and Manuel Alfonso Garzon-Castrill ´ on´ (all – from Colombia) https://doi.org/10.3390/publications9040046 which provides a kind of summary of a general overview of research, which is one way or another related to this issue. According to the data of the authors, who used the Scopus database to find and systematize them, intellectual capital was important and relevant in scientific publications over the last six years, which were linked by the number of authors by institutions. To some extent, this problem is also explored in the next article -Research Productivity in Emerging Economies: Empirical Evidence from Kazakhstan presented by Timur Narbaev and Diana Amirbekova (Kazakhstan) https://doi.org/10.3390/publications9040051, in which, using the scientometric approach, the situation with research activities in the Republic of Kazakhstan and the dynamics of its development over the past 30 years is discussed. Although among the entire array of scientific publications in modern fundamental science, scientific articles are the most important component, in applied science related to the creation and implementation of technical and technological innovations, patents for various intellectual property objects, and above all for various inventions, are no less important. A scientometric analysis of patent literature in the People's Republic of China (PRC) issued in the first 20 years of the 21st century (more than 2500 patents) was performed in the article - A Scientometric Assessment of Agri-Food Technology for Research Activity and Productivity by Jakkrit Thavorn, Veera Muangsin, Chupun Gowanit and Nongnuj Muangsin (all – from Thailand) https://doi.org/10.3390/publications9040057. The results presented in it show that in all technology clusters, PRC is ahead of other countries in terms of the number of patents. Interestingly, almost all of China's patents are used to commercialize technologies domestically, while other countries tend to apply for patents abroad to exploit the legal opportunities associated with them.

The most recent publication in the given Special Issue is the systematic review -What Proportion of Systematic Reviews and Meta-Analyses Published in the Annals of Surgery Provide Definitive Conclusions—A Systematic Review and Bibliometric Analysis presented by Matthew G. Davey, Martin S. Davey, Aoife J. Lowery and Michael J. Kerin (all - from Ireland) https://doi.org/10.3390/publications10020019, which provides a systematization and bibliometric analysis of systematic reviews and meta-analyses published in the journal "Annals of Surgery" over a 10-year acceptable period, and the unambiguity of the final statements of these reviews published in this journal has been evaluated. It is symbolic in its own way that such an article, as a kind of "review of reviews", completes this Special Issue.

As can be seen from the above, the authors of these articles are researchers from different countries of three continents—Europe, America and Asia—working in the field of science/scientometrics and the sociology of science. Thus, there is every reason to consider this Special Issue as an international endeavor and to hope that its materials will be useful to its future readers from different countries.

Oleg V. Mikhailov, 22 June 2022

> **Oleg V. Mikhailov** *Editor*

#### *Editorial* **Introduction from the Guest Editor of Special Issue "Modern Problems of Scientometric Assessment of Publication Activity"**

**Oleg V. Mikhailov**

> Department of Analytical Chemistry, Certification and Quality Management, Kazan National Research Technological University, K. Marx Street 68, 420015 Kazan, Russia; olegmkhlv@gmail.com

In any branch of intellectual activity that claims to be called the word "science", there are two approaches to describe the phenomena and objects associated with it; namely, qualitative analysis and quantitative analysis. In such an interdisciplinary science as modern analytical chemistry, these concepts are generally fundamental in the systematics of methods for establishing the composition and structure of matter. A similar state of affairs should take place in the branch of knowledge that studies the functioning of science as such—Sciencelogy (science of science), where one of the key problems is an adequate assessment of scientific activity and the contribution to a particular branch of science of an individual scientist and scientific teams. Consequently, in the general methodology of such an assessment, two components must be present—a qualitative assessment, based mainly on opinions about this very activity on the part of other people (mainly those who have the moral right to consider themselves representatives of the scientific community), and a quantitative assessment, on the basis of which is no longer public opinion, but some objective indicators of scientific activity, which no longer depend on this opinion.

The problem of assessing the quality of the scientific activity of both an individual specific scientist and scientific collectives takes its origins almost from the moment of the birth of science itself as such, and at all times was one of the most urgent—and at the same time the most difficult—of its problems. For a long time, in any branch of science, only a qualitative component was used to evaluate scientific activity and achievements in this field of any scientist–researcher in the scientific world, the mechanism of which was actually unknown. The dominance of the qualitative assessment of scientific activity and scientific achievements has existed for centuries. However, it could not be eternal, because ignoring its quantitative component inevitably made this assessment itself, firstly, one-sided, and secondly—and this is the main thing—subjective, regardless of who carried it out. For it to acquire an objective character, it is necessary to find some quantitative indicators based on which it would be possible to evaluate the scientific activity, and above all such an important factor in it as publication activity, since any scientist leaves a memory of themselves primarily due to their published scientific works. (Therefore, we would not have known anything about the outstanding Roman astronomer Claudius Ptolemy, if not for his epoch-making work "Almagest" (from Arabic - , al-kitabu-l-mijisti)

in 13 books, culminating in the creation of a geocentric system of the world, because neither the lifetime appearance of this scientist nor even the years of his birth and death are reliably known). Nevertheless, quantitative criteria for assessing scientific activity by the scientific community began to be developed only starting from the second half of the 20th century, when the pursuit of science became a fairly widespread phenomenon and an urgen<sup>t</sup> requirement of the time became the need for its objective assessment using certain quantitative parameters that did not depend on any subjective factors. Such an assessment is of particular importance when it comes to certain "marks of distinction" of an individual scientist or research team, whether it is funding scientific research in the form of gran<sup>t</sup> support, awarding scientific prizes, medals, academic degrees and titles, etc. Without it, already in the near future, science is threatened by the fact that those researchers who are

**Citation:** Mikhailov, O.V.

Introduction from the Guest Editor of Special Issue "Modern Problems of Scientometric Assessment of Publication Activity". *Publications* **2021**, *9*, 19. https://doi.org/ 10.3390/publications9020019

Received: 25 February 2021 Accepted: 4 May 2021 Published: 8 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

creating it now and are able to create it in the future, but whose talent and achievements do not receive a due assessment from both their colleagues and the scientific community generally, will begin to leave science and stop coming in it.

In principle, various options for assessing any type of creative activity are possible; however, in all its spheres, the most objective assessment is based on the final result, and not on the procedure for achieving it and the efforts expended on it. Ideally, an objective assessment of the quality of the scientific activity, it seems to us, should resemble the procedure for identifying winners in sports competitions, when the best are determined, focusing on certain specific objective quantitative indicators achieved by athletes (run time at an appropriate distance, range of a javelin or hammer throw, height in pole vaulting, etc.). In such cases, as a rule, there are no problems (for example, the advantage of one runner over another at a 100-m distance will be objectively recorded by a stopwatch). Let us emphasize: precisely objective, because when quantitative but subjective indicators are used to determine the winners in any kind of sport, arbitrariness is inevitable to one degree or another, as it has been the case to this day, in particular, in figure skating, artistic and rhythmic gymnastics. Of course, science at all times is not a sport, and whether we like it or not, it was based, is based and, apparently, will always be based on the domination of the authority of scientists, but this is not a reason to exclude the quantitative component of evaluating scientific activity. However, before putting into effect this very quantitative assessment of scientific activity, it is first necessary to solve the fundamental question of which digital indicator (or totality of indicators) should be used as the basis for such an assessment to make it really objective, and not some kind of "game into numbers". When choosing such an indicator, one should remember the sayings of two completely different scientists, namely the grea<sup>t</sup> physicist and Nobel laureate Albert Einstein: "*Everything should be made as simple as possible, but not too simple*" and one of the largest economists of the 20th–21st centuries, Charles Goodhart, Professor Emeritus of London School of Economics and Political Science: "*When a target becomes a goal, it stops being a good indicator*" (which is nothing more than a direct consequence of the so-called Goodhart Law: "*Any observed statistical regularity will tend to collapse once the pressure is placed upon it for control purposes*"), which, in our opinion, are also directly related to the problem of a quantitative assessment of scientific activity.

The most objective criterion for evaluating the activities of any scientist, probably, should be the worth of those scientific works that were created with his personal participation, for the development of both the corresponding separate branch of science and science as a whole. However, it is extremely difficult (if it is possible at all) to determine this value on a quantitative level. To a large extent (although not always), the value is related to the demand for the scientist's work, which is determined by the measure of interest in them from their other colleagues (and not only them). However, the demand is also difficult (if not impossible) to characterize with any quantitative parameter; on the other hand, not one or two cases are known when the demand and value of the scientist's work for the development of the corresponding branch of science ultimately turned out to be almost different "poles" (such as, for example, the phlogiston theory of the German physicist G. Stahl, which for more than a hundred years was very much in demand by chemists for the interpretation of many experimental data, but already in the 19th century it had no scientific value). The next step in the search for a common criterion for the quantitative assessment of scientific activity was the appeal of creative thought to the phenomenon of *citation*, which in one way or another characterizes the degree of mention (citing) of the works of the corresponding scientist in the media and, above all, in the scientific press. The correlation between the value, demand and citation of scientific papers can be schematically shown as follows:

# **WORTH**

↓ *Correlation: rather yes than no*

**DEMAND**

↓*Correlation: rather no than yes*

# **CITATION**

To some extent, the correlation between demand and citation is akin to such a statement: "The taller a person is in stature, the stronger he is." On average, this is probably true, but a comparison of the strength of two randomly chosen people solely by their height is often erroneous. Be that as it may, in the second half of the 20th century, citation and related parameters (known as "bibliometric indices") began to attract increased attention, and not only from specialists in the field of science and the sociology of science but even among those scientists and researchers whose activities, it would seem, are very distant from the just indicated branches of science. In this regard, it is interesting to note that the term "bibliometric index" (which is often also known as "scientometric index"), despite its wide distribution and use in the scientific community, has not ye<sup>t</sup> received a generally accepted interpretation in the scientific literature; moreover (that is also quite remarkable), and there are essentially no discussions about its definition either. At least two options for its interpretation-in the "narrow" sense and in the "wide" may be given, however. Within the framework of the first of them, a bibliometric index should be understood as any quantitative parameter which in one way or another characterizes the publication activity of an author or a group of authors, but which at the same time is necessarily (directly or indirectly) related to the citation of their works; within the framework of the second, there is a quantitative parameter, which characterizes the publication activity of an author or a group of authors, but in the determination of which, citation already is not taken into account explicitly. Wherein, in both in that and in the other variant, some bibliometric parameters can be an integer (such as, for example, the Hirsch index or the total number of publications for a certain period of time), while others can be non-integer (in particular, the sum of share citations for all publications of the author, the average number of co-authors in his publications). At present, the number of bibliometric indices proposed by various authors is already several dozen; at the same time, that is very symptomatic, the overwhelming majority of them appeared after the publication of the work of J.E. Hirsch [1] which served as a kind of "catalyst" for the process of both creating new and improving existing indices (the author of this article also made a modest contribution to this process).

In modern sciencelogy, it is customary to distinguish the following two categories of fundamental indicators related to the citation:

	- - the total number of works (articles) of the researcher in the database of various scientometric systems and institutions (*Web of Science*, *Scopus, etc.*);
	- - the number of links to specific articles in the relevant database and the total number of links to these articles;
	- - Hirsch index (*h*-index), also known as "hirsch".

Namely, these parameters, by the way, are key parameters in each of these two most authoritative international citation databases–*Web of Sciences* and *Scopus*;

	- - the impact factor of the publication (IF in *Web of Science*, CiteScore in *Scopus*);
	- - time of "cited half-life" of the publication (cited half-life);
	- - the total number of references to articles in this edition.

The basic parameter, based on the values of which the personal citation of a researcher is assessed, is the total number of references to the works (publications) of a given person in various scientific editions (both periodic and non-periodic). Being the earliest in terms of the time of its introduction into "scientific turnover" in comparison with the others, it remained the only one for quite a long time (in fact, until the beginning of the 21st century). However, in itself, the personal citation of the works of both a particular researcher and any research team, no matter how grea<sup>t</sup> it may be, cannot ye<sup>t</sup> serve as evidence of the significance and worth of the scientific works they have performed and even their demand by any part of the scientific community. The only obvious advantage of a researcher's personal citation index is the following: if his works are cited very little or not at all cited, then they are most likely of little interest or even unnecessary to anyone. Along with this lack of the personal citation index of the researcher, several others can also be noted, such as:


The second category of citation parameters is associated with the citation of those editions (journals, books, collections of scientific papers and other publication sources) in which the works of this author were published. The situation here is more complicated, because to this species of citation, contributions are made by not only and not so much the author of the article in a particular journal, as other authors whose articles are published in the same journal. Moreover, in many cases, a much larger contribution to the citation of this edition is made by authors of publications in other scientific journals citing articles from the given journal. The citation parameters here can also be different. Additionally, here it remains to be understood how adequately they can assess the citation rate of both individual authors of articles and scientific editions where they were published.

Despite the very significant advances in this specific area of sociology, the problem of an adequate and objective assessment of scientific activity and publication activity is still rather far from being solved. Additionally, although, it should be noted that the totality bibliometric indicators, of course, are far from ideal for assessing the scientific activity of a human individual, but in the general case it is still better than an assessment only subjective, whoever carried it out. That is why the key task of the given Special Issue is to familiarize its readers with the latest achievements both in the search for new, more advanced bibliometric indicators and in the improvement of existing ones.

In connection with the aforesaid, this Special Issue includes mainly original articles and communications devoted to improving the quantitative assessment of scientific and publication activity of researchers and research teams using various bibliometric indices (both new, original ones and those already proposed earlier). Along with this, it contains articles that contain proposals for the development and improvement of indices characterizing the authority of scientific periodicals (journals) and articles of a critical character related to the application of these indices in practice.

**Funding:** This research received no external funding.

**Data Availability Statement:** There are no new data in this article.

**Conflicts of Interest:** The authors declare no conflict of interest.

# **Reference**

1. Hirsch, J.E. An index to quantify an individual's scientific research output. *Proc. Nat. Acad. Sci. USA* **2005**, *102*, 16569–16572. [CrossRef] [PubMed]

#### *Viewpoint* **Conference Accreditation and Need of a Bibliometric Measure to Distinguish Predatory Conferences**

**Pooyan Makvandi 1,\*, Anahita Nodehi 2 and Franklin R. Tay 3**


**Abstract:** Academic conferences offer scientists the opportunity to share their findings and knowledge with other researchers. However, the number of conferences is rapidly increasing globally and many unsolicited e-mails are received from conference organizers. These e-mails take time for researchers to read and ascertain their legitimacy. Because not every conference is of high quality, there is a need for young researchers and scholars to recognize the so-called "predatory conferences" which make a profit from unsuspecting researchers without the core purpose of advancing science or collaboration. Unlike journals that possess accreditation indices, there is no appropriate accreditation for international conferences. Here, a bibliometric measure is proposed that enables scholars to evaluate conference quality before attending.

**Keywords:** conference indicator; conference impact factor; conference accreditation; bibliometric measure

# **1. Introduction**

Academic conferences offer scientists the opportunity to share their findings and knowledge with other researchers. Conferences are organized by institutions or societies, and in rare cases, by individuals [1]. There is an increasing tendency for researchers to receive invitations from unsolicited conferences. The organizers of these so-called "predatory conferences" lure researchers, especially young scientists, to attend their conferences by sending out one or more emails that invite the scholars to be plenary speakers in those conferences (Figure 1) [2–4]. Jeffrey Beall is the first person to use the term "predatory meetings". The term was used in the same context as "predatory publications". He explained that some companies organize conferences to invite researchers from all over the world to present their papers. These organizers exploit the need for researchers to publish papers in proceedings or affiliated journals by asking for a significant conference attendance charge, using low quality conference business models [2]. Interested readers are referred to excellent reference sources for conference enhancement tips [5,6] and the implications of predatory conferences [4,7]. Early-career academics and scholars from developing countries are the most vulnerable to these predatory meeting invitations. Readers can easily identify some of the introductions used in those electronic communications:


**Citation:** Makvandi, P.; Nodehi, A.; Tay, F.R. Conference Accreditation and Need of a Bibliometric Measure to Distinguish Predatory Conferences. *Publications* **2021**, *9*, 16. https:// doi.org/10.3390/publications9020016

Academic Editor: Oleg V. Mikhailov

Received: 1 March 2021 Accepted: 21 April 2021 Published: 22 April 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

5

• I just wanted to check that if you have received my previous mail that I sent a couple of weeks back. We have not heard back and wanted to make sure it went through your inbox . . .

**Figure 1.** Predatory conferences target scientists. In practice, predatory conferences quickly accept even poor quality submissions without peer review and without control of nonsensical content, while asking for high attendance fees. They may utilize conference names that are similar to the names of more established conferences to attract academics and promote meetings with unrelated images copied from the Internet.

Similar to invitations from predatory conferences, there is also a notable increase in the number of invitations from predatory journals [8–10]. Whereas reputable international journals possess accreditation indices such as impact factor [11,12], source normalized impact factor (SNIP) [13,14], Scimago journal rank (SJR) [15], Eigenfactor Score (ES) or Hirsch index (h-index) [16], conferences do not have comparable accreditation indices. Although some conference ranking metrics are available (e.g., http://www.conferenceranks.com/; http://portal.core.edu.au/conf-ranks/, accessed on February 2021), not all reputable conferences are amenable to search (e.g., European Society of Biomaterials Conference; Forbes Women's Summit). Usually, search conferences are restricted to specific fields only. In addition, the number of conferences are growing at an exponential rate, which makes website updates on a daily or even a weekly basis virtually impossible.

There is a pressing need for a system that evaluates the academic quality of international conferences [17]. Prior art only discusses the dilapidation of predatory conferences without offering a solution. The objective of the present letter is to address potential methods of evaluating conferences and to offer suggestions on conference evaluation. The authors propose a new accreditation scheme for conferences which may be useful for scientists, especially for young scholars, to identify high-level conferences.

#### **2. Potential Solutions**

Although some institutions do evaluate the credibility of conferences, such evaluations are not conducted on all conferences. In some instances, the conference organizer has to apply for the conference accreditation. This is not a mandatory process, unlike journal accreditation. The following are suggestions for enhancing the quality of conference accreditation:

(I) A conference must have a unique name with a registered International Standard Serial Number (ISSN). This is comparable with journal ISSN, in which there are no two journals with identical names. The conference title should be devoid of a period descriptor that references it as part of an ongoing conference series. For example, "European Conference on Biomaterials" is preferred over "30th European Conference on Biomaterials". If one utilizes a descriptor that represents a continued series, such as "30th European Conference

on Biomaterials", the conference will have its own ISSN. It follows that another ISSN will be issued for the "31st European Conference on Biomaterials". This is not how journals are cited. Each journal has its individual ISSN but issues within the same journal do not have their own ISSNs. Therefore, each conference should have a unique name with a registered ISSN number.

Avoiding citation for a conference: poster and slide presentations in conferences are not peer reviewed in depth and are rarely accessible to scholars. Consequently, citation of posters or slides should be avoided. The information below may be employed for referring a conference abstract/paper.

Author names. *Title* of the study. Conference name. Series number. Year. City and Country. Publisher.

For example: P. Makvandi, F.R. Tay. *Injectable antibacterial hydrogels for potential applications in drug delivery*. European Conference on Biomaterials, 2019, 30th series, Dresden, Germany, Elsevier.

It should be stressed that a conference title and a series number should only be used once for a particular conference, similar to the name of a journal. No two conferences should have the same title.

(II) If the original article has previously been published in a journal, it has to be mentioned in the conference abstract by referring to the electronic link of the published paper. Such a strategy helps to reduce redundant citation of one's previously published research. This is because since many researchers present their results at more than one conference.

#### **3. How to Accredit**

If there is a persistent handle or DOI of a previously published paper in the conference abstract, citation of the original paper may be used as an index to distinguish the quality of the presented paper at the conference. Accordingly, the h-index [18,19] of a conference may be used. In addition, the average number of the presented abstracts, including oral and poster presentations, may be employed along with other criteria such as CiteScore [20], impact factor [21,22], source normalized impact per paper (SNIP) [21], in conjunction with the conference h-index. In this manner, one does not need to know the number of accepted abstracts in the conference because such information is already expressed by the h-index. If a conference presentation is generated from more than one published paper (even from different journals), the average number of citations of the original published papers may be used.

Because this type of accreditation depends on previously published papers, the term "secondary" may be added before the indices. For example, secondary CiteScore (SSC), secondary impact factor (SIF), secondary Hirsch index (Sh-index) may be used to differentiate between the previously published papers and the abstracts (Table 1).


**Table 1.** List of 3 presentations that come from 4 previously published papers to be introduced at a conference.

\* Average citation for presentation 1 is 6+7 2 = 6.5; † Average h-index for presentation 1 is (<sup>14</sup>+<sup>9</sup>) 2 = 11.5 ≈ 11.

Some conferences accept findings that have not been published or were presented at other conferences. In this case, the abstract will not be linked to a previously published paper. It should be noted that presentations of the same findings to the same audience should be avoided. However, different parts of a previously published paper or different aspects of a clinical trial may be presented in different conferences.

Where the secondary CiteScore (SSC) for the conference is calculated based on the average:

$$\text{SSC}: \frac{6.5 + 10 + 3}{3} = 6.5\tag{1}$$

In addition, the secondary h-index (Sh-index) is the minimum value of *h* such that the given conference has published *h* papers that have each been cited at least *h* times. In the example, the calculated Sh-index for the conference is 3 because there are three publications that each has at least three citations.

The present letter proposed a bibliometric measure that enables all academicians to evaluate conference quality before attending. Publishing an article may take a long time (e.g., more than one year) in some disciplines. During this period, there may be new publications that may be cited as references. Hence, presenting the paper is an excellent opportunity to identify the strength of an idea and additional research that has been accomplished in a particular field. It has to be mentioned, however, that that is no evaluation available for unpublished and nonpeer reviewed manuscripts. Hence, only published studies may be used for bibliometric measurement.

It has to be pointed out that popular accreditation systems such as "impact factor" have their own disadvantages. For instance, "impact factor" depends on the size of the field/discipline. A larger community who work would draw more citations than the one having a small number of publications. Thus, these limitations motivated academic members to introduce other bibliometric measurements. To date, there is no universal acceptance of the accreditation systems. Thus, it may not be possible to solve the issue completely till there is a new accreditation system for conferences. Nevertheless, this present letter will help researchers identify and avoid participating in predatory conferences. Therefore, our proposed bibliometric measurement has its pros and cons. Based on our opinion, this letter brings the predatory conferences to the attention of scholars to stimulate some thought about this issue.

**Author Contributions:** Writing—original draft preparation, P.M., A.N.; writing—review and editing, P.M., A.N., and F.R.T. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** No new data were created or analyzed in this study. Data sharing is not applicable to this article.

**Conflicts of Interest:** The authors declare no conflict of interest.

# **References**


## *Article* **Russian University Teachers' Perceptions of Scientometrics**

**Andrei V. Grinëv 1,\*, Daria S. Bylieva 2 and Victoria V. Lobatyuk 3**


**Abstract:** This article is devoted to the attitude of Russian university teachers toward scientometrics and its indicators, which have been imposed on them by university administrations and the state since 2012. In addition to substantiating the problem's urgency, the article contains a brief critical outline of the main scientometric parameters and their application in practice in Russia. To evaluate this, 283 people from leading universities in Russia (included in Program 5-100) were questioned. As the study showed, faculties of Russian universities understand the specifics of scientometrics, relate to it relatively positively, and over the past years have been able to adapt to the new requirements of the administration regarding implementing scientometric tasks and standards. The higher the position and scholarly qualification of a respondent, the more complete the knowledge about scientometrics. Scholars in the humanities know and relate comparatively better to scientometrics than representatives of technical and general scientific specialties.

**Keywords:** scientometrics; scientometric indicators; Russian professors; sociological polls; scientometric politics

# **1. Introduction**

Digitalization affects increasing areas of human life; quantitative indicators are becoming dominant in areas that previously seemed not intended for this [1]. Scientometrics is a special discipline that researches scholarship using mathematical methods, data collecting, and statistical processing of bibliographic information (the number of published scientific papers, citations, etc.). Despite using mathematical methods, equations, and mathematical analysis, scientometrics can hardly be classified as a full-fledged science since its main indicators rarely give a complete and objective picture of the scientific achievements of specific researchers (which will be discussed below). Nevertheless, scientometric data are now widely used in various countries, and in Russia, since 2012, there has been a real fetishization of scientometric parameters, at least in the country's leading universities.

The origin of bibliometrics dates back to the end of the 19th century, the scientometrics taking shape in the middle of the 1950s in the United States, founded by the American linguist Eugene Garfield [2]. In 1960, he organized the Institute of Scientific Information (ISI), which since 1963 has regularly published bibliographic indexes of scholarly citation (Science Citation Index). Such bibliographic information was much needed by universities, rating agencies, and other organizations directly or indirectly connected with education and scholarship and, therefore, willing to pay for it [3]. It is not surprising that three decades later, scientometrics in the United States has become a thriving and profitable discipline, especially after the takeover of ISI by the Thomson Reuters Corporation in 1992 and forming the world-famous bibliographic database (BDB) Web of Science (WoS), which Clarivate Analytics took over in October 2016.

11

**Citation:** Grinëv, A.V.; Bylieva, D.S.; Lobatyuk, V.V. Russian University Teachers' Perceptions of Scientometrics. *Publications* **2021**, *9*, 22. https://doi.org/10.3390/ publications9020022

Academic Editor: Oleg V. Mikhailov

Received: 7 April 2021 Accepted: 25 May 2021 Published: 27 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Europe was somewhat late in joining this bibliometric business. In 1995, Elsevier Publishing Corporation, based in the Netherlands, created the largest current bibliographic and abstract database Scopus, which indexes more than 24 thousand scholarly journals, materials from the most authoritative international and national conferences, and monographs published by major scientific publishers, mainly in English. Only in 2005 did Russia, on the initiative of the Russian electronic library (eLIBRARY), receive its own bibliographic database of the Russian Science Citation Index (RSCI).

Now in Russia, the bibliographic information and the results of its mathematical processing obtained from all three databases (WoS, Scopus, RSCI) are actively used in planning scientific work and reports, distributing grants, and encouraging leading scientists and scientific departments of universities and institutes, with their ranking and financing depending on the achieved scientometric indicators. At the same time, ministerial and university bureaucrats often treat scientometric parameters and final figures uncritically, allowing various ill-considered decisions and abuse of scientometric information. For example, in January 2020, the Ministry of Science and Higher Education of the Russian Federation sent out a directive letter no. MH-8/6-SK "On adjusting the State Task taking into account the methodology for calculating the integrated score of publication performance." An analysis of the table attached to the letter, where the scores for various publications were indicated, shows that the ministerial requirements for scientometric indicators have reached complete absurdity. Thus, according to the table, one article published in a journal indexed by the WoS Core Collection database and assigned to the first quartile (Q1) is almost equal to 20 articles registered in the Scopus database (regardless of quartiles) or 40 articles from Russian journals or 20 monographs! However, the work and time expenditures for writing even a small monograph are incomparable with the similar costs for writing even the most high-quality article. Moreover, it is the monographs that are most appreciated by specialists in the field of humanitarian knowledge. Moreover, although then the ministerial table underwent correction, taking into account the specifics of the humanitarian disciplines, the problem of the acceptable use of scientometric indicators does not lose its relevance.

The object of this study is the reflection of the teaching staff on scientometrics and scientometric requirements. Currently, the scientometrics in elite Russian universities are not just a trivial calculation of numbers, their analysis, and interpretation, but a reason for making important managerial decisions with the appropriate reflection of the scientific community.

#### **2. Literature Review**

Scientometrics draws its material for mathematical calculations and indexes from bibliographic databases that use three main indicators: (1) the number of publications of the author, (2) the number of citations of his works, and (3) the Hirsch index. In other words, the minimum standard set. For organizations and their divisions, these indicators are usually added to the number of the staff who published their works, with various additional weighting factors and calculations introduced if necessary. In addition to the three main indicators, other markers, metrics, and graphs can be used.

Besides the three main indicators, other markers, metrics, and graphs can be used. For example, Web of Science and Scopus use charts to clearly show the number of publications by year and their citations in the author's profile. In addition, on the Publons platform, authors can find the average figure of citations of their works and the average number of citations per year in the WoS system. RSCI provides several dozen metrics, including the *h*-index without self-citation, the number of articles and citations in foreign journals, the number of publications over the past five years, and so on (for scientometric indicators, see more detail [4]). However, all these and other scientometric parameters are of secondary importance compared to the three main ones, and they are usually not used at all in Russian universities in practice and are unknown to either the administration or the teaching staff.

In addition to scientometric indicators, an expert assessment is also used in Russia and other countries, but usually in reviews in journals (or reviews on monographs). Of course, this kind of expert analysis indirectly supplements and corrects scientometric statistics. In university practice, expert analysis is usually not applied, but primitive quantitative indicators are used, for example, the number of publications indexed in the Scopus or WoS databases for the previous three calendar years; Hirsch index according to BDB Scopus or RSCI, etc.

Let us start with the total number of publications—the most common indicator of the effectiveness of the scientific activity of a scholar. Not all of the scholar's works are recorded in bibliographic databases, sometimes reducing the final figure and significantly. The leading international databases Web of Science and Scopus introduced artificial restrictions on the registration of scholarly papers: articles published in peer-reviewed journals, mainly in English, have absolute priority, which results in discrimination against representatives of non-English-speaking countries [5,6].

In addition to the frequent underestimation of a scholar's total number of publications, another important disadvantage of the main scientometric indicator is the ignoring of complexity, volume, and quality of scholarly work. After all, a simple mechanical calculation of publications erases the difference between a monograph, journal article, review, etc. Therefore, such a simple indicator as the total number of publications must, on one hand, be detailed, indicating the nature of the publications, and, on the other hand, must be supplemented with other scientometric indicators.

Here it is also necessary to touch upon the issue of co-authorship. This problem is especially relevant for the natural, medical, and technical sciences since humanities specialists usually write their work individually or in small teams. A classic example is an article published after research at the Large Hadron Collider and the discovery of the Higgs boson: almost 3000 physicists are formally considered the authors of this article! [7] (p. 7). It is impossible to imagine that all these people wrote one article. Although fractional counting methods have already been developed in scientometrics [8], they are rarely used in university practice. The main bibliometric databases also give equal co-authorship to all contributors (real and more often imaginary) of the indexed publication.

The problem of co-authorship often, especially in Russia, also lies in the possibility of various abuses, for example, when co-authors forcibly (or voluntarily—to assist in the publication of an article) include the leadership of a department or university in the absence of a real contribution to scientific research [9] (p. 276). In addition, false co-authorship distorts scientometric results, and sometimes very significantly, leading to citation fraud, contributing to corruption, and so on.

Even this brief outline of the most universally used scientometric indicator demonstrates that the number of author's works appearing in one or another bibliographic database rarely reflects the real number of his/her publications (underestimating or, conversely, overestimating through the mechanism of co-authorship). In parallel, scientometrics offers another indicator that is designed to solve the research impact. This is the citation, which means considering the reference to the author's work by another author or group of authors. It is believed that the more citations, the higher the demand for the scholarly work, and thus its quality. Currently, the citation index (CI) is a generally recognized indicator of the significance of a scholar's work in the scholarly world. Although it has no fewer disadvantages than the previous main scientometric indicator, it appears in all the databases because not all of the author's works are recorded in the leading databases Web of Science and Scopus. Accordingly, the number of citations considered will be artificially reduced. Again, there is the problem of co-authorship: the citation index does not take into account the personal contribution of the author (when calculating the CI, that is, it does not matter if fifty people wrote the article or there was only one author) [10] (pp. 135–141).

In scientometrics, it has long been noted that citation depends on the branch of scientific knowledge and the culture of citation: most often, doctors and biologists are cited, and least often, historians and mathematicians. In addition, the number of citations

depends very significantly on the scholarly topic and specialization, even within a single science. For example, it is obvious that in ethnography, a larger topic or study devoted to a larger ethnic group will gather a larger "crop" of citations: links to an article about the material culture of the Chinese will be significantly more than to a publication about totemism among the Haida-Kaigani Indians. Among other things, the number of citations received can be influenced by the "scientific fashion" and personal relationships between scholars (the role of the subjective factor is especially grea<sup>t</sup> in Russia with its tradition of informal scholarly relations and corruption).

There are several other obvious disadvantages to the citation index. In particular, the CI registers references even to those works that are subjected to fair criticism and thus reaches the point of complete absurdity: a negative reference, which, in theory (after verification by experts) should be credited with a minus sign, on the contrary, brings the criticized author an additional citation. However, the "father of scientometrics", Eugene Garfield did not consider this a significant problem since, in his opinion, scholars are not inclined to be distracted by refuting frivolous works [11] (p. 45). He also believed that there is no need to be afraid of self-citation, which from his point of view is to a certain extent justified and considered by all bibliographic databases without exception. Indeed, references to your own work can be useful (within reasonable limits) by referring the reader to more detailed information. However, it is clearly not necessary to take them into account in scientometric calculations since, on this basis, an artificial increase in the citation is possible. Thus, the introduction in Italy in 2010 of the rule to take into account citations when holding the position of professor has led to a sharp increase in self-citation, especially among sociologists [12]. According to a special study, the more people quote themselves, the more often they are quoted by other scholars [13] (p. 433). Wherein men are about one and a half times more likely to refer to their own work than women [14].

A very significant disadvantage of the citation index is, in our opinion, that a single reference to specific work is usually considered in the list of literature used in the article. However, there may be several references to this publication in the text. An equally serious disadvantage of the CI is that it can be artificially increased by various manipulations, for example, when colleagues agree to quote each other's results. Such unethical methods are not uncommon in Russia and sometimes abroad.

Despite all these disadvantages, the citation of scholarly papers formed the basis of another scientometric index—the impact factor (IF) of journals. The impact factor is a formal numerical indicator of the importance of a scientific journal, which shows how many times, on average, each article published in it is cited over the next two years after publication. The introduction of the impact factor contributed to a better selection of scientific journals by the WoS database, where they are divided into four categories—quartiles—from Q1 (highest) to Q4 (lowest). Currently, the SCImago Journal Rank (SJR) is a convenient and visual system that demonstrates quartiles of journals. It is a publicly accessible portal developed by the University of Granada (Spain) with ratings of journals and countries and is associated with the Scopus database [15].

In addition to the impact factor, the citation of scholarly papers forms the basis of the last of the main scientometric indicators, the Hirsch Index—the h-index. This was developed in 2005 by Jorge Hirsch from the University of San Diego (CA, USA) to assess a researcher's scientific productivity that can be given by such simple characteristics as the total number of publications and citations. According to the h-index, a scientist has index h if h of his or her Np papers have at least h citations each and the other (Np-h) papers have ≤h citations each [16] (p. 16569). Although the h-index has undeniable positive qualities, such as ease of calculation and a relatively adequate assessment of the scholarly productivity of the researcher, it is not without many disadvantages. Thus, the h-index number cannot exceed the total number of the author's works; it does not take into account information on the most important highly cited works; two people with the same h-index value may have a total (summary) citation rate (IC), which differs tenfold, etc. [10].

As we can see, none of the three main scientometric parameters is perfect, and their thoughtless use can lead to various misunderstandings and abuses (see in detail [17]). However, the warnings of scientometrics experts had no effect in Russia, where a real fetishization of scientometric indicators began in 2012, which could not but affect the position of teachers in the country's leading universities. The pressure of the state bureaucracy was especially intensified under President Vladimir Putin, including strengthening the powers of the Ministry of Higher Education and university administration. To prevent a further decline in the country's prestige in the field of international publication activity, decree no. 599, signed by President Putin, "On Measures to Implement State Policy in the Field of Education and Science," was issued on 7 May 2012. It set a goal to increase Russian researchers' work in the total number of publications in world scholarly journals indexed in the Web of Science to 2.44% by 2015.

After the publication of the 2012 presidential decree, all subsequent state policies in the field of scholarship began to adapt to it, including the forced introduction of basic scientometric indicators, which began receiving much more attention. Another consequence of the decree was the inclusion of Russian universities in the rating race both at home and abroad. In 2013, the ambitious governmen<sup>t</sup> program "5-100-2020" was launched, according to which by 2020, the five best Russian universities should be included in the top 100 universities globally. Considerable funds were allocated to 21 elite Russian universities for this project, including to increase the publication activity and citation of their employees and scholars. In general, the experience was quite successful, and according to WoS data for 2017, Russia's share was 2.56% of the world's scholarly publications, which corresponded to 13th place globally [18,19] (p. 828). Now, in 2021, according to the SJR rating, Russia is in 12th place globally in terms of the number of scientific publications, slightly ahead of South Korea and behind Spain.

In this situation, Russian scholars and teachers are forced to adapt to the state and university bureaucracy regulations. Similar processes are observed in other countries. For example, the rush toward rating indicators in Pakistan, where a similar program to include five of the best universities in the top 300 higher educational establishments of the world had been adopted, led not only to the monetary stimulation of publication activity but also to various abuses and the deterioration of the quality of scientific publications [20] (pp. 442–447). Likewise, in China, young scientists are forced to publish in journals indexed by prestigious international databases [21]. Even in Italy, some authors, albeit insignificant, either unknowingly or quite deliberately give their works to so-called "predatory" journals for faster publication and indexing of articles for getting a citation [22] (pp. 14–15, 26).

The same is observed in Russia. To meet targets, not always honest methods are used that will allow one to achieve, and sometimes even exceed, the formal requirements of superiors. In recent years, the number of articles with multiple authors has increased significantly (especially among scholars in the humanities). The usual method has been to divide a large article into several smaller ones to increase the total number of publications; publication of the same work, but with different titles and minimal changes in content, has the same effect. The unrestrained mutual citation has also been used [23] (pp. 64–66).

## **3. Methods**

In preparing this article, such standard theoretical methods of scholarship were used as induction and deduction, analysis and synthesis, a systematic approach, and the comparative–typological and comparative–analytical methods.

In addition to theoretical methods, practical methods, such as working with documents, analysis of printed and electronic sources of information, and especially computerassisted web interviewing, were widely used in writing this article. The use of the latter helped to gather the main blocks of information on the research topic. In addition, statistical and mathematical methods were also used when processing questionnaires and respondent's answers.

For a more detailed study of the attitude of Russian university teachers to scientometrics and its indicators, the authors developed a questionnaire of 22 questions for a survey, the results of which are given in the Appendix A (see Table A1). The survey, conducted in the first three months of 2020, involved 283 respondents. In the Russian Federation in 2020, 227 thousand teachers worked in all universities; this information was published in the statistical collection "Education in Figures: 2020". This short statistical compilation is the main source of information on the entire system of Russian education. The collection uses data from the Ministry of Education of the Russian Federation, the Ministry of Education and Science of the Russian Federation, and the Federal Treasury, as well as its own developments at the Institute for Statistical Research and Economics of Knowledge of the National Research University Higher School of Economics. Thus, observing a confidence level of 90 and a confidence interval of 5%, we have the minimum required sample size of 272 respondents (283 university professors were interviewed in the study). In addition, forming the sample being influenced by the number of teachers with a university degree should have been at the average for the Russian Federation (74.1%). The distribution of respondents by universities is presented as follows: Peter the Great St. Petersburg Polytechnic University (30 respondents), ITMO University (30 respondents), Ural Federal University (30 respondents), University of Tyumen (30 respondents), the National Research Tomsk State University (30 respondents), the Herzen State Pedagogical University of Russia ( 30 respondents), St Petersburg University, Saint Petersburg State University (30 respondents), etc. (17 universities in total, including Nizhny Novgorod State University, Far Eastern Federal University, Kazan Federal University, Moscow University of Physics and Technology, Moscow Pedagogical State University, Higher School of Economics, Novosibirsk State University, Kemerovo State University, Ugra State University, Surgut State University (73 respondents). These are mainly educational institutions included in the 5-100 Program, and therefore, their teachers were generally better oriented in the problems of scientometrics than representatives of ordinary Russian universities (the Herzen State Pedagogical University of Russia was chosen as a control University not included in the 5-100 Program, which provided 10% of respondents). Statistical materials were collected both through direct questionnaires and through Internet surveys. At the same time, the number of respondents from provincial universities—150 people—slightly exceeded the number of respondents from St. Petersburg (133). Among the respondents, just over half were women—50.5%, men—45.6%, and only a very few refused to answer the question about their gender (3.5%) or indicated another gender—0.4%. The age distribution gave the following figures: young people under 34 years of age made up 36.7%, middle-aged (35–49 years)—39.9%, and 50 years and older—23.4%, which roughly reflects the gender and age structure of teachers at Russian universities. As for ethnicity, the vast majority of respondents identified themselves as Russian—96.5%; several people identified themselves as Jews, Kazakhs, Tatars, or Ukrainians. As for the respondents' professional and official structure, professors made up 10.6%, associate professors—44.2%, senior teachers—19.8%, and assistants—25.4%, which approximately corresponds to the standard number of each category in a normal Russian university. Half of the respondents identified themselves as scholars in the humanities—49.8%, and 50.2% identified themselves as natural and technical sciences representatives. Of the respondents, 46.3% preferred not to reveal their specialty, indicating only which sciences could be attributed. Information on those who indicate their specialty is presented in the Appendix A with information on the universities (see Appendix A, Table A2).

#### **4. Analysis of the Perception of Scientometric Indicators in Leading Russian Universities**

Interest in the problem of scholars' attitudes to scientometric indicators arose initially in the West in the 1990s [24]. One of the most extensive studies of this phenomenon was done in 2012 when an Internet survey of 1704 researchers representing all branches of scholarship from 86 countries was conducted. However, this survey concerned only one scientometric indicator—the impact factor. The results showed that the positive attitude of

scholars to it only slightly exceeded the negative one, but for 88% of the respondents, the IF is important or very important for evaluating scholarly performance in their country [25] (p. 286–289). Research in this direction is continuing, although not very intensively. For example, a comparative analysis of the surveys of 420 humanities scholars in Australia and Sweden concerning bibliometric (scientometric) indicators was recently conducted. The survey found, in particular, that a third (32%) of respondents used scientometric parameters for evaluation or self-promotion in attachments and resumes [26] (p. 927).

In Russia, almost the only work devoted to the topic of interest was an article published in 2016 by Igor Filippov, "How Scholars of Humanities Profile Evaluate Scientometrics," in the journal Siberian Historical Research. Forty people from among the humanities folk took part in the interview he conducted, i.e., the representativeness of the research is clearly insufficient, which the author of the article admitted. At the same time, all of Filippov's respondents were united in the opinion that the proposed new ways of evaluating scientific activity using scientometric indicators are unsatisfactory since they differ in a high degree of formalism, do not allow evaluation of the merits of the work of scholars, and therefore, are unfair. At the same time, the interviews revealed that many respondents were not well aware of scientometrics, its goals, heuristic capabilities, limits, and experience of application: many, even experts, were unable to report their data in the Russian science citation index or in foreign bibliometric databases (specific figures are not given in Filippov's article) [27] (p. 14).

After a preliminary review of the main parameters of respondents, we will go directly to the results of their survey. To the question, "Do you know what 'scientometrics' is?" the most popular answer was: "Yes, I know very well"—54.8% of responses; another 35% of respondents noted "vaguely imagine," and 10.2%—"do not know." Thus, a little more than half of the Russian university teachers are very familiar with scientometrics, and only 10% do not know about it. At the same time, the share of those who do not have a clear knowledge of this discipline is significant—just over one-third. At the same time, only 18.7% of respondents heard the term "scientometrics" for the first time in 2012 when scientometric indicators were widely introduced into practice, while the rest gradually became acquainted with it only recently. By year, it looks like this: in 2012–2013: 11.7%, in 2014–2015: 19.8%, in 2016–2017: 19.4%, in 2018–2019: 15.9% and 14.5% of respondents learned the term "scientometrics" only during our survey. From the given response figures, it can be seen that familiarity with the term gradually increased up to 2016, after which it began to decline.

Most respondents have a neutral attitude to scientometrics (56.9%); 26.9% have a positive attitude to this discipline, and 16.2% have a negative attitude. These figures show that, in general, the teaching community has already adapted to the administration's requirements and is relatively favorable to scientometrics (definitely—about a quarter), and opponents are outnumbered. It is also likely that the desire of some teachers, especially from provincial universities, to indirectly demonstrate their loyalty to the university authorities (whose primary concern is to formally increase the scientometric indicators for reporting to the ministry) has affected this situation.

Of the respondents, 63.3% could boast a good knowledge of all bibliometric databases (RSCI, Web of Science, and Scopus). "Something familiar, but there are no clear ideas" was the answer of 17% of respondents. Only 14.1% of respondents know well the Russian science citation index, 3.2% do not know any of the bibliometric databases at all, and 2.4% of respondents gave their own version of the answer, which in terms of semantic content is closer to a good knowledge of all three databases. Thus, only a very small percentage of Russian teachers do not know anything about bibliometric databases.

Next in our questionnaire were clarifying questions related to scientometric indicators. To the question, "Do you know what the 'h-index' is?" "yes" was answered by 77.4%, "very approximately"—17%, "no"—5.6%. It is interesting to note that respondents know the h-index much better than scientometrics itself. Perhaps the reason lies in the exoticsounding term that periodically pops up in the scientific press, administration orders,

and private conversations of teachers. As for knowing the value of their own h-index, Russian university teachers show much less awareness: in the RSCI database, about a third—32.5%—do not know the value of their own h-index. The situation is even worse regarding knowledge of their indicators in international databases: more than half of Russian respondents have no idea about their h-index figure in the Web of Science (61.5%) and in Scopus (56.2%). This is not surprising, given that not all Russian teachers, and especially young ones, have registered citations in international databases and, accordingly, a nonzero h-index.

To the question, "Do you know what the journal's impact factor and its quartile are?" a little more than half of the respondents answered: "I know both terms very well and understand their meaning" (55.1%). Almost 30% of respondents have a vague idea about them ("I have encountered them somewhere"—29.7%), and 11.3% first learned about the existence of these terms during the survey. The last figure agrees well with the negative answer about knowledge of the term "scientometrics" (10.2%) in one of the previous questions.

The next question was: "Do you keep track of your scientometric indicators in the main bibliographic databases?" It turned out that a little more than half of the respondents monitor their scientometric indicators from time to time (55.1%), 20.9% monitor constantly, and never—almost a quarter (24%). The last figure is very significant and surprising. An additional review of the questionnaire showed that of the 67 people, who ignore their data in WoS, Scopus, and RSCI, most are people, who occupy the lower levels of the official hierarchy (assistants and senior teachers—88%), and most are young men, who, due to their age, do not ye<sup>t</sup> have decent publication indicators that it would make sense to monitor.

To the question, "How does managing your university treat scientometric indicators?" 44.2% of respondents answered: "The university managemen<sup>t</sup> pays some attention to scientometric indicators," 41% believe that the administration regularly monitors the scientometric indicators of teachers, and according to 14.8% of respondents, it ignores them. Thus, most teachers (85.2%) are aware of administrative control and are probably trying to adapt to it, including correcting their publication strategy. At the same time, the authors should note a relatively high percentage of responses (almost 15%) that deny monitoring of scientometric parameters by the university administration. It is obvious that the respondents who answered this way are at the lowest levels of the official hierarchy and simply do not know the administrative policy of university management. It is also striking that most such respondents (56%) are concentrated at the Herzen State Pedagogical University of Russia, where control over the scientometric results of teachers is of secondary importance since the university is not included in the 5-100 Program.

According to our survey, most universities have incentives for high scientometric achievements. Almost half of the respondents (46.7%) reported that their superiors sometimes reward subordinates for good scientometric reporting, and 31.4% said that such incentives are regularly based on the school year's results and during recertification. At the same time, 21.9% of respondents stated that there is no reward for high scientometric indicators. Most likely, we are again faced with the results of the responses of young teachers and specialists, who still have very few publications, their scientometric indicators being minimal. Therefore, they cannot count on any awards from their superiors, and thus they have the illusion that the university does not have any incentive system for scholarly achievements.

To the question, "What specific sanctions are applied to stimulate the increase of scientometric indicators in your university?" the following information was received: the administration of more than a quarter of universities pays bonuses for publications indexed in the WoS and/or Scopus database (28.6%), and almost the same number of universities do not use any sanctions at all (28.3%). One-fifth of respondents (19.8%) pointed to the link between scientometric indicators and wages, while another 3.2% found it difficult to answer. The remaining 20.1% of the surveyed teachers offered their own options, including dismissal/non-promotion to pass the competition due to the lack of publications indexed in the WoS and Scopus databases and various sanctions for different academic branches or departments of the university.

At the end of the general survey, respondents were asked to evaluate the attitude toward using scientometric indicators in universities on a scale from negative to positive in the range from 1 to 5. The results are as follows: 1: 13.5%, 2: 19.2%, 3: 33.9%, 4: 21.7%, 5: 11.7%. Summing up the negative and positive responses, we ge<sup>t</sup> 32.7% and 33.4%, which indicates that the number of respondents who are sympathetic to using scientometric indicators is only 0.7% higher than the number of opponents of this practice. This information is presented in Figure 1.

**Figure 1.** Attitude of respondents to the use of scientometric indicators in the university.

After reviewing the overall indicators and figures, let us ge<sup>t</sup> a little more detailed picture and analyze the answers to the main questions in the questionnaire in terms of gender, age, department, and job affiliation. The data obtained during the respondents' responses show that women are better acquainted with scientometrics than men. For example, almost 63% of women and only 45% of men know this discipline very well; 30.1% and 41% have vague ideas, respectively, and 7% of women and 14% of men are not familiar with scientometrics at all. This ratio in favor of the fair sex can be explained by the greater responsibility inherent in women (this quality is developed by generations of women, who bear the main concern for the welfare of their offspring). At the same time, the greater responsibility and awareness of women affected the answers to other questions in our questionnaire. Although in general, a neutral attitude to scientometrics prevails in almost equal proportions between women (55.9%) and men (58.1%), simultaneously, a significant share of women (32.2%) perceive this discipline positively, which is not true for men (only 21.7%), and also, the number of men, who have a negative attitude to scientometrics (20.2%) is almost double that of women (11.9%). In addition, women are more familiar with bibliometric databases and with the h-index, and they know the value of this index better in their author profile in the RSCI, but slightly worse than men in the WoS and Scopus database. Women are also more attentive to the dynamics of their scientometric indicators in bibliometric databases: 27.3% carry out constant monitoring (men—only 14.7%). Moreover, if the opinion of both sexes mostly coincides in assessing the control of the university administration over scientometric indicators, there are clear discrepancies in the responses to the question about incentives for high scientific achievements (for example, 41.3% of women and only 21.75% of men noted the presence of incentives at the end of the academic year). Some, but not critical, variation between the sexes is observed in responses to the question about sanctions used to stimulate scientometric indicators. Finally, as

expected, the proportion of women positively evaluating using scientometric indicators in universities (36.5%) exceeds the share of those who negatively perceive such practices (26.4%); among men, the proportion is quite different: 40.6% of them think negatively about using scientometric indicators, and only 29.4% approve of their implementation.

Age also has a certain impact on the perception of scientometrics and its main parameters. For example, people of middle age (34–50 years) know best what scientometrics is—66.3%, and only 1.8% of them do not have any idea about it; older people (over 50 years old) have slightly worse knowledge, while the weakest indicators are among young people: only 39.4% of them are well versed in scientometrics, and 18.3% know almost nothing about it. Of respondents, middle-aged people have a positive attitude to scientometrics—34.6% of respondents (only 8.8% have a negative attitude, while the figure for young people is twice as high—16.3%). Again, primarily middle-aged people demonstrate knowledge of bibliometric databases (RSCI, WoS, and Scopus)—77.9% of respondents; awareness is slightly worse among the elderly—62.1%, and young people are noticeably weaker in orientation in the databases—48.1%. In answer to the question: "Do you know what the 'h-index' is?" the leaders are again middle-aged people with 88.5% positive responses; but the ignorance of this index was shown primarily by the elderly—15% of them do not have any idea about it; among the youth, only 4.8% have no idea. At the same time, all three age categories show poor knowledge of the h-index value in their author's profile in various bibliometric databases. However, even here, representatives of the middle generation are ahead—only 22.8% do not know the value of their h-index in the RSCI, while the share of young people who do not have an idea about it reaches 40% and the elderly—35.8%. Even worse is the situation with the personal Hirsch index in foreign databases: among young people, 71.4% do not know their figure in WoS, among the middle-aged—about half (50.9%), and among the elderly—61.2%. In light of what has already been said, the answers to the question, "Do you keep track of your scientometric indicators in the main bibliographic databases?" were fairly predictable. Among young people, only 17.3% of respondents constantly carry out such monitoring, and 37.5% are never interested in it; among middle-aged people, figures were 26.6% and 10.6%, respectively, and among the elderly—16.7% and 25.8%. Here it can repeat what was already mentioned above: young people, due to natural causes, usually have nothing or almost nothing to track in bibliometric databases. However, the number of young people who have a negative attitude toward using scientometric parameters in universities is only slightly higher than the number of supporters (a ratio of 27.1% and 25.5%), but among the elderly, the number of opponents of using scientometric indicators is almost 2 times higher—52.4%. Obviously, the more conservative older generation is skeptical of scientometric innovations, and therefore, there are half as many supporters of scientometric standards among them—25.3%. In contrast, among the middle age group, the number of adherents of scientometric parameters is 43%, while the number of opponents is only 21%, i.e., a mirror ratio compared to the older generation.

There was a definite surprise with the questionnaire analysis regarding the criterion of humanities folk/technicians. Thus, it turned out that scholars in the humanities are relatively better at knowing what scientometrics is than representatives of technical and general scientific specialties: 59.6% against 50%, while on the other hand, 14.8% of technicians and only 5.6% of humanities folk do not know anything about this discipline. Moreover, if the number of both who have a negative attitude toward scientometrics is approximately the same (15.5% and 17%), then for scholars in the humanities, who have a positive attitude toward scientometrics (32.6%) significantly exceeds the share of techies (21.1%). The weaker interest of representatives of natural and technical disciplines in scientometrics may be since they are used to collective scientific work, which brings equal scientometric bonuses to all participants of the project and, therefore, does not stimulate interest in evaluating personal scientific contributions. Some techies, for example, at ITMO University, work in other firms (often with higher earnings) in parallel with their work at the university, and therefore, the data of scientometrics is not critical for their career and material prosperity. However, when answering other questions in the questionnaire,

the differences between humanities folk and technicians are generally insignificant. As our study shows, scholars in the humanities are twice as aware of their data in the RSCI and more often monitor their indicators in the main BDB. Again, on the issue of using scientometric indicators in universities, there is a discrepancy in the views of humanities folk and techies: if only 27.5% of scholars in the humanities perceive introducing metrics negatively, then 36.5% of techies do, and, conversely, 40.1% of humanities folk and 29% of techies welcome introducing scientometric indicators.

Now let us look at the main questions of our questionnaire through the prism of answers from people who are at different levels of the professional and official hierarchy. As a result of analyzing the answers to the question about scientometrics and familiarity with the name of the discipline, a linear pattern emerges: the higher the position and scholarly qualification, the more complete the knowledge. Hence, if only 16.7% of assistants know what scientometrics is, and 30.6% do not have the slightest idea about it, while among professors, the corresponding figures are 86.7% and 0%. In this regard, the attitude toward scientometrics is also very revealing: among university employees of lower categories (assistants, senior teachers), the negative perception of the discipline is clearly predominant—22.2% against 13.9%, and 14.3% of the responses in which scientometrics was evaluated from a positive point of view. Associate professors, by contrast, have a positive perception of scientometrics (36%), and only 8.8% of them have a negative attitude toward it. It is even better perceived by professors (43.3%), though there are many people who have a negative attitude to scientometrics—20% (a figure close to the indicators of assistants and senior teachers). The opposition to scientometrics on the part of lower categories of university employees has explained above—these usually are young people who do not have a significant number of publications and citations and, therefore, are not worthy of scientometric attention. On the other hand, professors and associate professors usually have sufficient symbolic "capital" in the form of publications and citations. This thesis is confirmed by the answers about the respondents' knowledge of bibliometric databases: 29.2% of assistants, 48% of senior teachers, 82.4% of associate professors and 93.4% of professors know all the databases. A similar linear progression is built when answering the question about h-index knowledge (52.8% of assistants and 96.7% of professors are familiar with it). A similar result is observed when answering other questions. Only sometimes do associate professors begin to challenge the palm of superiority (from professors), in particular, when answering the question, "Do you keep track of your scientometric indicators in the main bibliographic databases?" 32% of associate professors admitted to constant monitoring of their scientometric data, while professors gave only 23.3% such answers. At the same time, according to the opinion of a significant share of assistants (20.8%) and senior teachers (28.6%), the university administration ignores scientometric indicators, which confirms the above hypothesis about the relationship of age/position (youth/low status) with the denial of the remuneration system for high scientometric indicators. It is characteristic that only 4.8% of associate professors deny such a system in their universities, and as for professors, there was not a single one who would say that the university administration ignores scientometric data. It is not surprising that 37.3% of assistant professors and 42% of senior teachers do not approve of using scientometric indicators in universities (24.2% and 22%, respectively, have the opposite view), while associate professors are clear supporters of using scientometric data: 51.2% (against only 24%). However, most professors negatively affect metrics (46.7%; only 36.6% have a positive attitude). It may be due to a certain conservatism inherent in older people or a deeper understanding of the shortcomings and formalism of scientometrics criteria. More detailed information on these issues is presented in Figure 2.

**Figure 2.** Differentiation of awareness of scientometric indicators depending on the position held at the university.

Finally, we need to consider the attitude toward scientometrics and its indicators in terms of the respondents' capital/regional affiliation. It is hardly appropriate to analyze in detail the main answers of teachers of St. Petersburg and provincial universities since the latter is superior in all parameters. Thus, among representatives of universities in the northern capital, only 44.4% know what scientometrics is, while among their provincial colleagues, this figure reaches 64%; all three bibliometric databases are well-known by 55.6% of St. Petersburg respondents and 70% of teachers at provincial universities. At the same time, residents of St. Petersburg evaluate scientometrics mostly negatively—39.2% of negative reviews against 29.3% of positive ones, while in the provinces, the opposite picture is observed (27.4% and 36.8%). These figures can be explained by the fact that teachers in provincial universities want to make a better impression and protect themselves from the displeasure of their superiors. On the other hand, the most depressing indicators among St. Petersburg teachers were given by employees of the Herzen State Pedagogical University, which was selected as a control institution not included in the 5-100 Program. For example, out of 30 respondents from this university, only 3 people know well what scientometrics is, and the rest either have a vague idea of what it is (16) or do not know at all (9); again, only three respondents know all three bibliometric databases, etc. This indicates that scientometrics and its data have not ye<sup>t</sup> been adequately applied in ordinary Russian universities, in contrast to a limited number of leading universities in the country.

# **5. Discussion**

The research conducted has shown that among the teachers of the leading Russian universities, the vast majority have more or less clear ideas about scientometrics. In general, respondents have a relatively positive perception of this discipline and do not object to using its indicators. This shows that in just a few years, the representatives of the teaching corps have managed to adapt to the requirements of the university and ministerial administrations. The survey revealed two statistical patterns: the better teachers know scientometrics, the better they feel about it, and vice versa; the younger the respondent and lower their position, the more negatively they feel about scientometrics and the less they know about its parameters. At the same time, the fact of a better attitude toward scientometrics and knowledge of scientometric standards and their monitoring by representatives of the humanities, in contrast to natural and technical disciplines, turned out to be surprising. Similarly, but even more clearly, the teachers of provincial universities are superior in

all respects to their colleagues from St. Petersburg. This was probably due in some way to the inclusion of data from the Herzen State Pedagogical University of Russia—which was selected as a control organization that was not included in the 5-100 Program—in the general sample of St. Petersburg universities. As expected, scientometric indicators at HSPU play a secondary role, and the knowledge of its teachers about scientometrics is significantly inferior to the knowledge of representatives of leading universities included in the governmen<sup>t</sup> program.

Summing up the results of our research, we should say that we are only at the very beginning of the road, and in the future, we need more extensive mass surveys and indepth interviews, as well as a comparison of Russian data with similar data from foreign colleagues for a comprehensive illumination of the chosen topic

**Author Contributions:** Conceptualization, A.V.G. and D.S.B.; methodology, A.V.G. and V.V.L.; validation, A.V.G. and V.V.L., formal analysis, D.S.B. and V.V.L.; investigation, V.V.L. and D.S.B.; data curation, A.V.G. and V.V.L.; writing—original draft preparation, A.V.G., D.S.B. and V.V.L.; writing—review and editing, A.V.G. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Data Availability Statement:** Research data can be provided by the authors upon request.

**Acknowledgments:** The authors wish to thank Richard L. Bland (University of Oregon, USA) for his excellent translation of the manuscript of this article.

**Conflicts of Interest:** The authors declare no conflict of interest.

## **Appendix A**


#### **Table A1.** Survey form and results.



#### **Table A2.** Data on the main universities studied.



#### **Table A2.** *Cont*.

# **References**


#### *Communication* **Author-Suggested, Weighted Citation Index: A Novel Approach for Determining the Contribution of Individual Researchers**

**Tanmoy Konar**

> West Bengal Police Housing and Infrastructure Development Corporation Limited, Salt Lake, Kolkata 700091, India; 3tanmoykonar@gmail.com

**Abstract:** A novel scientometric index, named 'author-suggested, weighted citation index' (*Aw*-index) is proposed to indicate the scientific contribution of any individual researcher. For calculation of the *Aw*-index, it is suggested that during the submission of a scholarly article, the corresponding author would provide a statement, agreed upon by all the authors, containing weightage factors against each author of the article. The author who contributed more to the article would secure a higher weightage factor. The summation of the weightage factors of all the authors of an article should be unity. The citation points a researcher receives from a scholarly publication is the product of his/her weightage factor for that article and the total number of citations of the article. The *Aw*-index of any individual researcher is the summation of the citation points he/she receives for all his/her publications as an author. The *Aw*-index provides the opportunity to the group of authors of a multi-authored article to determine the quantum of partial citations to be attributed to each of them. Through an illustrative example, a comparison of the proposed index with the major scientometric indexes is presented to highlight the advantages of the *Aw*-index.

**Keywords:** scientometric indexes; scholarly publications; scientific contribution of individual; authorsuggested weighted citation index

# **1. Introduction**

The scientific contribution made by a researcher is often approximated in terms of the impact of his/her scholarly publications. This has become an important parameter for appointment in academic positions, research collaboration, receipt of research grants, etc. [1–8]. The impact of the scholarly publications of a researcher is related to the citations of his/her publications quantified in terms of different scientometric indexes such as cumulative citations, *h*-index [9,10], *i*10-index [11], etc. The cumulative citations provide the total number of citations received by all of the scholarly publications of a researcher. The *h*-index of a researcher is defined as the highest value of *h* such that the researcher has at least *h* publications, each of that have been cited at least *h* times. On the other hand, the *i*10-index of a researcher indicates the number of publications authored by him/her with at least 10 citations. However, as these indexes do not adjust their values for multiauthored publications, sometimes they may put forward a misleading picture. In the present age of rising multi-authored publications [12–17], much research is devoted to determining the co-authorship-adjusted impact of a researcher. However, the scientific community is still divided on the methodology to be adopted to quantify the proportion of credit to be attributed to a particular author of a multi-authored scholarly article. Several indexes, such as, *hI*-index [18], *hf*-index [19], *hm*-index [20,21], etc. have been proposed for determination of co-authorship-adjusted impact of a researcher. The *hI*-index is determined by dividing the *h*-index by the average number of authors of the publications in the *h*-core (that is, in the *h*-index defining set of publications). To determine the *hf*-index, fractional citations are derived by dividing the number of citations by the number of authors for each publication. The *hf*-index is the number of publications of a researcher for which the fractional citations are at least equal to *hf* . For determination of

**Citation:** Konar, T. Author-Suggested, Weighted Citation Index: A Novel Approach for Determining the Contribution of Individual Researchers. *Publications* **2021**, *9*, 30. https://doi.org/10.3390/ publications9030030

Academic Editor: Oleg V. Mikhailov

Received: 17 May 2021 Accepted: 7 July 2021 Published: 12 July 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

the *hm*-index, the publication list of a researcher is sorted by the number of citations with the publication having the highest number of citations ranked first. Then, the effective rank of a publication is determined as, *reff*(*r*) = ∑*r <sup>n</sup>*=<sup>1</sup>[1/*a*(*r*)]. Here, *a*(*r*) is the number of authors of the *r*th ranked publication. The *hm*-index is the value of the *reff* of the highest ranked publication for which the number of citations is not less than *reff* . There are more such imperial formulation-based approaches that provide partial citations to each author when a multi-authored scholarly article is cited in another article. *L*-index [22], *k*-index [23], eigenfactor-derived scoring system [24], *hIa*-index [25], *RA*-index [26], pure *h*-index [27] represent a few examples. The main drawback of these approaches is that all the authors of a multi-authored article are given equal weightage. In a slightly different approach, the *Z*-index [28,29] and the *Ab*-index [30] provide additional weightage to the first author and the corresponding author, and equal weightage to all other authors.

Recently many of the leading journals have made it compulsory to share the detailed description of the contributions of each author to the published article through Contributor Role Taxonomy (CRediT) [31–34]. From the analysis of CRediT statement of a large number of scholarly publications, it is observed that the contributions of all the authors are not always equal in a multi-authored publication. In view of this, a novel scientometric index in the form of the 'author-suggested, weighted citation index' ( *Aw*-index) is proposed. The *Aw*-index is expected to quantify the scientific contribution of any individual researcher, taking his/her possible authorships in multi-authored publications into account with appropriate weightage suggested by the authors of the publications themselves.

# **2. Methods**

The *Aw*-index requires the corresponding author to provide a 'contribution weightage statement', containing a weightage factor against each author of the article, during the submission process of any scholarly article. The 'contribution weightage statement' should be agreed upon by all the authors. Let *N* be the number of authors of a scholarly publication and the weightage factor of the *i*th author is *wi*. Then, *wi* should satisfy the following two conditions.

$$0 < w\_i \le 1,\tag{1}$$

$$\sum\_{i=1}^{N} w\_i = 1,\tag{2}$$

The author who contributed more to the article would secure a higher weightage factor. The author of a single-authored article would ge<sup>t</sup> a weightage factor equal to 1. Later, when an article would be cited, the article would receive 1 citation point for each citation. While calculating the *Aw*-index, the citation point would be distributed among the authors of the article based on the weightage factor.

For determination of the weightage factors of the authors of an article, intellectual impact should be given paramount importance. However, the co-authors may select a few additional parameters based on mutual agreement. For articles with fewer authors, the weightage factors for each co-author can be determined in a straightforward approach fulfilling the conditions mentioned above. However, when a large number of authors contribute to an article [12,14,15], the weightage factors of the authors may be determined through an indirect approach. For this scenario, it is proposed that the authors should be grouped based on the activities they are involved in. For example, one group may be involved in experimental works, the other group may be doing numerical simulations, another group may be involved in data acquisition, a group may be developing and validating a mathematical model, and so on. The weightage factors for each group would be determined first. After that, the weightage factor of a group would be further divided into the members of the group based on their contributions within the group. It may so happen that a particular researcher is involved in more than one group. Then, his/her total weightage factor for the article would be the summation of the weightage factors he/she would receive from different groups.

Now, consider a scholarly publication gets a total *C* number of citations. Then, the contribution of that publication to the *Aw*-index of the *i*th author would be *C* × *wi* citation points. The *Aw*-index of any researcher would be the summation of the citation points he/she receives from all the articles in which he/she had contributed as an author or as a co-author.

Let *X* be the number of scholarly publications a researcher produces as author or co-author. His/her *n*th publication, for which the weightage factor of the researcher is *wn*, receives a total *Cn* number of citations. Then the *Aw*-index of the researcher can be expressed by the following.

$$A\_w - \text{index} = \sum\_{n=1}^{X} \mathbb{C}\_n w\_n \tag{3}$$

In Equation (3), *wn* indicates the contribution factor and *Cn* indicates the quality factor as a better scholarly publication is expected to have higher citations. Through the summation of citation points of all the publications of the researchers, the quantity factor is also taken care of in the *Aw*-index. Thus, the *Aw*-index is expected to become a useful indicator of the scientific contribution of any individual researcher as it gives an estimate of the significance, importance, and broad impact of a researcher's cumulative scientific effort.

# **3. Results**

Let *R*1 to *R*6 be a group of six researchers. They collectively produced ten scholarly publications designated as *p*1 to *p*10, each of which has at least two researchers from the group of six as authors. The lists of authors of the publications are given in Table 1. For simplicity, it is assumed that none of the six researchers considered for the present study have produced any other scholarly publication as authors or co-authors.

**Table 1.** Lists of authors for publications *p*1 to *p*10.


Now, for the determination of the *Aw*-index, the weightage factors of the researchers for the publications are required. It may be noted that the *Aw*-index is a new concept and presently the journals do not have the provision for submission of the 'contribution weightage statement' for the authors during the submission of scholarly articles. Hence, to illustrate the concept of the *Aw*-index, let us assume that the values of weightage factors, *w*, of the researchers for the publications are as given in Table 2. The number of citations received by the publications are also given in Table 2.

Now the *Aw*-indexes of the researchers considered for the present study are calculated using Equation (3) and plotted in Figure 1. Among the six researchers, *R*6 has the highest *Aw*-index of 25.8. On the other hand, *R*4 has the lowest *Aw*-index of 6.3. From Table 1, it can be observed that both *R*1 and *R*6 have contributed to all 10 scholarly publications under consideration. However, as *R*6 contributed more than *R*1 in most of the publications (see Table 2), *Aw*-index of *R*6 is much higher than that of *R*1. Again, *R*2, despite having a lesser number of publications, has a higher *Aw*-index than *R*1. This is because *R*2 has a high proportion of contribution in the publications he/she features as co-author. This ability to include the proportion of contribution in the scientometric indexes of individual researchers is the main feature of *Aw*-index.


**Table 2.** Weightage factors of the researchers and number of citations for publications *p*1 to *p*10.

**Figure 1.** *Aw* -index of the six researchers considered for present study.

Table 3 shows the comparison of *Aw*-index with the major scientometric indexes for the six researchers. Both the indexes that do not make adjustment for multi-authorship, such as cumulative citations, *h*-index, and *i*10-index, and the indexes that make adjustment for multi-authorship, such as *hI*-index, *hf*-index, and *hm*-index are considered for the comparative study. Table 3 shows that all the indexes except the *Aw*-index have an identical value for the researchers *R*1 and *R*6. Only the *Aw*-index is able to quantitatively identify that *R*6 contributed more than *R*1 in most of their publications. Researcher *R*2 ranked fourth among the group as per cumulative citations, *h*-index and *hI*-index. In contrast, his/her rank is third as per *i*10-index and *hf*-index. The *hm*-index places him/her in the fifth spot. Although *R*2 has a lesser number of publications, he/she put up more effort behind those publications than most of his/her co-authors. This ensures higher weightage factors for him/her in those publications (see Table 2). As commonly used bibliometric indexes, provide equal weightage to all the co-authors, they do not indicate the extra effort of *R*2. On the other hand, the *Aw*-index gives recognition of the extra effort of *R*2 and thus places him/her in the second spot among the group of researchers. The *Aw*-index combines (a) the research effort of an individual behind his/her publications with (b) the quality of his/her publications in terms of citation and (c) the quantity of his/her publications as it includes all of his/her publications in the calculation.


**Table 3.** Comparison of *Aw*-index with major bibliometric indexes.

# **4. Discussions**

The *Aw*-index is developed to enhance clarity in the attribution of credit to the researchers for the publications they produced as authors or co-authors. The *Aw*-index covers the quality and quantity of publications as well as the research effort of an individual researcher behind the publications. It is expected that upon adaptation by the journals collectively, the *Aw*-index could become a reliable indicator of the scientific contribution of individual researchers in the future. It is not possible to start using the *Aw*-index immediately, as the weightage factors of the authors of already published articles are not available with the journals. It is anticipated that the journals would take time and require review before they start asking the authors for submission of their weightage factors on contribution. A similar thing has happened with the attribution of contributorship. In the year 1997, Rennie et al. [35] proposed the concept of contributorship, ultimately leading to the development of CRediT [31] in 2014. Now, most of the leading journals are publishing author contributions statements with the articles. In the same line, it is presumed that the implementation of the *Aw*-index by the journals would take some time.

The concept of the weightage factor is expected to reduce the chances of denial of authorship on the ground of smaller contributions. When the contribution of a researcher in an article is small, instead of denying authorship, he/she may be given authorship with a smaller weightage factor. With the system of weightage factor in place, one researcher with a larger contribution would be more open to accept a smaller contributor as co-author because the effort of the larger contributor would be recognized with a higher weightage factor. However, and only if a smaller contributor fulfills the minimum requirements for authorship, he/she could be included as co-author with appropriate weightage factor. There are several guidelines for the minimum requirements for authorship [36–40]. However, a universal guideline in this regard is ye<sup>t</sup> to be developed. It may be noted that the determination of minimum contribution for granting authorship is out of the present scope of this article.

In order to be sure about the correctness of the assigned weightage on contributions, the journals may frame a rule that makes submission of the 'contribution weightage statement' signed by all the co-authors compulsory during the initial submission of a manuscript. Alternatively, the journals may develop an automatic verification system through auto-generated e-mails. It may be noted that many journals already follow a system for verification of authorship of the submitted articles through auto-generated e-mails. The process of verification of the 'contribution weightage statement' can easily be integrated with that system.

Sometimes there may be conflict among the co-authors on different issues [41], including the distribution of the weightage factors. To avoid this, co-authors should discuss and agree on the goals of collaboration, roles of individuals, guidelines for authorship, contingencies and communication strategies, and methods for handling conflicts, including conflicts of interest [42], at the early phase of the research process. As the research progresses, the roles of co-authors may change and even co-authors may be added or

dropped. This may lead to variation in the expected weightage factor of a co-author. All such alterations should be done through open, honest, and respectful discussion [43].

The risk of a co-author agrees to a manipulated 'contribution weightage statement' under some kind of pressure cannot be completely ruled out. However, when the coauthors adhere to the best authorship practices and guidelines [38,44–47] this kind of situation would not occur. Moreover, studies to develop a more objective approach for determination of percentage contributions of the co-authors may be taken up in the future to avoid any manipulation or conflict.

# **5. Conclusions**

To quantify the scientific contribution of any individual researcher, a new scientometric index, named *Aw*-index, is proposed. The concept of *Aw*-index is unique as it determines the weighted partial citations for each author of a multi-authored article based on the suggestion of the group of authors of the article. It is proposed that a statement containing a weightage factor against each author would be submitted during the submission process of any scholarly article. The weightage factor would be given to a particular author based on the quantum of the contribution of that author, and the statement of weightage factors would be approved by all the contributing authors. The *Aw*-index of an individual researcher would be the summation of the citation points, given by the product of his/her weightage factor, and the total number of citations for an article for all of his/her scholarly publications. The *Aw*-index is expected to provide an authentic evaluation of the scientific contribution of a researcher, as the group of researchers who produces an article is the best judge to determine the proportion of contribution made by each member of the group in the article. The method for calculation of the *Aw*-index of any individual researcher is described and, through an illustrative example, the effectiveness of the *Aw*-index is shown. A comparison among the *Aw*-index and other commonly adopted scientometric indexes is presented. It is shown that the *Aw*-index of a researcher depends upon the quality of his/her publications, the quantity of his/her publications, and the proportion of his/her contributions in those publications. The main drawback of *Aw*-index is that it cannot be calculated for the already published articles as, presently, the journals do not have the provision for submission of the statement containing weightage factors for the authors. However, upon adaptation by the journals collectively, *Aw*-index has the potential to become a reliable indicator of the scientific contribution of individual researchers in the future.

**Funding:** This research received no external funding.

**Data Availability Statement:** All data are available upon contacting the author.

**Acknowledgments:** The author thanks the peer reviewers for pointing out areas where the article needed clarification.

**Conflicts of Interest:** The author declares no conflict of interest.

# **References**


#### *Review* **Intellectual Capital: A Review and Bibliometric Analysis**

**Wilder Quintero-Quintero \*, Ana Beatriz Blanco-Ariza and Manuel Alfonso Garzón-Castrillón**

> Doctorado en Administración, Facultad de Administración y Negocios, Universidad Simón Bolívar, Barranquilla 080020, Colombia; ablanco1@unisimonbolivar.edu.co (A.B.B.-A.); mgarzon2@unisimonbolivar.edu.co (M.A.G.-C.)

**\*** Correspondence: quinterowilder@ufpso.edu.co; Tel.: +57-315-724-9833

**Abstract:** Intellectual capital is managed by competencies and the development of information and communication technologies, which have seen high growth and impact in higher education institutions related to scientific publications. The main objective of this study was to provide a summary of the general review of studies related to intellectual capital around the world. Methodology: the Bibliometric analysis was carried out using the Bibliometrix library and BiblioShiny platform of the RStudio® software through the data obtained from the Scopus database. Findings: in total, 389 documents in the Scopus database used "capital", "intellectual", "research" and "institutions" as keywords with a growth rate of 2.34% every year from 1947 to 2021. The publications were written by around 866 authors, mainly from the USA, the UK, and Spain. Original value: the data obtained show that intellectual capital has been important and relevant in the scientific publications of the last six years, which were related by the number of authors by institutions.

**Keywords:** bibliometric; human capital; universities; trends; higher education

#### **Citation:** Quintero-Quintero, W.; Blanco-Ariza, A.; Garzón-Castrillón, M. Intellectual Capital: A Review and Bibliometric Analysis. *Publications* **2021**, *9*, 46. https://doi.org/ 10.3390/publications9040046

Academic Editor: Oleg V. Mikhailov

Received: 8 September 2021 Accepted: 12 October 2021 Published: 19 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

#### **1. Introduction and Overview**

The study of intellectual capital at a global level appeared from the 1960s, with the rise of the knowledge economy, the process of managemen<sup>t</sup> by competencies, and the development of information and communication technologies (ICTs), to generate competitive advantage in economies that prioritize knowledge and learning in organizations in different sectors. The evolution of the study on intellectual capital has been established with the conviction of valuing intangible assets in organizations. In 1963, the term "human asset accounting" was used to include people in the financial statements of organizations, recognizing the potential value for companies and fixed assets [1]. Later, in 1967, the term intellectual capital appeared for the first time, established by the economist J.K. Galbraith, who considered it as the result of an "intellectual action" rather than just knowledge, creating value as another asset in the traditional economy [2]. In the 1970s, the term "human asset" appeared to refer to people who collaborate in organizations, proposed by Flam Holtz in different studies [3].

The historical development on the study of intellectual capital was consolidated during the 1990s and later. Nonaka and Takeuchi [4] defined it as the "Ability of a company to generate new knowledge, disseminate it among the organization members and materialize it in products, services, and systems". Similarly, intellectual capital is made up of three dimensions. The first is human capital, which is related to employing skills. The second is structural capital, which is related to the internal component. The third is relational capital, which is constituted by the external component of the organizations [5].

On the other hand, in the mid-1990s, four stages were established on intellectual capital. Considering the first stage, intellectual capital and its importance in creating competitive advantages in companies is created. There is also interest in the measurement of intellectual capital through some attempts at the creation of norms and standards for empirical investigations [6,7]. The second stage arose at the end of the 1990s, where efforts were

developed to recognize intellectual capital as an academic discipline. Different models were developed for their measurement to increase the competitiveness of organizations [8,9]. The third stage was presented in 2004, with the interest of measuring the commercial implications of intellectual capital in practice through empirical work to determine its measurement and to compare theory with practice [8,10,11]. Finally, the fourth stage is a complement to the previous stage, focusing on the study of the future of intellectual capital, in which it is intended to go beyond CI reports by expanding to broader open and collaborative ecosystems to understand ethical, social, and environmental impacts according to an ecosystem approach to IC [12,13]. In this same period, the concept of structural capital, and its distinction from human capital, appeared, where structural capital is considered a product of human capital and is mainly made up of client capital, innovation capital, and intellectual property of the human resources linked to the organization [8,14,15].

The emergence of intellectual capital is based mainly on the theory of resources and capacities of organizations. According to Reed et al. [16], the intellectual capital approach allows defining the intangible resources and capacities that organizations must possess to obtain a competitive advantage. It identifies three dimensions of intellectual capital, thus providing greater precision. For Reed et al. [16] and Foss et al. [17], the theory based on the intellectual capital of organizations represents a specific aspect of the more general theory of resources and capabilities, considering three resources that have been theoretically linked to competitive advantage through the knowledge created and accumulated in the three components of the capital of the company: in its people (human capital), its social relations (social capital), and its systems and processes (organizational capital) [18].

Considering the evolution throughout history on the importance of this issue in organizations, intellectual capital is considered the organizational knowledge and organizational processes necessary for the competitiveness of companies. Therefore, both should be pursued jointly. In this way, Ratogi [19] argues that intellectual capital (IC) and knowledge managemen<sup>t</sup> (KM) represent organizational activities related to knowledge from stock to knowledge management. Knowledge managemen<sup>t</sup> and intellectual capital are naturally connected in a bidirectional way, where IC represents the stock of knowledge in terms of human capital, structural capital, and relational capital [7,20].

#### *1.1. Studies about Intellectual Capital around the World*

Several studies have been developed in different organizations and sectors of the world economy. Spanish companies [21], technology and ICT companies [22], and Pymes [23] have found that IC positively influences human resource managemen<sup>t</sup> practices based on knowledge and performance in innovation, partially favoring structural and relational capital through human capital. Likewise, the IC components referring to client capital, structural capital, social capital, technological capital, and spiritual capital are positively related to the organizational performance of higher education institutions.

Intellectual capital in higher education institutions (HEIs), according to Leitnet et al. [24] and Sanchez et al. [25], is called the set of intangible assets that allow educational institutions to transform material, financial, and human resources into a system capable of creating value for their clients. Therefore, they are the most valuable resources that teachers, researchers, administrative personnel, managers, and students possess, including their organizational relationships and routines. This set of intangible assets is constituted by its processes; its capacity for innovation; the patents and intellectual property rights obtained; the tacit knowledge of its members; their capacities and skills; the recognition of society; its networks of collaborators, allies, and contacts; and scientific research processes, among other resources. Intellectual capital is the set of intangibles that "allows an organization to transform a set of material, financial and human resources into a system capable of creating value for stakeholders". Ramírez-Córcoles Y. and Manzaneque-Lizano [26] argued that HEIs and research organizations are the ideal organizations to apply the theory of intellectual capital since these institutions create knowledge through scientific research or teaching processes.

Giustina et al. [15], determined that the conceptual framework for IC managemen<sup>t</sup> creates multi-stakeholder participation within the university network, where the main components are the ultimate goal of a university (what); the collective human capital to achieve the goal (who); the processes activated within the university (how); and, finally, the motivations behind achieving the goal (why). Likewise, Guistina et al. [27] studied the collective intelligence approach to managing intellectual capital, determining that IC managemen<sup>t</sup> must change and incorporate an ecosystem perspective, reflecting the fourth stage of IC research.

Elena-Mădălina et al. [28] analyzed the points of view and practices of 210 university students from European developing countries, concluding that the policies and practices of universities have a positive and significant impact on the assessment of Internet-based IC components and showing that 63% of the professional and organizational competitiveness of HEIs is determined by the exploitation of IC integrated into online academic networks. Zhuravlev et al. [29] evaluated the effectiveness of education as the most important factor in forming IQ, based on modern Russian and foreign studies, on historical examples from the late nineteenth century, concluding that the labor productivity of more-educated workers is higher than the less-educated. Therefore, continuing education is the most important factor influencing employees' earnings and attitudes towards work, labor efficiency, and development of the state economy.

Passaro et al. [30] investigated the impact of higher education on the emergence of entrepreneurial intention and human capital in students and academics, concluding that there are significant differences between the two samples concerning the level and the specific characteristics of entrepreneurial education that are the key factors for the development of business intent and human capital. Di and Corsi [31] analyzed the contribution of intellectual capital to the development of the third mission in 71 Italian universities financed by the governmen<sup>t</sup> in the period 2004–2014, concluding that there is a significant revelation of IC in the quality assessment model; in the same way, it was identified that the activities of the third mission have a positive impact on the university ecosystem, with a relevant performance of structural capital and relational capital in the development of the third mission. Veltri and Puntillo [32] analyzed whether the performance managemen<sup>t</sup> systems (PMS) of the universities consider IC managemen<sup>t</sup> as a criterion to evaluate their managers, a case study from the Universidad de Calabria, which is far from considering IC substantially as a key criterion to evaluate your managers.

Ramirez et al. [33] proposed a model of an intellectual capital product of a study carried out in Spanish public universities to indicate which intangible elements it is necessary to measure, and a new framework for the measurement and managemen<sup>t</sup> of intellectual capital was presented that helps universities in the method of presenting useful information for its stakeholders, contributing to greater transparency, accountability, and comparability in the higher education sector. Eugenia et al. [34] demonstrated that intellectual capital positively and directly influences sustainable development practices, and said practices contribute significantly to the quality of life in 738 students and 587 professors/researchers in seven Portuguese higher education institutions. Chatterji and Kiran [35] studied how universities can create a knowledge economy, using data collected from 13 universities in North India. The findings reveal that human capital has a significant influence on the performance of a university, and relational capital partially mediates this effect.

Naranjo and Chu [36] designed a model and instrument to measure its structural capital at the Universidad Nacional Autónoma de México (UNAM) from 2011 to 2012. The results showed that the university has the human and technical resources necessary to generate a competitive advantage, since it permanently invests in technological infrastructure and the R+D+i process, becoming a grea<sup>t</sup> advantageand taking into account strengths in terms of structural capital, which are based on measurements of communication channels and annual studies on the culture and organizational climate.

Kichuk et al. [37] studied the impact of the knowledge economy in the education sector through the classifications in international rankings of Ukrainian universities, demonstrating the importance of IC managemen<sup>t</sup> in higher education institutions in the current knowledge economy and the capacity they possess to increase the competitive advantage in the positioning and quality of these institutions worldwide.

The study developed by Nicolò et al. [38] established the relationship between academic performance and the voluntary disclosure of Intellectual Capital (IC) in 59 Italian public universities, showing that educational institutions with the highest academic performance transfer or disseminate a greater quantity and quality of information on IC based on its subcomponents (human capital, structural capital, relational capital). In the same way, Aversano et al. [39] showed that the dissemination of IQ in 60 Italian public universities is carried out through human capital in a quantitative way related to the strategic framework and organizational performance. On the other hand, Brusca et al. [40] developed a comparison between the disclosure of intellectual capital (IC) on the web pages of 128 European universities (Greece 22, Italy 58, and Spain 48) and its correlation with the academic classifications of the World Ranking, showing that the universities that reveal the most information about IC are the largest according to the number of students, demonstrating a positive correlation between the level of diffusion of IC on the web and the academic ranking of universities. Ramírez et al. [41] examined the disclosure on the web of 50 Spanish public universities in 2016, showing that human capital was the most publicized category and, to a lesser extent, relational capital, according to the size and internationality of the university with the purpose of satisfying the information needs of their stakeholders. It was recommended that Spanish universities present higher-quality information on their financial relationships, student satisfaction, and collaboration between universities and stakeholders.

Yudianto et al. [42] showed that good university governance and IC positively and significantly influence the performance of state universities–legal entities (SU–LE) and state universities–public service agencies (SU–PSA) in Indonesia, which contributes to developing science and technology, increasing the competitiveness of these institutions. In the same way, Limón et al. found 102 researchers from four state universities in Mexico in the area of business and administration, and they concluded that these institutions are a source of knowledge. Therefore, the IC has greater relevance to obtain a competitive advantage and improve the performance that allows them to generate value for customers.

#### *1.2. Intellectual Capital (IC) on Scientific Production*

Scientific production is the direct result of research activity, from which products such as research articles, books, book chapters, patents, utility models and technological products, architecture, and design, among others, are derived [43,44]. Through the literature review on intellectual capital, the following hypotheses (H) were established, taking into account Figure 1:

**Figure 1.** Intellectual capital scheme.

**Hypothesis 1 (H1).** *Human capital has a positive and significant effect on intellectual capital.*

**Hypothesis 2 (H2).** *Structural capital has a positive and significant effect on intellectual capital.*

**Hypothesis 3 (H3).** *Relational capital has a positive and significant effect on intellectual capital.*

**Hypothesis 4 (H4).** *Intellectual capital has a positive and significant effect on competitive advantage in HEIs.*

**Hypothesis 5 (H5).** *Intellectual capital has a positive and significant effect on scientific production with the number of authors.*

On the other hand, the main component principals on intellectual capital are H1, H2, and H3 according to the results obtained by Inkinen [45]. A high relation on the IC, according to the interactions, combinations, and mediations in the organization performance and innovation of the organizations, was present.

#### 1.2.1. Human Capital (HC)

Human capital is created by employees by their inherent and acquired knowledge, skills, talents, and competencies. In this way, HC can be considered as a dynamic index and a very important factor for the prosperity of the organization today [46]. On the other hand, HC refers to the knowledge (explicit or tacit and individual or social) that people and groups possess and their ability to generate it, which is useful for the strategic purpose (mission and vision) of the organization. Ultimately, human capital is integrated by what people and groups know and learn and whether they share that knowledge with others. Once codified, they can benefit the organization. Within human capital and appropriate to the characteristics of each organization, elements such as values and attitudes, aptitudes (knowing), and capacities (knowing how to do) can be considered, according to Bueno et al. [47].

From Bontis [20], HC is a production factor in the organization and is a combination of intelligence, knowledge, and skills, which provide each organization its special character. People are elements of the organization that are capable of learning, innovating, thinking creatively, initiating, and making changes. Simultaneously, it is a necessary assumption for successful long-term performance in the market because it acts as a source of innovation and strategic renewal in organizations.

Human capital in higher education institutions, according to Ramirez et al. [33], Casanueva and Gallego [48], and Secundo et al. [49], is the set of explicit and tacit knowledge of the personnel of universities and public research bodies (professors, researchers, managers, and administration and services personnel) acquired through formal and informal education and updating processes included in their activities [50].

#### 1.2.2. Structural Capital (SC)

Structural capital or organizational capital presents the institutional knowledge created and owned by the organization that is stored in databases, manuals, etc. In this type of capital, there are work processes, organizational norms, technological processes, know-how, brand, etc. [46]. According to Bueno et al. [51], structural capital is the set of knowledge and intangible assets derived from action processes that are the organization's property and that remain there when people leave it. It is made up of organizational capital (a set of intangibles of an explicit and implicit nature) and technological capital (a set of intangibles directly linked to the development of the organization technical system).

Structural capital in higher education institutions, according to Casanueva and Gallego [48] and Secundo et al. [49], is the explicit knowledge related to the internal process of dissemination, communication, and managemen<sup>t</sup> of scientific and technical knowledge in universities. It also integrates the incorporated, internalized, systematized, and processed knowledge of each institution through a succession of organizational routines, evaluating

variables with culture, strategy, organizational structure, intellectual property, technologies, support processes, and recruitment of knowledge and innovation processes.

#### 1.2.3. Relational Capital (RC)

Relational capital, or social capital, presents sets and flows of knowledge resulting from relationships within the organization and outside of it. They are characteristics of the life of society (relationships, norms, expectations, and responsibilities) that allow participants to work together effectively in the achievement of goals. It is related to the institutions, relationships, and norms that create the quality and quantity of social interactions in society [46].

According to Bueno et al. [51], RC can be defined as the set of knowledge incorporated into the organization and the people who make it up as a consequence of the value derived from the number and quality of relationships that are continuously maintained with the different market agents and with the society in general. It is made up of business capital (which refers to the value of the relationships it maintains with the main agents linked to its fundamental business process representing the organization) and social capital (which refers to the value that the relationships represents for the organization). It is maintained with the other social agents that act in its environment.

Relational capital in HEIs, according to Casanueva and Gallego [48] and Secundo et al. [49], establishes the broad set of economic, political, and institutional relations developed and maintained between the university and non-academic partners: companies, NGOs, public authorities, local government, and society in general, and it also collects how the university is perceived: its image, attractiveness, reliability, etc.

#### *1.3. Models for the Measurement of Intellectual Capital*

Regarding the measurement of intellectual capital, there are various models according to the strategic–corporate approach for higher education institutions, as shown in Table 1.

Martin et al. [57], Gernard and Nick [58], and Ramirez and Manzaneque [26] have established that the different models of intellectual capital propose different typologies according to the characteristics, needs, and types of organizations, where most of the research carried out highlights three major components concerning human capital, structural capital, and relational capital. In some cases, structural capital is subdivided into organizational capital and technological capital.

Intellectual capital in education has provided evidence of growing academic interest as a relevant field of research. However, in Colombia, there are few related jobs in the higher education sector. The proposed measurement approaches have been developed for the most part in the European context, making their adaptation and application difficult in Colombian universities. Likewise, empirical studies on the nature of the interrelationships between the dimensions of intellectual capital (human, structural, and relational capital) and their effect on the performance of universities have not yielded sufficient evidence [59].

In this review article, a Bibliometric analysis was carried out considering the data collected directly from the *Scopus* database on the topic of intellectual capital, and thus it analyzed growth trends over the years about the publication of scientific material (articles and books mainly), journals in which the authors publish, the main countries of publication, and collaboration networks. Likewise, this study provides perspectives and trends on this important field in the social sciences regarding the implications of human, relational, and social capital and structural elements involved in intellectual capital.


**Table 1.** Intellectual capital models of the strategic–corporate approach.

#### **2. Materials and Methods**

#### *2.1. Bibliometric Analysis*

The Bibliometric analysis was carried out using the Bibliometrix library and BiblioShiny platform of the RStudio® software [60]. Besides, the VOSviewer software was used to obtain the relationship between countries and keywords. Figure 2 shows the principal steps of the workflow applied to the data analysis obtained from the *Scopus* database.

#### *2.2. Data Collection*

The data was compiled on 19 January 2021 directly from *Scopus* database scientific publications on intellectual capital in higher education institutions for documents published from 1947 with the following search equation in general way: (capital AND intellectual AND research AND institutions), as nowadays it is one of the important data sources to obtain scientific publications [64]. The results of the Bibliometrix analysis were used to expose the most relevant topics across the time using specific keywords and quantitative information of the publications and journals (title, abstract, author, keywords, total citation per document, and filiation, among others).

**Figure 2.** Workflow of the Bibliometric analysis obtained from García-León et al. [61–63].

#### **3. Results and Discussions**

#### *3.1. Statistical Results*

Table 2 shows the general results that were analyzed in the R studio software. Considering the methodology of Figure 2, a publication time from 1947 to 2021 was observed with an annual growth rate: 2.34%. Additionally, the types of documents, the authors, and the collaboration between authors for the period of time were studied.

The historical development of scientific production related to intellectual capital, as shown in Figure 3, is evident given that the first publication on the topic studied was in 1947 with fluctuations and few publications until 2005. Still, in 2015, the increase in publications on this topic increased considerably, with an average of 25 articles per year. On the other hand, 389 documents were published by 866 authors from different countries globally, with an average citation per year of 1.39 according to the statistical analysis established in Table 2. Note that this number of documents included mainly articles, books, and book chapters.

Throughout history, there have been events that have negatively and positively affected the development of publications related to the topic of IC, such as the case of the Second World War (1939–1945), where there was a reduction in scientific publications in almost all countries in all areas of knowledge; while in the years 1972 to 1974 publications on issues related to organizations and human capital increased. The appearance of the.COM and the strengthening of technology and research companies between 1997 and 2001, which promoted the development of science and technology in this important area of organizational administration. It can be observed that from this period scientific publications related to IC increased considerably.


 **Figure 3.** Accumulated articles across time for intellectual capital.

On the other hand, the conceptualization of knowledge managemen<sup>t</sup> is of recent creation, taking into account that from the 1950s onwards, studies and definitions of the most relevant theories on the subject began, where it can be specified that its origin begins to take shape from the managemen<sup>t</sup> by competencies and the development of information and communication technologies (ICTs), thus generating a competitive advantage in the knowledge society, especially in economies where importance was given to learning and knowledge [65]. Scientific production and the dissemination of knowledge occurred mainly at the end of the 1980s, with the development of scientific research and the advancement of science, where new means were created for the dissemination of knowledge coupled with the significant advance of the ICTs and the internet, as drivers of growth in productivity and the economy, establishing networks for the global connection of knowledge.

Taking the above into account, the first companies to adopt knowledge managemen<sup>t</sup> practices were those of audit services such as Andersen Consulting or Ernst and Young and manufacturers such as General Electric or Hewlett-Packard [66]. In this way, Millares and Puerta [67] sugges<sup>t</sup> that in these companies, the knowledge of human talent is the basis for generating competitive advantages in this type of organization, as well as later an accelerated increase in knowledge managemen<sup>t</sup> practices in various sectors of the economy and mainly in large companies and higher education institutions.

#### *3.2. Keyword Evolution*

Keywords were analyzed directly from the published documents, taking into account the frequency of appearance of the most used keywords, as shown in Figure 4. In this way, it was evident that the three most used keywords were "societies and institutions", "knowledge management", and "intellectual capital" with an average frequency of 80 times, which is of interest considering the analysis of this topic for higher education institutions, which involves different topics of knowledge.

Taking into account the antecedents on knowledge and intellectual capital, it was considered that this subject of study is of recent creation, because from the 1950s onwards studies and definitions on knowledge began to be carried out by Drucker [68], with the term "knowledge workers" in organizations in 1959 and "Personal Knowledge" in 1967. In the same way, there are the beginnings of intellectual capital in 1963 where the term "accounting of human assets" appears [1], and, in 1967, the term "intellectual capital" was used for the first time [2]. In the 1980s and 1990s, significant contributions were obtained on knowledge managemen<sup>t</sup> where conferences, book publications, and knowledge business practices began to be held, as was the case of the first three conferences on the subject held in 1987, 1992, and 1993 and the publication of the book "The Knowledge-Creating Company" in 1995 by Nonaka and Takeuchi [4]. Similarly, companies such as Dow Chemical and Skandia, as well as consulting firms such as McKinsey, Ernst & Young, and IBM Consulting, appointed "knowledge managers" and "directors of intellectual capital".

Dumay [69] made a critical reflection on the future of intellectual capital, concluding that different authors should focus on revealing what was "previously secret or unknown" in organizations, which implies abandoning reporting so that stakeholders understand how an organization considers ethical, social, and environmental impacts by an ecosystem approach to IC. Inkinen [45] developed a literature review to measure the influence of IQ on the performance of the company, obtaining as a result that IQ influences significantly through interactions, combinations, and mediations in the performance of the organization. In the same way, IQ is significantly related to the innovation performance of companies.

Figure 5 shows the co-occurrence between the keywords. Five clusters of keywords can be observed, the most important being in the central point, intellectual capital, followed by knowledge managemen<sup>t</sup> (red color), societies and institutions (color green), and, to a lesser extent, a cluster related to competition and education and universities, (colors: purple, blue, and yellow, respectively). This figure determines the importance of the thematic areas or topics related to the analyzed documents and thus relates to the concept of intellectual capital.

**Figure 5.** Co-occurrence between the keywords.

Figure 6 shows two specific dendrograms for the keywords, which relate areas on human resources and organizations (blue lines) and everything related to intellectual capital and knowledge managemen<sup>t</sup> (red lines).

There are various approaches or classifications on intellectual capital, of which the most appropriate for the education sector is the strategic–corporate approach, which is immersed in the mission of higher education institutions. In this sense, it must be strategically linked to the fulfillment of the mission of research and generation of knowledge, based on the capacity of these institutions to produce and transfer knowledge to society. Therefore, it is viewed as a dynamic system of intangible assets essential for scientific production by Leitner and Warden [24] Sanchez and Susana [70]. In this way, the managemen<sup>t</sup> of intellectual capital as a tool provides added value to institutions, as well as to their impact

and positioning strategy, thus contributing to the scientific performance of human capital and the codification of knowledge through scientific publications in universities [59]. For Bueno [71], the arrival of the information society and its evolution towards that of knowledge have placed intangible resources in one of the primary sources of creating a sustainable competitive advantage for organizations and generating value and future performance. In this context, intellectual capital is a strategic perspective of the "account and reason" of organization intangibles.

**Figure 6.** Topic dendrogram for keywords.

Figure 7 shows the clusters of keywords by appearance; it should be noted that the conceptual structure tries to explain the main themes and trends in the scientific world in a specific area, that is, what science talks about. Figures 6 and 7 show that the keywords used by the authors defined two conceptual clusters (or themes). These clusters show a minimum cluster frequency of five per thousand documents and a minimum number of 250 repetitions per keyword.

#### *3.3. Source's Significance*

Figure 8 shows the most important journals in which articles on the subject of intellectual capital have been published. Note that the "Journal of Intellectual Capital" is the most relevant journal because it has an h-index of 18 with an impact factor of 1.18 being its quartile Q1. Subsequently, it is followed by the journal of the "Proceedings of the European Conference on Knowledge Management" with a lower impact factor but with the highest number of publications on this topic studied in the Bibliometric analysis.

Taking into account the results of the significance of the 20 most important journals, Figure 9 shows the 20 most-cited journals; in the same way, the "Journal of Intellectual Capital" prevailed among the others, thus corroborating the importance of this journal in this area of knowledge, which helps to strengthen institutions and companies.

**Figure 7.** Keyword clusters by appearance.

**Figure 8.** Importance of the 20 main sources.

**Source**

**Figure 9.** Total citations by source.

#### *3.4. Top Authors*

In the collection of documents, there are 369 authors, where 30.1% (111) had only one publication. However, Figure 10 shows the top 20 authors in the intellectual capital topic. Bontis, Edwinsson, and Dumay presented the highest citations on the subject, with around 240, 136, and 128, respectively, for the period of time analyzed.

**Figure 10.** Relevance of the author by the number of citations.

The most important journal on the topic of intellectual capital is the "Journal Intellectual Capital", which presents a large number of citations for the 20 most important articles, which are related to study topics such as performance, transformation, knowledge, advantages, competitions, and reviews on intellectual capital in countries, as shown in Table 3. It is evident that both the H\_index and the citations of the publications are influenced by the HC and EC of the organizations, as well as the impact that the subject can have in this area of knowledge for IC managemen<sup>t</sup> in institutions and organizations worldwide.


**Table 3.** Most relevant authors.

Figure 11 shows the evolution over time of the authors related to the subject of intellectual capital. It can be observed that the growth between the number of authors per document and citations was relevant since 2002. The authors Bontis N of the DeGroote School of Business, Hamilton, Canada and Matos F of the Instituto Universitario de Lisboa (ISCTE-IUL), Lisboa, Portugal, Instituto Universitario de Lisboa were the most active authors related to the 20 most important articles in terms of the publications analyzed as a result of the bibliometric analysis.

**Figure 11.** Time evolution of the 20 top authors in intellectual capital.

The studies carried out on the influence of intellectual capital on scientific production and the dissemination of knowledge at a global level in higher education institutions are very scarce. Similarly, there are many empirical studies in the Scopus database in the period of 2015 to 2020 whose level of importance is reflected in the number of citations of these publications and the H\_index of the authors. The five most important published documents on intellectual capital at the international level are described below.

Dumay [70] is the most relevant author about the subject of intellectual capital in institutions with 162 citations, and they made a critical reflection on the future of intellectual capital, concluding that the different authors should concentrate on revealing what "was previously secret or unknown" in organizations, which implies abandoning reporting so that stakeholders understand how an organization considers ethical, social, and environmental impacts according to an ecosystem approach to IC, taking into account that IC currently expands its limits to the broader ecosystem to "go beyond IC reporting".

Kianto et al. [21] demonstrated that IC in 180 Spanish companies positively influences human-resource-management practices based on knowledge and performance in innovation. In this way, it favors structural and relational capital partially through human capital, and, in turn, human capital favors innovation performance by improving structural and relational capital. Sirinuch [72] conducted a study on 213 technology companies listed on five stock exchanges in the countries of the Association of Southeast Asian Nations (ASEAN), showing that IC is positively related to market value, which indicates that companies with a higher IC have a higher market value. Khalique et al. [23] showed in 247 Pakistani SMEs in Gujranwala and Gujarat that the components of intellectual capital referring to client capital, structural capital, social capital, technological capital, and spiritual capital are positively related to the organizational performance of SMEs operating in the electrical and electronic products manufacturing sector in Pakistan.

Regarding the collaboration networks between the authors on intellectual capital, Figure 12 shows two collaboration networks: the green network led by Matos F from the Instituto Universitario de Lisboa, which works with Secundo G from Universita del Salento, Lecce, Italy, which is related to the red collaboration network. These collaboration networks increase the visibility of scientific publications in different areas of knowledge and, in this way, reduce knowledge gaps from different points of view and perspectives.

**Figure 12.** Collaboration networks between authors.

Figure 13 shows the Three-Fields plot for the reference–authors–keywords; it was evident how all the authors included the subject of intellectual capital in their articles as well as, secondly, knowledge managemen<sup>t</sup> and, thirdly, universities. These were included in publications that were developed since 1997, which have taken grea<sup>t</sup> interest and have been relevant to this important area of administration.

**Figure 13.** Three-Fields plot for the reference–authors–keywords.

#### *3.5. Top Institutions and Countries*

In order to identify the most important institutions worldwide on the topic of intellectual capital, Figure 14 shows the top 20 institutions, taking into account the affiliations of the authors. Islamic Azad, McMaster, and California Universities are the most relevant institutions with 23 articles in total. For the specific case of Colombia, the Atlantic and Medellín universities reported very few articles on this topic, at around six. Thus, it was observed that the analyzed topic presents few publications by institutions; however, according to Figure 3, the growth rate has been increasing, which is significant for this area of study.

**Figure 14.** Top affiliation of institutions by authors.

From the Bibliometric analysis and the results shown in Figure 15, it was obtained as a result that the United States, Canada, and the United Kingdom are the most cited countries in the topic studied, with 1468, 731, and 727, respectively; they were followed by other countries such as Spain and Australia with 321 and 311 citations, respectively, which were approximately 50% less than the first three countries. It was determined that, regardless of the number of publications in a country, these articles are cited by the works that are being developed, thus being the importance and quality of the documents studied.

**Figure 15.** Frequency of appearance of countries and collaboration.

The productivity of knowledge workers is the most significant contribution to be made in the 21st century. In this way, Druker [68] argues that in the 20th century, production equipment was considered the most valuable asset of a company. In contrast, for the 21st century, the most valuable asset of an institution is human resources and productivity. That is, knowledge makes organizations more productive (IC and HC). According to the above, in developed countries, the main challenge is to make knowledge workers more productive and not manual workers. This is becoming the central challenge for organizations, bearing in mind that such knowledge workers are rapidly becoming the largest group in the workforce of all developed countries, where productivity is most often dependent on future professionals and, indeed, the future survival of developed economies.

Figure 16 shows the collaboration networks between countries, where six collaboration networks were observed (yellow, red, purple, blue, light blue, and green), the most important being the USA, the UK, Spain, and Italy. This shows that the subject of intellectual capital has been studied by relevant countries, which reveals the importance of analyzing the aspects that this subject involves in institutions and universities worldwide. In the case of Colombia, collaboration networks are presented mainly with Mexico, Canada, and Spain, where the most important publications related to IC have been generated.

On the other hand, in Colombia, the study on the influence of intellectual capital on scientific production and disseminating knowledge has occurred to a lesser extent. Research has been carried out independently from the 1980s and 1990s, thanks to the rise of information and communication technologies. Still, articulation is not visualized in said study variables. Simultaneously, in higher education institutions, there is an absence of documents published on said topics studied, reflecting a lack of interest on the part of the national scientific community.

**Figure 16.** Networks of collaborations by countries.

#### **4. Conclusions and Trends**

The scientific publications associated with intellectual capital were reviewed with the aid of advanced data analysis and graphics across time using Bibliometric analysis. It was found that research in this field topic has entered a stage of accelerated increase from 2015, and therefore, the values are still growing. The major contribution to this research topic predominates in the USA, the UK, Spain, and many other countries that appear on the statistical results of the sources of data analyzed.

The analysis of the keywords showed that various aspects of intellectual capital have been developed and analyzed over the last 74 years, and the latest top studies are associated with intellectual capital and knowledge management, followed by societies and institutions, and, to a lesser extent, clusters related to competition, education, and universities. Moreover, time evolution in keywords research showed that intellectual capital is still predominant.

The quantity and quality of intellectual capital are related to scientific publications, which are directly associated with the quantification and qualification of the personnel working in public institutions of higher education. In this way, it was established that the greater the number of authors, the greater the publications that will be substantial to their number.

The study of intellectual capital has occurred mainly in the business sector and, to a lesser extent, in the education sector, where there were only eight scientific publications in the 2015–2020 period in the Scopus database, thus reflecting a lack of interest in the study of this topic by the scientific community in higher education institutions. This takes into account that it is considered as one of the important tools for the development and strengthening of public or private organizations.

The IC in higher education institutions (HEIs) contributes significantly to their competitiveness and corporate image, in the sense that these institutions are measured by their academic products in terms of the mobility of students and graduates; in research through the categorization of researchers and research groups, and the production and dissemination of knowledge at the national and international level; and also by extension programs and products. Therefore, the HEIs, having high levels of IC contained in trained and innovative human resources; a robust structural capital in organizational, technological, and research processes; and relational capital with academic and research networks with its stakeholders, generating an impact on the academic community and society in general, provides these institutions with better organizational learning, more efficient performance, and higher quality in their academic and research processes. Therefore, both these educational organizations are more competitive locally, nationally, and internationally.

The theoretical implications of this study offer a general review of the literature on the development of intellectual capital research around the world at a general level and its consequences in HEIs through a bibliometric analysis in the *Scopus* database. This allows identification of the components of the main concept, and models of capital also determine the level of growth of the research carried out historically, the concurrence of the keywords through clusters, prominent journals, institutions, countries, and scientific collaboration on this topic of study. Therefore, the main practical implications of this study fall directly on researchers, teachers, and students of higher education institutions, regarding the theoretical foundation and historical development of intellectual capital research. In addition, the methodology used in this study may be used to obtain similar results in other contexts and organizations.

The limitations of this research are oriented to the bibliometric analysis, which was carried out using only the *Scopus* database, which has grea<sup>t</sup> academic and scientific prestige with wide coverage in the publication of scientific articles. There are other databases with other publications on the subject studied that can serve as a basis for other research of this type.

Future research directions in this topic of study could focus on the relationship of intellectual capital with other study variables such as scientific production, knowledge management, and innovation since most current studies only focus on the measurement of intellectual capital from its three main components such as human capital, structural capital, and relational capital in commercial and academic organizations. In addition, it is proposed that future research study intellectual capital as a strategic resource in HEIs as creative organizations and disseminators of knowledge, where the identification, measurement, and development of IC generate value and sustainable competitive advantage from the strategic direction to make better decisions for the future.

**Author Contributions:** W.Q.-Q.: investigation, collected data, funding acquisition, formal analysis, original draft, writing—review & editing. A.B.B.-A.: analysis tools, formal analysis, other contributions. M.A.G.-C.: contributed data, formal analysis, other contributions. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** This work was supported by the research entitled "Influencia del capital intelectual en la producción y difusión del conocimiento científico de los investigadores de Instituciones de Educación Superior Públicas del Nororiente Colombiano". Doctorado en administración Universidad Simón bolívar, Barranquilla, Colombia.

**Conflicts of Interest:** The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this article.

# **References**


# *Article* **Research Productivity in Emerging Economies: Empirical Evidence from Kazakhstan**

**Timur Narbaev \* and Diana Amirbekova**

> Business School, Kazakh British Technical University, Almaty 050000, Kazakhstan; d.amirbekova@kbtu.kz **\***

 Correspondence: t.narbaev@kbtu.kz

**Abstract:** The growth of the Higher Education and Science (HES) sector is positively associated with its research productivity and has a high potential in emerging countries. To explore such research productivity, this study offers a comprehensive analysis of the scientific literature from Kazakhstan. Our methods included descriptive analysis, network analysis, and author-based productivity analysis (by Lotka's law) of 23,371 articles from Scopus, published during 1991–2020, and across 25 subject areas. The results of the descriptive analysis showed a substantial increase in the number of and citations to the literature since 2011 in almost all subject areas. However, the network analysis found that research in natural sciences was more developed in topical relationships and international collaborations than research in arts and humanities, social, and medical sciences. The Lotka's law application revealed that the overall scientific literature in Kazakhstan did not reach its necessary stage of maturity. Additionally, some subject areas demonstrated greater contribution to the overall knowledge base, while others were less productive or lagging in their development. Our findings, useful for researchers and policymakers in emerging countries, can be exemplary in understanding the results of policy reforms aimed to improve the HES sector in emerging countries.

**Keywords:** citation analysis; emerging country; Kazakhstan; Lotka's law; network analysis; publication trend; research productivity; scientometrics

# **1. Introduction**

The growth of educational and scientific performance is positively associated with the research productivity of a country and contributes to its economic development [1]. One of the crucial reforms that post-Soviet countries undertook in the Higher Education and Science (HES) sector was the financing of the local science and its integration into the international scientific community [2,3]. Among these countries, Kazakhstan is one of the few that has built a relatively robust research infrastructure, including support through grants, access of researchers to research mobility programs, earlier application of the Bologna processes, and other measures to increase its research performance indicators [3]. In turn, such measures have resulted in the improved scientific engagemen<sup>t</sup> of local researchers in the international arena [4]. This has all led to increased research published in international peer-reviewed outlets, improved productivity of the local researchers, and raised the scientometric indexes of the country.

A few studies have analyzed the development and trends in research productivity in the HES sector using scientometric approaches in Kazakhstan. To reveal issues related to research productivity and science in selected post-Soviet countries, Suleymenov et al. [5], using publication records in Scopus, performed several types of analysis. They analyzed trends in seven selected research areas with most publications and identified development trends for the Commonwealth of Independent States (CIS) member countries. The CIS members are nine countries: Armenia, Azerbaijan, Belarus, Kazakhstan, Kyrgyzstan, Moldova, Russia, Tajikistan, and Uzbekistan. They revealed the citation rate per paper, average citation rate, publication rate per 10,000 people, and the potential growth for

**Citation:** Narbaev, T.; Amirbekova, D. Research Productivity in Emerging Economies: Empirical Evidence from Kazakhstan. *Publications* **2021**, *9*, 51. https://doi.org/10.3390/ publications9040051

 Academic Editor: Oleg V. Mikhailov

Received: 1 September 2021 Accepted: 3 November 2021 Published: 5 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Kazakhstan and other CIS countries compared to the global level. Kuzhabekova and Lee [1] assessed 361 publications to identify available contributions of international faculty employed at the Kazakhstani universities and how they contribute to local research capacity building. Using a combination of bibliometric, social network analysis, and content analysis methods, they found the growing role of global research networks, knowledge development, and research dissemination in the HES sector in the country. Focusing on managemen<sup>t</sup> literature, Narbaev [6] assessed the productivity of the project managemen<sup>t</sup> discipline in the country through the application of a co-word analysis on 826 articles sourced from Scopus. The network analysis was applied to visualize the scientometric trends in this field. He found that project managemen<sup>t</sup> research in the country was in its infancy stage and was correlated to the project orientation of the society.

Applications of scientometric methods to analyze growing literature and research productivity have been gaining researchers' interest on the global scale. For example, to examine publication patterns of Brazil, Russia, India, and China (BRIC countries), Guevara and Mendoza [7] used a network analysis technique and built maps of authors collaboration and subject area interaction. They revealed that China and Russia are the top publishers of scientific literature and more specialized in physics and astronomy. Brazil was the most collaborative country with a developing economy, while India was grouped with the developed countries and had a more diversified network of research areas. Hinojo-Lucena et al. [8] used several bibliometric tools to evaluate the impact of artificial intelligence (A.I.) in higher education globally. They applied widely used scientometric methods, such as the Price law and Lotka's law, on the dataset of 132 articles and revealed that A.I. applications and the associated research were growing in the field. Applying productivity analysis, collaboration analysis, and citation analysis methods, Macchi Silva et al. [9] examined more than 700 papers on the competence-based management literature sourced from the Web of Science and Scopus databases. Their findings showed that collaborations between researchers did not necessarily lead to strong coauthorships and that the most cited papers were in diverse areas of the literature, implying the interdisciplinary landscape of the competence-based managemen<sup>t</sup> literature.

The major findings of the above studies stress the importance of using scientometric approaches to investigate research trends and productivity in a country's HES sector. Country-wide scientometric studies contribute to understanding a growing body of knowledge and decision-making for its effective research policy [10–12]. On the one hand, the studies show that various methods to analyze research productivity exist and that some countries lack applications of advanced scientometric tools. On the other hand, a wide variety of scientometric methods are available that could be used to study research productivity and development trends.

The current state of the scientometric literature shows a lack of studies dedicated to analyzing scholarly literature and research productivity in Kazakhstan, including applications of methods and techniques available for such scientometric studies. To fill this research gap, in this study, we aim to reveal scientific trends and analyze the research productivity of Kazakhstan and provide implications for science managemen<sup>t</sup> in emerging countries. Using data from 23,371 articles sourced from Scopus and published during 1991–2020, we conduct descriptive analysis, network analysis, and author-based research productivity analysis of the country's scientific potential. These analyses are performed both at the country level and across numerous subject areas of the collected articles.

The remainder of our paper is structured as follows. Next, we introduce the dataset collected and three types of analytical methods applied. Then, we report and discuss our detailed results. Lastly, we summarize our study with a discussion of significant findings, research limitations, and contributions to the body of knowledge.

#### **2. Materials and Methods**

Table 1 presents an outline of our research methodology. We followed a general approach of a review study, applicable also for scientometric research, established by the PRISMA declaration [13]. The scientometric methods used for descriptive analysis, network analysis, and author-based research productivity analysis in this study are similar to those used in previous research [8,14,15].


**Table 1.** Outline of the research methodology.

#### *2.1. Materials Collection and Their Screening*

In Step 1 of our methodology (Table 1), we limited our search to articles in English and published in peer-reviewed journals indexed in Scopus. We looked for articles where at least one of the co-author's affiliation was Kazakhstan. Our search covered the period from 1991 (which marks the independence of Kazakhstan) to 2020.

In Scopus, we used its advanced search function using a field code "AFFILCOUN-TRY(KAZAKHSTAN)". Further, we applied the following filters: year of publication (1991– 2020), document type (article and review), source type (journal), and language (English). This search resulted in an initial set of 24,284 articles by 156,405 authors with titles, abstracts, keywords, and bibliographic details. Using the subject area category function of Scopus, we grouped all the articles into 27 subject areas. Then, we excluded 2 subject areas: Dentistry as all authors had published only one article, and Multidisciplinary as it was unrelated to a specific research field. This screening resulted in 23,371 articles by 150,708 authors and representing 25 subject areas. Lastly, we exported our dataset from

Scopus in a tab-delimited CSV format and utilized it to conduct the analyses in Steps 2 through 4 (Table 1).

## *2.2. Methods*

#### 2.2.1. Descriptive Analysis

In Step 2 of our methodology, we conducted a descriptive analysis of the articles to identify the overall research trends and productivity in Kazakhstan. This involved the analysis of the distribution of the articles by year of publication, the number of articles, etc., which are presented in Table 1 [16,17]. In the literature, similar studies used such a descriptive analysis on Brazilian scientific output [18], understanding the impact of sustainability performance indicators [19], and introduced a framework to assess the productivity of a research area [20]. In addition to this analysis, based on the bibliographic details of the articles, we performed an analysis of publishers, collaborating countries, and funding sponsors of science.

#### 2.2.2. Network Analysis

The scientometrics research is linked to data visualization [21]. One of the methods to visualize trends in a given research area is a network analysis [22]. This analysis refers to using a network of closely related attributes, such as keywords co-occurrence analysis. A keyword is a critical attribute of a publication that may represent a research topic, and it provides essential information about research trends in a field [23,24]. In our study, a keyword co-occurrence network was built to represent topics, identify the relationships between these topics, and define clusters of closely related topics within a subject area (Step 3 in Table 1). This type of analysis demonstrates the interaction within and between clusters based on keywords in each subject area. A cluster represents a collection of closely related elements (topics) that are homogeneous [25]. In this study, each constructed network represented a subject area (defined by Scopus classification). Each network had several clusters to represent closely related topics. In order to build such networks, we used the VOSViewer package. To perform this technical task, we downloaded the articles from Scopus for each subject area separately and constructed the networks with clusters using the co-word analysis function of VOSViewer. This function is performed using keywords extracted from the Scopus database and applies a counting method in the VOSViewer. The counting method is "full counting" where each keyword has the same weight, without any influence on the number of keywords for each article. Given that some subject areas had a scarcity of articles (with only a few keywords) affiliated with Kazakhstan, we kept the minimum number of co-occurrences for a keyword as 1.

#### 2.2.3. Author-Based Research Productivity Analysis

An analysis of a country's research productivity is as critical as an assessment of publication and topical trends for a given research field. It is reflected by the number of publications scholars contribute to an overall knowledge base within a specific time frame [26]. Several methods are available to evaluate author-based research productivity, including Lotka's law [27,28].

In Step 4 (Table 1), we used this law to assess the scholarly productivity of the researchers from Kazakhstan and to evaluate the relative productivity (development) of 25 subject areas. Lotka's law uses the number of articles and the number of authors in a given subject area and presents the frequency of publication by authors for this area [29]. It is defined as per Equation (1).

$$\mathbf{f}(\mathbf{x}) = \mathbf{k}/\mathbf{x}^{\mathbf{n}},\tag{1}$$

where f(x) calculates the number of authors contributing x articles each, x is the number of articles by an author, k is a given constant which represents the number of authors who published only one article, and n is the parameter which represents the distribution of the research productivity (articles) by all authors.

In this equation, theoretically, the *n*-parameter is equal to about 2. If so, according to this law, about 60% of all authors in a given subject area make a single contribution (represented by the k-constant as 0.60), about 25% (1/2ˆ2), 2 contributions, about 11% (1/3ˆ2), 3 contributions, etc. [30,31]. The relationship between the *n*-parameter and k-constant implies that the number of scholars publishing a given number of articles is fixed to the number of scholars publishing only one article. In the literature, Voos [32] applied Lotka's law in the information science literature and found that the *n*-parameter was 3.5. Pao [33] empirically tested this law on the number of research fields and determined that the parameter value ranged from 1.8 to 3.8. Therefore, the case with the *n*-parameter equal to about 2 is considered a generalization [30,34]. It is regarded that those subject areas with higher *n*-parameter values are less developed (less maturely represented by fewer researchers), while subject areas with lower *n*-parameter values are more established (more maturely represented by more researchers).

In this study, we applied Lotka's law to evaluate the country's research productivity and calculated the values for the *n*-parameter and k-constant for all 25 subject areas.

We should note that deciding on which subject areas to analyze has been a long process of learning and trying. The main concern was that Lotka's law has been primarily applied in engineering and I.T. fields or has rarely been used for several subject areas at once. Initially, we took only a few of the most representative subject areas by the highest number of published articles. However, selecting such subject areas does not mean that the remaining areas are not essential or productive. Moreover, we aimed to reveal the overall trend in the country, which would serve as exemplary for other emerging economies. This was not limited to a few subject areas. Therefore, after a thorough review of the reported literature on using Lotka's law in different fields and countries, we kept all 25 subject areas defined by Scopus. For this, we downloaded the articles from Scopus in a tab-delimited CSV format into VosViewer. Then, we calculated the number of authors and the number of articles they published by simple counting for each subject area. Lastly, we applied Lotka's law on Excel to analyze the author-based research productivity.

#### **3. Results and Discussion**

#### *3.1. Descriptive Analysis*

In this section, we report and discuss the results of our descriptive analysis (Step 2 in Table 1). Figure 1 shows the pattern of published articles from Kazakhstan during 1991–2020. Overall, the trend in the publications was unnoticeable in the first half of the 1990s. During 1996–2010, the research output was about 237.5 articles per year with no apparent changes in the number of publications. However, we can see a stable increase since 2011, with a rate of about 32.0 percent per year. Such an increase is the result of the implementation of a number of essential policies and laws in the HES sector. These include the State Program of Educational Development (2011–2020), the Law on Science (2011), the Law on Commercialization (2015), and the State Program for Education and Science Development (2016–2019). For example, the Law on Science was enacted to reevaluate new scientific directions, improve publication quality, and set standards for awarding academic degrees and titles [35,36]. Additionally, the State Programs (2011–2020, 2016–2019) set key targets relevant to the country's research performance and contributed to its productivity. Some critical targets are the increase in the number of the local HES institutions in the global Quacquarelli Symonds (Q.S.) World University ranking (2 institutions were in the 2015 ranking, 10 in the 2020 ranking); the percentage of academic staff who publish in non-zero impact factor journals (the target of 3.25% for 2015 was achieved in advance in 2013); and the percent of academic staff who engage in research (8% was in 2011, 27% in 2014) [37]. Other requirements established in 2012 include publishing at least one paper in a journal with the two-year journal impact factor being above zero or indexed in Scopus (to award a Ph.D. degree) and publishing at least two and three articles in journals with a journal impact factor of above zero (to award associate professor and professor titles, respectively) [35]. Additionally, the gran<sup>t</sup> funding scheme by the Ministry of Education

and Science started considering the quantity and quality of the applicant's publications in international journals indexed in the Web of Science or Scopus databases. Based on such policies that promote an increase in the quantity and quality of publications from the local researchers, we can expect that the growth of articles in international journals will continue in the near future.

**Figure 1.** Research publication trend of Kazakhstan during 1991–2020.

Table 2 represents a summary of the descriptive analysis for all subject areas. About 11% of all articles were published in Physics and Astronomy, while about 9% were in Social Sciences. A few areas have more than 5% of the articles published—Engineering, Chemistry, Biochemistry, Genetics and Molecular biology, Material Sciences, and Mathematics. This finding demonstrates the high productivity rate of these seven subject areas compared to the others and shows the solid contribution of their authors. Additionally, the overall contribution of authors in natural and engineering sciences is much higher compared to the ones in the subject area of Social Sciences, Economics, Econometrics and Finance, and Business, Management, and Accounting. The most significant number of articles was published in Physics and Astronomy (4522 articles), while the lowest was in Nursing (96 articles).

In terms of the author analysis, the top subject area is Medicine (19,612 authors). Overall, the average number of authors per article across all subject areas is 4.21. The maximum number of authors per article is in Medicine (10.19), while the lowest is in Mathematics (1.36). This demonstrates the collaboration intensity of co-authors in a given subject area.

The citation analysis helps to reveal the most influential subject areas in Kazakhstan. Additionally, it demonstrates the recognition of authors affiliated with Kazakhstan in the research community [38]. The analysis reveals few subject areas where the local scholars are influential in their global research community. The number of citations per article (the citation rate) is a long-term indicator of the quality of research in a published article [4]. Given the importance of this indicator, it could be changed over time and some of the subject areas may see a positive trend in the future [39]. Additionally, to understand the pattern of the distribution of citations by articles in the subject areas, the citation distribution for the top five subject areas with the largest number of articles was constructed, given in Figure A1 in Appendix A1. To evaluate the citation impact and distribution, Bornmann and Williams [40] used the percentage of papers that received the largest number of citations. In addition to this, the mean and median numbers of citations per article are given where the former is used to show the average of the citations in the subject area and the latter demonstrates the middle point in the distribution. They can be used to assess the relative

dispersion/skewness of the distribution. Overall, we note that the top 10% of the articles in these subject areas received between 57.1% and 66.7% of all citations.


**Table 2.** Distribution of the number of articles, authors, and citations by subject area.

1The total number of articles in this table (41,524) is different from the total number of articles in the study (23,371). This is because a single article in Scopus may be indexed in more than one subject area, e.g., in Chemistry and Materials science, simultaneously.

> The international rankings of the HES sector, such as by Q.S., look at the citation rate as an indicator of an institution's performance. Therefore, the growth in the number of articles and citations to the studies of the researchers affiliated with Kazakhstan has a considerable contribution to such rankings. Additionally, collaborations and co-authorship with more countries allow researchers from Kazakhstan to become integrated into the global research community where funding comes from various international sources. The current trends in local research productivity reveal that the areas related to agriculture, engineering, and medicine may experience tremendous growth in the coming years. Moreover, publications in reputed international journals indexed in Scopus and Web of Science and the increase in citation rates are some of the most essential criteria in the evaluation and funding of research proposals, awarding of Ph.D. degrees, and the promotion of faculty and researchers, not only in Kazakhstan [41], but also in other emerging countries [7,42,43]. These are the measures and policies regulated by the Ministry of Education and Science of the Republic of Kazakhstan.

> Table 3 provides the details of the top five publishers, collaborating countries, and funding sponsors of the top five subject areas of science in Kazakhstan. The results of this analysis for the remaining subject areas are given in Table S1 in Supplementary Materials. Among the major publishers of research from Kazakhstan are Elsevier, Springer, and al-Farabi Kazakh State National University. It is noted that the choice of a publisher also depends on the specificity of a subject area. The analysis of collaborating countries shows the variety of partnerships, although most papers are published in collaboration with the Russian Federation, United States of America, United Kingdom, and Germany.

Based on the analysis of the number of the collaborating countries for each subject area, we observe that international collaboration is higher in the subject areas representing natural sciences than in the subject areas representing arts and humanities, social, and medical sciences. In part, this can be due to the fact that the researchers in natural sciences participate in more projects funded by international donor organizations or foreign partner universities [44]. The most recognized sources of science funding in Kazakhstan are the Ministry of Education and Science of the Republic of Kazakhstan, the Russian Foundation for Basic Research, and Nazarbayev University. Overall, along with the increase in the funding of science, an increased interest in research globalization contributed to the growth in the number of researchers who publish in international journals.

**Table 3.** Top 5 publishers, collaborating countries, and funding sponsors of the top 5 subject areas.


#### *3.2. Network Analysis*

This section presents the results and findings of our network analysis (Step 3 in Table 1). As noted in Section 2.2.2, the network analysis included construction and visualization of the keyword co-occurrence networks for all subject areas. Each network comprised several clusters with closely related keywords that represented some topics. Figure 2 presents a sample network for the subject area Physics and Astronomy. This network has eight clusters that represent independent streams of research in this subject area. Close links in the network demonstrate interconnections that exist between clusters. Some of the keywords belong to several clusters. Based on the network visualization, the most representative keywords are ions, irradiation, temperature, scanning electron microscopy, silicon, hydrogen, carbon, mathematical models, crystal structure, and electrons. Similar networks for all 25 subject areas are provided in Figures S1–S25 in Supplementary Materials.

Table 4 summarizes the key attributes of the keyword co-occurrence networks in our study. We note that Physics and Astronomy, Biochemistry, Genetics and Molecular biology, and Medicine are characterized by a large number of keywords in comparison to other subject areas and those with stronger links.



**Table 4.** *Cont.*


**Table 4.** *Cont.*

Medicine has the largest number of collaboration links (153,993), which implies strong collaboration in this subject area. The areas close in research scope to Medicine, such as Immunology and Microbiology and Pharmacology, Toxicology, and Pharmaceutics, have comparatively fewer links.

Business, Management, and Accounting is characterized by uneven sporadic interconnections between words, which means that this area is comparatively less productive and collaborative in Kazakhstan. A related subject area Economics, Econometrics, and Finance demonstrates a low level of interconnectivity among its keywords, and therefore a low level of interconnections and links. The subject areas Arts and Humanities, Energy, and Decision Sciences demonstrate a similar number of keywords, although Energy has more links and fewer clusters compared to the other mentioned subject areas. There are more links in Energy, which means keywords are more interconnected; therefore, the number of clusters is lower.

Materials science demonstrates a high level of interconnections among its keywords and their relation to each other. The size of the clusters in Chemistry is quite large with close connections inside the clusters and among its keywords. We report the same pattern also for Chemical Engineering. Some major topics in these subject areas are similar or occur concurrently. Biochemistry, Genetics, and Molecular Biology shows more developed collaboration and interconnection between its keywords, similar to Environmental Science and Agriculture and Biological Sciences.

Overall, our network analysis in this step reveals the overall development and current trends in these subject areas. Overall, such subject areas as Physics and Astronomy, Biochemistry, Genetics, and Molecular biology, Medicine, and Chemistry are more developed, while such areas as Social Sciences, Business, Management, and Accounting, Arts and Humanities, Neuroscience, and Psychology are less developed. Overall, the

subject areas representing natural sciences are well developed, with dense clusters and topical relationships. This can be noted from the number of keywords and links in Table 4 where each keyword may represent a topic or line of research and each link represents a relationship between such two keywords. Overall, such a finding can be corroborated by the contributions of the Soviet school of science in natural sciences, which had a profound impact on the global scientific community which continue in present Kazakhstan.

#### *3.3. Author-Based Research Productivity Analysis*

This section provides the results and findings of our author-based research productivity analysis (Step 4 in Table 1). Lotka's law (Equation (1)) was used to assess the research productivity of the scholars from Kazakhstan. As introduced in Section 2.2.3, it was assessed using the frequency distribution of the number of articles published by unique author names. We demonstrate the subject area Art and Humanities as an example. For this subject area, overall, 1539 articles were published by 4520 unique authors (Table 5). This altogether gives 5716 co-author occurrences in 1539 articles since one author may publish more than one article or multiple authors (co-authorship) may represent one article. Such distribution represents the overall authorship pattern for this subject area. The range for the frequency distribution is such that 3711 authors published one article each, 584 authors published two articles each, and so on. In the limit of this range, there is only one unique author who published 15 articles. The aim from building this distribution table is to find the value for the *n*-parameter, which ideally should fit the predicted number of authors to the actual number of authors. For this, the difference between the total number of authors (actual) and the total number of authors (predicted) must be equal or close to 0. For the subject area Art and Humanities, the *n*-parameter is equal to 2.89. The other results in Table 5 when the *n*-parameter is equal to 2 are given for demonstration purposes only since this is the theoretical (benchmark) value of the *n*-parameter reported in the literature.

**Table 5.** Results of the author-based research productivity analysis by the Lotka's law. A sample calculation for the subject area Art and Humanities.


Table 6 presents the summary results of the application of Lotka's law for all 25 subject areas. We can observe that the values for the *n*-parameter range from 2.05 in Medicine to 3.85 in Neuroscience. Overall, the subject areas are categorized into four groups, with an increment of 0.50 for the parameter value. Along with the results of the *n*-parameter, we report the results for the k-constant, which represents the associated percentage of the authors who published only one article for each subject area. We note the relative correlation between these two measures: the higher the *n*-parameter value, the higher the

k-constant value, which implies a given subject area is less mature, represented by the smaller number of researchers (check in Table 2).


**Table 6.** Summary of the author-based research productivity analysis by Lotka's law.

> The subject area Medicine with the low *n*-parameter = 2.05 has the lowest percentage of authors (63.73%) who published one article, which suggests it is the most established subject area with the largest number of researchers (19,612 authors in Table 2) in Kazakhstan. The average percentage for the subject areas in Group 1 is 71.03%, which is higher than the average percentage for overall Kazakhstan, 82.36%.

> There are more subject areas in Group 2 whose *n*-parameter range is 2.51–3.00. Compared to the previous group, related more to natural and pure sciences, this group is represented by the subject areas related to social sciences, arts, humanities, and computer science. This group is less mature than the previous one, with the *n*-parameter = 2.84 and the percentage of the authors who published only one article being 82.34%. An exception is the subject area Energy, with a percentage of 90.79%.

> Groups 3 and 4 have only three subject areas each. Group 3 has the percentage of authors publishing only one article close to Group 2. The percentage for Group 4 is much

higher, suggesting 91.24% of all the researchers in the subject areas Decision sciences, Health professions, and Neuroscience published only one article.

Overall, we can observe the following findings from the application of Lotka's law. First, the results of its application to evaluate the research productivity of the scholars across the subject areas in Kazakhstan confirm that the *n*-parameter = 2.00 is a benchmark. The closer the subject area's *n*-parameter value to 2.00 (Table 6), the more developed the subject area is. Second, the finding from this law's application can be corroborated by the finding from the other two analyses in our study (reported in Sections 3.1 and 3.2). For example, we can observe that the subject areas in Group 1 are more developed as per the results of the Lotka's law application (Table 6) than the other subject areas in our study. Apparently, these subject areas are also more productive (in the number of articles), impactful (by the number of citations), and collaborative/networked (see Tables 2 and 4, Table S1 in Supplementary Materials, Figures S1–S25 in Supplementary Materials). Similar analysis of the subject areas in the other groups reaffirms this finding. Third, Lotka's law application also suggests that the number of authors who published only one article in a given subject area is not enough to ensure the overall productivity of this area. Overall, we observe that the overall science sector in Kazakhstan did not reach its necessary stage of maturity, as shown by the average value of the *n*-parameter of 3.01 for all the subject areas in Table 6. Recalling from Section 2.2.3 of the current paper, the higher *n*-parameter value implies less developed areas (less maturely represented by fewer researchers), while the lower *n*-parameter value implies the more established area (more maturely represented by more researchers). On the other hand, this also implies an opportunity for research growth in the near future that may fill the current gap in the development of the listed subject areas in Kazakhstan.

# **4. Conclusions**

The development of a country's scientific potential is based on its research productivity and quality. Recent trends in the HES sector of Kazakhstan, such as an increase in science funding, access of researchers to research mobility programs, and globalization of the local research, have resulted in the country's improved research performance. To reveal associated trends and characteristics of the research productivity of the country, in this study, we offered a comprehensive analysis of scientific literature from Kazakhstan. Our research scope included the descriptive analysis, network analysis, and author-based research productivity analysis of 23,371 articles sourced from Scopus, published during 1991–2020, and across 25 subject areas.

The results of the descriptive analysis revealed substantial growth in research publications in Scopus since 2011. The average annual growth rate of 32 percent in the past 10 years indicates a stable and robust contribution of researchers affiliated with Kazakhstan. In terms of research quality, the results of the citation analysis showed the subject areas that contribute more to the research body of knowledge. These are Physics and Astronomy, Engineering, Medicine, and Immunology and Microbiology subject areas, which are recognized by the research community on the global scale. Moreover, the collaboration patterns as co-authorship with counterparts from other countries showed that local researchers in the subject areas related to agriculture, engineering, and medicine may experience tremendous growth in the coming years. The increase in the number of such publications in English and since 2011 results from the implementation of some crucial policies and requirements of the governmen<sup>t</sup> in the HES sector. In particular, this includes the State Program of Educational Development (2011–2020), the Law on Science (2011), the Law on Commercialization (2015), and the State Program for Education and Science Development (2016–2019). For example, the Law on Science was enacted to reevaluate new scientific directions, improve publication quality, and set standards for awarding academic degrees and titles. The State Programs (2011–2020, 2016–2019) set key targets which were relevant to the country's research performance and contributed to its productivity. Other policies include publishing at least one paper in a journal with a two-year journal impact factor

above zero or indexed in Scopus (to award a Ph.D. degree) and publishing at least two and three articles in journals with a journal impact factor above zero (to award associate professor and professor titles, respectively). Publications in international journals indexed in Scopus and Web of Science and the increase in citation rates are essential criteria for funding research proposals, awarding Ph.D. degrees, and promoting faculty and researchers in HES institutions in Kazakhstan. In addition, implementing a set of requirements for research gran<sup>t</sup> holders and rigid rules in competitions (e.g., the country's best university or faculty member) continues to both push and motivate the HES institutions and local researchers to increase their research output.

The findings from the network analysis showed that the topical relationships and research collaborations in some subject areas are stronger and denser (e.g., Physics and Astronomy, Biochemistry, Genetics, and Molecular Biology, Medicine, Chemistry), while other areas (e.g., Social Sciences, Business, Management, and Accounting, Arts and Humanities, Neuroscience, and Psychology) are less established. We found that, on average, the subject areas representing natural sciences are more developed than the subject areas representing social and medical sciences. We corroborate this finding with the development of the Soviet school of science that had a crucial impact in the former areas than the latter, which continues in present Kazakhstan. The findings from this analysis can help understand the scientometric characteristics of the science sector and identify the areas for prospect growth through a more profound analysis of the factors that enable their development.

Lastly, we assessed the research productivity of local scholars and evaluated the relative research maturity of all subject areas. Applying Lotka's model with its *n*-parameter (with the value of 2.00 as a theoretical benchmark) and k-constant (representing an associated percentage of the authors who published only one article in a given subject area), we found that the overall science sector in Kazakhstan did not reach its necessary stage of productivity. On the other hand, this suggests that the country has potential in its publication output which would lead to its scientific maturity. Additionally, based on the associated values of their *n*-parameter, we grouped the 25 subject areas into four distinct groups. Some subject areas demonstrated greater productivity and contribution to the HES sector in Kazakhstan, while the others were less productive. This all implies an opportunity for research growth in the near future which may fill the current gap in the development of lagging subject areas in the country.

We acknowledge some limitations that can be considered in future research. A single article in our study may have represented more than one subject area. This is because articles in Scopus may be indexed in more than one subject area, which is especially true for allied areas, e.g., Chemistry and Materials science. Additionally, in this study, we used quantitative methods and our findings are based on statistical analysis. In future research, the scope of this study can be extended, or findings can be confirmed by using qualitative approaches (e.g., interviews) or by analyzing non-academic policy materials or reports.

The findings and implications from our study can be helpful for the international research community, policymakers in the HES sector, and serve as exemplary for other emerging countries. They can be used to understand the results of structural and policy reforms aimed to improve the country's HES sector. Additionally, understanding the current state of the research productivity and scientific maturity is crucial in building a more sustainable research environment for a country.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10.3 390/publications9040051/s1, Figures S1–S25: A keywords co-occurrence networks for all 25 subject areas, Table S1: Top 5 publishers, collaborating countries, and funding sponsors of the remaining 20 subject areas.

**Author Contributions:** Conceptualization, T.N.; methodology, T.N. and D.A.; software, D.A.; validation, T.N. and D.A.; formal analysis, T.N. and D.A.; investigation, T.N. and D.A.; resources, T.N. and D.A.; data curation, D.A.; writing—original draft preparation, D.A.; writing—review and editing, T.N.; visualization, D.A.; supervision, T.N.; project administration, T.N.; funding acquisition, T.N. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Science Committee of the Ministry of Education and Science of the Republic of Kazakhstan, gran<sup>t</sup> number AP08856031.

**Data Availability Statement:** The data presented in this study are available in the article and the Supplementary Material.

**Acknowledgments:** The authors are thankful to Damina Sabi (a researcher at Business School, Kazakh-British Technical University, Almaty, Kazakhstan) for helping with data collection and analysis in the earlier drafts of this project.

**Conflicts of Interest:** The authors declare no conflict of interest.

# **Appendix A**

(**b**) 

**Figure A1.** *Cont*.

**Figure A1.** Citation distribution for the top five subject areas with the largest number of articles. The subject areas are: (**a**) Physics and Astronomy; (**b**) Social Sciences; (**c**) Engineering; (**d**) Chemistry; (**e**) Materials Science.

(**e**)

# **References**


## *Article* **A Scientometric Assessment of Agri-Food Technology for Research Activity and Productivity**

**Jakkrit Thavorn 1, Veera Muangsin 2,\*, Chupun Gowanit 1 and Nongnuj Muangsin 3**


**Abstract:** In accordance with the UN Sustainable Development Goals (SDGs), several SDGs target global food issues, including zero hunger (food security and sustainable agriculture), responsible consumption and production (food losses), climate action (greenhouse gas emissions from food waste), and partnerships for the goals (research collaboration). As such, it is vital to identify technology and market opportunities to support advanced development by exploring scientific and technological research on such SDGs. The significance of technological innovation and evaluations of activity, productivity, and collaboration aids and guides future research streams. Motivated by the growing severity of the global food waste crisis, this paper focuses on the case study of shelf-life extension technology for food and applies a scientometric analysis of patents based on text mining. VantagePoint was used to analyze 2516 patents issued between 2000 and 2020, with the aim of understanding the conceptual structure of knowledge and the social relationships among key players. The results indicate that the technology is experiencing a period of growth, and it can be clustered into five technology sectors. Across all technology clusters, China outperformed other countries in terms of the number of patents. Almost all of China's patents applied for technology commercialization domestically, whereas other countries tended to apply for patents overseas to exploit opportunities. The findings have implications for both policymaking and strategic decision-making using a multi-layered network innovation system.

**Keywords:** scientometrics; scientific activity; technology assessment; research collaboration; patent analysis; bibliometric indicators; sustainable development goals

#### **1. Current Issues in Agri-Food Industry and Technology**

Leaders from 193 countries around the world initiated a plan known as the Sustainable Development Goals (SDGs) together with the United Nations Development Program (UNDP). In total, there are 17 SDGs, each focusing on creating a future without poverty, hunger, or insecurity. Several SDGs target global food issues, including zero hunger (food security and sustainable agriculture), responsible consumption and production (food losses), climate action (greenhouse gas emissions from food waste), and partnerships for the goals (research collaboration).

At present, various global trends are influencing food security and the degree to which food and agricultural systems are sustainable. By 2050, the global population is estimated to reach approximately 10 billion, which has been forecasted to correspond to a growth in agricultural demand by 50% over 2013 [1]. Considered in relation to other sectors, growth in the agricultural sector is two to four times more effective in raising individual incomes. Agricultural activities also play a pivotal role in economic growth, as reflected by the fact that 4% of worldwide gross domestic product (GDP) is based on agriculture, with this

**Citation:** Thavorn, J.; Muangsin, V.; Gowanit, C.; Muangsin, N. A Scientometric Assessment of Agri-Food Technology for Research Activity and Productivity. *Publications* **2021**, *9*, 57. https://doi.org/10.3390/ publications9040057

Academic Editor: Oleg V. Mikhailov

Received: 5 November 2021 Accepted: 16 December 2021 Published: 20 December 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

figure exceeding 25% in several developing countries [2]. Nevertheless, growth facilitated by agriculture has been noted for its potentially damaging impact on food safety and food security. Available data indicate that agriculture and land use changes account for approximately 25% of climate change effects observed in recent years (e.g., via greenhouse gas emissions). These activities produce levels of waste and pollution that cannot be sustained in the long term [3]. At the same time, it is anticipated that climate change will directly influence food nutrition and food quality in the future. Given that food waste and food losses occur for approximately 33% of all food produced around the world, it is crucial to recognize that resolving challenges surrounding food waste and loss is essential for enhancing not only food and nutrition security but also the situation regarding climate change and environmental stress [3].

A range of factors can be considered causes of food loss and waste. Over 80% of food loss and waste derives from post-harvest managemen<sup>t</sup> activities such as processing, packaging, distribution, and consumption, and this waste stems from limitations in terms of shelf-life length [4]. Shelf-life extension technology can delay the spoiling of food and, in this way, lead to the prevention of food loss and waste. Of note, fresh vegetables and fruits, which offer high nutritional and health benefits, have become increasingly popular globally, resulting in a recent global pattern of higher consumption and greater investment in R&D activities. In terms of post-harvest processing activities, these have a detrimental impact on shelf-life length in terms of surface browning and lower nutritional content, which has raised public health concerns in some jurisdictions [5]. Hence, the development and dynamics of R&D in this issue have to be examined as a way to enhance food safety, minimize food waste, and improve consumer protection.

#### **2. Scientific and Technology Opportunities based on Scientometric Analysis**

Technology opportunity analysis (TOA) was originally proposed by Cooper and Schendel [6] as a tool for helping organizations to counter threats (e.g., disruption) arising from novel technologies. It refers to the group of activities that can lower uncertainty regarding technology. TOA enables the establishment of competitive advantages by forecasting trends, obtaining key technology information, and learning about research and development (R&D) opportunities [7]. Furthermore, TOA has the potential to facilitate technological progress either generally or in a particular discipline [8]. Obtaining insights of this kind in a timely manner is crucial for establishing a competitive advantage in both strategic planning and operational aspects.

To improve TOA performance, it is possible to leverage tools from the big data era. In particular, a data-driven TOA process involving the comprehensive analysis of technical documentation (e.g., patents) can be utilized to assist decision making [9]. Patents, which can be regarded as specific types of technical documents, are fundamental in knowledgebased economies, and they have garnered significant attention in the literature on technology competition and technology monitoring [8,10]. Patents contain more than 80% of technical information worldwide and are a trigger for new ideas and solutions [11]. The widespread application of information from patents includes areas such as technology forecasting, technology policy, technology assessment, and innovation improvement [8]. Patents serve as the legal basis for intellectual property rights, but they also contain rich content and detailed information relating to a specific novel technology [12].

Patent information is used for various purposes, ranging from legal to technological to managerial purposes. These uses include evaluating the originality or evolution of technology, finding competitors, determining the capacity for innovation, assisting in the design of patent planning and strategy, conducting quality analysis of patents, and finding patents with substantial promise [8]. As a case in point, directors within organizations can use patent information to safeguard against investing in R&D projects that will not yield benefits [13], coordinate R&D projects that lead to critical patents [14], and gain insight into popular or impactful technologies [15].

One characteristic of patents is an International Patent Classification (IPC) code, as defined by the World Intellectual Property Organization (WIPO) [16]. It is an index that serves the purpose of classifying inventions in a standardized way. An IPC code indicates the technology area that an invention belongs to and provides a hierarchically organized system of symbols that are language-independent, which can be used to classify patents and utility models. A definition of IPC codes is given at the WIPO's webpage (https://www.wipo.int/classifications/ipc/, accessed on 30 July 2021). As a classification tool, IPC codes are routinely applied in patent offices worldwide. IPC codes are categorized into five levels: sections (A-H), classes, subclasses, main groups, and subgroups. Based on an analysis of the distribution of IPC codes, it is possible to establish an understanding of the main technical areas in any given field. As a consequence, the analysis of IPC codes can also help in understanding the research areas and knowledge flow of the technology.

The value of patents is critical for competition in today's markets. If a patent can lead to commercial opportunities such as new product launches or new technology licenses, it is considered to have business value [17]. Patents are also a fine-grained source of information relating to markets, and at the level of countries, patents are reflective of the country's capacity for technological innovation [17]. For this reason, the number of patents associated with the country can also be viewed as a proxy for the country's level of innovation and technology.

In addition, a citation is defined as a reference to prior work (i.e., prior art) that is regarded as relevant for an ongoing patent application [15]. There are two main types of citations: backward citations and forward citations. Backward citations refer to patents that are cited by a specific patent, whereas forward citations refer to patents that cite a specific patent [15]. Commonly used indicators that assist in predicting the technological value and commercial viability of a patent are patent citations and the state-of-the-art they include, as well as the frequency of citing previous documents [15]. To be more specific, forward citations are frequently used as a proxy for the value of the patent. That is to say, a patent with a significant number of forward citations has a greater likelihood compared to those with a limited number of forward citations of leading to a competitive advantage and playing a key role in a particular field of technology [18]. Understanding the economic value of a patent, as well as its significance, aids in investigating the connection between firm performance and the number of forward citations. In the research undertaken by Chen and Chang [19], the researchers confirmed that citation value is positively associated with market value. Nevertheless, the authors found that when patent citations exceed an optimal threshold value, they are negatively associated with corporate market value owing to the R&D spillover effect. For this reason, tracking the forward citations of any given patent application enables the identification of emerging competitors, potential infringers, and future licensing opportunities. The total number of citations can be used to determine the market value of the technology of the active or influencing assignees.

Text mining has been applied throughout the literature for the analysis of technology opportunities, tracing or monitoring the evolution of the technology, and identifying upcoming trends [20,21]. As a case in point, Chae and Gim [22] developed a model for investigating technical inventions and promoting competition in innovation from patent applications on the basis of patent classification systems. The authors proposed a hierarchically organized technological taxonomy of the classification of each patent, which detailed key developments in patent applications. In the research of Liu et al. [23], the authors undertook social network analysis to examine developments in patent collaboration in China in the field of smart technology for smart grids. For the purpose of identifying the positions of technology in a network (e.g., in terms of the greatest importance, the most influencers, and the most interconnections), the researchers calculated indicators such as betweenness centrality and degree centrality. Given the growth in the value and significance of patents, a range of analysis and search systems have been developed. Further analysis system studies are required to analyze patents worldwide from a diverse perspective. Hence, this study focuses on patents, which have emerged as important not

only for business value but also for social value, and it analyzes them from a policymaking and technology managemen<sup>t</sup> point of view, highlighting the implications from patent analysis.

Despite the value of patent technology opportunity analysis in enabling the identification of technology competition and the investigation of strategies for business and technology development [24], the literature on patents for shelf-life extension technology is—to the best of our knowledge—limited. Most prior studies have focused on analyzing supply chains that handle fruits, vegetables, and other foodstuffs, and they have not targeted the question of novel technology for shelf-life extension [25–27]. As a case in point, Tatry, Fournier, Jeannequin, and Dosba [25] reviewed the literature on fruit and vegetable species to identify key actors, topics, and species, thereby establishing an accurate picture of the state of the research landscape. In the research undertaken by Daim, Rueda, Martin, and Gerdsri [26], the focus was the patent analysis of food safety technologies, which enabled the authors to predict upcoming technologies, market responses, and commercial successes in this sector. Hence, the study's results have strong empirical grounding and may motivate researchers to engage in deeper scientometric studies in other fields.

In light of several factors—namely, the rapid expansion in the number of patents for shelf-life extension technology, the growing attention paid toward industrial applications in this area, and the emergence of innovative technologies such as bio-based technologies [28,29]—the following research questions are important to pursue:


With these questions in mind, this paper presents a scientometric text mining approach known as technology intelligence—which leverages both qualitative and quantitative methods—to assist in patent analysis. The model offers methods that can be used to understand the conceptual structure in exploring the development of technology areas, as well as the social structure in terms of networks and collaboration patterns at a multi-level perspective (i.e., country-level, organization-level, and so on). This process empirically applies patents associated with fruit shelf-life extension technology, which is a problematic area in post-harvesting managemen<sup>t</sup> in agriculture, with the aim of establishing a holistic understanding—encompassing micro to macro views—and presenting insights from a meta-perspective that facilitates comprehension of the current state, development, and trends in shelf-life extension technology research.

#### **3. Methodology and Data**

The concept of technology intelligence, which refers to the activity of extracting crucial decision-making information to promote innovation [7], was used in this research. Additionally, technology intelligence enables researchers to gain insight into technological developments that produce competitive advantages [30]. Therefore, scientometric analysis, which refers to technology intelligence as an approach for technology opportunity analysis, was used for patent analysis. The use of this approach is valuable in finding existing areas in which technology is under development (e.g., specific technology fields and sectors), as well as exploring trends in research network and collaboration. In particular, this can yield benefits for governments, corporations, and universities in terms of supporting and guiding R&D. Scientometric patent text mining was used to yield insights from the analysis of raw big data pertaining to patents. As previously noted, the focal point of the research is post-harvest food management, particularly technology for fruit shelf-life extension. This focus was selected to illuminate existing technologies in the field, as well as key players, thus enabling future collaborations for the improvement of food quality, security, and safety. Considerations relating to this study's materials and methods are discussed in the next sections.

#### *3.1. Data Source*

When investigating the state of technical resource distribution and the development features of a certain area of technology, it is possible to organize complex technical information into comprehensible and logical statistics based on an analysis of patent data [31]. For this reason, applying scientometric analysis to patent data holds significant promise. In this research, we specified keywords associated with this technology. Following the Boolean approach described by Porter et al. [32], and also with the assistance of an expert researcher, search terms were built up from initial search strings. The following search term was used to retrieve patent documents from TotalPatent One: 'TITLE-ABS ("shelf life" OR shelf-life OR "storage life") AND (extension\* OR extend\* OR increas\* OR improv\* OR prolong\* OR pro-long\*) AND (fruit\*) AND (postharvest\* OR post-harvest\* OR fresh\* OR "fresh cut" OR fresh-cut)'. In this search term, the asterisk guarantees that the search will not exclude variants of the words.

The data collection process was conducted throughout April 2021. One of the most comprehensive patent databases available online, TotalPatent One (https://www.totalpatentone. com), was accessed on 1 April 2021 and used as the data source for retrieving patent documents, and the search was restricted to the period between 2000 and 2020. TotalPatent One, a patent search platform, was used because of high coverage data with more than 100 patent authorities [33]. The rationale for selecting 2000 as the start year was based on a finding from our prior analysis, which indicated that prior to 2000, a regulation existed that prevented universities and governmen<sup>t</sup> agencies from owning patents, and the number of patents was limited [34]. It is also important to note that as a result of the lag period between filing and publishing patents, the number of relevant patents identified in this study was not completed. Nevertheless, this did not influence our analysis of this area of technology. At the end of the data collection process, 2516 patents were identified, which were subsequently imported into the text mining software for data analysis.

#### *3.2. Data Cleaning*

As an essential pre-processing step before data analysis, data cleaning was applied to eliminate errors and duplication arising from variability in expressions and names. Unmatched data were combined to facilitate standardization, and the "List Cleanup" tool— paired with a manual cleanup—was applied to unify country, assignee, and inventor names. As a case in point, certain applicant names are the same but they are expressed differently; for this reason, text manipulation algorithms were applied to the applicant names and they also underwent careful manual inspection. Every applicant's name was converted into a term with the same meaning, which was also the case for terms such as "Co.", "Co", "Limited", "ltd." and "Ltd", which were substituted with empty strings.

#### *3.3. Data Analysis*

Porter and Cunningham's [35] nine-step text mining approach—beginning with problem identification and ending with utilization—was applied to analyze the data. Additionally, given that quantitative methods for text mining applied in isolation are unable to yield insights from the data, qualitative data—specifically, the evaluations of domain-specific experts—were used to lend greater robustness, depth, and credibility to the results. This process is based on a more concise and general adaptation of our previous research [34,36]. The framework's scientometric process is outlined in Figure 1.

VantagePoint version 13.1 (https://www.thevantagepoint.com/), from Search Technology, Inc., located in Norcross, GA, USA, was used as the text mining software for data analysis. The rationale for using this software was that it is capable of managing big data (i.e., the number of patents retrieved from TotalPatent One), and it also offers a useful suite for refining, investigating, and reporting on information. Additionally, the software can perform a range of scientometric procedures—ranging from the simple to the sophisticated—that are valuable in enabling the identification of patterns, relationships, and trends, which are, in turn, essential for the classification and visual representation of

big data. VantagePoint is also capable of inferring relationships between data fields (e.g., the connections between assignees, countries, and technology development areas, as well as their corresponding collaborations, citations, and organizations) [37].

To generate a data overview, scientometric indicators and descriptive statistics were used, examples of which include publication years, country productivity, assignee productivity, annual patent growth, and analysis of filing years. More advanced techniques were also used for knowledge synthesis, including conceptual structure and social structure. Conceptual structure illuminates what science and technology focus on by classifying technology fields and sectors, and social structure focuses on the interactions between actors (i.e., via analysis of assignees and countries), collaboration patterns, and technology and market opportunities [38]. As a case in point, to identify technology sectors and technology fields, International Patent Classification (IPC) codes were used for the following purposes: first, for the analysis of market of interest, where these refer to the countries in which most patents were filed by non-residents (i.e., filings by entities that were not domiciled in the target country); and second, for assignee analysis, where the assignments that were universities, governmen<sup>t</sup> agencies, individuals, and companies were regarded as understanding the role in collaboration and its technology development opportunities. The details of the data analysis are the following.

First, for the statistical analysis of technology evolution (Section 4.1), the trend line was constructed using MS Excel to understand the evolution and growth rate of numbers of patents. This helped to establish a clear picture regarding the stage of the technology life cycle (TLC). Furthermore, to understand the evolution in each stage of the TLC, it is notable that statistical tools are available to implement tests to explore the differences between each stage of technology development. For this reason, we examined whether specific patent indices (numbers of patents and values of examination periods) are associated with

different values in the identified TLC phases (here, the emerging phase and the growth phase). The tested indices were acquired from bibliometric information from the database that was exported to the software for the analysis. An interesting question relates to the issue of whether the length of the process of examination (i.e., the time elapsed between the year a patent was filed and the year it was published) influences the two phases. Additionally, to comparatively examine the means of two groups (in this case, emerging phase and growth phase), we applied an independent samples *t*-test using IBM's SPSS (version 22) from IBM Corporation, located in Endicott, NY, United States. This made it possible to generate statistical evidence suggesting that the associated population means were significantly different or not. Specifically, for the independent samples *t*-test, Levene's test was initially performed to identify whether variance in the length of the examination process was equal due to different formulations in calculating *p*-value in *t*-test analysis. The cutoff point for the *p*-value was set at 0.05. For *p*-values less than 0.05, this indicates that the mean values of the examination periods are statistically significant between a given two TLC stages.

Second, for IPC code analysis (Section 4.2), data clustering techniques, which are techniques for data mining analysis, were used to identify structure in multivariate datasets. The K-means clustering algorithm can be applied in various areas with beneficial effects, and the rising level of computing power has resulted in the greater availability of large datasets [39]. Data clustering utilizes partitioning-based techniques, which rely on the iterative movement of data points from cluster to cluster. Data clustering leads to the division of the data points in a dataset into non-overlapping clusters or groups based on their characteristics. The idea is to generate clusters of data points that are highly similar within the group and minimally similar between the groups [40]. Thus, we applied this method to patent data by classifying three attributes (IPC codes, technology sectors, and technology fields) to group patents into clusters. We initially conducted data analysis to gain the profiles of patents in our dataset and then conducted a cluster validation process to find an optimal number of clusters in patent data [41]. As a result, we set five clusters (k = 5) according to the cluster validation process.

Third, for market opportunity analysis by patent filings (Section 4.3), we applied the patent filings profiles (e.g., origin countries and targeted countries) to understand market opportunities for technology commercialization. In this research, we focus primarily on market analysis by exploring both origin and target countries, and we especially seek to gain insight into the nature of countries' potential markets. As a result, we can obtain information between original countries and targeted countries in terms of whether they focus on domestic or international markets.

Fourth, for the market analysis using numbers of citations (Section 4.4), we applied forward citation analysis to explore the trends regarding patent applications that enable the identification of opportunities, namely, competitors, potential infringers, and future licensing opportunities.

Last, for the collaboration analysis (Section 4.4), and for the purpose of evaluating and identifying collaboration in technological development, VantagePoint was applied to construct a cluster map reflecting the collaboration network shown in the retrieved patents. In a cluster map, the connecting lines represent collaborative research groups in which both an assignee and co-assignees are mentioned in the patents. Furthermore, the yellow nodes correspond to the number of patents, but where the size would be too large, numerical values are shown.

#### *3.4. Data Visualization*

Data visualization was undertaken after the analysis process. In particular, to gain insights into the development and evolution of technology, graphs, clusters, and maps were applied, principally because they serve as a decision-making aid. Data visualizations of this kind are expected to play an essential role in guiding executives and managers within corporations, governments, and higher education institutions to develop strategies for R&D, as well as to direct future planning and network formation.

The basic visualization tool applied in this research is the graph (e.g., line chart, bar chart, pie chart, donut chart, etc.), which is employed in the data representations. The advanced visualization tool used in this research is the cluster map (Section 5.3). This map is based on co-occurrence analysis, which assists in creating lists (called nodes) of items by combining all the terms to generate clusters. The sizes of nodes refer to the numbers of records and the linkage lines refer to the relationship degree [36]. The map helps readers understand the groups of interested items and their relationships.

#### **4. Results and Discussion**

#### *4.1. Evolution over Time*

Based on the patents retrieved from TotalPatent One, Figure 2 provides an overview of the number of patents published per year (bar chart), the cumulative frequency representation of these numbers (solid line chart), and the trendline of growth (dash line chart). A total of 2516 patents were published on shelf-life extension technology between 2000 and 2020. Based on the technology life cycle (TLC), it is possible to separate the technological development into two phases, which are the "emerging phase" and "growth phase". In Figure 2, the emerging phase, where the growth rate increases linearly, lasts from 2000 to 2007, whereas the growth phase begins in 2008 and continues through until 2020. The number of patent publications reached a peak in 2008, and most of these were from corporations where patents filed in 2006 were ultimately granted in 2008 (see Section 3.3 for details). This may be attributable to the fact that Achour [42] proposed a novel indicator, the Global Stability Index (GSI), in 2006 (with a pre-published release in 2005), which can be used to quantify the decline in quality of a foodstuff during storage or commercialization. GSI enables food shelf-life to be estimated effectively by integrating diverse attributes of food into one measure. It has been shown to yield favorable results compared to the traditional procedure of accelerated shelf-life testing (ASLT) [43]. The novel method may have made it easier to undertake more sensitive and precise experiments to quantify food shelf-life, becoming one of the factors to attract researchers to this technology.

**Figure 2.** Technology life cycle of shelf-life extension technology with respect to chronological development.

Figure 2 indicates that over the period from 2000 to 2020, the rate at which patents were published increased significantly. Between 2000 and 2007, an average of 41 patents was published each year, but between 2008 and 2020, this increased to 168 patents per year. This rate of growth corresponds to a polynomial curve obtained from MS Excel (R<sup>2</sup> = 99.6%). Taken together, these data sugges<sup>t</sup> that shelf-life extension technology is growing in popularity among players and is associated with promising possibilities both for R&D and commercialization. The reduction in the number of patents published in 2020 stems from the fact that the data were incomplete at the time the research was undertaken. It is noteworthy that the patents published since 2015 represent 50% of the total number of patents retrieved from TotalPatent One. Continuous patent applications and rising numbers are an indicator of the maturity phase in the TLC [44], but shelf-life extension technology has not ye<sup>t</sup> attained maturity. To summarize, patents concerning shelf-life extension technology indicate that this is an emerging field marked by growing popularity.

A useful area of investigation is the identification of patent indices that display typically different values at each phase in the TLC of a given technology. We sought to determine whether our case study was consistent with the results from prior studies. Descriptive statistics relating to this issue are given in Table 1. On average, the examination periods for the emerging and growth phase were 3.18 ± 2.07 and 2.39 ± 1.97 years, respectively.

**Table 1.** Descriptive analysis of patents at different TLC stages.


As shown in Table 2, the results of Levene's test and the *t*-test led to the conclusion that the variances and mean values of examination periods between the two groups were statistically different. Hence, this serves as strong evidence indicating that the time for the examination process of the two phases was different at the significance level of 0.05. There are several possible explanations to account for the finding that the examination process lasts significantly longer in the emerging phase compared to the growth phase. Haupt et al. [45] explained that at the outset of any given technological development (i.e., in the emerging phase of the TLC), applicants often submit broad claims with the intention of limiting opportunities for subsequent patents. In turn, this increases the length of the examination process. It is also notable that the longer examination times associated with the emerging phase can be accounted for by referencing the fact that the examiners lack specific experience concerning the technology at the emerging stage [45]. Moreover, after shorter examination processes in the growth stage, Haupt, Kloyer, and Lange [45] expected a longer average duration for the maturity stage because the applications have to be compared to a higher technological standard; however, our technology life cycle has not reached that stage. At the same time, there are diverse determinants that may influence the examination process, including application characteristics (e.g., total number of classifications), applicant characteristics (e.g., applicant type), and environmental characteristics (e.g., heterogeneity in the technology area) [46].

**Table 2.** Statistical testing between TLC stages.


*4.2. Technology Topic Analysis*

4.2.1. Overall Technological Development

IPC codes can be used to identify key technologies and emerging technologies. As shown in Figure 3, the technical topics in shelf-life extension patents focused primarily on section A (Human Necessities; 2168 pieces, 74.2% of records), section C (Chemistry and Metallurgy; 356 pieces, 12.2%), and section B (Performing operations; Transporting; 288 pieces, 9.9%). The remaining patents were in section F (Mechanical Engineering, Lighting, Heating, Weapons, and Blasting; 41 pieces, 1.4%), section G (Physics; 36 pieces, 1.22%), and other categories. The fact that most patents were in section A is consistent with this section's focus on foodstuffs, including both products (e.g., fruits) and processes (e.g., treatment and nutrition modification) for preservation such as disinfectants and to prevent the growth of organisms.

**Figure 3.** IPC codes derived from patents database.

For the detailed analysis, we used the proportion of IPC subgroups of subfields to indicate the leading ten IPC subgroups, which are shown in Appendix A (Table A1). Approximately 9380 records of IPC subgroups were identified in the analyzed report, which indicates that a non-obvious disparity exists in the patent applications among different IPC codes. The IPC in the first place occupied 4.2%, whereas several other IPC subgroups occupied less than 3%. We also noted that the IPC subgroups that count only a single time over the analysis period amounted to 598 pieces, occupying 6.4%, which resulted in a reduction of intensity in the leading IPC subgroups. Those one-time cited subgroups can be considered inactive fields. Based on the IPC subgroups that corresponded to the number of patents, it was found that the technical topics for global patents focused primarily on enzymes, organic compounds, and microorganisms (A23B 7/154); coating protective layers, or compositions or apparatus (A23B 7/16); and fruit or vegetable products with preparation or treatment (A23L 19/00), and so on.

#### 4.2.2. Technological Sectors and Fields

Data clustering was used in this research to identify patent data characteristics. Three variables (or attributes) were used to group the patent datasets: IPC codes, technology sectors, and technology fields. In total, 1403 IPC codes in the subclass level were distributed

in each cluster and calculated as a percentage. There are the five clusters (or technology sectors) showing a group of patents, including chemistry (73.8%), mechanical engineering (17.2%), instruments (2.6%), electrical engineering (2.5%), and other fields (3.8%). Data clustering led to the identification of the two largest numbers of patents and three clusters with a relatively small number of patents. The "chemistry" cluster had the largest number of patents. This cluster can be considered to represent an adequate technical sector as it is associated with a large number of registered patents. By contrast, the "instrument", "electrical engineering", and "other fields" clusters can be viewed as inadequate technology clusters due to their limited numbers of patents. Every cluster contains data that reflect the relationship between IPC codes and the key terms extracted from patent titles. Both can be used to describe the technologies, inventions, and influencers that are useful for R&D and technology managemen<sup>t</sup> in the future.

Table 3 provides an overview of patent cluster characteristics based on the attributes of the technology sector, technology field, and IPC subclass code. Each of the clusters comprises specific technology fields and IPC codes. The IPC subclass codes serve as a representative of the inventions shown in each technology field, and it is possible to use these codes to assess the connections between technologies. In particular, this can be achieved by utilizing association rule mining to discover relationships among technological developments. In this study, it was found that the main group and subgroup under the IPC subclass codes resulted in different technology sectors and fields that are not presented in this paper.


**Table 3.** Five technology sectors based on K-means clustering algorithm.


**Table 3.** *Cont.*

The next step in our analysis involved applying various processes to extract key terms from the titles of patents. These processes included tokenizing, stop word filtering, transforming cases, and stemming. An example of patent titles is given in Table A2 (Appendix A). Additionally, Figure 3 shows that the five shelf-life extension technology clusters were largely concentrated in terms of IPC distribution. There was also a clear disparity in the patent applications across the IPC codes. The diverse nature of the cluster distribution reflects the fact that, due to the complex business environment [28], the demand for effective and universal technology is increasing. It also appears to be the case that most patents in the field of shelf-life extension technology are the products of interdisciplinary academic research. For this reason, emphasizing interdisciplinary collaboration is worthwhile among inventors and researchers to produce new viewpoints and lead to favorable research outcomes.

#### *4.3. Country-Level Analysis*

#### 4.3.1. Countries' Productivity

The purpose of this section is to present a general overview of development trends in shelf-life extension technology. Models such as the PESTEL framework, which is an acronym for a series of factors (i.e., political, economic, social, technological, environmental, and legal), reflect the fact that in any particular technological area, the competitive capacities and resources of countries differ. Table 4 shows the evolution trends of the 10 leading countries in terms of the number of published patents. The top countries during the emerging phase, each with more than 30 patents, were China, the United States, and Australia, whereas for the growth phase, the share of the number of patents associated with the United States and Australia declined. Of note, China's share of patents increased significantly to 62.4%. Additionally, Russia, which was the second-leading country, grew from 18 patents to 141 patents in the later phase, which led to a slight increase in the proportion of Russian patents. In total throughout the years, the ten leading countries held more than 80% of the existing patents. In terms of the total number of patents, China accounts for more than 50% of the global total, and as such is the leader in the field. The results indicate that the greatest number of patents was associated with the Asian region, including China, India, Korea, and Japan. In the EU, Russia alone was noticeable in terms

of its patenting activities. Whereas Canada and the United States played a significant role, the main player in South America was Brazil.


**Table 4.** Top ten countries in terms of number of patents \*.

Note: NP = Number of patents. \* The results were obtained and analyzed based on data retrieved from the database.

Figure 4 illustrates that during the emerging phase, all of the countries contributed patents closely (see Table 4). After 2007, the leading five countries, with the exception of China, grew gradually in terms of the number of patents. However, the rate of patent publication in China grew significantly over the two periods, leading to Chinese supremacy in this technological area. Specifically, China's dominant position in the growth phase, accounting for 62.4% of the percentage share, was preceded by a share of 14.3% in the emerging phase. One way to account for this result is by referencing China's recent emergence as a leading exporter in the fruit market. China's status in this area has produced development chances for Chinese inventors and applicants in technology development. It is also noteworthy that China's evaluation system strongly values patents [47]. Regarding Russia, it is notable that a peak occurred in terms of the number of patents published in 2008, amounting to over 100. This observation is consistent with Figure 2 wherein a clear peak occurred in 2008. In terms of detailed analysis, the patents published in 2008 were originally filed in 2006. These patents are those where the applicants used a novel method for the storage of different fruits. Most of the patents were filed by Kvasenkov Oleg Ivanovich, a member of the Russian Federation and the Russian Food Institute, who completed the paperwork as an individual, and ranked as one of the leading 100 patent applicants globally in 2016 [48]. In addition to China, the United States is also notable in terms of the number of patent applicants published across the emerging and growth phases.

Regarding R&D collaboration for patent production, our study identified no collaboration for patents across countries. That is to say, the applicants for each patent were always affiliated with a single country rather than multiple countries. Additionally, collaboration was not found at the level of continents, indicating that countries have nationally-bounded technology development that does not leverage the advantages of geological distance. Whenever companies, researchers, or inventors seek to create or develop products or services that are categorized into different technology sectors, it is necessary for them to explore whether competitors or other assignees are patent holders; this safeguards against conflicts in terms of intellectual property. The number of patents can be viewed as a proxy for the level of technological development in a given area in each country.

Figure 5 provides an overview of the leading countries in the respective technology sectors based on the number of patents in each cluster. The figure indicates that China is the greatest contributor across every sector, reflecting China's strong influence on technological development. The United States ranked in the top five countries across all sectors with the exception of the instruments sector. Our analysis also revealed that other countries produce technology and patents in diverse technology sectors. As a case in point, in

the field of mechanical engineering, which is concerned with machinery manufacturing, Russia was ranked as the second-greatest influence. The competitive advantages that Russia benefits from primarily relate to its advancement in machine-tool and equipment construction [49]. In the case of India, it is noteworthy that India's rise as a substantial economy in recent decades is significantly reflected in developments in the equipment and instruments sector and electricity sector, which form the core of the Indian electronics industry [50]. India's growing exports to Canada, Australia, Germany, and the United States are facilitating industry growth in industrial electronics (e.g., process control equipment, analytical instruments, automation instruments, and measuring and test equipment) [50].

**Figure 4.** Patent publications for top five countries from Table 4 (CN = China, RU = Russia, US = The United States, AU = Australia, and IN = India).

**Figure 5.** Top five countries in terms of numbers of patents categorized into five clusters in which the same color refers to the same country (Note: CN = China, RU = Russia, US = The United States, AU = Australia, IN = India, CA = Canada, DE = Germany, and TW = Taiwan).

4.3.2. Market Opportunities by International Patent Filings

The market refers to the destination of technology development. Patent analysis was applied to illuminate the nature of the target markets. A patent family is commonly defined as a set of patents filed in various countries with the aim of safeguarding a single invention [51]. When an entity aims to protect its invention in several jurisdictions, it

is necessary to obtain patents in all corresponding patent offices. Due to this, the initial filing (also known as the priority filing), which is submitted to serve as protection for the invention, is succeeded by a sequence of filings, together constituting a patent family. In view of this, it is possible to pair patent family data in order to investigate international technology markets. In this analysis, the origin and target countries of a patent, as well as organizations, are represented in Table 5 by country or area codes of each applicant's country information. Along with countries, the World Intellectual Property Organization (WIPO) and the European Patent Office (EPO) are key organizations in which countries can apply for patents, and so statistical analysis is included here for them. As a case in point, the Patent Cooperation Treaty (PCT) system helps applicants in establishing international patent protection [48]. Therefore, Table 5 presents the number of patents in target countries or organizations of this technology-related patents from the main origin countries. In the columns of Table 5, the number of patents applied for by the countries is shown.



Note: CN = China, RU = Russia, US = The United States, AU = Australia, IN = India, CA = Canada, KR = Korea, MX = Mexico, BR = Brazil, JP = Japan, WO = WIPO, PC = Pacific Islands, GB = United Kingdom, EP = EPO, and IL = Israel.

> Broadly speaking, it is possible to separate patent applicants into two groups: (1) applicants who primarily apply for patents locally, including China, Russia, the United States, India, and so on; and (2) applicants who primarily apply for the patents internationally, including Australia, Canada, Korea, Mexico, Brazil, and Japan. In the first group, China's assignees applied for almost all of the patents in their home country. Although China made patent applications in the United States, the Pacific Islands, and WIPO, the numbers are significantly lower compared to applications from Canada and Mexico. This might indicate that the future market of China is primarily situated domestically. Notably, this is consistent with the suggestion that China will serve as a leading fruit export country, where the technology must be used domestically for exporting. The possibilities for China internationally appear not to be developed at present, which can be attributed to the domestic priority. Countries such as China, Russia, the United States, and India rank in the top countries for domestic patent applications. It has been noted that these countries typically hold a small number of family patent applications, as well as different patent applications, to protect various novel technologies.

> In terms of the second group, it is possible for patent families to reflect patent value. This is because the overseas filing of patents typically leads to greater costs in the case of the applicant (i.e., due to patent office fees, patent attorney bills, and translation costs). The consequence of this is that applicants only seek to protect their inventions in other countries if it is worth it along the dimensions of cost, effort, and time [52]. Based on patent family size, it is reasonable to sugges<sup>t</sup> that Australia, Canada, Korea, Mexico, Brazil, and Japan have relatively highly valued patents, which is potentially closer to commercialization. As a case in point, compared to the number of local patent applications, the total number of

international patent applications from Australia and Canada was almost 10 to 14 times greater. This is indirectly reflective of the fact that applicants in this group are strongly internationalized, and they typically apply in multiple countries for patent protection for a single invention. Due to this, Australia and Canada are characterized by intense market competition. At the same time, the interesting target countries are the Pacific Islands (i.e., Australia and New Zealand). China, the United States, and Canada are seeking to submit international patent applications in this region. This stems from the competitive advantages associated with fruit cultivation such as various types of fruits planted in Australia and New Zealand, which results in effective commercialization and the growth of fruit exports [53].

To be specific, applicants who made applications to other countries or organizations primarily center on the United States and WIPO. For the purpose of reducing cost and simplifying the process of submitting other patent applications, almost all applicants made patent filings in WIPO. In the process of making only a single international patent application, it is possible for applicants to protect their novel technologies and inventions in multiple jurisdictions. WIPO is the primary target for international patent applications and, due to this, the patents granted are referred to as PCT patents. When an application for a PCT patent is made, patent rights associated with the PCT patent have validity in every PCT member mentioned [54]. Currently, there are approximately 800 pieces of PCT patents associated with shelf-life extension technology, which indicates that each country is engaging in international competition, which is expected to intensify in the coming years.

#### *4.4. Assignee-Level Analysis*

#### 4.4.1. Assignees' Productivity

The following four types of players were identified as assignees: companies (50.7%), universities (23.1%), individual researchers (where the applicant was the researcher's name; 20.1%), and governments (including state-owned R&D institutions; 6.1%). Most contributions were submitted by companies engaging in R&D for the commercialization of shelf-life extension technologies for agricultural food. Two main types of products were observed that are involved in business-to-customer (B2C) and business-to-business (B2B), including the food packaging film and chemicals used during the post-harvesting process.

As illustrated in Figure 6, companies constituted the most influential assignee during the emerging phase (61.2%), whereas during the growth phase, academic institutions and universities developed considerably in terms of their patenting activities (increasing their share from 8.5% to 25.3%). Historically, higher education institutions have sought to bypass the costs associated with publishing patents and, instead, have prioritized research and the publication of scholarly literature to derive comparable benefits to those resulting from holding patents. As shown, this indicates that state-of-the-art technology in universities is reaching a higher technology readiness level (TRL) in terms of viable commercial applications [55], thereby increasing the number of patents.

**Figure 6.** Four types of assignees in view of the technology life cycle.

At present, higher education institutions have technology transfer offices (TTOs) to assist the commercialization of research output (e.g., in terms of licensing, spin-offs, and patents) [56]. This accounts for the fact that universities published a greater number of patents during the growth phase compared to the emerging phase. Although universities can own patents, the effective commercial application of any patents held depends on effective strategizing and business model formation. On filing patents, universities may experience difficulties when seeking licenses because the technology may represent only an initial breakthrough in a lengthy process of development [57]. In many cases, higher education institutions will need to wait many years to receive a return on their investment.

In our organization-level focus on assignees, individual researchers or investors were excluded for the purpose of identifying the role played by corporations, universities, and other institutions in the development of shelf-life extension technology. Figure 7 provides an overview of the leading 10 applicants with respect to the number of patents. In this figure, the country code presented after the assignee name is indicative of the country of origin. The figure indicates that assignees 1, 4, and 10 were from corporations, whereas the remainder were from universities. The leading players in this space are Chinese universities seeking out opportunities to commercialize patents (e.g., licensing and spinoffs). Nevertheless, the leading ten applicants included a single US firm (ranked first), a Canadian firm (ranked fourth), and an Israeli firm (ranked tenth). The top-ranked firm, Mantrose-Haeuser, specializes in the development of edible coatings and specialty products for the food, industrial, and agricultural industries.

Chemistry Mechanical Engineering Instruments Electrical Engineering Other Fields

**Figure 7.** Top ten registered applicants for shelf-life extension technology according to technology sectors (Number one to ten is based on ranking in terms of number of patents). Note: CN = China, US = The United States, ES = Spain, and IL = Israel.

#### 4.4.2. Market Opportunities by Number of Citations

Table 6 provides an overview of the leading 10 shelf-life extension technology patents in terms of the number of forward citations. Each of these patents was published in the growth phase of the technology life cycle. Of note, eight of the ten patents are held by China, with one belonging to the United States and another to Japan, which reflects China's competitive advantage in terms of highly-cited patents. A total of four of the patents are held by corporations, whereas six are held by universities and research institutions. Hence, it is reasonable to conclude that universities and research institutions constitute the core

driver of technology strength in this area. As the table indicates, the top-ranked assignee in terms of forward citations is concerned with a method for generating fresh-cut fruits and vegetables by leveraging gas and ultra-high pressure technology, which was submitted by China Agricultural University. Additionally, Zhejiang University holds two of the top ten cited patents, which reflects the university's status as an important industry player.

**Table 6.** Top ten forward citing patents.


Patents from Zhejiang University, CN103583675A and CN103583675B, also shown in Table 6, are concerned with prolonging the shelf-life of Chinese bayberry fruits (e.g., waxberry and yumberry) by applying low-temperature environments. Bayberry fruits are crucial economically and rich nutritionally, and they have been harvested in southern China for two millennia [58]. Bayberries, along with processed products that involve bayberries, undergo exportation to numerous countries (e.g., France, Spain, and Singapore), and the volume of exports has increased in recent years [59]. A particularly notable point is that all of the patents were filed in China. However, there are a number of assignees from other countries (e.g., Japan and the United States) that have identified market opportunities and sought to protect them in China. In the case of NatureSeal, this organization has played a critical role in the development of shelf-life extension technology globally. The patent CN106998716A, which is held by NatureSeal, facilitates corrosion-proof and tasteenhancing cutting of fresh products, including agricultural products. For CN107529769A, which is held by Japan's Maruha Nichiro Corporation, the invention is concerned with extending the shelf-life of strawberries.

#### 4.4.3. Collaboration Opportunities

The results indicate that most assignees of patents were individual players (i.e., either single corporations or universities). Furthermore, collaborative activities among the main players were limited relative to the total number of patents. This is indicative of the fact that collaborative activities were common within countries but not between countries. The primary individual player was corporations, followed by universities. This is inconsistent with the research collaboration found in the academic literature, where collaboration at diverse levels is observed (e.g., at the level of individuals, organizations, and countries) [36,60]. In this case, patents are inextricably linked to laws and regulations, technology benefits, and commercial opportunities, which is a fact that may lead to conflict among collaborators. We note that the use of different database sources may influence the results.

Although the relationships between entities were limited, some research collaborations among other players were identified (Figure 8). The most robust relationship was observed between corporations and researchers or inventors (64 patents named together as an assignee). This is typically seen because researchers who work within corporations often negotiate to be listed as an assignee when a patent is filed. At the same time, it is common for companies to create employment contracts for inventors stipulating that the

inventor's patent rights will be secured and protected. In the event that a corporate entity decides to be the assignee of the patent before establishing a sizeable portfolio, this may lead to complexities if their industry rivals create a "patent fence" around a technology area (i.e., strategically submitting a sequence of patents to disrupt R&D avenues) [61]. To safeguard against risks of this kind, companies can leverage their employees' contracts regarding patent rights, thereby filing patents in the employee's rather than the organization's name. This helps to avoid revealing the organization name in patents and, notably, a common practice is for companies to do this initially, and only afterward to transfer the assignee name from the name of the inventor to the name of the company. This enables organizations to buy time with which they can develop a sufficiently robust portfolio for patent enforcement [62].

**Figure 8.** Collaboration mapping of groups of assignees.

Regarding intra-collaboration within groups (i.e., corporate–corporate collaboration or university–university collaboration), no evidence of intra-collaboration was identified across the leading 20 entities. This reflects the fact that intellectual property such as ownership emerges as one of the issues in the context of negotiation. Even though companies frequently pursue patent applications independently, it is still necessary for them to collaborate with other entities as a consequence of limitations in terms of time, budget, and human resources. The number of collaborations between corporations and universities, as well as between corporations and governments, amounted to approximately 35 patents in each pair. As influenced by the input of experts, private companies can lower R&D costs by recruiting collaborating with other players (e.g., universities) because, for example, this prevents them from having to invest in expensive equipment or facilities.

Evidence was also found that our analysis for collaborative activities indicated three players (i.e., corporations, universities, and governments). In particular, the patent "Coating Agent for Fresh-Cut Fruit and the Manufacturing Method Thereof" emerged from collaborative activities among these three key players in Korea. At this point, the concept of the triple helix innovation model is worth noting, which stipulates that the universitygovernment-industry helix is a source of economic development, knowledge development, and growth in innovation [63]. Of note, the notion of academic entrepreneurialism is linked to the triple helix concept. In 1980, Bayh-Dole Act (or Patent and Trademark Law Amendments Act) changed legislation relating to US intellectual property ownership [64]. In particular, the new legislation made it possible for government-funded researchers to register patents based on their findings and confer licenses onto other parties. As such, this development enabled researchers and universities to register patents. Additionally, after

the reduction of state research funding, it was necessary for research institutions to seek funding from corporations [63].

# **5. Implications**

#### *5.1. Technology Development Opportunities*

With strong implications for technology opportunities, the outcomes of scientometric analysis indicate connections in available patent data. Technology sectors (i.e., clusters) and technology fields comprise the influential technologies, and opportunities exist to develop novel inventions and technologies. Organizations seek to identify patents to explore the concept and technology, which enables them to create new products and services and, at the same time, to ensure they do not violate intellectual property. Patent managemen<sup>t</sup> is crucial for organizations that depend on R&D to generate novel technology for their development. Organizations can explore the gap of technologies that have not been renewed, which is referred to as the "freedom to operate" (FTO) or "white space" [65], but technological development may be obstructed by patent holders. Companies can aim for partnerships regarding technology transfer, which could serve as a strategic approach to patent acquisition that is intended to commercialize or eventually protect patents [66]. It is possible to use the core technology to produce new developments in terms of products and services, and this does not lead to intellectual property violations. Additionally, when an organization can identify the white space or freedom to operate, it is also necessary to strive to assume a leading position in the field to maximize the value of the technology opportunity.

In the event that an assignee has more effective patent distribution, particularly in terms of white spaces, then they will benefit from a greater competitive advantage in terms of technical strength and, in this way, become a leader in the field. Regarding universities and research institutions, it is essential for them to increase the robustness of technology transfer and the industry-university-research system. To be specific, it is worthwhile for universities and research institutions to operate technology transfer offices (TTO), the purpose of which is to foster collaborative activities with corporations and industry, as well as to license inventions to industry for technology commercialization [8].

#### *5.2. Collaboration Opportunities*

With significant implications for collaborative activities, it is possible for universities, corporations, and governments to use the products of international collaborative networks [67] to gain insight into the overall trajectory of research and the evolution of patents worldwide.

First, it is essential for governments to allocate funding and resources to enable the cooperative innovation of different entity types (in particular, university-industrygovernmen<sup>t</sup> interactions). It is particularly crucial to motivate universities and research institutions to engage actively in collaborative innovation and technology development. As a case in point, mobilizing collaborative efforts between universities and research institutions can increase the strength of their capabilities and advocate individual innovation as a hub for industries to engage for the advisory in advanced technology development, thus leading to the promotion of collaborative innovation in this field. An interactive innovation model (e.g., incubators) has developed technology and business ideas into an array of firms, and to form research centers by combining diverse R&D entities from universities, governments, and industries, thus leading to the creation of a networked entity [68].

Second, it is reasonable to change the proportional structure of different collaborative relationships regarding patents in this field. It is an essential attempt to foster intracollaboration on patents between universities and universities or, alternatively, between research institutions and research institutions. It is noticeable that universities and research institutions have strong and independent R&D capabilities. For this reason, they can achieve robust cooperative alliances by beginning from intra-group collaboration, which stems from the fact that their goals are aligned. Furthermore, integration and resource

allocation in terms of human resources, technology, knowledge, and information can strengthen hub quality.

Third, it is essential to reinforce the frequency and intensity of collaboration by forming strategies in view of governmen<sup>t</sup> policies, which can lead to win-win situations. As a case in point, the talent mobility (TM) mechanism in policy has emerged as an issue of intense interest for universities, policymakers, and industries. This is because the model has substantial utility for innovation and is critical for researchers, particularly when the knowledge area has applied components in technology, science, and innovation for business commercialization [69].

Fourth, key influencers (e.g., large corporations) can offer support to establish several large-scale associations for collaborative innovation. It is possible to initiate the associations based on governmen<sup>t</sup> policy support, thus motivating corporations to foster associations to ensure they are at the center of a network in the same or diverse subjects. Owing to the establishment of these associations, authority and power can grow into various regions both domestically and overseas. As a case in point, Qiao, et al. [70] reported that when corporations are members of industry association networks, this strongly influences innovation and, in turn, performance in a positive way.

Finally, it is crucial to incentivize marginal entities to participate in communities for innovation, to establish cooperative relationships that foster technological innovation, to increase the robustness of knowledge and information sharing, and to facilitate long-term improvements in technology innovation for patent collaboration networks in this area.

#### *5.3. Innovation Ecosystem*

Figure 9 illustrates a multi-layered network innovation system, grounded on a set of patents, that has the capability to provide an account of the characteristics of the future innovation system [71]. To show the relationship, four layers are included, ranging from the business perspective (e.g., analysis at the country-level) to the technology perspective (e.g., fields of technology). In the case of the first layer (i.e., the layer at the top), this corresponds to the leading countries in the technology. As for the second layer, this shows the leading ten players, thus locating the principal actors in the innovation system. Hence, patent holders that were identified using the proposed approach in this study are shown. The third layer focuses on the level of technology sectors, and it visualizes their relationships on the basis of the similarity of their patents using IPC codes. In this layer, node size reflects the total number of patents associated with the respective sectors, and those with more patents are considered as having a greater level of activity in the creation of novel technologies. The fourth layer focuses on the field of technology, where relationships are established on the basis of technology field co-occurrence analysis. In the event that two fields are found frequently in the patents, these are considered to be interrelated.

In Figure 9, the two leading countries (China and United States) in patents for shelf-life extension technology are shown in the top layer. Examining this multi-layered network assists in knowing about active actors, interesting technological fields, and their association to technology sectors and different players. Based on this information, organizations can identify critical areas for R&D. Additionally, patent information performs a critical function in connecting innovation actors and areas of innovation technology. As a case in point, Mantrose-Haeuser, which was identified as a leading corporation in shelf-life extension technology, focused exclusively on the chemistry and mechanical engineering sectors, which reflects their positioning in the market. These sectors are in the domains of food chemistry and specialized machinery, both of which are associated with substantial future promise.

**Figure 9.** A multi-layered network within an innovation ecosystem.

#### *5.4. Limitations and Recommendations for Scientometric Analysis*

Just as it is important to acknowledge the contributions of this research, it is also worthwhile to state several notable limitations. First, given that most areas of technology develop at a rapid pace and the patent landscape concerning any technology will expand and change over time, this analysis is not timeless. Therefore, further research, including patent roadmaps, should be undertaken to identify changes that may influence strategic directions, particularly as shelf-life extension technology transitions from the current growth phase into the maturity phase. The second limitation is that the patents included in this study were retrieved from the TotalPatent One database only. The availability of other sources, including local patent offices, means that more sources could have been considered to analyze and compare results. Despite this, TotalPatent One is one of the most comprehensive databases available, which means that the results obtained in this analysis can be generalized. By comparing with various databases from local patent offices, it may help to gain a deeper understanding of the research activities from each country.

#### **6. Conclusions and Future Research**

The purpose of this study was to present a scientometric analysis of patents relating to the field of shelf-life extension technology. The analysis leveraged data mining techniques and focused on both conceptual analysis (technology clustering) and social analysis (productivity, opportunity, and competitive advantage). It also used a multi-level analysis approach encompassing both the country-level and entity-level, where the analysis was informed by the technology life cycle.

The growth in the number of patents published since the year 2000 reflects the fact that technology-related patents have been receiving an increasingly large amount of attention in both the research community and industry. The analysis indicates that in the growth phase of this technology, the time required for the patent examination process has generally been lower compared to the time required in the preceding emerging phase.

The clustering algorithm was used to identify group similarities in the retrieved patent data. The five technology sectors constituted the focus groups, where each group contained varying IPC subclasses, as well as diverse fields. As a result, the following five technology sectors were identified: chemistry, instruments, electrical engineering, mechanical engineering, and other fields. Chemistry was a notable technology sector for its role in the synthesis of novel chemicals that extend shelf-life, with many new developments relying on chemical-based and bio-based technology. The second most notable sector concerning patents for shelf-life extension technology was that of mechanical engineering, which seeks to develop innovative methods and physical techniques.

Regarding the social structure and network, our results indicate that although China is currently the most significant contributor to this technology, collaboration of all kinds on patent applications is not extensively apparent. As such, the patent collaboration network yielded by our analysis is small and not sufficiently dense to maximize collaborative innovation. Based on this study's analysis of patent collaboration, both corporation– university and corporation–government patent collaborations accounted for a substantial proportion of the identified collaborative relationships, with far fewer instances of other types of collaborative relationships.

Based on these findings, it is reasonable to conclude that in the field of shelf-life extension technology, there are significant differences in the proportions of collaborative relationship types formed by different patent applicants. Further to the results of our analysis, two corporations were identified as having key positions in patent collaboration with their local governmen<sup>t</sup> and local university, respectively. Thus, the patent collaboration network in this field has these corporations at its core. Nevertheless, marginal entities were identified in the network, including small enterprises, universities, and individual researchers, but a patent collaboration network dominated by several cores has not ye<sup>t</sup> been established.

For future research, workshops for strategy design and implementation can be undertaken with representatives from governments, universities, and industries. Such workshops may assist governments in their policymaking efforts to reinforce national progress in innovation and technology. In addition, a technology roadmap can be formulated to align research directions (i.e., short-, medium-, and long-term plans) with the advancement of this technology. In terms of the scientometric process, it is recommended to analyze the scientific publications relating to this technology, as well as to compare the results with the patents. This can help to illuminate the linkage between scientific development and technology development.

**Author Contributions:** Conceptualization, V.M. and N.M.; methodology, J.T. and V.M.; software, J.T.; validation, V.M.; formal analysis, J.T.; investigation, V.M.; resources, V.M. and C.G.; writing—original draft preparation, J.T.; writing—review and editing, V.M., C.G. and N.M.; visualization, J.T. and N.M.; supervision, V.M., C.G. and N.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was supported by Thailand Science Research and Innovation (TSRI) through its Research and Researchers for Industries (RRI) Scholarship Program, gran<sup>t</sup> number PhD62I0011.

**Data Availability Statement:** The data supporting the results are available upon request from the corresponding author.

**Acknowledgments:** The following institutions supported this research: Technopreneurship and Innovation Management Program at the Graduate School, Chulalongkorn University, Thailand; and the Thailand Science Research and Innovation (TSRI) through its Research and Researchers for

Industries (RRI) Scholarship Program. Search Technology's VantagePoint is also acknowledged for generating the results.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

# **Appendix A**

**Table A1.** Major topic distribution of the patents.


**Table A2.** Examples of patent titles from the technology sector.



#### **Table A2.** *Cont.*

# **References**


#### *Systematic Review* **What Proportion of Systematic Reviews and Meta-Analyses Published in the** *Annals of Surgery* **Provide Definitive Conclusions—A Systematic Review and Bibliometric Analysis**

**Matthew G. Davey 1,\*, Martin S. Davey 2, Aoife J. Lowery 1,2 and Michael J. Kerin 1,2**

> 1 Discipline of Surgery, The Lambe Institution of Translational Research, National University of Ireland, H91 YR71 Galway, Ireland; aoife.lowery@nuigalway.ie (A.J.L.); michael.kerin@nuigalway.ie (M.J.K.)

> 2 Department of Surgery, Galway University Hospitals, H91 YR71 Galway, Ireland; martindavey@rcsi.ie

**\*** Correspondence: m.davey7@nuigalway.ie; Tel.: +353-91524411

**Abstract: Objective:** To perform a systematic review and bibliometric analysis of systematic reviews and meta-analyses published in the *Annals of Surgery* during a 10-year eligibility period and determine the unambiguity of concluding statements of these reviews published in the journal. **Background:** Systematic reviews and meta-analyses integrate clinically pertinent results from several studies to replicate large-volume, 'real world' scenarios. While the assimilation of results from multiple high-quality trials are at the summit of the evidence-base, the increasing prevalence of reviews using low-to-moderate levels of evidence (LOE) limit the ability to make evidence-based conclusions. In surgery, increasing LOE are typically associated with publication in the highest impact surgical journals (e.g., *Annals of Surgery*). **Methods:** A systematic review was performed as per PRISMA guidelines. An electronic search of the *Annals of Surgery* for articles published between 2011 and 2020 was conducted. Descriptive statistics were used. **Results:** In total, 186 systematic reviews (with or without meta-analyses) were published in the *Annals of Surgery* between 2011 and 2020 (131 systematic reviews with meta-analyses (70.4%) and 55 without meta-analyses (29.6%)). Study data were from 22,656,192 subjects. In total, 94 studies were from European research institutes (50.5%) and 58 were from North American institutes (31.2%). Overall, 75.3% of studies provided conclusive statements (140/186). Year of publication (*P* = 0.969), country of publication (*P* = 0.971), region of publication (*P* = 0.416), LOE (*P* = 0.342), surgery performed (*P* = 0.736), and two-year impact factor (IF) (*P* = 0.251) failed to correlate with conclusive statements. Of note, 80.9% (106/131) of meta-analyses and 61.8% of systematic reviews (34/55) provided conclusive statements (*P* = 0.009, †). **Conclusions:** Over 75% of systematic reviews published in the *Annals of Surgery* culminated in conclusive statements. Interestingly, meta-analyses were more likely to provide conclusive statements than systematic reviews, while LOE and IF failed to do so.

**Keywords:** systematic review; meta-analysis; academic surgery

# **1. Introduction**

Synthetic reviews (i.e., systematic reviews and meta-analyses) involve a thorough interrogation of studies published by previous authors to provide a comprehensive consensus based on real-world findings in relation to a predetermined research question. The value of such studies is their ability to integrate clinically pertinent results from several studies or trials, using the robustness of larger data to inform results, outcomes, and overarching consensus. In the world of surgery, the overarching intention of such analysis is to synthesize realistic, large-volume approximations of clinical reality, which may then inform best-practice for prospective candidates requiring surgical interventions.

While the assimilation of results from multiple well-designed, high-quality trials (i.e., randomized controlled trials, or RCTs) are placed at the peak of the evidence-base [1], there has been a recent increase in the number of systematic reviews and meta-analyses

**Citation:** Davey, M.G.; Davey, M.S.; Lowery, A.J.; Kerin, M.J. What Proportion of Systematic Reviews and Meta-Analyses Published in the *Annals of Surgery* Provide Definitive Conclusions—A Systematic Review and Bibliometric Analysis. *Publications* **2022**, *10*, 1085. https://doi.org/10.3390/ publications10020019

Academic Editor: Oleg V. Mikhailov

Received: 16 February 2022 Accepted: 25 April 2022 Published: 28 April 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

being performed to integrate low-to-moderate levels of evidence (LOE), with the ambition to provide consensus from the included data [2,3]. However, higher risks of bias, lower methodological quality, and heterogeneous results impact the validity and the meaningfulness of the conclusions which may be drawn. Despite this, the publication of such studies continues to increase, which often leads to the authors of such studies being unable to report definitive results of their synthetic review. Thus, many published systematic reviews and meta-analyses have evoked scrutiny among expert members of the scientific community and has led to the growing perception that the mass production of such articles has reached 'epidemic proportions' [4]. Moreover, some consider the majority of such articles to be 'unnecessary, misleading, and/or conflicted', leading to the dissemination of redundant and uninformative data [4]. An illustrative example of this conceptualization has been captured in the recent work by Harris et al., which was published in *Arthroscopy* [5]. Following their extensive review of what the authors considered to be the six top-ranking orthopedic science journals, the authors concluded that nearly one-third of published systematic reviews and meta-analysis provided ambiguous conclusions.

The authors of the current study acknowledge the reputation of the *Annals of Surgery* and recognize the quality and importance of the data published in this top-ranking journal in the field of surgery. While Harris et al. focused solely on investigating the conclusiveness of orthopedic research, the authors of the current study sought to evaluate whether this concept was pertinent to the synthetic reviews on topics published in the *Annals of Surgery.* Accordingly, the primary aim of the current study was to assess the concluding statements of systematic reviews and meta-analyses published in the *Annals of Surgery* to assess the impact on the concluding statements on the practice of surgery. In surgery, increasing LOE is typically associated with publication in the highest impact surgical journals (i.e., *Annals of Surgery*). Therefore, our secondary aim was to determine the impact of LOE and other factors (such as country of origin, year of publication, type of review, etc.) on the concluding statements from the studies published in the journal. Our hypothesis was that at least 50% of included systematic reviews and meta-analyses would provide comprehensive conclusive statements and that LOE would correlate with the conclusive statements of studies, due to the high quality of research articles published in the *Annals of Surgery*.

# **2. Methods**

#### *2.1. Preparation and Study Criteria*

Studies meeting the following inclusion criteria were included in this analysis: (1) articles had to be published as full-text manuscripts in the *Annals of Surgery*; (2) articles had to have been published between the years 2011 and 2020; and (3) articles must have been a systematic review with or without meta-analysis. Studies meeting the following criteria were excluded from this analysis: (1) articles not published in the *Annals of Surgery* journal; (2) any study that was not a systematic review with or without meta-analysis; (3) studies published outside the determined search period; or (4) published conference proceedings or abstracts (including proceedings from the European Surgical Association and the American Surgical Association annual conferences).

This review was not prospectively registered with the international prospective register of systematic reviews (PROSPERO) as the results of this review do not have a direct link to human health.

#### *2.2. Search Strategy*

A systematic review was performed in accordance with the preferred reporting items for systematic reviews and meta-analysis (PRISMA) checklist [6]. A formal search was performed by two independent reviewers using the predefined search strategy, which was designed by the senior author (M.J.K.). The first and second authors (M.G.D. and M.S.D.) conducted a comprehensive manual electronic search of the *Annals of Surgery* journal for systematic reviews and meta-analyses published in the journal between the years 2011 and 2020. All titles published in the journal were initially screened, and all systemic reviews

(with or without meta-analysis) published in the journal were included and had their abstracts and full texts reviewed. Each reviewer read the title of each manuscript published in the *Annals of Surgery* between the years 2011 and 2020 and identified whether the studies were systematic review with or without meta-analysis. Each reviewer read the retrieved manuscripts to ensure all inclusion criteria was met, before extracting the following data: (1) first author name; (2) year of publication; (3) country; (4) region; (5) level of evidence; (6) whether it was a systematic review or meta-analysis; (7) number of included patients or participants; (8) the two-year impact factor (IF) of each systematic review; and (9) the conclusiveness of each study. In case of discrepancies in opinion between both reviewers, a third reviewer was asked to arbitrate (A.J.L.).

# *2.3. Definitions*


#### *2.4. Statistical Analyses*

Descriptive statistics were used to determine the association between the study details and conclusiveness of studies. Chi-squared (χ2) and Fisher's exact (†) tests were used, as appropriate [8]. Differences in two-year IF between conclusive and inconclusive studies were measured using independent samples *t*-test (‡). Subgroup analysis was performed based on region of publication, surgical specialties, type of study (i.e., systematic review or meta-analysis), and on LOE. All tests of significance were two-tailed, with *P* < 0.050 indicating statistical significance. Data were analyzed using SPSS™ (IBM SPSS Statistics for Mac, Version 26.0. Armonk, NY, USA) version 26.

# **3. Results**

#### *3.1. Study Characteristics*

In total, 186 systematic reviews (with or without meta-analyses) were published in the *Annals of Surgery* between the years 2011 and 2020. These studies included data in relation to an estimated 22,656,192 subjects/patients. Overall, 94 of the included studies were from European research institutes (50.5%), 58 from North American research institutes (31.2%), 18 from Asian research institutes (9.7%), and 16 from research institutes in Australia and New Zealand (8.6%). Research facilities in the United Kingdom published the most systematic reviews and meta-analyses (18.8%, 35/186), followed closely by the United States (16.1%, 30/186), and Canada (15.1%, 28/186). The mean number of published systematic reviews and meta-analyses was 18.6 per year (median 19, range: 15–25 studies). The majority of included studies were level III evidence (59.7%, 111/186), with 24.2% of included studies providing level I evidence (45/186). General and gastrointestinal surgery was the most common type of surgery with systematic reviews and meta-analyses published in the *Annals of Surgery* (55.4%, 103/186). In total, there were 131 systematic reviews with meta-analyses (70.4%) and 55 systematic reviews without meta-analyses (29.6%) included. Study characteristics from the 186 included studies are outlined in Supplementary Material Table S1.

#### *3.2. Conclusive Conclusions*

Of the 186 included systematic reviews and meta-analyses, 75.3% provided conclusive evidence in their conclusions (140/186). Year of publication (*P* = 0.969, χ2), country of publication (*P* = 0.971, χ2), region of publication (*P* = 0.416, χ2), overall LOE (*P* = 0.342, χ2), higher LOE (*P* = 0.465, †), type of surgery (*P* = 0.736, χ2), and two-year IF (*P* = 0.251, ‡) were not associated with yielding conclusive statements in their publications in the *Annals of Surgery* (Supplementary Material Table S2). Meta-analyses were more likely to yield conclusive conclusions than systematic reviews (*P* = 0.009, †).

#### *3.3. Subgroup Analyses—Region of Publication*

We performed a subgroup analysis based on region of publication of the synthetic reviews included in this study. When evaluating each region independently, the LOE failed to significantly impact the conclusiveness of studies, irrespective of region (all *P* > 0.050). Additionally, when analyzing studies published from Australia and New Zealand, 100.0% of studies performed in the fields of breast surgery, academic surgery, and gastrointestinal surgery all yielded conclusive conclusions (*P* = 0.026, χ2). For each of the other regions, surgical specialty failed to impact the conclusiveness of studies performed (all *P* > 0.050). For studies published from European surgical facilities, meta-analyses trended towards significance for being more likely to yield conclusive conclusions (*P* = 0.074, χ2). The type of study performed failed to impact the conclusiveness of studies published from other regions (Supplementary Material Table S3).

#### *3.4. Subgroup Analyses—Level of Evidence*

When performing a subgroup analysis based on the LOE, surgical specialty failed to significantly impact the conclusiveness of studies included in this systematic review (all *P* > 0.050, χ2). For studies included that were of level III evidence, meta-analyses were significantly more likely to provide conclusive conclusions compared to traditional systematic reviews (*P* = 0.016, †). Otherwise, the type of study performed failed to impact the conclusiveness of the studies (all *P* > 0.050, †). All correlations between other subgroups and LOE are outlined in Supplementary Material Table S3.

#### *3.5. Subgroup Analyses—Study Type*

In this study, the type of study (systematic review or meta-analysis) failed to influence the conclusiveness of studies based on surgical specialty (both *P* > 0.050, χ2). For systematic reviews and meta-analyses independently, all other study parameters failed to impact the conclusiveness of included studies, as outlined in Supplementary Material Table S3.

# **4. Discussion**

The most important finding in this systematic review of systematic reviews and metaanalyses published in the *Annals of Surgery* over a 10-year eligibility period is that over 75% of the 186 included studies that yielded conclusive conclusions to their articles. This result highlights the value of synthetic reviews published in the *Annals of Surgery* and supports the authors' null hypothesis suggesting that over 50% of such studies published in the journal would provide indecisive conclusive statements. These results support the journal as one that provides strong definitive conclusions on most synthetic reviews, particularly when compared to similar, previously conducted studies (e.g., Harris et al. reported one in three

studies that failed to provide definitive conclusions in their previous analysis of orthopedic literature). Interestingly, study characteristics (such as country of publication, region of publication, LOE, type of surgery, and two-year IF) failed to inform the conclusiveness of published synthetic reviews during this time period in the *Annals of Surgery*. Conversely, meta-analyses published in the journal were more likely to yield conclusive conclusions when compared to traditional systematic reviews ( *P* = 0.009, †). While the number of synthetic reviews published in the *Annals of Surgery* increased marginally during the 10-year eligibility period, the proportion of studies with conclusive conclusions remained stable (*P* = 0.969, χ2), indicating consistency of these published studies during this time period.

As previously outlined, more than three-quarters of systematic reviews and metaanalyses published in this journal provided conclusive statements to their study. This is an interesting finding, albeit one that is somewhat predictable as the *Annals of Surgery* has traditionally been renowned as the most prestigious academic journal in the field of academic surgery and is consistently ranked as the highest ranking surgical journal using both SCImago and Resurchify journal ranking metrics [9,10]. Furthermore, Agha et al. previously reported the median IF of 1.526 for the 193 surgical Thomas Reuters Journal Citation Reports (2014) [11], which is considerably lower than the current IF for the *Annals of Surgery* (IF of the journal at the time of writing is 12.969 (2021)). Therefore, it is fair to assume that synthetic reviews of higher LOE are more likely to be published in this journal, which one may intuitively expect to impact the authors' likelihood to provide conclusive statements to their review.

Of note, both LOE and the two-year IF failed to correlate with the conclusiveness of studies published in the *Annals of Surgery* in this systematic review. These are interesting findings; members of the academic community have the tendency to rely on IF as a proxy of the quality of a journal compared to competing journals in the same field [12], with the *Annals of Surgery* being considered among those at the summit of surgical journals internationally. Panesar et al. previously established that just 5.6% of studies published in four of the highest ranking surgical journals by IF (*Annals of Surgery, Archives of Surgery, British Journal of Surgery,* and *Annals of the Royal College of Surgeons*) are RCTs in design (63/1135) [13]. This implies that in the absence of well-designed RCTs being published in such journals, it is plausible that there is an overall 'dilution' of the quality of published studies, with those of moderate methodological quality being published [14]. Even in a high-ranking journal, such as the *Annals of Surgery*, this 'dilution' is somewhat evident in the results of the current study. Overall, 69.4% of included systematic reviews and meta-analyses were of level III evidence or lower (129/186), with just 21.2% of studies representing level I evidence (45/186), a finding that was surprising given the high-ranking and reputable profile of the journal.

Interestingly, meta-analyses published in the *Annals of Surgery* were significantly more likely to provide conclusions than traditional systematic reviews ( *P* = 0.009, †). Moreover, meta-analyses of level III evidence were significantly more likely to be conclusive than traditional systematic review articles ( *P* = 0.016, †). This emphasizes the value of utilizing meta-analysis methodology in providing consensus for surgical research [15]. While both systematic reviews and meta-analyses are useful in integrating data in large volumes from several clinical studies to replicate 'real world' scenarios, meta-analyses have the advantage of providing outcome measures which provide a more precise estimate of the treatment effect when compared to simple pooled analyses used to report results in conventional systematic reviews [16]. Moreover, meta-analyses also examine the degree of variability (or 'heterogeneity') of included data, while accurately establishing the impact of treatment effects, which is crucial in the field of academic surgery, in order to provide greater scientific rationalization of results yielded. This is evident from the results of the current study where 80.9% of meta-analyses versus 61.8% of systematic reviews provide conclusive statements to their analyses ( *P* = 0.009, †). Thus, this study supports the use of meta-analysis methodology where feasible in order to improve the ability to provide conclusive results to surgical research questions.

## **5. Limitations**

The current systematic review of systematic reviews and meta-analyses published in the *Annals of Surgery* suffers from several limitations. Firstly, and most importantly, the authors of this study subjectively adjudicated the conclusiveness of the published systematic reviews and meta-analyses in the journal, without applying a reliable scoring system to fairly judge 'conclusiveness'. This is due to the requirement of several methodologies used to score 'conclusiveness', requiring the analyst to have the raw data from the study. This is an obvious shortcoming of this study. Secondly, although this analysis was performed by two independent reviewers, the study design makes presumptions as to the reliability of this assessment, with no formal appraisal of intra- and inter-observer agreement. Thirdly, the failure for LOE and IF to influence the conclusiveness of studies published may be considered a blunt instrument when determining the actual clinical impact these studies may have on influencing or challenging current practice in the field of surgery. Finally, this study fails to evaluate the overall methodology or the risk of bias/quality assessments of the 186 included studies, which limits the conclusions which may successfully be drawn from the data presented in the current study.

## **6. Conclusions**

In conclusion, over 75% of systematic reviews and meta-analyses published in the *Annals of Surgery* during a 10-year period yielded conclusive conclusions statements. Study characteristics, such as country of origin, region of origin, surgical specialty, LOE, and two-year IF, failed to impact the conclusiveness of published studies. Interestingly, metaanalyses were more likely to provide conclusive statements than systematic reviews. This systematic review emphasizes the value of performing comprehensive synthetic reviews capable of providing informative data to the reader, which may be interpreted in a 'conclusive' manner to impact clinical practice.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/ 10.3390/publications9020019/s1, Table S1: List of included systematic review and meta-analysis included in this systematic review by year of publication, Table S2: Correlations between study characteristics and the conclusiveness of systematic review and meta-analyses published in the Annals of Surgery which were included in the current study, Table S3: Correlations between study characteristics based on subgroups and the conclusiveness of systematic review and meta-analyses published in the Annals of Surgery which were included in the current study. Full reference list for the included 186 studies available in S4.

**Author Contributions:** Conceptualization, M.G.D. and M.S.D.; methodology, M.G.D. and M.S.D.; software, M.G.D.; validation, M.G.D., M.S.D. and A.J.L.; formal analysis, M.G.D.; data curation, M.G.D. and M.S.D.; writing—original draft preparation, M.G.D.; writing—review and editing, M.G.D., M.S.D., A.J.L. and M.J.K.; visualization, M.G.D. and M.J.K.; supervision, M.J.K.; funding acquisition, M.G.D. All authors have read and agreed to the published version of the manuscript.

**Funding:** M.G.D. received funding from the National Breast Cancer Research Institute, Ireland.

**Data Availability Statement:** Data made available upon reasonable request from the corresponding author (M.G.D.).

**Conflicts of Interest:** None of the authors have any conflict of interest.

# **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Publications* Editorial Office E-mail: publications@mdpi.com www.mdpi.com/journal/publications

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel: +41 61 683 77 34

www.mdpi.com

ISBN 978-3-0365-4832-6