A Bibliometric Analysis of the Scientific Output of EU Pharmacy Departments

Abstract

A bibliometric study of 25 EU pharmacy departments showed that the top two department members (in terms of the number of articles in which the two top staff members are author (or co-author) over a 14-year period from 1998 through 2012) had h-indices of 14 (mean) / 9 (median) and 12 (mean) / 8.5 (median). These were similar to values published for pharmacy department members in the USA. Global data for departments showed lower values as they were affected by the very skewed nature of the distribution with 16% of department members accounting for 76% of the department’s publications.

1. Introduction

Strategic planning and assessment are becoming more and more essential in academic pharmacy during a time of socio-economic difficulty. Measurement of outcomes can help determine how well goals are being met and—ultimately—how well (limited) resources are being used. The measurement of scientific output is an essential element of the review process. However, whilst the relevance of the output of scientists who earn a Nobel Prize is unquestionable, for the vast majority of scientists the problem remains of how to quantify the cumulative impact and relevance of an individual’s scientific research output. Agencies that control quality assurance in research output in pharmacy departments do not have standards for output although attempts are being made to propose and possibly introduce such standards [1,2,3,4]. Several methods have been used, such as the amount of grant support, the number of books, abstracts and/or articles published; the number of citations of such publications. Hirsch (2005) [5] has suggested the use of the “Hirsch” or “h” index defined as “the number of papers with a citation number >h”. Furthermore, as scientific production will vary with the number of years of activity, Hirsch [5] also proposed the use of the “m quotient” defined as: h/number of years of publishing.

These parameters have been applied to pharmacy department publications in the USA [1,2,3] but to our knowledge, although pharmacology / pharmacy journals have been evaluated [6], similar studies have not been made on pharmacy departments in the EU.

This paper reports on the use of these indices for the examination of the published output of the staff of pharmacy departments in the EU. For each department the h and m indices for the two top staff members in terms of the number of articles in which they are author (or co-author) were calculated. The use of h and m indices is generally restricted to individual scientists. Hirsch [5] suggested that the h and m indices could also be applied to groups of individuals. Thus h and m indices were calculated for departments using “address = city of origin of the department” in the Thomson-Reuters “Web of Knowledge” portal [7]. The Science Citation Index Expanded (SCI-EXPANDED) database was used, limiting the search to articles in English. Most articles (>25%) were published in the pharmacology/pharmacy area, with slightly less (>20%) published in the chemistry research area. As EU pharmacy departments are of widely ranging size (see results) “m/number of staff” was also used. There was one department per EU country. The departments chosen were members of the EU PHARMINE (“Pharmacy Education in Europe”) network [8]. It should be noted that there is no EU ranking of pharmacy departments so no selection on the basis of ranking was made. It should also be noted that in some EU countries such as Estonia, there is only one department, and that in two countries (Cyprus and Luxembourg) there is no pharmacy department. In countries with more than one department the one chosen was the one that is member of PHARMINE. Results were compared to those published for the USA [2,3].

2. Experimental Section

In the first case, for individual scientists, the numbers of published articles, citations, citations for the most cited article, h-index and m-quotient were calculated for the top two—in terms of number of articles they authored or co-authored over the period 1998–2012 (15 years)—department staff members. Articles were thoroughly reviewed to ensure that they all had the same author, address, and concerned a pharmaceutical topic.

In the second case, for departments, the address (AD) search parameters were fixed as: AD = (pharmacy SAME city). In this case, the search time span was 1994–2012 (19 years). The top 10 papers (in terms of citations) were checked as having a pharmacy department address and a “pharmaceutical” topic.

Parametric and non-parametric statistics were used. The Kolmogorov-Smirnov statistic (KS) was calculated to test for continuous, one-dimensional probability distributions [9]. Distributions were also tested for skewness (measure of lack of symmetry of distribution) and kurtosis (measure of whether the data are peaked or flat relative to a normal distribution).

Medians, means, tests of normality (Kolmogorov-Smirnov (KS) normality test, skewness, kurtosis) for articles published, citations, number of citations per article, number of citations for the most cited article, h-indices and m quotients were calculated. In some cases, variability was calculated as coefficient of variation (CV % = standard deviation of the mean / mean × 100). Transformation of data to normalize distributions was not used. Parametric correlation and linear regression were used to investigate relationships between variables. Probability was fixed at p < 0.05.

GraphPad v6.01 [10] was used to perform all statistical calculations.

3. Results and Discussion

3.1. Numbers of Published Articles, Citations, and h-Indices for Top Two Department Staff Members

Table 1. Medians, means, tests of normality (Kolmogorov-Smirnov (KS) normality test, skewness, kurtosis) for numbers of articles published, total number of citations, h-index, number of citations per article, and number of citations for most cited article for top two department members (in terms of number of articles authored or co-authored) (1st author/co-author: top panel, 2nd author/co-author: bottom panel) of EU pharmacy departments.

1st author/co-author
	Numbers of articles published	Total number of citations	h-index	Number of citations per article	Number of citations for most cited article
Number of values	25†	25	25	25	25
25% Percentile	17	116	5.5	5.5	33
Median	49	249	9	10	55
75% Percentile	79	1752	23	20	157
Mean	56	971	14	13	109
Std. Deviation	46	1312	11	10	128
Std. Error of Mean	9.1	262	2.2	2.1	26
KS normality test
KS distance	0.13	0.27	0.25	0.19	0.28
p value	0.2	< 0.0001 ****	0.0004 ***	0.0196	< 0.0001 ****
Passed normality test (alpha=0.05)?	Yes	No	No	No	No
Skewness††	0.9	1.8 *	0.92 *	0.91 *	2.2 *
Kurtosis†††	0.19	3.3 *	−0.22	-0.44	5.4 *
2nd author/co-author
	Numbers of articles published	Total number of citations	h-index	Number of citations per article	Number of citations for most cited article
Number of values	24†	24	24	24	24
25% Percentile	13	87	5.3	0.063	23
Median	28	221	8.5	0.32	40
75% Percentile	70	757	18	4.6	81
Mean	45	661	12	3.3	66
Std. Deviation	39	965	10	6.9	66
Std. Error of Mean	8	197	2.1	1.4	14
KS normality test
KS distance	0.2	0.25	0.18	0.32	0.21
P value	0.0147 *	0.0003 ***	0.0355	< 0.0001 ****	0.0072 **
Passed normality test (alpha=0.05)?	No	No	No	No	No
Skewness††	0.93 *	2	1.2*	2.8 *	1.8 *
Kurtosis†††	0.078	2.7 *	0.82	7.6 *	2.9 *

^† Of the 27 EU member states, 2 (Cyprus and Luxembourg) do not have pharmacy departments ; ^†† (upper panel) Percentage points for n = 25 and alpha = 0.05: 0.71; ^††† Percentage points for n=25 and alpha = 0.05: 2.32; Probability given as: * = p < 0.05, ** = p < 0.01, *** = p < 0.001, **** = p < 0.0001, Ns: not significant. In the lower panel only 24/25 2nd author/co-authors could be identified; in 1/25 cases (from a pharmacy and medicine department) the exact affiliation of the 2nd author/co-author could not be determined.

Table 1. Medians, means, tests of normality (Kolmogorov-Smirnov (KS) normality test, skewness, kurtosis) for numbers of articles published, total number of citations, h-index, number of citations per article, and number of citations for most cited article for top two department members (in terms of number of articles authored or co-authored) (1st author/co-author: top panel, 2nd author/co-author: bottom panel) of EU pharmacy departments.

1st author/co-author
	Numbers of articles published	Total number of citations	h-index	Number of citations per article	Number of citations for most cited article
Number of values	25†	25	25	25	25
25% Percentile	17	116	5.5	5.5	33
Median	49	249	9	10	55
75% Percentile	79	1752	23	20	157
Mean	56	971	14	13	109
Std. Deviation	46	1312	11	10	128
Std. Error of Mean	9.1	262	2.2	2.1	26
KS normality test
KS distance	0.13	0.27	0.25	0.19	0.28
p value	0.2	< 0.0001 ****	0.0004 ***	0.0196	< 0.0001 ****
Passed normality test (alpha=0.05)?	Yes	No	No	No	No
Skewness††	0.9	1.8 *	0.92 *	0.91 *	2.2 *
Kurtosis†††	0.19	3.3 *	−0.22	-0.44	5.4 *
2nd author/co-author
	Numbers of articles published	Total number of citations	h-index	Number of citations per article	Number of citations for most cited article
Number of values	24†	24	24	24	24
25% Percentile	13	87	5.3	0.063	23
Median	28	221	8.5	0.32	40
75% Percentile	70	757	18	4.6	81
Mean	45	661	12	3.3	66
Std. Deviation	39	965	10	6.9	66
Std. Error of Mean	8	197	2.1	1.4	14
KS normality test
KS distance	0.2	0.25	0.18	0.32	0.21
P value	0.0147 *	0.0003 ***	0.0355	< 0.0001 ****	0.0072 **
Passed normality test (alpha=0.05)?	No	No	No	No	No
Skewness††	0.93 *	2	1.2*	2.8 *	1.8 *
Kurtosis†††	0.078	2.7 *	0.82	7.6 *	2.9 *

^† Of the 27 EU member states, 2 (Cyprus and Luxembourg) do not have pharmacy departments ; ^†† (upper panel) Percentage points for n = 25 and alpha = 0.05: 0.71; ^††† Percentage points for n=25 and alpha = 0.05: 2.32; Probability given as: * = p < 0.05, ** = p < 0.01, *** = p < 0.001, **** = p < 0.0001, Ns: not significant. In the lower panel only 24/25 2nd author/co-authors could be identified; in 1/25 cases (from a pharmacy and medicine department) the exact affiliation of the 2nd author/co-author could not be determined.

Top department scientists (in terms of number of articles authored or co-authored) had a median of 49 and a mean of 56 articles published up to 2012. The median in terms of the total number of citations amounts to 249, the mean to 971; the median in terms of the citations per article amounts to 10, the mean to 13. The median in terms of the citations for the most cited article amounts to 9, the mean to 14. The second best department scientists (in terms of number of articles authored or co-authored) had a median of 28 and a mean of 45 articles published. In 1/25 departments a 2nd author/co-author could not be clearly identified. The total number of citations had a median of 221 and a mean of 661; there were 0.3 (median) and 3.3 (mean) citations per article and 40 (median), 66 (mean) citations for the most cited article. The h-index had a median of 8.5 and a mean of 12.

The number of articles authored or co-authored by the two top department members represented 24 (median) and 34 (mean) percentage of the total published output of the department. The number of citations obtained for articles published by the two top publishers represented 26 (median) and 39 (mean) percentage of the total number of citations for the published output of the department. With a mean number of department staff of 80 (see below) this means that 2/80 = 2.5% of the staff are involved in 34% of the articles published by departments

In a subgroup of the 4 departments (staff numbers 91 ± 23) publishing the highest number of articles, the output of the top 15 (16.5%) principal publishing staff members accounted for 76±6% of the total published output of the department.

3.2. Numbers of Staff, Published Articles, Citations, h-Indices and m-Quotients for Departments

The data for the number of staff per department passed all normality tests and the median (70) and mean (80) were similar; variability was high (coefficient of variability (CV) = 69%). The number of articles published had a median of 363 and a mean of 511; this data failed the KS normality test showing skewness with values bunched to the left at low numbers of articles published, and a long right tail at high numbers. The same was true for the total number of citations (median, 3507, and mean, 6571). The number of citations per article had a median of 9 and a mean of 11; the distribution was flat with a right side tail. The number of citations for the most cited article had a similar distribution with a median of 125 and a mean of 238. Values for the h-index passed the KS test but nonetheless showed skewness with a right side tail; median h-index was 25 and mean 30.

Table 2. Medians, means, tests of normality (Kolmogorov-Smirnov (KS) normality test, skewness, kurtosis) for numbers of staff, articles published, citations, number of citations per article, number of citations for most cited article, and m quotient / number of staff of EU pharmacy departments.

	Staff	Articles	Citations	h-index	Number of citations per article	Number of citations for most cited article	m quotient / number of staff
Number of values	25†	25	25	25	25	25	25
25% Percentile	28	70	750	16	5.5	88	0.014
Median	70	363	3507	25	9	125	0.022
75% Percentile	122	797	9333	43	14	308	0.049
Mean	80	511	6571	30	11	238	0.029
Std. Deviation	55	473	9115	20	7.5	280	0.02
Std. Error of Mean	11	95	1823	4.0	1.5	56	0.004
KS normality test
KS distance	0.14	0.18	0.28	0.12	0.19	0.24	0.2
P value	> 0.10	0.0424*	< 0.0001****	> 0.10	0.0171*	0.0007***	0.0086**
Passed normality test (alpha=0.05)?	Yes	No	No	Yes	No	No	No
Skewness††	0.61	0.71*	2.2*	0.86*	1.5*	2.4*	0.84*
Kurtosis†††	−0.35	−0.48	5.5*	0.39	2.8*	6*	−0.57

^† Of the 27 EU member states, 2 (Cyprus and Luxembourg) do not have pharmacy departments; ^†† Percentage points for n = 25 and alpha = 0.05: 0.71; ^††† Percentage points for n = 25 and alpha = 0.05: 2.32; Probability given as: * = p < 0.05, ** = p < 0.01, *** = p < 0.001, **** = p < 0.0001, Ns: not significant.

Table 2. Medians, means, tests of normality (Kolmogorov-Smirnov (KS) normality test, skewness, kurtosis) for numbers of staff, articles published, citations, number of citations per article, number of citations for most cited article, and m quotient / number of staff of EU pharmacy departments.

	Staff	Articles	Citations	h-index	Number of citations per article	Number of citations for most cited article	m quotient / number of staff
Number of values	25†	25	25	25	25	25	25
25% Percentile	28	70	750	16	5.5	88	0.014
Median	70	363	3507	25	9	125	0.022
75% Percentile	122	797	9333	43	14	308	0.049
Mean	80	511	6571	30	11	238	0.029
Std. Deviation	55	473	9115	20	7.5	280	0.02
Std. Error of Mean	11	95	1823	4.0	1.5	56	0.004
KS normality test
KS distance	0.14	0.18	0.28	0.12	0.19	0.24	0.2
P value	> 0.10	0.0424*	< 0.0001****	> 0.10	0.0171*	0.0007***	0.0086**
Passed normality test (alpha=0.05)?	Yes	No	No	Yes	No	No	No
Skewness††	0.61	0.71*	2.2*	0.86*	1.5*	2.4*	0.84*
Kurtosis†††	−0.35	−0.48	5.5*	0.39	2.8*	6*	−0.57

^† Of the 27 EU member states, 2 (Cyprus and Luxembourg) do not have pharmacy departments; ^†† Percentage points for n = 25 and alpha = 0.05: 0.71; ^††† Percentage points for n = 25 and alpha = 0.05: 2.32; Probability given as: * = p < 0.05, ** = p < 0.01, *** = p < 0.001, **** = p < 0.0001, Ns: not significant.

Analysis was done for publications from 1994 through 2012 in 20 departments; in the 5 remaining departments the period was shorter. These departments were created post-1994. The m-quotients had a normal distribution with a median of 2.00 and a mean of 2.05; variability was high (CV = 60%). The variable “m / staff number” had a distribution with a right side tail; the median was 0.022 and the mean 0.029.

In some cases the median is very different from the mean (e.g., Table 1, total number of citations for the first author: 249 versus 971, a 4-fold difference), whereas in others the difference is less (e.g. Table 1, number of citations article: 10 versus 13, a 1.3-fold difference). This can be explained by the degree of deviation form normality as expressed by the KS distance: 0.27 in the case of the former, and 0.19 in the case of the latter, and by the skewness index: 1.8 versus 0.91. Other examples with low KS distance and skewness index, and hence less deviation form normality, show a similar phenomenon, e.g., for Table 1, number of articles published, the KS distance and skewness index are low and the mean and median similar.

3.3. Discussion

When the data presented here for individual scientists are compared to those published by the group of Thompson and Nahata [2,3] (Table 3) it is seen that various performance indicators are similar showing that performance is comparable in the EU and the USA.

A question can be posed as to whether the top authors/co-authors are in the biggest departments with (possibly) the greatest research resources. Although—as to be expected—there was a positive correlation between number of articles published and number of staff (r² = 0.41, p < 0.05), there were no significant correlations between number of staff and (1) number of citations, (2) number of citations per article, (3) number of citations for most cited article, and (4) h-index. Qualitative performance was therefore not significantly different between small and large departments.

Table 3. Comparison of data with that of pharmacy staff members in the USA.

	USA		EU
Reference	[3]	[2]	This work	This work
	Professor at a research-intensive pharmacy department	Pharmacy practice chairs	1st publisher	2nd publisher
Articles published / year	3.2/2.0	1.4/0.8	4.0/3.5	3.2/2.0
Citations / year	49/13	Not available	69/18	47/16
Citations / article	7.1/4.2	7.9/6.7	13/10	3.3/0.3
m-quotient	2.5/2.0	0.36/0.30	1.0/0.6	0.9/0.6

Data are given as mean/median. In reference 2 the period of analysis was 1965-2008; in reference 3 the period was 2005–2009.

Table 3. Comparison of data with that of pharmacy staff members in the USA.

	USA		EU
Reference	[3]	[2]	This work	This work
	Professor at a research-intensive pharmacy department	Pharmacy practice chairs	1st publisher	2nd publisher
Articles published / year	3.2/2.0	1.4/0.8	4.0/3.5	3.2/2.0
Citations / year	49/13	Not available	69/18	47/16
Citations / article	7.1/4.2	7.9/6.7	13/10	3.3/0.3
m-quotient	2.5/2.0	0.36/0.30	1.0/0.6	0.9/0.6

Data are given as mean/median. In reference 2 the period of analysis was 1965-2008; in reference 3 the period was 2005–2009.

The starting point for the evaluation of departments was the department address. This approach will include present and former staff members. Another way of calculating h-indices for groups has been proposed by Schubert [11] based on calculating a series of h-indices at successively higher levels. This method was used by Arencibia-Jorge, et al. [12], to evaluate the scientific performance over a short period (2001–2005) of the Cuban National Scientific Research Centre by starting with the h-indices of individuals and passing on through that of departments to that of the whole center. This has the advantage of clearly identifying individuals. Albeit such an advantage could be a disadvantage when evaluating performance over a long period and over many institutions as was done here. In such a case it is difficult to identify former staff that have left or retired from the department. The global approach used here will overcome such difficulties but will be distorted by staff members that change name, and departments and/or universities that change their name or affiliation. This approach will involve not only pharmacy departments, but also departments of clinical pharmacy, and hospital pharmacies.

The approach caused difficulties in the case of the 9/25 cases in which pharmacy is not an independent, individual department but just another department alongside others such as surgery and cardiology—all of which had the same “pharmacy and medicine” school address, with the same street address.

Calculations based on a group as defined by address and not on individuals mean that in any given department there may be staff with degrees in areas other than pharmacy (such as medicine, pharmacology and chemistry); such persons are included. Pharmacy department staff members who do their research in another department and use that address are not included.

Although the global approach to calculation of h-indices has been criticised¹° the correlation between the individual h-indices for the top scientist (in terms of number of articles authored or co-authored) and the global m quotients for their departments showed a correlation coefficient of r² = 0.72 (p < 0.001) and linear regression analysis of the same variables showed that only 2/25 were outside the 90% prediction band. This would suggest that the two parameters are related. A proviso has to be added here as there is discussion—but no agreement—as to whether linear regression analysis is robust enough to be applied to skewed data.

The size of the pharmacy departments in the EU is small (mean 80, median 70) but production is high with 511 (mean) / 363 (median) articles published over the 19 year period. Distributions are not normal with significant skew (values bunched to the left with a long right side tail) and often significant kurtosis (flattening of distributions). Thus distributions are pulled to the right by the better performing departments. The number of articles published by the two top staff members represented 24 (median) and 34 (mean) percentage of the total published output of the department. Furthermore in a subgroup of the 4 departments (mean staff numbers 91 ± 23) publishing the highest number of articles, the output of the top 15 (16%) publishing staff accounted for 76% ± 6% of the total published output of the department. This is also seen when comparing data for departments to that for department members. Based on the m-quotient / number of staff for departments the “average” department member would have an h-index for a 15 year career of 15 × 0.029 = 0.44 whereas the actual values for the top two authors/co-authors were 14 and 12, respectively. This emphasizes that the global h-index for departments calculated here includes many staff members with little or no output.

A proviso on methodology has to be added, however. In Table 1 the number of citations per article for the 1st author/co-author was 10 (median) / 13 (mean), whereas in Table 2 for departments these figures were similar (9 / 11). This can be explained by the fact that for Table 1 data were collected on the basis of number of articles, whereas in Table 2 data were collected on the basis of number of citations. Thus those with the largest number of articles are not necessarily those with the largest number of citations.

This study was performed in pharmacy using the Web of Knowledge database. Journal indicators vary among disciplines and databases [13]. Gorraiz and Schloegl [1] compared the ways in which pharmacy (and pharmacology) journals were evaluated by the Web of Science/Knowledge database and by the Scopus database [14]; they found that differences between the two databases were within a tolerable margin for most journals.

Finally, in a recent paper from Ding and co-workers at the China Pharmaceutical University in Nanjing, it was concluded that multiple bibliometric indicators showed that China and other Asian countries were lagging behind the USA and European countries in terms of the quality of their scientific output [15]. They went on to argue that this may partially explain the poor performance of research and development in the pharmaceutical industry. A corollary of this statement is that the relatively high quality of scientific output in pharmacy departments in the USA [2,3] and in the EU (this paper) may partially explain the high performance of pharmaceutical research and development in these two regions (see figures published by the European Federation of Pharmaceutical Industries and Associations [16]. Thus, coming back to the beginning of this paper, spending limited resources on EU (or American) pharmacy departments appears to be a good investment.

4. Conclusions

A bibliometric study of the two top scientists in EU pharmacy departments gave h-indices of 14 (mean)/9 (median) for the first and 12 (mean) / 8.5 (median) for the second, similar to values published from pharmacy departments in the USA. Results compared to those from other regions of the world showed that performance in the EU—as in the USA—is at a relatively high level. This may partially explain the good performance of research and development in the European and American pharmaceutical industries. Data for departments gave much lower values as they were affected by the skewed nature of the distribution with 16% of department staff involved in 76% of the department’s publications. The results showed that qualitative performance of individuals or departments is not related to the size of the department. Scientists from some small departments had a performance equal to that of those from much larger departments showing that resources were not the only determinate for performance. Finally the paper discussed the strengths and the weaknesses of the approach used here for evaluating the performance of departments compared to other published methods.