Altmetric Behaviour over a Two-Year Observation Period: A Longitudinal Cohort Study in Orthodontic Research

Abstract

Background: Alternative metrics have been proposed to estimate the impact of research on the academic and social environment. The objective of the current study was to analyze the longitudinal behavior of Altmetric resources related to online engagement in orthodontic research and to explore their correlation with citations over time. Methods: The Dimensions App was searched in December 2019 and December 2021 for published items belonging to orthodontic journals listed in the Journal Citation Reports (JCR) from 2014 to 2018. Items with an AAS (Altmetric Attention Score) equal to or greater than one were selected and screened for data related to authorship and publication. The breakdown of the different Altmeric resources was collected in 2019 and updated in 2021. Citations were retrieved from Web of Science (WOS) and Scopus at the same time interval. Results: The best performing journals were Progress in Orthodontics and the European Journal of Orthodontics at both time points, with a mean AAS per published item of 1.74 and 1.63, respectively, in 2021. The topics with the highest online engagement display a change over time, while the study design remained randomized clinical trials (RCTs) in both observations. Tweets, Facebook posts, and blogs showed a very slight increase over time, while News Outlets, patent data, and policy sources longitudinally showed a significant increase. No or poor correlation was found between altmetrics and citation except for Mendeley reader count. Conclusions: Tweets, Facebook, and Blog mentions can be considered attention trackers. News Outlets, patents, and policy sources are time dependent data. Mendeley reader count, can help to identify the article with a future citation potential.

1. Introduction

Since the first journal impact metric was presented in 1975, citations have been used as indicators of research quality. Citation metrics have been extensively used in the academic environment to assess the scientific production and performance of researchers, departments, and institutions. So far, hiring, promotion, and funding policies, as well as award selection, have been based mainly on these data [1]. Although considered a key indicator of research quality, citation-based metrics are not free from criticism. The Declaration on Research Assessment (DORA) released in San Francisco in 2012, highlighted the pitfalls of decision-making based only on JCR findings. The JCR is an annual report on the citation impact of a defined set of journals at a given moment in time, the jounal list is updated every year by Clarivariate, a private stakeholder on the basis of 24 criteria of supposed objectivity, selectivity, reliability and transparency.

The declaration that had first been released by a small group of researchers and editors, convened at the American Society for Cell Biology, has since then been endorsed by more than 21,923 individuals and approximately 2569 organizations and institutions [2].

The Journal Impact Factor (JIF) is an index designed by a private enterprise as a profit service without prior consultation of the affected research stakeholders, such as researchers and scholars. The index was designed as a tool to help librarians identify which journals to include in the repository and was not intended as an indicator of the scientific quality of research [3]. Moreover, citation-based metrics have a series of limitations as they do not evaluate the reason behind the article citation; they can be manipulated by editorial policies and self-citation practice; a long time is generally needed for an article to gather a significant number of citations; the so-called Matthew effect can alter citation behavior, attracting citations towards already top-cited cited papers.

Citation-based metrics should therefore be reconsidered, funding bodies and higher education institutions should recognize and reward aspects related to community participation, teamwork, and the emotional and practical effort that is part of the research process and usually falls on the youngest members of a research group [4]. All these valuable aspects are not reflected in the citation count. To better define the impact of research outside the boundaries of the academic context and partly address the limitations of classical citation-based metrics, new metrics should be considered.

Altmetrics allows us to track the immediate online attention of a research item thanks to a real-time assessment that displays how an article is shared on the Web. For this reason, almetrics are considered immediacy indicators that can react in the very short term to the interest around a research item, in contrast to classic citations that are highly time dependent [5].

An Overview of Altmetric

The flow of information among researchers and scholars has undergone a major change over the past few decades. In the “Social Web”, scholars can still cite a paper, but can also upload it on their website, Tweet it, bookmark it, blog it, download it, share it or post an online review. The general public can also react to scientific findings in an easy and fast way, thanks to the wide dissemination of social media platforms and their easy accessibility. Online activity surrounding a published item can be tracked and can help to define the impact of an article in a different, broader and updated way [6]. When the impact of research output on social networks is assessed, it should be considered that the general public could have limited knowledge on a large number of scientific topics. Altmetric (Altmetric LLP, London, UK) is an altmetrics aggregator that tracks the online activity about a published item on social media (Facebook, Twitter, Blogs, Google+); mainstream media (the New York Times, The Guardian); scientific media (Scientific American, New Scientist); online reference managers (Mendeley, CiteULike). A specific algorithm calculates the Altmetric Attention Score (AAS) of a published item, based on its mentions in the different Altmetric resources [7]. Additionally, Almetric generates a graphic summary of the impact of the article in which the different colors represent a different altmetric resource (Figure 1), allowing the reader to quickly assess which resources contributed the most to the global score. Altmetric has already been implemented by many publishers in the scientific field. Altmetric is a subscription service that provides all the services and tools of the application to customers. Through the Dimensions app it is possible to retrieve the AAS and other basic Altmetric data free of charge [1].

Based on AAS, the reader can filter published articles and detect which items are provoking greater online engagement [8]. According to what Elmore reported in 2018, many young researchers are already accustomed to disseminating their publications through social media platforms [9]. It should be remarked that most funding institutions, such as the US National Science Foundation, are currently expecting that the funded scientific projects have to be relevant from a scientific standpoint but also to present a social impact. According to some authors, Altmetric can not only track the online dissemination of research on social and mainstream media, but can also play a role in the definition of its social impact [10].

The potential for manipulating data to obtain better ratings is one of the key issues with Altmetric. The manipulation of metrics based on social media is a special problem because of the accessibility of these platforms and the existence of spam providers that make profits selling votes or tweets from registered users. By locating. Altmetric claims to employ a special algorithm that can identify fabricated attention patterns or spotting social media automation tools, in order to reduce the impact of such manipulation [11]. The tendency of Altmetric to favor funny or sexually explicit titles or content might be a disadvantage. According to Di Girolamo and Reynders (2017), some title qualities or having an intriguing title may also affect the chance of receiving a high AAS [12]. Because of Altmetric’s unique characteristics and the vast potential public reach of social media, the researcher may be tempted to intentionally or unintentionally misreport the results of their research in order to affect the effect it may have on social media dissemination. A reliability check can be performed manually on every published item by the individual researcher since in the Altmetric pay per use App it is possible to access the links of every social media citation contributing to the final AAS.

To our best knowledge, the longitudinal behavior of Altmetric resources has not yet been studied in any field of medicine or dentistry. The objective of the current study was to evaluate and analyze the longitudinal behavior of Altmetric resources used to track online attention to research in the field of Orthodontics. Furthermore, we wish to longitudinally explore the correlation between WOS citations, Scopus citations, and the different Altmetric resources.

2. Materials and Methods

A first search was carried out, in December 2019, in the inCites JCR database to select orthodontic journals that were included in the category of dentistry, oral surgery, and medicine of the JCR during the period from 2014 to 2018. The online interest generated by the orthodontic research outputs, was observed and tracked through the Dimensions free app https://app.dimensions.ai/discover/publication (accessed on 2 December 2019) in the Dimensions database. The search was limited to the nine journals listed in the JCR in 2018, which were the American Journal of Orthodontics & Dentofacial Orthopedics (AJODO), The Angle Orthodontist, The European Journal of Orthodontics (EJO), Progress in Orthodontics, Korean Journal of Orthodontics (KJO), Orthodontics & Craniofacial Research (OCR), Journal of Orofacial Orthopedics/Fortschritte der Kieferorthopädie, Seminars in Orthodontics, and the Australian Orthodontic Journal. The Dimension App was used to carry out the search and the following filters were applied: publication year (2018 or 2017 or 2016 or 2015 or 2014); source title (American Journal of Orthodontics & Dentofacial Orthopedics OR The European Journal of Orthodontics OR The Angle Orthodontist OR Korean Journal of Orthodontics OR Orthodontics & Craniofacial Research OR Journal of Orofacial Orthopedics/Fortschritte der Kieferorthopädie OR Progress in Orthodontics OR Seminars in Orthodontics OR the Australian Orthodontic Journal).

The search performed in December 2019 was restricted to items published from January 2014 to December 2018. A total of 4301 published items were identified by the search process and ordered by the Altmetrics Attention Score’s highest first option. A CSV file (Comma-Separated Values) with the AAS of the selected items was downloaded from the Dimension web app. Data were exported to an Excel data sheet (Microsoft Office for Mac version 16.43). Three members of the study group cross-checked the number of items retrieved with the one reported in PubMed, and items such as adverts, letters, replies, table of contents and obituaries, were excluded from the final sample, which included 3678 published items. The same members of the study group screened items with positive AAS (AAS ≥ 1) and extracted by consensus information on the following: (1) article title; (2) source title; (3) DOI; (4) time interval since publication; (5) Altmetric Attention Score; (6) number of authors and affiliations; (7) type of the affiliation of the corresponding author, i.e., university or other; and (8) country/region of origin as defined by the authors’ institutional affiliations, i.e., USA, Spain, Brazil. The contributions of the different Altmetric resources to the AAS of a selected item were registered after accessing the Altmetric data by means of the Dimensions search engine. Citation counts were gathered from the science citation index expanded of WOS (property of Clarivariate) and from Scopus (property of Elsevier BV).

For published items with an AAS greater than five, the following data were also recorded: (a) article topic, i.e., Periodontics-Orthodontics interaction, Bone anchor, or others; (b) study design, i.e., system Case report, Observational cross-sectional, or others. The categorization of the topic was mainly based on the categories proposed by other bibliometric studies published in the field of orthodontics [13]. If the study design was not reported in the title or abstract, the full text was screened and the study design was identified using the decision tree reported by Grimes and Schulz in 2002 [14]. In December 2021 a second search was performed on the Dimension Web app by the members of the research team introducing the DOI or the article title of the 3678 items included in the 2019 sample. This search allowed us to retrieve and register the updated information about their online engagement. All the data related to the included items are available and fully accessible Mendeley Data, https://data.mendeley.com/datasets/3p73knstfj/2 (accessed on 29 December 2022).

Statistical Analysis

The published items, journals, and Altmeric resources included in the search were described using descriptive statistics that employed counts and proportions. The correlation between the citations for individual articles in 2021, and the magnitude of the different Altmetric resources in 2019 was investigated using Pearson’s correlation analysis. Pearson’s correlation coefficient (r) < 0.3 was considered poor, 0.3–0.5 as low, 0.5–0.7 as moderate, 0.7–0.9 as high, and >0.9 as very high. p < 0.05 was considered statistically significant. To better understand the correlation between the value of the different Altmetric resources in 2019 and the citations in 2021 we compared in terms of WOS citations the published items that displayed in 2019 a positive value of an Altmetric resource (Tweets, Facebook posts, News Outlets, Patents, and policy sources) with the ones that got no mentions. The median number of WOS citations in the two groups was compared by means of a Mann-Whitney test for two independent random samples. Box and whisker graphs were plotted for the altmetric resources analyzed and are available in Supplementary Figure S1. All statistical analyses were performed using IBM SPSS Statistics version 25 software (IBM Corp., Armonk, NY, USA).

3. Results

According to the data presented in

Table 1. Impact factor (IF), Number of published items (N), total Altmetric AAS, Number of citations in WOS and Scopus, Mean AAS per article, and Number of articles with a positive AAS (N/AAS) and AAS range for each of the studied journals are presented. The data are segregated per year (2019 and 2021); (Journal titles are expressed according to their ISO Abbreviation).

						Citations
				AAS		WOS		SCOPUS					N/AAS Range
JOURNAL	IF 2022	N	Year	N	∆ (%)	N	∆ (%)	N	∆ (%)	AAS/Article	N/AAS (%)	∆ (%)	1	2–5	6–10	>10
Am J Orthod Dentofacial Orthop	3.0	1378	19	1155	29.96	5648	112	6335	105.7	0.84	427 (30.5)	29 (2.07)	262	121	26	18
			21	1501		11,971		13,031		1.09	456 (32.5)		272	127	34	23
Angle Orthod	3.4	676	19	716	26.96	3412	126.9	3805	125.9	1.06	182 (25.9)	37 (5.27)	110	49	17	6
			21	909		7740		8594		1.34	219 (31.2)		120	63	24	12
Eur J Orthod	2.6	471	19	650	17.85	2446	123.1	2753	113.4	1.38	186 (38.7)	9 (1.87)	92	52	28	14
			21	766		5457		5874		1.63	195 (40.5)		94	52	31	16
Orthod Craniofac Res	3.1	206	19	155	56.77	941	117.5	1078	103.3	0.75	67 (31.9)	2 (0.95)	39	17	9	2
			21	243		2047		2191		1.18	69 (32.85)		37	21	5	6
Prog Orthod	4.8	224	19	326	19.33	1194	139.2	1368	141.4	1.46	146 (65.2)	3 (1.34)	72	69	1	4
			21	389		2856		3302		1.74	149 (66.5)		68	69	6	6
J Orofac Orthop	1.7	222	19	41	109.8	610	146.2	668	134.6	0.18	28 (12.2)	12 (5.22)	23	4	1	0
			21	86		1502		1567		0.39	40 (17.4)		23	14	3	0
Semin Orthod	4.2	181	19	85	20	301	124.3	330	129.7	0.47	30 (14.2)	4 (1.89)	21	8	0	1
			21	102		675		758		0.56	34 (16)		21	10	2	1
Korean J Orthod	1.9	217	19	143	39.86	878	93.05	934	107	0.66	18 (8.5)	3 (1.42)	15	2	0	1
			21	200		1695		1933		0.92	21 (9.95)		13	7	0	1
Aust Orthod J	2.1	103	19	1	100	179	84.36	181	98.34	0.01	1 (0.9%)	1 (0.85)	1	0	0	0
			21	2		330		359		0.02	2 (1.71)		2	0	0	0
TOTAL		3678	19	3272	28.3	15,609	119.6	17,452	115.5				635	322	82	46
			21	4198		34273		37,609					650	363	105	65

after two years, the AAS of the published items increased by 28.3%, but only 98 (2.5%) new items gained a positive AAS in the same time period. The Angle Orthodontist and The Journal of Orofacial Orthopedics were the journals that showed the highest increase, with 5.27% and 5.22% of the publications that achieved positive AAS.

Only 42 more articles managed to reach an AAS greater than five from 2019 to 2021. In the same time span, the count of Classic citations in both the evaluated database (WOS and Scopus) increased between 110.70 and 114.25%. Progress in Orthodontics and EJO also presented the highest AAS per article in 2021.

Comparing the breakdown of different Altmetric data resources after two years (

Table 2. Breakdown of different Altmetric data resources of the selected journals from 2014 to 2017 (N), and the normalized contribution of each resource is expressed as percentage (%). The data are segregated per year (2019 and 2021); (Journal titles are expressed according to their ISO Abbreviation).

JOURNAL	Year		Twitter	Facebook	Mendeley	Google+	Blog	News Outlets	Policy Source	Wikipedia	Video Uploader	Patent	Peer Reviews
Am J Orthod Dentofacial Orthop	19	N	527	245	14,990	12	72	10	0	13	8	3	4
		%	58.95	27.4		1.34	8.05	1.12		1.45	0.98	0.34	0.45
	21	N	535	262	29,648	12	80	258	8	16	8	15	4
		%	44.65	21.86		1	6.67	21.53	0.66	1.33	0.66	1.25	0.33
Angle Orthod	19	N	166	89	6435	13	40	30	1	3	3	4	0
		%	47.56	25.5		3.72	11.46	8.6	0.29	0.86	0.86	1.15	0
	21	N	187	96	13,433	13	49	37	22	3	4	16	0
		%	43.79	22.48		3.04	11.47	8.66	5.15	0.7	0.93	3.74	0
Eur J Orthod	19	N	185	154	6489	10	76	1	2	5	0	0	0
		%	42.73	35.57		2.31	17.55	0.23	0.46	1.15	0	0
	21	N	228	157	13,305	10	76	2	4	10	1	53	1
		%	42.07	28.97		1.84	14.02	0.37	1	2	0.18	9.78	0.18
Orthod Craniofac Res	19	N	99	35	1870	0	15	0	0	1	1	4	0
		%	63.87	22.58		0	9.68	0	0	0.65	0.65	2.58	0
	21	N	96	43	3721	0	17	0	5	2	1	11	0
		%	48.97	21.93		0	8.67	0	2.55	1.02	0.65	5.6	0
Prog Orthod	19	N	241	168	5225	12	5	2	0	0	0	0	0
		%	56.31	39		2.8	1.17	0.47	0	0	0	0	0
	21	N	240	228	10,809	12	6	5	4	5	0	1	0
		%	47.9	45.5		2.4	1.19	0.99	0.79	0.99	0	0.19	0
J Orofac Orthop	19	N	20	14	222	1	0	1	0	1	1	0	1
		%	51.28	35.9		2.56	0	2.56	0	2.56	2.56	0	2.56
	21	N	21	15	831	1	1	1	7	4	0	4	1
		%	38.18	27.27		1.81	1.81	1.81	12.72	7.27	0	7.27	1.81
Semin Orthod	19	N	38	7	880	1	1	4	0	1	0	0	0
		%	73.08	13.46		1.92	1.92	7.69	0	1.92	0	0	0
	21	N	43	7	1777	1	1	4	1	2	0	2	0
		%	70.49	11.47		1.63	1.63	6.55	1.63	3.27	0	3.27	0
Korean J Orthod	19	N	17	4	476	0	1	14	0	1	3	2	0
		%	40.48	9.52		0	2.38	33.33	0	2.38	7.14	4.76	0
	21	N	20	4	791	0	1	20	0	4	0	2	0
		%	39.21	7.84		0	1.96	39.21	0	7.84	0	3.92	0
Aust Orthod J	19	N	0	1	31	0	0	0	0	0	0	0	0
		%	0	100		0	0	0	0	0	0	0	0
	21	N	0	2	94	0	0	0	0	0	0	0	0
		%	0	100		0	0	0	0	0	0	0	0
TOTAL	19	N	1293	717	36,618	49	210	62	3	25	16	13	5
		%	54.03	29.96		2.05	8.78	2.59	0.13	1.04	0.67	0.54	0.21
	21	N	1370	814	74,409	49	231	327	51	46	14	104	6
		%	45.48	27.02		1.62	7.66	10.85	1.69	1.52	0.46	3.45	0.19

), we can appreciate that the increase in Tweets is anecdotical in most of the included journals except for the EJO with 43 new tweets and for the Angle Orthodontist with 20 new tweets while in all journals the weight of Tweets in percentage on the total of the Altmetric resources decreased. The same behavior is displayed by Facebook posts with the exception of Progress in Orthodontics, which displayed a 60-post increase, followed by the AJODO with 17 new posts. In all journals except Progress in Orthodontics the weight of Facebook posts in percentage on the total of the Altmetric resources decreased. Google+; Video Uploader; Blogs and Peer reviews display at a global level no or a very slight fluctuation in the studied period, while other Altmetric resources presented a significant increase. News Outlets showed a five-fold increase passing from 61 to 328, but the increase was not homogeneous between the different journals and involved especially the AJODO that passed from 10 to 259 news outlets. Policy sources increased from 3 to 50, in this case, the increase was slight and homogeneous in most journals except for the Angle Orthodontist whose articles were cited 21 times in new policy sources. Only the KJO and the AJO articles were not related to any policy source at the end of the studied period. Patents increased from 13 to 104 with the EJO accounting for almost half of them (53), followed by the Angle Orthodontist with 16 patents.

Although Mendeley reader number is not taken into account for the AAS calculation, we can appreciate that the 102% increase of this item is similar to the increase displayed by classic citations. The behavior of the different altmetrics along the studied period can be visually observed in the radar graph (Figure 2) and compared with the one displayed by the classic citations and Mendeley readers number (Figure 3).

When the items are stratified by topic (

Table 3. Summary statistics for Articles Topic and Study design, number (N) of articles per topic and study design, Altmetric Attention Score (Total), AAS per Item, and citation count in Web of Science (Total) Scopus (Total) are presented. Data are segregated per year (2019 and 2021).

Topic	N		AAS		AAS/Item		Citations
							WOS		SCOPUS
	2019	2021	2019	2021	2019	2021	2019	2021	2019	2021
OHRQOL	13	15	335	348	25.8	23.20	213	486	224	531
Oral hygiene Caries and white spot prevention	12	16	129	198	10.8	12.38	129	343	139	376
Tooth movement acceleration (Vibrational, laser, corticotomy)	11	13	128	145	11.6	11.15	154	362	171	419
Class II fixed or removable functional appliances	11	12	108	121	9.8	9.31	148	331	167	366
Stability and relapse/Retention/Fixed and removable retainers	10	11	93	100	9.3	9.09	96	217	99	232
Eruption problems: impaction, canine ectopic eruption/number problems	7	9	69	87	9.9	9.67	56	141	66	147
Psychological and psychosocial aspects in patients	6	6	87	88	14.5	14.67	96	184	110	208
Periodontics-Orthodontics Interaction	6	7	57	105	9.5	15.00	47	115	61	131
Injuries and complications during treatment on both hard and soft tissues	5	5	197	250	39.4	50.00	9	38	9	40
Clear Aligners	4	5	54	70	13.5	14.00	98	311	97	343
Brackets design, friction, self-ligating	4	4	49	49	12.3	12.25	65	88	58	84
Habit and Myfunctional problems influence on orofacial structures	4	5	44	64	11	12.80	21	62	17	61
CBCT/Digital model/3D technology	4	6	39	53	9.8	8.83	58	173	67	199
Early/interceptive treatment	4	4	32	35	8	8.75	29	48	29	55
Bone anchor	4	8	32	64	8	8.00	27	155	30	171
Archwires, elastomers, resins and other materials: effectiveness, biochemistry, biology, toxicity	3	5	54	75	18	15.00	8	61	11	76
Orthopeadic treatment of Skeletal Class III	3	4	35	47	11.7	11.75	34	88	29	92
Aesthetic Soft tissues—profile evaluation, smile evaluation	3	4	27	36	9	9.00	27	57	30	67
Sleep disorders/breathing	2	3	31	41	15.5	13.67	7	32	8	32
Artificial intelligence application to orthodontic treatment and diagnosis	2	2	21	21	10.5	10.50	10	47	11	60
Vertical alterations: open bite	2	3	21	29	10.5	9.67	8	31	11	37
Bilbliometric		3		46		15.33		75		82
Orthodontic Research Methodology		2		16		8.00		68		73
Patient Compliance		2		17		8.50		38		37
Lip and Cleft Palate		2		14		7.00		37		41
Biology of tooth movement		2		26		13.00		61		68
Others	8	12	83	269	10.4	22.42	110	233	117	247
							Citations
Study Design	N		AAS		AAS/Item		WOS		SCOPUS
	2019	2021	2019	2021	2019	2021	2019	2021	2019	2021
RCTs	32	38	521	581	16.3	15.29	344	792	359	860
Systematic review	30	31	340	343	11.3	11.06	480	1052	535	1176
Observational (cross-sectional, longitudinal, cohort)	28	43	311	504	11.1	11.72	211	681	224	761
Systematic review and meta-analysis	25	29	282	352	11.3	11.73	294	836	324	900
CCTs	5	5	61	61	12.2	12.20	42	147	65	164
Meta-epidemiological study	2	2	23	24	11.5	12.00	19	46	24	48
Case series/Case-report	2	4	14	32	7	8.00	18	42	18	38
In vitro/Basic	1	6	122	237	122	39.50	1	61	1	78
Editorial	1	2	41	52	41	26.00	3	6	2	5
Bibliometric	1	3	13	46	13	15.33	3	75	2	82
Scoping Review	1	1	8	8	8	8.00	2	5	1	5
Review		6		195		32.50		139		158

), the highest AAS is displayed by articles about OHRQoL followed by Others due to the extremely high online attention attracted by the article of Carlson D.S. (2015) on the Evolving concepts of heredity and genetics in orthodontics [15].

This article, the second for the AAS magnitude in the sample, is a new entry in relation to what was observed in 2019. Its high AAS is due to the large number of news outlets in which the item was cited. The topic Injuries and complications during treatment on both hard and soft tissues own the highest AAS per published item. When the items were stratified by study design, the highest AAS was displayed by RCTs as in 2019 followed by Observational studies, in contrast to what was observed in 2019 when Systematic reviews were the second study design for AAS. In both 2019 and 2021 assessments, the highest AAS per article was displayed by the In vitro/Basic study dealing with the Effect of carbonated water manufactured by a soda carbonator on etched or sealed enamel. Consistent with what was observed in 2019, the top AAS article was on Pain control in orthodontics using a micropulse vibration device: A randomized clinical trial [16]. It should be highlighted that the AAS of the article remained the same as observed in 2019 (191). In 2021 narrative reviews, which were absent in the top tier in 2019, acquired a relevant position in terms of number (6) and AAS per item, being the second study design with the highest AAS per item in the sample. The correlation between WOS and Scopus citations in 2021 was very high (r = 0.977; p = 0.000), in agreement with the one observed in 2019 (r = 0.972; p = 0.000). To explore the predictive values of the magnitude of the different Altmetric resources on classic citations we studied the correlation between WOS citations in 2021 and the 2019 count of Tweets (r = 0.202; p = 0.000), Facebook posts (r = 0.158; p = 0.000), News Outlets (r = 0.023; p = 0.158), Policy Sources (r = 0.023; p = 0.162), and Patents (r = 0.017; p = 0.283). The correlation is significant but poor or low for all the explored resources. Although its value is not included in the AAS calculation, we studied the correlation between Mendeley reader count in 2019 and WOS citations in 2021 (r = 0.496; p = 0.000). In this case, the correlation is moderate-low and significant. When the group of publications that in 2019 presented an Altmetric resource score equal to 0 and the group with scores greater than or equal to 1 are compared for the citations gathered in WOS until December 2021 (

Table 4. Comparison for citations in WOS 2021 between the groups with Altmetric resources score equal to 0 and greater than or equal to 1 in 2019. (N) number of published items in the group, (Mean) Mean number of WOS citations in the group, (W) Non parametric Mann-Whitney Wilcoxon, significant at p < 0.05.

Comparison for Citations in WOS 2021
	Twitter 2019 = 0	Twitter 2019 ≥ 0	W	p
N	2901	777	13.124	0.000
Mean	8.438	13.343
SD	9.399	15.713
Median	6.00	9.00
	Facebook 2019 = 0	Facebook 2019 ≥ 0	W	p
N	3259	419	84,2349.0	0.000
Mean	8.776	14.995
SD	9.984	17.263
Median	6.00	11.00
	News Outlets 2019 = 0	News Outlets 2019 ≥ 0	W	p
N	3662	16	39,145.5	0.009
Mean	9.467	17.875
SD	11.231	15.248
Median	6.00	10.50
	Policy Sources 2019 = 0	Policy Sources 2019 ≥ 0	W	p
N	3675	3	8900.0	0.046
Mean	9.497	18.666
SD	11.264	7.094
Median	6.00	20.00
	Patents 2019 = 0	Patents 2019 ≥ 0	W	p
N	3672	6	14,975.0	0.088
Mean	9.485	21.5
SD	11.243	18.13
Median	6.00	20.50
	Mendeley 2019 = 0	Mendeley 2019 ≥ 0	W	p
N	2417	1261	1.9	0.00
Mean	7.691	12.81
SD	8.942	13.999
Median	5.00	9.00

), a statistically significant difference in citations is observed for all the resources explored except Patents.

According to the data gathered in 2021, a total of 55 items presented an AAS decrease during the studied period, and in most of the cases (46) this was due to a decrease in the registered number of Tweets. A decrease was also noted for Mendeley readers in 241 published items.

4. Discussion

Most studies on the top cited articles in dentistry show that at least a decade is needed for an article to express all its potential in terms of citation and in case it turns into a citation classic [17]. Older articles usually display a greater number of citations than the newest ones, despite their actual impact, whereas the impact of more recent papers is normally underrated, maybe due to their recent release, since time has been too short for citations to be accumulated [13]. Compared to the behavior of traditional citations, Altmetric stands out for its immediacy, which allows recently published items to get a significant recognition and prominence in the short term [18]. In previous studies on top-Altmetric papers, most of the published items belonged to the last five years, while none of the publications, in Citation classic studies, was published in such a recent time period [17,19,20]. This finding is in line with all previous Altmetric studies that emphasized the widespread and evident presence of recently published literature [21,22]. Although there is little data on the life-cycle of Altmetric resources, it seems realistic to expect that a published item will reach the end of its social media buzz-cycle within a year after publication [6]. To study the life cycle of Altmetric we designed this observational longitudinal study. To our best knowledge, this is the first study on the longitudinal behavior of altmetrics in the current dental and orthodontic literature at a global level. Unlike classic bibliometric indicators that only consider mentions (citations) and production (publications) of research outputs, altmetric indicators take into account a broader and more complex set of factors related to the usage, mentioning, sharing, and bookmarking of a published item. This broader analysis of research impact raises more theoretical concerns, since each indicator represents a distinct action that occurs at different times and in different settings [23].

In our sample, the increase in Tweets is anecdotical in most of the journals, being this finding in agreement with what was reported by other authors. Eysenbach, G. in 2011 highlighted that most articles receive tweets within 60 days after publication [24]. Ortega J.L. in 2018 reported that in his sample Tweets display an abrupt increase during the first 90 days and then remain constant demonstrating a short half-life [25]. In our sample, Facebook posts follow the same behavior as Tweets and can be considered as early attention trackers. It should be considered that Facebook had, in the first trimester of 2022, 2.91 billion monthly active users, which represents for almost a third of the world population [26]. As reported by Erdt et al. in 2016, despite the large number of users, it seems that research outputs are not widely disseminated on Facebook. In their review, only 7.7% of published items were shared on Facebook, significantly less than the 59.2% reported for Mendeley users or the 24.3% for Twitter [27]. It should be emphasized that some social media, such as Twitter or Facebook, are censored or blocked in some key nations, such as China or Russia [28], which could affect the number of users who can actually use these socials and, consequently, the global data from which the AAS is calculated.

Blogs display very slight fluctuation during the studied period, in agreement with what was reported by other authors who outlined how the life cycle of Blog mentions reaches its roof after 90 days (1.9%), showing the same rapid and short life cycle that tweets and Facebook posts [25]. Shema et al. in 2014 reported that the vast majority of articles cited on a blog were published during the same year of publication [29].

In our sample News Outlets, Patents, and Policy sources appeared as the Altmetric resources that increased the most over time. Erdt M. et al. 2016 in agreement with our findings reported that policy sources were one of the slower resources to accumulate, in contrast to Tweets, Facebook posts, or Blogs which were early indicators of online attention [27]. In light of our results, it seems questionable to insert such items in the AAS calculation since they did not display the immediacy that Altmetric is supposed to have and that is a key distinguishing factor when compared to classic citations.

Consistent with what Ortega proposed in 2020 the dissemination of a published item on Facebook, Twitter or mainstream media provides an overview of the influence on public opinion while the indicators such as patents or policy sources highlight the knowledge transfer from the researcher to society being possible indicators of societal impact [27]. According to our data when publications with Altmetric resources score equal to 0 are compared with those with a score greater than or equal to 1 in 2019, are compared for citations in WOS 2021, a statistically significant difference in citations is observed for most of the resources explored except Patents, a similar finding was reported by other authors in a different field of medicine [30]. This finding is interesting and highlights how Altmetric resources can indicate the tendency for an article to be cited, but not being mentioned on social media platforms is not per se an indication of a low citation tendency. As a matter of fact, in our sample, some of the WOS top cited articles received few or no mentions, and we could find poor or no correlation between future citations and mentions in Altmetric resources (Supplementary Figure S1). Our results are consistent with what was previously reported by Livas and Delli in 2018 but in disagreement with what was reported by Hassan et al. in 2023, that observed a positive correlation between citation counts and AAS, Tweets, and Mendeley readers count [18,31]. When analyzing Hassan data, it should be underlined that they included articles published only in 2018 and they included only original articles that usually get more citations than other published items that were included by other studies in the orthodontic field [18,32]. Although this is the first study to explore this type of correlation and we could not find data to contrast our findings. However, when citations and altmetrics mentions are compared with a cross-sectional approach, similar findings were reported in Emergency Medicine or Ophthalmology about the correlation between Twitter mentions and citations in Scopus (Spearman r ranging from 0.30 to 0.42) [30,33]. The findings for Facebook posts or News Outlets were also similar with poor to moderate correlation. The results appear in line with other authors who reported a weak or no correlation between Altmetric score and citations [34,35]. Altmetric may be able to gauge the societal impact of the items published in orthodontic journals and the online dissemination of research, despite its inherent limitation. It might provide a deeper understanding of the impacts of research when combined with conventional measures. It should be underlined that measuring very different aspects of the scores in the classic citations database and Altmeric do not have a direct relation.

In our sample the number of the Mendeley reader, it was the only resource that displayed a moderate and significant correlation with future citations, this result is in agreement with what was reported by other authors [36] and was highlighted previously in the orthodontic field [32]. Mendeley readers count seems to be within its limitations a useful indicator to highlight the published items with future citation potential [36].

The negative fluctuations that especially involved Twitter and Mendeley data, over the observation period, raise a concern about data quality or accuracy. Yu et al. in 2021 reported that errors in Facebook posts or Tweet attributable to the user, the platform, or the altmetric aggregator could involve 11.2 to 17.2% of the data [37]. Their findings can justify the fluctuation present in our sample and highlight how these databases can be considered reliable, although there is still room for further improvement.

Lastly, it is interesting to discuss the role of social research networks as ResearchGate or Academia.edu, both launched in 2008, have been able to attract the interest of an increasing number of users among researchers and academics over the last decade reaching respectively 20 million users and 175 million users. They offer citations count using a similar dynamic to classic citations, and they also provide alternative metrics based on mentions, reads, recommendations, downloads, or acknowledgments. Although these social networks do not track altmetrics, they allow authors to upload preprints enabling them to provide early impact evidence for their new research outputs. Social research networks can help increase the visibility of researchers, helping younger members of research teams and researchers working in countries with limited access to literature databases to draw attention to their work. Some authors reported that posting on a social research network site attracts more citations than posting on other parts of the public web [38]. Some authors suggested that social research networks are not yet reliable to track for early citation impact indicators. Like many web-gathered indicators, they can potentially be manipulated by uploading non-peer reviewed or fake documents and hence should be carefully interpreted for research evaluation [39,40].

5. Conclusions

In the Orthodontic field, the AAS and the mentions in the Altmetric resources used for its calculation display a slow increase over time when compared to the behavior of classic citations.

Tweets, Facebook and Blog mentions follow a similar pattern over time and stop increasing shortly after publication thus can be considered early attention trackers. Along the two-year observation period, News Outlets, patents data and policy sources present a significant increase over time and thus cannot be considered as early attention trackers. According to our findings, it seems questionable to account for such items in the AAS calculation since they did not display the immediacy that Altmetric is supposed to have and that is a key distinguishing factor when compared to classic citations.

All the Altmetric resources analyzed presented a poor or low correlation with future citations. Only the Number of Mendeley readers, a resource not included in the AAS calculation, presented a moderate and significant correlation with WOS citations. This correlation could help identify the article with future citation potential.