You are currently viewing a new version of our website. To view the old version click .
MDPIMDPI
Search all articles
Informatics
Download PDF
  • Article
  • Open Access

8 December 2017

Relative Quality and Popularity Evaluation of Multilingual Wikipedia Articles

,
and
Department of Information Systems, Poznań University of Economics and Business, 61-875 Poznań, Poland
*
Author to whom correspondence should be addressed.
†
Current address: al. Niepodległości 10, 61-875 Poznań, Poland
‡
These authors contributed equally to this work.
Informatics2017, 4(4), 43;https://doi.org/10.3390/informatics4040043
This article belongs to the Special Issue Quality Management in Big Data
Version Notes
Order Reprints

Abstract

Despite the fact that Wikipedia is often criticized for its poor quality, it continues to be one of the most popular knowledge bases in the world. Articles in this free encyclopedia on various topics can be created and edited in about 300 different language versions independently. Our research has showed that in language sensitive topics, the quality of information can be relatively better in the relevant language versions. However, in most cases, it is difficult for the Wikipedia readers to determine the language affiliation of the described subject. Additionally, each language edition of Wikipedia can have own rules in the manual assessing of the content’s quality. There are also differences in grading schemes between language versions: some use a 6–8 grade system to assess articles, and some are limited to 2–3. This makes automatic quality comparison of articles between various languages a challenging task, particularly if we take into account a large number of unassessed articles; some of the Wikipedia language editions have over 99% of articles without a quality grade. The paper presents the results of a relative quality and popularity assessment of over 28 million articles in 44 selected language versions. Comparative analysis of the quality and the popularity of articles in popular topics was also conducted. Additionally, the correlation between quality and popularity of Wikipedia articles of selected topics in various languages was investigated. The proposed method allows us to find articles with information of better quality that can be used to automatically enrich other language editions of Wikipedia.

1. Introduction

Sustaining accurate, complete, reliable, and up-to-date information on the Web is very important, particularly during the development of collaborative platforms and the growth of their popularity. These platforms allow Internet users to create content without special technical skills. Despite the fact that even anonymous users can participate in content addition, information in these knowledge bases can be not only abundant but also trustworthy [1].
Wikipedia is one of the best examples of such collaborative platforms. This encyclopedia became a popular source of information on different topics. Nowadays, it is the fifth most visited page in the world (https://www.alexa.com/siteinfo/wikipedia.org). The pages of this online knowledge base often appear among the first in search results using Google, Bing, Yandex, and other search engines. There are about 300 language editions on Wikipedia with over 46 million articles, which cover all subjects of human activity (https://meta.wikimedia.org/wiki/List_of_Wikipedias). The English edition is the largest and consists of over 5.4 million articles.
Despite the popularity and large volume of information in free access, Wikipedia is often criticized for unreliable content (more information about criticism of Wikipedia can be found on the page https://en.wikipedia.org/wiki/Criticism_of_Wikipedia). This is due primarily to the fact that everyone can participate in the creation and editing of the articles without proving competence or education. Changes made by users (even anonymous users) are immediately available for a wide range of Wikipedia readers. There is no professional editorial control. Articles about the same subject can be edited independently in each language version. Therefore, we can observe a difference in quality between languages depending on the described topic.
In order to help readers quickly determine the quality of the content, the Wikipedia community has defined a grading system for assessing the quality of the articles. Each language version of Wikipedia can have its own rules and standards for writing. In many language versions, there are special awards for articles of the highest quality. In English Wikipedia, these articles are labeled as “featured articles” (FAs)—they must be well-written with appropriate structure, comprehensive, well-researched with reliable sources, and present views fairly and without bias (https://en.wikipedia.org/wiki/Wikipedia:Featured_article_criteria). Another distinction—“good article” (GA)—can be awarded to an article that has not met the criteria for a FA but was close enough. These awards used in English Wikipedia often have equivalents in other language editions of Wikipedia. For example, for FA and GA awards in German Wikipedia are “exzellente artikel” and “lesenswerte artikel”, respectively. However, the share of the best articles in each Wikipedia language is relatively small—on average, around 0.3% in each language.
In some language editions of Wikipedia, there are also other quality grades, which can reflect the maturity of an article. In English Wikipedia, in addition to the highest FA and GA grades, there are also A-class, B-class, C-class, start, and stub. In Russian Wikipedia, there are additionally “solid article”, “I level”, “II level”, “III level”, and “IV level” grades. Polish Wikipedia has three additional grades: “four”, “start”, and “stub”. Equivalent classes between language versions can have differences in assessing standards. For example, in some language versions, for high grades, there is a limitation on the article’s length. Therefore, each language version can have its own quality model, even if those languages have the same number of grades.
The observation is then that there are differences between Wikipedia languages in grading schemes, and not all language versions have a developed system of quality grades for articles. For example, one of the largest versions of Wikipedia is the German edition, and it has only two highest grades—equivalent to FA and GA. Differences in quality grades do not allow us to directly compare the quality of the articles between the various language versions. An additional challenge is a large number of articles without grades. For example, in German and Polish Wikipedia, over 99% of articles are unassessed (i.e., over 2 million and 1.2 million articles, respectively).
The goal of this paper is to research the relation between the quality of Wikipedia articles and their popularity. Our hypothesis is that relative popularity is positively correlated with the relative quality of an article. We introduce a method of quality assessment of Wikipedia articles as a synthetic measure, on a scale between 0 and 100. This approach is used to evaluate more than 28 million articles in 44 language versions of Wikipedia. In addition, a comparison of quality between the articles in different languages on selected topics is conducted. The paper also presents results of the estimation of relative popularity of these articles. This makes it possible to study the association between quality and popularity in each language–topic pair.
The paper is structured as follows. Section 2 describes related work concerning both the quality and popularity of Wikipedia articles. Section 3 introduces a synthetic measure used by us to assess the quality of articles, and we present various statistics. Section 4 explains how popularity is measured. In Section 5, we study the association between quality and popularity. Section 6 presents the results of the quality and popularity assessment of Wikipedia articles in 44 languages on different topics. Section 7 concludes the paper.

3. Quality Measure

Many of the existing studies solve the problem of automatic quality assessment of articles as a classification task: articles can be marked as complete or incomplete [3,4,5,6,7,9]. This is a large limitation for comparing articles in different languages, as it is not possible to show to what degree one article is better than the other if both are tagged with the same class (e.g., incomplete). Additionally, it is necessary to take into account different standards in the quality assessment met in various language editions of Wikipedia, defined by each community.
In order to build a synthetic measure, we chose five important content-based metrics:
  • len—article length (in bytes);
  • ref—number of references;
  • img—number of images;
  • hdr—number of first- and second-level headers;
  • ral—the ratio of the number of references to the article length.
These metrics previously have showed high prediction power in quality assessment of English Wikipedia [3], as well as for other language editions of Wikipedia [4,5]. According to our findings, the above metrics are positively correlated with the quality grades [4,5,21] (see Figure 1).
Figure 1. Distribution of metrics in articles of each quality class in English Wikipedia (featured article (FA)—the highest grade; stub—the lowest). Source: own calculation.
Considering over 4 million articles with assigned quality classes in English Wikipedia, we have calculated the values of proposed metrics by quality classes. We can observe that the values of the metrics increase with an increase in the quality (stub—the lowest; FA—the highest). Table 1 presents medians of each metric of all articles in a particular quality class. As a side note, we do not take into account the A-class, because this class is usually assigned to articles that already have a FA or GA grade. We also excluded 111,412 articles that had two or more different quality grades assigned by various Wikipedia projects.
Table 1. Median of metrics in each quality class in English Wikipedia. Source: own calculations.
In addition to the above metrics, which were used in our previous work [21], we also decided to take into account special quality flaw templates, which can indicate some problems as identified by Wikipedia editors in a considered article. There are 12 types of this template in English Wikipedia, for example, verifiability, the style of writing, the structure, and neutrality [10]. We conducted a preliminary analysis of the best articles for finding quality flaw templates. It turned out that articles with a FA grade virtually did not contain important quality flaw templates. Therefore, including this additional metric is important for decreasing the quality score for articles with high values of content-based metrics and some quality problems at the same time.

3.1. Language Versions

We applied the following selection criteria for language editions of Wikipedia: (a) more than 100,000 articles and (b) editing depth value higher than 20. The latter value reflects the depth of collaborativeness, that is, how frequently articles are updated (https://meta.wikimedia.org/wiki/Wikipedia_article_depth). This descriptor is highly relevant for Wikipedia. These criteria were met by 44 language versions. The list of languages along with a number of extracted articles and redirects is presented in Table 2.
Table 2. Number of articles and redirects in considered language versions of Wikipedia.

3.2. Metrics Extraction

We used our own parser to extract the six considered metrics. This parser uses some of the files from Wikipedia dumps (a complete copy of all Wikimedia wikis, in the form of Wikitext source, raw database tables in SQL and metadata embedded in XML can be found at https://dumps.wikimedia.org/). Below is list of the files that were used by our parser for metrics extraction:
  • {lang}wiki-latest-pages-articles.xml.bz2—Recombined articles, templates, media/file descriptions, and primary meta-pages. Used for calculation of articles’ length, number of headers and references.
  • {lang}wiki-latest-imagelinks.sql.gz—Wiki media/files usage records. Used in calculation of number of images in articles.
  • {lang}wiki-latest-templatelinks.sql.gz—Wiki template inclusion link records. Used in calculation of number of quality flaw templates and for searching of articles with selected infoboxes (topics).
  • {lang}wiki-latest-redirect.sql.gz—Redirect list. Used for determining articles’ name that redirects to other articles.
  • {lang}wiki-latest-langlinks.sql.gz—Wiki interlanguage link records. Used for determining name(s) of the article in other language version(s).
In the above file names, {lang} refers to the language code of the Wikipedia edition (as described in Table 2). Thus, for each language version, we downloaded and then processed these five compressed files.
To obtain the most complete list of language links of each article, it is necessary to follow language links from each language version. For example, if an article in a given language has Wikilinks to relevant articles in other languages, one needs to check if the links are mutual. An additional challenge was to overcome redirections in language links of the articles. Summarizing, we collected about 19.3 million language link sets, and 5.6 million remained after removing duplicates. Further refining, on the basis of the similarity analysis, reduced the number of articles to 4.2 million interlanguage link sets.
In the case of counting quality flaws, we had to take into account various names of templates that pointed to specific English counterparts. For this purpose, we used interlanguage links in important quality flaw templates in English Wikipedia to obtain automatically appropriate names for these templates in other languages.
In this paper, we have used the Wikipedia dumps from September 2017.

3.3. Building Quality Measure

As described in [21], often we can observe a positive correlation between the article quality and the value of each of the five considered quality metrics (article length, number of references, images, headers, and references per length). Figure 1 shows how the distribution of articles varies depending on metrics values considered by the example from the largest English Wikipedia version, which is noticeable if we consider the same number of articles with different quality grades.
As mentioned previously, English Wikipedia is the biggest edition, has an extensive grading system, and has a large number of assessed articles. The less developed languages (e.g., Belarusian, Georgian, Serbian, and Czech) do not always behave similarly to their more developed counterparts. However, taking into account the presence of the highest FA grade in all considered language versions of Wikipedia, we could calculate the median value of these best articles in each language. Medians for each considered metric and language versions are shown in Table 3.
Table 3. Median metrics values in the highest quality class in various Wikipedia languages. Source: own calculation.
The above values were then used as thresholds in our quality measure. As proposed in [21], on the basis of the medians, we normalized each metric, in particular, the Wikipedia language version, according to the following rule: if the value of the given metric in a given language exceeded the threshold, it was set to 100 points; otherwise its value was linearly scaled to reflect the relation of the value to the median value. For example, if the median for the number of references in Japanese Wikipedia was 118, any article with a larger number of references would score 100 for this metric; an article with 59 references would score proportionally 50 points after normalizing.
Changing the value of any metric in a particular Wikipedia language version would have a different effect on the normalized value. For each language version of Wikipedia, each metric could play an important role in assessing the quality; therefore we first counted the normalized metrics average (NMA) by the following formula:
N M A = 1 c ∑ i = 1 c n m i
where n m i is a normalized metric m i and c is the number of metrics.
Next we took into account the number of quality flaw templates QFT in the considered article (if they existed) and our final formula for the quality measure reads as follows:
Q u a l i t y S c o r e = N M A − N M A ∗ 0.05 ∗ Q F T
In articles with a high quality score value (e.g., 90 points), each quality flaw template reduced the quality score by 5% (for one such template in our example, the article had 85.5 points). This way, if an article had the maximum values of a particular metric but at the same time had quality flaw template(s), this would not allow it to obtain the maximum value of the quality score (100).
After the assessing of more than 28 million articles in 44 considered language editions of Wikipedia, we found that most of the articles obtained scores of between 0 and 30 points. Figure 2 shows the distribution of articles in this scale (a more detailed and interactive chart is found on the Web page: http://data.lewoniewski.info/informatics2017/).
Figure 2. Distribution of articles depending on quality score in each language version of Wikipedia. The medians of the quality scores for each language edition are in parentheses. Source: own calculation.

4. Popularity Measure

The quality of Wikipedia articles can change over time. This is particularly true for articles that contain time-sensitive information. If they are not regularly updated or are updated with delays, their quality will decrease over time. A lower quality will be observed particularly in comparison to equivalent articles in other languages that may be updated regularly. We can expect that more-popular language versions of the article will be verified by authors more often and, if necessary, can be updated faster than less-popular language versions. To some extent, this is reflected in the Wikipedia article depth measure. Concluding, it can be useful to consider popularity metrics of the articles.
Similarly to other studies [19,20], we have used the page view information in order to measure the popularity of articles. Wikipedia records data on users visiting their pages in all language versions every hour to special compressed files (https://dumps.wikimedia.org/other/analytics/). In order to measure the popularity of articles, we downloaded these data files with statistics for the last year (from September 2016 to August 2017)—about 442 GB of compressed raw data.
We define the following popularity metrics:
  • tp—total popularity: total number of visits during the considered period;
  • sp—stable popularity: stable number of visits, which is calculated as the median of daily visits during the considered period.
In order to calculate the relative popularity, we normalized both metrics with regard to maximum values of popularity metrics in corresponding articles in other languages. Thus, for the popularity metric p of the particular article with v numbers of language versions, the language l p ∗ with the maximum value can be found by the formulas:
l t p ∗ = arg max v = 1 . . n t p ( v ) , l s p ∗ = arg max v = 1 . . n s p ( v )
Now, in order to calculate the relative popularity R P (on a scale between 0 and 100) of the selected language version l of the article, we counted using the average of the normalized popularity metrics t p and s p :
R P ( l ) = t p ( l ) t p ( l t p ∗ ) × 50 + s p ( l ) s p ( l s p ∗ ) × 50
We consider an example. We suppose we have three language versions of the article—en, de, and fr. For each language, we have the following popularity metrics:
  • total popularity t p ( e n ) = 2000 , t p ( d e ) = 1000 , and t p ( f r ) = 500 ;
  • stable popularity s p ( e n ) = 30 , s p ( d e ) = 40 , and t p ( f r ) = 20 .
English (1) has the highest value of the t p metric; therefore l t p ∗ = e n , and we normalize using the value t p ( l t p ∗ ) = 2000 : t p ∗ ( e n ) = 2000 2000 = 1 , t p ∗ ( d e ) = 1000 2000 = 0.5 , and t p ∗ ( f r ) = 500 2000 = 0.25 .
German (2) has the highest value of the s p metric; therefore we normalize using the value s p ( l s p ∗ ) = 40 for l s p ∗ = d e : s p ∗ ( e n ) = 30 40 = 0.75 , s p ∗ ( d e ) = 40 40 = 1 , and s p ∗ ( f r ) = 20 40 = 0.5 .
Now substituting the normalized values into Equation (4), we obtain the following values of the relative popularity measure for each considered language version of the article:
  • R P ( e n ) = 1 × 50 + 0.75 × 50 = 87.5 ;
  • R P ( d e ) = 0.5 × 50 + 1 × 50 = 75 ;
  • R P ( f r ) = 0.25 × 50 + 0.5 × 50 = 37.5 .

5. Wikipedia Articles’ Assessment

In this section, we present the results of the quality and popularity assessment of Wikipedia articles in 44 languages on different topics: companies, films, persons, universities, and video games.

5.1. Dataset

Wikipedia provides a system of categories, specific to each language, that allows for the grouping of articles. Thus, each language version of Wikipedia usually has its own structure of categories and own practices concerning their assignment. For example, in some languages, it is customary to tag an article with more than 20 categories; in others, the number can be limited to 2–5 categories. The quality of structure of categories also differs among languages. For example, in some language versions, articles about people, events, transport and other topics can be assigned to just one category.
A more reliable approach for classification is based on the infobox system. An infobox is a table, located usually at the top right-hand corner of an article, that concisely presents main facts about the subject. Depending on the topic described, infoboxes have different names. This allows other popular knowledge bases (e.g., DBpedia, https://dbpedia.org) to develop detailed ontology on the basis of these Wikipedia templates [22]. Popular infoboxes usually have their own names in various languages. For the purpose of our research, we have chosen 12 different infobox types on the basis of popularity in English Wikipedia. Using interwiki links, we extracted infobox names in other language versions. Table 4 shows that almost all languages of Wikipedia have equivalents of popular infoboxes in the English version.
Table 4. Number of considered language versions of Wikipedia with particular infobox. Source: own calculation.
In order to define groups of the articles that described the same topic, we extracted lists of articles separately for each infobox in a particular language version. In some languages, the lack of an infobox does not mean the absence of articles on a given topic. For example, German Wikipedia does not use infoboxes for people (office holders, musicians, etc.). Moreover, there is no obligation to add an infobox at all. However, it is often considered an important element of an article’s quality. In such cases, we can use interwiki links from identified articles in some languages to reach articles in other versions. Results of the above procedure are presented in Table 5, which presents the number of articles on a particular topic in the analyzed Wikipedia languages.
Table 5. Number of articles on particular topic in various Wikipedia languages. Source: own calculations.
Table 6 presents the results from another perspective. Here we can find out, for each topic, the number of articles that were translated to a given number of languages. As data is best interpreted using visual cues, we also present the phenomenon in Figure 3 (logarithmic scale on vertical axis).
Table 6. Number of articles that have a certain number of language versions (NoL) in particular topics. Source: own calculations.
Figure 3. Number of articles that have a certain number of language versions in particular topics. Source: own calculations.
Another possibility to analyze the data on language versions from Table 5 is to show overlaps between a group of three languages using Venn diagrams. These show how many articles specific languages have in common (see Figure 4).
Figure 4. Coverage of articles that describe universities in different languages. Source: own calculation. Other interactive Venn diagrams for this paper with different topics and languages are available on the following Web page: http://data.lewoniewski.info/informatics2017/vn/.

5.2. Quality Assessment

For all articles from our dataset, we calculated a synthetic measure of quality as described in Section 3.3. Table 7 presents the average quality scores of articles for each topic in 44 Wikipedia language editions.
Table 7. Average quality scores of articles for each topic in 44 Wikipedia language editions. Source: own calculations.
If we consider the distribution of the quality scores of Wikipedia articles, we can also observe differences across language versions and topics. Figure 5 presents the distribution of quality scores for three Wikipedia language versions (English, German, and French) in 12 considered topics (charts for other languages are available from the Web page: http://data.lewoniewski.info/informatics2017/).
Figure 5. Distribution of quality scores for three Wikipedia language versions (English, German, and French) in 12 considered topics. Source: own calculation.

5.3. Popularity Assessment

Our goal is to look for correlation between quality and popularity. Therefore, we also collected data about popularity as described in Section 4. In Table 8, we present the average popularity metric t p for articles in each topic in 44 Wikipedia language editions.
Table 8. Average popularity metric t p in articles for each topic in 44 Wikipedia language editions. Source: own calculations.

6. Association between Quality and Popularity

In this section, we present a comparison of the quality and popularity of Wikipedia articles in different languages.
As there were additional requirements for relations between languages, we have conducted the analysis on a subset of Wikipedia articles. We selected only those articles in each topic that had at least three language versions (cf. Table 6). We further analyzed combinations of a language and a topic—a pair. Table 9 presents the top 25 pairs with a share of articles, which had the highest quality in comparison to other languages (full data is presented in Table A1 in the Appendix). For example, the first row of this table should be interpreted as follows: regarding the topic “videogame”, 60.5% of articles according to our quality score were best described in the English version.
Table 9. Top 25 language–topic pairs with share of articles that have the highest quality in comparison to other languages (articles with at least three language versions were considered). Source: own calculations.
An analogous table was prepared for popularity. Table 10 presents the top 25 language–topic pairs with the share of articles that attracted the greatest popularity in comparison to other languages (full data is presented in Table A2 in the Appendix). Similarly to the previous table, the first row of this table should be interpreted as follows: regarding the topic “album”, 85.8% of articles had the English version as the most popular (attracted the greatest number of visits).
Table 10. Top 25 language versions and topics with share of articles that have the highest popularity in comparison to other languages (articles with at least three language versions were considered). Source: own calculations.
The goal of our research is to analyze the association between quality and popularity. We have done this on two levels, using appropriate statistics, both parametric and non-parametric.
We first present results of a parametric test using a phi coefficient, calculated for each language–topic pair. This is a measure of association for two binary variables. Our variables were coded as follows: if an article about a specific topic in a given language was of the highest quality among all languages, then it was assigned a value of 1 (high score); otherwise, it was assigned 0 (low score). Popularity was coded similarly: if an article about a specific topic in a given language was the most popular among all languages, then it was assigned a value of 1 (high score); otherwise it was assigned 0 (low score).
Then, the phi coefficient was calculated by the following formula:
ϕ = n 11 n 00 − n 10 n 01 n 1 n 0 n 0 n 1
where n 11 is the number of articles of high quality and popularity scores, n 10 is the number of articles that have a high quality and low popularity score, n 01 is the number of articles that have a low quality and high popularity score, and n 00 is the number of articles that have low quality and popularity scores.
Depending on the language and topic, the correlation may differ significantly. Table 11 shows the top 25 language–topic pairs with the highest correlation coefficients (full data is presented in Table A3 in the Appendix).
Table 11. Top 25 language versions and topics with the highest phi coefficients between articles with the highest quality and popularity (articles with at least three language versions were considered). Source: own calculations.
The problem with the phi coefficient, a special case of Pearson’s correlation coefficient, is that the results have a high granularity and that it cannot be easily generalized. Therefore, we also set up another experiment, in which we estimated the association between the quality and popularity within a topic. For every topic, we prepared two lists of languages: one ordered by the share of articles that were of highest quality (see Table A1), and the other ordered by the share of articles that were the most popular (see Table A2). These lists were effectively ranks. We wished to know whether the order of the languages was similar, which would support the hypothesis that quality and popularity are associated. For this purpose, we used Spearman’s rank correlation coefficient between shares of articles (also used by [19] in similar tasks). The results are presented in Table 12.
Table 12. Spearman’s rank correlation coefficients for shares of the articles of the highest quality and popularity, on various topics. Source: own calculation using [23].
Spearman’s rank correlation assesses the strength of a link between two sets of considered data, which in our case reached 0.87 (for the topic “company”). The results show that depending on the topic, we could find a different correlation between quality and popularity, but a coefficient of no less then 0.61 (for the topic “settlement”). All associations were statistically significant (as shown by p-values). Overall, the results of our calculations supported the hypothesis that there is an association between high quality of articles and their popularity. However, the association strength depends on the topic and the language version of Wikipedia.

7. Conclusions and Future Work

In this paper, we have described how the quality and popularity of Wikipedia articles can be measured across different languages. Depending on the topic and language, different correlations can be observed between the quality and popularity of Wikipedia articles. This can be due to several reasons.
First, there are differences between Wikipedia language communities in terms of the number of experts in each area. In less-developed language versions of Wikipedia, there are a small number of experts (or even an absence of) in some topics. This can be observed particularly in domains not specific for a given language community. Therefore, for any created article, there is a greater chance to award the highest grade through an assessment procedure than would be possible in a more-developed language version. More-developed versions, having a larger user base, are more demanding and hence more critical. It is more difficult to obtain a high-quality grade when there are more eyes watching.
Second, quality can evolve over time. We suppose that a non-popular article once received a high grade from the community in a less-developed language version. Because it is not popular, there is no incentive to update this article. The same subject in another language can be much more popular, and therefore facts can be updated regularly. In the less-developed language versions, we observe a discrepancy between the “graded” quality and the real quality. The factor that can help in this distinction is popularity.
Third, a large number of unassessed articles make it difficult to build accurate quality models on the basis of awards provided by users. Except for English and French Wikipedia, most of the language versions have a large number of unrated articles. In such models, important metrics are often related to the volume of information (e.g., articles’ length, number of references, and images). Unfortunately, these metrics cannot measure other quality dimensions of the article content, such as, for example, timeliness.
The approach for quality assessment presented in this paper takes into account the specifics of the best articles of each language version of Wikipedia. By considering the popularity measure, we can improve the process of identification of language versions with the highest quality.
The proposed quality and popularity measures can be helpful particularly in automated knowledge extraction from Wikipedia articles. One of these such solutions is DBpedia. The problem that is often encountered is a conflict resolution, which is necessary when various language versions concerning the same subject have conflicting information [24]. Our quality metrics can help in building more-effective conflict-resolution strategies for data fusing. An example of such conflict in DBpedia is presented in Figure 6.
Figure 6. Infobox about Basel with its data sources and its extraction to DBpedia from different Wikipedia language versions.
Conflict resolution is a first step towards the overall objective of enriching less-developed Wikipedia language versions, for which the appropriate information is of poor quality or is even absent. Figure 7 shows the general scheme of enrichment of information by transferring values from infoboxes of language versions with the highest quality and popularity scores. Before transferring values of particular parameters of an infobox, the information is compared to other language versions, and versions with higher quality and popularity scores will have a higher influence (weight) on selecting the proper value.
Figure 7. Scheme of information enrichment of Wikipedia infobox on the basis of quality and popularity assessment of other language versions by an example of Basel city. Source: own calculations from September 2017.
The methods proposed in the paper have practical implications. The synthetic quality measure is used in the WikiRank service (http://wikirank.net), which assesses and compare articles in the various language versions of Wikipedia. A quality and popularity assessment of an article can help to evaluate the quality of its important part—the infobox. Such evaluation is used in the Infoboxes service (http://infoboxes.net).
Some of the presented metrics can be expanded. For example, by analyzing the similarity of sources in Wikipedia articles across languages, we can also evaluate the quality of their content [25]. Furthermore, the references themselves can have their own quality metrics (e.g., impact factor), which can be used as an indirect indicator of the article’s quality. For popularity measurementsm it can be useful to add some metrics related to link analysis in Wikipedia articles [26]. In the future, we plan to continue studies on new metrics and their extraction methods for improving the Wikipedia article quality assessment model.

Author Contributions

K.W. and W.L. conceived the research problem; W.L. conducted state of the art analysis; K.W. proposed research methodology and designed the experiments, starting from hypotheses to be verified statistically; W.L. collected data and performed the analysis; W.L. and K.W. interpreted the results; W.A. provided an overall guidance.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Appendix A.1

Table
Table A1. Shares of Wikipedia articles with the highest quality score compared with other language versions (articles with at least three language versions were considered). Source: own calculations.
Table A1. Shares of Wikipedia articles with the highest quality score compared with other language versions (articles with at least three language versions were considered). Source: own calculations.
Lang.AlbumComp.FilmFootb.Music.Offic.PersonSettl.TaxoboxTelev.Univ.Videog.
ar0.00.0020.0010.0080.0010.0030.0070.0010.0040.0010.0070.004
az0.00.00.0010.0010.0010.0040.0010.0060.0010.00.0040.0
be0.00.00.00.0030.00.0010.00.00.00.00.00.0
bg0.0030.0010.0060.0030.0040.0030.0040.010.030.0020.0050.0
ca0.0010.0030.030.0120.0040.0040.040.0050.0330.0040.0040.008
cs0.0030.0050.0020.0160.0090.0020.0060.0030.0060.0020.0020.002
da0.0010.0020.0010.0030.0020.0010.00.00.00.0010.00.0
de0.0320.1330.1470.0910.0350.0130.0380.0150.080.0360.0710.047
el0.0050.0020.0030.0050.0040.0060.0060.0020.0020.0020.0020.002
en0.5550.4970.4370.2910.490.3930.3870.2120.2710.4780.4350.605
es0.0580.0270.0170.0530.0570.0620.0620.0460.1390.0730.0390.025
et0.0010.0040.0010.0020.0030.0070.0040.0030.0030.0010.0040.0
fa0.0010.0020.0020.0030.0020.0080.0030.0120.00.0020.0050.001
fi0.0180.0130.0140.0070.0120.0090.0120.0050.0090.0050.0060.006
fr0.0320.050.060.0520.0670.0650.0770.1150.0260.0380.060.051
gl0.00.0010.00.00.0010.0010.0010.0010.0020.00.0010.0
he0.0020.0060.0050.0030.0080.0140.0110.0020.0010.0060.0090.001
hi0.0010.0020.0060.00.0010.0020.0010.0020.00.0020.0020.0
hr0.0080.0050.0070.0070.0080.010.0070.0160.0020.0040.0030.001
hu0.0110.0060.0080.0150.010.0130.010.0150.0090.0110.0030.005
hy0.00.00.0010.00.0010.0020.0010.0190.00.00.00.0
id0.0040.0040.0070.0020.0050.0050.0030.0010.0020.0070.0070.001
it0.0630.030.0660.1810.0510.0590.10.0310.0420.0570.0140.027
ja0.0110.0460.0170.0360.0350.0190.0180.0040.0060.0430.0470.053
ka0.010.0010.0010.0010.0010.0020.0010.0030.010.00.0030.001
ko0.0030.0070.0010.0030.0050.0050.0030.0010.0020.0110.0130.004
lt0.0010.0020.0010.0030.0030.0060.0020.0040.0010.00.0010.0
no0.0060.0080.0110.0110.0120.0140.0330.0110.0070.0060.0060.004
pl0.0380.0160.0180.0270.0380.060.040.0880.0340.0240.0250.015
pt0.0430.0150.0270.0240.020.0180.0190.0280.0330.0350.0130.018
ro0.0030.0030.0020.0040.0030.0040.0030.0170.0290.0020.0030.001
ru0.0340.0320.0310.0570.040.0780.040.0390.0260.0190.040.045
sh0.00.0010.0050.00.0010.0050.0020.0360.00.0010.0020.0
simple0.0020.0020.0020.0060.0040.0060.0040.0040.0010.0030.0040.002
sl0.0030.0010.0010.00.0030.0040.0030.0170.00.00.00.001
sr0.0010.0010.0010.0010.0020.0030.0020.0020.0130.0020.0010.001
ta0.00.0020.0010.00.0010.0050.0020.0020.0010.0010.0030.0
th0.0010.0010.0010.0010.0020.0030.0010.0010.0020.0020.0080.002
tr0.0020.0030.0020.0110.0050.0070.0040.0060.0020.0040.0080.002
uk0.0190.0190.0210.0410.0170.0370.020.1510.0110.0070.0230.01
ur0.00.00.00.00.00.0010.0010.0370.00.00.0020.0
uz0.00.00.00.00.00.0010.00.0010.00.00.00.0
vi0.0010.0010.0020.0010.0020.0020.0010.0170.1350.0040.0040.002
zh0.0220.0440.0320.0120.0290.0320.0190.0090.0250.1050.1090.052

Appendix A.2

Table
Table A2. Shares of Wikipedia articles with the highest popularity compared with other language versions (articles with at least three language versions were considered). Source: own calculations.
Table A2. Shares of Wikipedia articles with the highest popularity compared with other language versions (articles with at least three language versions were considered). Source: own calculations.
Lang.AlbumComp.FilmFootb.Music.Offic.PersonSettl.TaxoboxTelev.Univ.Videog.
ar0.00.0010.0020.0040.0030.0030.0060.00.0010.0010.010.0
az0.00.00.00.00.0010.0020.00.00.00.00.0040.0
be0.00.00.00.00.00.00.00.00.00.00.00.0
bg0.00.00.0010.0010.0020.0030.0010.0020.00.0010.0030.0
ca0.00.00.00.00.00.00.0010.00.00.00.00.0
cs0.00.0030.0030.0060.0060.0010.0040.0010.0010.0010.0030.0
da0.00.0010.0010.0010.0020.0030.00.00.00.0010.0010.0
de0.0050.060.0320.0410.0080.0040.0170.0080.0330.0230.0320.004
el0.00.0010.0010.0020.0010.0050.0020.0020.00.0010.0010.0
en0.8580.6630.7330.5520.6760.5440.6240.3910.7360.6440.5080.857
es0.0190.0240.020.0890.0460.0690.0650.0520.0610.0950.0510.007
et0.00.0010.00.00.0010.0020.0010.0040.00.00.0010.0
fa0.00.0010.0020.0040.0020.0060.0040.0050.00.0010.0080.0
fi0.0030.0040.0010.0010.0060.0050.0030.0020.0010.0010.0020.0
fr0.0110.0320.0430.0350.0290.0370.0490.1130.0330.0220.0510.005
gl0.00.00.00.00.00.00.00.00.00.00.00.0
he0.00.0020.0010.0020.0030.0040.0020.0030.0010.0010.0030.0
hi0.00.00.00.00.00.00.00.00.00.00.00.0
hr0.0020.0010.0010.00.0040.0040.0020.0110.00.0020.0010.0
hu0.00.0010.0010.0050.0030.0060.0040.0190.0010.0010.0030.0
hy0.00.00.00.00.00.00.00.00.00.00.0010.0
id0.00.0020.00.00.0030.0040.0010.0020.0080.00.0050.0
it0.0110.0180.0370.0350.0190.0270.0380.0240.0060.0120.0120.004
ja0.0580.0770.0350.060.060.0220.0290.0070.0140.080.0930.075
ka0.00.00.00.00.00.0020.00.00.00.00.00.0
ko0.00.0060.00.0020.0020.0040.0020.0010.0010.0060.0110.0
lt0.00.0010.00.00.0010.0020.0010.0010.00.00.0020.0
no0.00.0020.0010.0030.0030.0030.0040.0030.0010.0010.0010.0
pl0.0030.010.0060.0150.0110.0320.0180.0750.0150.0050.0160.001
pt0.0070.0120.0040.030.0140.0120.0130.0250.0080.0190.0170.002
ro0.00.0010.0010.00.0030.0050.0020.0350.0010.00.0020.0
ru0.0130.050.0510.0960.0590.1420.0840.1690.0520.0290.0760.036
sh0.00.00.00.00.00.00.00.0010.00.00.00.0
simple0.00.00.00.00.00.00.00.00.00.00.00.0
sl0.00.00.00.00.0010.0010.0010.0040.00.00.00.0
sr0.0010.00.0040.00.0030.0050.0030.0140.00.0020.0050.0
ta0.00.00.00.00.00.00.00.00.00.00.00.0
th0.00.0020.0010.0010.0020.0020.0010.0030.0020.0010.0080.0
tr0.0010.0040.0030.010.0060.0110.0050.0120.00.0060.0120.0
uk0.00.0010.00.00.0010.0020.0010.0070.0010.00.0090.0
ur0.00.00.00.00.00.00.00.00.00.00.00.0
uz0.00.00.00.00.00.00.00.00.00.00.00.0
vi0.00.0010.00.00.0020.0020.0010.0020.0070.0010.0060.0
zh0.0060.0170.0130.0020.0150.0250.010.0020.0150.0420.0420.007

Appendix A.3

Table
Table A3. Phi correlation coefficients of articles with the highest quality and popularity in selected Wikipedia languages. Source: own calculations.
Table A3. Phi correlation coefficients of articles with the highest quality and popularity in selected Wikipedia languages. Source: own calculations.
Lang.AlbumComp.FilmFootb.Music.Offic.PersonSettl.TaxoboxTelev.Univ.Videog.
ar0.3340.1990.2430.0860.3800.1770.2410.1040.2740.4570.390—
az—−0.0060.4630.1010.3500.2630.2260.0630.158−0.0050.458−0.010
be0.3040.263——0.2690.0990.1610.0280.349———
bg0.1400.3670.2230.0970.5340.4880.2440.3580.1390.6020.603—
ca0.1800.0510.0410.1860.0730.0330.1700.1670.0530.2880.1810.088
cs0.2110.4550.3310.2790.4050.4000.4060.2090.1780.5800.6400.269
da0.4270.3720.3580.2250.4190.4610.1620.2890.1820.4770.4110.405
de0.3340.5230.3380.3640.2630.3410.3090.5120.2070.5350.5930.215
el0.1960.3030.5640.2460.2380.5110.3350.0260.0660.5220.134—
en0.6360.6250.4450.4340.5760.5650.3900.4540.3630.6300.6760.589
es0.2620.4660.2950.3370.4360.5670.3150.6420.4190.5430.6830.185
et0.2070.3320.1890.1510.4610.3620.2830.2110.1570.6500.359—
fa0.2980.4020.3860.2200.5950.4550.4580.1610.1410.6550.6010.181
fi0.3250.4430.2540.2500.4560.5570.2990.3860.1660.5110.3240.176
fr0.2380.4300.2370.2630.3430.3620.3300.3700.2040.4570.6360.205
gl−0.0010.1030.065−0.0050.0240.0830.2470.0820.1020.367——
he0.3430.5040.2810.5250.4590.4100.3250.2560.1790.5860.552—
hi——0.074—−0.015−0.0060.0740.0150.200−0.0270.026—
hr0.2620.4210.2570.4020.5930.4000.3520.4090.3410.4430.456—
hu0.1030.4670.2520.4170.4250.4580.4190.5460.0930.3690.5370.094
hy0.163−0.0040.198−0.0020.2200.2450.124−0.003−0.0070.1680.341—
id0.2490.3620.1900.2120.4350.5820.3220.6560.0330.4660.705—
it0.2480.4380.1370.3440.3370.4400.3130.4660.2830.2540.5360.159
ja0.3090.6370.4550.5940.5620.4820.4920.4470.3420.5320.6140.379
ka0.052—0.2680.0200.1760.2010.1950.501−0.0010.1110.214—
ko0.2130.4540.1900.3190.3690.5670.4170.5520.1760.4260.5110.150
lt0.1330.2470.2320.0880.3530.3960.3690.5410.2350.3180.558—
no0.1490.3380.1910.0360.2530.3530.1550.1700.5360.3510.3420.227
pl0.3190.6070.2560.2920.4140.4240.3390.5240.1340.3040.7170.160
pt0.3340.4740.1790.4440.4960.4860.3670.5530.0640.6320.7070.252
ro0.1290.4450.4410.1010.4980.3700.3980.1130.4350.2590.473—
ru0.2680.4600.3920.3650.4140.4560.3300.4210.3600.4700.6070.296
sh—−0.0070.075−0.0010.0670.0270.1750.011−0.004−0.0050.495—
simple0.0980.0910.137−0.0070.049—0.0710.0770.074———
sl0.2140.4360.642−0.0050.3290.2300.3290.1130.140−0.0260.597—
sr0.1220.1890.3050.0450.3300.2510.3010.1130.0810.3980.556—
ta−0.012−0.0130.010——0.0380.0390.0330.212—0.082—
th0.1960.4400.4070.2260.6840.7620.4810.4950.1510.5440.8380.173
tr0.4510.3670.4530.3660.5070.3970.4050.5070.2900.5600.6770.163
uk0.1040.1880.0830.1100.2060.2160.2010.2040.0910.1840.391—
ur—————0.1040.0440.013————
uz————0.2360.3140.2010.0730.338———
vi0.1210.3020.2610.4200.6070.4810.3690.1420.3720.2090.7190.096
zh0.3630.4230.3780.2500.4560.6150.4100.1850.1260.5500.4620.243

References

  1. Staub, T.; Hodel, T. Wikipedia vs. Academia: An Investigation into the Role of the Internet in Education, with a Special Focus on Wikipedia. Univ. J. Educ. Res. 2016, 4, 349–354. [Google Scholar] [CrossRef]
  2. Blumenstock, J.E. Size matters: Word count as a measure of quality on wikipedia. In Proceedings of the 17th International Conference on World Wide Web, Beijing, China, 21–25 April 2008; pp. 1095–1096. [Google Scholar]
  3. Warncke-wang, M.; Cosley, D.; Riedl, J. Tell Me More: An Actionable Quality Model for Wikipedia. In Proceedings of the 9th International Symposium on Open Collaboration, Hong Kong, China, 5–7 August 2013; pp. 1–10. [Google Scholar]
  4. Węcel, K.; Lewoniewski, W. Modelling the Quality of Attributes in Wikipedia Infoboxes. In Business Information Systems Workshops; Abramowicz, W., Ed.; Lecture Notes in Business Information Processing; Springer International Publishing: Cham, Switzerland, 2015; Volume 228, pp. 308–320. [Google Scholar]
  5. Lewoniewski, W.; Węcel, K.; Abramowicz, W. Quality and Importance of Wikipedia Articles in Different Languages. In Information and Software Technologies: 22nd International Conference, ICIST 2016, Druskininkai, Lithuania, October 13-15, 2016, Proceedings; Springer International Publishing: Cham, Switzerland, 2016; pp. 613–624. [Google Scholar]
  6. Lex, E.; Voelske, M.; Errecalde, M.; Ferretti, E.; Cagnina, L.; Horn, C.; Stein, B.; Granitzer, M. Measuring the quality of web content using factual information. In Proceedings of the 2nd Joint WICOW/AIRWeb Workshop on Web Quality—WebQuality’12, Lyon, France, 16–20 April 2012; p. 7. [Google Scholar]
  7. Khairova, N.; Lewoniewski, W.; Węcel, K. Estimating the Quality of Articles in Russian Wikipedia Using the Logical-Linguistic Model of Fact Extraction. In Business Information Systems: 20th International Conference, BIS 2017, Poznan, Poland, June 28–30, 2017, Proceedings; Abramowicz, W., Ed.; Springer International Publishing: Cham, Switzerland, 2017; pp. 28–40. [Google Scholar]
  8. Lipka, N.; Stein, B. Identifying Featured Articles in Wikipedia: Writing Style Matters. In Proceedings of the 19th International Conference on World Wide Web (2010), Raleigh, NC, USA, 26–30 April 2010; pp. 1147–1148. [Google Scholar]
  9. Xu, Y.; Luo, T. Measuring article quality in Wikipedia: Lexical clue model. In Proceedings of the 2011 3rd Symposium on Web Society (SWS), Port Elizabeth, South Africa, 26–28 October 2011; pp. 141–146. [Google Scholar]
  10. Anderka, M. Analyzing and Predicting Quality Flaws in User-generated Content: The Case of Wikipedia. Ph.D. Thesis, Bauhaus-Universitaet, Weimar, Germany, 2013. [Google Scholar]
  11. Wu, G.; Harrigan, M.; Cunningham, P. Characterizing Wikipedia Pages Using Edit Network Motif Profiles. In Proceedings of the 3rd International Workshop on Search and Mining User-generated Contents, Glasgow, UK, 24–28 October 2011; pp. 45–52. [Google Scholar]
  12. Suzuki, Y.; Nakamura, S. Assessing the Quality of Wikipedia Editors Through Crowdsourcing. In Proceedings of the 25th International Conference Companion on World Wide Web, Montreal, QC, Canada, 11–15 April 2016; International World Wide Web Conferences Steering Committee: Geneva, Switzerland, 2016; pp. 1001–1006. [Google Scholar]
  13. Wilkinson, D.M.; Huberman, B.A. Cooperation and quality in wikipedia. In Proceedings of the 2007 International Symposium on Wikis WikiSym 07, Montreal, QC, Canada, 21–23 October 2007; pp. 157–164. [Google Scholar]
  14. Ingawale, M.; Dutta, A.; Roy, R.; Seetharaman, P. Network analysis of user generated content quality in Wikipedia. Online Inf. Rev. 2013, 37, 602–619. [Google Scholar] [CrossRef]
  15. Halfaker, A.; Taraborelli, D. Artificial Intelligence Service “ORES” Gives Wikipedians X-Ray Specs to See Through Bad Edits. Available online: https://blog.wikimedia.org/2015/11/30/artificial-intelligence-x-ray-specs/ (accessed on 31 October 2017).
  16. Dang, Q.V.; Ignat, C.L. Quality assessment of Wikipedia articles without feature engineering. In Proceedings of the 2016 IEEE/ACM Joint Conference on Digital Libraries (JCDL), Newark, NJ, USA, 19–23 June 2016; pp. 27–30. [Google Scholar]
  17. Dalip, D.H.; Lima, H.; Gonçalves, M.A.; Cristo, M.; Calado, P. Quality assessment of collaborative content with minimal information. In Proceedings of the IEEE/ACM Joint Conference on Digital Libraries, London, UK, 8–12 September 2014; pp. 201–210. [Google Scholar]
  18. Reinoso, A.J. Temporal and Behavioral Patterns in the Use of Wikipedia. Ph.D. Thesis, Universidad Rey Juan Carlos, Madrid, Spain, 2011. Available online: https://gsyc.urjc.es/~ajreinoso/thesis/ (accessed on 31 October 2017).
  19. Lehmann, J.; Müller-Birn, C.; Laniado, D.; Lalmas, M.; Kaltenbrunner, A. Reader preferences and behavior on wikipedia. In Proceedings of the 25th ACM Conference on Hypertext and Social Media, Santiago, Chile, 1–4 September 2014; pp. 88–97. [Google Scholar]
  20. Warncke-Wang, M.; Ranjan, V.; Terveen, L.G.; Hecht, B.J. Misalignment Between Supply and Demand of Quality Content in Peer Production Communities. In Proceedings of the ICWSM, Oxford, UK, 26–29 May 2015; pp. 493–502. [Google Scholar]
  21. Lewoniewski, W.; Węcel, K. Relative Quality Assessment of Wikipedia Articles in Different Languages Using Synthetic Measure. In Business Information Systems Workshops: BIS 2017 International Workshops, Poznań, Poland, June 28-30, 2017, Revised Papers; Abramowicz, W., Ed.; Springer International Publishing: Cham, Switzerland, 2017; pp. 282–292. [Google Scholar]
  22. Bizer, C.; Lehmann, J.; Kobilarov, G.; Auer, S.; Becker, C.; Cyganiak, R.; Hellmann, S. DBpedia-A crystallization point for the Web of Data. Web Semant. Sci. Serv. Agents World Wide Web 2009, 7, 154–165. [Google Scholar] [CrossRef]
  23. Wessa, R. Spearman Rank Correlation (v1.0.3) in Free Statistics Software (v1.2.1), Office for Research Development and Education. Available online: https://www.wessa.net/rwasp_spearman.wasp/ (accessed on 31 October 2017).
  24. Bryl, V.; Bizer, C. Learning conflict resolution strategies for cross-language wikipedia data fusion. In Proceedings of the 23rd International Conference on World Wide Web, Seoul, Korea, 7–11 April 2014; pp. 1129–1134. [Google Scholar]
  25. Lewoniewski, W.; Węcel, K.; Abramowicz, W. Analysis of References Across Wikipedia Languages. In Information and Software Technologies: 23rd International Conference, ICIST 2017, Druskininkai, Lithuania, October 12–14, 2017, Proceedings; Damaševičius, R., Mikašytė, V., Eds.; Springer International Publishing: Cham, Switzerland, 2017; pp. 561–573. [Google Scholar]
  26. Hanada, R.; Cristo, M.; Pimentel, M.D.G.C. How do metrics of link analysis correlate to quality, relevance and popularity in wikipedia? In Proceedings of the 19th Brazilian Symposium on Multimedia and the Web, Salvador, Brazil, 5–8 November 2013; pp. 105–112. [Google Scholar]

Article Metrics

Citations

Article Access Statistics

Journal Statistics
Multiple requests from the same IP address are counted as one view.
Download PDF
Requirements and Pitfalls in AAL Projects. Guide to Self-Criticism for Developers from Experience
Previous
Web-Based Scientific Exploration and Analysis of 3D Scanned Cuneiform Datasets for Collaborative Research
Next