A Quantitative Comparison of Exoplanet Catalogs

Abstract

In this study, we investigated the differences between four commonly-used exoplanet catalogs (exoplanet.eu; exoplanetarchive.ipac.caltech.edu; openexoplanetcatalogue.com; exoplanets.org) using a Kolmogorov–Smirnov (KS) test. We found a relatively good agreement in terms of the planetary parameters (mass, radius, period) and stellar properties (mass, temperature, metallicity), although a more careful analysis of the overlap and unique parts of each catalog revealed some differences. We quantified the statistical impact of these differences and their potential cause. We concluded that although statistical studies are unlikely to be significantly affected by the choice of catalog, it would be desirable to have one consistent catalog accepted by the general exoplanet community as a base for exoplanet statistics and comparison with theoretical predictions.

1. Introduction

Since the detection of ‘51 Peg b’, the first exoplanet around a main sequence star [1], many more planets around other stars have been discovered. Currently, more than 3500 exoplanets have been detected in our galaxy. The diversity of these exoplanets in terms of orbital and physical properties is overwhelming. This diversity challenges planet formation and evolution theories, which were tuned originally to explain the planets in our Solar System [2,3].

Several groups took it upon themselves to label and classify the known exoplanets, and compile catalogs to provide the scientific community with a comprehensive working tool to access the data and perform statistical studies of the exoplanet sample (hereafter, exostatistics). These databases include information about the physical properties of the planets, as well as their host stars. Analysis of this information is constantly improving our understanding of planet formation mechanisms [4], protoplanetary disks [5], and planetary composition and internal structure [6,7]. At the moment, several exoplanet catalogs are available and are used by the community (see [8]).

The most widely-used exoplanet catalogs are:

• The Extrasolar Planets Encyclopaedia, www.exoplanet.eu ([9]; hereafter, EU).
• The NASA Exoplanet Archive, https://exoplanetarchive.ipac.caltech.edu ([10]; hereafter, ARCHIVE).
• The Open Exoplanet Catalogue, www.openexoplanetcatalogue.com/ (hereafter, OPEN).
• The Exoplanet Data Explorer, www.exoplanets.org ([11]; hereafter, ORG).

These catalogs include data from ground-based observations as well as space missions such as CoRoT, Kepler, and K2. The available data in these catalogs are comprehensive and include the physical properties of the host star, available information on the planetary physical properties, and the referenced confirmation paper or other mentioned source.

The different teams of each catalog use different criteria to include a planet, which are usually based on the physical properties of the planet or statistical thresholds (see

Table 1. The exoplanets catalogs inclusion criteria.

Catalog	Object Mass Criterion	Reference Criteria
EU	<$60M_J\pm 1\sigma$	Published or submitted to peer-reviewed journals or announced in conferences by professional astronomers.
ARCHIVE	<$30M_J$	Accepted, refereed paper.
OPEN	Not listed	Open-source.
ORG	<$24M_J$	Carefully vetted, peer-reviewed journal papers.

). Furthermore, each catalog has a different approach to displaying the database. For example, ARCHIVE designates a set of default parameters for each planet. This set is extracted from a single published reference to ensure internal consistency. Additional values published in other papers can only be found by viewing the pages dedicated to individual planets, where multiple sets of parameters are displayed. As a result, the ARCHIVE table provides a self-consistent set of parameters for any system, with missing values when the information is unavailable. On the other hand, EU uses a table displaying information on specific planet extracted from different sources, thus making for a more complete parameter set, though not necessarily self-consistent.

Many exostatistics papers use one of these catalogs as their source of observational data. Nevertheless, so far, the different catalogs have not been compared in terms of their possible differences and potential biases and selection effects that might affect inferred results and conclusions.

In this work, we present a simple statistical comparison between the different exoplanet catalogs. We mainly focus on the EU, ARCHIVE and OPEN catalogs. The database of the ORG catalog contains a single and reliable set of parameters for each planet. However, since it has not been updated for a couple of years now (see website and discussion in Reference [8]), we perform only a coarse comparison. As discussed in Reference [8], there are plans to restart regular updates in the near future.

2. Methods

We have downloaded lists of confirmed planets from the following four catalogs: EU, ARCHIVE, OPEN and ORG on 3 April 2018. As discussed previously, because of the different planetary mass criteria of each catalog (see Table 1), we set $10M_J$ as an upper bound for the planetary mass, to strictly exclude any potential brown dwarfs. Thus, we avoided any biases that might emerge from the different mass cutoffs the catalogs use.

The parameters we use in order to compare the catalogs are the stellar mass ($M_{*}$), surface temperature ($T_{eff}$) and metallicity ($\left[Fe/H\right]$), and planetary mass ($M_p$), radius ($R_p$) and orbital period (Period). We chose this set of six parameters because they are the fundamental parameters that are most easily available from current photometric and spectroscopic detection methods [12]. Physically, these parameters provide basic, broad information about the planetary system [6]. The process of deriving the stellar properties involves a collection of literature values for atmospheric properties (temperature, surface gravity, and metallicity) derived from different observational techniques (photometry, spectroscopy, asteroseismology, and exoplanet transits), and then fitting them to stellar isochrones (e.g., References [13,14]). The stellar properties are then used in the derivation of almost all planet properties from radial velocities (RV), transits or transit timing variation (TTV) data. Thus, a reliable estimate of these parameters is crucial for the quality of the planet properties estimate (e.g., Reference [15]).

In the framework of this analysis, we compared separately (and in combination) the planetary properties of the confirmed planets from the listed catalogs. In addition, we performed a comparison between planetary systems by examining the distributions of stellar and planetary properties of the main star and each system’s first detected planet. By doing so, we were able to find the biases of the planet properties emerging from the stellar properties. There was no sense in comparing the stellar parameters of all the confirmed planets since it is possible to unintentionally give more weight to multi-planetary systems when performing the analysis.

Table 2. The total number of listed planets with different stellar and planet properties for each exoplanet catalog.

Number of Objects	EU	ARCHIVE	OPEN	ORG
All planets	3757	3708	3524	2950
With planetary mass	1327	1276	1275	2917
With planetary radius	2807	2912	2731	2438
With planetary orbital period	3505	3564	3371	2920
With planetary mass, radius and orbital period	655	576	558	2436
All systems	2652	2693	2556	2200
With stellar mass $M_{*}$	2465	2571	2476	1929
With stellar metallicity $\left[Fe/H\right]$	2444	2581	2088	1877
With stellar surface temperature $T_{eff}$	2503	2424	2469	2162

lists the total number of confirmed planets and systems of each catalog as a function of the different stellar and planet properties. The significant variability of those numbers raised the following questions: Does the catalog with the largest number of planets include all the listed objects of the other catalogs? How different is the distribution of planets from two different catalogs?

Most of the statistical work in this analysis is based on comparing the different sets using a two-sample Kolmogorov–Smirnov test (hereafter, KS test, [16]). Broadly speaking, the p-value of the KS test indicates to what extent two samples can be considered to be drawn from the same distribution—a high p-value indicates a good agreement. It is sensitive to differences in both shape and location of the empirical distribution functions of the two samples. A KS test comparison between two catalogs would compare the distributions of all available estimates of one of the planetary properties mentioned above. If a specific object’s quantity is unavailable, we excluded the object from the comparison pertaining to this property. In cases where only lower/upper bounds were available, we set it to the listed value instead of excluding it. Thus, there were some cases in which two catalogs agreed on the planetary nature of a specific object, yet, since one of the catalogs had a missing value for some property, we excluded this planet from the test.

For each pair of catalogs, ‘A’ and ‘B’, we compiled three subsets: (1) The overlap subset, including all objects listed in both catalogs; (2) the unique ‘A’ subset, including only the planets that are unique to catalog ‘A’ and not listed in catalog ‘B’; and (3) the unique ‘B’ subset, including only the planets that are unique to catalog ‘B’ and not listed in catalog ‘A’. We then applied the KS test to compare the three subsets. We performed this analysis for each one of the six parameters separately, as well as a comparison of subsets that include information about the planetary mass, radius and orbital period. Figure 1 describes the methodology applied to compare the different catalogs.

A problem we often encountered in our analysis is the possibility of different planet names and aliases listed in each of the catalogs. The differences are typically caused by spaces, uppercase/lowercase mix-up, and numbers or unique symbols that are used in spelling the stellar and planet names. Yet, in most cases, we found a given object to have two different names in two different catalogs. To overcome this difficulty, we first cross-matched the different objects according to their stellar names using the SIMBAD database, and then identified the different planetary names that populate each system. This practice imparted us with the following unfortunate conclusion: Currently, there is no consensus on an unambiguous method to mark a specific planet and to find its aliases in a convenient way. Therefore, there is no reliable way to cross-match two planetary tables.

3. Results

In general, we found the distributions of planets and systems from the different catalogs to be quite similar. An exception was the ORG catalog, where its missing planetary mass values were derived from some theoretical mass–radius (M–R) relation. See Appendix A for the detailed and complete analysis with the quantitative results (

Table A1. p-values of the different catalogs’ KS test for various planetary properties.

Planet Property	EU-ORG	EU-ARCHIVE	ARCHIVE-ORG	EU-OPEN	OPEN-ARCHIVE	OPEN-ORG
$M_P$	<10−10	0.12	<10−10	0.30	0.99	<10−10
$R_P$	6 × 10−4	0.21	0.18	0.82	0.03	3 × 10−4
$Period$	0.32	0.37	0.96	0.99	0.74	0.73

and

Table A8. Same as Table A1, but for planets with combined information of planetary properties.

Planet Property	EU-ORG	EU-ARCHIVE	ARCHIVE-ORG	EU-OPEN	OPEN-ARCHIVE	OPEN-ORG
$M_P$	<10−10	0.73	<10−10	0.69	0.92	<10−10
$R_P$	<10−10	0.98	<10−10	0.21	0.24	<10−10
$Period$	<10−10	0.99	<10−10	0.81	0.58	<10−10

; Figure A1 and Figure A7). To infer the differences between the catalogs, we compared the derived populations of the overlap and unique subsets for two catalogs at a time, as discussed above. We found the distributions of the overlapping planets between the different catalogs to be similar. However, there were significant differences between the overlap and unique subsets. In Figure 2, a kernel density estimation (KDE) [17] for the probability density functions (PDF) of the mass, radius and orbital period for the different subsets is given. The solid blue curves represent the three overlap subsets when comparing each pair of the EU-ARCHIVE-OPEN catalogs. The brown curves represent the unique subsets, where the dashed, dashed-dotted and dotted curves correspond to the unique EU, ARCHIVE and OPEN subsets, respectively. Evidently, the overlap and unique subsets seem to have very different distributions for the three planet properties.

Table 3. The p-values from KS tests wherein the distributions of the overlap vs. unique subsets of planetary mass, radius and orbital period properties are compared. The three numbers in brackets in each cell represent the p-value according to this order: ($M_P, R_P, Period$).

	Overlap OPEN-ARCHIVE	Overlap EU-OPEN	Overlap ARCHIVE-EU
Unique EU	---	(<10−10, <10−10, <10−10)	(<10−10, <10−10, <10−10)
Unique ARCHIVE	(3 × 10−8, 0.03, 2 × 10−3)	---	(0.69, 0.01, 0.02)
Unique OPEN	(<10−10, <10−10, 10−9)	(0.034, 0.74, 4 × 10−8)	---

presents the KS tests p-values for the different comparisons between KS tests. We found a low p-value for most comparison tests.

As for the property of planetary mass, we noted that a higher density of small mass planets ($M_p~10M_{\oplus }$) populate the unique subsets. We found most of these planets to be TTV planets, detected in multi-planetary systems. Usually, the mass of a planet detected through TTV is not resolved directly, and in practice, is degenerate with the planet’s eccentric orbit [18]. As a result, we have many small TTV planets with an upper mass limit, rather than a nominal value. However, there is no uniform approach to displaying this mass limit. At times, the catalogs choose to omit the value altogether, while at other times it is displayed as an upper limit. However, in many cases, especially with the EU catalog, the mass limit is reported as a valid nominal estimate. Due to this difference in the mass property criteria, we found many small mass planets biasing the unique subset towards smaller masses.

For the planet radii and orbital period distributions, we found the EU and OPEN unique subsets to have a relative higher density of planets in radii values of $R_p~1R_J$, and periods of $Period~1000\mathrm{days}$. Examining the planets that comprise these subsets suggests the reason for this difference is probably related to the different inclusion criteria the catalogs use. Some examples are: Listed planets where the confirmation paper has used some theoretical M–R relations to infer the planetary radius (or mass), some unusual large radii planets suffering strong tidal forces due to their proximity to the parent stars (in short periods), planets with ‘strange’ transiting light curves that make the planetary detection more controversial, etc. As for the higher consistency, in terms of the overlap and unique parts of the ARCHIVE catalog, we found it to be somewhat artificial. By examining the ARCHIVE unique planets, we found over 75% of them to be K2 planets, suggesting the reason for the good agreement with the overlap subsets related to the simple fact: Most of the overlap planets are transiting planets (Kepler and K2), with similar properties as the K2 planets.

Performing a similar analysis for the subset of planets in which all three properties of planetary mass, radius and orbital period were available, we found the unique and overlap subsets were different again (Figure 3), although having slightly higher p-values (

Table 4. Same as Table 3, but for the subset of planets in which all three properties of planetary mass, radius and orbital period are available.

	Overlap OPEN-ARCHIVE	Overlap EU-OPEN	Overlap ARCHIVE-EU
Unique EU	---	(2 × 10−8, <10−10, 8 × 10−6)	(3 × 10−7, 8 × 10−7, 0.23)
Unique ARCHIVE	(3 × 10−5, <10−10, 3 × 10−11)	---	(0.02, 3 × 10−7, 8 × 10−4)
Unique OPEN	(6 × 10−5, 7 × 10−5, 6 × 10−4)	(6 × 10−3, 2 × 10−7, 10−3)	---

) than the individual property comparisons presented above.

In this case, most of the overlap planets had large masses and radii, with short periods. This in itself is a bias, caused by the combined sensitivity of the RV and transit detection methods [19]. As for the distributions of the unique subsets, we found them to be compromised by a higher number of TTV-detected planets, causing the distributions to be almost uniform in the regime of both small and large planets. Another interesting aspect we observed about these unique subsets was their relative similarity between each other, as displayed quantitative in

Table A10. p-values of the different catalogs’ KS test for the planetary MRP properties.

Catalog	Planetary Property	Overlap vs. UA	Overlap vs. UB	UA vs. UB
	$M_P$	2 × 10−8	6 × 10−3	0.49
EU-OPEN	$R_P$	<10−10	2 × 10−7	0.46
	$Period$	8 × 10−6	10−3	0.23
	$M_P$	6 × 10−5	3 × 10−5	0.34
OPEN-ARCHIVE	$R_P$	7 × 10−5	<10−10	2 × 10−3
	$Period$	6 × 10−4	3 × 10−11	0.05
	$M_P$	0.02	3 × 10−7	0.51
ARCHIVE-EU	$R_P$	3 × 10−7	8 × 10−7	0.02
	$Period$	8 × 10−4	0.23	0.05

displaying the unique subsets p-values. In spite of this relative similarity, these subsets populate different planets, causing this resemblance to be a product of the TTV biases in the catalogs planet mass inclusion criteria.

After examining the planetary systems according to the stellar properties, we again found similarity between the overlap subsets and significant differences between the unique to overlap subsets (Figure 4). We noted that the unique distributions of stellar mass and surface temperature were biased towards smaller mass and lower temperature stars (K, M-stars). Spectroscopy of small stars (especially M-stars) is challenging because of their intrinsic faintness and high activity [20]. As a result, detection of planets around these stars is purportedly more difficult and somewhat controversial, thus explaining why we found a higher relative number of these objects in the unique subsets. As for the stellar metallicity, we found it to be an unreliable property when analyzing possible biases in the catalog comparison. Although the metallicity is an important property, providing an early record of the chemical composition of the initial protoplanetary disk [21], the planetary detection methods do not rely on this property directly. The catalogs usually reported this property with high relative errors, probably linked to the imprecise derivation that is used to determine the metallicity value. Adding to this the fact that each catalog referred to different stellar survey sources, we found the highest inconsistency between catalogs to be in this parameter (when compared with the other stellar and planetary properties). Nevertheless, we still found the following trends: The unique subsets that included larger planets, especially in terms of the OPEN catalog, was biased towards higher metallicity values, as expected from previous studies [22,23]. On the other hand, small planets, especially listed in the unique ARCHIVE catalog, had a wider distribution of metallicities [24]. As before, the p-values for the different KS tests of stellar properties in the overlap vs. unique subsets are provided in

Table 5. The p-values obtained from KS tests comparing the distributions of the overlap vs. unique subsets of stellar mass, metallicity and surface temperature properties. The three numbers in brackets of each cell represent the p-value according to this order: ($M_{*},\left[Fe/H\right],T_{*}$).

	Overlap OPEN-ARCHIVE	Overlap EU-OPEN	Overlap ARCHIVE-EU
Unique EU	---	(<10−10, <10−10, 9 × 10−10)	(5 × 10−7, 0.19, 0.02)
Unique ARCHIVE	(<10−10, <10−10, <10−10)	---	(2 × 10−6, 2 × 10−5, 2 × 10−4)
Unique OPEN	(0.12, 0.02, 0.09)	(0.06, 0.04, 9 × 10−4)	---

. The seemingly improved mean p-value results of the OPEN catalog are caused by its relative smaller number of listed stellar property objects.

Although it would be reasonable to detect most biases in the exoplanets catalogs in the extreme ends of the examined distributions, the analysis we have presented used a quantitative approach to study the biases the catalogs possess. We conclude that the biases in the catalogs are caused by: Some missing or obsolete information about a planets’ properties (e.g., the system of ‘Kepler 53 b,c’ is labeled in the ARCHIVE catalog with main planetary masses of $M_P$ < $18 M_J$ and $M_P$ < $15 M_J$, for planets ‘b’, ‘c’ respectively [25], however a later paper [26] finds the masses to be much smaller $M_P~0.18 M_J$, $M_P~0.11 M_J$, respectively, as listed in both EU and OPEN); model dependent planetary information based on some theoretical assumption and not a direct measurement (e.g., ‘Kepler-446 b,c,d’: Both EU and OPEN display their mass property, however, according to the reference article [27], the given value is only a coarse estimate for the planets’ masses and expected RV semi-amplitude signatures using recent empirically-measured data); roughly estimated measurements, which is especially relevant for stellar parameters; or approximated upper limits as the nominal value (e.g., ‘Kepler-114 c’ is a TTV detected mass planet. The EU catalog displays its mass to be $M_p~40M_{\oplus }$ [26]. However, since this measurement is an upper limit, we find it to be inconsistent with the nominal measurement the ARCHIVE displays of $M_p~2.8M_{\oplus }$ [28], while the OPEN catalog does not include this planet).

4. Summary & Discussion

Our analysis suggests that, although the main exoplanet catalogs overlap significantly, which results in similar distributions for most astrophysical parameters, the small discrepancies between the subsets highlight some of the catalogs’ biases. These biases can best be seen in the extreme ends of the examined distributions of small mass, long orbital period planets or small stars (less than our sun). These biases do not only result from different numbers of confirmed planets in each catalog, but mainly from contributing factors, related to the data collection policy of each catalog, such as: The process each catalog uses to present and collect the properties of a specific planet, the decision whether to include a controversial object as a planet, or the routine maintenance each catalog team performs to its current listed planets.

Furthermore, in our analysis, we excluded planets with masses larger than $M_p>10M_J$. However the different catalogs use different mass boundaries, which also adds to their different biases. Unfortunately, most of the biases we found are due to the use of various subjective criteria in compiling and maintaining the database. Although all catalogs usually include in their database planets announced in peer-reviewed publications, this should not be the only criterion for a confirmed planet. We suggest that the explosive growth in the known planet population in recent years once again highlights the need for a more rigorous and objective mechanism to tag planets as confirmed. The differences among the catalogs demonstrate that there are conflicting views in the community regarding such criteria. The International Astronomical Union (IAU) is an objective and well-accepted authority by the community, and we therefore suggest that a central catalog could be maintained by Division F (Planetary Systems and Bio-astronomy) of the IAU, and specifically its Commission F2 (Exoplanets and the Solar Systems). Discussions within the commission should resolve the various differences and arrive at a system that can be agreed upon.

After performing this analysis and scrutinizing the different calculated biases, we can carefully make the following statements:

• The ARCHIVE catalog is the most up-to-date catalog, with recent Kepler and K2 planet discoveries. It is also the least biased catalog in terms of the interpretation of the mass upper limit, being the true value or the adoption of a model-based value instead of a genuine measurement. Another interesting feature the catalog has is a list of “removed targets” displaying objects that had been listed in the catalog but were removed, suggesting a more rigorous process applied by the ARCHIVE team.
• The EU catalog is less restrictive when listing the planetary properties, and therefore could include imprecise estimates. The EU catalog differs the most with the overlap subsets, probably due to its more permissive acceptance criteria and the use of mixed sources of information. However, it has the most wide and large coverage of planets.
• The OPEN catalog is somewhere in the middle, between ARCHIVE and EU. In some cases, we find that it resembles the EU subsets, while in others the ARCHIVE. This might not be surprising, given that this catalog is an open-source catalog which is managed and updated by the astronomical community. Although its interface is elegant and user friendly, it has its drawbacks, especially the lack of detection reference and a smaller planet population.

Finally, while each catalog suffers biases, for an exostatistics work, there should not be too much difference among the databases, since the planet population (especially the one compared in this work) is large enough to wash out the small biases and discrepancies. Nevertheless, we find the fusion of catalogs (the overlap subset) a powerful tool as a starting point for increasing the reliability of exostatistics research. A promising platform seems to be the Data & Analysis Center for Exoplanets (DACE) database (https://dace.unige.ch), which includes a linked table to commonly-used exoplanet catalogs. DACE offers an accessible option to check the properties of a specific planet listed in different catalogs, and to compare its properties as they are displayed on the catalogs.

Besides a careful and detailed inspection of each exoplanet related paper confirmation, other useful techniques that can be used to increase the confidence of some exoplanet databases is to check other related parameters such as: Discovery date and update times, which can solve issues of “catch-up” times between catalogs and the rate by which they upload new exoplanets; a measure of the velocity semi-amplitude K parameter can suggest the mass measurement is truly deduced from a RV measurement and not derived from some theoretical model; a TTV flag with reported eccentricity parameter can suggest the reported mass measurement is probably not an upper limit, but some nominal value.