The Parallel Compact Object CALculator: An Efficient General Relativistic Initial Data Solver for Compact Objects

Abstract

Every numerical general relativistic investigation starts from the solution of the initial value equations at a given time. Astrophysically relevant initial values for different systems lead to distinct sets of equations that obey specific assumptions tied to the particular problem. Therefore, a robust and efficient solver for a variety of strongly gravitating sources is needed. In this work, we present the OpenMP version of the Compact Object CALculator (COCAL) on shared memory processors. We performed extensive profiling of the core COCAL modules in order to identify bottlenecks in efficiency, which we addressed. Using modest resources, the new parallel code achieves speedups of approximately one order of magnitude relative to the original serial COCAL code, which is crucial for parameter studies of computationally expensive systems such as magnetized neutron stars, as well as its further development towards more realistic scenarios. As a novel example of our new code, we compute a binary quark system where each companion has a dimensionless spin of $0.43$ aligned with the orbital angular momentum.

1. Introduction

Gravitational wave astronomy was launched in 2015 with the first-ever gravitational wave detection of the inspiral and merger from a binary black hole system, as reported by the LIGO scientific collaboration—event GW150914 [1]. Two years later, the simultaneous detection of gravitational waves from an inspiraling binary neutron star system, event GW170817, and its postmerger emission of electromagnetic radiation spurred the era of multimessenger astronomy [2,3,4,5,6]. Merging binary neutron stars and black hole–neutron stars are not only important sources of gravitational radiation, but also promising candidates for coincident electromagnetic counterparts, which could give new insight into their sources. To understand these observations and, in particular, to understand the physics of matter under extreme conditions, it is crucial to compare them with predictions from theoretical modeling which, due to the complexity of the underlying physical phenomena, are largely numerical in nature.

Numerical modeling of strongly gravitating systems typically involves two steps: first, the ab initio calculation of the initial values that describe the astrophysical system at a given time; and second, the evolution of these initial data in order to describe it at later times. These two general relativistic calculations involve two distinct codes: one that solves the elliptic problem to find out the initial data and another that solves the hyperbolic one that simulates their evolution. For these reasons, we have developed the Compact Object CALculator (COCAL) code, which is a serial code that solves for any compact object in equilibrium or quasiequilibrim in a variety of settings and/or mathematical formulations. In particular, COCAL has been used to calculate: (i) rotating neutron stars and quark stars (axisymmetric or triaxial, uniformly or differentially rotating) [7,8,9,10,11,12]; (ii) binary black holes in quasicircular orbits [13,14,15]; (iii) binary neutron stars and quark stars (irrotational or spinning) in quasicircular orbits [16,17]; (iv) magnetized rotating neutron stars with generic mixed poloidal and toroidal magnetic fields and a magnetosphere [18,19,20]; and (v) self-gravitating, tilted black hole-disks (where the angular momentum of the disk is tilted with respect to the angular momentum of the black hole) [21]. The main characteristics of the COCAL code is the use of finite differences and a Green’s function approach, as first developed in [22] for neutron stars and in [23] for black holes, to achieve a convergent solution through a Picard type of iteration, known as the Komatsu–Eriguchi–Hachisu (KEH) method [24,25]. The field equations are solved in spherical coordinates in multiple patches, and a smooth solution is obtained everywhere through boundary surface integrals. For the fluid equations, surface-fitted coordinates are being implemented that allow for an accurate representation of the neutron star surface, which is important in order to impose boundary conditions.

Similar to COCAL, there are a number of other codes whose purposes are to solve for initial data in general relativity. For binary as well as single compact object codes, these include the TwoPunctures [26], LORENE [27], KADATH/FUKA [28,29,30], SGRID [31], Spells [32], Elliptica [33], and NRPyElliptic [34] codes. Each code uses its own numerical methods, and depending on the problem, it may use different mathematical formulations too.

Compact systems (i)–(v) employ different mathematical formulations that nevertheless share common characteristics. For example, one well-known method for the calculation of initial data [35] is the Isenberg–Wilson–Mathews (IWM) [36,37,38,39] formulation, where the spatial three-dimensional metric is assumed to be conformally flat and the trace of the extrinsic curvature is zero. This method is applied from rotating neutron/quark stars [7,11] to binary black holes [13,14,15] and binary neutron stars [16,17] and results in five elliptic equations, i.e., a truncated system of the Einstein field equations. Notwithstanding the approximate nature of the method, its robustness and accuracy (especially for neutron stars) make the IWM method a workhorse in initial data construction. On the other hand, more intricate methods have been developed that can treat the full Einstein system, both for neutron stars and black holes. In [8], rotating axisymmetric or triaxial neutron stars were computed using the waveless method [40,41], while in [18,19,20], this method was employed to calculate generic magnetized equilibria with mixed poloidal and toroidal magnetic fields as well as a force-free magnetosphere. In [42,43], the same method was used to compute binary neutron stars in quasicircular orbits. This method is referred to as the “waveless” formulation. For spacetimes that contain a black hole, a new method has been developed in [21] that solves the complete initial data equations. Contrary to the IWM method, the formulations that try to address the whole Einstein system [21,40] result to a significantly larger number of elliptic equations. For neutron star spacetimes [40], we have fourteen nonlinear Poisson-like equations, with four of them resulting from imposing gauge conditions. For black hole spacetimes, we have a total of seventeen equations, with four of them resulting from imposing gauge conditions and another three being used for an additional decomposition of the extrinsic curvature [21]. The inclusion of magnetohydrodynamics will increase the total number of elliptic equations to be solved even more. Therefore, it is evident that in order to be able to study these astrophysically realistic systems in a systematic manner, an efficient parallelization scheme for the COCAL code needs to be developed.

In this paper, we present the OpenMP version of the COCAL code, which we call the Parallel Compact Object CALculator or PCOCAL for short. For the first time, we have systematically analyzed the whole serial COCAL code and identified the bottlenecks both for single objects (e.g., rotating neutron stars) as well as for binaries (e.g., binary neutron stars) using either the IWM or the waveless formulation. In order to implement the OpenMP directives for parallelization, we had to revise and sometimes completely rewrite many of the original subroutines. Using modest resources of 30–40 cores, both the single rotating neutron star module as well as the binary neutron star one achieve a speedup of an order of magnitude. In the binary case, we observed a smaller speedup than in the single star case, mainly due to the communication between different coordinate systems, which is absent in the latter calculation. As an application of our new code, we computed for the first time a spinning binary quark star system where each companion has a dimensionless spin of $0.43$ aligned with the orbital angular momentum.

In the equations presented, Greek indices are taken to run from 0 to 3 while Latin indices run from 1 to 3. We use a signature $(-,+,+,+)$ for the spacetime line element and a system of units in which $c=G=M_{\odot }=1$ (unless explicitly shown).

2. The Initial Value Equations

In this section, we summarize the 3 + 1 decomposition of Einstein’s equations for two representative problems that will be discussed below: (i) single rotating neutron stars in the waveless formalism [8,40] and (ii) spinning binary neutron stars in the IWM formalism [16,17]. While details on these methods can be found in the aforementioned papers, here we provide a broad overview of the equations that need to be solved in order to discuss the time footprint of each one in the iteration procedure, the possible bottlenecks with the parallelization, and the optimizations we perform.

We assume that a spacetime $\mathcal{M}$ is foliated by a family of space-like hypersurfaces ${\Sigma }_t$, parametrized by a time coordinate $t\in \mathbb{R}$ as $\mathcal{M}=\mathbb{R}\times {\Sigma }_t$. The future-pointing unit normal one form to ${\Sigma }_t$, $n_{\alpha }=-\alpha {\nabla }_{\alpha }t$, is related to the generator of time translations $t^{\alpha }$ as

$$t^{\alpha }:=\alpha n^{\alpha }+{\beta }^{\alpha },$$

(1)

where $t^{\alpha }{\nabla }_{\alpha }t=1$. Here, $\alpha$ and ${\beta }^{\alpha }$ are, respectively, the lapse function and the shift vector, which is spatial, ${\beta }^{\alpha }{\nabla }_{\alpha }t=0$. The projection tensor ${\gamma }_{\beta }^{\alpha }$ to ${\Sigma }_t$ is introduced as ${\gamma }_{\beta }^{\alpha }:=g_{\beta }^{\alpha }+n^{\alpha }{n}_{\beta }$. The induced spatial metric $g_{ij}$ on ${\Sigma }_t$ is the projection tensor restricted to it. By introducing a conformal spatial geometry with spatial metric ${\tilde{\gamma }}_{ij}={\psi }^{-4}{\gamma }_{ij}$, where $\psi$ is the conformal factor, we can write the line element on a chart $\{t,x^i\}$ of ${\Sigma }_t$ as

$$d{s}^2=-{\alpha }^{2}d{t}^2+{\psi }^4{\tilde{\gamma }}_{ij}(d{x}^i+{\beta }^{i}dt)(d{x}^j+{\beta }^{j}dt).$$

(2)

The conformal rescaling is determined from a condition $\tilde{\gamma }=f$, where $\tilde{\gamma }$ and f are determinants of the rescaled spatial metric ${\tilde{\gamma }}_{ij}$ and the flat metric $f_{ij}$, respectively. We denote by $h_{ij}$ and $h^{ij}$ the differences between the conformal and flat metric

$${\tilde{\gamma }}_{ij}:=f_{ij}+h_{ij},{\tilde{\gamma }}^{ij}:=f^{ij}+h^{ij}.$$

(3)

The extrinsic curvature of each slice ${\Sigma }_t$ is defined by

$$\begin{array}{c}\hfill K_{ij}=-\frac{1}{2}{\mathcal{L}}_{\mathbf{n}}{\gamma }_{ij}\end{array}$$

(4)

where ${\mathcal{L}}_{\mathbf{n}}$ is the Lie derivative along the normal vector on ${\Sigma }_t$. Hereafter, we denote the trace of $K_{ij}$ by K and the trace-free part of $K_{ij}$ by $A_{ij}:=K_{ij}-\frac{1}{3}{\gamma }_{ij}K$. We define the conformal extrinsic curvature ${\tilde{A}}_{ij}={\psi }^{-4}{A}_{ij}$, similar to the conformal spatial metric.

In this paper, we consider perfect fluid spacetimes, whose stress–energy tensor is written as

$$T^{\alpha \beta }=(ϵ+p)u^{\alpha }{u}^{\beta }+p{g}^{\alpha \beta },$$

(5)

where $ϵ$ is the energy density and p is the pressure as measured by the co-moving observer, i.e., an observer with 4-velocity $u^{\alpha }$. As discussed in [18,19], in the case where we have magnetized neutron stars and total stress–energy tensor in addition to the perfect fluid contribution, Equation (5) will include the contribution of the magnetic field. We decompose the Einstein’s equations ${\mathcal{E}}_{\alpha \beta }:=G_{\alpha \beta }-8\pi T_{\alpha \beta }=0$ along the hypersurface as well as along its normal $n^{\alpha }$ as follows,

$$\begin{array}{ccc}& {\mathcal{E}}_{\alpha \beta }{n}^{\alpha }{n}^{\beta }=0,& \hfill \end{array}$$

(6)

$$\begin{array}{ccc}& {\mathcal{E}}_{\alpha \beta }{\gamma }_i^{\alpha }{n}^{\beta }=0,& \hfill \end{array}$$

(7)

$$\begin{array}{ccc}& {\mathcal{E}}_{\alpha \beta }\left({\gamma }^{\alpha \beta }+\frac{1}{2}{n}^{\alpha }{n}^{\beta }\right)=0,& \hfill \end{array}$$

(8)

$$\begin{array}{ccc}& {\mathcal{E}}_{\alpha \beta }\left({\gamma }_i^{\alpha }{\gamma }_j^{\beta }-\frac{1}{3}{\gamma }_{ij}{\gamma }^{\alpha \beta }\right)=0,& \hfill \end{array}$$

(9)

which correspond, respectively, to the Hamiltonian constraint, momentum constraint, spatial trace part (combined with the Hamiltonian constraint), and spatial trace-free part.

As shown in [40], the above set of field Equations (6)–(9) are reduced to elliptic (Poisson) equations for $\{\psi ,{\tilde{\beta }}_i,\alpha \psi ,h_{ij}\}$, respectively, as follows,

$$\begin{array}{ccc}\hfill \mathring{\Delta }\psi & =& {\mathcal{S}}_{\mathrm{H}},\hfill \end{array}$$

(10)

$$\begin{array}{ccc}\hfill \mathring{\Delta }{\tilde{\beta }}_i& =& {\mathcal{S}}_i,\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill \mathring{\Delta }\left(\alpha \psi \right)& =& {\mathcal{S}}_{\mathrm{tr}},\hfill \end{array}$$

(12)

$$\begin{array}{ccc}\hfill \mathring{\Delta }{h}_{ij}& =& {\mathcal{S}}_{ij},\hfill \end{array}$$

(13)

where $\mathring{\Delta }$ is the flat metric (in arbitrary coordinates) Laplacian, defined by $\mathring{\Delta }=f^{ij}{\mathring{D}}_i{\mathring{D}}_j$. The sources ${\mathcal{S}}_{\mathrm{H}}$, ${\mathcal{S}}_i$, ${\mathcal{S}}_{\mathrm{tr}}$, and ${\mathcal{S}}_{ij}$ [21] depend nonlinearly on the unknown potentials $\{\psi ,{\tilde{\beta }}_i,\alpha \psi ,h_{ij}\}$ and are written in Appendix A. In the COCAL code, we use the Cartesian components of the elliptic Equations (10)–(13), on spherical grids (see Section 3). Equations (10)–(13) must be supplied with conditions on the boundary of our computational region. Because we will only consider isolated single or binary stars, the boundary conditions for the gravitational equations will only be at spatial infinity, where we impose asymptotic flatness, i.e.,

$$\underset{r\to \infty }{lim}\alpha =\underset{r\to \infty }{lim}\psi =1,\underset{r\to \infty }{lim}{\beta }^i=\underset{r\to \infty }{lim}{h}_{ij}=0.$$

(14)

In case a black hole is present, COCAL uses excision, and boundary conditions are applied on the excised surface (typically a sphere). Depending on the method, we have employed Robin–Dirichlet, apparent horizon [13,14,23], or simply Kerr–Schild [21] boundary conditions.

2.1. Rotating Neutron Star in the Waveless Formalism

Coordinate gauge conditions such as the maximal slicing and generalized Dirac gauge

$$\begin{array}{ccc}& K=0& \hfill \end{array}$$

(15)

$$\begin{array}{ccc}& H^i=0& \hfill \end{array}$$

(16)

are assumed for rotating neutron stars. As described in Appendix A, $H^i={\mathring{D}}_j{\tilde{\gamma }}^{ij}$. These conditions simplify Equations (A1)–(A4) significantly. In particular, ${\tilde{\mathcal{R}}}_{ij}^{\mathrm{LI}}=0$. The asymptotic behavior of the metric potentials becomes a Coulomb-type fall-off,

$$\begin{array}{cc}\hfill \psi -1=O\left(r^{-1}\right),& \alpha -1=O\left(r^{-1}\right),\end{array}$$

(17)

$$\begin{array}{cc}\hfill {\beta }^i=O\left(r^{-2}\right),& h_{ij}=O\left(r^{-1}\right).\end{array}$$

(18)

Our choice of ${\beta }^a$ is the shift in an (asymptotically) inertial frame.

The Bianchi identity implies ${\nabla }_{\alpha }{T}^{\alpha \beta }=0$. Under the assumption of rest mass conservation ${\nabla }_{\alpha }\left(\rho u^{\alpha }\right)=0$ and an isentropic law ${\nabla }_{\alpha }s=0$, it leads to the relativistic Euler equation

$$u^{\beta }{\omega }_{\beta \alpha }=0,\mathrm{where}{\omega }_{\alpha \beta }:={\nabla }_{\alpha }\left(h{u}_{\beta }\right)-{\nabla }_{\beta }\left(h{u}_{\alpha }\right)$$

(19)

is the relativistic vorticity and $h:=(ϵ+p)/\rho$ the relativistic specific enthalpy. We consider two cases of single rotating stars, both of which are described by a 4-velocity

$$u^{\alpha }=u^t{k}^{\alpha }.$$

(20)

Here, $k^{\alpha }=t^{\alpha }+\Omega {\varphi }^{\alpha }$ is the helical symmetry vector, $\Omega$ is the constant angular velocity of the star, and ${\varphi }^{\alpha }$ is the generator of rotational symmetry. For uniformly rotating stars, it is $u^{\beta }{\omega }_{\beta \alpha }=-u^t{\nabla }_{\alpha }\left(h{u}_{\beta }{k}^{\beta }\right)$.

2.1.1. Axisymmetric Rotating Neutron Stars

For stationary and axisymmetric systems, we impose time symmetry on both the three dimensional metric as well as the extrinsic curvature

$${\mathcal{L}}_{\mathbf{t}}{\gamma }_{ij}={\mathcal{L}}_{\mathbf{t}}{K}_{ij}=0.$$

(21)

Note that we do not explicitly impose the axisymmetry on our formulation. Under these asumptions, ${\tilde{u}}_{ij}=0$ in Equation (A2) and ${\mathcal{L}}_{\alpha \mathbf{n}}(\cdots )=-{\mathcal{L}}_{\beta }(\cdots )$ for Equation (A4). Note that the last terms involve second order derivatives of the shift vector ${\tilde{\beta }}^i$.

The stationarity condition for the fluid variables

$${\mathcal{L}}_{\mathbf{t}}\left(h{u}_{\alpha }\right)={\mathcal{L}}_{\mathbf{t}}\rho =0,$$

(22)

reduce the Euler Equation (19) to the simple algebraic equation

$$\frac{h}{u^t}=C,$$

(23)

where C is a constant to be determined. Note that $u^t$ is computed by the normalization condition $u^{\alpha }{u}_{\alpha }=-1$. In this work, we also assume that neutron stars are “cold”, i.e., they can be described by a zero-temperature equation of state (EOS),

$$ϵ=ϵ\left(\rho \right),p=p\left(\rho \right),$$

(24)

or, equivalently, $p=p\left(ϵ\right)$. This kind of one-parameter EOS is called barotropic. Equations (10)–(13) under gauge conditions (15) and (16) and Equation (23) determine all gravitational and fluid variables.

2.1.2. Triaxial Rotating Neutron Stars

For nonaxisymmetric rotating neutron stars, we impose time symmetry on the three metric but helical symmetry (stationarity in the rotating frame) on the extrinsic curvature

$${\mathcal{L}}_{\mathbf{t}}{\gamma }_{ij}=0,\mathrm{and}{\mathcal{L}}_{\mathbf{k}}{K}_{ij}=0.$$

(25)

Under the conditions of (25), we have ${\tilde{u}}_{ij}=0$ in Equation (A2) and ${\mathcal{L}}_{\alpha \mathbf{n}}(\cdots )=-{\mathcal{L}}_{\omega }(\cdots )$, where ${\omega }^{\alpha }={\beta }^{\alpha }+\Omega {\varphi }^{\alpha }$ is the so-called corotating shift vector ${\omega }^{\alpha }$.

The helical symmetry condition for the fluid variables

$${\mathcal{L}}_{\mathbf{k}}\left(h{u}_{\alpha }\right)={\mathcal{L}}_{\mathbf{k}}\rho =0,$$

(26)

reduce Euler Equations (19)–(23) similarly to the stationary and axisymmetric case.

2.2. Binary Neutron Stars in the IWM Formalism

Binary neutron stars are computed within the IWM formalism, where

$$h_{ij}=0,\mathrm{and}K=0,$$

(27)

which means that Equation (13) is absent, while in Equations (10)–(12), the magenta terms in the sources (A1)–(A3) are zero. The spacetime helical symmetry

$${\mathcal{L}}_{\mathbf{k}}{\gamma }_{ij}={\mathcal{L}}_{\mathbf{k}}{K}_{ij}=0$$

(28)

implies that Equation (A10) can be written as

$${\tilde{A}}_{ij}=\frac{1}{2\alpha }{\left(\tilde{\mathbb{L}}\tilde{\omega }\right)}_{ij}$$

(29)

where ${\omega }^i$ the corotating shift.

The 4-velocity of the fluid here can be written as

$$u^{\alpha }:=u^t(k^{\alpha }+V^{\alpha }),$$

(30)

where $V^i$ is the fluid velocity in the corotating frame and the helical Killing vector $k^{\alpha }$ refers to a binary system having orbital angular velocity $\Omega$. For irrotational binaries [44,45,46,47], we have ${\omega }_{\alpha \beta }=0$, so the specific enthalpy current $h{u}_{\alpha }$ can be derived from a potential $h{u}_{\alpha }={\nabla }_{\alpha }\Phi$. In order to allow for arbitrary spinning binary configurations, a 3-vector $s^i$ is introduced according to [48]

$${\widehat{u}}_i:={\gamma }_i^{\alpha }h{u}_{\alpha }=D_i\Phi +s_i,$$

(31)

where the $D_i\Phi$ part corresponds to the “irrotational part” of the flow and $s^i$ the “spinning part” of the flow. Equations (30) and (31) yield

$$V^i=\frac{D^i\Phi +s^i}{h{u}^t}-{\omega }^i.$$

(32)

For arbitrary spinning binaries under the assumptions of helical symmetry

$${\mathcal{L}}_{\mathbf{k}}\left(h{u}_{\alpha }\right)=0$$

(33)

and the additional assumption for the spin of the neutron star

$${\mathcal{L}}_{\mathit{V}}\left(s_{\alpha }\right)=0,$$

(34)

yields

$$\frac{h}{u^t}+V^j{D}_j\Phi =C,$$

(35)

where C is a constant to be determined.

The conservation of rest mass and Equation (32) for the fluid velocity will produce an extra elliptic equation for the fluid potential $\Phi$

$$\begin{array}{ccc}\hfill {\nabla }^2\Phi & =& -\frac{2}{\psi }{\partial }_i\psi {\partial }^i\Phi +{\psi }^4{\omega }^i{\partial }_i\left(h{u}^t\right)+[{\psi }^{4}h{u}^t{\omega }^i-{\partial }^i\Phi ]{\partial }_{i}ln\left(\frac{\alpha \rho }{h}\right)\hfill \\ & \phantom{=}& -{\psi }^4\left[{\partial }_i{s}^i+s^i{\partial }_{i}ln\left(\frac{\alpha \rho {\psi }^6}{h}\right)\right]:={\mathcal{S}}_{\Phi }.\hfill \end{array}$$

(36)

Note that because $h_{ij}=0$, the conformal geometry is flat ${\tilde{\gamma }}_{ij}=f_{ij}$ and therefore ${\partial }_i\Phi ={\tilde{D}}_i\Phi$. The boundary for the fluid is represented by the surface of the star; hence, the boundary condition for Equation (36) will be of the von Neumann type, that is, in terms of derivatives of the rest mass density and of $\Phi$

$${\left[\left({\psi }^{4}h{u}^t{\omega }^i-{\partial }^i\Phi -{\psi }^4{s}^i\right){\partial }_i\rho \right]}_{\mathrm{surf}.}=0.$$

(37)

Equations (10)–(12) under a maximal slicing gauge condition and Equations (35) and (36) determine all gravitational and fluid variables. As in the rotating star case, $u^t$ is determined from the normalization of the 4-velocity. A barotropic EOS is assumed.

3. Numerical Methods

All elliptic equations are solved using the representation theorem of partial differential equations through a Picard type of iteration. Starting from

$${\nabla }^{2}f=S$$

(38)

where S is a nonlinear function of f (here, f can be any of $\{\psi ,\alpha ,{\tilde{\beta }}_i,h_{ij},\Phi \}$ and S any combination of them), and by using an appropriate Green’s function

$${\nabla }^{2}G(x,x^{\prime })=-4\pi \delta (x-x^{\prime }),$$

(39)

that satisfies certain boundary conditions, a solution for f can be written as

$$f\left(x\right)=-\frac{1}{4\pi }{\int }_{V}G(x,x^{\prime })S\left(x^{\prime }\right)d^3{x}^{\prime }+\frac{1}{4\pi }{\int }_{\partial V}\left[G(x,x^{\prime }){\nabla }^{\prime a}f\left(x^{\prime }\right)-f\left(x^{\prime }\right){\nabla }^{\prime a}G(x,x^{\prime })\right]d{S}_a^{\prime }.$$

(40)

where V is the domain of integration, $x,x^{\prime }\in V\subseteq {\Sigma }_t$, the initial space-like hypersurface. The volume V and its boundary $\partial V$ depend on the coordinate system used and are different for isolated rotating neutron stars and binary neutron stars. We will explain this difference in the next sections. This method is widely known as the KEH method [24,25].

For the evaluation of the integrals in Equation (40), a multipole expansion of $G(x,x^{\prime })$ in associated Legendre functions, $P_{\ell }^m$, on spherical coordinates is used;

$$G(x,x^{\prime }):=\sum _{\ell =0}^{\infty }{g}_{\ell }(r,r^{\prime })\sum _{m=0}^{\ell }{ϵ}_m\frac{(\ell -m)!}{(\ell +m)!}\times P_{\ell }^m(cos\theta )P_{\ell }^m(cos{\theta }^{\prime })cos\left[m(\varphi -{\varphi }^{\prime })\right],$$

(41)

where the radial Green’s function $g_{\ell }(r,r^{\prime })$ depends on boundary conditions and the coefficients ${ϵ}_m$ are ${ϵ}_0=1$ and ${ϵ}_m=2$ for $m\ge 1$. In practice, the sum over ℓ is truncated at $\ell =L$. The standard value used for single rotating stars as well as binaries is $L=12$. On the other hand, highly rotating black holes and magnetized neutron stars require a larger $L\lesssim 60$.

3.1. Single Rotating Neutron Star

For a single rotating neutron star, the only elliptic equations to be solved are the gravitational Equations (10)–(13). We employ a single spherical coordinate system that covers the region $[r_a,r_b]\times [0,\pi ]\times [0,2\pi ]$, where $r_a=0$ and $r_b\sim O\left({10}^{6}M\right)$, with M being the total mass of the system. The boundary conditions (14) are applied at $r=r_b$. Coordinate grids $(r_i,{\theta }_j,{\varphi }_k)$ with $i=0,\cdots ,N_r$, $j=0,\cdots ,N_{\theta }$, and $k=0,\cdots ,N_{\varphi }$, are freely specifiable except for the points at the boundary of the computational domain. The grid setup is the same as in [8]: the radial grid intervals $\Delta r_i:=r_i-r_{i-1}$ are constant when $r_i\in [r_a,r_c]$$(i=1,\cdots ,N_r^{\mathrm{m}})$, which typically extends up to ∼$1.25R_e$, where $R_e$ is the equatorial radius of the star, and increase thereafter in a geometric progression with $\Delta r_i=k\Delta r_{i-1}$. Here, k is a constant, and $r_i\in [r_c,r_b]$$(i=N_r^{\mathrm{m}}+1,\cdots ,N_r)$. For the angular coordinate grids $({\theta }_j,{\varphi }_k)$, we choose equally spaced grids. Definitions of the parameters for the grid setups are listed in

Table 1. Summary of parameters used for single rotating star configurations. $R(\theta ,\varphi )$ denotes the neutron star surface.

${\mathit{r}}_{\mathit{a}}$:	Radial coordinate where the grid ${\mathit{r}}_{\mathit{i}}$ starts.
$r_b$:	Radial coordinate where the grid $r_i$ ends.
$r_c$:	Radial coordinate between $r_a$ and $r_b$ where the grid changes from equidistant to non-equidistant.
$N_r$:	Total number of intervals $\Delta r_i$ between $r_a$ and $r_b$.
$N_r^{\mathrm{m}}$:	Number of intervals $\Delta r_i$ in $[r_a,r_c]$.
$N_r^{\mathrm{f}}$:	Number of intervals $\Delta r_i$ in $[0,R(\theta ,\varphi \left)\right]$.
$N_{\theta }$:	Total number of intervals $\Delta {\theta }_i$ for $\theta \in [0,\pi ]$.
$N_{\varphi }$:	Total number of intervals $\Delta {\varphi }_i$ for $\varphi \in [0,2\pi ]$.
L:	Number of multipole in the Legendre expansion.

.

When applying Equation (40), we use the Green’s function $G(x,x^{\prime })=1/|x-x^{\prime }|$. For the quadratures, a 2nd order midpoint rule is used in r and $\varphi$ integrations, and a 4th order midpoint rule is used for $\theta$ integration. We also use a 2nd order finite difference formula for the r, $\theta$, and $\varphi$ derivatives evaluated at the midpoints $(r_{i+\frac{1}{2}},{\theta }_{j+\frac{1}{2}},{\varphi }_{k+\frac{1}{2}})=((r_i+r_{i+1})/2,({\theta }_i+{\theta }_{i+1})/2,({\varphi }_i+{\varphi }_{i+1})/2)$, except for a 3rd order finite difference formula used in the first radial derivative evaluated at the midpoints. For derivatives at the grid points $(r_i,{\theta }_j,{\varphi }_k)$, a 4th order finite difference formula is used. We found [13,15] that these choices are necessary in order to maintain an overall second-order accuracy for the COCAL code and a small error near the possible black hole excised surface [15]. Note that we use the midpoint rule for numerical quadrature formula and hence compute the source terms, Equation (A1)–(A4), at the midpoints of the grids.

A relaxation parameter $\lambda$ is used when updating a newly computed variable $f\in \{\psi ,\alpha ,{\tilde{\beta }}_i,h_{ij},\Phi ,\rho \}$. If $f^{\left(n\right)}\left(x\right)$ is the value at the n-th iteration, and $\widehat{f}\left(x\right)$ the current result of the Poisson solver, Equation (40), then the $(n+1)$-th iteration value will be

$$f^{(n+1)}\left(x\right):=\lambda \widehat{f}\left(x\right)+(1-\lambda )f^{\left(n\right)}\left(x\right),$$

(42)

where $0.1\le \lambda \le 0.4$. Usually, $\lambda =0.4$ for all variables except for $\Phi$, where $\lambda =0.1$. The criterion used by the cocal to stop the iteration for a variable f is based on the relative error between two successive iterations and is given by

$${\mathtt{E}}^f\left(x\right)=2\frac{|f^{\left(n\right)}\left(x\right)-f^{(n-1)}\left(x\right)|}{|f^{\left(n\right)}\left(x\right)|+|f^{(n-1)}\left(x\right)|}<{10}^{-6},$$

(43)

for all points of the grids and all variables. The coupled elliptic Equations (10)–(13) are solved sequentially (starting from $\psi$, followed by $\alpha$, ${\beta }^i$, and finally the $h_{ij}$) at every iteration, and Equation (42) is applied separately to each potential before moving to the next elliptic equation.

3.2. Binary Systems

For binary configurations, the hypersurface ${\Sigma }_t$ is covered by at least three coordinate systems, as explained in detail in [13,14,15,16]. As an example of a general binary system, we show in Figure 1 a black hole–neutron star ($x-y$ cross-section) that consists of the following: (i) Coordinate system COCP-BH (top-left coordinate system) centered around the black hole extending from an inner sphere $S_a$ (small black circle at its center) of radius $r_a$ to a sphere $S_b$ of radius $r_b$ (red outer circle). The surface $S_a$ denotes the black hole region and is excised from the grid. If M is the mass of the system, we have $r_a\sim O\left(M\right)$ while $r_b\sim O\left(100M\right)$; (ii) Coordinate system COCP-NS (top-right coordinate system) centered around the neutron star, which extends from $r_a=0$ to a sphere $S_b$ of radius $r_b$ (blue outer circle). Both COCP-BH and COCP-NS contain an excised sphere $S_e$ (although we use the common symbol $S_e$, these spheres are different in the two coordinate systems), which is introduced to improve the angular resolution and reduce the number of multipoles for resolving the companion object in a given coordinate system; and (iii) Coordinate system ARCP, which is positioned at the center of mass of the system and extends from an inner sphere $S_a$ of radius $r_a\sim O\left(10M\right)$ (green inner circle) to an outer sphere $S_b$ of radius $r_b\sim \left({10}^{6}M\right)$ (outer black circle). Definitions of the parameters for the grid setups are listed in

Table 2. Summary of the grid the parameters used for the binary systems and, in particular, the binary neutron stars computed here. $R(\theta ,\varphi )$ denotes the neutron star surface. Every patch has its own set of parameters above.

${\mathit{r}}_{\mathit{a}}$:	Radial coordinate where the radial grids start. For the COCP-NS patch it is ${\mathit{r}}_{\mathit{a}}=0$.
$r_b$:	Radial coordinate where the radial grids end.
$r_c$:	Center of mass point. Excised sphere is located at $2r_c$ in the COCP patch.
$r_e$:	Radius of the excised sphere. Only in the COCP patch.
$N_r$:	Number of intervals $\Delta r_i$ in $[r_a,r_b]$.
$N_r^{\mathrm{f}}$:	Number of intervals $\Delta r_i$ in $[0,R(\theta ,\varphi \left)\right]$ for the COCP-NS patch or in $[r_a,r_a+1]$ for the ARCP patch.
$N_r^{\mathrm{m}}$:	Number of intervals $\Delta r_i$ in $[r_a,r_c]$.
$N_{\theta }$:	Number of intervals $\Delta {\theta }_j$ in $[0,\pi ]$.
$N_{\varphi }$:	Number of intervals $\Delta {\varphi }_k$ in $[0,2\pi ]$.
L:	Order of included multipoles.

.

In binary systems, elliptic Equations (10)–(12) are solved separately in all three coordinate systems (in Figure 1), and a smooth solution emerges in an iterative process. On the other hand, Equation (36) is only solved in coordinate systems where a neutron star exists. Therefore, the computational cost of a binary system is significantly larger than that of a single rotating star. Convergence to a smooth solution is achieved because in any given coordinate system, in the calculation of the surface integrals in Equation (40), we use the values of the potential functions f from another coordinate system as indicated by the red, green, and blue arrows in Figure 1. For example, when we calculate the contribution of the surface integral on the red sphere $S_e$ inside the COCP-NS coordinate system (top-right), we use the values of the potentials on the corresponding red sphere from the COCP-BH coordinate system (top-left) as indicated by the red curved arrow. In this way, a given potential ($\alpha ,\psi ,{\beta }^i$) from one coordinate system is communicated to all others, and at the end of the iterative process, a smooth solution is obtained in the whole computational domain. For a detailed explanation of how the KEH method works in multiple coordinate systems, see [13,14,23].

4. Speedup and Efficiency Results

The original serial COCAL consists of more than 400,000 lines of Fortran 90 code, and in order to use OpenMP parallelization, many of its loops had to be rewritten in a way that would efficiently utilize OpenMP’s capabilities. In particular, in multiple loops, the code inside the innermost one had to be written in a way that is independent between threads. For example, in a typical triple loop over the r (loop 1), $\theta$ (loop 2), and $\varphi$ (loop 3) coordinates, any code between loop 1 and loop 2 and/or between loop 2 and loop 3 is written only inside the innermost loop 3. The reason is to utilize the collapse clause, which collapses the multiple loops into a single one, which is then divided among the multiple threads. Most of our loops were three-dimensional, but there were many that were four or even five-dimensional, which result in large speedups through OpenMP parallelization. For example, in a binary system, Figure 1, the calculation of the surface integral on the excised sphere $S_e$ uses the Legendre functions with respect to the coordinate system at the center of $S_e$. Assuming ${\theta }_e$ is the spherical angle with respect to the z-axis at the center of $S_e$, the term $cos\left({\theta }_e\right)$ will depend on all coordinates $r,\theta ,\varphi$ of that particular patch; therefore, these functions become five-dimensional, as they depend on the Legendre indices $\ell ,m$ too.

Another commonly used command is the reduction clause, which is used to perform summations, for example in the calculation of the volume and surface integrals. In this case, the code is written in a way that the summation appears only inside the innermost loop so that a combination of the collapse and reduction clauses can execute this operation in multiple threads. The reduction clause is also used in finding the maximum error in Equation (43) for every variable in each coordinate system. The private clause is used extensively for local nested-loop variables so that multiple threads executing a parallel region are independently calculated. Listing a variable as private causes each thread that executes that construct to receive a new temporary variable of the same type, and the multiple loop can be performed independently in parallel.

Below, we will describe the two main modules of the PCOCAL code: (a) the parallelized single rotating star module (Section 2.1); and (b) the parallelized binary neutron star module (Section 2.2). For the single star solver (a), the computational infrastructure is straightforward (one spherical grid), but the mathematical formalism used requires the solution of 14 elliptic equations. On the other hand, for the binary solver (b), a simpler mathematical formalism requires the solution of six elliptic equations in a more intricate computational domain, which involves at least three coordinate systems (see Figure 1).

In order to quantify the speedup and efficiency of PCOCAL, we define the following measures:

• $T_p\left(n\right)$: CPU wall-clock time of parallelized part of the code using n threads.
• $T_s$: CPU wall-clock time of serial (nonparallelized) part of the code. Input/Output (IO) and copying between arrays are excluded.
• $T_{\mathrm{IO}}$: CPU wall-clock time of serial IO.
• $T_m$: CPU wall-clock time of memory copy between arrays.

The reason for timing $T_m$ is because in COCAL/PCOCAL, every main variable (e.g., the 3D conformal factor $\psi$) exists in all coordinate systems (which are at least three) and is stored in a higher dimensional array (for the case of $\psi$, in a 4D array) where the extra index determines the patch (COCP-1, COCP-2, or ARCP). When calculating quantities in one patch where we want to access a quantity in another patch, an array copy is necessary. The time for these copies is measured by $T_m$.

The total time for a calculation is $T=T_p+T_s+T_{\mathrm{IO}}+T_m$. We define the speedup of the code using n threads as

$$S_p\left(n\right)=\frac{T_p\left(1\right)}{T_p\left(n\right)},S\left(n\right)=\frac{T\left(1\right)}{T\left(n\right)}$$

(44)

while the efficiency is defined as

$$E_p\left(n\right)=\frac{S_p\left(n\right)}{n},E\left(n\right)=\frac{S\left(n\right)}{n}.$$

(45)

The speedup $S_p\left(n\right)$ and efficiency $E_p\left(n\right)$ refer to the parallelized part of code (which constitutes the large majority of the solver), while $S\left(n\right)$ and $E\left(n\right)$ refer to the total speedup and efficiency, which include the serial parts of the code, the input/output, and memory handling. Note that for the tests performed in this paper, the input/output routines were neither optimized nor minimized (for diagnostic purposes), and thus, the total speedup shown underestimates the real one.

To make sure that the parallelized PCOCAL code produces the same results as COCAL itself, we perform a pointwise check for every parallelized subroutine against the serial code and confirm that the differences between the two codes are ≲${10}^{-12}$ in all variables.

Most of the runs have been performed in server A, which has 36 cores with 2 threads per core (72 threads total) and an Intel(R) Xeon(R) Gold 6254 CPU 3.10 GHz (Intel, Santa Clara, CA, USA). We have also used server B, which has 40 cores with 2 threads per core (80 threads total) and an Intel(R) Xeon(R) Gold 6242R CPU 3.10 GHz. Both servers are dual-socket. An Intel compiler (2021.3.0 20210609) is being used.

4.1. Single Rotating Neutron Star

Following the methods of Section 2.1 and Section 3.1, we test our new code in three grid resolutions, as can be seen in

Table 3. Three different resolutions used for the single rotating star tests. The parameters are shown in Table 1. The number of points that cover the largest star radius is $N_r^{\mathrm{f}}$. Resolutions H2, H3, and H4 correspondingly use 2, 10.3, and 80.1 GB of RAM.

Type	${\mathit{r}}_{\mathit{a}}$	${\mathit{r}}_{\mathit{b}}$	${\mathit{r}}_{\mathit{c}}$	${\mathit{N}}_{\mathit{r}}$	${\mathit{N}}_{\mathit{r}}^{\mathbf{m}}$	${\mathit{N}}_{\mathit{r}}^{\mathbf{f}}$	${\mathit{N}}_{\mathit{\theta }}$	${\mathit{N}}_{\mathit{\varphi }}$
H2	0	${10}^6$	1.25	192	80	64	48	48
H3	0	${10}^6$	1.25	384	160	128	96	96
H4	0	${10}^6$	1.25	768	320	256	192	192

. Resolution H2 has $N_r\times N_{\theta }\times N_{\varphi }=$ 442,368 intervals (number of points in the $r,\theta ,\varphi$ directions are $N_r+1$, $N_{\theta }+1$, and $N_{\varphi }+1$, respectively), while resolutions H3, H4 have twice and four times as many intervals in every dimension, i.e., 8 and 64 times more intervals in total, respectively. The default value of terms in the Legendre expansions is $L=12$. Below, we also investigate the performance of the code with respect to L.

In Algorithm 1, we sketch the most salient steps taken for a solution. In parentheses with green fonts, we show the percentage of time needed for a given calculation per iteration using a single core using the H3 resolution. Lines that do not have a number indicate that the time it took for completion was less than $1\%$. The rest of the time is spent in the calculation of diagnostics as well as further output. Steps 2–12 are repeated iteratively until condition 13 is fulfilled, and hence, convergence to a solution is achieved. As we can see, the most time-consuming routines are the computation of the momentum constraint sources ${\mathcal{S}}_i$ (15%), the calculation of the sources ${\mathcal{S}}_{ij}$ (13%), followed by the calculation of the conformal Ricci tensor ${\tilde{\mathcal{R}}}_{ij}$ (11%) and the Poisson solvers themselves (11%). The reason that the three source arrays ${\mathcal{S}}_i$ of the momentum constraint took more time to be computed than the six source arrays ${\mathcal{S}}_{ij}$ was mainly due to the modular way that COCAL implements various mathematical formulations. In particular, for the waveless formulation, the sources (right-hand side of the Poisson equations) are split into two parts: (i) the part that comes from the conformal flat part of the metric and (ii) the ones that come from the nonconformal one. In this way, one has the flexibility of choosing a specific method (conformal flat vs. nonconformal flat) without repeating the writing of a code. The disadvantage is that triple loops all over the gridpoints may be repeated; therefore, speed is sacrificed in view of modularity. In the computation of the momentum constraint sources ${\mathcal{S}}_i$ for the waveless formulation (which is not conformally flat), both the fluid part and the gravitational part of the source computation are performed twice, which leads to a larger time footprint than the six source arrays ${\mathcal{S}}_{ij}$. Overall the bottleneck for the rotating neutron star module in the waveless formulation is the computation of the sources (lines 2–8) rather than the Poisson solvers (line 9). As we will see, this is in contrast with the binary neutron star module in the conformal flat approximation, where the latter dominates over the former.

Algorithm 1 Rotating star in the waveless formalism
1:	procedure RNS
2:	Interpolate variables to SFC	▹$(8\%)$
3:	Compute $K_{ij}(8\%)$, $C_{kj}^i(8\%)$, ${\tilde{\mathcal{R}}}_{ij}(11\%)$	▹$(27\%)$
4:	Compute volume sources ${\mathcal{S}}_{\mathrm{H}}$, Equation (A1)	▹$(5\%)$
5:	Compute volume sources ${\mathcal{S}}_{\mathrm{tr}}$, Equation (A3)	▹$(6\%)$
6:	Compute volume sources ${\mathcal{S}}_i$, Equation (A2)	▹$(15\%)$
7:	Compute volume sources ${\mathcal{S}}_{ij}$, Equation (A4)	▹$(13\%)$
8:	Compute right-hand side of Equation (A22)	▹$(3\%)$
9:	Compute $\psi ,\alpha ,{\tilde{\beta }}_i,h_{ij},{\xi }^i$, Equation (40)	▹$(11\%)$
10:	Variable update, Equation (42)	▹$(2\%)$
11:	Use Equation (23) to compute the rest mass density $\rho$.
12:	Compute ${\mathtt{E}}^{\psi },{\mathtt{E}}^{\alpha },{\mathtt{E}}_i^{\beta },{\mathtt{E}}_{ij}^h,{\mathtt{E}}^{\rho }$ Equation (43)	▹$(1\%)$
13:	if $({\mathtt{E}}^{\psi },{\mathtt{E}}^{\alpha },{\mathtt{E}}_i^{\beta },{\mathtt{E}}_{ij}^h,{\mathtt{E}}^{\rho }<{10}^{-6})$ then
14:	exit

In Figure 2, we plot in the upper two panels the speedup $S_p$ (top) and efficiency $E_p$ (bottom) of the parallelized part of the code (solid lines), as well as the corresponding measures S and E for the total code (dashed lines). As we mentioned above, S underestimates the real speedup as the input/output routines were neither optimized nor minimized (for diagnostic purposes). Thus, in the discussion below, we focus on the speedup $S_p$ and efficiency $E_p$. One characteristic of the speedup is that for all resolutions, it reaches a maximum at ∼36 threads and drops slightly afterwards, from which point it continues to increase. The maximum speedup of the parallelized part of the code is 18–20 times the serial one (when ≲60 threads are used) which can be achieved with a minimum of ∼36 threads. At that point, the efficiency is ∼50%. At maximum speedup, the whole rotating neutron star code (including the serial routines that are nonparallelized, such as the calculation of the parameters $\Omega$ in Equation (20), C in Equation (23), and the coordinate scaling $R_0$ [8,16]) is ∼12, 13, and 14 times faster than the serial analog for resolutions H2, H3, and H4 and occurs at 36, 34, and 35 threads, respectively. This can be seen in Figure 2 on the lower three panels, where blue bars signify the parallelized routines of the code, red bars the serial part of the code, and yellow bars the memory copy between arrays. From this plot, we see that the parallelized code constitutes the vast majority of the total number of subroutines and is responsible for the achieved speedup. Also, we observe that the higher the resolution (H4), the larger the total speedup, which is a promising result for high-resolution campaigns.

In all tests presented in Figure 2, we used $L=12$ terms in the Legendre expansion, Equation (41), and therefore all integrals, Equation (40). It is experimentally found that such a number of terms leads to accurate results without compromising the speed of the code. $L=12$ is commonly used in all nonmagnetized rotating star calculations [7,8,9,10,11,12] as well as in the binary neutron star calculations [16,17] that we will mention in the next section. On the other hand, accurate magnetized rotating neutron stars or black hole-disk solutions require a larger number (≲60) of terms to be included in the Legendre expansions. In Figure 3, we plot the time of a single iteration (blue dots), as well as the time for the Poisson solvers alone (red dots), using the H3 resolution and a single core. Increasing the number of Legendre terms L increases the time spent on the Poisson solvers as

$$\mathrm{Time}\approx 0.32{(L-12)}^{1.28}+c$$

(46)

where $c=9.52$. This fitting function is plotted by a solid yellow line. At the same time, the time spent on the whole iteration is also fitted by the same relation Equation (46), but $c=88.63$ (as reported in Algorithm 1, line 9, the percentage of time spent for the Poisson solvers with respect to the whole iteration when $L=12$ is approximately $11\%$) and is plotted with a purple curve. In other words, for a given resolution, the increase in the number of Legendre terms essentially affects the Poisson solvers as well as the whole iteration in the same manner and scale as ∼$L^{1.28}$.

In order to evaluate our new code with respect to the number L, we performed the same tests using the H4 resolution for three different numbers of Legendre terms $L=12,24,48$. The results are shown in the upper two panels of Figure 4 with red, blue, and yellow curves in server A. In addition, we show with a cyan curve the speedup and efficiency of the $L=24$ run in server B. The first conclusion is that the speedup is approximately preserved when the number of Legendre terms is increased. All runs in server A have a maximum at ∼35 threads. Beyond that point, and similarly to Figure 2, the speedup drops and then starts to increase again. This behavior is qualitatively the same as the run with $L=24$ at server B, with the maximum now happening at 40 threads and a speedup of larger than 20. Given than the server A CPUs have 36 cores while the server B CPUs have 40, we conclude that peak speedup with minimum amount of threads happens approximately at the number of cores a CPU has. The larger the number of cores, the larger the speedup will be. On the lower three panels, we plot the total performance time for $L=12,24,48$ in the H4 resolution and server A. We find that the maximum speedup of the total rotating neutron star code (including the routines that are nonparallelized) is ∼14, 13, and 14 times faster than the serial analog in the H4 resolutions for $L=12,24,48$ and occurs at 35, 36, and 35 threads, respectively. Therefore, the speedup is preserved when the number of Legendre terms is increased for the whole code as well, which is expected, given the fact that the parallelization scheme covers all of the main components of PCOCAL.

4.2. Binary Neutron Stars

Following the methods of Section 2.2 and Section 4, we test our new code in three resolutions as in

Table 4. Three different grid structure parameters used for the circular binary computation in COCAL. The amount of RAM used in each resolution is written in the first column. All variables are explained in Table 2, and the distances are in normalized quantities [16]. There are two COCP grids centered around each neutron star and one ARCP centered at the center of mass of the system (see Figure 1). Resolutions E2.5, E3.0, and E3.5 correspondingly use 5.0, 9.9, and 30.6 GB of RAM.

Type	Patch	${\mathit{r}}_{\mathit{a}}$	${\mathit{r}}_{\mathit{b}}$	${\mathit{r}}_{\mathit{c}}$	${\mathit{r}}_{\mathit{e}}$	${\mathit{N}}_{\mathit{r}}^{\mathbf{f}}$	${\mathit{N}}_{\mathit{r}}^{\mathbf{m}}$	${\mathit{N}}_{\mathit{r}}$	${\mathit{N}}_{\mathit{\theta }}$	${\mathit{N}}_{\mathit{\varphi }}$	$\mathit{L}$
E2.5	$\mathrm{COCP}$	$0.0$	40	$2.51$	$2.375$	80	151	288	72	72	12
	$\mathrm{ARCP}$	$5.0$	${10}^6$	$6.25$	−	24	30	288	72	72	12
E3.0	$\mathrm{COCP}$	$0.0$	40	$2.51$	$2.35$	100	188	384	96	96	12
	$\mathrm{ARCP}$	$5.0$	${10}^6$	$6.25$	−	32	40	384	96	96	12
E3.5	$\mathrm{COCP}$	$0.0$	40	$2.51$	$2.355$	152	286	576	144	144	12
	$\mathrm{ARCP}$	$5.0$	${10}^6$	$6.25$	−	48	60	576	144	144	12

. Resolution E2.5 has $N_r\times N_{\theta }\times N_{\varphi }=$ 1,492,992 intervals in each of the three coordinates systems (two COCPs and one ARCP in Figure 1), while resolutions E3.0 and E3.5 have ∼1.3, and 2 times as many intervals in every dimension, i.e., $2.4$ and 8 times more intervals in total, respectively. In all binary cases, we use $L=12$ as the number of terms in the Legendre expansions. Notice that the BH coordinate system (COCP-BH, top-left, red patch in Figure 1) is now replaced by an NS coordinate system, similar to the blue, top-right patch COCP-NS. Therefore, there is no inner surface $S_a$ (no boundary conditions), and $r_a=0$ for the red patch in Figure 1 for the binary neutron/quark case.

In Algorithm 2, we sketch the most important steps taken for a binary neutron star solution using a single core in the E3.0 resolution. As in Algorithm 1, we report the percentage of time needed for the completion of each step with green fonts inside a parenthesis, while when such number is absent, it means that the time it took for completion was less than $1\%$. The rest of the time is spent in the calculation of diagnostics as well as further output. Steps 2–14 are repeated iteratively until the condition in line 15 is fulfilled. The main differences between the RNS and NSNS modules are: (i) Steps 3–15 are performed in three coordinate systems (two for COCP-NS and the ARCP) as seen in Figure 1 instead of one in the RNS module; (ii) the calculation of every potential $\psi ,\alpha ,{\tilde{\beta }}_i$ in the two COCP-NS coordinate systems is significantly more involved because of the excised sphere $S_e$ (the reason for the existence of $S_e$ in the COCP patches is explained in detail in [13,14,15]); (iii) the additional calculation of the velocity potential $\Phi$ (∼4%) (absent in the RNS module) is another important difference between the NSNS module and the RNS one; (iv) because for binary neutron stars we solve for conformally flat initial data, potentials $h_{ij}=0$ and, in addition, the source terms (Equation (A1)–(A3)) are significantly simpler as the magenta terms are zero.

As mentioned above, surface integrals (see Equation (40)) on $S_e$ do not exist in the single coordinate system of the RNS module. When computing those integrals as well as the surface integrals at $S_b$ and $S_a$ in a given coordinate system (e.g., on COCP-NS-1), the integrands are computed using the variables of another coordinate system (e.g., from COCP-NS-2). In this way, solutions between coordinate systems communicate between each other in order to achieve a smooth solution everywhere. Therefore, to compute the integrands on the surface integrals at $S_a$, $S_b$, and $S_e$, three-dimensional interpolations from nearby points of another coordinate system is needed. In total, the Poisson solvers take ∼41% of the time of an iteration. The time of the Poisson solvers in each COCP-NS patch is ∼17%, while in the ARCP, it is ∼7%. The big difference between them is due to the fact that patch ARCP does not have an excised sphere $S_e$. The surface integral in the off-center sphere $S_e$ is the most time-consuming operation (∼11%), and it involves the computation of the corresponding associated Legendre functions, which are five-dimensional arrays of ($r,\theta ,\varphi ,\ell ,m$). In the current implementation, this time-consuming operation is performed on the fly every time such an integral is calculated in order to have a smaller memory footprint. Note that these functions do not change during the iteration procedure and in principle can only be calculated once. Such an array in the E3.5 resolution will be ∼9 GB.

Algorithm 2 Binary neutron star
1:	procedure NSNS
2:	for all Coordinate Systems A do
3:	Interpolate variables from SFC	▹$(3\%)$
4:	Compute $K_{ij}$	▹$(9\%)$
5:	Compute volume source Equation (A1), ${\mathcal{S}}_{\mathrm{H}}$
6:	Compute volume source Equation (A3), ${\mathcal{S}}_{\mathrm{tr}}$
7:	Compute volume source Equation (A2), ${\mathcal{S}}_i$	▹$(6\%)$
8:	Compute sources on excised surface $S_e$	▹$(1\%)$
9:	Compute $\psi ,\alpha ,{\tilde{\beta }}_i$ Equation (40)	▹$(41\%)$
10:	Variable update Equation (42)	▹$(1\%)$
11:	Use Equation (35) to compute the rest mass density $\rho$.
12:	Compute $\Phi$ Equations (36) and (40)	▹$(4\%)$
13:	Compute ${\mathtt{E}}^{\psi }\left(A\right),{\mathtt{E}}^{\alpha }\left(A\right),{\mathtt{E}}_i^{\beta }\left(A\right),{\mathtt{E}}^{\rho }\left(A\right),{\mathtt{E}}^{\Phi }\left(A\right)$	▹$(1\%)$
14:	Variable copy between CSs	▹$(2\%)$
15:	if $({\mathtt{E}}^{\psi }\left(A\right),{\mathtt{E}}^{\alpha }\left(A\right),{\mathtt{E}}_i^{\beta }\left(A\right),{\mathtt{E}}^{\rho }\left(A\right),{\mathtt{E}}^{\Phi }\left(A\right)<{10}^{-6}$
	for all A) then
16:	exit

In Figure 5, we plot in the upper two panels the speedup $S_p$ (top) and efficiency $E_p$ (bottom) of the parallelized part of the binary NSNS code (solid lines), as well as the corresponding measures S and E for the total code (dashed lines). Similar to the RNS module, Figure 2, we find that for all resolutions the speedup increases until a certain number of threads; then, it exhibits a sudden small drop, from which point it continues to increase with the number of threads. One difference with respect to Figure 2 is that the speedup curves are distinct for resolutions E2.5, E3.0, and E3.0, while for the RNS module we observe a broad overlap of these curves between resolutions H2, H3, H4. The reason for this behavior is the communication between the different coordinate systems in the binary code, which is absent in the single neutron star module. Despite that, the speedup of the parallelized code reaches values of 12–16 using just ∼36 threads. The efficiency at that point is ∼40%. This speedup can further increase if more than 60 threads are used. Another finding in Figure 5 is the fact that for the highest resolution E3.5, the speedup is broadly constant from ∼26 to ∼35 threads, which is due to the domination of data communication between the different grids in the calculation process. In the lower three panels of Figure 5, we also see that the total binary code (including the routines that are nonparallelized) is six, five, and five times faster than the serial analogue for resolutions E2.5, E3.0, and E3.5 and occurs at 33, 34, and 26 threads, respectively. The distribution of times in the whole code can be seen on the lower three panels of Figure 5. In the future, we will address the data communication and memory handling in order to make the binary speedup scale similarly to the RNS code. In practice, a binary neutron star in quasicircular orbit that took approximately six days of continuous computation in the E3.0 resolution with the COCAL code; with PCOCAL, it needs approximately one day using a modest amount of ∼30 threads.

The differences between the PCOCAL and COCAL for a binary neutron or quark star are ≲${10}^{-12}$ in all coordinate systems for the gravitational variables $\psi ,\alpha ,{\beta }^i$ and fluid variables $\rho ,\Phi$, as well as for the constants $\Omega ,C$ and $R_0$ that appear in the calculation.

5. Spinning Binary Quark Stars

Given the fact that the state of matter at supranuclear densities is an open question, there is the possibility that conventional neutron stars contain in their core a phase transition to strange quark matter, i.e., up, down, and strange free quarks. A quark-gluon plasma has been observed in heavy ion collision at CERN and RHIC [49]. Also, Bodmer and Witten [50,51] pointed out that it is possible that the ground state of matter at zero pressure and a large baryon number is not iron but strange quark matter. Thus, from a theoretical standpoint, one can have pure quark stars. In [11,12,52,53,54], we have computed rotating isolated axisymmetric, triaxial, and differentially rotating quark stars, while in [55], irrotational binary quark stars have been evolved in grid-based hydrodynamics. Here, we present an example using the PCOCAL code of a spinning binary quark star, where each companion has a spin along the axis of the orbital angular momentum.

As in our previous studies, the quark EOS is the MIT model with $p=\frac{1}{3}(ϵ-4B)$ or in a parametric form

$$\begin{array}{cc}& p=K{\rho }^{4/3}-B,\end{array}$$

(47)

$$\begin{array}{cc}& ϵ=3K{\rho }^{4/3}+B\end{array}$$

(48)

where $B={ϵ}_s/4$ is the bag constant, which is set to $52.5\mathrm{MeV}{\mathrm{fm}}^{-3}$, and ${ϵ}_s$ is the surface energy density. The parameter $\rho$ could be regarded as the rest mass density and K is constant. As explained in [54], changing K does not affect the EOS; thus, we choose this constant such that at the surface ${\rho }_s={ϵ}_s$, i.e., $h_s=1$. This is accomplished for

$$K={\left(\frac{c^8}{256B}\right)}^{\frac{1}{3}}=3.119\times {10}^{15}\frac{c{m}^3}{s^2{g}^{1/3}}.$$

(49)

Assuming ${\rho }_s=1.4{\rho }_{\mathrm{nuc}}$, this EOS predicts a maximum spherical mass of a quark star of $M_{\max }^{\mathrm{sph}}=2.1M_{\odot }$ with $R_{\max }^{\mathrm{sph}}=11.5\mathrm{km}$.

Using the formalism described in Section 2.2 with a spin vector

$$s^i=0.13(-y_s,x_s,0)$$

(50)

($x_s,y_s$ are coordinates with respect to the quark star center) in Equation (31), and solving Equations (10)–(12), (31), (35) and (36), we compute here a spinning binary quark star at $37.3\mathrm{km}$ separation and Arnowitt–Deser–Misner mass of $M=2.84$. The quasilocal spin [17] of each companion is $J_{\mathrm{ql}}=0.858$ along the axis of orbital motion. This spin angular momentum corresponds to an approximate dimensionless spin of $J_{\mathrm{ql}}/{(M/2)}^2=0.43$. In Figure 6, we plot the rest mass density (left quark star), velocity potential (right quark star), contour plots of the conformal factor $\psi$ (green dashed lines), and velocity potential (brown dashed lines), as well as the fluid velocity with respect to the corotating frame. The direction of the velocity arrows show that the spin angular momentum is along the positive z-axis (contrary to the irrotational case) in accordance with Equation (50). From the density colorbar, we see that the central density ${\rho }_c$ is approximately $1.55$ times the surface density ${\rho }_s$. The nonzero surface density is characteristic of the self-bound quark stars, contrary to their neutron star analogs. For nonspinning (irrotational) binaries of the same surface density (and rest mass), the central density is higher and ${\rho }_c/{\rho }_s=1.63$. This behavior is similar to binary neutron stars as well as to isolated rotating neutron/quark stars. Note that in order to find the quasiequilibrium solution, a root finding method over the central quark star densities is needed, and therefore, a number of cycles (∼10) of Algorithm 2 is performed. Pointwise comparison between PCOCAL and COCAL solutions show differences ≲${10}^{-12}$ for all variables.

6. Conclusions

In this work, we presented a new efficient parallelized code, PCOCAL, for general relativistic compact object initial data. PCOCAL is based on the serial COCAL code (which has been tested for a wide range of astrophysical scenarios) and employs extensive OpenMP practices. To optimize PCOCAL, a thorough profiling of the serial COCAL code is performed for the first time. Both the single and binary neutron star modules have been examined in detail, and timing of each subroutine was performed. The parallelized module that solves the full set of Einstein equations for a single rotating neutron star is found to be 12–14 times faster than the corresponding serial code, with the parallel part being 18–20 times faster than the serial one. The parallelized module that solves for a binary neutron star system in quasiequilibrim is 5–6 times faster than the corresponding serial code, with the parallel part being 12–16 times faster than the serial one. As a novel example, we computed initial data for a spinning binary quark star system that could be a binary neutron star mimicker. The achieved speedup will help us explore a larger set of problems that require intensive calculations, as for example in the calculation of magnetized neutron stars.

As a step forward, we plan to revisit the data communication and memory handling of the binary module in order to achieve better speedups for the highest resolutions, as in the single rotating star module. In addition, we will perform domain decomposition in order to implement a combined MPI-OpenMP scheme.