Dynamic Tomographic Reconstruction of Deforming Volumes

Abstract

The motion of a sample while being scanned in a tomograph prevents its proper volume reconstruction. In the present study, a procedure is proposed that aims at estimating both the kinematics of the sample and its standard 3D imaging from a standard acquisition protocol (no more projection than for a rigid specimen). The proposed procedure is a staggered two-step algorithm where the volume is first reconstructed using a “Dynamic Reconstruction” technique, a variant of Algebraic Reconstruction Technique (ART) compensating for a “frozen” determination of the motion, followed by a Projection-based Digital Volume Correlation (P-DVC) algorithm that estimates the space/time displacement field, with a “frozen” microstructure and shape of the sample. Additionally, this procedure is combined with a multi-scale approach that is essential for a proper separation between motion and microstructure. A proof-of-concept of the validity and performance of this approach is proposed based on two virtual examples. The studied cases involve a small number of projections, large strains, up to 25%, and noise.

1. Introduction

Tomography is a non-destructive imaging technique that enables the visualization of the bulk of the observed specimen. Tomography is now widely used in many fields ( e.g., medical imaging for diagnostic [1], biology [2], material science [3], etc.), performed with various waves ( e.g., X-ray, neutron, electron, terahertz, optics, ultrasound, etc.) depending on the experiment and material absorption and or scattering. Different instruments have been developed with different flux, space and time resolutions ( e.g., for X-rays medical scanners, synchrotron, lab-CT, etc.) giving access to a wide range of imaging devices and performances.

To image the 3D structure, the specimen rotates over 180° or 360° with respect the source-detector pair and at a series of distributed angles radiographs are acquired. Radiographs are transformed with dark-fields and white-fields, to extract the relative beam absorption, transformed with a logarithm (Beer-Lambert law) or more sophisticated treatments for beam hardening, in order to obtain so-called projections of the local coefficient of absorption of the sample. The collection of projections at all angles constitutes a so-called sinogram. Then, from the sinogram, reconstruction algorithms [4] have been developed to reconstruct the 3D imaged volume. This technique relies on the strict satisfaction of conditions, in particular concerning the geometry of the set-up and the motion of the sample as a rigid rotation with the prescribed axis and angles.

The required time for a full 3D scan varies depending on the flux (and exposition time), type of camera and rotation speed of the device. Since the beginning of the development of these techniques, the time required to acquire a tomographic scan has constantly decreased [5]. Recent papers have reported on ultra-fast tomographies, at up to 20 Hz in synchrotron beamlines, that allow extremely fast processes to be captured [6,7].

Motion of the sample during the scan is one of the main issue of tomography that leads to poor quality, blurry volumes [8]. This is the case for medical imaging (as the patient or imaged organ may move), in vivo measurements [9], for electron tomography [10] (because of the extremely small scales of observation, one cannot guarantee a fixed rotation axis at nanometer accuracy) for usually minute to hour long acquisitions, fast mechanical behavior or continuous in situ experiments [6]. Wrong or imprecise estimates of the calibration parameters (that may even vary along the scan) can also be seen as motions in the sinogram space and have the same deleterious consequences for the volume reconstruction.

Sophisticated methods have been developed to avoid or limit motion perturbations especially for periodic motion, for instance using a trigger for acquisition of radiographs based on a specific signal to captures always the same phase as can be done for cardiac or respiratory motion in medical imaging [11,12,13].

Many works have been devoted to correcting imperfect acquisitions as a post-processing treatment. For automatic (re)calibration, online methods, based on the motion of the sample itself during the scanning process, have been applied as a post-process after reconstruction to evaluate a corrected set of calibration parameters [14]. When dealing with electron tomography, (TEM or STEM), the voxel scale makes this problem quite limiting. The identified motion of the specimen is often regularized as being composed of rigid body motions [15,16]. However, in addition to accounting for the slight deviation of the rigid body motion of the sample from the ideal perfect rotation, motion description can be enriched by taking into account more precisely the physics of the electron trajectory in inhomogeneous magnetic fields leading to distortions [17], or sample warping due to irradiation [18,19] for electron tomography. These kinematic degrees of freedom have to be inferred at each projection, and for this fiducial markers (such as gold nanoparticles) are used. In all those cases, deformations can be treated as a slight perturbation, with strains of order of a few ${\displaystyle {10}^{-3}}$ at most. Similarly, in Optical Projection Tomography, OPT, Zhu et al. [20] face similar reconstruction artifacts due to motion for in vivo imaging. Motion is here regularized in time as a polynomial series, and the coefficient describing motions—essentially rigid body motions—are determined from robust quantities (geometric moments) that can be computed over the entire region of interest.

Very early, corrections were also applied in the sinogram space ([21]), with affine transforms [22]. Projection-based measurement methods (e.g., Projection-based Digital Volume Correlation (P-DVC) [23], 3D–2D registration [24,25]) have been developed to correct for rigid body motions (due to a rigid patient motion or variation of calibration parameters) from the radiograph data directly.

Yet, a deforming body with a significant strain and variation in time is a much more demanding case. Projection-based Digital Volume Correlation (P-DVC) has been shown to address part of the problem with complex 4D—3D space + time—kinematic identification [26,27,28]. First if the reference 3D geometry is well known, the displacement field can be evaluated on the fly as the sample is being deformed. This method requires a high quality reference volume and a series of deformed projections. A single projection per motion state is required to capture the full 4D (space-time) kinematics. Alternatively, imperfect acquisition conditions (but no sample strain) can also be corrected using a similar technique, without a pre-determined 3D reference geometry [23], considering that the deforming projection stack is the one used for the reconstruction.

Similar developments have been carried out very early in the context of medical imaging where periodic motion is frequent (heart beat, breathing). In particular, Refs. [29,30,31,32,33] have proposed to determine the motion of the sample from projection data. Small amplitude displacement fields with a periodic modulation in time were considered and identified using highly regularized kinematic models.

However, very often, a reference reconstructed volume is known, and is used as a prior for determining the motion [29,30,31,33,34]. This is often the case for radio-therapy treatment where the key issue is to irradiate the targeted region, in spite of a spurious motion, and hence the goal is to identify the displacement field in 3D, and a fast determination is more valuable than a very precise one.

In a similar spirit, [31,35,36] do not consider a reference to be known but rather use a phase signal (say from an electrocardiogram) to extract from a long sinogram projections coming from a similar phase of the motion, and reconstruct a low quality volume for a series of phase. Registration of the reconstructed volumes [35,36] allows the displacement field to be estimated and interpolated for the entire range of accessed phase. Then, back-correcting for this motion a deformed reconstruction grid is obtained [37] on which the projection data can be backprojected using a classical FBP/FDK algorithms [4]. In this way, each ray follows the deformed sample at each projection angle. The obtained volume has a better quality than the initial one (more details and sharper edges). Katsevich [38] proposed a mathematical study of the generalized inverse Radon transform, using a modified filtered backprojection, showing convergence in appropriate space. Further mathematical consideration lead Hahn [39] to focus on smooth boundaries of subdomains in the volume as the latters produce singularities in projections (in sinogram space, a diverging density appears along tangent planes) that can be tracked in time easily. In the following, inverse Radon transforms and filtered backprojection will not be considered, although they constitute an attractive alternative to the modified algebraic reconstruction algorithm used hereafter.

Dynamic tomography methods based on multiple volume acquisitions have been recently developed. Ruhlandt  et al. [40] proposed an approach without prior knowledge of a phase for each angle, nor of a reference volume, developed for phase contrast imaging at a synchrotron facility. The displacement field that animates a volume at time ${\displaystyle t}$ is measured from the analysis of the motion-blurred reconstructed volume at time ${\displaystyle t-1}$ and ${\displaystyle t+1}$, then interpolated linearly. A full 4D space-time ‘movie’ of the phenomenon could be obtained. This method however requires the use of many acquired 3D volumes for the displacement field measurement, thus a high dose. The measured displacement has a relatively small amplitude compared to the volume texture characteristic scale. A criterion based on the image reconstruction quality is not easy to set and the quality has to be appreciated visually. A similar recent technique [41] deals with the correction of a volume using Digital Volume Correlation and an extended Simultaneous Algebraic Reconstruction Technique (SART) algorithm. To be able to correct a single rotation volume, the authors sub-sampled the acquired projections in 2 sub-acquisitions from which the motion is evaluated and further involved in the reconstruction strategy. This technique is however not suited to large and irregular displacements. In [42], the volume sub-sampling is performed more easily because of an especially designed sampling acquisition strategy (that cannot be adapted to any tomography). One displacement field, constant in time, is estimated for each successive pair of reconstructed volumes and is used to correct the reconstruction procedure. In this latter reference, although the tackled displacements and deformations are important during the entire test, the incremental displacement between all acquired volumes is small.

In most of these studies, the displacement and strain fields between scans was relatively small (strain of approximately ${\displaystyle 1\%}$ and uniformly distributed), and often the time (or phase) is believed to be known.

The present study proposes a strategy to reconstruct both the reference geometry and its large motion from a single sinogram. No periodic signal is used to constrain the kinematics. The recorded projections are the data that drive the measurement of the kinematic field, as is proposed in P-DVC. This however requires a “model,” here a reconstructed 3D volume, to be known in order to measure the displacement field. It is proposed here to “learn” this model from the projection data itself using a multiscale approach.

The standard reconstruction methods are briefly presented in Section 2, so that the introduction of motion can be cast in a similar framework. Section 3 details the joint determination of the reconstructed image and the motion experienced during the scan. The latter algorithm makes use of ideas comparable to those of P-DVC for the motion, and Algebraic Reconstruction Techniques (ART) for the microstructure and exploits a multiscale approach to disentangle microstructure and motion from the sinogram. Two virtual test cases of moving samples validate the procedure (Section 4). The first example is performed with the Shepp-Logan phantom with large deformation up to 20%. The second example is a checkerboard with a more complex temporal pulsating motion.

2. Motionless X-ray Tomography

X-ray tomographic reconstruction is based on the relative beam intensity attenuation for each discrete detector position ${\displaystyle \mathit{r}=[r,z]}$ (where ${\displaystyle z}$ is parallel to the specimen rotation axis, and ${\displaystyle r}$ is perpendicular to it) and rotation angle. For simplicity, and because the present paper is a proof of concept, the displacement field is assumed to lie in a plane perpendicular to the rotation axis, so that each slice ${\displaystyle z}$ remains independent from its neighbors, and the problem turns two dimensional. Hence, only one line of the detector is considered, for a unique value of ${\displaystyle z}$ (omitted from now on).

Let us briefly recall the principle of tomography for a parallel beam: a projection ${\displaystyle p(\mathit{r},\theta )}$ is defined as the line integral of ${\displaystyle f\left(\mathit{x}\right)}$ along a direction ${\displaystyle {\mathit{e}}_{\theta }}$, or

$$p(\mathit{r},\theta )={\int }_{\mathcal{D}(\mathit{r},\theta )}f\left(\mathit{x}\right)\mathrm{d}\mathit{x}$$

(1)

where ${\displaystyle \mathcal{D}(\mathit{r},\theta )}$ is the line parallel to ${\displaystyle {\mathit{e}}_{\theta }}$ hitting the detector plane at position ${\displaystyle \mathit{r}}$. Different projection and interpolation algorithms exist. In the following procedure, the Matlab function radon.m is used.

Tomography consists of recording a set of ${\displaystyle N_{\theta }}$ projections ${\displaystyle p(\mathit{r},\theta )}$ for a collection of angles ${\displaystyle \theta \left(t\right)}$ as the sample is rotated over a complete (or half) rotation about a fixed axis parallel to the detector plane. For a still sample, and a continuous rotation, ${\displaystyle p(\mathit{r},\theta (t\left)\right)}$, written ${\displaystyle p(\mathit{r},\theta )}$, is the Radon transform of ${\displaystyle f\left(\mathit{x}\right)}$, ${\displaystyle p(\mathit{r},\theta )=\mathcal{R}\left[f\right(\mathit{x}\left)\right]}$ and hence the ${\displaystyle f\left(\mathit{x}\right)}$ can be computed from an inverse Radon transform, ${\displaystyle f\left(\mathit{x}\right)={\mathcal{R}}^{-1}\left[p(\mathit{r},\theta )\right]}$. Let us introduce the indicator function ${\displaystyle {\mathcal{I}}_{\mathcal{E}}\left(\mathit{x}\right)}$ of the domain ${\displaystyle \mathcal{E}}$ within which the volume is to be reconstructed. The ray length in ${\displaystyle \mathcal{E}}$ for a specific detector position ${\displaystyle \mathit{r}}$ and rotation angle ${\displaystyle \theta }$, is simply ${\displaystyle L(\mathit{r},\theta )={\mathcal{R}}_{\theta }\left[{\mathcal{I}}_{\mathcal{E}}\left(\mathit{x}\right)\right]}$. It is useful to introduce the backprojection operator, ${\displaystyle {\mathcal{B}}_{\theta }}$, which to each point ${\displaystyle \mathit{x}}$ of the line ${\displaystyle \mathcal{D}(\mathit{r},\theta )}$ within ${\displaystyle \mathcal{E}}$, gives a value ${\displaystyle 1/L(\mathit{r},\theta )}$. Thus for any ${\displaystyle p(r,\theta )}$, ${\displaystyle {\mathcal{R}}_{\theta }\left[{\mathcal{B}}_{\theta }\left[p(\mathit{r},\theta )\right]\right]=p(\mathit{r},\theta )}$.

Tomography is now a very mature field and numerous powerful algorithms have been devised in order to deal with a discrete set of angles, with fan-bean or cone-beam projections [43], with laminography [44], etc.

However, ${\displaystyle f\left(\mathit{x}\right)}$ is always assumed to stand for a rigid and still object (independent of time or rotation angle). From the collection of acquired projections, different algorithms exist to reconstruct the 3D volume [4] and fall into two categories: Fourier-domain algorithms and algebraic algorithms.

Fourier space reconstructions

With Filtered Back-Projection (FBP), each projection, ${\displaystyle p(\mathit{r},\theta )}$ is first “filtered” with a ramp, or Ram-Lak filter, eventually windowed. Ignoring such windowing, in Fourier space, ${\displaystyle \mathcal{F}\left[p\right(\mathit{r},\theta \left)\right](\mathit{k},\theta )}$ is multiplied by ${\displaystyle \left|\mathit{k}\right|}$, inverse Fourier transformed, and then back-projected in real space, thereby producing a field ${\displaystyle g_{\theta }\left(\mathit{x}\right)}$ that is invariant along the direction ${\displaystyle {\mathit{e}}_{\theta }}$. These fields ${\displaystyle g_{\theta }\left(\mathit{x}\right)}$ are simply summed over all visited angles ${\displaystyle \theta }$, producing the sought initial image, ${\displaystyle f\left(\mathit{x}\right)}$

$$f\left(\mathit{x}\right)=\sum _{\theta =1}^{N_{\theta }}{g}_{\theta }\left(\mathit{x}\right)$$

(2)

Iterative reconstructions

Other reconstruction methods have received much attention, namely iterative algebraic approaches which tolerate deviations from the ideal conditions of the previous Fourier space reconstruction such as for instance having access to a continuous range of angles, covering the entire half (or full) rotation. Those methods exploit the linear structure of the problem to solve, but for computational efficiency, they avoid the writing of the linear system. They are based on the minimization over volumes, ${\displaystyle \psi \left(\mathit{x}\right)}$, of the functional, ${\displaystyle {\mathsf{\Gamma }}_{\mathrm{ART}}\left[\psi \right]}$, equal to the quadratic norm of the difference ${\displaystyle \rho (\mathit{t},\theta )}$ between the acquired projections and the projected reconstructed volume

$$\begin{array}{ccc}\hfill {\mathsf{\Gamma }}_{\mathrm{ART}}\left[\psi \right]& =& \hfill {\displaystyle \sum _{\mathit{r},\theta }}{\parallel \rho (\mathit{t},\theta )\parallel }^2\hfill \\ \hfill \rho (\mathit{t},\theta )& =& \hfill p(\mathit{r},\theta )-{\mathcal{R}}_{\theta }\left[\psi \left(\mathit{x}\right)\right]\hfill \end{array}$$

(3)

then

$$f=\underset{\psi }{\mathrm{Argmin}}{\mathsf{\Gamma }}_{\mathrm{ART}}\left[\psi \right]$$

(4)

Additional prior information may easily be added to this functional through regularization, in order to compensate limited angle range for projections, or coarse sampling for example. This generally leads to better quality reconstructions than FBP algorithms at the expense of a higher computational cost.

To solve this huge linear inverse problem, ART algorithms essentially consist of iterative updates of the volume. Successively visiting each angle, the projection of the volume is compared with the acquired one. The difference is back projected and used to correct the volume (sometimes multiplied by a damping coefficient, not considered in our case). Faster convergence rate is observed when angles are not sampled in consecutive order but rather with a large difference between successive angles. This can be achieved for instance with a permutation of the angle order. A convergence criterion on the functional value can be used to stop the number of iterations (${\displaystyle {\mathsf{\Gamma }}_{\mathrm{ART}}\left[f\right]<ϵ}$), with ${\displaystyle ϵ}$, a threshold value with respect to noise and artifact acquisition. Generally a few iterations (${\displaystyle N_{\mathrm{ART}}}$) are required for convergence. The algorithm for this method is detailed in Algorithm 1.

Algorithm 1: Standard algebraic reconstruction procedure, $f\leftarrow \mathrm{ART}\left(p\right)$.

During the reconstruction procedure, additional information, defined as constraints, can be added. Those regularizations allow the reconstruction of high quality volume with a few or missing angles, noisy projections and artifacts etc. This may come from prior knowledge of the different phases of the sample (as DART algorithms proposed by [45], reconstruction with binary images [46], Total Variation [47]), dictionary learning [48], etc. However, because those regularizations are independent from the following proposed reconstruction with motion compensation, it is not considered hereafter apart from the positivity constraint ${\displaystyle f\left(\mathit{x}\right)\leftarrow max\left(\psi \right(\mathit{x}),0)}$.

3. Data Driven Reconstruction of Non-Rigid Samples

It is proposed to study a specimen that moves during the acquisition with a space/time displacement field ${\displaystyle \mathit{u}(\mathit{x},t)}$ such that, at any time, the sample is expressed with respect to a reference state ${\displaystyle f(\mathit{x}+\mathit{u}(\mathit{x},t\left)\right)}$.

For a still object ${\displaystyle f}$ and ${\displaystyle p}$ are bijectively related to each other through the inverse Radon transform. The introduction of motion causes a non-trivial nullspace and thus the loss of bijectivity. The reconstruction of the volume from the previously introduced algorithms ( i.e., ignoring motion) leads to a low quality, blurry, volume.

It is to be noted that the FBP reconstruction procedure has been extended to take motion into account in [35,36,40] The driving idea is to apply the back-projection step on the currently deformed geometry of the to-be-reconstructed sample, or equivalently to transport the back-projection onto the initial geometry, unwarping the motion, so that the X-ray beam would then follow non-straight paths. In Ref. [40], the motion is estimated from the registration of two reconstructions of the volume at different instants of time and linear interpolation.

$$f\left(\mathit{x}\right)=\sum _{t=1}^{N_{\theta }}{g}_{\theta \left(t\right)}(\mathit{x}-\mathit{u}(\mathit{x},t))$$

(5)

Because this approach requires different volumes to estimate the displacement field, it is not suited when the motion is fast and when only a single scan can be acquired. Moreover, it is difficult to estimate a quality criterion but visual on the reconstructed volumes thus on the measured kinematics.

A recently developed Digital Volume Correlation (DVC) procedure called Projection-based DVC [26] allows to identify the 4D [49] (space-time) displacement field of sample from an initially reconstructed volume and its moving projections. An extension of this method has been applied to an online calibration ( i.e., calibrated from the sample motion during the scanning process) of the tomograph [23]. An initial (blurry) reconstruction was performed from a set of initial parameters. The comparison between the projection of the blurry sample and the acquired projections is, in addition to the acquisition noise and artifacts, the signature the erroneous projection geometry parameters that can be identified and corrected. The sample could be re-positioned for each angle by a rigid body motion. Because the motion was simple and of low amplitude, the correction could be applied on the sinogram itself leading to very significant improvement on the quality of the reconstruction. However, more complex displacements, or larger amplitudes (involving larger displacement variations perpendicular to the ray) would render the corrections on the projection inaccessible.

It is proposed to introduce here a new two-step algorithm based on ART reconstruction on the one hand and P-DVC on the other hand to identify both a complex and large displacement field and volume texture with a single scan performed on a moving and deforming object. The ART functional is naturally extended to account for the motion as

$${\mathsf{\Gamma }}_{\mathrm{motion}-\mathrm{ART}}[\psi ,\mathit{v}]=\sum _{\mathit{r},t}{\parallel {\mathcal{R}}_{\theta \left(t\right)}\left[\psi (\mathit{x}+\mathit{v}(\mathit{x},t))\right]-p(\mathit{r},t)\parallel }^2$$

(6)

where the summation over time extends over the ${\displaystyle N_t}$ acquired projections (and not just a full rotation) then

$$(f,\mathit{u})=\underset{\psi ,\mathit{v}}{\mathrm{Argmin}}{\mathsf{\Gamma }}_{\mathrm{motion}-\mathrm{ART}}[\psi \left(\mathit{x}\right),\mathit{v}(\mathit{x},t)]$$

(7)

The updating procedure (indexed by ${\displaystyle l}$) is split into two parts that are repeated alternatively:

• A volume reconstruction from an iterative dynamic ART algorithm assuming a known motion (described in Section 3.1);
• An update of the motion from P-DVC with a given reconstructed sample (described in Section 3.2).

However, as such, this procedure does not tolerate large displacement amplitudes. To increase the robustness and fast convergence, a multi-scale approach is coupled to the previous two-step procedure, resolving first the large scale features of both microstructure and motion, and progressively enriching the description with finer details. The complete multi-scale procedure is described in Section 3.3.

3.1. Dynamic Reconstruction

The dynamic reconstruction used in this article is an extension of the standard ART algorithm, and will follow the same structure as Algorithm 1. Considering the inner “for” loop, at time ${\displaystyle t}$ (and angle ${\displaystyle \theta \left(t\right)}$, the volume is warped with the measured displacement field

$${\tilde{f}}^{(n-1)}(\mathit{x},t)=f^{(n-1)}(\mathit{x}+\mathit{u}(\mathit{x},t))$$

(8)

(initially ${\displaystyle \mathit{u}(\mathit{x},t)=\mathbf{0}}$). The computed projection of ${\displaystyle {\tilde{f}}^{(n-1)}}$ along ${\displaystyle \theta \left(t\right)}$ is compared with the recorded projection and the residual ( i.e., their difference)

$${\rho }^{\left(n\right)}(\mathit{r},t)=p(\mathit{r},t)-{\mathcal{R}}_{\theta \left(t\right)}{\tilde{f}}^{(n-1)}(\mathit{x},t)$$

(9)

is normalized and back-projected ${\displaystyle \Delta {\tilde{f}}^{\left(n\right)}={\mathcal{B}}_{\theta \left(t\right)}\left[{\rho }^{\left(n\right)}(\mathit{r},t)\right]}$.

Finally the correction term is unwarped to the frame of the undeformed state, ${\displaystyle {\widehat{\Delta f}}^{\left(n\right)}\left(\mathit{x}\right)=\Delta {\tilde{f}}^{\left(n\right)}(\mathit{x}-\mathit{u})}$ so that it matches the reference configuration and it is added to the volume, ${\displaystyle f^{\left(n\right)}=f^{(n-1)}+{\widehat{\Delta f}}^{\left(n\right)}}$. Let us emphasize that theoretically, ${\displaystyle {\widehat{\Delta f}}^{\left(n\right)}}$ should have been defined implicitly as obeying ${\displaystyle {\widehat{\Delta f}}^{\left(n\right)}(\mathit{x}+\mathit{u})=\Delta {\tilde{f}}^{\left(n\right)}\left(\mathit{x}\right)}$. The two expressions are equivalent only for small strains and rotations, otherwise the unwarping should involve the Eulerian rather than the Lagrangian displacement, and one can be computed from the other. Let us also note that for not too large strains and rotations, ignoring the difference between Eulerian and Lagrangian displacements simply slows down the convergence, but the final solution is not affected. In the present case, the choice was made to use the Eulerian registration to achieve the convergence for engineering strains as large as 20%. A convergence criterion has to be chosen as in the ART procedure. Nevertheless, the criterion based on the functional value cannot be used in this case as the reconstruction is unperfect. A convergence criterion based on the variation of the functional or a maximum number of iteration ${\displaystyle N_{\mathrm{DynART}}}$ can be set. The procedure is described in Algorithm 2.

Algorithm 2: Proposed motion-corrected algebraic reconstruction procedure, $f\leftarrow \mathrm{DynART}(p,\mathit{u})$.

As earlier mentioned, additional priors can be added in this procedure at the end of the inner “for” loop. In the following, only a positivity constraint for ${\displaystyle f}$ is added at each iteration.

3.2. Motion Identification

The full procedure is a staggered two-step process where alternatively the volume is reconstructed from a frozen displacement, and the motion is identified from a frozen estimate of the microstructure. The second step is described now.

At step ${\displaystyle l}$, the reconstructed volume, ${\displaystyle f_l\left(\mathit{x}\right)}$, although imperfect, is now considered as reliable. The projected residual fields ${\displaystyle {\rho }_l(\mathit{r},t)}$ (computed at the end of the previous procedure when the volume is no more updated) contains patterns that are the signature of an incomplete motion correction. For the identification of the displacement field, the functional for a given ${\displaystyle f}$ can be linearized around the previously identified displacement field ${\displaystyle {\mathit{u}}_l={\mathit{u}}_{l-1}+\delta \mathit{u}}$

$$\delta \mathit{u}=\underset{\delta \mathit{v}}{\mathrm{Argmin}}\sum _{\mathit{r},t}{\parallel {\mathcal{R}}_{\theta \left(t\right)}[\nabla {\tilde{f}}_l(\mathit{x},t)\delta \mathit{v}(\mathit{x},t)]-{\rho }_l(\mathit{r},t)\parallel }^2$$

(10)

For a better conditioning, the space and time dependencies of motion may be regularized, either using “weak regularization”, with a penalty on spatial or temporal rapid variation of the displacement field to be added to the above cost function, or reverting to “strong regularization” by choosing a parametrization space composed of smooth functions of space and time. At this regularization step, any additional information pertaining to the experiment ( e.g., synchronous measurements from sensors of different modalities such as force, pressure or temperature measurements, cardiac phase etc.) can be incorporated in the kinematic model through functional dependencies on such parameters. Qualitative features may also be incorporated, for instance, the sudden occurrence of a crack, may be accounted for by allowing a temporal discontinuity in concerned degrees of freedom for the kinematics.

The chosen reduced basis is composed respectively of ${\displaystyle N_{\tau }}$ time functions, ${\displaystyle {\phi }_i\left(t\right)}$, and ${\displaystyle N_s}$ vector spatial shape functions ${\displaystyle {\mathsf{\Phi }}_j\left(\mathit{x}\right)}$ such as

$$\mathit{u}(\mathit{x},t)=\sum _{i=1}^{N_{\tau }}\sum _{j=1}^{N_s}{\alpha }_{ij}{\phi }_i\left(t\right){\mathsf{\Phi }}_j\left(\mathit{x}\right)$$

(11)

with ${\displaystyle {\alpha }_{ij}}$ the time and space amplitudes that weight the basis functions. Setting ${\displaystyle {\phi }_i\left(0\right)=0}$, the reference state is at initial time or angle ${\displaystyle \theta =0}$, ${\displaystyle \mathit{u}(\mathit{x},0)=\mathbf{0}}$.

The minimization of the functional ${\displaystyle {\mathsf{\Gamma }}_{\mathrm{motion}-\mathrm{ART}}}$, Equation (6), with respect to the displacement parameters ${\displaystyle \delta \mathbf{\alpha }}$ is performed using Newton’s descent method. This procedure requires the computation of the advected image gradient and Hessian of ${\displaystyle {\mathsf{\Gamma }}_{\mathrm{motion}-\mathrm{ART}}}$. They are built from the projected sensitivities

$$S_{ij}(\mathit{r},t)={\displaystyle \frac{\partial {\mathcal{R}}_{\theta \left(t\right)}\tilde{f}(\mathit{x},t)}{\partial {\alpha }_{ij}}}={\phi }_i\left(t\right){\mathcal{R}}_{\theta \left(t\right)}[{\mathsf{\Phi }}_j\left(\mathit{x}\right)\nabla \tilde{f}(\mathit{x},t)]$$

(12)

Numerically, the sensitivities are computed from finite differences. The Hessian matrix and second member built from those sensitivities is

$$\begin{array}{ccc}\hfill H_{ijkl}& =& \hfill \sum _{\mathit{r},t}{S}_{ij}(\mathit{r},t)S_{kl}(\mathit{r},t)\hfill \end{array}$$

(13)

$$\begin{array}{ccc}\hfill b_{ij}& =& \hfill \sum _{\mathit{r},t}\rho (\mathit{r},t)S_{ij}(\mathit{r},t)\hfill \end{array}$$

(14)

thus the vector of displacement amplitude correction ${\displaystyle \mathbf{\delta }\mathbf{\alpha }}$ is the solution of the linear system

$$\left[\mathit{H}\right]\mathbf{\delta }\mathbf{\alpha }=\mathit{b}$$

(15)

from which the displacement is updated. This procedure is repeated until the projection residual is no longer decreasing. Algorithm 3 summarizes the determination of the displacement field. In the following test cases, a single iteration in this algorithm is performed before updating the volume.

Algorithm 3: Displacement identification procedure, $\mathit{u}\leftarrow \mathrm{PDVC}(f_{\lambda },p,{\mathit{u}}_0)$.

3.3. Multi-Scale Approach

If displacement magnitude is bounded by a length scale ${\displaystyle \lambda }$, one expects that the reconstruction is fair at a scale larger than ${\displaystyle \lambda }$. Hence, if the original image is convoluted with a Gaussian of width ${\displaystyle \lambda }$, it should well match its sinogram. One convenient property of the projection is that the projection of the convoluted image is the convolution of the original projection with a Gaussian of the same width. However, because of motion, this matching is not perfect but just fair. It means that one may estimate a better match by treating the deformation as a slight perturbation.

More precisely, the recorded projections are convoluted by the Gaussian of width ${\displaystyle \lambda }$,

$${\breve{p}}_{\lambda }(\mathit{r},\theta )=\sum _{{\mathit{r}}^{\prime }}{G}_{\lambda }\left({\mathit{r}}^{\prime }\right)p(\mathit{r}-{\mathit{r}}^{\prime },\theta )$$

(16)

where, ${\displaystyle G_{\lambda }\left(\mathit{r}\right)=1/\left(2\pi {\lambda }^2\right){exp(-|\mathit{r}|}^2/\left(2{\lambda }^2\right))}$.

Using the progressively identified displacement field, a more accurate determination of ${\displaystyle f}$ can be achieved using the above described reconstruction. Because a large part of the displacement is expected to be captured in ${\displaystyle \mathit{u}}$, the idea is to repeat the above procedure but with a smaller Gaussian filter, namely cutting down ${\displaystyle \lambda }$ by a factor of two. Thus at each iteration, the displacement correction being smaller and smaller, convergence to the actual displacement field is expected. A convergence criterion is chosen on the norm of the residual variation or on the norm of the displacement correction.

The summary of the complete procedure is described in Algorithm 4.

Algorithm 4: Complete dynamic tomography procedure, $(f,\mathit{u})\leftarrow \mathrm{DynTomo}\left(p\right)$.

4. Test Case

Two numerical test cases are proposed to validate the procedure. To build the input data, two geometries are chosen, and two kinematics (one per case) deformed and projected at all considered angle. The obtained projections are then corrupted by a white Gaussian noise (standard deviation of ${\displaystyle 1\%}$ of the gray level dynamic of the projections and are used as the virtual experimental inputs for our procedure.

Both examples are carried out on ${\displaystyle 512\times 512}$ pixel images. The beam is parallel, and only ${\displaystyle N_{\theta }=300}$ projections are acquired over a single 360° rotation.

• The first application corresponds to a moving Shepp-Logan phantom with large displacement magnitude (up to 37 pixels) and large engineering strains (27${\displaystyle \%}$). Large strains are chosen here in order to highlight the robustness of the proposed procedure as compared with previously studied examples where strains were about 1% [40].
• The second test is performed on a checkerboard with smaller displacements but a more complex time evolution composed of two separated modes: a steady drift superimposed to a high frequency pulsating motion.

In both test cases, the displacement bases chosen for the inverse problem were similar to the ones used for performing the direct problem, so that no additional model error (apart from noise) is introduced. The space functions ${\displaystyle \mathsf{\Phi }\left(\mathit{x}\right)}$ are composed of four C4 mesh elements (4-node square elements with bilinear interpolations). The space basis ${\displaystyle N_s}$ is hence composed of 18 degrees of freedom.

4.1. Shepp-Logan Phantom Case

In this test case, the Shepp-Logan phantom is used and deformed up to ${\displaystyle 27\%}$. For this test case, a single time evolution (linear drift in time) is applied. The imposed displacement field can be written

$$\mathit{v}(\mathit{x},\theta )=\theta /N_{\theta }·\mathit{v}\left(\mathit{x}\right)$$

(17)

The nine nodal displacements are given in

Table 1. Applied nodal displacements for ${\displaystyle \mathit{V}\left(\mathit{x}\right)}$ in pixels, in ${\displaystyle \mathit{x}}$ (left) and ${\displaystyle y}$ (right).

y\x	66	256	446		y\x	66	256	446
446	15	0	4		446	4	0	1
256	19	${\displaystyle -4}$	19		256	26	30	0
66	26	10	15		66	0	24	34

in ${\displaystyle \mathit{x}}$ and ${\displaystyle y}$ directions.

The reference and deformed phantoms are shown Figure 1. The maximum displacement amplitude is 37 pixels. The first reconstruction of the image (standard ART procedure), presented Figure 2a is very blurry. Some parts of the phantom are split in two. The initial projected residual fields are very high and stresses that the reconstruction is not properly performed.

Before using the proposed procedure, the multi-scale procedure presented Section 3.3 is applied to the projections to willingly blur the reconstruction. After 60 iterations ( i.e., volume updates), the displacement field has converged. The corrected reconstructed volume is presented Figure 3. The edges are sharp and the gray level amplitudes are correct. The projected residual fields (true metric of our procedure) is mostly composed of the white Gaussian noise meaning that the proposed procedure has been successful.

The displacement error computed on the nodal values displays a standard deviation of 3.10 pixel. This result validates the procedure.

As a last validation of the phantom reconstruction quality, the reconstruction is compared to the reference volume ${\displaystyle f}$. It is shown in Figure 4 that the reconstructed shape and positioning is very good. The final difference displays a “ghost” of the phantom that points out a small intensity error that does not appear in the residual fields.

4.2. Pulsating Checkerboard Case

This second test case is here based on a checkerboard composed of ${\displaystyle 8\times 8}$ squares of ${\displaystyle 35\times 35}$ pixels each. This square shaped pattern is chosen to exhibit reconstruction errors very clearly since sharp and straight boundaries are very easily detected, and hence the visual perception is a very severe test.

In this example, the imposed (supposed unknown) displacement field is composed of the sum of two parts:

• A pulsating motion: Temporally, a shifted cosine function (${\displaystyle 1-cos\dots )}$) (obeying the constraint of being null at time 0) evolution with a non-integer number of periods to avoid symmetry (here ${\displaystyle 2.35}$ periods during the full-rotation scan). Spatially, the displacement field is a centered dilatation/contraction;
• A linear drift in time for all nodes with random directions and amplitudes.

The applied displacement field can be written

$$\mathit{u}(\mathit{x},\theta )=\left(1-cos(2.35·2\pi ·\theta /N_{\theta })\right)·{\mathit{V}}_1\left(\mathit{x}\right)+\theta /N_{\theta }·{\mathit{V}}_2\left(\mathit{x}\right)$$

(18)

with the nodal values presented in

Table 2. Applied nodal displacements for ${\displaystyle {\mathit{V}}_1\left(\mathit{x}\right)}$ in pixels, in ${\displaystyle \mathit{x}}$ (left) and ${\displaystyle y}$ (right).

y\x	50	256	462		y\x	50	256	462
462	28	0	${\displaystyle -28}$		462	${\displaystyle -28}$	0	${\displaystyle -28}$
256	0	0	0		256	0	0	0
50	28	0	${\displaystyle -28}$		50	28	0	28

and

Table 3. Applied nodal displacements for ${\displaystyle {\mathit{V}}_2\left(\mathit{x}\right)}$ in pixels, in ${\displaystyle \mathit{x}}$ (left) and ${\displaystyle y}$ (right).

y\x	50	256	462		y\x	50	256	462
462	-13	22	${\displaystyle -17}$		462	${\displaystyle 17}$	${\displaystyle -22}$	9
256	${\displaystyle -22}$	34	17		256	22	22	${\displaystyle -17}$
50	17	−30	22		50	${\displaystyle -22}$	17	0

.

The nodal displacement vectors ${\displaystyle {\mathit{V}}_1\left(\mathit{x}\right)}$ and ${\displaystyle {\mathit{V}}_2\left(\mathit{x}\right)}$ are shown in Figure 5a. The reference image, the deformed one at the end of the scan and the chosen C4 mesh are shown in Figure 5. The maximum strain is about 25%.

Because of the large motion amplitude, the initial reconstruction ( i.e., obtained from a standard ART procedure for which ${\displaystyle \mathit{u}(\mathit{x},\theta )=\mathbf{0}}$) is fuzzy and its quality is very poor as can be judged from Figure 6a. The projection of this blurred volume is compared with the initial projection to generate the initial projected residual fields ${\displaystyle \rho (\mathit{r},t)}$ (see Figure 6b).

After 60 iterations, ( i.e., 60 updates of the reconstruction) — performed in approximately 2 h—the 38 degrees of freedom that drive the displacement field (18 spatial times 2 temporal degrees of freedom) have converged to a steady value. A small standard deviation of the displacement field error with respect to the prescribed displacement of less than 1.2 pixel remains at the end. Considering the large imposed motion amplitude, the estimated kinematics is deemed quite satisfactory.

The final reconstruction and projected residuals are shown in Figure 7. The reconstruction has sharp edges and its constituting squares have been correctly reconstructed. Zooms in the initial and corrected specimen are shown in Figure 8. The projected residual field, where all features of the initial sinogram have been completely erased, and only white Gaussian noise remains, means that the reconstruction has been quite successful.

To correctly appreciate the quality of the achieved volume, a difference with the initial perfect one is shown Figure 9. This difference highlights a perfect positioning of the reconstruction, and only slight discrepancies of the gray level intensity on the bright squares are visible.

The full procedure (i.e., 60 complete iterations for 512 ${\displaystyle \times }$ 512 pixel images) is performed in approximately 1 h. What takes time (and iterations) is the computation of the sensitivities that requires the deformed volumes over the entire range of time. However, it is worth mentioning that the code has not been optimized, since only a proof of concept was aimed at.

5. Discussion and Conclusions

An innovative algorithm is presented to perform simultaneously a dynamic reconstruction of a moving sample with the identification of the full 2D space and time displacement field. The method is derived from Algebraic Reconstruction Techniques coupled with Projection based Digital Volume Correlation. The iterative algorithm is based on two steps:

• For a given displacement field, a dynamic algebraic reconstruction algorithm is proposed. Each iteration of the procedure consists in comparing the acquired projection with the projected warped volume (deformed with current displacement field). The projected residual is backprojected, unwarped to match the reference space and added to the volume;
• For a given reference volume, a P-DVC analysis allows the displacement field to be identified. The projection of the (unperfect) warped volume is compared with the acquired projections. The residual can be read as motion using the computed sensitivity fields. An update of the displacement field is then performed.

A multiscale procedure has been proposed as an essential ingredient to properly correct large displacements. The acquired projections are first convoluted with a Gaussian kernel of large width (low pass filter) to increase its correlation length and capture large corrections from the linearized P-DVC functional. The Gaussian filter is then progressively reduced, following the residual norm evolution, to identify finer details.

The post treatment procedure, that exploits the same data as a standard acquisition (same number of projections and standard projection operator), has been tested with two challenging numerical examples (with large displacements and strains). The first is a Shepp Logan phantom with large displacement fields (up to 1/4 of the phantom length). The second is a checkerboard with a pulsating motion in time. Both examples are corrupted by a white Gaussian noise that probes the robustness with respect to the acquisition noise. The two applications show a nearly perfect identification of the displacement field and dynamic reconstruction. Performed with a parallel projection algorithm for simplicity, the exact same method can be applied with any projection model.

The proposed dynamic reconstruction algorithm has been devised as an extension of the ART algorithms. It is convenient with those approaches to include in the process an a priori knowledge of the scanned specimen (assumption on the gray levels, its variations, the number of phases, its sizes, etc.). Many different regularization have been proposed in the literature that enable to obtain high quality reconstructions, with less artifacts, from less projections or missing angles, etc. Because those regularizations are independent of the current algorithms, it was chosen not to implement them and focus on the proposed method performances without any ’additional help’. Nevertheless, they are fully compatible with the proposed approach and can be implemented in a transparent fashion. When aiming to perform ultra-fast acquisitions with a few angles, they would certainly be very precious to accelerate convergence, and improve reconstruction quality.

In the proposed examples, the optical flow was kept constant. Some applications may require to include a gray level variation model. A perspective of this work could be the scan of in situ mechanical test with high strains, the identified deformation could be used to correct for absorption evolution of the material considering a constant beam intensity.

The proposed procedure shows performances that can be beneficial to numerous fields. The clear reconstruction of the moving sample allows for qualitative and quantitative analyses:

• Combined with Digital Volume Correlation [50] between well reconstructed volumes;
• Combined with image segmentation for diagnosis from radiology;
• Combined with ultra-fast tomography acquisition as recently available from some synchrotron beam-lines [6,51].

This is key for data assimilation [52] and model identification and validation in material science [53] with CT-scan as already developed with MRI [54].