Next Article in Journal
Preliminary Tests and Results Concerning Integration of Sentinel-2 and Landsat-8 OLI for Crop Monitoring
Next Article in Special Issue
Illusion and Illusoriness of Color and Coloration
Previous Article in Journal
Detection and Classification of Land Crude Oil Spills Using Color Segmentation and Texture Analysis
Previous Article in Special Issue
The Academy Color Encoding System (ACES): A Professional Color-Management Framework for Production, Post-Production and Archival of Still and Motion Pictures
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Exemplar-Based Face Colorization Using Image Morphing

1
Department of Mathematics, Technische Universität Kaiserslautern, Paul-Ehrlich-Str. 31, 67663 Kaiserslautern, Germany
2
Laboratoire Lorrain de Recherche en Informatique et ses Applications, UMR CNRS 7503, Université de Lorraine, INRIA projet Magrit, 54500 Vandoeuvre-les-Nancy, France
3
Fraunhofer ITWM, Fraunhofer-Platz 1, 67663 Kaiserslautern, Germany
*
Author to whom correspondence should be addressed.
J. Imaging 2017, 3(4), 48; https://doi.org/10.3390/jimaging3040048
Submission received: 30 May 2017 / Revised: 18 September 2017 / Accepted: 19 October 2017 / Published: 31 October 2017
(This article belongs to the Special Issue Color Image Processing)

Abstract

:
Colorization of gray-scale images relies on prior color information. Exemplar-based methods use a color image as source of such information. Then the colors of the source image are transferred to the gray-scale target image. In the literature, this transfer is mainly guided by texture descriptors. Face images usually contain few texture so that the common approaches frequently fail. In this paper, we propose a new method taking the geometric structure of the images rather their texture into account such that it is more reliable for faces. Our approach is based on image morphing and relies on the YUV color space. First, a correspondence mapping between the luminance Y channel of the color source image and the gray-scale target image is computed. This mapping is based on the time discrete metamorphosis model suggested by Berkels, Effland and Rumpf. We provide a new finite difference approach for the numerical computation of the mapping. Then, the chrominance U,V channels of the source image are transferred via this correspondence map to the target image. A possible postprocessing step by a variational model is developed to further improve the results. To keep the contrast special attention is paid to make the postprocessing unbiased. Our numerical experiments show that our morphing based approach clearly outperforms state-of-the-art methods.

1. Introduction

Colorization consists in adding color information to a gray-scale images. This technique is used for instance by the cinema industry to make old productions more attractive. As usual we can consider a gray-scale image as luminance channel Y of an RGB image [1,2,3,4]. The Y channel is defined as a weighted average of the RGB channels, see, e.g., [5]:
Y = 0.299 R + 0.587 G + 0.114 B .
In addition to the luminance channel, two chrominance channels, called U and V, enable to recover the RGB image. Recovering an RGB image from the luminance channel alone is an ill-posed problem and requires additional information [3,6]. This information is provided in the literature in two different ways, namely by manual [1,4,7] or exemplar-based [3,6,8] methods. In the first one the user augments the image by some color strokes as basis for the algorithm to compute the color of each pixel. The colorization of a complex scene by a manual prior can be a tedious work for the user [2]. In the second approach a color image is used as a source of information. Here the results strongly depend on the choice of the image. Therefore it is often called semi-automatic [1].
In this paper, we focus on the exemplar-based methods. A common back-bones of these techniques is the matching of the images. First, the color source image is transformed to a gray-scale image, referred to as template which is compared with the input gray-scale image called target. The main issue of the exemplar-based colorization consists in matching the pixels of the template and the target gray-scale images. The basic hypothesis in the literature is that the color content is similar in similar texture patches [8]. Then the main challenge is the choice of appropriate texture descriptors.
In the literature, the exemplar-based methods come along with various matching procedures. For instance, Gupta et al. [6] use SURF and Gabor features, Irony et al. [2] use DCT, Welsh et al. [8] use standard-deviation, etc. Some of them are done after an automatic segmentation, whereas others use local information. The seminal paper on exemplar-based colorization by Welsh et al. [8] was inspired by the texture synthesis of Efros and Leung [9] and uses basic descriptors for image patches (intensity and standard-deviation) to describe the local texture. Pierre et al. [3] proposed an exemplar-based framework based on various metrics between patches to produce a couple of colorization results. Then a variational model with total variation like regularization is applied to choose between the different results in one pixel with a spatial regularity assumption. The approach of Irony et al. [2] is built on the segmentation of the images by a mean-shift algorithm. The matching between segments of the images is computed from DCT descriptors which analyse the textures. The method of Gupta et al. [6] is rather similar. Here, an over-segmentation (SLIC, see, e.g., [10]) is used instead of the mean-shift algorithm. The comparison between textures in done by SURF and Gabor features. Chen et al. [11] proposed a segmentation approach based on Bayesian image matching which can also deal with smooth images including faces. The authors pointed out that the colorization of faces is a particular hard problem. However, their approach uses a manual matching between objects to skirt the problem of smooth parts. Charpiat et al. [12] ensured spatial coherency without segmenting, but their method involves many complex steps. The texture discrimination is mainly based on SURF descriptors. In the method of Chia et al. [13], the user has manually to segment and label the objects and the algorithm finds similar segments in a set of images available in the internet. Recently a convolutional neural network (CNN) has been used for colorization by Zhang et al. [14] with promising results. Here the colorization is computed from a local description of the image. However, no regularization is applied to ensure a spatial coherence. This produces ”halo effects” near strong contours. All the described methods efficiently distinguish textures and possibly correct them with variational approaches, but fail when similar textures have to be colorized with different colors. This case arises naturally for face images. Here the smooth skin is considered nearly as a constant part. Thus, as we show in this paper, when the target image contains constant parts outside the face, the texture-based methods fail.
In this paper, we propose a new technique for the colorization of face images guided by image morphing. Our framework relies on the hypothesis that the global shape of faces is similar. The matching of the two images is performed by computing a morphing map between the target and the template image. Image morphing is a generic term for smooth image transition which is an old problem in image processing and computer vision. For example in feature based morphing only specific features are mapped to each other and the whole deformation is then calculated by interpolation. Such method [15] was used by the film industry for example in the movie Willow. For an overview of this and similar techniques see also [16,17]. A special kind of image morphing, the so-called metamorphosis was proposed by Miller, Trouvé and Younes [18,19,20]. The metamorphosis model can be considered as an extension of the flow of diffeomorphism model and its large deformation diffeomorphic metric mapping framework [21,22,23,24,25] in which each image pixel is transported along a flow determined by a diffeomorphism. As an extension the metamorphosis model allows the variation of image intensities along trajectories of the flow. Shooting methods for the metamorphosis model were developed e.g., in [26]. For a metamorphosis regression model and corresponding shooting methods we refer to [27]. A comprehensive overview over the topic is given in the book [28] as well as the review article [29], for a historic account see also [30]. In this paper, we build up on the time discrete metamorphosis model by Berkels, Effland and Rumpf [31]. In contrast to these authors we apply a finite difference approach for the computation of the morphing maps. This involves the solution of a chain of registration problems in each iteration step. There exists a rich literature on registration problems, e.g., [32,33,34,35], and we refer to the books of Modersitzki [36,37] for an overview.
Having the morphing map available, the chrominance channels can be transported by this map to the target image, while preserving its luminance channel. This gives very good results and outperforms state-of-the-art methods. For some images we can further improve the quality by applying a variational post-processing step. Our variational model incorporates a total variation like regularization term which takes the edges of the target image into account. This was also proposed by one of the authors of this paper in [3]. The method is accomplished by adapting a procedure of Deledalle et al. [38] to keep the contrast of the otherwise biased total variation method.
The outline of the paper is as follows: in Section 2 we sketch the ideas of the morphing approach. In particular, we show how the morphing map is computed with an alternating minimization algorithm and describe our finite difference approach. Details of the computation are shifted to the Appendix. Section 3 deals with the color transfer. Having the morphing map at hand we explain the transfer of the chrominance values. In particular we address the necessary scalings. Sometimes it is useful to apply a variational model with a modified total variation regularization as post-processing step to remove possible spatial inconsistencies. Such a procedure is developed in Section 4. Numerical experiments demonstrate the very good performance of our algorithm in Section 5. The paper ends with conclusions in Section 6.

2. Image Morphing

Our colorization method is based on the time discrete image metamorphosis model [31]. We briefly explain the model used also in the numerical part of [31] in Section 2.1 and describe our numerical realization for digital images by a finite difference approach in Section 2.2. Note that in [31] a finite element approach for the spatial discretization was proposed without a detailed description. Finite element methods are highly flexible and can be also applied, e.g., for shape metamorphosis. However, having the rectangular structure of the image grid in mind, we propose to use finite differences for the spatial discretization. Then, in Step 1 of our alternating algorithm, we can build up on registration methods proposed e.g., by Haber and Modersitzki [34].

2.1. Morphing Model Based on [31]

Let Ω R 2 be an open, bounded domain with Lipschitz continuous boundary. We are given a gray-value template image I temp : Ω R and a target image I tar : Ω R which are supposed to be continuously differentiable and compactly supported. For K 2 set
I 0 : = I temp , I K : = I tar .
In our application, the template image will be the luminance channel of the color source image and the target image the gray-scale image we want to colorize. We want to find a sequence of K 1 images I together with a sequence of mappings φ on Ω , i.e.,
I : = ( I 1 , , I K 1 ) , φ : = ( φ 1 , , φ K ) ,
such that
I k I k 1 φ k for   all x Ω ,
see Figure 1, and the deformations φ k have a small linearized elastic potential defined below. To this end, we suppose for k = 1 , , K that φ k is related to the displacement v k by
φ k ( x ) = x v k ( x ) = x v k , 1 ( x ) y v k , 2 ( x ) , x = ( x , y ) T Ω ,
and set v ( v 1 , , v K ) . The (Cauchy) strain tensor of the displacement v = ( v 1 , v 2 ) T : Ω R 2 is defined by
ε ( v ) : = 1 2 ( v + v T ) = x v 1 1 2 ( y v 1 + x v 2 ) 1 2 ( y v 1 + x v 2 ) y v 2 ,
where v denotes the Jacobian of v. The linearized elastic potential is given by
S ( v ) : = Ω μ trace ε T ( v ) ε ( v ) + λ 2 trace ε ( v ) 2 d x ,
where μ , λ > 0 . Then we want to minimize
J ( I , v ) : = k = 1 K Ω | I k I k 1 φ k | 2 + S ( v k ) d x , subject   to I 0 = I temp , I K = I tar .
This functional was used in the numerical part of [31]. Note that the term S ( v ) may be accomplished by a higher order derivative
Ω | D m v k ( x ) | 2 d x , m > 2 ,
which ensures in the time continuous setting that φ is indeed a diffeomorphism, see [22]. In the time discrete setting the additional term does not guaranty that φ k , k = 1 , , K , is a diffeomorphism. Hence we do not include (4) into our functional (3), as the linearized elastic potential S is sufficient for our purposes.
The minimizer ( I , v ) of the functional provides us with both a sequence of images I along the approximate geodesic path and a sequence of displacements v managing the transport of the gray values through this image sequence. For finding a minimizer of (3) we alternate the minimization over I and v :
  • Fixing I and minimizing over v leads to the following K single registration problems:
    arg min v k J I ( v k ) : = Ω | I k I k 1 φ k | 2 + S ( v k ) d x , k = 1 , , K ,
    where φ k is related to v k by (1).
  • Fixing v , resp., φ leads to solving the following image sequence problem
    arg min I J φ ( I ) : = k = 1 K Ω | I k I k 1 φ k | 2 d x .
    This can be done via the linear system of equations arising from Euler-Lagrange equation of the functional which we describe in the Appendix A.
Note that Miller et al. [39] considered for two given image sequences I : = ( I j ) j = 1 N and J : = ( J j ) j = 1 N (e.g., related to corresponding image patches) the registration problem
arg min v Ω j = 1 N | I j J j φ | 2 + S ( v ) d x ,
where φ is related to v by (1). In contrast to our problem these authors search for the same mapping v and the N template-target pairs are known.

2.2. Space Discrete Morphing Model

When dealing with digital images we have to work in a spatially discrete setting. Let
G p : = { 1 , , n 1 } × { 1 , , n 2 }
be the (primal) image grid, i.e., I ref : G p R and I temp : G p R . We discretize the integrals on the integration domain Ω ¯ : = [ 1 2 , n 1 + 1 2 ] × [ 1 2 , n 2 + 1 2 ] by the midpoint quadrature rule, i.e., with pixel values defined on G p . For discretizing the operators in (2) we work as usual on staggered grids. For the application of mimetic grid techniques in optical flow computation see also [40]. Let
G d : = { 3 2 , , n 1 1 2 } × { 3 2 , , n 2 1 2 } .
be the (inner) dual grid, i.e., G p shifted by 1 2 in each direction, and
G 1 : = { 3 2 , , n 1 1 2 } × { 1 , , n 2 } , G 2 : = { 1 , , n 1 } × { 1 2 , , n 2 3 2 } .
We start by considering the registration problems in the first alternating step (5) and turn to the image sequence problem in the second step (6) with (A2) afterwards.

Solution of the Registration Problems

Let us fix k { 1 , , K } in (5) and abbreviate T : = I k 1 , R : = I k as template and reference image, resp., and v : = v k . Then we have to find a minimizer of the continuous registration problem
J ( v ) : = Ω | R ( x ) T x v ( x ) | 2 + S ( v ) d x
where the elastic potential in (2) can be rewritten as
S ( v ) = μ Ω ( x v 1 ) 2 + 1 2 y v 1 + x v 2 2 + ( y v 2 ) 2 d x + λ 2 Ω x v 1 + y v 2 2 d x .
For the spatial discrete setting we assume that T , R : G p R s are given. Further, we consider v = ( v 1 , v 2 ) T with v 1 : G 1 R and v 2 : G 2 R . In contrast to [34] we assume no flow over the image boundary and set
v 1 ( 3 2 , y ) = v 1 ( n 1 1 2 , y ) = 0 , y { 1 , , n 2 } , v 2 ( x , 3 2 ) = v 2 ( x , n 2 1 2 ) = 0 , x { 1 , , n 1 } .
See Figure 2 for an illustration. We approximate x v 1 for x = ( x , y ) T G p by
( D 1 , x v 1 ) ( x , y ) : = 0 , x = 1 , v 1 ( x + 1 2 , y ) v 1 ( x 1 2 , y ) , x = 2 , , n 1 1 , 0 , x = n 1 ,
and y v 1 for x { 1 , n 1 1 } and y { 1 , , n 2 1 } by
( D 1 , y v 1 ) ( x + 1 2 , y + 1 2 ) = v 1 ( x + 1 2 , y + 1 ) v 1 ( x + 1 2 , y ) , x = 1 , , n 1 1 ,
and similarly for the derivatives with respect to v 2 . Then we obtain
S ( v ) = x G p μ ( D 1 , x v 1 ) 2 ( x ) + ( D 2 , y v 2 ) 2 ( x ) + λ 2 ( D 1 , x v 1 ) ( x ) + ( D 2 , y v 2 ) ( x ) 2 + x G d μ 2 ( D 1 , y v 1 ) ( x ) + ( D 2 , x v 2 ) ( x ) 2 .
To discretize the data term in (7) we need to approximate T x v ( x ) . Since v is not defined on G p we use instead of v its averaged version P v : = G p R 2 given by
( P 1 v 1 ) ( x , y ) : = 0 , x = 1 1 2 v 1 ( x 1 2 , y ) + v 1 ( x + 1 2 , y ) , x = 2 , , n 1 1 , 0 , x = n 1
and similarly for P 2 v 2 in y-direction. Let the continuous function T ˜ on Ω : = [ 1 , n 1 ] × [ 1 , n 2 ] be constructed from given values of T on G p by interpolation. Here we restrict our attention to bilinear interpolations, see (A3), but more sophisticated approaches are possible. In summary, the discrete registration problem reads
arg min v J ( v ) : = x G p | R ( x ) T ˜ x ( P v ) ( x ) | 2 + S ( v ) ,
where S ( v ) is given by (8). We solve this nonlinear least squares problem by a Gauss-Newton like method described in the Appendix B.

Multilevel Strategy

As usual in optical flow and image registration we apply a coarse-to-fine strategy. First we iteratively smooth and downsample our given images. On the coarsest level we perform a registration to obtain a deformation on the coarse grid. We apply a bilinear interpolation v of this deformation to construct intermediate images on the finer level by
I k ( x ) = T + k K ˜ ( R φ 1 T ) x k K ˜ v ( x ) , φ ( x ) = x v ( x ) ,
where T , R are the start and end images at the new level. Then we use the alternating algorithm on this level with K ˜ K images to obtain deformations and intermediate images on this level. Going to the finer level by bilinear interpolation, we construct more intermediate images by interpolating between neighboring images with (9). We repeat this process until we reach the maximal number K of images and finest level. The multilevel strategy of the algorithm is sketched in Algorithm 1.
Algorithm 1 Morphing Algorithm (informal).
1: T 0 : = T , R 0 : = R , Ω 0 : = Ω
2:create image stack ( T 0 , , T l ) , ( R 0 , , R l ) on ( Ω 0 , , Ω l ) by smoothing and downsampling
3:solve (3) for T l , R l with K = 1 , for v ˜
4: l l 1
5:use bilinear interpolation to get v on Ω l from v ˜
6:obtain K ˜ l images I l ( 0 ) from T l , R l , v by (9)
7:while l 0 do
8:  find image path I ˜ l and deformation path v ˜ l minimizing (3) with initialization I l ( 0 )
9:   l l 1
10:  if l > 0 then
11:   use bilinear interpolation to get I l and v l on Ω l
12:   for k = 1 , , K ˜ l do
13:    calculate K ˜ l intermediate images between I l , k 1 , I l , k with v l , k using (9)
14: I : = I 0

3. Face Colorization

In this section, we describe a method to colorize a gray-scale image based on the morphing map between the luminance channel of a source image (template) and the present image (target). The idea consists in transferring of the chrominance channels from the source image to the target one. To this end, we work in the YUV color space. For the importance of the chrominance channels in color imaging, see, e.g., [41,42]. While in RGB images the color channels are highly correlated, the YUV space shows a suitable decorrelation between the luminance channel Y and the two chrominance channels U , V . The transfer from the RGB space to YUV space is given by (see, e.g., [5])
Y U V = 0 , 299 0 , 587 0 , 114 0 , 14713 0 , 28886 0 , 436 0 , 615 0 , 51498 0 , 10001 R G B .
Most of the colorization approaches are based on the hypothesis that the target image is the luminance channel of the desired image. Thus, the image colorization process is based on the computation of the unknown chrominance channels.

3.1. Luminance Normalization

The first step of the algorithm consists in transforming the RGB source image to the Y U V image by (10). The range of the target gray-value image and the Y channel of the source image may differ making the meaningful comparison between these images not possible.
To tackle this issue, most of state-of-the art methods use a technique called luminance remapping which was introduced in [43]. This affine mapping between images which aims to fit the average and the standard deviation of the target and the template images is defined as
I temp : = var ( I tar ) var ( I Y ) I Y mean ( I Y ) + mean ( I tar ) ,
where mean is the average of the pixel values, and var is the empirical variance.

3.2. Chrominance Transfer by the Morphing Maps

Next we compute the morphing map between the two gray-scale images I temp and I tar with model (3). This results in the deformation sequence φ which produces the resulting map from the template image to the target one by concatenation
Φ = φ 1 φ 2 φ K .
Due to the discretization of the images, the map Φ is defined, for images of size n 1 × n 2 , on the discrete grid G : = { 1 , , n 1 } × { 1 , , n 2 } :
Φ : G [ 1 , n 1 ] × [ 1 , n 2 ] , x Φ ( x ) ,
where Φ ( x ) is the position in the source image which corresponds to the pixel x G in the target image. Now we colorize the target image by computing its chrominance channels, denoted by ( U tar ( x ) , V tar ( x ) ) at position x as
U tar ( x ) , V tar ( x ) : = U ( Φ ( x ) ) , V ( Φ ( x ) ) .
The chrominance channels of the target image are defined on the image grid G , but usually Φ ( x ) G . Therefore the values of the chrominance channels at Φ ( x ) have to be computed by interpolation. In our algorithm we use just bilinear interpolation which is defined for Φ ( x ) = ( p , q ) with ( p , q ) [ i , i + 1 ] × [ j , j + 1 ] , ( i , j ) { 1 , , n 1 1 } × { 1 , , n 2 1 } by
U ( Φ ( x ) ) = U ( p , q ) : = ( i + 1 p , p i ) U ( i , j ) U ( i , j + 1 ) U ( i + 1 , j ) U ( i + 1 , j + 1 ) j + 1 q q j .
Finally, we compute a colorized RGB image from its luminance I tar = Y tar and the chrominance channels (11) by the inverse of (10):
R ( x ) G ( x ) B ( x ) = 1 0 1 , 13983 1 0 , 39465 0 , 58060 1 2 , 03211 0 Y tar ( x ) U tar ( x ) V tar ( x )
for all x G . Figure 3 (Image segements with unified background of George W. Bush https://commons.wikimedia.org/wiki/File:George-W-Bush.jpeg and Barack Obama https://commons.wikimedia.org/wiki/File:Barack_Obama.jpg both in public domain.) summarizes our color transfer method.

4. Variational Methods for Chrominance Postprocessing

Sometimes the color transfer computed from the morphing map can be disturbed by artifacts. To improve the results, post-processing steps are usually applied in the literature.
Variational approaches are frequently applied in image colorization either directly or as a post-processing step, see, e.g., [1,3,44]. For instance, the technique of Gupta et al. [6] uses the chrominance diffusion approach of Levin et al. [1].
In this paper, we propose a variational method with a total variation based regularization as a post-processing step to remove possible artifacts. We build up on the model [3] suggested by one of the authors. This variational model uses a functional with a specific regularization term to avoid ”halo effects”. More precisely, we consider the minimizer of
u ^ = ( U ^ , V ^ ) = arg min ( U , V ) TV Y tar ( U , V ) + α Ω | U ( x ) U tar ( x ) | 2 + | V ( x ) V tar ( x ) | 2 d x ,
with
TV Y tar ( U , V ) : = Ω γ | Y tar | 2 + | U | 2 + | V | 2 d x .
The first term in (13) is a coupled total variation term which enforces the chrominance channels to have a contour at the same location as the target gray-value image. The data fidelity term is the classical squared L 2 -norm of the differences of the given and the desired chrominance channels. Note that the model in [3] contains an additional box constraint.
The parameter γ manages the coupling of the chrominance channels with the luminance one. It has been shown in [3] that a parameter value around 25 can be used for most images. The parameter α is related to the object size in the images. If some colors on large objects leak on small ones, this parameter has to be increased. On the other hand, if some artifacts are still present in the result, the value of α has to be decreased. We apply the primal-dual Algorithm 2 to find the minimizer of the strictly convex model (13). It uses an update on the step time parameters τ and σ , as proposed by Chambolle and Pock [45], as well as a relaxation parameter θ to speed-up the convergence. Here we use the abbreviation b : = ( U tar , V tar ) and u : = ( U , V ) . Further, p is the dual variable which is pixel-wise in R 6 . The parameters τ and σ are intern time step sizes. The operator div stands for the discrete divergence and ∇ for the discrete gradient. Further, the proximal mapping P B is given pixel-wise, for p R 6 by
P B p = p ^ max 1 , p ^ 2 2 , where p ^ : = p σ 0 0 0 0 x Y y Y .
Algorithm 2 Minimization of (13).
1: u 0 b , u ¯ 0 u 0
2: p 0 u 0
3: σ 0.001 , τ 20
4:for n > 0 do
5:   p n + 1 P B p n + σ u ¯ n
6:   u n + 1 u n + τ div ( p n + 1 ) + α b 1 + τ α
7:   θ = 1 / 1 + τ α
8:   τ = θ τ σ = σ / θ
9:   u ¯ n + 1 u n + 1 + θ ( u n + 1 u n )
10: u ^ u + .
As mentioned in the work of Deledalle et al. [38], the minimization of the TV- L 2 model produces a biased result. This bias causes a lost of contrast in the case of gray-scale images, whereas it is visible as a lost of colorfulness in the case of model (13). The authors of [38] describe an algorithm to remove such bias. In this paper, we propose to modify this method for our model (13) in order to enhance the result of Algorithm 2.
The CLEAR method of [38] relies on the refitting estimator R x ^ ( y ) of the data y from the biased estimation x ^ ( y ) :
R x ^ ( y ) arg min h H x ^ h ( y ) y 2 2
where H x ^ is defined as the set of maps h : R n R p satisfying, y R n :
h ( y ) = x ^ ( y ) + ρ J x ^ ( y ) ( y x ^ ( y ) ) , with ρ R ,
where J x ^ ( y ) is the Jacobian of the biased estimator with respect to the data y:
J x ^ ( y ) d = lim ε 0 x ^ ( y + ε d ) x ^ ( y ) ε .
J x ^ ( y ) contains some structural information of the biased estimator x ^ such as the jumps. A closed formula for ρ can be given:
ρ = J x ^ ( y ) ( δ ) | δ J x ^ ( y ) ( δ ) 2 2 if J x ^ ( y ) ( δ ) 0 1 otherwise . ,
where δ = y x ^ ( y ) . With Equation (15), the refitting process is as closer as possible to the original data.
The final algorithm is summarized in Algorithm 3. Note that it uses the result u ^ of Algorithm 1 as an input. The proximal mapping Π p ( p ˜ ) within the algorithm is defined pixel-wise, for variables p R 6 and p ˜ R 6 , as
Π p p ˜ = p ˜ ^ if p ^ < 1 1 p ^ p ˜ ^ p ^ , p ˜ ^ p ^ 2 p ^ otherwise ,
where p ^ and p ˜ ^ are defined as in (14).
Algorithm 3 Debiasing of Algorithm 2.
1: u 0 = b , u ¯ 0 = b
2: δ b u ^
3: u ˜ 0 = δ , u ˜ ¯ 0 = δ
4: p 0 u , p ˜ 0 u ˜
5: σ 0.001 , τ 20
6:for n 0 do
7:   p n + 1 P B p n + σ u ¯ n
8:   p ˜ n + 1 Π p n + σ u ¯ n p ˜ n + σ u ˜ ¯ n
9:   u n + 1 u n + τ div ( p n + 1 ) + α b 1 + τ α
10:   u ˜ n + 1 u ˜ n + τ div ( p ˜ n + 1 ) + α δ 1 + τ α
11:   θ = 1 / 1 + τ α
12:   τ = θ τ σ = σ / θ
13:   u ¯ n + 1 u n + 1 + θ ( u n + 1 u n )
14:   u ˜ ¯ n + 1 u ˜ n + 1 + θ ( u ˜ n + 1 u ˜ n )
15: ρ u ˜ + | δ u ˜ + 2 2 if u ˜ + 0 1 otherwise . ,
16: u debiased u ^ + ρ u ˜ + .
The results obtained at the different steps of the work-flow are presented for a particular image in Figure 4. First we demonstrate in Figure 4c that simply transforming the R , G , B channels via the morphing map gives no meaningful results. Letting our morphing map act to the chrominance channels of our source image and applying (12) with the luminance of our target image we get , e.g., Figure 4d which is already a good result. However, the forehead of Obama contains an artifact; a gray unsuitable color is visible here. After a post-processing of the chrominance channels by our variational method the artifacts disappear as can be seen in Figure 4e.

5. Numerical Examples

In this section, we compare our method with
-
the patch-based algorithm of Welsh et al. [8],
-
the patch-based method of Pierre et al. [3], and
-
the segmentation approach of Gupta et al. [6].
We implemented our morphing algorithm in Matlab 2016b and used the Matlab intern function for the bilinear interpolation. The parameters are summarized in Table 1. Here λ = μ and K are the parameters for the morphing step. The parameters γ and α appear in the variational model for post-processing and were only applied in three images. The number of deformations K depends on the size and details of the image, i.e., large detailed images need a larger K as small image having not so much details. Usually K = 24 was a good choice for our experiments. The parameter λ controls the smoothness of the deformation, i.e., a small value leads to large movements which might not be reasonable, while for large values the deformations become zero and the image metamorphosis is just a blending. In our experiments, a value of magnitude 10 2 lead to good results.
First we compare portraits in Figure 5 (Image segments of Steve Jobs https://commons.wikimedia.org/wiki/File:Steve_Jobs_Headshot_2010-CROP.jpg image courtesy of Matthew Yohe, Albert Einstein https://commons.wikimedia.org/wiki/File:Einstein_Portr_05936.jpg in public domain, Catherine Deneuve https://commons.wikimedia.org/wiki/File:Catherine_Deneuve_2010.jpg image courtesy of Georges Biard and Renée Deneuve https://commons.wikimedia.org/wiki/File:Ren%C3%A9e_Deneuve.jpg in public domain.) starting with the modern photographies in the first row. The approach of Welsh et al. [8] is based on a patch matching between images. The patch comparison is done with basic texture descriptors (intensity of the central pixel and standard-deviation of the patches). Since the background, as well as the skin are smooth, the distinction between them is unreliable if their intensities are similar. Moreover, since no regularization is used after the color transfer, some artifacts occur. For instance, some blue points appear on Obama’s face, see Figure 5, first row. The approach of Pierre et al. [3] is based on more sophisticated texture features for patches and applies a variational model with total variation like regularization. With this approach the artifacts mentioned above are less visible. Nevertheless, the forehead of Obama is purple which is unreliable. The method of Gupta et al. [6] uses texture descriptors after an over-segmentation, see, e.g., SLIC [10]. The texture descriptors are based on SURF and Gabor features. In the case of the Obama image, the descriptors are not able to distinguish the skin and other smooth parts, leading to a background color different from the source image. Our method is able to colorize the second image in a more reasonable way, i.e., face and background color are different and the tie gets a blue color. However, our methods is not perfect so far. For example part of the forehead of Obama becomes gray is due to the gray hair of Bush, which has a gray value closer to the forehead as to the hair of Obama.
The second and the third rows of Figure 5 focus on the colorization of old photographies. This challenging problem is a real application of image colorization which is sought, e.g., by the cinema industry. Note that the texture of old images are disturbed by the natural grain which is not the case in modern photography. Thus, the texture comparison is unreliable for this application. This issue is visible in all the comparison methods. For the portrait of Einstein the background is not colorized with the color of the source. Moreover, the color of the skin is different from those of the person in the source image. For the picture of Deneuve, the color of her lips is not transferred to the target image (Deneuve’s mother) with the state-of-the-art texture-based algorithms. With our morphing approach, the shapes of the faces are mapped. Thus, the lips, as well as the eyes and the skin are well colorized with a color similar to the source image. In the last row we have two images of the same person. Here the state-of-the-art-texture-based methods give unlikely results, especially the methods of Welsh et al. and Gupta et al. lead to a non smooth colorization of the background and the face, respectively. Our method provides a reasonable result with only small artifacts around the glasses.
In Figure 6 (Image segments of self-portraits of Vincent van Gogh https://commons.wikimedia.org/wiki/File:Vincent_Willem_van_Gogh_102.jpg and https://commons.wikimedia.org/wiki/File:Vincent_van_Gogh_-_Self-Portrait_-_Google_Art_Project.jpg both in public domain, a photography of a woman https://pixabay.com/en/woman-portrait-face-model-canon-659352/ licensed CC0, a drawing of a woman https://pixabay.com/en/black-and-white-drawing-woman-actor-1724363/ licensed CC0, a color image of Marilyn Monroe https://www.flickr.com/photos/7477245@N05/5272564106 created by Luiz Fernando Reis, and a drawing of Marilyn Monroe https://pixabay.com/en/marilyn-monroe-art-draw-marilyn-885229/ licensed CC0), we provide results including painted images. Note that we use the same Van Gogh self-portraits as in [31]. Due to the low contrast of the ear and suit to the background we add here the same segmentation information as in [31], which means our images are two dimensional during the calculation for the results shown in the first row of Figure 6. In these examples the similarity of the shapes between the source and target images is again more reliable than the matching of the textures so that only our morphing approach produces suitable results. Consider in particular the lips of the woman in the second and third row. A non post-processed result for the woman in the second row is shown in Figure 7. Comparing the two images we see that only small details change but most of the colorization is done by the morphing.
Figure 7 visualizes the path of the color transfer. As the morphing approach calculates at the same time an image and mapping path, we can not only show the final colorization result, but also the way the color is transported along this path. We illustrate this by the image in second row of Figure 6. The UV channels are transported via the mapping path of the Y channels of the source and the target images, where every second image along the path is shown. We see that even though the right part of the images undergoes large transformations, the eyes and the mouth are moved to the correct places. Note that the final image does not contain a post-processing, in contrast to those in the second row of Figure 6. However, the result is already quite good.
Figure 8 is included to evaluate the quality of our colorization method. We took two photographs of the same person in different poses. Assuming that the color of the person does not change due to the movement, we have a ground truth target image and can do a quantitative comparison using the PSNR. We see that the visual superiority of our result is confirmed by the much higher PSNR.
In Figure 9 we considered three RGB image pairs. Using the luminance of one of the images as target and the other one as source and conversely we colorized the target image and computed the PSNR. In Table 2 the PSNR values of the colorization with our method and the state-of-the-art methods are shown. The performance of our method is very convincing for all these example images.
Remark (Limitations of our approach).
In all our experiments our method leads to good results and outperforms the state-of-the-art colorization methods. However, the method is not perfect and the face images can not be arbitrarily deformed. Since the morphing is only done in the Y channel, the illumination of the faces should be similar. For example there is a shadow on the left side of the images in the second row of Figure 5. Mirroring one of the images would lead to bad matching results. The same holds true for the features themselves, so we can not expect to match bright hair perfectly to dark hair, as we see for example in Figure 4d.
Since the deformations are close to diffeomorphisms, features should not be excluded or appear. For example, the left eye in the last row of Figure 9 starts in the middle of the face and moves to the edge, so a part of the skin-color stays in the eye. Similarly in the second matching here the eye moves from the edge to the inside, so the skin obtains the color from the background.
By a proper initialization the faces should be aligned in such a way that there are no similar features atop of each other, e.g., a left eye in the source should not be atop of the right eye of the target. However with more prior knowledge we could use the given features as landmarks and perform a landmark matching, e.g., [46], for the initialization on the coarsest level.

6. Conclusions

In this paper, we propose an finite difference method to compute a morphing map between a source and a target gray-value image. This map enables us to transfer the color from the source image to the target one, based on the shapes of the faces. An additional variational post-processing step with a luminance guided total variation regularization and an update to make the result unbiased may be added to remove some possible artifacts. The results are very convincing and outperform state-of-the-art approaches on historical photographies.
Let us notice some special issues of our approach in view of an integration into a more global framework for an exemplar-based image and video colorization. First of all, our method works well on faces and on object with similar shapes, but when this hypothesis is not fulfilled, some artifacts can appear. Therefore, instead of using our algorithm on full images, a face detection algorithm can be used to focus on the face colorization. Let us remark that faces in image can be detected by efficient, recent methods, see, e.g., [47]. In future work, the method will be integrated into a complete framework for exemplar-based image and video colorization.
Second, the morphing map has to be computed between images with the same size. This issue can be easily solved with a simple interpolation of the images. Keeping the ratio between the width and the height of faces similar, the distortion produced by such interpolation is small enough to support our method.

Acknowledgments

Funding by the German Research Foundation (DFG) within the project STE 571/13-1 is gratefully acknowledged.

Author Contributions

All the three authors have substantially contributed to the paper. The paper is the outcome of many joint and fruitful discussions J. Persch and G. Steidl had with F. Pierre during the months the later was a Postdoc at the University at Kaiserslautern (Germany). Therefore it is not possible to split the contributions. However, J. Persch was the leading expert concerning morphing and F. Pierre concerning color image processing.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A. Solution of the Image Sequence Problem

Fixing v , resp., φ leads to the image sequence problem (6). In the following we show how this problem can be solved via the linear system of equations arising from Euler-Lagrange equation of the functional. We mainly follow [31].
The terms in J φ ( I ) containing I k are
Ω | I k φ k + 1 ( x ) I k + 1 ( x ) | 2 + | I k ( x ) I k 1 φ k ( x ) | 2 d x .
Assuming that φ k + 1 is a diffeomorphism, the variable transform y : = φ k + 1 ( x ) gives
Ω | I k ( x ) I k + 1 φ k + 1 1 ( x ) | 2 | det φ k + 1 1 φ k + 1 1 ( x ) | + | I k ( x ) I k 1 φ k ( x ) | 2 d x = Ω | I k ( x ) I k + 1 φ k + 1 1 ( x ) | 2 | det φ k + 1 φ k + 1 1 ( x ) | 1 + | I k ( x ) I k 1 φ k ( x ) | 2 d x
Setting
a k φ k + 1 1 ( x ) : = | det φ k + 1 φ k + 1 1 ( x ) | 1 ,
the Euler-Lagrange equations read for k = 1 , , K 1 as
I k ( x ) I k + 1 ( φ k + 1 1 ( x ) ) a k φ k + 1 1 ( x ) + I k ( x ) I k 1 φ k ( x ) = 0 , I k 1 φ k ( x ) + 1 + a k φ k + 1 1 ( x ) I k ( x ) a k φ k + 1 1 ( x ) I k + 1 ( φ k + 1 1 ( x ) ) = 0 .
We introduce
X K ( x ) : = x , X k 1 ( x ) : = φ k X k ( x ) , k = 1 , , K ,
which can be computed for each x Ω since the φ k , k = 1 , , K are given. Then (A1) can be rewritten for k = 1 , , K 1 as
I k 1 X k 1 ( x ) + ( 1 + a k X k + 1 ( x ) I k X k ( x ) a k X k + 1 ( x ) I k + 1 X k + 1 ( x ) = 0 .
In matrix-vector form, this reads with
F 0 = I temp X 0 ( x ) , F K = I ref ( x ) , F k I k X k ( x ) , k = 1 , , K 1
for fixed x Ω and F : = ( F 1 , , F K 1 ) T as
A F = ( F 0 , 0 , , 0 , a K 1 F K ) T , A : = tridiag ( 1 , 1 + a k , a k ) k = 1 K 1 .
Assuming that a k > 0 which is the case in practical applications, the matrix A is irreducible diagonal dominant and thus invertible.

Appendix B. Gauss-Newton Method for the Registation Problem

We consider the nonlinear least squares problem in (7)
arg min v J ( v ) : = x G p | R ( x ) T ˜ x ( P v ) ( x ) | 2 + S ( v ) .
Let us first write S ( v ) in a convenient matrix-vector notation. Using v 1 = v 1 ( x + 1 2 , y ) x , y = 1 n 1 1 , n 2 R n 1 1 , n 2 , v 2 = v 1 ( x , y + 1 2 ) x , y = 1 n 1 , n 2 1 R n 1 , n 2 1 and
D 1 , x : = 0 1 1 0 0 1 1 0 0 0 R n 1 × n 1 1 , D 1 , y : = 1 1 0 1 1 1 1 0 0 0 1 1 R n 2 2 × n 2 1 ,
and similarly D 2 , x R n 1 2 × n 1 1 and D 2 , y R n 2 × n 2 1 we obtain
S ( v ) = μ D 1 , x v 1 F 2 + v 2 D 2 , y T F 2 + λ 2 D 1 , x v 1 + v 2 D 2 , y T F 2 + μ 2 v 1 D 1 , y T + D 2 , x v 2 F 2
where · F denotes the Frobenius norm of matrices. Reshaping the v i , i = 1 , 2 columnwise into a vectors v 1 R ( n 1 1 ) n 2 and v 2 R n 1 ( n 2 1 ) (where we keep the notation) and using tensor products ⊗ of matrices
D 1 , x : = I n 2 D 1 , x , D 2 , x : = I n 2 1 D 2 , x , D 1 , y : = D 1 , x I n 1 1 , D 1 , y : = D 2 , y I n 1
the regularizing term can be rewritten as
S ( v ) = S v 2 2 , S μ D 1 , x 0 0 μ D 2 , y μ 2 D 1 , y μ 2 D 2 , x λ 2 D 1 , x λ 2 D 2 , y , v = v 1 v 2 .
The nonlinear data term in the registration problem with the columnwise reshaped image T ( v ) of T ˜ x ( P v ) ( x ) x G p and R of R ( x ) x G p reads
D ( v ) : = R T ( v ) 2 .
We linearize the term at a previous iterate v ( r ) , i.e.,
D ( v ) R T ( v ( r ) ) + v T ( v ( r ) ) G ( r ) v v ( r ) 2 2 .
We comment on the computation of G ( r ) below. Then we consider
R T ( v ( r ) ) + G ( r ) v v ( r ) 2 2 + S v 2 2 .
The gradient is given by
G ( r ) T G ( r ) + S T S v G ( r ) T G ( r ) v ( r ) G ( r ) T R T ( v ( r ) ) .
Setting the gradient to zero we obtain for v
v : = v ( r ) G ( r ) T G ( r ) + S T S 1 G ( r ) T T ( v ( r ) ) R + S T S v ( r ) = v ( r ) G ( r ) T G ( r ) + S T S 1 v J ( v ( r ) )
As we want to minimize J , the next iterate is given by
v ( r + 1 ) = v ( r ) 1 2 l G ( r ) T G ( r ) + S T S 1 v J ( v ( r ) ) ,
where l N is the smallest number such that J ( v ( k + 1 ) ) < J ( v ( k ) ) , if v J ( v ( r ) ) is not zero.
Finally we comment on the computation of
G ( r ) : = v T ( v ( r ) ) = P 1 T x T ˜ x P 1 v 1 ( x ) , y P 2 v 2 ( x ) P 2 T y T ˜ x P 1 v 1 ( x ) , y P 2 v 2 ( x ) .
Since T ˜ is computed by linear interpolation from T ( x ) x G p , Then we have for an arbitrary point ( p , q ) [ i , i + 1 ] × [ j , j + 1 ] , ( i , j ) { 1 , , n 1 1 } × { 1 , , n 2 1 } that
T ˜ ( p , q ) : = ( i + 1 p , p i ) T ( i , j ) T ( i , j + 1 ) T ( i + 1 , j ) T ( i + 1 , j + 1 ) j + 1 q q j .
Then
x T ˜ ( p , q ) = ( j + 1 q ) T ( i + 1 , j ) T ( i , j ) + ( q j ) T ( i + 1 , j + 1 ) T ( i , j ) , y T ˜ ( p , q ) = ( i + 1 p ) T ( i , j + 1 ) T ( i , j ) + ( p i ) T ( i + 1 , j + 1 ) T ( i + 1 , j ) .
Hence the derivatives can be calculated from the forward differences of T in x- resp. y-direction and appropriate weighting.

References

  1. Levin, A.; Lischinski, D.; Weiss, Y. Colorization using optimization. ACM Trans. Graph. 2004, 23, 689–694. [Google Scholar] [CrossRef]
  2. Irony, R.; Cohen-Or, D.; Lischinski, D. Colorization by example. In Proceedings of the 16th Eurographics Conference on Rendering Techniques, Konstanz, Germany, 29 June 29–1 July 2005; pp. 201–210. [Google Scholar]
  3. Pierre, F.; Aujol, J.F.; Bugeau, A.; Papadakis, N.; Ta, V.T. Luminance-chrominance model for image colorization. SIAM J. Imaging Sci. 2015, 8, 536–563. [Google Scholar] [CrossRef]
  4. Yatziv, L.; Sapiro, G. Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 2006, 15, 1120–1129. [Google Scholar] [CrossRef] [PubMed]
  5. Jack, K. Video Demystified: A Handbook for the Digital Engineer; Elsevier: Amsterdam, The Netherlands, 2011. [Google Scholar]
  6. Gupta, R.K.; Chia, A.Y.S.; Rajan, D.; Ng, E.S.; Zhiyong, H. Image colorization using similar images. In Proceedings of the 20th ACM International Conference on Multimedia, Nara, Japan, 29 October–2 November 2012; pp. 369–378. [Google Scholar]
  7. Horiuchi, T. Colorization algorithm using probabilistic relaxation. Image Vis. Comput. 2004, 22, 197–202. [Google Scholar] [CrossRef]
  8. Welsh, T.; Ashikhmin, M.; Mueller, K. Transferring color to greyscale images. ACM Trans. Graph. 2002, 21, 277–280. [Google Scholar] [CrossRef]
  9. Efros, A.A.; Leung, T.K. Texture synthesis by non-parametric sampling. In Proceedings of the 7th IEEE International Conference on Computer Vision, Kerkyra, Greece, 20–27 September 1999; Volume 2, pp. 1033–1038. [Google Scholar]
  10. Achanta, R.; Shaji, A.; Smith, K.; Lucchi, A.; Fua, P.; Süsstrunk, S. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 34, 2274–2282. [Google Scholar] [CrossRef] [PubMed]
  11. Chen, T.; Wang, Y.; Schillings, V.; Meinel, C. Grayscale image matting and colorization. In Proceedings of the Asian Conference on Computer Vision, Jeju Island, Korea, 27–30 January 2004; pp. 1164–1169. [Google Scholar]
  12. Charpiat, G.; Hofmann, M.; Schölkopf, B. Automatic image colorization via multimodal predictions. In European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2008; pp. 126–139. [Google Scholar]
  13. Chia, A.Y.S.; Zhuo, S.; Kumar, R.G.; Tai, Y.W.; Cho, S.Y.; Tan, P.; Lin, S. Semantic colorization with internet images. ACM Trans. Graph. 2011, 30, 156. [Google Scholar] [CrossRef]
  14. Zhang, R.; Isola, P.; Efros, A.A. Colorful image colorization. In European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2016; pp. 1–16. [Google Scholar]
  15. Smythe, D.B. A Two-Pass Mesh Warping Algorithm for Object Transformation and Image Interpolation; Technical report; ILM Technical Memo Department, Lucasfilm Ltd.: San Francisco, CA, USA, 1990. [Google Scholar]
  16. Wolberg, G. Digital Image Warping; IEEE Computer Society Press: Los Alamitos, CA, USA, 1990; Volume 10662. [Google Scholar]
  17. Wolberg, G. Image morphing: A survey. Vis. Comput. 1998, 14, 360–372. [Google Scholar] [CrossRef]
  18. Miller, M.I.; Younes, L. Group actions, homeomorphisms, and matching: A general framework. Int. J. Comput. Vis. 2001, 41, 61–84. [Google Scholar] [CrossRef]
  19. Trouvé, A.; Younes, L. Local geometry of deformable templates. SIAM J. Math. Anal. 2005, 37, 17–59. [Google Scholar] [CrossRef]
  20. Trouvé, A.; Younes, L. Metamorphoses through Lie group action. Found. Comput. Math. 2005, 5, 173–198. [Google Scholar] [CrossRef]
  21. Christensen, G.E.; Rabbitt, R.D.; Miller, M.I. Deformable templates using large deformation kinematics. IEEE Trans. Image Process. 1996, 5, 1435–1447. [Google Scholar] [CrossRef] [PubMed]
  22. Dupuis, P.; Grenander, U.; Miller, M.I. Variational problems on flows of diffeomorphisms for image matching. Q. Appl. Math. 1998, 56, 587–600. [Google Scholar] [CrossRef]
  23. Trouvé, A. An infinite dimensional group approach for physics based models in pattern recognition. Int. J. Comput. Vis. 1995, 28, 213–221. [Google Scholar] [CrossRef]
  24. Trouvé, A. Diffeomorphisms groups and pattern matching in image analysis. Int. J. Comput. Vis. 1998, 28, 213–221. [Google Scholar] [CrossRef]
  25. Beg, M.F.; Miller, M.I.; Trouvé, A.; Younes, L. Computing large deformation metric mappings via geodesic flows of diffeomorphisms. Int. J. Comput. Vis. 2005, 61, 139–157. [Google Scholar] [CrossRef]
  26. Richardson, C.L.; Younes, L. Metamorphosis of images in reproducing kernel Hilbert spaces. Adv. Comput. Math. 2016, 42, 573–603. [Google Scholar] [CrossRef]
  27. Hong, Y.; Joshi, S.; Sanchez, M.; Styner, M.; Niethammer, M. Metamorphic geodesic regression. Med. Image Comput. Comput. Assist. Interv. 2012, 15, 197–205. [Google Scholar] [PubMed]
  28. Younes, L. Shapes and Diffeomorphisms; Springer: Berlin, Germany, 2010. [Google Scholar]
  29. Miller, M.I.; Trouvé, A.; Younes, L. Hamiltonian systems and optimal control in computational anatomy: 100 years since D’Arcy Thompson. Annu. Rev. Biomed. Eng. 2015, 17, 447–509. [Google Scholar] [CrossRef] [PubMed]
  30. Miller, M.I.; Trouvé, A.; Younes, L. On the metrics and Euler-Lagrange equations of computational anatomy. Annu. Rev. Biomed. Eng. 2002, 4, 375–405. [Google Scholar] [CrossRef] [PubMed]
  31. Berkels, B.; Effland, A.; Rumpf, M. Time discrete geodesic paths in the space of images. SIAM J. Imaging Sci. 2015, 8, 1457–1488. [Google Scholar] [CrossRef]
  32. Christensen, G.E.; Johnson, H.J. Consistent image registration. IEEE Trans. Med. Imaging 2001, 20, 568–582. [Google Scholar] [CrossRef] [PubMed]
  33. Fischer, B.; Modersitzki, J. Curvature based image registration. J. Math. Imaging Vis. 2003, 18, 81–85. [Google Scholar] [CrossRef]
  34. Haber, E.; Modersitzki, J. A multilevel method for image registration. SIAM J. Sci. Comput. 2006, 27, 1594–1607. [Google Scholar] [CrossRef]
  35. Han, J.; Berkels, B.; Droske, M.; Hornegger, J.; Rumpf, M.; Schaller, C.; Scorzin, J.; Urbach, H. Mumford–Shah Model for one-to-one edge matching. IEEE Trans. Image Process. 2007, 16, 2720–2732. [Google Scholar] [CrossRef] [PubMed]
  36. Modersitzki, J. Numerical Methods for Image Registration; Oxford University Press: Oxford, UK, 2004. [Google Scholar]
  37. Modersitzki, J. FAIR: Flexible Algorithms for Image Registration; SIAM: Philadelphia, PA, USA, 2009. [Google Scholar]
  38. Deledalle, C.A.; Papadakis, N.; Salmon, J.; Vaiter, S. CLEAR: Covariant LEAst-square Refitting with applications to image restoration. SIAM J. Imaging Sci. 2017, 10, 243–284. [Google Scholar] [CrossRef]
  39. Miller, M.I.; Christensen, G.E.; Amit, Y.; Grenander, U. Mathematical textbook of deformable neuroanatomies. Proc. Natl. Acad. Sci. USA 1993, 90, 11944–11948. [Google Scholar] [CrossRef] [PubMed]
  40. Yuan, J.; Schnörr, C.; Steidl, G. Simultaneous higher order optical flow estimation and decomposition. SIAM J. Sci. Comput. 2007, 29, 2283–2304. [Google Scholar] [CrossRef]
  41. Bertalmio, M. Image Processing for Cinema; CRC Press: Boca Raton, FL, USA, 2014. [Google Scholar]
  42. Nikolova, M.; Steidl, G. Fast Hue and Range Preserving Histogram Specification: Theory and New Algorithms for Color Image Enhancement. IEEE Trans. Image Process. 2014, 23, 4087–4100. [Google Scholar] [CrossRef] [PubMed]
  43. Hertzmann, A.; Jacobs, C.E.; Oliver, N.; Curless, B.; Salesin, D.H. Image analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA, 12–17 August 2001; pp. 327–340. [Google Scholar]
  44. Peter, P.; Kaufhold, L.; Weickert, J. Turning diffusion-based image colorization into efficient color compression. IEEE Trans. Image Process. 2016, 26, 860–869. [Google Scholar] [CrossRef] [PubMed]
  45. Chambolle, A.; Pock, T. A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 2011, 40, 120–145. [Google Scholar] [CrossRef]
  46. Joshi, S.C.; Miller, M.I. Landmark matching via large deformation diffeomorphisms. IEEE Trans. Image Process. 2000, 9, 1357–1370. [Google Scholar] [CrossRef] [PubMed]
  47. Chen, D.; Hua, G.; Wen, F.; Sun, J. Supervised transformer network for efficient face detection. In European Conference on Computer Vision; Springer: Cham, Switzerland, 2016; pp. 122–138. [Google Scholar]
Figure 1. Illustration of the image path and the diffeomorphism path, where I 0 : = I temp and I K : = I tar are the given template and target images.
Figure 1. Illustration of the image path and the diffeomorphism path, where I 0 : = I temp and I K : = I tar are the given template and target images.
Jimaging 03 00048 g001
Figure 2. Illustration of the grids, where empty boxes mean zero movement.
Figure 2. Illustration of the grids, where empty boxes mean zero movement.
Jimaging 03 00048 g002
Figure 3. Overview of the color transfer. The mapping Φ is computed from Model (3) between the luminance channel of the source image and the target one. From this map, the chrominances of the source image are mapped. Finally, from these chrominances and the target image the colorization result is computed.
Figure 3. Overview of the color transfer. The mapping Φ is computed from Model (3) between the luminance channel of the source image and the target one. From this map, the chrominances of the source image are mapped. Finally, from these chrominances and the target image the colorization result is computed.
Jimaging 03 00048 g003
Figure 4. Illustration of the colorization steps of our algorithm. (a) The color source image; (b) The gray value target image; (c) The transport of the R , G , B channel via the morphing map is not suited for colorization; (d) The result with our morphing method is already very good; (e) It can be further improved by our variational post-processing.
Figure 4. Illustration of the colorization steps of our algorithm. (a) The color source image; (b) The gray value target image; (c) The transport of the R , G , B channel via the morphing map is not suited for colorization; (d) The result with our morphing method is already very good; (e) It can be further improved by our variational post-processing.
Jimaging 03 00048 g004
Figure 5. Comparison of our approach with state-of-the-art methods on photographies. In contrast to these methods our model is not based on texture comparisons, but on the morphing of the shapes. Therefore it is able to handle faces images, where the background has frequently a similar texture as the skin.
Figure 5. Comparison of our approach with state-of-the-art methods on photographies. In contrast to these methods our model is not based on texture comparisons, but on the morphing of the shapes. Therefore it is able to handle faces images, where the background has frequently a similar texture as the skin.
Jimaging 03 00048 g005
Figure 6. Results including painted faces. Only our morphing method is able to colorize the target images in an authentic way.
Figure 6. Results including painted faces. Only our morphing method is able to colorize the target images in an authentic way.
Jimaging 03 00048 g006
Figure 7. Color transport along the image path.
Figure 7. Color transport along the image path.
Jimaging 03 00048 g007
Figure 8. Results on a color image turned into a gray-scale one for a quantitative comparison. The qualitative comparisons with the state-of-the-art methods are confirmed by the PSNR measures.
Figure 8. Results on a color image turned into a gray-scale one for a quantitative comparison. The qualitative comparisons with the state-of-the-art methods are confirmed by the PSNR measures.
Jimaging 03 00048 g008
Figure 9. Multiple colorizations of known RGB-images with difference plots measured in Euclidean distance in R 3 .
Figure 9. Multiple colorizations of known RGB-images with difference plots measured in Euclidean distance in R 3 .
Jimaging 03 00048 g009
Table 1. Parameters for our numerical experiments.
Table 1. Parameters for our numerical experiments.
Image μ K γ α
Figure 5-1. row0.02524500.005
Figure 5-2. row0.0524250.005
Figure 5-3. row0.0512--
Figure 5-4. row0.0524--
Figure 6-1. row0.00532--
Figure 6-2. row0.007518500.05
Figure 6-3. row0.0424--
Figure 70.007518--
Figure 80.0125--
Figure 9-1. row0.00525--
Figure 9-2. row0.0125--
Figure 9-3. row0.0125--
Table 2. Comparison of the different PSNR values for the image pairs in Figure 9.
Table 2. Comparison of the different PSNR values for the image pairs in Figure 9.
GrayWelsh et al. [8]Gupta et al. [6]Pierre et al. [3]Our
Figure 9-1. row 1. pair24.802320.046726.352733.769444.7808
Figure 9-1. row 2. pair24.521823.951325.945734.423145.4682
Figure 9-2. row 1. pair24.378422.672927.558632.011941.1413
Figure 9-2. row 2. pair23.772123.237525.937530.139839.4254
Figure 9-3. row 1. pair24.595030.398524.311231.526342.3861
Figure 9-3. row 2. pair24.390727.781625.620731.898242.4092

Share and Cite

MDPI and ACS Style

Persch, J.; Pierre, F.; Steidl, G. Exemplar-Based Face Colorization Using Image Morphing. J. Imaging 2017, 3, 48. https://doi.org/10.3390/jimaging3040048

AMA Style

Persch J, Pierre F, Steidl G. Exemplar-Based Face Colorization Using Image Morphing. Journal of Imaging. 2017; 3(4):48. https://doi.org/10.3390/jimaging3040048

Chicago/Turabian Style

Persch, Johannes, Fabien Pierre, and Gabriele Steidl. 2017. "Exemplar-Based Face Colorization Using Image Morphing" Journal of Imaging 3, no. 4: 48. https://doi.org/10.3390/jimaging3040048

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop