Quantifying Perceived Facial Asymmetry to Enhance Physician–Patient Communications

Abstract

In cosmetic surgery, bridging the anticipation gap between the patients and the physicians can be challenging if there lacks objective and transparent information exchange during the decision-making and surgical process. Among all factors, facial symmetry is the most important for assessing facial attractiveness. The aim of this work is to promote communications between the two parties by providing a quadruple of quantitative measurements: overall asymmetry index (oAI), asymmetry vector, classification, and confidence vector, using an artificial neural network classifier to model people’s perception acquired from visual questionnaires concerning facial asymmetry. The questionnaire results exhibit a Cronbach’s Alpha value of 0.94 and categorize the respondents’ perception of each stimulus face into perceived normal (PN), perceived asymmetrically normal (PAN), and perceived abnormal (PA) categories. The trained classifier yields an overall root mean squared error < 0.01, and its result shows that the oAI is, in general, proportional to the degree of perceived asymmetry. However, there exist faces that are difficult to classify as either PN or PAN or either PAN or PA with competing confidence values. In such cases, oAI alone is not sufficient to articulate facial asymmetry. Assisting surgeon–patient conversations with the proposed asymmetry quadruple is advised to avoid or to mitigate potential medical disputes.

1. Introduction

Appearance is the most prominent stimulus to establish an impression in others [1]. Rubenstein et al. showed that humans develop their capability of perceiving attractiveness as early as in their infancy [2]. Cross-cultural and cross-age similarities exist in judging facial attractiveness [3]. Among all factors, facial symmetry is the most important factor that correlates facial attractiveness [4,5,6,7,8]. Perfect facial symmetry is only considered a theoretical existence, as most attractive individuals exhibit an asymmetric facial nature [9]. However, significant facial asymmetry can introduce aesthetic or even functional problems [10]. Cheong and Lo articulated the etiology of facial asymmetry as being congenital, developmental, or acquired; the clinical implications, evaluation, and treatment planning and management may vary accordingly [5,11].

People have grown more accepting of seeking craniofacial orthognathic, orthodontic, or plastic surgery for appearance improvements in recent years [12,13]. According to ISAPS Global Statistics, more than 11 million surgical procedures were performed in 2019, an increase of 20.6% from 2015, and there were 4,058,143 (35.7%) total face and head surgical procedures [14].

Although plastic surgery is expected to enhance one’s appearance and self-esteem, there exist consequential uncertainties due to psychological and physical discomfort for the patients. Harmonious physician–patient relationships are crucial to soothe this discomfort and to promote healthcare delivery [15]. However, anticipation gaps and insufficient decision-making communications have caused medical disputes [16,17,18,19]. According to the Department of Health, Taipei City, among Taipei’s 415 cosmetic surgery clinics in 2016, 116 of 374 medical disputes came from aesthetic medicine [20]

To bridge the anticipation gaps, quantitative measurements are needed to model perception of facial asymmetry. Related work and strategies in the below literature include: (1) determining how stimulus faces are collected—two-dimensional (2D) or three-dimensional (3D), real or simulated, with skin or without; (2) identifying facial landmarks automatically or semi-automatically, or maybe no landmarks; (3) defining a coordinate system and making measurements; (4) conducting questionnaire surveys to quantify perceptions of facial asymmetry; (5) computing the degree of asymmetry and classification.

Chu et al. employed a series of 2D images to present progressive asymmetry [21]. The results suggested that at least 3 mm of facial asymmetry on the oral commissure, brow, or both was required for the participants to notice the asymmetry. Naini et al. measured the nasolabial angle on 2D facial contours to determine whether a rhinoplasty surgery was needed [22]. The results showed that 2D stimulus facial images had their constraints. Meyer-Marcotty et al. performed a similar study on facial asymmetry, though utilizing 3D images [1]. Instead of employing images from different persons, they transformed the same image by incremental soft tissue alterations. They found that the nasal structure played a crucial role in perception of symmetry, and the asymmetry of chins greatly affected the facial appearance.

In clinical practices, asymmetry is usually measured by comparing bilateral facial features against the mid-sagittal plane. Quantitative studies of the face rely on the accurate positioning of facial landmarks or surface-based patches [23]. Masuoka et al. explored the correlation of facial asymmetry between cephalometric measurements and the frontal photograph evaluation conducted by orthodontists. The result argued that there existed a discrepancy between the two [24]. Ferrario et al. proposed quantitative metrics to assess pre- and post-operative differences [25]. Hajeer et al. compared the performance of bi-maxillary and maxillary osteotomy [26]. Pre- and post-orthognathic surgeries of 44 patients (20 Class III cases of bi-maxillary osteotomy; 12 Class III cases of maxillary osteotomy; 12 Class II cases of bi-maxillary osteotomy) were assessed. Meyer-Marcotty et al., Djordjevic et al., and Cevidanes et al. mirrored faces and aligned feature points [1,27,28]. Huang et al. proposed an asymmetry index (AI) for each individual facial landmark of interest [7]. Hsu et al. computed facial contour asymmetry instead of landmark asymmetry [29]. Lo et al. constructed a transfer learning model to score the asymmetry of facial contour maps and to assess the efficacy of an orthognathic surgery [30]. The above-mentioned studies employed the Frankfurt horizontal plane or the natural head position to derive 3D head coordinate systems.

Questionnaires were used to elicit and quantify people’s perception of facial asymmetry. Lee et al. claimed that non-experts’ assessments for facial asymmetry should be first considered because the ultimate demand and perception rested upon the patients [31]. Padwa et al. maintained similar comments [32]. Jackson et al. conducted visual questionnaires among professional orthodontists, general dentists, and laypersons. [8]. Their study showed that abilities to assess facial symmetry were different over varied professional backgrounds, and the orthodontists had the most profound capabilities of differentiating symmetrical from asymmetrical faces. Chu et al. employed playback of photographs to trigger conscious perception of the observers [21]. In the study of Naini et al., three groups of respondents, i.e., orthognathic patients, clinicians, and laypeople, were invited to assess how mandible and chin landmarks related to perceived asymmetry [22]. The result demonstrated that the perception of clinicians and patients were more critical than laypeople. McAvinchey et al. classified the participants into five groups: laypeople, dental students, dental care professionals, dental practitioners, and orthodontists [33].

To grade the degree of asymmetry, Yamamoto et al. invited oral surgeons and orthodontists to subjectively evaluate facial asymmetry and defined grade#0 as good symmetrical frontal view, grade#1 as little asymmetry, grade#2 as localized asymmetry, and grade#3 as marked asymmetry [34]. Meyer-Marcotty et al. adopted a six-point scale to rank facial symmetry (1: very symmetrical; 6: very asymmetrical) [1]. Masuoka et al. categorized the frontal facial image into two groups [24]. Patients in Group A exhibited symmetrical or little asymmetrical frontal views, without having to undergo a surgical treatment. Patients in Group B exhibited marked asymmetry and a surgical treatment was required. McAvinchey et al. requested the observers to classify the facial images displayed on the screen into normal, slightly abnormal but socially acceptable, or abnormal [33].

The building blocks to achieve the aim of this work are threefold. (1) Define a 3D facial coordinate system regardless of the head direction and compute asymmetry index vector ($\stackrel{\rightharpoonup }{\mathrm{AI}}$) for individual facial features and overall asymmetry index ($\mathrm{oAI}$). (2) Build an artificial neural network classifier to model perceived asymmetry acquired from the questionnaires and classify a face as perceived normal ($\mathrm{PN}$), perceived asymmetrically normal ($\mathrm{PAN}$), or perceived abnormal ($\mathrm{PA}$), with associated confidence vectors ($\stackrel{\rightharpoonup }{\mathrm{C}}$). (3) Propose the quadruple $\langle \mathrm{oAI}, \stackrel{\rightharpoonup }{\mathrm{AI}}, \mathrm{PN}\left|\mathrm{PAN}\right|\mathrm{PA}, \stackrel{\rightharpoonup }{\mathrm{C}}\rangle$ as a tool to promote transparent communications between the surgeon and the patient. The degree of overall facial asymmetry ($\mathrm{oAI}$) together with laypeople-deemed classification ($\mathrm{PN}\left|\mathrm{PAN}\right|\mathrm{PA}$) articulate the severity of asymmetry. The asymmetry index vector ($\stackrel{\rightharpoonup }{\mathrm{AI}}$) depicts individual degrees of facial features of concern. The confidence vector ($\stackrel{\rightharpoonup }{\mathrm{C}}$) reveals laypeople’s voting distribution over the three classes for a face.

2. Materials and Methods

Ethical approval for this study was obtained from the Institutional Review Board of Chang Gung Memorial Hospital, Taiwan, R.O.C. (102-1359B and 103-3130B)

Figure 1 illustrates the procedure used to model the perceived facial asymmetry. On the top-left corner is a 3D normal face whose skin texture is removed, and facial landmarks are identified. The head coordinate system is then computed. Walking along the upper row and going downwards, the normal face is morphed to generate stimulus faces with varied degrees of asymmetry, with which the proposed visual questionnaire surveys are conducted. Each stimulus face receives votes from the respondents’ asymmetry perception and is categorized as PN, PAN, or PA, with associated confidence vector ($\stackrel{\rightharpoonup }{\mathrm{C}}$). Walking downwards on the left, the asymmetry index vector ($\stackrel{\rightharpoonup }{\mathrm{AI}}$) is computed and fed as the input to train the artificial neural network classifier to learn the categorization on the right. The overall asymmetry ($\mathrm{oAI}$) is the weighted sum of individual asymmetry indices.

The remainder of this section depicts the proposed methods in details.

2.1. Acquisition of 3D Facial Images and Pre-Process

A 3dMD scanner, an ultra-fast non-invasive 3D cranial imaging system, is used to document high-precision facial surface with the capture speed of 1.5 ms per image. To prevent the respondents from being distracted by the subjects’ individual outlooks, facial hues, or skin quality, we decide to use only one subject’s 3D facial image, which is later morphed into a series of faces with different, controlled degrees of asymmetry. Furthermore, we remove the facial skin texture and make it monochrome. The inclusion criteria of the subject are of dental occlusion Angle Class I, no craniofacial deformity, no facial trauma history, no prior orthognathic surgery, and their face generally regarded as symmetric by three orthodontists.

2.2. Facial Landmarks and the Corresponding 3D Coordinate System

Twenty facial landmarks, including eight medial and six bilateral pairs, as shown in

Table 1. Twenty facial landmarks, abbreviations, and definitions.

Landmarks	ID (i)	L(i)	Definition
Glabella	1	G	Most prominent midline point between eyebrows
Nasion	2	N	Deepest point of nasal bridge
Pronasale	3	Prn	Most protruded point of the apex nasi
Subnasale	4	Sn	Midpoint of angle at columella base
Labial (superius)	5	Ls	Midpoint of the upper vermilion line
Labial (inferius)	6	Li	Midpoint of the lower vermilion line
Stomion	7	Sto	Midpoint of the mouth orifice
Menton	8	Me	Most inferior point on chin
Exocanthion (left and right)	9 *	Ex	Outer commissure of the eye fissure
Endocanthion (left and right)	10 *	En	Inner commissure of the eye fissure
Alar curvature (Left and Right)	11 *	Al	Most lateral point on alar contour
Cheilion (left and right)	12 *	Ch	Point located at lateral labial commissure
Zygion (left and right)	13 *	Zy	The most lateral extents of the zygomatic arches
Gonion (left and right)	14 *	Go	The inferior aspect of the mandible at its most acute point

* Bilateral landmarks.

, are identified from the acquired facial image [6]. Each landmark, denoted as L(i), is associated with an ID (i). IDs with star superscript indicate bilateral landmarks.

To mathematically construct the 3D coordinate system, the mid-sagittal plane of the face is defined as the orthogonal to the vector $\stackrel{\rightharpoonup }{E{x}_{r}E{x}_l}$ that connects right and left Exocanthi (ID = 9*; $E{x}_l$ and $E{x}_r$) and passes through Nasion (ID = 2; N). The intersection of the mid-sagittal plane and the vector $\stackrel{\rightharpoonup }{E{x}_{r}E{x}_l}$ is denoted as N’, which can be considered as the projection point of N along the mid-sagittal plane. The proposed head coordinate system is thus defined as (N, $\stackrel{\rightharpoonup }{x}$, $\stackrel{\rightharpoonup }{y}$, $\stackrel{\rightharpoonup }{z}$), where N is the origin; $\stackrel{\rightharpoonup }{x}=\stackrel{\rightharpoonup }{E{x}_{r}E{x}_l}/\Vert \stackrel{\rightharpoonup }{E{x}_{r}E{x}_l}\Vert$ (pointing to the right); $\stackrel{\rightharpoonup }{y}=\stackrel{\rightharpoonup }{N^{\prime }N}/\Vert \stackrel{\rightharpoonup }{N^{\prime }N}\Vert$ (pointing into the head); $\stackrel{\rightharpoonup }{z}=\stackrel{\rightharpoonup }{x}\times \stackrel{\rightharpoonup }{y}/\Vert \stackrel{\rightharpoonup }{x}\times \stackrel{\rightharpoonup }{y}\Vert$ (pointing upwards). The coordinates of all 20 landmarks can then be determined against (N, $\stackrel{\rightharpoonup }{x}$, $\stackrel{\rightharpoonup }{y}$, $\stackrel{\rightharpoonup }{z}$). Note that these 20 facial landmarks are later identified as 14 facial features (ID = 1, 2, 3, … 14).

2.3. Visual Questionnaire Surveys

To generate transformed faces with varied degrees of asymmetry, we perform combined counter-clockwise rolls (rotations of the Y-axis) of the nose and the chin [7]. In total, 8 nose rolls together with 8 chin rolls yield 64 stimulus faces with off-the-mid-sagittal distances covering from normal to serious asymmetry [33], as shown in

Table 2. Roll rotations of the nose and the chin.

	Nose(n)		Chin(c)
${\mathit{n}}_{\mathit{i}}$$\mathbf{or} {\mathit{c}}_{\mathit{j}}$	θ(°)	${\mathit{\delta }}_{\mathit{x}}\mathbf{\left(}\mathit{P}\mathit{r}\mathit{n}\mathbf{\right)}\mathbf{\left(}\mathbf{mm}\mathbf{\right)}$	θ(°)	${\mathit{\delta }}_{\mathit{x}}\mathbf{\left(}\mathit{M}\mathit{e}\mathbf{\right)}\mathbf{\left(}\mathbf{mm}\mathbf{\right)}$
01	0.5	0.35	0.56	1.14
03	1.5	1.06	1.68	3.42
05	2.5	1.77	2.81	5.70
07	3.5	2.48	3.93	7.97
09	4.5	3.19	5.05	10.25
11	5.5	3.90	6.17	12.52
13	6.5	4.60	7.29	14.78
15	7.5	5.31	8.42	17.04

. In our study, the off-the-mid-sagittal distance ranges are 0.35–5.31 mm for the nose and 1.14–17.04 mm for the chin. For example, a stimulus denoted as $n_{09}{c}_{03}$ indicates that the transformation involves 4.5° of nose roll and 1.68° of chin roll, and the corresponding Pronasale (Prn) and Menton (Me) displacements off the mid-sagittal plane are 3.19 mm (or ${\delta }_x\left(Prn\right)$) and 3.42 mm (or ${\delta }_x\left(Me\right)$), respectively. Figure 2a,b illustrate the facial coordinates and the counter-clockwise roll, and Figure 2c illustrates a series of deformed faces.

The 64 stimulus faces are randomly shuffled and compiled into a visual questionnaire. When conducting the questionnaires, each face is displayed for five seconds, sufficiently enough for a respondent to reach a decision [35,36]. A blank screen, lasting for two seconds, is shown between consecutive faces to remove residual stimuli. The study recruits 128 laypeople to serve as the questionnaire respondents. Respondents’ informed consents are obtained before taking the survey. A questionnaire survey would take less than 10 minutes, including time spent for opening, closing, and other miscellaneous preparations. Two simple questions for a face are asked: (Q1) Do you think this face is symmetrical? (Q2) Is it abnormally asymmetrical and you would consider a surgery? A face is rated as perceived normal (or $\mathrm{PN}$) if the answer to Q1 is YES. A face is rated as perceived asymmetrically normal (or $\mathrm{PAN}$) if the answer to Q1 is NO and the answer to Q2 is NO. A face is rated as perceived abnormal (or $\mathrm{PA}$) if the answer to Q1 is NO and the answer to Q2 is YES. All respondents’ answers are gathered, and each face is categorized as $\mathrm{PN}$, $\mathrm{PAN}$, or $\mathrm{PA}$, if it receives the most votes for that class. Furthermore, the percentage of votes each face receives over $\mathrm{PN}$, $\mathrm{PAN}$, and $\mathrm{PA}$, respectively, constitute the corresponding confidence vector ($\stackrel{\rightharpoonup }{\mathrm{C}}$). For example, a face receives 20% votes for $\mathrm{PN}$, 30% votes for $\mathrm{PAN}$, and 50% votes for $\mathrm{PA}$. This face is therefore categorized as $\mathrm{PA}$ with a confidence vector $\stackrel{\rightharpoonup }{\mathrm{C}}={\left[0.2, 0.3,0.5\right]}^T$.

2.4. Asymmetry Classifier

The asymmetry classifier is an artificial neural network using back-propagation to adjust its inter-layer weights. The model has 14 nodes on its input layer, 10 nodes on the hidden layer, and 2 target nodes. The 14 facial features fed into the input layer nodes are the 14 asymmetry indices (AI’s) computed from the 20 landmarks as defined in Equation (1). The number of hidden layer nodes being 10 is determined by cross-validation and by employing the rule of thumb: (a) The number of hidden neurons is between the size of the input layer and the size of the output layer; (b) The number of hidden neurons is approximately two-thirds of the size of the input layer, plus the size of the output layer. The two target nodes correspond to the binary coding of the three possible asymmetry classes: PN, PAN, and PA.

An asymmetry index of a feature is its distance off the mid-sagittal plane if it is a medial landmark. On the other hand, the asymmetry index for a pair of bilateral landmarks is defined as the root sum squared (RSS) of the disparities of both landmarks on x-, y-, and z-directions. Equation (1) is calculated as follows:

$${\mathrm{AI}}_i=\{\begin{array}{cc}{\delta }_x\left(\mathit{L}\left(i\right)\right)& , \mathrm{if} \mathit{L}\left(i\right)\in \left\{G,N,Prn,Sn,Ls,Li,Sto,Me\right\}\\ {\left({\left(\frac{L_x^l+L_x^r-2M_x}{2}\right)}^2+{\left(\frac{L_y^l-L_y^r}{2}\right)}^2+{\left(\frac{L_z^l-L_z^r}{2}\right)}^2\right)}^{1/2}& , \mathrm{if} \mathit{L}\left(i\right)\in \left\{Ex,En,Al,Ch,Zy,Go\right\}\end{array},$$

(1)

where $L=\mathit{L}\left(i\right)$ represents the landmark with ID being i, as shown in Table 1. ${\mathrm{AI}}_i$ denotes the asymmetry index of landmark $\mathit{L}\left(i\right)\in \left\{G,N,Prn,Sn,Ls,Li,Sto,Me\right\}{\displaystyle \cup }\left\{Ex,En,Al,Ch,Zy,Go\right\}$; ${\delta }_x\left(L\right)$ is the distance of landmark L off the mid-sagittal plane; $L_x^l$ denotes the x-coordinate of landmark L(left); $L_x^r$ is the x-coordinate of landmark L(right); $L_y^l$ denotes the y-coordinate of landmark L(left); $L_y^r$ is the y-coordinate of landmark L(right); $L_z^l$ denotes the z-coordinate of landmark L(left); $L_z^r$ is the z-coordinate of landmark L(right); $M_x$ denotes the x-coordinate of the mid-sagittal plane.

Specifically, in this study, we define mid-sagittal plane as the yz-plane, i.e., $M_x=0$; therefore, the definition of (1) can be simplified to Equation (2) as follows:

$${\mathrm{AI}}_i=\{\begin{array}{cc}{\delta }_x\left(\mathit{L}\left(i\right)\right)& , \mathrm{if} \mathit{L}\left(i\right)\in \left\{G,N,Prn,Sn,Ls,Li,Sto,Me\right\}\\ {\left({\left(\frac{L_x^l+L_x^r}{2}\right)}^2+{\left(\frac{L_y^l-L_y^r}{2}\right)}^2+{\left(\frac{L_z^l-L_z^r}{2}\right)}^2\right)}^{1/2}& , \mathrm{if} \mathit{L}\left(i\right)\in \left\{Ex,En,Al,Ch,Zy,Go\right\}\end{array}.$$

(2)

During the training process, each face’s 14 AIs are fed as the input to the under-constructed classifier, which adjusts its inter-layer weight matrices of the input and hidden and hidden and output layers to generate the desired target that is encoded from the corresponding questionnaire results. In combination, the 14 AIs of a face form an asymmetry index vector, $\stackrel{\rightharpoonup }{\mathrm{AI}}={\left[{\mathrm{AI}}_1,{ \mathrm{AI}}_2,\cdots ,{\mathrm{AI}}_{14}\right]}^T$. The $\stackrel{\rightharpoonup }{\mathrm{AI}}\prime \mathrm{s}$ of the 64 faces are iteratively applied until the classifier reaches convergence and its mean square error from the desired target is below a pre-defined threshold.

2.5. Overall Asymmetry Index (oAI) and Asymmetry Quadruple

The overall asymmetry index ($\mathrm{oAI}$) of a face is defined as the weighted sum of the asymmetry indices or as the inner product of the asymmetry index vector and the relative importance vector. The weight associated with a facial feature pertains its relative importance ($\stackrel{\rightharpoonup }{\mathrm{RI}}={\left[{\mathrm{RI}}_1,{ \mathrm{RI}}_2,\cdots ,{\mathrm{RI}}_{14}\right]}^T$) extracted from the constructed asymmetry classifier and can be computed as Equation (3) [37]:

$${\mathrm{RI}}_i=\frac{{{\displaystyle \sum }}_{j=1}^n\left(\frac{w_{ij}}{{{\displaystyle \sum }}_{i=1}^m{w}_{ij}}{w}_{jk}\right)}{{{\displaystyle \sum }}_{i=1}^m\left({{\displaystyle \sum }}_{j=1}^n\left(\frac{w_{ij}}{{{\displaystyle \sum }}_{i=1}^m{w}_{ij}}{w}_{jk}\right)\right)} \mathrm{and} {\displaystyle \sum }_{i=1}^m{\mathrm{RI}}_i=1,$$

(3)

where ${\mathrm{RI}}_i$ denotes the relative importance of landmark $\mathrm{ID}=i$, as defined in Table 1; $w_{ij}$ represents the weight of the connection between node i on the input layer and node j on the hidden layer; $w_{jk}$ denotes the weight of the connection between node k on the output layer and node j on the hidden layer; $i=1,2,\cdots , m \left(m=14\right)$; $j=1,2,\cdots , n \left(n=10\right)$; individual weights are represented as unsigned magnitude values.

The overall asymmetry index of a face is then defined as Equation (4):

$$oAI=\stackrel{\rightharpoonup }{AI} · \stackrel{\rightharpoonup }{RI} \text{or} \text{oAI}={{\displaystyle \sum }}_{i=1}^{14}\left({RI}_i\right)\left({AI}_i\right).$$

(4)

To put it all together, for a given face, overall asymmetry index, individual asymmetry indices (or asymmetry index vector), perceived asymmetry classification, and voting percentages of three categories (or associated confidence vector) constitute an asymmetry quadruple $\langle \mathrm{oAI}, \stackrel{\rightharpoonup }{\mathrm{AI}}, \mathrm{PN}\left|\mathrm{PAN}\right|\mathrm{PA}, \stackrel{\rightharpoonup }{\mathrm{C}}\rangle$ that quantitatively articulates the characteristics of the facial asymmetry.

3. Results

The study recruited 128 laypeople as the respondents and 113 (88.3%) of them replied with valid questionnaires, in which 43 (38%) are male and 70 (62%) are female. The invalid questionnaires include incomplete or apparently inconsistent answers, e.g., the respondent may consider a face as symmetrical but paradoxically call for a surgery. The questionnaire results present a Cronbach’s Alpha value of 0.944 [37]. The category of a face, $\mathrm{PN}$, $\mathrm{PAN}$, or $\mathrm{PA}$, is determined by which category receives the most votes from the respondents. Among the 64 stimulus faces, 9 faces are classified as $\mathrm{PN}$ (14%), 15 faces are classified as $\mathrm{PAN}$ (23%), and 40 faces are classified as $\mathrm{PA}$ (63%). The distribution of face categories is aligned with the process of generating stimulus faces, which are deformed from a normal face and present varied degrees of facial asymmetry.

The asymmetry classifier models the respondents’ perceptions to predict the perceived asymmetry of a face. The input layer takes the asymmetry index vector (or 14 asymmetry indices) of the corresponding 20 facial landmarks. The 14 asymmetry indices (${\mathrm{AI}}_i, i=1\dots 14$) of a stimulus face are computed using Equation (2). The classifier is trained with 75%, validated with 15%, and tested with 15% of the 3D stimulus face images and yields an overall accuracy >99% (root mean squared error 0.000059225). The resulted inter-layer weight matrix between the input and hidden layers (denoted as $I{W}^{1,1}$) is detailed in

Table 3. Weight matrix between input and hidden layers (${\left[I{W}^{1,1}\right]}_{10\times 14}$).

0.1572	−0.9305	0.1164	0.1997	−0.4651	−0.6731	0.1136	0.2616	−0.6093	−1.0658	−0.6998	−0.3851	−1.0148	−0.4178
1.0199	1.0967	−0.8636	−0.8685	0.3984	−1.3503	−1.8537	−1.8547	0.9310	0.0100	0.0306	−2.0512	1.0864	−1.0609
−1.1120	0.1507	−1.4890	−2.3591	−0.1408	0.9426	0.7155	1.1847	−0.4150	−0.4978	−2.3130	0.2910	−1.2981	0.2095
0.6825	0.0520	−0.1678	−1.4755	0.5084	−0.6806	0.1095	−0.5615	0.1468	0.4848	−0.7662	0.4254	0.4785	−0.5332
0.0814	0.2938	−1.8753	−2.6104	−0.4818	−2.1416	−1.1659	−1.1171	0.4416	0.4628	−2.5384	−2.0989	0.3153	−2.1255
−0.4233	−0.3582	0.3842	0.4808	0.1527	−1.2400	−0.5632	−1.4579	−0.1468	−0.3758	0.1567	−0.7575	0.7093	−1.7727
0.0191	0.2685	3.0053	2.9216	0.3100	0.3137	−0.4040	−1.0149	0.8525	0.5245	2.1812	−0.4173	−0.3188	0.1759
0.2811	0.9569	0.2252	0.0875	0.4633	−0.2850	−0.6547	−0.4486	0.6601	0.1391	0.7050	0.9174	0.0115	0.1564
1.1533	0.9079	0.4449	0.8827	0.7849	−0.9138	−1.5615	−0.8682	0.2965	−0.2506	0.5101	−1.6634	0.1175	−0.9619
0.1950	0.1871	0.8557	0.6553	−0.1134	0.0447	0.0944	0.4108	−0.3876	−0.7262	1.6998	0.3650	−0.0469	−0.0631

. The inter-layer weight matrix between the hidden and output layers (denoted as $L{W}^{2,1}$) is detailed in

Table 4. Weight matrix between hidden and output layers (${\left[L{W}^{2,1}\right]}_{2\times 10}$).

1.0615	0.4904	2.1091	−1.0133	−1.5437	−0.6059	2.7268	−1.1142	−3.4308	1.6700
1.1757	4.4862	−3.4968	−1.2510	−4.3407	−4.1067	−2.1687	−1.2679	−2.1432	0.6080

. The bias vectors added to the hidden and output layers are listed in

Table 5. Bias to hidden and output layers.

Hidden	−1.2863	−1.8825	1.2718	−0.9064	−0.3507	−0.4427	−0.3438	0.7215	0.8118	2.0438
Output	0.2226	2.4557

.

The relative importance RIs (${\mathrm{RI}}_i, i=1\dots 14$) are calculated using Equation (3) and are employed as the weights of facial features, respectively. As shown in

Table 6. Relative importance of the 14 facial features.

i	1	2	3	4	5	6	7	8	9	10	11	12	13	14
L(i)	G	N	Prn	Sn	Ls	Li	Sto	Me	Ex	En	Al	Ch	Zy	Go
RIi (%)	5.1	5.7	8.2	11.1	4.1	7.8	6.4	8.6	5.1	5.4	11.7	8.8	5.3	6.8
Ranking	13	9	5	2	14	6	8	4	12	10	1	3	11	7

, the RIs are rank-ordered, and it shows that alar curvature (Al) and subnasale (Sn) are relatively important when classifying perceived asymmetry: both having RIs greater than 0.10, which are significantly greater than those of the remaining facial features. It is also aligned with the proposed deformation of the normal face with the emphasis on the nose and chin regions, as suggested in [1,7,33].

The oAI of a stimulus face is a linear combination (SOP; sum of product) of the corresponding ${\mathrm{AI}}_i$ and ${\mathrm{RI}}_i$, as shown in Equation (4).

Table 7. Sixty-four stimulus faces and associated minimal asymmetry characteristics.

${\mathit{n}}_{\mathit{i}}{\mathit{c}}_{\mathit{j}}$	$(\mathbf{oAI}, \mathbf{PN}|\mathbf{PAN}|\mathbf{PA}, \widehat{\mathit{C}})$	${\mathit{n}}_{\mathit{i}}{\mathit{c}}_{\mathit{j}}$	$(\mathbf{oAI}, \mathbf{PN}|\mathbf{PAN}|\mathbf{PA}, \widehat{\mathit{C}})$
$n_{01}{c}_{01}$	(0.8182, $\mathrm{PN}$, 0.88)	$n_{09}{c}_{01}$	(2.2582, $\mathrm{PAN}$, 0.46)
$n_{01}{c}_{03}$	(1.5815, $\mathrm{PN}$, 0.74)	$n_{09}{c}_{03}$	(3.0214, $\mathrm{PAN}$, 0.45)
$n_{01}{c}_{05}$	(2.3462, $\mathrm{PAN}$, 0.53)	$n_{09}{c}_{05}$	(3.7861, $\mathrm{PAN}$, 0.52)
$n_{01}{c}_{07}$	(3.1108, $\mathrm{PAN}$, 0.51)	$n_{09}{c}_{07}$	(4.5507, $\mathrm{PA}$, 0.65)
$n_{01}{c}_{09}$	(3.8749, $\mathrm{PA}$, 0.73)	$n_{09}{c}_{09}$	(5.3148, $\mathrm{PA}$, 0.77)
$n_{01}{c}_{11}$	(4.6382, $\mathrm{PA}$, 0.86)	$n_{09}{c}_{11}$	(6.0782, $\mathrm{PA}$, 0.70)
$n_{01}{c}_{13}$	(5.4006, $\mathrm{PA}$, 0.83)	$n_{09}{c}_{13}$	(6.8405, $\mathrm{PA}$, 0.76)
$n_{01}{c}_{15}$	(6.1618, PA, 0.93)	$n_{09}{c}_{15}$	(7.6018, $\mathrm{PA}$, 0.93)
$n_{03}{c}_{01}$	(1.1772, $\mathrm{PN}$, 0.80)	$n_{11}{c}_{01}$	(2.6179, $\mathrm{PA}$, 0.35)
$n_{03}{c}_{03}$	(1.9404, $\mathrm{PN}$, 0.79)	$n_{11}{c}_{03}$	(3.3811, $\mathrm{PAN}$, 0.42)
$n_{03}{c}_{05}$	(2.7051, $\mathrm{PAN}$, 0.51)	$n_{11}{c}_{05}$	(4.1459, $\mathrm{PAN}$, 0.37)
$n_{03}{c}_{07}$	(3.4697, $\mathrm{PAN}$, 0.53)	$n_{11}{c}_{07}$	(4.9105, $\mathrm{PAN}$, 0.43)
$n_{03}{c}_{09}$	(4.2338, $\mathrm{PA}$, 0.60)	$n_{11}{c}_{09}$	(5.6746, $\mathrm{PA}$, 0.74)
$n_{03}{c}_{11}$	(4.9972, $\mathrm{PA}$, 0.73)	$n_{11}{c}_{11}$	(6.4379, $\mathrm{PA}$, 0.80)
$n_{03}{c}_{13}$	(5.7595, $\mathrm{PA}$, 0.91)	$n_{11}{c}_{13}$	(7.2003, $\mathrm{PA}$, 0.87)
$n_{03}{c}_{15}$	(6.5208, $\mathrm{PA}$, 0.92)	$n_{11}{c}_{15}$	(7.9615, $\mathrm{PA}$, 0.88)
$n_{05}{c}_{01}$	(1.5375, $\mathrm{PN}$, 0.66)	$n_{13}{c}_{01}$	(2.9771, $\mathrm{PN}$, 0.42)
$n_{05}{c}_{03}$	(2.3007, $\mathrm{PN}$, 0.46)	$n_{13}{c}_{03}$	(3.7403, $\mathrm{PAN}$, 041)
$n_{05}{c}_{05}$	(3.0654, $\mathrm{PAN}$, 0.50)	$n_{13}{c}_{05}$	(4.5050, $\mathrm{PA}$, 0.48)
$n_{05}{c}_{07}$	(3.8300, $\mathrm{PAN}$, 0.53)	$n_{13}{c}_{07}$	(5.2697, $\mathrm{PA}$, 0.67)
$n_{05}{c}_{09}$	(4.5941, $\mathrm{PA}$, 0.53)	$n_{13}{c}_{09}$	(6.0338, $\mathrm{PA}$, 0.61)
$n_{05}{c}_{11}$	(5.3575, $\mathrm{PA}$, 0.87)	$n_{13}{c}_{11}$	(6.7971, $\mathrm{PA}$, 0.69)
$n_{05}{c}_{13}$	(6.1199, $\mathrm{PA}$, 0.91)	$n_{13}{c}_{13}$	(7.5595, $\mathrm{PA}$, 0.90)
$n_{05}{c}_{15}$	(6.8811, $\mathrm{PA}$, 0.91)	$n_{13}{c}_{15}$	(8.3207, $\mathrm{PA}$, 0.90)
$n_{07}{c}_{01}$	(1.8979, $\mathrm{PN}$, 0.54)	$n_{15}{c}_{01}$	(3.3356, $\mathrm{PA}$, 0.61)
$n_{07}{c}_{03}$	(2.6612, $\mathrm{PN}$, 0.56)	$n_{15}{c}_{03}$	(4.0988, $\mathrm{PA}$, 0.58)
$n_{07}{c}_{05}$	(3.4259, $\mathrm{PAN}$, 0.43)	$n_{15}{c}_{05}$	(4.8635, $\mathrm{PA}$, 0.58)
$n_{07}{c}_{07}$	(4.1905, $\mathrm{PAN}$, 0.51)	$n_{15}{c}_{07}$	(5.6281, $\mathrm{PA}$, 0.58)
$n_{07}{c}_{09}$	(4.9546, $\mathrm{PA}$, 0.52)	$n_{15}{c}_{09}$	(6.3922, $\mathrm{PA}$, 0.81)
$n_{07}{c}_{11}$	(5.7179, $\mathrm{PA}$, 0.82)	$n_{15}{c}_{11}$	(7.1556, $\mathrm{PA}$, 0.78)
$n_{07}{c}_{13}$	(6.4803, $\mathrm{PA}$, 0.86)	$n_{15}{c}_{13}$	(7.9180, $\mathrm{PA}$, 0.86)
$n_{07}{c}_{15}$	(7.2415, $\mathrm{PA}$, 0.89)	$n_{15}{c}_{15}$	(8.6792, $\mathrm{PA}$, 0.91)

presents the overall asymmetry index ($\mathrm{oAI}$), classification of perceived asymmetry ($\mathrm{PN}\left|\mathrm{PAN}\right|\mathrm{PA}$), and the confidence value of the classification ($\widehat{\mathrm{C}}$: percentage of the voted classification) for the 64 stimulus faces. Please note that for brevity of demonstration, only partial asymmetry quadruple $\langle \mathrm{oAI}, \stackrel{\rightharpoonup }{\mathrm{AI}}, \mathrm{PN}\left|\mathrm{PAN}\right|\mathrm{PA}, \stackrel{\rightharpoonup }{\mathrm{C}}\rangle$ is shown: the detailed asymmetry index vector ($\stackrel{\rightharpoonup }{\mathrm{AI}}$) and detailed confidence vector ($\stackrel{\rightharpoonup }{\mathrm{C}}$) are excluded. For example, $n_{03}{n}_{15}$ is a stimulus face with a ${1.5}^{°}$ nose roll and a ${8.42}^{°}$ chin roll (as depicted in Table 2) from the normal face. Its entry in Table 7 is shown as (6.5208,$\mathrm{PA}$,0.92), i.e., its $\mathrm{oAI}$ is 6.5208 and it is perceived as abnormal ($\mathrm{PA}$) with a confidence vector of 0.92.

The range of oAI is from 0.8182 ($n_{01}{c}_{01}$) to 8.6792 ($n_{15}{c}_{15}$). The greater the $\mathrm{oAI}$ of a stimulus face, the greater the possibility of it being classified as $\mathrm{PAN}$ or $\mathrm{PA}$.

4. Discussion

We transformed a normal face rather than using a series of realistic asymmetric faces to maintain perception stability of the questionnaire respondents, similar to the approach proposed by Meyer-Marcotty et al. [1]. The degree of deformation can therefore be computed to correspond to the respondents’ ratings over different faces. The facial texture is removed before deformation in our study to mitigate perception distractions by skin color, rashes, moles, etc.

Table 8. Classifications of respondents’ perceived asymmetry (pre-revision).

${\mathit{n}}_{\mathit{i}}{\mathit{c}}_{\mathit{j}}$	$\mathit{j}$= 01	$\mathit{j}$ = 03	$\mathit{j}$ = 05	$\mathit{j}$ = 07	$\mathit{j}$ = 09	$\mathit{j}$ = 11	$\mathit{j}$ = 13	$\mathit{j}$ = 15
$i$ = 01	PN	PN	PAN	PAN	PA	PA	PA	PA
$i$= 03	PN	PN	PAN	PAN	PA	PA	PA	PA
$i$ = 05	PN	PN	PAN	PAN	PA	PA	PA	PA
$i$= 07	PN	PN	PAN	PAN	PA	PA	PA	PA
$i$= 09	PAN	PAN	PAN	PA	PA	PA	PA	PA
$i$= 11	PA	PAN	PAN	PAN	PA	PA	PA	PA
$i$= 13	PN	PAN	PA	PA	PA	PA	PA	PA
$i$ = 15	PA	PA	PA	PA	PA	PA	PA	PA

illustrates the classification results of the facial asymmetry for the 64 stimulus faces. The 8 × 8 matrix is arranged so that the left side shows varied degrees of nose rolls, whereas different degrees of chin rolls are shown on the top. As shown in Table 2, each increment of i denotes 0.5° of nasal roll, whereas each increment of j denotes 0.56° of chin roll. Presumably, the stimulus faces towards the upper left are likely to be categorized as $\mathrm{PN}$, where those towards the lower right are likely to be classified as $\mathrm{PAN}$ or $\mathrm{PA}$. The perceived asymmetry is, in general, aligned with this assumption, with the exceptions of the four red-shaded entries ($n_{09}{c}_{01}$, $n_{11}{c}_{01}$, $n_{13}{c}_{01}$, and $n_{11}{c}_{07}$), where there exhibit inconsistencies in the asymmetry classifications.

When exploring the confidence vector ($\stackrel{\rightharpoonup }{\mathrm{C}}$) of the four seemingly erroneous classifications, as depicted in

Table 9. Confidence vectors of the four likely misclassified stimulus faces.

$\stackrel{\rightharpoonup }{\mathrm{C}}$	PN	PAN	PA
$n_{09}{c}_{01}$	0.35	0.46 *	0.19
$n_{11}{c}_{01}$	0.33	0.32	0.35 *
$n_{13}{c}_{01}$	0.42 *	0.21	0.36
$n_{11}{c}_{07}$	0.15	0.43 *	0.42

Stars (*) denote the highest confidence; underlines (_) denote the second-highest confidence for a face.

, we find “ambiguous zones” lying around the PN–PAN and PAN–PA borders where there are competing values in corresponding confidence vectors. In Table 9, the highest confidence value in a confidence vector is starred (*); and the second highest confidence value is underlined. For example, for the stimulus face $n_{11}{c}_{07}$ that is associated with confidence vector ${\left[0.15, 0.43, 0.42\right]}^T$, its highest confidence value is starred (0.43*) and the second highest confidence value is underlined (0.42). The four stimulus faces demonstrate too-close-to-call situations. This means that respondents’ perceptions are divergent, and any classification decision may easily draw disagreement from the other camp. By reassigning the classifications of the four faces to the categories with the second highest votes, we obtain a revised matrix, as shown in

Table 10. Classifications of respondents’ perceived asymmetry (post-revision).

${\mathit{n}}_{\mathit{i}}{\mathit{c}}_{\mathit{j}}$	$\mathit{j}$ = 01	$\mathit{j}$ = 03	$\mathit{j}$ = 05	$\mathit{j}$ = 07	$\mathit{j}$ = 09	$\mathit{j}$ = 11	$\mathit{j}$ = 13	$\mathit{j}$ = 15
$i$= 01	PN	PN	PAN	PAN	PA	PA	PA	PA
$i$= 03	PN	PN	PAN	PAN	PA	PA	PA	PA
$i$= 05	PN	PN	PAN	PAN	PA	PA	PA	PA
$i$= 07	PN	PN	PAN	PAN	PA	PA	PA	PA
$i$= 09	PN	PAN	PAN	PA	PA	PA	PA	PA
$i$= 11	PN	PAN	PAN	PA	PA	PA	PA	PA
$i$= 13	PA	PAN	PA	PA	PA	PA	PA	PA
$i$ = 15	PA	PA	PA	PA	PA	PA	PA	PA

, in which the red alerts are gone. By meticulously re-classifying these faces by examining the competing confidence values for the other stimulus faces, even though some of them are not red alerted here, we obtain similar results.

The ambiguous zones phenomenon of asymmetry perception suggests: (1) that misclassifications can happen and there are no univocally correct answers; (2) most importantly, that objective and transparent articulations of the asymmetry characteristics between the physician and the patient during the course of assessment, decision making, surgical planning, and post-op are needed to avoid or mitigate potential medical disputes.

Figure 3 visualizes the relationship between $\mathrm{oAI}$ and asymmetry classification. The stimulus faces are arranged along the horizontal axis so that stimulus faces of the same classified category are clustered together: $\mathrm{PN}$ on the left ($\mathrm{oAI}$: 1.88 ± 0.69), $\mathrm{PAN}$ in the middle ($\mathrm{oAI}$: 3.43 ± 0.71), $\mathrm{PA}$ on the right ($\mathrm{oAI}$: 5.89 ± 1.39). Within a category, the faces are sorted in order of $\mathrm{oAI}$. Each graphic bar is associated with the confidence vector of a stimulus face. It illustrates the voting distribution (referring to the vertical percentage axis on the left) from the questionnaire surveys. Green segments relate to votes for $\mathrm{PN}$, yellow segments to $\mathrm{PAN}$, and red segments to $\mathrm{PA}$. The solid black curve across the plot depicts the $\mathrm{oAI}$s (referring to the $\mathrm{oAI}$ vertical axis on the right) of the corresponding stimulus faces.

The ambiguous zone phenomenon is again manifested in Figure 3. There are $\mathrm{oAI}$ drops around the inter-category areas ($\mathrm{PN}–\mathrm{PAN}$ and $\mathrm{PAN}–\mathrm{PA}$)—coinciding with where red alerts occur in Table 8. For example, the first $\mathrm{oAI}$ drop happens at $n_{09}{c}_{01}$, which is one of the four stimulus faces exhibiting red alerts in Table 8. It has an $\mathrm{oAI}$ of 2.258 but is perceptually classified as $\mathrm{PAN}$ instead of $\mathrm{PN}$. Its votes for $\mathrm{PN}$ and $\mathrm{PAN}$ are close (0.35 versus 0.46), and the corresponding green and yellow segments are visually of similar lengths. Such inconsistent asymmetry classifications exist when respondents’ subjective opinions are divisive.

Limitations and Strengths. Although quantifying facial asymmetry can be used as a helpful ancillary utensil, it is not the sole decision maker for surgical decisions. Comprehensive clinician grading, laypeople evaluations, patient-reported outcomes, and healthy patient–physician communications are just as important. The study models facial asymmetry for 3D images; however, 2D facial images are still widely utilized by facial plastic and reconstructive surgeons. The proposed model can be adapted for 2D images by (1) establishing 2D facial coordinate systems; (2) identifying the mid-face line instead of mid-sagittal plane; (3) computing the point-to-line distance instead of the point-to-plane distance; and (4) removing the Z-axis components in Equations (1) and (2). Furthermore, when conducting the questionnaire survey, the respondents rate asymmetry on faces with skin texture removed. Facial asymmetry among different races or different genders is not studied. Pre- and post-surgical asymmetry comparisons and how prominent facial features, such as skin colors, moles, mustaches, attractiveness, pimples, etc., affect the ratings of facial asymmetry are beyond the scope of this study.

5. Conclusions

In this study, we construct an artificial neural network model to address the perception of facial asymmetry. The resulted asymmetry quadruple $\langle \mathrm{oAI}, \stackrel{\rightharpoonup }{\mathrm{AI}}, \mathrm{PN}\left|\mathrm{PAN}\right|\mathrm{PA}, \stackrel{\rightharpoonup }{\mathrm{C}}\rangle$ can serve as a tool to establish more transparent communications between the physician and the patient and alleviate the anticipation gaps between the two parties. The $\mathrm{oAI}$ is an overall score of facial asymmetry and is a weighted sum of asymmetry indices of individual facial features. Before making surgical decision, the patient can weigh on their own asymmetry characteristics in terms of the asymmetry quadruple $\langle \mathrm{oAI}, \stackrel{\rightharpoonup }{\mathrm{AI}}, \mathrm{PN}\left|\mathrm{PAN}\right|\mathrm{PA}, \stackrel{\rightharpoonup }{\mathrm{C}}\rangle$. The ambiguous zone phenomenon should be taken into account as well. With such practice, the patients are more involved in the process and analysis of surgical decision making and undertake more knowledgeable risks. On the other hand, it is the physician’s responsibility to properly address the patient’s asymmetry characteristics and perform an $\mathrm{oAI}$-improving (or lower post-op oAI) surgery. Quantifying facial asymmetry can serve as an advisory tool during the surgical decision process, alongside comprehensive clinician grading, laypeople evaluations, and patient-reported outcomes.

Finally, the visualization of the ambiguous zones of asymmetry perception as depicted in Table 8 and Figure 3 helps explain why certain medical disputes are difficult to avoid. A thorough articulation of the proposed asymmetry quadruples $\langle \mathrm{oAI}, \stackrel{\rightharpoonup }{\mathrm{AI}}, \mathrm{PN}\left|\mathrm{PAN}\right|\mathrm{PA}, \stackrel{\rightharpoonup }{\mathrm{C}}\rangle$ is expected to improve physician–patient relationship.

6. Patents

The concept and preliminary study of this work have been awarded an invention patent (no. I595430) by the Intellectual Property Office, Ministry of Economic Affairs, Taiwan, ROC.